Took me some time but finally I managed to workaround all issues and got X11 running on real AArch64 hardware.
Two years ago I started X11 on AArch64 using emulator. But planned to make it on real hardware one day. And that day came today.
What took so long? Several things. First I lacked hardware - but APM Mustang arrived one day. Then lack of PCI-Express support but it got solved.
So I started collecting graphic cards. Finally ended with Radeon HD5450 and Geforce GTS250 - both with 512MB ram. After last firmware update (to 0.14) they even got whole memory assigned. But none of them worked ;(
After few discussions it got finally confirmed that something is going on with supporting more than 64MB of memory on PCI cards. I prefer not to go to details. Anyway I digged and found Matrox G550 card in local computer scrapyard. Seller wrote that it has 32MB ram (Linux says that only 16MB is present).
Bought card, inserted into pinkiepie (Mustang) and after kernel rebuild I got nice 1920x1080x32-60 framebuffer and X11 over it. Maybe it is not the fastest but it works.
Tomorrow will reconfigure kernel to get USB working (and submit patch for Fedora config) which will give me keyboard, mouse and audio.
Next step? XFCE, GNOME, KDE testing of course ;D And building MPlayer so I will be able to watch movies too.