Back in 2012 AArch64 was something new, unknown yet. There was no toolchain
support (so no gcc, binutils or glibc). And I got assigned to get some stuff
running around it.
As there was no hardware cross compilation was the only way. Which meant
OpenEmbedded as we wanted to have wide selection of software available.
I learnt how to use modern OE (with OE Core and layers) by building images for
ARMv7 and checking them on some boards I had floating around my desk.
Non-public toolchain work
Some time later first non-public patches for binutils and gcc arrived in my
inbox. Then eglibc ones. So I started building and on 12th September 2012 I was
able to build helloworld:
12:38 hrw@puchatek:aarch64-oe-linux$ ./aarch64-oe-linux-gcc ~/devel/sources/hello.c -o hello
12:38 hrw@puchatek:aarch64-oe-linux$ file hello
hello: ELF 64-bit LSB executable, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.39, not stripped
12:39 hrw@puchatek:aarch64-oe-linux$ objdump -f hello
hello: file format elf64-littleaarch64
architecture: aarch64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x00000000004003e0
Then images followed. Several people at Linaro (and outside) used those images
to test misc things.
At that moment we ran ARMv8 Fast models (quite slow system emulator from Arm).
There was a joke that Arm developers formed a queue for single core 10 GHz x86-64
cpus to get AArch64 running faster.
Toolchain became public
Then 1st October 2012 came. I entered Linaro office in Cambridge for AArch64
meeting and was greeted with “glibc patches went to public ML” information. So I
rebased my OpenEmbedded repository, updated patches, removed any traces of
non-public ones and published whole work.
Building on AArch64
My work above added support for AArch64 as a target architecture. But can it be
used as a host? One day I decided to check and ran OpenEmbedded on
After one small patch it worked fine.
As I had access to Arm Fast model I was able to play with graphics. So one day
in January 2013 I did a build and and started Xorg.
Through next years I had fun when people wrote that they got X11 running on
their AArch64 devices ;D
Two years later I had Applied Micro Mustang at home (still have it). Once it had
working PCI Express support I added graphics card and started X11 on hardware.
Then went debugging why Xorg requires configuration file and one day with help
from Dave Airlie, Mark Salter and Matthew Garrett I got two solutions for the
problem. Do not
remember did any of them went upstream but some time later problem was solved.
Few years later I met Dave Airlie at Linux Plumbers. We introduced to each other
and he said “ah, you are the ‘arm64 + radeon guy’” ;D
AArch64 Desktop week
One day in September 2015 I had an idea. PCIe worked, USB too. So I did AArch64
desktop week. Connected monitors, keyboard, mouse, speakers and used Mustang
instead of my x86-64 desktop.
It was fun.
First we had nothing. Then I added AArch64 target into OpenEmbedded.
Same month Arm released Foundation model so
anyone was able to play with AArch64 system. No screen, just storage, serial and
network but it was enough for some to even start building whole distributions
like Debian, Fedora, OpenSUSE, Ubuntu.
At that moment several patches were shared by all distributions as it was faster
way than waiting for upstreams. I saw multiple versions of some of them during
my journey of fixing packages in some distributions.
Debian and Ubuntu
In February 2013 Debian/Ubuntu team presented their AArch64 port.
It was their first architecture bootstrapped without using external toolchains.
Work was done in Ubuntu due to different approach to development than Debian
has. All work was merged back so some time later Debian also had AArch64 port.
Fedora team started early — October 2012, right after toolchain became public.
Used Fedora 17 packages and switched to Fedora 19 during work.
When I joined Red Hat in September 2013 one of my duties was fixing packages in
Fedora to get them built on AArch64.
In January 2014 first versions of QEMU support arrived and people moved from
using Foundation model. March/April OpenSUSE team did massive amount of builds
to get their distribution built that way.
Fedora bootstrap also meant RHEL 7 bootstrap. When I joined Red Hat there were
images ready to use in models. My work was testing them and fixing packages.
There were multiple times when AArch64 fix helped to build also on ppc64le and
Hardware I played with
First Linux capable hardware was announced in June 2013. I got access to it at
Red Hat. Building and debugging was much faster than using fast models ;D
Applied Micro Mustang
Soon Applied Micro Mustangs were everywhere. Distributions used them to build
packages etc. Even without support for half of hardware (no PCI Express, no
I got one in June 2014. Running UEFI firmware out of the box. At first months I
had a feeling that firmware is developed at Red Hat as we had fresh versions
often right after first patches for missing hardware functionality were written.
In reality it was maintained by Applied Micro and we had access to sources so
there were some internal changes in testing (that’s why I had firmware versions
All those graphics cards I collected to test how PCI Express works. Or testing
USB before it was even merged into Linux mainline kernel. Using virtualization
for development of armhf build fixes (8 cores, 12 gigabytes of ram and plenty of
storage beat all armv7 hardware I had).
I stopped using Mustang around 2018. It is still under my desk.
For those who use: make sure you have 3.06.25 firmware.
In February 2015 Linaro announced 96boards initiative. The plan was to make
small, unified SBC with different Arm chips. Both 32- and 64-bit ones.
First ones were ‘Consumer Edition’. Small, limited to basic connectivity. Now
there are tens of them. 32-bit, 64-bit, fpga etc. Choose your poison ;D
Second ones were ‘Enterprise Edition’. Few attempts existed, most of them did
not survived prototype phase. There was joke that full length PCI Express slot
and two USB ports requirements are there because I wanted to have AArch64
Too bad that nothing worth using came from EE spec.
As Linaro assignee I have access to several servers from Linaro members. Some
are mass-market ones, some never made to market. We had over hundred X-Gene1
based systems (mostly as m400 cartridges in HPe Moonshot chassis’) and shutdown
them in 2018 as they were getting more and more obsolete.
Main system I use for development is one of those ‘never went to mass-market’
ones. 46 cpu cores, 96 GB of ram make it nice machine for building container
images, Debian packages or running virtual machines in OpenStack.
For some time I was waiting for some desktop class hardware to have development
box more up-to-date than Mustang. Months turned into years. I no longer wait as
it looks like there will be no such thing.
Solidrun has made some attempts in this area. First with Macchiatobin and later
I did not used any of them.
When I (re)joined Linaro in 2016 I became part of team working on getting
OpenStack working on AArch64 hardware. We used Liberty, Mitaka, Newton releases
and then changed way we work and started contributing more. And more. Kolla,
Nova, Dib and other projects. Added aarch64 nodes to OpenDev CI.
The effect of it was Linaro Developer Cloud used by
hundreds of projects to speed-up their aarch64 porting, tens of projects hosting
their CI systems etc.
Two years later Amazon started offering aarch64 nodes in AWS.
I spent half of my life with Arm on AArch64. Had great moments like building
helloworld as one of first people outside of Arm Ltd. Got involved in far more
projects then ever thought. Met new friends, visited several places in the
world I would probably never go otherwise.
I also got grumpy and complained far too many times that AArch64 market is
‘cheap but limited sbc or fast but expensive servers and nearly nothing in
between’. Wrote some posts about missing systems targeting software developers
and lost hope that such will happen.
NOTE: It is 8 years of my work on AArch64. I work with Arm since 2004.