1. I/O plate for APM Mustang

    Applied Micro Mustang uses standard Mini-ITX form factor just like many PC mainboards. The problem is that contrary to PC ones it does not come with I/O plate. So I decided to make one using 3d printer.

    First version

    Loaded Tinkercad page, did quick measurements and sent STL to my brother-in-law Szymon for printing. That’s how v1 was born.

    It was not good:


    As you may see several ports were misaligned (or too small). So I moved ports a bit. The other problem was thickness — turned out that 1mm of plastic was too weak. Second version got printed as 2mm thick:


    Other filament made it look ugly. And shows that my measurements were wrong.


    My brother-in-law uses FreeCAD for his designs so I decided to recreate model of I/O plate in this tool. There was some cursing involved as their approach to doing holes in objects. On the other side positioning of holes was much easier to do.

    There were several changes and versions were moving fast. I decided to not put any 3d text on model for several reasons:

    • it looked shitty on dark filament
    • low resolution made it look even worse
    • FreeCAD way of doing 3d text feels overcomplicated

    Fifth version was 1mm shorter as I had to add space for screws in case:


    At this moment I left APM Mustang for my brother-in-law to make changes easier.

    In meantime we got some hints from people more involved in 3d printing and decided to do some other changes to how model is done.

    Mesh design

    FreeCAD has this idea of ‘mesh design’ - you select object, create mesh of it and then export it as STL. The problem is that this part is completely broken under Ubuntu (used by Szymon) — it creates mesh elements but with 0 points/faces.

    Seventh version

    6th version existed for a moment and then 7th came as first one using mesh:


    Not that it changed much ;D

    Final one

    Few more iterations, another set of measurements and finally we got version which went into use:


    It is still far from being perfect but does it’s job. As 3d printing n00b I have no idea why there are vents around holes for ports. Maybe something wrong in printer setup or slicer configuration. Suggestions are welcome.

    One button is for power, other for reset as case had just one button on front panel (which I do not use).

    Files to download (MIT licensed):

    Written by Marcin Juszkiewicz on
  2. Switched to BorgBackup

    The old joke says that there are two types of people:

    • those who make backups
    • those will will make backups

    I was in second group long time ago and then moved to first one.


    So far I used Duplicity along with it’s fronted called Duply. It was installed by default on Ubuntu systems and was quite easy to setup.

    But getting files restored from it was pain. And consistency was a problem. Especially when I wanted to restore one directory from quite old copy — turned out that one file (of many) was damaged so whole backup was useless…


    So I looked for alternatives and BorgBackup was one of suggestions.

    Features list was long, FOSS, several platforms, compression etc. There were two things which brought my attention:

    • deduplication
    • mountable backups with FUSE


    So why deduplication? Because it allows me to backup several machines into one place and separate copies of git repositories, source code will not take extra space:

                           Original size      Compressed size    Deduplicated size
    All archives:                2.02 TB              1.57 TB            118.85 GB
                           Unique chunks         Total chunks
    Chunk index:                 1060634             18670536

    Or on my server where backups are run hourly:

                           Original size      Compressed size    Deduplicated size
    All archives:                7.29 TB              5.36 TB             11.35 GB
                           Unique chunks         Total chunks
    Chunk index:                  359783            260310747

    Nice difference compared to Duplicity I used before.

    FUSE mounting of backups

    Instead of checking which options I have to use to restore that one directory from 2 months old backup I can now mount each backup using FUSE:

    borgmatic mount --archive puchatek-2021-01-01 --mount-point /tmp/del/1

    And then just copy whatever file(s) I want to restore. Very handy way.


    What Duply was to Duplicity, Borgmatic is to BorgBackup. Simple, easy to use frontend hiding most of internals behind easy to use command line interface.

    Configure once, then simple commands like “borgmatic info”, “borgmatic prune” etc. All needed things are stored in config file.

    Will it work for me?

    Time will show does it work for me. Machines’ backup copies are done, syncing them between machines need some improvements.

    Written by Marcin Juszkiewicz on
  3. System calls by kernel version

    As you know I have a page with list of Linux system calls which I usually update every rc1 release.

    Recently Pavel Šnajdr asked me on Twitter:

    Could you also track the first kernel version supporting each syscall? That’d make the table a lot more useful for assembler/calling_it_raw junkies :)

    I decided to give it a try.

    Decided to check every -rc1 release as this is point in history when most of new code is already merged. First kernel version with rc1 tag was v3.0-rc1 one. Small shell script later I got tables for each rc1 release between 3.0-rc1 and 5.11-rc1 and started digging.

    There were at least 80 system calls added in this time range (my method of checking was far from being perfect).

    Most active kernel versions

    Most active was 5.1 kernel with 24 new ones — mostly due to y2038 problem (21 time64 related calls added).

    Then 5.2 kernel added 6 new ones in filesystem handling.

    3.5 kernel had some Alpha related activity with 5 osf_* calls added of *stat* family. Also ‘kcmp’ appeared then.

    There were few kernel releases with 3 new system calls:

    • 3.17 added getrandom(), memfd_create() and seccomp()
    • 4.9 gave us pkey_alloc(), pkey_free() and pkey_mprotect()
    • 5.3 added clone3(), fp_udfiex_crtl() and pidfd_open()

    Two new calls were present in:

    kernel release system calls
    3.2 process_vm_readv() and process_vm_writev()
    3.3 cache_sync() and mq_getsetaddr()
    3.8 finit_module() and kern_features()
    3.9 arc_gettls() and arc_settls()
    3.14 sched_getattr() and sched_setattr()
    4.3 membarrier() and userfaultfd()
    4.6 preadv2() and pwritev2()
    4.18 io_pgetevents() and rseq()
    5.6 openat2() and pidfd_getfd()

    Sixteen kernel releases brought one system call each.

    Architecture specific calls

    Other arch specific entries:

    architecture system calls
    Alpha osf_*
    ARC arc_gettls() and arc_settls()
    Arm arm_fadvise64_64()
    OpenRISC or1k_atomic()
    S/390 s390_runtime_instr() and s390_sthyi()

    Those are usually hardware specific. Some get their name changed — for example Arm had arm_sync_file_range() which was later renamed to sync_file_range2() and used on several of other architectures.

    I do wonder why Alpha still has all those osf_* calls. Do people still use OSF/1 emulation there?


    Most of kernel releases bring some new system call(s). Usually it takes 2-3 releases for most architectures to keep up. Looks like there are just 153 (of 595) calls supported on all archs (but this is affected by fact that I keep information also for architectures removed from Linux kernel).

    Some system calls get dropped or renamed. I drop those from my table as well.

    My work on this table will continue. It does not take much of my time and provides useful information to several FOSS projects.

    Written by Marcin Juszkiewicz on
  4. So long, and thanks for all the fun

    During last days I tried to get my Applied Micro Mustang running again. And it looks like it is no more. Like that Norwegian Blue parrot.

    Tried some things

    By default Mustang outputs information on serial console. It does not here. Checked serial cables, serial to usb dongles. Nothing.

    Tried to load firmware from SD card instead of on-board flash. Nope.

    Time to put it to rest.

    How it looked

    When I got it in June 2014 it came in 1U server case. With several loud fans. Including one on cpu radiator. So I took the board out and put into PC Tower case. Also replaced 50mm processor fan with 80mm one:

    Top view of Mustang
    Top view of Mustang
    Side view
    Side view

    All that development…

    I did several things on it:

    Some of them were done for first time on AArch64.

    Board gave me lot of fun. I built countless software packages on it. For CentOS, Debian, Fedora, RHEL. Tested installers of each of them.

    Was running OpenStack on it since ‘liberty’ (especially after moving from 16GB to 32GB ram).

    What next?

    I am going to frame it. With few other devices which helped me during my career.


    It would be nice to replace Mustang with some newer AArch64 hardware. From what is available on mass market SolidRun HoneyComb looks closest. But I will wait for something with Armv8.4 cores to be able to play with nested virtualization.

    UPDATE: I got it working again.

    Written by Marcin Juszkiewicz on
  5. Standards in Arm space (part II)

    In first part I went from board files and ugly bootloaders to SBSA/SBBR and EBBR. Now let me try to explain how it evolve.

    BSA, BBR?

    During Arm DevSummit 2020 there was announcement of new standards for Arm devices:

    Arm is extending the system architecture standards compliance from servers to other segments of the market, edge and IoT. We introduce the new BSA specification with market segment-specific supplements and provide the operating system-oriented boot requirements recipes in the new BBR specification.

    BSA is meant to describe basic recommendations and requirements for hardware. Just like SBSA did it for servers before, BBR covers booting.


    Base System Architecture specifies hardware that software can rely on. Compliance is not required:

    Arm does not mandate compliance to this specification. However, Arm anticipates that OEMs, ODMs, cloud service providers and software providers will require compliance to maximize Out of Box software compatibility and reliability.

    According to BSA 1.0 (DEN0094A document) there are two supplements:

    • Server Base System Architecture (SBSA)
    • Client Base System Architecture (CBSA)

    Former is described in separate document (DEN0029E) covering AArch64 server requirements (look for spec v6.1+) while latter document (DEN0087) is not yet present on Arm developer website (I was told that you need to contact Arm for a copy).

    SBSA changes

    There are some interesting changes in specification. For example there is a table with hardware requirements for each SBSA level:

    Level A profile SMMU GIC
    3 v8.0 v2 or v3 v3.0
    4 v8.3 v3.0 v3.0
    5 v8.4 v3.2 v3.0
    6 v8.5 v3.2 v3.0

    Previous (v6) version of SBSA specification mentioned that all PEs (cpu cores) must implement XYZ introduced in Armv8.y version (I selected just a few):

    Level required features/extensions optional ones
    4 RAS (v8.2), 16-bit VMID, VHE pointer signing
    5 enhanced nested virt (v8.4), CS-BSA cryptography
    6 Armv8.5-PMU, restrictions on speculation Memory Tagging Extension

    So level 4 is now v8.3+ from v8.2+ before.

    Most of hardware requirements descriptions moved from SBSA to BSA. Due to this SBSA v6.1 spec is just 25 pages while SBSA v6.0 had 83 of them.

    BSA and SBSA checklists

    Both BSA and SBSA have now section with checklist. This allows to quickly check which components are required for ‘minimum BSA’ and each SBSA level.

    Funny part is when you compare ‘minimum BSA’ with SBSA level 3 — latter one lists operating system requirements up to B_PE_14 while former goes to B_PE_17. At beginning it feels like mistake but B_PE_15 to B_PE_17 describe optional parts (_15 is part of Armv8.3 so SBSA level 4+ and _16 and _17 are required for SBSA level 6).

    And the reason for above is backward compatibility. SBSA levels are defined as “previous one + some extras” so they can not be rebased on top of BSA. I wonder how it looks on CBSA.


    Base Boot Requirements specifies firmware requirements to make booting easy and predictable.

    BBR specification is highly based on SBBR one. It is visible in document number (DEN0044) as A-E versions were SBBR, from F it is BBR.

    According to BBR 1.0 (DEN0044F document) there are four recipes:

    • SBBR for servers
    • ESBBR which is SBBR with some potential exceptions (none are defined so far)
    • EBBR for those who can not into SBBR
    • LBBR for LinuxBoot based systems


    So what for ESBBR was invented when it is the same as SBBR? Probably those so called “Edge” devices — nearly server class hardware but something went wrong or vendor was lazy to go for full SBBR.


    LBBR stands for LinuxBoot BBR — systems where machine has very minimal firmware just to run Linux which initialize everything and then use kexec system calls to load final kernel image. Used by some datacenters that are compliant with the Open Compute Project (OCP).

    In theory LBBR system can load UEFI instead of kernel and be SBBR compliant.

    Required components

    Each recipe has own list of required components:

    PSCI/SMCCC yes yes yes yes
    Secondary Core Boot yes yes no no
    UEFI yes yes yes no
    ACPI yes yes optional (*) yes
    DeviceTree forbidden forbidden optional (*) no
    SMBIOS yes yes no yes

    *) EBBR system must provide ACPI or DeviceTree — both at same time are not allowed.

    SBBR changes

    SBBR 6.0 specification required SBSA hardware:

    This document defines the Boot and Runtime Services expected by an enterprise platform Operating System or hypervisor, for an SBSA-compliant Arm AArch64 server which follows the UEFI and ACPI specifications.

    In BBR 1.0 spec this requirement got wiped out:

    Systems using SBBR recipe must meet the requirements that are specified in section 5 (PSCI/SMCCC), section 6 (Secondary Core Boot), section 7 (UEFI), section 8 (ACPI), and section 9 (SMBIOS).

    SBBR-compliant systems must not present a DeviceTree binary to the operating system.

    Now any device can be made ‘SBBR compliant’. SBSA requirement got moved to SystemReady specification but this is material for other post.

    Secure and Trusted Boot

    SBBR v1.2 spec (DEN0044E document) had “Secure and Trusted Boot” subsection in UEFI section. It was removed from BBR 1.0 version.

    The reason is simple — it has own specification now: “Base Boot Security Requirements (BBSR)” (DEN0107 document) with more details in it. There will be separate test suite and certificate program for it.


    BSA/BBR feel a bit like cleaning process. Non-server hardware was not defined before so now BSA kind of does that. Several extensions from v8.5 are required to cover all those mitigation issues. Too bad that CBSA was not released at same time as BSA

    Embedded devices did not got updated specification yet. EBBR 1.0.1 is from August 2020 and does not even mention BBR. I would see it as part of BBR specification but was told that it is handled by other team so have to stay separate.

    Servers are like they were before — SBSA + SBBR cover like before. Unless you want Secure Boot as this is no longer defined. And for some server-like machines there is ESBBR allowing to make some exceptions (once they got defined).

    Several datacenter servers got own part with LBBR. Normal users would not even play with them so nothing to worry for them.

    Written by Marcin Juszkiewicz on
  6. Standards in Arm space (part I)

    One of things which made AArch64 servers so successful was agreeing on set of standards and keeping them implemented. But that was not always a case…

    Wild, Wild West

    I started working with Arm architecture in 2004. This was time when nearly every device required own kernel… You had those ‘board files’ inside of arch/arm directory, each vendor made own versions of same drivers etc.

    From distribution perspective it was nightmare. I was maintaining OpenZaurus at that time and with ten models supported we had to build whole set of kernels. Good that four of them were differing only by amount of memory and flash so we were able to handle them as one machine leaving checking details to kernel once it booted. PXA250 or PXA255 processor was also handled by kernel.

    Those times also meant different bootloaders. Zaurus ones were awful. We even had to ignore kernel cmdline it gave as it did not fit even into our 2.4.18-crappix kernels and was completely wrong once we moved to 2.6 line.

    Nokia 770/N8x0 had another one. Developer boards had RedBoot, U-Boot (if lucky) or whatever vendor invented. Some had a way to change and store boot commands, some did not. Space for kernel could be limited in a way that getting something which fits was a challenge.

    Basically for most of devices you had to handle booting, updates of kernels etc. separately.

    Linaro to the rescue

    In 2010 Arm with some partners created Linaro to improve Linux situation on Arm devices. I was one of first engineers there. We were present in many areas. Porting software, benchmarking, improving performance etc.

    And cleaning kernel/boot situation. I do not know how many people remember this post by Linus Torvalds:

    Gaah. Guys, this whole ARM thing is a f*cking pain in the ass.

    You need to stop stepping on each others toes. There is no way that your changes to those crazy clock-data files should constantly result in those annoying conflicts, just because different people in different ARM trees do some masturbatory renaming of some random device. Seriously.

    This was reaction to a moment when someone created another copy of some drivers. It was popular way to do things on Arm architecture — each vendor had their own version of PL011 serial driver etc.

    Some time later “arm-soc” subsystem was created to handle merging code touching device support, drivers etc. This allowed Russell King to concentrate on maintaining Arm architecture support.

    During next years most of vendor versions were merged into single ones. And moved where they belong — from arch/arm/ to drivers/ area of kernel.

    At some moment adding new board files was forbidden as Arm architecture was migrating into DeviceTree world.

    DeviceTree migration

    Why going into DeviceTree (DT in short)? What it gave us? Other than new problems?

    There were several such questions. The good part is that it was not something new to the Linux kernel. DT was already in use on Power architecture (and iirc SPARC). After some adaptations Arm devices became more maintainable.

    DeviceTree solved one crucial problem of Arm — lack of hardware discovery. System on Chip (SoC) can contain several controllers, processor cores etc. Before it was handled inside of ‘board file’ but also required building kernels per nearly each device. Now kernel was finally able to boot, parse DT information and get idea what is available and which drivers need to be used.

    That way one kernel was able to support several devices. And amount of them was bigger and bigger each release. At some moment you could build one kernel for all Arm v4 and v5 devices plus second one for v6 and v7 ones. Huge improvement.


    When it comes to bootloaders situation changed here as well. Most of ones used in past vanished and U-Boot became kind of ‘gold standard’. DeviceTree support was present but still each device had own way of booting. Different commands, storage options etc.

    Distributions handled that in miscellaneous ways. Extlinux support, ‘flash-kernel’ scripts etc.

    At some moment Dennis Gilmore took some time and introduced generic boot command for U-Boot. It was merged in July 2014. So instead of having different ways of handling stuff there was now one command on all devices (once they migrated).

    Kernel and initramfs were checked on sd/mmc/emmc, sata, scsi, ide, usb and then fallback to tftp. It was expanded since then to support several options and is now standard in U-Boot.

    AArch64 arrival

    At the beginning of 2013 several AArch64 systems started to appear. SBC ones followed what was on 32-bit Arm but servers were driven into different direction.


    They were supposed to be as boring as x86 one were. You unpack, put it into rack, connect standard power/network cables and boot it without worrying will it work or not. At same time provide administrators with same environment as they had on x86.

    So it meant UEFI as a firmware. ACPI as hardware description. And I simplified a bit.

    So to make it right it needed work on defining standards and then vendors to follow them.

    SBSA defined hardware

    First specification was Server Base System Architecture (SBSA in short). It defined hardware part — each AArch64 machine needs to use PL011 serial port, PL031 RTC, PL061 GPIO controller etc. And PCI Express support without quirks. Without it it can not be called server.

    SBSA has several levels of compliance. Nowadays level 3 is minimal version.

    Level 0 was funny as it covered only X-Gene1 boxes (SoC was older than specification).

    SBBR defined firmware

    Simplest definition of Server Base Boot Requirements specification? Server needs to run UEFI and use ACPI to describe hardware. And has to be SBSA compliant.

    Someone may ask why UEFI and ACPI. One reason is that they are present in x86 servers and aarch64 ones follow them as much as possible in behaviour. Other is that this way there are things which can be done with firmware help.

    But ACPI was x86 only so it needed to be adapted to AArch64 architecture. Work started by making ACPI an open specification under UEFI Forum agenda so it became open to anyone (it was Intel, Microsoft, Phoenix and Toshiba only before). There were many changes made since then. And several new tables defined.

    I heard several rumours about why ACPI. Someone said that ACPI was forced by Microsoft. In reality it was decision taken by all major distros and Microsoft.

    So what SBBR compliance gives? For start it allows to run generic distribution kernels out of the box. Each server SoC has same basic components and use same standards to boot system. So far Linux distributions, several *BSD systems and Microsoft Windows support SBBR machines out of the box.

    For example getting Qualcomm Centriq or Huawei TaiShan servers supported in Debian ‘buster’ was very easy task. Both booted with distribution kernel. Huawei one required enabling of on-board network card, Centriq had SAS controller module to enable to connect to storage (which was enabled on few other architectures already).

    EBBR for those who can not follow

    In short Embedded Base Boot Requirements are kind of SBBR for non-server class hardware.

    Device can use ACPI and/or DeviceTree to describe hardware. May boot whatever as long it provides EFI Boot Services to bootloader used by distributions (grub2, gummiboot etc).

    Specification feels made especially for distributions to make their life easier. This way there is one way to boot both SBC and SBBR compliant machines.

    Getting distribution kernel running on EBBR board is usually more work than it is with SBBR compliant server. All hardware specific options need to be found and enabled (from SoC support to all it’s drivers etc).

    BSA, BBR?

    During Arm DevSummit 2020 there was announcement of new standards for Arm devices:

    Arm is extending the system architecture standards compliance from servers to other segments of the market, edge and IoT. We introduce the new BSA specification with market segment-specific supplements and provide the operating system-oriented boot requirements recipes in the new BBR specification.

    They are described in second part of this article.

    Written by Marcin Juszkiewicz on
  7. Upgraded to Fedora 33

    I am running Fedora on my desktop since started working for Red Hat. Kind of ‘eat your own dogfood’ style despite fact that I am not active in Fedora development for some time.

    Fedora 33 reached Beta status so it was time to upgrade.

    Do it Debian style

    I use Debian since 1999 so got used to several ways of doing things which may not always fit into official Fedora guidelines.

    One of them is my way of upgrading to newer release:

    LANGUAGE=C dnf distrosync --releasever 33

    If there is a problem listed then I try to solve it with use of “—best” or even “—allow-erasing” options to check which packages are a problem. But this time it went smoothly.

    Rebooted and system works fine. Or so I thought…

    SSH keys

    Fedora ships OpenSSH 8.4p1 so if you used ‘ssh-rsa’ keys then you may need to generate newer ones. More info in OpenSSH 8.3 announcement.

    I got hit by this on Gerrit instances:

    $ git remote update
    Fetching gerrit
    marcin.juszkiewicz@review.linaro.org: Permission denied (publickey).

    One workaround is small addition to ssh configuration file:

    Host *
        PubkeyAcceptedKeyTypes +rsa-sha2-256,rsa-sha2-512

    TLS v1.2 is the new default

    New distribution version, new defaults. This time TLS v1.2 became default version. I was informed about it when wanted to send email and Thunderbird told me that it is unable to talk to my mail server…

    Logged to server, checked Postfix log and found this:

    connect from IP_ADDRESS
    SSL_accept error from IP_ADDRESS: -1
    warning: TLS library problem: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol:../ssl/statem/statem_srvr.c:1661:
    lost connection after CONNECT from IP_ADDRESS
    disconnect from IP_ADDRESS commands=0/0

    Looks nasty. Did some searching and changed Postfix config to accept TLSv1.2 on incoming connections.


    One of features of Fedora 33 is switch from dnsmasq to systemd-resolved for name resolution. On my system I have some local changes to former one to get internal Red Hat names resolved without using company DNS for everything. Therefore I reverted migration and keep using dnsmasq.

    One day I may be able to understand systemd-resolved documentation and migrate my local configuration to it.


    So far no other issues found. System works as it should. I still run kind of KDE desktop on X11.

    Written by Marcin Juszkiewicz on
  8. 8 years of my work on AArch64

    Back in 2012 AArch64 was something new, unknown yet. There was no toolchain support (so no gcc, binutils or glibc). And I got assigned to get some stuff running around it.


    As there was no hardware cross compilation was the only way. Which meant OpenEmbedded as we wanted to have wide selection of software available.

    I learnt how to use modern OE (with OE Core and layers) by building images for ARMv7 and checking them on some boards I had floating around my desk.

    Non-public toolchain work

    Some time later first non-public patches for binutils and gcc arrived in my inbox. Then eglibc ones. So I started building and on 12th September 2012 I was able to build helloworld:

    12:38 hrw@puchatek:aarch64-oe-linux$ ./aarch64-oe-linux-gcc ~/devel/sources/hello.c -o hello
    12:38 hrw@puchatek:aarch64-oe-linux$ file hello
    hello: ELF 64-bit LSB executable, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.39, not stripped
    12:39 hrw@puchatek:aarch64-oe-linux$ objdump -f hello
    hello:     file format elf64-littleaarch64
    architecture: aarch64, flags 0x00000112: 
    start address 0x00000000004003e0

    Then images followed. Several people at Linaro (and outside) used those images to test misc things.

    At that moment we ran ARMv8 Fast models (quite slow system emulator from Arm). There was a joke that Arm developers formed a queue for single core 10 GHz x86-64 cpus to get AArch64 running faster.

    Toolchain became public

    Then 1st October 2012 came. I entered Linaro office in Cambridge for AArch64 meeting and was greeted with “glibc patches went to public ML” information. So I rebased my OpenEmbedded repository, updated patches, removed any traces of non-public ones and published whole work.

    Building on AArch64

    My work above added support for AArch64 as a target architecture. But can it be used as a host? One day I decided to check and ran OpenEmbedded on AArch64.

    After one small patch it worked fine.

    X11 anyone?

    As I had access to Arm Fast model I was able to play with graphics. So one day in January 2013 I did a build and and started Xorg. Through next years I had fun when people wrote that they got X11 running on their AArch64 devices ;D

    Two years later I had Applied Micro Mustang at home (still have it). Once it had working PCI Express support I added graphics card and started X11 on hardware.

    Then went debugging why Xorg requires configuration file and one day with help from Dave Airlie, Mark Salter and Matthew Garrett I got two solutions for the problem. Do not remember did any of them went upstream but some time later problem was solved.

    Few years later I met Dave Airlie at Linux Plumbers. We introduced to each other and he said “ah, you are the ‘arm64 + radeon guy’” ;D

    AArch64 Desktop week

    One day in September 2015 I had an idea. PCIe worked, USB too. So I did AArch64 desktop week. Connected monitors, keyboard, mouse, speakers and used Mustang instead of my x86-64 desktop.

    It was fun.


    First we had nothing. Then I added AArch64 target into OpenEmbedded.

    Same month Arm released Foundation model so anyone was able to play with AArch64 system. No screen, just storage, serial and network but it was enough for some to even start building whole distributions like Debian, Fedora, OpenSUSE, Ubuntu.

    At that moment several patches were shared by all distributions as it was faster way than waiting for upstreams. I saw multiple versions of some of them during my journey of fixing packages in some distributions.

    Debian and Ubuntu

    In February 2013 Debian/Ubuntu team presented their AArch64 port. It was their first architecture bootstrapped without using external toolchains. Work was done in Ubuntu due to different approach to development than Debian has. All work was merged back so some time later Debian also had AArch64 port.


    Fedora team started early — October 2012, right after toolchain became public. Used Fedora 17 packages and switched to Fedora 19 during work.

    When I joined Red Hat in September 2013 one of my duties was fixing packages in Fedora to get them built on AArch64.


    In January 2014 first versions of QEMU support arrived and people moved from using Foundation model. March/April OpenSUSE team did massive amount of builds to get their distribution built that way.


    Fedora bootstrap also meant RHEL 7 bootstrap. When I joined Red Hat there were images ready to use in models. My work was testing them and fixing packages. There were multiple times when AArch64 fix helped to build also on ppc64le and s390x architectures.

    Hardware I played with

    First Linux capable hardware was announced in June 2013. I got access to it at Red Hat. Building and debugging was much faster than using fast models ;D

    Applied Micro Mustang

    Soon Applied Micro Mustangs were everywhere. Distributions used them to build packages etc. Even without support for half of hardware (no PCI Express, no USB).

    I got one in June 2014. Running UEFI firmware out of the box. At first months I had a feeling that firmware is developed at Red Hat as we had fresh versions often right after first patches for missing hardware functionality were written. In reality it was maintained by Applied Micro and we had access to sources so there were some internal changes in testing (that’s why I had firmware versions like ‘0.12-rh’).

    All those graphics cards I collected to test how PCI Express works. Or testing USB before it was even merged into Linux mainline kernel. Using virtualization for development of armhf build fixes (8 cores, 12 gigabytes of ram and plenty of storage beat all armv7 hardware I had).

    I stopped using Mustang around 2018. It is still under my desk.

    For those who use: make sure you have 3.06.25 firmware.


    In February 2015 Linaro announced 96boards initiative. The plan was to make small, unified SBC with different Arm chips. Both 32- and 64-bit ones.

    First ones were ‘Consumer Edition’. Small, limited to basic connectivity. Now there are tens of them. 32-bit, 64-bit, fpga etc. Choose your poison ;D

    Second ones were ‘Enterprise Edition’. Few attempts existed, most of them did not survived prototype phase. There was joke that full length PCI Express slot and two USB ports requirements are there because I wanted to have AArch64 desktop ;D

    Too bad that nothing worth using came from EE spec.


    As Linaro assignee I have access to several servers from Linaro members. Some are mass-market ones, some never made to market. We had over hundred X-Gene1 based systems (mostly as m400 cartridges in HPe Moonshot chassis’) and shutdown them in 2018 as they were getting more and more obsolete.

    Main system I use for development is one of those ‘never went to mass-market’ ones. 46 cpu cores, 96 GB of ram make it nice machine for building container images, Debian packages or running virtual machines in OpenStack.


    For some time I was waiting for some desktop class hardware to have development box more up-to-date than Mustang. Months turned into years. I no longer wait as it looks like there will be no such thing.

    Solidrun has made some attempts in this area. First with Macchiatobin and later with Honeycomb. I did not used any of them.


    When I (re)joined Linaro in 2016 I became part of team working on getting OpenStack working on AArch64 hardware. We used Liberty, Mitaka, Newton releases and then changed way we work and started contributing more. And more. Kolla, Nova, Dib and other projects. Added aarch64 nodes to OpenDev CI.

    The effect of it was Linaro Developer Cloud used by hundreds of projects to speed-up their aarch64 porting, tens of projects hosting their CI systems etc.

    Two years later Amazon started offering aarch64 nodes in AWS.


    I spent half of my life with Arm on AArch64. Had great moments like building helloworld as one of first people outside of Arm Ltd. Got involved in far more projects then ever thought. Met new friends, visited several places in the world I would probably never go otherwise.

    I also got grumpy and complained far too many times that AArch64 market is ‘cheap but limited sbc or fast but expensive servers and nearly nothing in between’. Wrote some posts about missing systems targeting software developers and lost hope that such will happen.

    NOTE: It is 8 years of my work on AArch64. I work with Arm since 2004.

    Written by Marcin Juszkiewicz on
  9. From a diary of AArch64 porter — drive-by coding

    Working on AArch64 often means changing code in some projects. I did that so many times that I am unable to say where I have some commits. Such thing got a name: drive-by coding.


    Drive-by coding is situation when you appear in some software project, do some changes, get them merged and then disappear to never be seen again.

    Let’s build something

    All starts from simple thing: I have/want to build some software. But for some reason it does not cooperate. Sometimes it is simple architecture check missing, sometimes atomic operations are not present, intrinsics are missing or anything else.

    First checks

    Then comes moment of looking at build errors and trying to work out some solution. Have I seen that bug before? Does it look familiar?

    If this is something new then quick Google search for error message. And checking bug reports/issues on project’s website/repo. There can be ready to use patches, information how to fix it or even some ideas why does it happen.

    If this is system call failure in some tests then I check my syscalls table are those ones handled on aarch64 and try to change code if they are not (legacy ones like open, symlink, rename).

    Simple fixes

    When I started working with AArch64 (in 2012) there were moments when many projects were easy to fix. If atomics were issue then copying them from Linux kernel was usually solution (if license allowed).

    Architecture checks with pile of #ifdef __X86_64__ or similar ones which are trying to do decide for simple things like “32/64” or “little/big endian”. Nowadays such ones do not happen as often as it was.

    SIMD intrinsics can be a problem. All those vst1q_f32_x2(), vld1q_f32_x2 and similar. I do not have to understand them to know that it usually means that C compiler lacks some backports as those functions were added into gcc and llvm already (like it was with Pytorch recently).

    Complex stuff

    There are moments when getting software to build needs something more complicated. Like I wrote above, I usually start with searching for error message and checking was it an issue in some other projects. And how it got solved. If I am lucky then patch can be done in short time and send for review upstream (once it builds and passes tests).

    Sometimes all I can do is reporting issue upstream and hope that someone will care enough to respond. Usually it ends with at least discussion on potential ways to fix, sometimes hints or even patches to test.

    Projects response

    Projects usually accept patches, review them and merge. In several cases it took longer than expected, sometimes there was larger amount of those so they remember me (at least for some time). It helps when I have something for those project again months/years later.

    There are projects where I prefer to forget that they exist. Complicated contribution rules, crazy CI setup, weird build systems (ever heard about ‘bazel’?). Or comments in ‘we do not give a shit about non-x86’ style (with a bit polished language). Been there, fixed something to get stuff working and do not want to go back.


    Drive-by coding’ reminds me going abroad for conferences. People think that you saw interesting places when in reality you spent most of time inside of hotel and/or conference centre.

    It is similar with code. I was in several projects, usually had no idea what they do, how they work. Came, looked shortly, fixed something and went back home.

    Written by Marcin Juszkiewicz on
  10. So your hardware is ServerReady?

    Recently I changed my assignment at Linaro. From Cloud to Server Architecture. Which means less time spent on Kolla things, more on server related things. And at start I got some project I managed to forget about :D

    SBSA reference platform in QEMU

    In 2017 someone got an idea to make a new machine for QEMU. Pure hardware emulation of SBSA compliant reference platform. Without using of virtio components.

    Hongbo Zhang wrote code and got it merged into QEMU, Radosław Biernacki wrote basic support for EDK2 (also merged upstream). Out of box it can boot to UEFI shell. Linux is not bootable due to lack of ACPI tables (DeviceTree is not an option here).

    ACPI tables in firmware

    Tanmay Jagdale works on adding ACPI tables in his fork of edk2-platforms. With this firmware Linux boots and can be used.

    Testing tools

    But what the point of just having reference platform if there is no testing? So I took a look and found two interesting tools:

    Server Base System Architecture — Architecture Compliance Suite

    SBSA ACS tool requires ACPI tables to be present to work. And once started it nicely checks how compliant your system is:

    FS0:\> Sbsa.efi -p
     SBSA Architecture Compliance Suite
        Version 2.4
     Starting tests for level  4 (Print level is  3)
     Creating Platform Information Tables
     PE_INFO: Number of PE detected       :    3
     GIC_INFO: Number of GICD             :    1
     GIC_INFO: Number of ITS              :    1
     TIMER_INFO: Number of system timers  :    0
     WATCHDOG_INFO: Number of Watchdogs   :    0
     PCIE_INFO: Number of ECAM regions    :    2
     SMMU_INFO: Number of SMMU CTRL       :    0
     Peripheral: Num of USB controllers   :    1
     Peripheral: Num of SATA controllers  :    1
     Peripheral: Num of UART controllers  :    1
          ***  Starting PE tests ***
       1 : Check for number of PE            : Result:  PASS
       2 : Check for SIMD extensions                PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
       3 : Check for 16-bit ASID support            PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
       4 : Check MMU Granule sizes                  PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
       5 : Check Cache Architecture                 PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
       6 : Check HW Coherence support               PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
       7 : Check Cryptographic extensions           PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
       8 : Check Little Endian support              PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
       9 : Check EL2 implementation                 PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
      10 : Check AARCH64 implementation             PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
      11 : Check PMU Overflow signal         : Result:  PASS
      12 : Check number of PMU counters             PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    0 for Level=  4 : Result:  --FAIL-- 1
      13 : Check Synchronous Watchpoints            PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
      14 : Check number of Breakpoints              PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
      15 : Check Arch symmetry across PE            PSCI_CPU_ON: failure
           Reg compare failed for PE index=1 for Register: CCSIDR_EL1
           Current PE value = 0x0         Other PE value = 0x100FBDB30E8
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
      16 : Check EL3 implementation                 PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
      17 : Check CRC32 instruction support          PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    1 for Level=  4 : Result:  --FAIL-- 129
      18 : Check for PMBIRQ signal
           SPE not supported on this PE      : Result:  -SKIPPED- 1
      19 : Check for RAS extension                  PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    0 for Level=  4 : Result:  --FAIL-- 1
      20 : Check for 16-Bit VMID                    PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    0 for Level=  4 : Result:  --FAIL-- 1
      21 : Check for Virtual host extensions        PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
           Failed on PE -    0 for Level=  4 : Result:  --FAIL-- 1
      22 : Stage 2 control of mem and cache         PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
    : Result:  -SKIPPED- 1
      23 : Check for nested virtualization          PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
    : Result:  -SKIPPED- 1
      24 : Support Page table map size change       PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
    : Result:  -SKIPPED- 1
      25 : Check for pointer signing                PSCI_CPU_ON: failure
      25 : Check for pointer signing                PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
    : Result:  -SKIPPED- 1
      26 : Check Activity monitors extension        PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
    : Result:  -SKIPPED- 1
      27 : Check for SHA3 and SHA512 support        PSCI_CPU_ON: failure
           PSCI_CPU_ON: failure
    : Result:  -SKIPPED- 1
          *** One or more PE tests have failed... ***
          ***  Starting GIC tests ***
     101 : Check GIC version                 : Result:  PASS
     102 : If PCIe, then GIC implements ITS  : Result:  PASS
     103 : GIC number of Security states(2)  : Result:  PASS
     104 : GIC Maintenance Interrupt
           Failed on PE -    0 for Level=  4 : Result:  --FAIL-- 1
          One or more GIC tests failed. Check Log
          *** Starting Timer tests ***
     201 : Check Counter Frequency           : Result:  PASS
     202 : Check EL0-Phy timer interrupt     : Result:  PASS
     203 : Check EL0-Virtual timer interrupt : Result:  PASS
     204 : Check EL2-phy timer interrupt     : Result:  PASS
     205 : Check EL2-Virtual timer interrupt
           v8.1 VHE not supported on this PE : Result:  -SKIPPED- 1
     206 : SYS Timer if PE Timer not ON
           PE Timers are not always-on.
           Failed on PE -    0 for Level=  4 : Result:  --FAIL-- 1
     207 : CNTCTLBase & CNTBaseN access
           No System timers are defined      : Result:  -SKIPPED- 1
         *** Skipping remaining System timer tests ***
          *** One or more tests have Failed/Skipped.***
          *** Starting Watchdog tests ***
     301 : Check NS Watchdog Accessibility
           No Watchdogs reported          0
           Failed on PE -    0 for Level=  4 : Result:  --FAIL-- 1
     302 : Check Watchdog WS0 interrupt
           No Watchdogs reported          0
           Failed on PE -    0 for Level=  4 : Result:  --FAIL-- 1
          ***One or more tests have failed... ***
          *** Starting PCIe tests ***
     401 : Check ECAM Presence               : Result:  PASS
     402 : Check ECAM value in MCFG table    : Result:  PASS
            Unexpected exception occured
            FAR reported = 0xEBDAB180
            ESR reported = 0x97800010
         Total Tests run  =   42;  Tests Passed  =   11  Tests Failed =   22
          *** SBSA tests complete. Reset the system. ***

    As you can see there is still a lot of work to do.

    ACPI Tables View

    This tool displays content of ACPI tables in hex/ascii format and then with information interpreted field by field.

    What makes it more useful is “-r 2” argument as it enables checking tables against Server Base Boot Requirements (SBBR) v1.2 specification. On SBSA reference platform with Tanmay’s firmware it lists two errors:

    ERROR: SBBR v1.2: Mandatory DBG2 table is missing
    ERROR: SBBR v1.2: Mandatory PPTT table is missing
    Table Statistics:
            2 Error(s)
            0 Warning(s)

    So situation looks good as those can be easily added.


    So we have code to check and tools to do that. Add one to another and you have a clean need for CI job. So I wrote one for Linaro CI infrastructure: “LDCG SBSA firmware“. It builds top of QEMU and EDK2, then boot it and run above tools. Results are sent to mailing list.


    The Arm ServerReady compliance program provides a solution for servers that “just works”, allowing partners to deploy Arm servers with confidence. The program is based on industry standards and the Server Base System Architecture (SBSA) and Server Base Boot Requirement (SBBR) specifications, alongside Arm’s Server Architectural Compliance Suite (ACS). Arm ServerReady ensures that Arm-based servers work out-of-the-box, offering seamless interoperability with standard operating systems, hypervisors, and software.

    In other words: if your hardware is SBSA complaint then you can go with SBBR compliance tests and then go and ask for certification sticker or sth like that.

    But if your hardware is not SBSA compliant then EBBR is all you can get. Far from being ServerReady. Never mind what people tries to say — ServerReady requires SBBR which requires SBSA.

    Future work

    More tests to integrate. ARM Enterprise ACS is next on my list.

    Written by Marcin Juszkiewicz on
Page 1 / 80
Older posts »