Marcin Juszkiewicz

Installed Debian ‘testing’ from netinst iso (2020.06.15) today. Fetched, wrote to USB pen drive and booted. Due to U-Boot being present in on-board SPI flash I did not had to mess with installation media.

Issues

There were some issues:

1. no graphics on default installer (known, someone promised to fix it)
2. grub refusing to install (bug against installer reported)
3. Panfrost failing to initialize

Serial console FTW!

Ok, this time I am joking. There are two choices: text and graphical installer. First option lacks kernel modules for graphics so only serial console is available. Graphical installer works fine.

EFI Grub and lack of EFI variables storage

As I booted board with U-Boot there was no EFI variables storage. Grub was not satisfied:

os-prober: debug: running /usr/lib/os-probes/50mounted-tests on /dev/sdb2
50mounted-tests: debug: mounted using GRUB fat filesystem driver
50mounted-tests: debug: running subtest /usr/lib/os-probes/mounted/40lsb
50mounted-tests: debug: running subtest /usr/lib/os-probes/mounted/90linux-distro
grub-installer: info: Installing grub on 'dummy'
grub-installer: info: grub-install does not support --no-floppy
grub-installer: info: Running chroot /target grub-install  --force "dummy"
grub-installer: Installing for arm64-efi platform.
grub-installer: grub-install: warning: Cannot set EFI variable Boot0000.
grub-installer: grub-install: warning: vars_set_variable: write() failed: Invalid argument.
grub-installer: grub-install: warning: _efi_set_variable_mode: ops->set_variable() failed: No such file or directory.
grub-installer: grub-install: error: failed to register the EFI boot entry: No such file or directory.
grub-installer: error: Running 'grub-install  --force "dummy"' failed.

How to work around it?

1. chroot into target system and run update-grub by hand
2. copy “/efi/debian/grubaa64.efi” to “/efi/boot/bootaa64.efi”

This way U-Boot will get efi binary to load in default location.

Panfrost initialization failure

Panfrost kernel module needs some devfreq governor. Kernel has four of them, Debian enables one. There are no dependencies between those modules which ends with this error:

panfrost ff9a0000.gpu: devfreq_add_device: Unable to find governor for the device
panfrost ff9a0000.gpu: [drm:panfrost_devfreq_init [panfrost]] *ERROR* Couldn't initialize GPU devfreq
panfrost ff9a0000.gpu: Fatal error during devfreq init
panfrost: probe of ff9a0000.gpu failed with error -22

Solution:

1. boot system
2. rmmod panfrost, modprobe governor_simpleondemand, modprobe panfrost
3. update-initramfs -u -kall

Good option at this phase is changing configuration of update-initramfs to include only needed kernel modules (by setting “MODULES=dep” in it’s configuration). This allowed me to shrink initramfs from 37 to 13 megabytes (removal of plymouth and ntfs-3g shrinked to 6.6 MB).

Final effect

Board boots directly to graphical login manager and then to KDE Plasma session. Some of OpenGL games work, some not (Nexuiz). Looks good.

SBBR or GTFO

Me.

But Arm world no longer ends on “SBBR compliant or complete mess”. For over a year there is new specification called EBBR (Embedded Base Boot Requirements).

WTH is EBBR?

In short it is kind of SBBR for devices which can not comply. So you still need to have some subset of UEFI Boot/Runtime Services but it can be provided by whatever bootloader you use. So U-Boot is fine as long it’s EFI implementation is enabled.

ACPI is not required but may be present. DeviceTree is perfectly fine. You may provide both or one of them.

Firmware can be stored wherever you wish. Even MBR partitioning is available if really needed.

Make it nice way

RockPro64 has 16MB of SPI flash on board. This is far more than needed for storing firmware (I remember time when it was enough for palmtop Linux).

During last month I sent a bunch of patches to U-Boot to make this board as comfortable to use as possible. Including storing of all firmware parts into on board SPI flash.

Needed files

To have U-Boot in SPI flash there you need to fetch two files:

• SPL + TPL
• U-Boot itself

Their sha256 sums:

3985f2ec63c2d31dc14a08bd19ed2766b9421f6c04294265d484413c33c6dccc  idbloader.img
35ec30c40164f00261ac058067f0a900ce749720b5772a759e66e401be336677  u-boot.itb

Store them as files on USB pen drive.

Flashing RockPro64 board

NOTE: I assume that you already have a way to boot your board to U-Boot shell (most common is to use microSD card with U-Boot on it).

Reboot board to U-Boot shell. Plug USB pen drive into any of RockPro64 USB ports.

Next do this set of commands to update U-Boot:

Hit any key to stop autoboot:  0 
=> usb start

=> ls usb 0:1
   163807   idbloader.img
   867908   u-boot.itb

2 file(s), 0 dir(s)

=> sf probe
SF: Detected gd25q128 with page size 256 Bytes, erase size 4 KiB, total 16 MiB

=> load usb 0:1 ${fdt_addr_r} idbloader.img
163807 bytes read in 16 ms (9.8 MiB/s)

=> sf update ${fdt_addr_r} 0 ${filesize}
device 0 offset 0x0, size 0x27fdf
163807 bytes written, 0 bytes skipped in 2.93s, speed 80066 B/s

=> load usb 0:1 ${fdt_addr_r} u-boot.itb
867908 bytes read in 53 ms (15.6 MiB/s)

=> sf update ${fdt_addr_r} 60000 ${filesize}
device 0 offset 0x60000, size 0xd3e44
863812 bytes written, 4096 bytes skipped in 11.476s, speed 77429 B/s

And reboot board.

After this your RockPro64 will have firmware stored in on board SPI flash. No need for wondering which offsets to use to store them on SD card etc.

Booting installation media

The nicest part of it is that no longer you need to mess with installation media. Fetch Debian/Fedora installer ISO, write it to USB pen drive, plug into port and reboot board.

Should work with any generic AArch64 installation media. Of course kernel on media needs to support RockPro64 board. I played with Debian ‘testing’ and Fedora 32 and rawhide and they booted fine.

Testing U-Boot on microSD

By default RockPro64 loads U-Boot from SPI flash. If you need/want to boot from microSD then follow instruction from official wiki:

If you mess-up your SPI and are unable to boot, jumpering pins 23 (CLK) and 25 pin (GND) on the PI-2-bus header will disable the SPI as a boot device.

You have to remove the jumper 2 seconds after having started your RP64 (before the white LED turns ON) otherwise the SPI will be missing and you won’t be able to flash it.

My setup has ‘disable SPI’ button next to ‘disconnect Tx serial line’ switch — both on small breadboard next to the board.

I work with OpenDev CI for a while. My first Kolla patches were over three years ago. We (Linaro) added AArch64 nodes few times — some nodes were taken down, some replaced, some added.

Speed or lack of it

Whenever you want to install some Python package using pip it is downloaded from Pypi (directly or mirror). If there is a binary package then you get it, if not then “noarch” package is fetched.

In worst case source tarball is downloaded and whole build process starts. You need to have all required compilers installed, development headers for Python and all required libraries and rest of needed tools. And then wait. And wait as some packages require a lot of time.

And then repeat it again and again as you are not allowed to upload packages into Pypi for projects you do not own.

Argh you, protobuf

There was a new release of protobuf package. OpenStack bot picked it up, sent patch for review and it got merged.

And all AArch64 CI jobs failed…

Turned out that protobuf 3.12.0 was released with x86 wheels only. No source tarball. At all.

This turned out to be new maintainer mistake — after 2-3 weeks it was fixed in 3.12.2 release.

Another CI job then

So I started looking at ‘requirements’ project and created a new CI job for it. To check are new package versions are available for AArch64. Took some time and several side updates as well (yak shaving all the way again).

Stuff got merged and works now.

Wheels cache

While working on above CI job I had a discussion with OpenDev infra team how to make it work properly. Turned out that there were old jobs doing exactly what I wanted: building wheels and caching them for next CI tasks.

It took several talks and patches from Ian Wienand, Clark Boylan, Jeremy ‘fungi’ Stanley and others. Several CI jobs got renamed, some were moved from one project to another. Servers got configuration changes etc.

Now we have wheels built for both x86-64 and AArch64 architectures. Covering CentOS 7/8, Debian ‘buster’ and Ubuntu ‘xenial/bionic/focal’ releases. For OpenStack ‘master’ and few stable branches.

Effect

Requirements project has quick ‘check-uc’ job running on AArch64 to make sure that all packages are available for both architectures. All OpenStack projects profit from it.

In Kolla ‘openstack-base’ image went from 23:49 to just 5:21 minutes. Whole Debian/source build is now 57 minutes instead of 2 hours 20 minutes.

Nice result, isn’t it?

Some time passed since I got that board. And some things changed as well.

U-Boot changes

I went through U-Boot and submitted a bunch of changes. Most of them were small tweaks to RockPro64 board configuration:

• enable USB OHCI host (so USB 1.1 things can be plugged directly into ports)
• enable USB keyboard
• enable RNG support
• enable SPI boot
• start USB support

With above changes I have all firmware parts stored in on-board SPI chip and do not have to worry about storing whatever on removable media. Also do not need serial console to play with U-Boot as everything can be done with USB keyboard and HDMI monitor.

Board configuration

To be able to work comfortably with RockPro64 board I mounted it in acrylic case from Pine A64 and then added some tweaks.

Breadboard wires and switches

Black and white cables are SPI clock and ground. Using button I can short them during boot so on-board flash gets ignored and U-Boot gets loaded from SD card.

Two orange wires are serial Rx line going through hardware switch. The reason is simple: for some reason board does not boot when Rx line is used during first power on (right after power plug insert, not after pressing Power button). So instead of disconnecting wire each time I can just move switch left, power on, wait for U-Boot messages, switch right and have both directions of serial console working.

Too bad that there no Power/Reset pins as I would move buttons to breadboard as well to make them more reachable.

USB hubs

Next to the board I have a bunch of USB hubs:

• USB-C to USB-A/HDMI/Power one from one of previous Linaro Connects
• USB 3.0 one connected with microUSB cable so work as USB 2.0 one
• USB 1.1 one

Each of them was in use when I played with enabling USB keyboard as I do not have USB 2.0 keyboard and needed a way to connect standard 1.1 one.

Other stuff

Tincantools SPI Hook works as serial console and there is PCI Express x8->x8 (open ended) riser so I can play with different PCIe cards.

And there is a jumper on ‘disable eMMC’ pins as I lack such module.

What next?

Next step would be testing distribution installers a bit. And check how they work. I did some attempts already and waiting for fixes to be merged.

When I was a kid there was a children’s book about Wojtek who wanted to be firefighter. It is part of culture for my generation.

I never wanted to follow Wojtek’s dreams. But during last years I became firefighter. And this is not a good thing in a long term.

CI failures

During last months we (Linaro) took care of AArch64 support in OpenStack infrastructure. There are nodes with CentOS 7 and 8, Debian ‘stretch’ and ‘buster, Ubuntu ‘xenial’, ‘bionic’ and ‘focal’. And several CI jobs in some projects (Disk Image Builder, Kolla, Nova and some other).

And those CI jobs tend to fail. As usual, right? Not quite…

Missing Python packages

One day when I joined irc in the morning I was greeted with “aarch64 ci fails — can you take a look?” from one of project developers.

Quick look into logs:

ERROR: Could not find a version that satisfies the requirement ntplib (from versions: none)
ERROR: No matching distribution found for ntplib

Then usual path. Pypi, check release, history, go to homepage, fill issue. Curse. And wait for upstream to fix problem. They fixed, CI was working again.

Last Monday — I started work ready to do something interesting. And then was greeted with same story “aarch64 ci fails, can you take a look?”.

ERROR: Could not find a version that satisfies the requirement protobuf==3.12.0 
(from versions: 2.0.0b0, 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0a2,
3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2.post1,
3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0, 3.1.0, 3.1.0.post1, 3.2.0rc1,
3.2.0rc1.post1, 3.2.0rc2, 3.2.0, 3.3.0, 3.4.0, 3.5.0.post1, 3.5.1, 3.5.2,
3.5.2.post1, 3.6.0, 3.6.1, 3.7.0rc2, 3.7.0rc3, 3.7.0, 3.7.1, 3.8.0rc1, 3.8.0,
3.9.0rc1, 3.9.0, 3.9.1, 3.9.2, 3.10.0rc1, 3.10.0, 3.11.0rc1, 3.11.0rc2, 3.11.0,
3.11.1, 3.11.2, 3.11.3)

Pypi, check, homepage, there was an issue already filled. So far no upstream response.

Problem got solved by moving all OpenStack projects to previous (3.11.3) release.

Missing/deprecated versions in distributions

So I started work on adding another CI job. This time for ‘requirements’ OpenStack project. To make sure that whatever Python package upgrade will be available on AArch64 as well.

As usual had to add a pile of distro dependencies to get ‘numpy’ and ‘scipy’ built correctly. And bump timeout to 2 hours. Build was going nice.

And then ‘confluent_kafka’ hit hard:

/tmp/pip-install-ld4fzu94/confluent-kafka/confluent_kafka/src/confluent_kafka.h:65:2: 
error: #error "confluent-kafka-python requires librdkafka v1.4.0 or later.
Install the latest version of librdkafka from the Confluent repositories, 
see http://docs.confluent.io/current/installation.html"                                                            

“librdkafka v1.4.0 or later” is not available in any distribution used on OpenStack infra nodes. Fantastic!

And repositories mentioned in error message are x86-64 only. Fun!

Sure, can do usual pypi, release, homepage, issue. Even went that way. Chain of GitHub projects building components into repositories. x86-64 only all the way.

External CI services

Which gets us to another thing — external CI services. You know: Azure pipelines, GitHub actions or Travis CI (alphabetical order). Used in misc ways by most FOSS projects nowadays.

Each of them has some kind of AArch64 support nowadays. But it looks like only Travis provides you with hardware. Azure and GitHub only can connect your external machines to their CI service.

Speed or lack of it

So you have a project where you need AArch64 support and upstream is already using Travis for their CI needs. Lucky you!

You work with project developers you get test suite running and then it time outs. It does not matter that CI machine you got have few cores because ‘pytest’ does not know how to run tests in parallel.

So you cut tests completely or partially. Or just abandon the idea.

No hardware, no binaries

If you are less lucky then you may get such answer from upstream. I had such in past few times. And I fully understand them — why support something when you can not even test does it work?

Firefighting or yak shaving?

When I discussed it with friends one of them mentioned that this reminds him more yak shaving than firefighting. To be honest it is both most of time.

AArch64 world has toys on cheap side, servers on expensive and some attempts in a middle.

Me.

I started working on AArch64 in 2012 year. Before hardware was available. Even before toolchain bits were available. Then first prototype server arrived at Red Hat, then it got replaced with Applied Micro Mustang. Then more servers, Mustang under desk. Joined Linaro (as Red Hat assignee) and started using their server lab.

And during all those years I never played with AArch64 SBC.

If it lacks proper storage and 16+GB ram then I am not interested.

Some months ago I decided to change it.

What to choose?

There are many AArch64 boards to choose from. Some are better, some are not. Popular ones and those with features. Like usual.

So I asked Peter Robinson and few other folks about their proposal and most of them agreed that Rockchip RK3399 looks most interesting. Has PCI Express slot on several boards, uses Mali T860 so can play with Panfrost driver, USB 3 ports are present etc.

RockPro64

After some market research I decided on RockPro64 from Pine64. Board has USB 3 in both A and C type, PCIe x4 slot is present and I can reuse acrylic case from Pine A64 board I bought few years ago. And it has “the most expensive chip in ARM world” present (16MB SPI flash for boot firmware).

Ordered in March, board arrived two days ago.

Setup

I digged in my boxes and found 12V/5A power supply (3A is recommended minimum), used some Ethernet cable flying under desk, old 64GB usb3 thumb drive for storage and my lovely IOGEAR GKM561R keyboard/trackpoint which I use for all boards bring-up.

Mounted board into acrylic case and started wiring extra stuff.

Disable SPI button

As boot firmware can be stored in onboard SPI chip there has to be a way to ignore it on boot. According to disable SPI note on vendor website it is a matter of crossing SPI_CLK with GND during boot.

So small breadboard, two wires, button and I am ready.

Serial console

As usual serial console is on pins. So breadboard got one of my cheap USB-Serial dongles plugged and then wires started.

Serial console setup post on forum says that 6/8/10 pins are all I need. Did them, connected USB cable and started picocom at 1.5Mbps speed:

Terminal ready
��tU�����UU���˕�U���TU J���UU���Q�UR�)���Q�
*�R��uT��*��U�
      �VQ��Y��ս+UJ.���I��zT����UQ])

Lovely! Turned out that CP2102 dongles are only “expected” to work at that speed. Digged deeper in drawers and found SPI Hook device from TinCanTools. It uses FT2232HL so should be fine with up to 12Mbps speed.

And worked without problems:

U-Boot TPL 2020.04 (Apr 20 2020 - 00:00:00)
Channel 0: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=16/15 CS=1 Die BW=16 Size=2048MB
Channel 1: LPDDR4, 50MHz
BW=32 Col=10 Bk=8 CS0 Row=16/15 CS=1 Die BW=16 Size=2048MB
256B stride
256B stride
lpddr4_set_rate: change freq to 400000000 mhz 0, 1
lpddr4_set_rate: change freq to 800000000 mhz 1, 0
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2020.04 (Apr 20 2020 - 00:00:00 +0000)
Trying to boot from MMC1

U-Boot 2020.04 (Apr 20 2020 - 00:00:00 +0000)

SoC: Rockchip rk3399
Reset cause: POR
Model: Pine64 RockPro64
DRAM:  3.9 GiB
PMIC:  RK808 
MMC:   dwmmc@fe320000: 1, sdhci@fe330000: 0
Loading Environment from MMC... Card did not respond to voltage select!
*** Warning - No block device, using default environment

So I was ready to play!

Mali

So when Arm will open source Mali drivers?

This question is present on each “Ask Arm Anything” sessions during Linaro Connect (ARM partners only session, badges are checked on room entry). And answer always is more or less “never”.

But there is Panfrost now. Fully open source driver for Mali family used in RK3399 SoC.

Booted board and it got stuck:

[   29.043106] panfrost ff9a0000.gpu: clock rate = 500000000
[   29.053073] rockchip-vop ff8f0000.vop: Adding to iommu group 1
[   29.058099] panfrost ff9a0000.gpu: mali-t860 id 0x860 major 0x2 minor 0x0 status 0x0
[   29.058948] panfrost ff9a0000.gpu: features: 00000000,100e77bf, issues: 00000000,24040400
[   29.059820] panfrost ff9a0000.gpu: Features: L2:0x07120206 Shader:0x00000000 Tiler:0x00000809 Mem:0x1 MMU:0x00002830 AS:0xff JS:0x7
[   29.061053] panfrost ff9a0000.gpu: shader_present=0xf l2_present=0x1
[   29.063559] rockchip-vop ff900000.vop: Adding to iommu group 2
[   29.147966] panfrost ff9a0000.gpu: devfreq_add_device: Unable to find governor for the device
[   29.151244] panfrost ff9a0000.gpu: [drm:panfrost_devfreq_init [panfrost]] *ERROR* Couldn't initialize GPU devfreq
[   29.173499] panfrost ff9a0000.gpu: Fatal error during devfreq init

Discussed issue with people on #panfrost IRC channel and they pointed me at “governor_simpleondemand” kernel module being missing. So I rebuilt initramfs with “dracut -v —force-drivers governor_simpleondemand —force” command and it booted to FullHD framebuffer.

Some work is needed here as monitor has 3440x1440 resolution so at least 2560x1440 or 2560x1080 should be used. But that can wait for now. I plan to move this board somewhere else on desk and connect to 24” full hd monitor.

Plans?

I do not plan to use that board for any serious stuff. More as a device to learn how EBBR world looks. Boot sequence with U-Boot loading Grub from EFI system partition looks good.

There are some things which need work. I want to have boot firmware stored in SPI flash as now I have 1GB microsd card in slot just to have U-Boot loaded.

There are some changes in U-Boot waiting for someone to test, USB-C port is not enabled at all (it should work as USB 3 and DisplayPort).

So who knows :D

Recently one of my friends worked on his websites and we exchanged some hints on how to make things better for accessibility and SEO. I checked my website and decided to work on another set of improvements.

Headers

First thing was headers (h1-h6) and their hierarchy. Loaded 51 articles, fixed their markup and then worked a bit on CSS to get styling like I wanted it to look.

Now there is just one H1 on website (page title), title of each article is in H2 and then H3-H5 are used in article content. Related posts are H3 as well as they are part of article.

Image captions

This thing was on my list for a while. In Wordpress times some articles used images with captions, some were not. Loaded 99 articles, fixed image titles and added ‘yafg’ plugin to Markdown to get all images described. Then some CSS styling and effect is quite ok:

Small tweaks

PageSpeed Insights complained that there can be a moment with no visible text (a flash of invisible text (FOIT) in webdev speak). Three small changes to get ‘font-display:swap’ in CSS and problem solved.

Also added some extra tags into head part of page — mostly OpenGraph junk.

Hope that it will be better usable now.

Each year has a day when people make jokes and watch the world burn.

2008: Let’s move to London

Twelve years ago, when I worked with London based OpenedHand company, I wrote about moving to London. So many people sent congratulations, offered help in getting settled etc. Next day I wrote that no, I am not moving to London.

2020: Let’s move to Vancouver

Two days ago I decided to repeat it some way. But it was late in the evening so I just wrote that on Facebook (at 00:14 local time):

12 years ago there was an idea to move to London. But it did not worked out. Looking at situation in Poland I am considering recent Amazon offer. Sure, Vancouver is far away but it is also nice city. And have snow during winter.

I can fly with one change for xmas or holidays.

There were many questions am I sure I want to do it. Even my wife asked ;D

One of friends thought that I was either serious or fishing for a raise (as for him message was written on 31st March due timezones).

AArch64 For Developers

When I wrote above post I got sick idea. Why not try to fool wider range? On easy target?

Quick check how ‘Prima Aprilis’ is called in English and hunt for good (and cheap) domain started. Turned out that Polish ones will be cheapest.

In the morning I bought ‘afd.biz.pl’ domain for about 4 EUR and started preparations. Free Bootstrap template, some pictures from the Internet and started working on page of not existing Polish startup specializing in computer boards design.

NUC

Many people in AArch64 industry would like to be able to buy NUC like computer. So it went first. Picture of some random Chinese product, some text and done.

Project date set to 1st April 2019, status ‘cancelled’ so simple reason: “lack of cheap PCI Express graphics chips which would fit on mainboard”. In my opinion it is valid reason as for AArch64 NUC to success either AMD Radeon or some NVidia GPU would need to be used. And I doubt that they can be bought in small quantities.

ATX mainboard

But AFD company had also more interesting project — full size ATX motherboard:

Design is an edited page of MSI X570-A PRO mainboard manual. Removed all RGB lights connectors, pump fan, additional 12V connector and few more elements. Copied some piece of I/O shield to not have a hole there.

Then I presented page to Leif for some review and he provided some hints on how to make it more believable.

Technical specification was too nice to be true (NUC mentioned above got subset of them):

• SBSA level 5, SBBR, ServerReady etc compliant
• Arm v8.4 cpu with 16 cores (Cortex-A72 like performance)
• CPU will be user upgradable
• cpu cooling is AMD AM4 compatible
• Up to 64GB of DDR4 memory
• ECC support not tested yet
• 32GB sticks not tested but should work and give 128GB ram
• Two m.2 slots (PCIe x4) for NVME storage
• PCIe setup: x16, x8, 3 x1 slots
• six SATA III ports
• on board 1GbE network
• 5.1 audio support
• two USB 2.0 on i/o shield + 4 on headers
• six USB 3.1 (4 type A, 2 type C) on i/o shield + 4 on headers
• serial console on I/O board

Final price was set to be around 800 EUR with an option to buy motherboard separately from CPU as there are companies where you can buy stuff quite easy if you fit in 500 USD.

I also added something for Openmoko fans — first 50 boards go to FOSS developers (free of cost).

Hints and guesses

Some hints that it is fake were present. Project date was set to 1st April 2020, CPU listed as ARMv8.4 while there are no such ones on a market.

I got several guesses from people. CPU being 16-core Ampere Skylark, mainboard being done by MSI using AMD southbridge for whole I/O:

Using AMD’s southbridges makes a lot of sense anyway, those are really just PCIe-multi-I/O chips anyway. That way, the CPU just needs to be connected to RAM, PCIe and a BIOS flash, while the southbridge can provide USB, SATA, audio, i2c, etc along with additional PCIe lanes.

AFD names

Of course AFD mean April Fools’ Day. For a moment I wanted to use AArch64 For Developers name on website but it did not fit to scheme. I also got Alternativ Für Datencenter as possible name.

Conclusion

It was fun. I may repeat is one day but with a bit more preparation.