Marcin Juszkiewicz

Recently I had a discussion with A13 and realized that people may have different perception of how AArch64 boards work:

Sahaj told me that you can just install generic images on honeycomb

it kinda blows my mind

How did we got to that point?

Servers are boring, right?

I started working on AArch64 in 2012. First in fast models written by Arm developers, then also in QEMU. Both used direct kernel boot method without any firmware or bootloaders.

In 2013 I moved from Canonical/Linaro to Red Hat. And there we got server from Applied Micro. I do not remember how it booted as I used it for building software. Some time later we had Mustangs and all of them were booting UEFI.

Then I got Mustang at home. Fedora, RHEL were booting fine. Then CentOS and Debian joined. All of them used grub-efi like my x86-64 desktop or laptop.

Time passed, I got other servers to work with. HPe M400, ThunderX, ThunderX2, Falkor, D05 etc. Each of them was running UEFI. Either Tianocore based or commercial one.

And to install operating system all I needed was to boot generic install media.

SBC hell

At same time SBC world was fighting with users. Each vendor/SoC/board had to be treated specially as there was no way to store firmware on board (as SPI flash is very expensive).

So depending on SBC your firmware could be written either:

• at some special offset from start of microSD card
• at the beginning of a partition of special type
• in a file on vfat partition of any type
• in a file on EFI System Partition (also using vfat)

Some offsets forced the use of “obsolete” MBR partitioning as there was no space for GPT information. While UEFI systems require GPT not MBR.

It also generated lot of wrong information like “this file needs to be named in UPPERCASE (on case insensitive filesystem)” or “needs to be first file written to a partition”. Some kind of “SBC boot voodoo”.

So each SBC required its own boot media — you could not take it to a board with some other SoC and expect it to start. Or you spend some time to create some kind of hybrid image which had a few bootloaders written. Easier way was to prepare a separate boot media images per SBC.

From time to time there was SBC with onboard flash available for storing firmware. Some people made use of it, others continued doing offset crap as they were used to it.

SBBR, EBBR came

Last years brought us several specifications from Arm. First was SBBR which stands for Server Base Boot Requirements. It said which features should be present in firmware (you can read more in my previous post about Arm standards).

As SBCs are not servers, a new specification was created for them: EBBR (E means Embedded). It basically says “try to follow what server does” and has some requirements either dropped or relaxed.

Both were designed to make distribution’s life easier. Never mind is it BSD, Linux or Microsoft Windows — they have to put EFI bootloader (like Grub-efi) in EFI System Partition and system will boot on any supported SBBR/EBBR hardware.

For example I have a USB pendrive with Debian “bullseye” installed. It boots fine on RockPro64 and Espressobin SBCs (both have EBBR compliant U-Boot stored in on-board flash) and on Mustang and HoneyComb (both with SBBR compliant UEFI in on-board flash).

Habits. Good, bad, forced.

So it looks like the way how AArch64 system should boot depends on what your habits are.

When you started from servers then SBBR/EBBR way is your way and you look weird at most of SBC systems with their offsets and “other mumbo jumbo”.

If all you used were SBC then going into SBBR/EBBR world can be “zOMG, it just magically works!”.

Note to SBC vendors

Most SBCs already follow the EBBR standard or can easily be made compliant. Never mind you are using mainline U-Boot or some own fork (and then consider upstreaming as board’s life may be longer than you expect).

Enable the CONFIG_DISTRO_DEFAULTS option in the config. Build U-Boot, store it to the board and boot. Then erase whatever environment you used before with “env default -a” command.

On next reboot your SBC will iterate over “boot_targets” variable and check for few standard boot files:

• extlinux/extlinux.conf
• boot.scr.uimg
• boot.scr
• /efi/boot/bootaa64.efi

When it gets something then it handles that and boots. If not then goes to another boot target.

This allows to handle basically every operating system used on Arm systems. And allows to boot generic install ISO (as long as OS on it supports the device).

Bonus points if your SBC has some on board flash or eMMC it can boot from. Then firmware can be stored there so user does not even have to worry about it.

Few years ago SolidRun released MACCHIATObin board. Nice fast cpu, PCI Express slot, several network ports. I did not buy it because it supported only 16 GB of memory and I wanted to be able to run OpenStack.

Time has passed, HoneyComb LX2 system appeared on AArch64 market. More cores, more memory. Again I haven’t bought it — my Ryzen 5 upgrade costed less than HoneyComb price is.

And when someone asked me for some serious AArch64 system to buy I was suggesting HoneyComb.

Let us look at hardware

So what do we have here?

• 16 Cortex-A72 cores
• 2 SO-DIMM slots (up to 64GB ram in total)
• USB 2.0 and 3.0 ports (as ports and/or headers)
• standard ATX power socket (no 12V AUX needed)
• 3 fan connectors (one with PWM, two with 12V)
• front panel connectors like on x86-64 motherboards
• M.2 slot for NVME (pcie x4)
• PCI Express slot (open x8 one so x16 card fits)
• MicroSD slot (for firmware)
• 4 SFP+ ports for 10GbE networking
• 1 GbE port
• 4 SATA ports
• serial console via microUSB port
• power/reset buttons

Lot of networking and there is even version with 100GbE port added: ClearFog CX LX2.

So how I got it?

I wrote that I did not bought it, right? Jon Nettleton (from SolidRun) contacted me recently and asked:

Morning. do you have any interest in a HoneyComb? I have some old stock boards available to the community. I figured it may help you out with your UEFI Qemu work.

We discussed about SBSA/SBBR stuff and I sent him an email with address information and shipping notes.

Some days passed and board arrived. I added spare NVME and two sticks of Kingston HyperX 2933 CL17 memory and it was ready to go (microsd card keeps firmware):

Let’s run something

Debian ‘bullseye’ booted right away. Again I used pendrive from my EBBR compliant RockPro64. Started without problems.

Network ports issue

Ok, there was one problem — on-board ethernet ports do not work yet with mainline nor distribution kernels so I had to dig out my old USB based network card.

There are patches for Linux kernel to get all ports running. May get merged into 5.13 kernel if things go nicely.

Plans?

I plan few things for HoneyComb:

• check several distributions how they handle AArch64 systems
• improve SBSA ACS code as HoneyComb is almost SBSA level 3 compliant (there are some places where error/warning messages break output)
• build, deploy and test OpenStack
• test software
• check how it works as AArch64 desktop (like I did with APM Mustang 6 years ago)

SBBR or GTFO

Me.

Yeah, right. But world is not so nice and there are many cheap SBC on market which are not SBBR compliant and probably never will. And with small amount of work they can do EBBR (Embedded Base Boot Requirements).

NOTE: I have similar post about EBBR on RockPro64 board.

WTH is EBBR?

It is specification for devices which are not servers and do not pretend to be such. U-Boot is all they have and with properly configured one they have some subset of EFI Boot/Runtime Services to load distribution bootloader (grub-efi usually) like it is done on servers.

ACPI is not required but may be present. DeviceTree is perfectly fine. You may provide both or one of them.

Firmware can be stored wherever you wish. Even MBR partitioning is available if really needed.

Few words about board itself

EspressoBin has 4MB of SPI flash on board. Less than on RockPro64 but still enough for storing firmware (U-Boot takes less than 1MB).

This SBC is nothing new — first version was released in 2016. There were several revisions with different memory type, amount of ram chips, type of them (ddr3 or ddr4), CPU speed and some more changes.

I got EspressoBin revision 5 with 1GB ram of ddr3 in 2 chips. And 1GHz processor.

It may sound silly that I repeat that information but it matters when you start building firmware for that board.

So let us build fresh firmware

This is Marvell so abandon all hope for sanity.

Thanks to Arm Trusted Firmware authors there is good documentation on how to build firmware for EspressoBin which guides step by step and explains all arguments you need. For me it was several git clone calls and then two make calls:

make -C u-boot CROSS_COMPILE=aarch64-linux-gnu- \
mvebu_espressobin-88f3720_defconfig u-boot.bin -j12

make -C trusted-firmware-a CROSS_COMPILE=aarch64-linux-gnu- \
CROSS_CM3=arm-none-linux-gnueabihf- PLAT=a3700 \
CLOCKSPRESET=CPU_1000_DDR_800 DDR_TOPOLOGY=2 \
MV_DDR_PATH=$PWD/marvell/mv-ddr-marvell/ \
WTP=$PWD/marvell/A3700-utils-marvell/ \
CRYPTOPP_PATH=$PWD/marvell/cryptopp/ \
BL33=$PWD/u-boot/u-boot.bin \
mrvl_flash -j12

And I had to install cross toolchain for 32-bit arm because the one I had was for building kernels/bootloaders only.

Is your U-Boot friendly or not?

First you need to check which version of U-Boot and hardware you have. Then check does it recognize SPI flash or not:

Marvell>> sf probe
SF: unrecognized JEDEC id bytes: 9d, 70, 16
Failed to initialize SPI flash at 0:0 (error -2)

I had bad luck as my board used SPI chip not recognized by any official U-Boot build…

Armbian to the rescue

I asked in few places did anyone had some experiences with this board. One of them was #debian-arm IRC channel where I got hint from Xogium that Armbian may have U-Boot builds.

And they have whole page about EspressoBin. With information how to choose firmware files etc.

So I downloaded archive with proper files for UART recovery. The important thing to remember is that once you move jumpers and load all firmware files over serial they are not written in SPI flash so reset of board means you start over.

Quick check is SPI flash detected:

Marvell>> sf probe
SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB

Yeah! Now can start USB and flash own firmware build:

Marvell>> bubt flash-image.bin spi usb
Burning U-Boot image "flash-image.bin" from "usb" to "spi"
Bus usb@58000: Register 2000104 NbrPorts 2
Starting the controller
USB XHCI 1.00
Bus usb@5e000: USB EHCI 1.00
scanning bus usb@58000 for devices... 1 USB Device(s) found
scanning bus usb@5e000 for devices... 2 USB Device(s) found
Image checksum...OK!
SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB
Erasing 991232 bytes (242 blocks) at offset 0 ...Done!
Writing 990944 bytes from 0x6000000 to offset 0 ...Done!

Quick reset and board boots to fresh, mainline U-Boot:

TIM-1.0
WTMI-devel-18.12.1-1a13f2f
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.108V
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
NOTICE:  BL1: Built : 17:11:19, Feb 15 2021
NOTICE:  BL1: Booting BL2
NOTICE:  BL2: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
NOTICE:  BL2: Built : 17:11:20, Feb 15 2021
NOTICE:  BL1: Booting BL31
NOTICE:  BL31: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
NOTICE:  BL31: Built : 18:07:02, Feb 15 2021


U-Boot 2021.01 (Feb 15 2021 - 19:25:41 +0100)

DRAM:  1 GiB
Comphy-0: USB3_HOST0    5 Gbps    
Comphy-1: PEX0          2.5 Gbps  
Comphy-2: SATA0         5 Gbps    
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs 
PCIE-0: Link down
MMC:   sdhci@d0000: 0, sdhci@d8000: 1
Loading Environment from SPIFlash... SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB
OK
Model: Globalscale Marvell ESPRESSOBin Board
Card did not respond to voltage select! : -110
Net:   eth0: neta@30000
Hit any key to stop autoboot:  0 

Final steps

OK, so SBC has fresh, mainline firmware. Nice. But still some stuff needs to be done.

First note MAC addresses of Ethernet ports. Use printenv command to check stored environment and note few variables:

eth1addr=f0:ad:4b:aa:97:01
eth2addr=f0:ad:4b:aa:97:02
eth3addr=f0:ad:4b:aa:97:03
ethaddr=f0:ad:4e:72:10:ef

Of course you may also skip that step and rely on random ones or choose own ones (I had router with C0:FF:EE:C0:FF:EE in past).

Then reset environment to default values stored in U-Boot binary and set those MAC addresses by hand:

=> env default -a -f
=> setenv eth1addr f0:ad:4b:aa:97:01
=> setenv eth2addr f0:ad:4b:aa:97:02
=> setenv eth3addr f0:ad:4b:aa:97:03
=> setenv ethaddr f0:ad:4e:72:10:ef
=> saveenv
Saving Environment to SPIFlash... Erasing SPI flash...Writing to SPI flash...done
OK
=> 

What EBBR brings?

Now your board is ready to boot Debian, Fedora and several other distribution install media with two commands:

=> set boot_targets usb0
=> boot

It will find EFI bootloader and start it. Just like on any other boring SBBR/EBBR system.

Distributions with old style ‘boot.scr’ script (like OpenWRT for example) will also work so no functionality loss.

My family got used to the fact that I am not available at beginning of February. Because of my FOSDEM trip. This year was not both not so different and different at same time.

Due to COVID-19 pandemic FOSDEM 2021 was online. So there was no reason for any trip other than to local shops to buy some Belgian beers. And I was not available to anyone during weekend.

Online? It will be terrible!

During 2020 I attended several online events. For some of them I prefer to not remember that I did it. Terrible recordings of talk, some had bandwidth issues. On some organizers did not managed to get presenters agree on online presence so some meetings had to be dropped when they were supposed to start.

Matrix to the rescue

FOSDEM team decided to organize some way for attendees to chat with each other. Matrix was setup on chat.fosdem.org with some rooms for basic FOSDEM stuff, room for each devroom, rooms for continuing discussion after talk… There was also “Virtual Janson Bar” one for food/drink discussions (renamed to “Virtual Delirium” for after conference hours).

Cloakroom had own channel, food trucks had another one. People were sending photos of their food/waffles/beer/etc.

FOSDEM @fosdem tweeted:

Oh and before you leave don’t forget to pickup your luggage and coat at the cloackroom https://chat.fosdem.org/#/room/#cloakroom:fosdem.org 🙃

Nearly every room had widget with Jitsi for video chat. It was used mostly for Q&A sessions and after talk discussions.

And it worked good. There were hot discussions with questions asked during talk and then replied during Q&A session. Links to many projects and interesting additional pages were posted that way.

Streaming all the way

One of things FOSDEM is famous for is networking at event. And live streaming of all rooms. This year was no different. My monitor’s screen was split to two Firefox windows: left side kept discussions on Matrix server, right side had live streaming schedule and video of currently attended talk. At same time my phone has “FOSDEM Companion” app started with bookmarks opened to make it easy to check which talks I wanted to see.

At some moments I had two videos started — one waited for start of presentation and second with some other talk running. Once new one started I closed watched one. Simple method of watching part of talk to see is it interesting or not.

There were some talks where I dropped during first few minutes and moved to other one. Something quite hard to do when you are in a middle of a room at normal FOSDEM.

Videos of talks will be available during next few days. I have a page with FOSDEM talks with slides/video links which will get updates during next days.

At same time at ULB…

Normally FOSDEM takes place in Brussels at ULB. There were some attendees there so we had messages like that on Matrix:

I went to tram and is was nearly empty. Did I messed timezones or what?

I got permission from Luilegeant to post some pictures from ULB so you may see what we missed this year:

Looks like typical FOSDEM weather ;D

Some final words

I enjoyed FOSDEM 2021. It was different that usual ones but adding Matrix for chat allowed to get that feeling of being with other attendees. I hope that online events in 2021 will copy it.

Shin Ice @_ShinIce tweeted

#FOSDEM21 is coming to an end and it was awesome as always 🤘

we have an estimation of ~33.6k attendees for the conference and ~20k attendees for today…impressive…and now take this numbers and try to fill the ULB 😱

Compare that with usual 8-9 thousands in previous years.

There were many people from other continents taking part in conference just because being online allowed them. For several of them it was night time all the time.

See you at ULB in 2022!

We have made Arm servers boring.

Jon Masters

Standards are boring. Satisfied users may not want to migrate to other boards the market tries to sell them.

So Arm market is flooded with piles of small board computers (SBC). Often they are compliant to standards only when it comes to connectors.

But our hardware is not standard

It is not a matter of ‘let produce UEFI ready hardware’ but rather ‘let write EDK2 firmware for boards we already have’.

Look at Raspberry/Pi then. It is shitty hardware but got popular. And group of people wrote UEFI firmware for it. Probably without vendor support even.

Start with EBBR

Each new board should be EBBR compliant at start. Which is easy — do ‘whatever hardware’ and put properly configured U-Boot on it. Upstreaming support for your small device should not be hard as you often base on some already existing hardware.

Add 16MB of SPI flash to store firmware. Your users will be able to boot ISO without wondering where on boot media they need to write bootloaders.

Then work on EDK2 for board. Do SMBIOS (easy) and keep your existing Device Tree. You are still EBBR. Remember about upstreaming your work — some people will complain, some will improve your code.

Add ACPI, go SBBR

Next step is moving from Device Tree to ACPI. May take some time to understand why there are so many tables and what ASL is. But as several other systems show it can be done.

And this brings you to SBBR compliance. Or SystemReady ES if you like marketing.

SBSA for future design

Doing new SoC tends to be “let us take previous one and improve a bit”. So this time change it a bit and make your next SoC compliant with SBSA level 3. All needed components are probably already included in your Arm license.

Grab EDK2 support you did for previous board. Look at QEMU SBSA Reference Platform support, look at other SBSA compliant hardware. Copy, reuse their drivers, their code.

Was it worth?

At the end you will have SBSA compliant hardware running SBBR compliant firmware.

Congratulations, your board is SystemReady SR compliant. Your marketing team may write that you are on same list as Ampere with their Altra server.

Users buy your hardware and can install whatever BSD, Linux distribution they want. Some will experiment with Microsoft Windows. Others may work on porting Haiku or other exotic operating system.

But none of them will have to think “how to get this shit running”. And they will tell friends that your device is as boring as it should be when it comes to running OS on it == more sales.

Applied Micro Mustang uses standard Mini-ITX form factor just like many PC mainboards. The problem is that contrary to PC ones it does not come with I/O plate. So I decided to make one using 3d printer.

First version

Loaded Tinkercad page, did quick measurements and sent STL to my brother-in-law Szymon for printing. That’s how v1 was born.

It was not good:

As you may see several ports were misaligned (or too small). So I moved ports a bit. The other problem was thickness — turned out that 1mm of plastic was too weak. Second version got printed as 2mm thick:

Other filament made it look ugly. And shows that my measurements were wrong.

FreeCAD

My brother-in-law uses FreeCAD for his designs so I decided to recreate model of I/O plate in this tool. There was some cursing involved as their approach to doing holes in objects. On the other side positioning of holes was much easier to do.

There were several changes and versions were moving fast. I decided to not put any 3d text on model for several reasons:

• it looked shitty on dark filament
• low resolution made it look even worse
• FreeCAD way of doing 3d text feels overcomplicated

Fifth version was 1mm shorter as I had to add space for screws in case:

At this moment I left APM Mustang for my brother-in-law to make changes easier.

In meantime we got some hints from people more involved in 3d printing and decided to do some other changes to how model is done.

Mesh design

FreeCAD has this idea of ‘mesh design’ - you select object, create mesh of it and then export it as STL. The problem is that this part is completely broken under Ubuntu (used by Szymon) — it creates mesh elements but with 0 points/faces.

Seventh version

6th version existed for a moment and then 7th came as first one using mesh:

Not that it changed much ;D

Final one

Few more iterations, another set of measurements and finally we got version which went into use:

It is still far from being perfect but does it’s job. As 3d printing n00b I have no idea why there are vents around holes for ports. Maybe something wrong in printer setup or slicer configuration. Suggestions are welcome.

One button is for power, other for reset as case had just one button on front panel (which I do not use).

Files to download (MIT licensed):

• FreeCAD
• STL

The old joke says that there are two types of people:

• those who make backups
• those will will make backups

I was in second group long time ago and then moved to first one.

Duplicity

So far I used Duplicity along with it’s fronted called Duply. It was installed by default on Ubuntu systems and was quite easy to setup.

But getting files restored from it was pain. And consistency was a problem. Especially when I wanted to restore one directory from quite old copy — turned out that one file (of many) was damaged so whole backup was useless…

BorgBackup

So I looked for alternatives and BorgBackup was one of suggestions.

Features list was long, FOSS, several platforms, compression etc. There were two things which brought my attention:

• deduplication
• mountable backups with FUSE

Deduplication

So why deduplication? Because it allows me to backup several machines into one place and separate copies of git repositories, source code will not take extra space:

                       Original size      Compressed size    Deduplicated size
All archives:                2.02 TB              1.57 TB            118.85 GB
                       Unique chunks         Total chunks
Chunk index:                 1060634             18670536

Or on my server where backups are run hourly:

                       Original size      Compressed size    Deduplicated size
All archives:                7.29 TB              5.36 TB             11.35 GB
                       Unique chunks         Total chunks
Chunk index:                  359783            260310747

Nice difference compared to Duplicity I used before.

FUSE mounting of backups

Instead of checking which options I have to use to restore that one directory from 2 months old backup I can now mount each backup using FUSE:

borgmatic mount --archive puchatek-2021-01-01 --mount-point /tmp/del/1

And then just copy whatever file(s) I want to restore. Very handy way.

Borgmatic

What Duply was to Duplicity, Borgmatic is to BorgBackup. Simple, easy to use frontend hiding most of internals behind easy to use command line interface.

Configure once, then simple commands like “borgmatic info”, “borgmatic prune” etc. All needed things are stored in config file.

Will it work for me?

Time will show does it work for me. Machines’ backup copies are done, syncing them between machines need some improvements.

As you know I have a page with list of Linux system calls which I usually update every rc1 release.

Recently Pavel Šnajdr asked me on Twitter:

Could you also track the first kernel version supporting each syscall? That’d make the table a lot more useful for assembler/calling_it_raw junkies :)

I decided to give it a try.

Decided to check every -rc1 release as this is point in history when most of new code is already merged. First kernel version with rc1 tag was v3.0-rc1 one. Small shell script later I got tables for each rc1 release between 3.0-rc1 and 5.11-rc1 and started digging.

There were at least 80 system calls added in this time range (my method of checking was far from being perfect).

Most active kernel versions

Most active was 5.1 kernel with 24 new ones — mostly due to y2038 problem (21 time64 related calls added).

Then 5.2 kernel added 6 new ones in filesystem handling.

3.5 kernel had some Alpha related activity with 5 osf_* calls added of *stat* family. Also ‘kcmp’ appeared then.

There were few kernel releases with 3 new system calls:

• 3.17 added getrandom(), memfd_create() and seccomp()
• 4.9 gave us pkey_alloc(), pkey_free() and pkey_mprotect()
• 5.3 added clone3(), fp_udfiex_crtl() and pidfd_open()

Two new calls were present in:

kernel release	system calls
3.2	process_vm_readv() and process_vm_writev()
3.3	cache_sync() and mq_getsetaddr()
3.8	finit_module() and kern_features()
3.9	arc_gettls() and arc_settls()
3.14	sched_getattr() and sched_setattr()
4.3	membarrier() and userfaultfd()
4.6	preadv2() and pwritev2()
4.18	io_pgetevents() and rseq()
5.6	openat2() and pidfd_getfd()

Sixteen kernel releases brought one system call each.

Architecture specific calls

Other arch specific entries:

architecture	system calls
Alpha	osf_*
ARC	arc_gettls() and arc_settls()
Arm	arm_fadvise64_64()
OpenRISC	or1k_atomic()
S/390	s390_runtime_instr() and s390_sthyi()

Those are usually hardware specific. Some get their name changed — for example Arm had arm_sync_file_range() which was later renamed to sync_file_range2() and used on several of other architectures.

I do wonder why Alpha still has all those osf_* calls. Do people still use OSF/1 emulation there?

Summary

Most of kernel releases bring some new system call(s). Usually it takes 2-3 releases for most architectures to keep up. Looks like there are just 153 (of 595) calls supported on all archs (but this is affected by fact that I keep information also for architectures removed from Linux kernel).

Some system calls get dropped or renamed. I drop those from my table as well.

My work on this table will continue. It does not take much of my time and provides useful information to several FOSS projects.