Marcin Juszkiewicz

Two days ago Arm announced Arm v9 architecture. Finally we can discuss it in open instead saying “I prefer not to talk about this” (because NDA etc.).

New things

There are several new things in v9 version. SVE2, Spectre/Meltdown like mitigations, memory tagging, realms… And some of them are present in v8 already (mitigations are v8.5 IIRC).

Hardware

But how it goes work in hardware? There was no new cpu core announcements so we need to wait for Arm Neoverse N2 derived designs (as it will be v9).

As usual mobile phones and tablets will get it first. Then probably Apple will put it into newer Macbooks. Decade later servers and workstations.

And there are always those machines in labs. Packed with NDAs, access queues etc…

$ uname -a
Linux bach 5.11.0-73 #1 SMP Fri Mar 12 11:34:12 UTC 2021 aarch64 GNU/Linux
$ head -n8 /proc/cpuinfo
processor       : 0
BogoMIPS        : 50.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp rme
                  asimdhp cpuid sb16 fp16 asimdrdm uafp jscvt fcma sb128 lrcpc
                  dcpop sha3 sm3 sm4 asimddp afd sha512 sve asimdfhm dit ilrcpc
                  rng flagm tme ssbs paca pacg sb dcpodp ac97 flagm2 frint mte
                  sve2 bf16 bti
CPU implementer : 0x4a
CPU architecture: 9
CPU variant     : 0x0
CPU part        : 0xf02
CPU revision    : 0

And sorry, no benchmarks allowed on not-mass-market hardware. The good part? It is still AArch64 so no recompilation required. Some software packages gets first set of SVE2 improvements soon.

That was only a joke

The truth is that even labs do not have such stuff. While most of features listed in output of /proc/cpuinfo exists in newer Arm v8 cores some of them were added there for fun:

• afd (April Fools’ Day)
• sb16 and sb128 (SoundBlaster 16/128)
• ac97 (yet another sound device)
• uafp (Use-After-Free Protection)

Thanks

I would like to thank a few people.

Arnd Bergmann pointed out that two fields related to CPU are wrong:

• implementer should be ASCII code (so I changed from 0xe3 to 0x4a (‘J’ as in Joke))
• part field is just 12 bits (so changed from 0x1f02 to - 0xf02)

Mark Brown spotted duplicated ‘rng’ feature. And Ryan Houdek who wrote about ‘tme’ feature.

Python wheels… Everyone loves them, many people curse when they are not available for their setup. I am in both groups every time I have to do something more complex with Python.

So today I will show how to build Python wheel in quick way.

What is manylinux?

Linux world has a lot of distributions. Even more when you add their releases into the mix. And they ship different versions of Python. At same time we have Pypi which works as repository of ready to use Python packages.

So manylinux idea was created to have minimal requirements for building Python packages. To make sure that you can install it on any distribution.

So far there were several versions built:

name	base distribution	PEP
manylinux1	CentOS 5	PEP 513
manylinux2010	CentOS 6	PEP 571
manylinux2014	CentOS 7	PEP 599
manylinux_2_24	Debian 9 ‘stretch’	PEP 600

As you see old releases are used to make sure that resulting binaries work on any newer distribution. manylinux2014 added non-x86 architectures (aarch64, ppc64le, s390x).

Each image contains several versions of Python binaries ready to use in /opt/python/ directory.

Manylinux images are distributed as container images on pypa repository on quay.io. Run under Docker, Kubernetes, Podman etc.

Source code is available in ‘manylinux’ repository on GitHub.

Let’s use!

My work requires me to build TensorFlow 1.5.15 version, which depends on NumPy 1.18.* version. None of them are available as wheels for AArch64 architecture.

So let me run container and install NumPy dependencies:

$ docker run -it -u root -v $PWD:/tmp/numpy quay.io/pypa/manylinux2014_aarch64
[root@fa339493a417 /]# cd /tmp/numpy/
[root@fa339493a417 /tmp/numpy/]# yum install -y blas-devel lapack-devel
[root@fa339493a417 /tmp/numpy/]# 

Image has several versions of Python installed and I want to build NumPy 1.18.5 for each of them so my build script is easy:

for py in /opt/python/cp3[6789]*
do
    pyver=`basename $py`
    $py/bin/python -mvenv $pyver
    source $pyver/bin/activate
    pip wheel numpy==1.18.5
    deactivate
done

Result is simple — I got set of wheel files. One per Python version. But it is not the end of work as NumPy libraries depend on blas/lapack we installed into system.

add libraries to the wheel

There is a tool we need to run: “auditwheel”. What it does is inspecting wheel file, all library symbols used, external libraries etc. Then it bundles required libraries into wheel file:

INFO:auditwheel.main_repair:Repairing numpy-1.18.5-cp39-cp39-linux_aarch64.whl
INFO:auditwheel.wheeltools:Previous filename tags: linux_aarch64
INFO:auditwheel.wheeltools:New filename tags: manylinux2014_aarch64
INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp39-cp39-linux_aarch64
INFO:auditwheel.wheeltools:New WHEEL info tags: cp39-cp39-manylinux2014_aarch64
INFO:auditwheel.main_repair:
Fixed-up wheel written to /root/wheelhouse/numpy-1.18.5-cp39-cp39-manylinux2014_aarch64.whl

File size changed from 13 467 772 to 16 806 338 bytes and resulting wheel can be installed on any distribution.

Let summarise

Manylinux is great tool to provide Python packages. It is easy to use on developer’s machine or on CI. And makes life of Python users much easier.

Small board computers (SBC) usually come with U-Boot as firmware. There could be some more components like Arm Trusted Firmware, OPTEE etc but what user interact with is the U-Boot itself.

Since 2016 there is the CONFIG_DISTRO_DEFAULTS option in U-Boot configuration. It selects defaults suitable for booting general purpose Linux distributions. Thanks to it board is able to boot most of OS installers out of the box without any user interaction.

How?

How does it know how to do that? There are several scripts and variables involved. Run “printenv” command in U-Boot shell and there you should see some of them named like “boot_*, bootcmd_* scan_dev_for_*”.

In my example I would use environment from RockPro64 running U-Boot 2021.01 version.

I will prettify all scripts for readability. Script contents may be expanded — in such case I will give name as a comment and then it’s content.

Let’s boot

First variable used by U-Boot is “bootcmd”. It reads it to know how to boot operating system on the board.

In out case this variable has “run distro_bootcmd” in it. So what is there on RockPro64 SBC:

setenv nvme_need_init
for target in ${boot_targets}
do 
    run bootcmd_${target}
done

It says that on-board NVME needs some initialization and then goes through set of scripts using order from “boot_targets” variable. On RockPro64 this variable sets “mmc0 mmc1 nvme0 usb0 pxe dhcp sf0” order which means:

• eMMC
• MicroSD
• NVME
• USB storage
• PXE
• DHCP
• SPI flash

Both eMMC and MicroSD look similar: ‘devnum=X; run mmc_boot’ — set MMC number and then try to boot by running ‘mmc_boot’ script:

if mmc dev ${devnum}; then 
    devtype=mmc; 
    run scan_dev_for_boot_part; 
fi

NVME one initialize PCIe subsystem (via “boot_pci_enum”), then scans for NVME devices (via “nvme_init”) and do the similar stuff (here with expanded scripts):

# boot_pci_enum
pci enum

# nvme_init
if ${nvme_need_init}; then 
    setenv nvme_need_init false;
    nvme scan;
fi

if nvme dev ${devnum}; then 
    devtype=nvme; 
    run scan_dev_for_boot_part; 
fi

USB booting goes with “usb_boot”:

usb start;
if usb dev ${devnum}; then 
    devtype=usb; 
    run scan_dev_for_boot_part;
fi

PXE network boot? Initialize USB, scan PCI, get network configuration, do PXE boot:

# boot_net_usb_start
usb start

# boot_pci_enum
pci enum

dhcp; 
if pxe get; then 
    pxe boot; 
fi

DHCP method feels like last resort one (do not ask me for meaning of all those variables):

# boot_net_usb_start
usb start

# boot_pci_enum
pci enum

if dhcp ${scriptaddr} ${boot_script_dhcp}; then 
    source ${scriptaddr}; 
fi;

setenv efi_fdtfile ${fdtfile}; 
setenv efi_old_vci ${bootp_vci};
setenv efi_old_arch ${bootp_arch};
setenv bootp_vci PXEClient:Arch:00011:UNDI:003000;
setenv bootp_arch 0xb;

if dhcp ${kernel_addr_r}; then 
    tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};

    if fdt addr ${fdt_addr_r}; then 
        bootefi ${kernel_addr_r} ${fdt_addr_r}; 
    else 
        bootefi ${kernel_addr_r} ${fdtcontroladdr};
    fi;
fi;

setenv bootp_vci ${efi_old_vci};
setenv bootp_arch ${efi_old_arch};
setenv efi_fdtfile;
setenv efi_old_arch;
setenv efi_old_vci;

And last method is SPI flash:

busnum=0

if sf probe ${busnum}; then
    devtype=sf;

    # run scan_sf_for_scripts; 
    ${devtype} read ${scriptaddr} ${script_offset_f} ${script_size_f}; 
    source ${scriptaddr}; 
    echo SCRIPT FAILED: continuing...
fi

Search for boot partition

Note how block devices end with one script: “scan_dev_for_boot_part”. What it does is quite simple:

part list ${devtype} ${devnum} -bootable devplist; 
env exists devplist || setenv devplist 1; 

for distro_bootpart in ${devplist}; do 
    if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then 
        run scan_dev_for_boot; 
    fi; 
done; 
setenv devplist

We know type and number of boot device from previous step so now we check for bootable partitions. Which means EFI System Partition for GPT disks and partitions marked as bootable in case of MBR. If none are present then first one is assumed to be bootable one.

Search for distribution boot information

Once we found boot partitions it is time to search for boot stuff with “scan_dev_for_boot” script:

echo Scanning ${devtype} ${devnum}:${distro_bootpart}...;
for prefix in ${boot_prefixes}; do 
    run scan_dev_for_extlinux; 
    run scan_dev_for_scripts; 
done;

run scan_dev_for_efi;

Old style OS configuration

First U-Boot checks for “extlinux/extlinux.conf” file, then go for old style “boot.scr” (in uimg and clear text formats). Both of them are checked in / and /boot/ directories of checked partition (those names are in “boot_prefixes” variable).

Let us look at it:

# scan_dev_for_extlinux
if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf};then 
    echo Found ${prefix}${boot_syslinux_conf}; 

    # run boot_extlinux; 
    sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}

    echo SCRIPT FAILED: continuing...; 
fi

# scan_dev_for_scripts
for script in ${boot_scripts}; do 
    if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then 
        echo Found U-Boot script ${prefix}${script}; 

        # run boot_a_script; 
        load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; 
        source ${scriptaddr}

        echo SCRIPT FAILED: continuing...; 
    fi; 
done

EFI compliant OS

And finally U-Boot checks for EFI style BootOrder variables and generic OS loader path:

# scan_dev_for_efi
setenv efi_fdtfile ${fdtfile};
for prefix in ${efi_dtb_prefixes}; do
    if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then 
        # run load_efi_dtb; 
        load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}
    fi;
done;

# run boot_efi_bootmgr;
if fdt addr ${fdt_addr_r}; then 
    bootefi bootmgr ${fdt_addr_r};
else 
    bootefi bootmgr;
fi

if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootaa64.efi; then
    echo Found EFI removable media binary efi/boot/bootaa64.efi; 

    # run boot_efi_binary; 
    load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootaa64.efi; 
    if fdt addr ${fdt_addr_r}; then 
        bootefi ${kernel_addr_r} ${fdt_addr_r};
    else 
        bootefi ${kernel_addr_r} ${fdtcontroladdr};
    fi

    echo EFI LOAD FAILED: continuing...;
fi; 
setenv efi_fdtfile

Booted

At this moment board should be in either OS or in OS loader (being EFI binary).

Final words

All that work on searching for boot media, boot scripts, boot configuration files, OS loaders, EFI BootOrder entries etc is done without any user interaction. Every bootable media is checked and tried.

If I would add SATA controller support into U-Boot binary then all disks connected to such would also be checked. Without any code/environment changes from my side.

So if your SBC has some weird setup then consider moving to distro generic one. Boot fresh mainline U-Boot, store copy of your existing environment (“printenv” shows it) and then reset to generic one with “env default -a” command. Probably would need to set MAC adresses for network interfaces.

I had some ideas for improvements on website. And finally found some time to implement them.

Series of posts

One of changes was implementing ‘series of posts’ to make it easier to find posts on one topic. For now there are two of them:

• Standards in Arm space
• From the diary of AArch64 porter

Each post in series has a list at the top. I may group some other posts into additional series.

Mobile devices fixes

From time to time I had email from Google bots that some things on my website need improvements. Most of it were about mobile devices. So I went through Lighthouse audits and did some changes:

• top menu is one entry per line
• clickable lists have more padding between entries
• removed ‘popular tags’ group from sidebar as it was not used
• display more entries in ‘recent posts’ sidebar section

About me section

Also added ‘About me’ section to the sidebar. I often give links to my blog posts in several instant messaging channels (IRC, Discord, Telegram) and when people realize that I wrote them there is strange moment:

<hrw> irc_user: https://marcin.juszkiewicz.com.pl/2020/06/17/ebbr-on-rockpro64/

<irc_user> hrw: yes, I know that article, that is why I want to try it, I’ll get the RockPro64 today if all goes well! :)

<irc_user> I wonder whether the images from uboot-images-armv8-2020.10-2.fc33.noarch.rpm also have EBBR support for rockpro64 though, without needing to download a random binary from Marcin’s website :)

<hrw> iirc Fedora images are not built for writing into SPI flash

<hrw> irc_user: and you reminded me that I need to add text block to website

<irc_user> hrw: ah, you are Marcin! :-)

<irc_user> now I feel stupid

<irc_user> excellent blog you have! thanks so much for that, I learned a lot!

<hrw> thx

Now info about my nick name is right at the top of page (unless on mobile).

Useful tables

I also added a list of tables from my side projects:

• BSA and SBSA checklist
• Linux system calls for all architectures

I know that they have some users but now both are more visible.

In the first part I went from board files and ugly bootloaders to SBSA/SBBR and EBBR. The second part went through BSA, BBR etc. TLAs and what changes they brought into OFLAs.

And both were about specifications written for developers (both hardware and software). This time I will write something about ones written for marketing people.

SBSA, SBBR, EBBR, LBBR etc

Who is gonna remember all those acronyms and what they mean? Only those who really have to. Rest of people needs something easier to remember.

ServerReady

Design a System that “Just Works”

In 2018 Arm brought ServerReady program. Name sounds much better than “hardware needs to comply with SBSA and firmware has to be SBBR compliant”, right? Ah, and “has to pass ACS” (which stands for Architectural Compliance Suite).

Yeah — one simple name instead of three acronyms. So imagine situation when you need to convince your boss that project needs a serious AArch64 machine for Continuous Integration builds. You can say “We buy XYZ because it is a Server Ready system” and they assume that it is a server so IT should be able to handle it.

Try to say “we buy XYZ because it is SBSA and SBBR compliant and passes ACS” and you can get asked about your mental health…

SystemReady

But not every AArch64 system is server class hardware. Or needs what whole UEFI, ACPI etc. things in firmware.

So in 2020 Arm came with SystemReady program. It is basically ServerReady renamed and extended to cover wider selection of hardware and firmware options.

It came around same time as BSA, BBR, LBBR etc. which I described in the second part already so will not repeat what those acronyms mean.

certification bands

There are four ‘bands’ defined:

Certification	Description	hardware specs	firmware spec
SystemReady SR	ServerReady	BSA + SBSA	SBBR
SystemReady ES	Embedded Server (*)	BSA	SBBR
SystemReady IR	IoT Ready	BSA	EBBR
SystemReady LS	LinuxBoot ServerReady	BSA + SBSA	LBBR

*) spec says “Embedded ServerReady” but it is probably an error as it was also mentioned as “Embedded Server” in few places outside of specification.

What that means for developers?

Certification	hardware type	usual firmware
SystemReady SR	server class	UEFI + ACPI
SystemReady ES	some SBC	UEFI + ACPI
SystemReady IR	some SBC	(UEFI or U-Boot) + DTB
SystemReady LS	server class	LinuxBoot

Where UEFI means Tianocore EDK2 or similar. And U-Boot needs EFI layer enabled (to fulfill EBBR requirements).

recertification

SystemReady specification says that SR systems are also ES compliant. There is no need for recertification if someone wants to put other sticker.

There are changes in progress. One of them is Devicetree requirement for IR band. So not every ES will be compliant with IR unless firmware be changed.

BTW — specification mentions 32-bit systems. But IoT Ready only as they are not covered by BSA.

Conclusion

Creation of ServerReady and later SystemReady specifications was good move. We got simple name which can be understood by mere mortals.

Developers and other interested people can go deeper and read about BSA, BBR, EBBR, LBBR, SBBR, SBSA and other TLAs and OFLAs.

Recently I had a discussion with A13 and realized that people may have different perception of how AArch64 boards work:

Sahaj told me that you can just install generic images on honeycomb

it kinda blows my mind

How did we got to that point?

Servers are boring, right?

I started working on AArch64 in 2012. First in fast models written by Arm developers, then also in QEMU. Both used direct kernel boot method without any firmware or bootloaders.

In 2013 I moved from Canonical/Linaro to Red Hat. And there we got server from Applied Micro. I do not remember how it booted as I used it for building software. Some time later we had Mustangs and all of them were booting UEFI.

Then I got Mustang at home. Fedora, RHEL were booting fine. Then CentOS and Debian joined. All of them used grub-efi like my x86-64 desktop or laptop.

Time passed, I got other servers to work with. HPe M400, ThunderX, ThunderX2, Falkor, D05 etc. Each of them was running UEFI. Either Tianocore based or commercial one.

And to install operating system all I needed was to boot generic install media.

SBC hell

At same time SBC world was fighting with users. Each vendor/SoC/board had to be treated specially as there was no way to store firmware on board (as SPI flash is very expensive).

So depending on SBC your firmware could be written either:

• at some special offset from start of microSD card
• at the beginning of a partition of special type
• in a file on vfat partition of any type
• in a file on EFI System Partition (also using vfat)

Some offsets forced the use of “obsolete” MBR partitioning as there was no space for GPT information. While UEFI systems require GPT not MBR.

It also generated lot of wrong information like “this file needs to be named in UPPERCASE (on case insensitive filesystem)” or “needs to be first file written to a partition”. Some kind of “SBC boot voodoo”.

So each SBC required its own boot media — you could not take it to a board with some other SoC and expect it to start. Or you spend some time to create some kind of hybrid image which had a few bootloaders written. Easier way was to prepare a separate boot media images per SBC.

From time to time there was SBC with onboard flash available for storing firmware. Some people made use of it, others continued doing offset crap as they were used to it.

SBBR, EBBR came

Last years brought us several specifications from Arm. First was SBBR which stands for Server Base Boot Requirements. It said which features should be present in firmware (you can read more in my previous post about Arm standards).

As SBCs are not servers, a new specification was created for them: EBBR (E means Embedded). It basically says “try to follow what server does” and has some requirements either dropped or relaxed.

Both were designed to make distribution’s life easier. Never mind is it BSD, Linux or Microsoft Windows — they have to put EFI bootloader (like Grub-efi) in EFI System Partition and system will boot on any supported SBBR/EBBR hardware.

For example I have a USB pendrive with Debian “bullseye” installed. It boots fine on RockPro64 and Espressobin SBCs (both have EBBR compliant U-Boot stored in on-board flash) and on Mustang and HoneyComb (both with SBBR compliant UEFI in on-board flash).

Habits. Good, bad, forced.

So it looks like the way how AArch64 system should boot depends on what your habits are.

When you started from servers then SBBR/EBBR way is your way and you look weird at most of SBC systems with their offsets and “other mumbo jumbo”.

If all you used were SBC then going into SBBR/EBBR world can be “zOMG, it just magically works!”.

Note to SBC vendors

Most SBCs already follow the EBBR standard or can easily be made compliant. Never mind you are using mainline U-Boot or some own fork (and then consider upstreaming as board’s life may be longer than you expect).

Enable the CONFIG_DISTRO_DEFAULTS option in the config. Build U-Boot, store it to the board and boot. Then erase whatever environment you used before with “env default -a” command.

On next reboot your SBC will iterate over “boot_targets” variable and check for few standard boot files:

• extlinux/extlinux.conf
• boot.scr.uimg
• boot.scr
• /efi/boot/bootaa64.efi

When it gets something then it handles that and boots. If not then goes to another boot target.

This allows to handle basically every operating system used on Arm systems. And allows to boot generic install ISO (as long as OS on it supports the device).

Bonus points if your SBC has some on board flash or eMMC it can boot from. Then firmware can be stored there so user does not even have to worry about it.

Few years ago SolidRun released MACCHIATObin board. Nice fast cpu, PCI Express slot, several network ports. I did not buy it because it supported only 16 GB of memory and I wanted to be able to run OpenStack.

Time has passed, HoneyComb LX2 system appeared on AArch64 market. More cores, more memory. Again I haven’t bought it — my Ryzen 5 upgrade costed less than HoneyComb price is.

And when someone asked me for some serious AArch64 system to buy I was suggesting HoneyComb.

Let us look at hardware

So what do we have here?

• 16 Cortex-A72 cores
• 2 SO-DIMM slots (up to 64GB ram in total)
• USB 2.0 and 3.0 ports (as ports and/or headers)
• standard ATX power socket (no 12V AUX needed)
• 3 fan connectors (one with PWM, two with 12V)
• front panel connectors like on x86-64 motherboards
• M.2 slot for NVME (pcie x4)
• PCI Express slot (open x8 one so x16 card fits)
• MicroSD slot (for firmware)
• 4 SFP+ ports for 10GbE networking
• 1 GbE port
• 4 SATA ports
• serial console via microUSB port
• power/reset buttons

Lot of networking and there is even version with 100GbE port added: ClearFog CX LX2.

So how I got it?

I wrote that I did not bought it, right? Jon Nettleton (from SolidRun) contacted me recently and asked:

Morning. do you have any interest in a HoneyComb? I have some old stock boards available to the community. I figured it may help you out with your UEFI Qemu work.

We discussed about SBSA/SBBR stuff and I sent him an email with address information and shipping notes.

Some days passed and board arrived. I added spare NVME and two sticks of Kingston HyperX 2933 CL17 memory and it was ready to go (microsd card keeps firmware):

Let’s run something

Debian ‘bullseye’ booted right away. Again I used pendrive from my EBBR compliant RockPro64. Started without problems.

Network ports issue

Ok, there was one problem — on-board ethernet ports do not work yet with mainline nor distribution kernels so I had to dig out my old USB based network card.

There are patches for Linux kernel to get all ports running. May get merged into 5.13 kernel if things go nicely.

Plans?

I plan few things for HoneyComb:

• check several distributions how they handle AArch64 systems
• improve SBSA ACS code as HoneyComb is almost SBSA level 3 compliant (there are some places where error/warning messages break output)
• build, deploy and test OpenStack
• test software
• check how it works as AArch64 desktop (like I did with APM Mustang 6 years ago)

SBBR or GTFO

Me.

Yeah, right. But world is not so nice and there are many cheap SBC on market which are not SBBR compliant and probably never will. And with small amount of work they can do EBBR (Embedded Base Boot Requirements).

NOTE: I have similar post about EBBR on RockPro64 board.

WTH is EBBR?

It is specification for devices which are not servers and do not pretend to be such. U-Boot is all they have and with properly configured one they have some subset of EFI Boot/Runtime Services to load distribution bootloader (grub-efi usually) like it is done on servers.

ACPI is not required but may be present. DeviceTree is perfectly fine. You may provide both or one of them.

Firmware can be stored wherever you wish. Even MBR partitioning is available if really needed.

Few words about board itself

EspressoBin has 4MB of SPI flash on board. Less than on RockPro64 but still enough for storing firmware (U-Boot takes less than 1MB).

This SBC is nothing new — first version was released in 2016. There were several revisions with different memory type, amount of ram chips, type of them (ddr3 or ddr4), CPU speed and some more changes.

I got EspressoBin revision 5 with 1GB ram of ddr3 in 2 chips. And 1GHz processor.

It may sound silly that I repeat that information but it matters when you start building firmware for that board.

So let us build fresh firmware

This is Marvell so abandon all hope for sanity.

Thanks to Arm Trusted Firmware authors there is good documentation on how to build firmware for EspressoBin which guides step by step and explains all arguments you need. For me it was several git clone calls and then two make calls:

make -C u-boot CROSS_COMPILE=aarch64-linux-gnu- \
mvebu_espressobin-88f3720_defconfig u-boot.bin -j12

make -C trusted-firmware-a CROSS_COMPILE=aarch64-linux-gnu- \
CROSS_CM3=arm-none-linux-gnueabihf- PLAT=a3700 \
CLOCKSPRESET=CPU_1000_DDR_800 DDR_TOPOLOGY=2 \
MV_DDR_PATH=$PWD/marvell/mv-ddr-marvell/ \
WTP=$PWD/marvell/A3700-utils-marvell/ \
CRYPTOPP_PATH=$PWD/marvell/cryptopp/ \
BL33=$PWD/u-boot/u-boot.bin \
mrvl_flash -j12

And I had to install cross toolchain for 32-bit arm because the one I had was for building kernels/bootloaders only.

Is your U-Boot friendly or not?

First you need to check which version of U-Boot and hardware you have. Then check does it recognize SPI flash or not:

Marvell>> sf probe
SF: unrecognized JEDEC id bytes: 9d, 70, 16
Failed to initialize SPI flash at 0:0 (error -2)

I had bad luck as my board used SPI chip not recognized by any official U-Boot build…

Armbian to the rescue

I asked in few places did anyone had some experiences with this board. One of them was #debian-arm IRC channel where I got hint from Xogium that Armbian may have U-Boot builds.

And they have whole page about EspressoBin. With information how to choose firmware files etc.

So I downloaded archive with proper files for UART recovery. The important thing to remember is that once you move jumpers and load all firmware files over serial they are not written in SPI flash so reset of board means you start over.

Quick check is SPI flash detected:

Marvell>> sf probe
SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB

Yeah! Now can start USB and flash own firmware build:

Marvell>> bubt flash-image.bin spi usb
Burning U-Boot image "flash-image.bin" from "usb" to "spi"
Bus usb@58000: Register 2000104 NbrPorts 2
Starting the controller
USB XHCI 1.00
Bus usb@5e000: USB EHCI 1.00
scanning bus usb@58000 for devices... 1 USB Device(s) found
scanning bus usb@5e000 for devices... 2 USB Device(s) found
Image checksum...OK!
SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB
Erasing 991232 bytes (242 blocks) at offset 0 ...Done!
Writing 990944 bytes from 0x6000000 to offset 0 ...Done!

Quick reset and board boots to fresh, mainline U-Boot:

TIM-1.0
WTMI-devel-18.12.1-1a13f2f
WTMI: system early-init
SVC REV: 5, CPU VDD voltage: 1.108V
NOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
NOTICE:  BL1: Built : 17:11:19, Feb 15 2021
NOTICE:  BL1: Booting BL2
NOTICE:  BL2: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
NOTICE:  BL2: Built : 17:11:20, Feb 15 2021
NOTICE:  BL1: Booting BL31
NOTICE:  BL31: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
NOTICE:  BL31: Built : 18:07:02, Feb 15 2021


U-Boot 2021.01 (Feb 15 2021 - 19:25:41 +0100)

DRAM:  1 GiB
Comphy-0: USB3_HOST0    5 Gbps    
Comphy-1: PEX0          2.5 Gbps  
Comphy-2: SATA0         5 Gbps    
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs 
PCIE-0: Link down
MMC:   sdhci@d0000: 0, sdhci@d8000: 1
Loading Environment from SPIFlash... SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB
OK
Model: Globalscale Marvell ESPRESSOBin Board
Card did not respond to voltage select! : -110
Net:   eth0: neta@30000
Hit any key to stop autoboot:  0 

Final steps

OK, so SBC has fresh, mainline firmware. Nice. But still some stuff needs to be done.

First note MAC addresses of Ethernet ports. Use printenv command to check stored environment and note few variables:

eth1addr=f0:ad:4b:aa:97:01
eth2addr=f0:ad:4b:aa:97:02
eth3addr=f0:ad:4b:aa:97:03
ethaddr=f0:ad:4e:72:10:ef

Of course you may also skip that step and rely on random ones or choose own ones (I had router with C0:FF:EE:C0:FF:EE in past).

Then reset environment to default values stored in U-Boot binary and set those MAC addresses by hand:

=> env default -a -f
=> setenv eth1addr f0:ad:4b:aa:97:01
=> setenv eth2addr f0:ad:4b:aa:97:02
=> setenv eth3addr f0:ad:4b:aa:97:03
=> setenv ethaddr f0:ad:4e:72:10:ef
=> saveenv
Saving Environment to SPIFlash... Erasing SPI flash...Writing to SPI flash...done
OK
=> 

What EBBR brings?

Now your board is ready to boot Debian, Fedora and several other distribution install media with two commands:

=> set boot_targets usb0
=> boot

It will find EFI bootloader and start it. Just like on any other boring SBBR/EBBR system.

Distributions with old style ‘boot.scr’ script (like OpenWRT for example) will also work so no functionality loss.