1. I got HoneyComb

    Few years ago SolidRun released MACCHIATObin board. Nice fast cpu, PCI Express slot, several network ports. I did not buy it because it supported only 16 GB of memory and I wanted to be able to run OpenStack.

    Time has passed, HoneyComb LX2 system appeared on AArch64 market. More cores, more memory. Again I haven’t bought it — my Ryzen 5 upgrade costed less than HoneyComb price is.

    And when someone asked me for some serious AArch64 system to buy I was suggesting HoneyComb.

    Let us look at hardware

    So what do we have here?

    • 16 Cortex-A72 cores
    • 2 SO-DIMM slots (up to 64GB ram in total)
    • USB 2.0 and 3.0 ports (as ports and/or headers)
    • standard ATX power socket (no 12V AUX needed)
    • 3 fan connectors (one with PWM, two with 12V)
    • front panel connectors like on x86-64 motherboards
    • M.2 slot for NVME (pcie x4)
    • PCI Express slot (open x8 one so x16 card fits)
    • MicroSD slot (for firmware)
    • 4 SFP+ ports for 10GbE networking
    • 1 GbE port
    • 4 SATA ports
    • serial console via microUSB port
    • power/reset buttons
    HoneyComb board layout
    HoneyComb board layout

    Lot of networking and there is even version with 100GbE port added: ClearFog CX LX2.

    So how I got it?

    I wrote that I did not bought it, right? Jon Nettleton (from SolidRun) contacted me recently and asked:

    Morning. do you have any interest in a HoneyComb? I have some old stock boards available to the community. I figured it may help you out with your UEFI Qemu work.

    We discussed about SBSA/SBBR stuff and I sent him an email with address information and shipping notes.

    Some days passed and board arrived. I added spare NVME and two sticks of Kingston HyperX 2933 CL17 memory and it was ready to go (microsd card keeps firmware):

    HoneyComb board ready to go
    HoneyComb board ready to go

    Let’s run something

    Debian ‘bullseye’ booted right away. Again I used pendrive from my EBBR compliant RockPro64. Started without problems.

    Network ports issue

    Ok, there was one problem — on-board ethernet ports do not work yet with mainline nor distribution kernels so I had to dig out my old USB based network card.

    There are patches for Linux kernel to get all ports running. May get merged into 5.13 kernel if things go nicely.


    I plan few things for HoneyComb:

    • check several distributions how they handle AArch64 systems
    • improve SBSA ACS code as HoneyComb is almost SBSA level 3 compliant (there are some places where error/warning messages break output)
    • build, deploy and test OpenStack
    • test software
    • check how it works as AArch64 desktop (like I did with APM Mustang 6 years ago)
    Written by Marcin Juszkiewicz on
  2. EBBR on EspressoBin

    SBBR or GTFO


    Yeah, right. But world is not so nice and there are many cheap SBC on market which are not SBBR compliant and probably never will. And with small amount of work they can do EBBR (Embedded Base Boot Requirements).

    NOTE: I have similar post about EBBR on RockPro64 board.

    WTH is EBBR?

    It is specification for devices which are not servers and do not pretend to be such. U-Boot is all they have and with properly configured one they have some subset of EFI Boot/Runtime Services to load distribution bootloader (grub-efi usually) like it is done on servers.

    ACPI is not required but may be present. DeviceTree is perfectly fine. You may provide both or one of them.

    Firmware can be stored wherever you wish. Even MBR partitioning is available if really needed.

    Few words about board itself

    EspressoBin has 4MB of SPI flash on board. Less than on RockPro64 but still enough for storing firmware (U-Boot takes less than 1MB).

    This SBC is nothing new — first version was released in 2016. There were several revisions with different memory type, amount of ram chips, type of them (ddr3 or ddr4), CPU speed and some more changes.

    I got EspressoBin revision 5 with 1GB ram of ddr3 in 2 chips. And 1GHz processor.

    It may sound silly that I repeat that information but it matters when you start building firmware for that board.

    So let us build fresh firmware

    This is Marvell so abandon all hope for sanity.

    Thanks to Arm Trusted Firmware authors there is good documentation on how to build firmware for EspressoBin which guides step by step and explains all arguments you need. For me it was several git clone calls and then two make calls:

    make -C u-boot CROSS_COMPILE=aarch64-linux-gnu- \
    mvebu_espressobin-88f3720_defconfig u-boot.bin -j12
    make -C trusted-firmware-a CROSS_COMPILE=aarch64-linux-gnu- \
    CROSS_CM3=arm-none-linux-gnueabihf- PLAT=a3700 \
    MV_DDR_PATH=$PWD/marvell/mv-ddr-marvell/ \
    WTP=$PWD/marvell/A3700-utils-marvell/ \
    CRYPTOPP_PATH=$PWD/marvell/cryptopp/ \
    BL33=$PWD/u-boot/u-boot.bin \
    mrvl_flash -j12

    And I had to install cross toolchain for 32-bit arm because the one I had was for building kernels/bootloaders only.

    Is your U-Boot friendly or not?

    First you need to check which version of U-Boot and hardware you have. Then check does it recognize SPI flash or not:

    Marvell>> sf probe
    SF: unrecognized JEDEC id bytes: 9d, 70, 16
    Failed to initialize SPI flash at 0:0 (error -2)

    I had bad luck as my board used SPI chip not recognized by any official U-Boot build…

    Armbian to the rescue

    I asked in few places did anyone had some experiences with this board. One of them was #debian-arm IRC channel where I got hint from Xogium that Armbian may have U-Boot builds.

    And they have whole page about EspressoBin. With information how to choose firmware files etc.

    So I downloaded archive with proper files for UART recovery. The important thing to remember is that once you move jumpers and load all firmware files over serial they are not written in SPI flash so reset of board means you start over.

    Quick check is SPI flash detected:

    Marvell>> sf probe
    SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB

    Yeah! Now can start USB and flash own firmware build:

    Marvell>> bubt flash-image.bin spi usb
    Burning U-Boot image "flash-image.bin" from "usb" to "spi"
    Bus usb@58000: Register 2000104 NbrPorts 2
    Starting the controller
    USB XHCI 1.00
    Bus usb@5e000: USB EHCI 1.00
    scanning bus usb@58000 for devices... 1 USB Device(s) found
    scanning bus usb@5e000 for devices... 2 USB Device(s) found
    Image checksum...OK!
    SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB
    Erasing 991232 bytes (242 blocks) at offset 0 ...Done!
    Writing 990944 bytes from 0x6000000 to offset 0 ...Done!

    Quick reset and board boots to fresh, mainline U-Boot:

    WTMI: system early-init
    SVC REV: 5, CPU VDD voltage: 1.108V
    NOTICE:  Booting Trusted Firmware
    NOTICE:  BL1: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
    NOTICE:  BL1: Built : 17:11:19, Feb 15 2021
    NOTICE:  BL1: Booting BL2
    NOTICE:  BL2: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
    NOTICE:  BL2: Built : 17:11:20, Feb 15 2021
    NOTICE:  BL1: Booting BL31
    NOTICE:  BL31: v2.4(release):v2.4-345-g04c122310 (Marvell-devel-18.12.2)
    NOTICE:  BL31: Built : 18:07:02, Feb 15 2021
    U-Boot 2021.01 (Feb 15 2021 - 19:25:41 +0100)
    DRAM:  1 GiB
    Comphy-0: USB3_HOST0    5 Gbps    
    Comphy-1: PEX0          2.5 Gbps  
    Comphy-2: SATA0         5 Gbps    
    SATA link 0 timeout.
    AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
    flags: ncq led only pmp fbss pio slum part sxs 
    PCIE-0: Link down
    MMC:   sdhci@d0000: 0, sdhci@d8000: 1
    Loading Environment from SPIFlash... SF: Detected is25wp032 with page size 256 Bytes, erase size 4 KiB, total 4 MiB
    Model: Globalscale Marvell ESPRESSOBin Board
    Card did not respond to voltage select! : -110
    Net:   eth0: neta@30000
    Hit any key to stop autoboot:  0 

    Final steps

    OK, so SBC has fresh, mainline firmware. Nice. But still some stuff needs to be done.

    First note MAC addresses of Ethernet ports. Use printenv command to check stored environment and note few variables:


    Of course you may also skip that step and rely on random ones or choose own ones (I had router with C0:FF:EE:C0:FF:EE in past).

    Then reset environment to default values stored in U-Boot binary and set those MAC addresses by hand:

    => env default -a -f
    => setenv eth1addr f0:ad:4b:aa:97:01
    => setenv eth2addr f0:ad:4b:aa:97:02
    => setenv eth3addr f0:ad:4b:aa:97:03
    => setenv ethaddr f0:ad:4e:72:10:ef
    => saveenv
    Saving Environment to SPIFlash... Erasing SPI flash...Writing to SPI flash...done

    What EBBR brings?

    Now your board is ready to boot Debian, Fedora and several other distribution install media with two commands:

    => set boot_targets usb0
    => boot

    It will find EFI bootloader and start it. Just like on any other boring SBBR/EBBR system.

    Distributions with old style ‘boot.scr’ script (like OpenWRT for example) will also work so no functionality loss.

    Written by Marcin Juszkiewicz on
  3. FOSDEM 2021 was the best online event ever

    My family got used to the fact that I am not available at beginning of February. Because of my FOSDEM trip. This year was not both not so different and different at same time.

    Due to COVID-19 pandemic FOSDEM 2021 was online. So there was no reason for any trip other than to local shops to buy some Belgian beers. And I was not available to anyone during weekend.

    Online? It will be terrible!

    During 2020 I attended several online events. For some of them I prefer to not remember that I did it. Terrible recordings of talk, some had bandwidth issues. On some organizers did not managed to get presenters agree on online presence so some meetings had to be dropped when they were supposed to start.

    Matrix to the rescue

    FOSDEM team decided to organize some way for attendees to chat with each other. Matrix was setup on chat.fosdem.org with some rooms for basic FOSDEM stuff, room for each devroom, rooms for continuing discussion after talk… There was also “Virtual Janson Bar” one for food/drink discussions (renamed to “Virtual Delirium” for after conference hours).

    Cloakroom had own channel, food trucks had another one. People were sending photos of their food/waffles/beer/etc.

    FOSDEM @fosdem tweeted:

    Oh and before you leave don’t forget to pickup your luggage and coat at the cloackroom https://chat.fosdem.org/#/room/#cloakroom:fosdem.org 🙃

    Nearly every room had widget with Jitsi for video chat. It was used mostly for Q&A sessions and after talk discussions.

    And it worked good. There were hot discussions with questions asked during talk and then replied during Q&A session. Links to many projects and interesting additional pages were posted that way.

    Streaming all the way

    One of things FOSDEM is famous for is networking at event. And live streaming of all rooms. This year was no different. My monitor’s screen was split to two Firefox windows: left side kept discussions on Matrix server, right side had live streaming schedule and video of currently attended talk. At same time my phone has “FOSDEM Companion” app started with bookmarks opened to make it easy to check which talks I wanted to see.

    At some moments I had two videos started — one waited for start of presentation and second with some other talk running. Once new one started I closed watched one. Simple method of watching part of talk to see is it interesting or not.

    There were some talks where I dropped during first few minutes and moved to other one. Something quite hard to do when you are in a middle of a room at normal FOSDEM.

    Videos of talks will be available during next few days. I have a page with FOSDEM talks with slides/video links which will get updates during next days.

    At same time at ULB

    Normally FOSDEM takes place in Brussels at ULB. There were some attendees there so we had messages like that on Matrix:

    I went to tram and is was nearly empty. Did I messed timezones or what?

    I got permission from Luilegeant to post some pictures from ULB so you may see what we missed this year:

    FOSDEM 2021
    FOSDEM 2021
    Entry to J building
    Entry to J building
    Janson bar was closed
    Janson bar was closed
    crane instead of food trucks
    crane instead of food trucks

    Looks like typical FOSDEM weather ;D

    Some final words

    I enjoyed FOSDEM 2021. It was different that usual ones but adding Matrix for chat allowed to get that feeling of being with other attendees. I hope that online events in 2021 will copy it.

    Shin Ice @_ShinIce tweeted

    #FOSDEM21 is coming to an end and it was awesome as always 🤘

    we have an estimation of ~33.6k attendees for the conference and ~20k attendees for today…impressive…and now take this numbers and try to fill the ULB 😱

    Compare that with usual 8-9 thousands in previous years.

    There were many people from other continents taking part in conference just because being online allowed them. For several of them it was night time all the time.

    See you at ULB in 2022!

    Written by Marcin Juszkiewicz on
  4. Standards are boring

    We have made Arm servers boring.

    Jon Masters

    Standards are boring. Satisfied users may not want to migrate to other boards the market tries to sell them.

    So Arm market is flooded with piles of small board computers (SBC). Often they are compliant to standards only when it comes to connectors.

    But our hardware is not standard

    It is not a matter of ‘let produce UEFI ready hardware’ but rather ‘let write EDK2 firmware for boards we already have’.

    Look at Raspberry/Pi then. It is shitty hardware but got popular. And group of people wrote UEFI firmware for it. Probably without vendor support even.

    Start with EBBR

    Each new board should be EBBR compliant at start. Which is easy — do ‘whatever hardware’ and put properly configured U-Boot on it. Upstreaming support for your small device should not be hard as you often base on some already existing hardware.

    Add 16MB of SPI flash to store firmware. Your users will be able to boot ISO without wondering where on boot media they need to write bootloaders.

    Then work on EDK2 for board. Do SMBIOS (easy) and keep your existing Device Tree. You are still EBBR. Remember about upstreaming your work — some people will complain, some will improve your code.

    Add ACPI, go SBBR

    Next step is moving from Device Tree to ACPI. May take some time to understand why there are so many tables and what ASL is. But as several other systems show it can be done.

    And this brings you to SBBR compliance. Or SystemReady ES if you like marketing.

    SBSA for future design

    Doing new SoC tends to be “let us take previous one and improve a bit”. So this time change it a bit and make your next SoC compliant with SBSA level 3. All needed components are probably already included in your Arm license.

    Grab EDK2 support you did for previous board. Look at QEMU SBSA Reference Platform support, look at other SBSA compliant hardware. Copy, reuse their drivers, their code.

    Was it worth?

    At the end you will have SBSA compliant hardware running SBBR compliant firmware.

    Congratulations, your board is SystemReady SR compliant. Your marketing team may write that you are on same list as Ampere with their Altra server.

    Users buy your hardware and can install whatever BSD, Linux distribution they want. Some will experiment with Microsoft Windows. Others may work on porting Haiku or other exotic operating system.

    But none of them will have to think “how to get this shit running”. And they will tell friends that your device is as boring as it should be when it comes to running OS on it == more sales.

    Written by Marcin Juszkiewicz on
  5. I/O plate for APM Mustang

    Applied Micro Mustang uses standard Mini-ITX form factor just like many PC mainboards. The problem is that contrary to PC ones it does not come with I/O plate. So I decided to make one using 3d printer.

    First version

    Loaded Tinkercad page, did quick measurements and sent STL to my brother-in-law Szymon for printing. That’s how v1 was born.

    It was not good:


    As you may see several ports were misaligned (or too small). So I moved ports a bit. The other problem was thickness — turned out that 1mm of plastic was too weak. Second version got printed as 2mm thick:


    Other filament made it look ugly. And shows that my measurements were wrong.


    My brother-in-law uses FreeCAD for his designs so I decided to recreate model of I/O plate in this tool. There was some cursing involved as their approach to doing holes in objects. On the other side positioning of holes was much easier to do.

    There were several changes and versions were moving fast. I decided to not put any 3d text on model for several reasons:

    • it looked shitty on dark filament
    • low resolution made it look even worse
    • FreeCAD way of doing 3d text feels overcomplicated

    Fifth version was 1mm shorter as I had to add space for screws in case:


    At this moment I left APM Mustang for my brother-in-law to make changes easier.

    In meantime we got some hints from people more involved in 3d printing and decided to do some other changes to how model is done.

    Mesh design

    FreeCAD has this idea of ‘mesh design’ - you select object, create mesh of it and then export it as STL. The problem is that this part is completely broken under Ubuntu (used by Szymon) — it creates mesh elements but with 0 points/faces.

    Seventh version

    6th version existed for a moment and then 7th came as first one using mesh:


    Not that it changed much ;D

    Final one

    Few more iterations, another set of measurements and finally we got version which went into use:


    It is still far from being perfect but does it’s job. As 3d printing n00b I have no idea why there are vents around holes for ports. Maybe something wrong in printer setup or slicer configuration. Suggestions are welcome.

    One button is for power, other for reset as case had just one button on front panel (which I do not use).

    Files to download (MIT licensed):

    Written by Marcin Juszkiewicz on
  6. Switched to BorgBackup

    The old joke says that there are two types of people:

    • those who make backups
    • those will will make backups

    I was in second group long time ago and then moved to first one.


    So far I used Duplicity along with it’s fronted called Duply. It was installed by default on Ubuntu systems and was quite easy to setup.

    But getting files restored from it was pain. And consistency was a problem. Especially when I wanted to restore one directory from quite old copy — turned out that one file (of many) was damaged so whole backup was useless…


    So I looked for alternatives and BorgBackup was one of suggestions.

    Features list was long, FOSS, several platforms, compression etc. There were two things which brought my attention:

    • deduplication
    • mountable backups with FUSE


    So why deduplication? Because it allows me to backup several machines into one place and separate copies of git repositories, source code will not take extra space:

                           Original size      Compressed size    Deduplicated size
    All archives:                2.02 TB              1.57 TB            118.85 GB
                           Unique chunks         Total chunks
    Chunk index:                 1060634             18670536

    Or on my server where backups are run hourly:

                           Original size      Compressed size    Deduplicated size
    All archives:                7.29 TB              5.36 TB             11.35 GB
                           Unique chunks         Total chunks
    Chunk index:                  359783            260310747

    Nice difference compared to Duplicity I used before.

    FUSE mounting of backups

    Instead of checking which options I have to use to restore that one directory from 2 months old backup I can now mount each backup using FUSE:

    borgmatic mount --archive puchatek-2021-01-01 --mount-point /tmp/del/1

    And then just copy whatever file(s) I want to restore. Very handy way.


    What Duply was to Duplicity, Borgmatic is to BorgBackup. Simple, easy to use frontend hiding most of internals behind easy to use command line interface.

    Configure once, then simple commands like “borgmatic info”, “borgmatic prune” etc. All needed things are stored in config file.

    Will it work for me?

    Time will show does it work for me. Machines’ backup copies are done, syncing them between machines need some improvements.

    Written by Marcin Juszkiewicz on
  7. System calls by kernel version

    As you know I have a page with list of Linux system calls which I usually update every rc1 release.

    Recently Pavel Šnajdr asked me on Twitter:

    Could you also track the first kernel version supporting each syscall? That’d make the table a lot more useful for assembler/calling_it_raw junkies :)

    I decided to give it a try.

    Decided to check every -rc1 release as this is point in history when most of new code is already merged. First kernel version with rc1 tag was v3.0-rc1 one. Small shell script later I got tables for each rc1 release between 3.0-rc1 and 5.11-rc1 and started digging.

    There were at least 80 system calls added in this time range (my method of checking was far from being perfect).

    Most active kernel versions

    Most active was 5.1 kernel with 24 new ones — mostly due to y2038 problem (21 time64 related calls added).

    Then 5.2 kernel added 6 new ones in filesystem handling.

    3.5 kernel had some Alpha related activity with 5 osf_* calls added of *stat* family. Also ‘kcmp’ appeared then.

    There were few kernel releases with 3 new system calls:

    • 3.17 added getrandom(), memfd_create() and seccomp()
    • 4.9 gave us pkey_alloc(), pkey_free() and pkey_mprotect()
    • 5.3 added clone3(), fp_udfiex_crtl() and pidfd_open()

    Two new calls were present in:

    kernel release system calls
    3.2 process_vm_readv() and process_vm_writev()
    3.3 cache_sync() and mq_getsetaddr()
    3.8 finit_module() and kern_features()
    3.9 arc_gettls() and arc_settls()
    3.14 sched_getattr() and sched_setattr()
    4.3 membarrier() and userfaultfd()
    4.6 preadv2() and pwritev2()
    4.18 io_pgetevents() and rseq()
    5.6 openat2() and pidfd_getfd()

    Sixteen kernel releases brought one system call each.

    Architecture specific calls

    Other arch specific entries:

    architecture system calls
    Alpha osf_*
    ARC arc_gettls() and arc_settls()
    Arm arm_fadvise64_64()
    OpenRISC or1k_atomic()
    S/390 s390_runtime_instr() and s390_sthyi()

    Those are usually hardware specific. Some get their name changed — for example Arm had arm_sync_file_range() which was later renamed to sync_file_range2() and used on several of other architectures.

    I do wonder why Alpha still has all those osf_* calls. Do people still use OSF/1 emulation there?


    Most of kernel releases bring some new system call(s). Usually it takes 2-3 releases for most architectures to keep up. Looks like there are just 153 (of 595) calls supported on all archs (but this is affected by fact that I keep information also for architectures removed from Linux kernel).

    Some system calls get dropped or renamed. I drop those from my table as well.

    My work on this table will continue. It does not take much of my time and provides useful information to several FOSS projects.

    Written by Marcin Juszkiewicz on
  8. So long, and thanks for all the fun

    During last days I tried to get my Applied Micro Mustang running again. And it looks like it is no more. Like that Norwegian Blue parrot.

    Tried some things

    By default Mustang outputs information on serial console. It does not here. Checked serial cables, serial to usb dongles. Nothing.

    Tried to load firmware from SD card instead of on-board flash. Nope.

    Time to put it to rest.

    How it looked

    When I got it in June 2014 it came in 1U server case. With several loud fans. Including one on cpu radiator. So I took the board out and put into PC Tower case. Also replaced 50mm processor fan with 80mm one:

    Top view of Mustang
    Top view of Mustang
    Side view
    Side view

    All that development…

    I did several things on it:

    Some of them were done for first time on AArch64.

    Board gave me lot of fun. I built countless software packages on it. For CentOS, Debian, Fedora, RHEL. Tested installers of each of them.

    Was running OpenStack on it since ‘liberty’ (especially after moving from 16GB to 32GB ram).

    What next?

    I am going to frame it. With few other devices which helped me during my career.


    It would be nice to replace Mustang with some newer AArch64 hardware. From what is available on mass market SolidRun HoneyComb looks closest. But I will wait for something with Armv8.4 cores to be able to play with nested virtualization.

    UPDATE: I got it working again.

    Written by Marcin Juszkiewicz on
Page 6 / 105