First 96boards Enterprise board which will be on a market?

I am at Linaro Connect in Budapest, Hungary. And on Arrow’s stand I noticed something I did not expected — 96boards Enterprise Edition form factor board.

In past Linaro presented ‘Husky’ and ‘Cello’ devboards in 96boards EE form factor. None of them ever reached production. Only few prototypes existed (had some of them in hands). Both products were complete failures.

Systart Oxalis LS1020A got announced about month ago. They target routers, IoT gateways type devices with it.

As you can see board has ports all over the edges but that’s fault of 96boards EE specification which mandate such broken designs. When I saw it first time my question was “where is PCIe slot?” but found out that (according to spec) it is optional. Board has mini-pcie slot on bottom side anyway.

Speaking of design… Oxalis is made from two parts: carrier board and SoM (System on Module). SoM is based on NXP Network Processor QorIQ® LS1012A processor (single ARM Cortex-A53 core running up to 800 MHz) with 64MB of SPI flash (space for bootloader!) and 1GB of memory. Carrier board gives two GbE network ports, two USB 3.0 connectors, standard 96boards header, one SATA port (with power!), microSD and mini-pcie slot (on bottom side).

The beauty of such design is that you can replace CPU board with something different. According to Dieter Kiermaier from Arrow there are plans for other SoM board in future.

Will it be success? Time will show. Will I buy it? Rather not as for my development I need 16GB ram. Will it have case? Not asked. When on market? May/June 2017.

My work on Kolla

During last month I was working on one of OpenStack projects: Kolla. My job was adding support for non-x86 architectures: aarch64 and ppc64le. Also resurrecting Debian support.

A bit of background

At Linaro we work on getting AArch64 (64-bit ARM, arm64) to be present in many places. We have at least two OpenStack instances running at the moment – on AArch64 hardware only.

First we used Debian/jessie and Openstack ‘liberty’ version. Was working. Not best but we helped many projects by providing virtual machines for porting software.

It was built from packages and later (when ‘mitaka’ was released) we moved to virtualenv per component. Out second “cloud” runs that. With proper Neutron networking, live migration and few other nice things.

But virtualenvs were done as quick solution. We decided to move to Docker containers for next release.

And Kolla was chosen as a tool for it. We do not like to reinvent the wheel again and again…

Non-x86 support in Kolla

The problem was typical: Kolla being x86-64 centric. As most of software nowadays. But thanks to work done by Sajauddin Mohammad I had something to use as a base for adding aarch64 support.

I took his patch, slashed out most of it and concentrated on getting minimal changes needed to get something built on AArch64 . Effect was sent for review and is now at 10th version.

Docker images started to appear. But at beginning I was building Ubuntu ones as Debian support was “basically abandoned, on a way out”. From CentOS guys I got confirmation that official Docker image will be generated (it is done already).

I spent some time on making sure that whole non-x86 support is free from any hardcoding wherever possible. As you can see in my working branch it went quite well. Most of arch related changes are related to “distro does not provide package ZYS for that architecture” or to handling of external repositories.

Debian support

And here we come to Debian support. At Linaro we decided to support two community based distributions: CentOS and Debian. But Debian was on a way out in Kolla…

As this was not related much to non-x86 work I decided to use one of x86-64 machines for that stuff.

First builds were against ‘jessie-backports’ base tag. I had to make a patch to tell APT that if I want backports then I really want them. It was sent for review as rest of patches.

Images were building but not so many as for Ubuntu. So I went through all of them and enabled Debian where it was possible. Resulting patch went for review as usual.

Effect was quite nice (on x86-64):

  • debian-binary: 158
  • debian-source: 201

But ‘jessie’ was missing several packages even with backports enabled. So after discussion with my team I decided to drop it and go for Debian/testing ‘stretch’ one instead. It is already frozen for release so no big changes are allowed. Patch in review of course.

At that moment I abandoned one of previous patches as ‘jessie-backports’ was not something I planned to support.

Turned out that ‘stretch’ images have a bit different set of packages installed than ‘jessie’ had. So ‘gnupg’ and ‘dirmngr’ were missing while we need them for importing GPG keys into APT. Proper patch went to review again.

Did rebuild on x86-64:

  • stretch-binary: 137
  • stretch-source: 195

A bit less than ‘jessie-backports’ had, right? Sure, but it also shows that I have to make a new build to check numbers (laptop already has ~1500 docker images generated by kolla).

Cleaning of old Power patch

Remember the patch which all that started from? I did not forgot it and after building all those images I went back to it.

Some parts are just fugly so I skipped them but others were useful if done properly. That’s how new changes were done and some updates to previous ones.

Then I managed to put remote hands on one of Power machines at Red Hat and started builds:

  • debian-binary: 134
  • debian-source: 184
  • ubuntu-binary: 147
  • ubuntu-source: 190

No CentOS builds as there was no centos/ppc64le image available.


Non-x86 support looks quite nice. There are some images which can not be built as they rely on external repositories so no aarch64 nor ppc64le packages to use.

Debian ‘stretch’ support is not perfect yet but it is something which I plan to maintain so situation will be going to improve. Note that most of my work will go into ‘source’ type of builds as we want to have same images for both Debian and CentOS systems.

System calls again

Few months ago I created a page with HTML table. For own use basically. Then presented it to the people and found out that it got useful for them. So started improving and improving so it became side project.

Yes, system calls again. I wrote about it in past but yesterday I rewrote code so it now uses Linux source so I can generate tables for far more architectures without need of other computers (either real or emulated).

Next step was work on presentation layer. Old version was just table with added sorting. Things were ugly when scrolled as header was gone. Now it sticks to the top of page so it is easier to note which column relates to which architecture.

Odd/even lines are coloured now which makes is easier to find numbers for syscall.

And speaking of searching — there is filter box now. You can type syscall name (or part of it) there and have table filtered. Same can be done with system call number as well. You used Valgrind and it said that has no idea how to handle syscall 145? Just enter number and you see that it is getresuid(), nfsservctl(), readv(), sched_getscheduler(), setreuid() or setrlimit() — depends which architecture you are testing.

You wonder what that that system call does? There are links to man pages provided.

Go here to check it out and comment here, open a new issue if you found a bug or would like to colaborate. Patches are welcome.

Linaro Connect: interesting hardware

Before going for Linaro Connect I had a plan to look at all those 96boards devices and write some complains/opinions about them. But it would be like shooting fish in a barrel so I decided against. But there were some interesting pieces of hardware there.

One of them was Macchiatobin board from SolidRun. I think that this is same as their Armada 8040 community board but after design changes. Standard Mini-ITX format, quad core Cortex-A72 cpu (with upto 2GHz clock), one normal DIMM slot (max 16GB, ships with 4GB), three Serial-ATA ports, PCI-Express x4 slot, one USB 3.0 port, microSD slot.

UPDATE: SolidRun confirmed – this is final design of their Armada 8040 community board.

Photo (done by Riku Voipio) shows which goodies are available:

Network interfaces from top to bottom are (if I remember correctly):

  • 10GbE (SFP + RJ-45)
  • 10GbE (SFP + RJ-45)
  • 2.5GbE (SFP)
  • 1GbE (RJ-45)

When it comes to software I was told that board is SBSA compliant so any normal distribution should work. Kernel, bootloaders (U-Boot and UEFI) are mainlined.

Price? 350USD. Looks like nice candidate for AArch64 development platform or NAS.

Other device was Gumstix Nodana 96BCE board which is 96boards complaint carrierboard for Intel Joule modules.

On top it looks like typical 96boards device (except USB C port):

But once reversed Intel Joule module is visible:

This is first non-ARM based 96boards device. Maybe even one of most compliant ones. At least from software perspective because when it comes to hardware then module makes it a bit too thick to fit in 96boards CE specification limits.

Note that 96boards Consumer Electronics specification does not require using ARM or AArch64 cpu.

AArch64 desktop hardware?

Soon there will be four years since I started working on AArch64 architecture. Lot of software things changed during that time. Lot in a hardware too. But machines availability still sucks badly.

In 2012 all we had was software model. It was slow, terribly slow. Common joke was AArch64 developers standing in a queue for 10GHz x86-64 cpus. So I was generating working binaries by using cross compilation. But many distributions only do native builds. In models. Imagine Qt4 building for 3-4 days…

In 2013 I got access to first server hardware. With first silicon version of CPU. Highly unstable, we could use just one core etc. GCC was crashing like hell but we managed to get stable build results from it. Qt4 was building in few hours now.

Then amount of hardware at Red Hat was growing and growing. Farms of APM Mustangs, AMD Seattle and several other servers appeared, got racked and available to use. In 2014 one Mustang even landed on my desk (as first such machine in Poland).

But this was server land. Each of those machines costed about 1000 USD (if not more). And availability was hard too.

Linaro tried to do something about it and created 96boards project.

First came ‘Consumer Edition’ range. Yet another small form factor boards with functionality stripped as much as possible. No Ethernet, no storage other than emmc/usb, low amount of memory, chips taken from mobile phones etc. But it was selling! But only because people were hungry to get ANYTHING with AArch64 cores. First was HiKey then DragonBoard410 got released. Then few other boards. All with same set of issues: non-mainline kernel, weird bootloaders, binary blobs for this or that…

Then so called ‘Enterprise Edition’ got announced. With another ridiculous form factor (and microATX as an option). And that was it. There was a leak of Husky board which shown how fucked up design it was. Ports all around the edges, memory above and under board and of course incompatible with any industrial form factor. I would like to know what they were smoking…

Time passed by. Husky got forgotten for another year. Then Cello was announced as a “new EE 96boards board” while it looked as redesigned Husky with two SATA ports less (because who needs more than two SATA, right?). Last time I heard about Cello it was still ‘maybe soon, maybe another two weeks’. Prototypes looked like hand soldered, USB controller mounted rotated, dead on-board Ethernet etc.

In meantime we got few devices from other companies. Pine64 had big campaign on Kickstarter and shipped to developers. Hardkernel started selling ODROID-C2, Geekbox released their TV box and probably something else got released as well. But all those boards were limited to 1-2GB of memory, often lacked SATA and used mobile processors with their own set of bootloaders etc causing extra work for distributions.

Overdrive 1000 was announced. Without any options for expansion it looked like SoftIron wanted customers to buy Overdrive 3000 if they want to use PCI Express card.

So we have 2016 now. Four years of my work on AArch64 passed. Most of distributions support this architecture by building on proper servers but most of this effort is not used because developers do not have sane hardware to play with (sane means expandable, supported by distributions, capable).

There is no standard form factor mainboards (mini-itx, microATX, ATX) available on mass market. 96boards failed here, server vendors are not interested, small Chinese companies prefer to release yet-another-fruit/Pi with mobile processor. Nothing, null, nada, nic.

Developers know where to buy normal computer cases, storage, memory, graphics cards, USB controllers, SATA controllers and peripherals. So vendors do not have to worry/deal with this part. But still there is nothing to put those cards into. No mainboards which can be mounted into normal PC case, have some graphics plugged in, few SSD/HDD connected, mouse/keyboard, monitors and just be used.

Sometimes it is really hard to convince software developers to make changes for platform they are unable to test on. And current hardware situation does not help. All those projects of hardware being available “in a cloud” helps only for subset of projects — ever tried to run GNOME/KDE session over the network? With OpenGL acceleration etc?

So where is my AArch64 workstation? In desktop or laptop form.

Post written after my Google+ post where similar discussion happened in comments.

My work on changing CirrOS images

What is CirrOS and why I was working on it? This was quite common question when I mentioned what I am working on during last weeks.

So, CirrOS is small image to run in a cloud. OpenStack developers use it to test their projects.

Technically it is yet another Frankenstein OS. Built using Buildroot 2015.05 uses uclibc or glibc (depending on target architecture). Then Ubuntu 16.04 kernel is applied on top and “grub” (also from Ubuntu) is used to make it bootable.

The problem was that it was not done in UEFI bootable way…

My first changes were: switch images to GPT, create EFI system partition and put some bootloader there. I first used CentOS “grub2-efi” packages (as they provided ready to use EFI binaries) and later switched to Ubuntu ones as upstream maintainer (Scott Moser) prefers to have all external binaries to came from one source.

When he was on vacations (so merge request had to wait) I started digging more and more into scripts.

Fixed getopt use as arguments passed between scripts were read partly via getopt, partially by assigning variables to ${X} (where X is a number).

All scripts were moved to use Bash (as /bin/sh in Ubuntu is usually Dash which is minimalist POSIX shell), whitespace got unified between all scripts and some other stuff happened as well.

At one moment all scripts had 1835 lines and my diff was 2250 lines (+1018/-603) long. Hopefully Scott was back and we got most of that stuff merged.

Recent (2016.07.21) images are available and work fine on all platforms. If someone uses them with OpenStack then please remember about setting “short_id” property to “ubuntu16.04” — otherwise there may be a problem with finding rootfs (no virtio-scsi in disk images).


architecture booting before booting after
aarch64 direct kernel UEFI or direct kernel
arm direct kernel UEFI or direct kernel
i386 BIOS or direct kernel BIOS, UEFI or direct kernel
powerpc direct kernel direct kernel
ppc64 direct kernel direct kernel
ppc64le direct kernel direct kernel
x86-64 BIOS or direct kernel BIOS, UEFI or direct kernel

My workflow for building big sets of RPM packages

In last months I did two rebuilds: NodeJS 4.x in Fedora and OpenStack Mitaka in CentOS. Both were targeting AArch64 and both were not done before. During latter one I was asked to write about my workflow, so will describe it with OpenStack one as base.

identify what needs to be done

At first I had to figure out what exactly needs to be built. Haïkel Guémar (aka number80) pointed me to openstack-mitaka directory on CentOS’ vault where all source packages are present. Also told me that EPEL repository is not required which helped a lot as it is not yet built for CentOS.

structure of sources

OpenStack set of packages in CentOS is split into two parts: “common” shared through all OpenStack versions and “openstack-mitaka” containing OpenStack Mitaka packages and build dependencies not covered by CentOS itself or “common” directory.

prepare build space

I used “mockchain” for such rebuilds. It is simple tool which does not do any ordering tricks just builds set of packages in given order and do it three times hoping that all build dependencies will be solved that way. Of course what got built once is not tried again.

To make things easier I used shell alias:

alias runmockchain mockchain -r default -l /home/hrw/rpmbuild/_rebuilds/openstack/mockchain-repo-centos

With this I did not have to remember about those two switches. Common call was “runmockchain –recurse srpms/*” which means “cycle packages three times and continue on failures”.

Results of all builds (packages and logs) were kept in “~/_rebuilds/openstack/mockchain-repo-centos/results/default/” subdirectories. I put all extra packages there to have all in one repository.

populate “noarch” packages

Then I copied x86-64 build of OpenStack Mitaka into “_IMPORT-openstack-mitaka/” to get all “noarch” packages for satisfying build dependencies. I built all those packages anyway but having them saved me several rebuilds.

extra rpm macros

When I started first build it turned out that some Python packages lack proper “Provides” fields. I was missing newer rpm build macros (“%python_provide” was added after Fedora 19 which was base for RHEL7). Asked Haïkel and added “rdo-rpm-macros” to mock configuration.

But had to scrap everything I built so far.

surprises and failures

Building big set of packages for new architecture most of time generate failures which were not present with x86-64 build. Same was this time as several build dependencies were missing or wrong.

packages missing in CentOS/aarch64

Some were from CentOS itself — I told Jim Perrin (aka Evolution) and he added them to build queue to fill gaps. I built them in meantime or (if they were “noarch”) imported into “_IMPORT-extras” or “_IMPORT-os” directories.

packages imported from other CBS tags

Other packages were originally imported from other tags at CBS (CentOS koji). For those I created directory named “_IMPORT-cbs”. And again — if they were “noarch” I just copied them. For rest I did full build (using “runmockchain”) and they end in same repository as rest of build.

For some packages it turned out that they got built long time ago with older versions of build dependencies and are not buildable from current versions. For them I tracked proper versions on CBS and imported/built (sometimes with their build dependencies and build dependencies of build dependencies).

downgrading packages

There was a package “python-flake8” which failed to build spitting out Python errors. I checked how this version was built on CBS and turned out that “python-mock” 1.3.0 (from “openstack-mitaka” repository) was too new… Downgraded it to 1.0 allowed me to build “python-flake8” one (upgrading of it is in queue).

merging fixes from Fedora

Both “galera” and “mariadb-galera” got AArch64 support merged from Fedora and got built with “.hrw1” added into “Release” field.


I did whole build in “~/rpmbuild/_rebuilds/openstack/” directory. Extra folders were:

  • mockchain-repo-centos/results/default/_HACKED-by-hew/
  • mockchain-repo-centos/results/default/_IMPORT-cbs/
  • mockchain-repo-centos/results/default/_IMPORT-extras/
  • mockchain-repo-centos/results/default/_IMPORT-openstack-mitaka/
  • mockchain-repo-centos/results/default/_IMPORT-os/

Vault ones were copy of OpenStack source packages and their build dependencies. Hacked ones got AArch64 support merged from Fedora. Both “extras” and “os” directories were for packages missing in CentOS/AArch64 repositories. CBS one was for source/noarch packages which had to be imported/rebuilt because they came from other CBS tags.

status page

In meantime I prepared web page with build results so anyone interested can see what builds, what not and check logs, packages etc. It has simple description and then table with list of builds (data can be sorted by clicking on column headers).


Whole job would take much more time if not help from CentOS developers: Haïkel Guémar, Alan Pevec, Jim Perrin, Karanbir Singh and others from #centos-devel and #centos-arm IRC channels.

How to speed up mock

Fedora and related distributions use “mock” to build packages. It creates chroot from “root cache” tarball, updates it, installs build dependencies and runs build. Similar to “pbuilder” under Debian. Whole build time can be long but there are some ways to get it faster.

kill fsync()

Everytime something calls “fsync()” storage slow downs because all writes need to be done. Build process goes in a chroot which will be removed at the end so why bother?

Run “dnf install nosync” and then enable it in “/etc/mock/site-defaults.cfg” file:

config_opts['nosync'] = True

local cache for packages

I do lot of builds. Often few builds of same package. And each build requires fetching of RPM packages from external repositories. So why not cache it?

In LAN I have one machine working as NAS. One of services running there is “www cache” with 10GB space which I use only for fetching packages — both in mock and system and I use it on all machines. This way I can recreate build chroot without waiting for external repositories.

config_opts['http_proxy'] = 'http://nas.lan:3128'

Note that this also requires editing mock distribution config files to not use “mirrorlist=” but “baseurl=” instead so same server will be used each time. There is a script to convert mock configuration files if you go that way.

decompress root cache

Mock keeps tarball of base chroot contents which gets unpacked at start of build. By default it is gzip compressed and unpacking takes time. On my systems I switched off compression to gain a bit at cost of storage:

config_opts['plugin_conf']['root_cache_opts']['compress_program'] = ""
config_opts['plugin_conf']['root_cache_opts']['extension'] = ""


If memory is not an issue then tmpfs can be used for builds. Mock has own plugin for it but I do not use it. Instead I decide on my own about mounting “/var/lib/mock” as tmpfs or not. Why? I only have 16GB ram in pinkiepie so 8-12GB can be spent on tmpfs while there are packages which would not fit during build.

parallel decompression

Sources are compressed. Nowadays CPU has more than one core. So why not using it with multithreaded gzip/bzip2 depackers? This time distro file (like “/etc/mock/default.cfg” one) needs to be edited:

config_opts['chroot_setup_cmd'] = 'install @buildsys-build /usr/bin/pigz /usr/bin/lbzip2'
config_opts['macros']['%__gzip'] = '/usr/bin/pigz'
config_opts['macros']['%__bzip2'] = '/usr/bin/lbzip2'

extra mockchain tip

For those who use “mockchain” a lot this shell snippet may help finding which packages failed:

for dir in *
    if [ -e $dir/fail ];then
        rm $dir/fail
        mv $dir _fail-$dir


With this set of changes I can do mock builds faster than before. Hope that it helps someone else too.

Failed to set MokListRT: Invalid Parameter

Somehow I managed to break UEFI environment on APM Mustang. As a result I was not able to enter boot manager menu nor UEFI shell. All I had was booting to 0001 boot entry (which was just installed Fedora 24 alpha).

After reboot I scrolled a bit to take a look at firmware output:

X-Gene Mustang Board
Boot firmware (version 1.1.0 built at 14:50:19 on Oct 20 2015)
TianoCore 1.1.0 UEFI 2.4.0 Oct 20 2015 14:49:32
CPU: APM ARM 64-bit Potenza Rev A3 2400MHz PCP 2400MHz
     SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
Board: X-Gene Mustang Board
Slimpro FW:
        Ver: 2.4 (build 2015/04/22)
        PMD: 950 mV
        SOC: 950 mV
Failed to set MokListRT: Invalid Parameter

Here screen got cleared instantly and grub was shown. I booted into one of installed systems and started playing with EFI boot manager:

17:38 root@pinkiepie-rawhide:~$ efibootmgr 
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0001,0004,0000
Boot0000  Fedora rawhide
Boot0001* Fedora
Boot0004* Red Hat Enterprise Linux

Note “0 seconds” timeout. I changed it to 5s (efibootmgr -t 5), rebooted and UEFI menu appeared again:

TianoCore 1.1.0 UEFI 2.4.0 Oct 20 2015 14:49:32
CPU: APM ARM 64-bit Potenza Rev A3 2400MHz PCP 2400MHz
     SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
Board: X-Gene Mustang Board
Slimpro FW:
        Ver: 2.4 (build 2015/04/22)
        PMD: 950 mV
        SOC: 950 mV
The default boot selection will start in   5 seconds
[1] Fedora rawhide
[2] Red Hat Enterprise Linux
[3] Fedora
[4] Shell
[5] Boot Manager
[6] Reboot
[7] Shutdown

So I can boot whatever I want again ;D

Cello: new AArch64 enterprise board from 96boards project

Few hours ago, somewhere in some hotel in Bangkok Linaro Connect has started. So during morning coffee I watched keynote and noticed that Jon Masters presented RHELSA 7.2 out of the box experience on Huskyboard. And then brand new board from 96boards project was announced: Cello.

Lot of people was expecting that this Linaro Connect will bring Huskyboard alive so people will finally have an option for some cheap board for all their AArch64 needs. Instead Cello was presented:


Compared to Husky (below) there are some hardware differences to notice but it is normal as 96borads enterprise specification only tells where to put ports.


I suppose that both boards were designed by different companies. Maybe it was a request from Linaro to ODM vendors to design and mass produce 96boards enterprise board and Husky was prototyped first but Cello won. Or maybe we will see Husky in distribution as well. Good part is: you can preorder Cello and get it delivered in Q2/2016.

Have to admit that I hoped for some industry standard board (96boards Enterprise specification mentions microATX) instead of this weird 96boards-only format which I ranted about already. Anyway 299 USD for quad-core Cortex-A57 with SATA, UEFI, ACPI, PCIe (and maybe few more four letter acronyms) does not sound bad but good luck with finding case for it ;(