My workflow for building big sets of RPM packages

In last months I did two rebuilds: NodeJS 4.x in Fedora and OpenStack Mitaka in CentOS. Both were targeting AArch64 and both were not done before. During latter one I was asked to write about my workflow, so will describe it with OpenStack one as base.

identify what needs to be done

At first I had to figure out what exactly needs to be built. Haïkel Guémar (aka number80) pointed me to openstack-mitaka directory on CentOS’ vault where all source packages are present. Also told me that EPEL repository is not required which helped a lot as it is not yet built for CentOS.

structure of sources

OpenStack set of packages in CentOS is split into two parts: “common” shared through all OpenStack versions and “openstack-mitaka” containing OpenStack Mitaka packages and build dependencies not covered by CentOS itself or “common” directory.

prepare build space

I used “mockchain” for such rebuilds. It is simple tool which does not do any ordering tricks just builds set of packages in given order and do it three times hoping that all build dependencies will be solved that way. Of course what got built once is not tried again.

To make things easier I used shell alias:

alias runmockchain mockchain -r default -l /home/hrw/rpmbuild/_rebuilds/openstack/mockchain-repo-centos

With this I did not have to remember about those two switches. Common call was “runmockchain –recurse srpms/*” which means “cycle packages three times and continue on failures”.

Results of all builds (packages and logs) were kept in “~/_rebuilds/openstack/mockchain-repo-centos/results/default/” subdirectories. I put all extra packages there to have all in one repository.

populate “noarch” packages

Then I copied x86-64 build of OpenStack Mitaka into “_IMPORT-openstack-mitaka/” to get all “noarch” packages for satisfying build dependencies. I built all those packages anyway but having them saved me several rebuilds.

extra rpm macros

When I started first build it turned out that some Python packages lack proper “Provides” fields. I was missing newer rpm build macros (“%python_provide” was added after Fedora 19 which was base for RHEL7). Asked Haïkel and added “rdo-rpm-macros” to mock configuration.

But had to scrap everything I built so far.

surprises and failures

Building big set of packages for new architecture most of time generate failures which were not present with x86-64 build. Same was this time as several build dependencies were missing or wrong.

packages missing in CentOS/aarch64

Some were from CentOS itself — I told Jim Perrin (aka Evolution) and he added them to build queue to fill gaps. I built them in meantime or (if they were “noarch”) imported into “_IMPORT-extras” or “_IMPORT-os” directories.

packages imported from other CBS tags

Other packages were originally imported from other tags at CBS (CentOS koji). For those I created directory named “_IMPORT-cbs”. And again — if they were “noarch” I just copied them. For rest I did full build (using “runmockchain”) and they end in same repository as rest of build.

For some packages it turned out that they got built long time ago with older versions of build dependencies and are not buildable from current versions. For them I tracked proper versions on CBS and imported/built (sometimes with their build dependencies and build dependencies of build dependencies).

downgrading packages

There was a package “python-flake8” which failed to build spitting out Python errors. I checked how this version was built on CBS and turned out that “python-mock” 1.3.0 (from “openstack-mitaka” repository) was too new… Downgraded it to 1.0 allowed me to build “python-flake8” one (upgrading of it is in queue).

merging fixes from Fedora

Both “galera” and “mariadb-galera” got AArch64 support merged from Fedora and got built with “.hrw1” added into “Release” field.

directories

I did whole build in “~/rpmbuild/_rebuilds/openstack/” directory. Extra folders were:

  • vault.centos.org/centos/7/cloud/Source/openstack-mitaka/common/
  • vault.centos.org/centos/7/cloud/Source/openstack-mitaka/
  • mockchain-repo-centos/results/default/_HACKED-by-hew/
  • mockchain-repo-centos/results/default/_IMPORT-cbs/
  • mockchain-repo-centos/results/default/_IMPORT-extras/
  • mockchain-repo-centos/results/default/_IMPORT-openstack-mitaka/
  • mockchain-repo-centos/results/default/_IMPORT-os/

Vault ones were copy of OpenStack source packages and their build dependencies. Hacked ones got AArch64 support merged from Fedora. Both “extras” and “os” directories were for packages missing in CentOS/AArch64 repositories. CBS one was for source/noarch packages which had to be imported/rebuilt because they came from other CBS tags.

status page

In meantime I prepared web page with build results so anyone interested can see what builds, what not and check logs, packages etc. It has simple description and then table with list of builds (data can be sorted by clicking on column headers).

thanks

Whole job would take much more time if not help from CentOS developers: Haïkel Guémar, Alan Pevec, Jim Perrin, Karanbir Singh and others from #centos-devel and #centos-arm IRC channels.

How to speed up mock

Fedora and related distributions use “mock” to build packages. It creates chroot from “root cache” tarball, updates it, installs build dependencies and runs build. Similar to “pbuilder” under Debian. Whole build time can be long but there are some ways to get it faster.

kill fsync()

Everytime something calls “fsync()” storage slow downs because all writes need to be done. Build process goes in a chroot which will be removed at the end so why bother?

Run “dnf install nosync” and then enable it in “/etc/mock/site-defaults.cfg” file:

config_opts['nosync'] = True

local cache for packages

I do lot of builds. Often few builds of same package. And each build requires fetching of RPM packages from external repositories. So why not cache it?

In LAN I have one machine working as NAS. One of services running there is “www cache” with 10GB space which I use only for fetching packages — both in mock and system and I use it on all machines. This way I can recreate build chroot without waiting for external repositories.

config_opts['http_proxy'] = 'http://nas.lan:3128'

Note that this also requires editing mock distribution config files to not use “mirrorlist=” but “baseurl=” instead so same server will be used each time. There is a script to convert mock configuration files if you go that way.

decompress root cache

Mock keeps tarball of base chroot contents which gets unpacked at start of build. By default it is gzip compressed and unpacking takes time. On my systems I switched off compression to gain a bit at cost of storage:

config_opts['plugin_conf']['root_cache_opts']['compress_program'] = ""
config_opts['plugin_conf']['root_cache_opts']['extension'] = ""

tmpfs

If memory is not an issue then tmpfs can be used for builds. Mock has own plugin for it but I do not use it. Instead I decide on my own about mounting “/var/lib/mock” as tmpfs or not. Why? I only have 16GB ram in pinkiepie so 8-12GB can be spent on tmpfs while there are packages which would not fit during build.

parallel decompression

Sources are compressed. Nowadays CPU has more than one core. So why not using it with multithreaded gzip/bzip2 depackers? This time distro file (like “/etc/mock/default.cfg” one) needs to be edited:

config_opts['chroot_setup_cmd'] = 'install @buildsys-build /usr/bin/pigz /usr/bin/lbzip2'
config_opts['macros']['%__gzip'] = '/usr/bin/pigz'
config_opts['macros']['%__bzip2'] = '/usr/bin/lbzip2'

extra mockchain tip

For those who use “mockchain” a lot this shell snippet may help finding which packages failed:

for dir in *
do 
    if [ -e $dir/fail ];then
        rm $dir/fail
        mv $dir _fail-$dir
    fi
done

summary

With this set of changes I can do mock builds faster than before. Hope that it helps someone else too.

Failed to set MokListRT: Invalid Parameter

Somehow I managed to break UEFI environment on APM Mustang. As a result I was not able to enter boot manager menu nor UEFI shell. All I had was booting to 0001 boot entry (which was just installed Fedora 24 alpha).

After reboot I scrolled a bit to take a look at firmware output:

X-Gene Mustang Board
Boot firmware (version 1.1.0 built at 14:50:19 on Oct 20 2015)
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3021001 I0
TianoCore 1.1.0 UEFI 2.4.0 Oct 20 2015 14:49:32
CPU: APM ARM 64-bit Potenza Rev A3 2400MHz PCP 2400MHz
     32 KB ICACHE, 32 KB DCACHE
     SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
Board: X-Gene Mustang Board
Slimpro FW:
        Ver: 2.4 (build 01.15.10.00 2015/04/22)
        PMD: 950 mV
        SOC: 950 mV
Failed to set MokListRT: Invalid Parameter

Here screen got cleared instantly and grub was shown. I booted into one of installed systems and started playing with EFI boot manager:

17:38 root@pinkiepie-rawhide:~$ efibootmgr 
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0001,0004,0000
Boot0000  Fedora rawhide
Boot0001* Fedora
Boot0004* Red Hat Enterprise Linux

Note “0 seconds” timeout. I changed it to 5s (efibootmgr -t 5), rebooted and UEFI menu appeared again:

TianoCore 1.1.0 UEFI 2.4.0 Oct 20 2015 14:49:32
CPU: APM ARM 64-bit Potenza Rev A3 2400MHz PCP 2400MHz
     32 KB ICACHE, 32 KB DCACHE
     SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
Board: X-Gene Mustang Board
Slimpro FW:
        Ver: 2.4 (build 01.15.10.00 2015/04/22)
        PMD: 950 mV
        SOC: 950 mV
The default boot selection will start in   5 seconds
[1] Fedora rawhide
[2] Red Hat Enterprise Linux
[3] Fedora
[4] Shell
[5] Boot Manager
[6] Reboot
[7] Shutdown
Start: 

So I can boot whatever I want again ;D

Cello: new AArch64 enterprise board from 96boards project

Few hours ago, somewhere in some hotel in Bangkok Linaro Connect has started. So during morning coffee I watched keynote and noticed that Jon Masters presented RHELSA 7.2 out of the box experience on Huskyboard. And then brand new board from 96boards project was announced: Cello.

Lot of people was expecting that this Linaro Connect will bring Huskyboard alive so people will finally have an option for some cheap board for all their AArch64 needs. Instead Cello was presented:

cello

Compared to Husky (below) there are some hardware differences to notice but it is normal as 96borads enterprise specification only tells where to put ports.

Huskey

I suppose that both boards were designed by different companies. Maybe it was a request from Linaro to ODM vendors to design and mass produce 96boards enterprise board and Husky was prototyped first but Cello won. Or maybe we will see Husky in distribution as well. Good part is: you can preorder Cello and get it delivered in Q2/2016.

Have to admit that I hoped for some industry standard board (96boards Enterprise specification mentions microATX) instead of this weird 96boards-only format which I ranted about already. Anyway 299 USD for quad-core Cortex-A57 with SATA, UEFI, ACPI, PCIe (and maybe few more four letter acronyms) does not sound bad but good luck with finding case for it ;(

From a diary of AArch64 porter — system calls

When userspace want to talk to kernel it can use system calls. Applications usually do it via wrappers in libc to not have to deal with different numbers on different architectures.

There are over four hundred system calls in Linux kernel. Many of them have “legacy” status and as such are not available on many new architectures. Riku Voipio from Linaro has nice slides about it.

So to make things easier for developers I worked on creating a table showing all system calls and their number/status on different architectures. But this table was generated partially by hand…

I changed code to keep a list of all syscalls in place, wrote generator of C code and added makefile on top. And then pushed it into a new git repo called syscalls-table. Code is MIT licensed.

Please go, check the code, run it on any architecture and send me pull requests. The only requirements for generating list of architecture specific syscalls list are bash and C compiler. Table generation requires Python but does not have to be run on target platform.

I may go back to Linaro

Few days ago my manager asked me if I would like to go back to Linaro. This time not as ‘Linaro-but-Canonical engineer’ but as ‘Red Hat assigned engineer’. That made me thinking…

Those three years at Linaro were good time. Learnt a lot about cross toolchains, got possibility to work on bootstrapping AArch64 support in OpenEmbedded, Debian/Ubuntu and then Fedora/RHEL. Met many skilled people from around the world, travelled into places which I would probably not visit on my own.

Going back sounds good. From my discussions with few people from Linaro there is more and more AArch64 related work there (and I have some hardware at home) so my rusty arm32 skills can rust in peace. Have to take a closer look at what exactly is on a plate there to take and find some place.

So if you work for Linaro and will be at FOSDEM (or devconf.cz) then I would love to talk.