System calls again

Few months ago I created a page with HTML table. For own use basically. Then presented it to the people and found out that it got useful for them. So started improving and improving so it became side project.

Yes, system calls again. I wrote about it in past but yesterday I rewrote code so it now uses Linux source so I can generate tables for far more architectures without need of other computers (either real or emulated).

Next step was work on presentation layer. Old version was just table with added sorting. Things were ugly when scrolled as header was gone. Now it sticks to the top of page so it is easier to note which column relates to which architecture.

Odd/even lines are coloured now which makes is easier to find numbers for syscall.

And speaking of searching — there is filter box now. You can type syscall name (or part of it) there and have table filtered. Same can be done with system call number as well. You used Valgrind and it said that has no idea how to handle syscall 145? Just enter number and you see that it is getresuid(), nfsservctl(), readv(), sched_getscheduler(), setreuid() or setrlimit() — depends which architecture you are testing.

You wonder what that that system call does? There are links to man pages provided.

Go here to check it out and comment here, open a new issue if you found a bug or would like to colaborate. Patches are welcome.

Linaro Connect: interesting hardware

Before going for Linaro Connect I had a plan to look at all those 96boards devices and write some complains/opinions about them. But it would be like shooting fish in a barrel so I decided against. But there were some interesting pieces of hardware there.

One of them was Macchiatobin board from SolidRun. I think that this is same as their Armada 8040 community board but after design changes. Standard Mini-ITX format, quad core Cortex-A72 cpu (with upto 2GHz clock), one normal DIMM slot (max 16GB, ships with 4GB), three Serial-ATA ports, PCI-Express x4 slot, one USB 3.0 port, microSD slot.

UPDATE: SolidRun confirmed – this is final design of their Armada 8040 community board.

Photo (done by Riku Voipio) shows which goodies are available:

Network interfaces from top to bottom are (if I remember correctly):

  • 10GbE (SFP + RJ-45)
  • 10GbE (SFP + RJ-45)
  • 2.5GbE (SFP)
  • 1GbE (RJ-45)

When it comes to software I was told that board is SBSA compliant so any normal distribution should work. Kernel, bootloaders (U-Boot and UEFI) are mainlined.

Price? 350USD. Looks like nice candidate for AArch64 development platform or NAS.

Other device was Gumstix Nodana 96BCE board which is 96boards complaint carrierboard for Intel Joule modules.

On top it looks like typical 96boards device (except USB C port):

But once reversed Intel Joule module is visible:

This is first non-ARM based 96boards device. Maybe even one of most compliant ones. At least from software perspective because when it comes to hardware then module makes it a bit too thick to fit in 96boards CE specification limits.

Note that 96boards Consumer Electronics specification does not require using ARM or AArch64 cpu.

Downgrading Fedora ‘rawhide’ -> Fedora 24

Time to downgrade my main desktop finally came. This time I decided to provide more details about process as my system uses Rawhide with set of external repositories (RPM Fusion (official + my rebuilds), few COPR ones etc).

So first step is to enable caching because dnf by default erases all downloaded packages. And downgrade involves fetching many of them. Edit ‘/etc/dnf/dnf.conf’ file and add “keepcache=1” line there.

First try:

10:48 root@puchatek:hrw# dnf --releasever 24 --disablerepo rawhide --enablerepo fedora --enablerepo updates  distro-sync  --best --allowerasing
Error: package python2-deltarpm-3.6-17.fc25.x86_64 requires deltarpm(x86-64) = 3.6-17.fc25, but none of the providers can be installed.
package libcrypt-nss-2.24.90-6.fc26.x86_64 requires glibc(x86-64) = 2.24.90-6.fc26, but none of the providers can be installed.
package gmp-c++-1:6.1.1-1.fc25.x86_64 requires gmp(x86-64) = 1:6.1.1-1.fc25, but none of the providers can be installed.
package iproute-tc-4.7.0-1.fc26.x86_64 requires iproute(x86-64) = 4.7.0-1.fc26, but none of the providers can be installed.
package ffmpeg-libs-3.1.1-1.fc26.x86_64 requires, but none of the providers can be installed.
package pcre-cpp-8.39-3.fc26.x86_64 requires pcre(x86-64) = 8.39-3.fc26, but none of the providers can be installed.
package perl-libintl-perl-1.26-1.fc25.x86_64 requires perl(:MODULE_COMPAT_5.24.0), but none of the providers can be installed.
package python3-rpm-4.13.0-0.rc1.46.fc26.x86_64 requires rpm(x86-64) = 4.13.0-0.rc1.46.fc26, but none of the providers can be installed.
package systemd-pam-231-4.fc26.x86_64 requires systemd = 231-4.fc26, but none of the providers can be installed.
package libvirt-libs-2.2.0-1.fc26.x86_64 requires, but none of the providers can be installed.
package glibc-2.24.90-6.fc26.i686 requires glibc-common = 2.24.90-6.fc26, but none of the providers can be installed.
package python2-rpm-4.13.0-0.rc1.46.fc26.x86_64 requires rpm(x86-64) = 4.13.0-0.rc1.46.fc26, but none of the providers can be installed.
nothing provides needed by ffmpeg-libs-2.6.3-1.fc22.x86_64.
package ffmpeg-libs-3.1.1-1.fc26.x86_64 requires, but none of the providers can be installed

As you see there is set of blockers. One of them is “ffmpeg-libs” from RPM Fusion, others are from normal Fedora repositories. One of reasons can be that packages got split/renamed since F24. Let’s try to handle some of them:

10:52 root@puchatek:hrw# dnf --releasever 24 --disablerepo rawhide --enablerepo fedora --enablerepo updates --best --allowerasing downgrade rpm glibc
Dependencies resolved.
 Package                      Arch     Version                 Repository  Size
 rpm-python                   x86_64   4.13.0-0.rc1.27.fc24    fedora     102 k
 rpm-python3                  x86_64   4.13.0-0.rc1.27.fc24    fedora     102 k
 libcrypt-nss                 i686     2.24.90-6.fc26          @rawhide    34 k
 libcrypt-nss                 x86_64   2.24.90-6.fc26          @rawhide    36 k
 python2-rpm                  x86_64   4.13.0-0.rc1.46.fc26    @rawhide   182 k
 python3-rpm                  x86_64   4.13.0-0.rc1.46.fc26    @rawhide   190 k
 glibc                        i686     2.23.1-10.fc24          updates    4.3 M
 glibc                        x86_64   2.23.1-10.fc24          updates    3.6 M
 glibc-common                 x86_64   2.23.1-10.fc24          updates    862 k
 glibc-devel                  i686     2.23.1-10.fc24          updates    936 k
 glibc-devel                  x86_64   2.23.1-10.fc24          updates    936 k
 glibc-headers                x86_64   2.23.1-10.fc24          updates    501 k
 glibc-langpack-en            x86_64   2.23.1-10.fc24          updates    276 k
 glibc-langpack-pl            x86_64   2.23.1-10.fc24          updates    133 k
 glibc-static                 x86_64   2.23.1-10.fc24          updates    1.5 M
 rpm                          x86_64   4.13.0-0.rc1.27.fc24    fedora     513 k
 rpm-build                    x86_64   4.13.0-0.rc1.27.fc24    fedora     139 k
 rpm-build-libs               x86_64   4.13.0-0.rc1.27.fc24    fedora     117 k
 rpm-libs                     x86_64   4.13.0-0.rc1.27.fc24    fedora     295 k
 rpm-plugin-selinux           x86_64   4.13.0-0.rc1.27.fc24    fedora      54 k
 rpm-plugin-systemd-inhibit   x86_64   4.13.0-0.rc1.27.fc24    fedora      54 k

Transaction Summary
Install     2 Packages
Remove      4 Packages
Downgrade  15 Packages

Total download size: 14 M
Is this ok [y/N]: 

Went fine. So next set of blockers goes:

10:54 root@puchatek:hrw# dnf --releasever 24 --disablerepo rawhide --enablerepo fedora --enablerepo updates --best --allowerasing downgrade *deltarpm* gmp* iproute*
Dependencies resolved.
 Package                 Arch         Version              Repository      Size
 python-deltarpm         x86_64       3.6-15.fc24          fedora          37 k
 gmp-c++                 x86_64       1:6.1.1-1.fc25       @rawhide        23 k
 iproute-tc              x86_64       4.7.0-1.fc26         @rawhide       660 k
 python2-deltarpm        x86_64       3.6-17.fc25          @rawhide        44 k
 deltarpm                x86_64       3.6-15.fc24          fedora          89 k
 gmp                     i686         1:6.1.1-1.fc24       updates        305 k
 gmp                     x86_64       1:6.1.1-1.fc24       updates        315 k
 gmp-devel               x86_64       1:6.1.1-1.fc24       updates        185 k
 iproute                 x86_64       4.4.0-3.fc24         fedora         658 k
 iptables                x86_64       1.4.21-16.fc24       fedora         425 k
 iptables-services       x86_64       1.4.21-16.fc24       fedora          53 k

Transaction Summary
Install    1 Package
Remove     3 Packages
Downgrade  7 Packages

Total download size: 2.0 M
Is this ok [y/N]: y
Downloading Packages:
(1/8): deltarpm-3.6-15.fc24.x86_64.rpm          139 kB/s |  89 kB     00:00    
(2/8): gmp-6.1.1-1.fc24.x86_64.rpm              384 kB/s | 315 kB     00:00    
(3/8): gmp-6.1.1-1.fc24.i686.rpm                320 kB/s | 305 kB     00:00    
(4/8): gmp-devel-6.1.1-1.fc24.x86_64.rpm        445 kB/s | 185 kB     00:00    
(5/8): iptables-services-1.4.21-16.fc24.x86_64. 437 kB/s |  53 kB     00:00    
(6/8): python-deltarpm-3.6-15.fc24.x86_64.rpm   303 kB/s |  37 kB     00:00    
(7/8): iptables-1.4.21-16.fc24.x86_64.rpm       1.0 MB/s | 425 kB     00:00    
(8/8): iproute-4.4.0-3.fc24.x86_64.rpm          1.1 MB/s | 658 kB     00:00    
Total                                           475 kB/s | 2.0 MB     00:04     
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction check error:
  file /usr/lib64/ from install of iptables-1.4.21-16.fc24.x86_64 conflicts with file from package iptables-libs-1.6.0-2.fc25.x86_64
  file /usr/lib64/ from install of iptables-1.4.21-16.fc24.x86_64 conflicts with file from package iptables-libs-1.6.0-2.fc25.x86_64
  file /usr/lib64/ from install of iptables-1.4.21-16.fc24.x86_64 conflicts with file from package iptables-libs-1.6.0-2.fc25.x86_64

Error Summary

So let’s go level deeper with packaging:

10:56 root@puchatek:packages# rpm -e iptables-libs
error: Failed dependencies:
        iptables-libs(x86-64) = 1.6.0-2.fc25 is needed by (installed) iptables-1.6.0-2.fc25.x86_64 is needed by (installed) iptables-1.6.0-2.fc25.x86_64 is needed by (installed) systemd-231-4.fc26.x86_64 is needed by (installed) systemd-container-231-4.fc26.x86_64 is needed by (installed) iptables-1.6.0-2.fc25.x86_64 is needed by (installed) iptables-1.6.0-2.fc25.x86_64 is needed by (installed) iproute-tc-4.7.0-1.fc26.x86_64
10:57 root@puchatek:packages# rpm -e iptables-libs --nodeps iproute-tc
10:57 root@puchatek:packages# rpm --oldpackage -U iptables-1.4.21-16.fc24.x86_64.rpm

And repeat previous dnf command as it works this time.

So next set of blockers has to go:

11:00 root@puchatek:packages# dnf --releasever 24 --disablerepo rawhide --enablerepo fedora --enablerepo updates --best --allowerasing downgrade systemd* libvirt* pcre*
Dependencies resolved.
 Package                           Arch     Version            Repository  Size
 systemd-compat-libs               x86_64   229-13.fc24        updates    152 k
 libvirt-libs                      x86_64   2.2.0-1.fc26       @rawhide    22 M
 pcre-cpp                          x86_64   8.39-3.fc26        @rawhide    39 k
 perl-Sys-Virt                     x86_64   2.2.0-1.fc26       @rawhide   824 k
 systemd-pam                       x86_64   231-4.fc26         @rawhide   282 k
 libvirt                           x86_64     updates     47 k
 libvirt-client                    x86_64     updates    4.4 M
 libvirt-daemon                    x86_64     updates    619 k
 libvirt-daemon-config-network     x86_64     updates     47 k
 libvirt-daemon-config-nwfilter    x86_64     updates     50 k
 libvirt-daemon-driver-interface   x86_64     updates     90 k
 libvirt-daemon-driver-libxl       x86_64     updates    163 k
 libvirt-daemon-driver-lxc         x86_64     updates    724 k
 libvirt-daemon-driver-network     x86_64     updates    241 k
 libvirt-daemon-driver-nodedev     x86_64     updates     89 k
 libvirt-daemon-driver-nwfilter    x86_64     updates    114 k
 libvirt-daemon-driver-qemu        x86_64     updates    528 k
 libvirt-daemon-driver-secret      x86_64     updates     82 k
 libvirt-daemon-driver-storage     x86_64     updates    274 k
 libvirt-daemon-driver-uml         x86_64     updates     98 k
 libvirt-daemon-driver-vbox        x86_64     updates    197 k
 libvirt-daemon-driver-xen         x86_64     updates    161 k
 libvirt-daemon-kvm                x86_64     updates     46 k
 libvirt-daemon-qemu               x86_64     updates     46 k
 libvirt-python                    x86_64   1.3.3-3.fc24       updates    255 k
 pcre                              i686     8.39-3.fc24        updates    413 k
 pcre                              x86_64   8.39-3.fc24        updates    404 k
 pcre-devel                        x86_64   8.39-3.fc24        updates    544 k
 pcre2                             x86_64   10.21-6.fc24       updates    414 k
 qemu                              x86_64   2:2.6.1-1.fc24     updates     63 k
 qemu-common                       x86_64   2:2.6.1-1.fc24     updates    323 k
 qemu-img                          x86_64   2:2.6.1-1.fc24     updates    828 k
 qemu-kvm                          x86_64   2:2.6.1-1.fc24     updates     62 k
 qemu-system-aarch64               x86_64   2:2.6.1-1.fc24     updates    2.5 M
 qemu-system-alpha                 x86_64   2:2.6.1-1.fc24     updates    1.9 M
 qemu-system-arm                   x86_64   2:2.6.1-1.fc24     updates    2.5 M
 qemu-system-cris                  x86_64   2:2.6.1-1.fc24     updates    1.4 M
 qemu-system-lm32                  x86_64   2:2.6.1-1.fc24     updates    1.4 M
 qemu-system-m68k                  x86_64   2:2.6.1-1.fc24     updates    1.9 M
 qemu-system-microblaze            x86_64   2:2.6.1-1.fc24     updates    2.7 M
 qemu-system-mips                  x86_64   2:2.6.1-1.fc24     updates    8.4 M
 qemu-system-moxie                 x86_64   2:2.6.1-1.fc24     updates    1.4 M
 qemu-system-or32                  x86_64   2:2.6.1-1.fc24     updates    1.4 M
 qemu-system-ppc                   x86_64   2:2.6.1-1.fc24     updates    6.8 M
 qemu-system-s390x                 x86_64   2:2.6.1-1.fc24     updates    1.7 M
 qemu-system-sh4                   x86_64   2:2.6.1-1.fc24     updates    3.7 M
 qemu-system-sparc                 x86_64   2:2.6.1-1.fc24     updates    3.3 M
 qemu-system-tricore               x86_64   2:2.6.1-1.fc24     updates    1.4 M
 qemu-system-unicore32             x86_64   2:2.6.1-1.fc24     updates    1.4 M
 qemu-system-x86                   x86_64   2:2.6.1-1.fc24     updates    4.5 M
 qemu-system-xtensa                x86_64   2:2.6.1-1.fc24     updates    2.7 M
 qemu-user                         x86_64   2:2.6.1-1.fc24     updates    8.3 M
 systemd                           x86_64   229-13.fc24        updates    5.1 M
 systemd-container                 x86_64   229-13.fc24        updates    1.0 M
 systemd-devel                     x86_64   229-13.fc24        updates    288 k
 systemd-libs                      i686     229-13.fc24        updates    482 k
 systemd-libs                      x86_64   229-13.fc24        updates    457 k
 systemd-udev                      x86_64   229-13.fc24        updates    1.2 M
 xen-libs                          x86_64   4.6.3-4.fc24       updates    574 k

Transaction Summary
Install     1 Package
Remove      4 Packages
Downgrade  54 Packages

Total download size: 80 M
Is this ok [y/N]:


Error: Transaction check error:
  file /usr/lib64/ from install of pcre-8.39-3.fc24.x86_64 conflicts with file from package pcre-utf32-8.39-3.fc26.x86_64
  file /usr/lib64/ from install of pcre-8.39-3.fc24.x86_64 conflicts with file from package pcre-utf16-8.39-3.fc26.x86_64

Let’s go back to rpm:

11:14 root@puchatek:packages# rpm -e pcre-utf16 pcre-utf32 --nodeps  
11:14 root@puchatek:packages# dnf --releasever 24 --disablerepo rawhide --enablerepo fedora --enablerepo updates --best --allowerasing downgrade -y "pcre*"    
Dependencies resolved.
 Package            Arch           Version               Repository        Size
 pcre-cpp           x86_64         8.39-3.fc26           @rawhide          39 k
 pcre               i686           8.39-3.fc24           updates          413 k
 pcre               x86_64         8.39-3.fc24           updates          404 k
 pcre-devel         x86_64         8.39-3.fc24           updates          544 k
 pcre2              x86_64         10.21-6.fc24          updates          414 k

Transaction Summary
Remove     1 Package
Downgrade  4 Packages

Total size: 1.7 M
Downloading Packages:

Now previous dnf command succeddes.

Last blocker is “perl-libintl-perl” package. In F24 it was named “perl-libintl” so dnf is not able to handle downgrading. Simplest way is removal of it. Will install removed packages later.

11:23 root@puchatek:packages# dnf remove perl-libintl-perl                      Dependencies resolved.
 Package                      Arch   Version                     Repository
 libpaper                     x86_64 1.1.24-12.fc24              @rawhide  83 k
 mutt                         x86_64 5:1.7.0-1.fc26              @rawhide 5.4 M
 perl-Class-Inspector         noarch 1.28-15.fc25                @rawhide  57 k
 perl-Class-Method-Modifiers  noarch 2.12-3.fc25                 @rawhide 100 k
 perl-Class-MethodMaker       x86_64 2.24-6.fc25                 @rawhide  20 M
 perl-Convert-BinHex          noarch 1.125-3.fc25                @rawhide  98 k
 perl-Data-Perl               noarch 0.002009-7.fc25             @rawhide  89 k
 perl-Devel-GlobalDestruction noarch 0.13-7.fc25                 @rawhide  17 k
 perl-Exporter-Tiny           noarch 0.042-6.fc25                @rawhide  78 k
 perl-File-ShareDir           noarch 1.102-7.fc25                @rawhide  19 k
 perl-Filter-Simple           noarch 0.92-365.fc25               @rawhide  50 k
 perl-GnuPG-Interface         noarch 0.52-5.fc25                 @rawhide 136 k
 perl-Import-Into             noarch 1.002005-3.fc25             @rawhide  20 k
 perl-List-MoreUtils          x86_64 0.416-1.fc25                @rawhide 200 k
 perl-MIME-tools              noarch 5.508-1.fc26                @rawhide 508 k
 perl-MailTools               noarch 2.18-2.fc25                 @rawhide 193 k
 perl-Moo                     noarch 2.002004-1.fc25             @rawhide 180 k
 perl-MooX-HandlesVia         noarch 0.001008-6.fc25             @rawhide  43 k
 perl-MooX-late               noarch 0.015-9.fc25                @rawhide  37 k
 perl-Net-IDN-Encode          x86_64 2.300-4.fc25                @rawhide 292 k
 perl-Role-Tiny               noarch 2.000003-3.fc25             @rawhide  39 k
 perl-SelfLoader              noarch 1.23-378.fc26               @rawhide  23 k
 perl-Text-Balanced           noarch 2.03-365.fc25               @rawhide 153 k
 perl-Type-Tiny               noarch 1.000005-7.fc25             @rawhide 581 k
 perl-Unicode-EastAsianWidth  noarch 1.33-8.fc25                 @rawhide  13 k
 perl-libintl-perl            x86_64 1.26-1.fc25                 @rawhide 4.2 M
 perl-strictures              noarch 2.000003-2.fc25             @rawhide  34 k
 pgp-tools                    x86_64 2.2-3.fc25                  @rawhide 449 k
 texinfo                      x86_64 6.1-3.fc25                  @rawhide 4.5 M
 tokyocabinet                 x86_64 1.4.48-6.fc24               @rawhide 1.3 M
 urlview                      x86_64 0.9-19.20131022git08767a.fc24
                                                                 @rawhide  49 k

Transaction Summary
Remove  31 Packages

Installed size: 39 M
Is this ok [y/N]: y

Ok. All blockers removed. Time to call distro-sync:

11:24 root@puchatek:packages# dnf --releasever 24 --disablerepo rawhide --enablerepo fedora --enablerepo updates  distro-sync  --best --allowerasing

[.. long list of packages ..]

Transaction Summary
Install      11 Packages
Upgrade      13 Packages
Remove        4 Packages
Downgrade  2224 Packages

Total size: 2.1 G
Total download size: 2.1 G
Is this ok [y/N]:

Error: Transaction check error:
  file /usr/lib64/ from install of lua-5.3.3-2.fc24.x86_64 conflicts with file from package lua-libs-5.3.3-3.fc25.x86_64

This is less fun… RPM needs Lua to work. So I went around. Unpacked Lua package from F24, then “rpm -e –nodeps lua-libs” and copied “lib64” from just unpacked package directly into system. Then downgraded “lua” using dnf.

Next step? Distro sync of course. This time it works. Now kernel needs to be taken care of.

I had 4.6.0 kernel installed from I have no idea when. Booted it and it allowed me to remove all “4.8-rc” kernels I got from rawhide. Then simple “dnf install kernel” to get 4.7.2 from updates repo and after one more reboot I got:

15:52 hrw@puchatek:~$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: Fedora
Description:    Fedora release 24 (Twenty Four)
Release:        24
Codename:       TwentyFour

As you see it took far more time than I expected it will. I learnt that ‘keepcache’ is not enabled (had to fetch 2GB of packages again), found ‘minrate’ dnf config option which helps me avoid slow Austrian mirror.

PowerVR is other way to say headless

Yesterday Acer announced convertible Chromebook R13, first MediaTek powered Chromebook. With AArch64 cpu cores. And PowerVR GPU.

As it was in the evening I did not notice PowerVR part and got excited. Finally some AArch64 Chromebook which people will be able to buy and do some development on. Specs were nice: 4GB of memory, 16/32/64GB of emmc storage, 13.3″ FullHD touchscreen display. But why they use that GPU :((

There are few graphics processing units in ARM/AArch64 world. Some of them have FOSS drivers (Ardeno, Tegra, Vivante), some are used with 2D units (Mali) and some just sucks (PowerVR).

Mali is kind of lost case as no one works on free driver for it (so-called “lima” looks like ARM Ltd secret task to get people from trying to do anything) but as it is paired with 2D unit users have working display. And there are binary blobs from ARM Ltd to get 3D acceleration working.

But PowerVR? I never heard about anyone working on free driver for it. I remember that it was used in Texas Instruments OMAP line. And that after few kernel releases it just stopped working when TI fired whole OMAP4 team so no one worked on getting it working with binary blobs.

So now MediaTek used it to make cpu for Chromebook… Sure it will work under ChromeOS as Google is good at keeping one kernel version for ages (my 2012 Samsung Chromebook still runs 3.8.11 one) so blobs will work. But good luck with it under other distributions and mainline kernel.

Heh, even Raspberry/Pi has working free driver nowadays…

Goodbye rawhide

During Flock I decided to start removing rawhide from my main development systems. Got too tired of funny things like X11 freezes , common applications segfaults or non-reacting programs. I had to create F24 virtual machine just to get LibreOffice Impress running to finish my slides…

So laptop went first:

19:51 root@kapturek:~# LANGUAGE=C dnf --releasever 24 distro-sync --allowerasing
Transaction Summary
Install      12 Packages
Upgrade     180 Packages
Remove       10 Packages
Downgrade  1330 Packages

Once it will be done and tested working I will do same with my main desktop. Then no more rawhide on my desktop/laptops. For development boards/servers I will keep rawhide but only there.

AArch64 desktop hardware?

Soon there will be four years since I started working on AArch64 architecture. Lot of software things changed during that time. Lot in a hardware too. But machines availability still sucks badly.

In 2012 all we had was software model. It was slow, terribly slow. Common joke was AArch64 developers standing in a queue for 10GHz x86-64 cpus. So I was generating working binaries by using cross compilation. But many distributions only do native builds. In models. Imagine Qt4 building for 3-4 days…

In 2013 I got access to first server hardware. With first silicon version of CPU. Highly unstable, we could use just one core etc. GCC was crashing like hell but we managed to get stable build results from it. Qt4 was building in few hours now.

Then amount of hardware at Red Hat was growing and growing. Farms of APM Mustangs, AMD Seattle and several other servers appeared, got racked and available to use. In 2014 one Mustang even landed on my desk (as first such machine in Poland).

But this was server land. Each of those machines costed about 1000 USD (if not more). And availability was hard too.

Linaro tried to do something about it and created 96boards project.

First came ‘Consumer Edition’ range. Yet another small form factor boards with functionality stripped as much as possible. No Ethernet, no storage other than emmc/usb, low amount of memory, chips taken from mobile phones etc. But it was selling! But only because people were hungry to get ANYTHING with AArch64 cores. First was HiKey then DragonBoard410 got released. Then few other boards. All with same set of issues: non-mainline kernel, weird bootloaders, binary blobs for this or that…

Then so called ‘Enterprise Edition’ got announced. With another ridiculous form factor (and microATX as an option). And that was it. There was a leak of Husky board which shown how fucked up design it was. Ports all around the edges, memory above and under board and of course incompatible with any industrial form factor. I would like to know what they were smoking…

Time passed by. Husky got forgotten for another year. Then Cello was announced as a “new EE 96boards board” while it looked as redesigned Husky with two SATA ports less (because who needs more than two SATA, right?). Last time I heard about Cello it was still ‘maybe soon, maybe another two weeks’. Prototypes looked like hand soldered, USB controller mounted rotated, dead on-board Ethernet etc.

In meantime we got few devices from other companies. Pine64 had big campaign on Kickstarter and shipped to developers. Hardkernel started selling ODROID-C2, Geekbox released their TV box and probably something else got released as well. But all those boards were limited to 1-2GB of memory, often lacked SATA and used mobile processors with their own set of bootloaders etc causing extra work for distributions.

Overdrive 1000 was announced. Without any options for expansion it looked like SoftIron wanted customers to buy Overdrive 3000 if they want to use PCI Express card.

So we have 2016 now. Four years of my work on AArch64 passed. Most of distributions support this architecture by building on proper servers but most of this effort is not used because developers do not have sane hardware to play with (sane means expandable, supported by distributions, capable).

There is no standard form factor mainboards (mini-itx, microATX, ATX) available on mass market. 96boards failed here, server vendors are not interested, small Chinese companies prefer to release yet-another-fruit/Pi with mobile processor. Nothing, null, nada, nic.

Developers know where to buy normal computer cases, storage, memory, graphics cards, USB controllers, SATA controllers and peripherals. So vendors do not have to worry/deal with this part. But still there is nothing to put those cards into. No mainboards which can be mounted into normal PC case, have some graphics plugged in, few SSD/HDD connected, mouse/keyboard, monitors and just be used.

Sometimes it is really hard to convince software developers to make changes for platform they are unable to test on. And current hardware situation does not help. All those projects of hardware being available “in a cloud” helps only for subset of projects — ever tried to run GNOME/KDE session over the network? With OpenGL acceleration etc?

So where is my AArch64 workstation? In desktop or laptop form.

Post written after my Google+ post where similar discussion happened in comments.

My work on changing CirrOS images

What is CirrOS and why I was working on it? This was quite common question when I mentioned what I am working on during last weeks.

So, CirrOS is small image to run in a cloud. OpenStack developers use it to test their projects.

Technically it is yet another Frankenstein OS. Built using Buildroot 2015.05 uses uclibc or glibc (depending on target architecture). Then Ubuntu 16.04 kernel is applied on top and “grub” (also from Ubuntu) is used to make it bootable.

The problem was that it was not done in UEFI bootable way…

My first changes were: switch images to GPT, create EFI system partition and put some bootloader there. I first used CentOS “grub2-efi” packages (as they provided ready to use EFI binaries) and later switched to Ubuntu ones as upstream maintainer (Scott Moser) prefers to have all external binaries to came from one source.

When he was on vacations (so merge request had to wait) I started digging more and more into scripts.

Fixed getopt use as arguments passed between scripts were read partly via getopt, partially by assigning variables to ${X} (where X is a number).

All scripts were moved to use Bash (as /bin/sh in Ubuntu is usually Dash which is minimalist POSIX shell), whitespace got unified between all scripts and some other stuff happened as well.

At one moment all scripts had 1835 lines and my diff was 2250 lines (+1018/-603) long. Hopefully Scott was back and we got most of that stuff merged.

Recent (2016.07.21) images are available and work fine on all platforms. If someone uses them with OpenStack then please remember about setting “short_id” property to “ubuntu16.04” — otherwise there may be a problem with finding rootfs (no virtio-scsi in disk images).


architecture booting before booting after
aarch64 direct kernel UEFI or direct kernel
arm direct kernel UEFI or direct kernel
i386 BIOS or direct kernel BIOS, UEFI or direct kernel
powerpc direct kernel direct kernel
ppc64 direct kernel direct kernel
ppc64le direct kernel direct kernel
x86-64 BIOS or direct kernel BIOS, UEFI or direct kernel

Debian On Chromebooks

Debian wiki has a section named “Debian On” where users can describe how to install Debian on any hardware. And there are several pages about Chromebooks.

It is great idea but how it is done is far from being great. People just copy pasted one of pages and did some adaptations leaving rest untouched.

So you can read “Do not play with ALSA mixer – you may fry your speakers!” warning which was valid on Samsung ARM Chromebook in 2012 but was quickly fixed by ChromeOS update.

Some of those pages link to my blog so people often ask me about installing Debian on Any Random Chromebook Model when I have only one – Samsung ARM Chromebook from 2012 year. And do not use it with Debian. I do not use it at all. Maybe will start again one day but it is just maybe.

So people: if you have issues with installing Debian/Fedora/Ubuntu/whatever-other-than-ChromeOS on your Chromebook then go to IRC, find your distribution channel and ask. Better chances for good answer than when you ask me.

My workflow for building big sets of RPM packages

In last months I did two rebuilds: NodeJS 4.x in Fedora and OpenStack Mitaka in CentOS. Both were targeting AArch64 and both were not done before. During latter one I was asked to write about my workflow, so will describe it with OpenStack one as base.

identify what needs to be done

At first I had to figure out what exactly needs to be built. Haïkel Guémar (aka number80) pointed me to openstack-mitaka directory on CentOS’ vault where all source packages are present. Also told me that EPEL repository is not required which helped a lot as it is not yet built for CentOS.

structure of sources

OpenStack set of packages in CentOS is split into two parts: “common” shared through all OpenStack versions and “openstack-mitaka” containing OpenStack Mitaka packages and build dependencies not covered by CentOS itself or “common” directory.

prepare build space

I used “mockchain” for such rebuilds. It is simple tool which does not do any ordering tricks just builds set of packages in given order and do it three times hoping that all build dependencies will be solved that way. Of course what got built once is not tried again.

To make things easier I used shell alias:

alias runmockchain mockchain -r default -l /home/hrw/rpmbuild/_rebuilds/openstack/mockchain-repo-centos

With this I did not have to remember about those two switches. Common call was “runmockchain –recurse srpms/*” which means “cycle packages three times and continue on failures”.

Results of all builds (packages and logs) were kept in “~/_rebuilds/openstack/mockchain-repo-centos/results/default/” subdirectories. I put all extra packages there to have all in one repository.

populate “noarch” packages

Then I copied x86-64 build of OpenStack Mitaka into “_IMPORT-openstack-mitaka/” to get all “noarch” packages for satisfying build dependencies. I built all those packages anyway but having them saved me several rebuilds.

extra rpm macros

When I started first build it turned out that some Python packages lack proper “Provides” fields. I was missing newer rpm build macros (“%python_provide” was added after Fedora 19 which was base for RHEL7). Asked Haïkel and added “rdo-rpm-macros” to mock configuration.

But had to scrap everything I built so far.

surprises and failures

Building big set of packages for new architecture most of time generate failures which were not present with x86-64 build. Same was this time as several build dependencies were missing or wrong.

packages missing in CentOS/aarch64

Some were from CentOS itself — I told Jim Perrin (aka Evolution) and he added them to build queue to fill gaps. I built them in meantime or (if they were “noarch”) imported into “_IMPORT-extras” or “_IMPORT-os” directories.

packages imported from other CBS tags

Other packages were originally imported from other tags at CBS (CentOS koji). For those I created directory named “_IMPORT-cbs”. And again — if they were “noarch” I just copied them. For rest I did full build (using “runmockchain”) and they end in same repository as rest of build.

For some packages it turned out that they got built long time ago with older versions of build dependencies and are not buildable from current versions. For them I tracked proper versions on CBS and imported/built (sometimes with their build dependencies and build dependencies of build dependencies).

downgrading packages

There was a package “python-flake8” which failed to build spitting out Python errors. I checked how this version was built on CBS and turned out that “python-mock” 1.3.0 (from “openstack-mitaka” repository) was too new… Downgraded it to 1.0 allowed me to build “python-flake8” one (upgrading of it is in queue).

merging fixes from Fedora

Both “galera” and “mariadb-galera” got AArch64 support merged from Fedora and got built with “.hrw1” added into “Release” field.


I did whole build in “~/rpmbuild/_rebuilds/openstack/” directory. Extra folders were:

  • mockchain-repo-centos/results/default/_HACKED-by-hew/
  • mockchain-repo-centos/results/default/_IMPORT-cbs/
  • mockchain-repo-centos/results/default/_IMPORT-extras/
  • mockchain-repo-centos/results/default/_IMPORT-openstack-mitaka/
  • mockchain-repo-centos/results/default/_IMPORT-os/

Vault ones were copy of OpenStack source packages and their build dependencies. Hacked ones got AArch64 support merged from Fedora. Both “extras” and “os” directories were for packages missing in CentOS/AArch64 repositories. CBS one was for source/noarch packages which had to be imported/rebuilt because they came from other CBS tags.

status page

In meantime I prepared web page with build results so anyone interested can see what builds, what not and check logs, packages etc. It has simple description and then table with list of builds (data can be sorted by clicking on column headers).


Whole job would take much more time if not help from CentOS developers: Haïkel Guémar, Alan Pevec, Jim Perrin, Karanbir Singh and others from #centos-devel and #centos-arm IRC channels.

How to speed up mock

Fedora and related distributions use “mock” to build packages. It creates chroot from “root cache” tarball, updates it, installs build dependencies and runs build. Similar to “pbuilder” under Debian. Whole build time can be long but there are some ways to get it faster.

kill fsync()

Everytime something calls “fsync()” storage slow downs because all writes need to be done. Build process goes in a chroot which will be removed at the end so why bother?

Run “dnf install nosync” and then enable it in “/etc/mock/site-defaults.cfg” file:

config_opts['nosync'] = True

local cache for packages

I do lot of builds. Often few builds of same package. And each build requires fetching of RPM packages from external repositories. So why not cache it?

In LAN I have one machine working as NAS. One of services running there is “www cache” with 10GB space which I use only for fetching packages — both in mock and system and I use it on all machines. This way I can recreate build chroot without waiting for external repositories.

config_opts['http_proxy'] = 'http://nas.lan:3128'

Note that this also requires editing mock distribution config files to not use “mirrorlist=” but “baseurl=” instead so same server will be used each time. There is a script to convert mock configuration files if you go that way.

decompress root cache

Mock keeps tarball of base chroot contents which gets unpacked at start of build. By default it is gzip compressed and unpacking takes time. On my systems I switched off compression to gain a bit at cost of storage:

config_opts['plugin_conf']['root_cache_opts']['compress_program'] = ""
config_opts['plugin_conf']['root_cache_opts']['extension'] = ""


If memory is not an issue then tmpfs can be used for builds. Mock has own plugin for it but I do not use it. Instead I decide on my own about mounting “/var/lib/mock” as tmpfs or not. Why? I only have 16GB ram in pinkiepie so 8-12GB can be spent on tmpfs while there are packages which would not fit during build.

parallel decompression

Sources are compressed. Nowadays CPU has more than one core. So why not using it with multithreaded gzip/bzip2 depackers? This time distro file (like “/etc/mock/default.cfg” one) needs to be edited:

config_opts['chroot_setup_cmd'] = 'install @buildsys-build /usr/bin/pigz /usr/bin/lbzip2'
config_opts['macros']['%__gzip'] = '/usr/bin/pigz'
config_opts['macros']['%__bzip2'] = '/usr/bin/lbzip2'

extra mockchain tip

For those who use “mockchain” a lot this shell snippet may help finding which packages failed:

for dir in *
    if [ -e $dir/fail ];then
        rm $dir/fail
        mv $dir _fail-$dir


With this set of changes I can do mock builds faster than before. Hope that it helps someone else too.