Is my work on Kolla done?

During last few months I was working on getting Kolla running on AArch64 and POWER architectures. It was a long journey with several bumps but finally ended.

When I started in January I had no idea how much work it will be and how it will go. Just followed my typical “give me something to build and I will build it” style. You can find some background information in previous post about my work on Kolla.

A lot of failures were present at beginning. Or rather: there was a small amount of images which built. So my first merged change was to do something with Kolla output ;D

  • build: sort list of built/failed images before printing

Debian support was marked for removal so I first checked how it looked, then enabled all possible images and migrated from ‘jessie’ to ‘stretch’ release. Reason was simple: ‘jessie’ (even with backports) lacked packages required to build some images.

  • debian: import key for download.ceph.com repository
  • debian: install gnupg and dirmngr needed for apt-key
  • debian: enable all images enabled for Ubuntu
  • handle rtslib(-fb) package names and dependencies
  • debian: move to stretch
  • Debian 8 was not released yet

Both YUM and APT package managers got some changes. For first one I took care to make sure that it fails if there were missing packages (which was very often during builds for aarch64/ppc64le). It allowed to catch some typo in ‘ironic-conductor’ image. In other words: I make YUM behave closer to APT (which always complain about missing packages). Then I made change for APT to behave more like YUM by making sure that update of packages lists was done before packages were installed.

  • ironic-conductor: add missing comma for centos/source build
  • make yum fail on missing packages
  • always update APT lists when install packages

Of course many images could not be built at all for aarch64/ppc64le architectures. Mostly due to lack of packages and/or external repositories. For each case I was checking is there some way for fixing it. Sometimes I had to disable image, sometimes update packages to newer version. There were also discussions with maintainers of external repositories on getting their stuff available for non-x86 architectures.

  • kubernetes: disable for architectures other than x86-64
  • gnocchi-base: add some devel packages for non-x86
  • ironic-pxe: handle non-x86 architectures
  • openstack-base: Percona-Server is x86-64 only
  • mariadb: handle lack of external repos on non x86
  • grafana: disable for non-x86
  • helm-repository: update to v2.3.0
  • helm-repository: make it work on non-x86
  • kubetoolbox: mark as x86-64 only
  • magnum-conductor: mark as x86-64 only
  • nova-libvirt: handle ppc64le
  • ceph: take care of ceph-fuse package availability
  • handle mariadb for aarch64/ubuntu/source
  • opendaylight: get it working on CentOS/non-x86
  • kolla-toolbox: use proper mariadb packages on CentOS/non-x86

At some moment I had over ten patches in review and all of them depended on the base one. So with any change I had to refresh whole series and reviewers had to review again… Painful it was. So I decided to split out the most basic stuff to get whole patch set split into separate ones. After “base_arch” variable was merged life became much simpler for reviewers and a bit more complicated for me as from now on each patch was kept in separate git branch.

  • add base_arch variable for future non-x86 work

At Linaro we support CentOS and Debian. Kolla supports CentOS/RHEL/OracleLinux, Debian and Ubuntu. I was not making builds with RHEL nor OracleLinux but had to make sure that Ubuntu ones work too. There was funny moment when I realised that everyone using Kolla/master was building images with Ocata packages instead of Pike ;D

  • Ubuntu: use Pike repository

But all those patches meant “nothing” without first one. Kolla had information about which packages are available for aarch64/ppc64le/x86-64 architectures but still had no idea that aarch64 or ppc64le exist. Finally the 50th revision of patch got merged so it now knows ;D

  • Support non-x86 architectures (aarch64, ppc64le)

I also learnt a lot about Gerrit and code reviews. OpenStack community members were very helpful with their comments and suggestions. We had hours of talk on #openstack-kolla IRC channel. Thanks goes to Alicja, Duong Ha-Quang, Jeffrey, Kurt, Mauricio, Michał, Qin Wang, Steven, Surya Prakash, Eduardo, Paul, Sajauddin and many others. You people rock!

So is my work on Kolla done now? Some of it is. But we need to test resulting images, make a Docker repository with them, update official documentation with information how to deploy on aarch64 boxes (hope that there will be no changes needed). Also need to make sure that OpenStack Kolla CI gets Debian based gates operational and provide them with 3rdparty AArch64 based CI so new changes could be checked.

System calls again

Few months ago I created a page with HTML table. For own use basically. Then presented it to the people and found out that it got useful for them. So started improving and improving so it became side project.

Yes, system calls again. I wrote about it in past but yesterday I rewrote code so it now uses Linux source so I can generate tables for far more architectures without need of other computers (either real or emulated).

Next step was work on presentation layer. Old version was just table with added sorting. Things were ugly when scrolled as header was gone. Now it sticks to the top of page so it is easier to note which column relates to which architecture.

Odd/even lines are coloured now which makes is easier to find numbers for syscall.

And speaking of searching — there is filter box now. You can type syscall name (or part of it) there and have table filtered. Same can be done with system call number as well. You used Valgrind and it said that has no idea how to handle syscall 145? Just enter number and you see that it is getresuid(), nfsservctl(), readv(), sched_getscheduler(), setreuid() or setrlimit() — depends which architecture you are testing.

You wonder what that that system call does? There are links to man pages provided.

Go here to check it out and comment here, open a new issue if you found a bug or would like to colaborate. Patches are welcome.

PowerVR is other way to say headless

Yesterday Acer announced convertible Chromebook R13, first MediaTek powered Chromebook. With AArch64 cpu cores. And PowerVR GPU.

As it was in the evening I did not notice PowerVR part and got excited. Finally some AArch64 Chromebook which people will be able to buy and do some development on. Specs were nice: 4GB of memory, 16/32/64GB of emmc storage, 13.3″ FullHD touchscreen display. But why they use that GPU :((

There are few graphics processing units in ARM/AArch64 world. Some of them have FOSS drivers (Ardeno, Tegra, Vivante), some are used with 2D units (Mali) and some just sucks (PowerVR).

Mali is kind of lost case as no one works on free driver for it (so-called “lima” looks like ARM Ltd secret task to get people from trying to do anything) but as it is paired with 2D unit users have working display. And there are binary blobs from ARM Ltd to get 3D acceleration working.

But PowerVR? I never heard about anyone working on free driver for it. I remember that it was used in Texas Instruments OMAP line. And that after few kernel releases it just stopped working when TI fired whole OMAP4 team so no one worked on getting it working with binary blobs.

So now MediaTek used it to make cpu for Chromebook… Sure it will work under ChromeOS as Google is good at keeping one kernel version for ages (my 2012 Samsung Chromebook still runs 3.8.11 one) so blobs will work. But good luck with it under other distributions and mainline kernel.

Heh, even Raspberry/Pi has working free driver nowadays…

Twenty five years of Linux

As I came back from PTO I had to dig into work mails. One of threads was about 25 years of Linux and there was a question “which was your first kernel” and I thought that it may be not an easy question to answer.

For me first was 2.0.2[6-8] (do not remember) on some Uni server where I got my first Linux account (normally used SunOS and text terminal). I remember that there was simple root exploit we used.

Then 2.0.36 on my Amiga 1200 (Debian ‘slink’) was first I run on my hardware.

2.2.10 was first I used for longer as it was Debian ‘potato’ m68k one.

2.3.47 was first I cross compiled (on i686/linux for m68k/linux). And it worked!

2.4.0-test5 was first I built for my x86 desktop once I moved from Amiga/AmigaOS to PC/Debian. I had Duron/600 desktop and old 386 desktop both running same version. Duron got newer ones later, 386 stayed with this one for about year when I returned it.

When I bougth Sharp Zaurus SL-5500 PDA 2.4.18-rmk7-pxa3-embeddix was running on it. So this is my first Linux version on mobile device. Next jump was 2.6.11 on Zaurus c760 as first 2.6 one on mobile.

During OpenZaurus maintaince I started upstreaming kernel patches. 2.6.17 was first with my patches in.

When I had that strange ProGear webpad I wrote backlight for it (based on someone’s code) and 2.6.21 was first with my driver in (and I removed it in 3.7).

Debian On Chromebooks

Debian wiki has a section named “Debian On” where users can describe how to install Debian on any hardware. And there are several pages about Chromebooks.

It is great idea but how it is done is far from being great. People just copy pasted one of pages and did some adaptations leaving rest untouched.

So you can read “Do not play with ALSA mixer – you may fry your speakers!” warning which was valid on Samsung ARM Chromebook in 2012 but was quickly fixed by ChromeOS update.

Some of those pages link to my blog so people often ask me about installing Debian on Any Random Chromebook Model when I have only one – Samsung ARM Chromebook from 2012 year. And do not use it with Debian. I do not use it at all. Maybe will start again one day but it is just maybe.

So people: if you have issues with installing Debian/Fedora/Ubuntu/whatever-other-than-ChromeOS on your Chromebook then go to IRC, find your distribution channel and ask. Better chances for good answer than when you ask me.

Why my device is not supported by distributions

During weekend I was in Puck, Poland at small conference called “Zimowisko linuksowe” (Linux winter camp) where I had a talk called “Dlaczego moje urządzenie nie jest obsługiwane przez dystrybucje” (Why my device is not supported by distributions).

In talk I presented how distributions (Debian, Fedora) handle ARM devices (one kernel for all, one image for all) and why it does not fit Raspberry/Pi or Chromebook. Also mentioned Roseapple/Pi as an example of how not to make support for device.

There were questions about suggested boards (most of people knew Raspberry/Pi and one or two other by name) and ARM powered laptops other than Chromebooks.

And then we went to celebrate birthdays of few friends who had them on same day.

You can download presentation in Polish or English (translation was requested by few folks from IRC after I annouced that there will be such talk).

I may go back to Linaro

Few days ago my manager asked me if I would like to go back to Linaro. This time not as ‘Linaro-but-Canonical engineer’ but as ‘Red Hat assigned engineer’. That made me thinking…

Those three years at Linaro were good time. Learnt a lot about cross toolchains, got possibility to work on bootstrapping AArch64 support in OpenEmbedded, Debian/Ubuntu and then Fedora/RHEL. Met many skilled people from around the world, travelled into places which I would probably not visit on my own.

Going back sounds good. From my discussions with few people from Linaro there is more and more AArch64 related work there (and I have some hardware at home) so my rusty arm32 skills can rust in peace. Have to take a closer look at what exactly is on a plate there to take and find some place.

So if you work for Linaro and will be at FOSDEM (or devconf.cz) then I would love to talk.

Ian Murdock passed away

I am not writing after people outside of my family die but when I read that Ian Murdock is no longer with us I got a feeling that I have to write few words.

Never met him but lot of things in my FOSS career happened because of his most famous project: Debian. For those who do not know: he was “ian” while “Deb” was from his girlfriend name Debra.

First GNU/Linux distribution installed: Debian. First on Amiga 1200, then on PC (where it was my main operating system for years). My first package was made for Debian (“tex-skak” – already removed from archive). I was considering applying for Debian Developer status but found OpenEmbedded first.

Debian way of handling non-free packages was something which allowed me to freely hack on anything I wanted as I knew that I can because someone else already checked licenses. Try that in PalmOS or Microsoft Windows worlds.

Sure, there were other distributions (Slackware, Red Hat Linux) in 90s but it was Debian which brought me to FOSS world. And still is my favorite (despite working for Red Hat).

Running VMs on Fedora/AArch64

There are moments when more than one machine would be handy. But AArch64 computers are not yet available in shop around a corner we have to go for other options. So this time let check how to get virtual machines working.

Requirements

For this I would use Fedora 22 on APM Mustang (other systems will be fine too). What else will be needed:

  • libvirtd running
  • virt-manager 1.1.0-7 (or higher) installed on AArch64 machine
  • UEFI for AArch64 from Gerd’s Hoffmann firmware repository
  • Fedora or Debian installation iso (Ubuntu does not provide such)
  • computer with X11 working (to control virt-manager)

UPDATE: starting from Fedora 23 UEFI package is in distribution repository. Packages are ‘edk2-arm’ for 32bit arm, ‘edk2-aarch64’ for 64bit arm and ‘edk2-ovmf’ for x86-64 architecture.

Is KVM working?

First we need to get KVM working — run “dmesg|grep -i kvm” after system boot. It should look like this:

hrw@pinkiepie-f22:~$ dmesg|grep -i kvm
[    0.261904] kvm [1]: interrupt-controller@780c0000 IRQ5
[    0.262011] kvm [1]: timer IRQ3
[    0.262026] kvm [1]: Hyp mode initialized successfully

But you can also get this:

[    0.343796] kvm [1]: GICV size 0x2000 not a multiple of page size 0x10000
[    0.343802] kvm [1]: error: no compatible GIC info found
[    0.343909] kvm [1]: error initializing Hyp mode: -6

In such case fixed DeviceTree blob from bug #1165290 would be needed. Fetch attached DTB, store as “/boot/mustang.dtb” and then edit “/etc/grub2-efi.cfg” file so kernel entry will look like this:

menuentry 'Fedora (4.0.0-0.rc5.git4.1.fc22.aarch64) 22 (Twenty Two)' --class fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-4.0.0-0.rc5.git2.4.1.fc22.aarch64-advanced-13e42c65-e2eb-4986-abf9-262e287842e4' {
        load_video
        insmod gzio
        insmod part_gpt
        insmod ext2
        set root='hd1,gpt32'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd1,gpt32 --hint-efi=hd1,gpt32 --hint-baremetal=ahci1,gpt32  13e42c65-e2eb-4986-abf9-262e287842e4
        else
          search --no-floppy --fs-uuid --set=root 13e42c65-e2eb-4986-abf9-262e287842e4
        fi
        linux /boot/vmlinuz-4.0.0-0.rc5.git4.1.fc22.aarch64 root=UUID=13e42c65-e2eb-4986-abf9-262e287842e4 ro  LANG=en_GB.UTF-8
        initrd /boot/initramfs-4.0.0-0.rc5.git4.1.fc22.aarch64.img
        devicetree /boot/mustang.dtb
}

After reboot KVM should work.

Software installation

Next step is installing VM software: “dnf install libvirt-daemon* virt-manager” will handle that. But to run Virt Manager we also need a way to see it. X11 forwarding over ssh to the rescue ;D After ssh connection I usually cheat with “sudo ln -sf ~hrw/.Xauthority /root/.Xauthority” to be able to run UI apps as root user.

UEFI firmware

Next phase is UEFI which allows us to boot virtual machine with ISO installation images (compared to kernel/initrd combo when there is no firmware/bootloader possibility). We will install one from repository provided by Gerd Hoffmann:

hrw@pinkiepie-f22:~$ sudo -s
root@pinkiepie-f22:hrw$ cd /etc/yum.repos.d/
root@pinkiepie-f22:yum.repos.d$ wget https://www.kraxel.org/repos/firmware.repo
root@pinkiepie-f22:yum.repos.d$ dnf install edk2.git-aarch64

Then libvirtd config change to give path for just installed firmware. Edit “/etc/libvirt/qemu.conf” file and at the end of file add this:

nvram = [
   "/usr/share/edk2.git/aarch64/QEMU_EFI-pflash.raw:/usr/share/edk2.git/aarch64/vars-template-pflash.raw"
]

Restart libvirtd via “systemctl restart libvirtd“.

Running Virtual Machine Manager

Now we can connect via “ssh -X” and run “sudo virt-manager“:

vm1

Next step is connection to libvirtd:

vm2vm3

Now we are ready for creating VMs. After pressing “Create a new VM” button we should see this:

vm4

And then creation of VM goes nearly like on x86 machines as there is no graphics only serial console.

But if you forgot to setup UEFI firmware then you will get this:

vm5

In such case get back to UEFI firmware step.

Installing Fedora 22 in VM

So let’s test how it works. Fedora 22 is in Beta phase now so why not test it?

vm6

vm10

vm11

2GB ram and 3 cpu cores should be more than enough ;D

vm12

And 10GB for minimal system:

vm13

vm14

But when it went to serial console it did not look good 🙁

vm15

I realized that I forgot to install fonts, but quick “dnf install dejavu*fonts” sorted that out:

vm16

Go for VNC controller installation.

After installation finish system runs just fine:

vm17

Summary

As you can see Fedora 22 has everything in place to get VM running on AArch64. UEFI firmware is the only thing out of distribution but that’s due to some license stuff on vfat implementation or something like that. I was running virtual machines with Debian ‘jessie’ and Fedora 22. Wanted to check Ubuntu but all I found was kernel/initrd combo (which is one of ways to boot in virt-manager) but it did not booted in VM.

Rawhide: unwanted baby in Fedora world?

For something about 15 years I was using Debian distribution and ones which derived from it (like Ubuntu). Basically whole time I used development versions of them and amount of issues was nearly not existing. Now I run Rawhide…

For those who do not know: Fedora world contains four distributions: Fedora, RHEL, CentOS and Rawhide. All new stuff goes to Rawhide which is then branched to make Fedora release. Every few years Red Hat forks released Fedora and uses it as a base for new RHEL release. Then CentOS guys create new release based on RHEL. At least this is how I see it — others will say “but rawhide is fedora”.

I think that the problem lies in development model. All new stuff goes to Rawhide but at same time nearly no one is using it anything can happen there. For example my KDE session lacks window decorations, Konsole5 freezes on any window resize and the common answer for such issues is “You should expect that in rawhide”.

Going into Fedora irc channels with questions is just waste of TCP/IP pockets because in a moment when you mention rawhide it is like everyone fired /ignore on you.

And it is some kind of fun (for some sick/weird definition of it) to watch how people start development of packages just after Fedora releases something. They upgrade and then start to seek what interesting happens in rawhide and can be built.

Each day I am closer to go back to Debian/Ubuntu for a desktop with just keeping Fedora in VM for development of some packages.