Marcin Juszkiewicz

Someone may say that I am main reason why CirrOS project does releases.

In 2016 I got task at Linaro to get it running on AArch64. More details are in my blog post ‘my work on changing CirrOS images’. Result was 0.4.0 release.

Last year I got another task at Linaro. So we released 0.5.0 version today.

But that’s not how it happened.

Multiple contributors

Since 0.4.0 release there were changes done by several developers.

Robin H. Johnson took care of kernel modules. Added new ones, updated names. Also added several new features.

Murilo Opsfelder Araujo fixed build on Ubuntu 16.04.3 as gcc changed preprocessor output.

Jens Harbott took care of lack of space for data read from config-drive.

Paul Martin upgraded CirrOS build system to BuildRoot 2019.02.1 and bumped kernel/grub versions.

Maciej Józefczyk took care of metadata requests.

Marcin Sobczyk fixed starting of Dropbear and dropped creation of DSS ssh key which was no longer supported.

My Linaro work

At Linaro I got Jira card with “Upgrade CirrOS’ kernel to Ubuntu 18.04’s kernel” title.

This was needed as 4.4 kernel was far too old and gave us several booting issues. Internally we had builds with 4.15 kernel but it should be done properly and upstream.

So I fetched code, did some test builds and started looking how to improve situation. Spoke with Scott Moser (owner of CirrOS project) and he told me about his plans to migrate from Launchpad to GitHub. So we did that in December 2019 and then fun started.

Continuous Integration

GitHub has several ways of adding CI to projects. First we tried GitHub Actions but turned out that it is paid service. Looked around and then I decided to go with Travis CI.

Scott generated all required keys and integration started. Soon we had every pull request going through CI. Then I added simple script (bin/test-boot) so each image was booted after build. Scott improved script and fixed Power boot issue.

Next step was caching downloads and ccache files. This was huge improvement!

In meantime Travis bumped free service to 5 simultaneous builders which got our builds even faster.

CirrOS supports building only under Ubuntu LTS. But I use Fedora so we merged two changes to make sure that proper ‘grub(2)-mkimage’ command is used.

Kernel changes

4.4 kernel had to go. First idea was to move to 4.18 from Ubuntu 18.04 release. But if we upgrade then why not going for HWE one? I checked 5.0 and 5.3 versions. As both worked fine we decided to go with newer one.

Modules changes

During start of CirrOS image several kernel modules are loaded. But there were several “no kernel module found” like messages for built-in ones.

We took care of it by querying /sys/module/ directory so now module loading is quiet process. At the end a list of loaded ones is printed.

VirtIO changes

Lot of things happened since 4.4 kernels. So we added several VirtIO modules.

One of results is working graphical console on AArch64. Thanks to ‘virtio-gpu’ providing framebuffer and ‘hid-generic’ handling usb input devices.

As lack of entropy is common issue in VM instances we added ‘virtio-rng’ module. No more ‘uninitialized urandom read’ messages from kernel.

Final words

Yesterday Scott created 0.5.0 tag and CI built all release images. Then I wrote release notes (based on ones from pre-releases). Kolla project got patch to move to use new version.

When next release? Looking at history someone may say 2023 as previous one was in 2016 year. But know knows. Maybe we will get someone with “please add s390x support” question ;D

Some time ago at one of Red Hat mailing lists someone asked “how has open source helped your career”. There were several interesting stories. I had mine as well.

2000

My first contribution to FOSS. It was updating Debian ‘potato’ installation guide for Amiga/m68k. I was writing article to new Amiga magazine called ‘eXec’ about installing Debian. So why not update official instruction at same time?

2002

Probably my first code contribution: small change to MPlayer. I completely forgot about it but as project was changing it’s license in 2017 I got an email about it.

2004

I bought my 3rd PDA (Sharp Zaurus SL-5500) and it was running Linux. I started building apps for it, hacking system to run better. Then cooperated with OpenZaurus distro developers and started contributing to OpenEmbedded build system. One day they gave me write access to repo and told to merge my changes.

When I stopped using OE few years later I was the 5th on list of top contributors.

I also count this year as first one of my FOSS career.

2005

Richard Jackson donated Zaurus c760 to me. As a gift for my OpenZaurus work. And then OPIE 1.2.2 release came due to my changes to make better use of VGA screen. I still have this device in running condition.

2006

Became release manager of OpenZaurus distribution, with team of users testing pre-release images. Released 3.5.4 (and later 3.5.4.1 and 3.5.4.2-rc) version.

Started my own consulting company. Got some serious customers. End of work as PHP programmer.

2007 - 2010

I am doing what was hobby as full time job. Full FOSS work. Different companies, ARM architecture for 95% of time. Mostly consulting around OpenEmbedded.

2010

Due to my ARM foss involvement Canonical hired me. Started working at Linaro as software engineer. Cleaned cross compilers in Ubuntu/Debian, several other things.

2012

Became one of first AArch64 developers. Published OpenEmbedded support for it right after all toolchain patches became public.

2013

Left Linaro and Canonical, wrote about it on blog and in less then hour got “send me your CV” from Jon Masters from Red Hat. Joined company, did lot of changes in RHEL 7 and Fedora — mostly fixing FTBFS on !x86 architectures.

2016

My manager asked me do I want to go back to Linaro. This time as Red Hat assignee. Went, met old chaps, working mostly around OpenStack. Still on 64bit Arm.

2017 - 2020

Lot of work in OpenStack. Some work on Big Data stuff for other team at Linaro. Countless projects where I worked on getting stuff working on AArch64.

Summary

My whole career is built on FOSS.

My x86(-64) desktop runs GNU/Linux since day one (September 2000) as main system. There was OpenDOS as second during studies due to some stuff.

I had MS Windows XP as second system on one of laptops. But that’s due to some Arm hardware bringup tool being available only for this OS (later also for Linux). My family and friends learnt that I am unable to help them with MS Windows issues as I do not know that OS.

FOSDEM. In my opinion the best IT conference. Each year. And I was there for 12th time.

Insane amount of talks (893 this time) allows to choose more than it is possible to see. Which is good because with thousands of attendees it is often impossible to enter the room. Having some headphones helps as everything is live streamed so later there are videos to download (mostly after conference as they go through review process).

Friday

Woke up at 3:45, shower, breakfast, taxi at 4:30, bus at 5:00, plane at 8:40 (TXL -> BRU). Met a friend at the bus, watched 2nd episode of “Star Trek: Picard” and tried to conserve energy for the rest of a day.

Landed, took a train to Brussels Nord, added some tickets to MOBIB card and went to CentOS Dojo as usual on FOSDEM’s Friday.

Cloud SIG talk turned into discussion how many projects are waiting for their packages. Thorsten’s talk about CentOS on desktop had some interesting points — you install once and use to the death of hardware. Then went to talk about Software Collections. It was Jan’s first presentation ever and went quite ok.

Skipped armhfp talk and went for some food. And then hours at Delirium. I know that crowd makes more and more people going somewhere else. I go there as it is easy way to discuss with friends or people who recognize me.

Saturday

Took 71 bus at first stop and then went for talks…

Thorsten gave a great talk about changes in Linux kernel over twenty years. I was a bit late so watched whole talk on my way back home.

Next one was about selfish contributors. By James Bottomley. Great one! Interesting comparisons. Worth watching.

I skipped some talks from my list and discussed with several friends instead.

Next one was in AW - Stylo editor. Looked as done by academics for academics. Interesting approach. Not something for me but I understand why it was created.

UEFI: edk2 and U-Boot. Two interesting talks one after another. I wonder will my home desktop pass SCT.

Wanted to attend talk about loading fonts but room was already full. So decided to visit Embedded room instead.

Talk about Yocto Project tools was boring. But it turned out that friend leads devroom so we had nice talk during break.

Sudo talk was full so decided to go back to the hotel. Dropped stuff and went for some food and beer with friends.

Sunday

Early wake up, breakfast and then we took a risk of getting 71 on 3rd stop. Interesting challenge. Managed to squeeze into the second bus.

First was one about Thunderbird. There are interesting changes coming. Enigmail and Lighting will be integrated and new UI will come too.

Then Community room. During first talk Matt listed several chat platforms used by current generation of contributors. IRC was not one of them. I am old.

Next was about ethics in Open Source. Kind of “are all four freedoms are always needed”. Worth watching.

Then pile of bad luck. Virtualization devroom was full. I planned to attend talk about virtio-fs which just went into kernel and lands in qemu.

Open Source Design one had long queue. Once I managed to enter room, I left. No seats so UI/UX tricks have to wait in queue of videos to watch.

Went back to the K building. Took polar from cloakroom (left there a day before) and went for “coreboot on AMD processors” talk. Met Marek so decided to allow other people go. It was “go for fun” category anyway as I no longer have any hardware supported by this project.

As I got tired, there could be just one decision: go to Janson to find a good seat for Maddog’s talk and all after.

Talk about postmarketOS and Maemo Leste showed that not much changed since Openmoko times. Long list of different attempts to make OS for mobile phones. And that there are people still using Nokia n900 “so called phone”.

Maddog was old as usual. Great talk. Definitely to watch if you missed.

And then was one about FOSDEM history. Several facts, funny moment with name of organizer of first OSDEM (watch video!). Have to watch it again as presenter was quite hard to understand. Turned out that attendees took “come in oldest FOSDEM t-shirt you have” serious as each year was covered!

Exhausted went to the hotel. Dropped stuff, ate something and went for beer with friends. And sleep. Without friends.

Monday

This time OpenEmbedded workshop organized by Philip Balister. Was good to meet some old friends.

Talks went from containers to BSPs, signing binaries and some other stuff I skipped.

Last talk was about past, present and future. Here I tried to help with some facts. And idea of collecting history of OE came. Will look at it - wiki page should be enough. Just finding facts and people who remember will take time.

Videos

During meeting I download set of videos for way back home.

“18 years of Blender” was great! I spent nearly an hour at TXL airport watching “Elephant’s dream”, “Big Buck Bunny”, “Sintel” and “Tears of steel”. Will watch rest of them the other day.

Turned out that talk about font loading was worth fetching. Have to check my blog and maybe do some tweaks.

One about coreboot was like I expected. Skipped most of it to check what was on slides.

Nouveau status update I ended even faster. Will check website is there any hope for using it on GTX 1050Ti as so far I use closed source one.

Summary

Will I go next year? Sure, I do — I can not afford to not be there.

Arm world is weird. You can have server (expensive, UEFI, out of the box supported by any mainline distribution, normal SATA/NVME storage) or SBC (cheap, usually U-Boot, not supported for months by any distribution, microsd/USB storage mostly). But I do not want to talk about servers today.

Bootloader(s) mess

So SBC… Which means playing with U-Boot. In the past it meant random fork of it. I have a feeling (can be wrong) that it changed. Board starts with mainline one or quite fresh fork with mainlining in progress. Similar with ATF (Arm Trusted Firmware - level0 bootloader).

So you have SBC, you have bootloader(s) built. The next step is taking microsd card, finding out which offsets to use. Then put bootloader(s) at proper places so CPU can read them and boot. Usually it also means that you have to use MBR style formatting as bootloader is where GPT would be.

Better way maybe?

Is there a better way? Sure it is. But it is very expensive. Or at least I have a feeling that it is for most of SBC vendors.

Very expensive solution

The solution is SPI flash chip. But it is very expensive - 3 (three) EUR for 512Mb (64MB) chip (if bought 1000 of them). Far too much for 99.99% of SBC probably.

Price taken from simple search on Digikey: W25Q512JV 512 Mb Serial Flash Memory.

64 megabytes is more than enough to store all. Bootloader(s), U-Boot/UEFI settings etc. You can even fit small OS image there.

Probably 64/128Mb will be enough even. And they are cheaper (128Mb is 1.7 EUR). Still too expensive…

“Whatever you do, do it upstream” is IMHO one of mottos worth following. So I went upstream wherever possible.

Python

2020 came, Python 2 is no more. So 2019 was full of cleaning and patching. I filled several bugs on projects, created patches to mark some Python packages as Py2 only (or <Py3.5 only). Some projects just dropped support for Py2 packages due to it.

pbr story

With move to Python 3 came some issues with Unicode characters in README files of packages used by OpenStack components. Filled some bugs, created patches and then, after several emails, someone found out that it is a bug in ‘pbr’ and fixed it ;D

manylinux2014

Installing Python packages on AArch64 can be painful. While “pip install numpy” takes few seconds on x86-64 machine it can take an hour on Arm64 + extra time to find out which libraries and compilers need to be installed.

So Python developers created PEP-599 to sort out situation and create ‘manylinux2014’ target which can be then used to build ‘wheels’ (aka binary Python packages) not only for x86(-64) but also for 32-bit Arm, 64-bit Arm, 64-bit Power (big and little endian) and Z/Arch (s390x).

Somehow I managed to be involved in it. Went through pull request on GitHub, build checked, fixed dependencies and some other issues, filled CentOS bugs, discussed with CentOS developers on fixes and then one day all that stuff got merged.

Now you can fetch “manylinux2014” images from “pypa” account on Quay.io container registry and build ‘wheel’ binaries for different architectures. If you lack target hardware then consider services like Travis CI which provide access (free for FOSS projects).

CirrOS

Speaking of Travis CI… Near the end of the year I got task from one of Linaro teams to update CirrOS images with kernels from Ubuntu 18.04 or later.

I fetched current source, built images and started wondering how to do it best. So started discussing with Scott Moser (main developer) and we agreed that such move have sense and that some changes are needed in this project.

In December we moved it from Launchpad/cirros to GitHub/cirros-dev/cirros. Opened several issues to not forget what we planned and I started looking at some of them.

Project is now using Travis CI service for tests. With all builds being run parallel we have checks done in less than and hour. Including boot test for aarch64, arm, i386 and x86_64 architectures (ppc/ppc64/ppc64le need work).

Doing releases on GitHub is one of next tasks. Current service will redirect.

Linaro ERP

After ERP 18.12 release we decided that there will be no more releases and Debian ‘buster’ will be used as it is (ERP based on Debian ‘stretch’). So I spent some time on working with Debian kernel maintainers.

Most important part? Merging kernel configuration changes. We had about twenty extra enabled to get our servers supported. Everything worth using got merged. In meantime we even found a bug in Linux-stable tree which took some time to fix.

We were ready for Debian ‘buster’ release. And it worked out of the box on all machines we supported ;D

Big Data

Bleh. Java, protobuf 2.5 and other madness… But as build engineer I get such stuff from time to time.

Last year it was mostly Apache Drill and Apache Arrow. In both cases I worked on their Debian packaging (not in-Debian one) to add/fix AArch64 support.

Other projects still wait for ‘let us move to newer protobuf’ but it will take years…

Quite long time ago I installed Planet Venus on my server to have RSS aggregator for my own use. More about in an old post about Developer planet.

Python 2 strikes again

But I forgot that it was written in Python 2… So once I removed Py2 from my server website updates stopped. No one noticed ;D

Yesterday I looked at situation — there are no updated versions, looks like no one plans to work on Python 3 porting. One of options was move to other aggregator written in Ruby but I do not know that language so it was out of question.

Let’s write own one

So I decided to take a look at Python libraries. Feedparser looked useful and after few hours of coding I had something working.

I called it Very Simple Planet Aggregator and put on my GitHub account.

Requirements are simple: Feedparser (parsing feeds), Jinja2 (templates) and sgmllib3k (to get relative urls working in feeds). Small bit of documentation is provided in repository. No tests so far.

Future

I need to add RSS template and do some tests with Feedparser provided feeds to check did I catched all possible errors.

And need to deploy it on server and move Developer Planet to use it.

Another year passed so let me try to summarize it. And a lot of things happened so it will be set of posts.

Kolla

We did two releases: ‘stein’ in July and ‘train’ in December. As usual a bit later than most of OpenStack projects as we have to wait for their releases, then for distributions to provide their binary packages, test everything etc.

2019 was also a year of moving to Python 3 in our project. Images went in rounds — first some in Stein, then rest in Train cycle. We managed to remove Python 2 support from Kolla-Ansible but have to wait for TripleO project before we drop Py2 in Kolla itself.

Also, we removed OracleLinux support. No one from active developers had access to it and jobs were failing on random moments. CentOS 7 support covered all needs in our opinion.

CentOS 8 was hoped to be ready for cloud use. But it was not due to many changes needed in build infrastructure. Now, after 8.1.1911 release, there is a hope that situation will change and SIGs will be able to build their packages.

On the other hand, we added support for IPv6 deployments. Linaro developers helped with testing as our second Linaro Developer Cloud will be IPv6 only and all resources will be provided for OpenStack infrastructure.

There was new OpenStack project called Kayobe. It used both Kolla and Kolla-Ansible (and some other projects) to deploy containered OpenStack on bare metal. Including provisioning. We accepted it as subproject of Kolla as some of our members work in all those projects.

Infrastructure

During 2019 I got huge amount of help from OpenStack infrastructure team. We changed mirroring of CentOS, Debian, Ubuntu and several external repositories for any of those distributions. AArch64 was added where it was needed, changes got reviewed quickly. Perfect job!

Continuous Integration

After long fight with issues we finally have working AArch64 CI for OpenStack projects. Linaro provided some vcpu in Linaro Developer Cloud, infrastructure admins created some nodes from it and then created ‘check-arm64’ pipeline on Zuul.

This way any OpenStack project can add tests on 64-bit Arm hardware without impacting their standard set of tests. We went that way due to low amount of nodes and it turned out to be very good move. Kolla(-ansible) itself is able to saturate all of them during developer heavy days.

More nodes are coming. Managed by Linaro on sponsored hardware. Pure IPv6!

Python runs everywhere, right? All those libraries are just one ‘pip install’ away. And we are used to it. Unless on AArch64.

On AArch64 when you do pip install SOMETHING you may end with “no compiler installed” or “No lapack/blas resources found.” messages. All due to lack of wheel files generated for this architecture… And even if you have all dependencies installed then building takes more time than it takes to install existing wheel file.

But there is a light!

PEP 599 defined “manylinux2014” target which is the first supporting something else than just “x86(-64)” architecture. The new arrivals are AArch64, PPC64 (Big and Little Endian) and 32-bit Arm.

I helped getting them working on AArch64. Do not remember how I ended there to be honest ;D

And those images are now available in Pypa repository on quay.io.

And there is more!

If your project uses Travis CI for testing and release then adding support for non-x86 architectures is just one edit away. All you need is entry in “arch” list of in “matrix”:

matrix:
  include:
   - os: linux
     arch: amd64
   - os: linux
     arch: arm64

And it really works!

I added support for it in “dumb-init” so now kolla does not need to use any workarounds but can grab binary straight from project’s releases. Took just few simple lines in “.travis.yml” file.

What is left to do now?

There are many Python projects out there. Most of them do not know that manylinux2014 got released and what it brings. Or that Travis CI can give them non-x86 architecture support. I slowly started creating issues, bug reports in them to make sure they are aware.