Marcin Juszkiewicz

OpenStack users are used to have graphical console available. They take it for granted even. But we lacked it…

We want graphics on AArch64

When we started working on OpenStack on 64-bit ARM there were many things missing. Most of them got sorted out already. One thing was still in a queue: graphical console. So two weeks ago I started looking at the issue.

Whenever someone tried to use it Nova reported one simple message: “No free USB ports.” You can ask what it has to console? I thought similar and started digging…

Long live architecture differences

As usual reason was simple: yet another aarch64 <> x86 difference in libvirt. Turned out that arm64 is one of those few architectures which do not have USB host controller in default setup. When Nova is told to provide graphical console it adds Spice (or VNC) graphics, video card and USB tablet. But in our case VM instance does not have any USB ports so VM start fails with “No free USB ports” message.

Let patch Nova

Solution was simple: let’s add USB host controller into VM instance. But where? Should libvirt do that or should Nova? I discussed it with libvirt developers and got mixed answers. Opened a bug for it and went to Nova hacking.

Turned out that Nova code for building guest configuration is not that complicated. I created a patch to add USB host controller and waited for code reviews. There were many suggestions, opinions and questions. So I rewrote code. And then again. And again. Finally 15th version got “looks good” opinion from all reviewers and got merged.

Final effect

And how result looks? Boring as it should:

Do you know what is the most popular PC machine nowadays? It is “simple” PCI based x86(-64) machine with i440fx chipset and some expansion cards. Everyone is using them daily. Often you do not even realise that things you do online are handled by machines from 90s.

What? 90s hardware in modern world?

Sounds like a heresy? Who would use 90s hardware in modern world? PCI cards were supposed to be replaced by PCI Express etc, right? No one uses USB 1.x host controllers, hard to find PS/2 mouse or keyboard in stores etc. And you all are wrong…

Meet virtual PC

Most of virtual machines in x86 world is a weird mix of 90s hardware with a bunch of PCI cards to keep them usable in modern world. Parallel ATA storage went to trash replaced by SCSI/SATA/virtual ones. Graphic card is usually simple framebuffer without any 3D acceleration (so like 90s) and typical PS/2 input devices connected. And you have USB 1.1 and 2.0 controllers with one tablet connected. Sounds like retro machines my friends prepare for retro events.

You can upgrade to USB 3.0 controller, graphic card with some 3D acceleration. Or add more memory and cpu cores that i440fx based PC owner ever dreamt about. But it is still 90s hardware.

What about some XXI century hardware?

Want to have something more modern? You can migrate to PCI Express. But nearly no one does that in virtual x86 world. And in AArch64 world we start from here.

And that’s the reason why working with developers of projects related to virtualization (qemu, libvirt, openstack) can be frustrating.

Issues with hotplug?

Hotplug issues? Which hotplug issues? My VM instance allows to plug 10 cards while it is running so where is a problem? The problem is that your machine is 90s hardware with simple PCI bus and 31 slots present on virtual mainboard while VM instance with PCI Express (aarch64, x86 with q35 model) has only TWO free slots present on motherboard. And once they are used no new slots arrive. Unless you shutdown, add free slots and power up again.

Architecture differences?

Or my recent stuff: adding USB host controller. x86 has it because someone ‘made a mistake’ in past and enabled it. But other architectures got sorted out and did not get it. Now all of them need to have it added when it is needed. Have a patch to Nova for it and I am rewriting it again and again to get it into acceptable form.

Partitioning is fun too. There are so many people who fear switching to GPT…

What connects those names? All of them are conference or team meeting names. Spread over Europe + Asia. And I will be on all of them this year.

First will be in Brno, Czechia. Terrible city to travel to but amount of people I can meet there is enormous. Some guys from my Red Hat team will be there, my boss’ boss (and boss of my boss’ boss) and several people from CentOS, Fedora, RHEL (and some other names) communities. Meetings, sessions… Nice to be there. Will be in A-Sport hotel as it is closest to Red Hat office where I have some meetings to attend.

Then FOSDEM. The only “week-long conference squeezed into two days” I know. Good luck with meeting me probably. As usual my list of sessions covers all buildings and have several conflicts. Will meet many people, miss even more, do some beers. This year I stay in Floris Arlequin Grand-Place hotel.

Next one? OpenStack PTG in Dublin, Ireland. Finally will meet all those developers reviewing my patches, helping me with understanding source code of several OpenStack projects. And probably answering several questions about state of AArch64 support. Conference hotel.

Linaro Connect. Hong Kong again. Not registered yet, not looked at flights. Have to do that sooner than later and checking which airline sucks less on intercontinental connections sucks too. With few members of SDI team I will talk about our journey through/with OpenStack on our beloved architecture. Conference hotel.

What after those? Probably OpenSource Day in Poland, maybe some other ones. Will see.

Let me introduce new awesome project: YADIBP. It is cool, foss, awesome, the one and only and invented here instead of there. And it does exactly what it has to do and in a way it has to be done. Truly cool and awesome.

Using that tool you can build disk images with several supported Linux distributions. Or maybe even with any BSD distribution. And Haiku or ReactOS. Patches for AmigaOS exist too!

Any architecture. Starting from 128 bit wide RUSC-VI to antique architectures like ia32 or m88k as long as you have either hardware or qemu port (patches for ARM fast models in progress).

Just fetch from git and use. Written in BASIC so it should work everywhere. And if you lack BASIC interpreter then you can run it as Python or Erlang. Our developers are so cool and awesome!

But let’s get back to reality — there are gazillions of projects of tool which does one simple thing: builds a disk image. And gazillion will be still written because some people have that “Not Invented Here” syndrome.

And I am getting tired of it.

I am working on getting OpenStack running on AArch64 architecture, right? So recently I went from “just” building images to also using them to deploy working “cloud” setup. And that resulted in new sets of patches, updates to patches, discussions…

OpenStack is supposed to make virtualization easier. Create accounts, give access to users and they will make virtual machines and use them without worrying what kind of hardware is below etc. But first you have to get it working. So this week instead of only taking care of Kolla and Kolla-ansible projects I also patched Nova. The component responsible for running virtual machines.

One patch was simple edit of existing one to make it comply with all comments. Took some time anyway as I had to write some proper bug description to make sure that reviewers will know why it is so important for us. And once merged we will have UEFI used as default boot method on AArch64. Without any play with hw_firmware_type=uefi property on images (which is easy to forget). But this was the easy one…

Imagine that you have a rack of random AArch64 hardware and want to run a “cloud”. You may end in a situation where you have a mix of servers for compute nodes (the ones where VM instances run). In Nova/libvirt it is handled by cpu_mode option:

It is also possible to request the host CPU model in two ways:

• “host-model” - this causes libvirt to identify the named CPU model which most closely matches the host from the above list, and then request additional CPU flags to complete the match. This should give close to maximum functionality/performance, which maintaining good reliability/compatibility if the guest is migrated to another host with slightly different host CPUs. Beware, due to the way libvirt detects host CPU, CPU configuration created using host-model may not work as expected. The guest CPU may confuse guest OS (i.e. even cause a kernel panic) by using a combination of CPU features and other parameters (such as CPUID level) that don’t work.

• “host-passthrough” - this causes libvirt to tell KVM to passthrough the host CPU with no modifications. The difference to host-model, instead of just matching feature flags, every last detail of the host CPU is matched. This gives absolutely best performance, and can be important to some apps which check low level CPU details, but it comes at a cost wrt migration. The guest can only be migrated to an exactly matching host CPU.

Nova assumes host-model when KVM/QEMU is used as hypervisor. And crashes terribly on AArch64 with:

libvirtError: unsupported configuration: CPU mode ‘host-model’ for aarch64 kvm domain on aarch64 host is not supported by hypervisor

Not nice, right? So I made a simple patch to get host-passthrough to be default on AArch64. But when something is so simple then it’s description is probably not so simple…

Reported bug on nova with some logs attached. Then digged for some information which would explain issue better. Found Ubuntu’s bug on libvirt from Ocata times. They used same workaround.

So I thought: let’s report a bug for libvirt and request support for host-model option. There I got link to an another bug in libvirt with set of information why it does not make sense.

The reason is simple. No one knows what you run on when you run Linux on AArch64 server. In theory there are fields in /proc/cpuinfo but still you do not know do cpu cores in compute01 are same as compute02 servers. At least from nova/libvirt/qemu perspective. This also blocks us from setting cpu_mode to custom and selecting cpu_model which could be some way of getting same cpu for each instance despite of types of compute node processor cores.

The good side is that VM instances will work. The problem may appear when you migrate VM to cpu with other core — it may work. Or may not. Good luck!

January

• Quiet time. Just after Linaro ERP release was done.

February

• FOSDEM as usual. This time with CentOS/RDO meeting.
• Took my daughter Mira to Touluse, France for winter holidays. There was about 17°C which was nice change after week of snow. Visited Aerospace museum and Space city. Nice trip.
• Started working on Kolla. Took few months to get non-x86 support merged and then I became core reviewer. Learnt a lot during those months.

March

• Linaro Connect in Budapest. Met friends, saw some nice places in a city. Arrow company presented several boards compatible with 96boards specifications including Enterprise Edition one.
• “Root Linux” conference in Kiev, Ukraine. Gave a talk about OpenStack on AArch64 (youtube video).
• OpenStack Day in Warsaw, Poland. Few interesting prelections, lot of talks with other developers.

April

• OpenStack, OpenStack and OpenStack. And rewriting patches all over again and again.

May

• Support for non-x86 support finally landed in Kolla. My goal was to get AArch64 supported and got POWER8/LE support as free bonus.

June

• With a friend we went for Sandra concert. Somehow I remembered city name wrong and instead of 1 hour drive it was 4 hour one. Still fun.
• Went for long weekend trip to Cologne, Paderborn, Düsseldorf. Ok, main reason was Paderborn where I visited Heinz Nixdorf MuseumsForum (described as biggest computer museum in the world) and Universität Paderborn where main Aminet server was hosted when I started my Amiga adventure. Museum was nice but I enjoyed Computer History Museum more.
• Built first set of OpenStack ‘Pike’ container images for Debian/AArch64.
• Started working on DPDK related issues. Then we got assignee to work on it and I only provided updated packages.

July

• Visited Linaro office in Harston for a release sprint. And we celebrated my birthday in a pub. Was great.

August

• Helped organizing Ingress Anomaly in Szczecin, Poland. We got aroung 1200 agents from many countries. My job was to take care of missions, mosaics and Mission Day. It was exhausting and we have a lot of fun.

September

• Attended Riverwash demoscene party.
• Went with a friends for a week in Bieszczady mountains. Nearly out of any civilisation related stuff.
• Another Linaro Connect. This time around San Francisco. And new 96boards Enterprise Edition hardware — this time in MicroATX form factor: Socionext SynQuacer with 24 cores.
• Spent a day in Computer History Museum and still have a feeling that it was not enough time to see everything ;D

October

• Attended Retrokomp/Load Error event.
• Donated blood for a first time. Will repeat.
• Started playing with Bigtop project. Lot of Java stuff, Docker images and porting issues. Reported several bugs, replaced their old AArch64 build slave with two new ones (all three run in Linaro Developer Cloud).

November

• Became core reviewer in Kolla project
• Another Ingress Anomaly — this time Berlin.

December

• Attended Silly Venture demoscene party. It is amazing what people can get running on those old Atari machines.

From time to time I try to change web browser (switch Firefox <> Chrome). This time it is moving to Firefox Quantum (v57). And have to say that I have very mixed opinion.

For years it was easy: Chrome is faster, Firefox has extensions which can alter how browser look, feel, work, behave. From Firefox Quantum it is gone. All add-ons have now be “so called” WebExtensions - no way to alter browser itself, only what is presented on web page can be changed.

Say good bye to switching tabs with mouse scroll - function was always missing in Firefox but there was extension for it. Same with tab grouping in tab bar - “Tree Style Tab” is now sidebar and original tab bar has to be disabled through userChrome.css file. Good that they got at least moved reload/stop button to the left side of location…

I will use it for week or two and see it stay or not on my desktop instead of Chrome. Have to admit that main reason for test is tab grouping function in Tree Style Tab as it allows me to get rid of multiple browser windows.

Also I have limited amount of extensions in use to just six ones related to ad blocking/privacy/user scripts.

Months of work, tens of patches, hundreds of changed lines. My whole work in Kolla project got rewarded this week. I am now one of core reviewers :)

What does it mean? I think that Gema summarised it best:

For those of you who don’t know, this means Kolla has recognised our contributions to the project as first class and are giving Marcin and ARM64 a vote of confidence, they realise we are there to stay.

I found it helpful in my daily work as now I can suggest my coworkers to send their patches directly instead of proxying it through me ;D