Tag: openstack

SnowpenStack in Dublin

During last week I was in Dublin, Ireland on OpenStack PTG. It was also the worst weather since 1982. There was snow and strong wind so conference quickly got renamed to SnowpenStack.

The main reasons for me to be there were:

  • meet all those developers who took some time and looked at my changes
  • discuss some other changes/plans
  • share aarch64 knowledge with OpenStack projects

Conference took place in the Croke Park stadium. We used meeting rooms on 4th, 5th, 6th floors. One day by mistake I took wrong stairs and ended on top of the stadium in just T-shirt… Quickly ran to an elevator to get back to proper floor ;D

The schedule was split into two parts: Monday and Tuesday were for mixed teams sessions while Wednesday — Friday were for discussions in teams. I spoke with Kolla, Nova and Infra teams mostly. There were some discussions with Ironic, Kuryr and some other ones too. Also met several Polish developers so there was a time to speak in native language ;D

On Tuesday I went to the city centre to buy some souvenirs for family (and 99th fridge magnet for myself). Launched Ingress, did one mosaic to see more of a city and after 11 kilometres I was back at the hotel just in time for a small party in GAA museum. And then pub trip with Polish guys. When I finally reached the hotel (about 01:30) there were still discussions in the lobby and I took a part in one of them.

Team discussions started on Wednesday. Visited Nova one summarizing ‘Queens’ release. Turned out that it went better than previous ones. The main problem was lack of reviews — not everyone likes to pester developers on IRC to get some attention to patches. I was asked few times for opinion as I was one of few fresh contributors.

Kolla sessions were a bit chaotic in my opinion. Recently chosen PTL was not present and the person supposed to replace him got stuck at home due to weather. One of the discussions I remember was about Ceph: should we keep using our images or rather move to ‘ceph-ansible’ instead. Final idea was to keep as it looked like there were more cons than pros with moving to ‘ceph-ansible’ images.

Discussed Arm64 support with Infra team. We (Linaro) provided them resources on one of our developer clouds to get aarch64 present in OpenStack CI gates. Turned out that machines work and some initial tests were done. I also got informed that diskimage-builder patches to add GPT/UEFI support will be reviewed soon.

And then there were some weather related issues. On Wednesday every attendee got an email with information that Irish government issued the Red Alert which strongly suggest to stay inside if you do not really have to go outside. And as attendance was not mandatory then people should first check are they comfortable with going to Croke Park (especially those who not stayed in the hotel nearby). Next day organization team announced that the venue we used will close after lunch to make sure that everyone is safe. And the whole conference moved to the hotel…

Imagine developers discussing EVERYWHERE in a hotel. Lobby was occupied by few teams, Infra found a table in library corner, Nova took Neutron and occupied breakfast room. Bar area was quite popular and soon some beers were visible here and there. Few teams went to meeting rooms etc. WiFi bandwidth was gone… Some time later hotel staff created a separate wireless network for our use. And then similar situation was on Friday.

On Wednesday other thing happened too: people started receiving information that their flights are cancelled. There were some connections on Thursday and then nothing was flying on Friday. Kudos to hotel staff to be aware of it — they stopped taking external reservations to make sure that PTG attendees have a place to stay for longer (as some people got rebooked to even Thursday).

And even on Saturday it was hard to get to the airport. No taxi going to the hotel due to snow on a street. But if you walked 500 meters then cab could be hailed on a street. Many people went for buses (line 700 was the only working one). The crowd on the airport was huge. Some of those people looked like they lived there (which was probably true). Several flights were delayed (even by 4-5 hours), other got cancelled but most of them were flying.

Despite the weather sitting in a hotel in Dublin was safe, walking around too as there were about 15-20 centimetres of snow on a street. There were several snowmen around, people had fun playing with snow. But at same time local news were informing that 30 000 homes lacked electricity and some people got stuck in their cars. There was no public transport, no trains, no buses. Much smaller amount of people on streets.

Was it worth attending? Yes. Will I attend next ones? Probably not as this is very developer related where I spend most of my OpenStack time around building it’s components or doing some testing.

OpenStack ‘Queens’ release done

OpenStack community released ‘queens’ version this week. IMHO it is quite important moment for AArch64 community as well because it works out of the box for us.

Gone are things like setting hw_firmware_type=uefi for each image you upload to Glance — Nova assumes UEFI to be the default firmware on AArch64 (unless you set the variable to different value for some reason). This simplifies things as users does not have to worry about and we should have less support questions on new setups of Linaro Developer Cloud (which will be based on ‘Queens’ instead of ‘Newton’).

There is a working graphical console if your guest image uses properly configured kernel (4.14 from Debian/stretch-backports works fine, 4.4 from Ubuntu/xenial (used by CirrOS) does not have graphics enabled). Handy feature which we were asked already by some users.

Sad thing is state of live migration on AArch64. It simply does not work through the whole stack (Nova, libvirt, QEMU) because we have no idea what exactly cpu we are running on and how it is compatible with other cpu cores. In theory live migration between same type of processors (like XGene1 -> XGene1) should be possible but we do not have even that level of information available. More information can be found in bug 1430987 reported against libvirt.

Less sad part? We set cpu_model to ‘host-passthrough’ by default now (in Nova) so nevermind which deployment method is used it should work out of the box.

When it comes to building (Kolla) and deploying (Kolla Ansible) most of the changes were done during Pike cycle. During Queens’ one most of the changes were small tweaks here and there. I think that our biggest change was convincing everyone in Kolla(-ansible) to migrate from MariaDB 10.0.x (usually from external repositories) to 10.1.x taken from distribution (Debian) or from RDO.

What will Rocky bring? Better hotplug for PCI Express machines (AArch64/virt, x86/q35 models) is one thing. I hope that live migration stuff situation will improve as well.

Hotplug in VM. Easy to say…

You run VM instance. Nevermind is it part of OpenStack setup or just local one started using Boxes, virt-manager, virsh or other that kind of fronted to libvirt daemon. And then you want to add some virtual hardware to it. And another card and one more controller…

Easy to imagine scenario, right? What can go wrong, you say? “No more available PCI slots.” message can happen. On second/third card/controller… But how? Why?

Like I wrote in one of my previous posts most of VM instances are 90s pc hardware virtual boxes. With simple PCI bus which accepts several cards to be added/removed at any moment.

But not on AArch64 architecture. Nor on x86-64 with Q35 machine type. What is a difference? Both are PCI Express machines. And by default they have far too small amount of pcie slots (called pcie-root-port in qemu/libvirt language). More about PCI Express support can be found in PCI topology and hotplug page of libvirt documentation.

So I wrote a patch to Nova to make sure that enough slots will be available. And then started testing. Tried few different approaches, discussed with upstream libvirt developers about ways of solving the problem and finally we selected the one and only proper way of doing it. Then discussed failures with UEFI developers. And went for help to Qemu authors. And explained what I want to achieve and why to everyone in each of those four projects. At some point I had seen pcie-root-port things everywhere…

Turned out that the method of fixing it is kind of simple: we have to create whole pcie structure with root port and slots. This tells libvirt to not try any automatic adding of slots (which may be tricky if not configured properly as you may end with too small amount of slots for basic addons).

Then I went with idea of using insane values. VM with one hundred PCIe slots? Sure. So I made one, booted it and then something weird happen: landed in UEFI shell instead of getting system booted. Why? How? Where is my storage? Network? Etc?

Turns out that Qemu has limits. And libvirt has limits… All ports/slots went into one bus and memory for MMCONFIG and/or I/O space was gone. There are two interesting threads about it on qemu-devel mailing list.

So I added magic number into my patch: 28 — this amount of pcie-root-port entries in my aarch64 VM instance was giving me bootable system. Have to check it on x86-64/q35 setup still but it should be more or less the same. I expect this patch to land in ‘Rocky’ (the next OpenStack release) and probably will have to find a way to get it into ‘Queens’ as well because this is what we are planning to use for next edition of Linaro Developer Cloud.

Conclusion? Hotplug may be complicated. But issues with it can be solved.

Graphical console in OpenStack/aarch64

OpenStack users are used to have graphical console available. They take it for granted even. But we lacked it…

When we started working on OpenStack on 64-bit ARM there were many things missing. Most of them got sorted out already. One thing was still in a queue: graphical console. So two weeks ago I started looking at the issue.

Whenever someone tried to use it Nova reported one simple message: “No free USB ports.” You can ask what it has to console? I thought similar and started digging…

As usual reason was simple: yet another aarch64<>x86 difference in libvirt. Turned out that arm64 is one of those few architectures which do not have USB host controller in default setup. When Nova is told to provide graphical console it adds Spice (or VNC) graphics, video card and USB tablet. But in our case VM instance does not have any USB ports so VM start fails with “No free USB ports” message.

Solution was simple: let’s add USB host controller into VM instance. But where? Should libvirt do that or should Nova? I discussed it with libvirt developers and got mixed answers. Opened a bug for it and went to Nova hacking.

Turned out that Nova code for building guest configuration is not that complicated. I created a patch to add USB host controller and waited for code reviews. There were many suggestions, opinions and questions. So I rewrote code. And then again. And again. Finally 15th version got “looks good” opinion from all reviewers and got merged.

And how result looks? Boring as it should:

OpenStack graphical console on aarch64

Everyone loves 90s PC hardware?

Do you know what is the most popular PC machine nowadays? It is “simple” PCI based x86(-64) machine with i440fx chipset and some expansion cards. Everyone is using them daily. Often you do not even realise that things you do online are handled by machines from 90s.

Sounds like a heresy? Who would use 90s hardware in modern world? PCI cards were supposed to be replaced by PCI Express etc, right? No one uses USB 1.x host controllers, hard to find PS/2 mouse or keyboard in stores etc. And you all are wrong…

Most of virtual machines in x86 world is a weird mix of 90s hardware with a bunch of PCI cards to keep them usable in modern world. Parallel ATA storage went to trash replaced by SCSI/SATA/virtual ones. Graphic card is usually simple framebuffer without any 3D acceleration (so like 90s) and typical PS/2 input devices connected. And you have USB 1.1 and 2.0 controllers with one tablet connected. Sounds like retro machines my friends prepare for retro events.

You can upgrade to USB 3.0 controller, graphic card with some 3D acceleration. Or add more memory and cpu cores that i440fx based PC owner ever dreamt about. But it is still 90s hardware.

Want to have something more modern? You can migrate to PCI Express. But nearly no one does that in x86 world. And in AArch64 world we start from here.

And that’s the reason why working with developers of projects related to virtualization (qemu, libvirt, openstack) can be frustrating.

Hotplug issues? Which hotplug issues? My VM instance allows to plug 10 cards while it is running so where is a problem? The problem is that your machine is 90s hardware with simple PCI bus and 31 slots present on virtual mainboard while VM instance with PCI Express (aarch64, x86 with q35 model) has only TWO free slots present on motherboard. And once they are used no new slots arrive. Unless you shutdown, add free slots and power up again.

Or my recent stuff: adding USB host controller. x86 has it because someone ‘made a mistake’ in past and enabled it. But other architectures got sorted out and did not get it. Now all of them need to have it added when it is needed. Have a patch to Nova for it and I am rewriting it again and again to get it into acceptable form.

Partitioning is fun too. There are so many people who fear switching to GPT…

Devconf.cz? FOSDEM? PTG? Linaro Connect?

What connects those names? All of them are conference or team meeting names. Spread over Europe + Asia. And I will be on all of them this year.

First will be in Brno, Czechia. Terrible city to travel to but amount of people I can meet there is enormous. Some guys from my Red Hat team will be there, my boss’ boss (and boss of my boss’ boss) and several people from CentOS, Fedora, RHEL (and some other names) communities. Meetings, sessions… Nice to be there. Will be in A-Sport hotel as it is closest to Red Hat office where I have some meetings to attend.

Then FOSDEM. The only “week-long conference squeezed into two days” I know. Good luck with meeting me probably. As usual my list of sessions covers all buildings and have several conflicts. Will meet many people, miss even more, do some beers. This year I stay in Floris Arlequin Grand-Place hotel.

Next one? OpenStack PTG in Dublin, Ireland. Finally will meet all those developers reviewing my patches, helping me with understanding source code of several OpenStack projects. And probably answering several questions about state of AArch64 support. Conference hotel.

Linaro Connect. Hong Kong again. Not registered yet, not looked at flights. Have to do that sooner than later and checking which airline sucks less on intercontinental connections sucks too. With few members of SDI team I will talk about our journey through/with OpenStack on our beloved architecture. Conference hotel.

What after those? Probably OpenSource Day in Poland, maybe some other ones. Will see.

YADIBP

Let me introduce new awesome project: YADIBP. It is cool, foss, awesome, the one and only and invented here instead of there. And it does exactly what it has to do and in a way it has to be done. Truly cool and awesome.

Using that tool you can build disk images with several supported Linux distributions. Or maybe even with any BSD distribution. And Haiku or ReactOS. Patches for AmigaOS exist too!

Any architecture. Starting from 128 bit wide RUSC-VI to antique architectures like ia32 or m88k as long as you have either hardware or qemu port (patches for ARM fast models in progress).

Just fetch from git and use. Written in BASIC so it should work everywhere. And if you lack BASIC interpreter then you can run it as Python or Erlang. Our developers are so cool and awesome!

But let’s get back to reality — there are gazillions of projects of tool which does one simple thing: builds a disk image. And gazillion will be still written because some people have that “Not Invented Here” syndrome.

And I am getting tired of it.

Today I was fighting with Nova. No idea who won…

I am working on getting OpenStack running on AArch64 architecture, right? So recently I went from “just” building images to also using them to deploy working “cloud” setup. And that resulted in new sets of patches, updates to patches, discussions…

OpenStack is supposed to make virtualization easier. Create accounts, give access to users and they will make virtual machines and use them without worrying what kind of hardware is below etc. But first you have to get it working. So this week instead of only taking care of Kolla and Kolla-ansible projects I also patched Nova. The component responsible for running virtual machines.

One patch was simple edit of existing one to make it comply with all comments. Took some time anyway as I had to write some proper bug description to make sure that reviewers will know why it is so important for us. And once merged we will have UEFI used as default boot method on AArch64. Without any play with hw_firmware_type=uefi property on images (which is easy to forget). But this was the easy one…

Imagine that you have a rack of random AArch64 hardware and want to run a “cloud”. You may end in a situation where you have a mix of servers for compute nodes (the ones where VM instances run). In Nova/libvirt it is handled by cpu_mode option:

It is also possible to request the host CPU model in two ways:

  • “host-model” – this causes libvirt to identify the named CPU model which most closely matches the host from the above list, and then request additional CPU flags to complete the match. This should give close to maximum functionality/performance, which maintaining good reliability/compatibility if the guest is migrated to another host with slightly different host CPUs. Beware, due to the way libvirt detects host CPU, CPU configuration created using host-model may not work as expected. The guest CPU may confuse guest OS (i.e. even cause a kernel panic) by using a combination of CPU features and other parameters (such as CPUID level) that don’t work.

  • “host-passthrough” – this causes libvirt to tell KVM to passthrough the host CPU with no modifications. The difference to host-model, instead of just matching feature flags, every last detail of the host CPU is matched. This gives absolutely best performance, and can be important to some apps which check low level CPU details, but it comes at a cost wrt migration. The guest can only be migrated to an exactly matching host CPU.

Nova assumes host-model when KVM/QEMU is used as hypervisor. And crashes terribly on AArch64 with:

libvirtError: unsupported configuration: CPU mode ‘host-model’ for aarch64 kvm domain on aarch64 host is not supported by hypervisor

Not nice, right? So I made a simple patch to get host-passthrough to be default on AArch64. But when something is so simple then it’s description is probably not so simple…

Reported bug on nova with some logs attached. Then digged for some information which would explain issue better. Found Ubuntu’s bug on libvirt from Ocata times. They used same workaround.

So I thought: let’s report a bug for libvirt and request support for host-model option. There I got link to an another bug in libvirt with set of information why it does not make sense.

The reason is simple. No one knows what you run on when you run Linux on AArch64 server. In theory there are fields in /proc/cpuinfo but still you do not know do cpu cores in compute01 are same as compute02 servers. At least from nova/libvirt/qemu perspective. This also blocks us from setting cpu_mode to custom and selecting cpu_model which could be some way of getting same cpu for each instance despite of types of compute node processor cores.

The good side is that VM instances will work. The problem may appear when you migrate VM to cpu with other core — it may work. Or may not. Good luck!

I am now core reviewer in Kolla

Months of work, tens of patches, hundreds of changed lines. My whole work in Kolla project got rewarded this week. I am now one of core reviewers 🙂

What does it mean? I think that Gema summarised it best:

For those of you who don’t know, this means Kolla has recognised our contributions to the project as first class and are giving Marcin and ARM64 a vote of confidence, they realise we are there to stay.

I found it helpful in my daily work as now I can suggest my coworkers to send their patches directly instead of proxying it through me ;D

Is my work on Kolla done?

During last few months I was working on getting Kolla running on AArch64 and POWER architectures. It was a long journey with several bumps but finally ended.

When I started in January I had no idea how much work it will be and how it will go. Just followed my typical “give me something to build and I will build it” style. You can find some background information in previous post about my work on Kolla.

A lot of failures were present at beginning. Or rather: there was a small amount of images which built. So my first merged change was to do something with Kolla output ;D

  • build: sort list of built/failed images before printing

Debian support was marked for removal so I first checked how it looked, then enabled all possible images and migrated from ‘jessie’ to ‘stretch’ release. Reason was simple: ‘jessie’ (even with backports) lacked packages required to build some images.

  • debian: import key for download.ceph.com repository
  • debian: install gnupg and dirmngr needed for apt-key
  • debian: enable all images enabled for Ubuntu
  • handle rtslib(-fb) package names and dependencies
  • debian: move to stretch
  • Debian 8 was not released yet

Both YUM and APT package managers got some changes. For first one I took care to make sure that it fails if there were missing packages (which was very often during builds for aarch64/ppc64le). It allowed to catch some typo in ‘ironic-conductor’ image. In other words: I make YUM behave closer to APT (which always complain about missing packages). Then I made change for APT to behave more like YUM by making sure that update of packages lists was done before packages were installed.

  • ironic-conductor: add missing comma for centos/source build
  • make yum fail on missing packages
  • always update APT lists when install packages

Of course many images could not be built at all for aarch64/ppc64le architectures. Mostly due to lack of packages and/or external repositories. For each case I was checking is there some way for fixing it. Sometimes I had to disable image, sometimes update packages to newer version. There were also discussions with maintainers of external repositories on getting their stuff available for non-x86 architectures.

  • kubernetes: disable for architectures other than x86-64
  • gnocchi-base: add some devel packages for non-x86
  • ironic-pxe: handle non-x86 architectures
  • openstack-base: Percona-Server is x86-64 only
  • mariadb: handle lack of external repos on non x86
  • grafana: disable for non-x86
  • helm-repository: update to v2.3.0
  • helm-repository: make it work on non-x86
  • kubetoolbox: mark as x86-64 only
  • magnum-conductor: mark as x86-64 only
  • nova-libvirt: handle ppc64le
  • ceph: take care of ceph-fuse package availability
  • handle mariadb for aarch64/ubuntu/source
  • opendaylight: get it working on CentOS/non-x86
  • kolla-toolbox: use proper mariadb packages on CentOS/non-x86

At some moment I had over ten patches in review and all of them depended on the base one. So with any change I had to refresh whole series and reviewers had to review again… Painful it was. So I decided to split out the most basic stuff to get whole patch set split into separate ones. After “base_arch” variable was merged life became much simpler for reviewers and a bit more complicated for me as from now on each patch was kept in separate git branch.

  • add base_arch variable for future non-x86 work

At Linaro we support CentOS and Debian. Kolla supports CentOS/RHEL/OracleLinux, Debian and Ubuntu. I was not making builds with RHEL nor OracleLinux but had to make sure that Ubuntu ones work too. There was funny moment when I realised that everyone using Kolla/master was building images with Ocata packages instead of Pike ;D

  • Ubuntu: use Pike repository

But all those patches meant “nothing” without first one. Kolla had information about which packages are available for aarch64/ppc64le/x86-64 architectures but still had no idea that aarch64 or ppc64le exist. Finally the 50th revision of patch got merged so it now knows ;D

  • Support non-x86 architectures (aarch64, ppc64le)

I also learnt a lot about Gerrit and code reviews. OpenStack community members were very helpful with their comments and suggestions. We had hours of talk on #openstack-kolla IRC channel. Thanks goes to Alicja, Duong Ha-Quang, Jeffrey, Kurt, Mauricio, Michał, Qin Wang, Steven, Surya Prakash, Eduardo, Paul, Sajauddin and many others. You people rock!

So is my work on Kolla done now? Some of it is. But we need to test resulting images, make a Docker repository with them, update official documentation with information how to deploy on aarch64 boxes (hope that there will be no changes needed). Also need to make sure that OpenStack Kolla CI gets Debian based gates operational and provide them with 3rdparty AArch64 based CI so new changes could be checked.