I work with OpenDev CI for a while. My first Kolla patches were over three years ago. We (Linaro) added AArch64 nodes few times — some nodes were taken down, some replaced, some added.
Speed or lack of it
Whenever you want to install some Python package using pip it is downloaded
from Pypi (directly or mirror). If there is a binary package then you get it, if
not then “noarch” package is fetched.
In worst case source tarball is downloaded and whole build process starts. You need to have all required compilers installed, development headers for Python and all required libraries and rest of needed tools. And then wait. And wait as some packages require a lot of time.
And then repeat it again and again as you are not allowed to upload packages into Pypi for projects you do not own.
Argh you, protobuf
There was a new release of protobuf package. OpenStack bot picked it up, sent patch for review and it got merged.
And all AArch64 CI jobs failed…
Turned out that protobuf 3.12.0 was released with x86 wheels only. No source tarball. At all.
This turned out to be new maintainer mistake — after 2-3 weeks it was fixed in 3.12.2 release.
Another CI job then
So I started looking at ‘requirements’ project and created a new CI job for it. To check are new package versions are available for AArch64. Took some time and several side updates as well (yak shaving all the way again).
Stuff got merged and works now.
Wheels cache
While working on above CI job I had a discussion with OpenDev infra team how to make it work properly. Turned out that there were old jobs doing exactly what I wanted: building wheels and caching them for next CI tasks.
It took several talks and patches from Ian Wienand, Clark Boylan, Jeremy ‘fungi’ Stanley and others. Several CI jobs got renamed, some were moved from one project to another. Servers got configuration changes etc.
Now we have wheels built for both x86-64 and AArch64 architectures. Covering CentOS 7/8, Debian ‘buster’ and Ubuntu ‘xenial/bionic/focal’ releases. For OpenStack ‘master’ and few stable branches.
Effect
Requirements project has quick ‘check-uc’ job running on AArch64 to make sure that all packages are available for both architectures. All OpenStack projects profit from it.
In Kolla ‘openstack-base’ image went from 23:49 to just 5:21 minutes. Whole Debian/source build is now 57 minutes instead of 2 hours 20 minutes.
Nice result, isn’t it?