Mass removal of image tags on Docker hub

At Linaro we moved from packaged OpenStack to virtualenv tarballs. Then we packaged those. But as it took us lot of maintenance time we switched to Docker container images for OpenStack and whatever it needs to run. And then we added CI job to our Jenkins to generate hundreds of images per build. So now we have lot of images with lot of tags…

Finding out which tags are latest is quite easy — you just have to go to Docker hub page of linaro/debian-source-base image and switch to tags view. But how to know which build is complete? We had some builds where all images except one got built and pushed. And the missing one is first in deployment… So whole set was b0rken.

How to remove those tags? One solution is to login to Docker hub website and go image by image and click all those tags to be removed. No one is so insane to suggest it. And we do not have credentials to do that as well.

So let’s handle it as we do that in SDI team: by automation. Docker has some API so it’s hub should have some too, right? Hmm…

I went through some pages, then issues, bug reports, random projects. Saw code in JavaScript, Ruby, Bash but nothing usable in Python. Some of projects assume that no one has more than one hundred of images (no paging in getting list of images) and limits itself to some queries.

Started reading docs and some code. Learnt that GET/POST are not the only methods of doing HTTP. There is also DELETE one which was exactly what I needed. Sorted out authentication, web paths and something started to work.

First version was simple: login and remove tag from image. Then added querying for whole list of images (with proper paging) and looping through the list with removal of requested tags from requested images:

15:53 (s) hrw@gossamer:docker$ ./ haerwu debian-source 5.0.0
haerwu/debian-source-memcached:5.0.0 removed
haerwu/debian-source-glance-api:5.0.0 removed
haerwu/debian-source-nova-api:5.0.0 removed
haerwu/debian-source-rabbitmq:5.0.0 removed
haerwu/debian-source-nova-consoleauth:5.0.0 removed
haerwu/debian-source-nova-placement-api:5.0.0 removed
haerwu/debian-source-glance-registry:5.0.0 removed
haerwu/debian-source-nova-compute:5.0.0 removed
haerwu/debian-source-keystone:5.0.0 removed
haerwu/debian-source-horizon:5.0.0 removed
haerwu/debian-source-neutron-dhcp-agent:5.0.0 removed
haerwu/debian-source-openvswitch-db-server:5.0.0 removed
haerwu/debian-source-neutron-metadata-agent:5.0.0 removed
haerwu/debian-source-heat-api:5.0.0 removed

Final version got MIT license as usual, I created git repo for it and pushed code. Next step? Probably creation of a job on Linaro CI to have a way of removing no longer supported builds. And some more helper scripts.

Is my work on Kolla done?

During last few months I was working on getting Kolla running on AArch64 and POWER architectures. It was a long journey with several bumps but finally ended.

When I started in January I had no idea how much work it will be and how it will go. Just followed my typical “give me something to build and I will build it” style. You can find some background information in previous post about my work on Kolla.

A lot of failures were present at beginning. Or rather: there was a small amount of images which built. So my first merged change was to do something with Kolla output ;D

  • build: sort list of built/failed images before printing

Debian support was marked for removal so I first checked how it looked, then enabled all possible images and migrated from ‘jessie’ to ‘stretch’ release. Reason was simple: ‘jessie’ (even with backports) lacked packages required to build some images.

  • debian: import key for repository
  • debian: install gnupg and dirmngr needed for apt-key
  • debian: enable all images enabled for Ubuntu
  • handle rtslib(-fb) package names and dependencies
  • debian: move to stretch
  • Debian 8 was not released yet

Both YUM and APT package managers got some changes. For first one I took care to make sure that it fails if there were missing packages (which was very often during builds for aarch64/ppc64le). It allowed to catch some typo in ‘ironic-conductor’ image. In other words: I make YUM behave closer to APT (which always complain about missing packages). Then I made change for APT to behave more like YUM by making sure that update of packages lists was done before packages were installed.

  • ironic-conductor: add missing comma for centos/source build
  • make yum fail on missing packages
  • always update APT lists when install packages

Of course many images could not be built at all for aarch64/ppc64le architectures. Mostly due to lack of packages and/or external repositories. For each case I was checking is there some way for fixing it. Sometimes I had to disable image, sometimes update packages to newer version. There were also discussions with maintainers of external repositories on getting their stuff available for non-x86 architectures.

  • kubernetes: disable for architectures other than x86-64
  • gnocchi-base: add some devel packages for non-x86
  • ironic-pxe: handle non-x86 architectures
  • openstack-base: Percona-Server is x86-64 only
  • mariadb: handle lack of external repos on non x86
  • grafana: disable for non-x86
  • helm-repository: update to v2.3.0
  • helm-repository: make it work on non-x86
  • kubetoolbox: mark as x86-64 only
  • magnum-conductor: mark as x86-64 only
  • nova-libvirt: handle ppc64le
  • ceph: take care of ceph-fuse package availability
  • handle mariadb for aarch64/ubuntu/source
  • opendaylight: get it working on CentOS/non-x86
  • kolla-toolbox: use proper mariadb packages on CentOS/non-x86

At some moment I had over ten patches in review and all of them depended on the base one. So with any change I had to refresh whole series and reviewers had to review again… Painful it was. So I decided to split out the most basic stuff to get whole patch set split into separate ones. After “base_arch” variable was merged life became much simpler for reviewers and a bit more complicated for me as from now on each patch was kept in separate git branch.

  • add base_arch variable for future non-x86 work

At Linaro we support CentOS and Debian. Kolla supports CentOS/RHEL/OracleLinux, Debian and Ubuntu. I was not making builds with RHEL nor OracleLinux but had to make sure that Ubuntu ones work too. There was funny moment when I realised that everyone using Kolla/master was building images with Ocata packages instead of Pike ;D

  • Ubuntu: use Pike repository

But all those patches meant “nothing” without first one. Kolla had information about which packages are available for aarch64/ppc64le/x86-64 architectures but still had no idea that aarch64 or ppc64le exist. Finally the 50th revision of patch got merged so it now knows ;D

  • Support non-x86 architectures (aarch64, ppc64le)

I also learnt a lot about Gerrit and code reviews. OpenStack community members were very helpful with their comments and suggestions. We had hours of talk on #openstack-kolla IRC channel. Thanks goes to Alicja, Duong Ha-Quang, Jeffrey, Kurt, Mauricio, MichaƂ, Qin Wang, Steven, Surya Prakash, Eduardo, Paul, Sajauddin and many others. You people rock!

So is my work on Kolla done now? Some of it is. But we need to test resulting images, make a Docker repository with them, update official documentation with information how to deploy on aarch64 boxes (hope that there will be no changes needed). Also need to make sure that OpenStack Kolla CI gets Debian based gates operational and provide them with 3rdparty AArch64 based CI so new changes could be checked.

My work on Kolla

During last month I was working on one of OpenStack projects: Kolla. My job was adding support for non-x86 architectures: aarch64 and ppc64le. Also resurrecting Debian support.

A bit of background

At Linaro we work on getting AArch64 (64-bit ARM, arm64) to be present in many places. We have at least two OpenStack instances running at the moment – on AArch64 hardware only.

First we used Debian/jessie and Openstack ‘liberty’ version. Was working. Not best but we helped many projects by providing virtual machines for porting software.

It was built from packages and later (when ‘mitaka’ was released) we moved to virtualenv per component. Out second “cloud” runs that. With proper Neutron networking, live migration and few other nice things.

But virtualenvs were done as quick solution. We decided to move to Docker containers for next release.

And Kolla was chosen as a tool for it. We do not like to reinvent the wheel again and again…

Non-x86 support in Kolla

The problem was typical: Kolla being x86-64 centric. As most of software nowadays. But thanks to work done by Sajauddin Mohammad I had something to use as a base for adding aarch64 support.

I took his patch, slashed out most of it and concentrated on getting minimal changes needed to get something built on AArch64 . Effect was sent for review and is now at 10th version.

Docker images started to appear. But at beginning I was building Ubuntu ones as Debian support was “basically abandoned, on a way out”. From CentOS guys I got confirmation that official Docker image will be generated (it is done already).

I spent some time on making sure that whole non-x86 support is free from any hardcoding wherever possible. As you can see in my working branch it went quite well. Most of arch related changes are related to “distro does not provide package ZYS for that architecture” or to handling of external repositories.

Debian support

And here we come to Debian support. At Linaro we decided to support two community based distributions: CentOS and Debian. But Debian was on a way out in Kolla…

As this was not related much to non-x86 work I decided to use one of x86-64 machines for that stuff.

First builds were against ‘jessie-backports’ base tag. I had to make a patch to tell APT that if I want backports then I really want them. It was sent for review as rest of patches.

Images were building but not so many as for Ubuntu. So I went through all of them and enabled Debian where it was possible. Resulting patch went for review as usual.

Effect was quite nice (on x86-64):

  • debian-binary: 158
  • debian-source: 201

But ‘jessie’ was missing several packages even with backports enabled. So after discussion with my team I decided to drop it and go for Debian/testing ‘stretch’ one instead. It is already frozen for release so no big changes are allowed. Patch in review of course.

At that moment I abandoned one of previous patches as ‘jessie-backports’ was not something I planned to support.

Turned out that ‘stretch’ images have a bit different set of packages installed than ‘jessie’ had. So ‘gnupg’ and ‘dirmngr’ were missing while we need them for importing GPG keys into APT. Proper patch went to review again.

Did rebuild on x86-64:

  • stretch-binary: 137
  • stretch-source: 195

A bit less than ‘jessie-backports’ had, right? Sure, but it also shows that I have to make a new build to check numbers (laptop already has ~1500 docker images generated by kolla).

Cleaning of old Power patch

Remember the patch which all that started from? I did not forgot it and after building all those images I went back to it.

Some parts are just fugly so I skipped them but others were useful if done properly. That’s how new changes were done and some updates to previous ones.

Then I managed to put remote hands on one of Power machines at Red Hat and started builds:

  • debian-binary: 134
  • debian-source: 184
  • ubuntu-binary: 147
  • ubuntu-source: 190

No CentOS builds as there was no centos/ppc64le image available.


Non-x86 support looks quite nice. There are some images which can not be built as they rely on external repositories so no aarch64 nor ppc64le packages to use.

Debian ‘stretch’ support is not perfect yet but it is something which I plan to maintain so situation will be going to improve. Note that most of my work will go into ‘source’ type of builds as we want to have same images for both Debian and CentOS systems.