From a diary of AArch64 porter — manylinux2014

This post is part 11 of the "From a diary of AArch64 porter" series:

  1. From a diary of AArch64 porter — autoconf
  2. From a diary of AArch64 porter — rpm packaging
  3. From a diary of AArch64 porter — testsuites
  4. From a diary of AArch64 porter — POSIX.1 functionality
  5. From a diary of AArch64 porter — PAGE_SIZE
  6. From a diary of AArch64 porter — vfp precision
  7. From a diary of AArch64 porter — system calls
  8. From a diary of AArch64 porter — parallel builds
  9. From a diary of AArch64 porter — firefighting
  10. From a diary of AArch64 porter — drive-by coding
  11. From a diary of AArch64 porter — manylinux2014
  12. From a diary of AArch64 porter — handling big patches
  13. From a diary of AArch64 porter — Arm CPU features table

Python wheels… Everyone loves them, many people curse when they are not available for their setup. I am in both groups every time I have to do something more complex with Python.

So today I will show how to build Python wheel in quick way.

What is manylinux?

Linux world has a lot of distributions. Even more when you add their releases into the mix. And they ship different versions of Python. At same time we have Pypi which works as repository of ready to use Python packages.

So manylinux idea was created to have minimal requirements for building Python packages. To make sure that you can install it on any distribution.

So far there were several versions built:

name base distribution PEP
manylinux1 CentOS 5 PEP 513
manylinux2010 CentOS 6 PEP 571
manylinux2014 CentOS 7 PEP 599
manylinux_2_24 Debian 9 ‘stretch’ PEP 600

As you see old releases are used to make sure that resulting binaries work on any newer distribution. manylinux2014 added non-x86 architectures (aarch64, ppc64le, s390x).

Each image contains several versions of Python binaries ready to use in /opt/python/ directory.

Manylinux images are distributed as container images on pypa repository on quay.io. Run under Docker, Kubernetes, Podman etc.

Source code is available in ‘manylinux’ repository on GitHub.

Let’s use!

My work requires me to build TensorFlow 1.5.15 version, which depends on NumPy 1.18.* version. None of them are available as wheels for AArch64 architecture.

So let me run container and install NumPy dependencies:

$ docker run -it -u root -v $PWD:/tmp/numpy quay.io/pypa/manylinux2014_aarch64
[root@fa339493a417 /]# cd /tmp/numpy/
[root@fa339493a417 /tmp/numpy/]# yum install -y blas-devel lapack-devel
[root@fa339493a417 /tmp/numpy/]# 

Image has several versions of Python installed and I want to build NumPy 1.18.5 for each of them so my build script is easy:

for py in /opt/python/cp3[6789]*
do
    pyver=`basename $py`
    $py/bin/python -mvenv $pyver
    source $pyver/bin/activate
    pip wheel numpy==1.18.5
    deactivate
done

Result is simple — I got set of wheel files. One per Python version. But it is not the end of work as NumPy libraries depend on blas/lapack we installed into system.

add libraries to the wheel

There is a tool we need to run: “auditwheel”. What it does is inspecting wheel file, all library symbols used, external libraries etc. Then it bundles required libraries into wheel file:

INFO:auditwheel.main_repair:Repairing numpy-1.18.5-cp39-cp39-linux_aarch64.whl
INFO:auditwheel.wheeltools:Previous filename tags: linux_aarch64
INFO:auditwheel.wheeltools:New filename tags: manylinux2014_aarch64
INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp39-cp39-linux_aarch64
INFO:auditwheel.wheeltools:New WHEEL info tags: cp39-cp39-manylinux2014_aarch64
INFO:auditwheel.main_repair:
Fixed-up wheel written to /root/wheelhouse/numpy-1.18.5-cp39-cp39-manylinux2014_aarch64.whl

File size changed from 13 467 772 to 16 806 338 bytes and resulting wheel can be installed on any distribution.

Let summarise

Manylinux is great tool to provide Python packages. It is easy to use on developer’s machine or on CI. And makes life of Python users much easier.

aarch64 linaro python