Marcin Juszkiewicz

This post is part 11 of the "From the diary of AArch64 porter" series:

1. From the diary of AArch64 porter — autoconf
2. From the diary of AArch64 porter — rpm packaging
3. From the diary of AArch64 porter — testsuites
4. From the diary of AArch64 porter — POSIX.1 functionality
5. From the diary of AArch64 porter — PAGE_SIZE
6. From the diary of AArch64 porter — vfp precision
7. From the diary of AArch64 porter — system calls
8. From the diary of AArch64 porter — parallel builds
9. From the diary of AArch64 porter — firefighting
10. From the diary of AArch64 porter — drive-by coding
11. From the diary of AArch64 porter — manylinux2014
12. From the diary of AArch64 porter — handling big patches
13. From the diary of AArch64 porter — Arm CPU features table

Python wheels… Everyone loves them, many people curse when they are not available for their setup. I am in both groups every time I have to do something more complex with Python.

So today I will show how to build Python wheel in quick way.

What is manylinux?

Linux world has a lot of distributions. Even more when you add their releases into the mix. And they ship different versions of Python. At same time we have Pypi which works as repository of ready to use Python packages.

So manylinux idea was created to have minimal requirements for building Python packages. To make sure that you can install it on any distribution.

So far there were several versions built:

As you see old releases are used to make sure that resulting binaries work on any newer distribution. manylinux2014 added non-x86 architectures (aarch64, ppc64le, s390x).

Each image contains several versions of Python binaries ready to use in /opt/python/ directory.

Manylinux images are distributed as container images on pypa repository on quay.io. Run under Docker, Kubernetes, Podman etc.

Source code is available in ‘manylinux’ repository on GitHub.

Let’s use!

My work requires me to build TensorFlow 1.5.15 version, which depends on NumPy 1.18.* version. None of them are available as wheels for AArch64 architecture.

So let me run container and install NumPy dependencies:

$ docker run -it -u root -v $PWD:/tmp/numpy quay.io/pypa/manylinux2014_aarch64
[root@fa339493a417 /]# cd /tmp/numpy/
[root@fa339493a417 /tmp/numpy/]# yum install -y blas-devel lapack-devel
[root@fa339493a417 /tmp/numpy/]# 

Image has several versions of Python installed and I want to build NumPy 1.18.5 for each of them so my build script is easy:

for py in /opt/python/cp3[6789]*
do
    pyver=`basename $py`
    $py/bin/python -mvenv $pyver
    source $pyver/bin/activate
    pip wheel numpy==1.18.5
    deactivate
done

Result is simple — I got set of wheel files. One per Python version. But it is not the end of work as NumPy libraries depend on blas/lapack we installed into system.

add libraries to the wheel

There is a tool we need to run: “auditwheel”. What it does is inspecting wheel file, all library symbols used, external libraries etc. Then it bundles required libraries into wheel file:

INFO:auditwheel.main_repair:Repairing numpy-1.18.5-cp39-cp39-linux_aarch64.whl
INFO:auditwheel.wheeltools:Previous filename tags: linux_aarch64
INFO:auditwheel.wheeltools:New filename tags: manylinux2014_aarch64
INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp39-cp39-linux_aarch64
INFO:auditwheel.wheeltools:New WHEEL info tags: cp39-cp39-manylinux2014_aarch64
INFO:auditwheel.main_repair:
Fixed-up wheel written to /root/wheelhouse/numpy-1.18.5-cp39-cp39-manylinux2014_aarch64.whl

File size changed from 13 467 772 to 16 806 338 bytes and resulting wheel can be installed on any distribution.

Let summarise

Manylinux is great tool to provide Python packages. It is easy to use on developer’s machine or on CI. And makes life of Python users much easier.