This post is part 11 of the "From a diary of AArch64 porter" series:
- From a diary of AArch64 porter — autoconf
- From a diary of AArch64 porter — rpm packaging
- From a diary of AArch64 porter — testsuites
- From a diary of AArch64 porter — POSIX.1 functionality
- From a diary of AArch64 porter — PAGE_SIZE
- From a diary of AArch64 porter — vfp precision
- From a diary of AArch64 porter — system calls
- From a diary of AArch64 porter — parallel builds
- From a diary of AArch64 porter — firefighting
- From a diary of AArch64 porter — drive-by coding
- From a diary of AArch64 porter — manylinux2014
- From a diary of AArch64 porter — handling big patches
- From a diary of AArch64 porter — Arm CPU features table
Python wheels… Everyone loves them, many people curse when they are not available for their setup. I am in both groups every time I have to do something more complex with Python.
So today I will show how to build Python wheel in quick way.
What is manylinux?
Linux world has a lot of distributions. Even more when you add their releases into the mix. And they ship different versions of Python. At same time we have Pypi which works as repository of ready to use Python packages.
So manylinux idea was created to have minimal requirements for building Python packages. To make sure that you can install it on any distribution.
So far there were several versions built:
|manylinux1||CentOS 5||PEP 513|
|manylinux2010||CentOS 6||PEP 571|
|manylinux2014||CentOS 7||PEP 599|
|manylinux_2_24||Debian 9 ‘stretch’||PEP 600|
As you see old releases are used to make sure that resulting binaries work on any newer distribution. manylinux2014 added non-x86 architectures (aarch64, ppc64le, s390x).
Each image contains several versions of Python binaries ready to use in /opt/python/ directory.
Manylinux images are distributed as container images on pypa repository on quay.io. Run under Docker, Kubernetes, Podman etc.
Source code is available in ‘manylinux’ repository on GitHub.
My work requires me to build TensorFlow 1.5.15 version, which depends on NumPy 1.18.* version. None of them are available as wheels for AArch64 architecture.
So let me run container and install NumPy dependencies:
$ docker run -it -u root -v $PWD:/tmp/numpy quay.io/pypa/manylinux2014_aarch64 [root@fa339493a417 /]# cd /tmp/numpy/ [root@fa339493a417 /tmp/numpy/]# yum install -y blas-devel lapack-devel [root@fa339493a417 /tmp/numpy/]#
Image has several versions of Python installed and I want to build NumPy 1.18.5 for each of them so my build script is easy:
for py in /opt/python/cp3* do pyver=`basename $py` $py/bin/python -mvenv $pyver source $pyver/bin/activate pip wheel numpy==1.18.5 deactivate done
Result is simple — I got set of wheel files. One per Python version. But it is not the end of work as NumPy libraries depend on blas/lapack we installed into system.
add libraries to the wheel
There is a tool we need to run: “auditwheel”. What it does is inspecting wheel file, all library symbols used, external libraries etc. Then it bundles required libraries into wheel file:
INFO:auditwheel.main_repair:Repairing numpy-1.18.5-cp39-cp39-linux_aarch64.whl INFO:auditwheel.wheeltools:Previous filename tags: linux_aarch64 INFO:auditwheel.wheeltools:New filename tags: manylinux2014_aarch64 INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp39-cp39-linux_aarch64 INFO:auditwheel.wheeltools:New WHEEL info tags: cp39-cp39-manylinux2014_aarch64 INFO:auditwheel.main_repair: Fixed-up wheel written to /root/wheelhouse/numpy-1.18.5-cp39-cp39-manylinux2014_aarch64.whl
File size changed from 13 467 772 to 16 806 338 bytes and resulting wheel can be installed on any distribution.
Manylinux is great tool to provide Python packages. It is easy to use on developer’s machine or on CI. And makes life of Python users much easier.