1. Installed top5000 Python packages on AArch64

    It was yet another boring week. I got some thanks for work we did to get Tensorflow running on AArch64 and was asked is there any other Python project which could use our help.

    I had some projects already on my list of things to check. And then found “Top PyPI Packages” website…

    Let install 5000 Python packages

    So I got an idea — let me grab list of top5000 PyPI packages and check how many of them have issues on AArch64.

    The plan was simple. Loop over list and do 3 tasks:

    • create virtualenv
    • install package
    • destroy virtualenv

    This way each package had the same environment and I did not had to worry about version conflicts.

    Virtualenv preparation

    To not repeat creation of virtualenv five thousand times I did it once:

    /opt/python/cp39-cp39/bin/python3 -mvenv venvs/test
    . venvs/test/bin/activate
    pip install -U pip wheel setuptools
    deactivate
    cp -a venvs/test venvs/clean
    

    This way I had newest versions of “pip”, “wheel” and “setuptools” as part of virtualenv. For a bit of speed “venvs” directory was mounted as “tmpfs”.

    Everything happened inside of “manylinux2014_aarch64” container (so CentOS 7 as OS) with Python 3.9 as interpreter.

    Phase 1

    First phase was simple installation of package wheel as it was available on PyPI:

    pip --only-binary :all: --no-compile ${name_of_package}
    

    So if package provided only source tarball/zip then it was marked as failed.

    There were 1569 packages which failed to pass this phase. Common issues (other than missing some development headers):

    • INFO: pip is looking at multiple versions of PACKAGE_NAME to determine which version is compatible with other requirements. This could take a while.
    • INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
    • INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
    • ERROR: Cannot install OTHER_PACKAGE_NAME because these package versions have conflicting dependencies.
    • ERROR: Could not find a version that satisfies the requirement OTHER_PACKAGE_NAME (from versions: none)
    • ERROR: Could not find a version that satisfies the requirement OTHER_PACKAGE_NAME==N.V.R (from ANOTHER_PACKAGE_NAME) (from versions: x.y, x.y.z, x.z.z)
    • ERROR: No matching distribution found for OTHER_PACKAGE_NAME

    Note that failure at this phase is allowed as I just want ready to use wheel files.

    Whole process took about 20 hours on Honeycomb.

    Phase 2

    The main difference was getting rid of “—only-binary” and “—no-compile” options from pip install calls.

    Still no additional development packages installed. Cache from phase 1 in use to not re-download/re-build existing wheel files.

    The main issue is how single threaded pip install is. Nevermind that Honeycomb has 16 cpu cores — only one is used (and this is Cortex-A72 so nothing fancy). This makes building times higher than they suppose to be:

    Building wheels for collected packages: pandas, typing
      Building wheel for pandas (setup.py): started
      Building wheel for pandas (setup.py): still running...
      Building wheel for pandas (setup.py): still running...
      Building wheel for pandas (setup.py): still running...
      Building wheel for pandas (setup.py): still running...
      Building wheel for pandas (setup.py): still running...
      Building wheel for pandas (setup.py): still running...
      Building wheel for pandas (setup.py): still running...
      Building wheel for pandas (setup.py): still running...
      Building wheel for pandas (setup.py): still running...
      Building wheel for pandas (setup.py): still running...
      Building wheel for pandas (setup.py): still running...
    

    There were 313 packages which failed to pass this phase. Issues were similar to those in phase 1 with one exception (as building packages was allowed):

    • ERROR: Could not build wheels for OTHER_PACKAGE_NAME, which is required to install pyproject.toml-based projects

    This phase took about 13 hours on Honeycomb.

    Phase 3

    About 6% packages left. Now it is time to install some development headers:

    • blas-devel
    • bzip2-devel
    • cairo-devel
    • cyrus-sasl-devel
    • gmp-devel
    • gobject-introspection-devel
    • graphviz-devel
    • gtk3-devel
    • httpd-devel
    • krb5-devel
    • lapack-devel
    • libcap-devel
    • libcurl-devel
    • libicu-devel
    • libjpeg-devel
    • libmemcached-devel
    • mariadb-devel
    • ncurses-devel
    • openldap-devel
    • openssl-devel
    • poppler-cpp-devel
    • postgresql-devel
    • protobuf-compiler
    • unixODBC-devel
    • xmlsec1-devel

    I created this list by checking how packages failed to build. It should be longer but CentOS 7 (base of “manylinux2014” container image) does not provide everything needed (for example up-to-date Rust compiler or LLVM).

    Before starting phase 3 run I removed all entries related to “pyobjc” as they are MacOS related so there is no need to waste time again.

    After 3.5 hours I had another 54 packages built.

    Phase 4

    Some packages are not present in CentOS 7 but are present in EPEL repository. So after enabling EPEL (yum install -y epel-release) I installed another set of development packages:

    • augeas-devel
    • boost-devel
    • cargo
    • gdal-devel
    • leptonica-devel
    • leveldb-devel
    • suitesparse-devel
    • portaudio-devel
    • proj
    • protobuf-devel
    • rust
    • zbar-devel

    Some of those packages should be installed in previous step. I did not caught them because build processes failed earlier.

    Before starting round I went through logs and removed everything:

    • failed with “No matching distribution for PACKAGE_NAME
    • failed with “use_2to3 is invalid” (aka “I need old setuptools”)
    • requiring Bazel
    • requiring tensorflow

    At the end I had about one hundred of failed to build packages. For different reasons:

    • missing build dependencies
    • expecting newer libraries than “manylinux2014” (CentOS 7) has
    • not listing all dependencies (everyone has “numpy” installed, right?)
    • being Python 2.7 only
    • using removed modules or classes
    • breaking install to say “this module is deprecated, use OTHER_NAME”
    • not supporting AArch64 architecture

    Summary

    One hundred of top five thousand packages equals two percent of failures. There were 13 failures in top 1000, another 14 in second thousand.

    Is 2% acceptable amount? I think that it is. Some improvements can still be made but nothing requiring shown. OK, would be nice to get Tensorflow for AArch64 released by upstream under same name (instead of “tensorflow_aarch64” builds done by team at Linaro).

    How to run it?

    After my tweet I had several comments and people wanted to run this test on other architectures, operating systems or devices. So I wrote simple script:

    #!/bin/bash
    
    echo "cleanup after previous runs"
    rm -rf venvs/* logs/*
    
    echo "Prepare clean virtualenv"
    python3 -mvenv venvs/test
    . venvs/test/bin/activate
    pip install -U pip wheel setuptools
    deactivate
    cp -a venvs/test venvs/clean
    
    echo "fetch and prepare top5000 list"
    rm top-pypi-packages-30-days.*
    wget https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.json
    grep project top-pypi-packages-30-days.json \
        |sed -e 's/"project": "\(.*\)"/\1/g' > top-pypi-packages-30-days.text
    
    echo "go through packages"
    mkdir -p logs
    for package in `cat top-pypi-packages-30-days.text`; do
        echo "processing ${package}"
        rm -rf venvs/test
        cp -a venvs/clean venvs/test
        source venvs/test/bin/activate
        pip install --no-input \
            -U --upgrade-strategy=only-if-needed \
            $package | tee logs/${package}.log
        deactivate
        echo "-----------------------------------------------------------------"
    done
    

    It should work on any operating system capable of running Python. All build dependencies need to be installed first. I suggest mounting “tmpfs” over “venvs/” directory as there will be lot of temporary i/o going on there.

    Once it finish just run grep to check how many packages were installed with success:

    grep "^Successfully installed" logs/*|wc -l
    

    Please share your results. Contact page lists several ways to catch me.

    Written by Marcin Juszkiewicz on
  2. Monitoring local network devices

    Due to current events it was hard to concentrate on work lately. So I decided to learn something new but still work related.

    In OpenStack Kolla project we provide images with Collectd, Grafana, Influxdb, Prometheus, Telegraf to give people options for monitoring their deployments. I never used any of them. Until now.

    Local setup

    At home I have some devices I could monitor for random data:

    • TrueNAS based NAS
    • OpenWRT based router
    • OpenWRT based access point
    • HoneyComb arm devbox
    • other Linux based computers
    • Home Assistant

    All machines can ping each other so sending data is not a problem.

    Software

    For software stack I decided to go for PIG — Prometheus, Influxdb, Grafana.

    I used instruction from blog posts written by Chris Smart:

    Read both — they have more details than I used to mention. Also more graphics.

    OpenWRT

    Turned out that OpenWRT devices can either use “collectd” or Prometheus node exporter to gather metrics. I had first one installed already (to have some graphs in webUI). If you do not then all you need is this set of commands:

    opkg update
    opkg install luci-app-statistics collectd collectd-mod-cpu \
                 collectd-mod-interface collectd-mod-iwinfo \
                 collectd-mod-load collectd-mod-memory \
                 collectd-mod-network collectd-mod-uptime
    
    /etc/init.d/luci_statistics enable
    /etc/init.d/collectd enable
    

    TrueNAS

    TrueNAS already has reporting in webUI. Can also send data to remote Graphite server. So again I had everything in place for gathering metrics.

    Next step was sorting out data collector and visualisation. There is community provided “Grafana & Influxdb” plugin. Installed it, gave jail a name, set to request own IP and got “grafana.lan” running in a minute.

    Configuration

    At this moment everything is installed for both gathering and visualisation of data. Time for some configuration.

    Logged into jail (“sudo iocage console grafana”) and fun started.

    Influxdb

    First Influxdb needed to be configured to accept both “collectd” data from OpenWRT nodes and “graphite” data from TrueNAS. Simple edit of “/usr/local/etc/influxd.conf” file to have this:

    [[collectd]]
      enabled = true
      bind-address = ":25826"
      database = "collectd"
      # retention-policy = ""
      #
      # The collectd service supports either scanning a directory for multiple types
      # db files, or specifying a single db file.
      typesdb = "/usr/local/share/collectd"
      #
      security-level = "none"
      # auth-file = "/etc/collectd/auth_file"
    

    and that:

    [[graphite]]
      # Determines whether the graphite endpoint is enabled.
      enabled = true
      database = "graphite"
      # retention-policy = ""
      bind-address = ":2003"
      protocol = "tcp"
      consistency-level = "one"
    

    Save, restart Influxdb (by “service influxd restart”). Next step was creation of databases (add username and password if needed):

    # influx
    > create database collectd
    > create database graphite
    > exit
    

    And data should be gathered from both OpenWRT (collectd) and TrueNAS (graphite) nodes.

    OpenWRT

    Here are two ways to do configure OpenWRT based system:

    First one is visiting webUI -> Statistics -> Setup -> Output plugins, then enabling “network” plugin and giving IP of InfluxDB server.

    OpenWRT WebUI -> Statistics -> Setup -> Output plugins
    OpenWRT WebUI -> Statistics -> Setup -> Output plugins
    OpenWRT WebUI Network output plugin configuration
    OpenWRT WebUI Network output plugin configuration

    Second one is simple edit of “/etc/collectd.conf” file:

    LoadPlugin network
    <Plugin network>
            Server "192.168.202.183" "25826"
    </Plugin>
    

    No idea why it wants IP address instead of FQDN.

    TrueNAS

    TrueNAS is simple — visit webUI -> System -> Reporting and give address (IP or FQDN) of remote graphite server.

    TrueNAS System Reporting configuration
    TrueNAS System Reporting configuration

    I enabled both “Report CPU usage in percent” and “Graphite separate instances” checkboxes because of Grafana dashboard I use.

    Grafana

    Logged into http://grafana.lan:3000, setup admin account and then setup data sources. Both will be “InfluxDB” type — one for ‘collectd’ database keeping metrics from OpenWRT nodes, second for ‘graphite’ one with data from TrueNAS.

    Use “http://localhost:8086” as URL, provide database names and name each data source in a way telling you which one is which.

    Visualisation

    Ok, software installed on all nodes, configuration files edited. Metrics flow to InfluxDB (you can check them using “show series” in Influx shell tool). Time to setup some dashboards in Grafana.

    This is moment where I went for ready ones:

    OpenWRT dashboard

    Hard to tell will I keep it as it provides lot of data on same graphs:

    Overloaded OpenWRT dashboard
    Overloaded OpenWRT dashboard

    I plan to take a look at it and keep what I really need, use some aliases to see WiFi network names instead of interface names, etc.

    TrueNAS dashboard

    Here some editing was needed. Dashboard itself needed timezone change, 24 hours clock, adding ‘da0’ hard disk to graphs and setting ZFS pool names.

    The other edit was in InfluxDB configuration (in “graphite” section) to get some metrics renamed:

      templates = [
        "*.app env.service.resource.measurement",
         "servers.* .host.resource.measurement.field*",
      ]
    

    Effect was nice:

    TrueNAS dashboard
    TrueNAS dashboard

    This one looks like good base. Also suggests me to visit my server wardrobe and check fans in NAS box.

    Future

    I need to alter both dashboards to make them show some really usable data. Then add some alerts. And more nodes — there is still no Prometheus in use here gathering metrics from my generic Linux machines. Nor my server.

    Written by Marcin Juszkiewicz on
  3. 2021 timeline

    2021 ended so let me try to mark some interesting moments.

    January

    • Started looking for a new flat. This time to buy not to rent.

    • As kind of experiment I started working on I/O plate for APM Mustang as it never had one. After several hours and versions finally I got something usable. Not perfect but good enough.

    February

    • Online FOSDEM happened. It was the best online conference so far.

    • Friend found me a nice flat. Went, saw, decided to buy.

    • Jon Nettleton from SolidRun asked do I have any interest in a HoneyComb. We discussed and few days later I got HoneyComb at home. Bought things like case and ram for it, used spare nvme and got it working. I use it as build machine for countless projects.

    • Replaced old Netgear router with EspressoBin. I got that SBC from a friend, printed 3d case and sorted out fresh firmware. Now it boots straight from SPI flash and reads operating system from microsd card.

    • After several discussions with developers I wrote how AArch64 perception change when all you used are SBC (compared to being used to AArch64 servers).

    March

    • Another ‘let me explain’ post. This time on U-Boot and generic distro boot as it turned out to be hard to understand/decode for some people.

    • I got an offer of Ampere eMag system. But due to my flat move and shipping/customs issues nothing came from it.

    April

    • Bought flat. Agreed with previous owners on moving date. They started painting walls.

    • Five years at Linaro passed. Still waiting for that glass award.

    • I wrote one tweet and it was enough to get extra AArch64 nodes for Opendev CI.

    • I got “Mars 2020 Helicopter Contributor” badge on GitHub. As one of ~12000 developers.

    • First dose of COVID-19 vaccine.

    May

    • Moved to my own flat. With help from my friends and moving company. Still have a few unpacked boxes.

    • As my daughter had 3 free days in a middle of a week we went for nice trip through museums.

    • Spent some time on designing new structure for home network. EspressoBin got replaced with very small x86-64 system (4 “igb” network interfaces). Finally reached 1Gbps speed on home link.

    June

    • Organizing my new home. Looking for good office furniture ended with buying VS 901 desk and organizer.

    • Second dose of COVID-19 vaccine.

    July

    • Got two Sharp Zaurus palmtops: SL-5500 (collie) and SL-5600 (poodle) from a friend. First model was a device which brought me to working full time on FOSS.

    • My 13 years old daughter asked me what “that huge phone with printer” she saw in anime was. Took me a moment or two to find out that it was fax machine…

    • Crashed my car. No one got hurt. Insurance paid what they had to and I later sold what was left of the car.

    August

    • Vacations! Rented Opel Astra combi and we went for a tour. Met some friends, visited new places (and some old ones). Good time.

    • Friends managed to organize demoscene party in COVID-19 times. So I went to Katowice, Poland and spent great time at Xenium 2021.

    September

    October

    • Left some projects maintained by Arm employees. Open source with closed development process is not something I am fan of. Especially when there is no serious review done on code contributions.

    • I got information that moderators on “Arm Software Developers” Discord are not fans of my comments there. Went through history, dropped most of my posts and left server. Was asked to go back some time later. No longer taking part of conversations there.

    November

    • I was positive! Too bad that it was result of COVID-19 test. 2 weeks of isolation as a result. Boring time with flu-like symptoms. Friends handled my shopping needs.

    • Migrated from Pocketbook Touch HD to Onyx Boox Poke 3. Just to migrate to Onyx Boox Nova 2 three weeks later. Reading books on 7.8” screen is the other experience.

    December

    • Polish demoscene became official Polish cultural heritage.

    • Wrote post about 25 years of using the Internet.

    • Started playing with configuration of my router to get native IPv6/IPv4 setup working. Took far too many attempts and curses but got it done (in 2022).

    • Third dose of COVID-19 vaccine (this time Moderna instead of Pfizer).

    Written by Marcin Juszkiewicz on
  4. My E-book readers

    During last days I had discussions about devices to read electronic books. You know: e-book readers like Amazon Kindle, Pocketbook. I am unable to count how many of them I bought and which models. But I know which I used.

    Very old times

    Long, long time ago I had Palm M105 and then Sony Clie SJ30 palmtops. Used both to read some electronic books. But small screen made it not comfortable.

    First try

    In 2011 I wanted to check how it feels to read e-books on proper e-ink device. Borrowed Amazon Kindle Keyboard from one of my friends and it was good.

    Month later I was in USA on Linaro Developer Summit and bought myself Kindle Keyboard. And got Kindle ‘no touch’ one too. First one went on shelf quickly as I used smaller one more often. Still have it — in storage now as battery finally gave up.

    Amount of books I bought and read in electronic form quickly passed amount of paper ones.

    Upgrading Kindle

    Amazon released e-ink device with touchscreen: Kindle Touch. So I bought it on next US visit (another Linaro conference). This time I also got few devices for my friends (no customs == profit) and sold my ‘no touch’ one.

    Then Kindle Paperwhite on next visit. Screen backlight was huge step. Reading books in buses, planes, trains became comfortable.

    I had one or two newer Kindle Paperwhite devices but for short time.

    E-Book subscription

    In 2015 I gave Kindle Paperwhite to my mother with some books on it. She enjoyed using device but there was a problem with selection of content…

    At around same time Legimi started their e-book subscription service. So I bought Inkbook Obsidian from them and gave my mother as Xmas gift. After some training she became a fan of both service and e-books.

    Goodbye Kindle, welcome Pocketbook

    At some moment Legimi started tests of ‘one account, two readers’ offer. I decided that it is good time to change device. Sold my Kindle Paperwhite and bought Pocketbook Touch Lux 3 instead.

    New device allowed me to use subscription on two e-book readers == less money spent on buying books.

    Few months later I had to buy another device as my daughter took Pocketbook from me and started using it. So I bought Pocketbook Touch HD for myself. And another Legimi subscription ;D

    This was also first e-book reader where I started experimenting with software. Coolreader is nice alternative to original reading application. My favourite option was “ignore publisher formatting” so each e-book looked the same. Of course if it was properly done which was not granted.

    Let go Android with Onyx

    Time passed, I cancelled my Legimi subscription in meantime. And PB Touch Lux 3 one day decided to not refresh screen. Like at all. Dead.

    So Mira got Pocketbook Touch HD and I started looking for something new for myself.

    Asked friends, did some research, watched countless review videos. And decided on Onyx Boox Poke 3. Still 6” 300dpi screen but with all fancy backlight things and Android 10.

    Did not even got used to it and replaced it with Onyx Boox Nova 2 from one of friends. 7.8” screen made a difference especially with pre-formatted PDF files.

    FBreader works great on it and allows to use OPDS catalogs directly from device. And amount of configuration options beats everything I used before.

    I miss one thing (compared to Kindle or Pocketbook) — there is no way to send e-books via e-mail straight to the device.

    Summary

    During those ten years of using e-book readers I have read countless amount of books. On trip to San Diego I have read above thousand pages on plane (one s-f book series was hard to put away).

    Bought hundreds of books in electronic form. And just a few paper ones just because they lacked any other form. I also feel unable to concentrate while reading paper ones — have to hold them in a way to not close etc…

    Written by Marcin Juszkiewicz on
  5. 25 years of using the Internet

    Twenty five years ago, in 1996 year, I failed exam for Computer Science studies. And went for Automation and Robotics instead. Next five years were funny, tough, hard and interesting. All at once as it is with studies.

    And I got access to the Internet there.

    Beginning

    One day during walking through university buildings I went to not visited yet corridor. There was a group of people sitting in front of some weird terminals. Turned out that those Hewlett Packard 2623A machines offer access to the Internet. All I had to do was to knock the system admins’ door, show some id and pay 10 PLN per month.

    Some time later I got access to terminal. Landed in SunOS without basically any UNIX experience (I was AmigaOS user at that time). Other users gave me some hints:

    • use “screen” as soon as you login
    • PINE is for email
    • Lynx is for web
    • Pico is your editor of choice
    • IRC is for chatting and you want Venom or Lice for it (popular ircii scripts)
    • use exit once you finish

    And I started spending time there. First weeks were tough — getting used to text interface and remembering useful commands and hints. One of most important was where to store extra files as account was 1.5 megabyte in size (with warning after crossing 1MB). Ah those /var/tmp/ or /var/news/ etc. subdirectories with everyone-can-write access :D The other was how to transfer them to floppy disks.

    Terminal knowledge

    None of terminals had a battery to keep configuration data. So often they were setup as 2400bps ones (if you powered it on and kept trashing Return key then SunOS finally appeared at such speed). One of things to learn was how to reconfigure it to 9600bps which meant quite “comfortable” work.

    Forget about using tools like Midnight Commander — it refreshed screen too often so most of time you saw only redrawing of characters.

    Also none of keys outside of alphanumeric part were usable.

    If you want more info about terminal type we used then visit HP Computer Museum page about HP262x terminal family page.

    Internet in dormitory

    When I was starting fourth year university connected one dormitory to the Internet. Surprise, surprise I lived there (it wasn’t default one for my faculty).

    During vacations I earned some good money and bought PCMCIA Ethernet card for my Amiga 1200. Oh, what change it was! No more queues to terminals. All those graphical tools and graphical web browser! And no more wondering where to go to grab files from my shell account.

    All that on 14” VGA mono monitor in 720x480 resolution.

    Commodore 128dcr can be online too

    There were two of us in the room and only one computer. So still the problem of access existed.

    One day I came out from visiting home and found out that my room mate bought Commodore 128dcr with 80 column monitor. We wired some serial cable and connected it to my Amiga. After fetching and sending some software over the wire we had 9600bps connection and 80x25 text terminal ready for use.

    Oh, those moments on IRC when we said that we use 8-bit Commodore ;D Too bad that we lack any photo from that time.

    Personal website

    I had some kind of personal website since basically forever. First version was my Lynx bookmarks with a bunch of extra text. This was quite popular type then.

    After studies my website moved between servers. Landed on some free hosting, then my friend’s server. Later on some paid space where I also had email in own domain and finally moved to self hosting in the cloud.

    The oldest copy I found in the Web Archive is from July 2003. It was on different domain that current page uses. And you can still use it to get here (there are five domains pointing here).

    I used several tools to maintain my page: php wiki, own code and over 16.5 years ago decided to give Wordpress a try. After over decade of using it moved to Pelican and it will stay that way for now.

    Future is in mobile

    I remember presentation of Nokia 9210 Communicator at some event at university. Device in your pocket with direct access to the Internet. It looked like future.

    Time passed, I met people with PalmOS devices. They used Irda for data transfer with their cellphones. Then Bluetooth with newer models. Integrated GSM modems in next ones…

    Cellphones got first versions of web browsers, mail apps… And all that moved faster and faster. We also started calling them smart phones.

    Nowadays I have multicore device in my pocket. With screen resolution higher than many people’s monitors, more memory than computers some of my friends use at work. And close to 4 terabytes of data allowance on my prepaid sim card. Something not imaginable for 20 years younger me.

    Written by Marcin Juszkiewicz on
  6. TensorFlow build times

    The big part of my work during last months was related to TensorFlow. We build it on Linaro CI. Resulting wheels are available on Linaro snapshots server.

    Versions

    When you dig there you can find those versions:

    • 1.5.x
    • 2.4.x
    • 2.5.x
    • 2.6.x
    • 2.7.x (git HEAD)

    How to install?

    TensorFlow requires several other Python packages and some of them are not distributed as AArch64 binary wheels. For this we have Python cache repository at Linaro snapshots.

    So how to install TensorFlow (2.6.0 or 2.7.0 version):

    $ export PIP_EXTRA_INDEX_URL=https://snapshots.linaro.org/ldcg/python-cache/
    $ pip install tensorflow-aarch64
    

    And it will be done. Package is renamed from “tensorflow-cpu” as there is a plan of uploading them to Pypi. For older versions please check our snapshots server.

    How do we build?

    Jenkins runs shell script which then starts container, installs Ansible and then the rest of build is done using it. Playbooks, roles and shell scripts are stored in Jenkins jobs repository.

    The process is quite simple — we choose which versions to build (from 1.5, 2.4, 2.5, 2.6, git HEAD selection) and then Ansible loops over them. All dependencies’ versions are stored in variables file.

    Whole work is done inside of “manylinux2014” container to get a way of building for wide selection of Python releases. Build covers versions from 3.6 to 3.9 one (we plan to enable 3.10 when possible) in one run.

    Build times

    To compare speed of several systems I have available I ran a build on each and compared to times on Linaro CI machine.

    Some details:

    Machine name processor cores threads memory note
    Oracle cloud A1 Altra 16 16 96 VM.Standard.A1.Flex
    Linaro CI ThunderX2 2x28 56 240 SMT disabled
    SolidRun HoneyComb LX2160 16 16 32
    my work laptop i7-8665U 4 8 32 SMT enabled
    my desktop Ryzen 5 3600 6 12 32 SMT enabled

    Build time includes fetching files from network. Python packages comes from either Pypi or Linaro Python cache repository.

    Procedure

    Install Docker, pull manylinux2014 image, fetch script from Linaro CI job git repository, run it.

    $ docker pull quay.io/pypa/manylinux2014_aarch64
    $ wget https://git.linaro.org/ci/job/configs.git/plain/ldcg-python-manylinux-tensorflow/build.sh
    $ mkdir BUILD
    $ cd BUILD
    $ export WORKSPACE=
    $ time build26=true bash ../build.sh
    

    BUILD directory created as job starts with cleaning workspace (stored in WORKSPACE variable).

    Results

    Machine name build time
    Oracle cloud A1 3:15:53.250
    Linaro CI 1:56:15.994
    SolidRun HoneyComb 7:29:01.922
    my work laptop 10:01:19.044
    my desktop 3:30:34.821
    Build time (in hours)
    Build time (in hours)

    Comments

    HoneyComb is a good system for development work. As long as there are other systems which will do hard work on building things.

    ThunderX2 is a workhorse. Give it something to do and it delivers. It was also very expensive (Avantek started with 13k USD for workstation).

    Oracle cloud instance used Ampere Altra cpu. Would not be surprised if it beats ThunderX2 system when more cores are used (16 cores was limit of free tier).

    I used my work laptop because it was available. Did not expected much from it. But in past benchmarks it was close to my previous desktop system.

    And with my desktop… It was quite cheap solution 2 years ago. Looks like price/performance is something where x86-64 is a king.

    Written by Marcin Juszkiewicz on
  7. Python package for system calls information

    Python is my favourite language. System calls table is my favourite side project. So why not merge them?

    That’s how “system-calls” Python package was born.

    Why?

    Mostly because I could. And thought that maybe someone else will find it useful. Also found out that it allows me to learn how to build own Python package, how Pypi uploads work etc. stuff. And I can remind myself how packaging works as having it in Debian and Fedora would be useful experience.

    What it can do?

    Module allows to check information about Linux system calls in several ways:

    • which number system call has on architecture XYZ
    • is system call supported on architecture XYZ

    Code for it is quite simple:

    import system_calls
    
    syscalls = system_calls.syscalls()
    
    for test_call in ['openat', 'osf_uadmin', 'nosuchcall']:
        try:
            print(f"System call '{test_call}' has number: {syscalls[test_call]}")
        except system_calls.NoSuchSystemCall:
            print(f"No such system call '{test_call}' on any architecture")
        except system_calls.NotSupportedSystemCall:
            print(f"System call '{test_call}' is not supported on this "
                  "architecture")
    

    Above example shows usage for host architecture. But treating “system_calls.syscalls” as dictionary does not allow to ask for data of different architecture. For such use there is “get()” method:

    import system_calls
    
    syscalls = system_calls.syscalls()
    
    for test_call in ['openat', 'osf_uadmin', 'nosuchcall']:
        try:
            print(f"System call '{test_call}' on arm64 has number: "
                  f"{syscalls.get(test_call, 'arm64')}")
        except system_calls.NoSuchSystemCall:
            print(f"No such system call '{test_call}' on any architecture")
        except system_calls.NotSupportedSystemCall:
            print(f"System call '{test_call}' is not supported on this "
                  "architecture")
    
    

    You can also use “get()” method for host architecture as second argument is optional.

    You can find more examples in project’s repository.

    Extras

    There is one bonus script in this project: “syscall”. Can be used to check for information about system calls for host or any supported architecture:

    $ syscall open
    On x86_64 system call number 2 is open()
    $ syscall open arm64
    On arm64 system call open() is not supported.
    $ syscall 32
    On x86_64 system call number 32 is dup()
    $ syscall 32 arm64
    On arm64 system call number 32 is flock()
    $ syscall openat mipso32
    On mipso32 system call number 4288 is openat()
    $ syscall open noa
    Architecture noa is not supported.
    $ syscall nos
    There is no such system call as nos().
    $
    

    Mistakes happen

    When I wrote first version I did not thought about uploading it to Pypi and sharing with other users. And when I started packaging it for Fedora I realized that I made some serious mistake.

    Package is called “system-calls” but code was stored in “syscalls” directory. Fixed that in 5.15.2 version so it is now stored in “system_calls” one.

    5.15.3 has all tests and examples fixed to use proper names.

    If you find some other issue then please report it in project’s repository.

    Packaging

    Fedora 35 and above have “python3-system-calls” package available in distribution repository.

    Debian users can use my private APT repository. Not decided yet do I want to became Debian Maintainer/Developer.

    Written by Marcin Juszkiewicz on
  8. Programming languages

    Today is Programmer’s Day. 13 years ago I wrote a post “Am I programmer” where I defined myself more as Developer than Programmer.

    Wikipedia definition of “Programmer” says:

    A computer programmer, sometimes called a software developer, a programmer or more recently a coder (especially in more informal contexts), is a person who creates computer software. The term computer programmer can refer to a specialist in one area of computers or to a generalist who writes code for many kinds of software.

    I still do not write software used directly by users — I usually build someone else’s software. But I used to do write programs and sometimes like to do something. In some programming language and I had a chance to use many of them.

    BASIC

    Probably most of people born in 70s started their experience with programming using BASIC on 8bit computers. If you were out of luck you could end with Commodore 64 and it’s “PET BASIC” without any way to make use of audio/video capabilities of computer.

    I used Atari 65XE. First original Atari BASIC, then moved to Turbo BASIC XL (was faster and better). I wrote several programs in my teenager years. Enhanced some of them with short routines written in 6502 assembler.

    Once moved to Amiga and then PC I had two times when BASIC knowledge helped. During studies one of teachers knew only GW-BASIC so we had to write programs using it. Other time was some kind of BASIC on PIC 16F84 microcontroller.

    Oh, and I wrote my first 8k intro in something resembling BASIC.

    Assembler

    Going lower in programming stack… During Atari years I learnt a bit of MOS 6502 assembly. Once moved to Amiga had quick lesson of m68k programming. Used it to understand how “Happy New Year! 96” virus was working. Then wrote simple tool (using High Speed Pascal compiler) to kill it in all infected binaries. Some months later I got antivirus software which did it properly.

    During studies we had lot of Zilog Z80 assembly. Mostly on some old controllers from some unnamed milk company (20 keys, 8 places of 7-segment display, ADC/DAC etc.). But, as I know how to drawing block diagrams, I always had beer from other students ;D One day someone brought something weird instead of Z80 code. Turned out it was 16-bit x86 code. Took me a few minutes of reading (it was simple program).

    Then came first real MCU: PIC 16F84. First we programmed it in BASIC (it was weird), then C and finally assembly. Small amount of instructions, simple names — was fun to program.

    I never learnt x86-64 or ARM assembly. There was no need.

    LOGO

    Oh, LOGO… And turtle drawings. On Atari you could have 4 (or 5) turtles drawing at same time.

    I even tried to write some simple game in LOGO. But need of calling garbage collecting too often made it worthless.

    It is also first language where I wrote code for someone on piece of paper. And it worked.

    C/C++

    Some people would split them into two, but not me. I learnt them in a way that I wrote some of my programs in pure C, some in some form of C++ etc. Depends on needs.

    During studies some programs were first written in pure C on my Amiga. Then I used someone’s PC to add Windows UI to it (Borland C++ Builder mostly). My master thesis was kind of terrible C++ and I am not proud of it.

    My probably most complicated application is my module player. I never released it despite spending about 13 years on it (from time to time). Used it to learn Qt programming a bit.

    Pascal

    I learnt Turbo Pascal in a school. A bit. Never wrote anything complex in this language. Used it few times on Amiga.

    One lab during studies involved Pascal programming. I wrote code, someone reviewed it and we run it. Robot which we were supposed to control went crazy and demolished room a bit. After simple fix we repeated and all went as supposed.

    During last year I played a bit with Mad-Pascal for Atari XL/XE (cross compiler runs on Windows or Linux). Maybe there will be something from it one day.

    AmigaE

    During my Amiga years I wanted to have some way of programming. No one from my friends had any C compiler, Pascal was not popular so I went with AmigaE.

    It was strange language. Kind of Pascal, kind of C. Allowed to mix assembly with AmigaE in one file.

    But compiler was fast and worked on 1MB machine (I had hard drive). All includes (called ‘modules’) were pre-compiled so building application was quick.

    I used it to write several applications. Released one of them in shareware model — full copy costed 5 USD (or 5 PLN for Poland). At the end of development I had about 100 users. It paid for some of my Amiga hardware.

    PHP

    After studies, when I had fresh Masters in Automation and Robotics, I had to decide which way to go. Turned out that my SQL knowledge from work done during previous years can be used. Just had to learn PHP.

    After 2 weeks of learning language I was at test period in one portal. And became ‘main programmer’ there. Not because I knew PHP better then the other guy. I was good at organizing work, sorting out project into steps etc. And being GNU/Linux user I had some knowledge useful to be ‘second after admin’.

    I worked in few companies writing PHP code. For several customers. And after about 6 years I dropped it and went into Embedded Linux territory.

    Nowadays I officially lack any knowledge of PHP language.

    Python

    Another “P” language… I started using Python in 2004. Right after I found OpenEmbedded as most of it’s code was (still is) Python.

    So for 17 years I am repeating that “one day I will learn Python properly” :D

    This is my favourite programming language. Used it in OpenEmbedded, OpenStack, this blog and countless projects. Nowadays if I have to write some script then most of time it is in Python.

    Minimal version I target is still 3.6 as this is what RHEL 8 uses.

    Shell

    Scripts. When not in Python then in shell. Most of time I use Bash. It works, I do not use most of it’s features.

    I can write scripts in plain POSIX shell. Not of fan but can. Some lessons learnt from years of working in embedded Linux area.

    Perl

    When “P” languages are mentioned, Perl has to be present. Never learnt it. Had to maintain some scripts in Perl in past. I even managed to make some improvements to them but no idea was it in proper way. It worked.

    Forth

    Reverse Polish Notation is crazy. It feels like source code written by master Yoda. I got Forth in Atari times and tried to write something. Probably never went behind ‘hello world’ ;D

    Java

    Never learnt. In past I found farm bot code for Ingress game. In Java. Took the code, expanded it, added several improvements. It worked fine. But became harder and harder to expand.

    So I rewrote it in Python 3 and forgot about Java.

    Javascript

    Just no. Like PHP.

    Other ones

    During years of building and porting software I had a chance to write something in several less popular languages. Erlang, Haskell, Rake, Ruby etc.

    Most of time it was something just a bit more complex than ‘hello world’ — to check does compiler/interpreter work.

    Written by Marcin Juszkiewicz on
Page 1 / 103
Older posts »