1. TensorFlow build times

    The big part of my work during last months was related to TensorFlow. We build it on Linaro CI. Resulting wheels are available on Linaro snapshots server.


    When you dig there you can find those versions:

    • 1.5.x
    • 2.4.x
    • 2.5.x
    • 2.6.x
    • 2.7.x (git HEAD)

    How to install?

    TensorFlow requires several other Python packages and some of them are not distributed as AArch64 binary wheels. For this we have Python cache repository at Linaro snapshots.

    So how to install TensorFlow (2.6.0 or 2.7.0 version):

    $ export PIP_EXTRA_INDEX_URL=https://snapshots.linaro.org/ldcg/python-cache/
    $ pip install tensorflow-aarch64

    And it will be done. Package is renamed from “tensorflow-cpu” as there is a plan of uploading them to Pypi. For older versions please check our snapshots server.

    How do we build?

    Jenkins runs shell script which then starts container, installs Ansible and then the rest of build is done using it. Playbooks, roles and shell scripts are stored in Jenkins jobs repository.

    The process is quite simple — we choose which versions to build (from 1.5, 2.4, 2.5, 2.6, git HEAD selection) and then Ansible loops over them. All dependencies’ versions are stored in variables file.

    Whole work is done inside of “manylinux2014” container to get a way of building for wide selection of Python releases. Build covers versions from 3.6 to 3.9 one (we plan to enable 3.10 when possible) in one run.

    Build times

    To compare speed of several systems I have available I ran a build on each and compared to times on Linaro CI machine.

    Some details:

    Machine name processor cores threads memory note
    Oracle cloud A1 Altra 16 16 96 VM.Standard.A1.Flex
    Linaro CI ThunderX2 2x28 56 240 SMT disabled
    SolidRun HoneyComb LX2160 16 16 32
    my work laptop i7-8665U 4 8 32 SMT enabled
    my desktop Ryzen 5 3600 6 12 32 SMT enabled

    Build time includes fetching files from network. Python packages comes from either Pypi or Linaro Python cache repository.


    Install Docker, pull manylinux2014 image, fetch script from Linaro CI job git repository, run it.

    $ docker pull quay.io/pypa/manylinux2014_aarch64
    $ wget https://git.linaro.org/ci/job/configs.git/plain/ldcg-python-manylinux-tensorflow/build.sh
    $ mkdir BUILD
    $ cd BUILD
    $ export WORKSPACE=
    $ time build26=true bash ../build.sh

    BUILD directory created as job starts with cleaning workspace (stored in WORKSPACE variable).


    Machine name build time
    Oracle cloud A1 3:15:53.250
    Linaro CI 1:56:15.994
    SolidRun HoneyComb 7:29:01.922
    my work laptop 10:01:19.044
    my desktop 3:30:34.821
    Build time (in hours)
    Build time (in hours)


    HoneyComb is a good system for development work. As long as there are other systems which will do hard work on building things.

    ThunderX2 is a workhorse. Give it something to do and it delivers. It was also very expensive (Avantek started with 13k USD for workstation).

    Oracle cloud instance used Ampere Altra cpu. Would not be surprised if it beats ThunderX2 system when more cores are used (16 cores was limit of free tier).

    I used my work laptop because it was available. Did not expected much from it. But in past benchmarks it was close to my previous desktop system.

    And with my desktop… It was quite cheap solution 2 years ago. Looks like price/performance is something where x86-64 is a king.

    Written by Marcin Juszkiewicz on
  2. Python package for system calls information

    Python is my favourite language. System calls table is my favourite side project. So why not merge them?

    That’s how “system-calls” Python package was born.


    Mostly because I could. And thought that maybe someone else will find it useful. Also found out that it allows me to learn how to build own Python package, how Pypi uploads work etc. stuff. And I can remind myself how packaging works as having it in Debian and Fedora would be useful experience.

    What it can do?

    Module allows to check information about Linux system calls in several ways:

    • which number system call has on architecture XYZ
    • is system call supported on architecture XYZ

    Code for it is quite simple:

    import system_calls
    syscalls = system_calls.syscalls()
    for test_call in ['openat', 'osf_uadmin', 'nosuchcall']:
            print(f"System call '{test_call}' has number: {syscalls[test_call]}")
        except system_calls.NoSuchSystemCall:
            print(f"No such system call '{test_call}' on any architecture")
        except system_calls.NotSupportedSystemCall:
            print(f"System call '{test_call}' is not supported on this "

    Above example shows usage for host architecture. But treating “system_calls.syscalls” as dictionary does not allow to ask for data of different architecture. For such use there is “get()” method:

    import system_calls
    syscalls = system_calls.syscalls()
    for test_call in ['openat', 'osf_uadmin', 'nosuchcall']:
            print(f"System call '{test_call}' on arm64 has number: "
                  f"{syscalls.get(test_call, 'arm64')}")
        except system_calls.NoSuchSystemCall:
            print(f"No such system call '{test_call}' on any architecture")
        except system_calls.NotSupportedSystemCall:
            print(f"System call '{test_call}' is not supported on this "

    You can also use “get()” method for host architecture as second argument is optional.

    You can find more examples in project’s repository.


    There is one bonus script in this project: “syscall”. Can be used to check for information about system calls for host or any supported architecture:

    $ syscall open
    On x86_64 system call number 2 is open()
    $ syscall open arm64
    On arm64 system call open() is not supported.
    $ syscall 32
    On x86_64 system call number 32 is dup()
    $ syscall 32 arm64
    On arm64 system call number 32 is flock()
    $ syscall openat mipso32
    On mipso32 system call number 4288 is openat()
    $ syscall open noa
    Architecture noa is not supported.
    $ syscall nos
    There is no such system call as nos().

    Mistakes happen

    When I wrote first version I did not thought about uploading it to Pypi and sharing with other users. And when I started packaging it for Fedora I realized that I made some serious mistake.

    Package is called “system-calls” but code is stored in “syscalls” directory. Fixed that in 5.15.2 version so it is now stored in “system_calls” one.

    5.15.3 has all tests and examples fixed to use proper names.

    If you find some other issue then please report it in project’s repository.


    I plan to make package for Debian and Fedora distributions. This will take some time probably as I have to remind myself few things.

    For now you can grab Fedora 34/35/rawhide and EPEL 8 compatible packages from my “system-calls COPR“.

    Written by Marcin Juszkiewicz on
  3. Programming languages

    Today is Programmer’s Day. 13 years ago I wrote a post “Am I programmer” where I defined myself more as Developer than Programmer.

    Wikipedia definition of “Programmer” says:

    A computer programmer, sometimes called a software developer, a programmer or more recently a coder (especially in more informal contexts), is a person who creates computer software. The term computer programmer can refer to a specialist in one area of computers or to a generalist who writes code for many kinds of software.

    I still do not write software used directly by users — I usually build someone else’s software. But I used to do write programs and sometimes like to do something. In some programming language and I had a chance to use many of them.


    Probably most of people born in 70s started their experience with programming using BASIC on 8bit computers. If you were out of luck you could end with Commodore 64 and it’s “PET BASIC” without any way to make use of audio/video capabilities of computer.

    I used Atari 65XE. First original Atari BASIC, then moved to Turbo BASIC XL (was faster and better). I wrote several programs in my teenager years. Enhanced some of them with short routines written in 6502 assembler.

    Once moved to Amiga and then PC I had two times when BASIC knowledge helped. During studies one of teachers knew only GW-BASIC so we had to write programs using it. Other time was some kind of BASIC on PIC 16F84 microcontroller.

    Oh, and I wrote my first 8k intro in something resembling BASIC.


    Going lower in programming stack… During Atari years I learnt a bit of MOS 6502 assembly. Once moved to Amiga had quick lesson of m68k programming. Used it to understand how “Happy New Year! 96” virus was working. Then wrote simple tool (using High Speed Pascal compiler) to kill it in all infected binaries. Some months later I got antivirus software which did it properly.

    During studies we had lot of Zilog Z80 assembly. Mostly on some old controllers from some unnamed milk company (20 keys, 8 places of 7-segment display, ADC/DAC etc.). But, as I know how to drawing block diagrams, I always had beer from other students ;D One day someone brought something weird instead of Z80 code. Turned out it was 16-bit x86 code. Took me a few minutes of reading (it was simple program).

    Then came first real MCU: PIC 16F84. First we programmed it in BASIC (it was weird), then C and finally assembly. Small amount of instructions, simple names — was fun to program.

    I never learnt x86-64 or ARM assembly. There was no need.


    Oh, LOGO… And turtle drawings. On Atari you could have 4 (or 5) turtles drawing at same time.

    I even tried to write some simple game in LOGO. But need of calling garbage collecting too often made it worthless.

    It is also first language where I wrote code for someone on piece of paper. And it worked.


    Some people would split them into two, but not me. I learnt them in a way that I wrote some of my programs in pure C, some in some form of C++ etc. Depends on needs.

    During studies some programs were first written in pure C on my Amiga. Then I used someone’s PC to add Windows UI to it (Borland C++ Builder mostly). My master thesis was kind of terrible C++ and I am not proud of it.

    My probably most complicated application is my module player. I never released it despite spending about 13 years on it (from time to time). Used it to learn Qt programming a bit.


    I learnt Turbo Pascal in a school. A bit. Never wrote anything complex in this language. Used it few times on Amiga.

    One lab during studies involved Pascal programming. I wrote code, someone reviewed it and we run it. Robot which we were supposed to control went crazy and demolished room a bit. After simple fix we repeated and all went as supposed.

    During last year I played a bit with Mad-Pascal for Atari XL/XE (cross compiler runs on Windows or Linux). Maybe there will be something from it one day.


    During my Amiga years I wanted to have some way of programming. No one from my friends had any C compiler, Pascal was not popular so I went with AmigaE.

    It was strange language. Kind of Pascal, kind of C. Allowed to mix assembly with AmigaE in one file.

    But compiler was fast and worked on 1MB machine (I had hard drive). All includes (called ‘modules’) were pre-compiled so building application was quick.

    I used it to write several applications. Released one of them in shareware model — full copy costed 5 USD (or 5 PLN for Poland). At the end of development I had about 100 users. It paid for some of my Amiga hardware.


    After studies, when I had fresh Masters in Automation and Robotics, I had to decide which way to go. Turned out that my SQL knowledge from work done during previous years can be used. Just had to learn PHP.

    After 2 weeks of learning language I was at test period in one portal. And became ‘main programmer’ there. Not because I knew PHP better then the other guy. I was good at organizing work, sorting out project into steps etc. And being GNU/Linux user I had some knowledge useful to be ‘second after admin’.

    I worked in few companies writing PHP code. For several customers. And after about 6 years I dropped it and went into Embedded Linux territory.

    Nowadays I officially lack any knowledge of PHP language.


    Another “P” language… I started using Python in 2004. Right after I found OpenEmbedded as most of it’s code was (still is) Python.

    So for 17 years I am repeating that “one day I will learn Python properly” :D

    This is my favourite programming language. Used it in OpenEmbedded, OpenStack, this blog and countless projects. Nowadays if I have to write some script then most of time it is in Python.

    Minimal version I target is still 3.6 as this is what RHEL 8 uses.


    Scripts. When not in Python then in shell. Most of time I use Bash. It works, I do not use most of it’s features.

    I can write scripts in plain POSIX shell. Not of fan but can. Some lessons learnt from years of working in embedded Linux area.


    When “P” languages are mentioned, Perl has to be present. Never learnt it. Had to maintain some scripts in Perl in past. I even managed to make some improvements to them but no idea was it in proper way. It worked.


    Reverse Polish Notation is crazy. It feels like source code written by master Yoda. I got Forth in Atari times and tried to write something. Probably never went behind ‘hello world’ ;D


    Never learnt. In past I found farm bot code for Ingress game. In Java. Took the code, expanded it, added several improvements. It worked fine. But became harder and harder to expand.

    So I rewrote it in Python 3 and forgot about Java.


    Just no. Like PHP.

    Other ones

    During years of building and porting software I had a chance to write something in several less popular languages. Erlang, Haskell, Rake, Ruby etc.

    Most of time it was something just a bit more complex than ‘hello world’ — to check does compiler/interpreter work.

    Written by Marcin Juszkiewicz on
  4. I moved to the new flat

    There are moments in a life when we want to change something. So I changed place of living.

    Moved out of rented flat into my own one. Still Szczecin, still suburbian part of it (I moved by about 1500 meters). Nothing fancy, plain 3 rooms but it is mine (once I pay bank’s part in next few years).

    It needs some changes as I am taller than previous owners. My list of ideas to make gets longer and longer each time I see something interesting in store of someone’s home.

    Server wardrobe

    Previous owners left several built-in wardrobes so I decided to use one for my servers. Moved my NAS, router, HoneyComb and Mustang into top shelf with all required cabling. This way none of them occupy my desk area.

    And why ‘server wardrobe’ name for it? In Polish we use ‘szafa serwerowa’ for ‘server rack’. But ‘szafa’ is also ‘wardrobe’ so that’s why :D

    New desk for my office

    In previous years I used a custom made desk. It was real monster — L-shaped with several shelves under. That allowed me to keep multiple systems running on/under desk (there were times when 16 ports in Ethernet switches were not enough).

    But it was also heavy and hard to manipulate/move. So it went to trash.

    For first days I used some table but it was not the right choice. In meantime I looked at several setups and desks to find which features to look for.

    Finally ended with buying VS series 901 desk. 180x90 cm gave me a lot of space, all cables go though dedicated gap into cable organiser under desk. And there I have Ethernet switch, laptop’s docking station (and it’s power supply), power supply for my monitor, usb hubs and all cables. No more floating cables! And all that for ~100€ for used one.

    Nicely organized desk
    Nicely organized desk

    Still need to buy some office storage unit with some drawers — it would go on left side of desk (where currently bathroom drawers are present). Looking through used office furniture again.

    Hall of fame

    On photo above you may see shelf behind my desk. I plan to organize some kind of ‘Hall of fame’ there for devices which helped with my carieer:

    • Sharp Zaurus SL-5500 (got one recently from Andrea Adami)
    • Atmel AT91SAM9263-EK developer board
    • Applied Micro Mustang

    And few other ones as I need to track where I have them first or where I can get them (anyone has spare ST NDK-15?).


    Due to some weird location of light switches I decided to speedup a bit and move most of home lights to controllable ones. To make things easier I went into system used by one of my friends: Trådfri by Ikea.

    Lights, motion sensor to get some dimmed light during night walk to toilet, remotes etc.


    It is good to have own space for living. I can alter it in any way I want. Without checking will it be useful for short people.

    Written by Marcin Juszkiewicz on
  5. My twenty plus years of IRC

    In 1996 I started studies at Białystok University of Technology. And one of early days I found that corridor with text terminals. Some time later I joined that crowd and started using HP-2623A term with SunOS account.

    And one of first applications crowd shown me was ircII (other were bash, screen, pine and ncftp). So I am IRC user for over 24 years now and this XKCD comics can be about me:

    2078: He announces that he's finally making the jump from screen+irssi to tmux+weechat.
    Team chat


    As wrote above I started with ircII. It was painful to use so quickly some scripts landed — Venom, Lice and some others. Tried Epic, Epic2000 and some other clients to finally end with Irssi. I have never been a fan of GUI based ones.

    There were moments when people used CTCP VERSION to check what clients other users use. In old Amiga days I usually had it set to similar one as AmIRC one but with version bumped above whatever was released. Simple trolling for all those curious people. Nowadays it simply replies with “telnet” and there was a day when I logged to IRC server and exchanged some messages using just telnet :D


    For years I was user of IRCnet. It was popular in Poland so why bother with checking other networks. But as time passed and I became more involved in FOSS projects there was a need to start using Freenode, then OFTC, Mozilla etc.

    Checked how old my accounts are as it nicely show when I started using which network.


    IRCnet was my first IRC network. Stopped using it few years ago as all channels I was on went quiet or migrated elsewhere (mostly Freenode).

    For years I was visible on Amiga channels #amisia, #amigapl where I met several friends and I am in contact with many of them still.


    The journey started on 15th May 2004. This was time when I started playing with OpenEmbedded and knew that this is a project where I will spend some of my free time (it became hobby, then job).

    It was a place where CentOS, Fedora, Linaro, OpenStack and several other projects were present.

    NickServ- Information on Hrw (account hrw):
    NickServ- Registered : Mar 15 10:59:47 2004 (17y 9w 6d ago)
    NickServ- Last addr  : ~hrw@redhat/hrw
    NickServ- vHost      : redhat/hrw
    NickServ- Last seen  : now
    NickServ- Flags      : HideMail
    NickServ- *** End of Info ***


    For me OFTC means Debian. Later also virtualization stuff as QEMU and libvirt folks sit there.

    NickServ- Nickname information for hrw (Marcin Juszkiewicz)
    NickServ- hrw is currently online
    NickServ-   Time registered: Fri 10 Jun 2011 17:43:55 +0000 (9y 11m 10d 14:47:05 ago)
    NickServ- Account last quit: Tue 18 May 2021 09:07:55 +0000 (1d 23:23:05 ago)
    NickServ- Last quit message: Remote host closed the connection
    NickServ-         Last host: 00019652.user.oftc.net
    NickServ-               URL: http://marcin.juszkiewicz.com.pl/
    NickServ-      Cloak string: Not set
    NickServ-          Language: English (0)
    NickServ-           ENFORCE: ON
    NickServ-            SECURE: OFF
    NickServ-           PRIVATE: ON
    NickServ-             CLOAK: ON
    NickServ-          VERIFIED: YES


    And since yesterday I am on Libera as well.

    NickServ- Information on hrw (account hrw):
    NickServ- Registered : May 19 15:26:18 2021 +0000 (17h 32m 35s ago)
    NickServ- Last seen  : now
    NickServ- Flags      : HideMail, Private
    NickServ- *** End of Info ***

    What next?

    I saw and used several instant messaging platforms. All of them were younger than IRC. Many of them are no longer popular, several are no longer existing. IRC survived so I continue to use it.

    Written by Marcin Juszkiewicz on
  6. What ‘a new computer’ is?

    During last months I had several discussions about buying a new computer. Helped few friends with choosing setup etc. And several of them were surprised when I told that I never bought ‘a new computer’.

    8-bit era

    My first computer was Atari 65XE which my parents bought as a new computer. I learnt BASIC, tried few other programming languages (Forth anyone?) and played games. But it was not platform for long use.


    Two years later I earned money collecting berries in Swedish forests and bought my first computer — used Amiga 600. And two weeks later added 425MB hard disk inside. Nice improvement, still same 12” green monitor. More programming, less games. Demoscene watching started.

    Time passed, I sold A600 and bought used Amiga 1200 instead. Same hdd and monitor (also same mouse as I liked old one more). Then some CPU accelerator card, then another, new ATA controller, new hdd, used cd-rom drive and used 14” VGA mono monitor.

    AmigaOS was great operating system but platform was dying, hardware was expensive and slow, no new software.

    Let move to PC

    Year 2000 was moment when I decided to abandon AmigaOS and bought a PC. Which reused hdd, cd-rom and monitor. In some sense it was a new computer but still kept something from previous system. In next years some minor/major upgrades happened.

    In 2006 I switched architecture of my desktop system. From x86 to x86-64. In as cheap as possible way. It was time when embedded Linux was starting to not be “just a hobby”.

    Years passed, processors, mainboards, memory amount, storage, cases, expansion cards, monitors etc. were changing. But still there was no point of ‘ok, let me buy whole new computer’ as most of components were reused.

    Do laptops count?

    There were several laptops in meantime. One of them (Asus UL30A) was even brand new. But, contrary to other users, I use laptops only during travels — at home they usually sit connected to power and sometimes are used as headless build machine (as I tend to use different Linux distribution than my desktop there).


    And none of Arm systems I use count. I bought Sharp Zaurus SL-5500A (no longer own), Nokia N810 tablet and some small board computers like Wandboard (sold), Raspberry/Pi 3 (sold) or RockPro64. Android devices do not count either.


    Who knows, maybe one day I will buy a new computer. Whole one — case, mainboard, processor, memory, storage, graphics. Or something NUC like.

    Just there is nothing interesting so far to make such buy. Or I am not lazy enough to just buy whole workstation instead of building it on my own ;D

    Written by Marcin Juszkiewicz on
  7. Sometimes one tweet is enough

    Two weeks ago I wrote on Twitter:

    Is there some company with spare AArch64 CPU cycles?

    Opendev (project behind OpenStack and some more) would make use of another aarch64 server offer.

    Current one is iirc paid by @Arm, hosted by @equinixmetal and operated by @LinaroOrg.

    Why I did that? Maybe frustration, maybe burnout. Hard to tell. But I did. Without targeting any Arm related company as I did not wanted to force anyone to do anything.


    A few hours later I got an email from Peter Pouliot from Ampere Computing. With information that they provided hardware to Oregon State University Open Source Lab (OSUOSL in short) and that we may get nodes there.

    As I have no idea how exactly Opendev infrastructure works I added Kevin Zhao to the list. He is Linaro employee working on all instances of Linaro Developer Cloud and he maintained all AArch64 resources provided to Opendev.


    Kevin added Opendev infra admins: Clark Boylan and Ian Wienand. Peter added Lance Alberson from OSUOSL. I was just one of addresses in emails looking how things go.

    And it went nice. If was pleasure to read how it goes. Two days, 8 emails, arrangements were made. Then changes to Opendev infrastructure configuration followed and week later ‘linaro-us’ was not the only provider of AArch64 nodes.


    Opendev has two providers of AArch64 nodes now:

    • linaro-us-regionone
    • osuosl-regionone

    First one is paid by Arm Ltd, hosted at Equinix Metal (formerly Packet) and operated by Kevin Zhao from Linaro.

    Second one runs on Ampere provided hardware and is operated by OSUOSL admins.

    check-arm64’ pipeline on Opendev CI gets less clogged. And I hope that more and more projects will use it to test their code not only on x86-64 ;D

    Written by Marcin Juszkiewicz on
  8. Five years @linaro.org

    Five years ago I got an email from Kristine with “Linaro Assignee On-Boarding” title. Those were busy years.


    According to stackalytics I did 1144 reviews so far:

    OpenStack project Reviews
    kolla 641
    kolla-ansible 445
    releases 14
    nova 11
    loci 9
    requirements 7
    devstack 4
    tripleo-ci 3
    pbr 2
    magnum 2

    Those were different ones — from simple fixes to new features. Sometimes it was hard to convince projects that my idea makes sense. There were patches with over 50 revisions. Some needed split to smaller ones, reordering etc.

    And at the end AArch64 is just another architecture in OpenStack family. Linaro Developer Cloud managed to pass official certifications etc.


    Countless projects. From suggesting ‘can you publish aarch64 wheel’ to adding code to make it happen. Or work on getting manylinux2014 working properly.

    Linaro CI

    We have Jenkins setup at Linaro. With several build machines attached to it. And countless amount of jobs running there. I maintain several ones and my job is taking care and adding new ones when needed.


    Due to my work at Linaro Enterprise Group (renamed later at Linaro Datacenter & Cloud Group) I dealt with many AArch64 server systems. From HPe Moonshot to Huawei D06. With Qualcomm Falkor in meantime. Used CentOS 7/8 on them. Debian 9/10/11. Added needed config entries to Debian kernel to get them working out of the box (iirc excluding M400 cartridges no one maintained any more).


    I did “OpenStack on AArch64” talk at three conferences in a row. First Linaro Connect LAS16 (as group talk), then on Pingwinaria in Poland and finally in Kiev, Ukraine. Same slides, translated English -> Polish -> English and updated each time.

    Since then I do not give lectures at conferences any more. Prefer to attend someone’s talks. Or spend time on hallway discussions. Linaro Connect events were good place for them (and hope they will be in future too).

    At the end of each there was listing of every person who worked at Linaro for 5 or 10 years. Due to pandemic I missed that part. But hope for that memorial glass ;D

    Written by Marcin Juszkiewicz on
Page 1 / 102
Older posts »