The end of “Mali question”?

For several Linaro Connect events we had sessions about state of graphics drivers on ARM platforms. I attended most of them and got a reputation of person asking problematic questions.

But situation has changed. With Panfrost project happening. It is a Foss driver for Arm Mali Midgard graphics chipset (Bifrost support on a way). It went from “wow, a triangle” to “we can play some games or run a desktop” in quite short time.

At BKK19 Linaro Connect we had “State of opensource drivers for mobile GPU chips” session. Freedreno, Etnaviv, vc4, v3d, Panfrost, Lima etc. What they target, what was already achieved, what are plans. Great progress across whole ARM world. And several questions from the audience. And interesting answers as well.

Mali then. Grant Likely from Arm told that they are looking how Panfrost is going. From company perspective both Midgard and Bifrost chips are “done, in a field” product which will not get changes. Still — engineering support goes entirely into their binary drivers as this is what their customers are using. Situation may change if those customers start asking for open drivers.

I do not use any Arm hardware with Mali GPU anymore. But hope that at next Linaro Connect instead of asking famous “Mali question” we will rather discuss how it runs on our devices.

How did I hacked Linaro Connect BKK19 puzzle

One of Linaro Connect traditions is a puzzle to solve. Created by Dave Pigott. And recent BKK19 event was not any different. There was puzzle announcement on the first day — right before first keynote. But no one could be first to answer at that time…


As usual before Connect I looked at a map and marked several locations in Bangkok as places to visit. Then took a look at the official BKK19 application. Installed it on my phone and started.

First screen had few paragraphs of text. Some information about the event and schedule. But there was also paragraph with information about the puzzle. WITH the link to it!

I clicked to get some redirection to Google Forms website. With information that this form is not available for users outside of organization. As I do not have work accounts on my phone I checked for redirection link and loaded it on my desktop. And landed into the puzzle.

Puzzle form

It was a bit different than version provided during Linaro Connect. There was a graphics with seven (official one had eight ones) columns of text. Under it was graphics with chess figures:

H1 G3 F1 H2 G4 E3 D1
A4 C3 B1 A3 C4 B6 A8
B5 A7 C8 D6 E8 F6 D5
G8 H6 F5 G7 H5 F4 E2
C6 A5 B7 C5 A6 B8 D7
F3 G1 H3 G5 H7 F8 E6
F7 H8 G6 H4 G2 E1 D3
A2 B4 C2 A1 B3 D2 E4

Dave later said that knight was not present in it but I did not noticed that. There was a plan to add that graphics into official puzzle if no one provides proper answer until Wednesday.

From text I noticed that it is chess related. Took a sheet of paper, draw 8×8 grid on it and started following each row on it with different markings each time. Hm… nothing came to my mind. Noted missing entries.

Help me Google, you are my only hope

Then started googling “knight chess puzzle” and got “knight’s tour” links on first page. Started reading what it is about. Then restarted tracking knight’s moves from the puzzle with adding missing ones. Turned out that this was it.

Let me mail Dave

I submitted “knight’s tour problem” as the answer and wrote to Dave:

I see that they are online already.

Are the ones there official ones or testing one?

Turned out that I found testing version. Which did not even collected emails when someone provided an answer. But there was one such sent so Dave marked it as my submission. And the link got removed from BKK19 application.

At pool bar

I arrived in Bangkok on Saturday. Met Dave at pool bar and we had a chat about the puzzle. Was fun to see how surprised people around were that first answer was already provided. I asked Dave to not give me the proper answer nor info was my answer good so I would not spoil other people.

During Connect few attendees asked me about the puzzle, how it went. Kept away from spoiling them.

And the winner is…

Then Friday happened with closing remarks session. Only a few people provided two words answer (“knight’s tour”) and few three word one (“closed knight’s tour”). My name was in “special mention” section.

Turned out that there is separate award for hacking a puzzle. It was 3rd time when it happened. I got a waterproof action camera (EZVIV S1C model) — will find some use for it sooner or later ;D

Whole puzzle was fun. Thanks go to Dave for creating it and providing me with a copy of both results slide and graphical hint so I could use it in blog post.

Moving Kolla images to Python 3

Python… To use 2.7 or to go for 3.x? To “be compatible” or to “use fancy new features”. Next year Python 2 gets finally unsupported upstream.

Kolla state

Kolla can run under Python 2.7 or 3.x and results will be the same. All those container images containing OpenStack components and whatever needed to get them running. But so far all of those images will use Python 2.7 inside…

I proposed to take care of it during Train cycle (we are at Stein now). Then we had a meeting and decided — let us do that in Stein! If I only knew how many issues it will bring…


First some cleaning and refactoring:

Distribution handling

Then created new variable to keep which exactly version of Python 3: distro_python_version. And then used wherever needed.

In meantime we got support for Red Hat Enterprise Linux 8 distribution. Where Python 3 is the only version available.

Finally we moved Ubuntu to use Stein UCA which also switched us to Python 3 in several images.

CI failures

And then hell gates opened…

We moved our CI jobs from Ubuntu 16.04 ‘xenial’ to Ubuntu 18.04 ‘bionic’. OMG. So many failures. When Mark Goddard found out why (too old ‘setuptools’) we had to pile up fixes to get CI back into working state.

Nova images were failing. Turned out that Debian/Ubuntu ‘qemu’ package is no longer metapackage but useless dummy. And then it brought ‘armv6l’ architecture to Nova so it started failing. There were three different patches to handle it and problem got solved.

OracleLinux image got repository information files renamed. And as we edit them we had to adapt.

Python 3 got enabled in another set of images.

Changing other projects

Kolla uses other projects, right? Are they Python 3 ready?

Had to patch several ones:

Some of those changes were requirements updates to mark “Python 2 only” components (enum34, functools32, UcsSdk) or getting rid of Unicode characters from files which should be be US-ASCII.

Bifrost one cleaned situation as we are using it within virtualenv so Python packages were going crazy. Now they handle it ;D

Current status

The most important patch switching all image to Python 3 for Debian/Ubuntu is still in review. Waiting for karbor fix getting merged and CentOS 7/OracleLinux 7 builds getting working again.

We also moved to Debian ‘buster’ release. It is now in ‘freeze’ state so no big changes allowed and provides us with many updates making Debian/Ubuntu blocks easy.

Python 2 stays 🙁

There are few images where we still have Python 2. Anything related to Ceph has it because Debian/Ubuntu ‘ceph-common’ package depends on Py2 packages. We install Py3 ones there.

We need Arm64 systems for developers. Again.

There are several posts and videos around recent discussion in “ARM announces Ares” thread on Real World Technologies website. People quote Linus Torvalds like crazy…


Yes, we need AArch64 (real Arm64 architecture name) systems for developers. SBBR compliant ones so their users will not have to deal with ‘how the hell should I boot it’. Instead get Linux distribution and install it like on current developer boxes (PXE, USB, CD etc).

SBBR: System Boot Base Requirements — specification which defines firmware of AArch64 server system. Which mean UEFI, ACPI on hardware compliant with SBSA (Server Base System Architecture) specification. In short: boring box which just works with any serious Linux/BSD distribution and for any random user looks and works like x86-64 box when it comes to booting.

Hardware formats

Some people would like small format — like Intel NUC, Apple Mini. Just open a box, put m.2 storage, few sticks of memory and go.

Other ones would like (micro)ATX format. You put it into standard case, add m.2 and sata storage, few sticks of memory, PCI Express graphics card and use.

So far there are no such systems in affordable prices. Or you have to deal with some issues.

Software issues

Anyway, you got hardware. Or access to. Or even bloody VM somewhere. So you start development. And often hit a wall…

Nowadays development culture ‘loves’ to use random binaries from the Internet. You do not wait until someone package software you would like to use, you grab compiled binaries from vendor/authors. Their website or github/gitlab release page.

And nope. x86-64 binaries only. So you dig for some aarch64 builds or look how to build it on your own. And you do not work on your project but on someone’s else one. Or you change your one to not use that external component.

Language extensions

Or your project is in Python/Ruby or other language with extensions compiled to native code. And simple “pip install scipy” which took 4s on your x86-64 machine now just fails…

So you install compilers, Python headers, check which libraries you should have installed, which versions of them, do your distro have them etc. Finally “pip install scipy” works. After several hours from your development time.


Getting AArch64 hardware for developers is important. When it happen? One day. Maybe even before people forget that such architecture existed.

We talk about it during each Linaro Connect. So far nothing serious came from it. We had some failed attempts like Cello or Husky. There is Synquacer with own set of issues. Some people use MACCHIATObin. Some still use Applied Micro Mustangs which should get a place in computer museums.

It is chicken and egg issue. No one makes affordable AArch64 systems because no one buys them. Because no one makes them. Hardware vendors concentrate on server market — no chips to choose for developer systems.

Three years of system calls table

Porting software often involves system calls. Usually their numbers differ between architectures. Some calls are missing, some are specific to platform. Normal stuff.

I knew that, you knew that, someone other knew. Or not. But looking into kernel/libc headers each time was boring so I created syscalls table for it. It was small project for personal use.

One day Arnd Bergmann sent me set of patches which rewrote table generation. From few architectures to which I had access (so could run binary) to every arch supported in Linux kernel. Then some architectures got dropped from kernel. But I kept their data in case someone needs (just moved it to the far right side).

Webpage look changed during those years. From ugly HTML table to table using DataTables framework. With plugins adding rows colouring, search option and few other tricks.

And several funny moments happened related to this table. At FOSDEM 2018 I visited Valgrind devroom and was greeted with “Ah, so you are that syscalls guy!” as it turned out that page was a great help. One of my friends was porting his lowlevel software from x86-64 to aarch64 one day and asked me “man, why there is no open() syscall on aarch64?”. Etc. etc.

I do not remember when last time used it for something. Keeping it updated every rc1 kernel release so anyone can see actual state. I know that people use it cause from time to time someone mentions it or gets directed to it.

Twelve years of remote work

Twelve years ago I stopped writing PHP code at work. And moved to paid embedded consultant role. Remote paid embedded consultant role even. And never moved to office since. Companies paying for my work were changing, flats/cities were changing, desks too.

People ask me how it is to work remote at each conference I attend. Often say that they could not work because of a company they work for or because they do not feel that it would fit them. Usually it comes to distractions, being at home etc.

So what you may need for remote work? It can differ. For me there are several requirements:

  • task oriented job
  • desk
  • comfortable chair
  • good monitor(s)
  • input devices
  • quiet environment
  • coffee machine access

Job requirements

First of all being remotee does not work when company does not allow to be out of office whole time. All those offers with “we allow one day per week to be remote” are a joke. It shows that company tries to follow trend but is not ready for it yet. In such cases most of the discussions are during physical meetings (so without remotees).

Work hours

Other thing is work hours. The first company I did remote work for was OpenedHand from UK. We had three time zones in use every day: UK, Europe, Finland. Which mean that if I start work between 8:00 and 10:00 then it is fine. But that is also common in non-remote work too. There I usually kept same/similar work hours that office guys did.

When I moved to work for other foreign companies (Red Hat Polska sp. z o.o. is Polish company but we can ignore that here) time difference usually got added.

Task oriented job

Time zone differences mean you can not keep same work hours as office. This is where task oriented job starts. You need to know what you have to do and when deadline is. And do that in time which fits you best.

This allows to take your child to a doctor or cinema during office work hours and spend other part of day on tasks.

If you bill by hours then some tool to mark hours/quarters you worked for customer is good to have.

With such kind of work come reports. Depends on company it can be weekly emails, jira cards, bugtracker issues etc.

I remember working for Vernier company where we had nine hours time difference. First we spent few hours on discussions what we need to do, how we split work into tasks. Then each day started with mail, ‘git svn pull’, changes, builds, commits, rebases and at the end of my day ‘git svn dcommit’. And email with list of done things, what needs work on their side, patches for review and plans for next day. This gave us 16 hours long developer days.

Free days

Depends on your contract you may or may not have days off available. If you have tasks to do then you can travel and work at your destination, right? I often took my daughter to my mother in a way that they were going to a beach or something while I was working.

Self discipline

This is where many of people give up. Not being in an office means you can do whatever because no one will notice, right? Yes and no at same time.

Wasting time

From one side it helps you waste time on whatever you want. But once you start piling not done work someone will notice. And you may have discussion with manager or lose contract.

When I worked on OpenEmbedded it was often “do some changes and have few hours of time due to build taking place”. Upgrading machine helped (or having remote access to powerful builders). Now my builds do not take so much time 😀

This was also a time when to learn something, clean a flat, do laundry or even watch some TV series episode.


Make sure that you have proper desk and chair. Read safety regulations and choose wisely.

Select good input devices. I use Microsoft Ergonomic Desktop 4000 keyboard and A4Tech Bloody ZL5 Sniper mouse. Huge mousepad helps (I use 35x45cm one).

Go for two/three monitors (same model if possible). Or one ultrawide (3440×1440 is nice). Mount them on arm to have space under them available.

RSI and other issues

Repetitive strain injury is something I would not wish even to enemies. Here is where combination of desk/chair/keyboard/mouse helps.

Do breaks during work — I use RSIbreak application to force me to do them. 20 seconds every ten minutes and 60 seconds every hour. Enough to look at something other than display (short ones) or walk a bit (make coffee, grab a fruit).

If you feel pain in your back do something about it. Massage helps. Pay a specialist to do it. And then repeat from time to time.

Check your sight yearly if glasses/lenses or bi-yearly if not.

Would I go back to an office?

No. Got used to work in environment that I control. Where I can choose if I want silence or some music. Where I meet other people when/if needed.

From sprint experiences I know that two-three days of work with some other folks exhausts me. Then I required headphones for the rest of week and a place to sit and work without interruptions.