Marcin Juszkiewicz

Python… To use 2.7 or to go for 3.x? To “be compatible” or to “use fancy new features”. Next year Python 2 gets finally unsupported upstream.

Kolla state

Kolla can run under Python 2.7 or 3.x and results will be the same. All those container images containing OpenStack components and whatever needed to get them running. But so far all of those images will use Python 2.7 inside…

I proposed to take care of it during Train cycle (we are at Stein now). Then we had a meeting and decided — let us do that in Stein! If I only knew how many issues it will bring…

Cleaning/refactoring

First some cleaning and refactoring:

• debian/ubuntu: drop mysql compat stuff, unify on openssl 1.1
• openstack-base: drop pika-pool
• Introduce ‘get_pip’ macro
• helm-repository: drop unused os_client_config variable

Distribution handling

Then created new variable to keep which exactly version of Python 3: distro_python_version. And then used wherever needed.

In meantime we got support for Red Hat Enterprise Linux 8 distribution. Where Python 3 is the only version available.

Finally we moved Ubuntu to use Stein UCA which also switched us to Python 3 in several images.

CI failures

And then hell gates opened…

We moved our CI jobs from Ubuntu 16.04 ‘xenial’ to Ubuntu 18.04 ‘bionic’. OMG. So many failures. When Mark Goddard found out why (too old ‘setuptools’) we had to pile up fixes to get CI back into working state.

Nova images were failing. Turned out that Debian/Ubuntu ‘qemu’ package is no longer metapackage but useless dummy. And then it brought ‘armv6l’ architecture to Nova so it started failing. There were three different patches to handle it and problem got solved.

OracleLinux image got repository information files renamed. And as we edit them we had to adapt.

Python 3 got enabled in another set of images.

Changing other projects

Kolla uses other projects, right? Are they Python 3 ready?

Had to patch several ones:

• bifrost (and test their patch)
• ironic
• karbor
• kuryr-libnetwork
• monasca (api and persister)
• vmtp

Some of those changes were requirements updates to mark “Python 2 only” components (enum34, functools32, UcsSdk) or getting rid of Unicode characters from files which should be be US-ASCII.

Bifrost one cleaned situation as we are using it within virtualenv so Python packages were going crazy. Now they handle it ;D

Current status

The most important patch switching all image to Python 3 for Debian/Ubuntu is still in review. Waiting for karbor fix getting merged and CentOS 7/OracleLinux 7 builds getting working again.

We also moved to Debian ‘buster’ release. It is now in ‘freeze’ state so no big changes allowed and provides us with many updates making Debian/Ubuntu blocks easy.

Python 2 stays :(

There are few images where we still have Python 2. Anything related to Ceph has it because Debian/Ubuntu ‘ceph-common’ package depends on Py2 packages. We install Py3 ones there.

There are several posts and videos around recent discussion in “ARM announces Ares” thread on Real World Technologies website. People quote Linus Torvalds like crazy…

SBBR

Yes, we need AArch64 (real Arm64 architecture name) systems for developers. SBBR compliant ones so their users will not have to deal with ‘how the hell should I boot it’. Instead get Linux distribution and install it like on current developer boxes (PXE, USB, CD etc).

SBBR: System Boot Base Requirements — specification which defines firmware of AArch64 server system. Which mean UEFI, ACPI on hardware compliant with SBSA (Server Base System Architecture) specification. In short: boring box which just works with any serious Linux/BSD distribution and for any random user looks and works like x86-64 box when it comes to booting.

Hardware formats

Some people would like small format — like Intel NUC, Apple Mini. Just open a box, put m.2 storage, few sticks of memory and go.

Other ones would like (micro)ATX format. You put it into standard case, add m.2 and sata storage, few sticks of memory, PCI Express graphics card and use.

So far there are no such systems in affordable prices. Or you have to deal with some issues.

Software issues

Anyway, you got hardware. Or access to. Or even bloody VM somewhere. So you start development. And often hit a wall…

Nowadays development culture ‘loves’ to use random binaries from the Internet. You do not wait until someone package software you would like to use, you grab compiled binaries from vendor/authors. Their website or github/gitlab release page.

And nope. x86-64 binaries only. So you dig for some aarch64 builds or look how to build it on your own. And you do not work on your project but on someone’s else one. Or you change your one to not use that external component.

Language extensions

Or your project is in Python/Ruby or other language with extensions compiled to native code. And simple “pip install scipy” which took 4s on your x86-64 machine now just fails…

So you install compilers, Python headers, check which libraries you should have installed, which versions of them, do your distro have them etc. Finally “pip install scipy” works. After several hours from your development time.

Conclusion(?)

Getting AArch64 hardware for developers is important. When it happen? One day. Maybe even before people forget that such architecture existed.

We talk about it during each Linaro Connect. So far nothing serious came from it. We had some failed attempts like Cello or Husky. There is Synquacer with own set of issues. Some people use MACCHIATObin. Some still use Applied Micro Mustangs which should get a place in computer museums.

It is chicken and egg issue. No one makes affordable AArch64 systems because no one buys them. Because no one makes them. Hardware vendors concentrate on server market — no chips to choose for developer systems.

Porting software often involves system calls. Usually their numbers differ between architectures. Some calls are missing, some are specific to platform. Normal stuff.

I knew that, you knew that, someone other knew. Or not. But looking into kernel/libc headers each time was boring so I created syscalls table for it. It was small project for personal use.

One day Arnd Bergmann sent me set of patches which rewrote table generation. From few architectures to which I had access (so could run binary) to every arch supported in Linux kernel. Then some architectures got dropped from kernel. But I kept their data in case someone needs (just moved it to the far right side).

Webpage look changed during those years. From ugly HTML table to table using DataTables framework. With plugins adding rows colouring, search option and few other tricks.

And several funny moments happened related to this table. At FOSDEM 2018 I visited Valgrind devroom and was greeted with “Ah, so you are that syscalls guy!” as it turned out that page was a great help. One of my friends was porting his lowlevel software from x86-64 to aarch64 one day and asked me “man, why there is no open() syscall on aarch64?”. Etc. etc.

I do not remember when last time used it for something. Keeping it updated every rc1 kernel release so anyone can see actual state. I know that people use it cause from time to time someone mentions it or gets directed to it.

Twelve years ago I stopped writing PHP code at work. And moved to paid embedded consultant role. Remote paid embedded consultant role even. And never moved to office since. Companies paying for my work were changing, flats/cities were changing, desks too.

People ask me how it is to work remote at each conference I attend. Often say that they could not work because of a company they work for or because they do not feel that it would fit them. Usually it comes to distractions, being at home etc.

So what you may need for remote work? It can differ. For me there are several requirements:

• task oriented job
• desk
• comfortable chair
• good monitor(s)
• input devices
• quiet environment
• coffee machine access

Job requirements

First of all being remotee does not work when company does not allow to be out of office whole time. All those offers with “we allow one day per week to be remote” are a joke. It shows that company tries to follow trend but is not ready for it yet. In such cases most of the discussions are during physical meetings (so without remotees).

Work hours

Other thing is work hours. The first company I did remote work for was OpenedHand from UK. We had three time zones in use every day: UK, Europe, Finland. Which mean that if I start work between 8:00 and 10:00 then it is fine. But that is also common in non-remote work too. There I usually kept same/similar work hours that office guys did.

When I moved to work for other foreign companies (Red Hat Polska sp. z o.o. is Polish company but we can ignore that here) time difference usually got added.

Task oriented job

Time zone differences mean you can not keep same work hours as office. This is where task oriented job starts. You need to know what you have to do and when deadline is. And do that in time which fits you best.

This allows to take your child to a doctor or cinema during office work hours and spend other part of day on tasks.

If you bill by hours then some tool to mark hours/quarters you worked for customer is good to have.

With such kind of work come reports. Depends on company it can be weekly emails, jira cards, bugtracker issues etc.

I remember working for Vernier company where we had nine hours time difference. First we spent few hours on discussions what we need to do, how we split work into tasks. Then each day started with mail, ‘git svn pull’, changes, builds, commits, rebases and at the end of my day ‘git svn dcommit’. And email with list of done things, what needs work on their side, patches for review and plans for next day. This gave us 16 hours long developer days.

Free days

Depends on your contract you may or may not have days off available. If you have tasks to do then you can travel and work at your destination, right? I often took my daughter to my mother in a way that they were going to a beach or something while I was working.

Self discipline

This is where many of people give up. Not being in an office means you can do whatever because no one will notice, right? Yes and no at same time.

Wasting time

From one side it helps you waste time on whatever you want. But once you start piling not done work someone will notice. And you may have discussion with manager or lose contract.

When I worked on OpenEmbedded it was often “do some changes and have few hours of time due to build taking place”. Upgrading machine helped (or having remote access to powerful builders). Now my builds do not take so much time :D

This was also a time when to learn something, clean a flat, do laundry or even watch some TV series episode.

Health

Make sure that you have proper desk and chair. Read safety regulations and choose wisely.

Select good input devices. I use Microsoft Ergonomic Desktop 4000 keyboard and A4Tech Bloody ZL5 Sniper mouse. Huge mousepad helps (I use 35x45cm one).

Go for two/three monitors (same model if possible). Or one ultrawide (3440x1440 is nice). Mount them on arm to have space under them available.

RSI and other issues

Repetitive strain injury is something I would not wish even to enemies. Here is where combination of desk/chair/keyboard/mouse helps.

Do breaks during work — I use RSIbreak application to force me to do them. 20 seconds every ten minutes and 60 seconds every hour. Enough to look at something other than display (short ones) or walk a bit (make coffee, grab a fruit).

If you feel pain in your back do something about it. Massage helps. Pay a specialist to do it. And then repeat from time to time.

Check your sight yearly if glasses/lenses or bi-yearly if not.

Would I go back to an office?

No. Got used to work in environment that I control. Where I can choose if I want silence or some music. Where I meet other people when/if needed.

From sprint experiences I know that two-three days of work with some other folks exhausts me. Then I required headphones for the rest of week and a place to sit and work without interruptions.

My current server is few years old. And now runs plain Debian.

Beginning

I started using that server during my work at Canonical. So it got Ubuntu installed. According to OVH panel it was 13.04 release. Then 13.10, 14.04 and finally 16.04 landed. In pain. Took me two days to get it working again (mail issues).

At that time I decided that it will not get any Ubuntu update. The plan was to upgrade to proper Debian release. And Buster will get frozen soon…

One day I took a list of installed packages and started “ubuntu:xenial” container. Test shown will it be big work to do such upgrade. Turned out that not that much.

Today I saw a post saying that php 7.1 goes into “security fixes only” mode. And I had 7.0 in use… So decided that ok, this is the time.

Let’s go with upgrade

Logged in, added Debian repository, APT keys and started with installation of 4.19 kernel. And rebooted to it.

Machine started without issues so I started upgrade. Used aptitude as usual. There were 10-20 conflicts to solve and then package installation started.

Few file conflicts was on a way but APT handled most of them without issues. Two or free packages I had to take care by hand.

Next step was replacing remaining Ubuntu packages with Debian ones. Or removing them completely. Easy, smooth work.

Getting services running

After copying php-fpm config files from 7.0 to 7.3 release my blog went online.

Then some edits to Courier auth daemon config files (adding “marker”) and mails started flowing in both directions. But if you got mail that my mail account was not found on a server then send it again.

Finally reboot. To make sure that everything works. Fingers crossed, “reboot”. Came back online like always. No issues.

Why Debian?

Someone may ask why not Fedora or RHEL or CentOS? I work at Red Hat now, right?

Yes, I do. But Debian is operating system I know most. It’s tools etc. Also upgrade was possible to do online. Otherwise I would have to start with reinstalation.

Now I have only one machine running Ubuntu. My wife’s laptop. But it is “no way” zone. It works for her and we have an agreement that I do not touch it. Unless requested.

About year ago friend convinced me to buy “Commodore: The Amiga Years” book by Brian Bagnall. It described how Commodore company looked right after Jack Tramiel left. Buying Amiga company, release of Amiga 1000 and then A500 and A2000 times.

In April I backed another project by Brian Bagnall: “Commodore The Final Years” book. This one describes 1987 - 1994 period. From A500/2000 releases to company end.

There were many stories written/told about Commodore company. People saying that Ali Mehdi was main reason it collapsed and other theories. Brian Bagnall’s book gives better explanation than any story I have read before.

So why C= ended?

For me this is wrong question. I would rather ask “How C= managed to survive so long?”…

It looked like a mix of terrible management with good engineers. All those people working on whatever they want to design. And then moved between projects. With complete lack of deadlines (at least sensible ones).

All those crazy machines

For example Commodore 64D… It was an idea of adding 1581 disk drive into C64 case. Side effect? C64 users buying 1581 disk drives (because software released on 3.5” floppies) and C64D people buying 1541 to get access to older releases on 5.25” floppies.

Or whole work on Commodore 65… Wet dream of some of my retro friends. 8bit machine with superior capabilities (compared to C64/C128/C+4) but several years too late. Huge amount of work hours spent on designing 4502 CPU, VIC-III and other chips, prototype boards, operating system, new BASIC…

Then Amiga 300 (released as A600) which got any expansion possibilities removed. Only to not give a chance for GVP to earn money on extensions. I used that model. Was terrible but allowed me to have hard drive so I bought it instead of A500+.

All that work done on AAA chipset. Which was far beyond everything when they started but then quickly became not-so-magical when PC market got SVGA cards, PCI slots…

Company related stories

Book is not only how many computer models were on designers’ tables. There are many stories related to the company, people working there. How Commodore interacted with communities etc. How people worked in 80s/90s. Internet/UUCP use in those years. Hobbies of Commodore employees and lot more.

Impact on technology market

Some readers may remember CDTV model. It was Amiga 500(+) with CD-ROM drive all put in HiFi like case. But not so many knows about CDTV-CR one. It was cost reduced version based on A600. And much cheaper CD-ROM drive based on some cheap ‘diskman’ like one with electronics done from scratch. According to the book it’s creation lowered prices of CD-ROM drives for the whole industry.

Conclusion

For me this book (and previous one) are must read for any Amiga fan. Lot of interesting details about released (and not released) models, accessories, some notes about software. Many stories from Commodore employees and people cooperating with C= company through years.

My Amiga story

I was Amiga user in 1995 - 1999 years. First Amiga 600 with 425MB hard drive and 2MB ram. Then A1200 which I moved to PC tower case. At the end it had 68040 cpu at 40MHz, 64MB ram and 17GB hard drive (connected to FastATA controller). AmigaOS was nice, I learnt a lot but hardware became slow so I sold it in 1999 and moved to PC running Debian.

Mutt website says: “All mail clients suck. This one sucks less.” Then Mutt-NG adds “Mutt sucks less - but it still sucks!”. And this is true. Thunderbird definitely sucks. But I still use it.

I have three email accounts: private one (own server, Dovecot + Postfix) and two work ones on Gmail. And 240GB SSD for /home partition so from time to time I have to clean something. This time it was ~/.thunderbird/ taking about 48GB…

But 48 is still better than 100GB it took in past. Still far too much. So let’s check why it takes so much.

The biggest folder had 57900 files (1.8GB total). Thunderbird said it has 1801 messages, 48.9MB in total. Each message stored over THIRTY times. Maildir format, Gmail account.

Went to another folder. 823MB in 77345 files on disk. Thunderbird said 3107 messages, 30.7MB in total. Maildir format again, same Gmail account.

Ok, let’s check Dovecot one. Turns out that this one is mbox based. The biggest one was 2.1GB on disk, 526MB according to Thunderbird.

For each of them, I went with “repair folder” button which is “drop whatever is on disk and fetch again from server”.

And then went through folders and disk usage went to even 14GB. But now it started to fetch everything so time will show how it ends.

Bug reported. Do not care much will it get solved or ignored.

Alternatives?

Evolution was worse than Thunderbird when I tried to use it one-two years ago. Do not remember details now.

KMail is not usable due to lack of OAuth support for SMTP. There was code written for it but it is not available in Fedora yet.

I am tired of useless discussions. Tired of “we are talking about servers and desktop, not toys” which needs to happen in EVERY arm64 discussion sooner or later. It was that way in “Qt: GL or GLES on arm64” thread on debian-arm ML or recently on #debian-boot when I tried to find out how to get graphical installer working on arm64.

There was a mistake done at some point probably. Maybe aarch64 should start with A72 cores, GICv3 and multicore server chips. And mobile market get fast v7 cores at same time. To make a clean split.

On arm64 Fedora has graphical installer for last few releases. Took a while to debug X11 and kernel to find out why it requires config file when it should not. We wrote some patches (better than ones in linked post) and got them merged. I can take Mustang, put graphics card and install operating system using keyboard, mouse and monitor. Just like it is on boring computers.

Debian? Same machine, same config — you need to grab serial cable and second computer. Because it is Arm system so it is supposed to be one of those small toy boards people give to kids to play with, right?

Sure, I could sit and discuss with people but it does not work. You always get someone with ‘I use R/Pi zero as a desktop’ (or other insane setup) and then thread dies as every normal person leaves.

So sorry, but I do not plan to spent any time on improving operating system installers. Never mind which distribution it will be.