Marcin Juszkiewicz

I have built my current desktop 7.5 years ago. Since then I did not had a need for a big hardware changes.

Upgrades

Machine (called ‘puchatek‘ (Winnie the Pooh in Polish)) had several upgrades in meantime:

• got maxxed at 32GB of memory
• 60GB Corsair SSD for / was installed 8 years ago
• 250GB Samsung Evo for /home was added 3 years ago
• graphic cards were changed from Radeon HD5450 via Radeon R7 240 to Nvidia GTX 1050 Ti

Issues

Size of system drive became an issue when I needed to build hundreds of container images. All that Kolla stuff…

One of solutions was replacing system drive with bigger one. So I tried to use PCI Express to m.2 adapter card and realised that x8 slot stopped working.

New mainboard

It was a time to replace motherboard. And it is impossible to find a brand new one with 1155 socket. So I went through used ones and found nice replacement — Asrock Z68 Extreme4.

What’s nice in it? PLX chip. It is PCI Express switch. So mainboard can have x16/x8+x8 slots, x4 slot, some x1 slots and several onboard components despite of only 24 pcie lanes available (16 from cpu, 8 from chipset).

This way I can have graphics card working in x16 slot (it goes with x8 anyway) and NVME drive in x4 slot. If I decide to go into SLI (two graphics cards) or 10GbE I have a slot for it.

PCI tree

PCI tree looks a bit different now:

• cpu

  • x16 slot with graphics card
  • x8 slot (empty)

• chipset

  • x4 slot with nvme
  • x1 link to Marvell SATA controller (disabled in firmware setup)
  • x1 link to Etron USB 3.0 host controller
  • x1 link to Etron USB 3.0 host controller
  • x1 link to PLX switch

• PLX switch

  • x1 slot with Renesas USB 3.0 host controller
  • x1 link to FireWire controller (disabled in firmware setup)
  • x1 link to Broadcom 1GbE controller
  • x1 slot (empty, covered by graphics card)

  • x1 link to PCIe to pci bridge

    • MOS 9845 Multi I/O controller with 6 serial ports
    • PCI slot (empty)

Hacking firmware

Latest available firmware was from 2012 and lacked any support for NVME boot. Thanks to other hackers it was not an issue. Only had to follow “how to add NVME booting into BIOS” instruction. After flashing modified firmware I could boot directly from NVME drive.

Final result

System boots from NVME now. 256GB of fast storage available for / and my container images. Spare PCIe x8 slot for future upgrades.

USB. Protocol which replaced random keyboard connectors, PS/2, ADB, gameport, serial and parallel ports (and many more). Sometimes expanded to “USB Sucks Badly“.

Five years ago buying USB 3.0 hub was a task as it was not so popular thing. Nowadays I have three SuperSpeed ones. One in LG monitor (webcam, phone, Bluetooth dongle), one on desk (watch charging/adb, Yubikey, card reader, pendrives etc) and that 7 port one from five years ago with random dongles in it.

Yesterday I replaced Gigabyte P67X-UD3-B3 mainboard with AsRock Z68 Extreme4 one. This gave me extra PCIe x4 slot where I can plug NVME storage. Built system, booted into Fedora and started using.

At some moment I had to login into one of systems with two factor authorization. I use Yubikey for it. Pressed the button and nothing was outputed…

Then I realized why previous configuration had that extra USB 3 controller:

usb 3-2.2: new full-speed USB device number 12 using xhci_hcd
usb 3-2.2: New USB device found, idVendor=1050, idProduct=0110, bcdDevice= 3.33
usb 3-2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 3-2.2: Product: Yubikey NEO OTP
usb 3-2.2: Manufacturer: Yubico
usb 3-2.2: Not enough bandwidth for new device state.
usb 3-2.2: can't set config #1, error -28

Yay, USB!

Plugged Renesas uPD720201 PCIe USB 3.0 host controller, moved all hubs to it and it works just fine:

usb 7-3.2: new full-speed USB device number 11 using xhci_hcd
usb 7-3.2: New USB device found, idVendor=1050, idProduct=0110, bcdDevice= 3.33
usb 7-3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 7-3.2: Product: Yubikey NEO OTP
usb 7-3.2: Manufacturer: Yubico
input: Yubico Yubikey NEO OTP as /devices/pci0000:00/0000:00:1c.7/0000:05:00.0/0000:06:01.0/0000:07:00.0/usb7/7-3/7-3.2/7-3.2:1.0/0003:1050:0110.0008/input/input30
hid-generic 0003:1050:0110.0008: input,hidraw6: USB HID v1.10 Keyboard [Yubico Yubikey NEO OTP] on usb-0000:07:00.0-3.2/input0

Then checked which USB 3.0 host controller is on mainboard:

11:29 (0s) hrw@puchatek:~$ lspci |grep -i usb
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
07:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)

Argh, Etron EJ168 again…

So if your mainboard has Etron EJ168 then consider adding Renesas card. Works just fine. Nevermind how slow/fast your USB devices are.

Nine years ago at 11:00 a group of developers gathered in a small room. I was one of them and did not knew anyone from a group before entering the room.

The meeting took place in Dolce La Hulpe Hotel and Resort in a village close to Brussels, Belgium. It was on first day of UDS-M.

This was the first meeting of NewCo developers. The organization now known as Linaro.

I do not remember who exactly was at that meeting so will not provide a list. We introduced ourselves, got some knowledge of who is who’s boss and what we will do from now. I just told “I’ll do what my boss (pointing to Steve Langasek) orders me to” as I was tired after whole night train trip.

For me this was the real beginning of Linaro. Not June 2010 when (at Computex) it was announced to the world. Neither 26th April 2010 — a day when I started working for Canonical as NewCo engineer.

I had some kind of personal website since started using Internet in 1996. First it was set of hand edited Lynx bookmarks, then were experiments with wikis. Finally in 2005 I started using WordPress. And it was in use for those 14 years. Until now…

WordPress is nice platform but I got tired of it. More and more plugins and themes became demo versions of commercial products. Also amount of JavaScript and CSS added to website made it harder and harder to maintain. At some point I said myself that it is enough. And started looking for alternatives.

Pelican

Here came Pelican — static site generator written in Python. I had few attempts to switch to it and finally managed to find some time and sorted out all issues.

Someone my ask why Pelican? Why not Jekkyl, Gatsby, Hexo, Wintersmith or other. For me reason is simple — it is Python. Language which I already know. So in case of need I can read source code and know how to change it (already sent one change and it got merged).

Conversion

The good side is import from WordPress went nice. As I used Markdown most of posts required changes. Implementation in Pelican differs from old Markdown Extra + SmartyPants I had in my blog.

Images

Then came images. Copied whatever I had on previous website and removed all thumbnails. Then decided to go with 700px wide ones and to not link to original photos. Boring work as almost every image in every post needed change. Some entries got pictures removed (most of time due to their low resolution).

This also shown how my blog was changing through all those years. Over 10 years ago adding 300x300px picture into blog post was normal thing. Now such graphics got either removed or replaced with 700px wide one.

Some posts had galleries inserted instead of pictures. This took a bit more time as I was grabbing filenames from database to replace gallery with set of photos. And removed some of them during.

Look and feel

When I was collecting ideas for a new platform I had few ideas:

• static generator
• no JavaScript
• minimal CSS
• similar look to WordPress version

Pelican solved first point. Handling rest was harder.

I took a look at existing Pelican themes and tried several ones. Finally decided to make own one — like WordPress “Spacious” one.

As a base I used “Simple” theme. Typical template with header, content, sidebar and footer. Elements put in CSS grid for most of screens and once screen width goes under 70em layout switches to “flex”. This allowed for simple responsive web design. All in ~2.5KB of CSS (plus some code for webfonts).

Archives

One of big changes (compared to WordPress) is a way of presenting archive posts. You can go into archives to see the list of all my blog posts like it was before. But if you go for a list of posts in a tag (like AArch64, Red Hat, Zaurus) then instead of posts with pagination you get list of posts presented in archive form.

This should make old content easier to find.

Comments

As you may notice there is no way to comment posts anymore. Amount of comments was lower every year so I decided to not bother with them in new website. I could add Disqus for example but is it worth for just a few entries per year?

To Do

There are some things I need to take care of still. Page about my fridge magnets collection is missing, some entries may get some formatting changes or small contents edit. No big edits of old posts as they show how awful my English was in past (not that it got any better).

For several Linaro Connect events we had sessions about state of graphics drivers on ARM platforms. I attended most of them and got a reputation of person asking problematic questions.

But situation has changed. With Panfrost project happening. It is a Foss driver for Arm Mali Midgard graphics chipset (Bifrost support on a way). It went from “wow, a triangle” to “we can play some games or run a desktop” in quite short time.

At BKK19 Linaro Connect we had “State of opensource drivers for mobile GPU chips” session. Freedreno, Etnaviv, vc4, v3d, Panfrost, Lima etc. What they target, what was already achieved, what are plans. Great progress across whole ARM world. And several questions from the audience. And interesting answers as well.

Mali then. Grant Likely from Arm told that they are looking how Panfrost is going. From company perspective both Midgard and Bifrost chips are “done, in a field” product which will not get changes. Still — engineering support goes entirely into their binary drivers as this is what their customers are using. Situation may change if those customers start asking for open drivers.

I do not use any Arm hardware with Mali GPU anymore. But hope that at next Linaro Connect instead of asking famous “Mali question” we will rather discuss how it runs on our devices.

One of Linaro Connect traditions is a puzzle to solve. Created by Dave Pigott. And recent BKK19 event was not any different. There was puzzle announcement on the first day — right before first keynote. But no one could be first to answer at that time…

Tuesday

As usual before Connect I looked at a map and marked several locations in Bangkok as places to visit. Then took a look at the official BKK19 application. Installed it on my phone and started.

First screen had few paragraphs of text. Some information about the event and schedule. But there was also paragraph with information about the puzzle. WITH the link to it!

I clicked to get some redirection to Google Forms website. With information that this form is not available for users outside of organization. As I do not have work accounts on my phone I checked for redirection link and loaded it on my desktop. And landed into the puzzle.

Puzzle form

It was a bit different than version provided during Linaro Connect. There was a graphics with seven (official one had eight ones) columns of text. Under it was graphics with chess figures:

H1 G3 F1 H2 G4 E3 D1
A4 C3 B1 A3 C4 B6 A8
B5 A7 C8 D6 E8 F6 D5
G8 H6 F5 G7 H5 F4 E2
C6 A5 B7 C5 A6 B8 D7
F3 G1 H3 G5 H7 F8 E6
F7 H8 G6 H4 G2 E1 D3
A2 B4 C2 A1 B3 D2 E4

Dave later said that knight was not present in it but I did not noticed that. There was a plan to add that graphics into official puzzle if no one provides proper answer until Wednesday.

From text I noticed that it is chess related. Took a sheet of paper, draw 8x8 grid on it and started following each row on it with different markings each time. Hm… nothing came to my mind. Noted missing entries.

Help me Google, you are my only hope

Then started googling “knight chess puzzle” and got “knight’s tour” links on first page. Started reading what it is about. Then restarted tracking knight’s moves from the puzzle with adding missing ones. Turned out that this was it.

Let me mail Dave

I submitted “knight’s tour problem” as the answer and wrote to Dave:

I see that they are online already.

Are the ones there official ones or testing one?

Turned out that I found testing version. Which did not even collected emails when someone provided an answer. But there was one such sent so Dave marked it as my submission. And the link got removed from BKK19 application.

At pool bar

I arrived in Bangkok on Saturday. Met Dave at pool bar and we had a chat about the puzzle. Was fun to see how surprised people around were that first answer was already provided. I asked Dave to not give me the proper answer nor info was my answer good so I would not spoil other people.

During Connect few attendees asked me about the puzzle, how it went. Kept away from spoiling them.

And the winner is…

Then Friday happened with closing remarks session. Only a few people provided two words answer (“knight’s tour”) and few three word one (“closed knight’s tour”). My name was in “special mention” section.

Turned out that there is separate award for hacking a puzzle. It was 3rd time when it happened. I got a waterproof action camera (EZVIV S1C model) — will find some use for it sooner or later ;D

Whole puzzle was fun. Thanks go to Dave for creating it and providing me with a copy of both results slide and graphical hint so I could use it in blog post.

Python… To use 2.7 or to go for 3.x? To “be compatible” or to “use fancy new features”. Next year Python 2 gets finally unsupported upstream.

Kolla state

Kolla can run under Python 2.7 or 3.x and results will be the same. All those container images containing OpenStack components and whatever needed to get them running. But so far all of those images will use Python 2.7 inside…

I proposed to take care of it during Train cycle (we are at Stein now). Then we had a meeting and decided — let us do that in Stein! If I only knew how many issues it will bring…

Cleaning/refactoring

First some cleaning and refactoring:

• debian/ubuntu: drop mysql compat stuff, unify on openssl 1.1
• openstack-base: drop pika-pool
• Introduce ‘get_pip’ macro
• helm-repository: drop unused os_client_config variable

Distribution handling

Then created new variable to keep which exactly version of Python 3: distro_python_version. And then used wherever needed.

In meantime we got support for Red Hat Enterprise Linux 8 distribution. Where Python 3 is the only version available.

Finally we moved Ubuntu to use Stein UCA which also switched us to Python 3 in several images.

CI failures

And then hell gates opened…

We moved our CI jobs from Ubuntu 16.04 ‘xenial’ to Ubuntu 18.04 ‘bionic’. OMG. So many failures. When Mark Goddard found out why (too old ‘setuptools’) we had to pile up fixes to get CI back into working state.

Nova images were failing. Turned out that Debian/Ubuntu ‘qemu’ package is no longer metapackage but useless dummy. And then it brought ‘armv6l’ architecture to Nova so it started failing. There were three different patches to handle it and problem got solved.

OracleLinux image got repository information files renamed. And as we edit them we had to adapt.

Python 3 got enabled in another set of images.

Changing other projects

Kolla uses other projects, right? Are they Python 3 ready?

Had to patch several ones:

• bifrost (and test their patch)
• ironic
• karbor
• kuryr-libnetwork
• monasca (api and persister)
• vmtp

Some of those changes were requirements updates to mark “Python 2 only” components (enum34, functools32, UcsSdk) or getting rid of Unicode characters from files which should be be US-ASCII.

Bifrost one cleaned situation as we are using it within virtualenv so Python packages were going crazy. Now they handle it ;D

Current status

The most important patch switching all image to Python 3 for Debian/Ubuntu is still in review. Waiting for karbor fix getting merged and CentOS 7/OracleLinux 7 builds getting working again.

We also moved to Debian ‘buster’ release. It is now in ‘freeze’ state so no big changes allowed and provides us with many updates making Debian/Ubuntu blocks easy.

Python 2 stays :(

There are few images where we still have Python 2. Anything related to Ceph has it because Debian/Ubuntu ‘ceph-common’ package depends on Py2 packages. We install Py3 ones there.

There are several posts and videos around recent discussion in “ARM announces Ares” thread on Real World Technologies website. People quote Linus Torvalds like crazy…

SBBR

Yes, we need AArch64 (real Arm64 architecture name) systems for developers. SBBR compliant ones so their users will not have to deal with ‘how the hell should I boot it’. Instead get Linux distribution and install it like on current developer boxes (PXE, USB, CD etc).

SBBR: System Boot Base Requirements — specification which defines firmware of AArch64 server system. Which mean UEFI, ACPI on hardware compliant with SBSA (Server Base System Architecture) specification. In short: boring box which just works with any serious Linux/BSD distribution and for any random user looks and works like x86-64 box when it comes to booting.

Hardware formats

Some people would like small format — like Intel NUC, Apple Mini. Just open a box, put m.2 storage, few sticks of memory and go.

Other ones would like (micro)ATX format. You put it into standard case, add m.2 and sata storage, few sticks of memory, PCI Express graphics card and use.

So far there are no such systems in affordable prices. Or you have to deal with some issues.

Software issues

Anyway, you got hardware. Or access to. Or even bloody VM somewhere. So you start development. And often hit a wall…

Nowadays development culture ‘loves’ to use random binaries from the Internet. You do not wait until someone package software you would like to use, you grab compiled binaries from vendor/authors. Their website or github/gitlab release page.

And nope. x86-64 binaries only. So you dig for some aarch64 builds or look how to build it on your own. And you do not work on your project but on someone’s else one. Or you change your one to not use that external component.

Language extensions

Or your project is in Python/Ruby or other language with extensions compiled to native code. And simple “pip install scipy” which took 4s on your x86-64 machine now just fails…

So you install compilers, Python headers, check which libraries you should have installed, which versions of them, do your distro have them etc. Finally “pip install scipy” works. After several hours from your development time.

Conclusion(?)

Getting AArch64 hardware for developers is important. When it happen? One day. Maybe even before people forget that such architecture existed.

We talk about it during each Linaro Connect. So far nothing serious came from it. We had some failed attempts like Cello or Husky. There is Synquacer with own set of issues. Some people use MACCHIATObin. Some still use Applied Micro Mustangs which should get a place in computer museums.

It is chicken and egg issue. No one makes affordable AArch64 systems because no one buys them. Because no one makes them. Hardware vendors concentrate on server market — no chips to choose for developer systems.