1. 2019 review: OpenStack

    Another year passed so let me try to summarize it. And a lot of things happened so it will be set of posts.


    We did two releases: ‘stein’ in July and ‘train’ in December. As usual a bit later than most of OpenStack projects as we have to wait for their releases, then for distributions to provide their binary packages, test everything etc.

    2019 was also a year of moving to Python 3 in our project. Images went in rounds — first some in Stein, then rest in Train cycle. We managed to remove Python 2 support from Kolla-Ansible but have to wait for TripleO project before we drop Py2 in Kolla itself.

    Also, we removed OracleLinux support. No one from active developers had access to it and jobs were failing on random moments. CentOS 7 support covered all needs in our opinion.

    CentOS 8 was hoped to be ready for cloud use. But it was not due to many changes needed in build infrastructure. Now, after 8.1.1911 release, there is a hope that situation will change and SIGs will be able to build their packages.

    On the other hand, we added support for IPv6 deployments. Linaro developers helped with testing as our second Linaro Developer Cloud will be IPv6 only and all resources will be provided for OpenStack infrastructure.

    There was new OpenStack project called Kayobe. It used both Kolla and Kolla-Ansible (and some other projects) to deploy containered OpenStack on bare metal. Including provisioning. We accepted it as subproject of Kolla as some of our members work in all those projects.


    During 2019 I got huge amount of help from OpenStack infrastructure team. We changed mirroring of CentOS, Debian, Ubuntu and several external repositories for any of those distributions. AArch64 was added where it was needed, changes got reviewed quickly. Perfect job!

    Continuous Integration

    After long fight with issues we finally have working AArch64 CI for OpenStack projects. Linaro provided some vcpu in Linaro Developer Cloud, infrastructure admins created some nodes from it and then created ‘check-arm64’ pipeline on Zuul.

    This way any OpenStack project can add tests on 64-bit Arm hardware without impacting their standard set of tests. We went that way due to low amount of nodes and it turned out to be very good move. Kolla(-ansible) itself is able to saturate all of them during developer heavy days.

    More nodes are coming. Managed by Linaro on sponsored hardware. Pure IPv6!

    Written by Marcin Juszkiewicz on
  2. Python and AArch64

    Python runs everywhere, right? All those libraries are just one ‘pip install’ away. And we are used to it. Unless on AArch64.

    On AArch64 when you do pip install SOMETHING you may end with “no compiler installed” or “No lapack/blas resources found.” messages. All due to lack of wheel files generated for this architecture… And even if you have all dependencies installed then building takes more time than it takes to install existing wheel file.

    But there is a light!

    PEP 599 defined “manylinux2014” target which is the first supporting something else than just “x86(-64)” architecture. The new arrivals are AArch64, PPC64 (Big and Little Endian) and 32-bit Arm.

    I helped getting them working on AArch64. Do not remember how I ended there to be honest ;D

    And those images are now available in Pypa repository on quay.io.

    And there is more!

    If your project uses Travis CI for testing and release then adding support for non-x86 architectures is just one edit away. All you need is entry in “arch” list of in “matrix”:

       - os: linux
         arch: amd64
       - os: linux
         arch: arm64

    And it really works!

    I added support for it in “dumb-init” so now kolla does not need to use any workarounds but can grab binary straight from project’s releases. Took just few simple lines in “.travis.yml” file.

    What is left to do now?

    There are many Python projects out there. Most of them do not know that manylinux2014 got released and what it brings. Or that Travis CI can give them non-x86 architecture support. I slowly started creating issues, bug reports in them to make sure they are aware.

    Written by Marcin Juszkiewicz on
  3. AMD again

    My i7 based desktop was about 8 years old and it was time to replace it. As there was no affordable aarch64 desktop class solution I have decided to go with AMD.

    Third generation of Ryzen processors brought several new features. Not that I care much of them. The important stuff is that they are faster than my Sandybridge i7. And that mainboards nowadays have m.2 slots so pcie-to-m.2 adapter cards are no longer needed.


    Looked at performance and price of new AMD cpus and chosen Ryzen 5 3600. Fast enough and still quite cheap compared to next in line. And it has more cores/threads than my i7.

    Stock cpu cooler is worthless junk as usual. Scythe Mugen 5 will replace it soon (ordered yesterday).


    Next was mainboard. First I thought about buying one with B450 chipset but then decided to bump budget and get x570 one. PCI Express 4.0 support, two m.2 slots, usb 3.2 ports etc. Should serve well for another few years.

    MSI X570-A Pro is what I went for. Nice black one, has all expansion slots I wanted. According to lspci there are 48 devices on my system (with graphics card inserted into PCIe slot and one NVME).

    mainboard partially assembled
    mainboard partially assembled


    Ram was next. This time new platform did not meant doubling amount of memory. Went with 32GB as two 16GB sticks so there is space for expansion if there will be any need.

    As with previous machines went with Kingston memory sticks. 3200MHz CL16 was proper balance between price and performance. Memtest86 told that they are fine (while looking funny on 34” panel).

    memtest86 on 34" LCD
    memtest86 on 34” LCD

    Old stuff

    Rest of my setup is from previous system. Case, nvme, ssd, Geforce 1050 Ti graphics card. My desktop always reused something from previous one. I could replace SATA cables with black ones to keep one colour scheme but this way I know which cable is which device.

    my underdesk desktop
    my underdesk desktop

    There were some no long needed cards. Soundblaster Live, usb 3 controller or pcie-to-m.2 adapter card went into storage box. Who knows, maybe one day will land in some other system.


    As with any new hardware there are some issues. Stock cpu cooler is crap as always. Getting cpu busy means lot of noise. Will solve that with Scythe Mugen 5.

    Other problem is usb related. For some reason one of my hubs (Anker 4 port) does not detect hot plugging devices. Cold plugged ones (present at system boot) work fine. Something to look closer one day.

    Why again?

    My first PC machine had AMD Duron cpu. 600 MHz overclocked to 850. Then Athlon XP. Athlon 64. Athlon 64 X2. The last one probably still has a use from time to time…

    Written by Marcin Juszkiewicz on
  4. What is wrong with all those AArch64 desktops?

    I have decided to finally upgrade my eight years old x86-64 desktop. Will stay with same architecture, just change CPU vendor from Intel to AMD.

    This move was delayed few times with “who knows, maybe finally there will be some affordable AArch64 desktop systems”. But time to say the ugly truth:

    There have never been, nor will be AArch64 desktop systems

    No one makes them because no one buys them. No one buys because there is no hardware to buy.

    Or we got to the point where most of people just need a phone? And developers are expected to use remote machines for everything?

    What is desktop system?

    For me desktop system is a computer in a box somewhere under desk. With some features:

    • fast storage (like m.2 NVME),
    • option for more than 16GB of RAM,
    • onboard audio able to handle 5.1/7.1 speaker set,
    • several USB ports,
    • gigabit Ethernet,
    • some PCI Express slots for potential upgrades.

    For example my eight years old x86-64 desktop has 256GB NVME in m.2 adapter (occupying PCIe x4 slot), 32GB of memory, additional USB 3.0 controller in the other PCI Express slot (as onboard Etron EJ168 one has issues with some devices). And SoundBlaster Live audio card in PCI slot. And some 1GbE port onboard (BCM57781 this time). And some SSDs attached.

    But those Avantek systems???

    Someone can say that there are AArch64 desktop systems on market. Avantek sells them. But are those machines desktop class systems?

    NO. Those are server mainboards in tower cases. And you basically have no idea what exactly you are going to buy as there is only graphics of case. No internal photos, no back photo, no user manual to download.

    Avantek 32 core Cavium ThunderX ARM Desktop

    No onboard audio. No mention about USB ports, two PCI Express slots (x16 + x8), no 1GbE network port. No mention about m.2 slot for NVME storage.

    Ampere eMAG 64bit Arm Workstation

    Specification may change without notice.
    Product may differ from images shown.

    In other words: “we have no idea what we sell”. Features listed put system a bit closer to desktop class as there are USB ports (just two), there is 1GbE network port. Just one PCI Express port (x8).

    Avantek ThunderX2 ARM Workstation - ThunderXStation

    4U rack/tower case sounds clearly like “we know it is server but will try to sell it as a desktop”. No m.2, no audio, no mention about USB ports, no 1GbE network.

    Other vendors

    So maybe there are other vendors? Depends on how much you are on “has to be SBBR compliant” side.

    Windows on Snapdragon

    Yay! Mobile chips based laptops! Sure, they have their niche. I know people who use them as daily laptops. For normal use and for development.

    But my work requires 16GB+ of RAM and working virtualization.

    Linaro Synquacker

    Or Synquacer or Linaro DeveloperBox or whatever you call that box filled with Cortex-A53 cpu cores. Where you have to choose “working graphics or working virtualization”.


    There are people using it as a desktop. 16GB ram is the limit. One PCIe slot. Lot of network interfaces would make it nice router probably.

    HoneyComb LX2K

    Looks interesting. Will see how it ends once get on market (preliminary version on sale now). Still requires USB audio dongle.

    Friend suggested that as CPU is on SoM card then other carrierboard can be designed. Sure, but not gonna happen probably.

    I have seen it already

    About four years ago I had that funny project: APM Mustang as a desktop (day 1, day 2, day 3). That system was kind of Frankenstein’s monster. Audio over some USB stereo-only dongle, all USB devices plugged into hubs due to only two ports on mainboard I/O panel.

    Looks like nothing changed…

    Written by Marcin Juszkiewicz on
  5. How to survive FOSDEM

    FOSDEM. Free Open Source Developers European Meeting. Every first weekend of February. In Brussels. At Université libre de Bruxelles.

    I used to say that FOSDEM is a week long conference in two days. I don’t think it is valid any longer. Should say two weeks now as last time it was something about 700 talks. Still during two days.

    I also used to say that I can not afford not being at FOSDEM. It is probably the easiest way to meet developers from all those projects I worked with or was interested in.

    But it is not just Saturday and Sunday. On Friday evening there is a beer event which nowadays takes place at street leading to Delirium Café. Can be hard to get there due to number of people but it is worth it. Easy way to meet far too many familiar faces. And get beer (once you managed to reach bar).

    But how to survive FOSDEM? Food. Logistics. Clothes. Logistics. Mobile app.


    Yes, logistics is one of key factors when it comes to this conference.

    Lectures take place in several buildings (AW, H, J, K, U). At same time. Getting from one to another takes time. You can meet people on a way and decide to discuss something rather than go for talk.

    Since few years ago there is even online navigation tool to show you which way to go from one place to another. Very handy especially if you have some mobility issues.

    FOSDEM map
    FOSDEM map

    Sometimes it is worth to be in some room session before just to have a seat. Limits are taken seriously and when info on door says “room full” then do not even try to sneak in. You may find some spot and watch live streaming — just remember about good headphones to not interrupt people around you.

    If you need a quiet place to sit for a bit with your laptop then you can go to cafeteria in F building (crowded, noisy) or sit in hacking room (quiet) in the middle of H building.


    Get place to sleep in city centre. “Crawling distance from Delirium Café” is perfect as this is where lot of people spend Friday’s and Saturday’s evening/night. Also several places to get various food, some grocery stores (just check opening times), pharmacies, GSM shops etc.

    Public transport

    Use whatever you normally use (Google Maps or so) to check for it. If possible avoid bus 71 if you go on popular time and not from the first stop. It gets full right on start and keeps that way until FOSDEM stop.

    IMPORTANT NOTE: Up to 2023 edition there were free buses from ULB to Brussels South (Midi) railway station on Sunday afternoon (15:30 - 19:30). It gave an easy way to get to BRU airport. They will be not present in 2024 ;(

    FOSDEM transportation page lists several options on how to get to/from ULB.

    One of options is getting to Etterbeek railway station (about 1.5 km walk) and then travelling wherever you need. This is my favourite way to go from FOSDEM to BRU airport on Sunday evening.


    When it comes to tickets…

    First of all check STIB ticket information for options as they change from time to time.

    Nowadays (2023) it looks like using bank card is easiest way. Just enter bus/metro/tram and tap to pay.

    Note that there are some single use tickets still in use if you prefer not to use bank card.

    Mobile application

    As long as your phone is able to run applications check Mobile schedule apps list at conference website and choose one. I use ‘FOSDEM Companion’ on my Android phone.

    Install application at least week before the event. Sync schedule and start checking which sessions to attend. Ignore fact that some will overlap. It is always better to have more listed than less — sometimes you sacrifice one due to being in wrong building or room will be full so you can take a quick look at list and go somewhere else.

    Internet access

    Once you reach FOSDEM there is no problem — conference area provides you fast wireless network access.

    If you are from one of EU countries then your GSM operator probably provides you with some data limit for “Roaming Like At Home” functionality.

    If you are from outside of EU then I suggest reading prepaid with data wiki page. GSM operators stores are in several places in city centre. Or you may even manage to find some at airport but easier to just grab one once you reach centre.


    Eat proper breakfast. Not “just a fruit and coffee” but some proper one. You may not have a time to stand in a queue for some FOSDEM food. It is not worth your time.

    Better option is to just have something with you. Get fruits, nuts, energy bars, chocolate. Some kind of quick calories.

    Or spend half an hour (or more) waiting for fast food. Still good opportunity to chat with people.


    Remember that Belgian beer can be stronger than your normal one. Do not drink too much or you will suffer and be too wasted during conference.


    FOSDEM is in February. Which means winter. So it may mean snow but it is Brussels so usually it is either nice or rainy. And you will go into building and out of building several times per day. Be prepared for it.

    Proper shoes to walk on concrete, steps but also snow, mud and grass. Winter cap, scarf, maybe even gloves. Do not have to be thick ones like for some Arctic winter. Assume few degrees above 0°C.

    Small backpack and bottle of water. Do not throw away once get empty. You are at university. You can refill your bottle at several places. Tap water is drinkable.


    Do not put your trash in random places. Use trashcans, inform organizers and volunteers if you see full ones. Put bottles into places for bottles.

    If you can then stay after ending session and help to clean up. It is free event so help those who organized it for you.


    FOSDEM is a free event. But there is swag. You can go to organizers’ desk and donate money to keep the event going. For 25€ you can get the official t-shirt.

    Several projects have their own stands — mostly in K building but there are also other ones in AW (mostly related to embedded space).

    If you are stickers fan (or your new laptop still have some space on case) then go on a tour, discuss with developers, see what they present. Grab leaflets, stickers, donate some money (usually in exchange to more premium swag) etc.

    Or some funny one like this license for VLC I got in 2019:

    Physical VLC license
    Physical VLC license

    Fiancé(e) or family

    If your companion is not interested in FOSDEM you may sign them to free tour(s). My wife came with me to conference once, went for tour each day and enjoyed them.

    Other events

    There are several meetings taking place right before and/or after FOSDEM.

    CentOS Project has event before FOSDEM. It is a nice opportunity to meet developers working on this distribution.

    After FOSDEM you can attend OpenEmbedded workshop of Buildroot developers meeting.

    Several other projects are having own meetings, developer days etc. You can check list at FOSDEM Fringe page.

    Written by Marcin Juszkiewicz on
  6. My first 8K intro

    I am on demoscene since 1997 when I attended “Intel Outside 4” party in Włocławek, Poland. But I have never released anything. Until Xenium 2019 party where I presented my first 8K intro. Written for Atari 2600 game console. In some kind of BASIC language…

    The idea

    The idea for it came about year ago during Riverwash demoscene party. Most of PC 64KB intros started with some kind of progress bar as code was generating textures, instruments and other stuff requiring calculations. I joked that it would be great to make something similar for Atari 2600 VCS.

    The fun is that Atari 2600 lacks any usable memory. It has 128 bytes (bytes, not kilobytes) of RAM. The only storage is cartridge with 4KB of ROM space (expandable by bankswitching to 32KB). So there is no point in any generating at other phase than development.

    batari Basic

    During July I took a look at available options and found ‘batari Basic’. It is BASIC-like language developed in 2005-2007 by Fred X. Quimby.

    My main source of help was a forum on Atari Age. Detailed language info came from useful page called ‘Random Terrain’. There is also “Visual BB which is IDE with some tools speeding up development.

    At party

    At start of “oldchool intro” competition I told to friend sitting next to me that I hope for at least four entries. He asked “why?” so I pointed him to the screen saying “because of it”:

    My first 8K intro announcement screen
    My first 8K intro announcement screen

    There were six entries so I though that I am safe and will have 5th entry maximum.


    Thanks to Xenium organizers I have my intro recorded as video:

    As you can see it has two parts. First one is playfield with one line changing every 30 frames (first version was for NTSC). Result is simple progress bar.

    Second part uses “titlescreen kernel” to display dino graphics. I used one provided by Xenium organizers (rescaled to 96x91 pixels). I used rainbow colours to have some colours on screen.


    Imagine my surprise during voting results announcement next day. My intro took 3rd place!

    Award for 3rd place
    Award for 3rd place

    Amazing for production without any effects, without any audio. And written in BASIC-like language without using any knowledge of 6502 assembly.

    Source code

    If someone wants to see how simple it was then source code is in “my first 8k” git repo on github. Enjoy!

    What next?

    During Xenium I got lot of technical information for VCS programming. And there were questions about my next production.

    Nothing to promise here. And not for Atari VCS. It is sick platform to program due to lack of any framebuffer memory so ‘race the beam’ is in use…

    I have some ideas for Atari XL/XE

    Written by Marcin Juszkiewicz on
  7. Kolla ‘stein’ released

    On last Monday we finally released Kolla and Kolla Ansible 8.0.0 ‘stein’. Took us longer than we planned but now it is done and ready for users.


    What got changed? Many things — details can be found in Kolla release notes and Kolla Ansible release notes.

    My work in this cycle was more reviews, less code. And lot of planning how to handle Python 3 migration for Debian/Ubuntu based images. At some point we decided that this stuff will be postponed to ‘train’ cycle. You can read more about it my previous post: Moving Kolla images to Python 3.

    Why so late

    Usually we release Kolla ‘two weeks’ after official OpenStack release. This allows us to switch to final release code of other projects, do some testing etc. This time it took us far more time :(

    Due to several issues (some core developers got more occupied with work, distributions changed dependencies in packages) it took us longer than ‘two weeks’ after official OpenStack ‘stein’ release. We added more tests for CI, handled partial Python 3 migration in Ubuntu Cloud Archive and more.Several fixes were made in ‘train’ and backported to ‘stein’.

    Now it is your turn — build, deploy, test, report ;D

    Written by Marcin Juszkiewicz on
  8. Upgraded my desktop a bit

    I have built my current desktop 7.5 years ago. Since then I did not had a need for a big hardware changes.


    Machine (called ‘puchatek‘ (Winnie the Pooh in Polish)) had several upgrades in meantime:

    • got maxxed at 32GB of memory
    • 60GB Corsair SSD for / was installed 8 years ago
    • 250GB Samsung Evo for /home was added 3 years ago
    • graphic cards were changed from Radeon HD5450 via Radeon R7 240 to Nvidia GTX 1050 Ti


    Size of system drive became an issue when I needed to build hundreds of container images. All that Kolla stuff…

    One of solutions was replacing system drive with bigger one. So I tried to use PCI Express to m.2 adapter card and realised that x8 slot stopped working.

    New mainboard

    It was a time to replace motherboard. And it is impossible to find a brand new one with 1155 socket. So I went through used ones and found nice replacement — Asrock Z68 Extreme4.

    What’s nice in it? PLX chip. It is PCI Express switch. So mainboard can have x16/x8+x8 slots, x4 slot, some x1 slots and several onboard components despite of only 24 pcie lanes available (16 from cpu, 8 from chipset).

    This way I can have graphics card working in x16 slot (it goes with x8 anyway) and NVME drive in x4 slot. If I decide to go into SLI (two graphics cards) or 10GbE I have a slot for it.

    PCI tree

    PCI tree looks a bit different now:

    • cpu

      • x16 slot with graphics card
      • x8 slot (empty)
    • chipset

      • x4 slot with nvme
      • x1 link to Marvell SATA controller (disabled in firmware setup)
      • x1 link to Etron USB 3.0 host controller
      • x1 link to Etron USB 3.0 host controller
      • x1 link to PLX switch
    • PLX switch

      • x1 slot with Renesas USB 3.0 host controller
      • x1 link to FireWire controller (disabled in firmware setup)
      • x1 link to Broadcom 1GbE controller
      • x1 slot (empty, covered by graphics card)
      • x1 link to PCIe to pci bridge

    Hacking firmware

    Latest available firmware was from 2012 and lacked any support for NVME boot. Thanks to other hackers it was not an issue. Only had to follow “how to add NVME booting into BIOS” instruction. After flashing modified firmware I could boot directly from NVME drive.

    Final result

    System boots from NVME now. 256GB of fast storage available for / and my container images. Spare PCIe x8 slot for future upgrades.

    Written by Marcin Juszkiewicz on
Page 10 / 105