1. The most expensive chip in the ARM world

    Arm world is weird. You can have server (expensive, UEFI, out of the box supported by any mainline distribution, normal SATA/NVME storage) or SBC (cheap, usually U-Boot, not supported for months by any distribution, microsd/USB storage mostly). But I do not want to talk about servers today.

    Bootloader(s) mess

    So SBC… Which means playing with U-Boot. In the past it meant random fork of it. I have a feeling (can be wrong) that it changed. Board starts with mainline one or quite fresh fork with mainlining in progress. Similar with ATF (Arm Trusted Firmware - level0 bootloader).

    So you have SBC, you have bootloader(s) built. The next step is taking microsd card, finding out which offsets to use. Then put bootloader(s) at proper places so CPU can read them and boot. Usually it also means that you have to use MBR style formatting as bootloader is where GPT would be.

    Better way maybe?

    Is there a better way? Sure it is. But it is very expensive. Or at least I have a feeling that it is for most of SBC vendors.

    Very expensive solution

    The solution is SPI flash chip. But it is very expensive - 3 (three) EUR for 512Mb (64MB) chip (if bought 1000 of them). Far too much for 99.99% of SBC probably.

    Price taken from simple search on Digikey: W25Q512JV 512 Mb Serial Flash Memory.

    64 megabytes is more than enough to store all. Bootloader(s), U-Boot/UEFI settings etc. You can even fit small OS image there.

    Probably 64/128Mb will be enough even. And they are cheaper (128Mb is 1.7 EUR). Still too expensive…

    Written by Marcin Juszkiewicz on
  2. 2019 review: FOSS projects

    Whatever you do, do it upstream” is IMHO one of mottos worth following. So I went upstream wherever possible.


    2020 came, Python 2 is no more. So 2019 was full of cleaning and patching. I filled several bugs on projects, created patches to mark some Python packages as Py2 only (or <Py3.5 only). Some projects just dropped support for Py2 packages due to it.

    pbr story

    With move to Python 3 came some issues with Unicode characters in README files of packages used by OpenStack components. Filled some bugs, created patches and then, after several emails, someone found out that it is a bug in ‘pbr’ and fixed it ;D


    Installing Python packages on AArch64 can be painful. While “pip install numpy” takes few seconds on x86-64 machine it can take an hour on Arm64 + extra time to find out which libraries and compilers need to be installed.

    So Python developers created PEP-599 to sort out situation and create ‘manylinux2014’ target which can be then used to build ‘wheels’ (aka binary Python packages) not only for x86(-64) but also for 32-bit Arm, 64-bit Arm, 64-bit Power (big and little endian) and Z/Arch (s390x).

    Somehow I managed to be involved in it. Went through pull request on GitHub, build checked, fixed dependencies and some other issues, filled CentOS bugs, discussed with CentOS developers on fixes and then one day all that stuff got merged.

    Now you can fetch “manylinux2014” images from “pypa” account on Quay.io container registry and build ‘wheel’ binaries for different architectures. If you lack target hardware then consider services like Travis CI which provide access (free for FOSS projects).


    Speaking of Travis CI… Near the end of the year I got task from one of Linaro teams to update CirrOS images with kernels from Ubuntu 18.04 or later.

    I fetched current source, built images and started wondering how to do it best. So started discussing with Scott Moser (main developer) and we agreed that such move have sense and that some changes are needed in this project.

    In December we moved it from Launchpad/cirros to GitHub/cirros-dev/cirros. Opened several issues to not forget what we planned and I started looking at some of them.

    Project is now using Travis CI service for tests. With all builds being run parallel we have checks done in less than and hour. Including boot test for aarch64, arm, i386 and x86_64 architectures (ppc/ppc64/ppc64le need work).

    Doing releases on GitHub is one of next tasks. Current service will redirect.

    Linaro ERP

    After ERP 18.12 release we decided that there will be no more releases and Debian ‘buster’ will be used as it is (ERP based on Debian ‘stretch’). So I spent some time on working with Debian kernel maintainers.

    Most important part? Merging kernel configuration changes. We had about twenty extra enabled to get our servers supported. Everything worth using got merged. In meantime we even found a bug in Linux-stable tree which took some time to fix.

    We were ready for Debian ‘buster’ release. And it worked out of the box on all machines we supported ;D

    Big Data

    Bleh. Java, protobuf 2.5 and other madness… But as build engineer I get such stuff from time to time.

    Last year it was mostly Apache Drill and Apache Arrow. In both cases I worked on their Debian packaging (not in-Debian one) to add/fix AArch64 support.

    Other projects still wait for ‘let us move to newer protobuf’ but it will take years…

    Written by Marcin Juszkiewicz on
  3. VSPA — Very Simple Planet Aggregator

    Quite long time ago I installed Planet Venus on my server to have RSS aggregator for my own use. More about in an old post about Developer planet.

    Python 2 strikes again

    But I forgot that it was written in Python 2… So once I removed Py2 from my server website updates stopped. No one noticed ;D

    Yesterday I looked at situation — there are no updated versions, looks like no one plans to work on Python 3 porting. One of options was move to other aggregator written in Ruby but I do not know that language so it was out of question.

    Let’s write own one

    So I decided to take a look at Python libraries. Feedparser looked useful and after few hours of coding I had something working.

    I called it Very Simple Planet Aggregator and put on my GitHub account.

    Requirements are simple: Feedparser (parsing feeds), Jinja2 (templates) and sgmllib3k (to get relative urls working in feeds). Small bit of documentation is provided in repository. No tests so far.


    I need to add RSS template and do some tests with Feedparser provided feeds to check did I catched all possible errors.

    And need to deploy it on server and move Developer Planet to use it.

    Written by Marcin Juszkiewicz on
  4. 2019 review: OpenStack

    Another year passed so let me try to summarize it. And a lot of things happened so it will be set of posts.


    We did two releases: ‘stein’ in July and ‘train’ in December. As usual a bit later than most of OpenStack projects as we have to wait for their releases, then for distributions to provide their binary packages, test everything etc.

    2019 was also a year of moving to Python 3 in our project. Images went in rounds — first some in Stein, then rest in Train cycle. We managed to remove Python 2 support from Kolla-Ansible but have to wait for TripleO project before we drop Py2 in Kolla itself.

    Also, we removed OracleLinux support. No one from active developers had access to it and jobs were failing on random moments. CentOS 7 support covered all needs in our opinion.

    CentOS 8 was hoped to be ready for cloud use. But it was not due to many changes needed in build infrastructure. Now, after 8.1.1911 release, there is a hope that situation will change and SIGs will be able to build their packages.

    On the other hand, we added support for IPv6 deployments. Linaro developers helped with testing as our second Linaro Developer Cloud will be IPv6 only and all resources will be provided for OpenStack infrastructure.

    There was new OpenStack project called Kayobe. It used both Kolla and Kolla-Ansible (and some other projects) to deploy containered OpenStack on bare metal. Including provisioning. We accepted it as subproject of Kolla as some of our members work in all those projects.


    During 2019 I got huge amount of help from OpenStack infrastructure team. We changed mirroring of CentOS, Debian, Ubuntu and several external repositories for any of those distributions. AArch64 was added where it was needed, changes got reviewed quickly. Perfect job!

    Continuous Integration

    After long fight with issues we finally have working AArch64 CI for OpenStack projects. Linaro provided some vcpu in Linaro Developer Cloud, infrastructure admins created some nodes from it and then created ‘check-arm64’ pipeline on Zuul.

    This way any OpenStack project can add tests on 64-bit Arm hardware without impacting their standard set of tests. We went that way due to low amount of nodes and it turned out to be very good move. Kolla(-ansible) itself is able to saturate all of them during developer heavy days.

    More nodes are coming. Managed by Linaro on sponsored hardware. Pure IPv6!

    Written by Marcin Juszkiewicz on
  5. Python and AArch64

    Python runs everywhere, right? All those libraries are just one ‘pip install’ away. And we are used to it. Unless on AArch64.

    On AArch64 when you do pip install SOMETHING you may end with “no compiler installed” or “No lapack/blas resources found.” messages. All due to lack of wheel files generated for this architecture… And even if you have all dependencies installed then building takes more time than it takes to install existing wheel file.

    But there is a light!

    PEP 599 defined “manylinux2014” target which is the first supporting something else than just “x86(-64)” architecture. The new arrivals are AArch64, PPC64 (Big and Little Endian) and 32-bit Arm.

    I helped getting them working on AArch64. Do not remember how I ended there to be honest ;D

    And those images are now available in Pypa repository on quay.io.

    And there is more!

    If your project uses Travis CI for testing and release then adding support for non-x86 architectures is just one edit away. All you need is entry in “arch” list of in “matrix”:

       - os: linux
         arch: amd64
       - os: linux
         arch: arm64

    And it really works!

    I added support for it in “dumb-init” so now kolla does not need to use any workarounds but can grab binary straight from project’s releases. Took just few simple lines in “.travis.yml” file.

    What is left to do now?

    There are many Python projects out there. Most of them do not know that manylinux2014 got released and what it brings. Or that Travis CI can give them non-x86 architecture support. I slowly started creating issues, bug reports in them to make sure they are aware.

    Written by Marcin Juszkiewicz on
  6. AMD again

    My i7 based desktop was about 8 years old and it was time to replace it. As there was no affordable aarch64 desktop class solution I have decided to go with AMD.

    Third generation of Ryzen processors brought several new features. Not that I care much of them. The important stuff is that they are faster than my Sandybridge i7. And that mainboards nowadays have m.2 slots so pcie-to-m.2 adapter cards are no longer needed.


    Looked at performance and price of new AMD cpus and chosen Ryzen 5 3600. Fast enough and still quite cheap compared to next in line. And it has more cores/threads than my i7.

    Stock cpu cooler is worthless junk as usual. Scythe Mugen 5 will replace it soon (ordered yesterday).


    Next was mainboard. First I thought about buying one with B450 chipset but then decided to bump budget and get x570 one. PCI Express 4.0 support, two m.2 slots, usb 3.2 ports etc. Should serve well for another few years.

    MSI X570-A Pro is what I went for. Nice black one, has all expansion slots I wanted. According to lspci there are 48 devices on my system (with graphics card inserted into PCIe slot and one NVME).

    mainboard partially assembled
    mainboard partially assembled


    Ram was next. This time new platform did not meant doubling amount of memory. Went with 32GB as two 16GB sticks so there is space for expansion if there will be any need.

    As with previous machines went with Kingston memory sticks. 3200MHz CL16 was proper balance between price and performance. Memtest86 told that they are fine (while looking funny on 34” panel).

    memtest86 on 34" LCD
    memtest86 on 34” LCD

    Old stuff

    Rest of my setup is from previous system. Case, nvme, ssd, Geforce 1050 Ti graphics card. My desktop always reused something from previous one. I could replace SATA cables with black ones to keep one colour scheme but this way I know which cable is which device.

    my underdesk desktop
    my underdesk desktop

    There were some no long needed cards. Soundblaster Live, usb 3 controller or pcie-to-m.2 adapter card went into storage box. Who knows, maybe one day will land in some other system.


    As with any new hardware there are some issues. Stock cpu cooler is crap as always. Getting cpu busy means lot of noise. Will solve that with Scythe Mugen 5.

    Other problem is usb related. For some reason one of my hubs (Anker 4 port) does not detect hot plugging devices. Cold plugged ones (present at system boot) work fine. Something to look closer one day.

    Why again?

    My first PC machine had AMD Duron cpu. 600 MHz overclocked to 850. Then Athlon XP. Athlon 64. Athlon 64 X2. The last one probably still has a use from time to time…

    Written by Marcin Juszkiewicz on
  7. What is wrong with all those AArch64 desktops?

    I have decided to finally upgrade my eight years old x86-64 desktop. Will stay with same architecture, just change CPU vendor from Intel to AMD.

    This move was delayed few times with “who knows, maybe finally there will be some affordable AArch64 desktop systems”. But time to say the ugly truth:

    There have never been, nor will be AArch64 desktop systems

    No one makes them because no one buys them. No one buys because there is no hardware to buy.

    Or we got to the point where most of people just need a phone? And developers are expected to use remote machines for everything?

    What is desktop system?

    For me desktop system is a computer in a box somewhere under desk. With some features:

    • fast storage (like m.2 NVME),
    • option for more than 16GB of RAM,
    • onboard audio able to handle 5.1/7.1 speaker set,
    • several USB ports,
    • gigabit Ethernet,
    • some PCI Express slots for potential upgrades.

    For example my eight years old x86-64 desktop has 256GB NVME in m.2 adapter (occupying PCIe x4 slot), 32GB of memory, additional USB 3.0 controller in the other PCI Express slot (as onboard Etron EJ168 one has issues with some devices). And SoundBlaster Live audio card in PCI slot. And some 1GbE port onboard (BCM57781 this time). And some SSDs attached.

    But those Avantek systems???

    Someone can say that there are AArch64 desktop systems on market. Avantek sells them. But are those machines desktop class systems?

    NO. Those are server mainboards in tower cases. And you basically have no idea what exactly you are going to buy as there is only graphics of case. No internal photos, no back photo, no user manual to download.

    Avantek 32 core Cavium ThunderX ARM Desktop

    No onboard audio. No mention about USB ports, two PCI Express slots (x16 + x8), no 1GbE network port. No mention about m.2 slot for NVME storage.

    Ampere eMAG 64bit Arm Workstation

    Specification may change without notice.
    Product may differ from images shown.

    In other words: “we have no idea what we sell”. Features listed put system a bit closer to desktop class as there are USB ports (just two), there is 1GbE network port. Just one PCI Express port (x8).

    Avantek ThunderX2 ARM Workstation - ThunderXStation

    4U rack/tower case sounds clearly like “we know it is server but will try to sell it as a desktop”. No m.2, no audio, no mention about USB ports, no 1GbE network.

    Other vendors

    So maybe there are other vendors? Depends on how much you are on “has to be SBBR compliant” side.

    Windows on Snapdragon

    Yay! Mobile chips based laptops! Sure, they have their niche. I know people who use them as daily laptops. For normal use and for development.

    But my work requires 16GB+ of RAM and working virtualization.

    Linaro Synquacker

    Or Synquacer or Linaro DeveloperBox or whatever you call that box filled with Cortex-A53 cpu cores. Where you have to choose “working graphics or working virtualization”.


    There are people using it as a desktop. 16GB ram is the limit. One PCIe slot. Lot of network interfaces would make it nice router probably.

    HoneyComb LX2K

    Looks interesting. Will see how it ends once get on market (preliminary version on sale now). Still requires USB audio dongle.

    Friend suggested that as CPU is on SoM card then other carrierboard can be designed. Sure, but not gonna happen probably.

    I have seen it already

    About four years ago I had that funny project: APM Mustang as a desktop (day 1, day 2, day 3). That system was kind of Frankenstein’s monster. Audio over some USB stereo-only dongle, all USB devices plugged into hubs due to only two ports on mainboard I/O panel.

    Looks like nothing changed…

    Written by Marcin Juszkiewicz on
  8. How to survive FOSDEM

    FOSDEM. Free Open Source Developers European Meeting. Every first weekend of February. In Brussels. At Université libre de Bruxelles.

    I used to say that FOSDEM is a week long conference in two days. I don’t think it is valid anymore. Should say two weeks now as last time it was something about 700 prelections. Still during two days.

    I also used to say that I can not afford not being at FOSDEM. It is probably the easiest way to meet developers from all those projects I worked with or was interested in.

    But it is not just Saturday and Sunday. On Friday evening there is a beer event which nowadays takes place at street leading to Delirium Café. Can be hard to get there due to amount of people but it is worth it. Easy way to meet far too many familiar faces. And get beer (once you managed to reach bar).

    But how to survive FOSDEM? Food. Logistics. Clothes. Logistics. Mobile app.


    Yes, logistics is one of key factors when it comes to this conference.

    Lectures take place in several buildings (AW, H, J, K, U). At same time. Getting from one to another takes time. You can meet people on a way and decide to discuss something rather than go for talk.

    Since few years ago there is even online navigation tool to show you which way to go from one place to another. Very handy especially if you have some mobility issues.

    FOSDEM map
    FOSDEM map

    Sometimes it is worth to be in some room session before just to have a seat. Limits are taken seriously and when info on door says “room full” then do not even try to sneak in. You may find some spot and watch live streaming — just remember about good headphones to not interrupt people around you.

    If you need a quiet place to sit for a bit with your laptop then you can go to cafeteria in F building (crowdy, noisy) or sit in hacking room (quiet) in the middle of H building.


    Get place to sleep in city centre. “Crawling distance from Delirium Café” is perfect as this is where lot of people spend Friday’s and Saturday’s evening/night. Also several places to get various food, some grocery stores (just check opening times), pharmacies, GSM shops etc.

    Public transport

    Use whatever you normaly use (Google Maps or so) to check for it. If possible avoid bus 71 if you go on popular time and not from first stop. It gets full right on start and keeps that way until FOSDEM stop.

    On Sunday afternoon (15:30 - 19:30) there are free buses from ULB to Brussels South (Midi) railway station. Easy way to get to BRU airport. Full of people so it is good to have some alternative routes in mind. One of options is getting to Etterbeek railway station (about 1.5 km walk) and then travelling wherever you need.

    When it comes to tickets you basically want MOBIB card. Costs 5€ and can be used each year. I bought mine at some metro station in city centre.

    For Friday to Sunday visit 72h ticket is imho best. You can check STIB ticket information for other options. Note that there are some single use tickets still in use (without MOBIB card).

    Mobile application

    As long as your phone is able to run applications check Mobile schedule apps list at conference website and choose one. I use ‘FOSDEM Companion’ on my Android phone.

    Install application at least week before the event. Sync schedule and start checking which sessions to attend. Ignore fact that some will overlap. It is always better to have more listed than less — sometimes you sacrifice one due to being in wrong building or room will be full so you can take a quick look at list and go somewhere else.

    Internet access

    Once you reach FOSDEM there is no problem — conference area provides you fast wireless network access.

    If you are from one of EU countries then your GSM operator probably provides you with some data limit for “Roaming Like At Home” functionality.

    If you are from outside of EU then I suggest reading prepaid with data wiki page. GSM operators stores are in several places in city centre. Or you may even manage to find some at airport but easier to just grab one once you reach centre.


    Eat proper breakfast. Not “just a fruit and coffee” but some proper one. You may not have a time to stand in a queue for some FOSDEM food. It is not worth your time.

    If you really have to then go to bar under J, have exact change and buy a sandwich. Not best food but you may get it quick.

    Better option is to just have something with you. Get fruits, nuts, energy bars, chocolate. Some kind of quick calories.

    Or spend half an hour (or more) waiting for fast food. Still good opportunity to chat with people.


    Remember that Belgian beer can be stronger than your normal one. Do not drink too much or you will suffer and be too wasted during conference.


    FOSDEM is in February. Which means winter. So it may mean snow but it is Brussels so usually it is either nice or rainy. And you will go into building and out of building several times per day. Be prepared for it.

    Proper shoes to walk on concrete, steps but also snow, mud and grass. Winter cap, scarf, maybe even gloves. Do not have to be thick ones like for some Arctic winter. Assume few degrees above 0°C.

    Small backpack and bottle of water. Do not throw away once get empty. You are at university. You can refill your bottle at several places. Tap water is drinkable.


    Do not put your trash in random places. Use trashcans, inform organizers and volunteers if you see full ones. Put bottles into places for bottles.

    If you can then stay after ending session and help to clean up. It is free event so help those who organized it for you.


    FOSDEM is a free event. But there is swag. You can go to organizers’ desk and donate money to keep the event going. For 25€ you can get the official t-shirt.

    Several projects have their own stands — mostly in K building but there are also other ones in AW (mostly related to embedded space).

    If you are stickers fan (or your new laptop still have some space on case) then go on a tour, discuss with developers, see what they present. Grab leaflets, stickers, donate some money (usually in exchange to more premium swag) etc.

    Or some funny one like this license for VLC I got in 2019:

    Physical VLC license
    Physical VLC license

    Fiancé(e) or family

    If your companion is not interested in FOSDEM you may sign them to free tour(s). My wife came with me to conference once, went for tour each day and enjoyed them.

    Other events

    There are several meetings taking place right before and/or after FOSDEM.

    CentOS Dojo on Friday is a nice opportunity to meet developers working on this distribution. Buildroot developers usually gather after FOSDEM.

    Several other projects are having own meetings, developer days etc. You can check list at FOSDEM Fringe page.

    Written by Marcin Juszkiewicz on
  9. My first 8K intro

    I am on demoscene since 1997 when I attended “Intel Outside 4” party in Włocławek, Poland. But I have never released anything. Until Xenium 2019 party where I presented my first 8K intro. Written for Atari 2600 game console. In some kind of BASIC language…

    The idea

    The idea for it came about year ago during Riverwash demoscene party. Most of PC 64KB intros started with some kind of progress bar as code was generating textures, instruments and other stuff requiring calculations. I joked that it would be great to make something similar for Atari 2600 VCS.

    The fun is that Atari 2600 lacks any usable memory. It has 128 bytes (bytes, not kilobytes) of RAM. The only storage is cartridge with 4KB of ROM space (expandable by bankswitching to 32KB). So there is no point in any generating at other phase than development.

    batari Basic

    During July I took a look at available options and found ‘batari Basic’. It is BASIC-like language developed in 2005-2007 by Fred X. Quimby.

    My main source of help was a forum on Atari Age. Detailed language info came from useful page called ‘Random Terrain’. There is also “Visual BB which is IDE with some tools speeding up development.


    Thanks to Xenium organizers I have my intro recorded as video:

    thumbnail for LChMOMF8GPw video

    As you can see it has two parts. First one is playfield with one line changing every 30 frames (first version was for NTSC). Result is simple progress bar.

    Second part uses “titlescreen kernel” to display dino graphics. I used one provided by Xenium organizers (rescaled to 96x91 pixels). I used rainbow colours to have some colours on screen.

    At party

    At start of “oldchool intro” competition I told to friend sitting next to me that I hope for at least four entries. He asked “why?” so I pointed him to the screen saying “because of it”:

    My first 8K intro announcement screen
    My first 8K intro announcement screen

    There were six entries so I though that I am safe and will have 5th entry maximum.


    Imagine my surprise during voting results announcement next day. My intro took 3rd place!

    Award for 3rd place
    Award for 3rd place

    Amazing for production without any effects, without any audio. And written in BASIC-like language without using any knowledge of 6502 assembly.

    Source code

    If someone wants to see how simple it was then source code is in “my first 8k” git repo on github. Enjoy!

    What next?

    During Xenium I got lot of technical information for VCS programming. And there were questions about my next production.

    Nothing to promise here. And not for Atari VCS. It is sick platform to program due to lack of any framebuffer memory so ‘race the beam’ is in use…

    I have some ideas for Atari XL/XE

    Written by Marcin Juszkiewicz on
  10. Kolla ‘stein’ released

    On last Monday we finally released Kolla and Kolla Ansible 8.0.0 ‘stein’. Took us longer than we planned but now it is done and ready for users.


    What got changed? Many things — details can be found in Kolla release notes and Kolla Ansible release notes.

    My work in this cycle was more reviews, less code. And lot of planning how to handle Python 3 migration for Debian/Ubuntu based images. At some point we decided that this stuff will be postponed to ‘train’ cycle. You can read more about it my previous post: Moving Kolla images to Python 3.

    Why so late

    Usually we release Kolla ‘two weeks’ after official OpenStack release. This allows us to switch to final release code of other projects, do some testing etc. This time it took us far more time :(

    Due to several issues (some core developers got more occupied with work, distributions changed dependencies in packages) it took us longer than ‘two weeks’ after official OpenStack ‘stein’ release. We added more tests for CI, handled partial Python 3 migration in Ubuntu Cloud Archive and more.Several fixes were made in ‘train’ and backported to ‘stein’.

    Now it is your turn — build, deploy, test, report ;D

    Written by Marcin Juszkiewicz on
« Newer posts
Page 3 / 79
Older posts »