Arm world is weird. You can have server (expensive, UEFI, out of the box
supported by any mainline distribution, normal SATA/NVME storage) or SBC (cheap,
usually U-Boot, not supported for months by any distribution, microsd/USB
storage mostly). But I do not want to talk about servers today.
Bootloader(s) mess
So SBC… Which means playing with U-Boot. In the past it meant random fork of it. I
have a feeling (can be wrong) that it changed. Board starts with mainline one or
quite fresh fork with mainlining in progress. Similar with ATF (Arm Trusted
Firmware - level0 bootloader).
So you have SBC, you have bootloader(s) built. The next step is taking microsd
card, finding out which offsets to use. Then put bootloader(s) at proper places
so CPU can read them and boot. Usually it also means that you have to use MBR
style formatting as bootloader is where GPT would be.
Better way maybe?
Is there a better way? Sure it is. But it is very expensive. Or at least I have
a feeling that it is for most of SBC vendors.
Very expensive solution
The solution is SPI flash chip. But it is very expensive - 3 (three) EUR for
512Mb (64MB) chip (if bought 1000 of them). Far too much for 99.99% of SBC probably.
Price taken from simple search on Digikey: W25Q512JV 512 Mb Serial Flash Memory.
64 megabytes is more than enough to store all. Bootloader(s), U-Boot/UEFI
settings etc. You can even fit small OS image there.
Probably 64/128Mb will be enough even. And they are cheaper (128Mb is 1.7 EUR).
Still too expensive…
“Whatever you do, do it upstream” is IMHO one of mottos worth following. So I
went upstream wherever possible.
Python
2020 came, Python 2 is no more. So 2019 was full of cleaning and patching. I
filled several bugs on projects, created patches to mark some Python packages
as Py2 only (or <Py3.5 only). Some projects just dropped support for Py2
packages due to it.
pbr story
With move to Python 3 came some issues with Unicode characters in README files
of packages used by OpenStack components. Filled some bugs, created patches and
then, after several emails, someone found out that it is a bug in ‘pbr’ and
fixed it ;D
manylinux2014
Installing Python packages on AArch64 can be painful. While “pip install numpy”
takes few seconds on x86-64 machine it can take an hour on Arm64 + extra time
to find out which libraries and compilers need to be installed.
So Python developers created
PEP-599 to sort out situation and
create ‘manylinux2014’ target which can be then used to build ‘wheels’ (aka
binary Python packages) not only for x86(-64) but also for 32-bit Arm, 64-bit
Arm, 64-bit Power (big and little endian) and Z/Arch (s390x).
Somehow I managed to be involved in it. Went through pull request on
GitHub, build checked, fixed
dependencies and some other issues, filled CentOS bugs, discussed with CentOS
developers on fixes and then one day all that stuff got merged.
Now you can fetch “manylinux2014” images from “pypa” account on Quay.io
container registry and build ‘wheel’
binaries for different architectures. If you lack target hardware then consider
services like Travis CI which provide access (free for FOSS projects).
CirrOS
Speaking of Travis CI… Near the end of the year I got task from one of Linaro
teams to update CirrOS images with kernels from Ubuntu 18.04 or later.
I fetched current source, built images and started wondering how to do it best.
So started discussing with Scott Moser (main developer) and we agreed that such
move have sense and that some changes are needed in this project.
In December we moved it from Launchpad/cirros
to GitHub/cirros-dev/cirros. Opened
several issues to not forget what we planned and I started looking at some of them.
Project is now using Travis CI service for tests. With all builds being run
parallel we have checks done in less than and hour. Including boot test for
aarch64, arm, i386 and x86_64 architectures (ppc/ppc64/ppc64le need work).
Doing releases on GitHub is one of next tasks. Current service will redirect.
Linaro ERP
After ERP 18.12 release we decided that there will be no more releases and
Debian ‘buster’ will be used as it is (ERP based on Debian ‘stretch’). So I
spent some time on working with Debian kernel maintainers.
Most important part? Merging kernel configuration changes. We had about twenty
extra enabled to get our servers supported. Everything worth using got merged.
In meantime we even found a bug in Linux-stable tree which took some time to fix.
We were ready for Debian ‘buster’ release. And it worked out of the box on all
machines we supported ;D
Big Data
Bleh. Java, protobuf 2.5 and other madness… But as build engineer I get such
stuff from time to time.
Last year it was mostly Apache Drill and Apache Arrow. In both cases I worked
on their Debian packaging (not in-Debian one) to add/fix AArch64 support.
Other projects still wait for ‘let us move to newer protobuf’ but it will take years…
But I forgot that it was written in Python 2… So once I removed Py2 from my
server website updates stopped. No one noticed ;D
Yesterday I looked at situation — there are no updated versions, looks like no
one plans to work on Python 3 porting. One of options was move to other
aggregator written in Ruby but I do not know that language so it was out of question.
Let’s write own one
So I decided to take a look at Python libraries. Feedparser looked useful and
after few hours of coding I had something working.
Requirements are simple: Feedparser (parsing feeds), Jinja2 (templates) and
sgmllib3k (to get relative urls working in feeds). Small bit of documentation is
provided in repository. No tests so far.
Future
I need to add RSS template and do some tests with Feedparser provided feeds to
check did I catched all possible errors.
And need to deploy it on server and move Developer
Planet to use it.
Another year passed so let me try to summarize it. And a lot of things happened
so it will be set of posts.
Kolla
We did two releases: ‘stein’ in July and ‘train’ in December. As usual a bit
later than most of OpenStack projects as we have to wait for their releases,
then for distributions to provide their binary packages, test everything etc.
2019 was also a year of moving to Python 3 in our project. Images went in
rounds — first some in Stein, then rest in Train cycle. We managed to remove
Python 2 support from Kolla-Ansible but have to wait for TripleO project before
we drop Py2 in Kolla itself.
Also, we removed OracleLinux support. No one from active developers had access
to it and jobs were failing on random moments. CentOS 7 support covered all
needs in our opinion.
CentOS 8 was hoped to be ready for cloud use. But it was not due to many
changes needed in build infrastructure. Now, after 8.1.1911 release, there is a
hope that situation will change and SIGs will be able to build their packages.
On the other hand, we added support for IPv6 deployments. Linaro developers
helped with testing as our second Linaro Developer Cloud will be IPv6 only and
all resources will be provided for OpenStack infrastructure.
There was new OpenStack project called Kayobe. It used both Kolla and
Kolla-Ansible (and some other projects) to deploy containered OpenStack on bare
metal. Including provisioning. We accepted it as subproject of Kolla as some of
our members work in all those projects.
Infrastructure
During 2019 I got huge amount of help from OpenStack infrastructure team. We
changed mirroring of CentOS, Debian, Ubuntu and several external repositories
for any of those distributions. AArch64 was added where it was needed, changes
got reviewed quickly. Perfect job!
Continuous Integration
After long fight with issues we finally have working AArch64 CI for OpenStack
projects. Linaro provided some vcpu in Linaro Developer Cloud, infrastructure
admins created some nodes from it and then created ‘check-arm64’ pipeline on Zuul.
This way any OpenStack project can add tests on 64-bit Arm hardware without
impacting their standard set of tests. We went that way due to low amount of
nodes and it turned out to be very good move. Kolla(-ansible) itself is able to
saturate all of them during developer heavy days.
More nodes are coming. Managed by Linaro on sponsored hardware. Pure IPv6!
Python runs everywhere, right? All those libraries are just one ‘pip install’
away. And we are used to it. Unless on AArch64.
On AArch64 when you do pip install SOMETHING you may end with “no compiler
installed” or “No lapack/blas resources found.” messages. All due to lack of
wheel files generated for this architecture… And even if you have all
dependencies installed then building takes more time than it takes to install
existing wheel file.
But there is a light!
PEP 599 defined “manylinux2014”
target which is the first supporting something else than just “x86(-64)”
architecture. The new arrivals are AArch64, PPC64 (Big and Little Endian) and
32-bit Arm.
I helped getting them working on
AArch64. Do not remember how I ended there to be honest ;D
If your project uses Travis CI for testing and release then adding support
for non-x86 architectures
is just one edit away. All you need is entry in “arch” list of in “matrix”:
matrix:
include:
- os: linux
arch: amd64
- os: linux
arch: arm64
And it really works!
I added support for it in “dumb-init” so now kolla does not need to use any
workarounds but can grab binary straight from project’s releases. Took just few
simple lines in “.travis.yml” file.
What is left to do now?
There are many Python projects out there. Most of them do not know that
manylinux2014 got released and what it brings. Or that Travis CI can give them
non-x86 architecture support. I slowly started creating issues, bug reports in
them to make sure they are aware.
My i7 based desktop was about 8 years old and it was time to replace it. As
there was no affordable aarch64 desktop class solution I have decided to go
with AMD.
Third generation of Ryzen processors brought several new features. Not that I
care much of them. The important stuff is that they are faster than my
Sandybridge i7. And that mainboards nowadays have m.2 slots so pcie-to-m.2
adapter cards are no longer needed.
Processor
Looked at performance and price of new AMD cpus and chosen Ryzen 5 3600. Fast
enough and still quite cheap compared to next in line. And it has more
cores/threads than my i7.
Stock cpu cooler is worthless junk as usual. Scythe Mugen 5 will replace it
soon (ordered yesterday).
Mainboard
Next was mainboard. First I thought about buying one with B450 chipset but then
decided to bump budget and get x570 one. PCI Express 4.0 support, two m.2
slots, usb 3.2 ports etc. Should serve well for another few years.
MSI X570-A Pro is what I went for. Nice black one, has all expansion slots I
wanted. According to lspci there are 48 devices on my system (with graphics
card inserted into PCIe slot and one NVME).
mainboard partially assembled
Memory
Ram was next. This time new platform did not meant doubling amount of memory.
Went with 32GB as two 16GB sticks so there is space for expansion if there will
be any need.
As with previous machines went with Kingston memory sticks. 3200MHz CL16 was
proper balance between price and performance. Memtest86 told that they are fine
(while looking funny on 34” panel).
memtest86 on 34” LCD
Old stuff
Rest of my setup is from previous system. Case, nvme, ssd, Geforce 1050 Ti
graphics card. My desktop always reused something from previous one. I could
replace SATA cables with black ones to keep one colour scheme but this way I
know which cable is which device.
my underdesk desktop
There were some no long needed cards. Soundblaster Live, usb 3 controller or
pcie-to-m.2 adapter card went into storage box. Who knows, maybe one day will
land in some other system.
Issues
As with any new hardware there are some issues. Stock cpu cooler is crap as
always. Getting cpu busy means lot of noise. Will solve that with Scythe Mugen 5.
Other problem is usb related. For some reason one of my hubs (Anker 4 port)
does not detect hot plugging devices. Cold plugged ones (present at system
boot) work fine. Something to look closer one day.
Why again?
My first PC machine had AMD Duron cpu. 600 MHz overclocked to 850. Then Athlon
XP. Athlon 64. Athlon 64 X2. The last one probably still has a use from time to time…
I have decided to finally upgrade my eight years old x86-64
desktop. Will stay with same architecture,
just change CPU vendor from Intel to AMD.
This move was delayed few times with “who knows, maybe finally there will be
some affordable AArch64 desktop systems”. But time to say the ugly truth:
There have never been, nor will be AArch64 desktop systems
No one makes them because no one buys them. No one buys because there is no
hardware to buy.
Or we got to the point where most of people just need a phone? And developers
are expected to use remote machines for everything?
What is desktop system?
For me desktop system is a computer in a box somewhere under desk. With some features:
fast storage (like m.2 NVME),
option for more than 16GB of RAM,
onboard audio able to handle 5.1/7.1 speaker set,
several USB ports,
gigabit Ethernet,
some PCI Express slots for potential upgrades.
For example my eight years old x86-64 desktop has 256GBNVME in m.2 adapter
(occupying PCIe x4 slot), 32GB of memory, additional USB 3.0 controller in the
other PCI Express slot (as onboard Etron EJ168 one has issues with some
devices). And SoundBlaster Live audio card in PCI slot. And some 1GbE port
onboard (BCM57781 this time). And some SSDs attached.
But those Avantek systems???
Someone can say that there are AArch64 desktop systems on market. Avantek sells
them. But are those machines
desktop class systems?
NO. Those are server mainboards in tower cases. And you basically have no idea
what exactly you are going to buy as there is only graphics of case. No internal
photos, no back photo, no user manual to download.
Avantek 32 core Cavium ThunderX ARM Desktop
No onboard audio. No mention about USB ports, two PCI Express slots (x16 + x8),
no 1GbE network port. No mention about m.2 slot for NVME storage.
Ampere eMAG 64bit Arm Workstation
Specification may change without notice.
Product may differ from images shown.
In other words: “we have no idea what we sell”. Features listed put system a bit
closer to desktop class as there are USB ports (just two), there is 1GbE network
port. Just onePCI Express port (x8).
Avantek ThunderX2 ARM Workstation - ThunderXStation
4U rack/tower case sounds clearly like “we know it is server but will try to
sell it as a desktop”. No m.2, no audio, no mention about USB ports, no 1GbE network.
Other vendors
So maybe there are other vendors? Depends on how much you are on “has to be SBBR
compliant” side.
Windows on Snapdragon
Yay! Mobile chips based laptops! Sure, they have their niche. I know people who
use them as daily laptops. For normal use and for development.
But my work requires 16GB+ of RAM and working virtualization.
Linaro Synquacker
Or Synquacer or Linaro DeveloperBox or whatever you call that box filled with
Cortex-A53 cpu cores. Where you have to choose “working graphics or
working virtualization”.
Macchiatobin
There are people using it as a desktop. 16GB ram is the limit. One PCIe slot.
Lot of network interfaces would make it nice router probably.
HoneyComb LX2K
Looks interesting. Will see how it ends once get on market (preliminary
version on sale now). Still requires USB audio dongle.
Friend suggested that as CPU is on SoM card then other carrierboard can be
designed. Sure, but not gonna happen probably.
I have seen it already
About four years ago I had that funny project: APM Mustang as a
desktop (day
1, day
2, day
3). That system was kind of
Frankenstein’s monster. Audio over some USB stereo-only dongle, all USB devices
plugged into hubs due to only two ports on mainboard I/O panel.
FOSDEM. Free Open Source Developers European Meeting. Every first weekend of
February. In Brussels. At Université libre de Bruxelles.
I used to say that FOSDEM is a week long conference in two days. I don’t think
it is valid any longer. Should say two weeks now as last time it was something
about 700 talks. Still during two days.
I also used to say that I can not afford not being at FOSDEM. It is probably the
easiest way to meet developers from all those projects I worked with or was
interested in.
But it is not just Saturday and Sunday. On Friday evening there is a beer event
which nowadays takes place at street leading to Delirium Café. Can be hard to
get there due to number of people but it is worth it. Easy way to meet far too
many familiar faces. And get beer (once you managed to reach bar).
But how to survive FOSDEM? Food. Logistics. Clothes. Logistics. Mobile app.
Logistics
Yes, logistics is one of key factors when it comes to this conference.
Lectures take place in several buildings (AW, H, J, K, U). At same time.
Getting from one to another takes time. You can meet people on a way and decide
to discuss something rather than go for talk.
Since few years ago there is even online navigation
tool to show you which way to go from one place to
another. Very handy especially if you have some mobility issues.
FOSDEM map
Sometimes it is worth to be in some room session before just to have a seat.
Limits are taken seriously and when info on door says “room full” then do not
even try to sneak in. You may find some spot and watch live
streaming — just remember about
good headphones to not interrupt people around you.
If you need a quiet place to sit for a bit with your laptop then you can go to
cafeteria in F building (crowded, noisy) or sit in hacking room (quiet) in the
middle of H building.
Hotel
Get place to sleep in city centre. “Crawling distance from Delirium Café” is
perfect as this is where lot of people spend Friday’s and Saturday’s
evening/night. Also several places to get various food, some grocery stores
(just check opening times), pharmacies, GSM shops etc.
Public transport
Use whatever you normally use (Google Maps or so) to check for it. If possible
avoid bus 71 if you go on popular time and not from the first stop. It gets full
right on start and keeps that way until FOSDEM stop.
IMPORTANTNOTE: Up to 2023 edition there were free buses from ULB to
Brussels South (Midi) railway station on Sunday afternoon (15:30 - 19:30). It
gave an easy way to get to BRU airport. They will be not present in 2024 ;(
One of options is getting to Etterbeek railway station (about 1.5 km walk) and
then travelling wherever you need. This is my favourite way to go from FOSDEM to
BRU airport on Sunday evening.
Nowadays (2023) it looks like using bank card is easiest way. Just enter
bus/metro/tram and tap to pay.
Note that there are some single use tickets still in use if you prefer not to
use bank card.
Mobile application
As long as your phone is able to run applications check Mobile schedule
apps list at conference website and
choose one. I use ‘FOSDEM Companion’ on my Android phone.
Install application at least week before the event. Sync schedule and start
checking which sessions to attend. Ignore fact that some will overlap. It is
always better to have more listed than less — sometimes you sacrifice one due to
being in wrong building or room will be full so you can take a quick look at
list and go somewhere else.
Internet access
Once you reach FOSDEM there is no problem — conference area provides you fast
wireless network access.
If you are from one of EU countries then your GSM operator probably provides you
with some data limit for “Roaming Like At Home” functionality.
If you are from outside of EU then I suggest reading prepaid with
data wiki page. GSM
operators stores are in several places in city centre. Or you may even manage to
find some at airport but easier to just grab one once you reach centre.
Food
Eat proper breakfast. Not “just a fruit and coffee” but some proper one. You
may not have a time to stand in a queue for some FOSDEM food. It is not worth
your time.
Better option is to just have something with you. Get fruits, nuts, energy
bars, chocolate. Some kind of quick calories.
Or spend half an hour (or more) waiting for fast food. Still good opportunity to
chat with people.
Beer
Remember that Belgian beer can be stronger than your normal one. Do not drink
too much or you will suffer and be too wasted during conference.
Clothes
FOSDEM is in February. Which means winter. So it may mean snow but it is
Brussels so usually it is either nice or rainy. And you will go into building
and out of building several times per day. Be prepared for it.
Proper shoes to walk on concrete, steps but also snow, mud and grass. Winter
cap, scarf, maybe even gloves. Do not have to be thick ones like for some Arctic
winter. Assume few degrees above 0°C.
Small backpack and bottle of water. Do not throw away once get empty. You are at
university. You can refill your bottle at several places. Tap water is drinkable.
Trash
Do not put your trash in random places. Use trashcans, inform organizers and
volunteers if you see full ones. Put bottles into places for bottles.
If you can then stay after ending session and help to clean up. It is free
event so help those who organized it for you.
Swag
FOSDEM is a free event. But there is swag. You can go to organizers’ desk and
donate money to keep the event going. For 25€ you can get the official t-shirt.
Several projects have their own stands — mostly in K building but there are also
other ones in AW (mostly related to embedded space).
If you are stickers fan (or your new laptop still have some space on case) then
go on a tour, discuss with developers, see what they present. Grab leaflets,
stickers, donate some money (usually in exchange to more premium swag) etc.
Or some funny one like this license for VLC I got in 2019:
Physical VLC license
Fiancé(e) or family
If your companion is not interested in FOSDEM you may sign them to free
tour(s). My wife came with me to
conference once, went for tour each day and enjoyed them.
Other events
There are several meetings taking place right before and/or after FOSDEM.
CentOS Project has event before FOSDEM. It is a nice opportunity to meet
developers working on this distribution.
After FOSDEM you can attend OpenEmbedded workshop of Buildroot developers meeting.
Several other projects are having own meetings, developer days etc. You can
check list at FOSDEM Fringe page.
