Linaro Connect: good to be back

One of things you do while you are at Linaro is going to Linaro Connect conferences. My previous one was 2012 Copenhagen one so it was good to be back.

All those people from different companies or projects… Some faces I recognized, some not. Several people recognized me, some said that my beard complicated it. Fun ;D

Discussions about random things, random hacking (not all Snow chromebooks are the same), talks to attend, talks to give…

And speaking of speaking — our team had a speak about OpenStack on AArch64. It was recorded but volume level is very low.

Complaining. Sure, there was some. And I got my badge upgraded just to show all those impersonators that I am back.

bagde with the main complainer text on it

There was jetlag as usual so I was a bit of excluded from evening events but those which I attended were great. Like team dinner with the Big Kahuna cheeseburger (with Sprite to drink).

Next year I have to organize trip in a way that I would do some personal sightseeing on a week before conference. According to rumours it would be in one of areas where I have some places to visit ;D

AArch64 desktop hardware?

Soon there will be four years since I started working on AArch64 architecture. Lot of software things changed during that time. Lot in a hardware too. But machines availability still sucks badly.

In 2012 all we had was software model. It was slow, terribly slow. Common joke was AArch64 developers standing in a queue for 10GHz x86-64 cpus. So I was generating working binaries by using cross compilation. But many distributions only do native builds. In models. Imagine Qt4 building for 3-4 days…

In 2013 I got access to first server hardware. With first silicon version of CPU. Highly unstable, we could use just one core etc. GCC was crashing like hell but we managed to get stable build results from it. Qt4 was building in few hours now.

Then amount of hardware at Red Hat was growing and growing. Farms of APM Mustangs, AMD Seattle and several other servers appeared, got racked and available to use. In 2014 one Mustang even landed on my desk (as first such machine in Poland).

But this was server land. Each of those machines costed about 1000 USD (if not more). And availability was hard too.

Linaro tried to do something about it and created 96boards project.

First came ‘Consumer Edition’ range. Yet another small form factor boards with functionality stripped as much as possible. No Ethernet, no storage other than emmc/usb, low amount of memory, chips taken from mobile phones etc. But it was selling! But only because people were hungry to get ANYTHING with AArch64 cores. First was HiKey then DragonBoard410 got released. Then few other boards. All with same set of issues: non-mainline kernel, weird bootloaders, binary blobs for this or that…

Then so called ‘Enterprise Edition’ got announced. With another ridiculous form factor (and microATX as an option). And that was it. There was a leak of Husky board which shown how fucked up design it was. Ports all around the edges, memory above and under board and of course incompatible with any industrial form factor. I would like to know what they were smoking…

Time passed by. Husky got forgotten for another year. Then Cello was announced as a “new EE 96boards board” while it looked as redesigned Husky with two SATA ports less (because who needs more than two SATA, right?). Last time I heard about Cello it was still ‘maybe soon, maybe another two weeks’. Prototypes looked like hand soldered, USB controller mounted rotated, dead on-board Ethernet etc.

In meantime we got few devices from other companies. Pine64 had big campaign on Kickstarter and shipped to developers. Hardkernel started selling ODROID-C2, Geekbox released their TV box and probably something else got released as well. But all those boards were limited to 1-2GB of memory, often lacked SATA and used mobile processors with their own set of bootloaders etc causing extra work for distributions.

Overdrive 1000 was announced. Without any options for expansion it looked like SoftIron wanted customers to buy Overdrive 3000 if they want to use PCI Express card.

So we have 2016 now. Four years of my work on AArch64 passed. Most of distributions support this architecture by building on proper servers but most of this effort is not used because developers do not have sane hardware to play with (sane means expandable, supported by distributions, capable).

There is no standard form factor mainboards (mini-itx, microATX, ATX) available on mass market. 96boards failed here, server vendors are not interested, small Chinese companies prefer to release yet-another-fruit/Pi with mobile processor. Nothing, null, nada, nic.

Developers know where to buy normal computer cases, storage, memory, graphics cards, USB controllers, SATA controllers and peripherals. So vendors do not have to worry/deal with this part. But still there is nothing to put those cards into. No mainboards which can be mounted into normal PC case, have some graphics plugged in, few SSD/HDD connected, mouse/keyboard, monitors and just be used.

Sometimes it is really hard to convince software developers to make changes for platform they are unable to test on. And current hardware situation does not help. All those projects of hardware being available “in a cloud” helps only for subset of projects — ever tried to run GNOME/KDE session over the network? With OpenGL acceleration etc?

So where is my AArch64 workstation? In desktop or laptop form.

Post written after my Google+ post where similar discussion happened in comments.

My work on changing CirrOS images

What is CirrOS and why I was working on it? This was quite common question when I mentioned what I am working on during last weeks.

So, CirrOS is small image to run in a cloud. OpenStack developers use it to test their projects.

Technically it is yet another Frankenstein OS. Built using Buildroot 2015.05 uses uclibc or glibc (depending on target architecture). Then Ubuntu 16.04 kernel is applied on top and “grub” (also from Ubuntu) is used to make it bootable.

The problem was that it was not done in UEFI bootable way…

My first changes were: switch images to GPT, create EFI system partition and put some bootloader there. I first used CentOS “grub2-efi” packages (as they provided ready to use EFI binaries) and later switched to Ubuntu ones as upstream maintainer (Scott Moser) prefers to have all external binaries to came from one source.

When he was on vacations (so merge request had to wait) I started digging more and more into scripts.

Fixed getopt use as arguments passed between scripts were read partly via getopt, partially by assigning variables to ${X} (where X is a number).

All scripts were moved to use Bash (as /bin/sh in Ubuntu is usually Dash which is minimalist POSIX shell), whitespace got unified between all scripts and some other stuff happened as well.

At one moment all scripts had 1835 lines and my diff was 2250 lines (+1018/-603) long. Hopefully Scott was back and we got most of that stuff merged.

Recent (2016.07.21) images are available and work fine on all platforms. If someone uses them with OpenStack then please remember about setting “short_id” property to “ubuntu16.04” — otherwise there may be a problem with finding rootfs (no virtio-scsi in disk images).

Summary:

architecture booting before booting after
aarch64 direct kernel UEFI or direct kernel
arm direct kernel UEFI or direct kernel
i386 BIOS or direct kernel BIOS, UEFI or direct kernel
powerpc direct kernel direct kernel
ppc64 direct kernel direct kernel
ppc64le direct kernel direct kernel
x86-64 BIOS or direct kernel BIOS, UEFI or direct kernel

Visiting UK again — Bletchley Park and Cambridge Beer Festival

For some time I had “visit Bletchley Park” on my ‘places to visit’ list. Some people told me that there is nothing interesting to see, some said that I should definitely go there. So I will. And will also grab some beers at Cambridge Beer Festival like three years ago.

Due to some family duties my visit will be short — landing on Saturday (21st May) and departing on Wednesday (25th May). First Bletchley park and then Cambridge from Sunday evening.

Plans are simple: walk, see old computers, walk, visit long time no see friends, walk, see not so old computers, maybe play some Ingress, meet other friends, drink some beers, exchange some hardware, buy some hardware etc.

This time will skip visiting Linaro office — they moved somewhere outside of Cambridge so it takes too much time to get there just to say “hi” and drink tea.

As usual I will be online so catch me via Hangouts, Telegram, Facebook, mail or call me if you want to meet.

Back @linaro.org

Six years ago I was one of first members of project which later got “Linaro” name. Today I am back. But in different form.

On 30th April 2010 I got email titled “Welcome to Linaro” and became software engineer at Linaro. Time shown that it was done in a way which helped to start project but was not liked by member companies. The plan was to leave in October 2012 but due to someone’s decision I stayed until May 2013.

Today I got “Linaro Assignee On-Boarding” email which means that I am still officially software engineer at Red Hat but assigned to work at Linaro. Same as people from other member companies.

I wonder will I get my unofficial title “main complainer at Linaro” back or do I have to deserve it again ;D

Waiting for 96 boards with MIPS or Arduino?

Linaro Connect in progress. Bubblegum 96 announced and present at 96boards website. But I am waiting for MIPS version…

Why MIPS? Looking how earlier Consumer Edition 96 boards devices got released it looks to me that compliance to specification is optional. And as there is no information that cpu has to be ARM (“Design is SoC independent (targets 32 or 64 bit SoCs)”) so let make MIPS one.

I thought about Atmega first but kernel support shall be provided and video output. On the other side — both (Linux kernel on AVR and video output) were already done in past on that platform so maybe it could be done.

You can run any random kernel release, have own set of ugly patches for bootloader. And you will get “96 boards officially certified” stamp on it.

In my opinion 96boards project should enforce “your SoC should have at least minimal support in linux-next kernel tree” rule before even looking at products. Actions Semi maybe is good in producing chips but looks like they have no idea how to take care of software.

Cello: new AArch64 enterprise board from 96boards project

Few hours ago, somewhere in some hotel in Bangkok Linaro Connect has started. So during morning coffee I watched keynote and noticed that Jon Masters presented RHELSA 7.2 out of the box experience on Huskyboard. And then brand new board from 96boards project was announced: Cello.

Lot of people was expecting that this Linaro Connect will bring Huskyboard alive so people will finally have an option for some cheap board for all their AArch64 needs. Instead Cello was presented:

cello

Compared to Husky (below) there are some hardware differences to notice but it is normal as 96borads enterprise specification only tells where to put ports.

Huskey

I suppose that both boards were designed by different companies. Maybe it was a request from Linaro to ODM vendors to design and mass produce 96boards enterprise board and Husky was prototyped first but Cello won. Or maybe we will see Husky in distribution as well. Good part is: you can preorder Cello and get it delivered in Q2/2016.

Have to admit that I hoped for some industry standard board (96boards Enterprise specification mentions microATX) instead of this weird 96boards-only format which I ranted about already. Anyway 299 USD for quad-core Cortex-A57 with SATA, UEFI, ACPI, PCIe (and maybe few more four letter acronyms) does not sound bad but good luck with finding case for it ;(

I may go back to Linaro

Few days ago my manager asked me if I would like to go back to Linaro. This time not as ‘Linaro-but-Canonical engineer’ but as ‘Red Hat assigned engineer’. That made me thinking…

Those three years at Linaro were good time. Learnt a lot about cross toolchains, got possibility to work on bootstrapping AArch64 support in OpenEmbedded, Debian/Ubuntu and then Fedora/RHEL. Met many skilled people from around the world, travelled into places which I would probably not visit on my own.

Going back sounds good. From my discussions with few people from Linaro there is more and more AArch64 related work there (and I have some hardware at home) so my rusty arm32 skills can rust in peace. Have to take a closer look at what exactly is on a plate there to take and find some place.

So if you work for Linaro and will be at FOSDEM (or devconf.cz) then I would love to talk.

96boards goes enterprise?

96boards is an idea from Linaro to produce some 32 and 64-bit ARM boards. So far there were two boards released in “consumer” format and few more announced of rumoured. The specification also lists “extended” version which has space for some more components.

But during Red Hat Summit there was announcement from AMD with mention of “enterprise” format:

How would you like an affordable and compact 160x120mm board to jump start your development efforts with AArch64? AMD and Linaro have been collaborating to develop a 96Boards Enterprise Edition (EE) specification that is ideal for the individual developer. Targeting the server and networking markets, the board will feature a 4-core AMD Opteron A1100 Series processor with two SO-DIMM memory slots, PCIe®, USB, SATA, and Gigabit Ethernet capabilities. Popular operating systems such as CentOS, Fedora, and Red Hat Enterprise Linux Server for ARM Development Preview are targeted for use with this particular board. Additional software downloads, updates, and a forum for software developers will be available via the 96Boards web site. The board is slated to be available in 2H 2015 from distribution partners worldwide and it will be supported through the Linaro Enterprise Group’s 96Boards.org site.

I do wonder where from they took idea to name yet-another-crazy-non-standard board format “Enterprise Edition”. In my understanding what enterprise user like is something which just works and comes with support and does not require crazy embedded nonsense hacks.

So when I saw post from Jeff Underhill with photos of the board I noticed few arghs.

Top view of AMD "Enterprise" board

Bottom view of AMD "Enterprise" board

First of course is board format. 160x120mm does not sound like any industrial format. Nano-ITX is 120×120, Mini-ITX is 170x170mm. But everyone knows that enterprise people love to be creative and make own cases. Why it was not done as 170×120 with partial compatibility with mini-itx cases?

Second thing (related to first) is connectors placing. With PCI-Express x16 slot (with x8 signals) I wonder how it will look when some cables go one side or the other while card sticks out of board. With SATA ports moved to the other side there would be space for USB and Ethernet ports so all cables would be in same area. Note also molex connector to give power to SATA disks.

Nice that there are two memory slots (DDR3 ECC SO-DIMM). But with second on the bottom we probably can say goodbye to all PC cases as it would not fit. Yay for creativity when it comes to cases (again).

There are holes to mount heatsink above CPU. From quick look I think that those for FM2 socket may fit.

HDMI connector suggests some graphics to be present. I did not heard about Radeon core inside AMD Seattle CPU but it could change since last check.

But even with those “issues” I would like to have that board 😉

96 boards again?

During Linaro Connect 2015 Asia there was announcement about new Linaro project called “96boards”. It is about making cheap ARM/AArch64 boards in same form factor and same placement of ports. And first board named HiKey was presented. Today third one — from Qualcomm. So we have two boards now (2/96 was not yet announced).

I prefer not to comment on form factor, lack of Ethernet, mobile phone cpus and other things people do not like but about software requirements.

96boards specification v1.0 says:

Minimum Software requirements for 96Boards certification will include:

  • Boot architecture (open source implementations are strongly recommended)
    • Support for bootloader such as U-Boot/FDT, UEFI/ACPI, UEFI/FDT
    • Support for a secure execution environment (optional)
    • Support for ARM Trusted Firmware (ARMv8), including PSCI APIs (optional)
  • Accelerated graphics support
    • Accelerated graphics drivers need to be fully supported either with open source code, or through royalty free binary drivers. If binary drivers are utilized, the vendor will provide support to provide updated drivers/libraries to support new mainline Linux kernel features.
  • Kernel
    • A kernel based on one of the following that is buildable from source code and any required binary blobs:
    • kernel.org latest “mainline” or “stable” kernel
    • The latest Google-supported Android kernel version
    • One of the last two kernel.org LTS kernels (for example Linaro LSK)
  • Operating system
    • The latest released (stable) version of one or more of the following open source distributions shall be made available for a 96Boards CE compliant design:
    • Android
    • Debian or Ubuntu
    • Fedora or Red Hat
    • An OpenEmbedded/Yocto build of a Linux distribution

I hoped that Linaro will be a place where free/open source software would matter. But it looks like “let release whatever you want as long as size and ports match” deal. Any blob as bootloader, binary graphics drivers (does someone remember TI OMAP line and PowerVR? Those boards run with raw framebuffer nowadays).

And that kernel requirement… HiKey uses cpu which is not in mainline kernel, so does Qualcomm one. Are they in AOSP kernel? Maybe. But does someone else than Android uses those trees for serious work? Latest I see in kernel-msm (which may not be proper place to check) is 3.10 which was released (in mainline) nearly 2 years ago…

I really wonder how “latest released (stable) version” of Debian/Fedora/Ubuntu can be made available for those boards when all those distributions use mainline kernel only (I do not count user generated remixes which are not supported by anyone).

So I wonder will 96/96 board came with mainline support, open bootloader and open drivers for everything. Time will show. Until that I am not so interested.