96 boards again?

During Linaro Connect 2015 Asia there was announcement about new Linaro project called “96boards”. It is about making cheap ARM/AArch64 boards in same form factor and same placement of ports. And first board named HiKey was presented. Today third one — from Qualcomm. So we have two boards now (2/96 was not yet announced).

I prefer not to comment on form factor, lack of Ethernet, mobile phone cpus and other things people do not like but about software requirements.

96boards specification v1.0 says:

Minimum Software requirements for 96Boards certification will include:

  • Boot architecture (open source implementations are strongly recommended)
    • Support for bootloader such as U-Boot/FDT, UEFI/ACPI, UEFI/FDT
    • Support for a secure execution environment (optional)
    • Support for ARM Trusted Firmware (ARMv8), including PSCI APIs (optional)
  • Accelerated graphics support
    • Accelerated graphics drivers need to be fully supported either with open source code, or through royalty free binary drivers. If binary drivers are utilized, the vendor will provide support to provide updated drivers/libraries to support new mainline Linux kernel features.
  • Kernel
    • A kernel based on one of the following that is buildable from source code and any required binary blobs:
    • kernel.org latest “mainline” or “stable” kernel
    • The latest Google-supported Android kernel version
    • One of the last two kernel.org LTS kernels (for example Linaro LSK)
  • Operating system
    • The latest released (stable) version of one or more of the following open source distributions shall be made available for a 96Boards CE compliant design:
    • Android
    • Debian or Ubuntu
    • Fedora or Red Hat
    • An OpenEmbedded/Yocto build of a Linux distribution

I hoped that Linaro will be a place where free/open source software would matter. But it looks like “let release whatever you want as long as size and ports match” deal. Any blob as bootloader, binary graphics drivers (does someone remember TI OMAP line and PowerVR? Those boards run with raw framebuffer nowadays).

And that kernel requirement… HiKey uses cpu which is not in mainline kernel, so does Qualcomm one. Are they in AOSP kernel? Maybe. But does someone else than Android uses those trees for serious work? Latest I see in kernel-msm (which may not be proper place to check) is 3.10 which was released (in mainline) nearly 2 years ago…

I really wonder how “latest released (stable) version” of Debian/Fedora/Ubuntu can be made available for those boards when all those distributions use mainline kernel only (I do not count user generated remixes which are not supported by anyone).

So I wonder will 96/96 board came with mainline support, open bootloader and open drivers for everything. Time will show. Until that I am not so interested.


So today Linaro announced their first board from 96boards project. It is named HiKey and is based on HiSilicon cpu for mobile phones.

I had an occasion to see that board during FOSDEM and decided to write something about it after it land on my desk (which will happen sooner or later). But I have read specification for this and next boards and decided to write few words from my perspective.

First thing? Footprint. Good that two sizes are available for designs as not everyone may want to squeeze into small one.

Second? Ports. 2015 year and no Ethernet, no SATA? Sure, first board is based on SoC from a mobile phone but there is no place on small board for them and extended version looks like not allow for extra ports too.

Next? Power supply. 8-18V in a world where everything is on 5V already. The only place where 12V is mentioned in spec is “external fan power”.

So as we are on voltage… Serial at pins and 1.8V level. Nice way of forcing everyone to buy new serial dongle (Arduino ones are 3.3 or 5V).

But assume that we got it powered and have serial connected. How to boot it? According to specs mainline kernel (or AOSP one or LTE one) has to be used. I wonder how HiSilicon cpu is supported in any of those. From what I read during day (on quite slow connection) it is still not in a kernel…

Graphics situation is still shitty. Vendor is allowed to provide binary blobs to get display working. Did they not learnt from OMAP? PowerVR again someone? But sure, plain framebuffer is all you need. OpenGL is for weak.

I prefer not to discuss about selection of signals on low/high connectors. There are more capable people for it. I only wonder why 2mm raster where nearly all boards I had played with had 2.500 one.

I like list of distributions listed as ones to choose. No longer Android/Ubuntu but also Debian, Fedora or OpenEmbedded based one

But give them time. It is just first board and next ones are announced. Marvell will produce one (they are in a Linaro group for it), other will (probably) follow. Hope that there will be something better.

2 years of AArch64 work

I do not remember exactly when I started working on ARMv8 stuff. Checked old emails from Linaro times and found that we discussed AArch64 bootstrap using OpenEmbedded during Linaro Connect Asia (June 2012). But it had to wait a bit…

First we took OpenEmbedded and created all tasks/images we needed but built them for 32-bit ARM. But during September we had all toolchain parts available: binutils was public, gcc was public, glibc was on a way to be released. I remember that moment when built first “helloworld” — probably as one of first people outside ARM and their hardware partners.

At first week of October we had ARMv8 sprint in Cambridge, UK (in Linaro and ARM offices). When I arrived and took a seat I got information that glibc just went public. Fetched, rebased my OpenEmbedded tree to drop traces of “private” patches and started build. Once finished all went public at git.linaro.org repository.

But we still lacked hardware… The only thing available was Versatile Express emulator which required license from ARM Ltd. But then free (but proprietary) emulator was released so everyone was able to boot our images. OMG it was so slow…

Then fun porting work started. Patched this, that, sent patches to OpenEmbedded and to upstream projects and time was going. And going.

In January 2013 I started X11 on emulated AArch64. Had to wait few months before other distributions went to that point.

February 2013 was nice moment as Debian/Ubuntu team presented their AArch64 port. It was their first architecture bootstrapped without using external toolchains. Work was done in Ubuntu due to different approach to development than Debian has. All work was merged back so some time later Debian also had AArch64 port.

It was March or April when OpenSUSE did mass build of whole distribution for AArch64. They had biggest amount of packages built for quite long time. But I did not tracked their progress too much.

And then 31st May came. A day when I left Linaro. But I was already after call with Red Hat so future looked quite bright ;D

June was month when first silicon was publicly presented. I do not know what Jon Masters was showing but it probably was some prototype from Applied Micro.

On 1st August I got officially hired by Red Hat and started month later. My wife was joking that next step would be Retired Software Engineer ;D

So I moved from OpenEmbedded to Fedora with my AArch64 work. Lot of work here was already done as Fedora developers were planning 64-bit ARM port few years before — when it was at design phase. So when Fedora 15 was bootstrapped for “armhf” it was done as preparation for AArch64. 64-bit ARM port was started in October 2012 with Fedora 17 packages (and switched to Fedora 19 during work).

My first task at Red Hat was getting Qt4 working properly. That beast took few days in foundation model… Good that we got first hardware then so it went faster. 1-2 months later and I had remote APM Mustang available for my porting work.

In January 2014 QEmu got AArch64 system emulation. People started migrating from foundation model.

Next months were full of hardware announcements. AMD, Cavium, Freescale, Marvell, Mediatek, NVidia, Qualcomm and others.

In meantime I decided to make crazy experiment with OpenEmbedded. I was first to use it to build for AArch64 so why not be first to build OE on 64-bit ARM?

And then June came. With APM Mustang for use at home. Finally X11 forwarding started to be useful. One of first things to do was running firefox on AArch64 just to make fun of running software which porting/upstreaming took me biggest amount of time.

Did not took me long to get idea of transforming APM Mustang (which I named “pinkiepie” as all machines at my home are named after cartoon characters) into ARMv8 desktop. Still waiting for PCI Express riser and USB host support.

Now we have October. Soon will be 2 years since people got foundation model available. And there are rumors about AArch64 development boards in production with prices below 100 USD. Will do what needed to get one of them on my desk 😉

10 years ago I got write access to OpenEmbedded

It was 8th May of 2004 when I did first push to OpenEmbedded repository. It was BitKeeper at that time but if someone wants to look then commit can be seen in git.

I will not write about my OE history as there are several posts about it on my blog already:

It was nice to be there through all those years to see how it grows. From a tool used by bunch of open source lovers who wanted to build stuff for own toys/devices, to a tool used by more and more companies. First ones like OpenedHand, Vernier. Then SoC vendors started to appear: Atmel, Texas Instruments and more. New architectures were added. New rewrites, updates (tons of those).

Speaking of updates… According to statistics from Ohloh.net I am still in top 5 contributors in OpenEmbedded and Yocto project 😉

There were commercial devices on a market with OpenEmbedded derived distributions running on them. I wonder how many Palm Pre users knew that they can build extra packages with OE. And that work was not lost — LG Electronics uses WebOS on their current TV sets and switched whole development team to use OpenEmbedded.

Since 2006 we got annual meetings and this year we have two of them: European as usual and North America one for first time (there was one few years ago during ELC but I do not remember was it official).

There is OpenEmbedded e.V. which is non-profit organization to take care of OE finances and infrastructure. I was one step from being one of its founders but birth of my daughter was more important 😉

And of course there is the Yocto project. Born from OpenedHand’s Poky helped to bring order into OpenEmbedded. Layers (which were discussed since 2006 at least) were created and enforced so recipes are better organized than it was before. It also helped with visibility. Note that when I write OpenEmbedded I mean OpenEmbedded and Yocto project as they are connected.

I remember days when Montavista was seen as kind of competitor (“kind of” because they were big and expensive while we were just a bunch of guys). Then they moved to OpenEmbedded and dropped own tools. Other company with such switch was Denx. 3 years ago they released ELDK 5.0 which was OE based and made several releases since then.

What future will bring? No idea but it will be bright. And I will still be somewhere nearby.

AArch64 can build OpenEmbedded

In 2012 I was porting OpenEmbedded to target AArch64 so I can say that I did first OE builds for that architecture.

But today I did kind of reverse thing.

In 2012 I was porting OpenEmbedded to target AArch64 so I can say that I did first OE builds for that architecture.

But today I did kind of reverse thing:

Build Configuration:
BB_VERSION        = "1.21.1"
BUILD_SYS         = "aarch64-linux"
TARGET_SYS        = "arm-oe-linux-gnueabi"
MACHINE           = "genericarmv7a"
DISTRO            = "nodistro"
DISTRO_VERSION    = "nodistro.0"
TUNE_FEATURES     = "armv7a vfp thumb neon callconvention-hard"
TARGET_FPU        = "vfp-neon"

Yes — I did build on AArch64 machine targeting ARMv7a system. Had to edit one patch (pseudo-native was set to use very old glibc symbols which are not available on 64-bit ARM) but after that build was running just fine.

I did not tested resulting binaries.

It is 10 years of Linux on ARM for me

It was somewhere between 7th and 11th February 2004 when I got package with my first Linux/ARM device. It was Sharp Zaurus SL-5500 (also named “collie”) and all started…

At that time I had Palm M105 (still own) and Sony CLIE SJ30 (both running PalmOS/m68k) but wanted hackable device. But I did not have idea what this device will do with my life.

Took me about three years to get to the point where I could abandon my daily work as PHP programmer and move to a bit risky business of embedded Linux consulting. But it was worth it. Not only from financial perspective (I paid more tax in first year then earned in previous) but also from my development. I met a lot of great hackers, people with knowledge which I did not have and I worked hard to be a part of that group.

I was a developer in multiple distributions: OpenZaurus, Poky Linux, Ångström, Debian, Maemo, Ubuntu. My patches landed also in many other embedded and “normal” ones. I patched uncountable amount of software packages to get them built and working. Sure, not all of those changes were sent upstream, some were just ugly hacks but this started to change one day.

Worked as distribution leader in OpenZaurus. My duties (still in free time only) were user support, maintaining repositories and images. I organized testing of pre-release images with over one hundred users — we had all supported devices covered. There was “updates” repository where we provided security fixes, kernel updates and other improvements. I also officially ended development of this distribution when we merged into Ångström.

I worked as one of main developers of Poky Linux which later became Yocto Linux. Learnt about build automation, QA control, build-after-commit workflow and many other things. During my work with OpenedHand I also spent some time on learning differences between British and American versions of English.

Worked with some companies based in USA. This allowed me to learn how to organize teamwork with people from quite far timezones (Vernier was based in Portland so 9 hours difference). It was useful then and still is as most of Red Hat ARM team is US based.

I remember moments when I had to explain what I am doing at work to some people (including my mom). For last 1.5 year I used to say “building software for computers which do not exist” but this is slowly changing as AArch64 hardware exists but is not on a mass market yet.

Now I got to a point when I am recognized at conferences by some random people when at FOSDEM 2007 I knew just few guys from OpenEmbedded (but connected many faces with names/nicknames there).

Played with more hardware then wanted. I still have some devices which I never booted (FRI2 for example). There are boards/devices which I would like to get rid of but most of them is so outdated that may go to electronic trash only.

But if I would have an option to move back that 10 years and think again about buying Sharp Zaurus SL-5500 I would not change it as it was one of the best things I did.

From a diary of AArch64 porter

One day I will go to software conference with an axe or a knife and will turn a place into slaughterhouse…

During last few weeks most of my work was related to fixing build issues on AArch64 platform. That’s what I do since September 2012. Just operating system changed from OpenEmbedded to Fedora. And there are days when I want to kill.

Kill who? Software developers. Some for shipping few years old copies of config.{guess,sub} files. Others for inventing crazy ways of abusing autoconf usage. My latest find was fakeroot.spec which has this precious jewel:

for type in sysv tcp; do
mkdir obj-$type
cd obj-$type
cat >> configure << 'EOF'
#! /bin/sh
exec ../configure "$@"
chmod +x configure
%configure \
  --disable-dependency-tracking \
  --disable-static \
  --libdir=%{_libdir}/libfakeroot \
  --with-ipc=$type \
cd ..

By default "%configure" macro updates config.{guess,sub} files. But it does it in place. So no luck here.

There are countless packages like that. Code for 3rd-party libraries bundled with code may have them as well.

So if your package uses config.{guess,sub} files then please take a look and do an update of them with new release.

My own company started 8th year today

Seven years ago I created my one person company. And it was one of best things I did in my life.

All started in 2006 when I started doing some small paid jobs around OpenEmbedded. Small things: solving build problems, updating recipes, adding new ones. But companies prefer to get invoice for such stuff instead of just giving money…

So one day I went to city hall and created what was then called “HaeRWu Marcin Juszkiewicz”. I changed name 2 years later and got rid of that ‘impossible to pronouce’ part.

There were many different clients for my consulting work. CELF was my first one, later I dropped my daily work and started remote work for OpenedHand. When they were acquired by Intel I got quite nice offer but preferred not to move to UK so went own way. From time perspective I do not know was it right decision 😉

So I looked at market around OpenEmbedded and started working with Bug Labs and few smaller jobs for other clients (some knew me from OpenedHand times). Also had job proposal from Canonical for their newly created ARM team but nothing came from it.

Time passed. One and half-year later Canonical made another attempt and this time I though “why not?”. So I went there just to be moved outside to a team which did not have any official name (other than NewCo or New Core which you may heard somewhere). And that team became Linaro some days later.

At Linaro I did lot of cleanup in Debian/Ubuntu toolchain components, added bootstrapable cross toolchain and fixed several packages (also created some new ones). But then, just when I was supposed to move to Canonical, new things came and AArch64 took my whole time.

ARMv8 work was great time. Learnt new things about OpenEmbedded, saw how project moved during those two years when I did not follow it’s development. Och it was good time.

But good things have to end one day. And so did my time at Linaro. But at around same time I started talking with several companies around Linaro to find a new place for me.

And I found it at Red Hat. Took a bit of time to get everything set up but I think that it was worth it. But due to the fact that I am employee not contractor I will suspend and in few months shutdown my consulting company.

It served me well. I came from being person not recognizable to someone who is known by people who I see for first time. It is good feeling 😉

On my own again

After 3 years at Linaro I have decided to not continue my trip with Canonical. So now I am back to be on my own again.

I will not write why I made such decision but also want to mention that time at Canonical/Linaro was good. I learnt some new tools and added some of them to “avoid if possible” list. From products created and developed at Canonical there are Bazaar and Unity. Both have replacements which I like more.

What next? Will see — I had some meetings and discussions. But I am open for job offers of course 😉 It can be Debian or OpenEmbedded or Ubuntu or other ARM Linux related as long I do not have to move.

State of Linaro layers for OpenEmbedded

As I will leave Linaro at the end of May I would like to write a summary of current state of Linaro layers for OpenEmbedded.

At Linaro we have 3 layers:

  1. meta-aarch64
  2. meta-linaro
  3. meta-linaro-toolchain

First one is BSP kind. I know that it had some issues which affected each build which had it in BBLAYERS but I fixed those issues. I would like to thank Khem Raj for pointing me at those.

We have git version of binutils there due to some changes which were not present in 2.23 line. But use of this version is not required as builds are fine with OE Core one.

We have “tune-armv8.inc” in this layer as well. There was attempt to merge that into OE Core but “/lib or /lib64” discussion started and at that time I decided to skip it. There are similar discussions at GCC and Glibc mailing lists. Once they sort that out OE tune file will be adapted by someone (I hope).

Rest of recipes can be split into 2-3 types. Few (like sysprof, emacs) just disable recipes for AArch64. Other have extra patches to add missing functionality or defines. And we have Linaro kernel for AArch64 there.

Second layer has ARMv7a(b) machine definitions used for our machine independent builds and some recipes.

There are no patches for OE recipes here. The only exception is busybox where we enable “dpkg(-deb)” command which we need for our tools used to merge rootfs with hardware support.

We have “recipes-extra” where we keep new recipes which may not be in a nicest state so are not yet merged into OpenEmbedded (or have no use there like “meta-toolchain-hhvm” one).

“recipes-linaro” is for our stuff. Images, automatic root shell on serial port etc.

And finally is toolchain layer. Everything here is related to gcc-linaro and Linaro binary cross toolchains (armv7a and aarch64 ones). GCC 4.6 and 4.7 is there but 4.6 one will be removed when 4.8 will be added into OE Core.

Who will maintain those layers after my leave? This was not decided yet. There are few guys at Linaro who know how to use OpenEmbedded but I think that most of them is outside of Builds and Baselines team.

If you have any questions then better ask now.