It is 10 years of Linux on ARM for me

It was somewhere between 7th and 11th February 2004 when I got package with my first Linux/ARM device. It was Sharp Zaurus SL-5500 (also named “collie”) and all started…

At that time I had Palm M105 (still own) and Sony CLIE SJ30 (both running PalmOS/m68k) but wanted hackable device. But I did not have idea what this device will do with my life.

Took me about three years to get to the point where I could abandon my daily work as PHP programmer and move to a bit risky business of embedded Linux consulting. But it was worth it. Not only from financial perspective (I paid more tax in first year then earned in previous) but also from my development. I met a lot of great hackers, people with knowledge which I did not have and I worked hard to be a part of that group.

I was a developer in multiple distributions: OpenZaurus, Poky Linux, Ångström, Debian, Maemo, Ubuntu. My patches landed also in many other embedded and “normal” ones. I patched uncountable amount of software packages to get them built and working. Sure, not all of those changes were sent upstream, some were just ugly hacks but this started to change one day.

Worked as distribution leader in OpenZaurus. My duties (still in free time only) were user support, maintaining repositories and images. I organized testing of pre-release images with over one hundred users — we had all supported devices covered. There was “updates” repository where we provided security fixes, kernel updates and other improvements. I also officially ended development of this distribution when we merged into Ångström.

I worked as one of main developers of Poky Linux which later became Yocto Linux. Learnt about build automation, QA control, build-after-commit workflow and many other things. During my work with OpenedHand I also spent some time on learning differences between British and American versions of English.

Worked with some companies based in USA. This allowed me to learn how to organize teamwork with people from quite far timezones (Vernier was based in Portland so 9 hours difference). It was useful then and still is as most of Red Hat ARM team is US based.

I remember moments when I had to explain what I am doing at work to some people (including my mom). For last 1.5 year I used to say “building software for computers which do not exist” but this is slowly changing as AArch64 hardware exists but is not on a mass market yet.

Now I got to a point when I am recognized at conferences by some random people when at FOSDEM 2007 I knew just few guys from OpenEmbedded (but connected many faces with names/nicknames there).

Played with more hardware then wanted. I still have some devices which I never booted (FRI2 for example). There are boards/devices which I would like to get rid of but most of them is so outdated that may go to electronic trash only.

But if I would have an option to move back that 10 years and think again about buying Sharp Zaurus SL-5500 I would not change it as it was one of the best things I did.

Another distribution said goodbye to ARMv5 devices

Fedora 18 just became EOL. Most of the people do not care as F20 is present so they can run it on their PCs. But there is a group of users which may care.

All those people with ARMv5t hardware are left with Debian/armel now as there is no other big distribution supporting their devices anymore. Someone will ask “what about Ångström or Gentoo?” but who sane would build Gentoo on armv5te?

I do not remember when last time I used something with arm926 core (or similar – like Kirkwood). Probably few years ago when helped friend to get Sheevaplug booting into Debian.

But there are still Sheevaplugs, Guruplugs, *plugs and QNAP devices out there serving their users with selected services. And some of their owners will have to decide what next…

Nine years of embedded Linux

Nine years ago I bought Sharp Zaurus SL-5500 as my first Linux PDA. And due to this I am where I am.

I could say that it started two years earlier when I saw PalmOS devices at local geek meetings. But it took me over year before Palm m105… Then was Sony Clie SJ30 — gorgeous device. High resolution, memory card, 16bit colour. Too bad that applications did not make use of it.

So I went for Linux. There were two options: Zaurus or iPaq. Went for former one as it had keyboard. It was good choice.

Quickly started development of packages and joined OpenEmbedded team. Then became one of OpenZaurus developers. After year or something took over release maintenance and released few last versions. 3.5.4(.1) were the best tested releases of OZ ever — I had over hundred testers for each RC image and they provided installation reports, bug reports and fixes. And it had unified installer for whole range of devices (took me several months to get it polished and few guys added own tweaks). When Ångström distribution started I was the one who officially ended OpenZaurus development.

And all that was in free time. But in mean time I created my consulting company. CELF was my first customer 😉

One nice evening I got question on irc and due to that I left dark side of IT and went from PHP programming to embedded Linux full-time. OpenedHand had interesting projects and clients with many devices. Imagine operating system + kernel + Python + GStreamer in 16 megabytes of flash… And I managed to get it done. While working for them I used proper developer boards (not only customer devices) and there were funny moments…

When we worked with ST Microelectronics on NDK-15 (later replaced by NHK-15 from ST Ericsson) I had to merge two kernel trees from two separate teams. Took me 2 days of mangling 20-30MB diffs but got it done. There are people at ST-E which reminded me this during one of Linaro Connects ;D

Also on GUADEC 2007 when we presented new interface for Openmoko phones NDK-15 had to wait for me as no one at stand was able to get it running (U-Boot config needed changes).

But then Intel acquired OpenedHand… The craziest trip of my life was return from London to my parents place. For three months I even had email but never used it due to problems with Intel corporate network and Linux (do not ask).

Next was Bug Labs and their BUG device. I cleaned their Poky trees, migrated to latest version and later to use OpenEmbedded directly. Less challenges but I also had few other customers at that time to keep me busy. Some of them were OH customers before and went to me for help.

Time passed, 2010 came. One day Canonical made another attempt to seduce me and this time I decided that it looks like good opportunity so I accepted. Sent BUG 2.0 prototype back to NYC and few weeks later I made crazy train trip to small nowhere near Brussels to meet my new coworkers from NewCore. 1-2 weeks later we got our current name: Linaro.

Total change… From embedded devices to ‘Yes, it is ARM. So what?’ kind as we support(ed) devices powerful enough to run normal desktop software. Many changes for me — from OpenEmbedded where you can (cross) build everything in few hours to Ubuntu packaging where sending package for inclusion into archive meant few hours of buildd queue and then few of build. But I learnt a lot here and met another set of hackers including grey beards ones 😉

And all that because I bought Sharp Zaurus SL-5500 nine years ago…

Unbricked my old SheevaPlug

Few months ago one of my friends borrowed SheevaPlug from me. About two weeks later he gave it back — bricked… I did not had time to play with it so it landed on shelf.

Yesterday I took it and decided to get it back to live. Requirements:

  • bricked SheevaPlug (v1.0 without SATA)
  • power cable
  • mini usb cable
  • usb thumb drive
  • OpenOCD (“apt-get install openocd”)
  • cross compiler (“apt-get install gcc-arm-linux-gnueabi” under Ubuntu)
  • U-Boot sources (HEAD of mainline)
  • Linux sources (also HEAD of mainline)
  • serial terminal (picocom, minicom, screen etc)
  • few terminals or terminal multiplexer (I used tmux)


  • Connected power and mini usb cables to SheevaPlug. Desktop recognized usb-serial device as /dev/ttyUSB1.
  • Connected to it with serial terminal. Nothing appeared there of course 😉
  • Run OpenOCD: “cd /tmp/;sudo openocd -f /usr/share/openocd/scripts/board/sheevaplug.cfg -s /usr/share/openocd/scripts”. SheevaPlug was detected.
  • Connected to OpenOCD: “telnet localhost 4444”.
  • Built U-boot:
export CROSS_COMPILE=arm-linux-gnueabi-
make mrproper
make sheevaplug_config
make u-boot.kwb
  • Copied “u-boot” to “/tmp/uboot.elf” and used “reset;sheevaplug_init;load_image u-boot.elf;resume 0x00600000” — landed in U-Boot 😉
  • There is “sheevaplug_reflash_uboot” macro but it was not working for me. So I used U-Boot to flash itself:
Marvell>> usb start
Marvell>> fatload usb 0:1 0x0800000 u-boot.kwb
Marvell>> nand erase 0x0 0xa0000
Marvell>> nand write 0x0800000 0x0 0xa0000
Marvell>> reset
  • Went to Ångström online image builder and built small busybox based image.
  • Unpacked tarball into /tmp/initfs, added /dev/ttyS0 node.
  • Built Linux kernel:
export CROSS_COMPILE=arm-linux-gnueabi-
make mrproper
make kirkwood_config
make menuconfig (set INITRAMFS_SOURCE to /dev/initfs)
make uImage
  • Copied “arch/arm/boot/uImage” to USB thumb drive and inserted it into SheevaPlug.
  • Booted image:
Marvell>> set ethaddr 'c0:ff:ee:c0:ff:ee'
Marvell>> set bootargs 'console /dev/ttyS0,115200 rw'
Marvell>> usb start;fatload usb 0:1 0x800000 /uImage;bootm 0x800000
  • Landed in nice and small Ångström distribution image 😉
  • Went to Ångström online image builder and built console image (task-base based).
  • Built Linux kernel (this time without initramfs):
export CROSS_COMPILE=arm-linux-gnueabi-
make menuconfig (unset INITRAMFS_SOURCE)
make uImage
  • Copied “arch/arm/boot/uImage” to USB thumb drive and inserted it into SheevaPlug.
  • Prepare NAND for UBI:
# mount none /dev -t devtmpfs
# udhcpc eth0
# opkg-cl update
# opkg-cl install mtd-utils
# ubiformat /dev/mtd2
# ubiattach -p /dev/mtd2
# ubimkvol /dev/ubi0 -N rootfs -s 490MiB
# ubiupdatevol /dev/ubi0_0 /media/sda1/angstrom-task-base.ubifs
# mount -t ubifs ubi0:rootfs /media/rootfs
# chown -R root:root /media/rootfs
# cp /media/sda1/uImage /media/rootfs/boot
# sync
# reboot
  • Another reconfiguration in U-Boot:
Marvell>> bootargs 'console=ttyS0,115200 rw ubi.mtd=2 rootfstype=ubifs root=ubi:rootfs'
Marvell>> bootcmd 'ubi part nand0,2; ubifsmount rootfs; ubifsload 0x800000 /boot/uImage;bootm 0x800000'
Marvell>> mtdids 'nand0=orion_nand'
Marvell>> set mtdparts 'mtdparts=orion_nand:512k(uboot),4m@1m(kernel),507m@5m(rootfs)'
Marvell>> save
Marvell>> reset

And now my SheevaPlug is operational again. Boots from NAND with latest U-Boot and Linux. There is around 440MB free still on NAND (not counting 4MB partition where kernel was expected to be). I can put it back on shelf now.

The only parts which I needed to compile were U-Boot and Linux kernel. I could skip bootloader and use binary image from Internet but prefer to know what my machines run (and building U-Boot is really easy). Initramfs support in Linux is real live saver as I did not had to play with initrd etc — just build image and boot it. The only problem was that devtmpfs was not auto mounted (even if option in kernel was selected).

I could also use one of those “easy installers” made by PlugComputer community but I found such solutions more complicated (fetching binaries, finding requirements etc) than the one I used.

Merging stuff from Poky into OpenEmbedded

Recently I started merging interesting stuff from OpenedHand’s Poky into OpenEmbedded. Now, with OE using GIT as storage for metadata it is much, much easier then it was in Monotone times.

I have over 3 thousands of revisions exported (using git format-patch) from Poky and I am reviewing them and adapt to add into OE. Useful ones are changed by simple shell script which adapts paths, change authors informations (I use Poky via git-svn so no real names/emails) and adds “(from Poky)” message to end of patch description. Then just git am and patch lands in OpenEmbedded.

For now I added newer APT and DPKG package tools, newer QEMU (not the latest but working with ARMv6/v7 instructions), U-Boot mkimage tool which does not use lot of Openmoko patches (just one is needed), Shared MIME Info which does not need any processing on target device and some tweaks here or there.

Next in queue are Maemo4 cleaned recipes (Diablo ones), binary locales for Angstrom powered ARMv6/7 based machines and miscellaneous tweaks or updates.

My private buildbot

One of great things which OpenedHand use to get Poky into working state is buildbot. But it is available only over VPN so recently I started my own buildbot for my projects.

Currently there are few build configurations enabled:

  • Poky build from scratch — qemuarm, c7x0, om-gta01, qemux86 images
  • Poky incremental (same targets as above)
  • Ångström incremental with same machines and base, console and x11 images

Soon more configurations will get added, some will change (probably will drop Poky ones). I am also wondering about providing public access to waterfall display but without rights to stop/start builders.

I hope that this will allow to catch some bad commits.

UPDATE: I added public read-only access to buildbot status page.

2.6.23 on Tosa

Thanks to work done by Dmitry Baryshkov we have 2.6.23 kernel working properly on Zaurus SL-6000 (Tosa). Ångström images are already present so users can test how does it works for them.

Today I reflashed Tosa with fresh build and did some testing. We still have problem with framebuffer (famous yellow lines) but otherwise it looks quite good. USB host is working very nice — I connected few devices:

root@tosa:~$ lsusb
Bus 1 Device 9: ID 1457:5122
Bus 1 Device 8: ID 046d:c70a Logitech, Inc.
Bus 1 Device 7: ID 046d:c70e Logitech, Inc.
Bus 1 Device 6: ID 046d:0b02 Logitech, Inc.
Bus 1 Device 5: ID 0a46:9601 Davicom Semiconductor, Inc.
Bus 1 Device 4: ID 05e3:0606 Genesys Logic, Inc.
Bus 1 Device 2: ID 0bb2:0302 Ambit Microsystems Corp.
Bus 1 Device 1: ID 0000:0000

Detailed checking as to wait until I charge this device to 100% because now it shutdown after few minutes.

x86 architecture mess

x86 architecture is total mess when it comes to naming. Basically there were “i386”, “i486”, “i586”, “i686” — nothing more was used. But this gives lot of problems.

First one — which optimisations can be used on “i686”? It has MMX for sure but does it have SSE? SSEv2? SSEv3? 3DNow! technology? It depends on cpu… for example PentiumPro (the first “i686”) has MMX but lack any other addons. Athlon64 will have most of them (or even all in newest cores) but under 32bit Linux it still be “i686”.

Second thing is how Linux recognize CPU. On ALIX board I have AMD Geode LX which has MMX and 3DNow! but lack SSE. According to some data it is “i686” but under Linux it is “i586”:

root@alix:~$ uname -m

Currently I am running Ångström on it but is it built with “i686” optimisations…

PC Engines Alix1c arrived

During GUADEC time Stelios Koroneos from Digital Opsis announced that PC Engines company donated two Alix1c boards to OpenEmbedded. As the board is interesting I wrote that if no one else will mail than I would like to play with it.

Today it picked it on post office and brought home. First reaction during unpacking: “Wow, this board is TINY!” because I had not used mini-ITX boards before. After connecting few cables (power, ethernet, vga, usb keyboard) and shorting “power switch” with screwdriver I got it booted and checked BIOS settings. Unfortunately I was unable to boot from PXE 🙁

I did search in drawer and took 256M CompactFlash card which I bought few years ago for “collie” and decided to install something on it. Decided that this time it will be Voyage Linux (Debian derived distribution for x86-based embedded platforms such as WRAP, ALIX and Soekris 45xx/48xx boards). Fetched their install CD (33M only) and started qemu with CF card as harddisk and that ISO. Few minutes later card was put into slot on Alix board and I got Voyage running.

Of course I could not resist and now board is upgraded to plain Debian ‘sid’ and 2.6.22 kernel ;D After wedding I will replace NSLU2 with Alix board and add few functions for it:

  • Samba server
  • Bluetooth Access Point
  • CUPS based print server
  • NFS server

During that time I also plan to move it from CF card and Debian to 2.5″ harddisk and Ångström.

Got GTA01B_v4

Two Neo1973 phones I just got my GTA01B_v4 from UPS courier. This is send as “developer update” as package contains:

  • GTA01B_v4 phone
  • battery (charged)
  • Europlug -> US plug adapter (no idea what for)

After powering phone I started ‘DM2’ application which allow to make few simple tests of hardware. And let Mickeyl and Sean stop talking that Neo1973 has nothing common with “The Matrix” movie — the sound sample used to test audio subsystem is Morpheus telling Neo that he is “The One”.

What do I plan to do when I have two phones? First I will upgrade Bv4 to newer software. Then I have to decide which one will run OpenMoko and which Ångström — so I will be able to compare (now both has OpenMoko distribution installed). And I wonder will Bv4 recognize my SimPlus prepaid card (it is normal 250 entries sim) — Bv3 do not like it.

This time I had to pay customs and UPS — 109 PLN (~30 EUR) in total. And as this time it was sent to me as person not as company I can not add this into company costs. But it does not really matter — I have phone and this is more important 🙂