Nine years of embedded Linux

Nine years ago I bought Sharp Zaurus SL-5500 as my first Linux PDA. And due to this I am where I am.

I could say that it started two years earlier when I saw PalmOS devices at local geek meetings. But it took me over year before Palm m105… Then was Sony Clie SJ30 — gorgeous device. High resolution, memory card, 16bit colour. Too bad that applications did not make use of it.

So I went for Linux. There were two options: Zaurus or iPaq. Went for former one as it had keyboard. It was good choice.

Quickly started development of packages and joined OpenEmbedded team. Then became one of OpenZaurus developers. After year or something took over release maintenance and released few last versions. 3.5.4(.1) were the best tested releases of OZ ever — I had over hundred testers for each RC image and they provided installation reports, bug reports and fixes. And it had unified installer for whole range of devices (took me several months to get it polished and few guys added own tweaks). When Ångström distribution started I was the one who officially ended OpenZaurus development.

And all that was in free time. But in mean time I created my consulting company. CELF was my first customer 😉

One nice evening I got question on irc and due to that I left dark side of IT and went from PHP programming to embedded Linux full-time. OpenedHand had interesting projects and clients with many devices. Imagine operating system + kernel + Python + GStreamer in 16 megabytes of flash… And I managed to get it done. While working for them I used proper developer boards (not only customer devices) and there were funny moments…

When we worked with ST Microelectronics on NDK-15 (later replaced by NHK-15 from ST Ericsson) I had to merge two kernel trees from two separate teams. Took me 2 days of mangling 20-30MB diffs but got it done. There are people at ST-E which reminded me this during one of Linaro Connects ;D

Also on GUADEC 2007 when we presented new interface for Openmoko phones NDK-15 had to wait for me as no one at stand was able to get it running (U-Boot config needed changes).

But then Intel acquired OpenedHand… The craziest trip of my life was return from London to my parents place. For three months I even had email but never used it due to problems with Intel corporate network and Linux (do not ask).

Next was Bug Labs and their BUG device. I cleaned their Poky trees, migrated to latest version and later to use OpenEmbedded directly. Less challenges but I also had few other customers at that time to keep me busy. Some of them were OH customers before and went to me for help.

Time passed, 2010 came. One day Canonical made another attempt to seduce me and this time I decided that it looks like good opportunity so I accepted. Sent BUG 2.0 prototype back to NYC and few weeks later I made crazy train trip to small nowhere near Brussels to meet my new coworkers from NewCore. 1-2 weeks later we got our current name: Linaro.

Total change… From embedded devices to ‘Yes, it is ARM. So what?’ kind as we support(ed) devices powerful enough to run normal desktop software. Many changes for me — from OpenEmbedded where you can (cross) build everything in few hours to Ubuntu packaging where sending package for inclusion into archive meant few hours of buildd queue and then few of build. But I learnt a lot here and met another set of hackers including grey beards ones 😉

And all that because I bought Sharp Zaurus SL-5500 nine years ago…

What interest me in ARM world

When I published my last post about ARM boards there were many questions and suggestions with interesting devices. Thank You all for it.

But there were also suggestions about ARM9 or ARM11 based devices. So I decided that it is good time to write what interest me now in ARM world.

But first some inventory. I had/used/have several devices with ARM cpu:

  • StrongARM (armv4) one:

    • Sharp Zaurus SL-5500 (which took me to ARM world)
  • ARM920 (armv4t) ones:

    • Openmoko GTA01 bv3, bv4 (s3c2410)
    • EDB9301 (EP9301 cpu)
    • Sim-One (EP9307)
  • ARM926 (armv5te) ones:

    • Sharp Zaurus sl-5600 (pxa250)
    • Sharp Zaurus c760/sl-6000 (pxa255)
    • Sharp Zaurus sl-c3000 (pxa272)
    • Sheevaplug (kirkwood)
    • Atmel devboards (at91sam9263, at91sam9m10)
    • ST-Microelectronics/ST-Ericsson NDK-15, NHK-15 (st88n15)
    • Nokia 770 (omap1710)
    • Linksys NSLU2 (ixp425 iirc)
  • ARM1136 (armv6) ones:

    • Nokia N810 (omap2430)
    • Bug r1.0, r1.2 (i.mx31)
  • Cortex-A8 (armv7a) ones:

    • Beagleboard B7, B7, C3 (omap3430)
    • Nokia N900 (omap3430)
    • Nexus S (exynos3)
    • Genesi Efika MX Smartbook (i.mx51)
    • Freescale Quickstart (i.mx53)
  • Cortex-A9 (armv7a) ones:

    • Pandaboard EA1, A1 (omap4430)
    • Archos G9 80 (omap4430)

All of that during last 8 years. Most of my ARM live so far was around ARM926 based devices (some of them still can not be listed here) and I do not want to go there again. Kirkwood core was fastest one with 1.2GHz clock and 512MB of RAM it was really fast machine. I only missed Serial ATA in my Sheevaplug (rev 1.0) but even with hard drive on USB it was nice improvement.

Then I played a bit with ARM11 processors. Ok, they were faster than most of ARM9 cpus but I already had experience with Sheevaplug. And after few months first Cortex-a8 board landed on my desk — I got Beagleboard B7 from Bug labs as test platform for their new device. This was improvement!

I still remember my reaction when connected it to normal LCD monitor and saw it used at 720p resolution (1680×1050 was a bit hard for omap3). Moved to Nokia N900 few months later and found that fast cpu means nothing when paired with slow storage and not enough memory for system.

So today I prefer to not look below Cortex-A9 (or comparable cores like ones from Qualcomm or Marvell). Hope to play one day with Cortex-A5 (which should replace ARM926 one day) just to see how low-end armv7a cpu behave.

And wait for ARMv8 to hit market.

PandaBoard: my story

It was 24th March 2010 when one friend asked me do I want to be added to beta testers list for new omap hardware. One of questions was “what would you like to have on board” so I replied:

  • hdmi out (does not care much about vga/svideo/composite out)
  • 2xSD slots (SD or microsd type)
  • ethernet (but rather not on usb)
  • serial on db9/icd10 + serial/jtag by miniusb (think sheevaplug)
  • OTG is not needed but can be present
  • BT would be nice but not required as I have 5 micro dongles here
  • few usb ports — if possible (not omap3530) on more then one hub
  • few leds (multicolor?) would be nice (bug 2.0 has 2xblue + 2xmulticolor)
  • few buttons including power/reset ones
  • and 5V 2.1/2.5mm power jack. I do not need power-on-otg because it require 500mA ports
  • onboard lcd+ts is not needed for me
  • ah… and mounting holes like in beagleboard so board can be mounted anywhere
  • connector with i2c/spi/gpio/etc/etc
  • I missed audio in/out
  • battery for rtc

And suggested to place most of connectors on 2 edges as it helps to organize desk. Atmel’s at91sam9m10 was given as example cause it has all connectors on top and left edge.

And time passed… At UDS-M TI people said that there will be cheap OMAP4 based board named PandaBoard. During dinner (later same day) I got added second time to early adopters list. I wonder how Rob Clark reacted when he saw me on a list already 😀

And again time passed… Ubuntu/ARM people were playing with prototypes of PandaBoard (ES1.0, ES2.0 6-layer etc) and I had occasion to play with boards during Ubuntu/Linaro platform sprint in Prague. It looked nice (if you did not looked at ES1.0 one) and was more or less working fine.

And finally at 15th September I was told that at the end of month there will be production run from which several boards will be shipped to early adopters and few selected projects. Board travelled half of the world, then got back to US and at the end of UDS-N I got it.

Arrived home, powered BeagleBoard C3 off and started to assemble new board. Panda got several accessories connected:

  • +5V 3.5A power supply
  • powered USB hub
  • small USB keyboard
  • wireless USB mouse
  • 20″ LCD monitor with 1680x1050px resolution (this is also connected to my desktop)
  • 320GB Serial-ATA hard drive in SATA->USB enclosure

Also connected Ethernet, serial (by usb-serial dongle + 2 usb extenders) and used one of floating SD cards to have place for bootloaders and kernel. Config is much nicer then it was when I used BeagleBoard.

As operating system I am using Ubuntu 11.04 ‘natty’ as this is current development version and I have some things to check under it. Anyway I plan to move backwards and install 10.10 ‘maverick’ as primary system cause this will allow me to test omap4 hardware acceleration of graphics and audio/video decoding.

What I am using it for? Package building and testing. So far rebuilt whole KDE4 but it was segfaulting all the time on EfikaMX Smartbook so I am waiting for official ones (as there are some things to fix there first).

PandaBoard: Beagleboard XM killer?

It was known since previous UDS that there will be OMAP4 based PandaBoard available for developers. And some time ago was started (for now with temporary website). Boards are still not available at distributors but there are some of them in different projects (like Ubuntu/ARM), some are on a way to new users (mine for example).

When final price was announced many people said that PandaBoard is BeagleBoard XM killer due to same (179USD) price. But is it? Let have a look.

First group of users for such boards are software developers. If they do not work for hardware companies then usually want to get more power for same price. So they will choose PandaBoard.

Second group would be companies which want to produce own hardware based on OMAP3/4. Here it depends on how soon OMAP4 chips will be available in small orders. As OMAP3 can be bought now and BBXM is available to buy many will choose it as this allow to get own hardware ready to market in less then year with having working platform for own developers so final device will start with ready software. One of such is BUG 2.0 which I used at prototype phase. It was designed after using BeagleBoards with BUGBoard extension as base for hardware development.

And Beagleboard XM is available to buy today — with fast CPU, 512MB ram, Ethernet, few USB ports it is big update to previous versions. I never used it — BB C3 is still my primary ARM development system. But in 2-3 weeks situation will change and BB will meet another C3 and one B7 versions in a box due to arrival of PandaBoard.

What makes a good developer board?

During FOSDEM 2010 I had discussion with Ulf Samuelsson from Atmel and few other guys about developer boards. What is required on them and what should be avoided. Some time later I had a talk with one person about new OMAP3 based board and what I would like to see on it. So I decided to write something in public.

So far I used mostly ARM developer boards from ST Microelectronics/ST Ericsson, Atmel, Cirrus Logic, Intel, Simple machines, Bug Labs, Texas Instruments. Some were better then others etc. But what ideal developer board should have? Let me try to create a list:

  • 2 serial ports (one can be null modem, second should have RTS/CTS/DTR lines)
  • working Ethernet not placed on USB bus (so it works when USB does not)
  • powered USB host port (more then one would be great)
  • USB device port
  • JTAG connector
  • one power input — +5V or +12V — other should be forbidden as those ones can be taken from PC PSU which can power multiple devices at same time
  • SD/MMC slot — even if it is over slow SPI (like on Sim.One — 250KB/s max)
  • GPIO pins
  • I²C bus
  • SPI bus
  • keypad with Up/Down/Left/Right + Enter at least
  • easily reachable reset button (pins to short are acceptable as micro switch can be put on them)
  • few LEDs controlled by system
  • all connectors put on one or two edges of board — top one + one of side ones are ok (Atmel at91sam9m10-ekes for example)
  • mounting holes (so board can be mounted on A5 sheet holder for presentation at stand)
  • backup battery for RTC
  • U-Boot
  • quite fresh kernel (not NHK-15 due to 2.6.20 kernel which is 3 years old now)

What to avoid:

  • female serial port connectors (Atmel NGW100) — most devs will find 3 null modem cables before straight one
  • flat cables which connect “debug boards” with main board (Openmoko phones, NHK-15 from ST Ericsson)
  • RJ45 connectors for serial console (Sim.One) — DB9 or properly done USB->RS232 adapter on-board are best
  • placing connectors on all edges (BeagleBoard — but it had to be cheap)
  • non standard bootloaders (U-Boot is what I prefer)
  • Ethernet on USB (Bug 2.0) — it is hard to use when you have problems with USB Host
  • jumpers (Atmel boards)
  • non standard connectors (Bug r1.2 and it’s Handylink crap — next versions use iPod connector which is easier to use)

I am trying to not cover should developer board contain display with touchscreen or not as it depends on type of board. But if screen is present then more then QVGA (320×240) would be nice (WVGA anyone?). Some kind of video out connector can also be used but would be nice to have one of VGA/DVI/HDMI so normal PC monitor can be used — Composite video and S-Video require searching for some kind of TV…

Which boards are my favorites? There are few:

  • FriendlyARM with WVGA screen — cheap product which gives access to everything needed to start with embedded Linux
  • Atmel AT91SAM9263-EK — my first own developer board
  • BeagleBoard Cx — has own problems but I like the power of it

Another job change

Since OpenedHand was acquired by Intel I worked with few customers. The biggest one was Bug Labs Inc. with which I spend lot of time on hacking Poky Linux and OpenEmbedded to make their BUG devices prosper in hacking community.

Thanks to developers there Java land is not so strange for me (not that I started to like it) and I know which projects exist in that area. Many of changes done for BUG landed in OpenEmbedded metadata and helped other projects. Last release of Poky ‘pinky’ (stable branch which we used with R1.4) was done due to out improvements and bug fixes (we got credit on them). It was great time and I really enjoyed it.

The open source companies have this nice feeling — developers work on code to make it better and better (as other people look at code) and are friendly to own employees and contractors.

What next? My first job where my experience from OpenEmbedded will be used in a project which does not derive from it. Yes — no OpenEmbedded, no Poky Linux. But it will be GNU/Linux still and still ARM architecture.

In 3 weeks from now I will work for Canonical as Foundations OS Engineer. The goal is to make Ubuntu/ARM fly on supported devices (armv7a only). This will be full time job but I hope that will have possibility to do some OE related things from time to time.

Transplanting bugLCDs

As you know from previous post about BUG 2.0 new device require using of new LCD module. I got one but without case.

Using screen case less is asking for problems — think of all short circuits it can do… So I decided to move new LCD into case of old LCD module (I have two of them).

So first let’s look at case donor: old LCD module. Front view:

Back view with all tools required (module was signed for FOSDEM as we had two bugs there and I wanted to know which modules/bugs are mine):

And dissasembled:

PCB view (notice plastic frame between PCB and LCD — it fits new module too but I isolated screen instead):

New LCD on left, old one on right. Large connector is for screen, smaller for touchscreen. And they are different then in old one:

Modules with screens attached. Old bugView modules also were using Sharp screens and I did not looked at my second one.

First problem during mounting — BUG 2.0 video slot is reversed… But few minutes with sharp knife solved that problem.

And this is how result looks. On left side is VonHippel module fitted into BMI adapter, on right new LCD in case from old one:

Yes — there are two pieces of isolation tape on board. Those blue LEDs are too bright so I had to make them less annoying.

Now I have screen working, X11 starts — time for touchscreen driver.

BUG 2.0 arrived

Some time ago there was decision that BUG 1.x will not be supported with next version of BUG Linux. As a result I ended in situation when I worked on handling device which I never saw.

It was not first time — in OpenedHand times I had this quite often but it is not a problem because my “hardware park” covers nearly whole ARM family:

  • armv4t (s3c2410 in Openmoko GTA01, EP3907 in Sim.One)
  • armv5te (at91sam9263, at91sam9m10 on Atmel boards, PXA255 in Zaurus c760, Sheeva in Marvell Sheevaplug, omap1510 in Nokia 770 tablet, ST88n15 in NHK-15)
  • armv6 (omap24xx in Nokia N810, i.mx31 in BUG 1.2/1.3)
  • armv7a (omap3530 in BeagleBoard B7/C3 and in Nokia N900)

So I am able to test binaries on other hardware or even in QEMU.

But few days ago I got information that developer version of BUG 2.0 will be sent to me. To make me more happy I ordered few books from Amazon to get them with package (inside US I got free posting). And today package was delivered by FedEx courier (their tracking page said Friday as delivery day).

Package reminds why recycling is easy: UPS package from Amazon (the one with books which I ordered) was repacked and got FedEx papers:

But box itself is not interesting — stuff in matters. After taking books out I got lot of packing bubbles and my eyes were presented with first level of things:

And then second one:

Everything unboxed:

And again but this time without packaging. From left-top to right-left:

  • BUGduino module
  • camera module
  • OMAP3 video module with HDMI and VGA outputs
  • new LCD module
  • LCD screen with touchscreen (for LCD module)
  • battery
  • BUG 2.0 dock with serial (miniUSB), Ethernet, USB host, JTAG connectors
  • BUG 2.0 rev. A
  • two BMI adapters
  • BMI cable extender

Look at new modules. First goes BUGduino which is Arduino thing with BUG connection. I do not know how it works but I knew that John Connolly did some programming for it.

New LCD module — QVGA like before.

Video module with HDMI and VGA outputs. This one is BUG 2.0 only as it uses OMAP3 signals and needs BMI slot with video signals. Yes, new BUG has only one slot for video — two screens configuration is not possible anymore. But hey — you can even connect 150″ LCD 😉

New camera module. I do not remember how many Mpx it has (old one had 2Mpx).

BUG 2.0 itself. Notice two microSD slots — one will be used for system, second is for user. There are just two buttons now, no LCD, no joystick. Also buzzer got removed in favour of headphone connector.

New hardware requires new BUGDock. What got changed? Serial is now present as miniUSB connector instead of DB9 so is easier to use with today computers (not everyone has 7 serial ports in desktop). Power and headphones connectors were removed because on-board ones are reachable. And JTAG connector is present. To tell the true I like old dock more then current one. But thats mostly because of angle connector instead of flat one. Anyway before BUG 2.0 will hit market there will be new dock for it.

And thats how system can look. BUG 2.0 with Dock and two modules connected by adapters.

Now I am waiting for Bug Labs guys to appear in the office to get informations how to boot it ;D

My expansion board for BeagleBoard

Using BeagleBoard as development platform means lot of cables as I need:

  • serial cable
  • power cable for BeagleBoard
  • USB hub
  • power cable for USB hub
  • USB Ethernet dongle
  • Ethernet cable

So by default my BB setup looks like this (here with B7 version so OTG used instead of EHCI):

BeagleBoard in a box

I decided that I need to do something with it. The idea was to make simple expansion board which will just gather all components on board connected to BeagleBoard. First sketch was this (without keeping aspect etc — just elements):

BeagleBoard expansion sketch

I presented it to few people with better electronic experience then mine and got few nice suggestions.

After few days of collecting elements and soldering/glueing I got first version working:


USB Ethernet dongle is glued to PCB, USB Hub needs mounting and it’s power cable has to be replaced by better solution. But the good thing is that I have power, serial, Ethernet connectors on one side without any special cables:


When BeagleBoard is placed on top all what is needed to get it working is connecting 3 cables: power, serial and EHCI:


Audio, S-Video and HDMI connectors are on same way as rest so all cables are in one side:


The problem appears when reversed BeagleBoard has to be mounted on top… I have such one with BeagleBUG extension. Extra distances helps but cables are not so nice then:


Notice that SD card in this config is harder to reach.


Serial cable is twisted but still do job.


Audio and video connectors are easy to reach but on other side of board ;(


Yes… that hub needs replacing — note ugly way of attaching power to it.

Anyway this works and allows me to have all cables in one place instead of jungle of them.

Poky Linux 3.1.2 released

Yesterday Richard Purdie released maintenance version of Poky ‘pinky’ branch. It contains mostly fixes to get it into buildable state in all distributions released since 3.1.1 was done.

It got over 50 changes during 1.5 year of development. Most of them were done by me as part of my work for Bug Labs company and their BUG Linux distribution. As policy of handling fixes requires to make them also in development branch it was more then just make a fix for ‘pinky’ — I also had to take care of ‘elroy’ (which had to be next stable version) and ‘master’. Some time ago support for ‘elroy’ was dropped anyway.

If you look at ‘git log’ output you will notice few authors other then just me. Thats because if fix was present in other sources such like Poky ‘master’ or OpenEmbedded I cherrypicked it and adapted to make it apply with keeping original author credits.

Did I add something new into it? Yes, few things were added:

  • SPLASH support in task-poky so you can use own bootsplash tool instead of psplash
  • warning for ‘/proc/sys/vm/mmap_min_addr’ not being set to 0 (which would break qemu)
  • Python 2.6 compatibility
  • BP/BPN variables which were used in Jalimo repository which we use as one of overlays
  • automatic resizing of ext2/3 images if rootfs do not fit in default size

What will future bring? I hope that new stable branch for Poky will be created in next few months so developers will be able to switch. I know that some companies did a move from ‘pinky’ to ‘master’ (or snapshot of it + own changes). We at Bug Labs are moving into OpenEmbedded ‘stable/2009’ as we need newer software and want some functionality which is not present in ‘pinky’.