Hotplug in VM. Easy to say…

You run VM instance. Nevermind is it part of OpenStack setup or just local one started using Boxes, virt-manager, virsh or other that kind of fronted to libvirt daemon. And then you want to add some virtual hardware to it. And another card and one more controller…

Easy to imagine scenario, right? What can go wrong, you say? “No more available PCI slots.” message can happen. On second/third card/controller… But how? Why?

Like I wrote in one of my previous posts most of VM instances are 90s pc hardware virtual boxes. With simple PCI bus which accepts several cards to be added/removed at any moment.

But not on AArch64 architecture. Nor on x86-64 with Q35 machine type. What is a difference? Both are PCI Express machines. And by default they have far too small amount of pcie slots (called pcie-root-port in qemu/libvirt language). More about PCI Express support can be found in PCI topology and hotplug page of libvirt documentation.

So I wrote a patch to Nova to make sure that enough slots will be available. And then started testing. Tried few different approaches, discussed with upstream libvirt developers about ways of solving the problem and finally we selected the one and only proper way of doing it. Then discussed failures with UEFI developers. And went for help to Qemu authors. And explained what I want to achieve and why to everyone in each of those four projects. At some point I had seen pcie-root-port things everywhere…

Turned out that the method of fixing it is kind of simple: we have to create whole pcie structure with root port and slots. This tells libvirt to not try any automatic adding of slots (which may be tricky if not configured properly as you may end with too small amount of slots for basic addons).

Then I went with idea of using insane values. VM with one hundred PCIe slots? Sure. So I made one, booted it and then something weird happen: landed in UEFI shell instead of getting system booted. Why? How? Where is my storage? Network? Etc?

Turns out that Qemu has limits. And libvirt has limits… All ports/slots went into one bus and memory for MMCONFIG and/or I/O space was gone. There are two interesting threads about it on qemu-devel mailing list.

So I added magic number into my patch: 28 — this amount of pcie-root-port entries in my aarch64 VM instance was giving me bootable system. Have to check it on x86-64/q35 setup still but it should be more or less the same. I expect this patch to land in ‘Rocky’ (the next OpenStack release) and probably will have to find a way to get it into ‘Queens’ as well because this is what we are planning to use for next edition of Linaro Developer Cloud.

Conclusion? Hotplug may be complicated. But issues with it can be solved.

Running 32-bit ARM virtual machine on AArch64 hardware

It was a matter of days and finally all pieces are done. Running 32-bit ARM virtual machines on 64-bit AArch64 hardware is possible and quite easy.


  • AArch64 hardware (I used APM Mustang as usual)
  • ARM rootfs (fetched Fedora 22 image with “virt-builder” tool)
  • ARM kernel and initramfs (I used Fedora 24 one)
  • Virt Manager (can be done from shell too)


Start “virt-manager” and add new machine:

Add new machine

Select rootfs, kernel, initramfs (dtb will be provided internally by qemu) and tell kernel where rootfs is:

Storage/boot options

Then set amount of memory and cores. I did 10GB of RAM and 8 cores. Save machine.

Let’s run

Open created machine and press Play button. It should boot:

Booted system

I upgraded F22 to F24 to have latest development system.

Is it fast?

If I would just boot and write about it then there will be questions about performance. I did build of gcc 5.3.1-3 using mock (standard Fedora way). On arm32 Fedora builder it took 19 hours, on AArch64 builder 4.5h only. On my machine AArch64 build took 9.5 hour and in this vm it took 12.5h (slow hdd used). So builder with memory and some fast storage will boost arm32 builds a lot.

Numbers from “openssl speed” shows performance similar to host cpu:

The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md2               1787.41k     3677.19k     5039.02k     5555.88k     5728.94k
mdc2                 0.00         0.00         0.00         0.00         0.00 
md4              24846.05k    81594.07k   226791.59k   418185.22k   554344.45k
md5              18881.79k    60907.46k   163927.55k   281694.58k   357168.47k
hmac(md5)        21345.25k    69033.83k   177675.52k   291996.33k   357250.39k
sha1             20776.17k    65099.46k   167091.03k   275240.62k   338582.71k
rmd160           15867.02k    42659.95k    88652.54k   123879.77k   140571.99k
rc4             167878.11k   186243.61k   191468.46k   192576.51k   193112.75k
des cbc          35418.48k    37327.19k    37803.69k    37954.56k    37991.77k
des ede3         13415.40k    13605.87k    13641.90k    13654.36k    13628.76k
idea cbc         36377.06k    38284.93k    38665.05k    38864.71k    39032.15k
seed cbc         42533.48k    43863.15k    44276.22k    44376.75k    44397.91k
rc2 cbc          29523.86k    30563.20k    30763.09k    30940.50k    30857.44k
rc5-32/12 cbc        0.00         0.00         0.00         0.00         0.00 
blowfish cbc     60512.96k    66274.07k    67889.66k    68273.15k    68302.17k
cast cbc         56795.77k    61845.42k    63236.86k    63251.11k    63445.82k
aes-128 cbc      61479.48k    65319.32k    67327.49k    67773.78k    66590.04k
aes-192 cbc      53337.95k    55916.74k    56583.34k    56957.61k    57024.51k
aes-256 cbc      46888.06k    48538.97k    49300.82k    49725.44k    50402.65k
camellia-128 cbc    59413.00k    62610.45k    63400.53k    63593.13k    63660.03k
camellia-192 cbc    47212.40k    49549.89k    50590.21k    50843.99k    50012.16k
camellia-256 cbc    47581.19k    49388.89k    50519.13k    49991.68k    50978.82k
sha256           27232.09k    64660.84k   119572.57k   151862.27k   164874.92k
sha512            9376.71k    37571.93k    54401.88k    74966.36k    84322.99k
whirlpool         3358.92k     6907.67k    11214.42k    13301.08k    14065.66k
aes-128 ige      60127.48k    65397.14k    67277.65k    67428.35k    67584.00k
aes-192 ige      52340.73k    56249.81k    57313.54k    57559.38k    57191.08k
aes-256 ige      46090.63k    48848.96k    49684.82k    49861.32k    49892.01k
ghash           150893.11k   171448.55k   177457.92k   179003.39k   179595.95k
                  sign    verify    sign/s verify/s
rsa  512 bits 0.000322s 0.000026s   3101.3  39214.9
rsa 1024 bits 0.001446s 0.000073s    691.7  13714.6
rsa 2048 bits 0.008511s 0.000251s    117.5   3987.5
rsa 4096 bits 0.058092s 0.000945s     17.2   1058.4
                  sign    verify    sign/s verify/s
dsa  512 bits 0.000272s 0.000297s   3680.6   3363.6
dsa 1024 bits 0.000739s 0.000897s   1353.1   1115.2
dsa 2048 bits 0.002762s 0.002903s    362.1    344.5
                              sign    verify    sign/s verify/s
 256 bit ecdsa (nistp256)   0.0005s   0.0019s   1977.8    538.3
 384 bit ecdsa (nistp384)   0.0015s   0.0057s    663.0    174.6
 521 bit ecdsa (nistp521)   0.0035s   0.0136s    286.8     73.4
                              op      op/s
 256 bit ecdh (nistp256)   0.0016s    616.0
 384 bit ecdh (nistp384)   0.0049s    204.8
 521 bit ecdh (nistp521)   0.0115s     87.2

2 years of AArch64 work

I do not remember exactly when I started working on ARMv8 stuff. Checked old emails from Linaro times and found that we discussed AArch64 bootstrap using OpenEmbedded during Linaro Connect Asia (June 2012). But it had to wait a bit…

First we took OpenEmbedded and created all tasks/images we needed but built them for 32-bit ARM. But during September we had all toolchain parts available: binutils was public, gcc was public, glibc was on a way to be released. I remember that moment when built first “helloworld” — probably as one of first people outside ARM and their hardware partners.

At first week of October we had ARMv8 sprint in Cambridge, UK (in Linaro and ARM offices). When I arrived and took a seat I got information that glibc just went public. Fetched, rebased my OpenEmbedded tree to drop traces of “private” patches and started build. Once finished all went public at repository.

But we still lacked hardware… The only thing available was Versatile Express emulator which required license from ARM Ltd. But then free (but proprietary) emulator was released so everyone was able to boot our images. OMG it was so slow…

Then fun porting work started. Patched this, that, sent patches to OpenEmbedded and to upstream projects and time was going. And going.

In January 2013 I started X11 on emulated AArch64. Had to wait few months before other distributions went to that point.

February 2013 was nice moment as Debian/Ubuntu team presented their AArch64 port. It was their first architecture bootstrapped without using external toolchains. Work was done in Ubuntu due to different approach to development than Debian has. All work was merged back so some time later Debian also had AArch64 port.

It was March or April when OpenSUSE did mass build of whole distribution for AArch64. They had biggest amount of packages built for quite long time. But I did not tracked their progress too much.

And then 31st May came. A day when I left Linaro. But I was already after call with Red Hat so future looked quite bright ;D

June was month when first silicon was publicly presented. I do not know what Jon Masters was showing but it probably was some prototype from Applied Micro.

On 1st August I got officially hired by Red Hat and started month later. My wife was joking that next step would be Retired Software Engineer ;D

So I moved from OpenEmbedded to Fedora with my AArch64 work. Lot of work here was already done as Fedora developers were planning 64-bit ARM port few years before — when it was at design phase. So when Fedora 15 was bootstrapped for “armhf” it was done as preparation for AArch64. 64-bit ARM port was started in October 2012 with Fedora 17 packages (and switched to Fedora 19 during work).

My first task at Red Hat was getting Qt4 working properly. That beast took few days in foundation model… Good that we got first hardware then so it went faster. 1-2 months later and I had remote APM Mustang available for my porting work.

In January 2014 QEmu got AArch64 system emulation. People started migrating from foundation model.

Next months were full of hardware announcements. AMD, Cavium, Freescale, Marvell, Mediatek, NVidia, Qualcomm and others.

In meantime I decided to make crazy experiment with OpenEmbedded. I was first to use it to build for AArch64 so why not be first to build OE on 64-bit ARM?

And then June came. With APM Mustang for use at home. Finally X11 forwarding started to be useful. One of first things to do was running firefox on AArch64 just to make fun of running software which porting/upstreaming took me biggest amount of time.

Did not took me long to get idea of transforming APM Mustang (which I named “pinkiepie” as all machines at my home are named after cartoon characters) into ARMv8 desktop. Still waiting for PCI Express riser and USB host support.

Now we have October. Soon will be 2 years since people got foundation model available. And there are rumors about AArch64 development boards in production with prices below 100 USD. Will do what needed to get one of them on my desk 😉

Palm SDK has leaked

As some of my readers already noticed there is WebOS SDK available for Palm Pre smartphone. Or rather “available” as it is leaked edition not normal release. I will not describe how to get it (using Google is enough) but rather how to get it run under Linux and a bit about what is inside.

The bad part is that SDK is (so far) available only as MS Windows binary. I did not tried it with WINE but used XP Pro installation on one of my machines to install it. There are two additional installations to be done first — VirtualBox 2.2.x is required by Palm SDK to be installed and Safari is required for “Palm Inspector” tool (which I did not got to work anyway).

After installation two important files: “palm_emulator_sdk_47.vmdk” and “palm_emulator_sdk_47.iso” can be found in “Documents and Settings/$USER/.VirtualBox/” subdirectories (they are also available in “Program Files/Palm/SDK/share/emulator/sdk47/” directory). There is also “Palm Emulator” icon on desktop which makes use of them.

So back to Linux. VirtualBox hard drive image and ISO needs to be copied to Linux machine and given for VirtualBox (configuration of virtual machine can be copied from MS Windows too). It is also possible to use QEMU to boot into SDK but it can be harder to find one with working mouse emulation. This is where their changes to QEMU or VirtualBox would be handy to get — adding 320x480px resolution to QEMU is few minutes work anyway (needs to change sources of vga bios and replace system one with tweaked copy).

What can be seen after boot? First error which I hope will be fixed in final release — “vga=864” kernel parameter results in “unknown video mode” message. Anyway other modes are working and I suggest 640x480x16 as it has the same height as Palm Pre screen. There is a one problem due that — screen has wrong calibration so it is hard to use UI.

Some applications are coded in ugly style — seems to have 320x480px resolution hardcoded (Google Maps, Video player, Dialer). But there are also others which resize properly (Contacts, desktop). To play with them few key shortcuts are useful to know:

  • TAB — runs launcher
  • Escape — “minimize” application to a card

Looks like other keys launch search tool — I have to admit that I did not searched yet for documentation.

But what is inside of image and why it works? Image was built for “qemux86” device by using OpenEmbedded build system — no new patches added. There is SSH daemon working in emulator so it is possible to login remotely and check what is in system. There are 697 packages installed (285 of them being kernel modules). Image looks like it was built on a same system as WebOS 1.0.3 image for Palm Pre (about which I already wrote few days ago).

Ah, I would forgot… There are few JavaScript examples in SDK if someone wants to know how to make “hello world” for WebOS.

Final thoughts — WebOS looks interesting and I would like to play more with it on real device. The bonus part is that it is even able to run classic PalmOS applications (but this is with 3rdparty application). Too bad that there is no GSM version yet.

Palm Pre and OpenEmbedded

Nearly month ago we got information that Palm company uses OpenEmbedded to build software for their Palm Pre smartphone. Two weeks later I read on Matthew Garret blog that there is root filesystem image available on one of Palm websites.

What is inside? Many interesting things — you can read about it on Matthew’s post so I will not repeat. If someone wants to look inside then please take a look at /usr/lib/ipkg/ directory. It contains packaging informations which shows which packages were installed, which files were used etc. It is clearly visible that they used OpenEmbedded with own overlay separated — OE was in /home/reviewdaemon/projects/nova/oe/foss and Palm overlay in oe/palm directory. The missing thing is IP address of “” server which is used to store additional sources (propertiary and free ones). What else we can find? Build information which says that rootfs was built at 2009.05.22 14:00:49. And according to audio setup files development started in September 2008 (if not earlier).

I wonder what kind of board they used as developer kit (they also used ARM Versatile emulation in QEMU).

UPDATE: Palm SDK uses “qemux86” device from OpenEmbedded to run WebOS.

Merging stuff from Poky into OpenEmbedded

Recently I started merging interesting stuff from OpenedHand’s Poky into OpenEmbedded. Now, with OE using GIT as storage for metadata it is much, much easier then it was in Monotone times.

I have over 3 thousands of revisions exported (using git format-patch) from Poky and I am reviewing them and adapt to add into OE. Useful ones are changed by simple shell script which adapts paths, change authors informations (I use Poky via git-svn so no real names/emails) and adds “(from Poky)” message to end of patch description. Then just git am and patch lands in OpenEmbedded.

For now I added newer APT and DPKG package tools, newer QEMU (not the latest but working with ARMv6/v7 instructions), U-Boot mkimage tool which does not use lot of Openmoko patches (just one is needed), Shared MIME Info which does not need any processing on target device and some tweaks here or there.

Next in queue are Maemo4 cleaned recipes (Diablo ones), binary locales for Angstrom powered ARMv6/7 based machines and miscellaneous tweaks or updates.

Nokia N8x0 emulation part II

My previous post about Nokia N800 tablet emulation became one of popular ones. On LinuxTag I shown Maemo booting in QEmu and it was met with nice response from community. But the problem remained — how to boot it when config.mtd which I used was not distributable…

Yesterday I solved that part. After studying how Maemo boots and why does QEmu restarts with wrong config.mtd I grabbed that partition from my N810 and tried again. This time OS2008/Chinook booted fine 🙂

What is needed? Tablet needs to have “no-lifeguard-reset” flag set. IT can be done by using flasher as this is one of R&D flags. I had it set on my N810 because I did experiments with booting from internal SD card in past.

Maemo OS2008 (Chinook) on emulated N800 - first screen Maemo OS2008 (Chinook) on emulated N800 - desktopMaemo OS2008 (Chinook) on emulated N800

Thanks to Faheem Pervez (more widely known as “qwerty12”) who sent me config.mtd dumps (without R&D and with “no-lifeguard-reset”) from his N800 I was able to confirm that this is all what is needed.

Next step will be updating qemu to more recent revision to get N810 emulation (which is present in HEAD) and getting Diablo booted.

UPDATE: Diablo booted on emulated N800 and N810:



Nokia N810 emulation is more useful as there is a keyboard attached so no need for use of onscreen input methods. There are some things to remember anyway:

  • Alt(Gr) behave like Fn (with sticky status)
  • no CapsLock (but Shift works like on N810 so no big loss)
  • no numeric row — to get “5” press “Alt+t” like on N810
  • some of other keys are also in weird places
  • Right Shift does not work (N810 has 2 Left Shifts)

NOTE: This is QEmu HEAD — no extra patches were needed to boot Chinook on emulated N800. To boot Diablo “hw/nseries.c” file needs to be edited to change partition info (initfs is twice as big compared to Chinook).

LinkedIn and checking email addresses

Some months ago Ross Burton wrote about checking email authors on LinkedIn:

I generally thought that LinkedIn was pretty useless for people like me. I have a community of like-minded associates available via Planet Gnome and so on, so apart from collecting friends it is pretty useless.

But recently it’s been becoming quite useful. For large companies it generally appears to be company policy that contact with open source projects is done via anonymous email domains, like GMail. This obviously makes it tricky to guess where someone is from when they appear on a mailing list… but LinkedIn to the rescue. Search for a name and hey presto, their CV!

Today I got interesting mail… It was technical question about my blog post “Recent Poky changes” where I wrote about updating QEMU in Poky to handle ARMv6/v7 rootfs. Question like question — but why it came to my OpenedHand email instead of private one? This is private blog…

The interesting part was mail author. As it came from private account which does not tell me anything so I did search on the LinkedIn. Result was nice — one of PDA vendors. I wonder when they will release phone with ARMv6 processor.

Anyway I answered and decided to share answer with other people which want to run ARMv6 Linux under QEMU. So to get it done few things are needed:

  1. recent Subversion snapshot of QEMU
  2. patch for Linux kernel to enable ARMv6 for ARM Integrator PB devboard
  3. ARMv6 rootfs
  4. some time to configure kernel

All those steps can be handled with Poky (or OpenEmbedded) of course. Kernel for “qemuarm” device use properly patched kernel — just kernel config change is needed to enable ARMv6 support. To get ARMv6 rootfs you can adapt “qemuarm” machine config to use proper optimizations (“” instead of “”).

Nokia N800 emulation

Few days ago Nokia N800 tablet emulation was released into public. Richard integrated it into Poky so now we have QEMU which can be used not only to test ARM images on ARM Versatile or Sharp Zaurus but also to run on Nokia N800 tablet. Of course it is not limited to Poky images — Maemo boots very nicely on it 🙂


Booting Poky is easy: runqemu nokia800 after building of “poky-image-sato” for “nokia800” machine. After few minutes (needed to create NAND Flash image and boot into JFFS2 rootfs) Poky desktop appears:

Poky on emulated N800 - first screen Poky on emulated N800 Poky on emulated N800 - Dates application


Booting Maemo takes few steps more now (will be improved).

  1. Edit “scripts/poky-qemu-internal” script and in line 154 change KERNELCMDLINE to boot from “/dev/mtdblock3” instead of “/dev/mtdblock4” as Poky do not use Maemo’s “initfs”.
  2. Get copy of “config” flash partition from N8x0 — simple “cat /dev/mtd1ro > config.mtd” is enough. Bad news: it does not work 🙁 And the one which works for me is not distributable as it does not came from device but was pre-generated somehow.
  3. Transfer it to the desktop.
  4. Grab OS2008 firmware image from Maemo website.
  5. Unpack firmware image to get kernel and images of “initfs” and “rootfs”.
  6. Use poky-nokia800-flashutil to generate NAND Flash image:
poky-nokia800-flashutil initfs.jffs2 maemo-image.qemuflash initfs
poky-nokia800-flashutil config.mtd   maemo-image.qemuflash config
poky-nokia800-flashutil rootfs.jffs2 maemo-image.qemuflash rootfs

Then “touch maemo-image” and run one command: poky-qemu zImage maemo-image to boot it.

Maemo OS2008 on emulated N800 - first screen Maemo OS2008 on emulated N800 - desktopMaemo OS2008 on emulated N800


Basic emulation works. There is no networking yet, DSP code is not emulated and few other limitations. But it is work in progress so expect improvements.

How to get it

Patch alone can be fetched from Poky repository.

Linux binaries of QEMU with N800 support can be built with Poky by bitbake qemu-sdk command. They will be also part of Poky Linux SDK.

UPDATE: poky-nokia800-flashutil instructions are fixed (thx to Yasser)

UPDATE: there is second part of that story.

ARMv6 booted in QEmu

Finally I booted ARMv6 rootfs in QEmu (used 2007-11-21 CVS one). All changes will be available in Poky in next few days.

OpenedHand Linux (Poky) 3.0+snapshot-20071206 qemuarm ttyAMA0

qemuarm login: root

root@qemuarm:~# uname -a
Linux qemuarm 2.6.23 #11 Fri Dec 7 19:59:57 CET 2007 armv6l unknown unknown GNU/Linux
root@qemuarm:~# cat /proc/cpuinfo
Processor       : ARMv6-compatible processor rev 3 (v6l)
BogoMIPS        : 524.28
Features        : swp half thumb fastmult vfp edsp java
CPU implementer : 0x41
CPU architecture: 6TEJ
CPU variant     : 0x1
CPU part        : 0xb36
CPU revision    : 3
Cache type      : write-through
Cache clean     : not required
Cache lockdown  : not supported
Cache format    : Harvard
I size          : 4096
I assoc         : 4
I line length   : 32
I sets          : 32
D size          : 65536
D assoc         : 4
D line length   : 32
D sets          : 512

Hardware        : ARM-Versatile PB
Revision        : 0000
Serial          : 0000000000000000