YADIBP

Let me introduce new awesome project: YADIBP. It is cool, foss, awesome, the one and only and invented here instead of there. And it does exactly what it has to do and in a way it has to be done. Truly cool and awesome.

Using that tool you can build disk images with several supported Linux distributions. Or maybe even with any BSD distribution. And Haiku or ReactOS. Patches for AmigaOS exist too!

Any architecture. Starting from 128 bit wide RUSC-VI to antique architectures like ia32 or m88k as long as you have either hardware or qemu port (patches for ARM fast models in progress).

Just fetch from git and use. Written in BASIC so it should work everywhere. And if you lack BASIC interpreter then you can run it as Python or Erlang. Our developers are so cool and awesome!

But let’s get back to reality — there are gazillions of projects of tool which does one simple thing: builds a disk image. And gazillion will be still written because some people have that “Not Invented Here” syndrome.

And I am getting tired of it.

Running 32-bit ARM virtual machine on AArch64 hardware

It was a matter of days and finally all pieces are done. Running 32-bit ARM virtual machines on 64-bit AArch64 hardware is possible and quite easy.

Requirements

  • AArch64 hardware (I used APM Mustang as usual)
  • ARM rootfs (fetched Fedora 22 image with “virt-builder” tool)
  • ARM kernel and initramfs (I used Fedora 24 one)
  • Virt Manager (can be done from shell too)

Configuration

Start “virt-manager” and add new machine:

Add new machine

Select rootfs, kernel, initramfs (dtb will be provided internally by qemu) and tell kernel where rootfs is:

Storage/boot options

Then set amount of memory and cores. I did 10GB of RAM and 8 cores. Save machine.

Let’s run

Open created machine and press Play button. It should boot:

Booted system

I upgraded F22 to F24 to have latest development system.

Is it fast?

If I would just boot and write about it then there will be questions about performance. I did build of gcc 5.3.1-3 using mock (standard Fedora way). On arm32 Fedora builder it took 19 hours, on AArch64 builder 4.5h only. On my machine AArch64 build took 9.5 hour and in this vm it took 12.5h (slow hdd used). So builder with memory and some fast storage will boost arm32 builds a lot.

Numbers from “openssl speed” shows performance similar to host cpu:

The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md2               1787.41k     3677.19k     5039.02k     5555.88k     5728.94k
mdc2                 0.00         0.00         0.00         0.00         0.00 
md4              24846.05k    81594.07k   226791.59k   418185.22k   554344.45k
md5              18881.79k    60907.46k   163927.55k   281694.58k   357168.47k
hmac(md5)        21345.25k    69033.83k   177675.52k   291996.33k   357250.39k
sha1             20776.17k    65099.46k   167091.03k   275240.62k   338582.71k
rmd160           15867.02k    42659.95k    88652.54k   123879.77k   140571.99k
rc4             167878.11k   186243.61k   191468.46k   192576.51k   193112.75k
des cbc          35418.48k    37327.19k    37803.69k    37954.56k    37991.77k
des ede3         13415.40k    13605.87k    13641.90k    13654.36k    13628.76k
idea cbc         36377.06k    38284.93k    38665.05k    38864.71k    39032.15k
seed cbc         42533.48k    43863.15k    44276.22k    44376.75k    44397.91k
rc2 cbc          29523.86k    30563.20k    30763.09k    30940.50k    30857.44k
rc5-32/12 cbc        0.00         0.00         0.00         0.00         0.00 
blowfish cbc     60512.96k    66274.07k    67889.66k    68273.15k    68302.17k
cast cbc         56795.77k    61845.42k    63236.86k    63251.11k    63445.82k
aes-128 cbc      61479.48k    65319.32k    67327.49k    67773.78k    66590.04k
aes-192 cbc      53337.95k    55916.74k    56583.34k    56957.61k    57024.51k
aes-256 cbc      46888.06k    48538.97k    49300.82k    49725.44k    50402.65k
camellia-128 cbc    59413.00k    62610.45k    63400.53k    63593.13k    63660.03k
camellia-192 cbc    47212.40k    49549.89k    50590.21k    50843.99k    50012.16k
camellia-256 cbc    47581.19k    49388.89k    50519.13k    49991.68k    50978.82k
sha256           27232.09k    64660.84k   119572.57k   151862.27k   164874.92k
sha512            9376.71k    37571.93k    54401.88k    74966.36k    84322.99k
whirlpool         3358.92k     6907.67k    11214.42k    13301.08k    14065.66k
aes-128 ige      60127.48k    65397.14k    67277.65k    67428.35k    67584.00k
aes-192 ige      52340.73k    56249.81k    57313.54k    57559.38k    57191.08k
aes-256 ige      46090.63k    48848.96k    49684.82k    49861.32k    49892.01k
ghash           150893.11k   171448.55k   177457.92k   179003.39k   179595.95k
                  sign    verify    sign/s verify/s
rsa  512 bits 0.000322s 0.000026s   3101.3  39214.9
rsa 1024 bits 0.001446s 0.000073s    691.7  13714.6
rsa 2048 bits 0.008511s 0.000251s    117.5   3987.5
rsa 4096 bits 0.058092s 0.000945s     17.2   1058.4
                  sign    verify    sign/s verify/s
dsa  512 bits 0.000272s 0.000297s   3680.6   3363.6
dsa 1024 bits 0.000739s 0.000897s   1353.1   1115.2
dsa 2048 bits 0.002762s 0.002903s    362.1    344.5
                              sign    verify    sign/s verify/s
 256 bit ecdsa (nistp256)   0.0005s   0.0019s   1977.8    538.3
 384 bit ecdsa (nistp384)   0.0015s   0.0057s    663.0    174.6
 521 bit ecdsa (nistp521)   0.0035s   0.0136s    286.8     73.4
                              op      op/s
 256 bit ecdh (nistp256)   0.0016s    616.0
 384 bit ecdh (nistp384)   0.0049s    204.8
 521 bit ecdh (nistp521)   0.0115s     87.2

Running VMs on Fedora/AArch64

There are moments when more than one machine would be handy. But AArch64 computers are not yet available in shop around a corner we have to go for other options. So this time let check how to get virtual machines working.

Requirements

For this I would use Fedora 22 on APM Mustang (other systems will be fine too). What else will be needed:

  • libvirtd running
  • virt-manager 1.1.0-7 (or higher) installed on AArch64 machine
  • UEFI for AArch64 from Gerd’s Hoffmann firmware repository
  • Fedora or Debian installation iso (Ubuntu does not provide such)
  • computer with X11 working (to control virt-manager)

UPDATE: starting from Fedora 23 UEFI package is in distribution repository. Packages are ‘edk2-arm’ for 32bit arm, ‘edk2-aarch64’ for 64bit arm and ‘edk2-ovmf’ for x86-64 architecture.

Is KVM working?

First we need to get KVM working — run “dmesg|grep -i kvm” after system boot. It should look like this:

hrw@pinkiepie-f22:~$ dmesg|grep -i kvm
[    0.261904] kvm [1]: interrupt-controller@780c0000 IRQ5
[    0.262011] kvm [1]: timer IRQ3
[    0.262026] kvm [1]: Hyp mode initialized successfully

But you can also get this:

[    0.343796] kvm [1]: GICV size 0x2000 not a multiple of page size 0x10000
[    0.343802] kvm [1]: error: no compatible GIC info found
[    0.343909] kvm [1]: error initializing Hyp mode: -6

In such case fixed DeviceTree blob from bug #1165290 would be needed. Fetch attached DTB, store as “/boot/mustang.dtb” and then edit “/etc/grub2-efi.cfg” file so kernel entry will look like this:

menuentry 'Fedora (4.0.0-0.rc5.git4.1.fc22.aarch64) 22 (Twenty Two)' --class fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-4.0.0-0.rc5.git2.4.1.fc22.aarch64-advanced-13e42c65-e2eb-4986-abf9-262e287842e4' {
        load_video
        insmod gzio
        insmod part_gpt
        insmod ext2
        set root='hd1,gpt32'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd1,gpt32 --hint-efi=hd1,gpt32 --hint-baremetal=ahci1,gpt32  13e42c65-e2eb-4986-abf9-262e287842e4
        else
          search --no-floppy --fs-uuid --set=root 13e42c65-e2eb-4986-abf9-262e287842e4
        fi
        linux /boot/vmlinuz-4.0.0-0.rc5.git4.1.fc22.aarch64 root=UUID=13e42c65-e2eb-4986-abf9-262e287842e4 ro  LANG=en_GB.UTF-8
        initrd /boot/initramfs-4.0.0-0.rc5.git4.1.fc22.aarch64.img
        devicetree /boot/mustang.dtb
}

After reboot KVM should work.

Software installation

Next step is installing VM software: “dnf install libvirt-daemon* virt-manager” will handle that. But to run Virt Manager we also need a way to see it. X11 forwarding over ssh to the rescue ;D After ssh connection I usually cheat with “sudo ln -sf ~hrw/.Xauthority /root/.Xauthority” to be able to run UI apps as root user.

UEFI firmware

Next phase is UEFI which allows us to boot virtual machine with ISO installation images (compared to kernel/initrd combo when there is no firmware/bootloader possibility). We will install one from repository provided by Gerd Hoffmann:

hrw@pinkiepie-f22:~$ sudo -s
root@pinkiepie-f22:hrw$ cd /etc/yum.repos.d/
root@pinkiepie-f22:yum.repos.d$ wget https://www.kraxel.org/repos/firmware.repo
root@pinkiepie-f22:yum.repos.d$ dnf install edk2.git-aarch64

Then libvirtd config change to give path for just installed firmware. Edit “/etc/libvirt/qemu.conf” file and at the end of file add this:

nvram = [
   "/usr/share/edk2.git/aarch64/QEMU_EFI-pflash.raw:/usr/share/edk2.git/aarch64/vars-template-pflash.raw"
]

Restart libvirtd via “systemctl restart libvirtd“.

Running Virtual Machine Manager

Now we can connect via “ssh -X” and run “sudo virt-manager“:

vm1

Next step is connection to libvirtd:

vm2vm3

Now we are ready for creating VMs. After pressing “Create a new VM” button we should see this:

vm4

And then creation of VM goes nearly like on x86 machines as there is no graphics only serial console.

But if you forgot to setup UEFI firmware then you will get this:

vm5

In such case get back to UEFI firmware step.

Installing Fedora 22 in VM

So let’s test how it works. Fedora 22 is in Beta phase now so why not test it?

vm6

vm10

vm11

2GB ram and 3 cpu cores should be more than enough ;D

vm12

And 10GB for minimal system:

vm13

vm14

But when it went to serial console it did not look good 🙁

vm15

I realized that I forgot to install fonts, but quick “dnf install dejavu*fonts” sorted that out:

vm16

Go for VNC controller installation.

After installation finish system runs just fine:

vm17

Summary

As you can see Fedora 22 has everything in place to get VM running on AArch64. UEFI firmware is the only thing out of distribution but that’s due to some license stuff on vfat implementation or something like that. I was running virtual machines with Debian ‘jessie’ and Fedora 22. Wanted to check Ubuntu but all I found was kernel/initrd combo (which is one of ways to boot in virt-manager) but it did not booted in VM.

I will stay with VirtualBox for a bit longer

VMWare, VirtualBox, Xen, KVM, LXC and there are probably several other ways to run virtual machine under Linux.

Years ago I used first one. Then moved to VirtualBox and still use it. But when it comes to rest of list I do not have too much experience. Xen? One of my servers in past was Xen instance. KVM? Maybe used few times to quick boot some ISO image. LXC? never played with.

From time to time I look at tools I use and check for better replacements. When moved to Fedora I decided to try “virt-manager” to move from VirtualBox to QEMU/KVM. Failed. Tried again…

Today, after “playing” with virt-manager in Fedora/rawhide I decided to stay with VirtualBox for longer. It may have some issues but runs my WinXP and Linux instances without any extra problems.

Sorry to say but Virt-manager feels like a tool for its creators only. Error messages which could be in any random language, insisting on using /var/lib/libvirt/images/ for disk images (or do lot of clicks instead) and depending on so many software packages that it will probably take months before someone will finally fix it in rawhide.

And no, I do not want to go to running QEMU/KVM by hand. It is good for booting image to play with. But I need a way to add/remove USB devices in running system. In an easy way.

And yes, I know “report all bugs” mantra…