We need some thermite…

Time goes and it is that time of year where Linaro Enterprise Group is working on a new release. And as usual jokes about lack of thermite starts…

Someone may ask “Why?”. Reason is simple: X-Gene 1 processor. I think that it’s hateclub grows and grows with time.

When it was released it was a nice processor. Eight cores, normal SATA, PCI Express, USB, DDR3 memory with ECC etc. It was used for distribution builders, development platforms etc. Not that there was any choice 😀

Nowadays with all those other AArch64 processors on a market it starts to be painful. PCI support requires quirks, serial console requires patching etc. We have X-Gene1 in Applied Micro Mustang servers and HPe Moonshot M400 cartridges. Maybe officially those machines are not listed as supported but we still use them so testing a new release work there has to be done.

And each time there are some issues to work around. Some could probably be fixed with firmware updates but I do not know do vendors still support that hardware.

So if you have some spare thermite (and a way to handle that legally) then contact us.

AArch64 desktop hardware?

Soon there will be four years since I started working on AArch64 architecture. Lot of software things changed during that time. Lot in a hardware too. But machines availability still sucks badly.

In 2012 all we had was software model. It was slow, terribly slow. Common joke was AArch64 developers standing in a queue for 10GHz x86-64 cpus. So I was generating working binaries by using cross compilation. But many distributions only do native builds. In models. Imagine Qt4 building for 3-4 days…

In 2013 I got access to first server hardware. With first silicon version of CPU. Highly unstable, we could use just one core etc. GCC was crashing like hell but we managed to get stable build results from it. Qt4 was building in few hours now.

Then amount of hardware at Red Hat was growing and growing. Farms of APM Mustangs, AMD Seattle and several other servers appeared, got racked and available to use. In 2014 one Mustang even landed on my desk (as first such machine in Poland).

But this was server land. Each of those machines costed about 1000 USD (if not more). And availability was hard too.

Linaro tried to do something about it and created 96boards project.

First came ‘Consumer Edition’ range. Yet another small form factor boards with functionality stripped as much as possible. No Ethernet, no storage other than emmc/usb, low amount of memory, chips taken from mobile phones etc. But it was selling! But only because people were hungry to get ANYTHING with AArch64 cores. First was HiKey then DragonBoard410 got released. Then few other boards. All with same set of issues: non-mainline kernel, weird bootloaders, binary blobs for this or that…

Then so called ‘Enterprise Edition’ got announced. With another ridiculous form factor (and microATX as an option). And that was it. There was a leak of Husky board which shown how fucked up design it was. Ports all around the edges, memory above and under board and of course incompatible with any industrial form factor. I would like to know what they were smoking…

Time passed by. Husky got forgotten for another year. Then Cello was announced as a “new EE 96boards board” while it looked as redesigned Husky with two SATA ports less (because who needs more than two SATA, right?). Last time I heard about Cello it was still ‘maybe soon, maybe another two weeks’. Prototypes looked like hand soldered, USB controller mounted rotated, dead on-board Ethernet etc.

In meantime we got few devices from other companies. Pine64 had big campaign on Kickstarter and shipped to developers. Hardkernel started selling ODROID-C2, Geekbox released their TV box and probably something else got released as well. But all those boards were limited to 1-2GB of memory, often lacked SATA and used mobile processors with their own set of bootloaders etc causing extra work for distributions.

Overdrive 1000 was announced. Without any options for expansion it looked like SoftIron wanted customers to buy Overdrive 3000 if they want to use PCI Express card.

So we have 2016 now. Four years of my work on AArch64 passed. Most of distributions support this architecture by building on proper servers but most of this effort is not used because developers do not have sane hardware to play with (sane means expandable, supported by distributions, capable).

There is no standard form factor mainboards (mini-itx, microATX, ATX) available on mass market. 96boards failed here, server vendors are not interested, small Chinese companies prefer to release yet-another-fruit/Pi with mobile processor. Nothing, null, nada, nic.

Developers know where to buy normal computer cases, storage, memory, graphics cards, USB controllers, SATA controllers and peripherals. So vendors do not have to worry/deal with this part. But still there is nothing to put those cards into. No mainboards which can be mounted into normal PC case, have some graphics plugged in, few SSD/HDD connected, mouse/keyboard, monitors and just be used.

Sometimes it is really hard to convince software developers to make changes for platform they are unable to test on. And current hardware situation does not help. All those projects of hardware being available “in a cloud” helps only for subset of projects — ever tried to run GNOME/KDE session over the network? With OpenGL acceleration etc?

So where is my AArch64 workstation? In desktop or laptop form.

Post written after my Google+ post where similar discussion happened in comments.

Failed to set MokListRT: Invalid Parameter

Somehow I managed to break UEFI environment on APM Mustang. As a result I was not able to enter boot manager menu nor UEFI shell. All I had was booting to 0001 boot entry (which was just installed Fedora 24 alpha).

After reboot I scrolled a bit to take a look at firmware output:

X-Gene Mustang Board
Boot firmware (version 1.1.0 built at 14:50:19 on Oct 20 2015)
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3021001 I0
TianoCore 1.1.0 UEFI 2.4.0 Oct 20 2015 14:49:32
CPU: APM ARM 64-bit Potenza Rev A3 2400MHz PCP 2400MHz
     32 KB ICACHE, 32 KB DCACHE
     SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
Board: X-Gene Mustang Board
Slimpro FW:
        Ver: 2.4 (build 01.15.10.00 2015/04/22)
        PMD: 950 mV
        SOC: 950 mV
Failed to set MokListRT: Invalid Parameter

Here screen got cleared instantly and grub was shown. I booted into one of installed systems and started playing with EFI boot manager:

17:38 root@pinkiepie-rawhide:~$ efibootmgr 
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0001,0004,0000
Boot0000  Fedora rawhide
Boot0001* Fedora
Boot0004* Red Hat Enterprise Linux

Note “0 seconds” timeout. I changed it to 5s (efibootmgr -t 5), rebooted and UEFI menu appeared again:

TianoCore 1.1.0 UEFI 2.4.0 Oct 20 2015 14:49:32
CPU: APM ARM 64-bit Potenza Rev A3 2400MHz PCP 2400MHz
     32 KB ICACHE, 32 KB DCACHE
     SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
Board: X-Gene Mustang Board
Slimpro FW:
        Ver: 2.4 (build 01.15.10.00 2015/04/22)
        PMD: 950 mV
        SOC: 950 mV
The default boot selection will start in   5 seconds
[1] Fedora rawhide
[2] Red Hat Enterprise Linux
[3] Fedora
[4] Shell
[5] Boot Manager
[6] Reboot
[7] Shutdown
Start: 

So I can boot whatever I want again ;D

Running 32-bit ARM virtual machine on AArch64 hardware

It was a matter of days and finally all pieces are done. Running 32-bit ARM virtual machines on 64-bit AArch64 hardware is possible and quite easy.

Requirements

  • AArch64 hardware (I used APM Mustang as usual)
  • ARM rootfs (fetched Fedora 22 image with “virt-builder” tool)
  • ARM kernel and initramfs (I used Fedora 24 one)
  • Virt Manager (can be done from shell too)

Configuration

Start “virt-manager” and add new machine:

Add new machine

Select rootfs, kernel, initramfs (dtb will be provided internally by qemu) and tell kernel where rootfs is:

Storage/boot options

Then set amount of memory and cores. I did 10GB of RAM and 8 cores. Save machine.

Let’s run

Open created machine and press Play button. It should boot:

Booted system

I upgraded F22 to F24 to have latest development system.

Is it fast?

If I would just boot and write about it then there will be questions about performance. I did build of gcc 5.3.1-3 using mock (standard Fedora way). On arm32 Fedora builder it took 19 hours, on AArch64 builder 4.5h only. On my machine AArch64 build took 9.5 hour and in this vm it took 12.5h (slow hdd used). So builder with memory and some fast storage will boost arm32 builds a lot.

Numbers from “openssl speed” shows performance similar to host cpu:

The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md2               1787.41k     3677.19k     5039.02k     5555.88k     5728.94k
mdc2                 0.00         0.00         0.00         0.00         0.00 
md4              24846.05k    81594.07k   226791.59k   418185.22k   554344.45k
md5              18881.79k    60907.46k   163927.55k   281694.58k   357168.47k
hmac(md5)        21345.25k    69033.83k   177675.52k   291996.33k   357250.39k
sha1             20776.17k    65099.46k   167091.03k   275240.62k   338582.71k
rmd160           15867.02k    42659.95k    88652.54k   123879.77k   140571.99k
rc4             167878.11k   186243.61k   191468.46k   192576.51k   193112.75k
des cbc          35418.48k    37327.19k    37803.69k    37954.56k    37991.77k
des ede3         13415.40k    13605.87k    13641.90k    13654.36k    13628.76k
idea cbc         36377.06k    38284.93k    38665.05k    38864.71k    39032.15k
seed cbc         42533.48k    43863.15k    44276.22k    44376.75k    44397.91k
rc2 cbc          29523.86k    30563.20k    30763.09k    30940.50k    30857.44k
rc5-32/12 cbc        0.00         0.00         0.00         0.00         0.00 
blowfish cbc     60512.96k    66274.07k    67889.66k    68273.15k    68302.17k
cast cbc         56795.77k    61845.42k    63236.86k    63251.11k    63445.82k
aes-128 cbc      61479.48k    65319.32k    67327.49k    67773.78k    66590.04k
aes-192 cbc      53337.95k    55916.74k    56583.34k    56957.61k    57024.51k
aes-256 cbc      46888.06k    48538.97k    49300.82k    49725.44k    50402.65k
camellia-128 cbc    59413.00k    62610.45k    63400.53k    63593.13k    63660.03k
camellia-192 cbc    47212.40k    49549.89k    50590.21k    50843.99k    50012.16k
camellia-256 cbc    47581.19k    49388.89k    50519.13k    49991.68k    50978.82k
sha256           27232.09k    64660.84k   119572.57k   151862.27k   164874.92k
sha512            9376.71k    37571.93k    54401.88k    74966.36k    84322.99k
whirlpool         3358.92k     6907.67k    11214.42k    13301.08k    14065.66k
aes-128 ige      60127.48k    65397.14k    67277.65k    67428.35k    67584.00k
aes-192 ige      52340.73k    56249.81k    57313.54k    57559.38k    57191.08k
aes-256 ige      46090.63k    48848.96k    49684.82k    49861.32k    49892.01k
ghash           150893.11k   171448.55k   177457.92k   179003.39k   179595.95k
                  sign    verify    sign/s verify/s
rsa  512 bits 0.000322s 0.000026s   3101.3  39214.9
rsa 1024 bits 0.001446s 0.000073s    691.7  13714.6
rsa 2048 bits 0.008511s 0.000251s    117.5   3987.5
rsa 4096 bits 0.058092s 0.000945s     17.2   1058.4
                  sign    verify    sign/s verify/s
dsa  512 bits 0.000272s 0.000297s   3680.6   3363.6
dsa 1024 bits 0.000739s 0.000897s   1353.1   1115.2
dsa 2048 bits 0.002762s 0.002903s    362.1    344.5
                              sign    verify    sign/s verify/s
 256 bit ecdsa (nistp256)   0.0005s   0.0019s   1977.8    538.3
 384 bit ecdsa (nistp384)   0.0015s   0.0057s    663.0    174.6
 521 bit ecdsa (nistp521)   0.0035s   0.0136s    286.8     73.4
                              op      op/s
 256 bit ecdh (nistp256)   0.0016s    616.0
 384 bit ecdh (nistp384)   0.0049s    204.8
 521 bit ecdh (nistp521)   0.0115s     87.2

Unbricking APM Mustang

Firmware update usually ends well. Previous (1.15.19) firmware failed to boot on some of Mustangs at Red Hat but worked fine on one under my desk. Yesterday I got 1.15.22 plus slimpro update and managed to get machine into non-bootable state (firmware works fine on other machines).

So how to get APM Mustang back into working state?

  • Get a SD card and connect it to an PC Linux machine with reader support.
  • Download Mustang software from MyAPM (1.5.19 was latest available there).
  • Unpack “mustang_sq_1.15.19.tar.xz” and then “mustang_binaries_1.15.19.tar.xz” tarballs.
  • Write the boot loader firmware to the SD card: “dd if=tianocore_media.img of=/dev/SDCARD“.
  • Take FAT formatted USB drive and put there some files from “mustang_binaries_1.15.19.tar.xz” archive (all into root directory):
    • potenza/apm_upgrade_tianocore.cmd
    • potenza/tianocore_media.img
    • potenza/UpgradeFirmware.efi
  • Power off your Mustang
  • Configure the Mustang to boot from SD card via these jumpers change:
    • Find HDR9 (close to HDR8, which is next to PCIe port)
    • Locate pin 11-12 and 17-18.
    • Connect 11-12 and 17-18 with jumpers
  • Insert SD card to Mustang SD port
  • Connect serial cable to Mustang and your PC.
  • Run minicom/picocom/screen/other-preferred-serial-terminal and connect to Mustang serial port
  • Power up Mustang and you should boot with SD UEFI firmware:
X-Gene Mustang Board
Boot firmware (version 1.1.0 built at 12:25:21 on Jun 22 2015)
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3021001 I0
TianoCore 1.1.0 UEFI 2.4.0 Jun 22 2015 12:24:25
CPU: APM ARM 64-bit Potenza Rev A3 1400MHz PCP 1400MHz
     32 KB ICACHE, 32 KB DCACHE
     SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
Board: X-Gene Mustang Board
The default boot selection will start in   2 second 
  • Press any key to get into UEFI menu.
  • Select “Shell” option and you will be greeted with a list of recognized block devices and filesystems. Check which is USB (“FS6” in my case).
Shell> fs6:
FS6:\> ls  
Directory of: FS6:\
08/04/2015  00:28              39,328  UpgradeFirmware.efi
08/27/2015  19:20                  56  apm_upgrade_tianocore.cmd
08/27/2015  19:20           2,098,176  tianocore_media.img
  • Flash firmware using “UpgradeFirmware.efi apm_upgrade_tianocore.cmd” command.
  • Power off
  • Change jumpers back to normal (11-12 and 17-18 to be open).
  • Eject SD card from Mustang
  • Power on

And your Mustang should be working again. You can also try to write other versions of firmware of course or grab files from internal hdd.

Fedora 23 and unsupported ARM/AArch64 devices

Week ago Fedora 23 got released. Also for ARM and AArch64 architectures. But it does not mean that it supports all possible hardware.

AArch64

There is the installation guide which lists two supported hardware platforms (besides QEMU):

  • Applied Micro Mustang
  • AMD Seattle

And then we got email from Clive Messer with question why we do not support 96Boards, ie. HiKey and Dragonboard as they are cheap and available.

I am not surprised with such question. It would be great to have support for both boards but their current state makes it quite hard. There is no support for them in mainline kernel, Dragonboard needs some firmware files which license forbids packaging it (note bolded part):

Distribution of the Redistributable Binary Code is subject to the following restrictions: (i) Redistributable Binary Code may only be distributed in binary format and may not be distributed in source code format: (ii) Redistributable Binary Code may not be distributed on a stand alone basis but may be redistributed in conjunction with and as a part of a software application created by you; (iii) the Redistributable Binary Code may only operate in conjunction with platforms incorporating Qualcomm Technologies, Inc. chipsets; (iv) redistribution of the Redistributable Binary Code must include the .txt file setting forth the terms and condition of this Agreement; (v) you may not use Qualcomm Technologies’ or its affiliates or subsidiaries name, logo or trademarks; and (vi) copyright, trademark, patent and any other notices that appear on the Materials may not be removed or obscured.

So even if we get mainline kernel working on it some things will not work without non-free files.

Then we have bootloaders. Hikey can be flashed with UEFI but (according to bootloader install documentation) you need to keep partitions in some magic way. Where is “there has to be one ef00 type partition formatted with FAT” as it is with other UEFI powered machines? Dragonboard 410c uses fastboot ;(

ARM

On ARM side there is common question about “most readily available and used board, with the most units sold and the biggest community” one. I think that developers from that community do not want their board supported in main distributions like Debian or Fedora.

Heresy? Do not think so. What needs to get board supported was told many times. Mainline kernel support, firmware blobs with redistribution license, drivers for graphics and sane bootloader (UEFI, U-Boot, maybe some other too).

Summary

So if you want to have your board supported then spend some time on mainlining your changes/drivers. And then come to us.

How to get Xserver running out of box on AArch64

As I want to have AArch64 desktop running with as small amount of changes needed as possible I decided that it is time to get Xserver to just run on APM Mustang.

Current setup

To get X11 running I need to have xorg.conf file. And this feels strange as on x86(-64) I got rid of it years ago.

Config snippet is small and probably could be even smaller:

Section "Device"
        Identifier " radeon"
        Driver  "radeon"
        BusID "PCI:1:0:0"
EndSection

Section "Screen"
        Identifier      "Screen"
        Device          "radeon"
EndSection

Section "DRI"
        Mode 0666
EndSection

Without it Xserver failed to find graphics card.

Searching for solution

I cloned Xserver git repository and started hacking. During several hours (split into few days) I added countless LogMessage() calls to source code, generated few patches and sent them to x-devel ML. And finally found out why it did not work.

Turns out that I was wrong — Xserver was able to find graphics card. But then it went to platform_find_pci_info() function, called pci_device_is_boot_vga() and rejected it.

Why? Because firmware from Applied Micro did not initialized card so kernel decided not to mark it as boot gfx one. I do not know is it possible to get UEFI to properly initialize pcie card on AArch64 architecture but there are two other ways to get it working.

hack Xserver

We can hack Xserver to not check for pci_device_is_boot_vga() or to use first available card if it returns false:

diff a/hw/xfree86/common/xf86platformBus.c b/hw/xfree86/common/xf86platformBus.c
index f1e9423..d88c58e 100644
--- a/hw/xfree86/common/xf86platformBus.c
+++ b/hw/xfree86/common/xf86platformBus.c
@@ -136,7 +136,8 @@ platform_find_pci_info()
     if (info) {
         pd->pdev = info;
         pci_device_probe(info);
-        if (pci_device_is_boot_vga(info)) {
+        if (pci_device_is_boot_vga(info) || xf86_num_platform_devices == 1)
+        {
             primaryBus.type = BUS_PLATFORM;
             primaryBus.id.plat = pd;
         }

This may not work on multi-gpu systems. In that case try removing “== 1” part.

hack Linux kernel

If firmware does not give us boot gfx card then maybe we can mark first one as such and everything will work? This is how PowerPC has it solved. So let’s take their code:

diff --git a/arch/arm64/kernel/pci.c b/arch/arm64/kernel/pci.c
index b3d098b..eea39ba 100644
--- a/arch/arm64/kernel/pci.c
+++ b/arch/arm64/kernel/pci.c
@@ -18,6 +18,7 @@
 #include <linux/of_pci.h>
 #include <linux/of_platform.h>
 #include <linux/slab.h>
+#include <linux/vgaarb.h>
 
 #include 
 
@@ -84,3 +85,15 @@ struct pci_bus *pci_acpi_scan_root()
        return NULL;
 }
 #endif
+
+static void fixup_vga(struct pci_dev *pdev)
+{
+       u16 cmd;
+
+       pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+       if ((cmd & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY)) || !vga_default_device())
+               vga_set_default_device(pdev);
+
+}
+DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_ANY_ID, PCI_ANY_ID,
+                             PCI_CLASS_DISPLAY_VGA, 8, fixup_vga);

Summary

Both hacks work. I can just run Xserver and get X11 working. But which one will get into upstream and then to Fedora and other Linux distributions? Time will show.

There are some issues with those solutions. If there are multiple graphics cards in a system then which one is primary one? Can their order change after firmware or kernel update?

Thanks goes to Dave Airlie for help with Xserver, Mark Salter for pointing me to PowerPC solution and Matthew Garrett for discussion about issues with kernel solution.

And now something more enterprise — CentOS on AArch64

Everyone can grab and install Fedora 21-22 on APM Mustang. But what if you want something more enterprise ready? Answer is simple: you can install CentOS 7 (at Alpha stage now).

Requirements are simple:

First fetch iso and then copy to USB thumb drive (simple “dd if=boot.iso of=/dev/sdc bs=4M” is enough). Then just reboot to UEFI shell and run “FS1:\EFI\BOOT\BOOTAA64.EFI” to get into installation.

All files will be read from USB (for Mustang you need official APM firmware 1.1.0 to get proper DTB) and installation goes like with Fedora, RHEL, CentOS — standard Anaconda.

Finally reboot and system is ready for use:

CentOS Linux 7 (Core)
Kernel 4.0.0-1.el7.aarch64 on an aarch64

pinkiepie login: 

AArch64 port is now in progress so not all packages are built yet but they will be.

Started X11 on AArch64 (hardware this time)

Took me some time but finally I managed to workaround all issues and got X11 running on real AArch64 hardware.

Two years ago I started X11 on AArch64 using emulator. But planned to make it on real hardware one day. And that day came today.

Screenshot

What took so long? Several things. First I lacked hardware – but APM Mustang arrived one day. Then lack of PCI-Express support but it got solved.

So I started collecting graphic cards. Finally ended with Radeon HD5450 and Geforce GTS250 – both with 512MB ram. After last firmware update (to 0.14) they even got whole memory assigned. But none of them worked ;(

After few discussions it got finally confirmed that something is going on with supporting more than 64MB of memory on PCI cards. I prefer not to go to details. Anyway I digged and found Matrox G550 card in local computer scrapyard. Seller wrote that it has 32MB ram (Linux says that only 16MB is present).

Bought card, inserted into pinkiepie (Mustang) and after kernel rebuild I got nice 1920x1080x32-60 framebuffer and X11 over it. Maybe it is not the fastest but it works.

Tomorrow will reconfigure kernel to get USB working (and submit patch for Fedora config) which will give me keyboard, mouse and audio.

Next step? XFCE, GNOME, KDE testing of course ;D And building MPlayer so I will be able to watch movies too.

Fedora 21 RC5 released for AArch64

Today Peter Robinsson released RC5 of Fedora 21 installation ISO for AArch64 architecture. As usual I grabbed it for tests.

As usual I used “clean Mustang, just hdd” method as it is one of easiest to setup. Grabbed fresh F21 RC5 ISO, prepared partitions and rebooted.

There was an issue with “fs0:\EFI\BOOT\BOOTAA64.EFI” as it tried to run “\EFI\BOOT\grubx64.efi” instead of “\EFI\BOOT\grubaa64.efi” but if second one is started then installer booted fine.

New EFI binary was added: MokManager.efi – can be used it for managing keys used to sign kernel modules. Not tried that yet but good to have.

Also noticed that ‘this is testing’ nagging requester got removed. Also nice side/top graphics with “fedora SERVER” information were added.

Installed without issues and rebooted into fresh Fedora 21 Server installation. Things look good for final release.