Marcin Juszkiewicz - virtualization

RISC-V is sloooow

2026-03-10T18:53:00+01:00

About 3 months ago I started working with RISC-V port of Fedora Linux. Many things happened during that time.

Triaging

I went through the Fedora RISC-V tracker entries, triaged most of them (at the moment 17 entries left in NEW) and tried to handle whatever possible.

Fedora packaging

My usual way of working involves fetching sources of a Fedora package (fedpkg clone -a) and then building it (fedpkg mockbuild -r fedora-43-riscv64). After some time, I check did it built and if not then I go through build logs to find out why.

Effect? At the moment, 86 pull requests sent for Fedora packages. From heavy packages like the “llvm15” to simple ones like the “iyfct” (some simple game). At the moment most of them were merged, and most of these got built for the Fedora 43. Then we can build them as well as we follow ‘f43-updates’ tag on the Fedora koji.

Slowness

Work on packages brings the hard, sometimes controversial, topic: speed. Or rather lack of it.

You see, the RISC-V hardware at the moment is slow. Which results in terrible build times — look at details of the binutils 2.45.1-4.fc43 package I took from koji (Fedora and RISC-V Fedora):

Architecture	Cores	Memory	Build time
aarch64	12	46 GB	36 minutes
i686	8	29 GB	25 minutes
ppc64le	10	37 GB	46 minutes
riscv64	8	16 GB	143 minutes
s390x	3	45 GB	37 minutes
x86_64	8	29 GB	29 minutes

That was StarFive VisionFive 2 board, while it has other strengths (such as upstreamed drivers), it is not the fastest available one. I asked around and one of porters did a built on Milk-V Megrez — it took 58 minutes.

Also worth mentioning is that the current build of RISC-V Fedora port is done with disabled LTO. To cut on memory usage and build times.

RISC-V builders have four or eight cores with 8, 16 or 32 GB of RAM (depending on a board). And those cores are usually compared to Arm Cortex-A55 ones. The lowest cpu cores in today’s Arm chips.

The UltraRISC UR-DP1000 SoC, present on the Milk-V Titan motherboard should improve situation a bit (and can have 64 GB ram). Similar with SpacemiT K3-based systems (but only 32 GB ram). Both will be an improvement, but not the final solution.

Hardware needs for Fedora inclusion

We need hardware capable of building above “binutils” package below one hour. With LTO enabled system-wide etc. to be on par with the other architectures. This is the speed-related requirement.

There is no point of going for inclusion with slow builders as this will make package maintainers complain. You see, in Fedora build results are released into repositories only when all architectures finish. And we had maintainers complaining about lack of speed of AArch64 builders in the past. Some developers may start excluding RISC-V architecture from their packages to not have to wait.

And any future builders need to be rackable and manageable like any other boring server (put in a rack, connect cables, install, do not touch any more). Because no one will go into a data centre to manually reboot an SBC-based builder.

Without systems fulfilling both requirements, we can not even plan for the RISC-V 64-bit architecture to became one of official, primary architectures in Fedora Linux.

I still use QEMU for local testing

Such long build times make my use of QEMU useful. My AArch64 desktop has 80 cores, so with the use of QEMU userspace riscv64 emulation, I can build the packages without buying RISC-V hardware. Still, there are timed out tests because single thread is slower than native one.

btop shows 80 cores being busy

There are package (like LLVM) which make real use of both available cores and memory. I am wondering how fast would it go on 192/384 cores of Ampere One-based system.

Still, I used QEMU for local builds/testing only. Fedora, like several other distributions, does native builds only.

Future plans

We plan to start building Fedora Linux 44. If things go well, we will use the same kernel image on all of our builders (the current ones use a mix of kernel versions). LTO will still be disabled.

When it comes to lack of speed… There are plans to bring new, faster builders. And probably assign some heavier packages to them.

From the diary of AArch64 porter — RISC-V

2025-12-04T18:47:00+01:00

Wait, what? RISC-V? In ‘the diary of AArch64 porter’? WTH?

Yes, I started working on Fedora packaging for the 64-bit RISC-V architecture port.

All started with discussion about Mock

About a week ago, one of my work colleagues asked me about my old post about speeding up Mock. We had a discussion, I pointed him to the Mock documentation, and gave some hints.

It turned out that he was working on RISC-V related changes to Fedora packages. As I had some spare cycles, I decided to take a look. And I sank…

State of the RISC-V Fedora port

The 64-bit RISC-V port of Fedora Linux is going quite well. There are over 90% of Fedora packages already built for that architecture. And there are several packages with the riscv64 specific changes, such as:

patches adding RISC-V support
disabling some parts of test suites
disabling some build options due to bootstrapping of some languages being in progress (like Java)
disabling of debug information due to some toolchain issues (there is a work-in-progress now to solve them)

Note that these changes are temporary. There are people working on solving toolchain issues, languages are being bootstrapped (there was a review of Java changes earlier this week), patches are being integrated upstream and in Fedora, and so on.

There is the Fedora RISC-V tracker website showing the progress of the port:

package name
current status (new, triaged, patch posted, patch merged, done)
version in RISC-V port Koji
version in Fedora Koji (F43 release is tracked now)
version in CentOS Stream 10
notes

This is a simple way to check what to work on. And there are several packages, not built yet due to use of “ExclusiveArch” setting in them.

My work

The quick look at work needed reminded me of the 2012-2014 period, when I worked on the same stuff but for AArch64 ports (OpenEmbedded, Debian/Ubuntu, Fedora/RHEL). So I had a knowledge, I knew the tools and started working.

In the beginning, I went through entries in the tracker and tried to triage as many packages as possible, so it will be more visible which ones need work and which can be ignored (for now). The tracker went from seven to over eighty triaged packages in a few days.

Then I looked at changes done by current porters. Which usually meant David Abdurachmanov. I used his changes as a base for the changes needed for Fedora packaging, while trying to minimise the amount of them to the minimum required.

I did over twenty pull requests to Fedora packages during a week of work.

Hardware?

But which hardware did I use to run those hundreds of builds? Was it HiFive Premier P550? Or maybe Milk-V Titan or another RISC-V SBC?

Nope. I used my 80-core, Altra-based, AArch64 desktop to run all those builds. With the QEMU userspace helper.

You see, Mock allows to run builds for foreign architectures — all you need is the proper qemu-user-static-* package and you are ready to go:

$ fedpkg mockbuild -r fedora-43-riscv64

You can extract the “fedora-43-riscv64” Mock config from the mock-riscv64-configs.patch hosted on Fedora RISC-V port forge. I hope that these configuration files may be found in the “mock-core-configs” in Fedora soon.

At some point I had 337 qemu-user-static-riscv processes running at same time. And you know what? It was still faster than a native build on 64-bit RISC-V hardware.

But, to be honest, I only compared a few builds, so it may be better with other builders. Fedora RISC-V Koji uses wide list of SBCs to build on:

Banana Pi BPI-F3
Milk-V Jupiter
Milk-V Megrez
SiFive HiFive Premier P550
StarFive VisionFive 2

Also note that using QEMU is not a solution for building a distribution. I used it only to check if package builds, and then scrap the results.

Future

Will I continue working on the RISC-V port of Fedora Linux? Probably yes. And, at some point, I will move to integrating those changes into CentOS Stream 10.

For sure I do not want to invest in RISC-V hardware. Existing models are not worth the money (in my opinion), incoming ones are still old (RVA20/RVA22) and they are slow. Maybe in two, three years there will be something fast enough.

ConfigurationManager in EDK2: just say no

2024-04-16T09:44:00+02:00

During my work on SBSA Reference Platform I have spent lot of time in firmware’s code. Which mostly meant Tianocore EDK2 as Trusted Firmware is quite small.

Writing all those ACPI tables by hand takes time. So I checked ConfigurationManager component which can do it for me.

Introduction

In 2018 Sami Mujawar from Arm contributed Dynamic Tables Framework to Tianocore EDK2 project. The goal was to have code which generates all ACPI tables from all those data structs describing hardware which EDK2 already has.

In 2023 I was writing code for IORT and GTDT tables to generate them from C. And started wondering about use of ConfigurationManager.

Mailed edk2-devel ML for pointers, documentation, hints. Got nothing in return, idea went to the shelf.

SBSA-Ref and multiple PCI Express buses

Last week I got SBSA-Ref system booting in NUMA configuration with three separate PCI Express buses. And started working on getting EDK2 firmware to recognize them as such.

Took me a day and pci command listed cards properly:

Shell> pci
   Seg  Bus  Dev  Func
   ---  ---  ---  ----
    00   00   00    00 ==> Bridge Device - Host/PCI bridge
             Vendor 1B36 Device 0008 Prog Interface 0
    00   00   01    00 ==> Network Controller - Ethernet controller
             Vendor 8086 Device 10D3 Prog Interface 0
    00   00   02    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   00   03    00 ==> Bridge Device - Host/PCI bridge
             Vendor 1B36 Device 000B Prog Interface 0
    00   00   04    00 ==> Bridge Device - Host/PCI bridge
             Vendor 1B36 Device 000B Prog Interface 0
    00   01   00    00 ==> Mass Storage Controller - Non-volatile memory subsystem
             Vendor 1B36 Device 0010 Prog Interface 2
    00   40   00    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   40   01    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   41   00    00 ==> Base System Peripherals - SD Host controller
             Vendor 1B36 Device 0007 Prog Interface 1
    00   42   00    00 ==> Display Controller - Other display controller
             Vendor 1234 Device 1111 Prog Interface 0
    00   80   00    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000E Prog Interface 0
    00   80   01    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   81   09    00 ==> Multimedia Device - Audio device
             Vendor 1274 Device 5000 Prog Interface 0
    00   81   10    00 ==> Network Controller - Ethernet controller
             Vendor 8086 Device 100E Prog Interface 0
    00   82   00    00 ==> Mass Storage Controller - Serial ATA controller
             Vendor 8086 Device 2922 Prog Interface 1

Three buses are: 0x00, 0x40 and 0x80. But then I had to tell Operating System about those. Which meant playing with ACPI tables code in C.

So idea came “what about trying ConfigurationManager?”.

Another try

Mailed edk2-devel ML again for pointers, documentation hints. And then looked at code written for N1SDP and started playing with ConfigurationManager…

ConfigurationManager.c has EDKII_PLATFORM_REPOSITORY_INFO struct with hundreds of lines of data (as another structs). From listing which ACPI tables I want to have (FADT, GTDT, APIC, SPCR, DBG2, IORT, MCFG, SRAT, DSDT, PPTT etc.) to listing all hardware details like GIC, PCIe, Timers, CPU and Memory information.

Then code for querying this struct. I thought that CM/DT (ConfigurationManager/DynamicTables) framework will have those already in EDK2 code but no — each platform has own set of functions. Another hundreds of lines to maintain.

Took some time to get it built, then started filling proper data and compared with ACPI tables I had previously. There were differences to sort out. But digging more and more into code I saw that I go deeper and deeper into rabbit hole…

Dynamic systems do not fit CM?

For platforms with dynamic hardware configuration (like SBSA-Ref) I needed to write code which would populate that struct with data on runtime. Check amount of cpu cores and write cpu information (with topology, cache etc), create all GIC structures and mappings. Then same for PCIe buses. Etc. Etc. etc…

STATIC
EFI_STATUS
EFIAPI
InitializePlatformRepository (
  IN  EDKII_PLATFORM_REPOSITORY_INFO  * CONST PlatRepoInfo
  )
{
  GicInfo                 GicInfo;
  CM_ARM_GIC_REDIST_INFO *GicRedistInfo;
  CM_ARM_GIC_ITS_INFO    *GicItsInfo;
  CM_ARM_SMMUV3_NODE     *SmmuV3Info;

  GetGicDetails(&GicInfo);
  PlatRepoInfo->GicDInfo.PhysicalBaseAddress = GicInfo.DistributorBase;

  GicRedistInfo = &PlatRepoInfo->GicRedistInfo[0];

  GicRedistInfo->DiscoveryRangeBaseAddress = GicInfo.RedistributorsBase;

  GicItsInfo = &PlatRepoInfo->GicItsInfo[0];
  GicItsInfo->PhysicalBaseAddress = GicInfo.ItsBase;

  SmmuV3Info = &PlatRepoInfo->SmmuV3Info[0];
  SmmuV3Info->BaseAddress = PcdGet64 (PcdSmmuBase);

  return EFI_SUCCESS;
}

Which in my case can mean even more code written to populate CM struct of structs than it would take to generate ACPI tables by hand.

Summary

ConfigurationManager and DynamicTables frameworks look tempting. There may be systems where it can be used with success. I know that I do not want to touch it again. All those structs of structs may look good for someone familiar with LISP or JSON but not for me.

DT-free EDK2 on SBSA Reference Platform

2024-04-04T13:26:00+02:00

During last weeks we worked on getting rid of DeviceTree from EDK2 on SBSA Reference Platform. And finally we managed!

All code is merged into upstream EDK2 repository.

What?

Someone may wonder where DeviceTree was in SBSA Reference Platform. Wasn’t it UEFI and ACPI platform?

Yes, from Operating System point of view it is UEFI and ACPI. But if you look deeper you will see DeviceTree hidden inside our chain of software components:

/dts-v1/;

/ {
        machine-version-minor = <0x03>;
        machine-version-major = <0x00>;
        #size-cells = <0x02>;
        #address-cells = <0x02>;
        compatible = "linux,sbsa-ref";

        chosen {
        };

        memory@10000000000 {
                reg = <0x100 0x00 0x00 0x40000000>;
                device_type = "memory";
        };

        intc {
                reg = <0x00 0x40060000 0x00 0x10000
                       0x00 0x40080000 0x00 0x4000000>;

                its {
                        reg = <0x00 0x44081000 0x00 0x20000>;
                };
        };

        cpus {
                #size-cells = <0x00>;
                #address-cells = <0x02>;

                cpu@0 {
                        reg = <0x00 0x00>;
                };

                cpu@1 {
                        reg = <0x00 0x01>;
                };

                cpu@2 {
                        reg = <0x00 0x02>;
                };

                cpu@3 {
                        reg = <0x00 0x03>;
                };
        };
};

It is very minimal one, providing us with only those information we need. It does not even pass any compliance checks. For example for Linux, GIC node (/intc/ one) should have gazillion of fields, but we only need addresses.

Trusted Firmware reads it, parses and provides information from it via Secure Monitor Calls (SMC) to upper firmware level (EDK2 in our case). DeviceTree is provided too but we do not read it any more.

Why?

Our goal is to treat software components a bit different then people may expect. QEMU is “virtual hardware” layer, TF-A provides interface to “embedded controller” (EC) layer and EDK2 is firmware layer on top.

On physical hardware firmware assumes some parts and asks EC for the rest of system information. QEMU does not give us that, while giving us a way to alter system configuration more than it would be possible on most of hardware platforms using a bunch of cli arguments.

EDK2 asks for CPU, GIC and Memory. When there is no info about processors or memory, it informs the user and shutdowns the system (such situation does not have a chance of happening but it works as an example).

Bonus stuff: NUMA

Bonus part of this work was adding firmware support for NUMA configuration. When QEMU is run with NUMA arguments then operating system gets whole memory and proper configuration information.

QEMU arguments used:

-smp 4,sockets=4,maxcpus=4
-m 4G,slots=2,maxmem=5G
-object memory-backend-ram,size=1G,id=m0
-object memory-backend-ram,size=3G,id=m1
-numa node,nodeid=0,cpus=0-1,memdev=m0
-numa node,nodeid=1,cpus=2,memdev=m1
-numa node,nodeid=2,cpus=3

How Operating System sees NUMA information:

root@sbsa-ref:~# numactl --hardware
available: 3 nodes (0-2)
node 0 cpus: 0 1
node 0 size: 975 MB
node 0 free: 840 MB
node 1 cpus: 2
node 1 size: 2950 MB
node 1 free: 2909 MB
node 2 cpus: 3
node 2 size: 0 MB
node 2 free: 0 MB
node distances:
node   0   1   2
  0:  10  20  20
  1:  20  10  20
  2:  20  20  10

What next?

There is CPU topology information in review queue. All those sockets, clusters, cores and threads. QEMU will pass it in DeviceTree, TF-A will give it via SMC and then EDK2 will put it in one of ACPI tables (PPTT == Processor Properties Topology Table).

If someone decide to write own firmware for SBSA Reference Platform (like port of U-Boot) then both DeviceTree and set of SMC calls will wait for them, ready to be used to gather hardware information.

Running SBSA Reference Platform

2024-03-25T11:21:00+01:00

Recently people asked me how to run SBSA Reference Platform for their own testing and development. Which shows that I should write some documentation.

But first let me blog about it…

Requirements

To run SBSA Reference Platform emulation you need:

QEMU (8.2+ recommended)
EDK2 firmware files

That’s all. Sure, some hardware resources would be handy but everyone has some kind of computer available, right?

QEMU

Nothing special is required as long as you have qemu-system-aarch64 binary available.

EDK2

We provide EDK2 binaries on CodeLinaro server. Go to “latest/edk2” directory, fetch both “SBSA_FLASH*” files, unpack them and you are ready to go. You may compare checksums (before unpacking) with values present in “latest/README.txt” file.

Those binaries are built from release versions of Trusted Firmware (TF-A) and Tianocore EDK2 plus latest “edk2-platforms” code (as this repo is not using tags).

Building EDK2

If you decide to build EDK2 on your own then we provide TF-A binaries in “edk2-non-osi” repository. I update those when it is needed.

Instructions to build EDK2 are provided in Qemu/SbsaQemu directory of “edk2-platforms” repository.

Running SBSA Reference Platform emulation

Note that this machine is fully emulated. Even on AArch64 systems where virtualization is available.

Let go through example QEMU command line:

qemu-system-aarch64
-machine sbsa-ref
-drive file=firmware/SBSA_FLASH0.fd,format=raw,if=pflash
-drive file=firmware/SBSA_FLASH1.fd,format=raw,if=pflash
-serial stdio
-device usb-kbd
-device usb-tablet
-cdrom disks/alpine-standard-3.19.1-aarch64.iso

At first we select “sbsa-ref” machine (defaults to four Neoverse-N1 cpu cores and 1GB of ram). Then we point to firmware files (order of them is important).

Serial console is useful for diagnostic output, just remember to not press Ctrl-C there unless you want to take whole emulation down.

USB devices are to have working keyboard and pointing device. USB tablet is more useful that USB mouse (-device usb-mouse adds it). If you want to run *BSD operating systems then I recommend to add USB mouse.

And the last entry adds Alpine 3.19.1 ISO image.

System boots to text console on graphical output. For some reason boot console is on serial port.

Adding hard disk

If you want to add hard disk then adding “-hdb disk.img” is enough (“hdb” as cdrom took 1st slot on the AHCI controller).

Handy thing is “virtual FAT drive” which allows to create guest’s drive from directory on the host:

-drive if=ide,file=fat:ro:DIRECTORY_ON_HOST,format=raw

This is useful for running EFI binaries as this drive is visible in UEFI environment. It is not present when operating system is booted.

Adding NVME drive

NVME is composed from two things:

PCIe device
storage drive

So let add it in a nice way, using PCIe root-port:

-device pcie-root-port,id=root_port_for_nvme1,chassis=2,slot=0
  -device nvme,serial=deadbeef,bus=root_port_for_nvme1,drive=nvme
     -drive file=disks/nvme.img,format=raw,id=nvme,if=none

Using NUMA configuration

QEMU can emulate Non-Uniform Memory Access (NUMA) setup. This usually means multisocket systems with memory available per cpu socket.

Example config:

-smp 4,sockets=4,maxcpus=4
-m 4G,slots=2,maxmem=5G
-object memory-backend-ram,size=1G,id=m0
-object memory-backend-ram,size=3G,id=m1
-numa node,nodeid=0,cpus=0-1,memdev=m0
-numa node,nodeid=1,cpus=2,memdev=m1
-numa node,nodeid=2,cpus=3

This adds four cpu sockets and 4GB of memory. First node has 2 cpu cores and 1GB ram, second node has 1 cpu and 3GB of ram and last node has 1 cpu without local memory.

Note that support for such setup in work in progress now (March 2024). We merged required code into TF-A and have set of patches for EDK2 in review. Without them you will see resources only from the first NUMA node.

Complex PCI Express setup

Our platform has GIC ITS support so we can try some complex PCI Express structures.

This example uses PCIe switch to add more PCIe slots and then (to complicate things) puts PCIe-to-PCI bridge into one of them to make use of old Intel e1000 network card:

-device pcie-root-port,id=root_port_for_switch1,chassis=0,slot=0
  -device x3130-upstream,id=up_port1,bus=root_port_for_switch1
    -device xio3130-downstream,id=down_port1,bus=up_port1,chassis=1,slot=0
      -device ac97,bus=down_port1
    -device xio3130-downstream,id=down_port2,bus=up_port1,chassis=1,slot=1
   -device pcie-pci-bridge,id=pci1,bus=down_port2
     -device e1000,bus=pci1,addr=2

Some helper scripts

During last year I wrote some helper scripts for working with SBSA Reference Platform testing. They are stored in sbsa-ref-status repository on GitHub.

May lack up-to-date documentation but can show my way of using the platform.

Summary

SBSA Reference Platform can be used for testing several things. From operating systems to (S)BSA compliance of the platform. Or to check how some things are emulated in QEMU. Or playing with PCIe setups (NUMA systems can have separate PCI Express buses but we do not handle it yet in firmware).

Have fun!

Testing *BSD on SBSA Reference Platform

2023-10-03T20:46:00+02:00

SystemReady specification mentions that system to be certified needs to be able to boot several operating systems:

In addition, OS installation and boot logs are required:

Windows PE boot log, from a GPT partitioned disk, is required.

VMware ESXi-Arm installation and boot logs are recommended.

Installation and boot logs from two of the Linux distros or BSDs are required.

All logs must be submitted using the ES/SR template.

In choosing the Linux distros or BSDs, maximize the coverage by diversifying the heritage. For example, the following shows the grouping of the heritage:

RHEL/Fedora/CentOS/AlmaLinux/Rocky Linux/Oracle Linux/Anolis OS

SLES/openSUSE

Ubuntu/Debian

CBL-Mariner

NetBSD/OpenBSD/FreeBSD

So during last week I went through *BSD ones.

OpenBSD

Started with “download OpenBSD” page and found out that there is no installation ISO for aarch64 architecture. Not good.

So I fetched miniroot73.img disk image instead and went on with booting:

>> OpenBSD/arm64 BOOTAA64 1.16
boot>
cannot open sd0a:/etc/random.seed: No such file or directory
booting sd0a:/bsd: 2798224+1058776+12709688+630920 [229059+91+651336+254968]=0x1
3ce628
FACP DBG2 MCFG SPCR APIC SSDT PPTT GTDT BGRT
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2023 OpenBSD. All rights reserved.  https://www.OpenBSD.org

OpenBSD 7.3 (RAMDISK) #1941: Sat Mar 25 14:42:22 MDT 2023
    deraadt@arm64.openbsd.org:/usr/src/sys/arch/arm64/compile/RAMDISK
real mem  = 4287451136 (4088MB)
avail mem = 4073807872 (3885MB)
random: boothowto does not indicate good seed
mainbus0 at root: ACPI
psci0 at mainbus0: PSCI 1.1, SMCCC 1.2
cpu0 at mainbus0 mpidr 0: ARM Cortex-A57 r1p0
cpu0: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
cpu0: 2048KB 64b/line 16-way L2 cache
cpu0: CRC32,SHA2,SHA1,AES+PMULL,ASID16
efi0 at mainbus0: UEFI 2.7
efi0: EFI Development Kit II / SbsaQemu rev 0x10000
smbios0 at efi0: SMBIOS 3.4.0
smbios0: vendor EFI Development Kit II / SbsaQemu version "1.0" date 09/15/2023
smbios0: QEMU QEMU SBSA-REF Machine
agintc0 at mainbus0 shift 4:3 nirq 256 nredist 2: "interrupt-controller"
agtimer0 at mainbus0: 62500 kHz
acpi0 at mainbus0: ACPI 6.0
acpi0: tables DSDT FACP DBG2 MCFG SPCR APIC SSDT PPTT GTDT BGRT
acpimcfg0 at acpi0
acpimcfg0: addr 0xf0000000, bus 0-255
pluart0 at acpi0 COM0 addr 0x60000000/0x1000 irq 33
pluart0: console
ahci0 at acpi0 AHC0 addr 0x60100000/0x10000 irq 42: AHCI 1.0
ahci0: port 0: 1.5Gb/s
ahci0: port 1: 1.5Gb/s
scsibus0 at ahci0: 32 targets
sd0 at scsibus0 targ 0 lun 0: <ATA, QEMU HARDDISK, 2.5+> t10.ATA_QEMU_HARDDISK_QM00001_
sd0: 43MB, 512 bytes/sector, 88064 sectors, thin
sd1 at scsibus0 targ 1 lun 0: <ATA, QEMU HARDDISK, 2.5+> t10.ATA_QEMU_HARDDISK_QM00003_
sd1: 504MB, 512 bytes/sector, 1032192 sectors, thin
ehci0 at acpi0 USB0 addr 0x60110000/0x10000 irq 43panic: uvm_fault failed: ffffff800034c3e8 esr 96000050 far ffffff8066ef5048

The operating system has halted.
Please press any key to reboot.

As you can see it hang on an attempt to initialize USB controller. Which shows that our move from EHCI to XHCI was not properly tested ;(

The problem was that our virtual hardware (QEMU) had XHCI (USB 3) controller on non-discoverable platform bus. But firmware (EDK2) tells that it was EHCI (USB 2) one.

This got solved with Yuquan Wang’s patch moving EDK2 to initiate and describe XHCI usb controller (change is already merged upstream). After rebuilding EDK2 OpenBSD booted fine right to the installation prompt (skipped previous messages):

xhci0 at acpi0 USB0 addr 0x60110000/0x10000 irq 43, xHCI 0.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root hub" rev 3.00/1.00 addr 1
acpipci0 at acpi0 PCI0
pci0 at acpipci0
0:1:0: rom address conflict 0xfffc0000/0x40000
0:2:0: rom address conflict 0xffff8000/0x8000
"Red Hat Host" rev 0x00 at pci0 dev 0 function 0 not configured
em0 at pci0 dev 1 function 0 "Intel 82574L" rev 0x00: msi, address 52:54:00:12:34:56
"Bochs VGA" rev 0x02 at pci0 dev 2 function 0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
simplefb0 at mainbus0: 1280x800, 32bpp
wsdisplay0 at simplefb0 mux 1
wsdisplay0: screen 0 added (std, vt100 emulation)
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Keyboard" rev 2.00/0.00 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0
wskbd0 at ukbd0 mux 1
wskbd0: connecting to wsdisplay0
uhidev1 at uhub0 port 2 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 2.00/0.00 addr 3
uhidev1: iclass 3/0
uhid at uhidev1 not configured
softraid0 at root
scsibus1 at softraid0: 256 targets
root on rd0a swap on rd0b dump on rd0b
WARNING: CHECK AND RESET THE DATE!
erase ^?, werase ^W, kill ^U, intr ^C, status ^T

Welcome to the OpenBSD/arm64 7.3 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell?

After this I added booting OpenBSD to QEMU tests for SBSA Reference Platform to make sure that we have something non-Linux based there.

FreeBSD

The next one was FreeBSD. And here situation started to be weird…

First I took 13.2 release. Used firmware with XHCI information and was greeted with:

Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.2-RELEASE releng/13.2-n254617-525ecfdad597 GENERIC arm64
FreeBSD clang version 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
VT(efifb): resolution 1280x800
module firmware already present!
real memory  = 4294967296 (4096 MB)
avail memory = 4160204800 (3967 MB)
Starting CPU 1 (1)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
random: entropy device external interface
MAP 100fbdf0000 mode 2 pages 128
MAP 100fbe70000 mode 2 pages 160
MAP 100fbf10000 mode 2 pages 80
MAP 100fbfb0000 mode 2 pages 80
MAP 100ff500000 mode 2 pages 400
MAP 100ff690000 mode 2 pages 592
MAP 10000000 mode 0 pages 1728
MAP 60010000 mode 0 pages 1
kbd0 at kbdmux0
acpi0: <LINARO SBSAQEMU>
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
acpi0: Could not update all GPEs: AE_NOT_CONFIGURED
psci0: <ARM Power State Co-ordination Interface Driver> on acpi0
gic0: <ARM Generic Interrupt Controller v3.0> iomem 0x40060000-0x4007ffff,0x40080000-0x4407ffff on acpi0
its0: <ARM GIC Interrupt Translation Service> mem 0x44081000-0x440a0fff on gic0
generic_timer0: <ARM Generic Timer> irq 3,4,5 on acpi0
Timecounter "ARM MPCore Timecounter" frequency 62500000 Hz quality 1000
Event timer "ARM MPCore Eventtimer" frequency 62500000 Hz quality 1000
efirtc0: <EFI Realtime Clock>
efirtc0: registered as a time-of-day clock, resolution 1.000000s
uart0: <PrimeCell UART (PL011)> iomem 0x60000000-0x60000fff irq 0 on acpi0
uart0: console (115200,n,8,1)
ahci0: <AHCI SATA controller> iomem 0x60100000-0x6010ffff irq 1 on acpi0
ahci0: AHCI v1.00 with 6 1.5Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
xhci0: <Generic USB 3.0 controller> iomem 0x60110000-0x6011ffff irq 2 on acpi0
xhci0: 32 bytes context size, 32-bit DMA

And it hang there…

Let check newer FreeBSD

Contacted people on #freebsd IRC channel and Mina Galić (meena on IRC) asked me to boot FreeBSD 14 or 15 images. So I tried both:

xhci0: <Generic USB 3.0 controller> iomem 0x60110000-0x6011ffff irq 2 on acpi0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0

System booted further. Note “64-bit DMA” information instead of “32-bit DMA” from 13.2 release. Reported bug 274237 for it. On the same day required change was identified and marked for potential backport.

AHCI issue

But that was not the only problem. Turned out that none of AHCI devices were found… So no way to run an installer:

ahci0: <AHCI SATA controller> iomem 0x60100000-0x6010ffff irq 1 on acpi0
ahci0: AHCI v1.00 with 6 1.5Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
[..]
Release APs...done
Trying to mount root from cd9660:/dev/iso9660/13_2_RELEASE_AARCH64_BO [ro]...
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
ahcich0: Poll timeout on slot 1 port 0
ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 170 serr 00000000 cmd 0000c017
(aprobe0:ahcich0:0:0:0): SOFT_RESET. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich0:0:0:0): CAM status: Command timeout
(aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted

I checked QEMU 7.2 (from Fedora package) and it booted fine. 8.0.5 failed, 8.0.0 booted. Hm… Started “git bisect” to find out which change broke it. After several rebuilds I found commit to blame:

commit 7bcd32128b227cee1fb39ff242d486ed9fff7648
Author: Niklas Cassel <niklas.cassel@wdc.com>
Date:   Fri Jun 9 16:08:40 2023 +0200

    hw/ide/ahci: simplify and document PxCI handling

    The AHCI spec states that:
    For NCQ, PxCI is cleared on command queued successfully.

Is it AArch64 only or not?

The next step: checking is it global problem or only aarch64 one.

I built x86-64 emulation component and checked Q35 machine (which also uses AHCI). And FreeBSD failed exactly same way. This made bug reporting a lot easier as there were several architectures and more users affected.

Mailed author and QEMU developers about it. Described the problem, gave exact command line arguments for QEMU etc. Niklas Cassel replied:

I will have a look at this.

So it will be done.

NetBSD

Here situation was a bit similar to FreeBSD one.

Fetched NetBSD 9.3 image and booted just to see it hang (removed printk.time from output):

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
    2018, 2019, 2020, 2021, 2022
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 9.3 (GENERIC64) #0: Thu Aug  4 15:30:37 UTC 2022
 mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/evbarm/compile/GENERIC64
total memory = 4075 MB
avail memory = 3929 MB
running cgd selftest aes-xts-256 aes-xts-512 done
armfdt0 (root)
simplebus0 at armfdt0: QEMU QEMU SBSA-REF Machine
simplebus1 at simplebus0
acpifdt0 at simplebus0
acpifdt0: using EFI runtime services for RTC
ACPI: RSDP 0x00000100FC020018 000024 (v02 LINARO)
ACPI: XSDT 0x00000100FC02FE98 00006C (v01 LINARO SBSAQEMU 20200810 LNRO 00000001)
ACPI: FACP 0x00000100FC02FB98 000114 (v06 LINARO SBSAQEMU 20200810 LNRO 00000001)
ACPI: DSDT 0x00000100FC02E998 000CD8 (v02 LINARO SBSAQEMU 20200810 INTL 20220331)
ACPI: DBG2 0x00000100FC02FA98 00005C (v00 LINARO SBSAQEMU 20200810 LNRO 00000001)
ACPI: MCFG 0x00000100FC02FE18 00003C (v01 LINARO SBSAQEMU 20200810 LNRO 00000001)
ACPI: SPCR 0x00000100FC02FF98 000050 (v02 LINARO SBSAQEMU 20200810 LNRO 00000001)
ACPI: IORT 0x00000100FC027518 0000DC (v00 LINARO SBSAQEMU 20200810 LNRO 00000001)
ACPI: APIC 0x00000100FC02E498 000108 (v04 LINARO SBSAQEMU 20200810 LNRO 00000001)
ACPI: SSDT 0x00000100FC02E898 000067 (v02 LINARO SBSAQEMU 20200810 LNRO 00000001)
ACPI: PPTT 0x00000100FC02FD18 0000B8 (v02 LINARO SBSAQEMU 20200810 LNRO 00000001)
ACPI: GTDT 0x00000100FC02E618 000084 (v03 LINARO SBSAQEMU 20200810 LNRO 00000001)
ACPI: 2 ACPI AML tables successfully acquired and loaded
acpi0 at acpifdt0: Intel ACPICA 20190405
cpu0 at acpi0: unknown CPU (ID = 0x411fd402)
cpu0: package 0, core 0, smt 0
cpu0: IC enabled, DC enabled, EL0/EL1 stack Alignment check enabled
cpu0: Cache Writeback Granule 16B, Exclusives Reservation Granule 16B
cpu0: Dcache line 64, Icache line 64
cpu0: L1 0KB/64B 4-way PIPT Instruction cache
cpu0: L1 0KB/64B 4-way PIPT Data cache
cpu0: L2 0KB/64B 8-way PIPT Unified cache
cpu0: revID=0x0, 4k table, 16k table, 64k table, 16bit ASID
cpu0: auxID=0x1011111110212120, GICv3, CRC32, SHA1, AES+PMULL, rounding, NaN propagation, denormals, 32x64bitRegs, Fused Multiply-Add
cpu1 at acpi0: unknown CPU (ID = 0x411fd402)
cpu1: package 0, core 1, smt 0
gicvthree0 at acpi0: GICv3
gicvthree0: ITS #0 at 0x44081000
gicvthree0: ITS [#0] Devices table @ 0x10009210000/0x80000, Cacheable WA WB, Inner shareable
gicvthree0: ITS [#1] Collections table @ 0x10009290000/0x10000, Cacheable WA WB, Inner shareable

As 9.3 release is quite old I tested NetBSD 10-Beta:

gicvthree0 at acpi0: GICv3
gicvthree0: ITS #0 at 0x44081000
gicvthree0: ITS [#0] Devices table @ 0x10008a60000/0x80000, Cacheable WA WB, Inner shareable
gicvthree0: ITS [#1] Collections table @ 0x10008ae0000/0x10000, Cacheable WA WB, Inner shareable
gtmr0 at acpi0: irq 27
armgtmr0 at gtmr0: Generic Timer (62500 kHz, virtual)
plcom0 at acpi0 (COM0, ARMH0011-0): mem 0x60000000-0x60000fff irq 33
plcom0: console
[..]
NetBSD-10.0_BETA Install System

Went to #netbsd channel on IRC and started disussion. Michael van Elst (mlelstv on irc) gave me a helping hand and debugged the problem. Looks like kernel went into infinite loop on parsing GTDT table from ACPI. Newer branches of NetBSD have additional check there.

Filled bug 57642 for it. And, like in FreeBSD case, it looks like some backport to stable branch is needed.

Summary

Testing platforms for SystemReady compliance needs to include *BSD systems. Linux and NetBSD were fine with our USB controller mess — gave “something is wrong” message and went on. FreeBSD and OpenBSD systems were complaining and stopped booting process.

We also need to do more testing before merging big changes in future. This USB controller mess could be avoided or done better.

SBSA Reference Platform update

2023-09-15T15:30:00+02:00

There were several changes done since my previous post on the topic. So after some discussions I decided to write a post about it.

There are improvements, fixes and even issues with BSA specification.

Versioning related changes

SBSA Reference Platform (“sbsa-ref” in short) is now at version 0.3 one. Note that this is internal number. Machine name is still the same.

First bump was adding GIC data into (minimalistic) device-tree so firmware can configure it without using any magic numbers (as it was before).

Second update added GIC ITS (Interrupt Translation Services) support. Which means that we can have MSI-X interrupts and complex PCI Express setup.

Third time we said goodbye to USB 2.0 (EHCI) host controller. It never worked and only generated kernel warnings. XHCI (USB 3) controller is used instead now. EDK2 enablement is still work in progress.

Firmware improvements

Most of versioning updates involved firmware changes. Information about hardware details gets passed from virtual hardware level to operating system via standard defined ways:

Trusted Firmware (TF-A) gets minimalistic Device-Tree from QEMU
UEFI (EDK2) uses Secure Monitor Calls to get information from TF-A
operating system uses ACPI tables

This way we were able to get rid of part of “magic numbers” from firmware components.

CPU updates

We can use Neoverse V1 cpu core now. It uses Arm v8.4 architecture and brings SVE and a bunch of other interesting features. You may need to update Trusted Firmware to make use of it.

QEMU got Arm Cortex-A710 cpu core support. It is first Arm v9.0 core there. Due to 2⁴⁰ address space we cannot use it for sbsa-ref. But it prepares code for Neoverse N2/V2 cores.

PCI Express changes and disputes

SBSA Reference Platform passes most of BSA ACS tests from PCI Express module:

      *** Starting PCIe tests ***

Operating System View:
 801 : Check ECAM Presence                        : Result:  PASS
 802 : PE - ECAM Region accessibility check       : Result:  PASS
 803 : All EP/Sw under RP in same ECAM Region     : Result:  PASS
 804 : Check RootPort NP Memory Access            : Result:  PASS
 805 : Check RootPort P Memory Access             : Result:  PASS
 806 : Legacy int must be SPI & lvl-sensitive
       Checkpoint --  2                           : Result:  SKIPPED
 808 : Check all 1's for out of range             : Result:  PASS
 809 : Vendor specfic data are PCIe compliant     : Result:  PASS
 811 : Check RP Byte Enable Rules                 : Result:  PASS
 817 : Check Direct Transl P2P Support
       Checkpoint --  1                           : Result:  SKIPPED
 818 : Check RP Adv Error Report
       Checkpoint --  1                           : Result:  SKIPPED
 819 : RP must suprt ACS if P2P Txn are allow
       Checkpoint --  1                           : Result:  SKIPPED
 820 : Type 0/1 common config rule                : Result:  PASS
 821 : Type 0 config header rules                 : Result:  PASS
 822 : Check Type 1 config header rules
       BDF 0x400 : SLT attribute mismatch: 0xFF020100 instead of 0x20100
       BDF 0x500 : SLT attribute mismatch: 0xFF030300 instead of 0x30300
       BDF 0x600 : SLT attribute mismatch: 0xFF040400 instead of 0x40400
       BDF 0x700 : SLT attribute mismatch: 0xFF050500 instead of 0x50500
       BDF 0x800 : SLT attribute mismatch: 0xFF060600 instead of 0x60600
       BDF 0x900 : SLT attribute mismatch: 0xFF080700 instead of 0x80700
       BDF 0x10000 : SLT attribute mismatch: 0xFF020201 instead of 0x20201
       Failed on PE -    0
       Checkpoint --  7                           : Result:  FAIL
 824 : Device capabilities reg rule               : Result:  PASS
 825 : Device Control register rule               : Result:  PASS
 826 : Device cap 2 register rules                : Result:  PASS
 830 : Check Cmd Reg memory space enable
       BDF 400 MSE functionality failure
       Failed on PE -    0
       Checkpoint --  1                           : Result:  FAIL
 831 : Check Type0/1 BIST Register rule           : Result:  PASS
 832 : Check HDR CapPtr Register rule             : Result:  PASS
 833 : Check Max payload size supported           : Result:  PASS
 835 : Check Function level reset                 : Result:  PASS
 836 : Check ARI forwarding enable rule           : Result:  PASS
 837 : Check Config Txn for RP in HB              : Result:  PASS
 838 : Check all RP in HB is in same ECAM         : Result:  PASS
 839 : Check MSI support for PCIe dev             : Result:  PASS
 840 : PCIe RC,PE - Same Inr Shareable Domain     : Result:  PASS
 841 : NP type-1 PCIe supp 32-bit only
       NP type-1 pcie is not 32-bit mem type
       Failed on PE -    0
       Checkpoint --  1                           : Result:  FAIL
 842 : PASID support atleast 16 bits
       Checkpoint --  3                           : Result:  SKIPPED

      One or more PCIe tests failed or were skipped.

     -------------------------------------------------------
     Total Tests run  =   30  Tests Passed  =   22  Tests Failed =    3
     -------------------------------------------------------

As you see some of them require work.

Root ports SLT issue

I reported problem with test 822 to QEMU developers and turned out that it is a bug there. I got patch from Michael S. Tsirkin (one of QEMU PCI maintainers) and it made test pass. I hope it will be merged soon.

PCIe to PCI bridge issue

I wonder how many SBSA physical platforms will use one of those. Probably none, but my testing setup has one.

And it makes test 841 fail. This time problem requires more discussion because BSA specification writes (chapter E.2 PCI Express Memory Space):

When PCI Express memory space is mapped as normal memory, the system must support unaligned accesses to that region. PCI Type 1 headers, used in PCI-to-PCI bridges, and therefore in root ports and switches, have to be programmed with the address space resources claimed by the given bridge. For non-prefetchable (NP) memory, Type 1 headers only support 32-bit addresses. This implies that endpoints on the other end of a PCI-to-PCI bridge only support 32-bit NP BARs.

On the other side we have PCI Express Base Specification Revision 6.0 which, in chapter 7.5.1.2.1, says that BAR can be either 32 or 64-bit long:

Base Address registers that map into Memory Space can be 32 bits or 64 bits wide (to support mapping into a 64-bit address space) with bit 0 hardwired to 0b. For Memory Base Address registers, bits 2 and 1 have an encoded meaning as shown in Table 7-9. Bit 3 should be set to 1b if the data is prefetchable and set to 0b otherwise. A Function is permitted to mark a range as prefetchable if there are no side effects on reads, the Function returns all bytes on reads regardless of the byte enables, and host bridges can merge processor writes into this range 150 without causing errors. Bits 3-0 are read-only.

Table 7-9 Memory Base Address Register Bits 2:1 Encoding

Bits 2:1(b) Meaning

00 Base register is 32 bits wide and can be mapped anywhere in the 32 address bit Memory Space.

01 Reserved

10 Base register is 64 bits wide and can be mapped anywhere in the 64 address bit Memory Space.

11 Reserved

Bits 2:1(b)	Meaning
00	Base register is 32 bits wide and can be mapped anywhere in the 32 address bit Memory Space.
01	Reserved
10	Base register is 64 bits wide and can be mapped anywhere in the 64 address bit Memory Space.
11	Reserved

And pcie-pci-bridge device in QEMU uses 64-bit BAR.

I opened support ticket for it at Arm. Will see how it ends.

Non-secure EL2 virtual timer

Arm v8.1 architecture brought Virtual Host Extension (VHE in short). And it added one more timer: non-secure EL2 virtual timer.

BSA ACS checks for it and we were failing:

 226 : Check NS EL2-Virt timer PPI Assignment         START

       NS EL2 Virtual timer interrupt 28 not received
       Failed on PE -    0
       B_PPI_02
       Checkpoint --  4                           : Result:  FAIL
         END

Turned out that everything to make it pass was already present in QEMU. Except code to enable it for our platform. Two lines of code were enough.

After I sent my small patch, Leif Lindholm extracted timer definitions to separate include file and cleaned code around it to make it easier to compare QEMU code with BSA specification.

Result? Test passes:

 226 : Check NS EL2-Virt timer PPI Assignment         START

       Received vir el2 interrupt
       B_PPI_02
                                       : Result:  PASS
         END

Summary

SBSA Reference Platform in QEMU gets better and better with time. We can emulate more complex systems, information about hardware details gets passed from virtual hardware level to operating system via standard defined ways.

Still have test failures but less than it was in past.

Versioning of sbsa-ref machine

2023-05-23T12:43:00+02:00

QEMU has emulation of several machines. One of them is “sbsa-ref” which stands for SBSA Reference Platform. The Arm server in simpler words.

In past I worked on it when my help was needed. We have CI jobs which run some tests (SBSA ACS, BSA ACS) and do some checks to see how we are with SBSA compliance.

Versioning?

One day there was discussion that we need a way to recognize variants of “sbsa-ref” in some sane way. The idea was to get rid of most of hardcoded values and provide a way to have data going from QEMU up to firmware.

We started with adding “platform version major/minor” fields into DeviceTree. Starting with “0.0” as value. And for some time nothing changed here as some of people working on SBSA Reference Platform changed jobs and other worked on other parts of it.

Note that this is different than other QEMU targets. We do not go “sbsa-ref-8.0”, “sbsa-ref-8.1” way as this would add maintenance work without any gain for us.

During last Linaro Connect we had some discussion on how we want to proceed. And some after (as not everyone got there — UK visa issues).

The plan

The plan is simple:

QEMU adds data into DeviceTree
TF-A parses DeviceTree and extracts data from it
TF-A provides Secure Monitor Calls (SMC) with data from DT
EDK2 uses SMC to gather data from TF-A
EDK2 creates ACPI tables
OS uses ACPI to get hardware information

Implementation

After setting the plan I created a bunch of Jira tickets and started writing code. Some changes were new, some were adapted from our work-in-progress ones.

0.0: Platform version SMC

Trusted Firmware (TF-A) reads DeviceTree from QEMU and provides platform version (PV from now on) up to firmware via SMC. EDK2 reads it and does nothing (as expected).

0.1: GIC data SMC

Firmware knows which platform version we are on so it can do something about it. So we bump the value in QEMU and provide Arm GIC addresses via another SMCs.

TF-A uses those values instead of hardcoded ones to initialize GIC. Then EDK2 does the same.

If such firmware boots on older QEMU then hardcoded values are used and machine is operational still.

0.2: GIC ITS SMC

Here things start to be more interesting. We add Interrupt Translation Service support to GIC. Which means we have LPI, MSI(-X) etc. In other words: have normal, working PCI Express with root ports, proper interrupts etc.

From code side it is like previous step: QEMU adds address to DT, TF-A reads it and provides via SMC to EDK2.

If such firmware boots on older QEMU then ITS is not initialized as it was not present in PV 0.0 system.

0.x: PCIe SMC

Normal PCI Express is present. So let get rid of hardcoded values. Similar steps and behaviour like above.

0.y: go PCIe!

At this step we have normal, working PCI Express structure. So let get rid of some platform devices and replace them with expansion cards:

AHCI (sata storage)
EHCI (usb 2.0)

We can use “ich9-ahci” card instead of former and “qemu-xhci” for latter one.

This step is EDK2 only as we do not touch those parts in TF-A. No real code yet as it needs adding some conditions to existing ASL code so operating system will not get information in DSDT table.

Again: if booted on lower PV then hardcoded values are used.

Other changes

Recently some additional changes to “sbsa-ref” were merged.

We exchanged graphics card from basic VGA one on legacy PCI bus to Bochs one (which uses PCI Express). From firmware or OS view not much changed as both were supported already.

Other change was default processor. QEMU 8.0 brought emulation of Arm Neoverse-N1 cpu. It is enabled in TF-A for a while so we switched to use it by default (instead of ancient Cortex-A57). With move from arm v8.0 to v8.2 we got several cpu features and functionalities.

Future

The above steps are cleanup preparing “sbsa-ref” for future work. We want to be able to change hardware definition more. For example to select exact GIC model (like GIC-600) instead of generic “arm-gic-v3” one.

SBSA Reference Platform is system where most of expansion is expected to happen by adding PCI Express cards.