This post is part 2 of the "SBSA Reference Platform in QEMU" series:
- Versioning of sbsa-ref machine
- SBSA Reference Platform update
- Testing *BSD on SBSA Reference Platform
There were several changes done since my previous post on the topic. So after some discussions I decided to write a post about it.
There are improvements, fixes and even issues with BSA specification.
Versioning related changes
SBSA Reference Platform (“sbsa-ref” in short) is now at version 0.3 one. Note that this is internal number. Machine name is still the same.
First bump was adding GIC data into (minimalistic) device-tree so firmware can configure it without using any magic numbers (as it was before).
Second update added GIC ITS (Interrupt Translation Services) support. Which means that we can have MSI-X interrupts and complex PCI Express setup.
Third time we said goodbye to USB 2.0 (EHCI) host controller. It never worked and only generated kernel warnings. XHCI (USB 3) controller is used instead now. EDK2 enablement is still work in progress.
Most of versioning updates involved firmware changes. Information about hardware details gets passed from virtual hardware level to operating system via standard defined ways:
- Trusted Firmware (TF-A) gets minimalistic Device-Tree from QEMU
- UEFI (EDK2) uses Secure Monitor Calls to get information from TF-A
- operating system uses ACPI tables
This way we were able to get rid of part of “magic numbers” from firmware components.
We can use Neoverse V1 cpu core now. It uses Arm v8.4 architecture and brings SVE and a bunch of other interesting features. You may need to update Trusted Firmware to make use of it.
QEMU got Arm Cortex-A710 cpu core support. It is first Arm v9.0 core there. Due to 240 address space we cannot use it for sbsa-ref. But it prepares code for Neoverse N2/V2 cores.
PCI Express changes and disputes
SBSA Reference Platform passes most of BSA ACS tests from PCI Express module:
*** Starting PCIe tests *** Operating System View: 801 : Check ECAM Presence : Result: PASS 802 : PE - ECAM Region accessibility check : Result: PASS 803 : All EP/Sw under RP in same ECAM Region : Result: PASS 804 : Check RootPort NP Memory Access : Result: PASS 805 : Check RootPort P Memory Access : Result: PASS 806 : Legacy int must be SPI & lvl-sensitive Checkpoint -- 2 : Result: SKIPPED 808 : Check all 1's for out of range : Result: PASS 809 : Vendor specfic data are PCIe compliant : Result: PASS 811 : Check RP Byte Enable Rules : Result: PASS 817 : Check Direct Transl P2P Support Checkpoint -- 1 : Result: SKIPPED 818 : Check RP Adv Error Report Checkpoint -- 1 : Result: SKIPPED 819 : RP must suprt ACS if P2P Txn are allow Checkpoint -- 1 : Result: SKIPPED 820 : Type 0/1 common config rule : Result: PASS 821 : Type 0 config header rules : Result: PASS 822 : Check Type 1 config header rules BDF 0x400 : SLT attribute mismatch: 0xFF020100 instead of 0x20100 BDF 0x500 : SLT attribute mismatch: 0xFF030300 instead of 0x30300 BDF 0x600 : SLT attribute mismatch: 0xFF040400 instead of 0x40400 BDF 0x700 : SLT attribute mismatch: 0xFF050500 instead of 0x50500 BDF 0x800 : SLT attribute mismatch: 0xFF060600 instead of 0x60600 BDF 0x900 : SLT attribute mismatch: 0xFF080700 instead of 0x80700 BDF 0x10000 : SLT attribute mismatch: 0xFF020201 instead of 0x20201 Failed on PE - 0 Checkpoint -- 7 : Result: FAIL 824 : Device capabilities reg rule : Result: PASS 825 : Device Control register rule : Result: PASS 826 : Device cap 2 register rules : Result: PASS 830 : Check Cmd Reg memory space enable BDF 400 MSE functionality failure Failed on PE - 0 Checkpoint -- 1 : Result: FAIL 831 : Check Type0/1 BIST Register rule : Result: PASS 832 : Check HDR CapPtr Register rule : Result: PASS 833 : Check Max payload size supported : Result: PASS 835 : Check Function level reset : Result: PASS 836 : Check ARI forwarding enable rule : Result: PASS 837 : Check Config Txn for RP in HB : Result: PASS 838 : Check all RP in HB is in same ECAM : Result: PASS 839 : Check MSI support for PCIe dev : Result: PASS 840 : PCIe RC,PE - Same Inr Shareable Domain : Result: PASS 841 : NP type-1 PCIe supp 32-bit only NP type-1 pcie is not 32-bit mem type Failed on PE - 0 Checkpoint -- 1 : Result: FAIL 842 : PASID support atleast 16 bits Checkpoint -- 3 : Result: SKIPPED One or more PCIe tests failed or were skipped. ------------------------------------------------------- Total Tests run = 30 Tests Passed = 22 Tests Failed = 3 -------------------------------------------------------
As you see some of them require work.
Root ports SLT issue
I reported problem with test 822 to QEMU developers and turned out that it is a bug there. I got patch from Michael S. Tsirkin (one of QEMU PCI maintainers) and it made test pass. I hope it will be merged soon.
PCIe to PCI bridge issue
I wonder how many SBSA physical platforms will use one of those. Probably none, but my testing setup has one.
And it makes test 841 fail. This time problem requires more discussion because BSA specification writes (chapter E.2 PCI Express Memory Space):
When PCI Express memory space is mapped as normal memory, the system must support unaligned accesses to that region. PCI Type 1 headers, used in PCI-to-PCI bridges, and therefore in root ports and switches, have to be programmed with the address space resources claimed by the given bridge. For non-prefetchable (NP) memory, Type 1 headers only support 32-bit addresses. This implies that endpoints on the other end of a PCI-to-PCI bridge only support 32-bit NP BARs.
On the other side we have PCI Express Base Specification Revision 6.0 which, in chapter 220.127.116.11.1, says that BAR can be either 32 or 64-bit long:
Base Address registers that map into Memory Space can be 32 bits or 64 bits wide (to support mapping into a 64-bit address space) with bit 0 hardwired to 0b. For Memory Base Address registers, bits 2 and 1 have an encoded meaning as shown in Table 7-9. Bit 3 should be set to 1b if the data is prefetchable and set to 0b otherwise. A Function is permitted to mark a range as prefetchable if there are no side effects on reads, the Function returns all bytes on reads regardless of the byte enables, and host bridges can merge processor writes into this range 150 without causing errors. Bits 3-0 are read-only.
Table 7-9 Memory Base Address Register Bits 2:1 Encoding
Bits 2:1(b) Meaning 00 Base register is 32 bits wide and can be mapped anywhere in the 32 address bit Memory Space. 01 Reserved 10 Base register is 64 bits wide and can be mapped anywhere in the 64 address bit Memory Space. 11 Reserved
And pcie-pci-bridge device in QEMU uses 64-bit BAR.
I opened support ticket for it at Arm. Will see how it ends.
Non-secure EL2 virtual timer
Arm v8.1 architecture brought Virtual Host Extension (VHE in short). And it added one more timer: non-secure EL2 virtual timer.
BSA ACS checks for it and we were failing:
226 : Check NS EL2-Virt timer PPI Assignment START NS EL2 Virtual timer interrupt 28 not received Failed on PE - 0 B_PPI_02 Checkpoint -- 4 : Result: FAIL END
Turned out that everything to make it pass was already present in QEMU. Except code to enable it for our platform. Two lines of code were enough.
After I sent my small patch, Leif Lindholm extracted timer definitions to separate include file and cleaned code around it to make it easier to compare QEMU code with BSA specification.
Result? Test passes:
226 : Check NS EL2-Virt timer PPI Assignment START Received vir el2 interrupt B_PPI_02 : Result: PASS END
SBSA Reference Platform in QEMU gets better and better with time. We can emulate more complex systems, information about hardware details gets passed from virtual hardware level to operating system via standard defined ways.
Still have test failures but less than it was in past.