Marcin Juszkiewicz - linaro

Future of BSA/PC-BSA/SBSA checklist page

2025-01-05T20:03:00+01:00

During last years Arm released some specifications in an effort to help organise standardise their market. We got Server Base System Architecture (SBSA), then Base System Architecture (BSA) and finally PC Base System Architecture (PC-BSA) one. Both SBSA and PC-BSA extend BSA (SBSA 7.0 was rebased on top of BSA).

Did these documents changed the market? That’s a discussion for another time.

The checklist

Each of mentioned specifications comes with a compliance checklist referencing document sections such as B_PE_11 which states:

Each PE must implement a minimum of six breakpoints, two of which must be able to match virtual address, contextID or VMID.

To visualize these checklists I created the Arm BSA/PC-BSA/SBSA checklist page where this data is presented as a table. How it is generated will be explained below.

Running ACS

Following the checklist manually is a difficult task so Arm released Architecture Compliance Suites (ACS in short) for BSA and SBSA specifications which run tests and tell whether your hardware is compliant. PC-BSA so far does not have own ACS yet.

NOTE: I ran compliance suites only on UEFI+ACPI systems so do not know how BSA ACS behaves on DeviceTree based ones.

At the start, ACPI tables are parsed and hardware details are checked, such as GIC, SMMU, PCIe etc. You then get a summary of the information:

 Creating Platform Information Tables
 PE_INFO: Number of PE detected       :    4
 GIC INFO: GIC version                :    v3
 GIC_INFO: Number of GICD             :    1
 GIC_INFO: Number of GICR RD          :    1
 GIC_INFO: Number of ITS              :    1
 TIMER_INFO: System Counter frequency :    1000 MHz
 TIMER_INFO: Number of system timers  :    0
 WATCHDOG_INFO: Number of Watchdogs   :    1
 PCIE_INFO: Number of ECAM regions    :    1
 PCIE_INFO: Number of BDFs found      :    4
 PCIE_INFO: Number of RCiEP           :    2
 PCIE_INFO: Number of RCEC            :    0
 PCIE_INFO: Number of EP              :    1
 PCIE_INFO: Number of RP              :    1
 PCIE_INFO: Number of iEP_EP          :    0
 PCIE_INFO: Number of iEP_RP          :    0
 PCIE_INFO: Number of UP of switch    :    0
 PCIE_INFO: Number of DP of switch    :    0
 PCIE_INFO: Number of PCI/PCIe Bridge :    0
 PCIE_INFO: Number of PCIe/PCI Bridge :    0
 SMMU_INFO: Number of SMMU CTRL       :    1
 SMMU_INFO: SMMU index 00 version     :    v3.1
 Peripheral: Num of USB controllers   :    1
 Peripheral: Num of SATA controllers  :    1
 Peripheral: Num of UART controllers  :    1

The format may differ a bit between both BSA ACS and SBSA ACS but data is there.

Then tests run in groups (PE, Memory map, GIC, SMMU, PCIe etc.). Each test can return one of 3 values: PASS, FAIL or SKIPPED (usually when the requirements to run the test are not met):

  28 : Check Fine Grain Trap Support         
       Failed on PE -    0
       S_L7PE_01
       Checkpoint --  1                           : Result:  FAIL
  29 : Check for ECV support                 
       Failed on PE -    0
       S_L7PE_02
       Checkpoint --  1                           : Result:  FAIL
  30 : Check for AMU Support                 
       Failed on PE -    0
       S_L7PE_03
       Checkpoint --  1                           : Result:  FAIL
  31 : Checks ASIMD Int8 matrix multiplc          : Result:  PASS
  32 : Check for BFLOAT16 extension               : Result:  PASS
  33 : Check PAuth2, FPAC & FPACCOMBINE           : Result:  PASS
  34 : Check for SVE Int8 matrix multiplc         : Result:  PASS
  35 : Check for data gathering hint              : Result:  PASS
  36 : Check WFE Fine tune delay feature     
       Recommened WFE fine-tuning delay feature not implemented
       S_L7PE_09
       Checkpoint --  2                           : Result:  SKIPPED

As you can see when test does not pass you get a tag (like S_L7PE_01) pointing to relevant specification.

Verbose mode

Each ACS can be run in verbose mode by adding “-v 1” to the command line. Amount of detail is increased to the level that logging output is highly recommended. You can compare my older logs:

Back to table stuff

In my sbsa-ref-status repository I have scripts that gather and parse data for QEMU’s SBSA Reference Platform (sbsa-ref in short). The result is set of YAML files (status-bsa.yml and status-sbsa.yml) that contain information how the tests went:

31:
  level: '7'
  status:
    cortex-a57: FAIL
    cortex-a72: FAIL
    max: PASS
    neoverse-n1: FAIL
    neoverse-n2: PASS
    neoverse-v1: PASS
  tags: S_L7PE_04
  title: Checks ASIMD Int8 matrix multiplc

As you see test for S_L7PE_04 passed on some cpu core models and failed on old ones. This pattern continues for other tests and tags. Several entries have only SKIPPED values because the hardware lacked something required to run them.

Those scripts should work with results from other hardware.

Test coverage

Some tags do not have coverage in the ACS. Some tests check for things that are not present in checklists present in the specifications. In these cases it is important to look into ACS documentation:

Both of these expand on the checklists from the specifications with additional information. Which SBSA level test was written for, is it tested in the UEFI or Linux environment, is additional exerciser card required etc.

I used both to expand status-(s)bsa.yml files to ensure all tested entries are listed.

Generation of checklist page

To generate a page with checklist table I use another YAML file: xbsa-checklist.yml. This file maps tags from the specifications into groups and subgroups and keeps information on whether tag required for BSA v1/v2, PC-BSA or SBSA levels. I wrote it by hand and it needs to be updated with every specification update.

Next generate-xbsa.py needs to be run which generates HTML page with the table.

Caveats

Changes to ACS may alter the test numbers or tags used. I reported several issues against both BSA ACS and SBSA ACS about it and they were handled. At this moment all tests have tags assigned.

If the generated page lists “ACS only tests” entries it means one of status-*.yml file needs to be updated because some unhandled tag was used. Or ACS change had error in tag name.

Tests may be renamed — in such cases status files will get new ones. When test numbers change (which is rare) then manual checking may be required.

PC-BSA ACS?

According to Arm developers (BSA ACS issue #395) there will be PC-BSA ACS in first week of December. Once it is released, a new parser will need to be written (like one for BSA ACS or for SBSA ACS) and the page generator updated to use this information.

Future?

This page is one of projects I plan to abandon in 2025. It was a useful tool for checking what needed to be done for the SBSA Reference Platform, either on virtual hardware (QEMU) or in firmware (TF-A + EDK2).

None of Arm hardware I use at home is SBSA compliant. Running BSA ACS on some of those causes them to hang. I do not expect this situation to change in the coming months.

The current page will remain online but I do not plan to invest time in updating it.

Leaving Linaro for 3rd time

2024-12-18T16:26:00+01:00

Some time ago I was informed that Red Hat will not prolong membership for Linaro DataCentre Group. Which for me means end of my 2nd adventure with Linaro.

Third time???

I was in Linaro from April 2010 to end of May 2013 and then from April 2016 to end of current month. So two times.

But I was leaving Linaro twice in past. First in October 2012 when someone decided that it is not the time yet for me to go. And then in May 2013 when I finally left.

The work

Those eight and half years of Linaro work were a good time.

OpenStack

First we were doing OpenStack. Went from “needs hacks” to be ready out of the box for use. During 6 years I did hundreds of patches and countless reviews.

In meantime cloud providers started providing Arm instances and pressure to keep OpenStack working became smaller. Why maintain whole infrastructure when you can rent virtual one?

Part of that jobs was extending CirrOS images to behave properly on UEFI systems (AArch64 and x86-64). Defined CI jobs, handled migration to Github and helped with several releases.

There was some work done on distribution images as well.

SBSA Reference Platform

Then moved to SBSA Reference Platform. This was interesting in the beginning. I had a feeling of being more of a manager than developer. Had to collect ideas from everyone who worked on it and get something working and being acceptable by upstream.

This ended as internal versioning of platform and then we started adding more features and upgrading platform whenever something interesting landed in QEMU. Now sbsa-ref uses Neoverse-N2 cpu by default, can be used with NUMA setup, have defined cpu topology and more.

Learnt firmware stuff (TF-A, EDK2), reviewed countless patches etc.

Too bad that during those years I was not able to buy any SBSA compliant hardware below 2000 EUR :(

Continuous Integration

One of my tasks during whole Linaro time was handling Continuous Integration. Defining new jobs, taking care of old ones, maintaining machines we used as Jenkins runners.

This took me into interesting places. There was a lot of Python. I even managed to be involved in manylinux images used to build Python packages.

What’s next?

I am moving back to Red Hat. There are some positions open where I may fit so have to take a look and choose.

Arm laptops for normal users?

2024-08-16T17:33:00+02:00

There are discussions in development circles about Arm powered laptops since forever. But most of time they do not mention “normal” users. Like your parents, spouses, kids who are not developers. People who turn computer on (cold boot or from suspend does not matter) and expect them to “just work”.

My teenage daughter is one of them. Her current laptop is one of Thinkpad models, previous one was Thinkpad as well. Fedora Linux as operating system serves her needs just fine. But despite my 20 years of work with Arm architecture I am unable to get Arm based laptop for her.

Choices

There are several Arm powered laptops on a market:

Macbooks
Windows on Arm laptops
Chromebooks
SBCs with screen like Pinebook Pro

And all of them have issues when it comes to using Fedora Linux on them.

Macbooks

Thanks to Asahi Linux team we can run Fedora Linux on M1/M2 based Macbooks. Which means second hand market as Apple does not sell those models any more (unless 8GB of ram is enough for you).

There are many things which are not supported:

Thunderbolt
DisplayPort over Thunderbolt
HDMI audio (work in progress)

So you pay for hardware and have features which you cannot use. I use Macbook Pro 2021 (with M1 Pro cpu) for local development and stopped checking how work goes.

Windows on Arm laptops

Qualcomm managed to convince Microsoft to not offer licenses for other vendors which means all we can have are Snapdragon based laptops. Which may work nice under MS Windows but if you want to use Linux then “good luck” is all you can get from me.

Some things work, some do not. I was told that Thinkpad x13s is one of best supported models. Johan Hovold has a Thinkpad x13s status page which lists what works and what needs to be done to have some kind of working laptop.

Definitely not a system for daily use for normal Linux user.

Chromebooks

Laptop to run web browser and Android apps. If this is all you need then go for it. But avoid if you are “normal” user and want to run Linux.

Finding how to enable running anything other than ChromeOS may involve digging through Internet pages, finding how to override ‘write protection’ etc.

Just no. Also Arm ones are usually ram limited.

Pinebook Pro and other SBCs with screen

Those are systems for developers only. Normal users should avoid using them as those systems require someone who knows how to prepare them to work at all.

Find/build proper firmware, put it properly in device (SPI Flash or storage media), keeping things up-to-date may end with partially not working device etc.

For developers those are ‘issues’ to workaround/solve but for normal users it may be ‘update went in background and now all I have is black screen’.

And like with Chromebooks you may be limited by ram size (Pinebook Pro has only 4GB ram).

Conclusion

If you are a normal user who wants to run Linux on a laptop then maybe stay away from Arm powered ones. Leave them for developers and check once/twice per year to see how situation looks.

Linaro Connect MAD24

2024-06-14T13:04:00+02:00

Nowadays we have only one Connect per year. And this year we met in Madrid, Spain.

How did it go for me? Good, better than the previous one.

I attended most of the talks I wanted to see, spoke with countless people, and met most of the people on my “to meet” list.

SystemReady related talks

For me, the main topic at Linaro Connect was SystemReady. There was a track for SystemReady IR on the first day and the next days included additional presentations.

Does SystemReady IR “Just Work”?

Jon Humphreys from Texas Instruments gave an introduction what SystemReady is. Explained how systems operated before it, the definition of “Just Works”, and some aspects of testing. He then discussed issues with the current certification process, such as missing firmware files used during testing and problems with errata documents. He also offered recommendations on how to improve the situation.

Looking back on SystemReady IR

Vincent Stehlé from Arm spoke a lot about of history and timelines of involved specifications, the U-Boot features timeline and several statistics about SystemReady certifications along with lessons learned.

30.9% of certified systems were for IR band (only SR was higher with 36%), mostly for older versions of the specification.

We learned that there are three certification labs besides Arm itself. And the plan is to eventually remove Arm from this role.

It was interesting to see which distributions were used during IR certification. Fedora Linux leads with 33.9%, followed by openSUSE at 33% and Debian at 21.1%. Other distributions include Ubuntu, SLES, Rocky Linux and RHEL.

U-Boot for SystemReady IR — Status and struggles

Ilias Apalodimas from Linaro presented the history of U-Boot getting features required for SystemReady IR support. He highlighted some common issues and offered suggestions for improvement.

This talk nicely expanded on the topics mentioned in the previous two talks.

SystemReady as default option for BSPs: Findings from the last SystemReady IoT workshop

Ilias Apalodimas, Pere Garcia Gutiérrez and Peter Robinson presented conclusions from SystemReady’s Advisory Committee workshop.

One idea was that SoC vendors should provide BSP setup in a way that the resulting firmware would already be SystemReady compliant. This would make it easier for OEM/ODM vendors, with some following their own paths while others adhered to the guidelines.

Qualcomm and SystemReady IR, an overview of Qualcomm support in U-Boot and what the future

Caleb Connolly from Linaro shared the story how Qualcomm support in U-Boot improved from terrible to good. He mentioned several quirks and issues encountered along the way.

Arm System Architecture and SystemReady Update

Dong Wei from Arm provided updates on changes to the SystemReady specifications.

I somehow missed that session. It was on my list, but I got distracted by a discussion.

He explained how changes are managed and reminded us how SystemReady bands depend on specifications. Then presented planned changes to BSA, such as creation of VBSA for virtual environments and increase of requirements for future versions.

Another change mentioned was that the SystemReady LS and LBBR recipe of BBR might be deprecated due to lack of interest.

There will be some changes for the SBSA requirements. So far there are no plans to create SBSA level 8, but if a silicon vendor can fulfil all future requirements, Arm may consider publishing SBSA level 8.

The BBSR (Boot Base Security Requirements) specification may become a requirement for future SystemReady versions.

SBSA Reference Platform in QEMU status update

I gave a talk presenting what we achieved in the past year regarding the SBSA Reference Platform in QEMU, Trusted Firmware and EDK2 projects.

I discussed changes to virtual hardware, firmware and the methods we use to transfer configuration data between layers. I explained why I use “Datatree” term instead of “Devicetree” and outlined our plans for the future.

Other sessions

I attended other sessions as well and watched several ones I missed due to discussions or schedule conflicts. Here I want to write about some of them.

TianoCore community update

Leif Lindholm provided an introduction and updates on the TianoCore EDK2 project.

He covered which specifications it implements, how repositories are structured, the licenses used and the communication methods. Also discussed updates to SSL/TLS libraries and toolchain profiles, contribution rules and recent changes to there rules.

Additionally, he outlined plans for changes to edk2-platforms such as creating stable tags and implementing automated CI.

An Arm laptop project conclusion

Johan Hovold shared information about the state of Linux on the Lenovo Thinkpad x13s laptop (which comes with Windows for Arm by default).

The good part is that most of the support is already merged into upstream projects and the required kernel configuration changes have been included in several distributions.

The bad part? There is still a lot of work to be done. I wonder when vendors will learn how to write proper ACPI tables (x13s uses Devicetree to run Linux).

Rethinking the kernel system call entry

Arnd Bergmann from Linaro spoke about his work on rethinking the Linux kernel system call entry.

He covered how system calls are currently defined now in the Linux kernel. Then showed how he changed it to be more readable.

I have my system calls table but had never looked how kernel code for them looks. Turned out that it can be quite intimidating with macros expanded to macros resulting in complex C code.

Arnd showed that it can be made readable. I hope that his work will land in the Linux kernel soon.

End words

Linaro Connect MAD24 was a success. I got ideas for blog posts, met people who read my posts.

And if you want to watch more talks from conference then there is official Linaro Connect MAD24 playlist on YouTube.

Visit Linaro Resources Hub for video recordings and presentation slides.

ConfigurationManager in EDK2: just say no

2024-04-16T09:44:00+02:00

During my work on SBSA Reference Platform I have spent lot of time in firmware’s code. Which mostly meant Tianocore EDK2 as Trusted Firmware is quite small.

Writing all those ACPI tables by hand takes time. So I checked ConfigurationManager component which can do it for me.

Introduction

In 2018 Sami Mujawar from Arm contributed Dynamic Tables Framework to Tianocore EDK2 project. The goal was to have code which generates all ACPI tables from all those data structs describing hardware which EDK2 already has.

In 2023 I was writing code for IORT and GTDT tables to generate them from C. And started wondering about use of ConfigurationManager.

Mailed edk2-devel ML for pointers, documentation, hints. Got nothing in return, idea went to the shelf.

SBSA-Ref and multiple PCI Express buses

Last week I got SBSA-Ref system booting in NUMA configuration with three separate PCI Express buses. And started working on getting EDK2 firmware to recognize them as such.

Took me a day and pci command listed cards properly:

Shell> pci
   Seg  Bus  Dev  Func
   ---  ---  ---  ----
    00   00   00    00 ==> Bridge Device - Host/PCI bridge
             Vendor 1B36 Device 0008 Prog Interface 0
    00   00   01    00 ==> Network Controller - Ethernet controller
             Vendor 8086 Device 10D3 Prog Interface 0
    00   00   02    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   00   03    00 ==> Bridge Device - Host/PCI bridge
             Vendor 1B36 Device 000B Prog Interface 0
    00   00   04    00 ==> Bridge Device - Host/PCI bridge
             Vendor 1B36 Device 000B Prog Interface 0
    00   01   00    00 ==> Mass Storage Controller - Non-volatile memory subsystem
             Vendor 1B36 Device 0010 Prog Interface 2
    00   40   00    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   40   01    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   41   00    00 ==> Base System Peripherals - SD Host controller
             Vendor 1B36 Device 0007 Prog Interface 1
    00   42   00    00 ==> Display Controller - Other display controller
             Vendor 1234 Device 1111 Prog Interface 0
    00   80   00    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000E Prog Interface 0
    00   80   01    00 ==> Bridge Device - PCI/PCI bridge
             Vendor 1B36 Device 000C Prog Interface 0
    00   81   09    00 ==> Multimedia Device - Audio device
             Vendor 1274 Device 5000 Prog Interface 0
    00   81   10    00 ==> Network Controller - Ethernet controller
             Vendor 8086 Device 100E Prog Interface 0
    00   82   00    00 ==> Mass Storage Controller - Serial ATA controller
             Vendor 8086 Device 2922 Prog Interface 1

Three buses are: 0x00, 0x40 and 0x80. But then I had to tell Operating System about those. Which meant playing with ACPI tables code in C.

So idea came “what about trying ConfigurationManager?”.

Another try

Mailed edk2-devel ML again for pointers, documentation hints. And then looked at code written for N1SDP and started playing with ConfigurationManager…

ConfigurationManager.c has EDKII_PLATFORM_REPOSITORY_INFO struct with hundreds of lines of data (as another structs). From listing which ACPI tables I want to have (FADT, GTDT, APIC, SPCR, DBG2, IORT, MCFG, SRAT, DSDT, PPTT etc.) to listing all hardware details like GIC, PCIe, Timers, CPU and Memory information.

Then code for querying this struct. I thought that CM/DT (ConfigurationManager/DynamicTables) framework will have those already in EDK2 code but no — each platform has own set of functions. Another hundreds of lines to maintain.

Took some time to get it built, then started filling proper data and compared with ACPI tables I had previously. There were differences to sort out. But digging more and more into code I saw that I go deeper and deeper into rabbit hole…

Dynamic systems do not fit CM?

For platforms with dynamic hardware configuration (like SBSA-Ref) I needed to write code which would populate that struct with data on runtime. Check amount of cpu cores and write cpu information (with topology, cache etc), create all GIC structures and mappings. Then same for PCIe buses. Etc. Etc. etc…

STATIC
EFI_STATUS
EFIAPI
InitializePlatformRepository (
  IN  EDKII_PLATFORM_REPOSITORY_INFO  * CONST PlatRepoInfo
  )
{
  GicInfo                 GicInfo;
  CM_ARM_GIC_REDIST_INFO *GicRedistInfo;
  CM_ARM_GIC_ITS_INFO    *GicItsInfo;
  CM_ARM_SMMUV3_NODE     *SmmuV3Info;

  GetGicDetails(&GicInfo);
  PlatRepoInfo->GicDInfo.PhysicalBaseAddress = GicInfo.DistributorBase;

  GicRedistInfo = &PlatRepoInfo->GicRedistInfo[0];

  GicRedistInfo->DiscoveryRangeBaseAddress = GicInfo.RedistributorsBase;

  GicItsInfo = &PlatRepoInfo->GicItsInfo[0];
  GicItsInfo->PhysicalBaseAddress = GicInfo.ItsBase;

  SmmuV3Info = &PlatRepoInfo->SmmuV3Info[0];
  SmmuV3Info->BaseAddress = PcdGet64 (PcdSmmuBase);

  return EFI_SUCCESS;
}

Which in my case can mean even more code written to populate CM struct of structs than it would take to generate ACPI tables by hand.

Summary

ConfigurationManager and DynamicTables frameworks look tempting. There may be systems where it can be used with success. I know that I do not want to touch it again. All those structs of structs may look good for someone familiar with LISP or JSON but not for me.

DT-free EDK2 on SBSA Reference Platform

2024-04-04T13:26:00+02:00

During last weeks we worked on getting rid of DeviceTree from EDK2 on SBSA Reference Platform. And finally we managed!

All code is merged into upstream EDK2 repository.

What?

Someone may wonder where DeviceTree was in SBSA Reference Platform. Wasn’t it UEFI and ACPI platform?

Yes, from Operating System point of view it is UEFI and ACPI. But if you look deeper you will see DeviceTree hidden inside our chain of software components:

/dts-v1/;

/ {
        machine-version-minor = <0x03>;
        machine-version-major = <0x00>;
        #size-cells = <0x02>;
        #address-cells = <0x02>;
        compatible = "linux,sbsa-ref";

        chosen {
        };

        memory@10000000000 {
                reg = <0x100 0x00 0x00 0x40000000>;
                device_type = "memory";
        };

        intc {
                reg = <0x00 0x40060000 0x00 0x10000
                       0x00 0x40080000 0x00 0x4000000>;

                its {
                        reg = <0x00 0x44081000 0x00 0x20000>;
                };
        };

        cpus {
                #size-cells = <0x00>;
                #address-cells = <0x02>;

                cpu@0 {
                        reg = <0x00 0x00>;
                };

                cpu@1 {
                        reg = <0x00 0x01>;
                };

                cpu@2 {
                        reg = <0x00 0x02>;
                };

                cpu@3 {
                        reg = <0x00 0x03>;
                };
        };
};

It is very minimal one, providing us with only those information we need. It does not even pass any compliance checks. For example for Linux, GIC node (/intc/ one) should have gazillion of fields, but we only need addresses.

Trusted Firmware reads it, parses and provides information from it via Secure Monitor Calls (SMC) to upper firmware level (EDK2 in our case). DeviceTree is provided too but we do not read it any more.

Why?

Our goal is to treat software components a bit different then people may expect. QEMU is “virtual hardware” layer, TF-A provides interface to “embedded controller” (EC) layer and EDK2 is firmware layer on top.

On physical hardware firmware assumes some parts and asks EC for the rest of system information. QEMU does not give us that, while giving us a way to alter system configuration more than it would be possible on most of hardware platforms using a bunch of cli arguments.

EDK2 asks for CPU, GIC and Memory. When there is no info about processors or memory, it informs the user and shutdowns the system (such situation does not have a chance of happening but it works as an example).

Bonus stuff: NUMA

Bonus part of this work was adding firmware support for NUMA configuration. When QEMU is run with NUMA arguments then operating system gets whole memory and proper configuration information.

QEMU arguments used:

-smp 4,sockets=4,maxcpus=4
-m 4G,slots=2,maxmem=5G
-object memory-backend-ram,size=1G,id=m0
-object memory-backend-ram,size=3G,id=m1
-numa node,nodeid=0,cpus=0-1,memdev=m0
-numa node,nodeid=1,cpus=2,memdev=m1
-numa node,nodeid=2,cpus=3

How Operating System sees NUMA information:

root@sbsa-ref:~# numactl --hardware
available: 3 nodes (0-2)
node 0 cpus: 0 1
node 0 size: 975 MB
node 0 free: 840 MB
node 1 cpus: 2
node 1 size: 2950 MB
node 1 free: 2909 MB
node 2 cpus: 3
node 2 size: 0 MB
node 2 free: 0 MB
node distances:
node   0   1   2
  0:  10  20  20
  1:  20  10  20
  2:  20  20  10

What next?

There is CPU topology information in review queue. All those sockets, clusters, cores and threads. QEMU will pass it in DeviceTree, TF-A will give it via SMC and then EDK2 will put it in one of ACPI tables (PPTT == Processor Properties Topology Table).

If someone decide to write own firmware for SBSA Reference Platform (like port of U-Boot) then both DeviceTree and set of SMC calls will wait for them, ready to be used to gather hardware information.

Running SBSA Reference Platform

2024-03-25T11:21:00+01:00

Recently people asked me how to run SBSA Reference Platform for their own testing and development. Which shows that I should write some documentation.

But first let me blog about it…

Requirements

To run SBSA Reference Platform emulation you need:

QEMU (8.2+ recommended)
EDK2 firmware files

That’s all. Sure, some hardware resources would be handy but everyone has some kind of computer available, right?

QEMU

Nothing special is required as long as you have qemu-system-aarch64 binary available.

EDK2

We provide EDK2 binaries on CodeLinaro server. Go to “latest/edk2” directory, fetch both “SBSA_FLASH*” files, unpack them and you are ready to go. You may compare checksums (before unpacking) with values present in “latest/README.txt” file.

Those binaries are built from release versions of Trusted Firmware (TF-A) and Tianocore EDK2 plus latest “edk2-platforms” code (as this repo is not using tags).

Building EDK2

If you decide to build EDK2 on your own then we provide TF-A binaries in “edk2-non-osi” repository. I update those when it is needed.

Instructions to build EDK2 are provided in Qemu/SbsaQemu directory of “edk2-platforms” repository.

Running SBSA Reference Platform emulation

Note that this machine is fully emulated. Even on AArch64 systems where virtualization is available.

Let go through example QEMU command line:

qemu-system-aarch64
-machine sbsa-ref
-drive file=firmware/SBSA_FLASH0.fd,format=raw,if=pflash
-drive file=firmware/SBSA_FLASH1.fd,format=raw,if=pflash
-serial stdio
-device usb-kbd
-device usb-tablet
-cdrom disks/alpine-standard-3.19.1-aarch64.iso

At first we select “sbsa-ref” machine (defaults to four Neoverse-N1 cpu cores and 1GB of ram). Then we point to firmware files (order of them is important).

Serial console is useful for diagnostic output, just remember to not press Ctrl-C there unless you want to take whole emulation down.

USB devices are to have working keyboard and pointing device. USB tablet is more useful that USB mouse (-device usb-mouse adds it). If you want to run *BSD operating systems then I recommend to add USB mouse.

And the last entry adds Alpine 3.19.1 ISO image.

System boots to text console on graphical output. For some reason boot console is on serial port.

Adding hard disk

If you want to add hard disk then adding “-hdb disk.img” is enough (“hdb” as cdrom took 1st slot on the AHCI controller).

Handy thing is “virtual FAT drive” which allows to create guest’s drive from directory on the host:

-drive if=ide,file=fat:ro:DIRECTORY_ON_HOST,format=raw

This is useful for running EFI binaries as this drive is visible in UEFI environment. It is not present when operating system is booted.

Adding NVME drive

NVME is composed from two things:

PCIe device
storage drive

So let add it in a nice way, using PCIe root-port:

-device pcie-root-port,id=root_port_for_nvme1,chassis=2,slot=0
  -device nvme,serial=deadbeef,bus=root_port_for_nvme1,drive=nvme
     -drive file=disks/nvme.img,format=raw,id=nvme,if=none

Using NUMA configuration

QEMU can emulate Non-Uniform Memory Access (NUMA) setup. This usually means multisocket systems with memory available per cpu socket.

Example config:

-smp 4,sockets=4,maxcpus=4
-m 4G,slots=2,maxmem=5G
-object memory-backend-ram,size=1G,id=m0
-object memory-backend-ram,size=3G,id=m1
-numa node,nodeid=0,cpus=0-1,memdev=m0
-numa node,nodeid=1,cpus=2,memdev=m1
-numa node,nodeid=2,cpus=3

This adds four cpu sockets and 4GB of memory. First node has 2 cpu cores and 1GB ram, second node has 1 cpu and 3GB of ram and last node has 1 cpu without local memory.

Note that support for such setup in work in progress now (March 2024). We merged required code into TF-A and have set of patches for EDK2 in review. Without them you will see resources only from the first NUMA node.

Complex PCI Express setup

Our platform has GIC ITS support so we can try some complex PCI Express structures.

This example uses PCIe switch to add more PCIe slots and then (to complicate things) puts PCIe-to-PCI bridge into one of them to make use of old Intel e1000 network card:

-device pcie-root-port,id=root_port_for_switch1,chassis=0,slot=0
  -device x3130-upstream,id=up_port1,bus=root_port_for_switch1
    -device xio3130-downstream,id=down_port1,bus=up_port1,chassis=1,slot=0
      -device ac97,bus=down_port1
    -device xio3130-downstream,id=down_port2,bus=up_port1,chassis=1,slot=1
   -device pcie-pci-bridge,id=pci1,bus=down_port2
     -device e1000,bus=pci1,addr=2

Some helper scripts

During last year I wrote some helper scripts for working with SBSA Reference Platform testing. They are stored in sbsa-ref-status repository on GitHub.

May lack up-to-date documentation but can show my way of using the platform.

Summary

SBSA Reference Platform can be used for testing several things. From operating systems to (S)BSA compliance of the platform. Or to check how some things are emulated in QEMU. Or playing with PCIe setups (NUMA systems can have separate PCI Express buses but we do not handle it yet in firmware).

Have fun!

Twenty years of my work with Arm architecture

2024-02-12T12:06:00+01:00

Twenty years ago I bought Sharp Zaurus SL-5500 Linux PDA. With StrongARM SA1110 inside. And I had no idea how it will end…

Hobby

At first it was a hobby. Finding what I can do with it (compared to previous devices running PalmOS) was fun. SharpROM, OpenZaurus 3.2, OpenZaurus 3.3-pre1 etc. New apps, taskbar plugins…

Then I wanted to build something. Found an OpenZaurus toolchain, tried it and went for help. And got “forget this toolchain, we have a new toy: OpenEmbedded”.

And I sunk in…

Took me some time to get familiar with that (lot of n00b questions asked). Then many attempts to get a bootable image. Turned out that my image was one of first properly booting on a SL-5500 device. So I got write access to OpenEmbedded with “now you can merge all of it” message.

OpenZaurus

I spent three years working on OpenZaurus distribution. In my free time. Started as a user, went through being developer and ended as a release manager. Through those years I moved from my SL-5500 to the c760 (donated by user). Had several other Zaurus models on my desk in meantime.

Those were interesting years. I learnt a lot about structure of Linux filesystem, how packaging works, all those differences between build-time and install-time dependencies, sorting out upgrades etc. And how to cooperate with other developers and users. Also how to avoid those who deserve to be ignored.

Embedded Linux developer

At one moment working with Arm architecture stopped being hobby and became daily work.

Matthew Allum from OpenedHand proposed me a full time contract so I left my daily job (as a PHP programmer) and started working on Poky Linux and its version of OpenEmbedded.

New devices, new interesting challenges, new co-workers and customers. Proper development boards (those big ones with lot of interesting connectors for even more expensive equipment), prototypes of misc devices…

After OpenedHand there were other companies interested in my services. Bug Labs with their Java based system on top of Poky Linux, Vernier with school palmtops and others. Still embedded Linux.

Invited speaker

My first conference was FOSDEM 2007. And then were several other ones but I was an attendee rather than speaker.

2009 changed it — Andrea Gallo from ST-Ericsson invited me to give a talk at their workshop in Grenoble. My presentation was about supporting their developer board NHK-15 in Poky Linux and OpenEmbedded.

Next day was ELCE where I gave a talk called “Hacking with OpenEmbedded”. It was also conference where I met Leif Lindholm and Jon Masters for the first time.

Linaro

2010 came, I had over three years of paid embedded Linux work behind me. Canonical asked and we signed papers. Then another papers and I became one of the first 20 people working at Linaro. To work on improving Arm support in Linux.

It was time when working with Arm meant embedded work, just using main distributions instead of embedded ones. All those device specific kernels, images etc.

I did cross compilation toolchains for Debian/Ubuntu, AArch64 bring up using OpenEmbedded and several other tasks.

Red Hat

In 2013 I ended my contract with Canonical, left Linaro and joined Red Hat. And met the other side of Arm world.

I was not doing developer boards or SBC systems. The project was to get Red Hat Enterprise Linux distribution running on 64-bit Arm servers. Before they even existed…

It was a fun time. Native builds in very slow system emulators took hours/days. Prototype servers were faster but also very unstable.

Anyway, we delivered. RHEL 7.2 had AArch64 support. Countless packages got fixed both in RHEL and Fedora.

Linaro again

In 2016 I joined Linaro again (this time as Red Hat engineer). To work on Arm servers. For data centers and clouds.

Virtualization, OpenStack, Big Data, emulation of Arm servers etc.

Acknowledgements

There were many people who helped me on this journey.

First of all Anna Wagner-Juszkiewicz — without her patience and forcing me to create my own company I would not go that far.

Then OpenEmbedded team members:

Philip Balister
Phil Blundell
Florian Boor
Holger Freyther
Koen Kooi
Christopher Larson
Mickey Lauer
Richard Purdie
Holger Schurig
and countless contributors

Some work changes related people:

Cliff Brake (my first freelance OE jobs)
Tim Bird (CELF was my first customer)
Matthew Allum (full time contract so I left web development)
Christian Reis (Canonical/Linaro job)
Jon Masters (“come, work at Red Hat with us”)

Linaro folks:

Fathi Boudra
Andrea Gallo
Gema Gomez
Lee Jones
Leif Lindholm
Sahaj Sarup
Ebba Simpson
Riku Voipio
Linus Walleij
Wookey

Of course I am unable to list everyone. During those twenty years I met many people, some became friends, some became adversaries. There are faces/names I remember (and more of those I should remember). And there are people who recognize me when I do not recognize them (sorry folks).

Hall of fame shelf

There is a plan to frame several devices from my past:

Sharp Zaurus SL-5500 (from which all that started)
Atmel AT91SAM9263EK (first developer board I owned)
Applied Micro Mustang (first AArch64 server)

And who knows, maybe some other hardware but first would need to get it. Some already got recycled, some had to be destroyed.

Plans for next years?

I plan to continue working on Arm architecture. It brings me fun still.

And who knows, maybe one day some standards compliant AArch64 based system will replace my x86-64 desktop. All those UEFI, ACPI, SBSA, SBBR, SystemReady buzzwords in working piece of hardware. Just without spending thousands of €€€ for it.