Marcin Juszkiewicz

During last years lot of work went into design of SIMD instructions for different cpus. X86-64 has some, Power has and so does AArch64.

But why I am writing about it? Simple — build failures. Or rather test failures. Especially in scientific software like HepMC or alglib. They build fine but that’s all.

The difference was small — something like e-15 or smaller but it was enough to make tests fail. And what to blame? SIMD of course.

AArch64 has FMA (fused multiply-add) instructions which speed up calculations but result is more precise than traditional way when all operations are done one by one. This is enough for tests to fail :-(

But there is solution for it. To degrade you need to add “-ffp-context=off” to compiler flags. This disables use of FMA so results of tests are same as on x86-64 (on pre-Haswell/Bulldozer cores). As a bonus it works on powerpc64(le) and s390(x) too.

Thanks goes to David Abdurachmanov and Andrew Pinski for finding out which exactly flag needs to be used (instead of -fno-expensive-optimisations I used before).

During fixing software to make it build and run on AArch64 sooner or later you can meet magical constant named PAGE_SIZE. And in most situations it will be used in a wrong way.

What it does is simple — tell how big page size is. But it does not work that way on AArch64 architecture as we have different values possible: 4K, 16K (may not be supported in all cpus) and 64K with latter being used in Fedora and other distributions. There were some packages which we built at time of 4K kernel being used and then wondered why things fail under 64K kernel…

But how to handle it as a userspace software developer? Simplest solution is to use sysconf(_SC_PAGESIZE) function call (same as getpagesize()). But remember to not hardcode anything based on what you get as a result. Otherwise your application can misbehave when run on kernel with other PAGE_SIZE size.

The good part is that if someone uses PAGE_SIZE constant in code then it will just do not compile on AArch64 as it is not present in system headers. From what I checked sys/user.h header has it defined on some platforms and does not on other so it can not be assumed as available.

UPDATE: added 16K page size which may not be supported in some cpus.

As I want to have AArch64 desktop running with as small amount of changes needed as possible I decided that it is time to get Xserver to just run on APM Mustang.

Current setup

To get X11 running I need to have xorg.conf file. And this feels strange as on x86(-64) I got rid of it years ago.

Config snippet is small and probably could be even smaller:

Section "Device"
        Identifier " radeon"
        Driver  "radeon"
        BusID "PCI:1:0:0"
EndSection

Section "Screen"
        Identifier      "Screen"
        Device          "radeon"
EndSection

Section "DRI"
        Mode 0666
EndSection

Without it Xserver failed to find graphics card.

Searching for solution

I cloned Xserver git repository and started hacking. During several hours (split into few days) I added countless LogMessage() calls to source code, generated few patches and sent them to x-devel ML. And finally found out why it did not work.

Turns out that I was wrong — Xserver was able to find graphics card. But then it went to platform_find_pci_info() function, called pci_device_is_boot_vga() and rejected it.

Why? Because firmware from Applied Micro did not initialized card so kernel decided not to mark it as boot gfx one. I do not know is it possible to get UEFI to properly initialize pcie card on AArch64 architecture but there are two other ways to get it working.

hack Xserver

We can hack Xserver to not check for pci_device_is_boot_vga() or to use first available card if it returns false:

diff a/hw/xfree86/common/xf86platformBus.c b/hw/xfree86/common/xf86platformBus.c
index f1e9423..d88c58e 100644
--- a/hw/xfree86/common/xf86platformBus.c
+++ b/hw/xfree86/common/xf86platformBus.c
@@ -136,7 +136,8 @@ platform_find_pci_info()
     if (info) {
         pd->pdev = info;
         pci_device_probe(info);
-        if (pci_device_is_boot_vga(info)) {
+        if (pci_device_is_boot_vga(info) || xf86_num_platform_devices == 1)
+        {
             primaryBus.type = BUS_PLATFORM;
             primaryBus.id.plat = pd;
         }

This may not work on multi-gpu systems. In that case try removing “== 1” part.

hack Linux kernel

If firmware does not give us boot gfx card then maybe we can mark first one as such and everything will work? This is how PowerPC has it solved. So let’s take their code:

diff --git a/arch/arm64/kernel/pci.c b/arch/arm64/kernel/pci.c
index b3d098b..eea39ba 100644
--- a/arch/arm64/kernel/pci.c
+++ b/arch/arm64/kernel/pci.c
@@ -18,6 +18,7 @@
 #include <linux/of_pci.h>
 #include <linux/of_platform.h>
 #include <linux/slab.h>
+#include <linux/vgaarb.h>

 #include

@@ -84,3 +85,15 @@ struct pci_bus *pci_acpi_scan_root()
        return NULL;
 }
 #endif
+
+static void fixup_vga(struct pci_dev *pdev)
+{
+       u16 cmd;
+
+       pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+       if ((cmd & (PCI_COMMAND_IO | PCI_COMMAND_MEMORY)) || !vga_default_device())
+               vga_set_default_device(pdev);
+
+}
+DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_ANY_ID, PCI_ANY_ID,
+                             PCI_CLASS_DISPLAY_VGA, 8, fixup_vga);

Summary

Both hacks work. I can just run Xserver and get X11 working. But which one will get into upstream and then to Fedora and other Linux distributions? Time will show.

There are some issues with those solutions. If there are multiple graphics cards in a system then which one is primary one? Can their order change after firmware or kernel update?

Thanks goes to Dave Airlie for help with Xserver, Mark Salter for pointing me to PowerPC solution and Matthew Garrett for discussion about issues with kernel solution.

During years of development GCC got several switches which are considered obsolete/deprecated now. And as such they are not available for new ports. Guess what? AArch64 has such status too.

One of switches is “-posix” one. It is not needed anymore as “_POSIX_SOURCE” macro deprecated it:

Macro: _POSIX_SOURCE

If you define this macro, then the functionality from the POSIX.1 standard (IEEE Standard 1003.1) is available, as well as all of the ISO C facilities.

But it happens sometimes (I saw it in pdfedit 0.4.5 which is so old that it still uses Qt3). So if you find it somewhere then please save world with “s/-posix/-D_POSIX_SOURCE/g” :)

Week ago I fetched git repositories of all Fedora packages. This took a while as there are over 26 thousands of them. Now I am going through them and check/add AArch64 bits.

One of changes is Valgrind support. As there is only one architecture in Fedora which does not have Valgrind support I plan to submit pile of patches which will enable it for all except s390 (32-bit s/390 - we hope it will die sooner than later).

Usual piece of spec file looks like this:

  %ifarch %{ix86} x86_64
  BuildRequires: valgrind
  %endif

My change will look like this:

  %ifnarch s390
  BuildRequires: valgrind
  %endif

Sure, some maintainers do not like to look at their packages on architectures other than x86(-64) because they do not own hardware, do not like being hit by bugs in valgrind or have other excuses. In such situations I will be fine with note in spec file about it - like “cscppc” package maintainer did.

I am aware that some packages may fail due to bug in valgrind implementation on some secondary architecture. But then we can let know developers so it will get improved (while in same time package may get this build dependency disabled for a while).

I am not travelling as much as in previous year, but still try to visit some interesting places when I can. As it usually mean going abroad new sim cards are bought for such visits. Or not.

During last weeks I was in Germany, Iceland, USA and Canada. Needed Internet access in each of them but not everywhere that meant new sim card.

In Germany I am using ALDI Talk card which I bought few months ago. Cost was 12€ with 10€ credit on card. Internet access comes with 500MB data limit per day for 2€ and it usually covers my needs (it gets insanely slow after reaching limit).

Iceland was typical “one time use” card so I went for Nova as it was first available one. Costed one or two thousands ISK and gave me enough data for my 24h visit.

Then USA. Country where even prepaid is expensive. Bought T-Mobile sim card in 7/11 for 47USD including 35USD of credit of which 30USD went for 5GB data package. Getting card running requires registration with codes from starter kit. During 11 days of stay I used about 3GB of it. There were also huge amount of open wifi networks.

Canada was next on list. No sim card this time as 30CAD + 10CAD for 1GB of data was total rip-off. Living on open wifi networks sucks. With my next visit I wait until they got some sane prices ;D

How it looks in other countries?

Poland - buy any sim card in kiosk for 5PLN (1.25€), put into device and usually 1GB of data is available.

Czech Republic - last time I used O² with 375MB data package but do not remember price for it.

Belgium - 10€ on Lycamobile card (card available for free in shops) gave me 3GB data (for 7€).

Finland - Saunalahti card costs 4€ with 6€ credit. Maximum price for Internet access per day is 1.70€ so one card is fine for 3 days. Then you just buy next one or top up with minimum 10€.

I have also cards from UK, Hong Kong, Lithuania and Latvia but do not remember details any more. But all can be always checked on prepaid data sim card wiki.

96boards is an idea from Linaro to produce some 32 and 64-bit ARM boards. So far there were two boards released in “consumer” format and few more announced of rumoured. The specification also lists “extended” version which has space for some more components.

But during Red Hat Summit there was announcement from AMD with mention of “enterprise” format:

How would you like an affordable and compact 160x120mm board to jump start your development efforts with AArch64? AMD and Linaro have been collaborating to develop a 96Boards Enterprise Edition (EE) specification that is ideal for the individual developer. Targeting the server and networking markets, the board will feature a 4-core AMD Opteron A1100 Series processor with two SO-DIMM memory slots, PCIe®, USB, SATA, and Gigabit Ethernet capabilities. Popular operating systems such as CentOS, Fedora, and Red Hat Enterprise Linux Server for ARM Development Preview are targeted for use with this particular board. Additional software downloads, updates, and a forum for software developers will be available via the 96Boards web site. The board is slated to be available in 2H 2015 from distribution partners worldwide and it will be supported through the Linaro Enterprise Group’s 96Boards.org site.

I do wonder where from they took idea to name yet-another-crazy-non-standard board format “Enterprise Edition”. In my understanding what enterprise user like is something which just works and comes with support and does not require crazy embedded nonsense hacks.

So when I saw post from Jeff Underhill with photos of the board I noticed few arghs.

First of course is board format. 160x120mm does not sound like any industrial format. Nano-ITX is 120x120, Mini-ITX is 170x170mm. But everyone knows that enterprise people love to be creative and make own cases. Why it was not done as 170x120 with partial compatibility with mini-itx cases?

Second thing (related to first) is connectors placing. With PCI-Express x16 slot (with x8 signals) I wonder how it will look when some cables go one side or the other while card sticks out of board. With SATA ports moved to the other side there would be space for USB and Ethernet ports so all cables would be in same area. Note also molex connector to give power to SATA disks.

Nice that there are two memory slots (DDR3 ECC SO-DIMM). But with second on the bottom we probably can say goodbye to all PC cases as it would not fit. Yay for creativity when it comes to cases (again).

There are holes to mount heatsink above CPU. From quick look I think that those for FM2 socket may fit.

HDMI connector suggests some graphics to be present. I did not heard about Radeon core inside AMD Seattle CPU but it could change since last check.

But even with those “issues” I would like to have that board ;)

Few years ago I told that I would not fly to USA. Then I changed job and had to fly there for conference. And another one and another… But it was always travel just for conference and one or two days of sightseeing. But not this year.

In August I am going to USA for Fedora Flock conference which takes place in Rochester, NY, USA on 12-15th. But I plan to spend a bit more time on american continent…

First flight is Berlin, Germany to Reykjavík, Iceland where I could just change flight and go to Boston, MA, USA. But why would I? Price of tickets does not change if I stay there for 24h. So I will do some sightseeing, buy a fridge magnet to my collection and then fly.

Boston is a place where I would like to spend few days. Take a look at city, visit Cambridge, MA just because it is younger brother of Cambridge, UK etc. Hope to go to Westford to visit Red Hat office and met some team members and co-workers there. Nothing planned yet, not even hotels.

Then trip to Rochester, NY. Probably by train or bus. Then conference, sightseeing, meeting people etc. I share hotel room with Paul Whalen so probably there will be some interesting discussions ;)

Another part will be visiting Niagara Falls. Hope for both US and Canadian sides. From there I go to Toronto and then on 19th August flying back home.

If someone want to meet me then just ping me. Hints for Canadian prepaid are welcome (in US T-Mobile with 30$ plan should be fine).