Tag Archives: linaro

2 years of AArch64 work

I do not remember exactly when I started working on ARMv8 stuff. Checked old emails from Linaro times and found that we discussed AArch64 bootstrap using OpenEmbedded during Linaro Connect Asia (June 2012). But it had to wait a bit…

First we took OpenEmbedded and created all tasks/images we needed but built them for 32-bit ARM. But during September we had all toolchain parts available: binutils was public, gcc was public, glibc was on a way to be released. I remember that moment when built first “helloworld” — probably as one of first people outside ARM and their hardware partners.

At first week of October we had ARMv8 sprint in Cambridge, UK (in Linaro and ARM offices). When I arrived and took a seat I got information that glibc just went public. Fetched, rebased my OpenEmbedded tree to drop traces of “private” patches and started build. Once finished all went public at git.linaro.org repository.

But we still lacked hardware… The only thing available was Versatile Express emulator which required license from ARM Ltd. But then free (but proprietary) emulator was released so everyone was able to boot our images. OMG it was so slow…

Then fun porting work started. Patched this, that, sent patches to OpenEmbedded and to upstream projects and time was going. And going.

In January 2013 I started X11 on emulated AArch64. Had to wait few months before other distributions went to that point.

February 2013 was nice moment as Debian/Ubuntu team presented their AArch64 port. It was their first architecture bootstrapped without using external toolchains. Work was done in Ubuntu due to different approach to development than Debian has. All work was merged back so some time later Debian also had AArch64 port.

It was March or April when OpenSUSE did mass build of whole distribution for AArch64. They had biggest amount of packages built for quite long time. But I did not tracked their progress too much.

And then 31st May came. A day when I left Linaro. But I was already after call with Red Hat so future looked quite bright ;D

June was month when first silicon was publicly presented. I do not know what Jon Masters was showing but it probably was some prototype from Applied Micro.

On 1st August I got officially hired by Red Hat and started month later. My wife was joking that next step would be Retired Software Engineer ;D

So I moved from OpenEmbedded to Fedora with my AArch64 work. Lot of work here was already done as Fedora developers were planning 64-bit ARM port few years before — when it was at design phase. So when Fedora 15 was bootstrapped for “armhf” it was done as preparation for AArch64. 64-bit ARM port was started in October 2012 with Fedora 17 packages (and switched to Fedora 19 during work).

My first task at Red Hat was getting Qt4 working properly. That beast took few days in foundation model… Good that we got first hardware then so it went faster. 1-2 months later and I had remote APM Mustang available for my porting work.

In January 2014 QEmu got AArch64 system emulation. People started migrating from foundation model.

Next months were full of hardware announcements. AMD, Cavium, Freescale, Marvell, Mediatek, NVidia, Qualcomm and others.

In meantime I decided to make crazy experiment with OpenEmbedded. I was first to use it to build for AArch64 so why not be first to build OE on 64-bit ARM?

And then June came. With APM Mustang for use at home. Finally X11 forwarding started to be useful. One of first things to do was running firefox on AArch64 just to make fun of running software which porting/upstreaming took me biggest amount of time.

Did not took me long to get idea of transforming APM Mustang (which I named “pinkiepie” as all machines at my home are named after cartoon characters) into ARMv8 desktop. Still waiting for PCI Express riser and USB host support.

Now we have October. Soon will be 2 years since people got foundation model available. And there are rumors about AArch64 development boards in production with prices below 100 USD. Will do what needed to get one of them on my desk ;)

It is 10 years of Linux on ARM for me

It was somewhere between 7th and 11th February 2004 when I got package with my first Linux/ARM device. It was Sharp Zaurus SL-5500 (also named “collie”) and all started…

At that time I had Palm M105 (still own) and Sony CLIE SJ30 (both running PalmOS/m68k) but wanted hackable device. But I did not have idea what this device will do with my life.

Took me about three years to get to the point where I could abandon my daily work as PHP programmer and move to a bit risky business of embedded Linux consulting. But it was worth it. Not only from financial perspective (I paid more tax in first year then earned in previous) but also from my development. I met a lot of great hackers, people with knowledge which I did not have and I worked hard to be a part of that group.

I was a developer in multiple distributions: OpenZaurus, Poky Linux, Ångström, Debian, Maemo, Ubuntu. My patches landed also in many other embedded and “normal” ones. I patched uncountable amount of software packages to get them built and working. Sure, not all of those changes were sent upstream, some were just ugly hacks but this started to change one day.

Worked as distribution leader in OpenZaurus. My duties (still in free time only) were user support, maintaining repositories and images. I organized testing of pre-release images with over one hundred users — we had all supported devices covered. There was “updates” repository where we provided security fixes, kernel updates and other improvements. I also officially ended development of this distribution when we merged into Ångström.

I worked as one of main developers of Poky Linux which later became Yocto Linux. Learnt about build automation, QA control, build-after-commit workflow and many other things. During my work with OpenedHand I also spent some time on learning differences between British and American versions of English.

Worked with some companies based in USA. This allowed me to learn how to organize teamwork with people from quite far timezones (Vernier was based in Portland so 9 hours difference). It was useful then and still is as most of Red Hat ARM team is US based.

I remember moments when I had to explain what I am doing at work to some people (including my mom). For last 1.5 year I used to say “building software for computers which do not exist” but this is slowly changing as AArch64 hardware exists but is not on a mass market yet.

Now I got to a point when I am recognized at conferences by some random people when at FOSDEM 2007 I knew just few guys from OpenEmbedded (but connected many faces with names/nicknames there).

Played with more hardware then wanted. I still have some devices which I never booted (FRI2 for example). There are boards/devices which I would like to get rid of but most of them is so outdated that may go to electronic trash only.

But if I would have an option to move back that 10 years and think again about buying Sharp Zaurus SL-5500 I would not change it as it was one of the best things I did.

My own company started 8th year today

Seven years ago I created my one person company. And it was one of best things I did in my life.

All started in 2006 when I started doing some small paid jobs around OpenEmbedded. Small things: solving build problems, updating recipes, adding new ones. But companies prefer to get invoice for such stuff instead of just giving money…

So one day I went to city hall and created what was then called “HaeRWu Marcin Juszkiewicz”. I changed name 2 years later and got rid of that ‘impossible to pronouce’ part.

There were many different clients for my consulting work. CELF was my first one, later I dropped my daily work and started remote work for OpenedHand. When they were acquired by Intel I got quite nice offer but preferred not to move to UK so went own way. From time perspective I do not know was it right decision ;)

So I looked at market around OpenEmbedded and started working with Bug Labs and few smaller jobs for other clients (some knew me from OpenedHand times). Also had job proposal from Canonical for their newly created ARM team but nothing came from it.

Time passed. One and half-year later Canonical made another attempt and this time I though “why not?”. So I went there just to be moved outside to a team which did not have any official name (other than NewCo or New Core which you may heard somewhere). And that team became Linaro some days later.

At Linaro I did lot of cleanup in Debian/Ubuntu toolchain components, added bootstrapable cross toolchain and fixed several packages (also created some new ones). But then, just when I was supposed to move to Canonical, new things came and AArch64 took my whole time.

ARMv8 work was great time. Learnt new things about OpenEmbedded, saw how project moved during those two years when I did not follow it’s development. Och it was good time.

But good things have to end one day. And so did my time at Linaro. But at around same time I started talking with several companies around Linaro to find a new place for me.

And I found it at Red Hat. Took a bit of time to get everything set up but I think that it was worth it. But due to the fact that I am employee not contractor I will suspend and in few months shutdown my consulting company.

It served me well. I came from being person not recognizable to someone who is known by people who I see for first time. It is good feeling ;)

Remote Linaro Connect

As I left Linaro I am not at Linaro Connect in Dublin, Ireland. But decided to access at least keynotes (which were always interesting) and probably also some sessions.

George Grey introduction speech was fine. Number, standard Linaro information (what it is, how many people etc). Worth watching if you want some updates but may be skipped.

Then James Bottomley from Parallels spoke about server side of computing. Unix, Windows NT, Linux, Itanium, AMD64/x86-64, Atom, 64-bit ARM are good keywords for his presentation. I liked few things:

  • Itanium iceberg description (why IA64 was disaster without IA32 compatibility)
  • Atom contra ARM “power fight” (hard to tell which one will be better for servers when it comes to energy use)
  • mentioning of Blackadder (I know what it is but never watched more then one episode)

There were some issues with bandwidth so there are few moments in video where audio/video stops and you get group photo from previous Linaro Connect instead. But this is “normal” on first day and I hope that will get fixed by network team.

There are few sessions today which I plan to take a look. ARMv8 Status one and the one about Linux scheduler.

On my own again

After 3 years at Linaro I have decided to not continue my trip with Canonical. So now I am back to be on my own again.

I will not write why I made such decision but also want to mention that time at Canonical/Linaro was good. I learnt some new tools and added some of them to “avoid if possible” list. From products created and developed at Canonical there are Bazaar and Unity. Both have replacements which I like more.

What next? Will see — I had some meetings and discussions. But I am open for job offers of course ;) It can be Debian or OpenEmbedded or Ubuntu or other ARM Linux related as long I do not have to move.

RedHat and real AArch64 hardware today

In around 3 hours from now Jon Masters from RedHat will have first live multi-node cluster 64-bit ARM silicon demo running Fedora. On real hardware…

It amazing how it went from new architecture announcement though simulators, boostrapping distributions to running those on real hardware. When I was working on AArch64 we were said that it will take one more year before we see devices not emulators or FPGAs (which I heard were slower than simulator).

I hope to work on AArch64 support again — one day in a future.

BTW — there will be no live streaming but Jon wrote that there will be video posted in short time after.


When last time I was in Cambridge we had a discussion about ARM processors. Paweł used term “ARMology” then. And with recent announcement of Cortex-A12 cpu core I thought that it may be a good idea to write a blog post about it.

Please note that my knowledge of ARM processors started in 2003 so I can make mistakes in everything older. Tried to understand articles about old times but sometimes they do not keep one version of story.

Ancient times

ARM1 got released in 1985 as CPU add-on to BBC Micro manufactured by Acorn Computers Ltd. as result of few years of research work. They wanted to have new processor to replace ageing 6502 used in BBC Micro and Acorn Electron and none of existing ones did not fit their requirements. Note that it was not market product but rather development tool made available for selected users.

But it was ARM2 which landed in new computers — Acorn Archimedes (1987 year). Had multiply instructions added so new version of instruction set was created: ARMv2. Just 8MHz clock but remember that it was first computer with new CPU…

Then ARM3 came — with cache controller integrated and 25MHz clock. ISA was bumped to ARMv2a due to SWP instruction added. And it was released in another Acorn computer: A5000. This was also used in Acorn A4 which was first ARM powered laptop (but term “ARM Powered” was created few years later). I hope that one day I will be able to play with all those old machines…

There was also ARM250 processor with ARMv2a instruction set like in ARM3 but no cache controller. But it is worth mentioning as it can be seen as first SoC due to ARM, MEMC, VIDC, IOC chips integrated in one piece of silicon. This allowed to create budget versions of computers.

ARM Ltd.

In 1990 Acorn, Apple and VLSI co-founded Advanced RISC Machines Ltd. company which took over research and development of ARM processors. Their business model was simple: “we work on cpu cores and other companies pay us license costs to make chips”.

Their first cpu was ARM60 with new instruction set: ARMv3. It had 32bit address space (compared to 26bit in older versions), was endian agnostic (so both big and little endian was possible) and there were other improvements.

Please note lack of ARM4 and ARM5 processors. I heard some rumours about that but will not repeat them here as some of them just do not fit when compared against facts.

ARM610 was powering Apple Newton PDA and first Acorn RiscPC machines where it was replaced by ARM710 (still ARMv3 instruction set but ~30% faster).

First licensees

You can create new processor cores but someone has to buy them and manufacture… In 1992 GEC Plessey and Sharp licensed ARM technology, next year added Cirrus Logic and Texas Instruments, then AKM (Asahi Kasei Microsystems) and Samsung joined in 1994 and then others…

From that list I recognize only Cirrus Logic (used their crazy EP93xx family), TI and Samsung as vendors of processors ;D


One of next cpu cores was ARM7TDMI (Thumb+Debug+Multiplier+ICE) which added new instruction set: Thumb.

The Thumb instructions were not only to improve code density, but also to bring the power of the ARM into cheaper devices which may primarily only have a 16 bit datapath on the circuit board (for 32 bit paths are costlier). When in Thumb mode, the processor executes Thumb instructions. While most of these instructions directly map onto normal ARM instructions, the space saving is by reducing the number of options and possibilities available — for example, conditional execution is lost, only branches can be conditional. Fewer registers can be directly accessed in many instructions, etc. However, given all of this, good Thumb code can perform extremely well in a 16 bit world (as each instruction is a 16 bit entity and can be loaded directly).

ARM7TDMI landed nearly everywhere – MP3 players, cell phones, microwaves and any place where microcontroller could be used. I heard that few years ago half of ARM Ltd. income was from license costs of this cpu core…


But ARM7 did not ended at ARM7TDMI… There was ARM7EJ-S core which used ARMv5TE instruction set and also ARM720T and ARM740T with ARMv4T. You can run Linux on Cirrus Logic CLPS711x/EP721x/EP731x ones ;)

According to ARM Ltd. page about ARM7 the ARM7 family is the world’s most widely used 32-bit embedded processor family, with more than 170 silicon licensees and over 10 Billion units shipped since its introduction in 1994.


I heard that ARM8 is one of those things you should not ask ARM Ltd. people about. Nothing strange when you look at history…

ARM810 processor made use of ARMv4 instruction set and had 72MHz clock. At same time DEC released StrongARM with 200MHz clock… 1996 was definitively year of StrongARM.

In 2004 I bought my first Linux/ARM powered device: Sharp Zaurus SL-5500.


Ah ARM9… this was huge family of processor cores…

ARM moved from a von Neumann architecture (Princeton architecture) to a Harvard architecture with separate instruction and data buses (and caches), significantly increasing its potential speed.

There were two different instruction sets used in this family: ARMv4T and ARMv5TE. Also some kind of Java support was added in the latter one but who knows how to use it — ARM keeps details of Jazelle behind doors which can be open only with huge amount of money.


Here we have ARM9TDMI, ARM920T, ARM922T, ARM925T and ARM940T cores. I mostly saw 920T one in far too many chips.

My collection includes:

  • ep93xx from Cirrus Logic (with their sick VFP unit)
  • omap1510 from Texas Instruments
  • s3c2410 from Samsung (note that some s3c2xxx processors are ARMv5T)


Note: by ARMv5T I mean every cpu never mind which extensions it has built-in (Enhanced DSP, Jazelle etc).

I consider this one to be most popular one (probably after ARM7TDMI). Countless companies had own processors based on those cores (mostly on ARM926EJ-S one). You can get them even in QFP form so hand soldering is possible. CPU frequency goes over 1GHz with Kirkwood cores from Marvell.

In my collection I have:

  • at91sam9263 from Atmel
  • pxa255 from Intel
  • st88n15 from ST Microelectronics

Had also at91sam9m10, Kirkwood based Sheevaplug and ixp425 based NSLU2 but they found new home.


Another quiet moment in ARM history. ARM1020E, ARM1022E, ARM1026EJ-S cores existed but did not looked popular.

UPDATE: Conexant uses ARM10 core in their next generation DSL CPE systems such as bridge/routers, wireless DSL routers and DSL VoIP IADs.


Released in 2002 as four new cores: ARM1136J, ARM1156T2, ARM1176JZ and ARM11 MPCore. Several improvements over ARM9 family including optional VFP unit. New instruction set: ARMv6 (and ARMv6K extensions). There was also Thumb2 support in arm1156 core (but I do not know did someone made chips with it). arm1176 core got TrustZone support.

I have:

  • omap2430 from Texas Instruments
  • i.mx35 from Freescale

Currently most popular chip with this family is BCM2835 GPU which got arm1136 cpu core on die because there was some space left and none of Cortex-A processor core fit there.


New family of processor cores was announced in 2004 with Cortex-M3 as first cpu. There are three branches:

  • Aplication
  • Realtime
  • Microcontroller

All of them (with exception of Cortex-M0 which is ARMv6) use new instruction sets: ARMv7 and Thumb-2 (some from R/M lines are Thumb-2 only). Several cpu modules were announced (some with newer cores):

  • NEON for SIMD operations
  • VFP3 and VFP4
  • Jazelle RCT (aka ThumbEE).
  • LPAE for more then 4GB ram support (Cortex A7/12/15)
  • virtualization support (A7/12/15)
  • big.LITTLE
  • TrustZone

I will not cover R/M lines as did not played with them.


Announced in 2006 single core ARMv7a processor core. Released in chips by Texas Instruments, Samsung, Allwinner, Apple, Freescale, Rockchip and probably few others.

Has higher clocks than ARM11 cores and achieves roughly twice the instructions executed per clock cycle due to dual-issue superscalar design.

So far collected:

  • am3358 from Texas Instruments
  • i.mx515 from Freescale
  • omap3530 from Texas Instruments


First multiple core design in Cortex family. Allows up to 4 cores in one processor. Announced in 2007. Looks like most of companies which had previous cores licensed also this one but there were also new vendors.

There are also single core Cortex-A9 processors on a market.

I have products based on omap4430 from Texas Instruments and Tegra3 from NVidia.


Announced around the end of 2009 (I remember discussion about something new from ARM with someone at ELC/E). Up to 4 cores, mostly for use in all designs where ARM9 and ARM11 cores were used. In other words new low-end cpu with modern instruction set.


The fastest (so far) core in ARMv7a part of Cortex family. Up to 4 cores. Announced in 2010 and expanded ARM line with several new things:

  • 40-bit LPAE which extends address range to 1TB (but 32-bit per process)
  • VFPv4
  • Hardware virtualization support
  • TrustZone security extensions

I have Chromebook with Exynos5250 cpu and have to admit that it is best device for ARM software development. Fast, portable and hackable.


Announced in 2011. Younger brother of Cortex-A15 design. Slower but eats much less power.


Announced in 2013 as modern replacement for Cortex-A9 designs. Has everything from Cortex-A15/A7 and is ~40% faster than Cortex-A9 at same clock frequency. No chips on a market yet.


That’s interesting part which was announced in 2011. It is not new core but combination of them. Vendor can mix Cortex-A7/12/15 cores to have kind of dual-multicore processor which runs different cores for different needs. For example normal operation on A7 to save energy but go up for A15 when more processing power is needed. And amount of cores in each of them does not even have to match.

It is also possible to make use of all cores all together which may result in 8-core ARM processor scheduling tasks on different cpu cores.

There are few implementations already: ARM TC2 testing platform, HiSilicon K3V3, Samsung Exynos 5 Octa and Renesas Mobile MP6530 were announced. They differ in amount of cores but all (except TC2) use the same amount of A7/A15 cores.


In 2011 ARM announced new 64-bit architecture called AArch64. There will be two cores: Cortex-A53 and Cortex-A57 and big.LITTLE combination will be possible as well.

Lot of things got changed here. VFP and NEON are parts of standard. Lot of work went into making sure that all designs will not be so fragmented like 32-bit architecture is.

I worked on AArch64 bootstrapping in OpenEmbedded build system and did also porting of several applications.

Hope to see hardware in 2014 with possibility to play with it to check how it will play compared to current systems.

Other designs

ARM Ltd. is not the only company which releases new cpu cores. That’s due to fact that there are few types of license you can buy. Most vendors just buy licence for existing core and make use of it in their designs. But some companies (Intel, Marvell, Qualcomm, Microsoft, Apple, Faraday and others) paid for ‘architectural license’ which allows to design own cores.


Probably oldest one was StrongARM made by DEC, later sold to Intel where it was used as a base for XScale family with ARMv5TEJ instruction set. Later IWMMXT got added in PXA27x line.

In 2006 Intel sold whole ARM line to Marvell which released newer processor lines and later moved to own designs.

There were few lines in this family:

  • Application Processors (with the prefix PXA).
  • I/O Processors (with the prefix IOP)
  • Network Processors (with the prefix IXP)
  • Control Plane Processors (with the prefix IXC).
  • Consumer Electronics Processors (with the prefix CE).

One day I will undust my Sharp Zaurus c760 just to check how recent kernels work on PXA255 ;D


Their Feroceon/PJ1/PJ4 cores were independent ARMv5TE implementations. Feroceon was Marvell’s own ARM9 compatible CPU in Kirkwood and others, while PJ1 was based on that and replaced XScale in later PXA chips. PJ4 is the ARMv7 compatible version used in all modern Marvell designs, both the embedded and the PXA side.


Company known mostly from wireless networks (GSM/CDMA/3G) released first ARM based processors in 2007. First ones were based on ARM11 core (ARMv6 instruction set) and in next year also ARMv7a were available. Their high-end designs (Scorpion and Krait) are similar to Cortex family but have different performance. Company also has Cortex-A5 and A7 in low-end products.

Nexus 4 uses Snapdragon S4 Pro and I also have S4 Plus based Snapdragon development board.


Faraday Technology Corporation released own processors which used ARMv4 instruction set (ARMv5TE in newer cores). They were FA510, FA526, FA626 for v4 and FA606TE, FA626TE, FMP626TE and FA726TE for v5te. Note that FMP626TE is dual core!

They also have license for Cortex-A5 and A9 cores.

Project Denver

Quoting Wikipedia article about Project Denver:

Project Denver is an ARM architecture CPU being designed by Nvidia, targeted at personal computers, servers, and supercomputers. The CPU package will include an Nvidia GPU on-chip.

The existence of Project Denver was revealed at the 2011 Consumer Electronics Show. In a March 4, 2011 Q&A article CEO Jen-Hsun Huang revealed that Project Denver is a five year 64-bit ARM architecture CPU development on which hundreds of engineers had already worked for three and half years and which also has 32-bit ARM architecture backward compatibility.

The Project Denver CPU may internally translate the ARM instructions to an internal instruction set, using firmware in the CPU.


AppliedMicro announced that they will release AArch64 processors based on own cores.

Final note

If you spotted any mistakes please write in comments and I will do my best to fix them. If you have something interesting to add also please do a comment.

I used several sources to collect data for this post. Wikipedia articles helped me with details about Acorn products and ARM listings. ARM infocenter provided other information. Dates were taken from Wikipedia or ARM Company Milestones page. Ancient times part based on The ARM Family and The history of the ARM CPU articles. The history of the ARM architecture was interesting and helpful as well.

Please do not copy this article without providing author information. Took me quite long time to finish it.


8 June evening

Thanks to notes from Arnd Bergmann I did some changes:

  • added ARM7, Marvell, Faraday, Project Denver, X-Gene sections
  • fixed Cortex-A5 to be up to 4 cores instead of single.
  • mentioned Conexant in ARM10 section.
  • improved Qualcomm section to mention which cores are original ARM ones, which are modified.

David Alan Gilbert mentioned that ARM1 was not freely available on a market. Added note about it.