1. Kolla ‘stein’ released

    On last Monday we finally released Kolla and Kolla Ansible 8.0.0 ‘stein’. Took us longer than we planned but now it is done and ready for users.

    Changes

    What got changed? Many things — details can be found in Kolla release notes and Kolla Ansible release notes.

    My work in this cycle was more reviews, less code. And lot of planning how to handle Python 3 migration for Debian/Ubuntu based images. At some point we decided that this stuff will be postponed to ‘train’ cycle. You can read more about it my previous post: Moving Kolla images to Python 3.

    Why so late

    Usually we release Kolla ‘two weeks’ after official OpenStack release. This allows us to switch to final release code of other projects, do some testing etc. This time it took us far more time :(

    Due to several issues (some core developers got more occupied with work, distributions changed dependencies in packages) it took us longer than ‘two weeks’ after official OpenStack ‘stein’ release. We added more tests for CI, handled partial Python 3 migration in Ubuntu Cloud Archive and more.Several fixes were made in ‘train’ and backported to ‘stein’.

    Now it is your turn — build, deploy, test, report ;D

    Written by Marcin Juszkiewicz on
  2. Upgraded my desktop a bit

    I have built my current desktop 7.5 years ago. Since then I did not had a need for a big hardware changes.

    Upgrades

    Machine (called ‘puchatek‘ (Winnie the Pooh in Polish)) had several upgrades in meantime:

    • got maxxed at 32GB of memory
    • 60GB Corsair SSD for / was installed 8 years ago
    • 250GB Samsung Evo for /home was added 3 years ago
    • graphic cards were changed from Radeon HD5450 via Radeon R7 240 to Nvidia GTX 1050 Ti

    Issues

    Size of system drive became an issue when I needed to build hundreds of container images. All that Kolla stuff…

    One of solutions was replacing system drive with bigger one. So I tried to use PCI Express to m.2 adapter card and realised that x8 slot stopped working.

    New mainboard

    It was a time to replace motherboard. And it is impossible to find a brand new one with 1155 socket. So I went through used ones and found nice replacement — Asrock Z68 Extreme4.

    What’s nice in it? PLX chip. It is PCI Express switch. So mainboard can have x16/x8+x8 slots, x4 slot, some x1 slots and several onboard components despite of only 24 pcie lanes available (16 from cpu, 8 from chipset).

    This way I can have graphics card working in x16 slot (it goes with x8 anyway) and NVME drive in x4 slot. If I decide to go into SLI (two graphics cards) or 10GbE I have a slot for it.

    PCI tree

    PCI tree looks a bit different now:

    • cpu

      • x16 slot with graphics card
      • x8 slot (empty)
    • chipset

      • x4 slot with nvme
      • x1 link to Marvell SATA controller (disabled in firmware setup)
      • x1 link to Etron USB 3.0 host controller
      • x1 link to Etron USB 3.0 host controller
      • x1 link to PLX switch
    • PLX switch

      • x1 slot with Renesas USB 3.0 host controller
      • x1 link to FireWire controller (disabled in firmware setup)
      • x1 link to Broadcom 1GbE controller
      • x1 slot (empty, covered by graphics card)
      • x1 link to PCIe to pci bridge

    Hacking firmware

    Latest available firmware was from 2012 and lacked any support for NVME boot. Thanks to other hackers it was not an issue. Only had to follow “how to add NVME booting into BIOS” instruction. After flashing modified firmware I could boot directly from NVME drive.

    Final result

    System boots from NVME now. 256GB of fast storage available for / and my container images. Spare PCIe x8 slot for future upgrades.

    Written by Marcin Juszkiewicz on
  3. Not enough bandwidth for new device state

    USB. Protocol which replaced random keyboard connectors, PS/2, ADB, gameport, serial and parallel ports (and many more). Sometimes expanded to “USB Sucks Badly“.

    Five years ago buying USB 3.0 hub was a task as it was not so popular thing. Nowadays I have three SuperSpeed ones. One in LG monitor (webcam, phone, Bluetooth dongle), one on desk (watch charging/adb, Yubikey, card reader, pendrives etc) and that 7 port one from five years ago with random dongles in it.

    Yesterday I replaced Gigabyte P67X-UD3-B3 mainboard with AsRock Z68 Extreme4 one. This gave me extra PCIe x4 slot where I can plug NVME storage. Built system, booted into Fedora and started using.

    At some moment I had to login into one of systems with two factor authorization. I use Yubikey for it. Pressed the button and nothing was outputed…

    Then I realized why previous configuration had that extra USB 3 controller:

    usb 3-2.2: new full-speed USB device number 12 using xhci_hcd
    usb 3-2.2: New USB device found, idVendor=1050, idProduct=0110, bcdDevice= 3.33
    usb 3-2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
    usb 3-2.2: Product: Yubikey NEO OTP
    usb 3-2.2: Manufacturer: Yubico
    usb 3-2.2: Not enough bandwidth for new device state.
    usb 3-2.2: can't set config #1, error -28
    

    Yay, USB!

    Plugged Renesas uPD720201 PCIe USB 3.0 host controller, moved all hubs to it and it works just fine:

    usb 7-3.2: new full-speed USB device number 11 using xhci_hcd
    usb 7-3.2: New USB device found, idVendor=1050, idProduct=0110, bcdDevice= 3.33
    usb 7-3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
    usb 7-3.2: Product: Yubikey NEO OTP
    usb 7-3.2: Manufacturer: Yubico
    input: Yubico Yubikey NEO OTP as /devices/pci0000:00/0000:00:1c.7/0000:05:00.0/0000:06:01.0/0000:07:00.0/usb7/7-3/7-3.2/7-3.2:1.0/0003:1050:0110.0008/input/input30
    hid-generic 0003:1050:0110.0008: input,hidraw6: USB HID v1.10 Keyboard [Yubico Yubikey NEO OTP] on usb-0000:07:00.0-3.2/input0
    

    Then checked which USB 3.0 host controller is on mainboard:

    11:29 (0s) hrw@puchatek:~$ lspci |grep -i usb
    00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
    00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
    03:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
    04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
    07:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
    

    Argh, Etron EJ168 again…

    So if your mainboard has Etron EJ168 then consider adding Renesas card. Works just fine. Nevermind how slow/fast your USB devices are.

    Written by Marcin Juszkiewicz on
  4. Nine years of Linaro

    Nine years ago at 11:00 a group of developers gathered in a small room. I was one of them and did not knew anyone from a group before entering the room.

    The meeting took place in Dolce La Hulpe Hotel and Resort in a village close to Brussels, Belgium. It was on first day of UDS-M.

    This was the first meeting of NewCo developers. The organization now known as Linaro.

    I do not remember who exactly was at that meeting so will not provide a list. We introduced ourselves, got some knowledge of who is who’s boss and what we will do from now. I just told “I’ll do what my boss (pointing to Steve Langasek) orders me to” as I was tired after whole night train trip.

    For me this was the real beginning of Linaro. Not June 2010 when (at Computex) it was announced to the world. Neither 26th April 2010 — a day when I started working for Canonical as NewCo engineer.

    Written by Marcin Juszkiewicz on
  5. Good bye WordPress

    I had some kind of personal website since started using Internet in 1996. First it was set of hand edited Lynx bookmarks, then were experiments with wikis. Finally in 2005 I started using WordPress. And it was in use for those 14 years. Until now…

    WordPress is nice platform but I got tired of it. More and more plugins and themes became demo versions of commercial products. Also amount of JavaScript and CSS added to website made it harder and harder to maintain. At some point I said myself that it is enough. And started looking for alternatives.

    Pelican

    Here came Pelican — static site generator written in Python. I had few attempts to switch to it and finally managed to find some time and sorted out all issues.

    Someone my ask why Pelican? Why not Jekkyl, Gatsby, Hexo, Wintersmith or other. For me reason is simple — it is Python. Language which I already know. So in case of need I can read source code and know how to change it (already sent one change and it got merged).

    Conversion

    The good side is import from WordPress went nice. As I used Markdown most of posts required changes. Implementation in Pelican differs from old Markdown Extra + SmartyPants I had in my blog.

    Images

    Then came images. Copied whatever I had on previous website and removed all thumbnails. Then decided to go with 700px wide ones and to not link to original photos. Boring work as almost every image in every post needed change. Some entries got pictures removed (most of time due to their low resolution).

    This also shown how my blog was changing through all those years. Over 10 years ago adding 300x300px picture into blog post was normal thing. Now such graphics got either removed or replaced with 700px wide one.

    Some posts had galleries inserted instead of pictures. This took a bit more time as I was grabbing filenames from database to replace gallery with set of photos. And removed some of them during.

    Look and feel

    When I was collecting ideas for a new platform I had few ideas:

    • static generator
    • no JavaScript
    • minimal CSS
    • similar look to WordPress version

    Pelican solved first point. Handling rest was harder.

    I took a look at existing Pelican themes and tried several ones. Finally decided to make own one — like WordPress “Spacious” one.

    As a base I used “Simple” theme. Typical template with header, content, sidebar and footer. Elements put in CSS grid for most of screens and once screen width goes under 70em layout switches to “flex”. This allowed for simple responsive web design. All in ~2.5KB of CSS (plus some code for webfonts).

    Archives

    One of big changes (compared to WordPress) is a way of presenting archive posts. You can go into archives to see the list of all my blog posts like it was before. But if you go for a list of posts in a tag (like AArch64, Red Hat, Zaurus) then instead of posts with pagination you get list of posts presented in archive form.

    This should make old content easier to find.

    Comments

    As you may notice there is no way to comment posts anymore. Amount of comments was lower every year so I decided to not bother with them in new website. I could add Disqus for example but is it worth for just a few entries per year?

    To Do

    There are some things I need to take care of still. Page about my fridge magnets collection is missing, some entries may get some formatting changes or small contents edit. No big edits of old posts as they show how awful my English was in past (not that it got any better).

    Written by Marcin Juszkiewicz on
  6. The end of “Mali question”?

    For several Linaro Connect events we had sessions about state of graphics drivers on ARM platforms. I attended most of them and got a reputation of person asking problematic questions.

    But situation has changed. With Panfrost project happening. It is a Foss driver for Arm Mali Midgard graphics chipset (Bifrost support on a way). It went from “wow, a triangle” to “we can play some games or run a desktop” in quite short time.

    At BKK19 Linaro Connect we had “State of opensource drivers for mobile GPU chips” session. Freedreno, Etnaviv, vc4, v3d, Panfrost, Lima etc. What they target, what was already achieved, what are plans. Great progress across whole ARM world. And several questions from the audience. And interesting answers as well.

    thumbnail for VTgDP3yNXI0 video

    Mali then. Grant Likely from Arm told that they are looking how Panfrost is going. From company perspective both Midgard and Bifrost chips are “done, in a field” product which will not get changes. Still — engineering support goes entirely into their binary drivers as this is what their customers are using. Situation may change if those customers start asking for open drivers.

    I do not use any Arm hardware with Mali GPU anymore. But hope that at next Linaro Connect instead of asking famous “Mali question” we will rather discuss how it runs on our devices.

    Written by Marcin Juszkiewicz on
  7. How did I hacked Linaro Connect BKK19 puzzle

    One of Linaro Connect traditions is a puzzle to solve. Created by Dave Pigott. And recent BKK19 event was not any different. There was puzzle announcement on the first day — right before first keynote. But no one could be first to answer at that time…

    Tuesday

    As usual before Connect I looked at a map and marked several locations in Bangkok as places to visit. Then took a look at the official BKK19 application. Installed it on my phone and started.

    First screen had few paragraphs of text. Some information about the event and schedule. But there was also paragraph with information about the puzzle. WITH the link to it!

    I clicked to get some redirection to Google Forms website. With information that this form is not available for users outside of organization. As I do not have work accounts on my phone I checked for redirection link and loaded it on my desktop. And landed into the puzzle.

    Puzzle form

    It was a bit different than version provided during Linaro Connect. There was a graphics with seven (official one had eight ones) columns of text. Under it was graphics with chess figures:

    H1 G3 F1 H2 G4 E3 D1
    A4 C3 B1 A3 C4 B6 A8
    B5 A7 C8 D6 E8 F6 D5
    G8 H6 F5 G7 H5 F4 E2
    C6 A5 B7 C5 A6 B8 D7
    F3 G1 H3 G5 H7 F8 E6
    F7 H8 G6 H4 G2 E1 D3
    A2 B4 C2 A1 B3 D2 E4
    

    Graphical hint

    Dave later said that knight was not present in it but I did not noticed that. There was a plan to add that graphics into official puzzle if no one provides proper answer until Wednesday.

    From text I noticed that it is chess related. Took a sheet of paper, draw 8x8 grid on it and started following each row on it with different markings each time. Hm… nothing came to my mind. Noted missing entries.

    Help me Google, you are my only hope

    Then started googling “knight chess puzzle” and got “knight’s tour” links on first page. Started reading what it is about. Then restarted tracking knight’s moves from the puzzle with adding missing ones. Turned out that this was it.

    Let me mail Dave

    I submitted “knight’s tour problem” as the answer and wrote to Dave:

    I see that they are online already.

    Are the ones there official ones or testing one?

    Turned out that I found testing version. Which did not even collected emails when someone provided an answer. But there was one such sent so Dave marked it as my submission. And the link got removed from BKK19 application.

    At pool bar

    I arrived in Bangkok on Saturday. Met Dave at pool bar and we had a chat about the puzzle. Was fun to see how surprised people around were that first answer was already provided. I asked Dave to not give me the proper answer nor info was my answer good so I would not spoil other people.

    During Connect few attendees asked me about the puzzle, how it went. Kept away from spoiling them.

    And the winner is…

    Then Friday happened with closing remarks session. Only a few people provided two words answer (“knight’s tour”) and few three word one (“closed knight’s tour”). My name was in “special mention” section.

    Puzzle winners slide

    Turned out that there is separate award for hacking a puzzle. It was 3rd time when it happened. I got a waterproof action camera (EZVIV S1C model) — will find some use for it sooner or later ;D

    Whole puzzle was fun. Thanks go to Dave for creating it and providing me with a copy of both results slide and graphical hint so I could use it in blog post.

    Written by Marcin Juszkiewicz on
  8. Moving Kolla images to Python 3

    Python… To use 2.7 or to go for 3.x? To “be compatible” or to “use fancy new features”. Next year Python 2 gets finally unsupported upstream.

    Kolla state

    Kolla can run under Python 2.7 or 3.x and results will be the same. All those container images containing OpenStack components and whatever needed to get them running. But so far all of those images will use Python 2.7 inside…

    I proposed to take care of it during Train cycle (we are at Stein now). Then we had a meeting and decided — let us do that in Stein! If I only knew how many issues it will bring…

    Cleaning/refactoring

    First some cleaning and refactoring:

    Distribution handling

    Then created new variable to keep which exactly version of Python 3: distro_python_version. And then used wherever needed.

    In meantime we got support for Red Hat Enterprise Linux 8 distribution. Where Python 3 is the only version available.

    Finally we moved Ubuntu to use Stein UCA which also switched us to Python 3 in several images.

    CI failures

    And then hell gates opened…

    We moved our CI jobs from Ubuntu 16.04 ‘xenial’ to Ubuntu 18.04 ‘bionic’. OMG. So many failures. When Mark Goddard found out why (too old ‘setuptools’) we had to pile up fixes to get CI back into working state.

    Nova images were failing. Turned out that Debian/Ubuntu ‘qemu’ package is no longer metapackage but useless dummy. And then it brought ‘armv6l’ architecture to Nova so it started failing. There were three different patches to handle it and problem got solved.

    OracleLinux image got repository information files renamed. And as we edit them we had to adapt.

    Python 3 got enabled in another set of images.

    Changing other projects

    Kolla uses other projects, right? Are they Python 3 ready?

    Had to patch several ones:

    Some of those changes were requirements updates to mark “Python 2 only” components (enum34, functools32, UcsSdk) or getting rid of Unicode characters from files which should be be US-ASCII.

    Bifrost one cleaned situation as we are using it within virtualenv so Python packages were going crazy. Now they handle it ;D

    Current status

    The most important patch switching all image to Python 3 for Debian/Ubuntu is still in review. Waiting for karbor fix getting merged and CentOS 7/OracleLinux 7 builds getting working again.

    We also moved to Debian ‘buster’ release. It is now in ‘freeze’ state so no big changes allowed and provides us with many updates making Debian/Ubuntu blocks easy.

    Python 2 stays :(

    There are few images where we still have Python 2. Anything related to Ceph has it because Debian/Ubuntu ‘ceph-common’ package depends on Py2 packages. We install Py3 ones there.

    Written by Marcin Juszkiewicz on
  9. We need Arm64 systems for developers. Again.

    There are several posts and videos around recent discussion in ARM announces Ares” thread on Real World Technologies website. People quote Linus Torvalds like crazy…

    SBBR

    Yes, we need AArch64 (real Arm64 architecture name) systems for developers. SBBR compliant ones so their users will not have to deal with ‘how the hell should I boot it’. Instead get Linux distribution and install it like on current developer boxes (PXE, USB, CD etc).

    SBBR: System Boot Base Requirements — specification which defines firmware of AArch64 server system. Which mean UEFI, ACPI on hardware compliant with SBSA (Server Base System Architecture) specification. In short: boring box which just works with any serious Linux/BSD distribution and for any random user looks and works like x86-64 box when it comes to booting.

    Hardware formats

    Some people would like small format — like Intel NUC, Apple Mini. Just open a box, put m.2 storage, few sticks of memory and go.

    Other ones would like (micro)ATX format. You put it into standard case, add m.2 and sata storage, few sticks of memory, PCI Express graphics card and use.

    So far there are no such systems in affordable prices. Or you have to deal with some issues.

    Software issues

    Anyway, you got hardware. Or access to. Or even bloody VM somewhere. So you start development. And often hit a wall…

    Nowadays development culture ‘loves’ to use random binaries from the Internet. You do not wait until someone package software you would like to use, you grab compiled binaries from vendor/authors. Their website or github/gitlab release page.

    And nope. x86-64 binaries only. So you dig for some aarch64 builds or look how to build it on your own. And you do not work on your project but on someone’s else one. Or you change your one to not use that external component.

    Language extensions

    Or your project is in Python/Ruby or other language with extensions compiled to native code. And simple “pip install scipy” which took 4s on your x86-64 machine now just fails…

    So you install compilers, Python headers, check which libraries you should have installed, which versions of them, do your distro have them etc. Finally “pip install scipy” works. After several hours from your development time.

    Conclusion(?)

    Getting AArch64 hardware for developers is important. When it happen? One day. Maybe even before people forget that such architecture existed.

    We talk about it during each Linaro Connect. So far nothing serious came from it. We had some failed attempts like Cello or Husky. There is Synquacer with own set of issues. Some people use MACCHIATObin. Some still use Applied Micro Mustangs which should get a place in computer museums.

    It is chicken and egg issue. No one makes affordable AArch64 systems because no one buys them. Because no one makes them. Hardware vendors concentrate on server market — no chips to choose for developer systems.

    Written by Marcin Juszkiewicz on
  10. Three years of system calls table

    Porting software often involves system calls. Usually their numbers differ between architectures. Some calls are missing, some are specific to platform. Normal stuff.

    I knew that, you knew that, someone other knew. Or not. But looking into kernel/libc headers each time was boring so I created syscalls table for it. It was small project for personal use.

    One day Arnd Bergmann sent me set of patches which rewrote table generation. From few architectures to which I had access (so could run binary) to every arch supported in Linux kernel. Then some architectures got dropped from kernel. But I kept their data in case someone needs (just moved it to the far right side).

    Webpage look changed during those years. From ugly HTML table to table using DataTables framework. With plugins adding rows colouring, search option and few other tricks.

    And several funny moments happened related to this table. At FOSDEM 2018 I visited Valgrind devroom and was greeted with “Ah, so you are that syscalls guy!” as it turned out that page was a great help. One of my friends was porting his lowlevel software from x86-64 to aarch64 one day and asked me “man, why there is no open() syscall on aarch64?”. Etc. etc.

    I do not remember when last time used it for something. Keeping it updated every rc1 kernel release so anyone can see actual state. I know that people use it cause from time to time someone mentions it or gets directed to it.

    Written by Marcin Juszkiewicz on
Page 1 / 76
Older posts »