Latest Posts

Is the warrant canary still warranted?

Somebody is actually watching Raptor's warrant canary, and mentioned it hasn't been updated in 6 months (as of this writing the last date is March 3, 2019). Although my usual tendency is to glance at it before installing a firmware update, 1.06 is over a year old, so I hadn't noticed myself.

Conceptually, the warrant canary helps to protect purchasers by acting as a negative indicator if they are under a gag order regarding a subpoena or other state actor legal action: if the canary disappears or isn't reupped, then caveat emptor. Raptor's response in the Twitter thread suggests that the failure to update was inadvertent and my gut impression is this is probably true, but the real question is how likely Talos or Blackbird owners are to be targets for state-level threats. We're using niche machines here but the OpenPOWER workstation userbase tends to be more cognizant of how it can be monitored, and if we weren't on watchlists before OpenPOWER started getting more popular, especially in certain countries the number of workstations may now be at a level where such concerns are no longer preposterous.

Raptor, in the same thread, is asking users to speak up about whether the warrant canary is still useful. (They mention a cost/benefit ratio; I'm interested to hear what the cost is. Is it time, money, both?) Lest one think a smaller company could be pushed around more easily, I don't think size is really a factor here; in fact, I'd argue that a bigger company is even less likely to care about such things because of increased bureaucracy and potentially competing internal priorities over government contracts. I agree their point we really should get comfortable with rolling our own firmware is very well taken, but by the same token it's not necessarily a small task for an individual to audit Raptor's tree either. Particularly for critical or time-sensitive updates we will still have some level of vendor dependency and it would be nice to have the canary in those circumstances when using a pre-built firmware package becomes necessary, so put my vote down as "please keep it." We're using these machines for a reason, and the more failsafes there are, the more we're better protected from Mayhem — like meow.

(*not sponsored or endorsed by Allstate)

Ubuntu 19.10 available

Ubuntu 19.10 is now available with the vaguely unwieldy name "EoanErmine" based on kernel 5.3 and GNOME 3.34. An interesting improvement in this release is their expanded cross-compilation toolchain allowing building for s390x, ppc64le, riscv and ARM targets, which hopefully will expand the number of ports and pre-compiled packages on this platform; another interesting one is experimental ZFS on root support. Although an official desktop release of Ubuntu for ppc64le still doesn't exist, the release notes do say that "[t]he ppc64el [sic] ... live-server ISO images are now considered production ready and are the preferred media to install Ubuntu Server on bare metal" (excellent!), so download the server ISO, and then for your workstation you can convert it to desktop Ubuntu.

librsvg2 issue on ppc64le

If you are using Fedora, keep an eye on bug 1756838 where an LLVM 8 codegen issue is suspected with ppc64le causing an apparently faulty assertion in librsvg2. Unfortunately, this library is heavily used by (at least) GNOME and Xfce, meaning the issue may well make your desktop environment unusable -- for example, my Blackbird with the faulty library couldn't open the Applications drawer without crashing gnome-shell. Unfortunately, reducing the codegen issue has not been trivial.

The faulty build is librsvg2-2.46.0-2. If you keep, or downgrade to, librsvg2-2.45.90-1, this version is unaffected because it was built with an earlier toolchain. At least for Fedora, there appear to be no ABI changes between 2.45.90 and 2.46.0 (thanks to Dan Horák for confirming this) and there are no known or at least visible security issues in the earlier version, so it is currently safe to stay there.

On Fedora, if you are on F30 and have not yet been affected, you may wish to consider putting exclude=librsvg2 into /etc/dnf/dnf.conf to inhibit updates to it until further notice. If you have been affected, you can attempt to downgrade to 2.45.90, though interestingly on my (unaffected) Talos II that has not updated to the bad version,

% strings /usr/lib64/ | fgrep 2.4 | grep fc

It is possible F31 may smooth this over with LLVM 9, which should arrive later this month, and doesn't appear to suffer from this problem.

This may not affect other distributions with older toolchains. If your distribution is also affected, please post in the comments.

CentOS 8 and CentOS Stream (The Freshmaker)

Yeah, okay, we've had a lot to say about Red Hat derivatives lately. On the heels of CentOS 7's latest service release now comes CentOS 8 in a new minty flavour CentOS Stream, "a midstream distribution that provides a cleared-path for participation in creating the next version of RHEL," rebranding the "classic" CentOS build from RHEL as CentOS Linux. Mentally translating, the intention appears to be as a staging area for updates from Fedora mainline to trickle into minor releases of RHEL (and thence to mainline CentOS), using CentOS Stream to more gradually introduce updates and incorporate user feedback in a rolling release fashion rather than the typical all-at-once version churn that previously resulted. You know, like chewy candy mints that make things fresher the moment you pop one in your mouth.

That said, mainline Fedora is plenty stable for (my) daily use on POWER9 and elsewhere (we're not talking bleeding-edge saddle sore Rawhide, kids), and Fedora will still be the ultimate upstream, so while I think this will help CentOS developers dogfood changes more gradually I'm having difficulty envisioning the small slice of conservative-but-not-that-conservative users this will appeal to as a daily driver. More likely people will simply regard it as the "public beta" channel for CentOS and RHEL, and I think that will be the actual role it serves regardless of the frilly language.

The CentOS Download site is not currently showing Power ISA (or other AltArch) builds for either CentOS 8 or CentOS Stream yet, but I expect these to emerge soon. It will be interesting to see if big-endian ppc64 is still supported when they do, but there should be POWER9 and "generic" little-endian builds at minimum.

Low-level change to Firefox 70 and ESR coming

If you are using Firefox on 64-bit Power, you'll want to know about bug 1576303 which will be landing soon on the beta and ESR68 trees to be incorporated into 70 and the next ESR respectively. This fixes a long-standing issue with intermittent and difficult to trace crashes (thanks to Ted Campbell at Mozilla for figuring out the root cause and Dan Horák for providing the hardware access) due to what in retrospect was a blatant violation of the ELF ABI in xpconnect, which glues JavaScript to native XPCOM. This needed several dodgy workarounds until we found the actual culprit.

The patch is well tested on multiple little-endian systems including this Talos II, but because it's an issue with register allocation in function calls the issue also theoretically affects big-endian Power even though we haven't seen any reports. I'm pretty sure the code I wrote will work for big-endian but none of my big-endian Power systems run mainline Firefox (and TenFourFox even on the G5 is 32-bit, where the problem isn't present). If you're using a big-endian system, you may want to pull a current release and make sure there is no regression in the browser with the changes; if there is and you can bisect to it, post in the bug so we can do a follow-up fix. On the other hand, if you're building from an old ESR such as 52 (the last non-Rust-required one), you may want to backport this fix because the problem has been there pretty much since it was first written.

Stuff like this actually proves Linus Torvalds' point that "as long as everybody does cross-development, the platform won't be all that stable." Linus was talking about ARM-based servers being undercut by a dearth of ARM-based PCs, but the point is also true here: 64-bit Power may do well in the data center but it was rarely used for workstations other than the Power Mac G5 and the small number of non-Apple PowerPC 970 towers, meaning this bug went undiscovered until people like us finally started dogfooding Power-based desktops again. (For that matter, the official PowerPC Mac OS X builds of Firefox were also always 32-bit, even on the G5, so no one would have noticed it there.) There's just no substitute for improving the quality and quantity of software for Power ISA like having one under your desk, and as the number of machines increases I expect we'll get more of these ugly corner bugs ironed out in other packages too.

CentOS 7-1908 available

CentOS 7-1908 is now available; this is a maintenance release with multiple updated components derived from Red Hat Enterprise Linux 7.7. Particularly interesting is that there are no less than three Power ISA downloads available, one for big-endian ppc64 (though POWER7 and up only: sorry G5 owners), one for ppc64le and a special build for POWER9 (which appears to also be little-endian), each with its own Everything, NetInstall and Minimal flavours.

Linux 5.3 for POWER, and ppc64le gets a Fedora Desktop

Linus always says that no Linux release is a feature release and numbers are purely bookkeeping instead of goalposts, but Linux 5.3 has landed. There are many changes for the x86 side of the fence that I won't mention here, but in platform-agnostic changes, 5.3 adds support for the AMD Navi GPU in amdgpu, allows loading of xz-compressed firmware files, further improves the situation with process ID reuse with additional expansions to pidfd (including polling support), refinements to the scheduler by supporting clamped processor clock ranges, and support for as a valid IPv4 range, allowing another 16 million IPv4 addresses while IPv6 continues to not set the world on fire.

Power ISA-specific changes in this release are relatively few but still noteworthy. Besides support for LZMA and LZO-compressed uImages, there is now Power ISA support for HAVE_ARCH_HUGE_VMAP, which enables (as the name would suggest) huge virtual memory mappings. With additional code in a future kernel, this should facilitate upcoming performance improvements. There is also additional /proc support for getting statistics on how virtual CPUs are dispatched to physical cores by systems using the Power hypervisor.

Meanwhile, this won't make much difference to people like me who have been using Fedora for awhile, but if you want to experiment with other distros on your POWER9 system Fedora is working on Live and Workstation ISOs for ppc64le. Currently this is Rawhide only (which is what will become F32) and you can of course already install from Server and switch to the Workstation flavour, or install over the network. However, it's just another positive indicator that IBM's purchase of Red Hat will continue facilitating improvements in Linux in general and Fedora/RHEL support for OpenPOWER in particular, especially as the installed base of POWER9 workstations like our T2s and Blackbirds continues to grow in numbers. In fact, although we don't have statistics, it's still quite possible (counting box for box) that there are now more discrete POWER9 workstations in operation out there than there are servers.

A beginner's guide to hacking Microwatt

Many improvements have occurred in Microwatt, the little VHDL Power ISA softcore, so far the easiest way — particularly for us hobbyists — of getting an OpenPOWER core in hardware you can play with. (The logo is not an official logo for Microwatt, but I figured it would be fun to try my hand at one in Krita.) Even though it still has many known and acknowledged deficiencies it's actually pretty easy to get it up and running in simulation, and easier still on POWER9 hardware where the toolchain is already ready to go.

I'm no VHDL genius personally, but this seemed like as good a time as any to learn. ghdl is available for most distros, though Fedora 30 and earlier curiously lack it for ppc64le; fortunately, Dan Horák's builds work fine. So let's get the basics up. If you're on F30 as I am, install ghdl from his repo first. These URLs may vary; they are what was current at the time of this article.

% sudo dnf install
% sudo dnf install

For Debian and other distros, install from your package manager as appropriate.

Next, let's install Microwatt and MicroPython and make sure all that works. This is essentially the same demo Anton showed at the OpenPOWER summit. If you are doing this on an inferior x86_64 system (or at least something that isn't POWER8 or POWER9), you will need to have a Power ISA C cross-compilation toolchain installed to properly build MicroPython. Adjust the make -jXX to your number of threads. This sequence of commands will end up with microwatt/ and micropython/ installed in separate directories at the same filesystem depth (in my case, ~/src). Keep it this way because we will be adding one more project at the end.

% git clone git://
Cloning into 'microwatt'...
Resolving deltas: 100% (818/818), done.
% cd microwatt
% make -j24
ghdl -a --std=08 decode_types.vhdl
% cd ..
% git clone git://
Cloning into 'micropython'...
Resolving deltas: 100% (52248/52248), done.
% cd micropython
% git checkout powerpc
Already on 'powerpc'
Your branch is up to date with 'origin/powerpc'.
% cd ports/powerpc
% make -j24
mkdir -p build/genhdr
MISC freezing bytecode
CC build/_frozen_mpy.c
LINK build/firmware.elf
% cd ../../../microwatt
% ln -s ../micropython/ports/powerpc/build/firmware.bin simple_ram_behavioural.bin
% ./core_tb > /dev/null
MicroPython v1.11-320-g7747411e9 on 2019-09-14; bare-metal with POWERPC
Type "help()" for more information.
>>> 1+2

The simulation is rather slow, made worse by all the copious debugging output (which here is sent to the bitbucket), but it does work as advertised. To make core_tb stop, you will probably need to kill it from another terminal session, depending on your shell. (I had to.)

Let's now turn to adding instructions to Microwatt. Since having to manually kill the simulation is annoying — it would be nice if the simulation could gracefully halt under program control — we'll implement a wait instruction as an educational example. This instruction is new in ISA 3.0B; the ISA book explains its operation as that it "causes instruction fetching and execution to be suspended. Instruction fetching and execution are resumed when the events specified by the WC field [the wait condition, its sole constant parameter] occur." Strictly speaking probably the stop instruction would have the most authentic semantics — "The thread is placed into power-saving mode and execution is stopped." — but for obvious reasons this is a privileged instruction because this would completely halt that hardware thread until a system reset or other system-level event. Also, it doesn't take any parameters, so it's not as nice an illustration.

wait's sole supported WC field code is 0b00; this causes the instruction to "[r]esume instruction fetching and execution when an exception, an event-based branch exception, or a platform notify occurs." In practical circumstances, if you execute wait from a userspace program, these events happen all the time and the instruction seems like a no-op.

% more test.c
#include <stdio.h>

int main(int argc, char** argv) {
  __asm__("wait 0\n");
  fprintf(stderr, "ok\n");
  return 0;
% gcc -o test test.c
% ./test

However, on a little core doing nothing else, it well might be a terminal instruction sequence, so since we can run it from userspace anyway let's go ahead and implement a hal-fassed version of it which will cause the simulation to conclude gracefully. This is the diff that does so, applied against ab34c483. Let's analyze it piece by piece.

First, let us note that there's already code in Microwatt for an ungraceful exit, such as when you execute an undefined instruction; this terminates with an error. We could simply use that, but I'd prefer to do something cleaner, so we'll define a new signal for halting.

Next, we will define the opcode format in the instruction decoder. Conveniently, the instructions td and tdi (trap doubleword and trap doubleword immediate, respectively) have a similar encoding where their common constant argument — the "TO" or trap operation bits — occupies the same bit field. (Note that td et al. allow five bits here but wait only takes the two least significant bits with the other three reserved. We will handwave this away since they are invariably encoded as zero.) To get these bits decoded for us, we specify that the first constant argument is encoded as TOO. You can see other encodings for registers and immediates in the surrounding templates.

Next, we tell Microwatt how to identify the opcode. The bit fields for the opcode pieces are simply cribbed from the ISA book.

Next, we add the actual symbols for the instruction and the operation, thus linking them up with the decoder.

Then, we write the operation's logic. For illustrative purposes, since only 0b00 is allowed and other bit combinations are reserved, we will have the simulation assert and ungracefully terminate on other values using the existing code. Otherwise, we set the halted signal.

Finally, we write the code to actually gracefully halt when the halted signal appears, using the built-in VHDL test bench function stop() (coincidentally named, as it happens).

With this patch applied, rebuild Microwatt with a make. To test it, we'll need something that actually executes this instruction, so let's make a simple "hello world" type example using pieces from MicroPython and Microwatt's own built-in "hello world." A small assembly language stub (in both of these examples, head.S) acts as a trampoline into whatever our main() is, detecting if we are running it within QEMU or from the VHDL test bench. However, we won't have a libc and we'll need routines to actually send and receive data with the "serial console" presented by the core. We also need a couple hints for the linker to make a binary we can actually run in the simulator.

I've compiled all of these pieces into a Github project "Microhello," which you can use as a scaffold for your own programs to run on the core. I've tried to make it a little more modularized than the Microwatt "Hello World" example as well. Clone it at the same depth as microwatt/ and micropython/, then do make runrun to replace the symbolic link to the MicroPython binary with Microhello:

% git clone git://
% cd microhello
% make runrun
cc -I. -g -Wall -std=c99 -msoft-float -mno-string -mno-multiple -mno-vsx -mno-altivec -mlittle-endian -fno-stack-protector -mstrict-align -ffreestanding -Os -fdata-sections -ffunction-sections -c -o build/main.o main.c
cc -I. -g -Wall -std=c99 -msoft-float -mno-string -mno-multiple -mno-vsx -mno-altivec -mlittle-endian -fno-stack-protector -mstrict-align -ffreestanding -Os -fdata-sections -ffunction-sections -c -o build/uart_core.o uart_core.c
cc -I. -g -Wall -std=c99 -msoft-float -mno-string -mno-multiple -mno-vsx -mno-altivec -mlittle-endian -fno-stack-protector -mstrict-align -ffreestanding -Os -fdata-sections -ffunction-sections -c -o build/string.o string.c
cc head.S -c -o build/head.o
ld -N -T -o build/firmware.elf build/main.o build/uart_core.o build/string.o build/head.o
size build/firmware.elf

text   data    bss    dec    hex filename
6508      0     24   6532   1984 build/firmware.elf
objcopy -O binary build/firmware.elf build/firmware.bin
( cd ../microwatt && rm -f simple_ram_behavioural.bin )
/usr/bin/make run
make[1]: Entering directory '/home/censored/src/microhello'
( cd ../microwatt && \
ln -s ../microhello/build/firmware.bin simple_ram_behavioural.bin && \
./core_tb > /dev/null )
PowerPC to the People

We neatly came to a halt. Yay!

The serial console library is in uart_core.c and a basic implementation of puts() (and strlen()) is in string.c. The main() is very simple. Minus the comments, here is main.c in its entirety:

#include "uart_core.h"
#include "string.h"

int main(int argc, char** argv) {

  puts("PowerPC to the People");
  __asm__("wait 0\n");
  return 0;

The trampoline uses the start of execution to determine what mode to initialize the serial console in, passing that to main() in r3, which in the Power ABI is the first argument to the function (argc). We then puts() the string and execute a wait 0 to terminate. Easy.

To prove the argument is being evaluated, change the instruction to wait 3 and re-run with make runrun. Notice how it terminates:

PowerPC to the People
make[1]: *** [Makefile:16: run] Error 1

If you run ./core_tb (in the microwatt/) directory without sending the output to /dev/null, you will see the message from our implementation in the log with the invalid wait condition.

Lastly, if you remove the wait instruction entirely and re-run with make runrun, then the test bench will loop forever echoing our string repeatedly, bouncing in and out of our code on the trampoline, until you kill it.

Microwatt is fun, simple, easy to experiment with and a great way to better understand what Power ISA does under the hood. While its performance is no barnburner, as a pedagogical aid it's a great little proof of concept, and it can certainly be the basis for something bigger. In a future article we'll actually synthesize this core and do a little more with it in actual hardware.

Firefox 69 on POWER

A brief note to say so far no major issues with Firefox 69 on Power ISA and this post is being made from it on my T2. (We're still dealing with bug 1576303 for Firefox 70, however.) As with Fx68, the working build configurations for ppc64le are unchanged from Fx67.

The VMX eagle is landing in Firefox 70 (plus: which core should open the door?)

Hugo Landau has an interesting take on the new open-OpenPOWER world. He points out, correctly, that Power ISA is a big win for open architectures because it has maturity in both the embedded and server spaces, but he'd like to see an actual production core opened as well (Microwatt is a lovely MVP and a great proof of concept but it is clearly for experimentation, not for production).

His suggestion is a softcore version of the PPC 405. PowerPC 4xx is a very common embedded CPU family indeed (the POWER8 OCC even has one inside of it), and in the days IBM was even willing to make it available to academia and researchers. He also suggests open-sourcing Mambo, IBM's currently proprietary simulator.

Open-sourcing Mambo is especially appealing to me trying to do simulation work of my own and not being able to do it on a POWER9! (It claims there is a POWER9 version for Debian, but the install directions and download area strictly show x86_64.) I also think there would be little non-IBM IP to stand in the way of doing so. On the other hand, although opening up the 405 would be admirable, I'm not sure how much it would accomplish in practice: it's 32-bit, not 64-bit; it's strictly big-endian (I like big-endian personally and three of the five systems on this KVM are big-endian, but we all know where the market's going and OpenPOWER in particular emphasizes LE); and it lacks VMX, a/k/a AltiVec. That brings us to Firefox.

In the TenFourFox world myself and several contributors did a fair bit of work on AltiVec acceleration to beef up performance on G4 and G5 systems. (Editorial note: I only use the term AltiVec for Apple systems and chips made by Motorola/Freescale, since Apple used both Motorola/Freescale and IBM parts, and Motorola/Freescale (now NXP) owned the trademark. IBM never owned nor licensed this trademark and always called it VMX, so in OpenPOWER, it's VMX. For that matter neither did P.A. Semi, so the PA6T has VMX too. Even with the G5, although its vector unit was popularly called AltiVec, IBM never officially referred to it by that name.) There are also many opportunities for VMX acceleration in mainline Firefox; depending on your compiler settings, these might get silently enabled already (such as qcms). libpng even has support for VSX. However, many in-tree components either never had the build-system glue written to turn on VMX support (libjpeg, libpng) or they're based on custom SIMD code Mozilla wrote that has no Power ISA equivalent.

For Firefox 70, build system support for VMX, VSX and VSX-3 compiler flags plus runtime detection is now available, written by yours truly, along with the first of the TenFourFox patches I updated and upstreamed to mainline Firefox (this one for fast scanning of text fragments for wide characters). I'm also hoping the libjpeg VMX enablement patch lands in time for merge with several more VMX patches to come. My work on the Firefox Power JIT is somewhat slowed by my continuing responsibilities to TenFourFox and compiler issues such as bug 1576303, which is why I wanted to get a couple quick wins with VMX stuff I already had on the shelf.

Allow me to close the loop on our core digression, though. In bug 817058 I'm asked a question by one of the Mozilla devs: can they just assume every Power chip someone is running Firefox on would support VMX? The answer, even for 64-bit Power, is no, because of poor choices like the AmigaOne X5000 running the QorIQ P5020 which has no SIMD. However, Rust supports compiling for SIMD and Power is a supported architecture, which means Rust supports VMX too, and Mozilla would be foolish not to take advantage of that. Assuming SIMD features are "just present" will become increasingly common and that means that continuing to run parts that don't have VMX (let alone VSX) will become an even bigger losing game on the desktop than it already was. Rather than the 405 I'd personally like to see something like the G5 itself be made openly available: it's POWER4, so it's 64-bit and largely upwardly compatible but wouldn't be a commercially competitive product at the high tiers IBM cares about, it has a VMX unit but it's IBM's (co-developed from the G4/7400), and it's fairly well-understood. Downclock it to reduce power consumption and it could even be a credible upper-end embedded chip. The only thing it lacks is a true little-endian mode.

More on Firefox 69 when it is officially released next week.

Support from a silicon turnip

Raptor is asking users to do a brief checklist before asking for support to help streamline problem determination. I think this just makes good sense, but although it's reasonable to assume most users would have another system around that can talk to the BMC (I use my Quad G5 for this), it might be nice in a future firmware version to have some sort of confidence testing. I'm not sure how that would look necessarily in implementation but I know when I was trying to determine why my kernel was freaking out that eliminating hardware as a cause, however unlikely, would have been helpful.

Ordinarily this would merit merely a brief informational item, except let's consider it in the context of my earlier underdeveloped pontification: Raptor must now have enough of an installed base that streamlining support is now necessary. I'm an early adopter; my Talos II is serial #12 and my Blackbird is serial #75. Back in those distant bygone days of 2018 with the early firmware that ran like a wind tunnel, I pretty much conversed with support directly over E-mail (I suspect it was Tim himself) and handled everything that way, but that clearly wouldn't scale beyond a certain number of even technically adept users. (I did comment at the time that it was the best support I'd ever had with any computer system and I still think so.)

We don't know how many Talos-family systems are out there, and Raptor is not a public company, so sales figures are kept close to the vest. I don't really begrudge them this, either, because pro-Power bigots like me would still use the platform even if we were the only ones out there and haters gonna hate whether there's 10 or 10 million. (However, if people want to post serial numbers in the comments, we can find the highest one and make an educated guess.) I think we can safely assume the support volume is not being driven by poor quality, so if support volume has increased to a critical mass where changes must be made, then that must be due to enough machines out there actually being used. And as I said in the prior article, moving enough machines is the only sane way to get the cost down. I stand by my musings that a good second workstation-market supplier could have advantages for both volume and market stability, but we also don't want some knockoff company that isn't beholden to open libre computing principles sucking the life from this segment with a race to the bottom.

Having said that: I hear things and people tell me stuff, and while I'm sworn to secrecy right now, I am permitted to say obliquely that a promising development in moving more machines is afoot. I think that's as much as I should say on the subject but if it pans out, I think all of us in the OpenPOWER world will be very, very pleased.