Introducing Kestrel

The new product tease we reported on from Raptor has a name: Kestrel. We theorized it was a FlexVer peripheral, but Raptor says it isn't, though it says "it's one of the most critical components of one." The image posted on Twitter (reproduced here) requires connection to the LPC, I2C (both platform control and AVSBus) and FSI signals, the latter of which will require either soldering or voltage conversion. Seems a strange omission for why that wouldn't simply be included on the board. Another curious omission is that the image also adds it has not yet been tested on the flagship Talos II (or presumably the T2 Lite), just the Blackbird, though reading the little what's available I don't see why it wouldn't work.

So, after all that, what does it do? The connection to those buses suggests some sort of low-level system monitor. If you really want to get down to the lowest level of what your system is executing, this is probably the device you want on one of the few systems that lets you do it as a supported feature. Here's a schematic of what the POWER9 is doing on bootup (what IBM docs call IPL, or the Initial Program Load):

The FSI connection is the biggest key here, which is the OpenPOWER Flexible Support Interface. This interface is active very early in standby mode, with clock signals available shortly after the machine is connected to power and the BMC is coming up — before even "Step 0" on this flowchart. Among many other things, the FSI triggers when the POWER9 Self-Boot Engine starts executing from its fused-in OTPROM, which contains the first instructions the POWER9 executes, and the BMC uses the FSI to determine which side of the SEEPROM the main CPUs should boot from. The FSI and IPC connections also allow monitoring the PNOR and other low-level traffic. With all these busses running through the Kestrel, you should be able follow the BMC and main CPUs all the way from standby to the end of IPL.

What we still don't know is if it will let you actually manipulate these signals, which could be a very powerful tool. Yes, you could potentially wreak havoc on your machine but I think soldering connections wrong would have a similar effect, so why not give us the keys to the store? Even if it's merely as a monitor, however, it could certainly be a way to have confidence a machine has not been tampered with and for hardware designers to understand better how present-day OpenPOWER systems operate.

The first Raptor tease of 2021

If you're wondering what Raptor has in the works next, it's not a new Condor: it's apparently something a little smaller. The company is teasing a new device for "for anyone interested in FPGA, open HDL, open FPGA tooling, low level IBM OpenPOWER POWER9 initialization, and minimal roots of trust." This doesn't sound like a computer; it may well be a peripheral for the FlexVer connector. If such a device could help improve the currently arduous secure boot procedure, this would be another big step forward in owner control and security.

Fedora 3-4K?

As previously mentioned, Fedora is the standard distribution we run right now on our two Raptor systems at Orbiting Floodgap HQ and, being one of the first distros to work more or less out of the box on POWER9 systems, is probably one of the most frequently deployed distributions on OpenPOWER workstations (Debian and Void PowerPC, of course, being in the mix there too).

Besides the big-endian and little-endian splits, the 64K vs 4K page size variation is now another notable fault line. Much software assumes the dominant little-endian byte ordering just as much other software, often the same software, assumes the dominant 4K page size, despite significant performance reasons for 64K pages on some platforms and not just on ppc64/ppc64le. Void is prominently 4K, for example, but Fedora uses a 64K page kernel as shipped, meaning significant software like the Wine-QEMU fusion Hangover can't currently run on Fedora as presently configured.

What about making Fedora Workstation 4K? There are clear advantages to keeping Server on a 64K page size, especially since those systems are likely to have the large memory volumes where the performance benefits of big pages would be most visible, but there are increasing compatibility disadvantages of having workstations on 4K. In a proposal mailing list thread, Daniel Pocock points out that 4K is needed both for the open-source Nvidia Nouveau driver as well as for the AMD RX 5700 (and probably other similar cards based on the same GPU generation), and Peter Robinson notes that the aarch64 spin went 4K for other reasons particularly acute on RPis and similar devices (though it remains 64K on RHEL 7 and 8). And all of this is independent of the compatibility issues that are already well-known and some non-trivial to fix. Is it time for a 4K spin of Fedora 34?

Compiling a 4K kernel is as "simple" as setting CONFIG_PPC_64K_PAGES=n but there is also the confidence and regression testing to prove a longstanding 64K platform doesn't itself have unknown assumptions going the other way, to say nothing of the risk of marginalizing 64K page systems when (and arguably unlike big vs little) performance differences of substance may exist and of what amounts to essentially fragmenting an already comparatively niche architecture. (As observed by another developer in the thread, "We have avoided doing so for much larger target markets than the power [sic] workstation market.") It may also be possible to solve this unofficially in a relatively maintainable fashion with a Copr kernel package, though I didn't see an existing one and I don't have much personal experience with this (perhaps someone who knows of such a package or the process can chime in).

Still, part of the justification for ppc64le on a long-standing big-endian platform was to meet existing software where it was and 4K pages may force the same change. If a 4K Fedora Workstation existed, I'd certainly use it: I have no especial loyalty to Fedora, but it's what I'm used to, and I'd rather stick with it right now than deal with an inconvenient migration ... at least until something vital comes along that I need to run.

Void PPC goes little-endian ... on 32-bit

It is frequently forgotten that just as 64-bit Power ISA comes in both big-endian (powerpc64, ppc64) and little-endian (ppc64le) variants, you can also have 32-bit little-endian (ppcle) as well as the classic 32-bit big-endian PowerPC most often encountered in Power Mac hardware. But such 32-bit little-endian systems have historically been quite rare (I struggle even to think of any), and distributions supporting them even more so.

So leave it to those wacky VoidPPC developers to not only spin a ppcle variant, but to even make it useful. It should be pointed out that such a distribution can run on little-endian OpenPOWER because there isn't the massive gulf between 32-bit and 64-bit PowerPC like there would be for x86 and x86_64. Accordingly, Daniel Kolesa demonstrated on Twitter a Blackbird running Void Linux but 32-bit little endian. Why do this on a 64-bit capable platform? Because it allows you to run certain 32-bit specific emulation systems like Box86 that due to their design don't run on 64-bit platforms (especially as Void is a 4K page system). And, true to form, the Blackbird was running Unreal Tournament 32-bit through Box86 at "fairly playable speed" — a game which was only ever made available for Linux as a 32-bit x86 binary (the PowerPC binaries were only for MacOS and Amiga). Such a feat with apparently acceptable performance is even more impressive given that Box86 doesn't have JIT support for ppcle either.

The ppcle spin (both glibc and musl variants) is already listed on the stats page and the port should be available shortly. Because of the unusualness of the architecture, cross-compilation depending on LLVM (and things that use it like Rust) may not yet work, but a significant slice of ports appear to already be built.

Linux 5.10

Linus Torvalds has tagged Linux 5.10, which will be the next long term support release. Despite a relatively small merge window, it includes nearly 14,000 commits.

Big new features in 5.10 include an ext4 performance improvement by reducing the amount of journal metadata written for crash recovery (unfortunately this feature right now needs to be enabled at the time the volume is mkfsed), secure ring sharing for io_uring, an API for manager processes to supply memory hints to other processes (I'm sure that won't be abused by anyone), support for static calls with in-place code patching as a better post-Spectre function pointer replacement than retpolines, widened timestamps on xfs to handle Y2038 (this wasn't already a thing? but requires explicit transitioning of old filesystems), lots of BPF improvements and many new drivers.

On the Power ISA specific side, 5.10 adds further support for POWER10 (shallow stop states and new watchpoint features), a fix for the three of you using a POWER9 in HPT mode with 4K pages and more than 16TB of RAM (or RAM attached to a second node), a filter for RTAS firmware calls to protect kernel memory, and better topology-aware scheduling on POWER9 and POWER10.

The saddest thing announced in this kernel, however, is the end of support for the original PowerPC 601. This is understandable as the 601 was always intended as a stepping-stone CPU and had important differences from classic PowerPC as later implemented in the 603, which had to be accounted for in system software. Nevertheless, if you've still got a Power Mac 6100 in your closet running Linux, your last call in LTS kernels is 5.8 (or 5.9, if you need the features).

There is no CentOS 8, there is only Stream

When IBM bought Red Hat in 2018 (who of course maintains Red Hat Enterprise Linux as a paid product and Fedora as its free community upstream, and since 2014 the de facto free version of RHEL, CentOS), as a Fedora user since it first booted on the Talos II, I had high hopes that finally OpenPOWER would be a first-class citizen on par with x86_64 under IBM's hopefully gentle goading. No more of this being in the alternative architectures penalty box for Fedora Workstation, for example. The idea was that IBM would see Red Hat's free open products as a logical extension of OpenPOWER and exploit the obvious synergy by having a free distribution available as a preferred choice on an open platform (and then the customers who want greater support and enterprise features could pony up). Win-win, right?

Well, first-class OpenPOWER Fedora still hasn't happened, and while RHEL remains perfectly happy to take your POWER8/POWER9 money, CentOS — or at least CentOS the way you've understood it, i.e., RHEL without the price or support contracts — is dead. There is no CentOS, there is only Stream (after 2021, that is, though CentOS 7 will finish its lifecycle as usual).

Let's be a little less handwringy, though, as Red Hat could have done a better job explaining what this means. As I so presciently determined back in 2019, Stream was clearly positioned as the "public beta" for RHEL and, at that time, for CentOS. It still is; this "merely" means there will be no stable channel. I also presciently determined in that 2019 article that there would be a relatively small slice of people who want "just enough" innovation compared to us on the bloody (Fedora) or bleeding (Rawhide) edge, but still prefer somewhat more current updates than those on regular RHEL or CentOS. Unexpectedly, Red Hat appears to have solved this problem by just eliminating classic CentOS. But people willingly pay good money for RHEL — or, you know, use CentOS — because stuff doesn't break much. Stream eliminates that "doesn't break" guarantee, as much as a free distribution could make such a guarantee in the first place, though it's definitely much less churn than Fedora and for many users will still fill the bill. Unlike Fedora, on CentOS Stream 8 you won't be forced to dogfood Stream 9 any earlier than an RHEL 8 user would be, though you may be forced to deal with 8.x.

There are certainly other RHEL downstreams because there must be (it must be open: that's how CentOS started in the first place). Amusingly, some of these are other proprietary vendor Linuces (Oracle Linux, Hewlett-Packard Enterprise ClearOS, etc.). But, and now getting to the OpenPOWER specific portion of this article, none of the free (as in beer) options run on POWER8 or POWER9. Springdale Linux, one of the few free rebuilds, is strictly x86, and Oracle Linux is free to download but only supported on aarch64 and x86_64 (sorry, SPARC). ClearOS is free but stuck on RHEL 7 and doesn't run on OpenPOWER either, and while CloudLinux claims to run on anything RHEL does, it costs money, so you might as well run RHEL unless you need its specific value-added features. [UPDATE: CloudLinux is now announcing a free community version in Q1 2021. No word on OpenPOWER support, but we're hopeful. Thanks Dimitris Z for reporting it.]

That leaves the recently announced Rocky Linux, led by Gregory Kurtzer, founder of the CentOS project. Rocky Linux aims to basically be what CentOS was originally: a downstream build of RHEL, without the branding or the fees. But all it is right now is an idea and no downloads are available, nor any indication that OpenPOWER will be supported, at least not as of this writing. When they come to a decision on that you'll hear it here first.

On the whole this announcement is probably of little concern for people using their OpenPOWER machines as workstations, because most of those that run a Red Hat derivative are probably running Fedora (yours truly included). A few will be running CentOS Stream, but that isn't going anywhere. Where this hurts is those individuals who wanted the superstability of CentOS without the supercost of RHEL, and that probably applies to a substantial number of people running OpenPOWER servers in high-availability environments. Many of these people will still be reasonably well served by Stream, but a few are so risk-averse that even Stream's small amount of turnover won't do for them either. That's no skin off IBM's nose because they'd rather have them as customers paying support fees, but it's not a good look for free computing, and it's not a good look for Red Hat specifically.

Which brings me to another prediction I made: "our worry is that the IBM monolith will affect Red Hat far more than the other way around." Is it my curse to always be right?