Showing posts from 2021

FreeBSD 13 and Guix for OpenPOWER

After a bit of downtime, we're back. And cool stuff has happened in our absence, the most notable being additional improvements to the increasingly mature OpenPOWER port of FreeBSD. 13-RELEASE, among other changes, officially introduces the 64-bit little-endian port (previously exclusively big-endian, which is still supported), experimental radix MMU support for POWER9 (hashed page tables are of course supported everywhere), XIVE interrupt support on POWER9 (about 10% faster), optimized memcpy(), memmove() and like-minded standard functions, and many stability and performance improvements. The releases notes say that "performance during bulk -a package building is at least 60% higher" which is very impressive. ISOs are available from their download server.

In addition, ppc64le support has been merged to the GNU Guix source tree, meaning with the next expected version 1.2.1 you'll hopefully be able to get a pre-built copy. It's been in development for several months and now it appears to be finally approaching reality. Like Guix the package manager, the GNU Guix System's most notable feature is its declarative service and package configuration, all on top of the GNU Shepherd init system and (right now) Linux 5.9. Currently there is still a reproducibility issue with gcc, rust is still at least somewhat experimental (which is relevant for librsvg) and many packages have not been tested. Still, since the Talos II and T2 Lite are GNU Respects Your Freedom systems, now you can run another GNU-free OS on them too and sooner than you think.

More 64K page problems and some solutions

Meanwhile, as the question of a 4K-page Fedora 34 remains as yet undecided, if you are using a more recent video card with your Linux OpenPOWER system Trung LĂȘ reports that kernel 5.11.x still crashes with 64K pages on his AMD R9 Nano. (Older cards, like the AMD WX7100 workstation GPU in this Talos II that Raptor sells as a BTO option, are unaffected may also be affected — see comments for more.) This is relevant since Fedora 33 is moving to 5.11. If you're a Fedora 33 user and wish to continue with the 5.10.x series until a fix for amdgpu emerges, kernels are available from his Github project.

In the meantime, speaking personally as one of those people who still use FireWire/IEEE-1394, 4K pages are at least part of the problem as to why FireWire cards don't seem to work in my F33 Talos II (I tried a Rosewill one first without success, and more recently an Iocrest card using a more typical Texas Instruments controller). Although a patch for 64K page support was initially submitted, it was rejected, and the followup patch was never tested. I'll be getting around to trying this myself and hopefully getting it into the kernel, but in the meantime report back if using the patch works for you (I still use some FireWire devices, particularly for video capture and legacy interchange with the Power Macs that lurk around here).

Tonight's game on OpenPOWER: The Original Strife Veteran's Edition

I'm a big fan of Strife, famously the last game to use id Software's Doom 3-D engine, and a nice hybrid of light RPG and heavy action. The engine might have been old and the plot was more shooting than Shakespeare, but hey: the voice in your ear is named Blackbird. You can't beat that!

I actually do own a retail copy of Strife from back in the day for MS-DOS; I bought it new and played it on the 486 I still keep around for such things. It plays just fine in Chocolate Doom on this POWER9, but I later heard about Strife: Veteran Edition on that fixed minor bugs with better music and improved graphics, and even threw in some extra enhancements and achievements, but still kept the plot, voice acting and character art. It had a Linux version, but clearly one for x86. But that's not a problem when you have the source code.

The source code builds largely uneventfully as long as you have the prerequisites (Fedora 33 and I did not test on big-endian). In particular, it will want cmake, SDL2, libogg, libtheora, libvorbis, zlib, libpng and OpenGL. However, it tries to link against libSD2_main which is no longer necessary; after you've run cmake and make, it will fail with No rule to make target 'SDL2_MAIN_LIBRARY-NOTFOUND'. To get around this, edit (in the build directory where you ran cmake) ./CMakeFiles/strife-ve.dir/link.txt and remove SDL2_MAIN_LIBRARY-NOTFOUND from the single long line link command, then edit ./CMakeFiles/strife-ve.dir/build.make and just delete the line strife-ve: SDL2_MAIN_LIBRARY-NOTFOUND. Run make again and it will link.

Since it built, I decided to spend $10 and try to extract the game assets from the GOG pack. GOG gives this to you as a behemoth 400 megabyte "shell script" which is really a wrapper for a ZIP archive with a MojoSetup installer. Irritatingly the installer is all just binaries, but you can feed it to unzip and it will break it apart. If we list the contents of the file, it will conveniently ignore the header and go right for the ZIP archive, and the money is in data/noarch/game. Thus, do unzip ./ data/noarch/game/'*' and the assets will be extracted to data/noarch/game.

If you want, you can just move the files in game/ (maintain the tree under it, don't flatten it) in with the POWER9 binary of strife-ve, but if you don't want all the x86 binary crap you don't need, a quick find . -name '*.so.*' -print | xargs rm and rm strife-ve before you copy it over should remove the bulk of it. Conveniently, the GOG assets also include the original DOS version (in DOS/) and all the relevant WADs so you can also run it in Chocolate Doom or our OpenPOWER-JIT DOSBox, and another copy of the source code just in case you lose it. Anyway, with everything moved, if you run ./strife-ve it should then just work.

Don't keep the Front waiting.


The RISC-V community is buoyed by Wave Computing, fresh from Chapter 11 bankruptcy, reemerging with the name of its subsidiary MIPS Technologies to develop ... RISC-V chips.

This actually says less about RISC-V than it says about the new MIPS Technologies. You'll recall that MIPS Technologies, formerly Wave Computing, suspended the MIPS Open Initiative, apparently to position it for sale before they went under, and nobody bit. Understandably so: the once great architecture that powered the SGI MIPS workstations in my office (I own an Indy, an Indigo2 and a Fuel) has now been relegated to the "too cheap for ARM" embedded market, which coincidentally is exactly where RISC-V has gotten most of its design wins so far.

And Wave MIPS Technologies isn't giving anything up. There is no mention of the licensing program for MIPS-the-architecture ending, which means it remains in operation, and it remains closed. Indeed, Tallwood Venture Capital probably demanded it, as a hedge in case their efforts with RISC-V aren't sufficiently profitable. MIPS Technologies will be entering a crowded field with other established players, notably SiFive, and not a lot of extant IP to suggest they will substantially leapfrog those existing designs in performance or power usage (if at all). In that sense, this announcement is best seen as a cynical way to capture public interest rather than an important engineering leap, and the RISC-V community should not in any way conclude they have gained a valuable partner. If anything, they've failed to avoid a new, shadier member of the ecosystem who actually took steps to make their previous products less open. That's not a good look for an architecture that has made openness its defining characteristic.

Juicing QEMU for fun, ??? and profit!

The number of packages and applications natively available for OpenPOWER continue to grow in just about every distro's package manager, and even if a prebuilt package doesn't exist even more will build from source. But emulation is still going to be a fact of life for Windows-only/x86/x86_64-only (maybe even aarch64-only) binaries we can't rebuild, and KVM only helps us with other Power ISA systems (in fact, it looks like KVM-PR broke and can't boot Mac OS X again, so I guess I'll be diving back into the source), so we need to wring as much speed out of QEMU's emulation engine as possible.

We are fortunate with QEMU in that there is ppc64le support in TCG, the Tiny Code Generator which implements a basic JIT, and the Power ISA TCG backend even emits those tasty newer POWER9 instructions to take better advantage of the processor. Without TCG, QEMU would be dreadfully slow when emulating a foreign architecture. However, unless IBM or some other OpenPOWER hardware developer implements instructions (a la Apple M1) in a future chip that specifically improve emulation of other CPUs (like, I dunno, x86_64), there's very little that can be done to improve the code the Power TCG backend generates and CPU emulation spends most of its time in TCG-generated code.

However, the software MMU that QEMU's CPU emulation uses has pre-compiled portions, and all the devices and components QEMU emulates (like the system bus, video, mass storage, USB, etc.) are also pre-compiled. This gives us an opportunity: with a little extra elbow grease, you can make a link-time-optimized and profile-guided-optimized (LTO-PGO) build of QEMU specific to the particular workload which can run the CPU anywhere from 3-8% faster and video and other devices up to 15% faster depending on the set of devices. While number crunching isn't substantially faster, and the modest CPU improvements don't improve user-mode emulation a great deal, full system emulation's general responsiveness improves and makes using more applications more feasible.

This process is not automated. For Firefox, we make LTO-PGO builds using the internal machinery and our patches for gcc compatibility, which is currently our preferred compiler on OpenPOWER systems. The Firefox build system generates a profiling build first, then automatically collects profiling data with it off a model workload and builds the optimized browser from that profile. QEMU doesn't have that infrastructure right now, but you can do it manually: you configure and compile a profiling build, run your workload with it to create a profile, and then configure and compile an optimized build with the profile thus generated.

I'll give instructions here for both QEMU 5.0 and 5.2, since 5.0 seems to be a bit more performant than 5.2 and has fewer build prerequisites, but 5.2 is more straightforward and we'll do it first. In these examples, I'm optimizing ppc-softmmu so that I can run Mac OS 9, which has never worked properly with KVM-PR; substitute with your desired target, such as x86_64-softmmu. Only do one target at a time, and you will want to do individual builds for each system image — even if you normally use the same executable binary for multiple OSes — because different code paths may be exercised with different workloads and/or configurations.

Let's start with making a profiling build. To do this, we'll add -fprofile-generate to the compiler flags (as well as -flto for LTO). For consistency we'll pass the same set of options to the C compiler, the C++ compiler and the linker (each will ignore options they don't need). In the QEMU source tree,

  • mkdir build
  • cd build
  • ../configure --extra-cflags="-O3 -mcpu=power9 -flto -fprofile-generate" \
    --extra-cxxflags="-O3 -mcpu=power9 -flto -fprofile-generate" \
    --extra-ldflags="-flto -fprofile-generate" --target-list=ppc-softmmu
  • make -j24 (or as appropriate: this is a dual-8 Talos II)

Wait for QEMU to build. When it finishes, back up your drive image because you may not be able to shut it down normally and it would suck to damage it inadvertently. With a backup copy saved, run the new QEMU as you ordinarily would on your target workload. For example, my classic script is (assuming you're still in the build directory)

./qemu-system-ppc -M mac99,accel=tcg,via=pmu -m 1536 -boot c \
-drive id=root,file=classic.img,format=qcow2,l2-cache-size=4M \
-usb -netdev tap,id=mynet0,ifname=tap0,script=no,downscript=no \
-device rtl8139,netdev=mynet0 -rtc base=localtime

You should use as close to your normal configuration as possible so that the device drivers you run are factored into the profile.

The first thing you'll notice is that QEMU is now really, really, really slow. Crust-of-the-earth-cooling slow. This is because it's storing all that profile data every time any block of compiled code is executed. As a result you will probably not be able to type or interact with the guest in any meaningful fashion, so let the system boot, grab a cup of a fortifying beverage and and wait for it to get as far as it can. For Mac OS 9, it took several minutes to get to the desktop; for OS X 10.4, it took about a quarter of an hour (with a lot of timeouts in a verbose boot) to even start the login window. At some point you will not be able to usefully proceed any further with the guest, but fortunately you backed up your drive image already, so you can simply close the window.

Go back to the build directory. This time we will tell gcc to build with the generated profile (-fprofile-use), though we will allow it to account for certain changes (-fprofile-correction) and allow compilation to occur even if a profile doesn't exist for a particular target (-Wno-missing-profile) so that it can get through configure cleanly:

  • make clean (this doesn't remove the profile .gcda files)
  • ../configure \ --extra-cflags="-O3 -mcpu=power9 -flto -fprofile-correction -fprofile-use -Wno-missing-profile" \
    --extra-cxxflags="-O3 -mcpu=power9 -flto -fprofile-use -fprofile-correction -Wno-missing-profile" \
    --extra-ldflags="-flto -fprofile-use -fprofile-correction -Wno-missing-profile" \
  • make -j24

Enjoy the new hotness. You should be able to see measurable improvements in the CPU emulation, but more importantly, boot times and responsiveness of the full system emulation should also be improved.

For 5.0.0, the process is a bit more complicated, but it's a bit quicker, so I found it worth it (and it's what I currently use for Mac OS 9). In the QEMU source tree, configure the build:

  • ./configure --extra-cflags="-O3 -mcpu=power9 -flto -fprofile-generate" \
    --extra-cxxflags="-O3 -mcpu=power9 -flto -fprofile-generate" \
    --extra-ldflags="-flto -fprofile-generate" --target-list=ppc-softmmu
  • make -j24

Run your profile as before. However, you need to preserve the profile before the rebuild because make clean will clobber it.

  • tar cvf instrumented.tar `find . -name '*.gcda' -print`
  • make clean
  • tar xf instrumented.tar
  • ../configure \ --extra-cflags="-O3 -mcpu=power9 -flto -fprofile-correction -fprofile-use -Wno-missing-profile" \
    --extra-cxxflags="-O3 -mcpu=power9 -flto -fprofile-use -fprofile-correction -Wno-missing-profile" \
    --extra-ldflags="-flto -fprofile-use -fprofile-correction -Wno-missing-profile" \
  • make -j24

Life's golden, and just a little bit zippier. It's not always possible to PGO all the things, but here's one where it makes a noticeable difference.

Firefox 86 on POWER

Firefox 86 is out, not only with multiple picture-in-picture (now have all the Weird Al videos open simultaneously!) and total cookie protection (not to be confused with other things called TCP) but also some noticeable performance improvements and finally gets rid of Backspace backing you up, a key I have never pressed to go back a page. Or, maybe those performance improvements are due to further improvements to our LTO-PGO recipe, which uses Fedora's work to get rid of the sidecar shell script. Now with this single patch, plus their change to nsTerminator.cpp to allow optimization to be unbounded by time, you can build a fully link- and profile-guided optimized version for OpenPOWER and gcc with much less work. Firefox 86 also incorporates our low-level Power-specific fix to xpconnect.

Our .mozconfigs are mostly the same except for purging a couple iffy options. Here's Optimized:

export CC=/usr/bin/gcc
export CXX=/usr/bin/g++

mk_add_options MOZ_MAKE_FLAGS="-j24"
ac_add_options --enable-application=browser
ac_add_options --enable-optimize="-O3 -mcpu=power9"
ac_add_options --enable-release
ac_add_options --enable-linker=bfd
ac_add_options --enable-lto=full
ac_add_options MOZ_PGO=1

# uncomment if you have it
#export GN=/home/censored/bin/gn
And here's Debug:
export CC=/usr/bin/gcc
export CXX=/usr/bin/g++

mk_add_options MOZ_MAKE_FLAGS="-j24"
ac_add_options --enable-application=browser
ac_add_options --enable-optimize="-Og -mcpu=power9"
ac_add_options --enable-debug
ac_add_options --enable-linker=bfd

# uncomment if you have it
#export GN=/home/censored/bin/gn

Blackbird supply chain likely to improve

(Thanks to a reader tip.) Although yours truly is a Talos man (and was ever since it was going to have a POWER8 in it), the Blackbird is certainly far more attractive in terms of price. Backorders due to COVID-19's effect on the global supply chain have plagued it for months, but Raptor management on IRC indicates that logjam may be breaking; the first sign just a few days ago is that the 18-core monster POWER9 v2s (DD2.3) were back in stock. Obviously 18-cores don't (routinely) go in Blackbirds, but their presence suggests the supply chain issues are resolving and that a minimum order from IBM was met.

Raptor is well aware that the Blackbirds, more so than the T2 and T2 Lite, are its leading workstation product, and said there was "lots of demand" too ("about the only positive in the whole pandemic-induced mess"). However, Raptor's Timothy Pearson in the same IRC chat also commented that "we're playing it safe and focusing more on the next generation products than taking risks with POWER9 ... I can categorically state that if COVID19 had never happened, we'd have already offered other chips and we'd have at least one other product on the market designed around P9 by now." The latter sounds like a reference to Condor, Raptor's cancelled LaGrange system, but as long as POWER10 still has openness concerns, what "other chips"?

Gentoo on little-endian

A nice write up by Martin Kukač on getting Gentoo to be happy on little-endian: even though many Linux distributions support LE, and some now only do, if you install Gentoo from the Minimal Installation CD and try to use the ppc64le stage 3 tarball there's an endian mismatch and it doesn't work (dies during the install steps with /bin/bash in incompatible format). The issue appears to be that the Minimal Installation CD itself is big-endian; there is currently no analogous little-endian image. Martin's brainwave was to complete the installation from an already running little-endian system (he used RiscySlack but Void should also work as well). Following his steps, the OS will build in little-endian mode from within the second OS, and then can be booted into it. Good to have the choice and a nice how-to.

A better theory on why there won't be an open POWER10 workstation for awhile

In our previous analysis we suspected that Raptor's indigestion over POWER10 was IBM failing to release some component of the firmware, meaning it wasn't a truly open platform after all. Raptor, under whatever NDA prohibited them, couldn't say, but there was enough to do some educated reading between the lines regarding the problem.

So hats off to Hugo Landau, who did his own research on the subject. As you will recall, for POWER8 IBM introduced the Centaur memory buffers which serve essentially as off-chip memory controllers and a fourth level of cache, and scale-up Cumulus POWER9s (not the Nimbus POWER9s in Raptor workstations) can use them too. This enables a lot of logic to be move off-die and can turn what is a critical high-speed and potentially error-prone parallel interface into a serial one. IBM expanded this into the vendor-neutral Open Memory Interface, or OMI, which halves the latency of Centaur (to 5ns) and runs up to 25Gbps per lane. With OMI RAM technology can advance separately from the CPU, and the processor can be completely agnostic about what it's attached to (as opposed to Cumulus, which only "speaks" Centaur, and our Nimbus systems which use commodity directly-attached DDR4 RAM through an on-chip controller).

We reported previously that at the 2019 OpenPOWER summit Microchip Technology was announced as the first vendor of OMI DDIMMs, and although Micron, Samsung and SMART Modular were listed as planning to release their own, so far the only vendor of OMI controllers appears to be Microchip. We haven't heard anything about a Nimbus-alike POWER10 yet with direct-attached memory, so we have to assume that at least the first wave of POWER10 processors will only use OMI. Hugo's discovery was a obscure Github repo that appears to contain the firmware for the Microchip OMI controller — and no source code. Read Hugo's article for the additional dirty details.

The concept of RAM that requires firmware binary blobs is frankly very disconcerting: I shouldn't have to explain to any regular reader of this blog that if you own the RAM, you own the store, and you could potentially own the RAM this way (even/especially with a vendor lock: see SolarWinds). I won't say how I have knowledge of this, but various other cues indicate to me Hugo has found the exact reason POWER10 can't be considered open under any reasonable definition.

POWER9 systems can't last forever, of course. If there were going to be a truly open POWER10 system, we'd either have to reverse-engineer the Microchip controller firmware or develop a separate open memory controller of "our" own. Likewise, I'm pretty sure Raptor doesn't want to be in the DDIMM business, so if a separate Raptor-specific controller were required it may be simpler to just have RAM on the board as a build-to-spec option. Either way, while I understand IBM's decision with OMI to cater to their bandwidth-hungry institutional customers, the implementation they've chosen may put those very same high-value customers at risk. We should be glad Raptor didn't make the same choice and fortunately POWER9 systems will still be able to hold their own for awhile.

Followup on Firefox 85 for POWER: new low-level fix

Shortly after posting my usual update on Firefox on POWER, I started to notice odd occasional tab crashes in Fx85 that weren't happening in Firefox 84. Dan Horák independently E-mailed me to report the same thing. After some digging, it turned out that our fix way back when for Firefox 70 was incomplete: although it renovated the glue that allows scripts to call native functions and fixed a lot of problems, it had an undiagnosed edge case where if we had a whole lot of float arguments we would spill parameters to the wrong place in the stack frame. Guess what type of function was now getting newly called?

This fix is now in the tree as bug 1690152; read that bug for the dirty details. You will need to apply it to Firefox 85 and rebuild, though I plan to ask to land this on beta 86 once it sticks and it will definitely be in Firefox 87. It should also be applied to ESR 78, though that older version doesn't exhibit the crashes to the frequency Fx85 does. This bug also only trips in optimized builds.

Firefox 85 on POWER

Firefox 85 declares war on supercookies, enables link preloading and adds improved developer tools (just in time, since Google's playing games with Chromium users again). This version builds and runs so far uneventfully on this Talos II. If you want a full PGO-LTO build, which I strongly recommend if you're going to bother building it yourself, grab the shell script from Firefox 82 if you haven't already and use this updated diff to enable PGO for gcc. Either way, the optimized and debug .mozconfigs I use are also unchanged from Fx82. At some point I'll get around to writing a upstreamable patch and then we won't have to keep carrying the diff around.

Introducing Kestrel, part II: it's a soft-BMC

Well, geez, guys, why didn't you just say so in the first place? Kestrel is a "soft" BMC replacement, meaning you can devise your own Baseboard Management Controller/service processor to bring an OpenPOWER system up from absolutely nothing but a Lattice ECP5. Now, that's cool!

The underpinnings are strongly based on the Microwatt soft core and as such makes it a true OpenPOWER processor itself, not ARM like the ASPEED BMC on shipping Raptor systems. Kestrel is not currently far enough along to bring up the On-Chip Controllers on the POWER9 (PowerPC 4xx-like cores), but this appears to merely be a matter of adding more IPMI command support. It is, however, enough to kick off the POWER9's Self-Boot Engines and go into Hostboot, so the basics absolutely work.

Right now I think this system is a little raw for general usage, and the soldering requirement on a several thousand dollar board that's badly backordered is not appealing. The whole dev stack is also intended for Raptor systems, though to be honest if you care about Kestrel you undoubtedly already own one. But Raptor is to be commended for making a shippable product out of Microwatt and making it truly open, as one would expect from them. What I'm interested to see, however, is whether future Raptor systems have Kestrels on board instead of the ASPEED. That would be really impressive in terms of owner control and would make the current valiant but vain efforts to neuter x86 firmware look even more pathetic.

There is no CentOS 8, there is only Stream and also this limited no-cost developer thing

Enough sting came from Red Hat's decision to jettison CentOS that Red Hat has backpedaled ... sort of. Instead of reviving CentOS in its prior RHEL-for-RHEL form, however, you'll just get Red Hat Enterprise Linux itself for no money out of pocket, provided you run it on a "small production workload" (16 systems or less, up from the prior single machine limitation) and sign up for the newly expanded Red Hat Developer Subscription program. Development teams can also take advantage of the free tier for cloud work. Either way, this "freemium" RHEL becomes available on February 1.

Is that enough? It's not ideal, but it is RHEL and you get it for $0 with a promise you won't become sales call fodder. However, Ars Technica is reporting that annual renewal is required, which doesn't sound that great for people who desire RHEL for long-term purposes (i.e., just about everyone who would use RHEL over CentOS Stream). On the other hand, while CloudLinux's free version now has its new name AlmaLinux, there's no official word yet on whether OpenPOWER is supported and RockyLinux's initial Q2 release will only be ARM and x86. If you're an OpenPOWER shop but the lesser stability guarantees of CentOS Stream really won't work for you, then freemium RHEL may be your only choice as long as your operation is small enough, at least for right now.

Introducing Kestrel

The new product tease we reported on from Raptor has a name: Kestrel. We theorized it was a FlexVer peripheral, but Raptor says it isn't, though it says "it's one of the most critical components of one." The image posted on Twitter (reproduced here) requires connection to the LPC, I2C (both platform control and AVSBus) and FSI signals, the latter of which will require either soldering or voltage conversion. Seems a strange omission for why that wouldn't simply be included on the board. Another curious omission is that the image also adds it has not yet been tested on the flagship Talos II (or presumably the T2 Lite), just the Blackbird, though reading the little what's available I don't see why it wouldn't work.

So, after all that, what does it do? The connection to those buses suggests some sort of low-level system monitor. If you really want to get down to the lowest level of what your system is executing, this is probably the device you want on one of the few systems that lets you do it as a supported feature. Here's a schematic of what the POWER9 is doing on bootup (what IBM docs call IPL, or the Initial Program Load):

The FSI connection is the biggest key here, which is the OpenPOWER Flexible Support Interface. This interface is active very early in standby mode, with clock signals available shortly after the machine is connected to power and the BMC is coming up — before even "Step 0" on this flowchart. Among many other things, the FSI triggers when the POWER9 Self-Boot Engine starts executing from its fused-in OTPROM, which contains the first instructions the POWER9 executes, and the BMC uses the FSI to determine which side of the SEEPROM the main CPUs should boot from. The FSI and IPC connections also allow monitoring the PNOR and other low-level traffic. With all these busses running through the Kestrel, you should be able follow the BMC and main CPUs all the way from standby to the end of IPL.

What we still don't know is if it will let you actually manipulate these signals, which could be a very powerful tool. Yes, you could potentially wreak havoc on your machine but I think soldering connections wrong would have a similar effect, so why not give us the keys to the store? Even if it's merely as a monitor, however, it could certainly be a way to have confidence a machine has not been tampered with and for hardware designers to understand better how present-day OpenPOWER systems operate.

The first Raptor tease of 2021

If you're wondering what Raptor has in the works next, it's not a new Condor: it's apparently something a little smaller. The company is teasing a new device for "for anyone interested in FPGA, open HDL, open FPGA tooling, low level IBM OpenPOWER POWER9 initialization, and minimal roots of trust." This doesn't sound like a computer; it may well be a peripheral for the FlexVer connector. If such a device could help improve the currently arduous secure boot procedure, this would be another big step forward in owner control and security.

Fedora 3-4K?

As previously mentioned, Fedora is the standard distribution we run right now on our two Raptor systems at Orbiting Floodgap HQ and, being one of the first distros to work more or less out of the box on POWER9 systems, is probably one of the most frequently deployed distributions on OpenPOWER workstations (Debian and Void PowerPC, of course, being in the mix there too).

Besides the big-endian and little-endian splits, the 64K vs 4K page size variation is now another notable fault line. Much software assumes the dominant little-endian byte ordering just as much other software, often the same software, assumes the dominant 4K page size, despite significant performance reasons for 64K pages on some platforms and not just on ppc64/ppc64le. Void is prominently 4K, for example, but Fedora uses a 64K page kernel as shipped, meaning significant software like the Wine-QEMU fusion Hangover can't currently run on Fedora as presently configured.

What about making Fedora Workstation 4K? There are clear advantages to keeping Server on a 64K page size, especially since those systems are likely to have the large memory volumes where the performance benefits of big pages would be most visible, but there are increasing compatibility disadvantages of having workstations on 4K. In a proposal mailing list thread, Daniel Pocock points out that 4K is needed both for the open-source Nvidia Nouveau driver as well as for the AMD RX 5700 (and probably other similar cards based on the same GPU generation), and Peter Robinson notes that the aarch64 spin went 4K for other reasons particularly acute on RPis and similar devices (though it remains 64K on RHEL 7 and 8). And all of this is independent of the compatibility issues that are already well-known and some non-trivial to fix. Is it time for a 4K spin of Fedora 34?

Compiling a 4K kernel is as "simple" as setting CONFIG_PPC_64K_PAGES=n but there is also the confidence and regression testing to prove a longstanding 64K platform doesn't itself have unknown assumptions going the other way, to say nothing of the risk of marginalizing 64K page systems when (and arguably unlike big vs little) performance differences of substance may exist and of what amounts to essentially fragmenting an already comparatively niche architecture. (As observed by another developer in the thread, "We have avoided doing so for much larger target markets than the power [sic] workstation market.") It may also be possible to solve this unofficially in a relatively maintainable fashion with a Copr kernel package, though I didn't see an existing one and I don't have much personal experience with this (perhaps someone who knows of such a package or the process can chime in).

Still, part of the justification for ppc64le on a long-standing big-endian platform was to meet existing software where it was and 4K pages may force the same change. If a 4K Fedora Workstation existed, I'd certainly use it: I have no especial loyalty to Fedora, but it's what I'm used to, and I'd rather stick with it right now than deal with an inconvenient migration ... at least until something vital comes along that I need to run.