Now that the Talos II is back in order and the Fedora 37 upgrade is largely behind me, it's now time to upgrade Firefox to version 108. There's some nice performance improvements here plus a hotkey for about:processes with Shift-Escape. Support for WebMIDI seems a little gratuitous, but what the hey (haven't tried it yet, the Macs mostly handle my music stuff), and there are also new CSS features. As before linking still requires Dan Horák's patch from bug 1775202 or the browser won't link on 64-bit Power ISA (alternatively put --disable-webrtc in your .mozconfig if you don't need WebRTC). Otherwise, we were able to eliminate one of our patches from the PGO-LTO diff, so use the new one for Firefox 108 and the .mozconfigs from Firefox 105.
Posts
When Petitboot barfs, everything's vomit
In its most literal sense this article is largely a precautionary tale, because unless you're a long-term Fedora user like me with a continuously updated older installation, it's very unlikely you have an XFS volume in your OpenPOWER box. But if the antique kernel in Petitboot ever starts barfing on your own filesystems or a device you install, you'll be in this state too, so here's how I got the Talos II working again.
It's pretty much been a constant that you need a second system to deal with glitches. For me, this is usually my trusty Quad G5 Power Mac sitting next to the T2 which is connected to its serial port (or to the BMC's), and this works when it's a problem you can resolve from the BMC side, which is many of them. It would be nice to power up a Talos or Blackbird and have the console automatically start up talking to the BMC instead of needing another system to do so but this is what we have, at least until Kestrel develops that capability.
Unfortunately, this wasn't one of those problems:
[Disk: nvme1n1p2 / 19a5d4e3-19f7-423f-a75b-5b15c8ee0bff]
Fedora (0-rescue-ee275f6a7d994c9981e4e1436b83172d) 30 (Workstation Edition)
Fedora Linux (5.18.13-200.fc36.ppc64le) 36 (Workstation Edition)
Fedora Linux (5.18.18-200.fc36.ppc64le) 36 (Workstation Edition)
(*) Fedora Linux (6.0.12-200.fc36.ppc64le) 36 (Workstation Edition)
System configuration
System status log
Language
Rescan devices
Retrieve config from URL
*Exit to shell
[fedora-root] Processing new Disk device[ 8.041704] XFS: Assertion failed: !
(fields & XFS_ILOG_DFORK) ||
(len == in_f->ilf_dsize), file: fs/xfs/xfs_log_recover.c, line: 3103
cpu 0x26: Vector: 700 (Program Check) at [c0002007e33171c0]
pc: c008000008dc46bc: assfail+0x54/0x60 [xfs]
lr: c008000008dc4694: assfail+0x2c/0x60 [xfs]
sp: c0002007e3317450
msr: 900000000282b033
current = 0xc0002007e32c3180
paca = 0xc0002007ff7f5900 irqmask: 0x03 irq_happened: 0x01
pid = 649, comm = pb-discover
kernel BUG at fs/xfs/xfs_message.c:110!
After the assertion appeared, Petitboot locked up (at least on the regular console) and the system wouldn't start from any device because Petitboot could not be coerced into ignoring it. I tried holding down the x key from the serial console to force it into the shell, and that worked — but it still tried to mount the volume anyway and died. This did bring up a live kernel debugging session as you can see in the screenshot, but since I wasn't sure what the XFS module would do at this point and didn't want to risk the filesystem, I just powered it down.
Something about the state of the root XFS volume after the Fedora 37 update was making it go wrong, and I haven't been the first to observe this, either. Recovering cleanly would at minimum require a system that can mount and examine the XFS volume, and the G5, which runs Mac OS X Tiger, isn't that system. (Maybe the SGI Fuel next to it with IRIX 6.5.30 is — though that's something to explore some other time when it isn't my primary computer's boot volume at stake.)
Fortunately I've also got a Blackbird that did complete its F37 upgrade successfully. So it's time to do a little shopping.
Both devices are USB 3.2 Gen 2 and came up as "SuperSpeed USB" connected to the Blackbird's rear USB ports. The Sabrent is a much nicer unit with high-quality metal construction that folds open and has an integrated heat spreader in the top. The "tool free" part is there's a small clip that rotates to hold the M.2 stick in (with a stopper in the package for smaller-sized sticks). But even though the Insignia was kludgier (pulls out instead of folds open, requires you to stick on a heat spreader, really clumsy turn clip), it supports USB Attached SCSI Protocol; dmesg indicated the Sabrent didn't respond to a UAS probe. If I could have combined the chipset in the Insignia with the case of the Sabrent, we'd have the perfect enclosure.
Both devices also worked in Petitboot — by which I mean having the tainted NVMe SSD plugged in while Petitboot came up would also crash the Blackbird.
Bringing up Fedora first and then connecting the enclosure after, we next get the T2's root volume up so it can be checked. Because both the Blackbird's boot drive and the T2's boot drive have the volume group name fedora, we'll need to rename the T2's. We list the volume groups with vgdisplay; the T2's starts with lO, so the commands are:
vgrename `vgdisplay | grep lO | awk '{print $3}'` tfed
lvchange -ay /dev/tfed/root
But xfs_repair /dev/tfed/root wouldn't try to fix it: it said there was a log entry that had to be replayed first. This can be done simply by mounting it, so
mount /dev/tfed/root /mnt
umount /mnt
xfs_repair /dev/tfed/root
This showed no errors, so I inactivated the root LV again with lvchange -an /dev/tfed/root, disconnected the NVMe stick, put it back in its PCIe carrier and reinstalled it in the T2. Petitboot didn't crash, but Fedora requires the logical volume be named fedora, so we enter the Petitboot shell first and finish up with
vgrename `vgdisplay | grep lO | awk '{print $3}'` fedora
and then boot.
Whose bug was this? Well, arguably, Fedora might not have properly unmounted the drive after the update, but the error appears to be minor in that simply mounting the drive (with a later kernel, admittedly) fixed up the issue. It's more important that Petitboot have a stable, well-tested codebase, so the decision to use an older kernel (though 5.5 is a little excessive) is not an unreasonable one, and this older kernel appears not to be able to do that kind of recovery.
But if Petitboot can't do it, it shouldn't just brick the system. There should be a way for a user to hold down a key and bypass the menu without mounting anything, and try to recover in the shell at that point, which you can do from the console. Similarly, if it barfs on a filesystem or an installed device, it should simply say so and ignore it, not panic. These computers are just too expensive to have vomit everywhere when something goes wrong — and you shouldn't have to have a whole second system around to clean up the mess.
Linux 6.1
I'm a little behind on stuff since I'm waiting for parts to get my T2 booting again (doing everything on my Mac laptop and my long-suffering Quad G5), but kernel version 6.1 came out, and there's some really good stuff on Power to mention.
But first the marquee general improvements: first, general support for Rust in kernel, which is now fairly mature on Power ISA (every Firefox build I make has it) and has obvious security benefits — assuming you're on a platform it supports, that is. The other change I think is a big one, possibly even bigger than Rust support, is the enhanced multi-generational LRU (Least Recently Used) memory page evictor: it's not on by default, but it ships as a configurable option, and some of the reports show some impressive performance wins. Finally, the new implementation of in-kernel maple trees means better cache hit rates and less lock contention for those kernel structures reimplemented with them (if you're 64-bit and have an MMU, which naturally we do), and I know people will appreciate the updates to AMD GPU support.
However, the Power-specific improvements are particularly interesting. If you're using the POWER9-and-up radix MMU (not available on the POWER8, nor if you need to use KVM-PR thanks Russell Currey for the correction: HPT already has this support), there's now the option of execute-only mapping (as opposed to read-execute which is supported with hashed page tables). Another important Power improvement is full support for 64-bit Power ISA under both hashed and radix MMUs with KFENCE, a "low-overhead sampling-based memory safety error detector of heap
use-after-free, invalid-free, and out-of-bounds access errors." Interestingly, 32-bit PowerPC was supported first!
To me, though, I'm most impressed with the exceptionally hard but worthy work done to rework system calls to use the new shared syscall wrapper implemented for s390, arm and x86 and obsolete the old legacy layer. This causes syscall handlers to take their parameters off the stack rather than relying on the state of the argument registers and r0, which is an obvious benefit if the registers are already on stack (such as for exception handling) because an additional stack frame wouldn't be needed, and further offers the opportunity to zero or sanitize them to prevent them from being used as a means to influence speculative execution (where expedient, and likely coming in 6.2). This has at most a minor performance boost, but it seems to be a definite security and maintainability gain, and best of all the new wrappers work on all PowerPC and Power ISA CPUs except the IBM Cell.
Expect to see it soon in Fedora and other leading-edge builds, and trickling down to other distros near you (full change list).
RHEL 9.1, AlmaLinux 9.1 and (eventually) RockyLinux
Red Hat Enterprise Linux 9.1 is out for those of you who pay money, but for those of you who don't, the new-CentOS-no-stream versions are coming up quick with AlmaLinux 9.1 now available for ppc64le and its other, less interesting architectures. As is typical for a point release RHEL 9.1 is largely a maintenance update, though it also includes support for Keylime, Red Hat's remote boot attestation solution. Keylime needs a TPM, though, which doesn't come with Raptor units as sold; there's nothing on the TPM headers on my three systems here. AlmaLinux's release notes are almost identical to RHEL's, including Keylime, and you can get ISOs from their mirrors or upgrade in place. RHEL customers will know their usual drill.
What about RockyLinux, the other white meat? 9.1 is still in beta as of this writing but is "passing ... to our testing team" for signoff. When ready, it should be ready for upgrades in place or new ISOs from the usual location.
As for CloudLinux, it seems to now be derived from AlmaLinux, so you might as well just use that.
Firefox 107 on POWER
Firefox 107 is out, a modest update, though there are some developer-facing changes. As before linking still requires Dan Horák's patch from bug 1775202 or the browser won't link on 64-bit Power ISA (alternatively put --disable-webrtc in your .mozconfig if you don't need WebRTC). Otherwise the build works with the .mozconfigs from Firefox 105 and the PGO-LTO patch from Firefox 101.
Fedora 37
Fedora 37 is out today. As I always say, it's usually one of the first mainstream distros to incorporate new changes and was one of the earliest distros to support POWER9 at all, so you should care about it because bugs and problems show up there first (if you don't like how the bleeding edge cuts your skin, try AlmaLinux or RockyLinux instead, which aim to occupy the niche old pre-Stream CentOS did). Chief amongst its changes is the new GNOME 43, which really does seem to have much better performance on OpenPOWER than previous releases (a big problem for the last couple) along with revised settings, toolkits and even more libadwaita-all-the-things, which means even fewer apps will respect your GTK theme. This also means Pantheon is no longer supported due to incompatibilities, so I guess I won't bother trying it again (admittedly it was definitely buggy even with GNOME the 42nd).
On the OpenPOWER side, the dust has largely settled from the 128-bit long double update, which was necessary pain and has translated into better package availability with the vast majority of regressions having been corrected. The kernel is up to 5.19, though 6.1 may arrive soon with possible good news for graphics card support on our systems. Some of us are tracking a problem with SATA PCIe cards, including ones Raptor ships as build-to-order options, using the Marvell 88SE9215/9235 SATA controllers (the earlier generation 88SE9128 seems unaffected), and I'll verify if this is still the case in the new kernel. The regression happened definitely by 5.15. gcc is up to 12.2.1.
Unfortunately, OpenPOWER is still in the AltArch penalty box (but along with aarch64 and s390x, so at least we have company), though that's better than ARMv7 (a.k.a. arm32, armhfp) which is no longer supported at all. I usually give it a week or two for any straggler packages to catch up and then I'll do our usual mini-review (here was the abbreviated one for F36). Note that I only test GNOME on my Blackbird; my daily driver Talos II is now KDE Plasma and not looking back. Should have made the jump ages ago.