Posts
ICYMI: Hugo Landau explains how the Broadcom BCM5719 was freed
Firefox 122 on POWER
Right now during our relocation I'm not always in the same ZIP code as my T2, but we've still got to keep it up to date. To that end Firefox 122 is out with some UI improvements and new Web platform support.
A number of changes have occurred between Fx121 and Fx122 which improve our situation in OpenPOWER world, most notably being we no longer need to drag our WebRTC build changes around (and/or you can remove --disable-webrtc in your .mozconfig). However, on Fedora I needed to add ac_add_options --with-libclang-path=/usr/lib64 to my .mozconfigs (or ./mach build would fail during configuration because Rust bindgen could not find libclang.so), and I also needed to effectively fix bug 1865993 to get PGO builds to work again on Python 3.12, which Fedora 39 ships with. You may not need to do either of these things depending on your distro. There are separate weird glitches due to certain other components being deprecated in Python 3.12 that do not otherwise affect the build.
To that end, here is the updated PGO-LTO patch I'm using, as well as the current .mozconfigs:
Optimized
export CC=/usr/bin/gcc export CXX=/usr/bin/g++ mk_add_options MOZ_MAKE_FLAGS="-j24" # or as you like ac_add_options --enable-application=browser ac_add_options --enable-optimize="-O3 -mcpu=power9 -fpermissive" ac_add_options --enable-release ac_add_options --enable-linker=bfd ac_add_options --enable-lto=full ac_add_options --without-wasm-sandboxed-libraries ac_add_options --with-libclang-path=/usr/lib64 ac_add_options MOZ_PGO=1 export GN=/home/censored/bin/gn # if you haz export RUSTC_OPT_LEVEL=2
Debug
export CC=/usr/bin/gcc export CXX=/usr/bin/g++ mk_add_options MOZ_MAKE_FLAGS="-j24" # or as you like ac_add_options --enable-application=browser ac_add_options --enable-optimize="-Og -mcpu=power9 -fpermissive -DXXH_NO_INLINE_HINTS=1" ac_add_options --enable-debug ac_add_options --enable-linker=bfd ac_add_options --without-wasm-sandboxed-libraries ac_add_options --with-libclang-path=/usr/lib64 export GN=/home/censored/bin/gn # if you haz
Fedora 39 mini-review on the Blackbird and Talos II (and other woes)
Also, as usual, recall both my T2 and Blackbird are configured to come up in a text boot instead of gdm and I start KDE manually from there. I still test GNOME on both systems, but my primary desktop environment has been KDE Plasma for several versions now, and I always recommend a non-graphical boot as a recovery mechanism in case your graphics card gets whacked by something or other. On Fedora this is easily done by ensuring the symlink /etc/systemd/system/default.target points to /lib/systemd/system/multi-user.target.
Unfortunately, dnf kernel updates still! don't seem to properly update grub's config (basically bug 1921479, showing messages like 0ed84c0-p94177c1: integer expression expected during the process), so the process remains largely unchanged from F38:
dnf upgrade --refresh # upgrade prior system and DNF
grub2-mkconfig -o /boot/grub2/grub.cfg # force grub to update
dnf install dnf-plugin-system-upgrade # install upgrade plugin if not already done
dnf system-upgrade download --refresh --releasever=39 # download F39 packages
dnf system-upgrade reboot # reboot into upgrader
I do the Blackbird first as a check, since it can afford to be down waiting for updates, but I can't have that happening with the Talos II since it's my primary workstation. Unfortunately, while the packages downloaded okay, the actual upgrade process didn't do anything. After a couple failed attempts where it rebooted back into 38 seemingly unchanged, I watched it like a hawk and observed the following:
I checked the CMOS battery coin cell and got a full 3.0 volts, and couldn't find anything otherwise wrong with the installer itself. The answer was labouriously going through the logs and finding that the system time was about two years in the past, confirmed in the Petitboot shell. This gets fixed in a normal boot by chrony synching up with NTP, but that apparently doesn't happen with the installer. Worse, it looked like everything had somehow gotten scrambled in the Blackbird's BMC because I couldn't log on with the admin password and fix the time (either from the SSH server or the web interface). Time to crack the case and connect to the BMC's serial port.
Here's what you might see in your terminal program. Make sure the terminal program is running and the serial port is connected before you apply power to the system to start up the BMC, or you won't be early enough to talk to U-Boot.
DRAM Init-V12-DDR4
0abc1-4Gb-Done
Read margin-DL:0.3725/DH:0.3803 CK (min:0.30)
U-Boot 2016.07 (Feb 19 2020 - 11:51:39 +0000)
Watchdog enabled
DRAM: 496 MiB
Flash: 32 MiB
In: serial
Out: serial
Err: serial
Net: aspeednic#0
Hit any key to stop autoboot: 0
ast# setenv bootargs console=ttyS4,115200n8 root=/dev/ram overlay-filesystem-in-ram rw
ast# boot
## Loading kernel from FIT Image at 20080000 ...
Using 'conf@aspeed-bmc-opp-blackbird.dtb' configuration
Trying 'kernel@1' kernel subimage
Description: Linux kernel
Type: Kernel Image
Compression: uncompressed
Data Start: 0x20080128
Data Size: 2656176 Bytes = 2.5 MiB
Architecture: ARM
OS: Linux
Load Address: 0x80001000
Entry Point: 0x80001000
Hash algo: sha1
Hash value: 1815ece74a2e27241c471b9ed87885071dd9e143
Verifying Hash Integrity ... sha1+ OK
## Loading ramdisk from FIT Image at 20080000 ...
Using 'conf@aspeed-bmc-opp-blackbird.dtb' configuration
Trying 'ramdisk@1' ramdisk subimage
Description: obmc-phosphor-initramfs
Type: RAMDisk Image
Compression: lzma compressed
Data Start: 0x20310f88
Data Size: 1583941 Bytes = 1.5 MiB
Architecture: ARM
OS: Linux
Load Address: unavailable
Entry Point: unavailable
Hash algo: sha1
Hash value: 55e0853d6ad703d5ea225837d85223a73f7cf3a4
Verifying Hash Integrity ... sha1
DRAM Init-V12-DDR4
0abc1-4Gb-Done
Read margin-DL:0.3745/DH:0.3784 CK (min:0.30)
U-Boot 2016.07 (Feb 19 2020 - 11:51:39 +0000)
Watchdog enabled
DRAM: 496 MiB
Flash: 32 MiB
In: serial
Out: serial
Err: serial
Net: aspeednic#0
Hit any key to stop autoboot: 0
ast# setenv bootargs console=ttyS4,115200n8 root=/dev/ram overlay-filesystem-in-ram rw
ast# boot
## Loading kernel from FIT Image at 20080000 ...
Using 'conf@aspeed-bmc-opp-blackbird.dtb' configuration
Trying 'kernel@1' kernel subimage
Description: Linux kernel
Type: Kernel Image
Compression: uncompressed
Data Start: 0x20080128
Data Size: 2656176 Bytes = 2.5 MiB
Architecture: ARM
OS: Linux
Load Address: 0x80001000
Entry Point: 0x80001000
Hash algo: sha1
Hash value: 1815ece74a2e27241c471b9ed87885071dd9e143
Verifying Hash Integrity ... sha1+ OK
## Loading ramdisk from FIT Image at 20080000 ...
Using 'conf@aspeed-bmc-opp-blackbird.dtb' configuration
Trying 'ramdisk@1' ramdisk subimage
Description: obmc-phosphor-initramfs
Type: RAMDisk Image
Compression: lzma compressed
Data Start: 0x20310f88
Data Size: 1583941 Bytes = 1.5 MiB
Architecture: ARM
OS: Linux
Load Address: unavailable
Entry Point: unavailable
Hash algo: sha1
Hash value: 55e0853d6ad703d5ea225837d85223a73f7cf3a4
Verifying Hash Integrity ... sha1+ OK
## Loading fdt from FIT Image at 20080000 ...
Using 'conf@aspeed-bmc-opp-blackbird.dtb' configuration
Trying 'fdt@aspeed-bmc-opp-blackbird.dtb' fdt subimage
Description: Flattened Device Tree blob
Type: Flat Device Tree
Compression: uncompressed
Data Start: 0x203089e8
Data Size: 34013 Bytes = 33.2 KiB
Architecture: ARM
Hash algo: sha1
Hash value: f53d8ad3a7c573a4903f910fd124507c73f0bbfb
Verifying Hash Integrity ... sha1+ OK
Booting using the fdt blob at 0x203089e8
Loading Kernel Image ... OK
Loading Ramdisk to 9ea16000, end 9eb98b45 ... OK
Loading Device Tree to 9ea0a000, end 9ea154dc ... OK
Starting kernel ...
[...]
Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) 0.1.0 blackbird ttyS4
blackbird login: root
Password: 0penBmc
root@blackbird:~# flash_eraseall /dev/mtd/rwfs
Erasing 64 Kibyte @ 400000 - 100% complete.
root@blackbird:~# reboot
Notice that it fell back to U-Boot once before actually going into the OpenBMC kernel. You need to set the bootargs both times (because the watchdog is aggressive and will reboot after even a brief period of inactivity, I recommand cutting and pasting that setenv command instead of typing it). Once that's done, the default 0penBmc password will work, and we can reset the persistent storage. You may need to powercycle the BMC after this too as the manual reboot may not be enough.
With that corrected, I was able to log into the web interface and set the time manually, though I also ensured the time owner was Host (i.e., not the BMC) so that hopefully the system can deal with this itself on startup again. Petitboot confirmed the time was correct and the install succeeded. On my Blackbird I got the usual graphical progress bar on the ASPEED BMC framebuffer; on my T2 with a BTO AMD WX7100, as long as you ensure the kernel is manually selected through Petitboot on startup, you'll still get to see the install log live as text. If you forget to manually select the kernel and the system comes up to an apparently black screen, you can either monitor on the serial port, or from a connected system viewing the serial console over the BMC's web server, or by logging into another VTY with CTRL-ALT-F2 or as appropriate as root and periodically issuing dnf system-upgrade log --number=-1 to watch log updates.
The update did not cause a stuck XFS log entry on the Blackbird, but after the reboot I still had to do one more grub2-mkconfig -o /boot/grub2/grub.cfg and a restart to ensure the right kernel and version were being used. Currently the kernel version as of this writing is 6.6.7.
Fedora 39 is apparently the last hurrah for the X11 session of KDE Plasma, largely because the Fedora KDE SIG doesn't want to deal with it. While Plasma 6 still has a legacy X11 session, KDE as provided in the Fedora spin won't have it — Wayland Wasteland or bust. (In particular, this will obsolete both kwin-x11, presumably including /usr/bin/kwin_x11 and /usr/bin/startplasma-x11, and plasma-workspace-wayland.) Naturally this directly affects me personally, so let's start with the system with the worst Wayland support, our stripped-down Blackbird. All it has is the ASPEED BMC framebuffer connected over HDMI.
Both GNOME 45 (GNOME's own X11 session support removal is further behind, and doesn't appear imminent) and Plasma 5 in F39 are still limited to 1024x768 with the on-board HDMI. This is a terrible limitation that yet again remains unaddressed, especially for those people trying to run their systems completely blobless, and also affects Arctic Tern which uses the same IT66121FN HDMI transceiver PHY. While performance remains stably improved over the horrid morass it used to be, further fixes appear to be stuck on a plateau.
Next was the Talos II, which has a custom KDE theme and a lot more packages. Its clock is fine, so that wasn't the problem, but the install left a stuck log entry on the XFS root again and Petitboot duly puked. I haven't (quite, anyway) gotten to the point of replacing the XFS root with ext4, but I now have a script to automatically do the volume group gyrations and XFS repair on the Blackbird instead of typing in commands, and for future upgrades I'll be doing ipmitool chassis bootdev safe (at the suggestion of a commenter) before running the installer so that if I can't scan for devices, I can at least restart, immediately jump into the Petitboot shell and do repairs if necessary. Note that turning off the automatic device scan can impair autobooting, such as me remotely bringing up the system from the BMC after a power failure, so I've set it back to normal with ipmitool chassis bootdev none after the upgrade and repair were successful.
So far F39 is running fine on the T2 as well, with only a couple old compat packages that didn't make the jump (meanwhile, for those of you hanging on to F37, that support is already over). It's F40 where the problems are expected to start for me with KDE, which as of this writing is scheduled for mid-April 2024. As usual no one cares, least of all the Wayland community, which has simply fallen back on strong-arm tactics to drive adoption and functionality be damned. I'll have to evaluate my choices then based on my app mix and how well Wayland-only KDE can handle them, and it won't be just OpenPOWER users like me in that boat.
Firefox 121
We're still in the process of finding a place to live at the new job and alternating back and forth to the tune of 400 miles each way. Still, this weekend I updated Firefox on the Talos II to Fx121, which fortunately also builds fine with the WebRTC patch from Fx116 (or --disable-webrtc in your .mozconfig), the PGO-LTO patch from Fx117 and the .mozconfigs from Firefox 105.
Unfortunately I had intended to also sit down with the Blackbird and do a test upgrade to Fedora 39 before doing so on the Talos II, but the Blackbird BMC's persistent storage seems to be hosed, the BMC password is whacked and the clock is permanently stuck in June 2022, causing signature checks on the upgrade to fail (even with --nopgpcheck). This is going to require a little work with a serial console and I just didn't have enough spare cycles over the weekend, so I'll do that over the Christmas holiday when we have a few free days. Hopefully I can also get some more work done on upstreaming the JIT at the same time.
Fedora 39
Fedora 39 is out, the Linux distro I use personally on my Talos II and Blackbird systems. Since I'm doing a lot of remote work with the new $DAYJOB it might be a bit before I can sit down with it but there will be the usual mini-review (here's what I had to say about Fedora 38). This release is based on kernel 6.5 and GNOME 45, with LLVM 17, Perl 5.38, glibc 2.38 and gcc 13.2. As usual, Fedora 37 will EOL one month after this release.
However, F39 isn't really the big news, and probably won't be a very exciting (or bumpy) release. Instead, the bigger and possibly more obnoxious change will occur with Fedora 40 when the Plasma spin (which I use instead of GNOME) is expected to completely drop X11 support, and the same is likely coming for GNOME 46 which will occur in the same version. It will be interesting, to say the least, to see how this affects people running OpenPOWER systems without GPUs to be completely blob-less. I've not been terribly happy with Wayland on the Blackbird's ASpeed BMC framebuffer and I haven't seen anything to indicate that its deficiencies have improved. Wonder how well it would work on the X1 ...
Firefox 119 and the next ppc64le JITeration
Although I've been a bit preoccupied lately with a new $DAYJOB which has required me to be remote, let's not bury the (larger) lede: the first iteration of the Firefox/SpiderMonkey ppc64le JIT is being evaluated by Mozilla to determine if the changes are acceptable. Please don't spam the Bugzilla entry with drive-by comments, but if you'd like to observe its progress, you can follow along in bug 1860412.
That doesn't mean, of course, that you can't try it yourself. The current JIT state for 115ESR now supports baseline Wasm as well as full optimizing Ion compilation for regular JS, and passes the complete test suite on Linux. It does not yet support POWER8, nor the optimizing Wasm compiler, so some applications will not run as well as they should (and obnoxiously asm.js code is not JITted at all in this configuration because it relies on the optimizing Wasm compiler, despite the fact it's regular JavaScript — for TenFourFox, which didn't support Wasm otherwise, I hacked JavaScript to simply compile asm.js with regular Ion). However, I do intend to add support for optimized Wasm and later POWER8, and with that said, the testers I've been seeding this with see good improvements for the vast majority of sites and no additional reproducible crashes so far.
If you'd like to give it a shot as well, then apply the new patches numerically and build as we did for Firefox 115, using the .mozconfigs from Firefox 105. For your convenience the JIT patch set already includes the PGO-LTO and WebRTC fixes for that version. If you don't want to roll your own browser (though I highly recommend it), then Dan Horák has you covered with a copr build for Fedora users. However, I don't intend to backport POWER8 or optimizing Wasm support to 115ESR; future work will be done on trunk, assuming Mozilla is fine with the existing changes. Do not post bugs with the ESR JIT to bug 1860412.
Apart from that, the other Firefox news is anticlimatic: Firefox 119 (I did a test build of Fx118 but hadn't tested enough to post about it) builds fine with the WebRTC patch from Fx116 (or --disable-webrtc in your .mozconfig), the PGO-LTO patch from Fx117 and the .mozconfigs from Firefox 105.