Latest Posts

Firefox 83 on POWER

LTO-PGO is still working great in Firefox 83, which expands in-browser PDF support, adds additional features to Picture-in-Picture (which is still one of my favourite tools in Firefox) and some miscellany developer changes. The exact same process, configs and patches to build a fully link-time and profile-guided optimized build work that were used in Firefox 82.

Dan Horák has filed three bugs (1679271, 1679272 and 1679273) for build failures related to the internal profiler, which is still not supported on ppc64, ppc64le or s390x (or, for that matter, 32-bit PowerPC). These targetted fixes should be landing well before release, but perhaps we should be thinking about how to get it working on OpenPOWER rather than having to play emergency games of whack-a-mole whenever the build blows up.

Guix port to OpenPOWER

Since we already have FSF RYF certification for the Talos II and T2 Lite, why not run an FSF package manager on it too? Tobias Platen has announced an official branch for a port of Guix to ppc64le, based on the existing 32-bit PowerPC port of Guix that can already apparently run on big-endian ppc64, harking back to his initial work in 2019. The port is a bigger effort than it might naïvely appear, though: an updated gcc and glibc is required (and possibly a different bootstrap gcc as well), plus potential 64-bit fixes to Guile and reworking and updates of the heavily x86-centric GNU Mes, the combination Scheme interpreter and C compiler, to bootstrap the full GNU Guix System. If you're interested in finding out more, watch Tobias' video.

Vikings' first order with Raptor is go

Quick news bite: Vikings has their first order in with Raptor and will go live with a selection of OpenPOWER systems in their on-line store as soon as they arrive. We don't know yet how many units they'll stock, but the fact they'll have some on the other side of the Atlantic is good news for current and potential European (and, hey, probably other) customers for whom shipping and exchange rates to and from the United States could be prohibitive. Given that Raptor still lists the Blackbird as backordered, it's most likely the initial selection will be T2 and T2 Lites, but as someone very happy with the T2 he's typing on, a Talos II is still a heck of a computer. We'll be sure to advise more about what's available and options for RMAs and support as information is revealed.

Fedora 33 mini-review on the Blackbird and Talos II

To avoid watching American election returns, it's time to report back on our traditional mini-review for the newest release of Fedora, F33. If you run it yourself hopefully this will help your upgrade go more smoothly, and even if you don't you should still care about it because bugs in packages and platforms usually pop up in Fedora's cutting edge first (after all, F28 was one of the first out-of-the-box distributions to even run on POWER9). Now that F33 has hit the release channel, F31 will become EOL in less than a month.

We test it on both the Blackbird and Talos II, for which T2 Lite owners will have a similar experience. However, one important configuration change for this review compared to my previous go-around for F32 is that I'm no longer running gdm on the Blackbird either (I've never run it on the T2). This was largely an accident of a F32 reinstall I did, where I installed the server variant and converted it to Fedora Workstation, same as I had originally done for my T2 back with F28. In this setup the system comes up in a text boot, you log in that way, and then manually startx or dbus-run-session gnome-session (with XDG_SESSION_TYPE=wayland or as appropriate) to launch GNOME. Besides speeding startup a bit, you avoid pitfalls with a graphical start and can much more easily recover without having to do an emergency boot into the installer. This and future reviews of Fedora will be done in this configuration which just eliminates a whole class of issues I used to have on the Blackbird in particular.

As before, the general upgrade steps are

sudo dnf upgrade --refresh # upgrade DNF
sudo dnf install dnf-plugin-system-upgrade # install upgrade plugin if not already done
sudo dnf system-upgrade download --refresh --releasever=33 # download F33 packages
sudo dnf system-upgrade reboot # reboot into upgrader

When the system reboots, manually select the kernel directly from Petitboot to get a more verbose boot rather than just waiting for it to automatically start. This let me watch the install in text mode for a change. If you don't do this, your system may go to a black screen; pick another VTY with CTRL-ALT-F2 or something, log in as root and periodically issue dnf system-upgrade log --number=-1 to watch the hot hot action.

The Blackbird is my "early warning" system to catch bad updates before I tank this daily driver T2. However, perhaps because it has a vanilla install of F32 on it, it updated without any problems whatsoever, and all applications that I usually use on it (Firefox, LibreOffice, etc.) ran without issues under Xorg in 1920x1080 with the usual manually specified xorg.conf. I didn't notice much performance improvement or change, but nothing seemed to regress particularly either.

I also usually do a token test of Wayland on the Blackbird as well, which because I run it "stripped" with no GPU and with only the BMC as a framebuffer, is invariably an unusable catastrophe. But, to my surprise, not this time:

I'm a Never Wayland and I acknowledge my biases, but there has been clear improvement in its useability without a GPU, which right now is essential to run these systems "fully libre" (or at least as cheaply as possible). I suspect the LLVM update is responsible and sufficiently juices llvmpipe accordingly, but regardless of the reason the system was much more responsive and all default applications seemed to work.

What didn't, though, was the screen resolution, which remained stuck at 1024x768 because support for the Blackbird's HDMI transceiver is still not in the shipping kernel. I grabbed edid-generator and tried making an EDID out of the known-working Xorg modeline, which was ignored at bootup and dmesg said it didn't even load:

platform VGA-1: Direct firmware load for edid/bmc.bin failed with error -2

(Yes, the port name is VGA-1, despite being connected over HDMI.)

I also tried video=VGA-1:1920x1080@60e on the kernel command line and while the text boot obligingly came up in 1920x1080, when I started the GNOME session it just hung and never jumped to the graphic display, so back to Xorg. But credit where credit is due that it's getting better, whether Wayland or LLVM is responsible for the improvement.

The T2 is more complex because I have a lot more packages installed and a somewhat customized GNOME theme. Although I lost a few packages in F32, no packages were broken or needed to be backed out for F33, and the 5.0GB installation proceeded uneventfully. With fast reboots off it also properly restarts as well.

Restarting into the new installation for the first time is usually where the problems start, though, and that's what happened again this go-around. The first problem was a crapload of SELinux warnings over and over, which turned out to be another permissions clash and was eventually fixed with a restorecon -RFv /var/lib/rpm after a lot of totally family friendly cursing. The second problem is the usual GNOME extension breakage as F33 moves from 3.36 to 3.38; Dash-To-Dock again refused to update and had to be manually reinstalled from the command line as well as User Themes. However, my hacked version of Argos survived without any changes. The drag-to-reorder feature new in 3.38 mostly works as advertised, though I'm used to apps moving to close the gap and that didn't seem to happen, but I do like the changes to Screenshot.

On the T2's BTO WX7100 GPU, GNOME 3.38 under Xorg was nice and snappy as always. I didn't notice really any performance improvement, but it seemed no worse either. Wayland did improve in this iteration and the games it used not to launch now seem to start properly under Xwayland, but it seemed a little less sprightly this time. Likewise, I'm highly reliant on appmodmap for my muscle memory which won't work with any current Wayland compositor, and while GNOME's new ability under Wayland to run multiple displays at different refresh rates is a nice new feature, I don't need it for my two displays. So back to Xorg. (If maintaining Xorg is such a paper cut with hydrochloric acid on it, then why don't we use Wayland for the low-level display stuff and just run everything in Xwayland on top of it? Why must we throw the baby out with the bathwater? I like all the hacks X lets me do.)

Anyway, F33 was a largely uneventful release and I consider that a positive: while the normal little polish issues are still there it didn't seem to require pulling more teeth than usual and overall has been working well for the last couple days. What I really want, though, is for 128-bit long doubles to finally arrive and I'd really like to see a push for this in F34. Me personally I'm tired of having to hack MAME all the time just to play the same games my G5 can with MacMAME, but there are more practical and less first-world-problemy concerns for needing this feature as well. And it would really be a boon to the platform if we weren't still stuck in the Alternative Architectures penalty box every time too.

Updates: Fedora 33, FreeBSD 12.2, Ubuntu 20.10

Hot on the heels of Ubuntu 20.10 and 20.04.1 LTS (download the server flavour, and convert it to desktop if you like) comes Fedora 33. Ubuntu 20.10 upgrades to kernel 5.8, GNOME 3.38, QEMU 5 and OpenStack Victoria with an installer fix for OpenPOWER; Fedora 33 remains on 5.8 (5.9 likely to follow) but also includes GNOME 3.38, glibc 2.32 and LLVM 11, and also defaults to btrfs on Workstation (watch out if you change to a 4K page size; Fedora uses 64K pages and filesystems generated on one are not currently compatible with the other). As previously mentioned Fedora is important to me personally because it's what I run on my own T2 and Blackbird, so once the packages and late breaking changes settle down I will do a mini-review (as I did for F32), but the change I've been waiting for (128-bit long doubles) is still not in F33 as they wait on glibc changes (maybe glibc 2.33).

And if you like your OpenPOWER systems but don't like Linux, FreeBSD 12.2 is out as well, with multiple security, bugfix and functionality upgrades for a wide variety of PowerPC and OpenPOWER-based systems. Big-endian is well-tested and little-endian is coming along (and snapshots should finally be in -CURRENT by the time you read this).

Firefox 82 on POWER goes PGO

You'll have noticed this post is rather tardy, since Firefox 82 has been out for the better part of a week, but I wanted to really drill down on a couple variables in our Firefox build configuration for OpenPOWER and also see if it was time to blow away a few persistent assumptions.

But let's not bury the lede here: after several days of screaming, ranting and scaring the cat with various failures, this blog post is finally being typed in a fully profile-guided and link-time optimized Firefox 82 tuned for POWER9 little-endian. Although it multiplies compile time by nearly a factor of 3 and the build process intermittently can consume a terrifying amount of memory, the PGO-LTO build is roughly 25% faster than the LTO-only build, which was already 4% faster than the "baseline" -O3 -mcpu=power9 build. That's worth an 84-minute coffee break! (-j24 on a dual-8 Talos II [64 threads], 64GB RAM.)

The problem with PGO and gcc (at least gcc 10, anyway) is that all the .gcda files end up in the same directory as the built objects in an instrumented build. The build system, which is now heavily clang-centric (despite the docs, gcc is clearly Tier 2, since this and other things don't work), does not know how to handle or transfer the resulting profile data and bombs after running the test load. We don't build with clang because in previous attempts it never managed to fully build the browser on ppc64le and I'm sceptical of its code quality on this platform anyway, but since I wanted to verify against a presumably working configuration I did try a clang build first to see if anything had changed. It breaks fairly early now, interestingly while compiling a Rust component:

4:33.00 error: /home/censored/src/mozilla-release/obj-powerpc64le-unknown-linux-gnu/release/deps/ undefined symbol: __muloti4
4:33.00 --> /home/censored/src/mozilla-release/third_party/rust/phf_macros/src/
4:33.00 227 | #[::proc_macro_hack::proc_macro_hack]
4:33.00    |      ^^^^^^^^^^^^^^^
4:33.00 error: aborting due to previous error
4:33.00 error: could not compile `phf_macros`.

So there's that. I'm not very proficient in Rust so I didn't do much more diagnosis at this point. Back to the hippo gcc.

What's needed is to hack the build system to copy the .gcda files generated during profiling out of instrumented/ into the regular build tree for the actual (second) build phase, which is essentially the solution proposed in bug 1601903 except without any explanation as to how you actually do it. The PGO driver is fortunately in a standalone Python script, so I decided to simply hijack that. At the end is code to coalesce the .profraw files from a successful instrumented clang build, which shouldn't be running anyway if the compiler is gcc, so I threw in a couple lines to terminate instead after it runs this shell script:

#!/bin/csh -f

set where=/tmp/mozgcda.tar

# all on one line yo
cd /home/censored/src/mozilla-release/obj-powerpc64le-unknown-linux-gnu/instrumented || exit
tar cvf $where `find . -name '*.gcda' -print`
cd ..
tar xvf $where
rm -f $where

This repopulates the .gcda files in the right place before we rebuild with the profile data, but because of this subterfuge, gcc thinks the generated profile is not consistent with the source and spams an incredible amount of complaint messages ... which made it difficult to spot the internal compiler error that the profile-guided rebuild triggered. This required another rebuild with some tweaks to turn that off and some other irrelevant warnings (I'll probably upstream at least one of these changes) so I could determine where the ICE was in the scrollback. Fortunately, it was in a test binary, so I just commented it out in the and it finally stuck. And so far, it's working impressively well. This may well be the fastest the browser can get while still lacking a JIT.

After all that, it's almost an anticlimax to mention that --disable-release is no longer needed in the build configs. You can put it in the Debug configuration if you want, but I now use --enable-release in optimized builds and it seems to work fine.

If you want to try compiling a PGO-LTO build yourself, here is a gist with the changes I made (they are all trivial). Save the shell script above as gccpgostub.csh in ~/src/mozilla-release and/or adjust paths as necessary, and make sure it is chmodded +x. Yes, there is no doubt a more elegant way to do this in Python itself but I hate Python and I was just trying to get it to work. Note that PGO builds can be exceptionally toolchain-dependent (and ICEs more so); while TestUtf8 was what triggered the ICE on my system (Fedora 32, gcc 10.2.1), it is entirely possible it will halt somewhere else in yours, and the PGO command line options may not work the same in earlier versions of the compiler.

Without further ado, the current .mozconfigs, starting with Optimized. Add ac_add_options MOZ_PGO=1 to enable PGO once you have patched your tree and deposited the script.

export CC=/usr/bin/gcc
export CXX=/usr/bin/g++

mk_add_options MOZ_MAKE_FLAGS="-j24"
ac_add_options --enable-application=browser
ac_add_options --enable-optimize="-O3 -mcpu=power9"
ac_add_options --enable-release
ac_add_options --enable-linker=bfd
ac_add_options --enable-lto=full

# this is implied by enable-release but left in to be explicit


export CC=/usr/bin/gcc
export CXX=/usr/bin/g++

mk_add_options MOZ_MAKE_FLAGS="-j24"
ac_add_options --enable-application=browser
ac_add_options --enable-optimize="-Og -mcpu=power9"
ac_add_options --enable-debug
ac_add_options --enable-linker=bfd