Latest Posts

Firefox 89 on POWER

Firefox 89 was released last week with much fanfare over its new interface, though being the curmudgeon I am I'm less enamoured of it. I like the improvements to menus and doorhangers but I'm a big user of compact tabs, which were deprecated, and even with compact mode surreptitously enabled the tab bar is still about a third or so bigger than Firefox 88 (see screenshot). There do seem to be some other performance improvements, though, plus the usual more lower-level changes and WebRender is now on by default for all Linux configurations, including for you fools out there trying to run Nvidia GPUs.

The chief problem is that Fx89 may not compile correctly with certain versions of gcc 11 (see bugs 1710235 and 1713968). For Fedora users if you aren't on 11.1.1-3 (the current version as of this writing) you won't be able to compile the browser at all, and you may not be able to compile it fully even then without putting a # pragma GCC diagnostic ignored "-Wnonnull" at the top of js/src/builtin/streams/PipeToState.cpp (I still can't; see bug 1713968). gcc 10 is unaffected. I used the same .mozconfigs and PGO-LTO optimization patches as we used for Firefox 88. With those changes the browser runs well.

While waiting for the updated gcc I decided to see if clang/clang++ could now build the browser completely on ppc64le (it couldn't before), even though gcc remains my preferred compiler as it generates higher performance objects. The answer is now it can and this time it did, merely by substituting clang for gcc in the .mozconfig, but even using the bfd linker it makes a defective Firefox that freezes or crashes outright on startup; it could not proceed to the second phase of PGO-LTO and the build system aborted with an opaque error -139. So much for that. For the time being I think I'd rather spend my free cycles on the OpenPOWER JavaScript JIT than figuring out why clang still sucks at this.

Some of you will also have noticed the Mac-style pulldown menus in the screenshot, even though this Talos II is running Fedora 34. This comes from firefox-appmenu, which since I build from source is trivial to patch in, and the Fildem global menu GNOME extension (additional tips) paired with my own custom gnome-shell theme. I don't relish adding another GNOME extension that Fedora 35 is certain to break, but it's kind of nice to engage my Mac mouse-le memory and it also gives me a little extra vertical room. You'll notice the window also lacks client-side decorations since I can just close the window with key combinations; this gives me a little extra horizontal tab room too. If you want that, don't apply this particular patch from the firefox-appmenu series and just use the other two .patches.

Progress on the OpenPOWER SpiderMonkey JIT


% gdb --args obj/dist/bin/js --no-baseline --no-ion --no-native-regexp --blinterp-eager -e 'print("hello world")'
GNU gdb (GDB) Fedora 10.1-14.fc34
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "ppc64le-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Find the GDB manual and other documentation resources online at:

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from obj/dist/bin/js...
(gdb) run
Starting program: obj/dist/bin/js --no-baseline --no-ion --no-native-regexp --blinterp-eager -e print\(\"hello\ world\"\)
warning: Expected absolute pathname for libpthread in the inferior, but got .gnu_debugdata for /lib64/
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
[New LWP 2797069]
[LWP 2797069 exited]
[New LWP 2797070]
[New LWP 2797071]
[New LWP 2797072]
[New LWP 2797073]
[New LWP 2797074]
[New LWP 2797075]
[New LWP 2797076]
[New LWP 2797077]
hello world
[LWP 2797072 exited]
[LWP 2797070 exited]
[LWP 2797074 exited]
[LWP 2797077 exited]
[LWP 2797073 exited]
[LWP 2797071 exited]
[LWP 2797076 exited]
[LWP 2797075 exited]
[Inferior 1 (process 2797041) exited normally]

This may not look like much, but it demonstrates that the current version of the OpenPOWER JavaScript JIT for Firefox can emit machine language instructions correctly (mostly — still more codegen bugs to shake out), handles the instruction cache correctly, handles ABI-compliant calls into the SpiderMonkey VM correctly (the IonMonkey JIT is not ABI-compliant except at those edges), and enters and exits routines without making a mess of the stack. Much of the code originates from TenFourFox's "IonPower" 32-bit PowerPC JIT, though obviously greatly expanded, and there is still ongoing work to make sure it is properly 64-bit aware and takes advantage of instructions available in later versions of the Power ISA. (No more spills to the stack to convert floating point, for example. Yay for VSX!)

Although it is only the lowest level of the JIT, what Mozilla calls the Baseline Interpreter, there is substantial code in common between the Baseline Interpreter and the second-stage Baseline Compiler. Because it has much less overhead compared to Baseline Compiler and to the full-fledged Ion JIT, the Baseline Interpreter can significantly improve page loads all by itself. In fact, my next step might be to get regular expressions and the OpenPOWER Baseline Interpreter to pass the test suite and then drag that into a current version of Firefox for continued work so that it can get banged on for reliability and improve performance for those people who want to build it (analogous to how we got PPCBC running first before full-fledged IonPower in TenFourFox). Eventually full Ion JIT and Wasm support should follow, though those both use rather different codepaths apart from the fundamental portions of the backend which still need to be shaped.

A big shout-out goes to Justin Hibbits, who took TenFourFox's code and merged it with the work I had initially done on JitPower way back in the Firefox 62 days but was never able to finish. With him having done most of the grunt work, I was able to get it to compile and then started attacking the various bugs in it.

Want to contribute? It's on Github. Tracing down bugs is labour-intensive, and involves a lot of emitting trap instructions and single-stepping in the debugger, but when you see those small steps add up into meaningful fixes (man, it was great to see those two words appear) it's really rewarding. I'm happy to give tips to anyone who wants to participate. Once it can pass the test suite at some JIT level, it will be time to forward-port it and if we can get our skates on it might even be possible to upstream it into the next Firefox ESR.

For better or worse, the Web is a runtime. Let's get OpenPOWER workstations running it better.

Tonight's game on OpenPOWER: Blake Stone Aliens of Gold

Everything is awful, so now that we've rescued a Blackbird let's go shoot more aliens. One of the more entertaining games based on id's Wolfenstein 3D engine was Apogee's Blake Stone: Aliens of Gold from 1993, developed by yet another set of apparent refugees from Softdisk, but that's another story for another day. It kept the basic formula but added subtle lighting effects and ceiling and floor textures, along with more varied environments, lots of (cartoony but fun to shoot) monsters, and a very helpful automap.

Ordinarily this wouldn't be worth a mention; that's what we have DOSBox for (see our article on adding an OpenPOWER JIT to DOSBox). Despite the fact that DOSBox does support this game, however, I do actually own a retail copy of Blake Stone from back in the day and it runs like dried snot even with a JIT.

Fortunately, the source code to Blake Stone was released back in the day as well after it was long believed to be lost, and an excellent SDL/OpenGL port called BStone is available which adds many graphical improvements, mouse look (well, side to side, anyway), and 16:9 modes as demonstrated in the screenshot. It also supports the IMHO inferior sequel, Planet Strike.

To start saving mankind, you can play the shareware version, but it's more fun to play with a retail copy (mine is the 1994 FormGen release, but the one you can still buy from Apogee will work), or extract the game assets from the DOSBox-based GOG installer. The CD or 3D Realms downloads are easiest to work with, as you can just copy the contents into a folder.

Clone the BStone Github project. You will need CMake and SDL 2.0.4 (or better) development headers and libraries. The CMake build recipe assumes that your SDL has a static libSDL2main.a library, which apparently the ones from Fedora, Slackware and possibly others don't, which may require modifying the SDL CMake component that comes with it (I had to). Then mkdir build, cd build and cmake .. to kick it off.

Once built you can start the game either from the directory where your Blake Stone files are (I have cd ~/stuff/dosbox/BSTONE && ~/src/bstone/build/src/bstone), or pass bstone the --data_dir option with a path (if it fails to detect the correct game, try passing --aog, --aog_sw or --ps). If you don't have OpenAL-capable hardware, disable OpenAL sound from the in-game configuration menu, or you may get random crashes during play. Don't shoot the informants.

Fedora 34 mini-review on the Blackbird and Talos II (it sucks)

Once again it's time to upgrade Floodgap's stable of Raptor systems to the latest release of Fedora, which is up to version 34 (see our prior review of Fedora 33). You may not necessarily run Fedora yourself, but the fact that it does run is important, because it tends to be very ahead of most distros and many problems are identified in it and fixed before moving to other less advanced ones. And boy howdy, are there problems this time. I'm going to get it over with and tl;dr myself right now: if you use GNOME as your desktop environment and you haven't upgraded yet, DON'T. F34 and in particular GNOME 40 are half-baked, and the problems don't seem specific to OpenPOWER and the hard work of folks like Dan Horák; these issues are more generalized. There is always that sense of dread over what's going to break during the update, and while I'm finally typing in Firefox on this updated Talos II, it took me hours to get everything glued back together and the desktop performance problems in particular are cramping my ability to use the system well. Fedora 33 will still be supported until a month after F35 comes out; it may be worth sticking with F33 for a couple months for the GNOME team to work on the remaining performance issues.

The problems started from the very beginning, even before actually updating. I do my updates initially on the Blackbird to shake out any major problems before doing it to my daily driver T2. As I explained previously, neither the Blackbird nor the T2 use gdm; they both boot to a text prompt, and we jump to GNOME with startx (or XDG_SESSION_TYPE=wayland dbus-run-session gnome-session if we want to explore the Wayland Wasteland). I do the upgrade at the text prompt so that there is minimal chance of interference. Our usual MO to update Fedora is, as root,

dnf upgrade --refresh # upgrade prior system and DNF
dnf install dnf-plugin-system-upgrade # install upgrade plugin if not already done
dnf system-upgrade download --refresh --releasever=34 # download F34 packages
dnf system-upgrade reboot # reboot into upgrader

If you do this with F34, however, you get a number of downgrades (unavoidable, apparently), missing groups and an instant conflict with iptables when you try to download the packages:

dnf suggests we add --best --allowerasing to deal with that. It doesn't work:
Neither does adding --skip-broken. The non-obvious solution is dnf system-upgrade download --refresh --releasever=34 --allowerasing, and just ignoring the duff package.
The Blackbird does not have a GPU; all video output is on the ASPEED BMC (using the Blackbird's HDMI port). Ordinarily I would select the new kernel from Petitboot when it restarts after the final command above to see a text log of the installation but this time we get an actual graphical install screen.
After the installation completed, the machine rebooted uneventfully and came up to the text prompt. I entered startx as usual and ...
At this point GNOME just plain hung up. There was no mouse pointer, though pressing ENTER on the keyboard triggered the button and put it back to the text prompt. Nothing unusual was in the Xorg logs, and journalctl -e showed only what seemed like a non-fatal glitch (Window manager warning: Unsupported session type). Well, maybe the time for the Wayland Wasteland was now. I did an exec bash (gnome-session doesn't properly handle using another shell, or you get weird errors like Unknown option: -l because it tries to be cute with the options to exec) and XDG_SESSION_TYPE=wayland dbus-run-session gnome-session, and Wayland does start:
However, it still doesn't support 1920x1080 on the Blackbird on-board HDMI, just 1024x768. It also seemed a little sluggish with the mouse. I exited it and tried to start gnome-session --debug --failsafe but it wouldn't initialize.

It then dawned on me that I was setting XDG_SESSION_TYPE manually for Wayland; I previously left it unset for X11. Setting XDG_SESSION_TYPE to x11 finally brought up GNOME 40 in X with a full 1080p display:

I put that into my .cshrc and that was one problem solved. The Applications drawer seemed a little slower to come up, though I have a very vanilla installation on this Blackbird on purpose and few apps are loaded, so I didn't try scrolling through the list or running lots of applications at once. (More on that in a moment.)

Just to see if anything shook out subsequently, I ran dnf upgrade again. This time the missing iptables compatibility packages came up:

That solves that mystery, so just ignore iptables during the initial download and the next time you run dnf after Fedora has been upgraded, it will clean up and install the right components. This whole sordid affair now shows up in the Release Notes.

Upgrading the Talos II is usually a much more complex undertaking anyway because I have custom GNOME themes and extensions installed on it and I always expect there will be some bustage. I don't like it, mind you, but I expect it. Armed with what I had learned from the Blackbird, I installed the packages on the T2 (some other groups also had "no match," though all of my optionally installed packages could and did upgrade) and rebooted.

Unlike the Blackbird, however, the installer still came up in a text screen as in prior upgrades when I selected that kernel from the Petitboot menu.
This machine has the BTO AMD WX7100 workstation card and does not use the ASPEED BMC framebuffer. If you don't select the kernel from the menu and just let the default go, you will get the usual black screen again, and as in prior versions you'll have to pick another VTY with CTRL-ALT-F2 or something, log in as root and periodically issue dnf system-upgrade log --number=-1 to watch.

I rebooted and started X (with XDG_SESSION_TYPE=x11), and GNOME came up, but it looked a little ... off.

If you noticed the weird pink-purple tint, you win the prize. However, my second monitor seemed to have a normal display (so did the Blackbird), and the difference is that my main display is colour-managed. When I selected the default profile, the tint went away but my colours weren't, you know, just right. I spent a few hours regenerating the profile with my Pantone huey manually with dispcal, but the same thing happened with the new profile.

The problem is the new colour transform matrix (CTM) support; the prior profile obviously worked fine in 3.38 but isn't compatible with 40. The proper way to solve this would be by letting GNOME make a new colour profile for you from the Settings app and it even allegedly supports the Pantone huey and other colourimeters. However, it has never (to my knowledge) worked properly on OpenPOWER (it crashes), so I've never been able to do this myself. Instead, my current solution is to just temporarily disable CTM with

xrandr --output DisplayPort-0 --set CTM 0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,1

(that's 0, 1, seven zeroes, 1, seven more zeroes, and 1). Adjust DisplayPort-0 to where your colour-managed display is connected. Note that every time you (re)start GNOME or its shell, it will forget this setting and you'll have to enter it again. It would be nice if the colour manager could work with OpenPOWER, but CTM should have never broken working profiles in the first place.

However, that all got solved later, because an even more pressing concern popped up first: the UI was slow as molasses. GNOME 40 defaults to the Activities overview on startup with nothing running. It takes literally several seconds to move from one page of activities/apps to the next. Several seconds. This problem is not unique to OpenPOWER, and occurs on both Wayland and Xorg, but a general fix is apparently months away.

The performance problems are not X11-specific. In fact, Wayland is even worse, because the mouse stutters even just moving it around. This is the first time Wayland is actually worse on the system with the GPU (the T2) rather than the system without one (the Blackbird), though I hardly consider this regression a positive development.

What am I doing about it? Well, what can I do about it, short of trying to fix it myself? GNOME is the default environment for Fedora Desktop, and while I could switch to KDE or Xfce (and I might!), these are serious regressions that are hitting a decent proportion of users and were even evident during the beta phase. Did QA just fall asleep or something? To top it off, even if it were working well, whose freaking bright idea was it to make you go to the upper left corner to click Activities, then back to the bar to click the Show Applications button, just to pull up what you have installed? I've started using the Applications menu that Fedora includes by default; at least that doesn't take a Presidential administration or two or wild sweeping mouse gestures just to show you a list of apps, even though it's still noticeably slower than 3.38.

The slowdowns are entirely specific to GNOME. Once you actually get an app started, like Firefox or a game, display speed is fine, so the problem clearly isn't pushing pixels; it's something higher level in GNOME. Switching all the core scheduling to performance made at most minimal difference. Similarly, (as root) echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level instead of auto made things a little better, but there is still no excuse for how bad it is generally. About the only thing that made more difference than that was simply turning animations off altogether in GNOME Tweaks. Nothing was smooth anymore, but it was about twice as fast at doing anything, so that's how I'm limping along for the time being.

With those significant problems on deck, the usual turmoil with custom themes and extensions is actually anticlimactic. I had to make some tweaks to my custom Tiger-like GNOME shell extension to fix the panel height and a weird glitch with slightly thicker border lines on the edges of the panel, which you can see in the screenshot below. Quite a few extensions could not automatically update to GNOME 40, either:

I've become irritated enough by this that I actually did set disable-extension-version-validation to true in dconf-editor, which made a couple start working immediately, including my beloved Argos custom script driver. For the others I downloaded the most current version of the shell system monitor and this fork of Dash-to-Dock, and manually installed them in ~/.local/share/gnome-shell/extensions/ (you may need to reset the GNOME shell with Alt+F2 and r to get gnome-extensions enable to actually see their UUIDs). A few I should have dispensed with earlier: No Topleft Hot Corner can now be simply replaced by gsettings set org.gnome.desktop.interface enable-hot-corners false, and AlternateTab's switcher behaviour now can be rigged manually from GNOME Settings.

I'm now more or less back where I started from, but working with apps is much less fluid and the desktop experience is undeniably inferior to prior releases, and I can't believe no one thought to blow the whistle during the test phase.

If you use Fedora purely as a command-line server, other than the initial hiccups with downloading packages, it seems to work. If you use KDE or Xfce or anything other than GNOME as your desktop, you're probably okay with F34 too, though I didn't test those (I may later). But if you use the default GNOME on Fedora, especially if you use Wayland, think twice about this update before installing it while you've still got some time with F33. Part of riding the bleeding edge is drawing blood now and then, but F34's wounds seem much more self-inflicted than usual. This is the worst Fedora update since I started using it in F28 and I'm not exaggerating in the slightest.

LibreBMC and Kestrel: two separate BMC tastes that taste separate separately

After our article earlier this week on LibreBMC and my concern as to what it meant for Raptor's own Kestrel "soft BMC" project, Raptor's Tim Pearson contacted me and advised that Kestrel is a standalone Raptor product on its own timetable that's still very much in development. In fact, the attached screenshot shows evolution from just last month with On-Chip Controller support now in the firmware, where it is able to read the temperature sensors and has scaffolding for monitoring fan RPM. Although Raptor makes it clear that Kestrel is and always will be open, and is developed on OpenPOWER systems using open tooling, it is mostly Raptor in-house code with the only non-Raptor bits being Zephyr (the OS), LiteX (the FPGA designer), Microwatt (the CPU core) and some minor components like I2C.

The other noteworthy thing is that Kestrel will indeed be offered as an aftermarket plug-and-play upgrade for existing Blackbird and Talos II systems, no soldering required. This is excellent news, because while LibreBMC is a very encouraging development and has wide-ranging implications beyond OpenPOWER systems, its basis on OpenBMC means a heavier installation that continues to be less well suited to a workstation environment (in terms of interface and sheer startup time amongst other things). While Antmicro's planned LibreBMC card may also work in Raptor systems, it's really meant for OpenPOWER servers as well as other server-class machines with onboard BMCs that aren't necessarily Power-based. Whether we like it or not, while there's a lot of nerd cred around OpenPOWER workstations, such systems presently represent a minority of the POWER9 installed base (let alone of all BMC-managed hardware). It thus makes sense that Raptor, currently the only manufacturer of OpenPOWER workstation machines, would be in the best position to create an open BMC that also is better tailored to improving the desktop user experience. As I've written here before, the envisioned improvements to fan control and user interface are very welcome, but Kestrel's promised 10-second power-to-IPL is a huge win for the desktop when ASPEED BMCs running LibreBMC right now take over a minute or more. LibreBMC's OpenPOWER core will certainly improve performance but I doubt it will be to that extent.

But I also suspect more good news than just an aftermarket upgrade is afoot. I pointed out in the LibreBMC article how much one of the physical form factors in particular looks an awful lot like an single-board computer, because really when you have a system-on-a-chip and on-board peripherals, you kind of have a "PowerPi" already. Raptor isn't saying, but Kestrel, like LibreBMC, will necessarily have its own SoC at its heart and thus could easily be the basis of another OpenPOWER SBC project, possibly one that might be even better suited for that environment. Plus, being Raptor, you know all the components will be open and blob-free (especially now that they've eliminated the blobs for the onboard Broadcom Ethernet, leaving just the Microsemi PM8068 as the last blob firmware component and only if you buy it as a BTO option). It will be worth watching to see when and how all this comes to market. Either way, getting your choice of BMC alternatives is a really great thing especially when the BMC is so critical to the system as a whole.

GNU Guix 1.3

As previously promised, support for POWER9 is now officially a part of the newly released Guix 1.3. Besides many performance and functionality improvements allowing you to Scheme your way to a full installation (or roll back one), the OpenPOWER port should run on any POWER9 PowerNV system (no word on POWER8 support currently, although the support is for generic powerpc64le). Note that as many binary substitutes aren't yet built for POWER9, support is still considered a "technology preview" and you will need to build many of the packages from source for the time being. It is expected this support will catch up as more build capacity comes online.

The only disappointing thing about this announcement is that there doesn't appear to be a way to install standalone Guix (i.e., Guix System) on OpenPOWER yet, nor are there any ISO images you can just pop in your workstation; you'll need to install some other bootable foreign distro and install Guix-the-package-manager on top of it. Presumably Debian, Fedora, Void and others will be suitable, but a full from-scratch install option will be needed to bring this port fully up to parity.