Latest Posts

assert_slb_presence aaargh_warnings_everywhere make_it_stop

We're tracking what seems to be a recent regression in Linux ppc64le (and probably big-endian as well, if we understand the actual cause) kernels from at least 4.20.5 and possibly a little earlier which throws recurrent kernel warnings to dmesg. Depending on your distro this may pass completely unnoticed except for your logs filling up a little faster, but systems that send notifications on such events may drive you up the wall (such as our Fedora 29 installation, where our testing of current Firefox trunk trips this assertion like mad). The output invariably looks like this:

[46425.991034] WARNING: CPU: 22 PID: 0 at arch/powerpc/mm/slb.c:74 assert_slb_presence+0x28/0x40
[46425.991039] WARNING: CPU: 18 PID: 0 at arch/powerpc/mm/slb.c:74 assert_slb_presence+0x28/0x40

followed by the usual debugging information. As the filename implies, this is related to the CPU's segment lookaside buffer, but failing the given assertion is otherwise harmless on the Talos. It looks like the bug has been there for a little while but at least as of 4.20.10 only occurs on CPUs that support the slbfee. instruction (POWER6 and up) and, if our understanding is correct, only on testing effective addresses with a particular bit set. If so, this patch should fix it, but there is no ETA.

In the meantime, if you're badly affected, one way to get the messages to temporarily quiet might be to twiddle your console logging level settings; see man klogctl for how this works. Alternatively, on a Red Hat-type system like ours (Fedora, CentOS, etc.), the notifications come from ABRT, so killall abrt-applet will temporarily quell the warnings (/usr/bin/abrt-applet --gapplication-service & to restart).

Ubuntu LTS 18.04.2 available

An updated release of the long-term support Ubuntu 18 (Bionic Beaver) is now available for ppc64el. Read the full changelog for 18.04.2. As with prior releases, Ubuntu 18 should "just work" on the Talos II. All Power ISA official releases of Ubuntu are Server branded and do not install a GUI by default.

The last POWER1 on Mars is dead

The Opportunity Rover, also known as the Mars Exploration Rover B (or MER-1), has finally been declared at end of mission today after 5,352 Mars solar days when NASA was not successfully able to re-establish contact. It had been apparently knocked off-line by a dust storm and was unable to restart either due to power loss or some other catastrophic failure. Originally intended for a 90 Mars solar day mission, its mission became almost 60 times longer than anticipated and it traveled nearly 30 miles on the surface in total. Spirit, or MER-2, its sister unit, had previously reached end of mission in 2010.

And why would we report that here? Because Opportunity and Spirit were both in fact powered by the POWER1, or more accurately a 20MHz BAE RAD6000, a radiation-hardened version of the original IBM RISC Single Chip CPU and the indirect ancestor of the PowerPC 601. There are a lot of POWER chips in space, both with the original RAD6000 and its successor the RAD750, a radiation-hardened version of the PowerPC G3.

That's not the end of Power ISA chips on Mars, though: Curiosity, which is running a pair of RAD750s (one main and one backup, plus two SPARC accessory CPUs), is still in operation at 2,319 Mars solar days and ticking. There is also the 2001 Mars Odyssey orbiter, which is still circling the planet with its own RAD6000 and is expected to have enough propellant to continue survey operations until 2025. Curiosity's design is likely to be reused for the Mars 2020 rover, meaning possibly even more Power chips will be exploring space and doing science where it counts millions of miles from home.


On a recent Hacker News discussion someone pointed me to this weird historical oddity: the AMD Opteron-socket-compatible POWER7 as reported in El Reg, circa 2006.

We use a POWER6 here at Floodgap for the main server, which as typical for RISC servers of those days uses a bespoke logic board and getting a replacement for it was quite expensive (as we found out when it blew one in 2014). Part of this was no doubt due to their low production volumes and in 2006 IBM was still producing x86 Xeon-based servers, so it made logical sense to try to consolidate their manufacturing. (Recall Apple did something similar with the Power Macintosh 4400 and the "Yellowknife"-derived "Gossamer" beige Power Macintosh G3, both of which were intended to use, or at least use more, off-the-shelf commodity PC components.)

What was particularly interesting about this concept, however, was that the envisioned AMD motherboard would also have accommodated SPARC processors, intended to attract IBM, Sun and Fujitsu at a time when Intel was planning to unify their own hardware for Xeon and Itanium (rip). In some respects it may have reflected an IBM perception that Itanium was potentially a threat to their RISC line and to achieve similar economies as Intel planned to.

Did this happen? Although the Register's article implies some prototyping was done, it doesn't look like it ever saw the light of day, and it's not clear why the agreement foundered. Indeed, the POWER7 systems I've all seen continued to use a custom board and I've never heard anything about SPARCs of that generation using such a common logic board either. In particular, the lowest level Power 720 820x machines — the ones that would have been most likely to use such a cost-reduced design — are in fact very similar to the POWER6 820x machines (including our local 8203-E4A), and there are even upgrade paths.

The idea didn't really die, though, because IBM finally opened up their architecture into OpenPOWER with the POWER8 and now anyone can make a board that a Power chip can go into. And, of course, one particular vendor's POWER9 workstation is what this very article is being typed on. Naturally this wasn't altruism on Big Blue's part; it was their attempt to build a larger multi-front ecosystem to combat x86 dominance in the server room, which would embiggen the pie for "big RISC" servers and thus IBM's slice of it. If it also caused Power chips to turn up in other environments, well, that would be more icing on the cake. While the "Opteron POWER7" looks like it never happened, and no one's putting Epyc chips in Talos IIs, at least some concept of a cross-vendor Power logic board did manage to survive and we OpenPOWER pioneers are the lucky beneficiaries today.

So long, Itanium

I have mixed feelings about Intel announcing the end of Itanic the Itanium. Everyone knew this was coming, of course, but Intel finally officially gave Itanium a death date of mid-2021 instead of keeping it limping along as it has for the past several years.

On the one hand, I don't like Itanium for killing PA-RISC. The first machine I ever had root on was a HP K250 running HP-UX 10.20, and I still have a PA-RISC laptop (an RDI PrecisionBook C160L) and a C8000 with dual PA-8900s, the most powerful Precision Architecture workstation HP ever made. I thought PA-RISC was a nice, clean instruction set with decent performance and HP seemed to at least try to keep up with PowerPC and SPARC, and I think it died before its time (there was even talk of using PA-RISC in Amiga computers!). HP is still the Itanium's only customer, mostly because they would probably roil the remnant HP-UX user base with another architecture switch. In fairness, it should be noted that it was HP themselves that didn't think there was any money in continuing with developing their own CPU (the same logic they applied to killing the DEC Alpha, another exceptional architecture murdered way too early), and that may have been true at the time, but Itanium has painted the Superdomes into a corner and HP-UX and OpenVMS will probably go the way of emulation like the Unisys mainframes have gone.

On the other hand, the end of IA-64 marks the end of an era, not only for non-x86 designs within Intel, but as the last major VLIW CPU (which Intel and HP called "EPIC"). GPUs are of course largely VLIW, VLIW chips still turn up in embedded systems applications and there are still oddballs like the Russian Elbrus, but as a CPU architecture load/store designs have largely won. Even your typical Intel chip is essentially a modern RISC-style core with an x86_64 instruction decoder bolted on (and of course all that other black box crap that you don't get with POWER9). Itanium made it worse with a pretty weaksauce x86 emulator and its unusual architecture choices that make it relatively resistant to Spectre-style attacks but difficult to optimize typical software applications for, a recurrent problem with VLIW compilers in general. This was one of the big reasons the SGI IA-64 workstations never really took off compared to the MIPS systems they replaced.

Big classic proprietary CPUs were a big part of my early career and losing Itanium is a sad whimper. But while Power ISA is a much less adventurous design, it's a much more performant one, and OpenPOWER means it will stay relevant for a long time to come. If you've got a Talos II under your desktop as I do, you chose right.

Alpine Linux 3.9.0 available

Alpine Linux 3.9.0 is now available. Unfortunately for platforms that aren't x86_64, Rust is not available and therefore packages downstream of it (most notoriously current versions of Firefox) are not available either. That includes their ppc64le support, which doesn't seem to support the POWER9 yet, though there is hope for Talos II support in a future release.

Alpine Linux's less typical internal configuration seems to be the sticky point. There had been requests to help with the Rust porting effort back in October, and while the ppc64 version could build, it apparently had (has?) crippling crash bugs. This is still better than their 64-bit ARM support, which apparently doesn't even compile.

Firefox 65 keeps up the POWER

Firefox 65, thanks to the continued heroic efforts of the Talos and Power ISA community, finally builds out of the box on this Fedora 29 little-endian Talos II without any patches. The marquee feature in this release is WebP support (as demonstrated in Google's WebP gallery). As always, make sure your Rust and C compilers are up to date, and you will probably need to update cbindgen in cargo. Among other bugs swatted, this version of Firefox fixes issues with jemalloc on systems that use a 64K page size (most though not all Linux distros currently supporting ppc64(le), including Fedora 29 which I'm typing in), so ac_add_options --disable-jemalloc should no longer be necessary in your .mozconfig.

Firefox 66 is currently not buildable due to bug 1521713, but hopefully we'll have that on beta before it goes to release (it's a matter of fixing some flubbed release asserts).

Meanwhile, I've completed the Ion code generator, Baseline code generator, shared inline cache code generator, trampoline, lowering and move emitter for the Firefox POWER9 little-endian JIT. Still left to do are the macroassembler and the low-level assembler, though that is actually fairly straightforward, just tedious. It was very helpful to crib off my own work for TenFourFox's IonPower and some TenFourFox code will live on in the POWER9 port even after the last Power Mac G5 has blown its liquid cooler. I don't think I'll make it in time for the next ESR (which should be Firefox 67), but I'm reasonably confident I can get it in around Firefox 69 or 70 based on my current rate of progress.

Flicker fixin' the WX7100

We talked about properly enabling the Command key so that Power Mac users could feel at home, so let's talk about flicker fixing so that Amiga users will feel at home. ;)

A gaming pet peeve of mine on modern systems is that full screen is still often the default. This is a workstation, damn it, not a freaking DOS box. There are other things running! But while working on a forthcoming article about Linux gaming on the Talos II (spoiler alert: it's a thing), my pet peeve was compounded upon returning to my usual display resolution when the AMD WX7100 workstation card started randomly flickering.

The first time this happened I solved it with a power cycle, but obviously I don't want to do this every time if I accidentally start a game with the wrong options. Apparently this regression in amdgpu was introduced somewhere around kernel 4.15. My best guess from tinkering a little with the low-level settings is that this issue is triggered by weird switches in refresh rates and seems to have something to do with rapid changes in memory frequency. Raptor sells this card as a BTO option for all models in the T2 line, so this is likely to be a source of annoyance for other Talos owners using their supported card. Unfortunately, it's not at all clear when it might get actually fixed.

If this happens to you, there are a few solutions. The one that works reliably for me is to hard-reselect my monitor's refresh rate with xrandr. My screen likes 60Hz and allows 50Hz, so I do (assuming your uid is in the right groups for this) xrandr -r 50 ; xrandr -r 60 to zap the screen back. This seems to do the job even when xrandr thinks I'm already at 60Hz. If the game didn't fix your resolution, you might add something like -s "1920x1080" or your geometry of choice.

If that doesn't work, disabling auto power management on the card seems to also help, though this may cause unexpected heat, power or performance changes. Something like (as root) echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level (or low, instead of auto) also fixed the issue on my system, suggesting the same root cause. You can also try limiting the memory clock to a fixed frequency, such as (as root) echo 2 > /sys/class/drm/card0/device/pp_dpm_mclk which pegs the clock at 2000MHz.

Now, if you'll excuse me, I have a date with shooting Nazis. In a window.

Introducing appmodmap: make the Command key work again (or, making Mac refugees into Talos owners)

At least in terms of sheer numbers, Power Macintoshes are still the most common Power ISA-based computers, and those of us who still used Power Macs until lately (while my trusty 13-year-old Quad G5 may no longer be my daily driver, it is still used, and it's still under my desk) may well be the Talos II's most natural audience: we gravitate to non-x86 architectures, we like PowerPC in particular, and many Power Macs nowadays run Linux too. Yes, of course there are other audiences for the T2 and it's the truly open libre nature of the architecture that's its strongest general selling point, but those folks attracted by "the next Power Mac" are going to come from the Apple world and it would be nice to help them feel more at home. Yes, the T2 has a unique advantage in that Mac OS X can be run under QEMU with KVM acceleration. But even virtualization has a speed penalty and doesn't always work, and native apps will be more current.

The screenshot at right shows GNOME OSC overlaid with my own tweaked GNOME shell theme, unmistakably still GNOME but evocative enough of early Mac OS X to make me feel at home. Having used Macs in some way or another since 1987, and still using a MacBook Air as my travel laptop (at least until 10.15 makes it impossible to run 32-bit apps, which is where I'll be getting off the fruit bus, thank you), the look isn't jarring switching back and forth and my friend Jon thought I was still on the G5 after a cursory glance at the screen.

However, the other deeply wired part of Mac users is the Command key. My muscle memory is incredibly ingrained with Command key combinations after all these years. I always feel a little hobbled on a regular PC keyboard, and since my KVM is shared between the Quad G5, the Talos II, an SGI Fuel and a Mirrored Drive Doors Power Mac G4 and I use an old-school white Mac keyboard for all of them, having to internally codeswitch back and forth from Ctrl-Q to Cmd-Q may be a first world problem but sure gets obnoxious after awhile. The tips in this article are by no means specific to the Talos systems, but speaking from my own experience I suggest they might make things a bit less alien for a Mac user in transition.

Naturally, by default most things use Control key combinations. Some users will just switch Control and Command but I find this a rather blunt solution that works for some apps but not others. GNOME's system software is somewhat more accommodating to Command key users than KDE, but at the application level rather few packages, GNOME, KDE or otherwise, will let you change their built-in keymappings. Even of those that do, it doesn't always work, or they don't allow the use of the "Super" (their alias) key except in certain specific situations. Fortunately GNOME Terminal does allow this, so I manually remapped my usual keystrokes to the standard Mac combinations. On the browser side, if you go into about:config in Firefox and set ui.key.accelKey to 91 (and possibly restart the browser and/or reset your profile), it will generally work there also mostly as you expect and menus will show the equivalent combinations with the "Win" key instead. This isn't a perfect solution since sometimes the keybindings don't stick in odd places, or some sites will sniff Linux from your user agent string and enforce Control key combinations (Blogger, ahem), but it largely works out of the box.

Unfortunately such applications are the exception and not the rule. Changing GNOME's keyboard settings to map Close Window to Super-Q (Command-Q) will at least make most other apps close in a Mac-like fashion but there isn't much otherwise for any of the other typical functions.

The usual advice people suggest is something like AutoKey. In fact, I did use AutoKey myself for awhile and I can't complain about the functionality it offers, but I found it rather fragile and prone to crashes during use and after a couple system updates it then stopped working completely. More to the point, it was labourious drudgery to manually reconstruct all the Command key combinations for the apps that needed them, and I never ended up finishing that work before it finally crapped out.

Really, all we need is just a way to dynamically determine which app is up and then transparently swap keys around as required based on it. As usual, if you want something done you do it yourself. The result is appmodmap.

appmodmap is a small daemon that watches what the topmost X11 window is and dispatches scripts to change system settings as required. I wrote it for the Talos and I haven't tested it on anything but Fedora, but it strictly uses documented calls and thus should work on anything with X11 and POSIX. It doesn't need elevated privileges and can simply run as you. As compiled out of the box (see the README), it uses setxkbmap and gsettings to alias the Control key to the Command key for the apps on my machine that need it and still provide things such as Super-Tab to switch applications regardless of whatever app is frontmost. When the remapping isn't required for an application, appmodmap will quietly switch everything back to the default, and thus most things "just work." Note that in this scheme already existing Super-key combinations may get temporarily waxed while Control-key aliasing is still in effect, so I had to change a few more shortcuts I need in GNOME's keyboard settings for things like taking screenshots (I remapped them to Alt/Option instead).

You can also use appmodmap as a general way to change any system setting dynamically based on the window class (just write the necessary primitives), but this was what I wrote it for originally, and I still include it as a useful demonstration. Best of all, rather than doing lots of hand work in the AutoKey interface, adding more applications to the daemon just means adding the window class hint name with the desired bitmask, then quickly rebuilding the daemon and restarting it.

Overall, speaking as a long-time Mac user, just a few tweaks made my T2 feel much more like I was used to and thus a lot more productive than having to untrain my fingers. If your primary interest is the Talos' strong commitment to freedom, you'll find a way to use it no matter what contortions the interface makes you do, but if you're a Power Mac user who's a little scared of Linux but yet needs more grunt than your trusty G-series Mac can muster, here's one less excuse for not moving on from the company that left you high and dry.

IBM's POWER9 retrospective is a little one-sided

I'm not going to fault IBM taking a victory lap with the POWER9 in their one-year anniversary retrospective. It's a kick-ass processor; that's why I'm typing this on one. I'm not even going to fault them for biasing it towards their own server products, because that's what IBM sells and they're a business and they want to sell their own stuff. And IBM maintaining financial health gives them the R&D capacity to make the POWER10 even more awesome, so bring on the salesdroids.

But while Google got a shout-out with their bespoke POWER9 Zaius server platform, now in production, IBM seems to have forgotten that Power ISA is making a triumphant return to the desktop in a form that's gotten at least as much press as Zaius/Barreleye, probably sold more units, and is actually in the hands of real end users who are using it as their real computers right now. Hmm, I wonder what computer that could be?

Let's consider the historical perspective: the last major Power desktop system that didn't come from IBM was of course the Power Mac G5 Quad, which was replaced by the Mac Pro in 2006. (The last Power workstations from IBM were the IntelliStation POWER 185 and 285. This pair of machines outlasted the Quad G5 until 2009 but they preferentially ran AIX, and weren't available in large numbers.) While PowerPC chips owned the game console market for a period of time (Wii/Wii U, PlayStation 3 and Xbox 360), Apple's Intel transition meant the only remaining third-party PowerPC desktops were the AmigaOne machines. I like Amigas fine, but these systems so far have been rather underpowered due to their use of embedded designs and fairly expensive due to the small market, boutique production runs and distinctly poor economies of scale. Frankly, depending on how big a detractor you are, PowerPC hadn't been competitive against x86 on the desktop since the early G4 days, and no one other than IBM had shipped a top-tier Power ISA desktop in over 12 years.

Now we have not only a Power ISA CPU that is performance-competitive with current x86_64 offerings, but an entire third-party libre workstation built around them that you can order and get shipped to your house right now. The cost of a full Talos II only seems steep until you consider how much a Xeon box in the same ballpark will run you, and the delta seems much more reasonable then. If Blackbird is successful at establishing a "low end" POWER9 machine with a more amenable price, we could see Power ISA start to become a major desktop player once again, especially as the software support situation continues to improve by leaps and bounds and people realize what a liability blackboxes like Intel Management Engine are. Even if you're not in the market for a T2 right now, you'll be more likely to have a real choice in workstations when you do. And that's good news for everybody.

Seems like IBM could have mentioned that.

Void Linux goes POWER9

A nice commit landed in Void Linux: 64-bit Power ISA support including big and little endian. Although you'll have to build it yourself and it looks like more work is needed for other packages, eventually the resulting binaries should boot on any Talos II system and the musl support is welcome for those who prefer to avoid issues with non-64-bit long double.

Related is work to get old-school 32-bit PowerPC working, so your beloved Power Mac can play too.