Posts

Showing posts from 2019

Firefox 66 on POWER


Firefox 66 builds out of the box as Firefox 65 did. In addition, as part of the porting effort, I'm running occasional check builds roughly weekly to make sure we intercept issues on nightly before they get into beta and require a higher threshold of approval for patches. So far no major issues and Mozilla continues to be very amenable to getting the occasional fixes in promptly.

The biggest change in Fx66 is the number of default content processes in this release has jumped from 4 to 8. This is good news on our massively parallel multicore systems generally, but it's possible that the differing memory usage may be what tickles that harmless kernel assertion (though on the other hand, setting it back down to 4 didn't eliminate them completely). If you're on Fedora as I am or another distro that elevates kernel warnings to notifications, you'll probably want to turn off those system notifications until your distro's kernel gets the fix in it; see that article for details.

A close look at the Raptor Blackbird and what I did at So Cal Linux Expo 17


Wow, what a swag haul (big box o'breakfast cereal for scale), and what a fun day at SCaLE 17x, the 17th annual So Cal Linux Expo! I'd actually never been before and now I see why people love to go!

SCaLE is at the lovely Pasadena Convention Center, just a short drive from Floodgap Orbiting HQ in sunny rainy southern California. If you're in the greater Los Angeles area, it's pretty accessible by Metro rail, and there are lots of restaurants and things to do if you're flying in from away. I had lunch with my good buddy Bill (lately of the Linux Journal) and two of his cow-orkers at Islands Burgers just north of the Convention Center.

Even if you just buy an expo-only ticket, there are a huge number of vendor booths from big names like IBM, VMware and even Microsoft (!) all the way down to open source projects like VLC, Krita and Inkscape. Would have loved to have seen a Mozilla booth, though. Just saying. I even wore my Mozilla grey hoodie.

I even renewed my Electronic Frontier Foundation membership in support of the great work they do (make mine Titanium).

But of course the star of the show for your humble writer was our friends at the OpenPOWER Foundation, and they came ready for action with our favourite heavy-duty free computing platform:

Hugh Blemings, executive director, kindly tolerated my shutterbugging and an endless parade of retakes to get everything just right. (By the way, did you notice that Timothy Pearson from Raptor is now on the OpenPOWER board?)

He brought in tow their Debian dual-four-core Talos II, which did presentation duty, and of course a prototype Raptor Blackbird motherboard! For those of you new to the blog, this is Raptor's lower-cost way to get into the Power ISA ecosystem. So let's have a detailed look.

Here's the mATX board itself and a side view of the ports. (I'll zoom in on some items of note in a moment.) You can see the ports for USB 3.0 (two rear with additional headers for two more), 4x SATA, 5.1 analogue audio, S/PDIF digital audio, 3x GigE and HDMI. You can also see the two RAM slots and the x8 and x16 PCIe slots. My grizzled old hacker heart was warmed to see that there is still a good old fashioned serial port there too.

The 2D framebuffer is provided by the AST2500 BMC (ARM11 based with a sidecar ColdFire core), at left, routed to HDMI via an ITE IT66121FN; all three Gigabit Ethernet ports are serviced by the Broadcom BCM5719, at right.

Blackbird has some additional hardware for higher-security applications. At left are the flash chips for the BMC and boot flash with hardware write-protect switches, meaning if you can secure the case, nothing's overwriting the firmware. (I looked on my early-model T2 and can't find any such switches, so this is definitely an improvement. Update: found them, just at a slightly different relative location. Timothy Pearson in E-mail notes, however: "In practice, the current firmware stack is a lot chattier with the Flash than we'd like, so there's still some work to be done before we can roll out write protect in official form to both platforms.") Further anti-tampering security is offered by a Raptor-specific FlexVer connector (PDF).

The single POWER9 CPU socket (four or eight cores), and an interesting unlabeled port. Hmmmmmmm. (Update: Timothy Pearson in E-mail identifies it as an FSI port. "Talos II has one just like it; you can plug an FSP box into that port. The FSP boxes are proprietary and available to IBM partners only (like us), however they're also being replaced with the BMC for the most part, which speaks the same protocol and incidentally can do the same kind of debugging tasks now that the FSP boxes were traditionally used for. We will probably continue to retain the connnector, since even if someone designs an open FSI box to plug into the port it could be useful for various low level hacking (in the good sense) tasks. Note that neither the BMC nor this connector can bypass secure boot if enabled, and certainly FlexVer would immediately thrown an attestation error if anyone even tried.")

And last but by no means least, a four-core POWER9 CPU and the exposed die, which Hugh had on display. What a gorgeous bit of silicon, amirite? I'm still planning a full review of this Power-on-a-budget system when my production unit gets here hopefully in just a couple months.

Some other fun stuff:

The One Laptop Per Child handcrank! It exists! I would have killed for this back in the Give One Get One days.

Purism had a nice showing. In addition to their very impressive line of free(r) laptops, they also had a prototype of the Librem 5 libre smartphone, currently scheduled for Q3 2019. I just bought a Pixel 3 (these pictures were taken with it), or I would be buying one of these. I might anyway. I also reminded them that some of us wouldn't mind buying a non-x86 laptop. Power would be nice, but ARM would be fine too. He duly took it under advisement.

Not to be outdone, System76 had a Thelio system on display. Weird site but sexy case. It would be high on my list if I could get the case by itself and slap a Talos-style system in it. How about it, 76?

Standing guard at the entrance to the the exhibit hall was this huge Tux mosaic, made out of AMD Opterons.

It may be a Linux expo, but that doesn't mean the BSD folks can't be there too. I got a couple "RUN BSD" stickers (a la RUN DMC) for my NetBSD machines and one of the fun flashing devil horns headbands. Still looking forward to the FreeBSD port to POWER9!

There were many great open source projects there, but the photographic winner was this particular famous project:

Rock those hats, guys!

And just a tiny selection more of the many vendors, from whom I shamelessly lifted free stuff, and without whose financial support the Expo would probably not be possible:

Special shoutout to the really plush setup the folks from IBM had:

Distros in the hizzouse:

See if you can figure out which one I wrote. No, go on. I'll wait.

The GNU Project/Free Software Foundation had an appropriately chaotic-good booth, with a very important message:

Finally, a few more open source projects to close us out:

Overall I came expecting to just take a couple pics of the Blackbird and leave, and instead I ended up having a blast with all the great exhibits, vendors and free junk to clutter up my house. Next year I'll be springing for the full show and hopefully bringing my wife and a couple friends. It's a great time to be in free computing and the interest has never been higher. I won't put any jokes in about this being the year of the Linux desktop, but I think it really is the year of Power ISA being back on the desktop. It's been gone for too long and it's good to see it roaring back.

Just watch out for the robots.

Ubuntu LTS 14.04.6 available


This is mostly relevant to our 32-bit PowerPC colleagues, but along with the recent updates to Ubuntu 16 and 18 comes what will likely be the final release of Ubuntu 14.04 LTS (read the change summary). This release is primarily security-focused, mostly to deal with the APT redirect vulnerability, though there are of course other fixes. PowerPC and POWER8 users still stuck on Ubuntu 14 should strongly consider upgrading to Ubuntu 16, which still supports 32-bit PowerPC, also supports POWER8 (but not POWER9), and is still receiving fixes as Ubuntu 14.04 will reach end of life in April 2019. Meanwhile, ISO images are available.

New Talos PowerAI SKU


If you, ahem, want to see how good the POWER9 is at computer vision -- or any other kind of deep learning -- Raptor now announces a new Talos II package, the PowerAI Development System (TL2PA1). This is a T2 Lite with a 4-core CPU, 32GB RAM, 128GB NVMe flash and most notably an NVIDIA RTX 2070 GPU. Debian is pre-installed.

Wait, did you say NVIDIA? Yes, because the intention with this system is to run IBM PowerAI Vision, and an NVIDIA NVLink-capable GPU is required (and so is a trial license). That automatically wouldn't make this a very good Talos workstation due to NVIDIA's historically poor open-source support (nouveau or bust since Power isn't supported by NVIDIA's proprietary driver), and to be sure, Raptor seems to be discouraging it for that purpose ("There is no way to add OpenGL support to the proprietary driver stack ... This system is designed for GPU compute, and while a minimal 2D framebuffer is supported 3D applications will fall back to non-accelerated LLVMPipe rendering"). However, if you want lots of threads and a system to run a high-performance computer vision platform, you've now got a choice which is at least freer than a comparably configured x86 box. Base price starts at $3450 and the SKU should be shipping soon.

A programming note: I'll be wandering the exhibit halls of the Southern California Linux Expo in Pasadena on Saturday. See you at booth 429 or be less of a nerd than I am.

Linux 5.0


Linux 5.0 has been released (what used to be "4.21"). Linus is very clear that no kernel release is a feature release and the 5.0 divide is more an arbitrary numerical cutoff than anything else, but there are some important advancements in Linux 5.0 such as improved AMD GPU support, support for AMD FreeSync, file system improvements (especially to encryption performance), continued Y2038 work and various additional device support. I suspect that Talos users will find improvements to the ASpeed BMC media driver particularly relevant; my guess is this is part of the kernel support that the Blackbird is waiting on.

Probably the most notable Power ISA-specific feature in Linux 5.0 final is support for the POWER On-Chip Controller in POWER8/POWER9, exposing temperature, frequency, power usage and other sensor data through hwmon. This will likely enable Talos-family owners to get even better environmental monitoring support for their machines. Other Power-specific changes include Spectre V2 mitigations for many NXP/Freescale Power CPUs, various KVM improvements and even some improvements for P. A. Semi chips (get that X1000 out and celebrate).

Read the full changelog if you dare, or an annotated summary.

Ubuntu LTS 16.04.6 available


Ubuntu LTS 16.04.6 is available (see the change summary for more details). Although 16.04.6 cannot boot on the Talos family, it does support POWER8, and is particularly relevant to our 32-bit PowerPC friends as it is the final Ubuntu release to support that architecture officially. All Power ISA official releases of Ubuntu are Server branded and do not install a GUI by default. Installation images are hosted at the Ubuntu download site.

Raptor, help us out here on the software side


I'll preamble this with a status report on the ppc64le Firefox JIT: stalled. The stall-out is because my time is currently occupied with Mozilla bug 1512162 which has to do with the connection between JavaScript and the native representation causing debug builds to assert. I wallpapered this for Firefox 66, and that seems to be okay, but the wallpaper just seems to have moved the badness around and the issue is more fundamental. This is a more serious issue, so it's my highest priority.

Unfortunately, work on this bug is getting stymied by the fact that POWER9 has hardware watchpoints disabled in the Linux kernel. Trying to catch who's writing the faulty values is nearly impossible in an application of Firefox's complexity without it because without hardware assistance gdb has to single step through the code. When you don't know even what haystack to look for the needle in, it's slow going, as in (no exaggeration) hundreds of times slower. I left Firefox to try to initialize overnight on my 8-core Talos II in the debugger. By the morning, 8 hours later, it hadn't even launched its first thread.

The reason they are disabled is an errata which causes watchpoints set on cache-inhibited memory (such as devices) to make the CPU halt with a checkstop. Arguably some sort of fault on this is correct behaviour, but a checkstop is catastrophic; it's the equivalent of stopping your car by driving it off the road into a wall. I don't fault the PowerPC kernel maintainers for taking this interim approach because without it an unprivileged user could instantly halt the machine, even inadvertently. Even if the kernel could detect that the watchpoint was pointing at a cache-inhibited address and return an error, a tricksy user could potentially set up the watchpoint to "good" memory and then change the memory mapping.

I talked to the PowerPC kernel maintainers about this and an interim solution we're sort of agreed on is to use a debugfs entry as a one-way switch so that workstation developers like me can turn on hardware watchpoints "at our own risk." When I'm ready to debug something that requires a proper watchpoint, then I create a debugfs file and the kernel will then allow the watchpoint in hardware until the next reboot. I'm the only user on the machine, so if I screw it up, it's only my pasty Roman sculptured tuckus (and filesystem).

But this isn't going to write itself. I need to scratch my own itch, get it working, get it accepted, get it actually in a kernel (instead of schlepping forward local changes), and then finish debugging Firefox, and then finish the JIT, in addition to my day job, my work on TenFourFox and not being in trouble with my lovely wife for not emerging from the back room for hours.

So now I'm going to be a little less than nice with Raptor. Raptor has been fairly public about their support for Chromium, ostensibly because of the (IMHO irresponsible) proliferation of Electron apps, but they've been very tepid on Firefox. I understand a small company can't do everything, but they have had time to do ports of Unreal Engine and WINE (though this in and of itself is not enough to make the QEMU-WINE fusion Hangover work, which I've been also trying to tinker with and have it about 70% building). These are fun things to do and are certainly interesting, but that makes statements like this a bit galling, and statements like this a bit disingenuous.

Mozilla has a chicken-egg problem when it comes to an architecture that until very recently had a very small desktop share: its share (and share of desktop users using Firefox) would doubtlessly increase with a Firefox JIT, but the resources to expend into writing that JIT can't be justified until there is a larger share. Furthermore, it rings hollow for Raptor to ding Mozilla in that tweet about not being sufficiently open when Google until literally days ago wouldn't land the existing POWER9 Chromium work when Mozilla has been allowing POWER9 (and other PowerPC) patches into Firefox as a tier-3 for pretty much its entire existence. You can put the open-source lipstick on the Google pig as much as you like but at the end of the day, it's still Google and it's still Google's repo. Freedom involves choice. I'm not going to slam the people who did hard work on the Chromium port, because it is hard work and unfortunately Electron is a thing despite my misgivings, but I am going to slam Raptor for endorsing it at Firefox's expense.

I'm not asking Raptor to do the Firefox JIT port, though I may be soliciting help to farm it out with my reduced number of available cycles. (Right now it's based on Firefox 62, which once it works there, we'll forward-port it to trunk. More on that in a future post, but I'll probably put my current work up on Github. If you're interested in contributing, post in the comments.) I am asking Raptor to endorse the effort, however, and I am asking them to become more involved with developer-facing features to allow those of us who are working on ports to do so more productively.

As an example, developing and getting the interim debugfs switch for hardware watchpoints into the kernel would be an enormous help to me personally (and would save me a great deal of time), and would probably be very beneficial for other developers. Nearly everyone on the LinuxPPC kernel team I talked to agreed this is a big deficiency and one that is realistically implementable. It would be nice if this could proceed in parallel so I'm not blocked on doing everything myself because on a scale of 0 to even, I just can't. I'm sure there are many other developer pain points that will appear as more people start working on Talos systems, and I'd like Raptor to also treat these requests with priority and dedicate resources to worthy ones to allow more port work and development to flourish.

Let me soften a little bit in conclusion by saying Raptor has a very hard row to hoe being a small company jumpstarting an entire ecosystem. I'm being hard on them because I'm glad they exist, I intend to continue being a customer, and as a long-time Power ISA bigot I want them to succeed. But I also want to see the principles of free computing embodied in the hardware for the Talos family appropriately manifested in software. I don't see that being adequately expressed in the choices they've made so far and I'd like that to change. Developers need to be prioritized and software choice needs to be facilitated. Let's see more of that so we can see more POWER9 adoption and a brighter future for desktop computing.

OpenSUSE 15.1 Beta available, just not for us (yet)


The beta for OpenSUSE 15.1 is now available, at least for x86_64. This isn't actually news on the ppc64(le) side because the Leap releases don't currently have a Power ISA port. However, if you want to run OpenSUSE on your Talos II, you can with the Tumbleweed releases, which is their rolling-release flavour and apparently (I'm told) works quite well on the hardware. Nevertheless, hopefully as the install base grows there will be more interest in a stable Leap release for Power ISA as well.

Blackbirds to ship Q2 2019


On Twitter Raptor is reporting Blackbird shipments will start occurring in Q2 2019 instead of Q1 as previously announced. However, manufacture of the first production batch is in progress (we have one on order and will be doing a review as soon as it arrives). The ASpeed BMC, which is built-in, apparently also needs upcoming Linux kernel support to route its output over HDMI (via an ITE device) when hotplugging a display, which implies current distros compatible with the T2 may not fully work on the Blackbird without a discrete graphics card until they are also updated. We'll be watching.

assert_slb_presence aaargh_warnings_everywhere make_it_stop


We're tracking what seems to be a recent regression in Linux ppc64le (and probably big-endian as well, if we understand the actual cause) kernels from at least 4.20.5 and possibly a little earlier which throws recurrent kernel warnings to dmesg. Depending on your distro this may pass completely unnoticed except for your logs filling up a little faster, but systems that send notifications on such events may drive you up the wall (such as our Fedora 29 installation, where our testing of current Firefox trunk trips this assertion like mad). The output invariably looks like this:

[46425.991034] WARNING: CPU: 22 PID: 0 at arch/powerpc/mm/slb.c:74 assert_slb_presence+0x28/0x40
[46425.991039] WARNING: CPU: 18 PID: 0 at arch/powerpc/mm/slb.c:74 assert_slb_presence+0x28/0x40

followed by the usual debugging information. As the filename implies, this is related to the CPU's segment lookaside buffer, but failing the given assertion is otherwise harmless on the Talos. It looks like the bug has been there for a little while but at least as of 4.20.10 only occurs on CPUs that support the slbfee. instruction (POWER6 and up) and, if our understanding is correct, only on testing effective addresses with a particular bit set. If so, this patch should fix it, but there is no ETA.

In the meantime, if you're badly affected, one way to get the messages to temporarily quiet might be to twiddle your console logging level settings; see man klogctl for how this works. Alternatively, on a Red Hat-type system like ours (Fedora, CentOS, etc.), the notifications come from ABRT, so killall abrt-applet will temporarily quell the warnings (/usr/bin/abrt-applet --gapplication-service & to restart).

Ubuntu LTS 18.04.2 available


An updated release of the long-term support Ubuntu 18 (Bionic Beaver) is now available for ppc64el. Read the full changelog for 18.04.2. As with prior releases, Ubuntu 18 should "just work" on the Talos II. All Power ISA official releases of Ubuntu are Server branded and do not install a GUI by default.

The last POWER1 on Mars is dead


The Opportunity Rover, also known as the Mars Exploration Rover B (or MER-1), has finally been declared at end of mission today after 5,352 Mars solar days when NASA was not successfully able to re-establish contact. It had been apparently knocked off-line by a dust storm and was unable to restart either due to power loss or some other catastrophic failure. Originally intended for a 90 Mars solar day mission, its mission became almost 60 times longer than anticipated and it traveled nearly 30 miles on the surface in total. Spirit, or MER-2, its sister unit, had previously reached end of mission in 2010.

And why would we report that here? Because Opportunity and Spirit were both in fact powered by the POWER1, or more accurately a 20MHz BAE RAD6000, a radiation-hardened version of the original IBM RISC Single Chip CPU and the indirect ancestor of the PowerPC 601. There are a lot of POWER chips in space, both with the original RAD6000 and its successor the RAD750, a radiation-hardened version of the PowerPC G3.

That's not the end of Power ISA chips on Mars, though: Curiosity, which is running a pair of RAD750s (one main and one backup, plus two SPARC accessory CPUs), is still in operation at 2,319 Mars solar days and ticking. There is also the 2001 Mars Odyssey orbiter, which is still circling the planet with its own RAD6000 and is expected to have enough propellant to continue survey operations until 2025. Curiosity's design is likely to be reused for the Mars 2020 rover, meaning possibly even more Power chips will be exploring space and doing science where it counts millions of miles from home.

The AMD POWER7?


On a recent Hacker News discussion someone pointed me to this weird historical oddity: the AMD Opteron-socket-compatible POWER7 as reported in El Reg, circa 2006.

We use a POWER6 here at Floodgap for the main server, which as typical for RISC servers of those days uses a bespoke logic board and getting a replacement for it was quite expensive (as we found out when it blew one in 2014). Part of this was no doubt due to their low production volumes and in 2006 IBM was still producing x86 Xeon-based servers, so it made logical sense to try to consolidate their manufacturing. (Recall Apple did something similar with the Power Macintosh 4400 and the "Yellowknife"-derived "Gossamer" beige Power Macintosh G3, both of which were intended to use, or at least use more, off-the-shelf commodity PC components.)

What was particularly interesting about this concept, however, was that the envisioned AMD motherboard would also have accommodated SPARC processors, intended to attract IBM, Sun and Fujitsu at a time when Intel was planning to unify their own hardware for Xeon and Itanium (rip). In some respects it may have reflected an IBM perception that Itanium was potentially a threat to their RISC line and to achieve similar economies as Intel planned to.

Did this happen? Although the Register's article implies some prototyping was done, it doesn't look like it ever saw the light of day, and it's not clear why the agreement foundered. Indeed, the POWER7 systems I've all seen continued to use a custom board and I've never heard anything about SPARCs of that generation using such a common logic board either. In particular, the lowest level Power 720 820x machines — the ones that would have been most likely to use such a cost-reduced design — are in fact very similar to the POWER6 820x machines (including our local 8203-E4A), and there are even upgrade paths.

The idea didn't really die, though, because IBM finally opened up their architecture into OpenPOWER with the POWER8 and now anyone can make a board that a Power chip can go into. And, of course, one particular vendor's POWER9 workstation is what this very article is being typed on. Naturally this wasn't altruism on Big Blue's part; it was their attempt to build a larger multi-front ecosystem to combat x86 dominance in the server room, which would embiggen the pie for "big RISC" servers and thus IBM's slice of it. If it also caused Power chips to turn up in other environments, well, that would be more icing on the cake. While the "Opteron POWER7" looks like it never happened, and no one's putting Epyc chips in Talos IIs, at least some concept of a cross-vendor Power logic board did manage to survive and we OpenPOWER pioneers are the lucky beneficiaries today.

So long, Itanium


I have mixed feelings about Intel announcing the end of Itanic the Itanium. Everyone knew this was coming, of course, but Intel finally officially gave Itanium a death date of mid-2021 instead of keeping it limping along as it has for the past several years.

On the one hand, I don't like Itanium for killing PA-RISC. The first machine I ever had root on was a HP K250 running HP-UX 10.20, and I still have a PA-RISC laptop (an RDI PrecisionBook C160L) and a C8000 with dual PA-8900s, the most powerful Precision Architecture workstation HP ever made. I thought PA-RISC was a nice, clean instruction set with decent performance and HP seemed to at least try to keep up with PowerPC and SPARC, and I think it died before its time (there was even talk of using PA-RISC in Amiga computers!). HP is still the Itanium's only customer, mostly because they would probably roil the remnant HP-UX user base with another architecture switch. In fairness, it should be noted that it was HP themselves that didn't think there was any money in continuing with developing their own CPU (the same logic they applied to killing the DEC Alpha, another exceptional architecture murdered way too early), and that may have been true at the time, but Itanium has painted the Superdomes into a corner and HP-UX and OpenVMS will probably go the way of emulation like the Unisys mainframes have gone.

On the other hand, the end of IA-64 marks the end of an era, not only for non-x86 designs within Intel, but as the last major VLIW CPU (which Intel and HP called "EPIC"). GPUs are of course largely VLIW, VLIW chips still turn up in embedded systems applications and there are still oddballs like the Russian Elbrus, but as a CPU architecture load/store designs have largely won. Even your typical Intel chip is essentially a modern RISC-style core with an x86_64 instruction decoder bolted on (and of course all that other black box crap that you don't get with POWER9). Itanium made it worse with a pretty weaksauce x86 emulator and its unusual architecture choices that make it relatively resistant to Spectre-style attacks but difficult to optimize typical software applications for, a recurrent problem with VLIW compilers in general. This was one of the big reasons the SGI IA-64 workstations never really took off compared to the MIPS systems they replaced.

Big classic proprietary CPUs were a big part of my early career and losing Itanium is a sad whimper. But while Power ISA is a much less adventurous design, it's a much more performant one, and OpenPOWER means it will stay relevant for a long time to come. If you've got a Talos II under your desktop as I do, you chose right.

Alpine Linux 3.9.0 available


Alpine Linux 3.9.0 is now available. Unfortunately for platforms that aren't x86_64, Rust is not available and therefore packages downstream of it (most notoriously current versions of Firefox) are not available either. That includes their ppc64le support, which doesn't seem to support the POWER9 yet, though there is hope for Talos II support in a future release.

Alpine Linux's less typical internal configuration seems to be the sticky point. There had been requests to help with the Rust porting effort back in October, and while the ppc64 version could build, it apparently had (has?) crippling crash bugs. This is still better than their 64-bit ARM support, which apparently doesn't even compile.

Firefox 65 keeps up the POWER


Firefox 65, thanks to the continued heroic efforts of the Talos and Power ISA community, finally builds out of the box on this Fedora 29 little-endian Talos II without any patches. The marquee feature in this release is WebP support (as demonstrated in Google's WebP gallery). As always, make sure your Rust and C compilers are up to date, and you will probably need to update cbindgen in cargo. Among other bugs swatted, this version of Firefox fixes issues with jemalloc on systems that use a 64K page size (most though not all Linux distros currently supporting ppc64(le), including Fedora 29 which I'm typing in), so ac_add_options --disable-jemalloc should no longer be necessary in your .mozconfig.

Firefox 66 is currently not buildable due to bug 1521713, but hopefully we'll have that on beta before it goes to release (it's a matter of fixing some flubbed release asserts).

Meanwhile, I've completed the Ion code generator, Baseline code generator, shared inline cache code generator, trampoline, lowering and move emitter for the Firefox POWER9 little-endian JIT. Still left to do are the macroassembler and the low-level assembler, though that is actually fairly straightforward, just tedious. It was very helpful to crib off my own work for TenFourFox's IonPower and some TenFourFox code will live on in the POWER9 port even after the last Power Mac G5 has blown its liquid cooler. I don't think I'll make it in time for the next ESR (which should be Firefox 67), but I'm reasonably confident I can get it in around Firefox 69 or 70 based on my current rate of progress.

Flicker fixin' the WX7100


We talked about properly enabling the Command key so that Power Mac users could feel at home, so let's talk about flicker fixing so that Amiga users will feel at home. ;)

A gaming pet peeve of mine on modern systems is that full screen is still often the default. This is a workstation, damn it, not a freaking DOS box. There are other things running! But while working on a forthcoming article about Linux gaming on the Talos II (spoiler alert: it's a thing), my pet peeve was compounded upon returning to my usual display resolution when the AMD WX7100 workstation card started randomly flickering.

The first time this happened I solved it with a power cycle, but obviously I don't want to do this every time if I accidentally start a game with the wrong options. Apparently this regression in amdgpu was introduced somewhere around kernel 4.15. My best guess from tinkering a little with the low-level settings is that this issue is triggered by weird switches in refresh rates and seems to have something to do with rapid changes in memory frequency. Raptor sells this card as a BTO option for all models in the T2 line, so this is likely to be a source of annoyance for other Talos owners using their supported card. Unfortunately, it's not at all clear when it might get actually fixed.

If this happens to you, there are a few solutions. The one that works reliably for me is to hard-reselect my monitor's refresh rate with xrandr. My screen likes 60Hz and allows 50Hz, so I do (assuming your uid is in the right groups for this) xrandr -r 50 ; xrandr -r 60 to zap the screen back. This seems to do the job even when xrandr thinks I'm already at 60Hz. If the game didn't fix your resolution, you might add something like -s "1920x1080" or your geometry of choice.

If that doesn't work, disabling auto power management on the card seems to also help, though this may cause unexpected heat, power or performance changes. Something like (as root) echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level (or low, instead of auto) also fixed the issue on my system, suggesting the same root cause. You can also try limiting the memory clock to a fixed frequency, such as (as root) echo 2 > /sys/class/drm/card0/device/pp_dpm_mclk which pegs the clock at 2000MHz.

Now, if you'll excuse me, I have a date with shooting Nazis. In a window.

Introducing appmodmap: make the Command key work again (or, making Mac refugees into Talos owners)


At least in terms of sheer numbers, Power Macintoshes are still the most common Power ISA-based computers, and those of us who still used Power Macs until lately (while my trusty 13-year-old Quad G5 may no longer be my daily driver, it is still used, and it's still under my desk) may well be the Talos II's most natural audience: we gravitate to non-x86 architectures, we like PowerPC in particular, and many Power Macs nowadays run Linux too. Yes, of course there are other audiences for the T2 and it's the truly open libre nature of the architecture that's its strongest general selling point, but those folks attracted by "the next Power Mac" are going to come from the Apple world and it would be nice to help them feel more at home. Yes, the T2 has a unique advantage in that Mac OS X can be run under QEMU with KVM acceleration. But even virtualization has a speed penalty and doesn't always work, and native apps will be more current.

The screenshot at right shows GNOME OSC overlaid with my own tweaked GNOME shell theme, unmistakably still GNOME but evocative enough of early Mac OS X to make me feel at home. Having used Macs in some way or another since 1987, and still using a MacBook Air as my travel laptop (at least until 10.15 makes it impossible to run 32-bit apps, which is where I'll be getting off the fruit bus, thank you), the look isn't jarring switching back and forth and my friend Jon thought I was still on the G5 after a cursory glance at the screen.

However, the other deeply wired part of Mac users is the Command key. My muscle memory is incredibly ingrained with Command key combinations after all these years. I always feel a little hobbled on a regular PC keyboard, and since my KVM is shared between the Quad G5, the Talos II, an SGI Fuel and a Mirrored Drive Doors Power Mac G4 and I use an old-school white Mac keyboard for all of them, having to internally codeswitch back and forth from Ctrl-Q to Cmd-Q may be a first world problem but sure gets obnoxious after awhile. The tips in this article are by no means specific to the Talos systems, but speaking from my own experience I suggest they might make things a bit less alien for a Mac user in transition.

Naturally, by default most things use Control key combinations. Some users will just switch Control and Command but I find this a rather blunt solution that works for some apps but not others. GNOME's system software is somewhat more accommodating to Command key users than KDE, but at the application level rather few packages, GNOME, KDE or otherwise, will let you change their built-in keymappings. Even of those that do, it doesn't always work, or they don't allow the use of the "Super" (their alias) key except in certain specific situations. Fortunately GNOME Terminal does allow this, so I manually remapped my usual keystrokes to the standard Mac combinations. On the browser side, if you go into about:config in Firefox and set ui.key.accelKey to 91 (and possibly restart the browser and/or reset your profile), it will generally work there also mostly as you expect and menus will show the equivalent combinations with the "Win" key instead. This isn't a perfect solution since sometimes the keybindings don't stick in odd places, or some sites will sniff Linux from your user agent string and enforce Control key combinations (Blogger, ahem), but it largely works out of the box.

Unfortunately such applications are the exception and not the rule. Changing GNOME's keyboard settings to map Close Window to Super-Q (Command-Q) will at least make most other apps close in a Mac-like fashion but there isn't much otherwise for any of the other typical functions.

The usual advice people suggest is something like AutoKey. In fact, I did use AutoKey myself for awhile and I can't complain about the functionality it offers, but I found it rather fragile and prone to crashes during use and after a couple system updates it then stopped working completely. More to the point, it was labourious drudgery to manually reconstruct all the Command key combinations for the apps that needed them, and I never ended up finishing that work before it finally crapped out.

Really, all we need is just a way to dynamically determine which app is up and then transparently swap keys around as required based on it. As usual, if you want something done you do it yourself. The result is appmodmap.

appmodmap is a small daemon that watches what the topmost X11 window is and dispatches scripts to change system settings as required. I wrote it for the Talos and I haven't tested it on anything but Fedora, but it strictly uses documented calls and thus should work on anything with X11 and POSIX. It doesn't need elevated privileges and can simply run as you. As compiled out of the box (see the README), it uses setxkbmap and gsettings to alias the Control key to the Command key for the apps on my machine that need it and still provide things such as Super-Tab to switch applications regardless of whatever app is frontmost. When the remapping isn't required for an application, appmodmap will quietly switch everything back to the default, and thus most things "just work." Note that in this scheme already existing Super-key combinations may get temporarily waxed while Control-key aliasing is still in effect, so I had to change a few more shortcuts I need in GNOME's keyboard settings for things like taking screenshots (I remapped them to Alt/Option instead).

You can also use appmodmap as a general way to change any system setting dynamically based on the window class (just write the necessary primitives), but this was what I wrote it for originally, and I still include it as a useful demonstration. Best of all, rather than doing lots of hand work in the AutoKey interface, adding more applications to the daemon just means adding the window class hint name with the desired bitmask, then quickly rebuilding the daemon and restarting it.

Overall, speaking as a long-time Mac user, just a few tweaks made my T2 feel much more like I was used to and thus a lot more productive than having to untrain my fingers. If your primary interest is the Talos' strong commitment to freedom, you'll find a way to use it no matter what contortions the interface makes you do, but if you're a Power Mac user who's a little scared of Linux but yet needs more grunt than your trusty G-series Mac can muster, here's one less excuse for not moving on from the company that left you high and dry.

IBM's POWER9 retrospective is a little one-sided


I'm not going to fault IBM taking a victory lap with the POWER9 in their one-year anniversary retrospective. It's a kick-ass processor; that's why I'm typing this on one. I'm not even going to fault them for biasing it towards their own server products, because that's what IBM sells and they're a business and they want to sell their own stuff. And IBM maintaining financial health gives them the R&D capacity to make the POWER10 even more awesome, so bring on the salesdroids.

But while Google got a shout-out with their bespoke POWER9 Zaius server platform, now in production, IBM seems to have forgotten that Power ISA is making a triumphant return to the desktop in a form that's gotten at least as much press as Zaius/Barreleye, probably sold more units, and is actually in the hands of real end users who are using it as their real computers right now. Hmm, I wonder what computer that could be?

Let's consider the historical perspective: the last major Power desktop system that didn't come from IBM was of course the Power Mac G5 Quad, which was replaced by the Mac Pro in 2006. (The last Power workstations from IBM were the IntelliStation POWER 185 and 285. This pair of machines outlasted the Quad G5 until 2009 but they preferentially ran AIX, and weren't available in large numbers.) While PowerPC chips owned the game console market for a period of time (Wii/Wii U, PlayStation 3 and Xbox 360), Apple's Intel transition meant the only remaining third-party PowerPC desktops were the AmigaOne machines. I like Amigas fine, but these systems so far have been rather underpowered due to their use of embedded designs and fairly expensive due to the small market, boutique production runs and distinctly poor economies of scale. Frankly, depending on how big a detractor you are, PowerPC hadn't been competitive against x86 on the desktop since the early G4 days, and no one other than IBM had shipped a top-tier Power ISA desktop in over 12 years.

Now we have not only a Power ISA CPU that is performance-competitive with current x86_64 offerings, but an entire third-party libre workstation built around them that you can order and get shipped to your house right now. The cost of a full Talos II only seems steep until you consider how much a Xeon box in the same ballpark will run you, and the delta seems much more reasonable then. If Blackbird is successful at establishing a "low end" POWER9 machine with a more amenable price, we could see Power ISA start to become a major desktop player once again, especially as the software support situation continues to improve by leaps and bounds and people realize what a liability blackboxes like Intel Management Engine are. Even if you're not in the market for a T2 right now, you'll be more likely to have a real choice in workstations when you do. And that's good news for everybody.

Seems like IBM could have mentioned that.

Void Linux goes POWER9


A nice commit landed in Void Linux: 64-bit Power ISA support including big and little endian. Although you'll have to build it yourself and it looks like more work is needed for other packages, eventually the resulting binaries should boot on any Talos II system and the musl support is welcome for those who prefer to avoid issues with non-64-bit long double.

Related is work to get old-school 32-bit PowerPC working, so your beloved Power Mac can play too.