Latest Posts

Firefox 66 on POWER

Firefox 66 builds out of the box as Firefox 65 did. In addition, as part of the porting effort, I'm running occasional check builds roughly weekly to make sure we intercept issues on nightly before they get into beta and require a higher threshold of approval for patches. So far no major issues and Mozilla continues to be very amenable to getting the occasional fixes in promptly.

The biggest change in Fx66 is the number of default content processes in this release has jumped from 4 to 8. This is good news on our massively parallel multicore systems generally, but it's possible that the differing memory usage may be what tickles that harmless kernel assertion (though on the other hand, setting it back down to 4 didn't eliminate them completely). If you're on Fedora as I am or another distro that elevates kernel warnings to notifications, you'll probably want to turn off those system notifications until your distro's kernel gets the fix in it; see that article for details.

A close look at the Raptor Blackbird and what I did at So Cal Linux Expo 17

Wow, what a swag haul (big box o'breakfast cereal for scale), and what a fun day at SCaLE 17x, the 17th annual So Cal Linux Expo! I'd actually never been before and now I see why people love to go!

SCaLE is at the lovely Pasadena Convention Center, just a short drive from Floodgap Orbiting HQ in sunny rainy southern California. If you're in the greater Los Angeles area, it's pretty accessible by Metro rail, and there are lots of restaurants and things to do if you're flying in from away. I had lunch with my good buddy Bill (lately of the Linux Journal) and two of his cow-orkers at Islands Burgers just north of the Convention Center.

Even if you just buy an expo-only ticket, there are a huge number of vendor booths from big names like IBM, VMware and even Microsoft (!) all the way down to open source projects like VLC, Krita and Inkscape. Would have loved to have seen a Mozilla booth, though. Just saying. I even wore my Mozilla grey hoodie.

I even renewed my Electronic Frontier Foundation membership in support of the great work they do (make mine Titanium).

But of course the star of the show for your humble writer was our friends at the OpenPOWER Foundation, and they came ready for action with our favourite heavy-duty free computing platform:

Hugh Blemings, executive director, kindly tolerated my shutterbugging and an endless parade of retakes to get everything just right. (By the way, did you notice that Timothy Pearson from Raptor is now on the OpenPOWER board?)

He brought in tow their Debian dual-four-core Talos II, which did presentation duty, and of course a prototype Raptor Blackbird motherboard! For those of you new to the blog, this is Raptor's lower-cost way to get into the Power ISA ecosystem. So let's have a detailed look.

Here's the mATX board itself and a side view of the ports. (I'll zoom in on some items of note in a moment.) You can see the ports for USB 3.0 (two rear with additional headers for two more), 4x SATA, 5.1 analogue audio, S/PDIF digital audio, 3x GigE and HDMI. You can also see the two RAM slots and the x8 and x16 PCIe slots. My grizzled old hacker heart was warmed to see that there is still a good old fashioned serial port there too.

The 2D framebuffer is provided by the AST2500 BMC (ARM11 based with a sidecar ColdFire core), at left, routed to HDMI via an ITE IT66121FN; all three Gigabit Ethernet ports are serviced by the Broadcom BCM5719, at right.

Blackbird has some additional hardware for higher-security applications. At left are the flash chips for the BMC and boot flash with hardware write-protect switches, meaning if you can secure the case, nothing's overwriting the firmware. (I looked on my early-model T2 and can't find any such switches, so this is definitely an improvement. Update: found them, just at a slightly different relative location. Timothy Pearson in E-mail notes, however: "In practice, the current firmware stack is a lot chattier with the Flash than we'd like, so there's still some work to be done before we can roll out write protect in official form to both platforms.") Further anti-tampering security is offered by a Raptor-specific FlexVer connector (PDF).

The single POWER9 CPU socket (four or eight cores), and an interesting unlabeled port. Hmmmmmmm. (Update: Timothy Pearson in E-mail identifies it as an FSI port. "Talos II has one just like it; you can plug an FSP box into that port. The FSP boxes are proprietary and available to IBM partners only (like us), however they're also being replaced with the BMC for the most part, which speaks the same protocol and incidentally can do the same kind of debugging tasks now that the FSP boxes were traditionally used for. We will probably continue to retain the connnector, since even if someone designs an open FSI box to plug into the port it could be useful for various low level hacking (in the good sense) tasks. Note that neither the BMC nor this connector can bypass secure boot if enabled, and certainly FlexVer would immediately thrown an attestation error if anyone even tried.")

And last but by no means least, a four-core POWER9 CPU and the exposed die, which Hugh had on display. What a gorgeous bit of silicon, amirite? I'm still planning a full review of this Power-on-a-budget system when my production unit gets here hopefully in just a couple months.

Some other fun stuff:

The One Laptop Per Child handcrank! It exists! I would have killed for this back in the Give One Get One days.

Purism had a nice showing. In addition to their very impressive line of free(r) laptops, they also had a prototype of the Librem 5 libre smartphone, currently scheduled for Q3 2019. I just bought a Pixel 3 (these pictures were taken with it), or I would be buying one of these. I might anyway. I also reminded them that some of us wouldn't mind buying a non-x86 laptop. Power would be nice, but ARM would be fine too. He duly took it under advisement.

Not to be outdone, System76 had a Thelio system on display. Weird site but sexy case. It would be high on my list if I could get the case by itself and slap a Talos-style system in it. How about it, 76?

Standing guard at the entrance to the the exhibit hall was this huge Tux mosaic, made out of AMD Opterons.

It may be a Linux expo, but that doesn't mean the BSD folks can't be there too. I got a couple "RUN BSD" stickers (a la RUN DMC) for my NetBSD machines and one of the fun flashing devil horns headbands. Still looking forward to the FreeBSD port to POWER9!

There were many great open source projects there, but the photographic winner was this particular famous project:

Rock those hats, guys!

And just a tiny selection more of the many vendors, from whom I shamelessly lifted free stuff, and without whose financial support the Expo would probably not be possible:

Special shoutout to the really plush setup the folks from IBM had:

Distros in the hizzouse:

See if you can figure out which one I wrote. No, go on. I'll wait.

The GNU Project/Free Software Foundation had an appropriately chaotic-good booth, with a very important message:

Finally, a few more open source projects to close us out:

Overall I came expecting to just take a couple pics of the Blackbird and leave, and instead I ended up having a blast with all the great exhibits, vendors and free junk to clutter up my house. Next year I'll be springing for the full show and hopefully bringing my wife and a couple friends. It's a great time to be in free computing and the interest has never been higher. I won't put any jokes in about this being the year of the Linux desktop, but I think it really is the year of Power ISA being back on the desktop. It's been gone for too long and it's good to see it roaring back.

Just watch out for the robots.

Ubuntu LTS 14.04.6 available

This is mostly relevant to our 32-bit PowerPC colleagues, but along with the recent updates to Ubuntu 16 and 18 comes what will likely be the final release of Ubuntu 14.04 LTS (read the change summary). This release is primarily security-focused, mostly to deal with the APT redirect vulnerability, though there are of course other fixes. PowerPC and POWER8 users still stuck on Ubuntu 14 should strongly consider upgrading to Ubuntu 16, which still supports 32-bit PowerPC, also supports POWER8 (but not POWER9), and is still receiving fixes as Ubuntu 14.04 will reach end of life in April 2019. Meanwhile, ISO images are available.

New Talos PowerAI SKU

If you, ahem, want to see how good the POWER9 is at computer vision -- or any other kind of deep learning -- Raptor now announces a new Talos II package, the PowerAI Development System (TL2PA1). This is a T2 Lite with a 4-core CPU, 32GB RAM, 128GB NVMe flash and most notably an NVIDIA RTX 2070 GPU. Debian is pre-installed.

Wait, did you say NVIDIA? Yes, because the intention with this system is to run IBM PowerAI Vision, and an NVIDIA NVLink-capable GPU is required (and so is a trial license). That automatically wouldn't make this a very good Talos workstation due to NVIDIA's historically poor open-source support (nouveau or bust since Power isn't supported by NVIDIA's proprietary driver), and to be sure, Raptor seems to be discouraging it for that purpose ("There is no way to add OpenGL support to the proprietary driver stack ... This system is designed for GPU compute, and while a minimal 2D framebuffer is supported 3D applications will fall back to non-accelerated LLVMPipe rendering"). However, if you want lots of threads and a system to run a high-performance computer vision platform, you've now got a choice which is at least freer than a comparably configured x86 box. Base price starts at $3450 and the SKU should be shipping soon.

A programming note: I'll be wandering the exhibit halls of the Southern California Linux Expo in Pasadena on Saturday. See you at booth 429 or be less of a nerd than I am.

Linux 5.0

Linux 5.0 has been released (what used to be "4.21"). Linus is very clear that no kernel release is a feature release and the 5.0 divide is more an arbitrary numerical cutoff than anything else, but there are some important advancements in Linux 5.0 such as improved AMD GPU support, support for AMD FreeSync, file system improvements (especially to encryption performance), continued Y2038 work and various additional device support. I suspect that Talos users will find improvements to the ASpeed BMC media driver particularly relevant; my guess is this is part of the kernel support that the Blackbird is waiting on.

Probably the most notable Power ISA-specific feature in Linux 5.0 final is support for the POWER On-Chip Controller in POWER8/POWER9, exposing temperature, frequency, power usage and other sensor data through hwmon. This will likely enable Talos-family owners to get even better environmental monitoring support for their machines. Other Power-specific changes include Spectre V2 mitigations for many NXP/Freescale Power CPUs, various KVM improvements and even some improvements for P. A. Semi chips (get that X1000 out and celebrate).

Read the full changelog if you dare, or an annotated summary.

Ubuntu LTS 16.04.6 available

Ubuntu LTS 16.04.6 is available (see the change summary for more details). Although 16.04.6 cannot boot on the Talos family, it does support POWER8, and is particularly relevant to our 32-bit PowerPC friends as it is the final Ubuntu release to support that architecture officially. All Power ISA official releases of Ubuntu are Server branded and do not install a GUI by default. Installation images are hosted at the Ubuntu download site.

Raptor, help us out here on the software side

I'll preamble this with a status report on the ppc64le Firefox JIT: stalled. The stall-out is because my time is currently occupied with Mozilla bug 1512162 which has to do with the connection between JavaScript and the native representation causing debug builds to assert. I wallpapered this for Firefox 66, and that seems to be okay, but the wallpaper just seems to have moved the badness around and the issue is more fundamental. This is a more serious issue, so it's my highest priority.

Unfortunately, work on this bug is getting stymied by the fact that POWER9 has hardware watchpoints disabled in the Linux kernel. Trying to catch who's writing the faulty values is nearly impossible in an application of Firefox's complexity without it because without hardware assistance gdb has to single step through the code. When you don't know even what haystack to look for the needle in, it's slow going, as in (no exaggeration) hundreds of times slower. I left Firefox to try to initialize overnight on my 8-core Talos II in the debugger. By the morning, 8 hours later, it hadn't even launched its first thread.

The reason they are disabled is an errata which causes watchpoints set on cache-inhibited memory (such as devices) to make the CPU halt with a checkstop. Arguably some sort of fault on this is correct behaviour, but a checkstop is catastrophic; it's the equivalent of stopping your car by driving it off the road into a wall. I don't fault the PowerPC kernel maintainers for taking this interim approach because without it an unprivileged user could instantly halt the machine, even inadvertently. Even if the kernel could detect that the watchpoint was pointing at a cache-inhibited address and return an error, a tricksy user could potentially set up the watchpoint to "good" memory and then change the memory mapping.

I talked to the PowerPC kernel maintainers about this and an interim solution we're sort of agreed on is to use a debugfs entry as a one-way switch so that workstation developers like me can turn on hardware watchpoints "at our own risk." When I'm ready to debug something that requires a proper watchpoint, then I create a debugfs file and the kernel will then allow the watchpoint in hardware until the next reboot. I'm the only user on the machine, so if I screw it up, it's only my pasty Roman sculptured tuckus (and filesystem).

But this isn't going to write itself. I need to scratch my own itch, get it working, get it accepted, get it actually in a kernel (instead of schlepping forward local changes), and then finish debugging Firefox, and then finish the JIT, in addition to my day job, my work on TenFourFox and not being in trouble with my lovely wife for not emerging from the back room for hours.

So now I'm going to be a little less than nice with Raptor. Raptor has been fairly public about their support for Chromium, ostensibly because of the (IMHO irresponsible) proliferation of Electron apps, but they've been very tepid on Firefox. I understand a small company can't do everything, but they have had time to do ports of Unreal Engine and WINE (though this in and of itself is not enough to make the QEMU-WINE fusion Hangover work, which I've been also trying to tinker with and have it about 70% building). These are fun things to do and are certainly interesting, but that makes statements like this a bit galling, and statements like this a bit disingenuous.

Mozilla has a chicken-egg problem when it comes to an architecture that until very recently had a very small desktop share: its share (and share of desktop users using Firefox) would doubtlessly increase with a Firefox JIT, but the resources to expend into writing that JIT can't be justified until there is a larger share. Furthermore, it rings hollow for Raptor to ding Mozilla in that tweet about not being sufficiently open when Google until literally days ago wouldn't land the existing POWER9 Chromium work when Mozilla has been allowing POWER9 (and other PowerPC) patches into Firefox as a tier-3 for pretty much its entire existence. You can put the open-source lipstick on the Google pig as much as you like but at the end of the day, it's still Google and it's still Google's repo. Freedom involves choice. I'm not going to slam the people who did hard work on the Chromium port, because it is hard work and unfortunately Electron is a thing despite my misgivings, but I am going to slam Raptor for endorsing it at Firefox's expense.

I'm not asking Raptor to do the Firefox JIT port, though I may be soliciting help to farm it out with my reduced number of available cycles. (Right now it's based on Firefox 62, which once it works there, we'll forward-port it to trunk. More on that in a future post, but I'll probably put my current work up on Github. If you're interested in contributing, post in the comments.) I am asking Raptor to endorse the effort, however, and I am asking them to become more involved with developer-facing features to allow those of us who are working on ports to do so more productively.

As an example, developing and getting the interim debugfs switch for hardware watchpoints into the kernel would be an enormous help to me personally (and would save me a great deal of time), and would probably be very beneficial for other developers. Nearly everyone on the LinuxPPC kernel team I talked to agreed this is a big deficiency and one that is realistically implementable. It would be nice if this could proceed in parallel so I'm not blocked on doing everything myself because on a scale of 0 to even, I just can't. I'm sure there are many other developer pain points that will appear as more people start working on Talos systems, and I'd like Raptor to also treat these requests with priority and dedicate resources to worthy ones to allow more port work and development to flourish.

Let me soften a little bit in conclusion by saying Raptor has a very hard row to hoe being a small company jumpstarting an entire ecosystem. I'm being hard on them because I'm glad they exist, I intend to continue being a customer, and as a long-time Power ISA bigot I want them to succeed. But I also want to see the principles of free computing embodied in the hardware for the Talos family appropriately manifested in software. I don't see that being adequately expressed in the choices they've made so far and I'd like that to change. Developers need to be prioritized and software choice needs to be facilitated. Let's see more of that so we can see more POWER9 adoption and a brighter future for desktop computing.

OpenSUSE 15.1 Beta available, just not for us (yet)

The beta for OpenSUSE 15.1 is now available, at least for x86_64. This isn't actually news on the ppc64(le) side because the Leap releases don't currently have a Power ISA port. However, if you want to run OpenSUSE on your Talos II, you can with the Tumbleweed releases, which is their rolling-release flavour and apparently (I'm told) works quite well on the hardware. Nevertheless, hopefully as the install base grows there will be more interest in a stable Leap release for Power ISA as well.

Blackbirds to ship Q2 2019

On Twitter Raptor is reporting Blackbird shipments will start occurring in Q2 2019 instead of Q1 as previously announced. However, manufacture of the first production batch is in progress (we have one on order and will be doing a review as soon as it arrives). The ASpeed BMC, which is built-in, apparently also needs upcoming Linux kernel support to route its output over HDMI (via an ITE device) when hotplugging a display, which implies current distros compatible with the T2 may not fully work on the Blackbird without a discrete graphics card until they are also updated. We'll be watching.

assert_slb_presence aaargh_warnings_everywhere make_it_stop

We're tracking what seems to be a recent regression in Linux ppc64le (and probably big-endian as well, if we understand the actual cause) kernels from at least 4.20.5 and possibly a little earlier which throws recurrent kernel warnings to dmesg. Depending on your distro this may pass completely unnoticed except for your logs filling up a little faster, but systems that send notifications on such events may drive you up the wall (such as our Fedora 29 installation, where our testing of current Firefox trunk trips this assertion like mad). The output invariably looks like this:

[46425.991034] WARNING: CPU: 22 PID: 0 at arch/powerpc/mm/slb.c:74 assert_slb_presence+0x28/0x40
[46425.991039] WARNING: CPU: 18 PID: 0 at arch/powerpc/mm/slb.c:74 assert_slb_presence+0x28/0x40

followed by the usual debugging information. As the filename implies, this is related to the CPU's segment lookaside buffer, but failing the given assertion is otherwise harmless on the Talos. It looks like the bug has been there for a little while but at least as of 4.20.10 only occurs on CPUs that support the slbfee. instruction (POWER6 and up) and, if our understanding is correct, only on testing effective addresses with a particular bit set. If so, this patch should fix it, but there is no ETA.

In the meantime, if you're badly affected, one way to get the messages to temporarily quiet might be to twiddle your console logging level settings; see man klogctl for how this works. Alternatively, on a Red Hat-type system like ours (Fedora, CentOS, etc.), the notifications come from ABRT, so killall abrt-applet will temporarily quell the warnings (/usr/bin/abrt-applet --gapplication-service & to restart).

Ubuntu LTS 18.04.2 available

An updated release of the long-term support Ubuntu 18 (Bionic Beaver) is now available for ppc64el. Read the full changelog for 18.04.2. As with prior releases, Ubuntu 18 should "just work" on the Talos II. All Power ISA official releases of Ubuntu are Server branded and do not install a GUI by default.