Showing posts from 2019


We haven't covered BSD a great deal in this blog even though I personally run NetBSD on three systems myself (two of which are in regular service), mostly because my system and I suspect the majority of the OpenPOWER install base is on Linux. However, FreeBSD 11.3 is now officially released and has fairly good support for 32-bit and 64-bit PowerPC on Power Mac hardware, so it's worth pointing out that 12.0 (and 13.0) has also been tested on the Blackbird and thus should also work on the Talos II. However, on the PowerPC wiki page -CURRENT is recommended for Blackbird, 12.0 is mandatory for OpenPOWER (thus 11.x won't work and presumably won't ever work), and X11 is currently listed "on Power8/Power9 [as] still a work in progress." Nevertheless, POWER8 systems also work, hardware support is improving and the OS offers another big-endian option for people preferring to run their systems that way, so hopefully Justin or Mark who are more versed in the FreeBSD world than I am have some comments about how well it works for others to explore.

Firefox 68 on POWER

Firefox 68 is out. I haven't had a chance to exhaustively test it on my ppc64le Talos II due to business trips and some family obligations, but on cursory testing the browser seems to function normally. Unfortunately our last minute latest workaround for (what is now clearly) a compiler bug in bug 1512162 did not make release, so you'll need to add it if you build from source; without it, some optimization levels may crash or behave adversely. We have not yet narrowed down the issue in gcc and on my last check clang still can't build the browser fully. Fortunately the fix did land on the new Extended Support Release 68, so individuals who prefer the ESR should be able to build as-is from there, and the fix also does not appear to be necessary on big-endian. Thanks to Dan Horák's usual quick work, the patch is also in the standard Fedora packages. The configurations I'm using are unchanged from Firefox 67.

DIAF, Amazon Music (and DRM)

It used to be that Amazon Music was a decent choice for playing the music you purchased. Not only did the AutoRip feature mean you had an automatic digital copy of participating CDs you purchased, playable from any web browser (I used TenFourFox for this purpose up until recently), but you still had the physical disc and discs you bought before got automatically added to your AutoRip library if Amazon got rights to do so. It was cool to watch my music library just fill in over the years from past purchases and still have the original CD if I needed it.

Well, turns out I'll need those CDs after all, because guess what Amazon Music does now?

"Amazon Music Unlimited" my pasty sculpted white butt. The message is almost intentionally misleading. What I've "disabled" in my browser is the Google Widevine EME component, because it doesn't exist for ppc64le, and while Amazon's community staff are as useless as ever that "deficiency" appears to be the real reason it won't work. Amazon, in fact, is claiming Linux on any platform isn't supported for the browser version or the dedicated client at all.

I wasn't going to take no for an answer. I used uBlock Origin to remove as many of the elements as I could. I couldn't get the blurring away easily but I was able to get into my old albums library and try to play something. It looked like it was starting, but no music issued forth. In the Browser console was this damning message:

No, you lying sack of filth. I didn't pref anything off. I didn't do anything. You did.

How did this work before? Amazon Music would say it required Flash, but it actually didn't (TenFourFox hasn't supported NPAPI plugins for years). The music files were just MP3. You could stream them or download them, and while some of the tracks were watermarked, I considered that a reasonable tradeoff for the convenience. Now it won't even let you in to download them.

I'm no Stallmanite. I could live with a compromise where music I don't own requires some sort of DRM, because I'll just preview it (at least for as long as they'll still allow it, which currently they still seem to), and I'll buy it if I want it. The problem is that Amazon has now effectively defined everything I've ever bought from them (and I have, in fact, bought a few tracks that I don't have a disc for) as "music I don't own." You can't even download them again despite Amazon's instructions because the browser client doesn't let you get there, even if you block the restraining elements. I'm not going to stop buying CDs from Amazon if they have a decent price, but I won't consider AutoRip as part of the value calculation anymore, and I certainly won't buy any form of digital music from them until this changes.

If there's going to be choices in computing, then this kind of crap has to stop. DRM isn't compatible with open source by definition. Worse, locking down a service that previously didn't enforce DRM is not only a still greater sin, but it's even potentially actionable. When DRM like Widevine is the only choice for playing content, then that means the only computers that can are the ones they control, and I wouldn't run some potentially untrustworthy blob on my Talos II anyway even if a ppc64le version were one day offered. Amazon Music can die in a fire.

Serious hash table kernel bug (CVE-2019-12817)

A while ago when I was running my Talos II in hashed page table (HPT) mode for KVM-PR purposes, I started noticing some weird kernel behaviour with recent versions of Firefox. In dmesg were weird messages like this:

[337262.237052] ida_free called for id=170 which is not allocated.
[337262.237089] WARNING: CPU: 6 PID: 12276 at lib/idr.c:519 ida_free+0x114/0x1e0

Initially these were annoying but seemed innocuous. But later on I started getting some weird lockups and I wasn't sure what was going on, so as part of an attempt to figure it out I switched the machine back into radix MMU mode (i.e., I removed disable_radix from the kernel arguments) and the problems disappeared. I reported the phenomenon and the backtraces to the good folks at OzLabs, and Michael Ellerman said he'd look into it.

Turns out the problem was a little deeper than just some weird kernel warnings. From Michael's detailed report, way back around 4.17 code was introduced to support mappings above a 512TB address in the event of a segment lookaside buffer miss. (The SLB has been a tricky devil before.) This situation might seem exotic at first glance, but such addresses are eminently possible in a 64-bit address space. The new code enabled a subtle bug: if a process allocates memory in that range with mmap(), and then forks a new process, the child erroneously maintains the parent's "context ID," a handle-like structure, to that memory mapping and both the child and parent can stomp on the same range of memory. This is obviously bad, but it gets worse when the child process exits, because all of the context IDs it had (including the one it incorrectly inherited) are now freed and thus sets up a use-after-free error where a subsequent unrelated process might get that context ID and access the original parent's space or vice versa. The kernel messages I was seeing was the kernel detecting the situation when both the child and parent exited before a subsequent process got the bogus context ID (and complaining about the double free). The hangups may well have been when the kernel didn't.

There are currently no known circulating exploits for this flaw but it's pretty clear this could be the basis of a nasty attack if a malicious sort were able to trigger such a mapping and then use it to victimize a subsequent process. All 64-bit POWER and PowerPC systems that use a 64K page size are vulnerable in HPT mode (but the Adelie Linux people can smirk a little here, because they use 4K pages, which are not vulnerable). Prior to POWER9, all systems are HPT, so this means everything is affected from the G5 on up including the PA6T and POWER4/5/6/7/8. Additionally, if you use KVM-PR on your POWER9, emulate a POWER8 guest in KVM-HV, or use KVM-PR within a KVM-HV guest, you are also vulnerable because your machine/guest must be using HPT mode. 32-bit PowerPC systems are unaffected.

Most POWER9 systems are probably using the radix MMU by default. If you aren't, then you should, at least temporarily (I haven't switched back to HPT on my own Talos yet, fortunately). dmesg will tell you in the first few lines:

[0.000000] dt-cpu-ftrs: setup for ISA 3000
[0.000000] dt-cpu-ftrs: not enabling: system-call-vectored (disabled or unsupported by kernel)
[0.000000] dt-cpu-ftrs: final cpu/mmu features = 0x0000f86f8f5fb1a7 0x3c006041
[0.000000] radix-mmu: Page sizes from device-tree:

If you see hash tables referenced instead, then you are in HPT mode, and you should remove disable_radix from your kernel arguments and restart.

If you are on a pre-POWER9 system, however, there is currently no effective mitigation, but the good news is that the patch has landed in the kernel tree. It has made it to the RC for 5.1.15, so it should enter Fedora quickly (however, my F30 system is still showing 5.1.12 as current as of 7:30pm Pacific). Update: Red Hat is tracking this as bug 1720616. Debian has an advisory page and is tracking the flaw. Ubuntu is tracking this issue as USN-4031-1 and already has updates.

Alpine Linux updated to 3.10

Alpine Linux, a lightweight distribution notable for its use of musl libc and busybox, has been updated to 3.10. Major changes in this release include ceph, a distributed object store and filesystem, and lightdm, a cross-desktop display manager, using kernel version 4.19.53. Note that Qt4, Mongodb and Truecrypt are removed from this release. Alpine supports ppc64le and should "just work" on POWER8 and POWER9 systems; you can download bootable images.

And now a bird that's not a Raptor: the H3 Falcon II

A post on the OpenPOWER blog caught my eye about the H3 Falcon II. This is not Raptor hardware; in fact, it's technically not even a computer, but rather a big rackmountable box of up to 16 GPUs. The Falcon II is PCIe 4.0 based and supports up to 31.5 GB/s bandwidth on each of its x16 GPU slots, with four host interconnects that can share them on demand. Obviously a lot of deep learning and other kinds of GPU-heavy tasks would be very interested in that much power that hopefully can be dynamically utilized as processing need requires.

POWER9 systems are well positioned to take advantage of this kind of hardware due to their prodigious I/O capacity and full support for PCIe 4.0, but although the machine is shown with Tesla V100 GPUs, their PCIe version currently "just" supports 3.0. AMD does have a PCIe 4.0 GPU, the data center-oriented Radeon Instinct MI60 and MI50, but let's not forget the Tesla has one other trick up its sleeve: NVLink 2.0, providing up to 150 GB/s bandwidth each way which is directly supported by the POWER9 as well. However, the Falcon II doesn't seem to offer NVLink.

The Falcon II is definitely an interesting unit and for POWER9-based datacenters looking at really heavily compute-bound learning tasks could be a more economical way of sharing lots of powerful GPUs between multiple nodes. It's not likely to achieve its fullest potential until PCIe 4.0 GPUs are more common and it lacks the flat-out crushing bandwidth of NVLink 2.0, but so far NVLink isn't shareable in the way this is and the Falcon II's NVMe-to-GPU link is truly innovative. That said, if you're in the kind of AI stratosphere where you would actually be cramming 16 $10,000 GPUs into one box, economy may not be the most pressing concern you'd probably have.

A couple followups to the Blackbird semi-review

Martin Kukač asked a few good clarifying questions about the Blackbird review; you can see his queries and my comments. I think it's worth clarifying that while I don't think the naked single-4 is the best Blackbird experience, especially where media is concerned, it's definitely useable as is (and certainly far faster than my long-suffering Quad G5).

For proof, Phoronix now has Blackbird benchmarks of that same basic single-4 (16 thread) system. They also tried to measure the performance impact of the Spectre mitigations, which Raptor enables on every machine they ship (both kernel and user-level), and discovered that while there is an impact on POWER9 of about 8% it is still rather less than the effect of the mitigations on Intel chips (about 18%, not counting microarchitectural mitigations required to deal with ZombieLoad et al). There's a little weirdness in the numbers in that sometimes the mitigated throughput was slightly better performing, possibly some unaccounted-for statistical artifact, but it's good to see that the price paid for better security on our systems is rather less than on x86_64. That said, if you really want to wring out that extra 8%, you can configure the protection level in the BMC based on your anticipated risk.

Finally, a big thumbs up to Red Hat, who fixed this ppc64le-specific bug in LibreOffice I reported via Dan Horák in record time.

A semi-review of the Raptor Blackbird: POWER9 on the cheap(er)

(*A big thanks to Tim Pearson at Raptor for his help with some of the technical questions, though I jealously guard my editorial integrity: this review was neither written nor approved by Raptor, and this machine was bought as a retail item without commercial consideration or discount.)

Much has been made and occasionally mocked of the Raptor Talos II's purchase price, which is hardly alone in the market even though I think you get what you pay for, but still admittedly eye-watering. (I'm saving pennies for a spare system and upgrading to a dual-8 instead of this dual-4, but that probably won't happen until I get my tax refund next year and I'm fortunate to make a decent living in a field largely unrelated to computing.) There are the usual folks who miss the point by stating that ARM and x86 machines are cheaper, and then there are the people who will miss the boat by waiting around for RISC-V, but while I think the people in the first category have their priorities mixed up they're also not wrong. The reason simply is that there are so many of them bought, sold and manufactured. No other architectures have the economy of scale of x86_64 and ARM, and therefore, no other architecture will have their price advantages for the foreseeable future. Boutique systems cost boutique bucks; the classic example is the PowerPC Amiga systems, which make it even worse by running embedded CPUs. If price or (perceived) value for dollar is your biggest concern, stop reading right now, because nothing in this review will convince you otherwise. Just don't ever complain someday when you don't have choices in computing architectures, because you did have a choice, and you chose cheap.

The point of the Blackbird is for people who either (like me) don't feel like feeding the Chipzilla monoculture further, or (like many others) prefer an alternative computing architecture that's open and fully auditable, and would prefer a smaller capital outlay. Maybe you're curious but you're not ready to make POWER9 your daily driver. Maybe you'd like to have one around as an "option" for playing around with. Maybe you have a need or interest to port software. Maybe some of your tasks would benefit from its auditability, but you don't need a full T2 for them. Or, more simply, maybe you see the value in the platform but this is all you can afford.

Now, finally, there's an existing lower-cost option. Not low cost, but lower cost: the price just plain won't be competitive with commodity systems and you shouldn't buy it with that expectation. I call this article a "semi-review" because it's not (primarily) about performance testing, benchmarks or other forms of male body part measurement; you can get that at Phoronix, and they have a Blackbird of their own they're testing. Instead, I'm here to ask two questions: how cheap can you get for a decent POWER9 system? And how good will that low-cost system be? There's no spoiler alert in saying I think this system is solid and a good value for the right kind of buyer, because really, what are you reading this blog for? I'll tell you what I paid, I'll tell you what I got for it, I'll tell you the ups and downs, and then I'll let you decide.

As with the Talos II the Blackbird is a fully open-source, auditable POWER9 system from the firmware up, but the biggest advantage of Blackbird over the T2, even more so than its price, is its size. The T2 is a hulking EATX monster and even the cut-down Lite is in the same form factor, but Blackbird is a lithe micro-ATX system and will fit in pretty much any compliant case. Cutting it down to size has some substantial impacts, though: there's a single socket only, and because only one CPU can be installed and the CPU handles directly-attached RAM and PCIe, you have fewer memory channels and PCIe lanes. That means a smaller RAM ceiling (just two DIMM slots for 256GB, vs 2TB in a loaded T2), fewer PCIe slots (a single x16 and x8 apiece versus three x16 and two x8), and of course fewer hardware threads. The 3.2/3.8GHz Sforza POWER9 sold for the Blackbird is the same CPU as the T2, so that means SMT-4 and a maximum of 88 threads with a 22-core part, but you'll only have one of them. Plus, this smaller board also has weaker power regulation, causing anything with more than 8 cores such as a 16-core or that 22-core beast to be unable to run at fullest performance (if at all).

That said, the Blackbird makes up for the limited expansion options with basic creature comforts on board, including USB 3.0, SATA, HDMI video out using the ASpeed BMC as a 2D framebuffer, 5.1 audio over analogue, S/PDIF, HDMI and Ethernet. (These devices are all blob-free except the NIC, by the way, and that last deficiency is already being worked on.) I decided in order to keep the cost really low that I'd just use the 2D BMC graphics and the on-board audio, a SATA SSD instead of NVMe like in this T2, and go with only 16GB of RAM.

I also thought about where to put it. We actually do have a need for a home theatre PC, and my wife is curious about Linux, so I decided to configure it that way and connect it into our existing DLP projection system. There's no Ethernet drop in the home theatre yet, so this machine will be wireless. Let's go shopping!

As of this writing the low-end 4-core (16 thread) Blackbird bundle starts at $1280 (all prices in US dollars). This includes the CPU and an I/O plate but does not include a heatsink or high-speed fan (HSF) assembly, which is extra and required and frankly Raptor should just build that into the price. I was fortunate that I got in on the Thanksgiving Black Friday special and got one for $1000. With the 2U heatsink, installation tool and shipping it came almost exactly to $1280 out of my pocket, so let's add another $280 to the current price for what you're likely to pay and budget $1560. I don't claim the prices below to be particular bargains; they're just what seemed decent on Amazon and your money/mileage may vary. Prices rounded up to the nearest whole dollar.

Blackbird Bundle (4-way CPU, heatsink, I/O plate and install tool, shipped to your door)
Micron MTA18ASF2G72PDZ-2G6D1 16GB DDR4-2666 ECC RDIMM
Seasonic Focus 650W 120mm Modular ATX PSU
(admittedly overkill)
SilverStone ML03B mATX Case
Samsung 860 EVO 500GB SATA III SSD
LG 14x BD-RW
Logitech K400 Wireless Combo Keyboard + Touchpad
Vonets VAP11G-300 WiFi Bridge
Arctic F8 80mm PWM case fans x 2
SSD bracket
Various random cables from my bag of crap
cheap as

That's a grand total of $2071 (I later spent an additional $22 which I'll explain in a minute) for what I would consider to be a basic system, not absolute barebones, but not generously configured either. Given that the major cost is the board itself, no matter what deals you find on similar items, you're largely guaranteed to be paying in the ballpark of two grand for a comparable loadout without a GPU.

One delivery exception and useless mailbox staff member later, I have a nice big box marked Fragile from the Raptor folks. If the NSA TAO got into this, it wasn't obvious from the outside.

Inside the big box was the 2U heatsink, the CPU (smaller white box), a 5/32" ballhead driver for cinching down the heatsink because I couldn't find where the other one had gone, and of course the Blackbird. I have serial #75, which seems high for an early order and I'm sad I missed one of the serial number 1 certificates.

Let's open up the Blackbird's box:

The DVD includes schematics, the manual, firmware builds and source code. There's also a paper info sheet, the test readout, the I/O plate, a couple cool stickers (hey, can we buy more of these OpenPOWER ones?) and, there it is, the motherboard. Other than the fact it's brand-spanking new, plus some additional markings and a couple of now-populated pin headers, at cursory glance I didn't see many changes in this Rev 1.01 board from the Rev 1.00 board I saw at SCaLE 17x, showing how mature the design was already by that stage.

The other very important thing in the box is a piece of receipt paper with red and black dot matrix printing containing the BMC factory root password. It is not 0penBmc as the manual states, and in fact nothing other than the slip of paper itself even references its existence. If you toss it out carelessly as I almost did, you won't be able to get into the BMC with anything short of a serial cable to its headers on the board. I applaud Raptor for their level of security, but it would have been nice to have been warned to watch for it, and the password on my unit had a capital O instead of a zero but the font doesn't differentiate them! (By the way, duh, I've changed the password. Don't bother writing in that you can see it.)

One item to note in particular on the board are the status LEDs, at the top left in this picture's orientation. If the case front panel does not have sufficient LEDs to display their state, and the one I bought doesn't, you may want to position this LED bank in such a way that you can see them easily. On my system they are visible through the side ventilation holes.

Let's pop the motherboard in the case, which I already prepared with the SSD, optical drive and PSU. We'll then remove (carefully) the plastic cap on the socket:

The CPU socket is a ZIF-type affair and has alignment notches so you can't put the CPU in wrong. It just drops in (though don't literally drop it, those delicate pins will bend).

The 2U heatsink is shown here. It just clears the top of the case in the mATX case I was using, but is not an active cooler, only passive. The manual mentioned an indium pad but that didn't sound necessary for the 4-core and indeed neither the 4 nor the 8 require one. The copper base with a couple small voids is shown, but doesn't seem to affect its ability to cool the chip. You don't need, and in fact should not use thermal compound, though I did polish the heat spreader with a microfibre cloth before installing the heatsink to remove fingerprints.

The side clasps grip the heatsink and should be level. Then insert your 5/32" ballhead in the top and turn clockwise. It requires a little bit of torque, but it's absolutely obvious when it's cinched down all the way because unless you're Ronda Rousey it won't turn any more.

At this point I also installed the two 80mm case fans and connected one to each of the two 4-pin PWM fan zones (if you have the heatsink-fan assembly, you would connect that instead to the zone closest to the CPU). For an HTPC we would definitely want as silent a system as possible, but Raptor doesn't recommend passive cooling of the rest of the components even with the 4-core because the fans will try to throttle down when they can and there may not be sufficient airflow. More about this in a moment.

One last gotcha is that you would think with one stick of RAM, it would go in slot A1. You would be wrong; the manual says it performs better in B1. But the single stick does work in A1. I won't tell you how I know, I just sense these things.

Anyway, let's put the lid on the case and install it in the home theatre rack. I put it on the bottom since it clears nicely there.

Connected to wall power, it took about a minute to bring the BMC up (until the rear status lights stopped pulsing). Now would be a good time to get in and change the OpenBMC default password provided on the slip of paper in the box. OpenBMC is accessible from the network on port 3, which is the one directly on top of the rear USB ports. Unfortunately, if you use a USB WiFi dongle, those ports will be powered down when the machine is, so there's no way to access it unless you set up some miniature Ethernet hardline network or plug into the serial port on the board. I suspected this would be an issue, hence the Vonets WiFi bridge which connects via Ethernet to the Blackbird and is USB-powered so I can power it independently from a wallwart. Because the BMC gets its address over DHCP, there may be a lag before it requests a lease and it may appear at varying addresses (I eventually tired of this and hardcoded a rule for its MAC address in the WiFi router). If your system will be hard-wired to Ethernet, though, there should be no problem. Note that even the Vonets devices are no panacea either because they will need separate configuration for the access point and password (I plugged it into my iBook G4 and used TenFourFox to set it up).

Once the BMC is ready, a quick press of the power button, and we have liftoff! Much as with the T2, the fans whir at full RPM on startup and then kick down at IPL as the temperature permits. Connected directly to the BMC's HDMI port, we get a graphical startup which is really quite cool. It shows all the steps, even some spurious meaningless errors which you can safely ignore. This image was displayed on the wall by my DLP projector, but the blinds were up, so apologies for the extra light.

And, about a minute and change later, here's our friend Petitboot:

Let's install the operating system. I don't know if I'll leave it on there and I'll probably experiment with some other OS choices, but I decided to install Fedora 30 on the Blackbird for review purposes so that I can compare the overall feel of the machine with my daily driver T2. (My T2 is a dual-4 with the BTO WX 7100 workstation card, 32GB of RAM and 1.5TB of NVMe flash.) So let's do that. It will also make a nice little stress test to see how it manages its own cooling.

I left it downloading Fedora Workstation and went to dinner, and came back about two hours later with the install complete but the fans now running full blast. That was not an encouraging sign.

However, that was not the biggest problem. The biggest problem was that the machine was almost unusable: besides sluggish animations, the mouse pointer skipped and the keyboard even stuttered keys, making entering passwords in GNOME a tedious and deliberate affair. This caused some, let's say, consternation on my part until it dawned on me I had not set this system up exactly like my full T2. Yes, it didn't have a discrete GPU, but it also was a direct install of Fedora Workstation rather than an install of Fedora Server that was turned into Workstation. This install was graphical turtles all the way down. That meant ... Wayland.

I opened up another VTY to tweak settings because typing into gnome-terminal was painful and error-prone, and a quick ps did indeed demonstrate the machine was in Wayland mode, which is the present default for Fedora Workstation on a graphical boot. That explained it: my T2 uses Xorg because it has a text boot and I run startx manually. I changed /etc/gdm/custom.conf to disable Wayland and restarted, this time into Xorg, and while animations were still not smooth they were much better than before. Best of all was that the keyboard and mouse pad were now working properly. If you don't have a GPU, and possibly even if you do, don't run Wayland on this machine.

Unfortunately, even with that sorted I couldn't increase the screen resolution to 1920x1080 (it was stuck at 1024x768), and audio wasn't playing through HDMI. Those could be addressed later but meanwhile I didn't want my new funbox to cook itself. Reported temperatures at idle looked like this:

Admittedly the location of the system is somewhat thermally constrained. The AV receiver doesn't run particularly hot and it's not flush on the top of the Blackbird, but mATX cases are cramped when they're loaded and the top vent is under the receiver (the side vents are where the fan mount points are). Coming from a system like the Quad G5 where temperatures over 70°C can be a sign of impending cooling system failure, this seemed worrisome. I asked Tim Pearson at Raptor about this and actually the POWER9 has a very wide temperature tolerance: 84°C as shown here falls well within its operating range and the thermal cutoff is, in his words, "well north of 100°C." This is reassuring but I would have preferred not to have a home theatre system that can also pop the popcorn while playing the movie, so I ordered a couple more fans and some splitters to take up the other two mount points ($22).

While waiting for those to arrive, the next order of business was the video, which was still stuck at 1024x768. Firefox from the Fedora repos worked fine for YouTube but only in 4:3.

After attempting to connect it directly to the DLP projector instead of through the AV receiver and getting nowhere, I started looking to see what the maximum resolution of the AST2500 BMC actually is, and bumbled into this Raptor wiki entry about getting Xorg to display in 1920x1080. Apparently you'll need to manually specify the settings for the time being because there isn't upstream support for the IT66121FN HDMI transceiver. Once this is done, though, it works:

HD playback from YouTube and Vimeo seems similar to the T2 in Firefox, maybe an occasional keyframe seek or dip in frame rate because of the smaller number of threads, but throughput is more than enough to be serviceable.

However, I couldn't say the same for VLC, which seemed strangely fill-rate limited trying to play commercial optical media. Playing both Atomic Blonde from DVD and Terminator 2 from Blu-ray in full screen 1080p generated roughly the same percentage of dropped frames in VLC's own statistics (around 6-8%). Disabling or changing interlacing settings didn't make a difference; turning down postprocessing, explicitly disabling hardware acceleration or trying different software framebuffer options for the video didn't help either. When reduced to a half-size window, however, no frames were dropped from either disc with VLC's default settings, suggesting that pushing pixels, not decoding, was hobbling playback.

This may have something to do with the fact that software LLVMpipe rendering at this resolution is a cruel, cruel joke. Xonotic, which runs at a ripping pace on my T2 with the WX 7100 card, is hobbled to a pathetic 5-10fps in software even with all the settings dialed down:

I spent a level or two getting whacked by the bots because running around and firing was a stuttery affair. Don't expect to game super hard on this either without a GPU unless you're talking about software-rendered classics. Unfortunately, it seems 1080p movie playback, at least on VLC, has similar limitations. Although mplayer doesn't seem to have any problems with full-screen scaling (I used mplayer -fs -x 1920 -y 1080 -zoom -ao alsa -afm hwac3), you have to know which title you want and it isn't as convenient or polished. I don't know why Firefox seemed to work okay and VLC didn't but I can live with that because streaming media will be this machine's primary task anyway.

Meanwhile, we still have the audio problem. My AV receiver does not have analogue 5.1 inputs, and what good is a home theatre without surround sound? The Blackbird does also offer S/PDIF, and my AV receiver has an input for that via TOSLINK, but being PCM only comes through in stereo. Tim suggested modifying /usr/bin/ on the BMC side to enable S/PDIF over HDMI, and provided a prototype script to do so. I'll post a partially working version as a gist, but my projector occasionally came up with a black screen from it and resolutions under 1920x1080 had a weird extraneous two pixels added, so I ended up reverting it and going back to TOSLINK. Apparently a reclocking step is needed per Tim which hopefully will occur in a future firmware release.

It turns out surround sound over S/PDIF is a perennial problem on Linux. The solution for me was creating a 5.1 lossy profile for libasound from this blog entry for Debian Wheezy, which more or less "just worked" on Fedora except I had to restart the machine to get it to stick. In pavucontrol I made sure that the S/PDIF profile was configured to stream AC-3 (Dolby Digital), DTS and MPEG, and having done so VLC was now able to play 5.1 surround from both Terminator 2 and Atomic Blonde with default settings. Even this didn't work absolutely flawlessly: if the audio stream was interrupted for some reason then the AV receiver went haywire and just played an annoying buzzing noise. But at least if you don't have analogue inputs on your receiver, you can still use a TOSLINK cable to get lossy 5.1.

A number of people have requested some idea of the system's power usage, so here are some rough unprofessionally obtained numbers. Remember, this is a low-end 4-core system with one RAM stick, no GPU, no PCIe cards and an SSD, so you should expect these numbers to reflect the Blackbird's minimum power usage (your loadout will not use less, and may use more). The Kill-A-Watt measured just over 3 watts with the Blackbird on standby plugged into the wall; powering it on, BMC bring-up and IPL topped out at around 60-80W, booting Fedora peaked at 127W, and idling in GNOME measured about 65W (I told you that 650W PSU I bought was overkill). Stressing the system vainly trying to play Xonotic only showed around 105W. The highest power draw I ever saw out of the Kill-A-Watt was 131W.

I installed the two new fans when they arrived and placed two fans (again, all Arctic F8 PWMs) using a splitter on each of the two zones for the full four this case will accommodate. With two fans on the CPU zone in the same environment, the cores could now be seen to visibly cool down at idle which wasn't happening before. Interestingly, the board didn't seem to be using the case fan zone much even though all four fans did indeed spool up at IPL as expected. After a few minutes letting it just sit there at the GNOME desktop, the cores dropped to 58°C and even the CPU fans gradually spun down to minimum. This wasn't silent, to be sure, though with a movie playing you'd never notice it. That said, making the machine work I could still get some 85°C-ish peaks out of it but nothing close to the thermal cutoff Tim mentioned.

In the five days I've had this so far, a couple of other trivial annoyances did pop up, though these are possibly unique to my specific circumstances. The first is that resolution switching periodically seemed to unsettle the projector, requiring some HDMI hot-replugging or even once or twice a full power down to get it to see anything. This could well be the projector, and may have absolutely nothing to do with the Blackbird, but it was still obnoxious and never happened before with the Blu-ray 3D player or the AV receiver's on-screen display. So far I can't discern an obvious pattern and this may disappear as kernel AST2500 support improves. Also, I completely cut power at the surge protector switch to protect the equipment when turning off the home theatre, but that means every time I fire it up and want to use the Blackbird I have to wait for the BMC to come back up again and then go through the startup sequence before I can even boot Fedora (about two to three minutes to a login prompt). Yes, this happens with a T2 as well, but in my case the T2 I'm typing this on is my daily driver and so it's always running anyway.

It's time to sum up, so let's answer the first of our two main questions: how cheap can you get for a decent POWER9 system? I think this is a decent POWER9 system and should meet many basic use cases as configured, but I think I've also demonstrated that it's at or near the functional minimum, and owing to the cost distribution in its particular bill of materials I don't think you can get a lot cheaper. Given that the board and CPU are about three-quarters of the total damages, there's not a whole lot more to economize on: you could cut down the RAM to 8GB or get a smaller PSU but you'd probably barely save $100, and even the SATA SSD, though not luxuriously large, wasn't all that expensive compared to a spinning disk.

In that case, let's answer the second, thornier question: how good will that low-cost system be? I'll be painfully frank and say I probably had unrealistic expectations when I chose to try to make this into a HTPC. Linux itself isn't a great choice, especially on a non-x86 platform. DVD playback on Linux is pretty much solved, but much commercial Blu-ray media is right out for lack of decryption keys, and because it's not x86 or ARM so is most DRM-ed content without closed-source black box binaries. While 5.1 mostly works over digital, the most trouble-free means of connecting it will be analogue, and I know I'm not the only person for whom that's not an option with their existing setup; the slow bring-up if you don't leave it on standby power all the time isn't a dealbreaker but is definitely annoying (I think shortening the startup time should be looked at for future lower-end designs), and at least for the time being the lack of a GPU in this loadout appears to limit the HD media options I can play comfortably. This box suffices fine for playing YouTube and Vimeo, which fortuitously will be the majority of its current tasks, and will be a nice training wheels system for my wife to experiment with Linux, work in LibreOffice and do other family computer tasks. If you choose something less fatty than GNOME you can probably wring a little more UI juice out of it, but I didn't notice much difference from my regular T2 in basic usage. More than that isn't really in the cards, and at least as-is, I felt I had to compromise a bit too much to make it into a credible media system.

As a deskside developer box, simple server or buildbot, though, even this relatively austere loadout is acceptable. The lack of a GPU isn't a major problem for such pursuits and 16GB of RAM is sufficient for even many intermediate tasks. With 16 threads and extrapolating from my dual-4 T2 which builds an optimized Firefox at -j24 in a little over a half hour, I can reasonably expect build times to be about double, which is good enough as a sidecar. It would be more than enough to serve data and media, even if (especially since) it's not playing it. In fact, because it's a secondary machine for me, I can run it full tilt since desktop responsiveness is comparatively unimportant. Plus, it's small and I don't care if I botch the installed OS, so this will likely be my test-bed machine for other distributions and hopefully the BSDs. If you want one purely as a secondary system or utility machine, then you're probably in the best position to make the most of this low-end config.

But it seems to me from my cursory observations of people waiting for their Blackbirds that they want this as a lower-cost way into the POWER9 ecosystem and to see how they like it as a primary system. In that case, let's just say it: you need to spend more than what I've spent here. I think based on this testing that to get a good experience of what POWER9 is and what it can do, you really need eight cores (currently, add about $330) and a GPU (you decide, though I stick with BTO choices since they're likely to be tested more, so that AMD WX7100 will set you back around $500 right now). I don't think there's enough threads for heavier usage with the 4-core, and gaming, media and desktop choices will expand dramatically with a discrete GPU rather than overworking poor old LLVMpipe. For that matter, 32GB wouldn't be a bad idea either. Due to what I consider the thermal constraints of such a larger spec, though (and given how constrained I found this machine to be with just 4 cores and no GPU), I would be leery of trying to get that all into an mATX case. I think you'd find it running louder and hotter than most people would like, and after my experiences so far, I wouldn't put a single-8 plus GPU in anything smaller than a regular ATX mini-tower.

With that done and dusted, now you're looking at a slightly larger footprint and a budget of about $3000 for what I would consider a system I could live with day-to-day. It won't be the same as my big T2, which because of its dual sockets has much more expansion capability (everything is NVMe and I still have room for the GPU, a RAID or SATA card, and FireWire), but I think the experience is comparable and it's still less than half of what a similarly equipped full T2 will cost. Just keep in mind if you find you like the POWER, and you might want a dual-18 or even a dual-22 someday, you won't get that on a Blackbird — you won't even get the fullest benefit of one of them. If you want dual GPUs, or lots of NVMe storage, forget it. Heck, if you just want a dual-8, forget it. What, you expected Raptor to cannibalize their own pro market? For someone like me, the big T2 would always have been my preferred option because it gives me the biggest bang and the most room to grow, but I've clearly got more money than sense and I already knew I wanted something with that level of power and flexibility. Even if they'd sold the Blackbird back then I'd still have bought the T2. But if there's no way you can spend that much today, even if you know you might in the future, then the Blackbird is what you should buy.

Don't get me wrong: even considering it didn't turn out as I planned, overall I like the Blackbird a lot, and while not a revolutionary design I think it's a strong, gutsy choice by Raptor to put their efforts into this sort of lower-end machine. It's clearly captured a lot of people's interest and I think Raptor will sell quite a few of them, which is great for the ecosystem and for the chances of having robust libre options on the desktop. I think it's certainly possible to get a lower-cost rock-bottom configuration like this to cover many people's uses, especially as a secondary computer, but I think you'll need to be a little more generous if you want this as your OpenPOWER workstation. Budget in the 8-core and a GPU and do it right, and you'll end up with a better priced option good enough to be your daily driver, yet running a performant architecture that's more open, less creepy, comparably specced and actually exists.

PowerPC to the people.

Flying the Blackbird

The Blackbird is now loaded into my home theatre rack. Just a couple tastes before the "full mini-review" later this week once I've ironed out a few other things.

The Mac Pro makes quesadillas again

It seems appropriate on the verge of the Blackbird's release that Apple would release the new "throwback" Mac Pro, back in a new smaller cheesegrater case that looks like a Power Mac G5 with an eating disorder. The "basic" eight-core system with 32GB of RAM, 256GB SSD and a Radeon Pro 580X will set you back $6000 plus tax and shipping. Funny, my eight-core Talos II with 32GB of RAM, a 500GB NVMe SSD and an AMD WX 7100 card shipped to my door cost me around $7300. Which would you rather buy? Which one would be a computer you actually, you know, own and can trust from the silicon up? Meanwhile, which one's OS is slowly moving to merge with its mobile version? Which one makes security choices for you? Do you want Raptor's T2 ... or Apple's?

Don't get me wrong: this is heaps better than the trash can, and might make a liveable system if I were still in the market for a Mac. But if you're actually considering buying one of these yet you think the Talos II is overpriced, I'm not sure what to tell you, plus you really have no excuse not to consider the Blackbird which can also offer a comparable loadout likely for less. My Blackbird arrives today after an untimely delivery exception, so I'm hoping to finish the review this week.

Void Comes Alive CD!

You don't have to be a racially confused puppet(*) to run Void Linux now; you just need a supported Power CPU and one of the brand-spanking-new Power ISA Void live ISOs with their interactive installer. As reported on Twitter (thanks reader Karl S), now that 32- and 64-bit efforts are getting unified you can run Void on pretty much any mainstream Power CPU from a G3 to a POWER9 and you can now bootstrap the system directly from the Live image instead of having to install something else first. Interestingly, while the 32-bit build will run without AltiVec, the 64-bit builds require it, meaning the only 64-bit system supported earlier than POWER6 is the G5 (POWER5 isn't). ISOs are available for any combination of 32 vs. 64 bits, big vs. little endian (on 64-bit) and musl vs. glibc. Package availability may temporarily vary based on your choice, though the goal is parity between all of them. Support for some of the unusual variant and console CPUs may come in the future.

Incidentally, got shipping notification for my Blackbird today!

(*apologies to Arrested Development fans)

Fedora 30 mini-review on the Talos II

It's upgrade time! We're Fedora users here at Floodgap Talospace because Fedora was one of the earliest distros to "just work" on the Talos II (we've run it since F28). This is not intended as a general review of Fedora 30, just to the things that are likely to matter to Talos users. There are relatively few things in F30 that are unique to Power ISA or the Talos particularly since 128-bit long double slipped to F31, but it's still an important update all the same.

Again, for those of you unfamiliar with Fedora, it is essentially the upstream for Red Hat (or Red Hat is its downstream, depending on your point of reference). It tends to incorporate new changes earlier than many distributions and is notable for having no LTS branch per se, since RHEL would in effect serve that role. Major releases come out roughly every six months and are maintained on an N+2 system; Fedora 28 will become unsupported about one month after F30 was released, which means this week. Fedora 31 is expected around the end of October 2019.

Although you can update through GNOME Software, I prefer to do it from a text boot at the command line to eliminate any variables. The steps are the same as for F29. Virtually everything should be available from the mirrors by now for ppc64le; I have a large number of packages installed and the only thing that didn't appear in the repos was my old Perl-Qt4 bindings (this required adding --allowerasing to dnf system-upgrade download --releasever=30 to remove rather than vainly attempt to update it). Remember that if you don't have your GPU's firmware loaded into Petitboot and you have the VGA disabled, you will not see the graphical installer. While the installation will proceed anyway, you will only be able to monitor it by pressing CTRL-ALT-F2 for an alternate console, logging in as root (other UIDs are locked out) and periodically issuing

dnf system-upgrade log --number=-1

which displays the log so far. The system will automatically reboot upon completion.

One improvement this time around was the black screen lockup didn't occur after the reboot as it did in F29; the system went straight through to the login prompt unattended.

As is usual for Fedora releases F30 includes an upgrade to GNOME, this time to version 3.32. This review isn't primarily about GNOME, which people have a love-hate relationship with, and I suspect the problems in this version aren't unique to the T2 even though I don't run Fedora on anything else. But this release does seem less polished and/or more troublesome than the GNOME update in F29. On the first start of GNOME, the Dock wasn't populated, items in Applications didn't show up and some extensions didn't load. Fortunately the second start fixed most of it. However, GNOME 3.32 also broke the Mac-alike theme I use, which now has weird proportions in gnome-terminal and gaps in Settings (and of course no one spends any time on documenting exactly what CSS theme selectors go with what widgets or I'd fix it myself). In addition, the UX changes are of dubious merit, particularly the new standard app icons which to my sensibilities are garish and ugly; the icons in 3.30 weren't particularly wonderful either but they were certainly easier on my eyes. Finally, GNOME Web just sits there and spins until you make the window close, so yeah, guess it'll just be Firefox now. However, I do suspect this is a Talos-specific problem; it also occurred in F29.

Ordinarily I use for windowing (a convenient startx brings me in from the text console). Since I hadn't really tried Wayland on Fedora on the T2 before, I started it up via dbus-run-session -- gnome-shell --display-server --wayland. It did seem to work fine for Firefox, GNOME Terminal and xterm, but oddly the Settings app wouldn't start, some games refused to run and window theming was inconsistent between Wayland-ready and XWayland apps. I didn't notice much of an advantage to it in terms of speed or stability, so while it's close to being a functional replacement I don't think it's quite ready for primetime. At least not for my usage, anyway.

Deepin is officially supported in F30 and is available for ppc64le from the standard Fedora repository (dnf install deepin-desktop), though GNOME hasn't made me hate it enough yet to try to hate something else.

Overall, F30 is some steps forward and some steps back, but if you're on Fedora you know the treadmill never stops. Fortunately, given that IBM owns Red Hat and Fedora is Red Hat's upstream, we can expect that Power ISA will be a well-supported platform on both Fedora and RHEL for the foreseeable future, and folks like Dan H who are more deeply embedded in the Fedora developer ecosystem help to maintain release quality. I look forward to F31 for further improvements but in the meantime F30 is still a solid release on the T2 and there's no good reason not to use it.

OpenSUSE updated to 15.1

OpenSUSE is updated to version 15.1. Among other updates are an improved graphics stack with backported later driver support, improvements to the YaST installation and configuration tool, and additional desktop refinements. OpenSUSE should "just work" on the Talos II; the ppc64le installation is available from the download Ports tab. See the release notes for more information on installation and upgrades.

Firefox 67 on POWER

Firefox 67 has been released and to my relief (though I now build smoketest builds of Firefox on ppc64le approximately weekly to find such problems) mostly builds uneventfully. It has a number of nice features, including enhanced content blocking, improved full keyboard accessibility and various performance improvements. The marquee GPU-accelerated WebRender isn't on most Linux systems yet but that's coming soon, hopefully. I haven't experimented with it yet myself but the existing GPU acceleration works fine on this Talos II with the BTO AMD WX7100 card (set layers.acceleration.force-enabled to true in about:config).

That brings up the first catch, because I did say mostly uneventfully: changes to profile handling. If you build from mozilla-release as I do and I recommend, you will end up with a "nightly release" version (assuming you don't pass --enable-release, which I advise you don't pass right now). Starting with Fx67 nightlies from any tree will try to create a new profile separate from your previous profile but the old one remains intact. You can explicitly select it from the Profile Manager (pass -P), or, if you know already which profile you want to use, you can specify it with -p (on my system the default profile is unimaginatively called default, ergo, -p default).

The second catch I haven't figured out the cause, whether it's a kernel or a Firefox bug, but periodically it will throw occasional but not infrequent warnings that look like this in dmesg (this is on a 5.0.x kernel):

[337262.237052] ida_free called for id=170 which is not allocated.
[337262.237089] WARNING: CPU: 6 PID: 12276 at lib/idr.c:519 ida_free+0x114/0x1e0

If you are on a distribution where kernel warnings get converted into notifications (like the Fedora machine I'm typing on), this can be rather obnoxious. If you are badly afflicted, you can temporarily turn them off with these instructions. I haven't found the root cause for it yet and it's hardly a great hardship, but it didn't occur in Firefox 66.

As far as the Firefox JIT for POWER9, I'm still plugging along, but other than a minor pull request to the documentation it's still 100% yours truly working on it. Of the remaining pieces the macro assembler is about 2/3rds written, leaving the low level assembler after that, and then trying to make it build. However, I'm also in the midst of a systems update for TenFourFox, which I still have a commitment to maintain in the short term, so any help will get it in your hands faster. Hopefully the commits make it clear how I'm translating the MIPS backend into POWER9, using all that 3.0B goodness (population count instructions! trailing zero count instructions! load PC in one instruction! it's an assembly language candy store!).

It's been a little while since I posted the .mozconfigs I use, so rather than direct you to old entries I'll just reproduce them here. Note that MOZ_PGO and MOZ_LTO don't seem to properly work and may generate defective binaries, thus their absence, and I explicitly pass --disable-release to an opt build because of various minor problems which hopefully we'll eventually smoke out. Adjust the number of cores as you like; this is a dual-4 system, so with 32 threads available I reserve 8 to let me still play Descent II during build runs. :)


export CC=/usr/bin/gcc
export CXX=/usr/bin/g++

mk_add_options MOZ_MAKE_FLAGS="-j24"
ac_add_options --enable-application=browser
ac_add_options --enable-optimize="-Og -mcpu=power9"
ac_add_options --enable-debug
ac_add_options --enable-linker=bfd

export GN=/usr/bin/gn # if you have it


export CC=/usr/bin/gcc
export CXX=/usr/bin/g++

mk_add_options MOZ_MAKE_FLAGS="-j24"
ac_add_options --enable-application=browser
ac_add_options --enable-optimize="-O3 -mcpu=power9"
ac_add_options --disable-release
ac_add_options --enable-linker=bfd

export GN=/usr/bin/gn # if you have it

What's missing in this picture

I've got the case (an inexpensive mATX Silverstone SST-ML03B), I've got the memory (16GB). The PSU, optical drive, wireless keyboard and WiFi should arrive next week. Now, what am I missing? Think think think!

Since the whole idea is a POWER9 system for the more price-sensitive, the trimmings cost about $500 on Amazon (minus tax and shipping) and could probably be found elsewhere for less. I also got in on the 4-core $999 Blackbird bundle special price, so with the 2U HSF and tooling that was $1090 before tax and shipping (now it would be roughly $1380) for a base outlay of about $1600. This is a nice attempt at a barebones 8-core for $1950, also apparently minus tax/shipping. Yes, I know you can get an Intel system for less, so don't even bother posting that. If price is your highest priority, you already know you're in the wrong place, but at least now price can still be a priority for what is a decent libre system regardless.

Obviously the aim for us here in the Floodgap household is to use it as an HTPC and that's how I'll be reviewing it. If you just want it as a workstation or to jam in a closet as a low-end server, you can almost certainly cut this parts list further.

ZombieLoad does not affect POWER9

If it's Tuesday, there must be yet another speculative execution attack debuting with a funny name and this Tuesday's entry is ZombieLoad. ZombieLoad works on the same conceptual basis of observable speculation flaws to exfiltrate data but implements it with a new class of Intel-specific side-channel attacks utilizing a technique the investigators termed MDS, or microarchitectural data sampling. While Spectre and Meltdown attack at the cache level, ZombieLoad targets Intel HyperThreading (HT), the company's implementation of symmetric multithreading, by trying to snoop on the processor's line fill buffers (LFBs) used to load the L1 cache itself. In this case, side-channel leakages of data are possible if the malicious process triggers certain specific and ultimately invalid loads from memory -- hence the nickname -- that require microcode assistance from the CPU; these have side-effects on the LFBs which can be observed by methods similar to Spectre by other processes sharing the same CPU core. Other internal buffers of potential value can also be sussed out by related MDS-style techniques.

Because of the limited bandwidth of the LFBs and the effectively streaming nature of the technique, an attacking process can't select arbitrary addresses and therefore can't easily read arbitrary memory. Nevertheless, targeting easily recognizable kinds of data can still make the attack feasible, even against kernelspace. For example, since URLs can be picked out of memory, this apparent proof of concept shows a separate process running on the same CPU victimizing Firefox to extract the URL as the user types it in. As the user types, the values of the individual keystrokes go through the LFB to the L1 cache, allowing the malicious process to observe the changes and extract characters. By its nature there is much less data available to the attacking process but that also means there is less data to scan, making real-time attacks like this more feasible combined with other attacks or social engineering.

However, ZombieLoad is pretty much irrelevant against POWER9 because the LFBs it attempts to monitor are specific to Intel's implementation of HyperThreading (which is true for really any other SMT implementation other than Intel's; the authors of the attack say they even tried on other SMT CPUs without success, almost certainly AMD, though it is not stated for certain that they tested on Power ISA). Even for unpatched Intel machines the actual risk from this (or even most speculative execution attacks, to be sure) is probably limited because it requires running a malicious process to do the snooping and such processes almost certainly have other, more reliable ways of pwning such machines. The decision to patch may simply come down to how much risk you're willing to tolerate: nearly every Intel chip since 2011 is apparently vulnerable and the performance impact of fixing ZombieLoad varies anywhere from Intel's rosy estimate of 3-9% to up to 40% if HT must be disabled completely.

Blackbird shipments start next week

The release of the Blackbird firmware indicates shipping is imminent and Raptor confirms on Twitter that shipments should start next week. Time to get those mATX cases!

Red Hat Enterprise Linux 8.0

The newly Big Blue Red Hat released Red Hat Enterprise Linux (RHEL) 8.0 today, based on Fedora 28. Because it's based on F28, this release of RHEL should "just work" on the Talos II (F28 was the first Fedora to support it), and mostly whatever applies to F28 applies to RHEL 8, including GNOME 3.28, Wayland-by-default and other changes. Although both big and little endian are apparently supported on the trial evaluation images, the documentation says only little-endian is supported, so it's possible not all parts of the site are in sync yet. Still, if you like your Linux corporate (and paying for it), now that Red Hat is an IBM product it's probably as corporate as you can get.

DAWR YOLO mode coming to kernel 5.2

Thanks to Michael Neuling at OzLabs who gave me the heads up and wrote up the patch. One of my pain points doing development on the POWER9 is that hardware watchpoints are disabled at the kernel level. This is because the CPU will checkstop if a watchpoint is set on cache-inhibited memory such as devices, and if a checkstop occurs will invariably bring the system down. The formal name for the special purpose register governing this feature (recall that Power ISA has three classes of registers, i.e., general purpose, floating point and special purpose) is the Data Access Watchpoint Register, or DAWR. There is no software workaround for this problem, and because a malicious local user could bring the system down without privileges by managing to provoke such a situation, setting such watchpoints via the DAWR is therefore currently disabled for safety. Unfortunately, software watchpoints are sometimes hundreds of times slower than hardware watchpoints and for certain debugging tasks are just about indispensable (such as JIT code generation).

IBM notes this issue as an erratum which implies they see it as a defect and therefore suggests it will be fixed in hardware in the future (it does not affect POWER8). Until then, Michael's patch enables "DAWR YOLO mode" for those of us (like me) who are single users on a workstation who know what we're doing, need hardware watchpoints to debug our software before the heat death of the universe, and accept the risk of system crashes. It creates a debugfs switch at /sys/kernel/debug/powerpc/dawr_enable_dangerous that enables the superuser to (mostly) freely turn access to the DAWR off and on; see the patch for more details. Fortunately this change has been finally queued for kernel version 5.2, which means I hopefully won't have to screw around with a custom kernel for much longer and is very good news for other developers in the same boat. Thanks, Michael!

A friendly FPGA reminder

Raptor is seeking beta testers for the upcoming 2.00 Talos II firmware, including a tease for users of the built-in VGA port. This brings up a question: flashing the PNOR and BMC is relatively straightforward (I've done it from my Quad G5), but flashing the FPGA requires a programmer and not all of us are handy with those -- or even have one. Do we need to do that too to try the beta?

Fortunately, Raptor's great support staff responded to my query and said (emphasis mine), "The FPGA firmware is largely independent of the BMC and PNOR. FPGA updates are only released to improve compatibility with PSUs and chassis components encountered in the wild, and at no point will a BMC nor PNOR update be released that is incompatible with the earliest FPGA revisions." That's very reassuring since I have an early T2 that is basically on the same FPGA flash it came from the factory with.

If that's the case, then, when should you update the FPGA? Raptor Support answered that too: "The only time you need to upgrade the FPGA is when you need functionality a new FPGA release provides, for example to activate the VGA disable jumper or to allow the system to boot with a different, previously problematic PSU."

Looks like I've got something to try over the weekend.

Fedora 30 released (and a big Void)

Not to be outdone by the release of Ubuntu 19.04, Fedora 30 has been released as well. We pay special attention to Fedora here at Talospace since this Talos II runs F29. As with our mini-review of F29, we will be doing a similar mini-review of F30 after a couple weeks when the package repositories should be caught up for ppc64le. Chief amongst the updates is GNOME 3.32, gcc 9, bash 5.0 and PHP 7.3; here is the full change set. One disappointment is that 128-bit long doubles did not make this release as previously scheduled and has been held over to F31, which affects building MAME with gcc (see that post for a possible workaround) and a few other things. It's not clear what caused the delay since the issue plagues relatively few packages overall, but it's just enough to be obnoxious when it does. Until then, though, watch for our mini-review once we're ready to update.

Meanwhile, if you like big ends and you cannot lie, the POWER9 Void Linux port can now boot in big-endian mode (the maintainer clarifies: with glibc). With a little bootstrapping help from Adelie Linux, now you can choose best you suits endianness which (deny can't brothers other you). The plan is to get the Void package repos for POWER9 at parity between the big and little endian versions and then let users pick what's most appropriate for their circumstances, including your choice between glibc and musl on both endiannesses. The big-endian version is also planned to have support for the PowerPC G5. For more information, the maintainer now has updated documentation.

A quick trip to IBM OzLabs

(Before we begin: this post was not sponsored, vetted or in any other way written in official collabouration with IBM; I'm writing this up strictly as a Power ISA bigot enthusiast and for no other purpose or consideration.)

Jetlagged greetings from a lovely trip now back at Floodgap Orbiting HQ in sunny southern California (this post partially composed on Air New Zealand's in-flight WiFi). Many thanks to Hugh Blemings of OpenPOWER whom I met at SCaLE 17x and suggested, since I was going to be visiting family in Australia and having a holiday in Canberra, that I head by IBM OzLabs (where he used to be manager). With the kind indulgence of Leonard Low, the current manager, it actually came together and on a warm autumn day last week my wife and I trooped down the National Circuit to the office.

OzLabs has a very long history as a Linux hacker collective and was one of the first commercial labs set up for Linux and Linux software support. In fact, it is rather infrequently reported that a visit to the Canberra Linux User Group, an indirect ancestor of OzLabs, is where the Linux Tux mascot originated (from a 1994 incident at the National Zoo & Aquarium where Linus Torvalds was actually bitten by a penguin; a somewhat inaccurate sign at the Zoo commemorates the injury). Formed at what was then the Australian division of Linuxcare in 1999, for a period of time OzLabs even provided a Linux CD burning service (in exchange for cookies, which my wife calls biscuits) along with Linux software development and support until the division shut down. Hugh was manager for part of this period along with its resurrection under IBM in 2001 as part of the IBM Linux Technology Center, which Leonard manages now.

The current OzLabs location is in an executive building and isn't purpose-built, as Leonard pointed out somewhat apologetically, but still gets the job done. Today the division concentrates primarily on Linux on Power support (including Skiboot and Petitboot, which actually originated from the Cell-based PlayStation 3) in addition to its hosted projects; in fact, Paul Mackerras, the original maintainer of PowerPC Linux, is today a senior technical staff member working on KVMPPC and very politely reviewed my trivial patches for KVM-PR a few months back.

Leonard greeted us at the entry and took us back where the magic happens.

Now, this is IBM, so there's still the corporate face. (I have a story about this: when I was an AIX sysdweeb back in the antediluvian days, we would regularly get visits from IBM salesdroids. However, a few years ago when I tried to buy my own POWER7 hardware personally, I couldn't get any VARs to take my money probably because I didn't need a service contract. I ended up settling for a lightly used POWER6 from a reseller; that box still runs Floodgap today. Now that I have an executive position in a large municipal department, though in a job unrelated to computing, upon hearing this story the IBM salesdroid servicing the municipal account gave me his card and told me to call him any time.) There are meeting rooms and a decent-sized auditorium space, where my wife was talking academia shop with Leonard while I shutterbugged.

A Thomas Watson-esque THINK mural dominates the wall (I have a THINK notepad as a gift from that salesdroid which I use for late night call notes). There's also a small display case nearby with several items, most notably the head and disk assembly from an IBM 3380 circa 1980. This assembly is a Model J with two actuators each accessing about 630MB each; the 3380's frame carried two of these assemblies for a grand total of 2.52GB in the AJ4 configuration, making the 3380 the first gigabyte storage device. The larger but slower Model K had 1890MB per actuator for roughly 7.5GB in a fully loaded AK4 frame, weighing a foot-flattening 250kg and pulling 6.6kW of power. This assembly alone weighs 32kg, so hope you had your hernia belt on while installing it. Due to its incredible rotational inertia the spindle was stopped by the equivalent of an automotive disc brake.

One disappointment: no photography inside the work area. Although I actually do hold a US security clearance, export regulations are such that pictures taken inside are not allowed, so to ease Leonard's heartburn all the pictures you see in this blog post were taken outside in the public space and I took no pictures within the secure area. However, I tapped out some copious notes and those I'll share with you.

In the prototype area Leonard showed us examples of Romulus and Witherspoon. The Romulus development reference design you should know very well by now: the Talos II is strongly based on it, and apparently the OzLabs developers like the T2 so much they're ordering more as workstations. (Maybe that's why it's currently backordered at Raptor, grr.) Witherspoon is described in Skiboot documentation as "a POWER9 system with NVLink2 attached GPUs"; it is the direct ancestor of the Monza-based AC922 used as nodes in the Summit and Sierra supercomputers (more at the end). There was also a small system with an FPGA prototype BMC under testing. Amusingly, the prototype room also had some historical items, including otherwise nondescript tower systems based on the PowerPC 604, 750 (G3) and 405, none of them Power Macs, and some workstation hardware I recognized from the beige days.

In the server room a POWER9 Zaius (a/k/a Barreleye G2) system was sprawled out on a table. This is an OpenCompute device developed jointly by Google and Rackspace as a successor to the original POWER8-based Barreleye. Although just 1U tall, the system we saw was too wide for IBM's racks, though I did rather like the removable drive bay. It takes LaGrange CPUs (more on Monza and LaGrange in a second).

We also saw the POWER8 Palmetto and Stratton prototypes in the racks, each in this SilverStone ATX case. The Palmetto design emerged as the Tyan GN70-BP010, the first customer-available OpenPOWER system; Stratton became the S821LC, with its close relative Briggs (get it?) as the S822.

Although the pamphlet I stole it from is dreadfully out of date (2008), since I couldn't photograph it you can get a small idea of the lab from this page out of IBM Australia's then-official brochure (warning: large PDF; usual disclaimers apply). While only some of the staff were there due to the Easter holidays, which was poor planning on my part, a relaxed and skilled atmosphere in the relatively open floor plan was evident. We also spotted the continuous integration display (all builds green!) and a modified xkcd that said "Petitboot" instead.

Michael Neuling, another IBM staffer, kindly provided some public chip samples to photograph and we took them outside the secure area. One of them was this POWER8 wafer which I took from a couple different angles. The two-ply "white" strip is test logic; the dies between them have six cores on a 22nm process. The Turismo POWER8 has one of these and the Murano POWER8 has two.

The POWER9 Nimbus scale-out family in OpenPOWER systems (on top of the POWER8's wafer carrier for size comparison, more or less), as elegantly hand-lettered by Mikey. Sforza is the chip we know and love in the T2 I'm typing this into; as implemented in Romulus it provides the most similarity to existing commodity designs and prioritizes PCIe, offering the most of all three (48 PCIe 4.0 lanes). LaGrange and Monza are in the larger form factors with double the memory channels of Sforza, with LaGrange also offering the biggest XBus bandwidth between processor sockets (two lanes, twice that of Monza and Sforza) and Monza the greatest OpenCAPI/NVLink throughput. Knowing this, it makes sense why Rackspace and Google went with LaGrange for Zaius, but IBM used Monza for Witherspoon/AC922 where GPU attachment mattered more.

At one point Raptor made an off-hand mention of a future LaGrange system, but so far nothing more has been heard.

Finally, a couple more items: the Top 500 certificates for Summit at Oak Ridge National Laboratory, TN and Sierra at Lawrence Livermore National Laboratory, CA, currently ranked numbers 1 and 2 as of this writing. Summit has 4,608 nodes based on the later 6-GPU AC922, each with dual 22-core Monza POWER9 CPUs and six NVIDIA Tesla V100 GPUs on a Mellanox dual-rail EDR InfiniBand network; Sierra has 4,320 nodes of an earlier revision with the same CPUs and four GPUs. Last is one of the many patents from the team, this one from 2015 honouring Andrew Bentley, a senior technology architect.

Although I couldn't show you everything, we're still very grateful we could drop by and see how the magic gets into our wonderful OpenPOWER machines. Thanks in particular to Paul, Mikey and Leonard for tolerating our silly questions and disturbing their quiet workplace, to Hugh for getting this all in motion, and all of the OzLabs inhabitants. We had a lovely time!

On our way out admiring the view from the auditorium (left: St Andrews Presbyterian Church; right: Parliament House) for the rest of our vacation, disturbed only by some officious stuffed shirt from the Attorney General's office that didn't like us photographing in a public street.