Posts

Latest Posts

One big happy Void


The PowerPC Void Linux project has officially merged its 32-bit and 64-bit Power offerings, though to be fair this was expected for awhile and just makes good sense. Meanwhile, substantial progress is being made on the ports and it looks like most packages are buildable, but actual package availability for the big-endian (32-bit and 64-bit) and musl flavours still lags ppc64le at least right now, so that G5 under your desk may have to wait a bit. Live CDs are still available.

OCC is the sound you make when throttled


Back from distant climes to find an interesting tweet from Raptor relating to the POWER9 OCC. The OCC, or On-Chip Controller, monitors power usage and thermal stability, and can surface this information to the kernel via cpufreq. Raptor is asking users who get throttling warnings in dmesg to report them, though I haven't seen any such issues on my thermally constrained Blackbird or on this cool-running Talos II, and it's not clear how widespread the issue actually is.

Meanwhile, users who get weird OCC-related crashes when the POWER9 is in a stop state are encouraged to upgrade to the latest firmware release candidate to pick up this fix. This apparently is being triggered by recent kernel versions that enable deep power saving modes.

FreeBSD on POWER


We haven't covered BSD a great deal in this blog even though I personally run NetBSD on three systems myself (two of which are in regular service), mostly because my system and I suspect the majority of the OpenPOWER install base is on Linux. However, FreeBSD 11.3 is now officially released and has fairly good support for 32-bit and 64-bit PowerPC on Power Mac hardware, so it's worth pointing out that 12.0 (and 13.0) has also been tested on the Blackbird and thus should also work on the Talos II. However, on the PowerPC wiki page -CURRENT is recommended for Blackbird, 12.0 is mandatory for OpenPOWER (thus 11.x won't work and presumably won't ever work), and X11 is currently listed "on Power8/Power9 [as] still a work in progress." Nevertheless, POWER8 systems also work, hardware support is improving and the OS offers another big-endian option for people preferring to run their systems that way, so hopefully Justin or Mark who are more versed in the FreeBSD world than I am have some comments about how well it works for others to explore.

Firefox 68 on POWER


Firefox 68 is out. I haven't had a chance to exhaustively test it on my ppc64le Talos II due to business trips and some family obligations, but on cursory testing the browser seems to function normally. Unfortunately our last minute latest workaround for (what is now clearly) a compiler bug in bug 1512162 did not make release, so you'll need to add it if you build from source; without it, some optimization levels may crash or behave adversely. We have not yet narrowed down the issue in gcc and on my last check clang still can't build the browser fully. Fortunately the fix did land on the new Extended Support Release 68, so individuals who prefer the ESR should be able to build as-is from there, and the fix also does not appear to be necessary on big-endian. Thanks to Dan Horák's usual quick work, the patch is also in the standard Fedora packages. The configurations I'm using are unchanged from Firefox 67.

DIAF, Amazon Music (and DRM)


It used to be that Amazon Music was a decent choice for playing the music you purchased. Not only did the AutoRip feature mean you had an automatic digital copy of participating CDs you purchased, playable from any web browser (I used TenFourFox for this purpose up until recently), but you still had the physical disc and discs you bought before got automatically added to your AutoRip library if Amazon got rights to do so. It was cool to watch my music library just fill in over the years from past purchases and still have the original CD if I needed it.

Well, turns out I'll need those CDs after all, because guess what Amazon Music does now?

"Amazon Music Unlimited" my pasty sculpted white butt. The message is almost intentionally misleading. What I've "disabled" in my browser is the Google Widevine EME component, because it doesn't exist for ppc64le, and while Amazon's community staff are as useless as ever that "deficiency" appears to be the real reason it won't work. Amazon, in fact, is claiming Linux on any platform isn't supported for the browser version or the dedicated client at all.

I wasn't going to take no for an answer. I used uBlock Origin to remove as many of the elements as I could. I couldn't get the blurring away easily but I was able to get into my old albums library and try to play something. It looked like it was starting, but no music issued forth. In the Browser console was this damning message:

No, you lying sack of filth. I didn't pref anything off. I didn't do anything. You did.

How did this work before? Amazon Music would say it required Flash, but it actually didn't (TenFourFox hasn't supported NPAPI plugins for years). The music files were just MP3. You could stream them or download them, and while some of the tracks were watermarked, I considered that a reasonable tradeoff for the convenience. Now it won't even let you in to download them.

I'm no Stallmanite. I could live with a compromise where music I don't own requires some sort of DRM, because I'll just preview it (at least for as long as they'll still allow it, which currently they still seem to), and I'll buy it if I want it. The problem is that Amazon has now effectively defined everything I've ever bought from them (and I have, in fact, bought a few tracks that I don't have a disc for) as "music I don't own." You can't even download them again despite Amazon's instructions because the browser client doesn't let you get there, even if you block the restraining elements. I'm not going to stop buying CDs from Amazon if they have a decent price, but I won't consider AutoRip as part of the value calculation anymore, and I certainly won't buy any form of digital music from them until this changes.

If there's going to be choices in computing, then this kind of crap has to stop. DRM isn't compatible with open source by definition. Worse, locking down a service that previously didn't enforce DRM is not only a still greater sin, but it's even potentially actionable. When DRM like Widevine is the only choice for playing content, then that means the only computers that can are the ones they control, and I wouldn't run some potentially untrustworthy blob on my Talos II anyway even if a ppc64le version were one day offered. Amazon Music can die in a fire.

Serious hash table kernel bug (CVE-2019-12817)


A while ago when I was running my Talos II in hashed page table (HPT) mode for KVM-PR purposes, I started noticing some weird kernel behaviour with recent versions of Firefox. In dmesg were weird messages like this:

[337262.237052] ida_free called for id=170 which is not allocated.
[337262.237089] WARNING: CPU: 6 PID: 12276 at lib/idr.c:519 ida_free+0x114/0x1e0

Initially these were annoying but seemed innocuous. But later on I started getting some weird lockups and I wasn't sure what was going on, so as part of an attempt to figure it out I switched the machine back into radix MMU mode (i.e., I removed disable_radix from the kernel arguments) and the problems disappeared. I reported the phenomenon and the backtraces to the good folks at OzLabs, and Michael Ellerman said he'd look into it.

Turns out the problem was a little deeper than just some weird kernel warnings. From Michael's detailed report, way back around 4.17 code was introduced to support mappings above a 512TB address in the event of a segment lookaside buffer miss. (The SLB has been a tricky devil before.) This situation might seem exotic at first glance, but such addresses are eminently possible in a 64-bit address space. The new code enabled a subtle bug: if a process allocates memory in that range with mmap(), and then forks a new process, the child erroneously maintains the parent's "context ID," a handle-like structure, to that memory mapping and both the child and parent can stomp on the same range of memory. This is obviously bad, but it gets worse when the child process exits, because all of the context IDs it had (including the one it incorrectly inherited) are now freed and thus sets up a use-after-free error where a subsequent unrelated process might get that context ID and access the original parent's space or vice versa. The kernel messages I was seeing was the kernel detecting the situation when both the child and parent exited before a subsequent process got the bogus context ID (and complaining about the double free). The hangups may well have been when the kernel didn't.

There are currently no known circulating exploits for this flaw but it's pretty clear this could be the basis of a nasty attack if a malicious sort were able to trigger such a mapping and then use it to victimize a subsequent process. All 64-bit POWER and PowerPC systems that use a 64K page size are vulnerable in HPT mode (but the Adelie Linux people can smirk a little here, because they use 4K pages, which are not vulnerable). Prior to POWER9, all systems are HPT, so this means everything is affected from the G5 on up including the PA6T and POWER4/5/6/7/8. Additionally, if you use KVM-PR on your POWER9, emulate a POWER8 guest in KVM-HV, or use KVM-PR within a KVM-HV guest, you are also vulnerable because your machine/guest must be using HPT mode. 32-bit PowerPC systems are unaffected.

Most POWER9 systems are probably using the radix MMU by default. If you aren't, then you should, at least temporarily (I haven't switched back to HPT on my own Talos yet, fortunately). dmesg will tell you in the first few lines:

[0.000000] dt-cpu-ftrs: setup for ISA 3000
[0.000000] dt-cpu-ftrs: not enabling: system-call-vectored (disabled or unsupported by kernel)
[0.000000] dt-cpu-ftrs: final cpu/mmu features = 0x0000f86f8f5fb1a7 0x3c006041
[0.000000] radix-mmu: Page sizes from device-tree:

If you see hash tables referenced instead, then you are in HPT mode, and you should remove disable_radix from your kernel arguments and restart.

If you are on a pre-POWER9 system, however, there is currently no effective mitigation, but the good news is that the patch has landed in the kernel tree. It has made it to the RC for 5.1.15, so it should enter Fedora quickly (however, my F30 system is still showing 5.1.12 as current as of 7:30pm Pacific). Update: Red Hat is tracking this as bug 1720616. Debian has an advisory page and is tracking the flaw. Ubuntu is tracking this issue as USN-4031-1 and already has updates.

Alpine Linux updated to 3.10


Alpine Linux, a lightweight distribution notable for its use of musl libc and busybox, has been updated to 3.10. Major changes in this release include ceph, a distributed object store and filesystem, and lightdm, a cross-desktop display manager, using kernel version 4.19.53. Note that Qt4, Mongodb and Truecrypt are removed from this release. Alpine supports ppc64le and should "just work" on POWER8 and POWER9 systems; you can download bootable images.

And now a bird that's not a Raptor: the H3 Falcon II


A post on the OpenPOWER blog caught my eye about the H3 Falcon II. This is not Raptor hardware; in fact, it's technically not even a computer, but rather a big rackmountable box of up to 16 GPUs. The Falcon II is PCIe 4.0 based and supports up to 31.5 GB/s bandwidth on each of its x16 GPU slots, with four host interconnects that can share them on demand. Obviously a lot of deep learning and other kinds of GPU-heavy tasks would be very interested in that much power that hopefully can be dynamically utilized as processing need requires.

POWER9 systems are well positioned to take advantage of this kind of hardware due to their prodigious I/O capacity and full support for PCIe 4.0, but although the machine is shown with Tesla V100 GPUs, their PCIe version currently "just" supports 3.0. AMD does have a PCIe 4.0 GPU, the data center-oriented Radeon Instinct MI60 and MI50, but let's not forget the Tesla has one other trick up its sleeve: NVLink 2.0, providing up to 150 GB/s bandwidth each way which is directly supported by the POWER9 as well. However, the Falcon II doesn't seem to offer NVLink.

The Falcon II is definitely an interesting unit and for POWER9-based datacenters looking at really heavily compute-bound learning tasks could be a more economical way of sharing lots of powerful GPUs between multiple nodes. It's not likely to achieve its fullest potential until PCIe 4.0 GPUs are more common and it lacks the flat-out crushing bandwidth of NVLink 2.0, but so far NVLink isn't shareable in the way this is and the Falcon II's NVMe-to-GPU link is truly innovative. That said, if you're in the kind of AI stratosphere where you would actually be cramming 16 $10,000 GPUs into one box, economy may not be the most pressing concern you'd probably have.

A couple followups to the Blackbird semi-review


Martin Kukač asked a few good clarifying questions about the Blackbird review; you can see his queries and my comments. I think it's worth clarifying that while I don't think the naked single-4 is the best Blackbird experience, especially where media is concerned, it's definitely useable as is (and certainly far faster than my long-suffering Quad G5).

For proof, Phoronix now has Blackbird benchmarks of that same basic single-4 (16 thread) system. They also tried to measure the performance impact of the Spectre mitigations, which Raptor enables on every machine they ship (both kernel and user-level), and discovered that while there is an impact on POWER9 of about 8% it is still rather less than the effect of the mitigations on Intel chips (about 18%, not counting microarchitectural mitigations required to deal with ZombieLoad et al). There's a little weirdness in the numbers in that sometimes the mitigated throughput was slightly better performing, possibly some unaccounted-for statistical artifact, but it's good to see that the price paid for better security on our systems is rather less than on x86_64. That said, if you really want to wring out that extra 8%, you can configure the protection level in the BMC based on your anticipated risk.

Finally, a big thumbs up to Red Hat, who fixed this ppc64le-specific bug in LibreOffice I reported via Dan Horák in record time.

A semi-review of the Raptor Blackbird: POWER9 on the cheap(er)


(*A big thanks to Tim Pearson at Raptor for his help with some of the technical questions, though I jealously guard my editorial integrity: this review was neither written nor approved by Raptor, and this machine was bought as a retail item without commercial consideration or discount.)

Much has been made and occasionally mocked of the Raptor Talos II's purchase price, which is hardly alone in the market even though I think you get what you pay for, but still admittedly eye-watering. (I'm saving pennies for a spare system and upgrading to a dual-8 instead of this dual-4, but that probably won't happen until I get my tax refund next year and I'm fortunate to make a decent living in a field largely unrelated to computing.) There are the usual folks who miss the point by stating that ARM and x86 machines are cheaper, and then there are the people who will miss the boat by waiting around for RISC-V, but while I think the people in the first category have their priorities mixed up they're also not wrong. The reason simply is that there are so many of them bought, sold and manufactured. No other architectures have the economy of scale of x86_64 and ARM, and therefore, no other architecture will have their price advantages for the foreseeable future. Boutique systems cost boutique bucks; the classic example is the PowerPC Amiga systems, which make it even worse by running embedded CPUs. If price or (perceived) value for dollar is your biggest concern, stop reading right now, because nothing in this review will convince you otherwise. Just don't ever complain someday when you don't have choices in computing architectures, because you did have a choice, and you chose cheap.

The point of the Blackbird is for people who either (like me) don't feel like feeding the Chipzilla monoculture further, or (like many others) prefer an alternative computing architecture that's open and fully auditable, and would prefer a smaller capital outlay. Maybe you're curious but you're not ready to make POWER9 your daily driver. Maybe you'd like to have one around as an "option" for playing around with. Maybe you have a need or interest to port software. Maybe some of your tasks would benefit from its auditability, but you don't need a full T2 for them. Or, more simply, maybe you see the value in the platform but this is all you can afford.

Now, finally, there's an existing lower-cost option. Not low cost, but lower cost: the price just plain won't be competitive with commodity systems and you shouldn't buy it with that expectation. I call this article a "semi-review" because it's not (primarily) about performance testing, benchmarks or other forms of male body part measurement; you can get that at Phoronix, and they have a Blackbird of their own they're testing. Instead, I'm here to ask two questions: how cheap can you get for a decent POWER9 system? And how good will that low-cost system be? There's no spoiler alert in saying I think this system is solid and a good value for the right kind of buyer, because really, what are you reading this blog for? I'll tell you what I paid, I'll tell you what I got for it, I'll tell you the ups and downs, and then I'll let you decide.

As with the Talos II the Blackbird is a fully open-source, auditable POWER9 system from the firmware up, but the biggest advantage of Blackbird over the T2, even more so than its price, is its size. The T2 is a hulking EATX monster and even the cut-down Lite is in the same form factor, but Blackbird is a lithe micro-ATX system and will fit in pretty much any compliant case. Cutting it down to size has some substantial impacts, though: there's a single socket only, and because only one CPU can be installed and the CPU handles directly-attached RAM and PCIe, you have fewer memory channels and PCIe lanes. That means a smaller RAM ceiling (just two DIMM slots for 256GB, vs 2TB in a loaded T2), fewer PCIe slots (a single x16 and x8 apiece versus three x16 and two x8), and of course fewer hardware threads. The 3.2/3.8GHz Sforza POWER9 sold for the Blackbird is the same CPU as the T2, so that means SMT-4 and a maximum of 88 threads with a 22-core part, but you'll only have one of them. Plus, this smaller board also has weaker power regulation, causing anything with more than 8 cores such as a 16-core or that 22-core beast to be unable to run at fullest performance (if at all).

That said, the Blackbird makes up for the limited expansion options with basic creature comforts on board, including USB 3.0, SATA, HDMI video out using the ASpeed BMC as a 2D framebuffer, 5.1 audio over analogue, S/PDIF, HDMI and Ethernet. (These devices are all blob-free except the NIC, by the way, and that last deficiency is already being worked on.) I decided in order to keep the cost really low that I'd just use the 2D BMC graphics and the on-board audio, a SATA SSD instead of NVMe like in this T2, and go with only 16GB of RAM.

I also thought about where to put it. We actually do have a need for a home theatre PC, and my wife is curious about Linux, so I decided to configure it that way and connect it into our existing DLP projection system. There's no Ethernet drop in the home theatre yet, so this machine will be wireless. Let's go shopping!

As of this writing the low-end 4-core (16 thread) Blackbird bundle starts at $1280 (all prices in US dollars). This includes the CPU and an I/O plate but does not include a heatsink or high-speed fan (HSF) assembly, which is extra and required and frankly Raptor should just build that into the price. I was fortunate that I got in on the Thanksgiving Black Friday special and got one for $1000. With the 2U heatsink, installation tool and shipping it came almost exactly to $1280 out of my pocket, so let's add another $280 to the current price for what you're likely to pay and budget $1560. I don't claim the prices below to be particular bargains; they're just what seemed decent on Amazon and your money/mileage may vary. Prices rounded up to the nearest whole dollar.

Blackbird Bundle (4-way CPU, heatsink, I/O plate and install tool, shipped to your door)
$1560
Micron MTA18ASF2G72PDZ-2G6D1 16GB DDR4-2666 ECC RDIMM
$121
Seasonic Focus 650W 120mm Modular ATX PSU
(admittedly overkill)
$94
SilverStone ML03B mATX Case
$88
Samsung 860 EVO 500GB SATA III SSD
$80
LG 14x BD-RW
$60
Logitech K400 Wireless Combo Keyboard + Touchpad
$25
Vonets VAP11G-300 WiFi Bridge
$20
Arctic F8 80mm PWM case fans x 2
$15
SSD bracket
$8
Various random cables from my bag of crap
cheap as

That's a grand total of $2071 (I later spent an additional $22 which I'll explain in a minute) for what I would consider to be a basic system, not absolute barebones, but not generously configured either. Given that the major cost is the board itself, no matter what deals you find on similar items, you're largely guaranteed to be paying in the ballpark of two grand for a comparable loadout without a GPU.

One delivery exception and useless mailbox staff member later, I have a nice big box marked Fragile from the Raptor folks. If the NSA TAO got into this, it wasn't obvious from the outside.

Inside the big box was the 2U heatsink, the CPU (smaller white box), a 5/32" ballhead driver for cinching down the heatsink because I couldn't find where the other one had gone, and of course the Blackbird. I have serial #75, which seems high for an early order and I'm sad I missed one of the serial number 1 certificates.

Let's open up the Blackbird's box:

The DVD includes schematics, the manual, firmware builds and source code. There's also a paper info sheet, the test readout, the I/O plate, a couple cool stickers (hey, can we buy more of these OpenPOWER ones?) and, there it is, the motherboard. Other than the fact it's brand-spanking new, plus some additional markings and a couple of now-populated pin headers, at cursory glance I didn't see many changes in this Rev 1.01 board from the Rev 1.00 board I saw at SCaLE 17x, showing how mature the design was already by that stage.

The other very important thing in the box is a piece of receipt paper with red and black dot matrix printing containing the BMC factory root password. It is not 0penBmc as the manual states, and in fact nothing other than the slip of paper itself even references its existence. If you toss it out carelessly as I almost did, you won't be able to get into the BMC with anything short of a serial cable to its headers on the board. I applaud Raptor for their level of security, but it would have been nice to have been warned to watch for it, and the password on my unit had a capital O instead of a zero but the font doesn't differentiate them! (By the way, duh, I've changed the password. Don't bother writing in that you can see it.)

One item to note in particular on the board are the status LEDs, at the top left in this picture's orientation. If the case front panel does not have sufficient LEDs to display their state, and the one I bought doesn't, you may want to position this LED bank in such a way that you can see them easily. On my system they are visible through the side ventilation holes.

Let's pop the motherboard in the case, which I already prepared with the SSD, optical drive and PSU. We'll then remove (carefully) the plastic cap on the socket:

The CPU socket is a ZIF-type affair and has alignment notches so you can't put the CPU in wrong. It just drops in (though don't literally drop it, those delicate pins will bend).

The 2U heatsink is shown here. It just clears the top of the case in the mATX case I was using, but is not an active cooler, only passive. The manual mentioned an indium pad but that didn't sound necessary for the 4-core and indeed neither the 4 nor the 8 require one. The copper base with a couple small voids is shown, but doesn't seem to affect its ability to cool the chip. You don't need, and in fact should not use thermal compound, though I did polish the heat spreader with a microfibre cloth before installing the heatsink to remove fingerprints.

The side clasps grip the heatsink and should be level. Then insert your 5/32" ballhead in the top and turn clockwise. It requires a little bit of torque, but it's absolutely obvious when it's cinched down all the way because unless you're Ronda Rousey it won't turn any more.

At this point I also installed the two 80mm case fans and connected one to each of the two 4-pin PWM fan zones (if you have the heatsink-fan assembly, you would connect that instead to the zone closest to the CPU). For an HTPC we would definitely want as silent a system as possible, but Raptor doesn't recommend passive cooling of the rest of the components even with the 4-core because the fans will try to throttle down when they can and there may not be sufficient airflow. More about this in a moment.

One last gotcha is that you would think with one stick of RAM, it would go in slot A1. You would be wrong; the manual says it performs better in B1. But the single stick does work in A1. I won't tell you how I know, I just sense these things.

Anyway, let's put the lid on the case and install it in the home theatre rack. I put it on the bottom since it clears nicely there.

Connected to wall power, it took about a minute to bring the BMC up (until the rear status lights stopped pulsing). Now would be a good time to get in and change the OpenBMC default password provided on the slip of paper in the box. OpenBMC is accessible from the network on port 3, which is the one directly on top of the rear USB ports. Unfortunately, if you use a USB WiFi dongle, those ports will be powered down when the machine is, so there's no way to access it unless you set up some miniature Ethernet hardline network or plug into the serial port on the board. I suspected this would be an issue, hence the Vonets WiFi bridge which connects via Ethernet to the Blackbird and is USB-powered so I can power it independently from a wallwart. Because the BMC gets its address over DHCP, there may be a lag before it requests a lease and it may appear at varying addresses (I eventually tired of this and hardcoded a rule for its MAC address in the WiFi router). If your system will be hard-wired to Ethernet, though, there should be no problem. Note that even the Vonets devices are no panacea either because they will need separate configuration for the access point and password (I plugged it into my iBook G4 and used TenFourFox to set it up).

Once the BMC is ready, a quick press of the power button, and we have liftoff! Much as with the T2, the fans whir at full RPM on startup and then kick down at IPL as the temperature permits. Connected directly to the BMC's HDMI port, we get a graphical startup which is really quite cool. It shows all the steps, even some spurious meaningless errors which you can safely ignore. This image was displayed on the wall by my DLP projector, but the blinds were up, so apologies for the extra light.

And, about a minute and change later, here's our friend Petitboot:

Let's install the operating system. I don't know if I'll leave it on there and I'll probably experiment with some other OS choices, but I decided to install Fedora 30 on the Blackbird for review purposes so that I can compare the overall feel of the machine with my daily driver T2. (My T2 is a dual-4 with the BTO WX 7100 workstation card, 32GB of RAM and 1.5TB of NVMe flash.) So let's do that. It will also make a nice little stress test to see how it manages its own cooling.

I left it downloading Fedora Workstation and went to dinner, and came back about two hours later with the install complete but the fans now running full blast. That was not an encouraging sign.

However, that was not the biggest problem. The biggest problem was that the machine was almost unusable: besides sluggish animations, the mouse pointer skipped and the keyboard even stuttered keys, making entering passwords in GNOME a tedious and deliberate affair. This caused some, let's say, consternation on my part until it dawned on me I had not set this system up exactly like my full T2. Yes, it didn't have a discrete GPU, but it also was a direct install of Fedora Workstation rather than an install of Fedora Server that was turned into Workstation. This install was graphical turtles all the way down. That meant ... Wayland.

I opened up another VTY to tweak settings because typing into gnome-terminal was painful and error-prone, and a quick ps did indeed demonstrate the machine was in Wayland mode, which is the present default for Fedora Workstation on a graphical boot. That explained it: my T2 uses Xorg because it has a text boot and I run startx manually. I changed /etc/gdm/custom.conf to disable Wayland and restarted, this time into Xorg, and while animations were still not smooth they were much better than before. Best of all was that the keyboard and mouse pad were now working properly. If you don't have a GPU, and possibly even if you do, don't run Wayland on this machine.

Unfortunately, even with that sorted I couldn't increase the screen resolution to 1920x1080 (it was stuck at 1024x768), and audio wasn't playing through HDMI. Those could be addressed later but meanwhile I didn't want my new funbox to cook itself. Reported temperatures at idle looked like this:

Admittedly the location of the system is somewhat thermally constrained. The AV receiver doesn't run particularly hot and it's not flush on the top of the Blackbird, but mATX cases are cramped when they're loaded and the top vent is under the receiver (the side vents are where the fan mount points are). Coming from a system like the Quad G5 where temperatures over 70°C can be a sign of impending cooling system failure, this seemed worrisome. I asked Tim Pearson at Raptor about this and actually the POWER9 has a very wide temperature tolerance: 84°C as shown here falls well within its operating range and the thermal cutoff is, in his words, "well north of 100°C." This is reassuring but I would have preferred not to have a home theatre system that can also pop the popcorn while playing the movie, so I ordered a couple more fans and some splitters to take up the other two mount points ($22).

While waiting for those to arrive, the next order of business was the video, which was still stuck at 1024x768. Firefox from the Fedora repos worked fine for YouTube but only in 4:3.

After attempting to connect it directly to the DLP projector instead of through the AV receiver and getting nowhere, I started looking to see what the maximum resolution of the AST2500 BMC actually is, and bumbled into this Raptor wiki entry about getting Xorg to display in 1920x1080. Apparently you'll need to manually specify the settings for the time being because there isn't upstream support for the IT66121FN HDMI transceiver. Once this is done, though, it works:

HD playback from YouTube and Vimeo seems similar to the T2 in Firefox, maybe an occasional keyframe seek or dip in frame rate because of the smaller number of threads, but throughput is more than enough to be serviceable.

However, I couldn't say the same for VLC, which seemed strangely fill-rate limited trying to play commercial optical media. Playing both Atomic Blonde from DVD and Terminator 2 from Blu-ray in full screen 1080p generated roughly the same percentage of dropped frames in VLC's own statistics (around 6-8%). Disabling or changing interlacing settings didn't make a difference; turning down postprocessing, explicitly disabling hardware acceleration or trying different software framebuffer options for the video didn't help either. When reduced to a half-size window, however, no frames were dropped from either disc with VLC's default settings, suggesting that pushing pixels, not decoding, was hobbling playback.

This may have something to do with the fact that software LLVMpipe rendering at this resolution is a cruel, cruel joke. Xonotic, which runs at a ripping pace on my T2 with the WX 7100 card, is hobbled to a pathetic 5-10fps in software even with all the settings dialed down:

I spent a level or two getting whacked by the bots because running around and firing was a stuttery affair. Don't expect to game super hard on this either without a GPU unless you're talking about software-rendered classics. Unfortunately, it seems 1080p movie playback, at least on VLC, has similar limitations. Although mplayer doesn't seem to have any problems with full-screen scaling (I used mplayer -fs -x 1920 -y 1080 -zoom -ao alsa -afm hwac3), you have to know which title you want and it isn't as convenient or polished. I don't know why Firefox seemed to work okay and VLC didn't but I can live with that because streaming media will be this machine's primary task anyway.

Meanwhile, we still have the audio problem. My AV receiver does not have analogue 5.1 inputs, and what good is a home theatre without surround sound? The Blackbird does also offer S/PDIF, and my AV receiver has an input for that via TOSLINK, but being PCM only comes through in stereo. Tim suggested modifying /usr/bin/lvds.sh on the BMC side to enable S/PDIF over HDMI, and provided a prototype script to do so. I'll post a partially working version as a gist, but my projector occasionally came up with a black screen from it and resolutions under 1920x1080 had a weird extraneous two pixels added, so I ended up reverting it and going back to TOSLINK. Apparently a reclocking step is needed per Tim which hopefully will occur in a future firmware release.

It turns out surround sound over S/PDIF is a perennial problem on Linux. The solution for me was creating a 5.1 lossy profile for libasound from this blog entry for Debian Wheezy, which more or less "just worked" on Fedora except I had to restart the machine to get it to stick. In pavucontrol I made sure that the S/PDIF profile was configured to stream AC-3 (Dolby Digital), DTS and MPEG, and having done so VLC was now able to play 5.1 surround from both Terminator 2 and Atomic Blonde with default settings. Even this didn't work absolutely flawlessly: if the audio stream was interrupted for some reason then the AV receiver went haywire and just played an annoying buzzing noise. But at least if you don't have analogue inputs on your receiver, you can still use a TOSLINK cable to get lossy 5.1.

A number of people have requested some idea of the system's power usage, so here are some rough unprofessionally obtained numbers. Remember, this is a low-end 4-core system with one RAM stick, no GPU, no PCIe cards and an SSD, so you should expect these numbers to reflect the Blackbird's minimum power usage (your loadout will not use less, and may use more). The Kill-A-Watt measured just over 3 watts with the Blackbird on standby plugged into the wall; powering it on, BMC bring-up and IPL topped out at around 60-80W, booting Fedora peaked at 127W, and idling in GNOME measured about 65W (I told you that 650W PSU I bought was overkill). Stressing the system vainly trying to play Xonotic only showed around 105W. The highest power draw I ever saw out of the Kill-A-Watt was 131W.

I installed the two new fans when they arrived and placed two fans (again, all Arctic F8 PWMs) using a splitter on each of the two zones for the full four this case will accommodate. With two fans on the CPU zone in the same environment, the cores could now be seen to visibly cool down at idle which wasn't happening before. Interestingly, the board didn't seem to be using the case fan zone much even though all four fans did indeed spool up at IPL as expected. After a few minutes letting it just sit there at the GNOME desktop, the cores dropped to 58°C and even the CPU fans gradually spun down to minimum. This wasn't silent, to be sure, though with a movie playing you'd never notice it. That said, making the machine work I could still get some 85°C-ish peaks out of it but nothing close to the thermal cutoff Tim mentioned.

In the five days I've had this so far, a couple of other trivial annoyances did pop up, though these are possibly unique to my specific circumstances. The first is that resolution switching periodically seemed to unsettle the projector, requiring some HDMI hot-replugging or even once or twice a full power down to get it to see anything. This could well be the projector, and may have absolutely nothing to do with the Blackbird, but it was still obnoxious and never happened before with the Blu-ray 3D player or the AV receiver's on-screen display. So far I can't discern an obvious pattern and this may disappear as kernel AST2500 support improves. Also, I completely cut power at the surge protector switch to protect the equipment when turning off the home theatre, but that means every time I fire it up and want to use the Blackbird I have to wait for the BMC to come back up again and then go through the startup sequence before I can even boot Fedora (about two to three minutes to a login prompt). Yes, this happens with a T2 as well, but in my case the T2 I'm typing this on is my daily driver and so it's always running anyway.

It's time to sum up, so let's answer the first of our two main questions: how cheap can you get for a decent POWER9 system? I think this is a decent POWER9 system and should meet many basic use cases as configured, but I think I've also demonstrated that it's at or near the functional minimum, and owing to the cost distribution in its particular bill of materials I don't think you can get a lot cheaper. Given that the board and CPU are about three-quarters of the total damages, there's not a whole lot more to economize on: you could cut down the RAM to 8GB or get a smaller PSU but you'd probably barely save $100, and even the SATA SSD, though not luxuriously large, wasn't all that expensive compared to a spinning disk.

In that case, let's answer the second, thornier question: how good will that low-cost system be? I'll be painfully frank and say I probably had unrealistic expectations when I chose to try to make this into a HTPC. Linux itself isn't a great choice, especially on a non-x86 platform. DVD playback on Linux is pretty much solved, but much commercial Blu-ray media is right out for lack of decryption keys, and because it's not x86 or ARM so is most DRM-ed content without closed-source black box binaries. While 5.1 mostly works over digital, the most trouble-free means of connecting it will be analogue, and I know I'm not the only person for whom that's not an option with their existing setup; the slow bring-up if you don't leave it on standby power all the time isn't a dealbreaker but is definitely annoying (I think shortening the startup time should be looked at for future lower-end designs), and at least for the time being the lack of a GPU in this loadout appears to limit the HD media options I can play comfortably. This box suffices fine for playing YouTube and Vimeo, which fortuitously will be the majority of its current tasks, and will be a nice training wheels system for my wife to experiment with Linux, work in LibreOffice and do other family computer tasks. If you choose something less fatty than GNOME you can probably wring a little more UI juice out of it, but I didn't notice much difference from my regular T2 in basic usage. More than that isn't really in the cards, and at least as-is, I felt I had to compromise a bit too much to make it into a credible media system.

As a deskside developer box, simple server or buildbot, though, even this relatively austere loadout is acceptable. The lack of a GPU isn't a major problem for such pursuits and 16GB of RAM is sufficient for even many intermediate tasks. With 16 threads and extrapolating from my dual-4 T2 which builds an optimized Firefox at -j24 in a little over a half hour, I can reasonably expect build times to be about double, which is good enough as a sidecar. It would be more than enough to serve data and media, even if (especially since) it's not playing it. In fact, because it's a secondary machine for me, I can run it full tilt since desktop responsiveness is comparatively unimportant. Plus, it's small and I don't care if I botch the installed OS, so this will likely be my test-bed machine for other distributions and hopefully the BSDs. If you want one purely as a secondary system or utility machine, then you're probably in the best position to make the most of this low-end config.

But it seems to me from my cursory observations of people waiting for their Blackbirds that they want this as a lower-cost way into the POWER9 ecosystem and to see how they like it as a primary system. In that case, let's just say it: you need to spend more than what I've spent here. I think based on this testing that to get a good experience of what POWER9 is and what it can do, you really need eight cores (currently, add about $330) and a GPU (you decide, though I stick with BTO choices since they're likely to be tested more, so that AMD WX7100 will set you back around $500 right now). I don't think there's enough threads for heavier usage with the 4-core, and gaming, media and desktop choices will expand dramatically with a discrete GPU rather than overworking poor old LLVMpipe. For that matter, 32GB wouldn't be a bad idea either. Due to what I consider the thermal constraints of such a larger spec, though (and given how constrained I found this machine to be with just 4 cores and no GPU), I would be leery of trying to get that all into an mATX case. I think you'd find it running louder and hotter than most people would like, and after my experiences so far, I wouldn't put a single-8 plus GPU in anything smaller than a regular ATX mini-tower.

With that done and dusted, now you're looking at a slightly larger footprint and a budget of about $3000 for what I would consider a system I could live with day-to-day. It won't be the same as my big T2, which because of its dual sockets has much more expansion capability (everything is NVMe and I still have room for the GPU, a RAID or SATA card, and FireWire), but I think the experience is comparable and it's still less than half of what a similarly equipped full T2 will cost. Just keep in mind if you find you like the POWER, and you might want a dual-18 or even a dual-22 someday, you won't get that on a Blackbird — you won't even get the fullest benefit of one of them. If you want dual GPUs, or lots of NVMe storage, forget it. Heck, if you just want a dual-8, forget it. What, you expected Raptor to cannibalize their own pro market? For someone like me, the big T2 would always have been my preferred option because it gives me the biggest bang and the most room to grow, but I've clearly got more money than sense and I already knew I wanted something with that level of power and flexibility. Even if they'd sold the Blackbird back then I'd still have bought the T2. But if there's no way you can spend that much today, even if you know you might in the future, then the Blackbird is what you should buy.

Don't get me wrong: even considering it didn't turn out as I planned, overall I like the Blackbird a lot, and while not a revolutionary design I think it's a strong, gutsy choice by Raptor to put their efforts into this sort of lower-end machine. It's clearly captured a lot of people's interest and I think Raptor will sell quite a few of them, which is great for the ecosystem and for the chances of having robust libre options on the desktop. I think it's certainly possible to get a lower-cost rock-bottom configuration like this to cover many people's uses, especially as a secondary computer, but I think you'll need to be a little more generous if you want this as your OpenPOWER workstation. Budget in the 8-core and a GPU and do it right, and you'll end up with a better priced option good enough to be your daily driver, yet running a performant architecture that's more open, less creepy, comparably specced and actually exists.

PowerPC to the people.

Flying the Blackbird


The Blackbird is now loaded into my home theatre rack. Just a couple tastes before the "full mini-review" later this week once I've ironed out a few other things.