Latest Posts

LibreBMC announced, end of Kestrel?

Oh, how timely. On the heels of our article yesterday on "little Power" comes an official announcement from the OpenPOWER Foundation for LibreBMC, a fully open and Power ISA-based BMC replacement.

All PowerNV systems currently use some form of ASPEED BMC to provide the baseboard management features necessary to run the system (what we old fogies used to call "service processors"), which on these machines includes IPMI, offering the PNOR flash ROM to the processors to start the machine, a 2D frame buffer, environmental monitoring, network, front panel and other on-board interconnects. They run their own operating system, almost always OpenBMC, and function as a computer-within-a-computer. The goal here is to replace the ASPEED devices with a multi-vendor supported option compliant to the Open Compute Project's Data Center Secure Module Specification, but would (from our perspective as users and operators) appear much the same. It would also be compatible with Power, x86 and ARM systems like current BMC offerings. However, unlike ASPEED chips which are ARM-based, LibreBMC would run on a Power ISA core, possibly even a descendant of Microwatt.

Some of the names announced in connection with LibreBMC are familiar: Yadro appeared in yesterday's piece, a Russian developer of high performance systems and storage solutions, and Google was one of the developers of the OCP DC-SCM spec and is undoubtedly advising on the high-level design. Raptor is also involved, which is very good news, because their involvement suggests there are unlikely to be hidden blobs and the LibreBMC will be truly an open device (the announcement says it's all open tooling using LiteX, and synthesizeable on Lattice and Xilinx FPGAs). If you want a "little Power" system-on-a-chip, you could do a whole lot worse than simply ripping this off into a little stand-alone developer board. Maybe this is where the PowerPi will come from.

But, oddly, despite their involvement Raptor isn't making LibreBMC: that's being done by Antmicro, who currently has RISC-V and ARM-based products and will be "developing the LibreBMC card," implying that LibreBMC will also be offered as a physical component instead of just IP and VHDL code. Specifications for the physical form factor are even in the DC-SCM specification: it has a special slot similarly sized to an OCP NIC, but differently keyed, and the vertical form factor option 1 in particular looks very much like Beagle Boards and RPis. The fact that LibreBMC will run OpenBMC also suggests this is not a direct port of Kestrel, Raptor's own prototype BMC replacement; Kestrel currently runs its own OS based on Zephyr.

If this is a quiet way of announcing Kestrel is throwing in with LibreBMC, this would disappoint me somewhat. Kestrel, though granted in a very early form, seemed a better fit for those of us on the small but visible Power workstation side if its promises on faster start times were to be believed. In particular its trimmer OS choice was an important difference: OpenBMC's problem (from my outsider view) is that it wants to be all things to all workloads, which means a bigger loadout with more features but slower start times and a much larger firmware footprint. LibreBMC's move to a higher-performance Power core should help but software always expands to meet the available CPU power. BangBMC is still out there and aimed to address performance by cutting out fat like dbus and systemd, and apparently is now facilitated by Raptor, but it hasn't gotten a great deal of traction and nobody ships it as default, for example. A bespoke fast-start workstation-oriented BMC would be a great fit for our desktop systems but LibreBMC doesn't sound like that. And if this is going to be the simultaneous basis of a little Power board as well, bootup time really has to be better.

There are also the practical aspects for current owners: OpenBMC isn't going to drop ASPEED support anytime soon since there's a large install base running it (on lots of machines, not just OpenPOWER), but future updates would understandably prioritize the new hotness. Kestrel, although you needed a soldering iron, could be retrofitted to existing boards. Especially since open POWER10 systems will be slower to arrive, what about a LibreBMC retrofit option (that doesn't involve board work) for existing machines? Will the Antmicro "card" offer this, or will there be an aftermarket product from Raptor or another vendor? We'd also like whatever hardware and software improvements the LibreBMC initiative will foster especially if we won't be replacing our systems for awhile.

Nevertheless, I wait to see what happens with cautious enthusiasm, because I think that the importance of the BMC to the system (it runs and sees everything, after all) really demands an open solution. Plus, the presence of a BMC means there's (going to be) an OpenPOWER SoC, meaning smaller Power solutions may be just around the corner. Whoever ends up making it, this is good news.

The strange bedfellows who want a little POWER

Thanks to reader Karl S for the initial report that got me down the rabbit hole, who discovered a quiet entry for a PowerPi workgroup at the OpenPOWER Foundation, clearly modeled on the ARM-based Raspberry Pi.

The obvious basis for a "little Power" device would be something like Microwatt, which Raptor is already using for Kestrel, and to which IBM staff regularly adds new features. While IBM also opened up other cores, notably the A2 family, Microwatt is (or at least will be) more like what's under our desk in terms of supported instructions and features, and is probably the easiest of the open synthesizeable cores to work with right now.

Still, Microwatt is "merely" a core, albeit a rapidly developing one, and not actually a product for sale or even a chip (though it may be in the near future). This makes the PowerPi concept particularly interesting in that it's trying to use real hardware to appeal to the same market segment that would rather experiment with something smaller, but end up unrealistically repelled by even Raptor's "low end" (relatively) pricing and reach instead for ARM or RISC-V boards because of their sensitivity to cost.

The PowerPi workgroup has four members. The first three are IBM (naturally); YADRO, a Russian manufacturer of high performance servers and storage solutions; and Van Tosh, a Belgian cloud solutions provider with their own RHEL offshoot. With the exception of IBM, none of them seem to have the expertise to work on a mini-POWER system, nor do their current sites indicate any particular proclivity.

But the fourth is noteworthy, a company called ChipEleven, a new, privately held VC-backed fabless design firm based in New Orleans that started just last year. Its CEO is a Master's student at Georgia Institute of Technology. They advertise a system on a chip with lots of peripheral interconnects, a quadcore ISA 3.0B-compliant CPU, an ARM Mali G77 GPU, onboard VGA up to 1920x1080, Ethernet, SPI flash and DDR4 RAM controller, with an ETA of 2023.

Does this sound a little familiar? It might, because an earlier project exists called Libre-SOC. Libre-SOC started with a RISC-V design to serve as a libre GPU, interestingly based on the CDC 6600, but shifted to OpenPOWER for higher performance and now plans to release an entire SoC instead. Raptor has provided material assistance (though Kestrel is proceeding on its own separate path) and development by volunteers is funded by NLNet donations. Allegedly, ChipEleven started with the Libre-SOC project and cleaved off, though to what extent (if any) ChipEleven's design derives from Libre-SOC is disputed.

It's not generally my habit to report on new speculative projects because a lot of them, even those with well-meaning and motivated people, will eventually flame out or their organizers lose interest. And ChipEleven may do just that: 2023 is a long way off (POWER10 will have certainly arrived by that point) and that doesn't mean there will be an actual physical product you can buy, let alone what it would cost. For its part, Libre-SOC wisely makes no promises. Still, the fact such a workgroup exists and has the backing of the OpenPOWER Foundation is notable, even if the current members are a motley bunch. Whether they can actually ship something is another story.

OpenBSD 6.9

Great past few days for new OpenPOWER operating system support: now the newest release of OpenBSD is available as well to complement Ubuntu and Fedora updates earlier this week. Many improvements have landed to the big-endian powerpc64 port since its début in OpenBSD 6.8, most notably framebuffer support for the ASPEED BMC, workarounds for addressing AMD GPUs over PCIe, power-saving mode for POWER9, and IPMI on PowerNV systems (which is all Raptor-family machines and quite a few others). General to all ports include an additional privacy guard for video devices (similar to the existing one for audio) among other webcam fixes, encrypted RAID-1, performance improvements to SMP, lots of additional hardware drivers, security fixes and updates to networking and crypto.

With this release, your BSD choices on OpenPOWER just got more solid between this and the mature FreeBSD port. Again, the real shame is why there's still no support for OpenPOWER in NetBSD.

Prefixed instructions and more in the OpenPOWER 64-bit ELF V2 ABI Specification

This is pretty nerdy, but you read this blog, so you have no room to talk. The ABI specification for 64-bit ELF v2, which is virtually everything running little-endian and many big-endian 64-bit Power systems, is now in draft for version 1.5. Although library interfaces are not in spec, the document defines a linking interface for executables and shared objects so that register usage and calling convenion is consistent. Rarely if ever would such updates include breaking changes and the small version number increment implies no large shifts, but those of us who write JITs and compilers pay attention to these updates since they may yield new opportunities to generate better code, and they also occasionally shed light on new or upcoming instructions.

Besides incorporating errata from the previous version 1.4, and moving vector programming to a separate document, the most interesting change here is related to POWER10. We've mentioned prefixed instructions briefly here before, a new class of load/store operations available in POWER10 and PowerISA 3.1 that effectively create 64-bit instructions: a first half (prefix) containing up to 18 bits of a displacement, and a second half (suffix) containing the lower 16. This enables to you to express as many as 34 bits of displacement for a memory address with an instruction like pld (as opposed to the 32-bit classical instruction ld with a 16-bit displacement); previously the maximum you could indicate was 26 bits, and only for instructions like unconditional branches that allowed it. However, the R bit in the prefix allows you to use the address of the prefix itself as the register against which the displacement is added, rather than having to have a general purpose register do it or (for non-local data) the dedicated TOC register. (Recall that in Power ISA the program counter is not a general purpose register.) This is actually a big deal for things like constant pools (embedding constants directly into many RISC-style instructions is generally unwieldy) because now you can just squirt local data right into whatever function fragment you're generating instead of having to keep track of it separately. This is mentioned briefly in the ISA 3.1 manual but the ABI spec makes it more prominent as a feature.

Unfortunately, various complexities in instruction dispatch make this less useful than it would appear to be. Prefixed instructions cannot be split over 64-byte (not bit!) instruction address boundaries or else an alignment exception occurs, which even if the OS handles it for you would be expensive. On the other hand, the CPU is clearly treating them as two 32-bit pieces because the prefix is always followed by the suffix regardless of the endianness (i.e., in little endian mode, the suffix is not in front of the prefix), and there are also debugging irregularities in that some suffixes are actually regular instructions that mean something different when used as a prefix. These can be detected by looking at bit 6 of the prefix (not the suffix), which if set indicates the suffix is a valid instruction that the prefix changes the behaviour of, but one wonders if more changes are to come and we don't need any more mscdfr (means something completely different for r0)-type situations in the ISA. A great example is pnop (yes, a prefixed no-operation instruction): you'd think the suffix would be ignored, and it mostly is, except if it's a branch instruction, rfebb, any context synchronizer other than isync or a service processor attention instruction. The ISA 3.1 book benignly says, "This restriction eases hardware implementation complexity." Well, thanks a lot! Does your head hurt too?

Again, I'm not a fan of introducing variable length instructions into what was a fairly regular instruction set and there are many important gotchas which to me seemed avoidable, but the displacement features are welcome and it makes certain on-the-metal programming tasks easier. Always watch these deceptively boring documents closely because there are sometimes valuable signals in their changelogs. Unfortunately, until the situation with POWER10 and OMI gets worked out, this is of largely academic interest.

Fedora 34

And hot on the heels of Ubuntu 21.04 is the latest iteration of Fedora, version 34. Fedora is of particular interest here at Floodgap Orbiting HQ, not only because it serves as an early warning indicator for problems on OpenPOWER as one of the most cutting-edge distros, but it's also the distro I'm typing this blog post into and personally use on a daily basis. Now that F34 has hit release, F32 will become EOL in one month as usual.

Most of us are interested in Workstation-level changes and the most notable is GNOME 40, which introduces new and sure-to-be-controversial changes to Activities (though if this helps multiple display management I'll be a believer), separation in the dash of running apps and favourites (good), and additional shortcuts and gesture support. Other system-wide changes include transparent by-default zstd compression for btrfs, routing all audio through PipeWire (including PulseAudio, JACK and legacy ALSA), enabling systemd-oomd by default, updating to glibc 2.33 as well as gcc 11, llvm 12 and binutils 2.35, upgrading to Ruby 3.0, and (another controversial one) using Wayland by default for KDE Plasma users as well. Another nice minor change is that kernel firmware files will now be compressed by default, saving a bit of space.

On the OpenPOWER side, however, specific platform improvements are rather thin on the ground. 128-bit long double got deferred again, which I've been tracking since Fedora 30 (!!) as certain packages like MAME require it to build out of the box, and there has been little appetite to consider a Workstation-specific 4K page option.

Because I confidently expect GNOME 40 will break all my extensions, and some minor interval is required to ensure all the packages are built for ppc64le, our usual mini-review for F34 will follow in a couple weeks on both Blackbird and T2 systems. Meanwhile, read how it went with F33 in preparation.

Ubuntu 21.04 and the expanding Wayland Wasteland

It's not really Power-specific to be disenchanted with Wayland; there are lots of people who don't like it even on majority platforms like x86_64. I also think that, much like the residual disdain for systemd, a fair amount of the backlash comes from some profoundly unwarranted scope creep in the project. X11 has a lot of historical cruft in its lower reaches which deserved at minimum a solid refactor and I do appreciate Wayland's engineering improvements, but Wayland throws the baby out with the bathwater by putting way too much on the back of the compositor, and as far as claims over security and network transparency are concerned one man's security hole is always another one's convenience. Most of the Wayland developers just throw up their hands when confronted with some functionality that was fine in X11 and say "patches welcome" and "don't expect us to scratch your itch," and then wonder why people get cheesed off when stuff quits working. It would be less aggravating if the process were less headlong but such is the state of Linux desktop development where only established players with their own priorities have traction.

That said, the problem is somewhat more acute on OpenPOWER because of the lack of a libre GPU. Right now, if you don't trust AMD or Nvidia, your solitary choice is the on-board ASPEED BMC and that gives you a 2D framebuffer, period. (Even Kestrel won't fix that.) Performance used to be abysmal under Wayland and now is tolerable, though there are still various problems, and even performance with a GPU seemed to regress a little in Fedora 33. I still don't use it anyway because no current Wayland compositor will tell you what the front window is, nor does it seem any of them care about that, even though X11 facilitated this for literally decades. Again, why not run everything through XWayland by default, let Wayland-friendly apps opt out, and get the best of both worlds for (nearly) free? Why intentionally p*ss everyone off by telling them their working edge cases don't matter?

Nevertheless, the Wayland Wasteland expands with Ubuntu 21.04 "Hirsute Hippo" (release notes), which now also makes Wayland the default as it has been in Fedora for many versions now. Fedora allows you to opt out by either running startx manually (as I do) or for those of you running gdm to set WaylandEnable=false in /etc/gdm/custom.conf, and this functionality will probably remain for as long as X is supported in Red Hat (I'm guessing end of support for RHEL 7, maybe 8). In Ubuntu currently you can do the same thing, but the file is /etc/gdm3/daemon.conf instead (or use the cog on the login screen, though the login screen would still come up in Wayland unless you set that flag). As before little-endian OpenPOWER systems (which Ubuntu calls ppc64el) are officially only offered a Server build for download, but you can then convert it to Desktop.

Should you upgrade? If you're happy with X11 on your OpenPOWER system and the performance is good, maybe you should just stick with 20.04, which is a Long Term Support release (21.04 isn't) and will get updates until 2025. But if the future really is the Wayland Wasteland, at least getting more people stuck in the sand will mean some of these rough spots could get smoothed over, and a better software-only rendering pipeline would at least improve the firmware-free use case. In the meantime, hello, X11: you may be ugly and everyone says you smell bad, but you've never gotten in my way.