Showing posts from February, 2020

Jon Masters, transparent ARM shill

I don't hate ARM. But I do hate cynical bloodymindedness.

Jon Masters' pronouncement of OpenPOWER as "dead" has been getting some press, and as far as this particular Power ISA bigot is concerned it's transparent twaddle. He's done a lot for ARM at Red Hat (lest we forget: a current subsidiary of IBM), but he's no longer at Red Hat: he's VP of Software at startup NUVIA, which is building ... a server-grade ARM chip. Knowing what he's planning on selling, that makes his "hot take" on OpenPOWER more than a little bit coloured by his own biases.

Despite being self-serving, though, not everything he points out is wrong. One valid concern is that currently the only manufacturer of high-performance OpenPOWER chips is IBM itself. We are fortunate in that Raptor is an accessible retail channel for these chips (and workstation-class systems), but most of the third-party builders are using Power in embedded applications, not high-performance desktops. Even in the Apple days the chip sources were pretty much just IBM and Motorola/Freescale, and for the G5 exclusively IBM (the brief existence of the PA6T notwithstanding); with the exception of Cell, the Power ISA game console generation was exclusively IBM too (i.e., Xenon, Gekko, Broadway and Espresso), and even Cell was an IBM co-design, so this is not a new issue. This is something that needs to be fixed and thanks to OpenPOWER being a royalty-free ISA there's a market opportunity here you don't have to pay IBM to exploit.

But to essentially argue it's okay to be open, but not that open is painfully self-serving. ARM can certainly compete in the server space; Apple's chips are already in striking distance even with their imposed limits on power consumption, and other companies have gotten into this business before. But none of them will be able to do it without paying ARM royalties, and with that investment in mind none of them want to do it without secret sauce (binary blob drivers) to deter competition. We're in a CPU age where what people think is the CPU is merely the target of a long line of intermediate operating steps and every one of these has firmware. On the Talos II I'm typing on, I can see the source code for every single boot stage. For Masters to argue that none of this matters until you pass into UEFI is like arguing that the Intel Management Engine, bless its little exposed backside, is somehow irrelevant, or that all the boot stages for POWER9 don't matter until you actually get to Petitboot, let alone all the sidecar auxiliary units like the GPEs and OCCs. Do we really need to go over again all the disastrous faults that have emerged in blackbox firmware you can't see or modify?

Masters knows this, too, and that makes his statements not just crap but disingenuous crap as well. (Perhaps he sees OpenPOWER as a threat?) Regardless, that also means you can confidently expect that NUVIA CPUs, if they ever even come out with a product (see also Calxeda), will be just as locked down as any other ARM core. So much for "reimagining silicon design."

Messing with the new 2.0 BMC

Tonight's attempt to upgrade the Blackbird to the new 2.0 BMC firmware did not go smoothly, though some of this was self-inflicted, and I'm also still flattened by whatever hellish virus has gripped me for the last month (it's not coronavirus, or at least not that coronavirus) which causes me little tolerance for glitches. TL;DR: the firmware basically works, but when I used it to reconfigure its IP address the BMC now can't see anything and nothing can see it, which has left me in somewhat of a foul mood [but see postscript]. I'll get it working when I'm feeling better and you should probably still update, but beware of these pitfalls.

Updating to 2.0 from any pre-2.0 version requires a complete flash of the BMC. Raptor warns against this generally because all your U-Boot/firmware settings will be reset, but in this case it's unavoidable. That brought us to the first problem: when sshed into the BMC, at the root prompt fw_printenv is supposed to show you the IPMI MAC address so you can reprogram it. On this Blackbird, however, it showed absolutely bupkis except for the serial port settings. After a brief moment of panic I realized I had a picture of the mainboard from the Blackbird semi-review and could enter it from that. Otherwise you'll have to drag the machine out, open it up and jot down the address printed on the board. Oddly, this does not reset anything else, including the BMC password or actual network settings. More about that in a moment. It did, however, change the ssh key.

Now updated, since my other main systems are Power Macs (and what better computer to be a "service processor" to your Blackbird than another PowerPC?), I decided to do further configuration through the BMC's new web interface in TenFourFox, which is essentially Firefox 45 with a lot of patches. These were done on my iBook G4 service laptop running the latest beta hot off the G5 in the backroom.

The first thing to keep in mind is that the certificate is self-signed. No biggie, just expect it.

The webapp appears to be written in Angular, and it's using JavaScript too recent for TenFourFox (which admittedly doesn't get along well with current React or AngularJS frameworks). Some stuff does work -- the IPMI sensor data loads -- but does not automatically update, and the server status never appeared. It might have been nicer to have a better fallback, especially for the NoScript people, so that data can be displayed even if it won't update until one reloads the page.

It didn't appear here either, even when directly queried.

Fortunately my main use case was to upload firmware through the web interface, so I decided to immediately update the PNOR (both out of necessity and as a useful test), and that worked. Just unpack the archive and upload the subarchive in the web_ipmi folder (the server will automatically unpack the .tar.gz and make the firmware available). TenFourFox threw a weird error at the end but the firmware uploaded, was verified, and could be activated.

IPL, showing the fans coming up. You can boot through the interface, but I just pushed the power button since I was sitting next to it.

The serial port output did not work on TenFourFox either, so I did it from Firefox on the MacBook Air, which I found technically disgusting but worked rather well. Fedora will happily run on the serial port. I was able to log in and look around from the BMC itself. Yes, using TenFourFox was a self-inflicted wound, but I thought it would have worked better than it did.

At this point I decided that I'd had enough mucking around with the Blackbird over WiFi and decided to give it a new static IP through the web interface and run it to the iBook over Ethernet. I did this from the Air, just in case the iBook screwed it up. The machine obligingly accepted the settings and then stopped responding on any address, even after a power cycle. Tomorrow I'll try to find a serial connector to talk to the board directly and try to start over from scratch. I would have your network settings finalized first before this update, as you probably should anyway.

I haven't tried doing this update on the Talos II. I might not anyway since my tax refund should be arriving and I'll be upgrading to a dual-8 system soon. I can't imagine there's much difference in firmware experience between the two systems, though.

The moral of the story is don't update firmware when you're ill.

POSTSCRIPT: ipmitool saves the day! After an obscure mention in an IBM technical manual I was reading for another purpose, it dawned on me that Petitboot (or, for that matter, Fedora) can set the BMC's address.

I started up the Blackbird and went into the Petitboot shell. A quick ipmitool lan print 1 showed what the problem was: the new web BMC interface claimed it had removed the old ZeroConf IP address, but had not, and that became the IP. Since the netmask was now all munged, nothing could see it and it couldn't see anything. So I forced the issue:

ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr [your IPv4 address]
ipmitool lan set 1 netmask [your IPv4 netmask]
ipmitool lan set 1 defgw ipaddr

A quick powercycle confirmed it stuck, and the web BMC answers correctly on the expected address. I still consider this a bug in web BMC but fortunately it's recoverable without digging out the serial cable.

The BMC is getting new tricks

UPDATE: And it's out! Talos II, Blackbird. I'll do the Blackbird update this weekend first. The web view of the serial port (for Petitboot) is particularly nice for those of us without GPU blobs in our firmware.

On Twitter Raptor is teasing an upcoming new BMC build for all Raptor family systems (Talos II, T2 Lite and Blackbird) offering web-based environmental monitoring and firmware updates. I'm a command line jockey myself and I didn't find the previous SSH-based means too onerous, but a web-accessible environmental monitoring system could be quite useful for centralized setups (especially if there's an API or some other means to scrape the data into a dashboard). Since this data is directly served from the BMC, it would be more complete than what ibmpowernv offers now and being directly pushed should be faster than ipmitool which can take several seconds to gather information (see our DIY GNOME IPMI fan tool for a real-world example). If you can't wait, it looks like this code is publicly available in Raptor's git tree right now, but you'll have to build it yourself (you should anyway) since there don't appear to be beta builds just yet.

Firefox 73 on POWER

... seems to just work. New in this release is better dev tools and additional CSS features. This release includes the fix for certain extensions that regressed in Fx71, and so far seems to be working fine on this Talos II. The debug and optimized mozconfigs I'm using are, as before, unchanged from Firefox 67.


It was pointed out on the Raptor discussion board that the ibmpowernv hwmon module doesn't report fan speed for Raptor family systems, and I suspect this is true for most things based on the Romulus reference design (you can only see the fan of the graphics card, and of course only if it's installed). This means most of the GNOME shell extensions to display system status won't display it. However, it is accessible by talking to the BMC over IPMI, so you should be able to get it that way. Here's a quick-and-dirty method to put your Blackbird or T2 fan speed(s) into your GNOME shell (and probably works fine for other systems with IPMI-accessible fans). This is using Fedora 31; adjust for your distro to taste.

  1. First, verify that you do have fans. You'll need to do this as root: sudo ipmitool sdr type fan

    This will show output like this, after a couple seconds:

    fan0   | DDh | ok  | 29.1 | 2100 revolution
    fan1   | DEh | ok  | 29.2 | 2100 revolution
    fan2   | DFh | ok  | 29.3 | 1900 revolution
    fan3   | E2h | ok  | 29.4 | 2000 revolution
    fan4   | E3h | ok  | 29.5 | 1700 revolution
    fan5   | E4h | ok  | 29.6 | 1700 revolution
    fan6   | E5h | ns  | 29.7 | Disabled
  2. We don't want to have to constantly query the BMC as root, so create a ipmi group and put yourself in it (with vigr, vigr -s and vipw as needed). Log out and log back in, and check groups to make sure you have ipmi privileges.

  3. Create a udev rule to make IPMI group-accessible by our new group ipmi. In /etc/udev/rules.d/99_my.rules, I have

    # allow ipmi to be seen by ipmi group
    KERNEL=="ipmi*", GROUP="ipmi", MODE="0660"

    Restart your system to make sure this sticks, and/or chgrp ipmi /dev/ipmi0 ; chmod 0660 /dev/ipmi0 to make the change live. You should now be able to just do ipmitool sdr type fan as your IPMI-group user.

Now that your system is configured, let's actually integrate the output. At some point I'll maybe make this into a full-fledged extension but for prototyping and playing around purposes, there is an easier way: Argos. Though sadly the maintainer is no longer a GNOME user, the extension seems to work fine still for this purpose as of this writing.

  1. Install the Argos GNOME extension if not already done. You may wish to chmod -x ~/.config/argos/ afterwards to get the demo menu out of your menu bar.

  2. Download this simple script to format the output from ipmitool into Argos BitBar output. Its only dependencies are bash, awk and ipmitool. It gets the IPMI information, caches it (because it's expensive), and then figures out the fastest fan and puts that into the Argos button (click that for all the fans in the system, as shown in the screenshot).

  3. The script goes into ~/.config/argos, but the filename will be based on where you want it and how quickly you want it to update itself. My filename is, which says set it to position six on the right side of the shell bar (this varies on other shell components you have there) and updates every 5 seconds.

  4. Once you have selected position and interval, chmod +x ~/.config/argos/[filename].sh, Argos will automatically see it, and it will start updating at the interval encoded in the filename. If it's in the wrong place, or you don't like how quickly or slowly it updates, just rename the file and Argos will "do the right thing" live.

Do the brew*

I've long tried to position the Talos family as an upgrade path for Power Mac owners, and here's another way: the macOS Homebrew package manager has been ported to OpenPOWER.

The concept is a bit involved but most of the work has been done for you. To bootstrap Ruby requires building a version from the portable Ruby recipe, or you can borrow a ppc64le build and patch the vendor install script to find it. At that point you should be able to patch brew itself with the three patches linked in the instructions. We look forward to seeing these patches getting into Homebrew proper!

(*The authors of Talospace do not endorse the large-scale drinking of alcoholic beverages unless you are an Asgardian god or Australian. And even then.)