Latest Posts

Broadcom BCM5719 libre firmware coming real soon

Posting from the Southern Hemisphere today, kudos to reader Mark J who sent in a heads-up on the progress the Ortega project has done on reverse-engineering the Broadcom BCM5719's firmware. If that number sounds familiar, it's because it's the very same NIC in the Talos II and Blackbird (see a photograph) and one of the few places left in the Talos family that binary blobs are required. Apparently, the reverse engineering effort is now believed sufficient to create a clean-room implementation.

And where is that implementation? Why, on Github, of course. As of this writing the current firmware is a work in progress, but now that the chip is much better understood it's very likely work will move much more quickly.

The BCM5719 is an interesting chip internally, implementing MIPS II (!) cores that apparently were once part of the receive and transmit machinery but are now mostly relegated to autoconfiguration-like tasks. A MIPS core exists for each port, but only a single application processor engine (APE) is present per chip. Hugo Landau's work on figuring out how to talk to the APE, the most essential component any open-source driver would need to interface with as it implements the sideband interface, is nothing short of heroic. A particularly noteworthy deficiency he discovered was that even though the firmware image for the APE's ARM Cortex-M3 has an RSA signature, nothing actually checks it! That would seem like a terrible rookie mistake on Broadcom's part but it's great news for us. A hearty congratulations to everyone's hard work on a very necessary project.

ArchLinux on POWER9

Another option is available for Linux on the Talos: Arch Linux. Although officially x86_64, alpha installation ISOs are now available which apparently "just work" on the Talos II. No word yet on available packages but if you like your Linux lean and mean on your machine that cost you much green, you'll like what we've seen. (I'll be here all week. Try the veal.) Update: The maintainer has a site up.

Blackbird delayed to May

On Twitter Raptor is reporting that due to supply chain disturbances from the USA-China trade spat, the lower-cost POWER9 Blackbird's ship date has been set back again, this time to around May 15. Raptor states the parts are finally in and logic board manufacturing is beginning, so we can assume this date is reasonably firm. However, functional Blackbird prototypes already exist on this side of the Pacific as evidenced by their additional offer of remote access for software testing, which Raptor says are "identical in all ways except HDMI DDC wiring," and sounds similar to the prototype we saw at SCaLE 17x. That bodes well for good operating system support at launch, which will be particularly critical for the lower-cost "tiny Talos" and hopefully its resulting larger audience. Ours is on order for review; with any luck we're near the first in line!

I wouldn't trust ME with a VISA either

Still think Intel chips are "secure enough"? Presenters at Black Hat Asia have revealed the existence of a previously undocumented Visualization of Internal Signals Architecture (VISA) feature, allowing the signals and internal operations of the chip to be captured at a highly granular level.

Now, this in and of itself isn't so strange. Many chips have debugging features like this, because many chips including our POWER9s are just that complicated, and to get into VISA requires admin-level rights (though one could reasonably ask why it wasn't hardware-disabled at the factory). Of course, if you can combine it with a vulnerability that would give you admin-level rights, like, oh, I dunno, the Intel Management Engine, then you have the entire system bus at your disposal. Although Intel argues that the exploit requires physical access as well, the researchers say no hardware modification is required and hint at other simpler methods of exploit.

A superficial analysis like The (Usually More Perceptive) Register's handwaves this away. After all, the exploit they use was fixed in 2017, and frankly if you could get into ME you'd just do that, so no problem, right? Except for three things: one, Intel didn't document this in the first place, so this is the first we've heard that VISA actually exists in shipping chips; two, what's to say this couldn't be exploited by the next Management Engine (or anything else) flaw that comes down the pike; and three, you can't turn ME or VISA off! You can do various things to hobble ME (including this utility from the authors of this very Black Hat presentation), but you can't get rid of it completely. While in practical terms attacking VISA is unlikely to be something widespread in the near future, Intel's attempt to lock it down under NDAs and secrecy doesn't speak well of how much you can trust an Intel CPU.

Most of all, how many of you wanna bet that this is the last little black box to be found in Intel chips? Yeah, I wouldn't trust ME with a VISA either.

Pitch into the Firefox JIT

On the suggestion of a few people, and the fact my time is somewhat more curtailed due to my attempts to do hal-fassed continuous integration testing on ppc64le Firefox and maintain TenFourFox for a still resilient 32-bit Power Mac crowd (and my day job, which as some of you know has pretty much zilch to do with this type of thing), I've uploaded the work so far on the Firefox JIT to a Github repo so that additional people who want to contribute can. The repo is live now.

I think I've documented most of it in the readme, but here are the highlights. Much like TenFourFox's IonPower, some of which was converted to 64-bit and transplated into this new JIT, the Power ISA JIT uses MIPS as a scaffold to hang new code off of. This is particularly handy since many of the MIPS instruction sequences have similar or direct correlates to PowerPC (like lui ori and addis ori), MIPS also uses a link register, and MIPS doesn't have some of the encoding idiosyncrasies of ARM.

The minimum viable product I'm shooting for is a little-endian 64-bit JIT with Wasm for the POWER9. It's not that big-endian isn't desirable or that it shouldn't run on anything earlier than a Talos, but I also know that for MVPs the last thing you want is having to boil the ocean and these are things that can be added after the fact. We also can use all the nice new P9 instructions and avoid endian glitches in the JIT core by concentrating on LE POWER9.

You can see the work I've done already; pretty much anything with a -ppc64le in it is a file I've either completed (the majority) or started work on (the MacroAssembler). The macro assembler, architectural definition and low-level direct assembler are the major pieces still left to do in the first draft before trying to get it to build. I've based this off Firefox 62 not because there's anything special about that version but because it was the version in release when I got started and having to react to code changes with succeeding versions would have slowed down the first draft. When we get it up there then we can forward port it and pick up any additional changes we need to make to the backend along the way.

Finally, this repo is merely a means to the eventual end of getting the JIT to live in the Mozilla tree and be maintained there. This is simply a temporary measure to get more collabourators and get it off the ground.

I'll still be working on this personally, of course, and I'll be doing it more once I get the current TenFourFox release into beta, but I think that we all want good browser choices on POWER9 and while Firefox is perfectly usable on my Talos, it's certainly not the best it could be. This way people can see progress is being made and those who have the technical chops can contribute to make a great platform even better. Further posts as we reach additional milestones. If you want to contribute, open issues and file PRs and I'll deal with them promptly.

Firefox 66 on POWER

Firefox 66 builds out of the box as Firefox 65 did. In addition, as part of the porting effort, I'm running occasional check builds roughly weekly to make sure we intercept issues on nightly before they get into beta and require a higher threshold of approval for patches. So far no major issues and Mozilla continues to be very amenable to getting the occasional fixes in promptly.

The biggest change in Fx66 is the number of default content processes in this release has jumped from 4 to 8. This is good news on our massively parallel multicore systems generally, but it's possible that the differing memory usage may be what tickles that harmless kernel assertion (though on the other hand, setting it back down to 4 didn't eliminate them completely). If you're on Fedora as I am or another distro that elevates kernel warnings to notifications, you'll probably want to turn off those system notifications until your distro's kernel gets the fix in it; see that article for details.

A close look at the Raptor Blackbird and what I did at So Cal Linux Expo 17

Wow, what a swag haul (big box o'breakfast cereal for scale), and what a fun day at SCaLE 17x, the 17th annual So Cal Linux Expo! I'd actually never been before and now I see why people love to go!

SCaLE is at the lovely Pasadena Convention Center, just a short drive from Floodgap Orbiting HQ in sunny rainy southern California. If you're in the greater Los Angeles area, it's pretty accessible by Metro rail, and there are lots of restaurants and things to do if you're flying in from away. I had lunch with my good buddy Bill (lately of the Linux Journal) and two of his cow-orkers at Islands Burgers just north of the Convention Center.

Even if you just buy an expo-only ticket, there are a huge number of vendor booths from big names like IBM, VMware and even Microsoft (!) all the way down to open source projects like VLC, Krita and Inkscape. Would have loved to have seen a Mozilla booth, though. Just saying. I even wore my Mozilla grey hoodie.

I even renewed my Electronic Frontier Foundation membership in support of the great work they do (make mine Titanium).

But of course the star of the show for your humble writer was our friends at the OpenPOWER Foundation, and they came ready for action with our favourite heavy-duty free computing platform:

Hugh Blemings, executive director, kindly tolerated my shutterbugging and an endless parade of retakes to get everything just right. (By the way, did you notice that Timothy Pearson from Raptor is now on the OpenPOWER board?)

He brought in tow their Debian dual-four-core Talos II, which did presentation duty, and of course a prototype Raptor Blackbird motherboard! For those of you new to the blog, this is Raptor's lower-cost way to get into the Power ISA ecosystem. So let's have a detailed look.

Here's the mATX board itself and a side view of the ports. (I'll zoom in on some items of note in a moment.) You can see the ports for USB 3.0 (two rear with additional headers for two more), 4x SATA, 5.1 analogue audio, S/PDIF digital audio, 3x GigE and HDMI. You can also see the two RAM slots and the x8 and x16 PCIe slots. My grizzled old hacker heart was warmed to see that there is still a good old fashioned serial port there too.

The 2D framebuffer is provided by the AST2500 BMC (ARM11 based with a sidecar ColdFire core), at left, routed to HDMI via an ITE IT66121FN; all three Gigabit Ethernet ports are serviced by the Broadcom BCM5719, at right.

Blackbird has some additional hardware for higher-security applications. At left are the flash chips for the BMC and boot flash with hardware write-protect switches, meaning if you can secure the case, nothing's overwriting the firmware. (I looked on my early-model T2 and can't find any such switches, so this is definitely an improvement. Update: found them, just at a slightly different relative location. Timothy Pearson in E-mail notes, however: "In practice, the current firmware stack is a lot chattier with the Flash than we'd like, so there's still some work to be done before we can roll out write protect in official form to both platforms.") Further anti-tampering security is offered by a Raptor-specific FlexVer connector (PDF).

The single POWER9 CPU socket (four or eight cores), and an interesting unlabeled port. Hmmmmmmm. (Update: Timothy Pearson in E-mail identifies it as an FSI port. "Talos II has one just like it; you can plug an FSP box into that port. The FSP boxes are proprietary and available to IBM partners only (like us), however they're also being replaced with the BMC for the most part, which speaks the same protocol and incidentally can do the same kind of debugging tasks now that the FSP boxes were traditionally used for. We will probably continue to retain the connnector, since even if someone designs an open FSI box to plug into the port it could be useful for various low level hacking (in the good sense) tasks. Note that neither the BMC nor this connector can bypass secure boot if enabled, and certainly FlexVer would immediately thrown an attestation error if anyone even tried.")

And last but by no means least, a four-core POWER9 CPU and the exposed die, which Hugh had on display. What a gorgeous bit of silicon, amirite? I'm still planning a full review of this Power-on-a-budget system when my production unit gets here hopefully in just a couple months.

Some other fun stuff:

The One Laptop Per Child handcrank! It exists! I would have killed for this back in the Give One Get One days.

Purism had a nice showing. In addition to their very impressive line of free(r) laptops, they also had a prototype of the Librem 5 libre smartphone, currently scheduled for Q3 2019. I just bought a Pixel 3 (these pictures were taken with it), or I would be buying one of these. I might anyway. I also reminded them that some of us wouldn't mind buying a non-x86 laptop. Power would be nice, but ARM would be fine too. He duly took it under advisement.

Not to be outdone, System76 had a Thelio system on display. Weird site but sexy case. It would be high on my list if I could get the case by itself and slap a Talos-style system in it. How about it, 76?

Standing guard at the entrance to the the exhibit hall was this huge Tux mosaic, made out of AMD Opterons.

It may be a Linux expo, but that doesn't mean the BSD folks can't be there too. I got a couple "RUN BSD" stickers (a la RUN DMC) for my NetBSD machines and one of the fun flashing devil horns headbands. Still looking forward to the FreeBSD port to POWER9!

There were many great open source projects there, but the photographic winner was this particular famous project:

Rock those hats, guys!

And just a tiny selection more of the many vendors, from whom I shamelessly lifted free stuff, and without whose financial support the Expo would probably not be possible:

Special shoutout to the really plush setup the folks from IBM had:

Distros in the hizzouse:

See if you can figure out which one I wrote. No, go on. I'll wait.

The GNU Project/Free Software Foundation had an appropriately chaotic-good booth, with a very important message:

Finally, a few more open source projects to close us out:

Overall I came expecting to just take a couple pics of the Blackbird and leave, and instead I ended up having a blast with all the great exhibits, vendors and free junk to clutter up my house. Next year I'll be springing for the full show and hopefully bringing my wife and a couple friends. It's a great time to be in free computing and the interest has never been higher. I won't put any jokes in about this being the year of the Linux desktop, but I think it really is the year of Power ISA being back on the desktop. It's been gone for too long and it's good to see it roaring back.

Just watch out for the robots.

Ubuntu LTS 14.04.6 available

This is mostly relevant to our 32-bit PowerPC colleagues, but along with the recent updates to Ubuntu 16 and 18 comes what will likely be the final release of Ubuntu 14.04 LTS (read the change summary). This release is primarily security-focused, mostly to deal with the APT redirect vulnerability, though there are of course other fixes. PowerPC and POWER8 users still stuck on Ubuntu 14 should strongly consider upgrading to Ubuntu 16, which still supports 32-bit PowerPC, also supports POWER8 (but not POWER9), and is still receiving fixes as Ubuntu 14.04 will reach end of life in April 2019. Meanwhile, ISO images are available.

New Talos PowerAI SKU

If you, ahem, want to see how good the POWER9 is at computer vision -- or any other kind of deep learning -- Raptor now announces a new Talos II package, the PowerAI Development System (TL2PA1). This is a T2 Lite with a 4-core CPU, 32GB RAM, 128GB NVMe flash and most notably an NVIDIA RTX 2070 GPU. Debian is pre-installed.

Wait, did you say NVIDIA? Yes, because the intention with this system is to run IBM PowerAI Vision, and an NVIDIA NVLink-capable GPU is required (and so is a trial license). That automatically wouldn't make this a very good Talos workstation due to NVIDIA's historically poor open-source support (nouveau or bust since Power isn't supported by NVIDIA's proprietary driver), and to be sure, Raptor seems to be discouraging it for that purpose ("There is no way to add OpenGL support to the proprietary driver stack ... This system is designed for GPU compute, and while a minimal 2D framebuffer is supported 3D applications will fall back to non-accelerated LLVMPipe rendering"). However, if you want lots of threads and a system to run a high-performance computer vision platform, you've now got a choice which is at least freer than a comparably configured x86 box. Base price starts at $3450 and the SKU should be shipping soon.

A programming note: I'll be wandering the exhibit halls of the Southern California Linux Expo in Pasadena on Saturday. See you at booth 429 or be less of a nerd than I am.

Linux 5.0

Linux 5.0 has been released (what used to be "4.21"). Linus is very clear that no kernel release is a feature release and the 5.0 divide is more an arbitrary numerical cutoff than anything else, but there are some important advancements in Linux 5.0 such as improved AMD GPU support, support for AMD FreeSync, file system improvements (especially to encryption performance), continued Y2038 work and various additional device support. I suspect that Talos users will find improvements to the ASpeed BMC media driver particularly relevant; my guess is this is part of the kernel support that the Blackbird is waiting on.

Probably the most notable Power ISA-specific feature in Linux 5.0 final is support for the POWER On-Chip Controller in POWER8/POWER9, exposing temperature, frequency, power usage and other sensor data through hwmon. This will likely enable Talos-family owners to get even better environmental monitoring support for their machines. Other Power-specific changes include Spectre V2 mitigations for many NXP/Freescale Power CPUs, various KVM improvements and even some improvements for P. A. Semi chips (get that X1000 out and celebrate).

Read the full changelog if you dare, or an annotated summary.

Ubuntu LTS 16.04.6 available

Ubuntu LTS 16.04.6 is available (see the change summary for more details). Although 16.04.6 cannot boot on the Talos family, it does support POWER8, and is particularly relevant to our 32-bit PowerPC friends as it is the final Ubuntu release to support that architecture officially. All Power ISA official releases of Ubuntu are Server branded and do not install a GUI by default. Installation images are hosted at the Ubuntu download site.