Showing posts from April, 2019

Fedora 30 released (and a big Void)

Not to be outdone by the release of Ubuntu 19.04, Fedora 30 has been released as well. We pay special attention to Fedora here at Talospace since this Talos II runs F29. As with our mini-review of F29, we will be doing a similar mini-review of F30 after a couple weeks when the package repositories should be caught up for ppc64le. Chief amongst the updates is GNOME 3.32, gcc 9, bash 5.0 and PHP 7.3; here is the full change set. One disappointment is that 128-bit long doubles did not make this release as previously scheduled and has been held over to F31, which affects building MAME with gcc (see that post for a possible workaround) and a few other things. It's not clear what caused the delay since the issue plagues relatively few packages overall, but it's just enough to be obnoxious when it does. Until then, though, watch for our mini-review once we're ready to update.

Meanwhile, if you like big ends and you cannot lie, the POWER9 Void Linux port can now boot in big-endian mode (the maintainer clarifies: with glibc). With a little bootstrapping help from Adelie Linux, now you can choose best you suits endianness which (deny can't brothers other you). The plan is to get the Void package repos for POWER9 at parity between the big and little endian versions and then let users pick what's most appropriate for their circumstances, including your choice between glibc and musl on both endiannesses. The big-endian version is also planned to have support for the PowerPC G5. For more information, the maintainer now has updated documentation.

A quick trip to IBM OzLabs

(Before we begin: this post was not sponsored, vetted or in any other way written in official collabouration with IBM; I'm writing this up strictly as a Power ISA bigot enthusiast and for no other purpose or consideration.)

Jetlagged greetings from a lovely trip now back at Floodgap Orbiting HQ in sunny southern California (this post partially composed on Air New Zealand's in-flight WiFi). Many thanks to Hugh Blemings of OpenPOWER whom I met at SCaLE 17x and suggested, since I was going to be visiting family in Australia and having a holiday in Canberra, that I head by IBM OzLabs (where he used to be manager). With the kind indulgence of Leonard Low, the current manager, it actually came together and on a warm autumn day last week my wife and I trooped down the National Circuit to the office.

OzLabs has a very long history as a Linux hacker collective and was one of the first commercial labs set up for Linux and Linux software support. In fact, it is rather infrequently reported that a visit to the Canberra Linux User Group, an indirect ancestor of OzLabs, is where the Linux Tux mascot originated (from a 1994 incident at the National Zoo & Aquarium where Linus Torvalds was actually bitten by a penguin; a somewhat inaccurate sign at the Zoo commemorates the injury). Formed at what was then the Australian division of Linuxcare in 1999, for a period of time OzLabs even provided a Linux CD burning service (in exchange for cookies, which my wife calls biscuits) along with Linux software development and support until the division shut down. Hugh was manager for part of this period along with its resurrection under IBM in 2001 as part of the IBM Linux Technology Center, which Leonard manages now.

The current OzLabs location is in an executive building and isn't purpose-built, as Leonard pointed out somewhat apologetically, but still gets the job done. Today the division concentrates primarily on Linux on Power support (including Skiboot and Petitboot, which actually originated from the Cell-based PlayStation 3) in addition to its hosted projects; in fact, Paul Mackerras, the original maintainer of PowerPC Linux, is today a senior technical staff member working on KVMPPC and very politely reviewed my trivial patches for KVM-PR a few months back.

Leonard greeted us at the entry and took us back where the magic happens.

Now, this is IBM, so there's still the corporate face. (I have a story about this: when I was an AIX sysdweeb back in the antediluvian days, we would regularly get visits from IBM salesdroids. However, a few years ago when I tried to buy my own POWER7 hardware personally, I couldn't get any VARs to take my money probably because I didn't need a service contract. I ended up settling for a lightly used POWER6 from a reseller; that box still runs Floodgap today. Now that I have an executive position in a large municipal department, though in a job unrelated to computing, upon hearing this story the IBM salesdroid servicing the municipal account gave me his card and told me to call him any time.) There are meeting rooms and a decent-sized auditorium space, where my wife was talking academia shop with Leonard while I shutterbugged.

A Thomas Watson-esque THINK mural dominates the wall (I have a THINK notepad as a gift from that salesdroid which I use for late night call notes). There's also a small display case nearby with several items, most notably the head and disk assembly from an IBM 3380 circa 1980. This assembly is a Model J with two actuators each accessing about 630MB each; the 3380's frame carried two of these assemblies for a grand total of 2.52GB in the AJ4 configuration, making the 3380 the first gigabyte storage device. The larger but slower Model K had 1890MB per actuator for roughly 7.5GB in a fully loaded AK4 frame, weighing a foot-flattening 250kg and pulling 6.6kW of power. This assembly alone weighs 32kg, so hope you had your hernia belt on while installing it. Due to its incredible rotational inertia the spindle was stopped by the equivalent of an automotive disc brake.

One disappointment: no photography inside the work area. Although I actually do hold a US security clearance, export regulations are such that pictures taken inside are not allowed, so to ease Leonard's heartburn all the pictures you see in this blog post were taken outside in the public space and I took no pictures within the secure area. However, I tapped out some copious notes and those I'll share with you.

In the prototype area Leonard showed us examples of Romulus and Witherspoon. The Romulus development reference design you should know very well by now: the Talos II is strongly based on it, and apparently the OzLabs developers like the T2 so much they're ordering more as workstations. (Maybe that's why it's currently backordered at Raptor, grr.) Witherspoon is described in Skiboot documentation as "a POWER9 system with NVLink2 attached GPUs"; it is the direct ancestor of the Monza-based AC922 used as nodes in the Summit and Sierra supercomputers (more at the end). There was also a small system with an FPGA prototype BMC under testing. Amusingly, the prototype room also had some historical items, including otherwise nondescript tower systems based on the PowerPC 604, 750 (G3) and 405, none of them Power Macs, and some workstation hardware I recognized from the beige days.

In the server room a POWER9 Zaius (a/k/a Barreleye G2) system was sprawled out on a table. This is an OpenCompute device developed jointly by Google and Rackspace as a successor to the original POWER8-based Barreleye. Although just 1U tall, the system we saw was too wide for IBM's racks, though I did rather like the removable drive bay. It takes LaGrange CPUs (more on Monza and LaGrange in a second).

We also saw the POWER8 Palmetto and Stratton prototypes in the racks, each in this SilverStone ATX case. The Palmetto design emerged as the Tyan GN70-BP010, the first customer-available OpenPOWER system; Stratton became the S821LC, with its close relative Briggs (get it?) as the S822.

Although the pamphlet I stole it from is dreadfully out of date (2008), since I couldn't photograph it you can get a small idea of the lab from this page out of IBM Australia's then-official brochure (warning: large PDF; usual disclaimers apply). While only some of the staff were there due to the Easter holidays, which was poor planning on my part, a relaxed and skilled atmosphere in the relatively open floor plan was evident. We also spotted the continuous integration display (all builds green!) and a modified xkcd that said "Petitboot" instead.

Michael Neuling, another IBM staffer, kindly provided some public chip samples to photograph and we took them outside the secure area. One of them was this POWER8 wafer which I took from a couple different angles. The two-ply "white" strip is test logic; the dies between them have six cores on a 22nm process. The Turismo POWER8 has one of these and the Murano POWER8 has two.

The POWER9 Nimbus scale-out family in OpenPOWER systems (on top of the POWER8's wafer carrier for size comparison, more or less), as elegantly hand-lettered by Mikey. Sforza is the chip we know and love in the T2 I'm typing this into; as implemented in Romulus it provides the most similarity to existing commodity designs and prioritizes PCIe, offering the most of all three (48 PCIe 4.0 lanes). LaGrange and Monza are in the larger form factors with double the memory channels of Sforza, with LaGrange also offering the biggest XBus bandwidth between processor sockets (two lanes, twice that of Monza and Sforza) and Monza the greatest OpenCAPI/NVLink throughput. Knowing this, it makes sense why Rackspace and Google went with LaGrange for Zaius, but IBM used Monza for Witherspoon/AC922 where GPU attachment mattered more.

At one point Raptor made an off-hand mention of a future LaGrange system, but so far nothing more has been heard.

Finally, a couple more items: the Top 500 certificates for Summit at Oak Ridge National Laboratory, TN and Sierra at Lawrence Livermore National Laboratory, CA, currently ranked numbers 1 and 2 as of this writing. Summit has 4,608 nodes based on the later 6-GPU AC922, each with dual 22-core Monza POWER9 CPUs and six NVIDIA Tesla V100 GPUs on a Mellanox dual-rail EDR InfiniBand network; Sierra has 4,320 nodes of an earlier revision with the same CPUs and four GPUs. Last is one of the many patents from the team, this one from 2015 honouring Andrew Bentley, a senior technology architect.

Although I couldn't show you everything, we're still very grateful we could drop by and see how the magic gets into our wonderful OpenPOWER machines. Thanks in particular to Paul, Mikey and Leonard for tolerating our silly questions and disturbing their quiet workplace, to Hugh for getting this all in motion, and all of the OzLabs inhabitants. We had a lovely time!

On our way out admiring the view from the auditorium (left: St Andrews Presbyterian Church; right: Parliament House) for the rest of our vacation, disturbed only by some officious stuffed shirt from the Attorney General's office that didn't like us photographing in a public street.

Broadcom BCM5719 libre firmware coming real soon

Posting from the Southern Hemisphere today, kudos to reader Mark J who sent in a heads-up on the progress the Ortega project has done on reverse-engineering the Broadcom BCM5719's firmware. If that number sounds familiar, it's because it's the very same NIC in the Talos II and Blackbird (see a photograph) and one of the few places left in the Talos family that binary blobs are required. Apparently, the reverse engineering effort is now believed sufficient to create a clean-room implementation.

And where is that implementation? Why, on Github, of course. As of this writing the current firmware is a work in progress, but now that the chip is much better understood it's very likely work will move much more quickly.

The BCM5719 is an interesting chip internally, implementing MIPS II (!) cores that apparently were once part of the receive and transmit machinery but are now mostly relegated to autoconfiguration-like tasks. A MIPS core exists for each port, but only a single application processor engine (APE) is present per chip. Hugo Landau's work on figuring out how to talk to the APE, the most essential component any open-source driver would need to interface with as it implements the sideband interface, is nothing short of heroic. A particularly noteworthy deficiency he discovered was that even though the firmware image for the APE's ARM Cortex-M3 has an RSA signature, nothing actually checks it! That would seem like a terrible rookie mistake on Broadcom's part but it's great news for us. A hearty congratulations to everyone's hard work on a very necessary project.

ArchLinux on POWER9

Another option is available for Linux on the Talos: Arch Linux. Although officially x86_64, alpha installation ISOs are now available which apparently "just work" on the Talos II. No word yet on available packages but if you like your Linux lean and mean on your machine that cost you much green, you'll like what we've seen. (I'll be here all week. Try the veal.) Update: The maintainer has a site up.

Blackbird delayed to May

On Twitter Raptor is reporting that due to supply chain disturbances from the USA-China trade spat, the lower-cost POWER9 Blackbird's ship date has been set back again, this time to around May 15. Raptor states the parts are finally in and logic board manufacturing is beginning, so we can assume this date is reasonably firm. However, functional Blackbird prototypes already exist on this side of the Pacific as evidenced by their additional offer of remote access for software testing, which Raptor says are "identical in all ways except HDMI DDC wiring," and sounds similar to the prototype we saw at SCaLE 17x. That bodes well for good operating system support at launch, which will be particularly critical for the lower-cost "tiny Talos" and hopefully its resulting larger audience. Ours is on order for review; with any luck we're near the first in line!