Showing posts from 2019

Firefox 67 on POWER

Firefox 67 has been released and to my relief (though I now build smoketest builds of Firefox on ppc64le approximately weekly to find such problems) mostly builds uneventfully. It has a number of nice features, including enhanced content blocking, improved full keyboard accessibility and various performance improvements. The marquee GPU-accelerated WebRender isn't on most Linux systems yet but that's coming soon, hopefully. I haven't experimented with it yet myself but the existing GPU acceleration works fine on this Talos II with the BTO AMD WX7100 card (set layers.acceleration.force-enabled to true in about:config).

That brings up the first catch, because I did say mostly uneventfully: changes to profile handling. If you build from mozilla-release as I do and I recommend, you will end up with a "nightly release" version (assuming you don't pass --enable-release, which I advise you don't pass right now). Starting with Fx67 nightlies from any tree will try to create a new profile separate from your previous profile but the old one remains intact. You can explicitly select it from the Profile Manager (pass -P), or, if you know already which profile you want to use, you can specify it with -p (on my system the default profile is unimaginatively called default, ergo, -p default).

The second catch I haven't figured out the cause, whether it's a kernel or a Firefox bug, but periodically it will throw occasional but not infrequent warnings that look like this in dmesg (this is on a 5.0.x kernel):

[337262.237052] ida_free called for id=170 which is not allocated.
[337262.237089] WARNING: CPU: 6 PID: 12276 at lib/idr.c:519 ida_free+0x114/0x1e0

If you are on a distribution where kernel warnings get converted into notifications (like the Fedora machine I'm typing on), this can be rather obnoxious. If you are badly afflicted, you can temporarily turn them off with these instructions. I haven't found the root cause for it yet and it's hardly a great hardship, but it didn't occur in Firefox 66.

As far as the Firefox JIT for POWER9, I'm still plugging along, but other than a minor pull request to the documentation it's still 100% yours truly working on it. Of the remaining pieces the macro assembler is about 2/3rds written, leaving the low level assembler after that, and then trying to make it build. However, I'm also in the midst of a systems update for TenFourFox, which I still have a commitment to maintain in the short term, so any help will get it in your hands faster. Hopefully the commits make it clear how I'm translating the MIPS backend into POWER9, using all that 3.0B goodness (population count instructions! trailing zero count instructions! load PC in one instruction! it's an assembly language candy store!).

It's been a little while since I posted the .mozconfigs I use, so rather than direct you to old entries I'll just reproduce them here. Note that MOZ_PGO and MOZ_LTO don't seem to properly work and may generate defective binaries, thus their absence, and I explicitly pass --disable-release to an opt build because of various minor problems which hopefully we'll eventually smoke out. Adjust the number of cores as you like; this is a dual-4 system, so with 32 threads available I reserve 8 to let me still play Descent II during build runs. :)


export CC=/usr/bin/gcc
export CXX=/usr/bin/g++

mk_add_options MOZ_MAKE_FLAGS="-j24"
ac_add_options --enable-application=browser
ac_add_options --enable-optimize="-Og -mcpu=power9"
ac_add_options --enable-debug
ac_add_options --enable-linker=bfd

export GN=/usr/bin/gn # if you have it


export CC=/usr/bin/gcc
export CXX=/usr/bin/g++

mk_add_options MOZ_MAKE_FLAGS="-j24"
ac_add_options --enable-application=browser
ac_add_options --enable-optimize="-O3 -mcpu=power9"
ac_add_options --disable-release
ac_add_options --enable-linker=bfd

export GN=/usr/bin/gn # if you have it

What's missing in this picture

I've got the case (an inexpensive mATX Silverstone SST-ML03B), I've got the memory (16GB). The PSU, optical drive, wireless keyboard and WiFi should arrive next week. Now, what am I missing? Think think think!

Since the whole idea is a POWER9 system for the more price-sensitive, the trimmings cost about $500 on Amazon (minus tax and shipping) and could probably be found elsewhere for less. I also got in on the 4-core $999 Blackbird bundle special price, so with the 2U HSF and tooling that was $1090 before tax and shipping (now it would be roughly $1380) for a base outlay of about $1600. This is a nice attempt at a barebones 8-core for $1950, also apparently minus tax/shipping. Yes, I know you can get an Intel system for less, so don't even bother posting that. If price is your highest priority, you already know you're in the wrong place, but at least now price can still be a priority for what is a decent libre system regardless.

Obviously the aim for us here in the Floodgap household is to use it as an HTPC and that's how I'll be reviewing it. If you just want it as a workstation or to jam in a closet as a low-end server, you can almost certainly cut this parts list further.

ZombieLoad does not affect POWER9

If it's Tuesday, there must be yet another speculative execution attack debuting with a funny name and this Tuesday's entry is ZombieLoad. ZombieLoad works on the same conceptual basis of observable speculation flaws to exfiltrate data but implements it with a new class of Intel-specific side-channel attacks utilizing a technique the investigators termed MDS, or microarchitectural data sampling. While Spectre and Meltdown attack at the cache level, ZombieLoad targets Intel HyperThreading (HT), the company's implementation of symmetric multithreading, by trying to snoop on the processor's line fill buffers (LFBs) used to load the L1 cache itself. In this case, side-channel leakages of data are possible if the malicious process triggers certain specific and ultimately invalid loads from memory -- hence the nickname -- that require microcode assistance from the CPU; these have side-effects on the LFBs which can be observed by methods similar to Spectre by other processes sharing the same CPU core. Other internal buffers of potential value can also be sussed out by related MDS-style techniques.

Because of the limited bandwidth of the LFBs and the effectively streaming nature of the technique, an attacking process can't select arbitrary addresses and therefore can't easily read arbitrary memory. Nevertheless, targeting easily recognizable kinds of data can still make the attack feasible, even against kernelspace. For example, since URLs can be picked out of memory, this apparent proof of concept shows a separate process running on the same CPU victimizing Firefox to extract the URL as the user types it in. As the user types, the values of the individual keystrokes go through the LFB to the L1 cache, allowing the malicious process to observe the changes and extract characters. By its nature there is much less data available to the attacking process but that also means there is less data to scan, making real-time attacks like this more feasible combined with other attacks or social engineering.

However, ZombieLoad is pretty much irrelevant against POWER9 because the LFBs it attempts to monitor are specific to Intel's implementation of HyperThreading (which is true for really any other SMT implementation other than Intel's; the authors of the attack say they even tried on other SMT CPUs without success, almost certainly AMD, though it is not stated for certain that they tested on Power ISA). Even for unpatched Intel machines the actual risk from this (or even most speculative execution attacks, to be sure) is probably limited because it requires running a malicious process to do the snooping and such processes almost certainly have other, more reliable ways of pwning such machines. The decision to patch may simply come down to how much risk you're willing to tolerate: nearly every Intel chip since 2011 is apparently vulnerable and the performance impact of fixing ZombieLoad varies anywhere from Intel's rosy estimate of 3-9% to up to 40% if HT must be disabled completely.

Blackbird shipments start next week

The release of the Blackbird firmware indicates shipping is imminent and Raptor confirms on Twitter that shipments should start next week. Time to get those mATX cases!

Red Hat Enterprise Linux 8.0

The newly Big Blue Red Hat released Red Hat Enterprise Linux (RHEL) 8.0 today, based on Fedora 28. Because it's based on F28, this release of RHEL should "just work" on the Talos II (F28 was the first Fedora to support it), and mostly whatever applies to F28 applies to RHEL 8, including GNOME 3.28, Wayland-by-default and other changes. Although both big and little endian are apparently supported on the trial evaluation images, the documentation says only little-endian is supported, so it's possible not all parts of the site are in sync yet. Still, if you like your Linux corporate (and paying for it), now that Red Hat is an IBM product it's probably as corporate as you can get.

DAWR YOLO mode coming to kernel 5.2

Thanks to Michael Neuling at OzLabs who gave me the heads up and wrote up the patch. One of my pain points doing development on the POWER9 is that hardware watchpoints are disabled at the kernel level. This is because the CPU will checkstop if a watchpoint is set on cache-inhibited memory such as devices, and if a checkstop occurs will invariably bring the system down. The formal name for the special purpose register governing this feature (recall that Power ISA has three classes of registers, i.e., general purpose, floating point and special purpose) is the Data Access Watchpoint Register, or DAWR. There is no software workaround for this problem, and because a malicious local user could bring the system down without privileges by managing to provoke such a situation, setting such watchpoints via the DAWR is therefore currently disabled for safety. Unfortunately, software watchpoints are sometimes hundreds of times slower than hardware watchpoints and for certain debugging tasks are just about indispensable (such as JIT code generation).

IBM notes this issue as an erratum which implies they see it as a defect and therefore suggests it will be fixed in hardware in the future (it does not affect POWER8). Until then, Michael's patch enables "DAWR YOLO mode" for those of us (like me) who are single users on a workstation who know what we're doing, need hardware watchpoints to debug our software before the heat death of the universe, and accept the risk of system crashes. It creates a debugfs switch at /sys/kernel/debug/powerpc/dawr_enable_dangerous that enables the superuser to (mostly) freely turn access to the DAWR off and on; see the patch for more details. Fortunately this change has been finally queued for kernel version 5.2, which means I hopefully won't have to screw around with a custom kernel for much longer and is very good news for other developers in the same boat. Thanks, Michael!

A friendly FPGA reminder

Raptor is seeking beta testers for the upcoming 2.00 Talos II firmware, including a tease for users of the built-in VGA port. This brings up a question: flashing the PNOR and BMC is relatively straightforward (I've done it from my Quad G5), but flashing the FPGA requires a programmer and not all of us are handy with those -- or even have one. Do we need to do that too to try the beta?

Fortunately, Raptor's great support staff responded to my query and said (emphasis mine), "The FPGA firmware is largely independent of the BMC and PNOR. FPGA updates are only released to improve compatibility with PSUs and chassis components encountered in the wild, and at no point will a BMC nor PNOR update be released that is incompatible with the earliest FPGA revisions." That's very reassuring since I have an early T2 that is basically on the same FPGA flash it came from the factory with.

If that's the case, then, when should you update the FPGA? Raptor Support answered that too: "The only time you need to upgrade the FPGA is when you need functionality a new FPGA release provides, for example to activate the VGA disable jumper or to allow the system to boot with a different, previously problematic PSU."

Looks like I've got something to try over the weekend.

Fedora 30 released (and a big Void)

Not to be outdone by the release of Ubuntu 19.04, Fedora 30 has been released as well. We pay special attention to Fedora here at Talospace since this Talos II runs F29. As with our mini-review of F29, we will be doing a similar mini-review of F30 after a couple weeks when the package repositories should be caught up for ppc64le. Chief amongst the updates is GNOME 3.32, gcc 9, bash 5.0 and PHP 7.3; here is the full change set. One disappointment is that 128-bit long doubles did not make this release as previously scheduled and has been held over to F31, which affects building MAME with gcc (see that post for a possible workaround) and a few other things. It's not clear what caused the delay since the issue plagues relatively few packages overall, but it's just enough to be obnoxious when it does. Until then, though, watch for our mini-review once we're ready to update.

Meanwhile, if you like big ends and you cannot lie, the POWER9 Void Linux port can now boot in big-endian mode (the maintainer clarifies: with glibc). With a little bootstrapping help from Adelie Linux, now you can choose best you suits endianness which (deny can't brothers other you). The plan is to get the Void package repos for POWER9 at parity between the big and little endian versions and then let users pick what's most appropriate for their circumstances, including your choice between glibc and musl on both endiannesses. The big-endian version is also planned to have support for the PowerPC G5. For more information, the maintainer now has updated documentation.

A quick trip to IBM OzLabs

(Before we begin: this post was not sponsored, vetted or in any other way written in official collabouration with IBM; I'm writing this up strictly as a Power ISA bigot enthusiast and for no other purpose or consideration.)

Jetlagged greetings from a lovely trip now back at Floodgap Orbiting HQ in sunny southern California (this post partially composed on Air New Zealand's in-flight WiFi). Many thanks to Hugh Blemings of OpenPOWER whom I met at SCaLE 17x and suggested, since I was going to be visiting family in Australia and having a holiday in Canberra, that I head by IBM OzLabs (where he used to be manager). With the kind indulgence of Leonard Low, the current manager, it actually came together and on a warm autumn day last week my wife and I trooped down the National Circuit to the office.

OzLabs has a very long history as a Linux hacker collective and was one of the first commercial labs set up for Linux and Linux software support. In fact, it is rather infrequently reported that a visit to the Canberra Linux User Group, an indirect ancestor of OzLabs, is where the Linux Tux mascot originated (from a 1994 incident at the National Zoo & Aquarium where Linus Torvalds was actually bitten by a penguin; a somewhat inaccurate sign at the Zoo commemorates the injury). Formed at what was then the Australian division of Linuxcare in 1999, for a period of time OzLabs even provided a Linux CD burning service (in exchange for cookies, which my wife calls biscuits) along with Linux software development and support until the division shut down. Hugh was manager for part of this period along with its resurrection under IBM in 2001 as part of the IBM Linux Technology Center, which Leonard manages now.

The current OzLabs location is in an executive building and isn't purpose-built, as Leonard pointed out somewhat apologetically, but still gets the job done. Today the division concentrates primarily on Linux on Power support (including Skiboot and Petitboot, which actually originated from the Cell-based PlayStation 3) in addition to its hosted projects; in fact, Paul Mackerras, the original maintainer of PowerPC Linux, is today a senior technical staff member working on KVMPPC and very politely reviewed my trivial patches for KVM-PR a few months back.

Leonard greeted us at the entry and took us back where the magic happens.

Now, this is IBM, so there's still the corporate face. (I have a story about this: when I was an AIX sysdweeb back in the antediluvian days, we would regularly get visits from IBM salesdroids. However, a few years ago when I tried to buy my own POWER7 hardware personally, I couldn't get any VARs to take my money probably because I didn't need a service contract. I ended up settling for a lightly used POWER6 from a reseller; that box still runs Floodgap today. Now that I have an executive position in a large municipal department, though in a job unrelated to computing, upon hearing this story the IBM salesdroid servicing the municipal account gave me his card and told me to call him any time.) There are meeting rooms and a decent-sized auditorium space, where my wife was talking academia shop with Leonard while I shutterbugged.

A Thomas Watson-esque THINK mural dominates the wall (I have a THINK notepad as a gift from that salesdroid which I use for late night call notes). There's also a small display case nearby with several items, most notably the head and disk assembly from an IBM 3380 circa 1980. This assembly is a Model J with two actuators each accessing about 630MB each; the 3380's frame carried two of these assemblies for a grand total of 2.52GB in the AJ4 configuration, making the 3380 the first gigabyte storage device. The larger but slower Model K had 1890MB per actuator for roughly 7.5GB in a fully loaded AK4 frame, weighing a foot-flattening 250kg and pulling 6.6kW of power. This assembly alone weighs 32kg, so hope you had your hernia belt on while installing it. Due to its incredible rotational inertia the spindle was stopped by the equivalent of an automotive disc brake.

One disappointment: no photography inside the work area. Although I actually do hold a US security clearance, export regulations are such that pictures taken inside are not allowed, so to ease Leonard's heartburn all the pictures you see in this blog post were taken outside in the public space and I took no pictures within the secure area. However, I tapped out some copious notes and those I'll share with you.

In the prototype area Leonard showed us examples of Romulus and Witherspoon. The Romulus development reference design you should know very well by now: the Talos II is strongly based on it, and apparently the OzLabs developers like the T2 so much they're ordering more as workstations. (Maybe that's why it's currently backordered at Raptor, grr.) Witherspoon is described in Skiboot documentation as "a POWER9 system with NVLink2 attached GPUs"; it is the direct ancestor of the Monza-based AC922 used as nodes in the Summit and Sierra supercomputers (more at the end). There was also a small system with an FPGA prototype BMC under testing. Amusingly, the prototype room also had some historical items, including otherwise nondescript tower systems based on the PowerPC 604, 750 (G3) and 405, none of them Power Macs, and some workstation hardware I recognized from the beige days.

In the server room a POWER9 Zaius (a/k/a Barreleye G2) system was sprawled out on a table. This is an OpenCompute device developed jointly by Google and Rackspace as a successor to the original POWER8-based Barreleye. Although just 1U tall, the system we saw was too wide for IBM's racks, though I did rather like the removable drive bay. It takes LaGrange CPUs (more on Monza and LaGrange in a second).

We also saw the POWER8 Palmetto and Stratton prototypes in the racks, each in this SilverStone ATX case. The Palmetto design emerged as the Tyan GN70-BP010, the first customer-available OpenPOWER system; Stratton became the S821LC, with its close relative Briggs (get it?) as the S822.

Although the pamphlet I stole it from is dreadfully out of date (2008), since I couldn't photograph it you can get a small idea of the lab from this page out of IBM Australia's then-official brochure (warning: large PDF; usual disclaimers apply). While only some of the staff were there due to the Easter holidays, which was poor planning on my part, a relaxed and skilled atmosphere in the relatively open floor plan was evident. We also spotted the continuous integration display (all builds green!) and a modified xkcd that said "Petitboot" instead.

Michael Neuling, another IBM staffer, kindly provided some public chip samples to photograph and we took them outside the secure area. One of them was this POWER8 wafer which I took from a couple different angles. The two-ply "white" strip is test logic; the dies between them have six cores on a 22nm process. The Turismo POWER8 has one of these and the Murano POWER8 has two.

The POWER9 Nimbus scale-out family in OpenPOWER systems (on top of the POWER8's wafer carrier for size comparison, more or less), as elegantly hand-lettered by Mikey. Sforza is the chip we know and love in the T2 I'm typing this into; as implemented in Romulus it provides the most similarity to existing commodity designs and prioritizes PCIe, offering the most of all three (48 PCIe 4.0 lanes). LaGrange and Monza are in the larger form factors with double the memory channels of Sforza, with LaGrange also offering the biggest XBus bandwidth between processor sockets (two lanes, twice that of Monza and Sforza) and Monza the greatest OpenCAPI/NVLink throughput. Knowing this, it makes sense why Rackspace and Google went with LaGrange for Zaius, but IBM used Monza for Witherspoon/AC922 where GPU attachment mattered more.

At one point Raptor made an off-hand mention of a future LaGrange system, but so far nothing more has been heard.

Finally, a couple more items: the Top 500 certificates for Summit at Oak Ridge National Laboratory, TN and Sierra at Lawrence Livermore National Laboratory, CA, currently ranked numbers 1 and 2 as of this writing. Summit has 4,608 nodes based on the later 6-GPU AC922, each with dual 22-core Monza POWER9 CPUs and six NVIDIA Tesla V100 GPUs on a Mellanox dual-rail EDR InfiniBand network; Sierra has 4,320 nodes of an earlier revision with the same CPUs and four GPUs. Last is one of the many patents from the team, this one from 2015 honouring Andrew Bentley, a senior technology architect.

Although I couldn't show you everything, we're still very grateful we could drop by and see how the magic gets into our wonderful OpenPOWER machines. Thanks in particular to Paul, Mikey and Leonard for tolerating our silly questions and disturbing their quiet workplace, to Hugh for getting this all in motion, and all of the OzLabs inhabitants. We had a lovely time!

On our way out admiring the view from the auditorium (left: St Andrews Presbyterian Church; right: Parliament House) for the rest of our vacation, disturbed only by some officious stuffed shirt from the Attorney General's office that didn't like us photographing in a public street.

Broadcom BCM5719 libre firmware coming real soon

Posting from the Southern Hemisphere today, kudos to reader Mark J who sent in a heads-up on the progress the Ortega project has done on reverse-engineering the Broadcom BCM5719's firmware. If that number sounds familiar, it's because it's the very same NIC in the Talos II and Blackbird (see a photograph) and one of the few places left in the Talos family that binary blobs are required. Apparently, the reverse engineering effort is now believed sufficient to create a clean-room implementation.

And where is that implementation? Why, on Github, of course. As of this writing the current firmware is a work in progress, but now that the chip is much better understood it's very likely work will move much more quickly.

The BCM5719 is an interesting chip internally, implementing MIPS II (!) cores that apparently were once part of the receive and transmit machinery but are now mostly relegated to autoconfiguration-like tasks. A MIPS core exists for each port, but only a single application processor engine (APE) is present per chip. Hugo Landau's work on figuring out how to talk to the APE, the most essential component any open-source driver would need to interface with as it implements the sideband interface, is nothing short of heroic. A particularly noteworthy deficiency he discovered was that even though the firmware image for the APE's ARM Cortex-M3 has an RSA signature, nothing actually checks it! That would seem like a terrible rookie mistake on Broadcom's part but it's great news for us. A hearty congratulations to everyone's hard work on a very necessary project.

ArchLinux on POWER9

Another option is available for Linux on the Talos: Arch Linux. Although officially x86_64, alpha installation ISOs are now available which apparently "just work" on the Talos II. No word yet on available packages but if you like your Linux lean and mean on your machine that cost you much green, you'll like what we've seen. (I'll be here all week. Try the veal.) Update: The maintainer has a site up.

Blackbird delayed to May

On Twitter Raptor is reporting that due to supply chain disturbances from the USA-China trade spat, the lower-cost POWER9 Blackbird's ship date has been set back again, this time to around May 15. Raptor states the parts are finally in and logic board manufacturing is beginning, so we can assume this date is reasonably firm. However, functional Blackbird prototypes already exist on this side of the Pacific as evidenced by their additional offer of remote access for software testing, which Raptor says are "identical in all ways except HDMI DDC wiring," and sounds similar to the prototype we saw at SCaLE 17x. That bodes well for good operating system support at launch, which will be particularly critical for the lower-cost "tiny Talos" and hopefully its resulting larger audience. Ours is on order for review; with any luck we're near the first in line!

I wouldn't trust ME with a VISA either

Still think Intel chips are "secure enough"? Presenters at Black Hat Asia have revealed the existence of a previously undocumented Visualization of Internal Signals Architecture (VISA) feature, allowing the signals and internal operations of the chip to be captured at a highly granular level.

Now, this in and of itself isn't so strange. Many chips have debugging features like this, because many chips including our POWER9s are just that complicated, and to get into VISA requires admin-level rights (though one could reasonably ask why it wasn't hardware-disabled at the factory). Of course, if you can combine it with a vulnerability that would give you admin-level rights, like, oh, I dunno, the Intel Management Engine, then you have the entire system bus at your disposal. Although Intel argues that the exploit requires physical access as well, the researchers say no hardware modification is required and hint at other simpler methods of exploit.

A superficial analysis like The (Usually More Perceptive) Register's handwaves this away. After all, the exploit they use was fixed in 2017, and frankly if you could get into ME you'd just do that, so no problem, right? Except for three things: one, Intel didn't document this in the first place, so this is the first we've heard that VISA actually exists in shipping chips; two, what's to say this couldn't be exploited by the next Management Engine (or anything else) flaw that comes down the pike; and three, you can't turn ME or VISA off! You can do various things to hobble ME (including this utility from the authors of this very Black Hat presentation), but you can't get rid of it completely. While in practical terms attacking VISA is unlikely to be something widespread in the near future, Intel's attempt to lock it down under NDAs and secrecy doesn't speak well of how much you can trust an Intel CPU.

Most of all, how many of you wanna bet that this is the last little black box to be found in Intel chips? Yeah, I wouldn't trust ME with a VISA either.

Pitch into the Firefox JIT

On the suggestion of a few people, and the fact my time is somewhat more curtailed due to my attempts to do hal-fassed continuous integration testing on ppc64le Firefox and maintain TenFourFox for a still resilient 32-bit Power Mac crowd (and my day job, which as some of you know has pretty much zilch to do with this type of thing), I've uploaded the work so far on the Firefox JIT to a Github repo so that additional people who want to contribute can. The repo is live now.

I think I've documented most of it in the readme, but here are the highlights. Much like TenFourFox's IonPower, some of which was converted to 64-bit and transplated into this new JIT, the Power ISA JIT uses MIPS as a scaffold to hang new code off of. This is particularly handy since many of the MIPS instruction sequences have similar or direct correlates to PowerPC (like lui ori and addis ori), MIPS also uses a link register, and MIPS doesn't have some of the encoding idiosyncrasies of ARM.

The minimum viable product I'm shooting for is a little-endian 64-bit JIT with Wasm for the POWER9. It's not that big-endian isn't desirable or that it shouldn't run on anything earlier than a Talos, but I also know that for MVPs the last thing you want is having to boil the ocean and these are things that can be added after the fact. We also can use all the nice new P9 instructions and avoid endian glitches in the JIT core by concentrating on LE POWER9.

You can see the work I've done already; pretty much anything with a -ppc64le in it is a file I've either completed (the majority) or started work on (the MacroAssembler). The macro assembler, architectural definition and low-level direct assembler are the major pieces still left to do in the first draft before trying to get it to build. I've based this off Firefox 62 not because there's anything special about that version but because it was the version in release when I got started and having to react to code changes with succeeding versions would have slowed down the first draft. When we get it up there then we can forward port it and pick up any additional changes we need to make to the backend along the way.

Finally, this repo is merely a means to the eventual end of getting the JIT to live in the Mozilla tree and be maintained there. This is simply a temporary measure to get more collabourators and get it off the ground.

I'll still be working on this personally, of course, and I'll be doing it more once I get the current TenFourFox release into beta, but I think that we all want good browser choices on POWER9 and while Firefox is perfectly usable on my Talos, it's certainly not the best it could be. This way people can see progress is being made and those who have the technical chops can contribute to make a great platform even better. Further posts as we reach additional milestones. If you want to contribute, open issues and file PRs and I'll deal with them promptly.

Firefox 66 on POWER

Firefox 66 builds out of the box as Firefox 65 did. In addition, as part of the porting effort, I'm running occasional check builds roughly weekly to make sure we intercept issues on nightly before they get into beta and require a higher threshold of approval for patches. So far no major issues and Mozilla continues to be very amenable to getting the occasional fixes in promptly.

The biggest change in Fx66 is the number of default content processes in this release has jumped from 4 to 8. This is good news on our massively parallel multicore systems generally, but it's possible that the differing memory usage may be what tickles that harmless kernel assertion (though on the other hand, setting it back down to 4 didn't eliminate them completely). If you're on Fedora as I am or another distro that elevates kernel warnings to notifications, you'll probably want to turn off those system notifications until your distro's kernel gets the fix in it; see that article for details.

A close look at the Raptor Blackbird and what I did at So Cal Linux Expo 17

Wow, what a swag haul (big box o'breakfast cereal for scale), and what a fun day at SCaLE 17x, the 17th annual So Cal Linux Expo! I'd actually never been before and now I see why people love to go!

SCaLE is at the lovely Pasadena Convention Center, just a short drive from Floodgap Orbiting HQ in sunny rainy southern California. If you're in the greater Los Angeles area, it's pretty accessible by Metro rail, and there are lots of restaurants and things to do if you're flying in from away. I had lunch with my good buddy Bill (lately of the Linux Journal) and two of his cow-orkers at Islands Burgers just north of the Convention Center.

Even if you just buy an expo-only ticket, there are a huge number of vendor booths from big names like IBM, VMware and even Microsoft (!) all the way down to open source projects like VLC, Krita and Inkscape. Would have loved to have seen a Mozilla booth, though. Just saying. I even wore my Mozilla grey hoodie.

I even renewed my Electronic Frontier Foundation membership in support of the great work they do (make mine Titanium).

But of course the star of the show for your humble writer was our friends at the OpenPOWER Foundation, and they came ready for action with our favourite heavy-duty free computing platform:

Hugh Blemings, executive director, kindly tolerated my shutterbugging and an endless parade of retakes to get everything just right. (By the way, did you notice that Timothy Pearson from Raptor is now on the OpenPOWER board?)

He brought in tow their Debian dual-four-core Talos II, which did presentation duty, and of course a prototype Raptor Blackbird motherboard! For those of you new to the blog, this is Raptor's lower-cost way to get into the Power ISA ecosystem. So let's have a detailed look.

Here's the mATX board itself and a side view of the ports. (I'll zoom in on some items of note in a moment.) You can see the ports for USB 3.0 (two rear with additional headers for two more), 4x SATA, 5.1 analogue audio, S/PDIF digital audio, 3x GigE and HDMI. You can also see the two RAM slots and the x8 and x16 PCIe slots. My grizzled old hacker heart was warmed to see that there is still a good old fashioned serial port there too.

The 2D framebuffer is provided by the AST2500 BMC (ARM11 based with a sidecar ColdFire core), at left, routed to HDMI via an ITE IT66121FN; all three Gigabit Ethernet ports are serviced by the Broadcom BCM5719, at right.

Blackbird has some additional hardware for higher-security applications. At left are the flash chips for the BMC and boot flash with hardware write-protect switches, meaning if you can secure the case, nothing's overwriting the firmware. (I looked on my early-model T2 and can't find any such switches, so this is definitely an improvement. Update: found them, just at a slightly different relative location. Timothy Pearson in E-mail notes, however: "In practice, the current firmware stack is a lot chattier with the Flash than we'd like, so there's still some work to be done before we can roll out write protect in official form to both platforms.") Further anti-tampering security is offered by a Raptor-specific FlexVer connector (PDF).

The single POWER9 CPU socket (four or eight cores), and an interesting unlabeled port. Hmmmmmmm. (Update: Timothy Pearson in E-mail identifies it as an FSI port. "Talos II has one just like it; you can plug an FSP box into that port. The FSP boxes are proprietary and available to IBM partners only (like us), however they're also being replaced with the BMC for the most part, which speaks the same protocol and incidentally can do the same kind of debugging tasks now that the FSP boxes were traditionally used for. We will probably continue to retain the connnector, since even if someone designs an open FSI box to plug into the port it could be useful for various low level hacking (in the good sense) tasks. Note that neither the BMC nor this connector can bypass secure boot if enabled, and certainly FlexVer would immediately thrown an attestation error if anyone even tried.")

And last but by no means least, a four-core POWER9 CPU and the exposed die, which Hugh had on display. What a gorgeous bit of silicon, amirite? I'm still planning a full review of this Power-on-a-budget system when my production unit gets here hopefully in just a couple months.

Some other fun stuff:

The One Laptop Per Child handcrank! It exists! I would have killed for this back in the Give One Get One days.

Purism had a nice showing. In addition to their very impressive line of free(r) laptops, they also had a prototype of the Librem 5 libre smartphone, currently scheduled for Q3 2019. I just bought a Pixel 3 (these pictures were taken with it), or I would be buying one of these. I might anyway. I also reminded them that some of us wouldn't mind buying a non-x86 laptop. Power would be nice, but ARM would be fine too. He duly took it under advisement.

Not to be outdone, System76 had a Thelio system on display. Weird site but sexy case. It would be high on my list if I could get the case by itself and slap a Talos-style system in it. How about it, 76?

Standing guard at the entrance to the the exhibit hall was this huge Tux mosaic, made out of AMD Opterons.

It may be a Linux expo, but that doesn't mean the BSD folks can't be there too. I got a couple "RUN BSD" stickers (a la RUN DMC) for my NetBSD machines and one of the fun flashing devil horns headbands. Still looking forward to the FreeBSD port to POWER9!

There were many great open source projects there, but the photographic winner was this particular famous project:

Rock those hats, guys!

And just a tiny selection more of the many vendors, from whom I shamelessly lifted free stuff, and without whose financial support the Expo would probably not be possible:

Special shoutout to the really plush setup the folks from IBM had:

Distros in the hizzouse:

See if you can figure out which one I wrote. No, go on. I'll wait.

The GNU Project/Free Software Foundation had an appropriately chaotic-good booth, with a very important message:

Finally, a few more open source projects to close us out:

Overall I came expecting to just take a couple pics of the Blackbird and leave, and instead I ended up having a blast with all the great exhibits, vendors and free junk to clutter up my house. Next year I'll be springing for the full show and hopefully bringing my wife and a couple friends. It's a great time to be in free computing and the interest has never been higher. I won't put any jokes in about this being the year of the Linux desktop, but I think it really is the year of Power ISA being back on the desktop. It's been gone for too long and it's good to see it roaring back.

Just watch out for the robots.