Making your Talos II into a Power Mac: KVMPPC for POWER9 (part 1)

UPDATE: On current systems (as of April 2020), see these errata.

Talospace is a spinoff from the TenFourFox Development blog, which for those unfamiliar with it, is a Firefox fork maintained for Power Macs running Mac OS X 10.4 and 10.5. It shouldn't be a surprise that the common architecture was a big plus for me, and it's possible to run OS X with reduced emulation overhead on the processor using the same Kernel-based Virtual Machine (KVM) scheme used for virtualization on other platforms.

Emulation is of course just one of the things us old Mac users would like working properly on the new Power hotness. The other is the damn Command key working like it's supposed to. We'll address that pain point in another "First Person" post coming soooooon.

Anyway, a brief digression before we begin, for those unfamiliar with how KVM works on Power ISA. KVMPPC comes in two flavours, KVM-PR ("PRoblem") and KVM-HV ("HyperVisor"); both work in big and little endian modes. KVM-HV is the more modern of the two and the most technically like hypervisors on other architectures. It uses the hardware support in later Power ISA CPUs, so it's overall faster, particularly when many supervisor-level instructions must be executed. However, it cannot be nested (you can't run a KVM-HV guest inside a KVM-HV guest, though you can run a KVM-PR guest; more on that in a moment), and most importantly, it supports only virtualizing the same processor generation or the one immediately prior. Since no version of OS X ran on a POWER8 (let alone a POWER9), we won't be dealing with it further for the purposes of this article.

That brings us to KVM-PR. Unlike KVM-HV, KVM-PR runs strictly in user mode, or what IBM docs refer to as the "Problem State." It does run as a kernel module, so it's not in userspace, but it does not depend on the hardware which powers KVM-HV and thus only runs user-level instructions. That means it must trap and emulate supervisor-level instructions on behalf of the guest, which is much slower. However, KVM-PR can also emulate other instructions and their desired behaviour, which theoretically allows it to act like any supported Power ISA or PowerPC CPU, including a G3, G4 or G5. Instructions which aren't supported natively are trapped and executed just like supervisor-level instructions, and everything else can still run on the metal. Because it's user mode, it can be nested (a KVM-PR guest can run inside of another KVM-PR guest, as well as inside a KVM-HV guest). KVM-PR was the original method of virtualization on PowerPC Linux, descending from the venerable old Mac-on-Linux project (which had its own peculiar hypercalls), and a specialized form of this method is how OS X runs Classic on 10.4 and earlier. This is the method we will use here.

Let's first talk about whether KVM is the way you want to go. For our Power Mac hardware emulation, we will use QEMU, which can use KVM (and KVMPPC) to accelerate the processor, and QEMU provides the rest of the platform. QEMU provides two platform profiles, g3beige, a Gossamer Beige Power Mac G3, as the name implies, and mac99, essentially a Sawtooth G4. We will only be using the mac99 platform since it provides the best combination of flexibility and compatibility.

QEMU also provides emulated USB devices. The most useful to us is the USB tablet, which allows QEMU to detect when the mouse is within the QEMU window without having to grab it and makes using the emulator a lot more seamless. Unfortunately, the USB tablet is only supported by 10.3 Panther and up. No version of PowerPC Mac OS currently has support for VirtIO devices yet either, so there is no graphics or disk acceleration. On the other hand, QEMU does provide an emulated RTL8139 network card, for which drivers are available for Mac OS 9 through 10.2 Jaguar and are built into at least 10.3 Panther and up, and with tun/tap runs with decent throughput. Sound is best described as a work in progress and graphics work but are basically a dumb framebuffer. Still, this is enough to get the OS off the ground and be useful.

KVMPPC does not work in all situations with QEMU. Most notoriously it does not work for booting Mac OS 9 and Rhapsody, and not 10.0 or 10.1 either, at least from disc. I've done some work on improving this and it gets mostly through the nanokernel startup in OS 9 but doesn't get any further yet. For these operating systems you will currently need to use TCG, QEMU's software CPU emulator, which runs by default if you don't ask for KVM. TCG does have JIT acceleration, and the JIT supports Power ISA, so while it's definitely slower it's at least somewhat better than it sounds. TCG also tends to run a little smoother than KVM since it's all within a user process, but compute-intensive tasks can run up to an order of magnitude slower. TCG is also involved if you run a completely alien inferior architecture like x86.

KVMPPC also tends to be problematic with heavy I/O loads. TCG can be noticeably faster when running installers, for example, or anything that involves substantial emulated disk access. This is probably due to the large amount of supervisor-level code that incurs a speed penalty with KVM-PR. I had better luck and faster install times installing things with QEMU using TCG, then shutting down and rebooting in QEMU with KVM to actually use them.

Finally, KVMPPC only works to mimic certain processors currently. G3 works for every system, and Nitro (G4 7410) works for most of them, but right now that's all. None will boot in KVMPPC with any G4 7450-series processor, and trying to start KVMPPC in 64-bit mode to emulate a G5 currently crashes my Talos. There is also no support for SMP, so our monstrous multi-core beasts will only present one CPU to the emulated OS. The processor you choose doesn't necessarily change the underlying vagaries of the architecture, though, which will be discussed in the next part as well.

Some specific notes on individual versions of Mac OS:

• Mac OS 9, Rhapsody, 10.0 Cheetah and 10.1 Puma do not currently boot on KVMPPC, at least not from CD. They also don't support the USB tablet, so you must click in the window to grab the mouse and keyboard, and hit Ctrl-Alt-G to release the grab to do something else. Rhapsody can be notoriously hard to install and requires multiple steps which I won't discuss here. For OS 9 I'll talk about a couple of glitches with QEMU in Part 2, since many of you will still want to run it even though there is no CPU acceleration.

• 10.2 Jaguar has various problems in KVMPPC, though it does work. Finder windows tend to glitch and not fully load when you doubleclick folders and devices on the desktop. Classic does not work in 10.2 with KVMPPC and aborts with a bus error. 10.2 also does not support the USB tablet, so you need to grab the mouse as with OS 9.

• 10.3 Panther and 10.4 Tiger both run well in KVMPPC. Later on we'll talk about a specific optimization to the operating system "commpage" to make them run even better. 10.3 runs better than 10.4, but 10.4 has better compatibility. Both support the USB tablet and have built-in support for the RTL8139 NIC. Classic will boot and run in both, but is noticeably slower than on a real machine (this is true of both TCG and KVM), though Classic is somewhat faster in Panther.

• 10.5 Leopard appears to work fine in KVMPPC. It supports everything that 10.3 and 10.4 do, though I haven't done the particular commpage optimization for 10.5 yet because I don't use, nor particularly like, Leopard personally. 10.5 obviously does not support Classic.

You'll need to do some preparation to get your Talos II to be an accelerated Mac with KVMPPC (this isn't needed if you're going to use TCG since it's purely userspace). The first is that you need to make sure your T2's MMU is in hash table mode, used by POWER8 and earlier CPU generations. The POWER9 introduces a new MMU mode called radix mode, but without going into the gory technical details, the particular memory mapping characteristics of radix mode mean certain tracts of memory cannot be properly manipulated by KVM-PR. All OSes that support the POWER9 in radix mode will support it in hash table mode. For Linux, just add disable_radix to your kernel command-line arguments. For my Fedora workstation, I just put it into GRUB, regenerated the configuration, and rebooted. If you did this right, dmesg will show a line like this:

[    0.000000] hash-mmu: Initializing hash mmu with SLB

You shouldn't see any mention of radix mode.

The next step is possibly to download a copy of the kernel source code. If you have kernel 4.17.x or earlier (as my Fedora 28 system does), you will need to apply patches to the KVMPPC kernel modules in that version to even get it to start. If you have kernel 4.18.x and up, the necessary patches should already be present for basic functionality, but you may still want to get the kernel source for some of the hack optimizations in this post that aren't (and probably won't ever be) included by default.

Let's assume for didactic purposes that you do need to patch the KVMPPC kernel module that comes with your distro. We will talk about adding the hacks to it a little later. This will taint your kernel. If your system behaves strangely and you are unable to unload the module, you may need to reboot.

1. Download and unpack the source archive, and cd into the root of the unpacked source archive.
2. Download this patch (written by yours truly) into the root of the source archive.
3. patch -p1 < that_patch.diff
4. cp /your/kernel/config .config (assuming you're still in the source archive)
5. make -j24 modules (or if you're one of the lucky scum with more cores, adjust as appropriate; I have two of the 4-core CPUs for 32 threads, but I like to leave a core free)
6. When the make has run to completion, edit include/generated/utsrelease.h to make sure it matches what appears in uname -r, or your kernel may refuse to load the module.
7. Regenerate the KVMPPC modules with the matching string: make -j24 SUBDIRS=arch/powerpc/kvm

Now you can load your custom modules:

cd arch/powerpc/kvm && sudo modprobe -r kvm_hv kvm_pr kvm && sudo insmod kvm.ko && sudo insmod kvm-pr.ko

and you should see something like this in dmesg:

[22198.130998] kvm: loading out-of-tree module taints kernel.
[22198.184535] kvm: module verification failed: signature and/or required key missing - tainting kernel

If you actually got an error message, you loaded the wrong thing, or you possibly forgot the patch (earlier KVMPPC versions won't even start KVM-PR on a POWER9).

If you already have the patches, chances are your OS already loaded KVM-PR. You can check this with lsmod. If it didn't, and trying to load it with sudo modprobe kvm_pr doesn't work, you may need to also patch your kernel modules with the steps above. On the other hand, if you see both kvm_pr and kvm_hv listed, do a sudo modprobe -r kvm_hv (unless you really do need it) to limit your system to KVM-PR and help to simplify the remaining steps in this article.

Next, the third step is to install QEMU. QEMU 3.0 is strongly advised; if your package source doesn't have it, then download and compile from source (and you get to do -O3 -mcpu=power9 anyway for great justice). Although QEMU 2.12 will mostly work for these examples, many bugs and edge cases were fixed in the Mac hardware emulation and some bugs can't be worked around easily any other way. You may also have to remove some command line options from the examples that were not supported in 2.12.

Create your base disk image according to the QEMU instructions and get out your OS X disc. I saw little value in using a raw disk image and it was substantially larger than a qcow2 image, so I'd just use that. We'll assume this is your chosen format for the remainder of this series.

In part 2, we'll talk about how to get all this actually booting.

Nested virtualization coming to POWER9

On the KVMPPC mailing list, Paul Mackerras posted for comments a new set of updates to KVM-HV allowing POWER9 systems in radix MMU mode to finally nest virtualization (i.e., run a virtualized POWER9 guest within another virtualized POWER9 guest through KVM-HV). This is not only a big boon to shops that run Power ISA virtual machines in terms of enhanced security and portability, but also offers the potential for improved debugging and development.

As you will no doubt recall from our previous series on turning your Talos into a Power Mac, the Kernel-based Virtual Machine functionality on Power ISA and PowerPC comes in two flavours: KVM-PR, which emulates supervisor instructions in software and thus is slower but more flexible and can be nested, and KVM-HV, which uses hardware hypervisor support in later Power ISA chips and is faster, but cannot emulate most earlier CPUs and previously could not be nested (though a KVM-PR guest can run within a KVM-HV guest, and additional KVM-PR guests within that).

With these patches, nested KVM-HV guests are now possible, and can run at nearly full speed. Let's define the base hypervisor to be at level 0 ("L0"). L0 can use the hardware virtualization support to run a guest at level 1 ("L1"). An L1 guest, however, currently cannot do the same thing, so it can't spawn any additional nested VMs under its own control. The trick with these patches is to add hypercalls to allow an L1 guest to ask the L0 hypervisor to create another guest on its behalf, but set up address translation that the L1 guest can manipulate. The new guest is actually another L1 guest, but it looks like an L2 guest because L0 will in effect translate the fake L2's addressing requests through the L1 guest that requested it using a combination of instruction emulation and paravirtualization. The emulated L2 guest should be able to then turn around and request a new VM itself, and the L0 hypervisor will make another L1 guest that the faux L2 guest can control that acts like an L3 guest, and thus turtles all the way down.

Because it is still inherently KVM-HV, however, it inherits all of its basic limitations such as only supporting the current processor generation and the one immediately preceding it. In addition, the current nested guest implementation relies on radix MMU mode, the default MMU mode of the POWER9 (KVM-PR requires hashed page table MMU mode), meaning it does not support earlier Power ISA generations that only support hashed page tables. The patches are out for comments on the mailing list and hopefully will be incorporated into the Linux kernel tree in the very near future.

Making your Talos II into a Power Mac: dcbz considered harmful (part 2)

In the first part of this article we talked about getting your Talos II prepped to emulate a Power Mac using KVMPPC, the kernel virtualization facility in Linux. Having followed the instructions in that article, you've got your kernel in hash table mode, you've got the KVM-PR kernel module loaded (and patched it if necessary), you installed (or built) QEMU, and you have a blank QEMU disk image ready to go.

For this part, we will assume you have chosen 10.3 Panther, 10.4 Tiger or 10.5 Leopard to install. I will discuss Leopard relatively little other than how to get you started in it; most of the rest applies to Leopard that applies to Tiger. I'll briefly discuss booting OS 9 with TCG at the end.

Before starting, since we will use tun/tap networking, make sure the interface is up before booting. On Fedora, I do something like this:

sudo ip tuntap add dev tap0 mode tap user [your username]
sudo ip link set tap0 up promisc on

and, if you use libvirt,

sudo brctl addif virbr0 tap0

For filesharing you could set up either Samba or Netatalk. I use Netatalk, since I'm more accustomed to AppleTalk and it enables my T2 to serve files over AFP to the other classic Macs here, and it also will work fine with Mac OS 9 if you want to use that at some point.

Let's begin by constructing the command line to boot your emulated Mac from disc and install the OS. Each OS does better currently with certain combinations of emulated CPU and hardware features. In addition, we also need to make sure that the emulator stays within a single core for better performance (you will get random system stalls if it moves over to another core and throughput will be generally impaired), so we need to set affinities appropriately.

We'll go with 10.4 for our example; substitute for your OS of choice where relevant. Start out with

taskset -a -c 0-3 qemu-system-ppc -M

This binds all of QEMU's threads to a single core (recall that the T2 Sforza cores are SMT-4, and each appear as logical CPUs, so everything must run on a single core this way). While QEMU spawns more than four threads, encompassing two cores (i.e., 0-7) has no noticeable performance benefit and can sometimes unsettle Mac OS X by making timing loops unpredictable.

For the -M option, we will specify mac99 and kvm. The OSes differ on what they prefer for the VIA. 10.3 and 10.4 need to run the emulated mac99 with an emulated CUDA chip onboard, or the OS is unable to detect the real-time clock. 10.5, however, requires the later PMU attached to the VIA. So that gets us to

taskset -a -c 0-3 qemu-system-ppc -M mac99,accel=kvm,via=cuda (10.3, 10.4)
taskset -a -c 0-3 qemu-system-ppc -M mac99,accel=kvm,via=pmu (10.5)

All three of these OSes work fine emulating a 7400-series G4. We will use the "Nitro" 7410 (-cpu nitro), which is a bit faster than the G3 (-cpu G3). 10.3 may have some problems with assigning more than 1.5GB of RAM (-m 1536), but 10.4 and 10.5 work fine with 2GB (-m 2048). Don't use more than 2GB of RAM; it will cause various problems. A verbose boot is helpful in case you accidentally did something wrong (-prom-env boot-args=-v). We'll specify our disk image and some tuning parameters (-drive file=[filename].img,format=qcow2,l2-cache-size=4M), and say boot from the CD or DVD (-boot d -cdrom "/dev/cdrom"). Lastly, we'll enable the emulated RTL8139 NIC and USB tablet (-netdev tap,id=mynet0,ifname=tap0,script=no,downscript=no -device rtl8139,netdev=mynet0 -usb -device usb-tablet) and use a sane screen resolution (-g 1024x768x32). For my 10.4 booter, the full command line looks like this (using the filenames I use on this system):

taskset -a -c 0-3 qemu-system-ppc -M mac99,accel=kvm,via=cuda -cpu nitro -m 2048 -prom-env boot-args=-v -boot d -cdrom /dev/cdrom -drive file=tigerhd.img,format=qcow2,l2-cache-size=4M -netdev tap,id=mynet0,ifname=tap0,script=no,downscript=no -device rtl8139,netdev=mynet0 -usb -device usb-tablet -g 1024x768x32

I strongly suggest saving this as a shell script so that you can make any necessary variations. Insert your OS CD or DVD and run the script. It should go into the installer. If it didn't, make sure your filenames are correct, that you have OpenBIOS installed (it comes with QEMU) in a location the emulator can see, and that the KVM kernel modules (both kvm and kvm_pr) are loaded by checking lsmod.

Once the installer has booted you can of course directly proceed to installation in KVM, but I actually recommend shutting down the emulated Mac at this point and bringing everything back up in TCG to get the OS installed. To do that, just use the same command line, but change accel=kvm to accel=tcg. As I mentioned in the first part, heavy I/O loads tend to be less performant on KVMPPC, and installing and upgrading an OS is a pretty heavy I/O load, so running it in TCG will complete the task more quickly and more reliably.

If you want to run Software Update to bring your emulated Mac up to date, it's probably best to also do this in TCG. You could also separately download one of the combo installers (such as the one for 10.4.11) and push that to the emulated Mac on your Samba or Netatalk AFP share.

When the OS is installed, remove the CD-ROM from your command line unless you want to keep it, and change the -boot argument to -boot c to boot from the emulated drive image.

Ta-daa!

For best results with video updates, make sure that the display settings inside System Preferences match your physical display. I'm in 32-bit colour, so I made sure that System Preferences was using Millions instead of Thousands of colours. Because of variabilities in timing, you may notice the OS X clock is close but may seem to run somewhat unsynchronized from your host's clock because of how the delay loop might have been calibrated at bootup. This is mostly just a nuisance.

The next step is optional, but hacks KVMPPC to improve performance of the emulated Mac. Right now we're actually fooling the operating system; we're not really a G4. In fact, the closest Power Mac relative to the T2's POWER9 is the G5, i.e., the PowerPC 970, which is essentially a POWER4 with some modifications for workstation duty and a bolted-on AltiVec unit. Even though we told the OS we're a G4, this doesn't change the attributes of the CPU, in particular for this case the specific instructions it does and does not support and how certain others are handled.

With "big POWER" IBM removed some of the PowerPC instructions that were infrequently used or scaled badly, such as dcba and mcrxr. You don't need to know what these do; just know they were used in some software, but as of the G5 ceased to exist in hardware. Additionally, the G5 and later big POWER designs (including the POWER9) also have a 128-byte cache line instead of the 32-byte cache line of the G3 and G4, which is relevant to the dcbz instruction as it zeroes an entire cache line and potentially spills it to memory. OS X has adaptations for dealing with these cases (an illegal instruction handler in the first case that simulates the instructions in software, and modified system routines in the second), but that only happens if OS X knows the machine is a G5. In this case, it doesn't, so these adaptations are never installed.

KVM-PR gets around the dcbz problem on later POWER designs, including the POWER9, by scanning every new code page in a 32-bit guest for the dcbz instruction and replacing it with an illegal one it can detect. (Remember, it's still a legal instruction; it just behaves differently.) When executed it faults and falls back to KVM-PR, which simulates a 32-byte dcbz instruction in software, and returns control to the guest. It's not a surprise that this process is quite slow, especially if it gets called in a loop. Unfortunately Apple does just exactly that for clearing memory and the instruction is a major portion of the OS' built-in implementation of bzero, which is also called by memset. This is a hot routine and needs to run fast. The G5 version knows about the cache line difference and accounts for it; the G4 and G3 versions don't, and we're using the G4 version.

Apple, however, also helped us out here a little bit by allowing us to guess where the routine is. This and other major components live in a section of memory called the "commpage," which is always located in the top eight pages of the 32-bit addressing space in every process. It is provided by the kernel as an optimization for fast access to important data and common routines. The bzero routine is virtually unchanged from 10.3 to 10.4, and both start with a very unique instruction (cmplwi cr7,r4,32). If we see this instruction in the commpage, we can be confident we have found bzero. And now that we've found it, we can modify it.

Recall I mentioned that KVM-PR must scan each new executable code page for the instruction and change it. We can alter KVM-PR to detect that unique leader instruction if it's mapping in the commpage, and then monkeypatch in a new routine that doesn't use dcbz and thus won't require slow simulation. To make it more reliable, we know where the location should be, so we'll only patch it if it's actually there. As a bonus we'll also map dcba to nop anywhere in an executable section so that it doesn't need a trip to a special handler either. That is what this patch does.

To build KVMPPC with this patch uses the same steps as we discussed for building and installing the kernel modules in part 1. This patch also applies with -p1.

Does it make a difference? You bet it does. On my system with Geekbench 32-bit on Mac OS X 10.4.11, it improved the overall benchmark by nearly 200 points over the unpatched version, almost all of it in (no surprise) the memory score.

This consequence of masquerading as a different CPU also carries over into which software you can run. Even though this is a G4, you actually have to run the G5 version of TenFourFox, which doesn't have any of the other illegal instructions that aren't patched (just be patient -- it will take TenFourFox almost a full minute to come up). If your software offers a G5 version, you should run that if you can. The discontinuity leads to amusing discrepancies like this one.

Interestingly, TCG on POWER9 actually had errors during SunSpider that the JIT in TenFourFox under KVMPPC doesn't, and even with the warmup was up to twice as slow as KVM at SunSpider. Go TenFourFox!

You'll find that performance is still fairly pedestrian even with KVMPPC. While the OS typically benchmarks my T2 as a "2.04GHz G4" (TCG usually gets computed as somewhere between "900 MHz" and "1.0GHz"), the actual throughput you get varies greatly on workload. Raw CPU performance is a bit better than my Quad G5 scores running single core in Reduced mode, though the Quad running full tilt easily surpasses it (the emulation overhead is only reduced, not eliminated). The numbers get a lot different in applications depending on how their workload is structured. For example, TenFourFox's G5 JIT in KVMPPC gets about 6800ms in SunSpider compared to around 3800ms on a "real" 1GHz iMac G4. Improving these numbers to get parity, and especially getting QEMU to support SMP, will need to be an area of active future development.

Lastly, I mentioned about the best way to run OS 9 on a Talos. Although limited to TCG, it's still pretty snappy, a testament to Mac OS 9's comparatively low system requirements. Mac OS 9 works better with the PMU than the CUDA (or you get problems with the mouse not responding to double clicks reliably) and is limited to 1.5GB of RAM. It also doesn't support the QEMU USB tablet, but it does support the RTL8139 with this driver. To get the driver installed, I actually just made an ISO image out of it, dropped it in the Extensions folder and rebooted it. My command line looks like this:

qemu-system-ppc -M mac99,accel=tcg,via=pmu -m 1536 -boot c -drive file=classic.img,format=qcow2,l2-cache-size=4M -usb -netdev tap,id=mynet0,ifname=tap0,script=no,downscript=no -device rtl8139,netdev=mynet0 -rtc base=localtime

Mac OS 9 uses a different real-time clock base, so this has an additional -rtc option. You can use any CPU you want since it's emulated; I just use the default G4 7400 here instead of specifying one.

Post questions or things you've discovered in the comments.

Some updated notes on KVMPPC

A couple quick notes on KVMPPC that came out of the last article on Mac OS 9 in QEMU:

• If you get a kernel Oops when unloading kvm_hv, you are bitten by this bug which should be fixed in recent kernel releases. (Thanks, Paul Mackerras.)

• However, you don't have to do that anymore to use KVM-PR anyway. With QEMU 4.2 (at least on this Talos II running Fedora 31), if you use qemu-system-ppc64 with -M accel=kvm and explicitly specify a CPU which requires KVM-PR (such as, in our previous articles, a Motorola G4/7410 with -cpu nitro), then it will be used without having to unload KVM-HV. In the examples in the second half of the article, just change qemu-system-ppc to qemu-system-ppc64 and it will now "just work" (otherwise you'll get a weird error trying to start QEMU). That is, of course, assuming you have your kernel in HPT mode instead of radix, which is the default on POWER9.

• Likewise, if you specify -cpu host (or power8 on a POWER9), KVM-HV should be used automatically, though you can explicitly specify you want HV with -M accel=kvm,kvm-type=HV as in our AIX on OpenPOWER example. Just don't tell IBM.

If you don't understand what all that meant, read our original article for a brief primer.

Making your Talos II into an IBM pSeries

This post has been updated with new information. Thanks to Zhuowei Zhang, the author of the post we reference, for pointing out QEMU did add SMP support for emulated pSeries hardware. Read on.

In our previous series on turning your Talos II into a Power Mac, we spent most of our time with the KVM-PR virtualizer, the "problem state" version of KVMPPC, which is lower performance but has no hardware dependencies and can emulate a great number of historical Power CPUs (including the G3 and G4, which were of most relevance to those articles).

Recently, however, someone pointed me to this blog post on running IBM's proprietary AIX operating system under QEMU and asked about how well this would work on the Talos II. AIX runs on IBM's own POWER hardware and thus affords a good opportunity for exploring KVM-HV, the hardware-assisted hypervisor flavour of KVMPPC, so let's find out.

Parenthetically I should say that I have a very long history with AIX: my first job out of college in 1997 was mostly working on a medium-size PA-RISC university server running HP-UX 10.20, but we also had a number of RS/6000 machines for E-mail running AIX 3.2.5 that I had access to as well. The RS/6000s are, of course, early implementations of the POWER architecture. In 1998, I ended up with an Apple Network Server 500 running AIX 4.1.4 (and later 4.1.5) that became the first floodgap.com until it was decommissioned in 2012. Its replacement was a 2-way SMT-2 IBM POWER6 p520 Express running AIX 6.1 TL.mumble with some hand-rolled patches, and this system still runs floodgap.com and gopher.floodgap.com today. I also have a couple of the oddball PowerPC ThinkPads, a ThinkPad "800" whose SCSI controller fuse got blown by a SCSI2SD upgrade, and a fully functional ThinkPad 860 with a German keyboard running AIX 4.1.5 as well.

I should also add that the licensing situation with AIX on non-IBM hardware is sticky. I may give the lawyers a heart attack with this oversimplification, but the salesdroids I worked with back in the day essentially had the rule that if you own IBM hardware that can run AIX, then you may run it, because you were considered to have an implicit license simply by possessing the hardware. This situation changed after IBM introduced pSeries hardware that was not allowed to run AIX, starting with the original POWER5 OpenPower machines: even though they are IBM hardware, they are not licensed for AIX, even though you allegedly could coerce AIX to run on at least a subset of these machines with some work.

This handwavy "some work" is what QEMU provides. There is enough of a pSeries-like environment to at least boot AIX, though some pieces are still missing and the kernel appears able to detect it's running under QEMU. However, whether it functions or not, it may not be legal to run an AIX installation on an OpenPOWER or PowerNV system like the Talos II even under virtualization because OpenPOWER and non-IBM Power ISA systems are explicitly not licensed for AIX. IBM is unlikely to come after you if you're just playing around with it, but you have been warned.

First of all, make sure your system is able to run QEMU under virtualization. You should be running at least kernel version 4.18 (my Fedora 28 T2 has 4.18.16) and QEMU 3.0. Check that kvm_hv shows up in lsmod to make sure it has loaded. You shouldn't need to make any modifications to it for this tutorial. If it hasn't loaded, try sudo modprobe kvm_hv to make sure the modules are enabled (check the dmesg if you get errors). There shouldn't be any problem if your kernel boots in HPT instead of radix MMU mode as mine does to enable KVM-PR.

Next, get bootable media. Although I have a set of install discs for AIX 7, the version I have is too old to boot on POWER9 systems (it's intended for when I get around to it with my POWER6), so for this demonstration we'll simply use the diagnostic image that the author of the blog post above uses. Although any of the diagnostic images compatible with POWER9 will work, download the CD72220.iso image to use the patch tool that author offers. This enables you to boot to a limited root shell to snoop around the filesystem. I haven't gotten around to updating the patcher for the more recent images, but this one will suffice for our purpose.

QEMU provides a graphical console and USB keyboard, but just like a real IBM system, only specific IBM-supplied devices are supported as the AIX console terminal (my own POWER6 requires a particular IBM USB keyboard and mouse, naturally provided at a confiscatory markup, to drive a console powered by a GXT145 graphics card). Since QEMU doesn't know how to provide these devices yet, we'll tell QEMU to provide an emulated serial terminal connected to one of the emulated system's VTYs instead, which will "just work." This emulated serial terminal is provided in the terminal session you run QEMU from, not the main QEMU window.

AIX will boot under TCG, the built-in JITted CPU emulation system. This is very slow but will demonstrate the speed differential versus running with hardware assistance. The same command line provided in the original blog post will work here too (I recommend keeping verbose booting enabled if you run with TCG so you can be reassured QEMU hasn't frozen); substitute your ISO filename below:

qemu-system-ppc64 -cpu POWER9 -machine pseries -m 2G -serial mon:stdio -cdrom iso/aix-72220-patched.iso -d guest_errors -prom-env "input-device=/vdevice/vty@71000000" -prom-env "output-device=/vdevice/vty@71000000" -prom-env "boot-command=dev / 0 0 s\" ibm,aix-diagnostics\" property boot cdrom:\ppc\chrp\bootfile.exe -s verbose"

When QEMU starts, just stay in the terminal session and minimize its graphical console; you won't be using it. Booting under TCG takes about seven minutes on my 32 thread (dual 4-core SMT-4) Talos II with QEMU built with -O3 -mcpu=power9. As the original author indicates, the boot will stall for some minutes (about six on my system) at the define_rspc step. You'll also notice four-digit hex codes appearing at the bottom of the terminal session representing the state of the bootloader which any AIX admin will recognize (real IBM hardware and the Apple Network Server display this on a front LCD or LED panel). Once the system prompts you to press 1 and press ENTER, do so, and it will either enter the diagnostics menu or the root shell depending on if you're using the patched ISO or not. This is sufficient to show it basically works but you will already appreciate this is dreadfully slow for any task of substance.

So, kill the QEMU process (or close the graphical console window) and let's bring it up with KVM-HV this time. SMP is supported, so let's give it four cores while we're at it to start with. You can continue to use a verbose boot if you want but this starts up so quickly you'll probably just find the messages annoying. As above, substitute your ISO filename below (if you get an error saying that the KVM type isn't supported and you know that kvm_hv is loaded, try booting it with just accel=kvm):

qemu-system-ppc64 -M accel=kvm,kvm-type=HV -cpu host -smp 4 -machine pseries -m 2G -serial mon:stdio -cdrom iso/aix-72220-patched.iso -d guest_errors -prom-env "input-device=/vdevice/vty@71000000" -prom-env "output-device=/vdevice/vty@71000000" -prom-env "boot-command=dev / 0 0 s\" ibm,aix-diagnostics\" property boot cdrom:\ppc\chrp\bootfile.exe"

Notice that we are using -cpu host. KVM-HV only supports virtualizing the actual CPU itself or the generation immediately before (-cpu power8 thus should work, but not -cpu power7 or before).

Once started, this virtualized boot shoots straight to the "press 1 on console" message in about 50 seconds on my box (!!), and all the way to the diags menu/root shell prompt in just under one minute. Much faster! As you explore the command line, do note that there are many missing binaries in the miniroot the diags disk provides and the terminal emulation (and my delete key: I manually backspaced with CTRL-H) have many glitches. This is to be expected since this disc was never meant to provide a shell environment and the components of the miniroot exist only to support the diagnostics front end. (In addition, it is not possible to actually configure the terminal correctly from the diags menu and therefore do anything useful, probably due to missing support in QEMU. Even if you enter a valid terminal type, the diagnostics front end will continue to complain the terminal was improperly initialized and prevent you from doing anything further.)

Nevertheless, once you get a root shell up, it's interesting to compare lsattr -E -lsys0 on real IBM hardware and on this emulated system. On my POWER6, here are some selected entries (I censored the system ID from the hardware VPD, nothing personal):

ent_capacity 2.00 Entitled processor capacity
frequency 2656000000 System Bus Frequency
fwversion IBM,EL350_149 Firmware version and revision levels
modelname IBM,8203-E4A Machine name
systemid IBM,{censored} Hardware system identifier

But some values are definitely different (and occasionally abnormal) on the emulated pSeries system. Some are even missing outright despite having a placeholder. Here are the corresponding ones from our virtualized 4-core box:

ent_capacity 4.00 Entitled processor capacity
frequency System Bus Frequency
fwversion SLOF,HEAD Firmware version and revision levels
modelname IBM pSeries (emulated by qemu) Machine name
systemid Hardware system identifier

The difference in entitled processor capacity is due to our command line options, but the CPU frequency is oddly unreported and the various other identifiers have different values or are unpopulated. This is possibly how the kernel was able to detect it's running under virtualization.

If you're curious what other hardware support is present, lsdev looks like this (with the given command line):

# lsdev
L2cache0   Available       L2 Cache
cd0        Available       N/A
mem0       Available       Memory
pci0       Available       PCI Bus
proc0      Available 00-00 Processor
proc8      Available 00-08 Processor
proc16     Available 00-16 Processor
proc24     Available 00-24 Processor
rcm0       Defined         Rendering Context Manager Subsystem
sys0       Available       System Object
sysplanar0 Available       System Planar
vio0       Available       Virtual I/O Bus
vsa0       Available       LPAR Virtual Serial Adapter
vscsi0     Available       N/A
vty0       Available       Asynchronous Terminal

The (in)famous AIX smit system configuration tool can be made to work from the command line; try something like TERM=vt100 /usr/bin/smitty to start it. As we say in the biz^tm, "smit happens."^tm Use CTRL-L to repaint the screen if needed; if you see key combinations like "Esc+0," press ESC, release it, and then quickly press the second key. Note that this version of smit is missing quite a few screens and not everything does anything.

To bring down the system cleanly, not like it really matters here, just type exit at the shell, eject the virtual CD if you want to (Y or N), and then indicate to halt the system (H). AIX will respond with Halt completed and QEMU will automatically exit.

IBM used to be a lot more interesting with AIX. AIX 4 in particular offered a lot of workstation features and even a few games (my ANS 500 has AIX ports of Quake and Abuse on it), but modern versions are intended as buttoned-down server OSes and any client functionality is either accidental or secondarily grafted on. That said, after AIX 5L it got a lot easier to build stuff on AIX (either with xlc or gcc) and my full-service POWER6 (web, gopher and E-mail) runs a good collection of servers and utilities I ported myself plus all my old binaries I built on the Apple Network Server without comment. AIX is definitely different (and arguably staid and humourless) and its underpinnings such as the ODM may not be immediately familiar, but it's a tough OS that can take punishment and run like a tank, and I have to admit that I do love the jackboots. Despite having my own real hardware, it is fun to see it boot and run on the Talos even if only in a limited sense.

Serious hash table kernel bug (CVE-2019-12817)

A while ago when I was running my Talos II in hashed page table (HPT) mode for KVM-PR purposes, I started noticing some weird kernel behaviour with recent versions of Firefox. In dmesg were weird messages like this:

[337262.237052] ida_free called for id=170 which is not allocated.
[337262.237089] WARNING: CPU: 6 PID: 12276 at lib/idr.c:519 ida_free+0x114/0x1e0

Initially these were annoying but seemed innocuous. But later on I started getting some weird lockups and I wasn't sure what was going on, so as part of an attempt to figure it out I switched the machine back into radix MMU mode (i.e., I removed disable_radix from the kernel arguments) and the problems disappeared. I reported the phenomenon and the backtraces to the good folks at OzLabs, and Michael Ellerman said he'd look into it.

Turns out the problem was a little deeper than just some weird kernel warnings. From Michael's detailed report, way back around 4.17 code was introduced to support mappings above a 512TB address in the event of a segment lookaside buffer miss. (The SLB has been a tricky devil before.) This situation might seem exotic at first glance, but such addresses are eminently possible in a 64-bit address space. The new code enabled a subtle bug: if a process allocates memory in that range with mmap(), and then forks a new process, the child erroneously maintains the parent's "context ID," a handle-like structure, to that memory mapping and both the child and parent can stomp on the same range of memory. This is obviously bad, but it gets worse when the child process exits, because all of the context IDs it had (including the one it incorrectly inherited) are now freed and thus sets up a use-after-free error where a subsequent unrelated process might get that context ID and access the original parent's space or vice versa. The kernel messages I was seeing was the kernel detecting the situation when both the child and parent exited before a subsequent process got the bogus context ID (and complaining about the double free). The hangups may well have been when the kernel didn't.

There are currently no known circulating exploits for this flaw but it's pretty clear this could be the basis of a nasty attack if a malicious sort were able to trigger such a mapping and then use it to victimize a subsequent process. All 64-bit POWER and PowerPC systems that use a 64K page size are vulnerable in HPT mode (but the Adelie Linux people can smirk a little here, because they use 4K pages, which are not vulnerable). Prior to POWER9, all systems are HPT, so this means everything is affected from the G5 on up including the PA6T and POWER4/5/6/7/8. Additionally, if you use KVM-PR on your POWER9, emulate a POWER8 guest in KVM-HV, or use KVM-PR within a KVM-HV guest, you are also vulnerable because your machine/guest must be using HPT mode. 32-bit PowerPC systems are unaffected.

Most POWER9 systems are probably using the radix MMU by default. If you aren't, then you should, at least temporarily (I haven't switched back to HPT on my own Talos yet, fortunately). dmesg will tell you in the first few lines:

[0.000000] dt-cpu-ftrs: setup for ISA 3000
[0.000000] dt-cpu-ftrs: not enabling: system-call-vectored (disabled or unsupported by kernel)
[0.000000] dt-cpu-ftrs: final cpu/mmu features = 0x0000f86f8f5fb1a7 0x3c006041
[0.000000] radix-mmu: Page sizes from device-tree:

If you see hash tables referenced instead, then you are in HPT mode, and you should remove disable_radix from your kernel arguments and restart.

If you are on a pre-POWER9 system, however, there is currently no effective mitigation, but the good news is that the patch has landed in the kernel tree. It has made it to the RC for 5.1.15, so it should enter Fedora quickly (however, my F30 system is still showing 5.1.12 as current as of 7:30pm Pacific). Update: Red Hat is tracking this as bug 1720616. Debian has an advisory page and is tracking the flaw. Ubuntu is tracking this issue as USN-4031-1 and already has updates.

It's Talos all the way down

Still can't bear the sticker shock of your very own Talos II, or even a itty bitty Blackbird? Why not do what we all do for the machines we can't own and emulate one instead? (And then decide you like it a lot, and save your pennies?)

QEMU 5.0.0 offers a machine model for the bare-metal PowerNV profile, to which the Raptor systems and other OpenPOWER POWER8 and POWER9 designs intended for Linux (i.e., not PowerVM machines) belong. Using the Talos II firmware image (mostly: one snag to be mentioned), you can boot the machine in QEMU and from there bring up an operating system in emulation. In this article we'll prove it works by bringing up Void Linux for Power (hi, Daniel!) in a variety of configurations. A set-up like this might be enough to test that your software or open-source package builds and runs on OpenPOWER, even if you don't own one yet. In a future article we'll talk about how you can boot your own code on the metal so you can port your favourite OS or build a unikernel.

(For the purposes of this article I'll assume an audience that isn't as familiar with OpenPOWER terminology as our usual readership. Kindly humour me.)

The emulation is imperfect, both if you're emulating it on a real Raptor family system or on an icky PC. While QEMU can emulate an AST2500 (i.e., the ARM-based Baseboard Management Controller, which acts as the service processor and provides the video framebuffer), and QEMU can also emulate a PowerNV system, it doesn't do both at the same time. That means the very lowest levels are actually being simulated here -- you can't watch Raptor's pretty Hostboot display, for example, and only the barest functions of the BMC are simulated enough to allow bring-up, not including the framebuffer. In fact, the hardware profiles we will use here do not in general match a real Raptor system either: we're just virtually plugging in PCI devices that give us necessary functionality, though of course none of the peripheral devices in a Raptor system is Raptor-proprietary. Finally, even though I have tagged this entry with KVM, KVM currently doesn't work right with the QEMU PowerNV machine model even though I'm pretty sure it should be technically possible. Sadly, I tried in vain to do so, could never get KVM-HV to be happy, and ended up kernel panicking the machine with KVM-PR. See if you can triumph where I have failed. In the meantime, naturally you can do everything here on a T2 or Blackbird as well because that's how I did it writing this article, but there is no special acceleration for those systems right now.

The first order of business is the first order of business with any emulator: get the ROMs. Fortunately, no one is going to bust you for pirating a set of these because we're an open platform, remember?

The two pieces required are Skiboot and Petitboot, both of which live in the system's PNOR flash. Skiboot contains OPAL, the OpenPOWER Abstraction Layer. It comes in after the BMC has turned on main power and started the Power CPUs' self-boot engines, which then IPL ("initial program load") Hostboot for the second-stage power-on sequence. When Hostboot completes, it chains into Skiboot, which initializes the PCIe host bus controllers (PHBs) and provides all the basic hardware calls needed by a guest kernel to support the platform. You can think of it as something like an overgrown BIOS. This is the lowest firmware level of an OpenPOWER system that QEMU currently supports emulating.

Skiboot lives only to service a kernel, so it immediately starts one. This initial payload is the bootloader for Petitboot, which is also stored in firmware. Petitboot has a small Linux root (Skiroot) and acts as a boot menu, finding bootable volumes on attached devices or over the network. Having found one (or you select one), it chains into it to start the main OS, and from then on Skiboot will provide platform services via OPAL for this final guest until the system is shut down or restarted. Because it's in firmware, Petitboot is always available, which can come in really handy when you're trying to do system recovery.

The first, best and most dedicated way is to build Skiboot and Petitboot yourself. They are open-source and the process is relatively well documented and automated, and you should know how to do this if you own an OpenPOWER machine anyhow. If you aren't doing this on a real OpenPOWER machine you'll need a cross-compiler, but most Linux distros offer such a package nowadays. Do keep in mind that if it looks like you're building a tiny Linux distro, well, that's because that's exactly what you're doing. The advantage here is you can fool around with the firmware at your leisure, but it requires a bit of an investment in disk space and time.

The second way assumes you have a more casual interest and would prefer to go with something prefab. It's possible if you (or, you know, your "friend") has a Raptor-family system to extract the necessary components right from the BMC prompt. Log into the BMC over SSH (or via direct serial connection) and type pflash -i. You'll see a list of all the partitions stored in the PNOR flash. The ones we want are PAYLOAD (which contains Skiboot) and BOOTKERNEL (which contains Skiroot and Petitboot). The exact addresses may vary from system to system and firmware to firmware.

root@bmc:~# pflash -P PAYLOAD -r /tmp/pnor.PAYLOAD --skip=4096
Reading to "/tmp/pnor.PAYLOAD" from 0x021a1000..0x022a1000 !
[==================================================] 100%
root@bmc:~# pflash -P BOOTKERNEL -r /tmp/pnor.BOOTKERNEL --skip=4096
Reading to "/tmp/pnor.BOOTKERNEL" from 0x022a1000..0x03821000 !
[==================================================] 100%

We skip the first 4K page to avoid the wrapping around each partition. pnor.PAYLOAD is actually compressed and needs to be uncompressed prior to use, so:

root@bmc:~# cd /tmp
root@bmc:/tmp# xz -d < pnor.PAYLOAD > skiboot.lid

Finally, scp both skiboot.lid and pnor.BOOTKERNEL to your desired system from the BMC.

Admittedly we just talked at length about the two ways most of you won't get the firmware, so let's talk about the third method and the way most of you will, i.e., you'll just download it. Currently there is an irregularity about Raptor's present Skiboot build for this purpose: it only boots if you are emulating a single POWER8. That's not a typo. If you use it to boot an emulated POWER9, the guest will simply panic, and the guest will go into a bootloop if you are emulating multiple POWER8 CPUs (necessary if you need a larger number of PCIe devices). This is undoubtedly a QEMU deficiency which will be corrected in future releases. In the meantime, if you just care about playing around using a single POWER8 on a terminal, then Raptor's builds (either from BMC flash or downloaded) will suffice. However, if you intend to emulate a POWER9 or SMP POWER8 system, download QEMU's own pre-built skiboot.lid and use that instead.

For Petitboot, we will extract that directly from Raptor's PNOR images. Assuming you didn't get it using the process above, download the current Talos II PNOR image and decompress it. In the shell_upgrade directory you will see the bzip2-compressed PNOR image. Uncompress that, leaving you with a filename like talos-ii-v2.00.pnor. Download my pnorex extractor tool (it's in Perl, because I'm one of those people) and run it on the PNOR image:

% pnorex talos-ii-v2.00.pnor
Version 1 PNOR archive with 33 entries.
Extracting PAYLOAD at offset 8601.
This is a xz format image.
Wrote 1020K successfully.
Extracting BOOTKERNEL at offset 8857.
This is an ELF executable image.
Wrote 22012K successfully.
Extracted 2 partitions successfully.

If you will be using Raptor's Skiroot, then uncompress pnor.PAYLOAD to skiroot.lid as above: xz -d < pnor.PAYLOAD > skiboot.lid

Now, with skiroot.lid (for this first example, either Raptor's or QEMU's) and pnor.BOOTKERNEL in the same folder, grab an ISO you want to boot. I used the prefab one Daniel offers on the Void Linux for Power site since I know it boots fine on OpenPOWER hardware. For our first example let's do a simple example of booting Void from a CD image on a POWER8 using the serial port. Our QEMU command line:

qemu-system-ppc64 -M powernv8 -m 4G -cpu power8 \
-nographic \
-bios ./skiboot.lid \
-kernel ./pnor.BOOTKERNEL \
-device ich9-ahci,id=ahci0 \
-drive id=cd0,media=cdrom,file=void-live-ppc64le-musl-20200411.iso,if=none \
-device ide-cd,bus=ahci0.0,drive=cd0

This configures a single-processor POWER8 system with 4GB of RAM, no graphics, and an Intel AHCI host controller with a single CD-ROM drive attached. The serial output should go to your terminal. It goes a little like this:

Here we are with Skiboot chaining into Petitboot. You can ignore the errors; there will be a lot of them since the platform is still incomplete. It will take a little bit of time to decompress the kernel (much slower than it would be on a regular system). You will notice a single device attached to the three available PCIe host bridges on the single POWER8 CPU, i.e., the host controller itself. Don't you just love that the vendor code for Intel is 8086?

This is Petitboot. When the bootable choices appear, cursor up to the starred option and press E before it autoboots, because we need to tell Void its console is the on-board serial port (otherwise it uses a VGA console: not sure whose bug that is).

Add console=hvc0 at the end, cursor down to OK and hit RETURN/ENTER a couple times to boot.

A successful login on your emulated baby POWER8. Ta-daa! To rudely pull the plug on the QEMU session, press Ctrl-A, and then X (QEMU: Terminated).

Let's now load out the POWER8. We would like to add a video card, an Ethernet card and a USB controller to our existing system, but POWER8 Turismo chips only offer enough PHBs for three PCI endpoints. How do we solve this problem? Easy: we'll add another processor!

At this point you will require the QEMU Skiboot and should use that where skiboot.lid appears in the remainder of this article. I use tun/tap networking in this example, which assumes you already have tap0 configured and up; change the -netdev setting if you want to use a different means of bridging the NIC. This example keeps the AHCI host controller and still displays debug output on the terminal, but uses the QEMU emulated VGA as a console instead and adds a good old Realtek 8139 NIC with a USB mouse and keyboard attached to a QEMU XHCI USB 3.0 controller.

qemu-system-ppc64 -M powernv8 -cpu power8 -m 4G -smp 2 \
-serial mon:stdio \
-device VGA \
-device ich9-ahci,id=ahci0,bus=pcie.0 \
-netdev tap,id=nic0,ifname=tap0,script=no,downscript=no \
-device rtl8139,netdev=nic0,bus=pcie.1 \
-device qemu-xhci,id=usb0,bus=pcie.2 \
-device usb-mouse \
-device usb-kbd \
-bios ./skiboot.lid \
-kernel ./pnor.BOOTKERNEL \
-drive id=cd0,media=cdrom,file=void-live-ppc64le-musl-20200411.iso,if=none \
-device ide-cd,bus=ahci0.0,drive=cd0

Let's spin this sucker like Superman's cape in a dryer:

The reason I keep the serial output is because the extra CPU adds around an extra minute on this T2 to get to Petitboot. Here, you will notice we now have six PHBs available, three per CPU, so now we have enough virtual PCI slots for the peripherals we require.

Petitboot shows up on both the 2D framebuffer and the serial terminal, and both work. You'll also see it probing the bridged Ethernet tap to see if it can boot that way, proving our Ethernet device is up and working. Whichever you use is where boot messages will go, so we'll use the framebuffer as console and start Void by cursoring up and selecting the starred option (thus also proving our USB devices work too).

Having booted Void, we can now demonstrate the PCI cards in the system, the attached peripherals and the number of CPUs. For the record, the DD2.3 POWER9 I'm typing this on shows its Spectre v2 status as "mitigated" with hardware acceleration.

Starting the Installer, which won't install anything because we haven't configured any storage to install to in our QEMU options. I'll leave that as an exercise to the reader.

If we switch to an emulated POWER9 system, Sforza CPUs support six PCI endpoints, so we get six PHBs. This means a single CPU is more than enough for our basic configuration without adding additional startup time. The QEMU command line to do so merely returns to single processor and changes the machine to powernv9 and the CPU to power9, i.e.,

qemu-system-ppc64 -M powernv9 -cpu power9 -m 4G \
-serial mon:stdio \
-device VGA \
-device ich9-ahci,id=ahci0,bus=pcie.0 \
-netdev tap,id=nic0,ifname=tap0,script=no,downscript=no \
-device rtl8139,netdev=nic0,bus=pcie.1 \
-device qemu-xhci,id=usb0,bus=pcie.2 \
-device usb-mouse \
-device usb-kbd \
-bios ./skiboot.lid \
-kernel ./pnor.BOOTKERNEL \
-drive id=cd0,media=cdrom,file=void-live-ppc64le-musl-20200411.iso,if=none \
-device ide-cd,bus=ahci0.0,drive=cd0

and it runs in the same way, but faster, because the emulation overhead is less. So let's totally do something stupid as our last parlour trick and run a POWER9 configuration with as many sockets as QEMU will let us hold (which right now is four). Note that these are all single-threaded cores, so this is still much less powerful than even a 4-core basic Blackbird.

./qemu-system-ppc64 -M powernv9 -cpu power9 -m 4G -smp 4 \
-serial mon:stdio \
-device VGA \
-device ich9-ahci,id=ahci0,bus=pcie.0 \
-netdev tap,id=nic0,ifname=tap0,script=no,downscript=no \
-device rtl8139,netdev=nic0,bus=pcie.1 \
-device qemu-xhci,id=usb0,bus=pcie.2 \
-device usb-mouse \
-device usb-kbd \
-bios ./skiboot.lid \
-kernel ./pnor.BOOTKERNEL \
-drive id=cd0,media=cdrom,file=void-live-ppc64le-musl-20200411.iso,if=none \
-device ide-cd,bus=ahci0.0,drive=cd0

With four emulated CPUs startup took over seven minutes from start to Petitboot on this dual-8 Talos II, so have patience if you're on a lesser workstation, but it does work:

You can see the watchdog complaining about the length of time OPAL calls are taking now (call 128 resets the XIVE VM interrupt controller on POWER9 chips). But we do have our four cores, and it's not impossibly slow on a beefy enough system (like another POWER9).

Incidentally, while the Power ISA emulation in QEMU allows SMT, it's very basic and not enough to get through the boot-up sequence, or at least not before the heat death of the universe. If you like listening to your cooling fans, see what happens when you try to emulate the biggest baddest dual-22 Talos II by adding -accel tcg,thread=multi -smp 176,threads=4,cores=22,sockets=2 to your QEMU command line. It's not pretty. That's why you should buy an OpenPOWER machine of your own instead of emulating one.

SheepSforza: the SheepShaver Power Mac emulator for OpenPOWER

I've always maintained (possibly by personal experience) that one of the natural future markets for OpenPOWER is one of the architecture's past markets: Power Mac users, particularly those who were irked by the Intel transition. I certainly was, even granting it made commercial sense at the time, and I ended up using a Power Mac G5 Quad for 13 years as a daily driver before I got the Raptor Talos II I use now.

I first touched a Mac in 1987 (it was my buddy's father's Mac Plus, and we spent hours messing around in HyperCard and System 6), though the first Mac I personally owned was a second-hand Macintosh IIsi. I upgraded from there to (briefly) a used Power Mac 7200, then traded it in for a used Power Mac 7300, then pimped that out, then a Power Mac G4 MDD, the first computer I personally bought new, and then the G5. With the exception of the G5 all of these systems natively ran the classic Mac OS, so I had a large investment in classic Mac software. Even my G5 only ever ran Mac OS X Tiger so that I still could run Classic applications (on top of the fact I preferred Tiger's interface to Leopard).

This is relevant because of the current state of Mac emulation: in general, the classic Mac OS is better supported than Mac OS X. SheepShaver started on BeOS and the PowerPC-based BeBox as a commercial product and pun on the Amiga 68K Mac emulator ShapeShifter; it only runs the classic Mac OS, and only then up to 9.0.4 (later versions require an MMU, which SheepShaver doesn't implement). It achieved surprisingly good speed on modest hardware by heavily patching the operating system (more later) and running most programs as native code directly on the BeBox's twin 603 CPUs, not unlike KVM-PR, though without using any special processor features (instead, this was achieved by patching out supervisor portions of the emulated Mac ROM and running all components, including the nanokernel, in the problem state — today we would call this paravirtualization). SheepShaver works on Mac OS X, too, allowing Power Macs with Leopard to run classic apps at near native speeds, though not as well integrated as the Classic Environment, of course. The ability of SheepShaver to run Mac apps directly on the processor accounts for some of its unusual design decisions that persist even on non-PowerPC architectures either running applications through its JIT compiler (on x86 and x86_64) or with an interpreter (everything else, including aarch64 and Apple silicon). SheepShaver led to Basilisk II, which is a 68K Macintosh emulator, before itself becoming open source. To this day both emulators share substantial amounts of code.

Other than SheepShaver and Basilisk II, other emulators include vMac's descendant Mini vMac (68K), MESS/MAME (68K), PearPC (PowerPC, an exception in that it can only run Mac OS X and various other free OSes), and of course QEMU. QEMU originally could only boot Mac OS X, but later added support for Mac OS 9, and can accelerate running at least OS X with KVM-PR (when it's not broken), though this has some edge glitches, and KVM-PR doesn't work on early versions of OS X or with OS 9 at all. In addition, QEMU has decent emulation fidelity and a well-supported "mini JIT" called TCG when KVM or other virtualizers can't or don't work, but it requires drivers on the guest OS side for device support, is slower to start up, and because of its full system emulation has non-trivial overhead for certain operations.

PearPC, SheepShaver and QEMU are your only options right now for emulating a Power Mac. Back in the day while using QEMU to run my old Mac applications, I explored the two other choices. Neither are officially maintained anymore, and even any unofficial updates to SheepShaver appear to be intermittent and fragmented. More to the point, at that time I couldn't get SheepShaver to run at all, even when I managed to make it compile (more about this in a moment). PearPC does build and appear to run on OpenPOWER systems, but its emulation speed is hideous; the JIT is limited to x86, and without it the project estimates it runs about 500 times slower than actual performance, which certainly matches my experience with it. I did some tinkering with adjusting refresh rates and other attempts to reduce the overhead, but it still ran abysmally bad, and I ended up abandoning it.

SheepShaver's long history was attractive, though, and having networking and native filesystem support would be a huge plus, so after some alterations to the source code to make it build on POWER9, I decided to look at why it wouldn't start. Its design is quite unusual because it originated as a "native" emulator without a JIT, running application code bare-metal on a 32-bit PowerPC CPU, and some of the architectural decisions made as a result have persisted. The classic Mac OS is notable for storing the state of certain important globals in very low memory, starting even from an effective address of 0, requiring any implementation of SheepShaver to ensure that virtual addresses that low can be mapped — even in emulation. However, the known trick of sudo sysctl vm.mmap_min_addr=0 and (if you're on SELinux, which being Fedora I am) sudo setsebool -P mmap_low_allowed 1 to allow it to map the lowest page of memory didn't get it started, and trying to use one of SheepShaver's alternate memory mapping schemes failed during configuration, so it was going to be "real" addressing or bust.

The problem turned out to be that SheepShaver is still pervasively 32-bit, even in a 64-bit configuration, the other major legacy hangover of its internal design. When memory is mmap(2)ed on many 64-bit operating systems with malloc() (and pretty much all ppc64 systems), it gets a full 64-bit address, which immediately crashes the emulator because it only deals with the least significant word. The workaround on x86_64 was to give a specific address to mmap(2) instead of letting it pick anywhere, so I stumbled onto an address that seemed to work for ppc64le and was able to narrow that down based on where SheepShaver's executable code maps. (For Fedora this value was 0x18000000. It should work for other Linuces, but may need adjustment on *BSDs.)

Naturally big-endian code cannot run natively on a little-endian processor. (Parenthetically, it should be possible to get SheepShaver running native on a big-endian POWER9, though you would need a SIGILL handler for PowerPC instructions no longer supported by 64-bit PPC like mcrxr, and something would need to be done about differing cache line sizes or dcbz is going to really ruin your day. I leave this exercise to the reader: I'm fairly confident this would work because G5 systems under Mac OS X run SheepShaver just dandy, but note that's because the operating system handlers are already doing this work for you, and the PowerPC 970 — but not the POWER4 — has a bit in a special HID register where dcbz can be made to act like a G4. Without these provisions G5 Power Macs wouldn't be compatible with Classic or any pre-G5 32-bit application.) SheepShaver does implement an emulated PowerPC CPU using a bespoke library called Kheperix, and Kheperix does have a JIT, but the JIT backend only seems to function for i386 and x86_64. Everything else runs in the interpreter.

Running under the interpreter, incredibly, isn't terrible like it was for PearPC. Kheperix's interpreter is pretty efficient but the main reason is because huge amounts of the operating system are patched out and shortcut into the emulator itself, very unlike QEMU or PearPC, particularly I/O and video. But this is also very flexible: QuickDraw acceleration is even supported, plus, as mentioned, networking and mounting a local directory as a Mac volume (no AppleShare needed), or even a disc in the optical drive, and some rudimentary clipboard synchronization. Recall that the normal state of a pre-OS X Power Mac is to be running 68K code, so SheepShaver achieves most of this magic by executing 68K A-line traps directly and jumping in and out of the Power Mac ROM 68K emulator. Functionally this isn't a problem because SheepShaver only operates as a uniprocessor system anyway, so the so-called Blue Task in the nanokernel is all there is. In fact, Kheperix doesn't emulate any PowerPC supervisor-level instructions except mfmsr (which returns a constant Machine State Register value of 0x0000f072, i.e., big endian, address translation, problem state), and it doesn't even implement SPRGs or most other supervisor-level SPRs. This is less of a functional problem than you might think because the PowerPC Mac OS generally runs drivers in user mode, though SheepShaver doesn't run much of this code anyway. Overall, while the interpreter isn't bad for an interpreter, there are notable gaps and some CPU-bound tasks like StuffIt Expander take way longer than they ought to (like close to five minutes to unpack a 37MB .sit).

I did explore writing a ppc64le JIT with the initial port. The JIT backends were originally auto-generated with a tool that actually pulled binary data from an ELF object and built a header file with the basic operations and processor machine code, and then patches it on the fly. This is a clever idea and should work for any arbitrary architecture, but it's early 2000's code and has trouble with relocation modes in binaries generated by later versions of gcc (and presumably clang), and it doesn't understand ppc64 at all. I did some initial work on this but it ended up being a rather bigger undertaking than I wanted to undertake right now even though it needs to be undertaken only once.

Instead, to juice the interpreter a bit more I looked for complex operations that could be turned into inline assembly language, because since we're running on Power ISA these operations can be trivially done "in hardware." The best example is floating point: things like fctiw/fctiwz, which convert an FPR to integer, or unusual operations like fres and frsqrte, all become one or at most a handful of instructions rather than doing the arithmetic and implementation details manually in C++. Even for operations like fused multiply-add which should be lowerable, a single fmadd or fnmadd more often than not is still faster than what the compiler generates. (C++ is still required for updating the register images, FPSCR/XER and condition codes, of course. However, since things like __builtin_fpclassify suck on Power ISA, at least in gcc, even some of that I rewrote in assembly too.) Population count (popcntw) and some other complex integer operations were accelerated the same way. AltiVec should be accelerable with VMX/VSX, but not many classic Mac applications use it, so I didn't bother this time around.

Another big improvement to apparent performance was to enable hardware cursor support. As configured SheepShaver will rely on the Mac OS to poll the mouse and draw the mouse pointer, and since it's triggering double-emulated 68K interrupts to poll ADB, it really kinda chugs without a JIT even with 60Hz video updates. SheepShaver has support for a hardware cursor which is kept in sync with MacOS's cursor and drawn natively. This works beautifully at least with SDL 2, so I've made it the default and exposed it in the GTK settings dialogue. I also turned off swapping Command and Option by default (and exposed that in the settings dialogue) so I could keep my Mac muscle memory, and adjusted the SDL backend to capture all keystrokes when SheepShaver is foreground so that key combinations don't get eaten by your window manager. Combined with triple-pumping the 1Hz interrupt to better synchronize with the host real-time clock, responsiveness seems pretty darn good now — in fact, in my experience even better than QEMU running Mac OS 9.2.2.

I also fixed the sound system, which didn't work at all initially, even though SheepShaver supports it (using, you guessed it, 68K interrupts to get buffers from the mixer). The issue was AUDIO_S16MSB doesn't seem to work for SDL audio, at least not with my crummy little USB audio device. I changed this to AUDIO_S16SYS, which shouldn't regress big-endian, but then means I need to byte-flip the shorts I get from the Mac side. Latency is substantially, though not totally, reduced by writing inline assembly which generates a byte-swap vector in VMX (lvsl has other uses!) and repeatedly vperms the buffer 16 bytes at a time in a tight loop. Since the usual size of audio data chunks is a full 16K, I also made a heavily unrolled version which does 128 bytes every iteration. Alert sounds in Mac OS no longer have gaps, and most audio plays with only minimal pauses because of fetches from the Mac audio mixer, which will get better when the JIT is written.

Last but not least, the build system now supports link-time optimization by default (pass --disable-lto to configure if you don't want this), and will detect POWER8 and POWER9 CPUs from /proc/cpuinfo and add the appropriate -mcpu flag (pass --disable-cpudetect to configure if you don't want this).

This is now good enough for me to run my personal "big four" productivity applications: Adobe Photoshop 6.0 (it can't run 7.0 since SheepShaver is limited to 9.0.4), Adobe FrameMaker, QuickTime VR Authoring Studio and QuarkXPress. Microsoft Office 98 and 2001 won't run on SheepShaver, but Word 5.1 does (in general 68K applications seem to be more compatible).

Best of all, they interact directly with the T2's filesystem — no need for a local installation of netatalk to exchange files.

Here's QTVR inside the Apple company store circa 1997 or so.

It's also good enough to run Doom, and even play it, though the music trips up a bit over itself. If you turn the music off, it runs rather better.

Marathon and Marathon 2 (for a good time, go over to your cat while she's sleeping and whisper "MARATHON!" in her ear) both play fine.

Unfortunately there are random intermittent things I think are gaps in emulation, and other more consistent things that definitely are. For example, you can hang the Mac immediately by running the Startup Disk CDEV: it makes a weird disk status call with csCode 51 and then locks up. I wrote a little kludge to try calling SysError with dsForcedQuit whenever that call appears, and it does indeed pop up a system error box, but it's blank and nothing else works. Even ExitToShell, while it did exit the app, also hangs things. I'm not sure if I did something wrong with my deep magic jumping back into the 68K fire or whether Startup Disk has the system in an unrecoverable state at that point, and neither possibility is unlikely or mutually exclusive. I left the code in to see if I could do something more with it later.

In fact, because exception handling is basically non-existent in Kheperix, whole classes of applications won't work or crash worse. Much as a system error can take down the entire Mac, an illegal instruction will cause SheepShaver to abort because Kheperix has no facility for dealing with it. For that matter, illegal memory accesses are simply ignored if you enable that option because there is no way to invoke the exception handler in the Mac ROM. You can't use guest-level debuggers as a result because any instructions like trap, twi, etc. are unimplemented: you could decode the instruction, but if the trap condition is satisfied, you can't do anything with it. Some sort of deep plumbing would be required to essentially do what the nanokernel is unable to, and without it you can forget about running CodeWarrior in debug mode, or using MacsBug at all (though maybe you can get away with RealBASIC or Future BASIC). QEMU is a full system emulator, so at least in theory it can do all of these things, and it already can run some applications SheepShaver will never be able to.

Still, it's amped up and functional enough that it's now my go-to Power Mac emulator for when I need one, so now you can go to it too. There are enough OpenPOWER-specific changes here that I've christened this modified version of SheepShaver "SheepSforza" after our favourite Nimbus processor series. The supported configuration is Linux, little-endian POWER9, SDL2, running Mac OS 9.0.4 with a Power Macintosh 7300 Old World ROM. (Which, conveniently, I now own several of including my original machine.) It should work fine with a New World ROM but I find I have better luck with Old World systems, and there is no practical difference in feature support. Configurations with 60Hz video and up to 1GB of RAM work just fine and even a lowly 4-core Blackbird will handle both with ease. Do not attempt to enable the PowerPC or 68K JITs: at best nothing will happen and at worst nothing will happen except a core dump. If you are transferring a disk image, ROM and settings from another installation of SheepShaver, you may wish to re-examine your settings (in particular, I strongly advise enabling the hardware mouse cursor).

To build, grab the source from Github and ensure you have development headers for SDL 2 (SDL 1.2 may work, but I don't recommend it) and GTK, then

cd SheepShaver/src/Unix
./configure
make -j24 # or as you like

and ./SheepShaver to start the emulator. (Don't forget to set low memory with sudo sysctl vm.mmap_min_addr=0 and, if you use SELinux, sudo setsebool -P mmap_low_allowed 1 or it will error out on startup.) A nice wiki explaining further use and configuration on Linux is available, if you are new to SheepShaver, except you should not turn the JIT on yet and slirp is the only currently supported networking option.

What's next to do, as I have time and inclination? Obviously I want to get the JIT running, and obviously I'd like to have a more reliable fashion of injecting code into the main execution context so that we can at least work around some of these bugs even if we can't fix them. SheepShaver does this sort of already but in side-execution contexts, so we can't take over the machine that way. I also want to figure out a way to suspend the emulator like QEMU can so that I can put it in "hibernation" when it doesn't need to be taking up CPU cycles. Since it isn't using a network share and SheepShaver syncs the RTC directly, it should behave just fine in this mode. There are also likely some opportunities for more explicit vectorization in the video update loop.

The other thing to fix is Basilisk II, which should work, but doesn't. There's probably a couple similarly fundamental problems to be solved there too, but once they are, many of the improvements in SheepShaver should work there as well (and because it's emulating a 68K, it's likely to run at a ripping pace on modern OpenPOWER systems even without a JIT).

Back to the Firefox JIT!

Juicing QEMU for fun, ??? and profit!

The number of packages and applications natively available for OpenPOWER continue to grow in just about every distro's package manager, and even if a prebuilt package doesn't exist even more will build from source. But emulation is still going to be a fact of life for Windows-only/x86/x86_64-only (maybe even aarch64-only) binaries we can't rebuild, and KVM only helps us with other Power ISA systems (in fact, it looks like KVM-PR broke and can't boot Mac OS X again, so I guess I'll be diving back into the source), so we need to wring as much speed out of QEMU's emulation engine as possible.

We are fortunate with QEMU in that there is ppc64le support in TCG, the Tiny Code Generator which implements a basic JIT, and the Power ISA TCG backend even emits those tasty newer POWER9 instructions to take better advantage of the processor. Without TCG, QEMU would be dreadfully slow when emulating a foreign architecture. However, unless IBM or some other OpenPOWER hardware developer implements instructions (a la Apple M1) in a future chip that specifically improve emulation of other CPUs (like, I dunno, x86_64), there's very little that can be done to improve the code the Power TCG backend generates and CPU emulation spends most of its time in TCG-generated code.

However, the software MMU that QEMU's CPU emulation uses has pre-compiled portions, and all the devices and components QEMU emulates (like the system bus, video, mass storage, USB, etc.) are also pre-compiled. This gives us an opportunity: with a little extra elbow grease, you can make a link-time-optimized and profile-guided-optimized (LTO-PGO) build of QEMU specific to the particular workload which can run the CPU anywhere from 3-8% faster and video and other devices up to 15% faster depending on the set of devices. While number crunching isn't substantially faster, and the modest CPU improvements don't improve user-mode emulation a great deal, full system emulation's general responsiveness improves and makes using more applications more feasible.

This process is not automated. For Firefox, we make LTO-PGO builds using the internal machinery and our patches for gcc compatibility, which is currently our preferred compiler on OpenPOWER systems. The Firefox build system generates a profiling build first, then automatically collects profiling data with it off a model workload and builds the optimized browser from that profile. QEMU doesn't have that infrastructure right now, but you can do it manually: you configure and compile a profiling build, run your workload with it to create a profile, and then configure and compile an optimized build with the profile thus generated.

I'll give instructions here for both QEMU 5.0 and 5.2, since 5.0 seems to be a bit more performant than 5.2 and has fewer build prerequisites, but 5.2 is more straightforward and we'll do it first. In these examples, I'm optimizing ppc-softmmu so that I can run Mac OS 9, which has never worked properly with KVM-PR; substitute with your desired target, such as x86_64-softmmu. Only do one target at a time, and you will want to do individual builds for each system image — even if you normally use the same executable binary for multiple OSes — because different code paths may be exercised with different workloads and/or configurations.

Let's start with making a profiling build. To do this, we'll add -fprofile-generate to the compiler flags (as well as -flto for LTO). For consistency we'll pass the same set of options to the C compiler, the C++ compiler and the linker (each will ignore options they don't need). In the QEMU source tree,

• mkdir build
• cd build
• ../configure --extra-cflags="-O3 -mcpu=power9 -flto -fprofile-generate" \
  --extra-cxxflags="-O3 -mcpu=power9 -flto -fprofile-generate" \
  --extra-ldflags="-flto -fprofile-generate" --target-list=ppc-softmmu
• make -j24 (or as appropriate: this is a dual-8 Talos II)

Wait for QEMU to build. When it finishes, back up your drive image because you may not be able to shut it down normally and it would suck to damage it inadvertently. With a backup copy saved, run the new QEMU as you ordinarily would on your target workload. For example, my classic script is (assuming you're still in the build directory)

./qemu-system-ppc -M mac99,accel=tcg,via=pmu -m 1536 -boot c \
-drive id=root,file=classic.img,format=qcow2,l2-cache-size=4M \
-usb -netdev tap,id=mynet0,ifname=tap0,script=no,downscript=no \
-device rtl8139,netdev=mynet0 -rtc base=localtime

You should use as close to your normal configuration as possible so that the device drivers you run are factored into the profile.

The first thing you'll notice is that QEMU is now really, really, really slow. Crust-of-the-earth-cooling slow. This is because it's storing all that profile data every time any block of compiled code is executed. As a result you will probably not be able to type or interact with the guest in any meaningful fashion, so let the system boot, grab a cup of a fortifying beverage and and wait for it to get as far as it can. For Mac OS 9, it took several minutes to get to the desktop; for OS X 10.4, it took about a quarter of an hour (with a lot of timeouts in a verbose boot) to even start the login window. At some point you will not be able to usefully proceed any further with the guest, but fortunately you backed up your drive image already, so you can simply close the window.

Go back to the build directory. This time we will tell gcc to build with the generated profile (-fprofile-use), though we will allow it to account for certain changes (-fprofile-correction) and allow compilation to occur even if a profile doesn't exist for a particular target (-Wno-missing-profile) so that it can get through configure cleanly:

• make clean (this doesn't remove the profile .gcda files)
• ../configure \ --extra-cflags="-O3 -mcpu=power9 -flto -fprofile-correction -fprofile-use -Wno-missing-profile" \
  --extra-cxxflags="-O3 -mcpu=power9 -flto -fprofile-use -fprofile-correction -Wno-missing-profile" \
  --extra-ldflags="-flto -fprofile-use -fprofile-correction -Wno-missing-profile" \
  --target-list=ppc-softmmu
• make -j24

Enjoy the new hotness. You should be able to see measurable improvements in the CPU emulation, but more importantly, boot times and responsiveness of the full system emulation should also be improved.

For 5.0.0, the process is a bit more complicated, but it's a bit quicker, so I found it worth it (and it's what I currently use for Mac OS 9). In the QEMU source tree, configure the build:

• ./configure --extra-cflags="-O3 -mcpu=power9 -flto -fprofile-generate" \
  --extra-cxxflags="-O3 -mcpu=power9 -flto -fprofile-generate" \
  --extra-ldflags="-flto -fprofile-generate" --target-list=ppc-softmmu
• make -j24

Run your profile as before. However, you need to preserve the profile before the rebuild because make clean will clobber it.

• tar cvf instrumented.tar `find . -name '*.gcda' -print`
• make clean
• tar xf instrumented.tar
• ../configure \ --extra-cflags="-O3 -mcpu=power9 -flto -fprofile-correction -fprofile-use -Wno-missing-profile" \
  --extra-cxxflags="-O3 -mcpu=power9 -flto -fprofile-use -fprofile-correction -Wno-missing-profile" \
  --extra-ldflags="-flto -fprofile-use -fprofile-correction -Wno-missing-profile" \
  --target-list=ppc-softmmu
• make -j24

Life's golden, and just a little bit zippier. It's not always possible to PGO all the things, but here's one where it makes a noticeable difference.