Posts

Showing posts from May, 2022

Firefox 101 on POWER


Firefox 101 is out, mostly of interest to developers but with a few liveability improvements. I suspect breakage of our LTO-PGO patch will be a regular occurrence and Fx101 indeed broke the diff again, but after updating rust, cargo and cbindgen the optimized build works fine with this revised PGO-LTO patch and the same .mozconfigs from Firefox 95.

I can't figure out where the bugs are in our POWER9 Ion JavaScript JIT implementation, though I still strongly suspect it's because we use sign-extended 32-bit ints. This is enough to pass the test suite but still breaks with the web. MIPS does too and we are strongly based on the MIPS port, but one wonders if MIPS suffers from the same issues, and I'm quite sure it's not tested anywhere near as well. As 32-bit arithmetic instructions aren't orthogonal in Power ISA this change would require some significant rearchitecting and I've just come off a really bad week at work, so I'll just try to pull up the current JIT to 102 so that those of you building the new ESR can continue to use Baseline Interpreter and Baseline Compiler, and then it will be time for more major surgery on the source code. You can help (please!).

Mini-review: The Clockwork Pi DevTerm R-01, or RISC-V on the go


This blog is unambiguously pro-Power ISA, not least because I'm a long-time PowerPC bigot to start with, but also I think OpenPOWER — POWER9 specifically — is currently the best option for a practical yet truly open computing platform: competitive performance, auditable stack, solid hardware, and good and steadily improving software support, so there. And that remains my official editorial position as I type this on my trusty Talos II.

But that doesn't mean I'm not other-RISC-curious, and RISC-V gets a lot of ink these days. I'll say for the record I believe much of that ink is hype: RISC-V is only a performance threat on the low end, there's a lot of sizzle and little steak in present hardware choices, and while the ISA may be open the actual implementations vary greatly on that point. I've observed that there are two main markets for OpenPOWER workstations, namely people who want to support alternatives to the x86-ARM duopoly, and those who want a truly libre auditable platform (with some natural overlap between these groups). RISC-V can scratch the itch of the first group, but it's arguable whether the ecosystem does so collectively for the second. That said, all hype machines, to borrow a cow-orker's trenchant expression, easily transform into self-licking ice cream cones and that sort of salivary momentum is why RISC-V is here to stay.

One other thing that RISC-V and OpenPOWER have in common, besides a royalty-free ISA, is that workstations are neither architecture's core market. POWER9 (and Power10 even more so) is still primarily a server and big-iron chip, and extant RISC-V cores mostly lurk in embedded applications (especially the "too cheap for MIPS" segment). But Raptor workstations at least have nerd awareness, while only the even-less-frequently-encountered HiFive Unmatched meets the definition of a RISC-V PC, and the others are just glorified evaluation boards. And even some people complain about the price of that.

For me, though, it wasn't the price that was the problem (I mean, I've got two T2 systems and a Raptor Blackbird and I'm saving my pennies for an Arctic Tern, so I'm all in on POWER); it was the form factor. I'm out of KVM slots and I have too many boxes around the office. If I was going to play with RISC-V as a use-it-like-a-computer user, it needed to be something that I could set up to mess with and tear it down to recover the desk space. Why, it could even be portable. That would be nice. There's no portable OpenPOWER option (yet?), so if there's a totable libre system out there other than those old bizarro Loongsons I'd love to rock one.

So when ClockworkPi announced they were making a RISC-V spin of their DevTerm "portable terminal," and for just US$240 to boot, I said, "Take my money. No, seriously, take it." So they took it, and yesterday about two months later it arrived, fresh from off the DHL boat from COVID-infested Hong Kong Hangzhou Manifest Tech Co Ltd (wait, did I buy a car or a computer?). Today I'll tell you about it.

Here's what this review isn't: in general it's not a review of the CPi DevTerm itself, though necessarily I'll mention some things of relevance. There are many of these reviews based on its previous iteration using aarch64 CPUs (mostly Cortex-A53 and A72) and the R-01 is literally just a DevTerm with a core module swap. Everything you'd like or hate about the form factor largely applies to both flavours, so refer to any of those existing reviews to determine whether you'd want a DevTerm at all regardless of what CPU's actually in it. Instead, I'm going to talk about this device specifically as a RISC-V general purpose computer, either if you'd just like a RISC-V machine to play with or to truly use as an alternative system on the road.

So, about that form factor. Although most people liken the DevTerm to the Tandy Radio Shack TRS-80 Model 100 (the most famous member of the Kyotronic 85 family), actually its design cues come more from the Model 100's close relative, the NEC PC-8201A. (Dig the control diamond: don't tell me that's a coincidence. For some reason CPi chose to make its codes separate from the cursor keys.) But I've used my PC-8201A on the road, for an entire month on Penang Island in Malaysia in 2000 where it was my only computer, and its full-size keyboard made it quite liveable. The DevTerm's a bit ... smaller — 65 percent regular size, to be exact. I have thin fingers (even when I was 45 pounds heavier) and wide hands, and I can sort of touch-type on it, or lift it in two hands and two-thumb-type with my hand width allowing my thumbs to just meet in the center of the keyboard. However, if you lack either of these attributes you may not enjoy the experience, though it has enough ports you could still potentially use it as a desktop system with external input devices instead. (The DevTerm keyboard is also not backlit, but I'm not going to ding it for that at this price point.)

Obligatory unboxing:

The box clearly advertises what's in it and, interestingly, the amount of RAM on the board (at 1GB this seems an odd thing to brag about).

Famously, it comes unassembled. Everything is in nice neat trays and it's actually rather fun to unpack. The plastic standoffs and holdfasts on sprues give it a delightful model-kit feel.

What's not included are a USB-C charger and two 18650 Li-Ion batteries. Neither of those is expensive or tough to find, but you'll need one or the other to actually power it up. There is also no paper for the thermal printer (!) it carries onboard, though any office supply store will have that too.

Case design, schematics and connectors are all GPL and on Github. The RISC-V "core," as CPi calls their CPU modules, sits on anti-static foam in one of the compartments:

CPi modules contain basically everything but storage and peripherals. In particular, the CPU/SoC, GPU (if present) and RAM are provisioned on a 200-pin SO-DIMM and can be swapped out (naturally you'd need to change the operating system at the same time, but that can be as simple as another microSD card). This is a strength of the design because if you don't like the experience with one CPU card (too slow? not compatible?), try another. CPi sells them inexpensively; if you already own an ARM DevTerm you can just buy the RISC-V module for US$29 and download the software. CPi kindly included a RISC-V build of their Ubuntu-based OS with this unit.

In this case, the CPU is an Allwinner D1, a single-core 1GHz RV64IMAFDCVU CPU based on the XuanTie C906 with on-board 2D graphics, DSP, audio/video, USB and SDIO. The DevTerm exposes HDMI, audio and USB external ports directly serviced by the D1. This is the same part in the Sipeed Nezha evaluation board and in a forthcoming SBC from Pine64. Impressively the RTL for the C906 core is open source and Apache-licensed, but unfortunately the rest of the design isn't: in particular the onboard graphics, DSP and peripheral controllers are only available as blobs, and only then if you register as a developer with Allwinner. For CPi's purposes this isn't a killer because their other CPU options are also blobularly blobtastic, but it's a minus in our book. The Hynix chip next to it is the RAM, a single gigabyte as stated. I cannot find any L2 cache at all, just the 32K L1 caches each for I+D.

Another minus is that the C906 and related cores, while they advertise vector instructions, predate the current RISC-V vector instruction standard. The instructions are similar but they are neither binary nor source compatible. On the other hand there's hardly anything supporting the current standard anyway, so this may not be a problem in the long run if the install base becomes large enough.

Getting out the clear orange scaffold and installing the screen, here's a size comparison against a DVD case to give you another idea about how big it is:

As you can see they're roughly the same size. If you'd find typing on a keyboard about half the size of a lengthwise DVD case difficult, the DevTerm is probably not for you. Anyway, let's finish putting it together.
Ta-daa! It took me about half an hour and it was pretty easy with no tools required. The manual goes step by step though they use a lot of part code shorthand that needed some flipping back and forth to check exactly what standoff, etc., I was supposed to be using where. While most of it can be taken apart as easily as it came together, some are glued stick-on components like the little on-board speakers and most notoriously the wireless antenna. I also didn't like the fact the flex cables are a little nervewracking to install and it took a little fiddling to convince myself they were properly seated (plus the video cable in particular has "SCREEN" silkscreened on it, but that end does not connect to the actual screen; it connects to the port marked "SCREEN" on the mainboard). Although it directs you to insert tiny screws to affix the core module to the board, there didn't seem to be any holes drilled on my board and the SO-DIMM socket seemed to hold the core just fine anyway. This might be something specific about the RISC-V module because the assembly manual is generic in scope. Finally, a pro-tip: it's easiest to put in the microSD card during assembly; it felt like I was inserting the card into an empty hole when I did it after, even though it did go in securely.

All that aside I give overall high marks to the fit and finish of the case, though it took me a little time to get the top to mate with the front lip of the bottom. And then there's those Frankensteiny Princess Leia earmuff closures on either side of the screen: they're cute and give it some personality, and they do hold the unit together, and I guess they're better than thumbscrews, but you almost expect them to have some sort of input device functionality and they don't. Missed opportunity, in my opinion. Plus, if you reopen the case the two halves of the closures come apart and have to be snapped back together, which is a little irritating. Once it is together, though, the case feels very sturdy. I liked how it felt in my hands; it didn't feel flimsy or fragile, and it was not excessively heavy even with the batteries in. Total weight according to my kitchen scale is 588g, or about a pound and a quarter.

Booting up! The manual warns it may take up to a minute, which wasn't too far off. The screen is very bright and legible, even considering the 1280x480 resolution is a little odd by modern standards (basically two VGA screens side by side), but it's very glossy and picks up hairs and fingerprints like a magnet. You probably want to have a microfibre cloth around in your bag for this thing.

And booted into CPi's bespoke Ubuntu variant, ClockworkOS. Despite the wiki (it's correct in the on-screen readme), the default username and password are both cpi.
Proof of uname:
ClockworkOS is very lightweight, and thank goodness it is for reasons we'll get into. There's no Wayland crap here; this is Xorg, as G-d Himself intended (especially because — as our frustrations with Wayland on our BMC-only 2D framebuffer Blackbird have proven — Wayland generally does not do well without a GPU even though it's doing better these days than it has). If the window manager looks throwback, it's because it's good old school twm. Fortunately the thumb trackball isn't too bad but there are key combinations for several marquee apps, which I found to be a thoughtful touch.

The status monitor on the right is also very handy, but you'll notice the clock is wrong, and the reason the clock is wrong is because it couldn't contact any time servers. Despite manually creating a network configuration for the house Wi-Fi, the DevTerm wouldn't connect from the front of the house (the Wi-Fi access point is in the server room in the middle) even though every other Wi-Fi capable device I own is able to do so. I was so perplexed by the range I ended up disassembling it again to check if I'd damaged the antenna or if it had come loose when I turned it over to put on the bottom, but the antenna was physically intact and the connector was snapped securely onto the mainboard. The only way it would connect was if it were closer to it. Various Wi-Fi issues have been reported with the DevTerm's relative GameShell, which appears to use the same sort of antenna, though I'm not sure if this is the same specific problem.

There are lots of fun pieces of software pre-installed like DOSBox and Chocolate Doom (and things like GIMP, Inkscape and Xfig if you want to do real work), so I fired up Doom because of course it plays Doom to get an idea of performance. The CPU immediately showed as pegged in the status monitor, which was not encouraging. Neither was gameplay:

I mean, the poor thing's even using a smaller viewport (by default: it started up that way) and the CPU is still straining at 99%. The framerate wasn't slide-show-slow and music and sound effects didn't seem to suffer (use Fn+the volume key to turn up the audio), but you could clearly see it painting each frame.

Web browsing was equally disappointing, but also for different reasons. Firefox on RISC-V is still a work in progress, although there was an Fx94 build at one point; there are patches for Chromium 104, though I wouldn't be caught dead using Chrome or anything derived from it, and neither Firefox nor Chromium are installed in any case. But it does have Qutebrowser, so let's try ... uh ...

Um, okay, how about ELinks?
This worked and might even be more appropriate for the display and CPU anyhow. It was very sprightly. Text for the win. I did go back to Qutebrowser and try QtWebKit despite the warning, and that does start, but ...
Besides the bad rendering, it was also just as slow as playing Doom was. At this point it would seem most appropriate to get an idea of how much oomph this thing actually has. For this we'll use CoreMark, since it's simple, easy to port and verifiable, and it's what many RISC-V vendors cite so this gives you comparison points. ClockworkPi kindly included development tools making it as simple as cloning it from Github and running make.
A couple benchmark results first, using default settings and reporting the highest score obtained: my NetBSD Macintosh IIci (25MHz Motorola 68030, no cache card) gets 8.3 iterations/second, my NetBSD Mac mini (1.5GHz PowerPC G4 7447A) gets 6073.9 iterations/second, my M1 MacBook Air gets 31713.3 iterations/second (single thread) and 171848.9 iterations/second (8 threads), and this dual-8 SMT-4 DD2.3 Raptor Talos II gets 14367.8 iterations/second (single thread) and 430078.6 iterations/second (64 threads). The D1 clocks in at ... 2232.5 iterations/second, just over a third of the performance of the G4 Mac mini, and I can run TenFourFox on that.

Another thing that performs badly on this device: shutting down. It takes literal, honest-to-goodness multiple minutes to power off cleanly. A surprise was the loud "click" it makes when it finally finishes. There doesn't appear to be an obvious way to make it sleep or suspend, though there is a screen saver which turns off the display after a period of inactivity.

I imagine some of these software problems will improve in later iterations, but as it stands this was my out-of-the-box user experience right now (in fairness CPi does say it is a "highly experimental" model, and actively steers new users away from it). I suppose you could try other distros but they may not support (fully) the DevTerm's onboard devices.

Now, something positive: battery life and power consumption seemed really good. The CPU may be weaksauce but it doesn't put a lot of power demand on the hardware either, nor is there a GPU to draw more juice, and the backlight can be easily changed with key combinations or terminal commands. And although there's a tiny little fan on the board I never heard it unless I put my ear right up against it and the device never seemed to get hot even when it was being held at its limits (which was a lot of the time). While the status monitor will show battery percentage (second from the bottom, above the uptime), cat /sys/class/power_supply/axp20x-battery/uevent will display a fuller, different set of statistics that don't always agree. Either way it's thrifty enough I'd estimate you can probably get eight, maybe 10 hours of runtime or more out of a full charge depending on load and battery quality. On the Kill-A-Watt the USB-C charger pulled between 11 and 14 watts depending on CPU load, though even with the CPU pegged in Chocolate Doom it only occasionally drew at the upper end of that range. Incidentally, since the batteries are removeable it's probably more efficient just to shut it down, take them out and stick them in a wall charger at the end of the day.

You might think after all the complaints I've made that I don't like this device. That is absolutely not the case; in fact, I've already become rather fond of it. I'll even go so far as to say that if you want an easy way to try RISC-V and want one you can use like a general purpose computer, and you're not already drunk on the Kool-Aid, then the DevTerm R-01 is your ticket. Clockwork Pi should be commended for offering it and charging less for it on top of that. Offered the choice between a HiFive Unmatched system and the DevTerm R-01, even considering the Unmatched will be somewhat more powerful, I'd still pick the DevTerm. Besides its obvious space and price advantages it's at least got enough grunt to serve as a terminal and do some very basic tasks and do it for hours on easily replaceable batteries, and it comes with sufficient developer tools out of the box that you can test your software on actually available RISC-V silicon today. Plus, with HDMI, USB and Bluetooth, you can just dock it as a desktop system if you don't need it to be mobile. While I certainly had my share of software problems, I suspect they are not at all unique to this particular implementation.

No, my objections here are primarily to the Allwinner D1. For as many claims as RISC-V's proponents make about openness, this chip isn't meaningfully so, and its underwhelming performance doesn't make it worth putting up with. I realize it's aggressively low-end but for crying out loud, it's getting its clock cleaned by a value-spec 2005 Power Mac (the U740 in the Unmatched gets by the G5, but by less than you'd think), and some software still doesn't work on the architecture yet — certainly more so than OpenPOWER. About all it's got going for it is that it can clearly do very well in a portable, power-constrained environment, and it's cheaper than ARM would be in that setting, yet despite such obvious shortcomings it and its progeny are the very chips here and in other upcoming products simply because they exist and survive. Is RISC-V going to be perennially bringing up the performance rear? (Hey, Raptor, want to make a souped-up Arctic Tern in this form factor?) Not for nothing but I don't see anyone selling those ballyhooed Micro Magic parts, and they may well be snake oil. Maybe some future RISC-V system will have sufficient performance, low power usage and full auditability to become a new and self-sustaining libre mobile option, but I'm having a hard time seeing any such CPU on the horizon.

The bottom line: the DevTerm R-01 is fun to play with and makes a stellar introductory RISC-V general purpose computer at a decent price that you can, for some values of "use," use. If you're already a DevTerm owner a measly US$29 for an R-01 module is a slam dunk, and even as a first-time buyer I feel my US$240 wasn't ill-spent. But after this first personal taste of RISC-V, I don't think OpenPOWER has much to worry about right now.

First production Kestrel and Arctic Terns? (OpenPOWER compute modules exist!)


Thanks, D, who's quicker on the draw than I am: this image popped up on the Raptor Wiki. Looks like Kestrel and Arctic Tern made it to full prototyping, and maybe on the way to production!

This image shows a Blackbird being brought up solely with the Kestrel soft BMC (the metadata says the ASPEED BMC was completely disabled), powered by a much more advanced design than the ECP5 card we saw in the last iteration, especially because this is now a set of custom boards. The PCIe carrier card has Ethernet and two HDMI ports to the left, and what looks like JTAG and serial (grey ribbon) on the right near the LEDs. The "hat" board has been incorporated into the carrier with the LPC, FSI (the black cables curling around out and back into frame go to the connector next to DIMM A1) and I2C signals, and a separate ribbon cable carries the TPM lines.

So far, this is mostly just moving components around. But new on this carrier card are two SO-DIMM style module slots, both populated with what looks like the same sort of card, though only the top one seems to be active. These modules are labeled "ARCTIC TERN ECP5 MODBMC (?) CARD AT1MB1 REV 1.01 (C)2022 RAPTOR COMPUTING SYSTEMS, LLC" (there was a rev 1.0? what did we miss?). This is clearly the CPU card on which the Kestrel soft BMC software runs. The BMC flash likely lives on these boards, but not the PNOR, which appears to be on flash chips on the carrier to the left of the LEDs. UPDATE: this thread says the modules have 1GB of DDR3 RAM each (!), and the CPU fan is directly connected to the carrier. They can be accessed remotely.

It really looks like it may be shipping in the very near future and I'm jazzed about how fast Kestrel reportedly can bring the system to power-ready. But there are two more exciting things about this: first, if this is laid out the way it appears to me to be, this means you can have two BMCs for a libre dual-monitor setup right out of the box, no extra PCI cards or firmware blobs required. (Suck it, Nvidia.) Second, and even more notable, this means that OpenPOWER compute modules may soon be a thing! Given Raptor Engineering, I'm sure hoping these will be sold for standalone projects, especially if the onboard Microwatt-core performance is competitive with RPi and other ARM boards. Maybe then the people whining about how much it costs to play with OpenPOWER will finally get something at a lower price point to play with (and then they can complain about that).

That said, we still don't know price or availability yet, and we don't know if there will be a way to add a Kestrel setup without using a precious PCIe slot (after all, the T2 Lite has only three, and the Blackbird just two; hopefully it can be configured to pull power from something else). But there's a lot of good things in this picture and we're looking forward to hearing more in what are no doubt soon-to-come official announcements.

Red Hat Enterprise Linux 9 and AlmaLinux 8.6


And more updates for Linux on OpenPOWER: RHEL 9 and AlmaLinux 8.6. AlmaLinux 8.6 continues on kernel 4.18 from version 8.5 with the usual bug fixes and security updates, plus updated streams for Perl, PHP and log4j and updated toolchains for gcc, LLVM, Rust and Go. ISOs are available. Currently your best bet on OpenPOWER for RHEL-style stability without the RHEL-style price, support for AlmaLinux 8.x is expected through at least 2029.

But hey, let's say in these inflationary times you think support contracts are a good investment. IBM-Red Hat's got you covered with RHEL 9, the first release based on CentOS Stream (version 9) instead of directly off Fedora, meaning this is actually F34 instead of the recently-released F36. This is particularly notable for ppc64le because you won't get the new long double support in F36 (it's "only" glibc 2.34), but then you won't have to deal with the GNOME 42 stupidity either. Fortunately application streams will cover keeping your toolchains current. If you're still on RHEL 8, though, you're still supported until at least 2030, and big-endian systems and others putting along on RHEL 7 will be supported through mid-2024 (with extended support planned for two years after that).

FreeBSD 13.1


FreeBSD 13.1 is now available, notable as it comes in both big-endian and little-endian flavours depending on how you swing (as well as 32-bit ports for Power Macs and the AmigaOne A1222). The biggest change in this version is that 13's shift to the PowerPC ELF ABIv2 makes it binary-incompatible with 12.x (so you may need to rebuild or relink; the release notes also suggest doing kldxref /boot/kernel after successful installation or upgrade). However, there are also fixes for the bootloader on the LE port, OpenSSL performance improvements, a serial console fix for the BE port, a fix for running FreeBSD with HPT superpages enabled on QEMU with TCG (if you're trying before you're buying), and — particularly of relevance to those of us on OpenPOWER hardware — a fix for the AST2500 console on bootup with recent OpenBMC firmware. It's a big update but one that makes OpenPOWER an even better citizen on FreeBSD (now if only NetBSD would get with the program, because at this point it's just embarrassing). Read the release notes, or download.

AOSC for old and new Power


Another choice in OpenPOWER distros, but with another choice for old-school 32-bit PowerPC, too. AOSC/OS (short for "Anthon Open Source Community") is an Debian (formerly OpenSUSE) derivative claiming to have a wide variety of packages and good port parity at the cost of larger space and a generally manual installation process. The desktop experience is KDE, though a server version is also available. Support for POWER8 and up is listed as "experimental" but is available for download, and most packages appear available to ppc64le.

However, another notable feature of AOSC/OS is its planned "retro" spin, including specific support for Power Macintosh, from the G3 through the G5 (the G5 using a 64-bit build). Unfortunately there isn't support yet for other PowerPC or bigendian systems, and the process is even clunkier since it requires another Linux installation as a trampoline, though the trampoline can be an old version. The retro spin also supports Loongson MIPS, Intel i486 and ARM going back to ARMv4; on desktop you have your choice of Trinity, a "spiritual successor" of KDE, or a "retro" X11 experience with IceWM. Currently these downloads are in the Alternative Downloads area.

Fedora 36 released


Fedora 36 is out. This distro is important to me personally as I run it on both my Raptors (my Talos II and my HTPC Blackbird), and even if you don't run it yourself it's a good early warning indicator of future platform issues because new support and features often hit Fedora first. However, I have mixed feelings about this milestone: we get LLVM 14 and gcc 12 and we're finally getting proper long doubles on ppc64le (as part of glibc 2.35), but it's also got GNOME 42, which means my customized, carefully curated to match my muscle memory Mac theme is screwed on GTK 4 apps (but in a worst of all worlds approach GTK 2/3 apps still use any custom theming while new hotness libadwaita apps don't). It might still be better than the alternatives, but it sucks, so in our upcoming usual mini-review (see my F35 one for an example) we might dive a bit into some of the alternative desktop environments offered on Fedora and see how they work on OpenPOWER — I'm thinking about Pantheon, myself. As usual, the review will come out in a week or so once any immediate breakers are identified and mirror sites have caught up. Meanwhile, here's the complete list of changes.

Firefox 100 on POWER


You know, it's not been a great weekend. Between striking out on some classic hardware projects, leaving printed circuit board corpses in my living room like some alternative universe fusion of William Gibson and Jeffrey Dahmer, one of the yard sprinkler valves has decided it will constantly water the hedge (a couple hundred bucks to fix) and I managed to re-injure my calf muscle sprinting to try to get a phone call (it went pop, I yelped and they hung up anyway). But Firefox is now at version 100, so there's that. Besides the pretty version window when you start it up, it has captions on picture-in-picture and various performance improvements.

Fx100 also broke our PGO-LTO gcc patches again; mach now needs to be patched to ignore how gcc captures profiling information or it will refuse to start a PGO build. This is rolled into the ensemble PGO-LTO patch, which works with the same .mozconfigs from FIrefox 95.

Between everything that's been going on and other projects I've wanted to pay attention to I don't think we're going to make the Fx102 ESR window for the POWER9 JIT. I'll still offer patches for 102ESR; you'll just have to apply them like you do for Firefox 91ESR. Meanwhile, I'll keep trying to get the last major bugs out as I have time, inclination and energy, but although I know people want this really badly, we need more eyes on the code than just me.