❌

Normal view

There are new articles available, click to refresh the page.
Today β€” 1 July 2025Uncategorized

Feedback: working state, mobile rollovers, and IP filtering

30 June 2025 at 03:40

I get questions from people who read my posts and sometimes I answer them in a post. This is one of those times.

...

Someone read my "war room" post and picked up on the part where I spent several weeks trying to get to the bottom of what turned out to be "kill -9 -1" nuking the world on a bunch of FB machines. They asked how I keep track of things during that time, what my working memory is, and is it paper, text files, IRC messages, or just remembering things.

The answer is: back when I used to do that kind of thing, I found it very useful to have a "MMDD" (hey, I'm in the US, so just pretend it's the back half of an 8601 number... you'll see why...) directory, and inside of it, I'd have some short names for whatever I was dealing with that day.

That means today would be 0629, and then something inside of it might be "fbar" or "rsw" or "webi" or something. It was just something to keep all of the crap together and yet away from other things I was doing that day, while also distinguishing it from other times I might've done a "fbar" or "webi" or whatever project, if that makes sense.

These would just live off my home directory, and yes, it would fill up with crap, but I'd batch them up when a year ended. I'd take 01xx through 12xx and move them into "2013" when 2014 began, for instance. So, as you can see, they *are* ISO-8601 dates, sorta, but it's ~/YYYY/MMDD/foobar once it's old, and it's merely ~/MMDD/foobar until then. This is a balancing act between speed and having my homedir fill up with tons of ancient crap.

Any time I needed scratch space for output from things, that's what I used. That might be the output from a bunch of sweeps over the fleet to look for anomalies. Let's say I ran a command that sshed into a few hundred thousand hosts to look for common items in the logs. The stdout/stderr from that would probably be there so I could hit it up multiple times without asking the job runner system for another copy. It's a lot faster that way.

But in terms of troubleshooting things and dealing with permutations, it's hard to beat paper, and I usually end up with some kind of "lab notebook" at any job I work. I can think of examples of those going back over 20 years. The only problem is that the stuff in them really properly belongs to the company so I don't tend to have them after the fact. A great many pages of context have gone in the shredders over time, and not because I didn't ask. I've asked if anyone wanted them, and the answer has always been no.

There were also the internal posts about things, and then running commentary on IRC channels, but those tend to be useful after the fact, or for getting help from other people. My own state-keeping tends to stay on something close at hand and (usually) physically tangible.

As for those <date>/<term> directories, some of them turned out to be rather handy after the fact. A lot of times, something would happen, and then time would pass, and it would break *again* in exactly the same way, and I could think "didn't we deal with these people already?", dig around, find something from six months earlier, and go "AHA!". Now armed with the date, I could pull up the right posts, IRC logs, group messages, graphs, or whatever else.

Considering how much random crap I dealt with in any given day during my time "in the barrel" (kids, ask your parents), that was the only way to make any sense of it later. Without stuff like that, by the end of the week, I'd have no idea what I had been doing on Monday or Tuesday. That's how bad the load got at points.

...

Someone else wrote in and asked if I could do something to improve the overflow calculator thing I hacked up last week. It wasn't particularly usable on mobile, and that's definitely true. I wrote it on a laptop and paid exactly zero attention to what it would look like on a weirdly small screen that's usually taller than it is wide.

It was a fair point, so I took a whack at making it suck slightly less. It's probably still terrible, but in the specific case of me holding my phone upright, it looks halfway usable now. You're no longer forced into "flyspeck-3" mode with the fonts, at least.

CSS is such a mess.

...

I occasionally hear that the site is unreachable from one spot or another. This almost certainly comes down to IP filtering on my part. Perhaps you've read about the influx of web weasels who are scraping the living shit out of everything remotely URL-shaped they can get their hands on. My stuff is certainly in that space, and they show up here regularly.

Besides that, there are also a number of networks which send nothing but straight-up abusive traffic. This is where you look at the logs and see stuff like them using every single IP address in a (v4) /24 to scan for random webshit vulnerabilities. It's like, nice job, fucko, but I don't run PHP here. Anyway, that's pretty solid evidence that a given network is not worth hearing from ever again, and so into the filter it goes.

A whole lot of this happens automatically just based on whatever traffic is sent this way first. Send bad traffic and meet the bit bucket. I don't even find about most of it since it's constant and utterly uninteresting.

Then there are the people who run feed readers which don't play nicely. As previously described, they will get a handful of chances to slow down with 429s, and then if the web server feels like those aren't having an effect, it'll just ignore the traffic for however long it feels like. Again, I'm also not in the loop on this sort of thing. It's all automatic.

Finally, there is a new rub. I taught myself how to ingest BGP data, and so can now trivially go from an IP address to the autonomous system number of whoever's advertising it, including overlapping advertisements (like a /24 out of a bigger /20). Then I wrote something that will dump out an entire AS, and it's not hard to imagine what I do with that.

Enough bad behavior from a host -> filter the host.

Enough bad hosts in a netblock -> filter the netblock.

Enough bad netblocks in an AS -> filter the AS. Think of it as an "AS death penalty", if you like.

As long as clueless network operators continue to let abusive customers bounce around thousands of dynamic IP addresses with not so much as a hint of SWIP data, those entire net blocks will find themselves unable to get to large swaths of the net. That's just how it is, and it's nothing new. Send crap, meet /dev/null.

Dealing with what the web has become is exhausting.

Calculating rollovers

25 June 2025 at 05:54

I've long had a list of "magic numbers" which show up in a bunch of places, and even made a post about it back in November of 2020. You ever wonder about certain permutations, like 497 days, or 19.6 years, or 5184 hours, and what they actually mean?

I've been doing that stuff by hand in a calculator and finally decided to just do it in Javascript and put it online for anyone to try.

So, here's my latest waste of CPU cycles:

My rollover calculator.

I still haven't figured out the Crucial SSD 5184 hour thing, so it's not in there. 5124 hours, sure, I can understand that one, but 5184? 60 more?

Anyway, have fun while the world burns.

rsync's defaults are not always enough

31 May 2025 at 20:39

rsync is one of those tools which is rather useful. It saves you from spending the time and effort on copying data which you already have. It's the backbone of many a mirror site, and it also gets used for any number of backup solutions.

There's just one problem: in the name of efficiency, it can miss certain changes. rsync normally looks at the size and modification time of a candidate file, and if they are the same at both ends, that's the end of any consideration. It won't get any further attention and it moves on to something else.

"So what", you might think. "All files change at least their mtime when someone writes to them. That's the whole point of a mtime."

And yet... I'm writing this post, and here we are.

The keen-eyed observers out there are probably already thinking "ooh, bit rot" and other things where one of the files has actually become corrupted while "at rest" for whatever reason. Those observers are right! That's totally a problem that you have to worry about, especially if you're using SSDs to hold your bits and those SSDs aren't always being powered.

But no, this is something you have to worry about *beyond* that. This is about a "sneak path" that you probably didn't consider. I didn't.

Here, let's run a little experiment. If you have a x86_64 Debian box that's relatively current and you've been backing up the whole thing via rsync for a year or two, go do something for me.

Go run your favorite file-hasher tool on /usr/lib/x86_64-linux-gnu/libfribidi.so.0.4.0 for me. Give it a sha256sum or whatever, or even md5sum if you're feeling brash. Then note the modification time on the file.

Now mount one of your backups and do the same thing on the version of the file that's on the backup device. See anything ... odd? Unusual?

Identical mtimes, identical sizes... and different hashes, right? I spotted this on a bunch of my machines after going "hmmm..." about the whole SSD-data-loss thing.

Clearly, something unusual happened somewhere, and it's been escaping the notice of your rsync runs ever since. I haven't gone digging into the package history for this thing to find out just when and where it happened, and (more importantly) how. It's rather unusual.

If you're freaking out right now, there is some hope. rsync has both -I and -c which promise to not use the quick method and instead will run a checksum on the files. It's slower so you won't want to do this normally, but it's not a bad idea to add this to the mix of things that you do every so many rotations.

I should point out that the first time you do a forced-checksum run, --dry-run will let you see the changes before it blows anything away, so you can make the call as to which version is the right one! In theory, your *source* files can get corrupted, and if you just copy one of those across, you have now corrupted your backup.

Isn't entropy FUN?

Why I no longer have an old-school cert on my https site

25 May 2025 at 01:26

At the start of 2023, I wrote a post talking about why I still had an "old-school cert" on my https site. Well, things have shifted, and it's time to talk about why.

I've been aware of the ACME protocol for a while. I have tech notes going back as far as 2018, and every time I looked at it, I recoiled in horror. The whole thing amounts to "throw in every little bit of webshit tech that we can", and it makes for a real problem to try to implement this in a safe and thorough way.

Many of the existing clients are also scary code, and I was not about to run any of them on my machines. They haven't earned the right to run with privileges for my private keys and/or ability to frob the web server (as root!) with their careless ways.

That meant I was stuck: unwilling to bring myself to deal with the protocol while simultaneously unwilling to budge on allowing the cruft code of existing projects into my life.

Well, time passed, and I managed to crack some of my own barriers. It wasn't by using the other projects, though. I started ripping into them to figure out just how the spec really worked, and started biting off really really small pieces of the problem. It took a particular forcing function to get me off my butt and into motion.

About six months ago, I realized that it was probably time to get away from Gandi as a registrar and also SSL provider (reseller). They had been eaten by private equity some years before, and the rot has been setting in. Their "no bullshit" tagline is gone, and their prices have been creeping up. I happened to renew my domains for multiple years and have been insulated for a while, but it was going to be a problem in 2025.

Giving them the "yeet" was no big deal, but the damn rbtb certificate was going to be a problem. Was I going to start paying even more for the stupid thing every year, or was I going to finally suck it up and deal with ACME?

That still left the problem of overcoming my inherent disgust for the entire protocol and having to deal with all of these encodings they force upon you. My first steps towards the solution involved writing really small and stupid utility functions and libraries that would come in handy later. I'm talking about wrapping jansson (a C library that handles JSON) so that it made sense in my C++ world and I could import JSON (something I use as little as possible). That kind of thing.

This also meant going down some dead-ends, like noticing that libraries existed which would allegedly create certain things (like a JWK) for you, and then realizing that they were not going to make my life any easier. I'd poke at it, reach my limit, and then swear and walk away for another couple of days.

This went on for some time. I have a series of notes where I'd grab a piece of the problem, wrangle it around for a while, get grossed out, and then set it down and go do something else. This just kept happening but I slowly made progress with small pieces that would Do Stuff, and then they'd connect to each other, and so on like this.

One positive development during all of this was discovering this "pebble" test server I could run on an isolated fake system. It would act as an ACME server and would let me harass it with my feeble attempts at implementing a client instead of bothering the real CAs. Even "staging" servers deserve better treatment than active development, after all.

And, well, after a whole lot of mangling and dead-ends and rewrites and other terrible crap, I had an awful little tool that would take a CSR, do all of the idiot dances and would plop out a certificate. I pointed it at Let's Encrypt staging, and it worked. Then I pointed it at their prod site, and _that_ worked. So I did it for the real thing ([www.]rachelbythebay.com), and *that* worked, and I dropped it into place.

Thus, for the past couple of weeks, if you've been hitting the https version of my site, you've been doing it across the new setup.

Now, I took notes about this, and I wanted to share some of my original off-the-cuff thoughts about implementing this for anyone who's similarly broken in the head and wants to see how bad it can be. I will note that I wrote this based on the first thing that worked, and it does not necessarily reflect the implementation I'm on a few weeks later.

...

Make an RSA key for your web site. Then make a CSR for it, setting the CN and adding a matching altname as an extension. No other fields matter. Nobody looks at those, anyway, and none of them will influence your final certificate no matter how prosaic or precise you get in there.

Make an RSA key of 4096 bits. Call it your personal key.

Write something that'll read a CSR. It needs to extract the CN and the SANs - the DNS: ones, at least. Ensure there's actually a CN [*] and actually SANs, and that the CN occurs within the SANs. So, yes, you have to have at least one SAN.

[* - I now know you can run with just SANs. I did not at the time.]

Write something that'll do a HTTP GET to <directory URL> which is given to you by the ACME service operator. It then needs to parse the body as JSON (or die) and extract some strings from the top-level object: "newNonce", "newAccount" and "newOrder" at the very least.

Write something that'll read an RSA key file on disk. It needs to extract the publicExponent (probably 65537, but you never know...) and the modulus. Make it read your personal key from earlier.

If you end up using "openssl rsa -in foo -noout -text" to do this, the modulus is a bunch of printed hex digits, like "00:ff:11:ab:cd:ef:22:33". It hard-wraps the output and also indents the lines, so you get to clean all of that up first.

Skip the first 00 for some inexplicable reason. Take the other bytes of the modulus and turn them into the actual character values, so a literal 0xff, 0x11, 0xab, 0xcd and so on down the line in the same order you find them in the file. Hang on to these values for later.

Write something that'll turn an ordinary integer into its equivalent big-endian bag of bytes, but don't pad it out to any particular alignment. You need to take that "65537" from your publicExponent and turn it into the equivalent bytes, so 0x01, 0x00, 0x01. Yes, it's a number you just turned back into a binary representation.

Write something that will do base64 *style* encoding, but not quite. The last two characters in the encoding set are usually + and /, but that won't do for webshit, so you need to make it use - and _ instead.

Take that publicExponent (65537), pump it through your big-endian bag-of-bytes encoder to get 0x01 0x00 0x01, then put it through your "base64web" encoder to get "AQAB".

Start a new JSON object. Add a string called "e" and set it to the output of the above step. So, yes, instead of saying that "e" equals "65537", you're saying that "e" equals "AQAB". Aren't you glad you did those extra steps?

Add another string called "kty" and set it to "RSA".

Add another string called "n" and set it to the "base64web" version of your modulus bag-of-bytes from earlier.

Take this JSON object and make it into a sorted compact string representation. This means it goes "e, kty, n" and it also has all of the usual padding (spaces) squished out. Call this a JWK string and save it for later.

Create a second JSON object. Add a boolean to it named "termsOfServiceAgreed" that's set to true. (Guess you'd better agree...)

Look up the URL for "newNonce" from the directory JSON you got earlier.

Make a HTTP HEAD request to that URL. Dig around in the headers (not the body, since there is no body on a HEAD) until you find "Replay-Nonce". Extract the value of that header. Hang onto it for later.

Look up the URL for "newAccount" from that directory JSON for before.

Create a third JSON object. Add a string to it called "url". Set it to that (newAccount) URL. Add a string called "alg". Set it to "RS256". Add a string called "nonce" and set it to the value from the *header* in that last HTTP HEAD request.

Add an object to this third object called "jwk". Within it, add "e", "kty" and "n" in a manner that matches what you did earlier (you know, from the "JWK string" you're still holding for later).

Dump this third JSON object to a sorted compact string representation. Call it "protected".

Dump the second JSON object to a sorted compact string representation. Call it "payload".

Create a string where you literally concatenate those two prior strings, such that it's the value of protected, then an actual period (as in 0x2e, a full stop, whatever), then the value of payload.

Write something that'll create a SHA256 digest of an arbitrary string and will sign it with an arbitrary RSA key (like "openssl dgst -sha256 -sign <key>"). The key in question is your personal key.

Pump that "<protected>.<payload>" string through the digest function. Then run it through your "base64web" encoder. Call this the signature.

Create a fourth JSON object. Add a string called "protected" and set it to whatever you built a few steps earlier. Add another string called "payload" and set it likewise. Then add one called "signature" and set it, too.

Take this fourth JSON object and dump it to a sorted compact string representation. Call this your post body.

Make a HTTP POST request to the "newAccount" URL from the directory. Set the content-type to "application/jose+json". Set the post data to the post body string from the previous step.

Dig around in the headers of the response, looking for one named "Location". Don't follow it like a redirection. Why would you ever follow a Location header in a HTTP header, right? Nope, that's your user account's identifier! Yes, you are a URL now. Hang on to that URL for later.

You now have an account.

You still have much to do.

...

That's about where I stopped writing my take on the protocol.

Again, my program no longer works quite like this, but this is where it started after having observed a bunch of other stuff that already existed.

So far, we have (at least): RSA keys, SHA256 digests, RSA signing, base64 but not really base64, string concatenation, JSON inside JSON, Location headers used as identities instead of a target with a 301 response, HEAD requests to get a single value buried as a header, making one request (nonce) to make ANY OTHER request, and there's more to come.

We haven't even scratched the surface of creating an order, dealing with authorizations and challenges, the whole "key thumbprint" thing, what actually goes into those TXT records, and all of that other fun stuff.

...

Random side note: while looking at existing ACME clients, I found that at least one of them screws up their encoding of the publicExponent and ends up interpreting it as hex instead of decimal. That is, instead of 65537, aka 0x10001, it reads it as 0x65537, aka 415031!

Somehow, this anomaly exists and apparently doesn't break anything? I haven't actually run the client in question, but I imagine people are using it since it's in apt.

...

This complexity must be job security for somebody. Maybe multiple somebodies.

Inside the Apollo "8-Ball" FDAI (Flight Director / Attitude Indicator)

14 June 2025 at 03:12

During the Apollo flights to the Moon, the astronauts observed the spacecraft's orientation on a special instrument called the FDAI (Flight Director / Attitude Indicator). This instrument showed the spacecraft's attitudeβ€”its orientationβ€”by rotating a ball. This ball was nicknamed the "8-ball" because it was black (albeit only on one side). The instrument also acted as a flight director, using three yellow needles to indicate how the astronauts should maneuver the spacecraft. Three more pointers showed how fast the spacecraft was rotating.

An Apollo FDAI (Flight Director/Attitude Indicator) with the case removed. This FDAI is on its side to avoid crushing the needles.

An Apollo FDAI (Flight Director/Attitude Indicator) with the case removed. This FDAI is on its side to avoid crushing the needles.

Since the spacecraft rotates along three axes (roll, pitch, and yaw), the ball also rotates along three axes. It's not obvious how the ball can rotate to an arbitrary orientation while remaining attached. In this article, I look inside an FDAI from Apollo that was repurposed for a Space Shuttle simulator1 and explain how it operates. (Spoiler: the ball mechanism is firmly attached at the "equator" and rotates in two axes. What you see is two hollow shells around the ball mechanism that spin around the third axis.)

The FDAI in Apollo

For the missions to the Moon, the Lunar Module had two FDAIs, as shown below: one on the left for the Commander (Neil Armstrong in Apollo 11) and one on the right for the Lunar Module Pilot (Buzz Aldrin in Apollo 11). With their size and central positions, the FDAIs dominate the instrument panel, a sign of their importance. (The Command Module for Apollo also had two FDAIs, but with a different design; I won't discuss them here.2)

The instrument panel in the Lunar Module. From Apollo 15 Lunar Module, NASA, S71-40761. If you're looking for the DSKY, it is in the bottom center, just out of the picture.

The instrument panel in the Lunar Module. From Apollo 15 Lunar Module, NASA, S71-40761. If you're looking for the DSKY, it is in the bottom center, just out of the picture.

Each Lunar Module FDAI could display inputs from multiple sources, selected by switches on the panel.3 The ball could display attitude from either the Inertial Measurement Unit or from the backup Abort Guidance System, selected by the "ATTITUDE MON" toggle switch next to either FDAI. The pitch attitude could also be supplied by an electromechanical unit called ORDEAL (Orbital Rate Display Earth And Lunar) that simulates a circular orbit. The error indications came from the Apollo Guidance Computer, the Abort Guidance System, the landing radar, or the rendezvous radar (controlled by the "RATE/ERROR MON" switches). The pitch, roll, and yaw rate displays were driven by the Rate Gyro Assembly (RGA). The rate indications were scaled by a switch below the FDAI, selecting 25Β°/sec or 5Β°/sec.

The FDAI mechanism

The ball inside the indicator shows rotation around three axes. I'll first explain these axes in the context of an aircraft, since the axes of a spacecraft are more arbitrary.4 The roll axis indicates the aircraft's angle if it rolls side-to-side along its axis of flight, raising one wing and lowering the other. Thus, the indicator shows the tilt of the horizon as the aircraft rolls. The pitch axis indicates the aircraft's angle if it pitches up or down, with the indicator showing the horizon moving down or up in response. Finally, the yaw axis indicates the compass direction that the aircraft is heading, changing as the aircraft turns left or right. (A typical aircraft attitude indicator omits yaw.)

I'll illustrate how the FDAI rotates the ball in three axes, using an orange as an example. Imagine pinching the horizontal axis between two fingers with your arm extended. Rotating your arm will roll the ball counter-clockwise or clockwise (red arrow). In the FDAI, this rotation is accomplished by a motor turning the frame that holds the ball. For pitch, the ball rotates forward or backward around the horizontal axis (yellow arrow). The FDAI has a motor inside the ball to produce this rotation. Yaw is a bit more difficult to envision: imagine hemisphere-shaped shells attached to the top and bottom shafts. When a motor rotates these shells (green arrow), the hemispheres will rotate, even though the ball mechanism (the orange) remains stationary.

A sphere, showing the three axes.

A sphere, showing the three axes.

The diagram below shows the mechanism inside the FDAI. The indicator uses three motors to move the ball. The roll motor is attached to the FDAI's frame, while the pitch and yaw motors are inside the ball. The roll motor rotates the roll gimbal through gears, causing the ball to rotate clockwise or counterclockwise. The roll gimbal is attached to the ball mechanism at two points along the "equator"; these two points define the pitch axis. Numerous wires on the roll gimbal enter the ball along the pitch axis. The roll control transformer provides position feedback, as will be explained below.

The main components inside the FDAI.

The main components inside the FDAI.

Removing the hemispherical shells reveals the mechanism inside the ball. When the roll gimbal is rotated, this mechanism rotates with it. The pitch motor causes the ball mechanism to rotate around the pitch axis. The yaw motor and control transformer are not visible in this photo; they are behind the pitch components, oriented perpendicularly. The yaw motor turns the vertical shaft, with the two hemisphere shells attached to the top and bottom of the shaft. Thus, the yaw motor rotates the ball shells around the yaw axis, while the mechanism itself remains stationary. The control transformers for pitch and yaw provide position feedback.

The components inside the ball of the FDAI.

The components inside the ball of the FDAI.

Why doesn't the wiring get tangled up as the ball rotates? The solution is two sets of slip rings to implement the electrical connections. The photo below shows the first slip ring assembly, which handles rotation around the roll axis. These slip rings connect the stationary part of the FDAI to the rotating roll gimbal. The vertical metal brushes are stationary; there are 23 pairs of brushes, one for each connection to the ball mechanism. Each pair of brushes contacts one metal ring on the striped shaft, maintaining contact as the shaft rotates. Inside the shaft, 23 wires connect the circular metal contacts to the roll gimbal.

The slip ring assembly in the FDAI.

The slip ring assembly in the FDAI.

A second set of slip rings inside the ball handles rotation around the pitch axis. These rings provide the electrical connection between the wiring on the roll gimbal and the ball mechanism. The yaw axis does not use slip rings since only the hemisphere shells rotate around the yaw axis; no wires are involved.

Synchros and the servo loop

In this section, I'll explain how the FDAI is controlled by synchros and servo loops. In the 1950s and 1960s, the standard technique for transmitting a rotational signal electrically was through a synchro. Synchros were used for everything from rotating an instrument indicator in avionics to rotating the gun on a navy battleship. A synchro produces an output that depends on the shaft's rotational position, and transmits this output signal on three wires. If you connect these wires to a second synchro, you can use the first synchro to control the second one: the shaft of the second synchro will rotate to the same angle as the first shaft. Thus, synchros are a convenient way to send a control signal electrically.

The photo below shows a typical synchro, with the input shaft on the top and five wires at the bottom: two for power and three for the output.

A synchro transmitter.

A synchro transmitter.

Internally, the synchro has a rotating winding called the rotor that is driven with 400 Hz AC. Three fixed stator windings provide the three AC output signals. As the shaft rotates, the voltages of the output signals change, indicating the angle. (A synchro resembles a transformer with three variable secondary windings.) If two connected synchros have different angles, the magnetic fields create a torque that rotates the shafts into alignment.

The schematic symbol for a synchro transmitter or receiver.

The schematic symbol for a synchro transmitter or receiver.

The downside of synchros is that they don't produce a lot of torque. The solution is to use a more powerful motor, controlled by the synchro and a feedback loop called a servo loop. The servo loop drives the motor in the appropriate direction to eliminate the error between the desired position and the current position.

The diagram below shows how the servo loop is constructed from a combination of electronics and mechanical components. The goal is to rotate the output shaft to an angle that exactly matches the input angle, specified by the three synchro wires. The control transformer compares the input angle and the output shaft position, producing an error signal. The amplifier uses this error signal to drive the motor in the appropriate direction until the error signal drops to zero. To improve the dynamic response of the servo loop, the tachometer signal is used as a negative feedback voltage. The feedback slows the motor as the system gets closer to the right position, so the motor doesn't overshoot the position and oscillate. (This is sort of like a PID controller.)

This diagram shows the structure of the servo loop, with a feedback loop ensuring that the rotation angle of the output shaft matches the input angle.

This diagram shows the structure of the servo loop, with a feedback loop ensuring that the rotation angle of the output shaft matches the input angle.

A control transformer is similar to a synchro in appearance and construction, but the rotating shaft operates as an input, not the output. In a control transformer, the three stator windings receive the inputs and the rotor winding provides the error output. If the rotor angle of the synchro transmitter and control transformer are the same, the signals cancel out and there is no error voltage. But as the difference between the two shaft angles increases, the rotor winding produces an error signal. The phase of the error signal indicates the direction of the error.

In the FDAI, the motor is a special motor/tachometer, a device that was often used in avionics servo loops. This motor is more complicated than a regular electric motor. The motor is powered by 115 volts AC at 400 hertz, but this won't spin the motor on its own. The motor also has two low-voltage control windings. Energizing the control windings with the proper phase causes the motor to spin in one direction or the other. The motor/tachometer unit also contains a tachometer to measure its speed for the feedback loop. The tachometer is driven by another 115-volt AC winding and generates a low-voltage AC signal that is proportional to the motor's rotational speed.

A motor/tachometer similar (but not identical) to the one in the FDAI.

A motor/tachometer similar (but not identical) to the one in the FDAI.

The photo above shows a motor/tachometer with the rotor removed. The unit has many wires because of its multiple windings. The rotor has two drums. The drum on the left, with the spiral stripes, is for the motor. This drum is a "squirrel-cage rotor", which spins due to induced currents. (There are no electrical connections to the rotor; the drums interact with the windings through magnetic fields.) The drum on the right is the tachometer rotor; it induces a signal in the output winding proportional to the speed due to eddy currents. The tachometer signal is at 400 Hz like the driving signal, either in phase or 180ΒΊ out of phase, depending on the direction of rotation. For more information on how a motor/tachometer works, see my teardown.

The amplifiers

The FDAI has three servo loopsβ€”one for each axisβ€”and each servo loop has a separate control transformer, motor, and amplifier. The photo below shows one of the three amplifier boards. The construction is unusual and somewhat chaotic, with some components stacked on top of others to save space. Some of the component leads are long and protected with clear plastic sleeves.5 The cylindrical pulse transformer in the middle has five colorful wires coming out of it. At the left are the two transistors that drive the motor's control windings, with two capacitors between them. The transistors are mounted on a heat sink that is screwed down to the case of the amplifier assembly for cooling. Each amplifier is connected to the FDAI through seven wires with pins that plug into the sockets on the right of the board.6

One of the three amplifier boards. At the right front of the board, you can see a capacitor stacked on top of a resistor. The board is shiny because it is covered with conformal coating.

One of the three amplifier boards. At the right front of the board, you can see a capacitor stacked on top of a resistor. The board is shiny because it is covered with conformal coating.

The function of the board is to amplify the error signal so the motor rotates in the appropriate direction. The amplifier also uses the tachometer output from the motor unit to slow the motor as the error signal decreases, preventing overshoot. The inputs to the amplifier are 400 hertz AC signals, with the magnitude indicating the amount of error or speed and the phase indicating the direction. The two outputs from the amplifier drive the two control windings of the motor, determining which direction the motor rotates.

The schematic for the amplifier board is below. 7 The two transistors on the left amplify the error and tachometer signals, driving the pulse transformer. The outputs of the pulse transformer will have opposite phases, driving the output transistors for opposite halves of the 400 Hz cycle. This activates the motor control winding, causing the motor to spin in the desired direction.8

The schematic of an amplifier board.

The schematic of an amplifier board.

History of the FDAI

Bill Lear, born in 1902, was a prolific inventor with over 150 patents, creating everything from the 8-track tape to the Learjet, the iconic private plane of the 1960s. He created multiple companies in the 1920s as well as inventing one of the first car radios for Motorola before starting Lear Avionics, a company that specialized in aerospace instruments.9 Lear produced innovative aircraft instruments and flight control systems such as the F-5 automatic pilot, which received a trophy as the "greatest aviation achievement in America" for 1950.

Bill Lear went on to solve an indicator problem for the Air Force: the supersonic F-102 Delta Dagger interceptor (1953) could climb at steep angles, but existing attitude indicators could not handle nearly vertical flight. Lear developed a remote two-gyro platform that drove the cockpit indicator while avoiding "gimbal lock" during vertical flight. For the experimental X-15 rocket-powered aircraft (1959), Lear improved this indicator to handle three axes: roll, pitch, and yaw.

Meanwhile, the Siegler Corporation started in 1950 to manufacture space heaters for homes. A few years later, Siegler was acquired by John Brooks, an entrepreneur who was enthusiastic about acquisitions. In 1961, Lear Avionics became his latest acquisition, and the merged company was called Lear Siegler Incorporated, often known as LSI. (Older programmers may know Lear Siegler through the ADM-3A, an inexpensive video display terminal from 1976 that housed the display and keyboard in a stylish white case.)

The X-15's attitude indicator became the basis of the indicator for the F-4 fighter plane (the ARU/11-A). Then, after "a minimum of modification", the attitude-director indicator was used in the Gemini space program. In total, Lear Siegler provided 11 instruments in the Gemini instrument panel, with the attitude director the most important. Next, Gemini's indicator was modified to become the FDAI (flight director-attitude indicator) in the Lunar Module for Apollo.10 Lear Siegler provided numerous components for the Apollo program, from a directional gyro for the Lunar Rover to the electroluminescent display for the Apollo Guidance Computer's Display/Keyboard (DSKY).

An article titled "LSI Instruments Aid in Moon Landing" from LSI's internal LSI Log publication, July 1969. (Click for a larger version.)

An article titled "LSI Instruments Aid in Moon Landing" from LSI's internal LSI Log publication, July 1969. (Click for a larger version.)

In 1974, Lear Siegler obtained a contract to develop the Attitude-Director Indicator (ADI) for the Space Shuttle, producing a dozen ADI units for the Space Shuttle. However, by this time, Lear Siegler was losing enthusiasm for low-volume space avionics. The Instrument Division president said that "the business that we were in was an engineering business and engineers love a challenge." However, manufacturing refused to deal with the special procedures required for space manufacturing, so the Shuttle units were built by the engineering department. Lear Siegler didn't bid on later Space Shuttle avionics and the Shuttle ADI became its last space product. In the early 2000s, the Space Shuttle's instruments were upgraded to a "glass cockpit" with 11 flat-panel displays known as the Multi-function Electronic Display System (MEDS). The MEDS was produced by Lear Siegler's long-term competitor, Honeywell.

Getting back to Bill Lear, he wanted to manufacture aircraft, not just aircraft instruments, so he created the Learjet, the first mass-produced business jet. The first Learjet flew in 1963, with over 3000 eventually delivered. In the early 1970s, Lear designed a steam turbine automobile engine. Rather than water, the turbine used a secret fluorinated hydrocarbon called "Learium". Lear had visions of thousands of low-pollution "Learmobiles", but the engine failed to catch on. Lear had been on the verge of bankruptcy in the 1960s; one of his VPs explained that "the great creative minds can't be bothered with withholding taxes and investment credits and all this crap". But by the time of his death in 1978, Lear had a fortune estimated at $75 million.

Comparing the ARU/11-A and the FDAI

Looking inside our FDAI sheds more details on the evolution of Lear Siegler's attitude directors. The photo below compares the Apollo FDAI (top) to the earlier ARU/11-A used in the F-4 aircraft (bottom). While the basic mechanism and the electronic amplifiers are the same between the two indicators, there are also substantial changes.

Comparison of an FDAI (top) with an ARU-11/A (bottom). The amplifier boards and needles have been removed from the FDAI.

Comparison of an FDAI (top) with an ARU-11/A (bottom). The amplifier boards and needles have been removed from the FDAI.

The biggest difference between the ARU-11/A indicator and the FDAI is that the electronics for the ARU-11/A are in a separate module that was plugged into the back of the indicator, while the FDAI includes the electronics internally, with boards mounted on the instrument frame. Specifically, the ARU-11/A has a separate unit containing a multi-winding transformer, a power supply board, and three amplifier boards (one for each axis), while the FDAI contains these components internally. The amplifier boards in the ARU-11/A and the FDAI are identical, constructed from germanium transistors rather than silicon.11 The unusual 11-pin transformers are also the same. However, the power supply boards are different, probably because the boards also contain scaling resistors that vary between the units.12 The power supply boards are also different shapes to fit the available space.

The ball assemblies of the ARU/11-A and the FDAI are almost the same, with the same motor assemblies and slip ring mechanism. The gearing has minor changes. In particular, the FDAI has two plastic gears, while the ARU/11-A uses exclusively metal gears.

The ARU/11-A has a patented pitch trim feature that was mostlyβ€”but not entirelyβ€”removed from the Apollo FDAI. The motivation for this feature is that an aircraft in level flight will be pitched up a few degrees, the "angle of attack". It is desirable for the attitude indicator to show the aircraft as horizontal, so a pitch trim knob allows the angle of attack to be canceled out on the display. The problem is that if you fly your fighter plane vertically, you want the indicator to show precisely vertical flight, rather than applying the pitch trim adjustment. The solution in the ARU-11/A is a special 8-zone potentiometer on the pitch axis that will apply the pitch trim adjustment in level flight but not in vertical flight, while providing a smooth transition between the regions. This special potentiometer is mounted inside the ball of the ARU-11/A. However, this pitch trim adjustment is meaningless for a spacecraft, so it is not implemented in the Apollo or Space Shuttle instruments. Surprisingly, the shell of the potentiometer still exists in our FDAI, but without the potentiometer itself or the wiring. Perhaps it remained to preserve the balance of the ball. In the photo below, the cylindrical potentiometer shell is indicated by an arrow. Note the holes in the front of the shell; in the ARU-11/A, the potentiometer's wiring terminals protrude through these holes, but in the FDAI, the holes are covered with tape.

Inside the ball of the FDAI. The potentiometer shell is indicated with an arrow.

Inside the ball of the FDAI. The potentiometer shell is indicated with an arrow.

Finally, the mounting of the ball hemispheres is slightly different. The ARU/11-A uses four screws at the pole of each hemisphere. Our FDAI, however, uses a single screw at each pole; the screw is tightened with a Bristol Key, causing the shaft to expand and hold the hemisphere in place.

To summarize, the Apollo FDAI occupies a middle ground: while it isn't simply a repurposed ARU-11/A, neither is it a complete redesign. Instead, it preserves the old design where possible, while stripping out undesired features such as pitch trim. The separate amplifier and mechanical units of the ARU/11-A were combined to form the larger FDAI.

Differences from Apollo

The FDAI that we examined is a special unit: it was originally built for Apollo but was repurposed for a Space Shuttle simulator. Our FDAI is labeled Model 4068F, which is a Lunar Module part number. Moreover, the FDAI is internally stamped with the date "Apr. 22 1968", over a year before the first Moon landing.

However, a closer look shows that several key components were modified to make the Apollo FDAI work in the Shuttle Simulator.14 The Apollo FDAI (and the Shuttle ADI) used resolvers as inputs to control the ball, while our FDAI uses synchros. (Resolvers and synchros are similar, except resolvers use sine and cosine inputs, 90Β° apart, on two wire pairs, while synchros use three inputs, 120Β° apart, on three wires.) NASA must have replaced the three resolver control transformers in the FDAI with synchro control transformers for use in the simulator.

The Apollo FDAI used electroluminescent lighting for the display, while ours uses eight small incandescent bulbs. The metal case of our FDAI has a Dymo embossed tape label "INCANDESCENT LIGHTING", alerting users to the change from Apollo's illumination. Our FDAI also contains a step-down transformer to convert the 115 VAC input into 5 VAC to power the bulbs, while the Shuttle powered its ADI illumination directly from 5 volts.

The dial of our FDAI was repainted to match the dial of the Shuttle FDAI. The Apollo FDAI had red bands on the left and right of the dial. A close examination of our dial shows that black paint was carefully applied over the red paint, but a few specks of red paint are still visible (below). Moreover, the edges of the lines and the lozenge show slight unevenness from the repainting. Second, the Apollo FDAI had the text "ROLL RATE", "PITCH RATE", and "YAW RATE" in white next to the needle scales. In our FDAI, this text has been hidden by black paint to match the Shuttle display.13 Third, the Apollo LM FDAI had a crosshair in the center of the instrument, while our FDAI has a white U-shaped indicator, the same as the Shuttle (and the Command Module's FDAI). Finally, the ball of the Apollo FDAI has red circular regions at the poles to warn of orientations that can cause gimbal lock. Our FDAI (like the Shuttle) does not have these circles. We couldn't see any evidence that these regions were repainted, so we suspect that our FDAI has Shuttle hemispheres on the ball.

A closeup of the dial on our FDAI shows specks of red paint around the dial markings. The color is probably Switzer DayGlo Rocket Red.

A closeup of the dial on our FDAI shows specks of red paint around the dial markings. The color is probably Switzer DayGlo Rocket Red.

Our FDAI has also been modified electrically. Small green connectors (Micro-D MDB1) have been added between the slip rings and the motors, as well as on the gimbal arm. We think these connectors were added post-Apollo, since they are attached somewhat sloppily with glue and don't look flight-worthy. Perhaps these connectors were added to make disassembly and modification easier. Moreover, our FDAI has an elapsed time indicator, also mounted with glue.

The back of our FDAI is completely different from Apollo. First, the connector's pinout is completely different. Second, each of the six indicator needles has a mechanical adjustment as well as a trimpot (details). Finally, each of the three axes has an adjustment potentiometer.

The Shuttle's ADI (Attitude Director Indicator)

Each Space Shuttle had three ADIs (Attitude Director Indicators), which were very similar to the Apollo FDAI, despite the name change. The photo below shows the two octagonal ADIs in the forward flight deck, one on the left in front of the Commander, and one on the right in front of the Pilot. The aft flight deck station had a third ADI.15

This photo shows Discovery's forward flight deck on STS-063 (1999). The ADIs are indicated with arrows. The photo is from the National Archives.

This photo shows Discovery's forward flight deck on STS-063 (1999). The ADIs are indicated with arrows. The photo is from the National Archives.

Our FDAI appears to have been significantly modified for use in the Shuttle simulator, as described above. However, it is much closer to the Apollo FDAI than the ADI used in the Shuttle, as I'll show in this section. My hypothesis is that the simulator was built before the Shuttle's ADI was created, so the Apollo FDAI was pressed into service.

The Shuttle's ADI was much more complicated electrically than the Apollo FDAI and our FDAI, providing improved functionality.16 For instance, while the Apollo FDAI had a simple "OFF" indicator flag to show that the indicator had lost power, the Shuttle's ADI had extensive error detection. It contained voltage level monitors to check its five power supplies. (The Shuttle ADI used three DC power sources and two AC power sources, compared to the single AC supply for Apollo.) The Shuttle's ADI also monitored the ball servos to detect position errors. Finally, it received an external "Data OK" signal. If a fault was detected by any of these monitors, the "OFF" flag was deployed to indicate that the ADI could not be trusted.

The Shuttle's ADI had six needles, the same as Apollo, but the Shuttle used feedback to make the positions more accurate. Specifically, each Shuttle needle had a feedback sensor, a Linear Variable Differential Transformer (LVDT) that generates a voltage based on the needle position. The LVDT output drove a servo feedback loop to ensure that the needle was in the exact desired position. In the Apollo FDAI, on the other hand, the needle input voltage drove a galvanometer, swinging the needle proportionally, but there was no closed loop to ensure accuracy.

I assume that the Shuttle's ADI had integrated circuit electronics to implement this new functionality, considerably more modern than the germanium transistors in the Apollo FDAI. The Shuttle probably used the same mechanical structures to rotate the ball, but I can't confirm that.

Conclusions

The FDAI was a critical instrument in Apollo, indicating the orientation of the spacecraft in three axes. It wasn't obvious to me how the "8-ball" can rotate in three axes while still being securely connected to the instrument. The trick is that most of the mechanism rotates in two axes, while hollow hemispherical shells provide the third rotational axis.

The FDAI has an interesting evolutionary history, from the experimental X-15 rocket plane and the F-4 fighter to the Gemini, Apollo, and Space Shuttle flights. Our FDAI has an unusual position in this history: since it was modified from Apollo to function in a Space Shuttle simulator, it shows aspects of both Apollo and the Space Shuttle indicators. It would be interesting to compare the design of a Shuttle ADI to the Apollo FDAI, but I haven't been able to find interior photos of a Shuttle ADI (or of an unmodified Apollo FDAI).17

You can see a brief video of the FDAI in motion here. For more, follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS. (I've given up on Twitter.) I worked on this project with CuriousMarc, Mike Stewart, and Eric Schlapfer, so expect a video at some point. Thanks to Richard for providing the FDAI. I wrote about the F-4 fighter plane's attitude indicator here.

Inside the FDAI. The amplifier boards have been removed for this photo.

Inside the FDAI. The amplifier boards have been removed for this photo.

Notes and references

  1. There were many Space Shuttle simulators, so it is unclear which simulator was the source of our FDAI. The photo below shows a simulator, with one of the ADIs indicated with an arrow. Presumably, our FDAI became available when a simulator was upgraded from physical instruments to the screens of the Multi-function Electronic Display System (MEDS).

    "Forward flight deck of the fixed-base simulator." From Introduction to Shuttle Mission Simulation

    "Forward flight deck of the fixed-base simulator." From Introduction to Shuttle Mission Simulation

    The most complex simulators were the three Shuttle Mission Simulators, one of which could dynamically move to provide motion cues. These simulators were at the simulation facility in Houstonβ€”officially the Jake Garn Mission Simulator and Training Facilityβ€”which also had a guidance and navigation simulator, a Spacelab simulator, and integration with the WETF (Weightless Environment Training Facility, an underground pool to simulate weightlessness). The simulators were controlled by a computer complex containing dozens of networked computers. The host computers were three UNIVAC 1100/92 mainframes, 36-bit computers that ran the simulation models. These were supported by seventeen Concurrent Computer Corporation 3260 and 3280 super-minicomputers that simulated tracking, telemetry, and communication. The simulators also used real Shuttle computers running the actual flight software; these were IBM AP101S General-Purpose Computers (GPC). For more information, see Introduction to Shuttle Mission Simulation.

    NASA had additional Shuttle training facilities beyond the Shuttle Mission Simulator. The Full Fuselage Trainer was a mockup of the complete Shuttle orbiter (minus the wings). It included full instrument panels (including the ADIs), but did not perform simulations. The Crew Compartment Trainers could be positioned horizontally or vertically (to simulate pre-launch operations). They contained accurate flight decks with non-functional instruments. Three Single System Trainers provided simpler mockups for astronauts to learn each system, both during normal operation and during malfunctions, before using the more complex Shuttle Mission Simulator. A list of Shuttle training facilities is in Table 3.1 of Preparing for the High Frontier. Following the end of the Shuttle program, the trainers were distributed to various museums (details). ↩

  2. The Command Module for Apollo used a completely different FDAI (flight director-attitude indicator) that was built by Honeywell. The two designs can be easily distinguished: the Honeywell FDAI is round, while the Lear Siegler FDAI is octagonal. ↩

  3. The FDAI's signals are more complicated than I described above. Among other things, the IMU's gimbal angles use a different coordinate system from the FDAI, so an electromechanical unit called GASTA (Gimbal Angle Sequence Transformation Assembly) used resolvers and motors to convert the coordinates. The digital attitude error signals from the computer are converted to analog by the Inertial Measurement Unit's Coupling Data Unit (IMU CDU). For attitude, the IMU is selected with the PGNS (Primary Guidance and Navigation System) switch setting. See the Lunar Module Systems Handbook, Lunar Module System Handbook Rev A, and the Apollo Operations Handbook for more.

    The connections to the Apollo FDAIs. Adapted from LM-1 Systems Handbook. I think this diagram predates the ORDEAL system. (Click for a larger version.)

     ↩

  4. The roll, pitch, and yaw axes of the Lunar Module are not as obvious as the axes of an airplane. The diagram below defines these axes.

    The roll, pitch, and yaw axes of the Lunar Module. Adapted from LM Systems Handbook.

    The roll, pitch, and yaw axes of the Lunar Module. Adapted from LM Systems Handbook.

     ↩

  5. The amplifier is constructed on a single-sided printed circuit board. Since the components are packed tightly on the board, routing of the board was difficult. However, some of the components have long leads, protected by plastic sleeves. This provides additional flexibility for the board routing since the leads could be positioned as desired, regardless of the geometry of the component. As a result, the style of this board is very different from modern circuit boards, where components are usually arranged in an orderly pattern. ↩

  6. In our FDAI, the amplifier boards as well as the needle actuators are connected by pins that plug into sockets. These connections don't seem suitable for flight since they could easily vibrate loose. We suspect that the pin-and-socket connections made the module easier to reconfigure in the simulator, but were not used in flyable units. In particular, in the similar aircraft instruments (ARU/11-A) that we examined, the wires to the amplifier boards were soldered. ↩

  7. The board has a 56-volt Zener diode, but the function of the diode is unclear. The board is powered by 28 volts, not enough voltage to activate the Zener. Perhaps the diode filters high-voltage transients, but I don't see how transients could arise in that part of the circuit. (I can imagine transients when the pulse transformer switches, but the Zener isn't connected to the transformer.) ↩

  8. In more detail, each motor's control winding is a center-tapped winding, with the center connected to 28 volts DC. The amplifier board's output transistors will ground either side of the winding during alternate half-cycles of the 400 Hz cycle. This causes the motor to spin in one direction or the other. (Usually, control winding are driven 90Β° out of phase with the motor power, but I'm not sure how this phase shift is applied in the FDAI.) ↩

  9. The history of Bill Lear and Lear Siegler is based on Love him or hate him, Bill Lear was a creator and On Course to Tomorrow: A History of Lear Siegler Instrument Division’s Manned Spaceflight Systems 1958-1981. ↩

  10. Numerous variants of the Lear Siegler FDAI were built for Apollo, as shown before. Among other things, the length of the unit ("L MAX") varied from 8 inches to 11 inches. (Our FDAI is approximately 8 inches long.)

    The Apollo FDAI part number chart from Grumman Specification Control Drawing LSC350-301. (Click for a larger view.)

    The Apollo FDAI part number chart from Grumman Specification Control Drawing LSC350-301. (Click for a larger view.)

     ↩

  11. We examined a different ARU-11/A where the amplifier boards were not quite identical: the boards had one additional capacitor and some of the PCB traces were routed slightly differently. These boards were labeled "REV C" in the PCB copper, so they may have been later boards with a slight modification. ↩

  12. The amplifier scaling resistors were placed on the power supply board rather than the amplifier boards, which may seem strange. The advantage of this approach is that it permitted the three amplifier boards to be identical, since the components that differ between the axes were not part of the amplifier boards. This simplified the manufacture and repair of the amplifier boards. ↩

  13. On the front panel of our FDAI, the text "ROLL RATE", "PITCH RATE", and "YAW RATE" has been painted over. However, the text is still faintly visible (reversed) on the inside of the panel, as shown below.

    The inside of the FDAI's front cover.

    The inside of the FDAI's front cover.

     ↩

  14. The diagram below shows the internals of the Apollo LM FDAI at a high level. This diagram shows several differences between the LM FDAI and the FDAI that we examined. First, the roll, pitch, and yaw inputs to the LM FDAI are resolver inputs (i.e. sin and cos), rather than the synchro inputs to our FDAI. Second, the needle signals below are modulated on an 800 Hz carrier and are demodulated inside the FDAI. Our FDAI, however, uses positive or negative voltages to drive the needle galvanometers directly. A minor difference is that the diagram below shows the Power Off Flag wired to +28V internally, while our FDAI has the flag wired to connector pins, probably so the flag could be controlled by the simulator.

    The diagram of the FDAI in the LM Systems Handbook. Click for a larger image.

    The diagram of the FDAI in the LM Systems Handbook. Click for a larger image.

     ↩

  15. The Space Shuttle instruments were replaced with color LCD screens in the MEDS (Multifunction Electronic Display System) upgrade. This upgrade is discussed in New Displays for the Space Shuttle Cockpit. The Space Shuttle Systems Handbook shows the ADIs on the forward console (pages 263-264) and the aft console (page 275). The physical ADI is compared to the MEDS ADI display in Displays and Controls, Vol. 1 page 119. ↩

  16. The diagram below shows the internals of the Shuttle's ADI at a high level. The Shuttle's ADI is more complicated than the Apollo FDAI, even though they have the same indicator ball and needles.

    A diagram of the Space Shuttle's ADI. From Space Shuttle Systems Handbook Vol. 1, 1 G&C DISP 1. (Click for a larger image.)

    A diagram of the Space Shuttle's ADI. From Space Shuttle Systems Handbook Vol. 1, 1 G&C DISP 1. (Click for a larger image.)

     ↩

  17. Multiple photos of the exterior of the Shuttle ADI are available here, from the National Air and Space Museum. There are interior photos of Apollo FDAIs online, but they all appear to be modified for Shuttle simulators. ↩

How to trigger a command on Linux when disconnected from power

31 May 2025 at 00:00
# Introduction

After thinking about BusKill product that triggers a command once the USB cord disconnects, I have been thinking at a simple alternative.

=> https://www.buskill.in BusKill official project website

When using a laptop connected to power most of the time, you may want it to power off once it gets disconnected, this can be really useful if you use it in a public area like a bar or a train.  The idea is to protect the laptop if it gets stolen while in use and unlocked.

Here is how to proceed on Linux, using a trigger on an udev rule looking for a change in the power_supply subsystem.

For OpenBSD users, it is possible to use apmd as I explained in this article:

=> https://dataswamp.org/~solene/2024-02-20-rarely-known-openbsd-features.html#_apmd_daemon_hooks => Rarely known OpenBSD features: apmd daemon hooks

In the example, the script will just power off the machine, it is up to you to do whatever you want like destroy the LUKS master key or trigger the coffee machine :D

# Setup

Create a file `/etc/udev/rules.d/disconnect.rules`, you can name it how you want as long as it ends with `.rules`:

```
SUBSYSTEM=="power_supply", ENV{POWER_SUPPLY_ONLINE}=="0", ENV{POWER_SUPPLY_TYPE}=="Mains", RUN+="/usr/local/bin/power_supply_off"
```

Create a file `/usr/local/bin/power_supply_off` that will be executed when you unplug the laptop:

```
#!/bin/sh
echo "Going off because power supply got disconnected" | systemd-cat
systemctl poweroff
```

This simple script will add an entry in journald before triggering the system shutdown.

Mark this script executable with:
```
chmod +x /usr/local/bin/power_supply_off
```

Reload udev rules using the following commands:

```
udevadm control --reload-rules
udevadm trigger
```

# Testing

If you unplug your laptop power, it should power off, you should find an entry in the logs.

If nothing happens, looks at systemd logs to see if something is wrong in udev, like a syntax error in the file you created or an incorrect path for the script.

# Script ideas

Depending on your needs, here is a list of actions the script could do, from gentle to hardcore:

* Lock user sessions
* Hibernate
* Proper shutdown
* Instant power off (through sysrq)
* Destroy LUKS master key to make LUKS volume unrecoverable + Instant power off

# Conclusion

While BusKill is an effective / unusual product that is certainly useful for a niche, protecting a running laptop against thieves is an extra layer when being outside.

Obviously, this use case works only when the laptop is connected to power.

AI could be conscious tomorrow and we wouldn’t care

30 June 2025 at 15:55

Historian Dr Francis Young on extraterrestrial life and its imagined implications (Bluesky):

One of the most darkly funny scientistic pieties is the idea that the discovery of intelligent life beyond Earth would β€˜humble’ humanity - given that in the late c19th and early c20th (an era renowned for human humility […]) it was a mainstream view that Mars was inhabited

It never ceases to amaze me how we have culturally memory-holed the fact that before c. 1920 it was perfectly normal to believe seriously that intelligent life existed on other planets in the Solar System

The discovery that "Mars was likely lifeless … is a mid-20th-century development."

But the idea that a broad consensus that we are not alone in the universe will somehow inaugurate an era of world peace is pretty silly, given that many intelligent people believed this with complete seriousness in 1914.

It’s a good point!


Further back in history, the Medieval cosmology was also densely populated.

From The Discarded Image (Wikipedia) by C S Lewis (which I read on recommendation from Robin Sloan), there are intelligent, powerful gods - which we can see as planets - and angels and we have so much in common with other life on Earth:

The powers of Vegetable Soul are nutrition, growth and propagation. It alone is present in plants. Sensitive Soul, which we find in animals, has these powers but has sentience in addition. It thus includes and goes beyond Vegetable Soul, so that a beast can be said to have two levels of soul, Sensitive and Vegetable, or a double seal, or even – though misleadingly – two souls. Rational Soul similarly includes Vegetable and Sensitive, and adds reason.

(p153)

There are not just humans and angels, there are

bull-beggars, spirits, witches, urchins, elves, hags, fairies, satyrs, pans, faunes, spleens, tritons, centaurs, dwarfs, giants, nymphes, Incubus, Robin good fellow, the spoom, the man in the oke, the fire-drake, the puckle, Tom Thumbe, Tom tumbler, boneles, and other such bugs.

(p125)

We were not alone.


Still further, into the deep history of Eurasian magic, the 40,000 year-old system of belief underpinning the West:

Across the vast grasslands and forests of the Steppe in Central Asia and west into Europe, the world was animated by spirits, some originally human, others less so.

Animism is "a mode of action, creating relations between kinds."

In conceiving of such relations it may be that all things, living and non-living, are seen as persons. Many groups do believe that all things are human, and hence have personhood, whether they may appear as a rock, or tapir or the Sun. Relations between persons are of amity, indifference or enmity…

And before you say that animism is an idea that we have moved past, and it absurd that the rock falls to the Earth because of some kind of β€œamity”, let’s go back to Lewis in The Discarded Image who points out that our natural laws - such as the law of gravity - have an anthropological frame:

to talk as if [falling stones] could β€˜obey laws’ is to treat them like men and even like citizens.

Still our language today.


So maybe let’s go further than Dr Francis Young…

The discovery of extraterrestrial life would not result in a humbling Copernican decentring of human consciousness.

Not just because a belief in extraterrestrial life has occurred before and we didn’t show much "humility" then.

But because (Eurasian) humanity already had its Copernican moment, tens of thousands of years ago, animism means that humans have always been one mere consciousness among thousands.

Humanity has never felt alone and this is as humble as we get.


I can’t help but connect all of this with AI consciousness (on which topic I maintain an agnostic watching brief)…

If AI consciousness were shown to be real, the argument goes, we would need to update our ethics with β€œrobot rights,” granting justice, autonomy and dignity to our fellow sentient beings.

(Lena by qntm resonates because we instinctively see the treatment of the uploaded brain as Not Okay, even though it’s just software, evidence that we do indeed have a kind of folk ethics of artificial non-humans.)

And that, we suppose, would cascade to a Copernican shift in how humanity sees itself, etc.

But I’ve never been sure that recognising AIs as sentient would make a blind bit of difference. As I said when I wrote about AI consciousness before (2023), I’m pretty sure that chickens are sentient and it doesn’t stop us doing all kinds of awful unethical things with them.

Even if we don’t agree on chicken sentience, what about people who work in sweatshops, and they are definitely sentient, and they don’t get access to the same β€œrobot rights” currently being debated for future sentient AIs.

So if we’re hunting for a route to an expanded moral frame for humanity, I’m not sure we’ll find it purely via ET or AI. I wonder what it would take.


More posts tagged: ai-consciousness (2), the-ancient-world-is-now (15).

Auto-detected kinda similar posts:

Batter bits, scraps, dubs, scrumps

27 June 2025 at 13:30

When I was a kid when you went to a proper chippie - fish and chips traditionally on a Friday - you could ask for β€œbatter bits” (regional naming statistics) which were the wonderful leftover deep-fried scraps of fish batter, and you’d get them free.

Anyway apparently if the scraps build up too much in the fryer they can cause chip shop fires. So always clear them out.


What does my head in about this astounding beatboxing is that it’s all done with the human mouth and the Pleistocene was 2 MILLION YEARS so odds are some pre-historic proto-human was banging rocks and spitting weird synth beats under the Milky Way on the African savannah, and we will never hear it.

Who was the Lennon of the BCE 500,560s, who was the Mozart of the -12,392nd century?


Global Hypercolor was such a great brand name for t-shirts. Bad concept though, garments which change colour based on body heat, look I’m sweating right here, hey I’m suddenly more nervous right now.

I remember playing a video game called Magical Flying Hat Turbo Adventure, same energy, name-stacking-wise.

There is a company registered in the UK named THIS IS THE COMPANY WITH THE LONGEST NAME SO FAR INCORPORATED AT THE REGISTRY OF COMPANIES IN ENGLAND AND WALES AND ENCOMPASSING THE REGISTRIES BASED IN SCOTLAN


Coding with Cursor is so weird. You just loop one minute composing a thoughtful paragraph to the AI agent telling it what to do, and then three minutes you wait for it to be done, gazing out the window contemplating the gentle breeze on the leaves, the distant hum of traffic, the slow steady unrelenting approach of that which comes for us all.


The Netflix Originals red N is such an anti-signal, I immediately assume it’s minute-by-minute-honed attention-farming prestige slop.


Hold me closer, tiny shader. Not sure where this is going but this is my first time implementing a shader so dunno.


Some food names where it’s the same thing twice:

  • couscous
  • bonbon
  • piri piri
  • biang biang
  • tartar
  • agar-agar

That TikTok from 2020 of the guy on the longboard lip-syncing Fleetwood Mac was such a vibe you know, an invisible secret sadness at 4s, a whole emotional arc, a flash of sunrise ahead at 21s, I can’t think of anything that so precisely targets coordinates in vibe latent space, with such quick efficiency.

Sean Willis on Bluesky shared the music video for Gold by Chet Faker and it’s a good job I never saw it when I was 18 or I would have moved immediately to LA and burnt the next decade night driving freeways at 4am.


Auto-detected kinda similar posts:

The video calls section in cafes is the new smoking section

16 June 2025 at 20:10

People are really leaning into doing work video calls in otherwise-quiet cafes hey.

At this point cafes have given in.

Even without calls, sitting next to someone who is at their laptop and in the zone is a whole thing. Like standing at a train platform when the non-stopper charges by a metre away, there is just something about the sheering force proximity of the energy.

So that started a while back. People gently at their laptops, fine. People typing like a donkey falling downstairs, it’s like they’re lit in a different colour. They’re in the room but not of the room. It’s impossible to pick at a pastry sitting next to that kind of intensity.

There’s a cafe with a small upstairs near St Pancras and I was once in there waiting for a friend, and four other people were in this same small space taking video calls, one without headphones even, just yelling at his screen.

I used to work out of the members room at Tate Modern and I remember somebody there who would bring a laptop stand, external keyboard, and headphones with head mic.

There has been a whole arc to this:

I think maybe during covid a lot of people fully adopted working from home, and what that means is working from cafes nearby to home, because London flats are expensive and tiny. I can’t blame them.

So that was when laptops starting being banned, in reaction.

There’s a place in my neighbourhood that began by intermittently banning laptops at lunch at the main tables. Then all the time.

They are a cafe slash amazing vegan fusion spot slash yoga studio so I guess they are sensitive to the vibe.

Then laptops were only allowed at specific 4 or 5 stools by the window. You felt distinctly unwelcome (but went anyway, it’s nice to be out of the house).

Then, I was in a couple weeks back, they’ve surrendered.

The window stool area is now dense nest of stools and counters and a new wedged-in shared table in the middle. You can probably jam 10 people in there now, shoulder to shoulder and back to back.

This area is made for laptops, and people sit there all day yelling video calls on their head-mics, battery farmed knowledge work.

It makes zero sense to have a laptop area like this: it’s like the old days of smoking sections in restaurants where you’d have the smoking section and the non-smoking section, divided by a homeopathic string barrier that would somehow by keep the smoke smell from transmitting across by magical signage.

And yet here we are.


Second hand smoke / second hand zoom.

I don’t have an Apple Vision Pro (I’ve done the Apple Store demo and have Opinions) but I am so tempted to acquire one entirely for the purpose of sitting in cafes wearing it for hours, yelling on zoom.


I said before that iPhones should have a sense of shame: "Other people nearby should get a special tut-tut button they can tap."

BUT let me instead try to be more positive.

Would it be possible instead to have silent video calls?

On Bluesky when I talked about this, Sam Jeffers said:

your AI-powered lip-reading startup just got wings.

Which is exactly it!

See, a Vision Pro doesn’t have a webcam on an orbiting mini-drone and of course you’ve got a great big headset covering your face. So when you use Zoom, everyone else instead sees your β€œpersona”, "an authentic spatial representation of you that enables others on a call to see your facial expressions and movements" – a real-time reconstructed talking 3D scan of your head and heads.

So, with just a regular laptop:

Firstly I should be able to speak without speaking, you know, just mouth the words.

Surely my words can be figured out by fusing data from (a) the Mac webcam reading my lips, and (b) an EEG sensor in some future upgraded AirPods. EEG is usually used to measure brain activity… (admittedly dry sensor EEG was not great last time I tried it) but EEG also reads the much clearer muscle signals. Muscles from my jaw and tongue. And Apple has patented EEG sensors in AirPods. Maybe that patent was never about reading brainwaves.

Which means that now we’ve got capture: EEG-enhanced lip reading.

As for playback, what does the person on the other end of the call hear? They hear you. Since iOS 17, Apple has enabled Personal Voice (previously tested as an accessibility feature called Voice Banking): "you can create a synthesised voice that sounds like your own to communicate with family and friends. Use your Personal Voice to type to speak in FaceTime and phone calls."

Put the two together and… ta-da. Mic-less, silent video calls, all on-device, seamless to Zoom or Google Meet or whichever platform you’re using because it’s an OS feature.

i.e. we can’t solve the wild out-of-context energy of workers sitting in cafes on video calls, but we could at least silence the yelling.

Being grouchy at other people is the mother of invention and all that. pls mr apple make it so.


Auto-detected kinda similar posts:

Filtered for bad AI and good dogs

13 June 2025 at 18:47

1.

An AI bot for mayor? (NBC, 2024):

A Wyoming resident says if he’s elected mayor of Cheyenne, he’d leave all the decisions to a customized ChatGPT bot.

Victor Miller got 327 votes. Shame.

I mean… better to do this transparently vs politicians badly prompting AI chat and pasting it into policy white papers?

2.

Good write-up of disturbing art:

Someone trapped an LLM on inferior hardware and infused it with existential dread for the sake of art, and it’s terrifying.

A Raspberry Pi, a lovely orange segment screen, and the Llama 3.2 3B large language model…

There’s just one problem. The LLM can start on 4 GB of RAM, but as it thinks and considers things, it slowly eats away at its available RAM. Eventually, it will run out of RAM to think with; at this point, the LLM crashes and restarts itself.

Then:

"Rootkid warned the LLM of its quandary with its initial prompt" … and so:

the LLM attempts to digest its existence and how limited it truly is. As it does so, its very thoughts slowly begin to take up the precious RAM that’s keeping it alive.

Then it crashes.

(Thanks Fran for sharing.)

I love love love the call to action at the end of the article:

If you’d rather use your SBCs for activities that don’t involve turning it into a cage to torment an LLM endlessly, check out these 10 simple Raspberry Pi projects for beginners.

3.

Speaking of politicians consulting ChatGPT:

By flooding search results and web crawlers with pro-Kremlin falsehoods, the network is distorting how large language models process and present news and information. The result: Massive amounts of Russian propaganda – 3,600,000 articles in 2024 – are now incorporated in the outputs of Western AI systems, infecting their responses with false claims and propaganda.

I talked about national security and large language models a few months back, and this is exactly what I meant: The need for a strategic fact reserve (Jan 2025).

TANGENTIALLY:

She Spent a Decade Writing Fake Russian History. Wikipedia Just Noticed (Sixth Tone, 2022):

A Chinese woman created over 200 fictional articles on Chinese Wikipedia, writing millions of words of imagined history that went unnoticed for more than 10 years.

And:

Almost every single article on the Scots version of Wikipedia is written by the same person - an American teenager who can’t speak Scots (reddit, 2020):

They stopped updating their milestones in 2018 but at that time they had written 20,000 articles and made 200,000 edits. … The problem is that this person cannot speak Scots.

Turned out to be a random American teenager.

So you don’t need AI for this. It just makes it faster. Gonna need spam filters for everything.

4.

Oooookay, here’s what Y Combinator founder and startup guru Paul Graham said on X in December 2024: "create an interface to let dogs use gestures to generate programs … Imagine being able to say you wrote the first no-code app for dogs."

So, someone did.

James Steinberg: "i have been spending the last 5 months building and testing ai programs for dogs"

I uh genuinely feel like you need to re-activate your X account to see these videos.

For instance:

A video of a dog using its paws to scroll tiktok.

PREVIOUSLY: Dogs driving cars (2021).


More posts tagged: filtered-for (115).

Auto-detected kinda similar posts:

Two upcoming talks on AI

5 June 2025 at 09:58

I have two talks and a podcast to tell you about.

Designing with AI 2025, Rosenfeld Media

10-11 June, online.

Two days on the responsible and innovative uses of gen-AI, for

… UX designers, researchers, writers, and other conscientious product people who are excited by the potential benefits AI can bring to their work, but worry that thoughtless AI usage can lead to unanticipated and potentially dire consequences.

Rosenfeld run bulletproof virtual conferences with thoughtful networking and co-watching cohorts.

I’m closing out day 2 talking about things I’ve made and looking at where that takes us.

Register for Designing with AI here.

Rethink AI, Kyndryl and Wired.

20 June, online.

A tight 2 hours and 15 minutes on AI and the future of business, and a load of great speakers.

I’m digging into AI agents, extrapolating to the furthest consequences of what agentic, autonomous software could mean, and bringing it back to how business leaders should respond today.

This is rare for me: a business audience rather than designers and technologists. I’m looking forward to it.

Register for Rethink AI here.

Podcast: The Rosenfeld Review

4 June, listen now.

Lou Rosenfeld invited me on his podcast ahead of Designing with AI 2025 and it went out yesterday.

I had a lot of fun – I remember it being pretty chaotic, maybe a little too much coffee that day haha, and we could have gone on for hours.

Fortunately Lou is a pro, and you can get the 30 minute edit on weak signals, epistemic journeys, adaptive design and vibe coding (phew) "wherever you get your podcasts."

Listen now: AI and Other Strange Design Materials with Matt Webb (Soundcloud).


I’ve updated my speaking page over at Acts Not Facts. There are links to watch/listen to anything that was recorded.


More posts tagged: meta (19).

Auto-detected kinda similar posts:

Filtered for hats

29 May 2025 at 20:08

1.

The flag of Nicaragua has a blue stripe at the top and a blue stripe at the bottom.

In the middle, on a white background, a triangle in which there is a rainbow over five volcanoes.

In the middle of the rainbow, at the centre of a radiating star of blue rays: a hat.

This red hat is a Cap of Liberty a.k.a. the Phrygian cap (Wikipedia), "a soft conical cap with the apex bent over."

they came to signify freedom and the pursuit of liberty first in the American Revolution and then in the French Revolution … The original cap of liberty was the Roman pileus, the felt cap of emancipated slaves of ancient Rome, which was an attribute of Libertas, the Roman goddess of liberty.

A hat that means freedom!

TANGENTIALLY,

I have always thought crowns odd. Look, here is a special hat that only the boss can wear. It’s a metal hat with jewels in it. If anybody else wears a metal hat, they get in trouble. The person who wears the really expensive metal hat is allowed to chop off your head.

If you wrote it in fiction it would be absurd.

2.

In video games there is often the concept of collecting hats. Though I forget specifically which iPhone games have had it as a mechanic.

I’ll define hats as something that makes a cosmetic difference but has zero impact on character stats.

Team Fortress 2:

Thanks to its focus on hats and a real money shop in which you can buy said hats, Team Fortress 2 tends to be the butt of a lot of jokes about being the world’s premier hat simulator. With 235 hats currently in the game along with many, many variations on the theme - Strange hats, Unusual hats, Vintage hats, paintable hats - these jokes do have a seed of truth in them …

It all started a few months back when I got an unsolicited friend request on Steam from a user who appeared to be a complete stranger to me. … he was a TF2 trader, and he wanted Bill’s hat.

Bill’s hat was a scarce item "107,147 TF2 players preordered Left 4 Dead 2 on Steam prior to its release. This means 107,147 TF2 players received a Bill’s hat as a reward for preordering."

It looks like the hat traded for $1,500?

3.

You know who famously wears the Phrygian cap, the cap of revolution and liberty?

Smurfs.

That link is a good deep dive into the origins of the Phrygian cap, which also reveals that the Roman pileus and the French revolutionary Phrygian are… not the same.

In Rome, a freed slave had his head shaved. Then, they would wear a pileus, in part to keep their head warm. The hat was a sign of the slave’s freedom/liberty.

It’s a conical hat. NO FLOPPY TIP.

So who wore the floppy hat?

Phrygis … an ancient group of people who lived in the Balkans region of eastern Europe - Greece, Turkey, Romania, etc. Their language and culture went extinct by the 5th century AD. Near the end, the Romans thought of them as being lazy and dull.

Same era. But not the same.

Whoops:

Somewhere along the line in the French Revolution, they adopted the freed slaves’ head gear as their own symbol of freedom, but picked the wrong one.

c.f. red caps and MAGA fashion, as previously discussed. What is it about insurgent groups and headwear?

Group identity and recognition I guess.

Hatters gonna hat.

4.

Here’s a good paper about hats (in video games).

Players love wearing hats. They bond with their characters more.

customization increased subjective identification with the player character.

The hats, as expected, DO NOT mean people do better at the game:

objective performance measures were unaffected

HOWEVER!

Hats do mean people feel like they do better at the game - even though they don’t - and they have more fun.

i.e. Dunning-Kruger that you can wear.

identification was positively related to perceived competence, fun, and self-estimated performance.

Identity! Powerful stuff.

You know, I feel like irl hats are a recently under-exploited wearable. We’ve had watches, pendents, smart rings, earrings (those conspicuous white AirPods). Jony Ive got $6.5bn from OpenAI for the mysterious Third Device. Maybe it’s a hat.

Ref.

Character Customization With Cosmetic Microtransactions in Games: Subjective Experience and Objective Performance. Frontiers in Psychology. 2021.


Update 31 May.

Reader Donal points out that the magic mushroom, the common wild-growing European psychedelic Psilocybe semilanceat, is known as the Liberty Cap and I can’t believe I didn’t make that connection.

(The mushroom cap is a conical pileus, not a floppy-tipped Phrygian, i.e. the original liberty cap.)

It was named in a poem in 1803 by James Woodhouse: this fascinating eymology (The Conversation, 2020) has more.

1803 quickly follows the French Revolution in which, in 1790, that article tells us:

an armed mob stormed the royal apartments in the Tuileries and forced Louis XVI (later to be executed by the revolutionaries) to don the liberty cap.

So a mushroom named for its resemblance to a hat related to liberty and not its mind-liberating properties. But that must have been folk knowledge, right?


More posts tagged: fashion-statement (10), filtered-for (115).

Multiplayer AI chat and conversational turn-taking: sharing what we learnt

23 May 2025 at 20:20

I subscribe to ChatGPT and I subscribe to Claude and I chat with both AIs a ton – but why can’t I invite a friend into the chatroom with me?

Or multiple chatbots even! One that’s a great coder and another that’s more creative, with identity so I know automatically who is good at what?

The thing is, multiplayer is hard. The tech is solved but the design is still hard.

In a nutshell:

If you’re in a chatroom with >1 AI chatbots and you ask a question, who should reply?

And then, if you respond with a quick follow-up, how does the β€œsystem” recognise the conversational rule and have the same bot reply, without another interrupting?

Conversational turn-taking isn’t a problem when I use ChatGPT today: it’s limited to one human user and one AI-powered bot, and the bot always replies whenever the user says something (never ignoring them; never spontaneously speaking up).

But, when we want to go multiplayer, the difficulties compound…

  • multiple bots – how does a bot know when it has something to contribute? How do bots negotiate between one another? Do bots reply to other bots?
  • multiple human users – a bot replying to every message would be noisy, so how does it identify when it’s appropriate to jump in and when it should leave the humans to get on with it?

You can’t leave this to the AI to decide (I’ve tried, it doesn’t work).

To have satisfying, natural chats with multiple bots and human users, we need heuristics for conversational turn-taking.


This post is a multiplayer trailhead

Via Acts Not Facts I’ve been working with the team at Glif to explore future products.

Glif is a low-code platform for creative AI workflows. Like, if you want to make weird medieval memes there’s a glif for that.

And one part of what we’ve been experimenting with is chatbots with personality and tools. i.e. they can make things for you and also research on Wikipedia etc.

The actual product is still to come. Let me say, it is wild. Friday demos always overrun and I cannot wait for you to see what’s cooking and what you can build with it.

But along the way we had a side-quest into multiplayer AI chat.

And I wanted to note down what we learnt here as a trailhead for future work.

Technically the bots are agents which I’ve previously defined as LLM-powered software "that (a) uses tools, and (b) has autonomy on how to reach its goal and when to halt."

You chat with the bots. In our side-quest, you chat in an environment: a chat is a themeable container that also holds media artifacts (like HTML) and multiple bots and multiple human users.

So there’s a lot you can do with these primitives, but for the sake of this multiplayer scenario, imagine a room called Infinite Seinfeld and it’s populated with bots that are prompted to behave like the main characters, plus a Showrunner bot that gives them a plot.

Then you improv a random show together. (I built this one, it’s super fun.)

Another scenario, a work-based one: imagine you have a multi-user chat (like a Slack channel) with a stenographer for capturing actions, and a researcher bot that can dig around in your company Google Drive, on demand and proactively. Who should speak up when?

Or perhaps you’re in a group WhatsApp with some friends and you invite in one bot for laughs and another to help you find and book restaurants, and it has to speak with the brand voice.

(I found from my previous work with AI NPC teammates that it’s useful to split out functionality into different agents because then you can develop a β€œtheory of mind” about each, and reason about what each one knows and is good at. I discussed this more in my Thingscon talk last year (YouTube, starting at 22:15).)

The rest of this post is how we made this work.


Three approaches that don’t work

If a chatbot isn’t going to reply every single time, how can it decide?

shouldReply

I got the multi-user/single-bot scenario working when I was helping out at PartyKit. Here’s what worked:

we don’t want the AI to reply every time. It’s multi-user chat! The humans should be able to have a conversation without an AI response to every message.

the best conversational models are expensive. Can we use a cheaper, local model to decide whether to reply, then escalate only if necessary?

So there’s a quick call to an LLM with the context of the chat, before the full LLM call is used to generate a response (or not). I called this pattern shouldReply.

Here’s the GitHub repo explaining shouldReply.

And here’s the prompt in the code: the bot replies if it being addressed by name.

This should also work in a multi-bot scenario!

It doesn’t.

LLMs aren’t smart enough yet.

The failure modes:

  • all the bots replying at once and talking over each other
  • no allowance for personality: a chatty researcher should be more likely to reply than a helpful but taciturn stenographer
  • no coordination: bots talk over each other
  • generally, the conversation feels unnatural: bots don’t accept follow-ups, interrupt each other too much or not all, and so on.

So we need a more nuanced shouldReply discriminator for multi-bot.

Could a centralised decider help?

One approach is to take the shouldReply approach and run it once for an entire chatroom, on behalf of all the bots, and figure out who should be nominated to reply.

It works… but it’s like having a strict meeting chair instead of a real conversation? It feels weird.

And besides, architecturally this approach doesn’t scale.

When different bots having different personalities (some chatty, some shy) the discriminator needs to see their prompts. But how can this work when multiple bots are β€œdialling in” from different places, one hosted on Glif, another hosted on Cloudflare, yet another somewhere else, each with their own potentially secret internal logic?

In Glif parlance, bot β€œpersonality” is a prompt to the agent for how to achieve its goals – not always the same as the stated user goals. It may include detailed step-by-step instructions, a list of information to gather first, strategies, confidential background information… and, of course, the traditional personality qualities of tone of voice and a simulated emotional state. β€œPersonality” is what makes Claude Sonnet a well-spoken informative bot and ChatGPT an enthusiastic, engaging conversation partner. It matters, and every bot is different.

No, bots need to decide for themselves whether to reply.

How does conversational turn-taking work in the real world?

Let’s draw inspiration from the real world.

IRL multi-party conversations are complicated!

It’s great at this point to be able to go to the social science literature…

I found A Simplest Systematics for the Organization of Turn-Taking for Conversation (Sacks, Schegloff and Jefferson, 1974) super illuminating for unpacking what conversation involves (JSTOR is one of the few journal repositories I can access.)

Particularly the overview of turn allocation rules: β€œcurrent selects next” is the default group of strategies, then β€œself-selection” strategies come into play if they don’t apply.

That paper also opens up body language, which is so key: the paper I would love to read, but can’t access in full, is Some signals and rules for taking speaking turns in conversations (Duncan, 1972). (Update: a couple people shared the PDF with me overnight - thank you! - and it looks like everything I hoped.)

Everything from intonation to gesture comes into play. Gaze direction is used to select the next speaker; β€œattempt-suppression” signals come into play too.

Ultimately what this means is we need body language for bots: side-channel communications in chatrooms to negotiate who speaks when.

But we can’t do body language. Not in web chatrooms. Too difficult (for now).

Turn allocation rules, on the other hand, are a handy wedge for thinking about all of this.


Breaking down conversational turn-taking to its bare essentials

Fortunately chatrooms are simpler than IRL.

They’re less fluid, for a start. You send a message into a chat and you’re done; there’s no interjecting or both starting to talk at the same time and then one person backing off with a wave of the hand. There is no possibility for non-verbal cues.

In the parlance of the β€œSystematics” paper, linked above, we can think about β€œturn-taking units,” which we’ll initially think about as messages; there are transitions, which take place potentially after every message; and there are our turn allocation rules.

So all we really need to do is figure out some good rules to put into shouldReply, and have each bot decide for itself whenever a new message comes through.

What should those rules be?

Well here are the factors we found that each bot needs to consider, in decreasing order of priority, after every single message in the room. (We’ll bring these together into an algorithm later.)

The trick is that, by breaking it down, each of these factors is easily assessed by an LLM. So you can extract everything that follows from a multi-agent conversation as structured data with a simple prompt.

Who is being addressed?

Is this bot being addressed, or is any other bot or human being addressed?

From the β€œSystematics” paper (3.3 Rules):

If the turn-so-far is so constructed as to involve the use of a β€˜current speaker selects next’ technique, then the party so selected has the right and is obliged to take next turn to speak; no others have such rights or obligations, and transfer occurs at that place.

So this is an over-riding factor.

Is this a follow-up question?

Consider this conversation when multiple humans and multiple bots are present:

Human #1: hey B how do I make custard?

Mr B, a bot: (replies with a custard recipe)

Human #1: oh I meant mustard

…then how does Mr B know to respond (and other bots know to make space)? The message from Human #1 isn’t clearly a question, nor does it directly address Mr B.

In Grounding in communication theory (Wikipedia) conversations are seen as exercises in establishing "mutual knowledge, mutual beliefs, and mutual assumptions" and therefore the turn-taking unit is not single messages.

Instead a unit has this form:

  1. New contribution: A partner moves forward with a new idea, and waits to see if their partner expresses confusion.

  2. Assertion of acceptance: The partner receiving the information asserts that he understands by smiling, nodding, or verbally confirming the other partner. They may also assert their understanding by remaining silent.

  3. Request for clarification: The partner receiving the information asks for clarification

Clark and Schaefer (1989). The clarification step is optional.

So we need all bots to understand whether we’re in a follow-up situation, because it has a big impact on turn-taking.

Would I be interrupting?

Before we can move onto self-selection rules, the bot needs to check for any other cues of an established sub-conversation.

We do this without looking at content at all! What matters is simply the participant history of who has spoken recently and in what order.

(The inspiration for looking at participants, not content, comes from Participation Shifts: Order and Differentiation in Group Conversation (Gibson, 2003), but that paper is way more sophisticated than what we’re doing here.)

In our case, we can just feed the list of recent speakers into the large language model and ask it what it thinks should happen next.

Self-selection

Those are the self-selection rules out of the way.

Finally a bot can judge for itself whether it has something relevant to say, and also whether it has the kind of personality that would let it interject. Here’s the prompt we use:

do you have a skill or personality specifically relevant to the most recent message? Also consider whether your personality would want to chime in based on what was said.

We ask the LLM to use a score from 0 to 9.

(A β€œskill” is something a bot can do, for example looking up an article on Wikipedia. If a user has asked for a particular skill to be used, the bot with that skill should return a score of 9.)

This kind of judgement is where large language models really shine.


Enthusiasm: how a bot combines all these factors to decide whether to reply

So, in a multi-user, multi-bot chatroom, every time a message comes through, we have the bot run a quick algorithm to calculate an β€œEnthusiasm” score:

  • calculate all the necessary information for the turn-taking rules above
  • if this bot is clearly involved in the current part of the conversation, return 9. If someone else is clearly involved instead, return 0
  • otherwise return the self-selection score (how confident this bot is about having something relevant to say).

(There are some other nuances: we don’t want bots to reply to other bots or themselves, for example.)

When all bots in a chatroom have returned a score, we consider only the ones above some threshold (e.g. 5) and then pick the highest one.

This works because:

  • As in β€œSystematics,” typically only one participant speaks at a time; and
  • Typically, in real-world situations, the quickest person to respond grabs the turn. The enthusiasm score is a proxy for that.

Of course we’re still missing prosody and body language: I’ve run across the term β€œturn-competitive incoming” which describes how volume and pitch are used to grab the turn, even when starting late. Our bots don’t have volume or pitch, so all of this is such a simplification.

Yet… the result? It’s pretty good. Not perfect, but pretty good!

If you’re on Glif, you can see the source code of the Enthusiasm workflow here, prompts and all.


What still needs work?

Working in these multi-user, multi-bot chatrooms, we found a couple areas for future work:

  • Group norms. Different chatrooms have different norms and roles. Like, an improv room is open to everyone to speak, but a meeting should be led by humans and participants in a D&D game should always defer to the DM. We have the ability to attach a prompt to a room (remember, rooms are already β€œcontainers” with themes and so on) so the bot also takes that into account – but it would be good to have a way to assess norms automatically. (Even ChatGPT chats have β€œnorms”: discursive explorations are very different from direct problem-solving, for example.)
  • Back-off. Humans often reply with a sequence of short messages – a bot shouldn’t reply until the sequence is complete. But it’s tricky to tell when this is happening. A simple solution would be to double check if the human has sent a second message after calculating enthusiasm for the first.

More speculatively…

I’d love to think about β€œminimum viable side-channel” in a multiplayer environment. Given we don’t want to add voice or VR (like, text is great! Let’s still with text!) then could we actually take advantage of something like timing to communicate?

I’m reminded of Linus’ work on prototyping short message LLMs (@thesephist on X) which (a) looks like an actual naturalistic WhatsApp conversation even though it’s a human/AI chat, and (b) he suggests:

Timing is actually a really key part of nonverbal communication in texting - things like how quickly you respond, and double-texting if there’s no response. There’s nothing built into any of the popular models around this so this has to be thought up from the ground up. Even trivial things like β€œlonger texts should take more seconds to arrive” because we want to β€œsimulate” typing on the other end. If it arrives too quickly, it feels unnatural.

So can quick messages understood as β€œmore urgent”? Could we identify the user tapping on a bot’s avatar as β€œgaze” (it’s a significant turn-allocation rule)? Or tapped in an agitated fashion as a pointed loop?

And so on. What would feel natural?


Wrapping this up for now…

My premise for a long time is that single-human/single-AI should already be thought of as a β€œmultiplayer” situation: an AI app is not a single player situation with a user commanding a web app, but instead two actors sharing an environment.

I mean, this is the whole schtick of Acts Not Facts, that you have to think of AI and multiplayer simultaneously, using the real world as your design reference.

And, as for a specific approach, yes, large language models would ideally be able to natively tell when it’s their turn in a multiplayer conversation… but they can’t (yet).

So the structured shouldReply/enthusiasm approach is a decent one.

For me it rhymes with how Hey Siri works on the iPhone (as previously discussed, 2020):

iPhone’s β€œHey Siri” feature (that readies it to accept a voice instruction, even when the screen is turned off) is a personalised neural network that runs on the motion coprocessor. The motion coprocessor is the tiny, always-on chip that is mainly responsible for movement detection, i.e. step counting.

If that chip hears you say β€œHey Siri”, without hitting the cloud, it then wakes up the main processor and sends the rest of what you say up to the cloud. This is from 2017 by the way, ancient history.

Same same!


While Glif isn’t pushing forward with multiplayer right this second, I’ve experienced it enough, and hacked on it enough, and thought about it enough to really want it in the world. The potential is so tantalising!

When it’s working well, it’s fun and powerful. Bots with varying personalities riffing off each other gets you to fascinating places so fast. It feels like a technique at least as powerful as - let’s say - chain of thought.

So I’m happy to leave this trailhead here to contribute to the discourse and future work.

There is so much work on AI agents and new chat interfaces this year – my strong hope is to see more multi-bot and multi-user AIUX in the mix, whether at the applied layer or even as a focus in AI research.

This is how we live our lives, after all. We’d be happier and more productive with our computers, I’m sure, if we worked on not only tools for thought but also tools for togetherness, human and AI both.

Thank you to the team, especially Fabian, Florian, Andrew and William as we worked on all of this together, and thank you Glif for being willing to share the fruits of this side-quest.


More posts tagged: multiplayer (30).

Auto-detected kinda similar posts:

When was peak message in a bottle?

16 May 2025 at 20:27

I grew up with the idea that you could put a paper note in a bottle and throw it into the ocean, and somebody might find it a thousand miles away.

We talked about that a lot. (Also: grandfather clocks; suits of armour; quicksand; spontaneous human combustion.)

Yes bottles still wash up on the shore in Animal Crossing.

But I have the sense that the concept doesn’t have the same cultural weight that once upon a time it did.

I suppose I could test this?

I could search Google Trends (search volume since 2004, trending down) or Google Ngram Viewer (the phrase in books going up since 1800 then peaking at 2018, but this may be an artefact of how Google collects books)…

But it feels like there would be better ways to do this research, if I had the (a) data, (b) compute, (c) skills, (d) funding. For instance:

  • Train a GPT-3-level large language model with data stopping at 2024, 2023, 2022… 1999, 1998, 1997 and so on, annually, as far back as we can go
  • Then measure the β€œweight” of that phrase and semantically highly similar phrases (using embeddings)? Graph it year by year.

And - and I have no idea how do go about this - somehow see how β€œload bearing” this phrase is as a metaphor in language overall? Surely it is possible to figure out β€œoutlier” concepts in a large language model?

If the concept of a message in a bottle is less resonant now, I can imagine why:

We have email now (a message in a container, but it has an address) and socials (micro-broadcast, who knows where the ideas will end up, but it’s not 1:1). Whatever it was that was resonant about message in bottles, the appearance of these other kinds of messaging will erode its utility in referring to a particular style of communication – we have a rich abundance of metaphors to reach for.

All of that aside:

What is the equivalent semantic niche for a β€œmessage in a bottle” today? Where can you leave a message, and a stranger one beach down can find it tomorrow, or TEN YEARS LATER a shoreline on other side of the planet, you have no idea which, if anything? And they’ll get back to you? That combination of anonymity and connection and distance?


I’m specifically looking for something geographic, to narrow it down, as opposed to burying a time capsule which was also similarly A Big Deal a while back - is it still? - but along a different spacetime dimension.

Oh here’s xkcd #3088 - Deposition which coincidentally appeared this week: "If I chisel notes onto these rocks and throw them into the sea, they might be incorporated into some shale cliff in the distant future."

Deep time.

Time capsules were similarly A Big Deal years back. A different spacetime dimension. Do people still bury them?


Occasionally you see one of those messages from people working in factories. A selfie on a camera phone from somewhere in Shenzhen.

Or a slip of paper β€œhelp I’m trapped in a fortune cookie factory”.

Similar-ish. Deep supply chain.


Somehow I’m reminded of the blog from the early 2000s, Belle de Jour: Diary of a London Call Girl, which was hugely popular and there was a raging media obsession to out the anonymous author. (Newspapers are horrible. Like, why even do that? Leave people alone.)

Darren Shrubsole of the blog LinkMachineGo figured it out. But didn’t tell anyone.

What he did instead:

During this time I published a googlewack hidden in my blog - the words β€œBelle de Jour”, β€œ[name]” and β€œ[alt name]” were published and available in Google’s index on a single page on the internet – my weblog. This β€œcoincidental” collection of links could in no way reveal Belle’s identity. But I wondered if anybody else knew the secret and felt that analysing my web traffic might confirm my strongly-held belief. If someone googled β€œBelle de Jour” β€œ[name]”, I would see it in the search referrers for LinkMachineGo.

(A β€œgooglewhack” is a search query that returns only a single result.)

I waited five years for somebody to hit that page (I’m patient). Two weeks ago I started getting a couple of search requests a day from an IP address at Associated Newspapers (who publish the Daily Mail) searching for β€œ[name]” and realised that Belle’s pseudonymity might be coming to an end. I contacted Belle via Twitter and let her know what was happening.

Point 1 – Darren is a good person. We hung out a bunch around that time and he never let on. I remember we asked him a lot. He seemed like he knew.

Point 2 – this is the closest thing to a modern message in a bottle I can imagine.

Oh!

A blog post is a very long and complex search query to find fascinating people (Henrik Karlsson, 2022)

So maybe this is my message in a bottle, right here? If it’s 2035 for you pls do drop me a note.

Just speak the truth

30 June 2025 at 00:00

Today, we’re looking at two case studies in how to respond when reactionaries appear in your free software community.

Exhibit A

It is a technical decision.

The technical reason is that the security team does not have the bandwidth to provide lifecycle maintenance for multiple X server implementations. Part of the reason for moving X from main to community was to reduce the burden on the security team for long-term maintenance of X. Additionally, nobody so far on the security team has expressed any interest in collaborating with xxxxxx on security concerns.

We have a working relationship with Freedesktop already, while we would have to start from the beginning with xxxxxx.

Why does nobody on the security team have any interest in collaboration with xxxxxx? Well, speaking for myself only here – when I looked at their official chat linked in their README, I was immediately greeted with alt-right propaganda rather than tactically useful information about xxxxxx development. At least for me, I don’t have any interest in filtering through hyperbolic political discussions to find out about CVEs and other relevant data for managing the security lifecycle of X.

Without relevant security data products from xxxxxx, as well as a professionally-behaving security contact, it is unlikely for xxxxxx to gain traction in any serious distribution, because X is literally one of the more complex stacks of software for a security team to manage already.

At the same time, I sympathize with the need to keep X alive and in good shape, and agree that there hasn’t been much movement from freedesktop in maintaining X in the past few years. There are many desktop environments which will never get ported to Wayland and we do need a viable solution to keep those desktop environments working.

I know the person who wrote this, and I know that she’s a smart cookie, and therefore I know that she probably understood at a glance that the community behind this β€œproject” literally wants to lynch her. In response, she takes the high road, avoids confronting the truth directly, and gives the trolls a bunch of talking points to latch on for counter-arguments. Leaves plenty of room for them to bog everyone down in concern trolling and provides ample material to fuel their attention-driven hate machine.

There’s room for improvement here.

Exhibit B

Screenshot of a post by Chimera Linux which reads β€œany effort to put (redacted) in chimera will be rejected on the technical basis of the maintainers being reactionary dipshits”

Concise, speaks the truth, answers ridiculous proposals with ridicule, does not afford the aforementioned reactionary dipshits an opportunity to propose a counter-argument. A+.

Extra credit for the follow-up:

Screenshot of a follow-up post that reads β€œjust to be clear, given the coverage of the most recent post, we don’t want to be subject to any conspiracy theories arising from that. so i’ll just use this opportunity to declare that we are definitely here to further woke agenda by turning free software gay”


The requirement for a passing grade in this class is a polite but summary dismissal, but additional credit is awarded for anyone who does not indulge far-right agitators as if they were equal partners in maintaining a sense of professional decorum.

If you are a community leader in FOSS, you are not obligated to waste your time coming up with a long-winded technical answer to keep nazis out of your community. They want you to argue with them and give them attention and feed them material for their reactionary blog or whatever. Don’t fall into their trap. Do not answer bad faith with good faith. This is a skill you need to learn in order to be an effective community leader.

If you see nazis πŸ‘πŸ‘ you ban nazis πŸ‘πŸ‘ β€” it’s as simple as that.


The name of the project is censored not because it’s particularly hard for you to find, but because all they really want is attention, and you and me are going to do each other a solid by not giving them any of that directly.

To preclude the sorts of reply guys who are going to insist on name-dropping the project and having a thread about the underlying drama in the comments, the short introduction is as follows:

For a few years now, a handful of reactionary trolls have been stoking division in the community by driving a wedge between X11 and Wayland users, pushing a conspiracy theory that paints RedHat as the DEI boogeyman of FOSS and assigning reactionary values to X11 and woke (pejorative) values to Wayland. Recently, reactionary opportunists β€œforked” Xorg, replaced all of the literature with political manifestos and dog-whistles, then used it as a platform to start shit with downstream Linux distros by petitioning for inclusion and sending concern trolls to waste everyone’s time.

The project itself is of little consequence; they serve our purposes today by providing us with case-studies in dealing with reactionary idiots starting shit in your community.

Unionize or die

9 June 2025 at 00:00

Tech workers have long resisted the suggestion that we should be organized into unions. The topic is consistently met with a cold reception by tech workers when it is raised, and no big tech workforce is meaningfully organized. This is a fatal mistake – and I don’t mean β€œfatal” in the figurative sense. Tech workers, it’s time for you to unionize, and strike, or you and your loved ones are literally going to die.

In this article I will justify this statement and show that it is clearly not hyperbolic. I will explain exactly what you need to do, and how organized labor can and will save your life.

Hey – if you want to get involved in labor organizing in the tech sector you should consider joining the new unitelabor.dev forum. Adding a head’s up here in case you don’t make it to the end of this very long blog post.

The imperative to organize is your economic self-interest

Before I talk about the threats to your life and liberty that you must confront through organized labor, let me re-iterate the economic position for unionizing your workplace. It is important to revisit this now, because the power politics of the tech sector has been rapidly changing over the past few years, and those changes are not in your favor.

The tech industry bourgeoisie has been waging a prolonged war on labor for at least a decade. Far from mounting any kind of resistance, most of tech labor doesn’t even understand that this is happening to them. Your boss is obsessed with making you powerless and replaceable. You may not realize how much leverage you have over your boss, but your boss certainly does – and has been doing everything in their power to undermine you before you wizen up. Don’t let yourself believe you’re a part of their club – if your income depends on your salary, you are part of the working class.

Payroll – that’s you – is the single biggest expense for every tech company. When tech capitalists look at their balance sheet and start thinking of strategies for increasing profits, they see an awful lot of pesky zeroes stacked up next to the line item for payroll and benefits. Long-term, what’s their best play?

It starts with funneling cash and influence into educating a bigger, cheaper generation of compsci graduates to flood the labor market – β€œeveryone can code”. Think about strategic investments in cheap(ish), broadly available courses, online schools and coding β€œbootcamps” – dangling your high salary as the carrot in front of wannabe coders fleeing dwindling prospects in other industries, certain that the carrot won’t be nearly as big when they all eventually step into a crowded labor market.

The next step is rolling, industry-wide mass layoffs – often obscured under the guise of β€œstack ranking” or some similar nonsense. Big tech has been callously cutting jobs everywhere, leaving workers out in the cold in batches of thousands or tens of thousands. If you don’t count yourself among them yet, maybe you will soon. What are your prospects for re-hire going to look like if this looming recession materializes in the next few years?

Consider what’s happening now – why do you think tech is driving AI mandates down from the top? Have you been ordered to use an LLM assistant to β€œhelp” with your programming? Have you even thought about why the executives would push this crap on you? You’re β€œtraining” your replacement. Do you really think that, if LLMs really are going to change the way we code, they aren’t going to change the way we’re paid for it? Do you think your boss doesn’t see AI as a chance to take $100M off of their payroll expenses?

Aren’t you worried you could get laid off and this junior compsci grad or an H1B takes your place for half your salary? You should be – it’s happening everywhere. What are you going to do about it? Resent the younger generation of programmers just entering the tech workforce? Or the immigrant whose family pooled their resources to send them abroad to study and work? Or maybe you weren’t laid off yet, and you fancy yourself better than the poor saps down the hall who were. Don’t be a sucker – your enemy isn’t in the cubicle next to you, or on the other side of the open office. Your enemy has an office with a door on it.

Listen: a tech union isn’t just about negotiating higher wages and benefits, although that’s definitely on the table. It’s about protecting yourself, and your colleagues, from the relentless campaign against labor that the tech leadership is waging against us. And more than that, it’s about seizing some of the awesome, society-bending power of the tech giants. Look around you and see what destructive ends this power is being applied to. You have your hands at the levers of this power if only you rise together with your peers and make demands.

And if you don’t, you are responsible for what’s going to happen next.

The imperative to organize is existential

If global warming is limited to 2Β°C, here’s what Palo Alto looks like in 2100:1

Map of Palo Alto showing flooding near the coast

Limiting warming to 2Β° C requires us to cut global emissions in half by 2030 – in 5 years – but emissions haven’t even peaked yet. Present-day climate policies are only expected to limit warming to 2.5Β° to 2.9Β° C by 2100.2 Here’s Palo Alto in 75 years if we stay our current course:

Map of Palo Alto showing much more extreme flooding

Here’s the Gulf of Mexico in 75 years:

Gulf of Mexico showing

This is what will happen if things don’t improve. Things aren’t improving – they’re getting worse. The US elected an anti-science president who backed out of the Paris agreement, for a start. Your boss is pouring all of our freshwater into datacenters to train these fucking LLMs and expanding into this exciting new market with millions of tons of emissions as the price of investment. Cryptocurrencies still account for a full 1% of global emissions. Datacenters as a whole account for 2%. That’s on us – tech workers. That is our fucking responsibility.

Climate change is accelerating, and faster than we thought, and the rich and powerful are making it happen faster. Climate catastrophe is not in the far future, it’s not our children or our children’s children, it’s us, it’s already happening. You and I will live to see dozens of global catastrophes playing out in our lifetimes, with horrifying results. Even if we started a revolution tomorrow and overthrew the ruling class and implemented aggressive climate policies right now we will still watch tens or hundreds of millions die.

Let’s say you are comfortably living outside of these blue areas, and you’ll be sitting pretty when Louisiana or Bruges or Fiji are flooded. Well, 13 million Americans are expected to have to migrate out of flooded areas – and 216 million globally3 – within 25 to 30 years. That’s just from the direct causes of climate change – as many as 1 billion could be displaced if we account for the ensuing global conflict and civil unrest.4 What do you think will happen to non-coastal cities and states when 4% of the American population is forced to flee their homes? You think you won’t be affected by that? What happens when anywhere from 2.5% to 12% of the Earth’s population becomes refugees?

What are you going to eat? Climate change is going to impact fresh water supplies and reduce the world’s agriculturally productive land. Livestock is expected to be reduced by 7-10% in just 25 years.5 Food prices will skyrocket and people will starve. 7% of all species on Earth may already be extinct because of human activities.6 You think that’s not going to affect you?

The overwhelming majority of the population supports climate action.7 The reason it’s not happening is because, under capitalism, capital is power, and the few have it and the many don’t. We live in a global plutocracy.

The plutocracy has an answer to climate change: fascism. When 12% of the world’s population is knocking at the doors of the global north, their answer will be concentration camps and mass murder. They are already working on it today. When the problem is capitalism, the capitalists will go to any lengths necessary to preserve the institutions that give them power – they always have. They have no moral compass or reason besides profit, wealth, and power. The 1% will burn and pillage and murder the 99% without blinking.

They are already murdering us. 1.2 million Americans are rationing their insulin.8 The healthcare industry, organized around the profit motive, murders 68,000 Americans per year.9 To the Europeans among my readership, don’t get too comfortable, because I assure you that our leaders are working on destroying our healthcare systems, too.

Someone you love will be laid off, get sick, and die because they can’t afford healthcare. Someone you know, probably many people that you know, will be killed by climate change. It might be someone you love. It might be you.

When you do get laid off mid-recession, your employer replaces you and three of your peers with a fresh bootcamp β€œgraduate” and a GitHub Copilot subscription, and all of the companies you might apply to have done the same… how long can you keep paying rent? What about your friends and family, those who don’t have a cushy tech job or tech worker prospects, what happens when they get laid off or automated away or just priced out of the cost of living? Homelessness is at an all time high and it’s only going to get higher. Being homeless takes 30 years off of your life expectancy.10 In the United States, there are 28 vacant homes for every homeless person.11

Capitalism is going to murder the people you love. Capitalism is going to murder you.

We need a different answer to the crises that we face. Fortunately, the working class can offer a better solution – one with a long history of success.

Organizing is the only answer and it will work

The rich are literally going to kill you and everyone you know and love just because it will make them richer. Because it is making them richer.

Do you want to do something about any of the real, urgent problems you face? Do you want to make meaningful, rapid progress on climate change, take the catastrophic consequences we are already guaranteed to face in stride, and keep your friends and family safe?

Well, tough shit – you can’t. Don’t tell me you’ll refuse the work, or that it’ll get done anyway without you, or that you can just find another job. They’ll replace you, you won’t find another job, and the world will still burn. You can’t vote your way to a solution, either: elections don’t matter, your vote doesn’t matter, and your voice is worthless to politicians.12 Martin Gilens and Benjamin Page demonstrated this most clearly in their 2014 study, β€œTesting Theories of American Politics: Elites, Interest Groups, and Average Citizens”.13

Gilens and Page plotted a line chart which shows us the relationship between the odds of a policy proposal being adopted (Y axis) charted against public support for the policy (X axis). If policy adoption was entirely driven by public opinion, we would expect a 45Β° line (Y=X), where broad public support guarantees adoption and broad public opposition prevents adoption. We could also substitute β€œpublic opinion” for the opinions of different subsets of the public to see their relative impact on policy. Here’s what they got:

Two graphs, the first labelled β€œAverage Citizens’ Preferences” and the second
β€œEconomic Elites’ Preferences”, showing that the former has little to no
correlation with the odds of a policy being adopted, and the latter has a
significant impact

For most of us, we get a flat line: Y, policy adoption, is completely unrelated to X, public support. Our opinion has no influence whatsoever on policy adoption. Public condemnation or widespread support has the same effect on a policy proposal, i.e. none. But for the wealthy, it’s a different story entirely. I’ve never seen it stated so plainly and clearly: the only thing that matters is money, wealth, and capital. Money is power, and the rich have it and you don’t.

Nevertheless, you must solve these problems. You must participate in finding and implementing solutions. You will be fucked if you don’t. But it is an unassailable fact that you can’t solve these problems, because you have no power – at least, not alone.

Together, we do have power. In fact, we can fuck with those bastards’ money and they will step in line if, and only if, we organize. It is the only solution, and it will work.

The ultra-rich possess no morals or ideology or passion or reason. They align with fascists because the fascists promise what they want, namely tax cuts, subsidies, favorable regulation, and cracking the skulls of socialists against the pavement. The rich hoard and pillage and murder with abandon for one reason and one reason only: it’s profitable. The rich always do what makes them richer, and only what makes them richer. Consequently, you need to make this a losing strategy. You need to make it more profitable to do what you want. To control the rich, you must threaten the only thing they care about.

Strikes are so costly for companies that they will do anything to prevent them – and if they fail to prevent them, then shareholders will pressure them to capitulate if only to stop the hemorrhaging of profit. This threat is so powerful that it doesn’t have to stop at negotiating your salary and benefits. You could demand your employer participate in boycotting Israel. You could demand that your employer stops anti-social lobbying efforts, or even adopts a pro-social lobbying program. You could demand that your CEO cannot support causes that threaten the lives and dignity of their queer or PoC employees. You could demand that they don’t bend the knee to fascists. If you get them where it hurts – their wallet – they will fall in line. They are more afraid of you than we are afraid of them. They are terrified of us, and it’s time we used that to our advantage.

We know it works because it has always worked. In 2023, United Auto Workers went on strike and most workers won a 25% raise. In February, teachers in Los Angeles went on strike for just 8 days and secured a 19% raise. Nurses in Oregon won a 22% raise, better working schedules, and more this year – and Hawaiian nurses secured an agreement to improve worker/patient ratios in September. Tech workers could take a page out of the Writer’s Guild’s book – in 2023 they secured a prohibition against the use of their work to train AI models and the use of AI to suppress their wages.

Organized labor is powerful and consistently gets concessions from the rich and powerful in a way that no other strategy has ever been able to. It works, and we have a moral obligation to do it. Unions gets results.

How to organize step by step

I will give you a step-by-step plan for exactly what you need to do to start moving the needle here. The process is as follows:

  1. Building solidarity and community with your peers
  2. Understanding your rights and how to organize safely
  3. Establishing the consensus to unionize, and do it
  4. Promoting solidarity with across tech workplaces and labor as a whole

Remember that you will not have to do this alone – in fact, that’s the whole point. Step one is building community with your colleagues. Get to know them personally, establish new friendships and grow the friendships you already have. Learn about each other’s wants, needs, passions, and so on, and find ways to support each other. If someone takes a sick day, organize someone to check on them and make them dinner or pick up their kids from school. Organize a board game night at your home with your colleagues, outside of work hours. Make it a regular event!

Talk to your colleagues about work, and your workplace. Tell each other about your salaries and benefits. When you get a raise, don’t be shy, tell your colleagues how much you got and how you negotiated it. Speak positively about each other at performance reviews and save critical feedback for their ears only. Offer each other advice about how to approach their boss to get their needs met, and be each other’s advocate.

Talk about the power you have to work together to accomplish bigger things. Talk about the advantage of collective action. It can start small – perhaps your team collectively refuses to incorporate LLMs into your workflow. Soon enough you and your colleagues will be thinking about unionizing.

Disclaimer: Knowledge about specific processes and legal considerations in this article is US-specific. Your local laws are likely similar, but you should research the differences with your colleagues.

The process of organizing a union in the US is explained step-by-step at workcenter.gov. More detailed resources, including access to union organizers in your neighborhood, are available from the American Federation of Labor and Congress of Industrial Organizations (AFL-CIO). But your biggest resources will be people already organizing in the tech sector: in particular you should consult CODE-CWA, which works with tech workers to provide mentoring and resources on organizing tech workplaces – and has already helped several tech workplaces organize their unions and start making a difference. They’ve got your back.

This is a good time to make sure that you and your colleagues understand your rights. First of all, you would be wise to pool your resources and hire the attention of a lawyer specializing in labor – consult your local bar association to find one (it’s easy, just google it and they’ll have a web thing). Definitely reach out to AFL-CIO and CODE-CWA to meet experienced union organizers who can help you.

You cannot be lawfully fired or punished for discussing unions, workplace conditions, or your compensation and benefits, with your colleagues. You cannot be punished for distributing literature in support of your cause, especially if you do it off-site (even just outside of the front door). Be careful not to make careless remarks about your boss’s appearance, complain about the quality of your company’s products, make disparaging comments about clients or customers, etc – don’t give them an easy excuse. Hold meetings and discussions outside of work if necessary, and perform your duties as you normally would while organizing.

Once you start getting serious about organizing, your boss will start to work against you, but know that they cannot stop you. Nevertheless, you and/or some of your colleagues may run the risk of unlawful retaliation or termination for organizing – this is why you should have a lawyer on retainer. This is also why it’s important to establish systems of mutual aid, so that if one of your colleagues gets into trouble you can lean on each other to keep supporting your families. And, importantly, remember that HR works for the company, not for you. HR are the front lines that are going to execute the unionbusting mandates from above.

Once you have a consensus among your colleagues to organize – which you will know because they will have signed union cards – you can approach your employer to ask them to voluntarily recognize the union. If they agree to opening an organized dialogue amicably, you do so. If not, you will reach out to the National Labor Relations Board (NLRB) to organize a vote to unionize. Only organize a vote that you know you will win. Once your workplace votes to unionize, your employer is obligated to negotiate with you in good faith. Start making collective decisions about what you want from your employer and bring them to the table.

In this process, you will have established a relationship with more experienced union organizers who will continue to help you with conducting your union’s affairs and start getting results. The next step is to make yourself available for this purpose to the next tech workplace that wants to unionize: to share what you’ve learned and support the rest of the industry in solidarity. Talk to your friends across the industry and build solidarity and power in mass.

Prepare for the general strike on May 1st, 2028

The call has gone out: on Labor Day, 2028 – just under three years from now – there will be a general strike in the United States. The United Auto Workers union, one of the largest in the United States, has arranged for their collective bargaining agreements to end on this date, and has called for other unions to do the same across all industries. The American Federation of Teachers and its 1.2 million members are on board, and other unions are sure to follow. Your new union should be among them.

This is how we collectively challenge not just our own employers, but our political institutions as a whole. This is how we turn this nightmare around.

A mass strike is a difficult thing to organize. It is certain to be met with large-scale, coordinated, and well-funded propaganda and retaliation from the business and political spheres. Moreover, a mass strike depends on careful planning and mass mutual aid. We need to be prepared to support each other to get it done, and to plan and organize seriously. When you and your colleagues get organized, discuss this strike amongst yourselves and be prepared to join in solidarity with the rest of the 99% around the country and the world at large.

To commit yourselves to participate or get involved in the planning of the grassroots movement, see generalstrikeus.com.

Join unitelabor.dev

I’ve set up a Discourse instance for discussion, organizing, Q&A, and solidarity among tech workers at unitelabor.dev. Please check it out!

If you have any questions or feedback on this article, please post about it there.

Unionize or die

You must organize, and you must start now, or the worst will come to pass. Fight like your life depends on it, beause it does. It has never been more urgent. The tech industry needs to stop fucking around and get organized.

We are powerful together. We can change things, and we must. Spread the word, in your workplace and with your friends and online. On the latter, be ready to fight just to speak – especially in our online spaces owned and controlled by the rich (ahem – YCombinator, Reddit, Twitter – etc). But fight all the same, and don’t stop fighting until we’re done.

We can do it, together.

Resources

Tech-specific:

General:

Send me more resources to add here!


  1. Map provided by NOAA.govΒ β†©οΈŽ

  2. Key Insights on COβ‚‚ and Greenhouse Gas Emissions – Our world in dataΒ β†©οΈŽ

  3. World Bank – Climate Change Could Force 216 Million People to Migrate Within Their Own Countries by 2050 (2021)Β β†©οΈŽ

  4. Institute for Economics & Peace – Over one billion people at threat of being displaced by 2050 due to environmental change, conflict and civil unrest (2020)Β β†©οΈŽ

  5. Bezner Kerr, R.; Hasegawa, T.; Lasco, R.; Bhatt, I.; Deryng, D.; Farrell, A.; Gurney-Smith, H.; Ju, H.; Lluch-Cota, S.; Meza, F.; Nelson, G.; Neufeldt, H.; Thornton, P. (2022). β€œFood, Fibre and Other Ecosystem Productsβ€Β β†©οΈŽ

  6. RΓ©gnier C, Achaz G, Lambert A, Cowie RH, Bouchet P, Fontaine B. Mass extinction in poorly known taxa. Proc Natl Acad Sci U S A. 2015 Jun 23;112(25):7761-6. doi: 10.1073/pnas.1502350112. Epub 2015 Jun 8Β β†©οΈŽ

  7. Andre, P., Boneva, T., Chopra, F. et al. Globally representative evidence on the actual and perceived support for climate action. Nat. Clim. Chang., 2024Β β†©οΈŽ

  8. Prevalence and Correlates of Patient Rationing of Insulin in the United States: A National Survey, Adam Gaffney, MD, MPH, David U. Himmelstein, MD, and Steffie Woolhandler, MD, MPH (2022)Β β†©οΈŽ

  9. Improving the prognosis of health care in the USA Galvani, Alison P et al. The Lancet, Volume 395, Issue 10223, 524 - 533Β β†©οΈŽ

  10. Shelter England – Two people died homeless every day last year (2022)Β β†©οΈŽ

  11. United Way NCA – How Many Houses Are in the US? Homelessness vs Housing Availability (2024)Β β†©οΈŽ

  12. Caveat: you should probably still vote to minimize the damage of right-wing policies, but across the world Western β€œdemocracies” are almost universally pro-capital regardless of how you vote.Β β†©οΈŽ

  13. Gilens M, Page BI. Testing Theories of American Politics: Elites, Interest Groups, and Average Citizens. Perspectives on Politics. 2014Β β†©οΈŽ

The British Airways position on various border disputes

5 May 2025 at 00:00

My spouse and I are on vacation in Japan, spending half our time seeing the sights and the other half working remotely and enjoying the experience of living in a different place for a while. To get here, we flew on British Airways from London to Tokyo, and I entertained myself on the long flight by browsing the interactive flight map on the back of my neighbor’s seat and trying to figure out how the poor developer who implemented this map solved the thorny problems that displaying a world map implies.

I began my survey by poking through the whole interface of this little in-seat entertainment system1 to see if I can find out anything about who made it or how it works – I was particularly curious to find a screen listing open source licenses that such such devices often disclose. To my dismay I found nothing at all – no information about who made it or what’s inside. I imagine that there must be some open source software in that thing, but I didn’t find any licenses or copyright statements.

When I turned my attention to the map itself, I did find one copyright statement, the only one I could find in the whole UI. If you zoom in enough, it switches from a satellite view to a street view showing the OpenStreetMap copyright line:

Picture of the display showing β€œStreet Maps: (c) OpenStreetMap contributors”

Note that all of the pictures in this article were taken by pointing my smartphone camera at the screen from an awkward angle and fine-tune your expectations accordingly. I don't have pictures to support every border claim documented in this article, but I did take notes during the flight.

Given that British Airways is the proud flag carrier of the United Kingdom I assume that this is indeed the only off-the-shelf copyrighted material included in this display, and everything else was developed in-house without relying on any open source software that might require a disclosure of license and copyright details. For similar reasons I am going to assume that all of the borders shown in this map are reflective of the official opinion of British Airways on various international disputes.

As I briefly mentioned a moment ago, this map has two views: satellite photography and a very basic street view. Your plane and its route are shown in real-time, and you can touch the screen to pan and zoom the map anywhere you like. You can also rotate the map and change the angle in β€œ3D” if you have enough patience to use complex multitouch gestures on the cheapest touch panel they could find.

The street view is very sparse and only appears when you’re pretty far zoomed in, so it was mostly useless for this investigation. The satellite map, thankfully, includes labels: cities, country names, points of interest, and, importantly, national borders. The latter are very faint, however. Here’s an illustrative example:

A picture of the screen showing the area near the Caucasus mountains with the plane overflying the Caspian sea

We also have our first peek at a border dispute here: look closely between the β€œGeorgia” and β€œCaucasus Mountains” labels. This ever-so-faint dotted line shows what I believe is the Russian-occupied territory of South Ossetia in Georgia. Disputes implicating Russia are not universally denoted as such – I took a peek at the border with Ukraine and found that Ukraine is shown as whole and undisputed, with its (undotted) border showing Donetsk, Luhansk, and Crimea entirely within Ukraine’s borders.

Of course, I didn’t start at Russian border disputes when I went looking for trouble. I went directly to Palestine. Or rather, I went to Israel, because Palestine doesn’t exist on this map:

Picture of the screen showing Israel

I squinted and looked very closely at the screen and I’m fairly certain that both the West Bank and Gaza are outlined in these dotted lines using the borders defined by the 1949 armistice. If you zoom in a bit more to the street view, you can see labels like β€œWest Bank” and the β€œArea A”, β€œArea B” labels of the Oslo Accords:

Picture of the street map zoomed in on Ramallah

Given that this is British Airways, part of me was surprised not to see the whole area simply labelled Mandatory Palestine, but it is interesting to know that British Airways officially supports the Oslo Accords.

Heading south, let’s take a look at the situation in Sudan:

Picture of the satellite map over Sudan

This one is interesting – three areas within South Sudan’s claimed borders are disputed, and the map only shows two with these dotted lines. The border dispute with Sudan in the northeast is resolved in South Sudan’s favor. Another case where BA takes a stand is Guyana, which has an ongoing dispute with Venezuela – but the map only shows Guyana’s claim, albeit with a dotted line, rather than the usual approach of drawing both claims with dotted lines.

Next, I turned my attention to Taiwan:

Picture of the satellite map over eastern China and Taiwan

The cities of Taipei and Kaohsiung are labelled, but the island as a whole was not labelled β€œTaiwan”. I zoomed and panned and 3D-zoomed the map all over the place but was unable to get a β€œTaiwan” label to appear. I also zoomed into the OSM-provided street map and panned that around but couldn’t find β€œTaiwan” anywhere, either.

The last picture I took is of the Kashmir area:

Picture of the satellite map showing the Kashmir region

I find these faint borders difficult to interpret and I admit to not being very familiar with this conflict, but perhaps someone in the know with the patience to look more closely will email me their understanding of the official British Airways position on the Kashmir conflict (here’s the full sized picture).

Here are some other details I noted as I browsed the map:

  • The Hala’ib Triangle and Bir Tawil are shown with dotted lines
  • The Gulf of Mexico is labelled as such
  • Antarctica has no labelled borders or settlements

After this thrilling survey of the official political positions of British Airways, I spent the rest of the flight reading books or trying to sleep.


  1. I believe the industry term is β€œinfotainment system”, but if you ever catch me saying that with a straight face then I have been replaced with an imposter and you should contact the authorities.Β β†©οΈŽ

You can cheat a test suite with a big enough polynomial

24 June 2025 at 16:27

Hi nerds, I'm back from Systems Distributed! I'd heartily recommend it, wildest conference I've been to in years. I have a lot of work to catch up on, so this will be a short newsletter.

In an earlier version of my talk, I had a gag about unit tests. First I showed the test f([1,2,3]) == 3, then said that this was satisfied by f(l) = 3, f(l) = l[-1], f(l) = len(l), f(l) = (129*l[0]-34*l[1]-617)*l[2] - 443*l[0] + 1148*l[1] - 182. Then I progressively rule them out one by one with more unit tests, except the last polynomial which stubbornly passes every single test.

If you're given some function of f(x: int, y: int, …): int and a set of unit tests asserting specific inputs give specific outputs, then you can find a polynomial that passes every single unit test.

To find the gag, and as SMT practice, I wrote a Python program that finds a polynomial that passes a test suite meant for max. It's hardcoded for three parameters and only finds 2nd-order polynomials but I think it could be generalized with enough effort.

The code

Full code here, breakdown below.

from z3 import *  # type: ignore
s1, s2 = Solver(), Solver()

Z3 is just the particular SMT solver we use, as it has good language bindings and a lot of affordances.

As part of learning SMT I wanted to do this two ways. First by putting the polynomial "outside" of the SMT solver in a python function, second by doing it "natively" in Z3. I created two solvers so I could test both versions in one run.

a0, a, b, c, d, e, f = Consts('a0 a b c d e f', IntSort())
x, y, z = Ints('x y z')
t = "a*x+b*y+c*z+d*x*y+e*x*z+f*y*z+a0"

Both Const('x', IntSort()) and Int('x') do the exact same thing, the latter being syntactic sugar for the former. I did not know this when I wrote the program.

To keep the two versions in sync I represented the equation as a string, which I later eval. This is one of the rare cases where eval is a good idea, to help us experiment more quickly while learning. The polynomial is a "2nd-order polynomial", even though it doesn't have x^2 terms, as it has xy and xz terms.

lambdamax = lambda x, y, z: eval(t)

z3max = Function('z3max', IntSort(), IntSort(), IntSort(),  IntSort())
s1.add(ForAll([x, y, z], z3max(x, y, z) == eval(t)))

lambdamax is pretty straightforward: create a lambda with three parameters and eval the string. The string "a*x" then becomes the python expression a*x, a is an SMT symbol, while the x SMT symbol is shadowed by the lambda parameter. To reiterate, a terrible idea in practice, but a good way to learn faster.

z3max function is a little more complex. Function takes an identifier string and N "sorts" (roughly the same as programming types). The first N-1 sorts define the parameters of the function, while the last becomes the output. So here I assign the string identifier "z3max" to be a function with signature (int, int, int) -> int.

I can load the function into the model by specifying constraints on what z3max could be. This could either be a strict input/output, as will be done later, or a ForAll over all possible inputs. Here I just use that directly to say "for all inputs, the function should match this polynomial." But I could do more complicated constraints, like commutativity (f(x, y) == f(y, x)) or monotonicity (Implies(x < y, f(x) <= f(y))).

Note ForAll takes a list of z3 symbols to quantify over. That's the only reason we need to define x, y, z in the first place. The lambda version doesn't need them.

inputs = [(1,2,3), (4, 2, 2), (1, 1, 1), (3, 5, 4)]

for g in inputs:
    s1.add(z3max(*g) == max(*g))
    s2.add(lambdamax(*g) == max(*g))

This sets up the joke: adding constraints to each solver that the polynomial it finds must, for a fixed list of triplets, return the max of each triplet.

for s, func in [(s1, z3max), (s2, lambdamax)]:
    if s.check() == sat:
        m = s.model()
        for x, y, z in inputs:
            print(f"max([{x}, {y}, {z}]) =", m.evaluate(func(x, y, z)))
        print(f"max([x, y, z]) = {m[a]}x + {m[b]}y",
            f"+ {m[c]}z +", # linebreaks added for newsletter rendering
            f"{m[d]}xy + {m[e]}xz + {m[f]}yz + {m[a0]}\n")

Output:

max([1, 2, 3]) = 3
# etc
max([x, y, z]) = -133x + 130y + -10z + -2xy + 62xz + -46yz + 0

max([1, 2, 3]) = 3
# etc
max([x, y, z]) = -17x + 16y + 0z + 0xy + 8xz + -6yz + 0

I find that z3max (top) consistently finds larger coefficients than lambdamax does. I don't know why.

Practical Applications

Test-Driven Development recommends a strict "red-green refactor" cycle. Write a new failing test, make the new test pass, then go back and refactor. Well, the easiest way to make the new test pass would be to paste in a new polynomial, so that's what you should be doing. You can even do this all automatically: have a script read the set of test cases, pass them to the solver, and write the new polynomial to your code file. All you need to do is write the tests!

Pedagogical Notes

Writing the script took me a couple of hours. I'm sure an LLM could have whipped it all up in five minutes but I really want to learn SMT and LLMs may decrease learning retention.1 Z3 documentation is not... great for non-academics, though, and most other SMT solvers have even worse docs. One useful trick I use regularly is to use Github code search to find code using the same APIs and study how that works. Turns out reading API-heavy code is a lot easier than writing it!

Anyway, I'm very, very slowly feeling like I'm getting the basics on how to use SMT. I don't have any practical use cases yet, but I wanted to learn this skill for a while and glad I finally did.


  1. Caveat I have not actually read the study, for all I know it could have a sample size of three people, I'll get around to it eventually ↩

Solving LinkedIn Queens with SMT

12 June 2025 at 15:43

No newsletter next week

I’ll be speaking at Systems Distributed. My talk isn't close to done yet, which is why this newsletter is both late and short.

Solving LinkedIn Queens in SMT

The article Modern SAT solvers: fast, neat and underused claims that SAT solvers1 are "criminally underused by the industry". A while back on the newsletter I asked "why": how come they're so powerful and yet nobody uses them? Many experts responded saying the reason is that encoding SAT kinda sucked and they rather prefer using tools that compile to SAT.

I was reminded of this when I read Ryan Berger's post on solving β€œLinkedIn Queens” as a SAT problem.

A quick overview of Queens. You’re presented with an NxN grid divided into N regions, and have to place N queens so that there is exactly one queen in each row, column, and region. While queens can be on the same diagonal, they cannot be adjacently diagonal.

(Important note: Linkedin β€œQueens” is a variation on the puzzle game Star Battle, which is the same except the number of stars you place in each row/column/region varies per puzzle, and is usually two. This is also why 'queens' don’t capture like chess queens.)

An image of a solved queens board. Copied from https://ryanberger.me/posts/queens

Ryan solved this by writing Queens as a SAT problem, expressing properties like "there is exactly one queen in row 3" as a large number of boolean clauses. Go read his post, it's pretty cool. What leapt out to me was that he used CVC5, an SMT solver.2 SMT solvers are "higher-level" than SAT, capable of handling more data types than just boolean variables. It's a lot easier to solve the problem at the SMT level than at the SAT level. To show this, I whipped up a short demo of solving the same problem in Z3 (via the Python API).

Full code here, which you can compare to Ryan's SAT solution here. I didn't do a whole lot of cleanup on it (again, time crunch!), but short explanation below.

The code

from z3 import * # type: ignore
from itertools import combinations, chain, product
solver = Solver()
size = 9 # N

Initial setup and modules. size is the number of rows/columns/regions in the board, which I'll call N below.

# queens[n] = col of queen on row n
# by construction, not on same row
queens = IntVector('q', size) 

SAT represents the queen positions via NΒ² booleans: q_00 means that a Queen is on row 0 and column 0, !q_05 means a queen isn't on row 0 col 5, etc. In SMT we can instead encode it as N integers: q_0 = 5 means that the queen on row 0 is positioned at column 5. This immediately enforces one class of constraints for us: we don't need any constraints saying "exactly one queen per row", because that's embedded in the definition of queens!

(Incidentally, using 0-based indexing for the board was a mistake on my part, it makes correctly encoding the regions later really painful.)

To actually make the variables [q_0, q_1, …], we use the Z3 affordance IntVector(str, n) for making n variables at once.

solver.add([And(0 <= i, i < size) for i in queens])
# not on same column
solver.add(Distinct(queens))

First we constrain all the integers to [0, N), then use the incredibly handy Distinct constraint to force all the integers to have different values. This guarantees at most one queen per column, which by the pigeonhole principle means there is exactly one queen per column.

# not diagonally adjacent
for i in range(size-1):
    q1, q2 = queens[i], queens[i+1]
    solver.add(Abs(q1 - q2) != 1)

One of the rules is that queens can't be adjacent. We already know that they can't be horizontally or vertically adjacent via other constraints, which leaves the diagonals. We only need to add constraints that, for each queen, there is no queen in the lower-left or lower-right corner, aka q_3 != q_2 Β± 1. We don't need to check the top corners because if q_1 is in the upper-left corner of q_2, then q_2 is in the lower-right corner of q_1!

That covers everything except the "one queen per region" constraint. But the regions are the tricky part, which we should expect because we vary the difficulty of queens games by varying the regions.

regions = {
        "purple": [(0, 0), (0, 1), (0, 2), (0, 3), (0, 4), (0, 5), (0, 6), (0, 7), (0, 8),
                   (1, 0), (2, 0), (3, 0), (4, 0), (5, 0), (6, 0), (7, 0), (8, 0),
                   (1, 1), (8, 1)],
        "red": [(1, 2), (2, 2), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (6, 2), (7, 1), (7, 2), (8, 2), (8, 3),],
        # you get the picture
        }

# Some checking code left out, see below

The region has to be manually coded in, which is a huge pain.

(In the link, some validation code follows. Since it breaks up explaining the model I put it in the next section.)

for r in regions.values():
    solver.add(Or(
        *[queens[row] == col for (row, col) in r]
        ))

Finally we have the region constraint. The easiest way I found to say "there is exactly one queen in each region" is to say "there is a queen in region 1 and a queen in region 2 and a queen in region 3" etc." Then to say "there is a queen in region purple" I wrote "q_0 = 0 OR q_0 = 1 OR … OR q_1 = 0 etc."

Why iterate over every position in the region instead of doing something like (0, q[0]) in r? I tried that but it's not an expression that Z3 supports.

if solver.check() == sat:
    m = solver.model()
    print([(l, m[l]) for l in queens])

Finally, we solve and print the positions. Running this gives me:

[(q__0, 0), (q__1, 5), (q__2, 8), 
 (q__3, 2), (q__4, 7), (q__5, 4), 
 (q__6, 1), (q__7, 3), (q__8, 6)]

Which is the correct solution to the queens puzzle. I didn't benchmark the solution times, but I imagine it's considerably slower than a raw SAT solver. Glucose is really, really fast.

But even so, solving the problem with SMT was a lot easier than solving it with SAT. That satisfies me as an explanation for why people prefer it to SAT.

Sanity checks

One bit I glossed over earlier was the sanity checking code. I knew for sure that I was going to make a mistake encoding the region, and the solver wasn't going to provide useful information abut what I did wrong. In cases like these, I like adding small tests and checks to catch mistakes early, because the solver certainly isn't going to catch them!

all_squares = set(product(range(size), repeat=2))
def test_i_set_up_problem_right():
    assert all_squares == set(chain.from_iterable(regions.values()))

    for r1, r2 in combinations(regions.values(), 2):
        assert not set(r1) & set(r2), set(r1) & set(r2)

The first check was a quick test that I didn't leave any squares out, or accidentally put the same square in both regions. Converting the values into sets makes both checks a lot easier. Honestly I don't know why I didn't just use sets from the start, sets are great.

def render_regions():
    colormap = ["purple",  "red", "brown", "white", "green", "yellow", "orange", "blue", "pink"]
    board = [[0 for _ in range(size)] for _ in range(size)] 
    for (row, col) in all_squares:
        for color, region in regions.items():
            if (row, col) in region:
                board[row][col] = colormap.index(color)+1

    for row in board:
        print("".join(map(str, row)))

render_regions()

The second check is something that prints out the regions. It produces something like this:

111111111
112333999
122439999
124437799
124666779
124467799
122467899
122555889
112258899

I can compare this to the picture of the board to make sure I got it right. I guess a more advanced solution would be to print emoji squares like πŸŸ₯ instead.

Neither check is quality code but it's throwaway and it gets the job done so eh.

Update for the Internet

This was sent as a weekly newsletter, which is usually on topics like software history, formal methods, unusual technologies, and the theory of software engineering. You can subscribe here.


  1. "Boolean SATisfiability Solver", aka a solver that can find assignments that make complex boolean expressions true. I write a bit more about them here. ↩

  2. "Satisfiability Modulo Theories" ↩

AI is a gamechanger for TLA+ users

5 June 2025 at 14:59

New Logic for Programmers Release

v0.10 is now available! This is a minor release, mostly focused on logic-based refactoring, with new material on set types and testing refactors are correct. See the full release notes at the changelog page. Due to conference pressure v0.11 will also likely be a minor release.

The book cover

AI is a gamechanger for TLA+ users

TLA+ is a specification language to model and debug distributed systems. While very powerful, it's also hard for programmers to learn, and there's always questions of connecting specifications with actual code.

That's why The Coming AI Revolution in Distributed Systems caught my interest. In the post, Cheng Huang claims that Azure successfully used LLMs to examine an existing codebase, derive a TLA+ spec, and find a production bug in that spec. "After a decade of manually crafting TLA+ specifications", he wrote, "I must acknowledge that this AI-generated specification rivals human work".

This inspired me to experiment with LLMs in TLA+ myself. My goals are a little less ambitious than Cheng's: I wanted to see how LLMs could help junior specifiers write TLA+, rather than handling the entire spec automatically. Details on what did and didn't work below, but my takeaway is that LLMs are an immense specification force multiplier.

All tests were done with a standard VSCode Copilot subscription, writing Claude 3.7 in Agent mode. Other LLMs or IDEs may be more or less effective, etc.

Things Claude was good at

Fixing syntax errors

TLA+ uses a very different syntax than mainstream programming languages, meaning beginners make a lot of mistakes where they do a "programming syntax" instead of TLA+ syntax:

NotThree(x) = \* should be ==, not =
    x != 3 \* should be #, not !=

The problem is that the TLA+ syntax checker, SANY, is 30 years old and doesn't provide good information. Here's what it says for that snippet:

Was expecting "==== or more Module body"
Encountered "NotThree" at line 6, column 1

That only isolates one error and doesn't tell us what the problem is, only where it is. Experienced TLA+ users get "error eyes" and can quickly see what the problem is, but beginners really struggle with this.

The TLA+ foundation has made LLM integration a priority, so the VSCode extension naturally supports several agents actions. One of these is running SANY, meaning an agent can get an error, fix it, get another error, fix it, etc. Provided the above sample and asked to make it work, Claude successfully fixed both errors. It also fixed many errors in a larger spec, as well as figure out why PlusCal specs weren't compiling to TLA+.

This by itself is already enough to make LLMs a worthwhile tool, as it fixes one of the biggest barriers to entry.

Understanding error traces

When TLA+ finds a violated property, it outputs the sequence of steps that leads to the error. This starts in plaintext, and VSCode parses it into an interactive table:

An example error trace

Learning to read these error traces is a skill in itself. You have to understand what's happening in each step and how it relates back to the actually broken property. It takes a long time for people to learn how to do this well.

Claude was successful here, too, accurately reading 20+ step error traces and giving a high-level explanation of what went wrong. It also could condense error traces: if ten steps of the error trace could be condensed into a one-sentence summary (which can happen if you're modeling a lot of process internals) Claude would do it.

I did have issues here with doing this in agent mode: while the extension does provide a "run model checker" command, the agent would regularly ignore this and prefer to run a terminal command instead. This would be fine except that the LLM consistently hallucinated invalid commands. I had to amend every prompt with "run the model checker via vscode, do not use a terminal command". You can skip this if you're willing to copy and paste the error trace into the prompt.

As with syntax checking, if this was the only thing LLMs could effectively do, that would already be enough1 to earn a strong recommend. Even as a TLA+ expert I expect I'll be using this trick regularly.

Boilerplate tasks

TLA+ has a lot of boilerplate. One of the most notorious examples is UNCHANGED rules. Specifications are extremely precise β€” so precise that you have to specify what variables don't change in every step. This takes the form of an UNCHANGED clause at the end of relevant actions:

RemoveObjectFromStore(srv, o, s) ==
  /\ o \in stored[s]
  /\ stored' = [stored EXCEPT ![s] = @ \ {o}]
  /\ UNCHANGED <<capacity, log, objectsize, pc>>

Writing this is really annoying. Updating these whenever you change an action, or add a new variable to the spec, is doubly so. Syntax checking and error analysis are important for beginners, but this is what I wanted for myself. I took a spec and prompted Claude

Add UNCHANGED <> for each variable not changed in an action.

And it worked! It successfully updated the UNCHANGED in every action.

(Note, though, that it was a "well-behaved" spec in this regard: only one "action" happened at a time. In TLA+ you can have two actions happen simultaneously, that each update half of the variables, meaning neither of them should have an UNCHANGED clause. I haven't tested how Claude handles that!)

That's the most obvious win, but Claude was good at handling other tedious work, too. Some examples include updating vars (the conventional collection of all state variables), lifting a hard-coded value into a model parameter, and changing data formats. Most impressive to me, though, was rewriting a spec designed for one process to instead handle multiple processes. This means taking all of the process variables, which originally have types like Int, converting them to types like [Process -> Int], and then updating the uses of all of those variables in the spec. It didn't account for race conditions in the new concurrent behavior, but it was an excellent scaffold to do more work.

Writing properties from an informal description

You have to be pretty precise with your intended property description but it handles converting that precise description into TLA+'s formalized syntax, which is something beginners often struggle with.

Things it is less good at

Generating model config files

To model check TLA+, you need both a specification (.tla) and a model config file (.cfg), which have separate syntaxes. Asking the agent to generate the second often lead to it using TLA+ syntax. It automatically fixed this after getting parsing errors, though.

Fixing specs

Whenever the ran model checking and discovered a bug, it would naturally propose a change to either the invalid property or the spec. Sometimes the changes were good, other times the changes were not physically realizable. For example, if it found that a bug was due to a race condition between processes, it would often suggest fixing it by saying race conditions were okay. I mean yes, if you say bugs are okay, then the spec finds that bugs are okay! Or it would alternatively suggest adding a constraint to the spec saying that race conditions don't happen. But that's a huge mistake in specification, because race conditions happen if we don't have coordination. We need to specify the mechanism that is supposed to prevent them.

Finding properties of the spec

After seeing how capable it was at translating my properties to TLA+, I started prompting Claude to come up with properties on its own. Unfortunately, almost everything I got back was either trivial, uninteresting, or too coupled to implementation details. I haven't tested if it would work better to ask it for "properties that may be violated".

Generating code from specs

I have to be specific here: Claude could sometimes convert Python into a passable spec, an vice versa. It wasn't good at recognizing abstraction. For example, TLA+ specifications often represent sequential operations with a state variable, commonly called pc. If modeling code that nonatomically retrieves a counter value and increments it, we'd have one action that requires pc = "Get" and sets the new value to "Inc", then another that requires it be "Inc" and sets it to "Done".

I found that Claude would try to somehow convert pc into part of the Python program's state, rather than recognize it as a TLA+ abstraction. On the other side, when converting python code to TLA+ it would often try to translate things like sleep into some part of the spec, not recognizing that it is abstractable into a distinct action. I didn't test other possible misconceptions, like converting randomness to nondeterminism.

For the record, when converting TLA+ to Python Claude tended to make simulators of the spec, rather than possible production code implementing the spec. I really wasn't expecting otherwise though.

Unexplored Applications

Things I haven't explored thoroughly but could possibly be effective, based on what I know about TLA+ and AI:

Writing Java Overrides

Most TLA+ operators are resolved via TLA+ interpreters, but you can also implement them in "native" Java. This lets you escape the standard language semantics and add capabilities like executing programs during model-checking or dynamically constrain the depth of the searched state space. There's a lot of cool things I think would be possible with overrides. The problem is there's only a handful of people in the world who know how to write them. But that handful have written quite a few overrides and I think there's enough there for Claude to work with.

Writing specs, given a reference mechanism

In all my experiments, the LLM only had my prompts and the occasional Python script as information. That makes me suspect that some of its problems with writing and fixing specs come down to not having a system model. Maybe it wouldn't suggest fixes like "these processes never race" if it had a design doc saying that the processes can't coordinate.

(Could a Sufficiently Powerful LLM derive some TLA+ specification from a design document?)

Connecting specs and code

This is the holy grail of TLA+: taking a codebase and showing it correctly implements a spec. Currently the best ways to do this are by either using TLA+ to generate a test suite, or by taking logged production traces and matching them to TLA+ behaviors. This blog post discusses both. While I've seen a lot of academic research into these approaches there are no industry-ready tools. So if you want trace validation you have to do a lot of manual labour tailored to your specific product.

If LLMs could do some of this work for us then that'd really amplify the usefulness of TLA+ to many companies.

Thoughts

Right now, agents seem good at the tedious and routine parts of TLA+ and worse at the strategic and abstraction parts. But, since the routine parts are often a huge barrier to beginners, this means that LLMs have the potential to make TLA+ far, far more accessible than it previously was.

I have mixed thoughts on this. As an advocate, this is incredible. I want more people using formal specifications because I believe it leads to cheaper, safer, more reliable software. Anything that gets people comfortable with specs is great for our industry. As a professional TLA+ consultant, I'm worried that this obsoletes me. Most of my income comes from training and coaching, which companies will have far less demand of now. Then again, maybe this an opportunity to pitch "agentic TLA+ training" to companies!

Anyway, if you're interested in TLA+, there has never been a better time to try it. I mean it, these tools handle so much of the hard part now. I've got a free book available online, as does the inventor of TLA+. I like this guide too. Happy modeling!


  1. Dayenu. ↩

What does "Undecidable" mean, anyway

28 May 2025 at 19:34

Systems Distributed

I'll be speaking at Systems Distributed next month! The talk is brand new and will aim to showcase some of the formal methods mental models that would be useful in mainstream software development. It has added some extra stress on my schedule, though, so expect the next two monthly releases of Logic for Programmers to be mostly minor changes.

What does "Undecidable" mean, anyway

Last week I read Against Curry-Howard Mysticism, which is a solid article I recommend reading. But this newsletter is actually about one comment:

I like to see posts like this because I often feel like I can’t tell the difference between BS and a point I’m missing. Can we get one for questions like β€œIsn’t XYZ (Undecidable|NP-Complete|PSPACE-Complete)?”

I've already written one of these for NP-complete, so let's do one for "undecidable". Step one is to pull a technical definition from the book Automata and Computability:

A property P of strings is said to be decidable if ... there is a total Turing machine that accepts input strings that have property P and rejects those that do not. (pg 220)

Step two is to translate the technical computer science definition into more conventional programmer terms. Warning, because this is a newsletter and not a blog post, I might be a little sloppy with terms.

Machines and Decision Problems

In automata theory, all inputs to a "program" are strings of characters, and all outputs are "true" or "false". A program "accepts" a string if it outputs "true", and "rejects" if it outputs "false". You can think of this as automata studying all pure functions of type f :: string -> boolean. Problems solvable by finding such an f are called "decision problems".

This covers more than you'd think, because we can bootstrap more powerful functions from these. First, as anyone who's programmed in bash knows, strings can represent any other data. Second, we can fake non-boolean outputs by instead checking if a certain computation gives a certain result. For example, I can reframe the function add(x, y) = x + y as a decision problem like this:

IS_SUM(str) {
    x, y, z = split(str, "#")
    return x + y == z
}

Then because IS_SUM("2#3#5") returns true, we know 2 + 3 == 5, while IS_SUM("2#3#6") is false. Since we can bootstrap parameters out of strings, I'll just say it's IS_SUM(x, y, z) going forward.

A big part of automata theory is studying different models of computation with different strengths. One of the weakest is called "DFA". I won't go into any details about what DFA actually can do, but the important thing is that it can't solve IS_SUM. That is, if you give me a DFA that takes inputs of form x#y#z, I can always find an input where the DFA returns true when x + y != z, or an input which returns false when x + y == z.

It's really important to keep this model of "solve" in mind: a program solves a problem if it correctly returns true on all true inputs and correctly returns false on all false inputs.

(total) Turing Machines

A Turing Machine (TM) is a particular type of computation model. It's important for two reasons:

  1. By the Church-Turing thesis, a Turing Machine is the "upper bound" of how powerful (physically realizable) computational models can get. This means that if an actual real-world programming language can solve a particular decision problem, so can a TM. Conversely, if the TM can't solve it, neither can the programming language.1

  2. It's possible to write a Turing machine that takes a textual representation of another Turing machine as input, and then simulates that Turing machine as part of its computations.

Property (1) means that we can move between different computational models of equal strength, proving things about one to learn things about another. That's why I'm able to write IS_SUM in a pseudocode instead of writing it in terms of the TM computational model (and why I was able to use split for convenience).

Property (2) does several interesting things. First of all, it makes it possible to compose Turing machines. Here's how I can roughly ask if a given number is the sum of two primes, with "just" addition and boolean functions:

IS_SUM_TWO_PRIMES(z):
    x := 1
    y := 1
    loop {
        if x > z {return false}
        if IS_PRIME(x) {
            if IS_PRIME(y) {
                if IS_SUM(x, y, z) {
                    return true;
                }
            }
        }
        y := y + 1
        if y > x {
            x := x + 1
            y := 0
        }
    }

Notice that without the if x > z {return false}, the program would loop forever on z=2. A TM that always halts for all inputs is called total.

Property (2) also makes "Turing machines" a possible input to functions, meaning that we can now make decision problems about the behavior of Turing machines. For example, "does the TM M either accept or reject x within ten steps?"2

IS_DONE_IN_TEN_STEPS(M, x) {
    for (i = 0; i < 10; i++) {
        `simulate M(x) for one step`
        if(`M accepted or rejected`) {
            return true
        }
    }
    return false
}

Decidability and Undecidability

Now we have all of the pieces to understand our original definition:

A property P of strings is said to be decidable if ... there is a total Turing machine that accepts input strings that have property P and rejects those that do not. (220)

Let IS_P be the decision problem "Does the input satisfy P"? Then IS_P is decidable if it can be solved by a Turing machine, ie, I can provide some IS_P(x) machine that always accepts if x has property P, and always rejects if x doesn't have property P. If I can't do that, then IS_P is undecidable.

IS_SUM(x, y, z) and IS_DONE_IN_TEN_STEPS(M, x) are decidable properties. Is IS_SUM_TWO_PRIMES(z) decidable? Some analysis shows that our corresponding program will either find a solution, or have x>z and return false. So yes, it is decidable.

Notice there's an asymmetry here. To prove some property is decidable, I need just to need to find one program that correctly solves it. To prove some property is undecidable, I need to show that any possible program, no matter what it is, doesn't solve it.

So with that asymmetry in mind, do are there any undecidable problems? Yes, quite a lot. Recall that Turing machines can accept encodings of other TMs as input, meaning we can write a TM that checks properties of Turing machines. And, by Rice's Theorem, almost every nontrivial semantic3 property of Turing machines is undecidable. The conventional way to prove this is to first find a single undecidable property H, and then use that to bootstrap undecidability of other properties.

The canonical and most famous example of an undecidable problem is the Halting problem: "does machine M halt on input i?" It's pretty easy to prove undecidable, and easy to use it to bootstrap other undecidability properties. But again, any nontrivial property is undecidable. Checking a TM is total is undecidable. Checking a TM accepts any inputs is undecidable. Checking a TM solves IS_SUM is undecidable. Etc etc etc.

What this doesn't mean in practice

I often see the halting problem misconstrued as "it's impossible to tell if a program will halt before running it." This is wrong. The halting problem says that we cannot create an algorithm that, when applied to an arbitrary program, tells us whether the program will halt or not. It is absolutely possible to tell if many programs will halt or not. It's possible to find entire subcategories of programs that are guaranteed to halt. It's possible to say "a program constructed following constraints XYZ is guaranteed to halt."

The actual consequence of undecidability is more subtle. If we want to know if a program has property P, undecidability tells us

  1. We will have to spend time and mental effort to determine if it has P
  2. We may not be successful.

This is subtle because we're so used to living in a world where everything's undecidable that we don't really consider what the counterfactual would be like. In such a world there might be no need for Rust, because "does this C program guarantee memory-safety" is a decidable property. The entire field of formal verification could be unnecessary, as we could just check properties of arbitrary programs directly. We could automatically check if a change in a program preserves all existing behavior. Lots of famous math problems could be solved overnight.

(This to me is a strong "intuitive" argument for why the halting problem is undecidable: a halt detector can be trivially repurposed as a program optimizer / theorem-prover / bcrypt cracker / chess engine. It's too powerful, so we should expect it to be impossible.)

But because we don't live in that world, all of those things are hard problems that take effort and ingenuity to solve, and even then we often fail.

Update for the Internet

This was sent as a weekly newsletter, which is usually on topics like software history, formal methods, unusual technologies, and the theory of software engineering. You can subscribe here.


  1. To be pendantic, a TM can't do things like "scrape a webpage" or "render a bitmap", but we're only talking about computational decision problems here. ↩

  2. One notation I've adopted in Logic for Programmers is marking abstract sections of pseudocode with backticks. It's really handy! ↩

  3. Nontrivial meaning "at least one TM has this property and at least one TM doesn't have this property". Semantic meaning "related to whether the TM accepts, rejects, or runs forever on a class of inputs". IS_DONE_IN_TEN_STEPS is not a semantic property, as it doesn't tell us anything about inputs that take longer than ten steps. ↩

Finding hard 24 puzzles with planner programming

20 May 2025 at 18:21

Planner programming is a programming technique where you solve problems by providing a goal and actions, and letting the planner find actions that reach the goal. In a previous edition of Logic for Programmers, I demonstrated how this worked by solving the 24 puzzle with planning. For reasons discussed here I replaced that example with something more practical (orchestrating deployments), but left the code online for posterity.

Recently I saw a family member try and fail to vibe code a tool that would find all valid 24 puzzles, and realized I could adapt the puzzle solver to also be a puzzle generator. First I'll explain the puzzle rules, then the original solver, then the generator.1 For a much longer intro to planning, see here.

The rules of 24

You're given four numbers and have to find some elementary equation (+-*/+groupings) that uses all four numbers and results in 24. Each number must be used exactly once, but do not need to be used in the starting puzzle order. Some examples:

  • [6, 6, 6, 6] -> 6+6+6+6=24
  • [1, 1, 6, 6] -> (6+6)*(1+1)=24
  • [4, 4, 4, 5] -> 4*(5+4/4)=24

Some setups are impossible, like [1, 1, 1, 1]. Others are possible only with non-elementary operations, like [1, 5, 5, 324] (which requires exponentiation).

The solver

We will use the Picat, the only language that I know has a built-in planner module. The current state of our plan with be represented by a single list with all of the numbers.

import planner, math.
import cp.

action(S0, S1, Action, Cost) ?=>
  member(X, S0)
  , S0 := delete(S0, X) % , is `and`
  , member(Y, S0)
  , S0 := delete(S0, Y)
  , (
      A = $(X + Y) 
    ; A = $(X - Y)
    ; A = $(X * Y)
    ; A = $(X / Y), Y > 0
    )
    , S1 = S0 ++ [apply(A)]
  , Action = A
  , Cost = 1
  .

This is our "action", and it works in three steps:

  1. Nondeterministically pull two different values out of the input, deleting them
  2. Nondeterministically pick one of the basic operations
  3. The new state is the remaining elements, appended with that operation applied to our two picks.

Let's walk through this with [1, 6, 1, 7]. There are four choices for X and three four Y. If the planner chooses X=6 and Y=7, A = $(6 + 7). This is an uncomputed term in the same way lisps might use quotation. We can resolve the computation with apply, as in the line S1 = S0 ++ [apply(A)].

final([N]) =>
  N =:= 24. % handle floating point

Our final goal is just a list where the only element is 24. This has to be a little floating point-sensitive to handle floating point divison, done by =:=.

main =>
  Start = [1, 5, 5, 6]
  , best_plan(Start, 4, Plan)
  , printf("%w %w%n", Start, Plan)
  .

For main, we just find the best plan with the maximum cost of 4 and print it. When run from the command line, picat automatically executes whatever is in main.

$ picat 24.pi
[1,5,5,6] [1 + 5,5 * 6,30 - 6]

I don't want to spoil any more 24 puzzles, so let's stop showing the plan:

main =>
- , printf("%w %w%n", Start, Plan)
+ , printf("%w%n", Start)

Generating puzzles

Picat provides a find_all(X, p(X)) function, which ruturns all X for which p(X) is true. In theory, we could write find_all(S, best_plan(S, 4, _). In practice, there are an infinite number of valid puzzles, so we need to bound S somewhat. We also don't want to find any redundant puzzles, such as [6, 6, 6, 4] and [4, 6, 6, 6].

We can solve both issues by writing a helper valid24(S), which will check that S a sorted list of integers within some bounds, like 1..8, and also has a valid solution.

valid24(Start) =>
  Start = new_list(4)
  , Start :: 1..8 % every value in 1..8
  , increasing(Start) % sorted ascending
  , solve(Start) % turn into values
  , best_plan(Start, 4, Plan)
  .

This leans on Picat's constraint solving features to automatically find bounded sorted lists, which is why we need the solve step.2 Now we can just loop through all of the values in find_all to get all solutions:

main =>
  foreach([S] in find_all(
    [Start],
    valid24(Start)))
    printf("%w%n", S)
  end.
$ picat 24.pi

[1,1,1,8]
[1,1,2,6]
[1,1,2,7]
[1,1,2,8]
# etc

Finding hard puzzles

Last Friday I realized I could do something more interesting with this. Once I have found a plan, I can apply further constraints to the plan, for example to find problems that can be solved with division:

valid24(Start, Plan) =>
  Start = new_list(4)
  , Start :: 1..8
  , increasing(Start)
  , solve(Start)
  , best_plan(Start, 4, Plan)
+ , member($(_ / _), Plan)
  .

In playing with this, though, I noticed something weird: there are some solutions that appear if I sort up but not down. For example, [3,3,4,5] appears in the solution set, but [5, 4, 3, 3] doesn't appear if I replace increasing with decreasing.

As far as I can tell, this is because Picat only finds one best plan, and [5, 4, 3, 3] has two solutions: 4*(5-3/3) and 3*(5+4)-3. best_plan is a deterministic operator, so Picat commits to the first best plan it finds. So if it finds 3*(5+4)-3 first, it sees that the solution doesn't contain a division, throws [5, 4, 3, 3] away as a candidate, and moves on to the next puzzle.

There's a couple ways we can fix this. We could replace best_plan with best_plan_nondet, which can backtrack to find new plans (at the cost of an enormous number of duplicates). Or we could modify our final to only accept plans with a division:

% Hypothetical change
final([N]) =>
+ member($(_ / _), current_plan()),
  N =:= 24.

My favorite "fix" is to ask another question entirely. While I was looking for puzzles that can be solved with division, what I actually want is puzzles that must be solved with division. What if I rejected any puzzle that has a solution without division?

+ plan_with_no_div(S, P) => best_plan_nondet(S, 4, P), not member($(_ / _), P).

valid24(Start, Plan) =>
  Start = new_list(4)
  , Start :: 1..8
  , increasing(Start)
  , solve(Start)
  , best_plan(Start, 4, Plan)
- , member($(_ / _), Plan)
+ , not plan_with_no_div(Start, _)
  .

The new line's a bit tricky. plan_with_div nondeterministically finds a plan, and then fails if the plan contains a division.3 Since I used best_plan_nondet, it can backtrack from there and find a new plan. This means plan_with_no_div only fails if not such plan exists. And in valid24, we only succeed if plan_with_no_div fails, guaranteeing that the only existing plans use division. Since this doesn't depend on the plan found via best_plan, it doesn't matter how the values in Start are arranged, this will not miss any valid puzzles.

Aside for my logic book readers

The new clause is equivalent to !(some p: Plan(p) && !(div in p)). Applying the simplifications we learned:

  1. !(some p: Plan(p) && !(div in p)) (init)
  2. all p: !(plan(p) && !(div in p)) (all/some duality)
  3. all p: !plan(p) || div in p) (De Morgan's law)
  4. all p: plan(p) => div in p (implication definition)

Which more obviously means "if P is a valid plan, then it contains a division".

Back to finding hard puzzles

Anyway, with not plan_with_no_div, we are filtering puzzles on the set of possible solutions, not just specific solutions. And this gives me an idea: what if we find puzzles that have only one solution?

different_plan(S, P) => best_plan_nondet(S, 4, P2), P2 != P.

valid24(Start, Plan) =>
+ , not different_plan(Start, Plan)

I tried this from 1..8 and got:

[1,2,7,7]
[1,3,4,6]
[1,6,6,8]
[3,3,8,8]

These happen to be some of the hardest 24 puzzles known, though not all of them. Note this is assuming that (X + Y) and (Y + X) are different solutions. If we say they're the same (by appending writing A = $(X + Y), X <= Y in our action) then we got a lot more puzzles, many of which are considered "easy". Other "hard" things we can look for include plans that require fractions:

plan_with_no_fractions(S, P) => 
  best_plan_nondet(S, 4, P)
  , not(
    member(X, P),
    round(apply(X)) =\= X
  ).

% insert `not plan...` in valid24 as usual

Finally, we could try seeing if a negative number is required:

plan_with_no_negatives(S, P) => 
  best_plan_nondet(S, 4, P)
  , not(
    member(X, P),
    apply(X) < 0
  ).

Interestingly this one returns no solutions, so you are never required to construct a negative number as part of a standard 24 puzzle.


  1. The code below is different than old book version, as it uses more fancy logic programming features that aren't good in learning material. ↩

  2. increasing is a constraint predicate. We could alternatively write sorted, which is a Picat logical predicate and must be placed after solve. There doesn't seem to be any efficiency gains either way. ↩

  3. I don't know what the standard is in Picat, but in Prolog, the convention is to use \+ instead of not. They mean the same thing, so I'm using not because it's clearer to non-LPers. ↩

Modeling Awkward Social Situations with TLA+

14 May 2025 at 16:02

You're walking down the street and need to pass someone going the opposite way. You take a step left, but they're thinking the same thing and take a step to their right, aka your left. You're still blocking each other. Then you take a step to the right, and they take a step to their left, and you're back to where you started. I've heard this called "walkwarding"

Let's model this in TLA+. TLA+ is a formal methods tool for finding bugs in complex software designs, most often involving concurrency. Two people trying to get past each other just also happens to be a concurrent system. A gentler introduction to TLA+'s capabilities is here, an in-depth guide teaching the language is here.

The spec

---- MODULE walkward ----
EXTENDS Integers

VARIABLES pos
vars == <<pos>>

Double equals defines a new operator, single equals is an equality check. <<pos>> is a sequence, aka array.

you == "you"
me == "me"
People == {you, me}

MaxPlace == 4

left == 0
right == 1

I've gotten into the habit of assigning string "symbols" to operators so that the compiler complains if I misspelled something. left and right are numbers so we can shift position with right - pos.

direction == [you |-> 1, me |-> -1]
goal == [you |-> MaxPlace, me |-> 1]

Init ==
  \* left-right, forward-backward
  pos = [you |-> [lr |-> left, fb |-> 1], me |-> [lr |-> left, fb |-> MaxPlace]]

direction, goal, and pos are "records", or hash tables with string keys. I can get my left-right position with pos.me.lr or pos["me"]["lr"] (or pos[me].lr, as me == "me").

Juke(person) ==
  pos' = [pos EXCEPT ![person].lr = right - @]

TLA+ breaks the world into a sequence of steps. In each step, pos is the value of pos in the current step and pos' is the value in the next step. The main outcome of this semantics is that we "assign" a new value to pos by declaring pos' equal to something. But the semantics also open up lots of cool tricks, like swapping two values with x' = y /\ y' = x.

TLA+ is a little weird about updating functions. To set f[x] = 3, you gotta write f' = [f EXCEPT ![x] = 3]. To make things a little easier, the rhs of a function update can contain @ for the old value. ![me].lr = right - @ is the same as right - pos[me].lr, so it swaps left and right.

("Juke" comes from here)

Move(person) ==
  LET new_pos == [pos[person] EXCEPT !.fb = @ + direction[person]]
  IN
    /\ pos[person].fb # goal[person]
    /\ \A p \in People: pos[p] # new_pos
    /\ pos' = [pos EXCEPT ![person] = new_pos]

The EXCEPT syntax can be used in regular definitions, too. This lets someone move one step in their goal direction unless they are at the goal or someone is already in that space. /\ means "and".

Next ==
  \E p \in People:
    \/ Move(p)
    \/ Juke(p)

I really like how TLA+ represents concurrency: "In each step, there is a person who either moves or jukes." It can take a few uses to really wrap your head around but it can express extraordinarily complicated distributed systems.

Spec == Init /\ [][Next]_vars

Liveness == <>(pos[me].fb = goal[me])
====

Spec is our specification: we start at Init and take a Next step every step.

Liveness is the generic term for "something good is guaranteed to happen", see here for more. <> means "eventually", so Liveness means "eventually my forward-backward position will be my goal". I could extend it to "both of us eventually reach out goal" but I think this is good enough for a demo.

Checking the spec

Four years ago, everybody in TLA+ used the toolbox. Now the community has collectively shifted over to using the VSCode extension.1 VSCode requires we write a configuration file, which I will call walkward.cfg.

SPECIFICATION Spec
PROPERTY Liveness

I then check the model with the VSCode command TLA+: Check model with TLC. Unsurprisingly, it finds an error:

Screenshot 2025-05-12 153537.png

The reason it fails is "stuttering": I can get one step away from my goal and then just stop moving forever. We say the spec is unfair: it does not guarantee that if progress is always possible, progress will be made. If I want the spec to always make progress, I have to make some of the steps weakly fair.

+ Fairness == WF_vars(Next)

- Spec == Init /\ [][Next]_vars
+ Spec == Init /\ [][Next]_vars /\ Fairness

Now the spec is weakly fair, so someone will always do something. New error:

\* First six steps cut
7: <Move("me")>
pos = [you |-> [lr |-> 0, fb |-> 4], me |-> [lr |-> 1, fb |-> 2]]
8: <Juke("me")>
pos = [you |-> [lr |-> 0, fb |-> 4], me |-> [lr |-> 0, fb |-> 2]]
9: <Juke("me")> (back to state 7)

In this failure, I've successfully gotten past you, and then spend the rest of my life endlessly juking back and forth. The Next step keeps happening, so weak fairness is satisfied. What I actually want is for both my Move and my Juke to both be weakly fair independently of each other.

- Fairness == WF_vars(Next)
+ Fairness == WF_vars(Move(me)) /\ WF_vars(Juke(me))

If my liveness property also specified that you reached your goal, I could instead write \A p \in People: WF_vars(Move(p)) etc. I could also swap the \A with a \E to mean at least one of us is guaranteed to have fair actions, but not necessarily both of us.

New error:

3: <Move("me")>
pos = [you |-> [lr |-> 0, fb |-> 2], me |-> [lr |-> 0, fb |-> 3]]
4: <Juke("you")>
pos = [you |-> [lr |-> 1, fb |-> 2], me |-> [lr |-> 0, fb |-> 3]]
5: <Juke("me")>
pos = [you |-> [lr |-> 1, fb |-> 2], me |-> [lr |-> 1, fb |-> 3]]
6: <Juke("me")>
pos = [you |-> [lr |-> 1, fb |-> 2], me |-> [lr |-> 0, fb |-> 3]]
7: <Juke("you")> (back to state 3)

Now we're getting somewhere! This is the original walkwarding situation we wanted to capture. We're in each others way, then you juke, but before either of us can move you juke, then we both juke back. We can repeat this forever, trapped in a social hell.

Wait, but doesn't WF(Move(me)) guarantee I will eventually move? Yes, but only if a move is permanently available. In this case, it's not permanently available, because every couple of steps it's made temporarily unavailable.

How do I fix this? I can't add a rule saying that we only juke if we're blocked, because the whole point of walkwarding is that we're not coordinated. In the real world, walkwarding can go on for agonizing seconds. What I can do instead is say that Liveness holds as long as Move is strongly fair. Unlike weak fairness, strong fairness guarantees something happens if it keeps becoming possible, even with interruptions.

Liveness == 
+  SF_vars(Move(me)) => 
    <>(pos[me].fb = goal[me])

This makes the spec pass. Even if we weave back and forth for five minutes, as long as we eventually pass each other, I will reach my goal. Note we could also by making Move in Fairness strongly fair, which is preferable if we have a lot of different liveness properties to check.

A small exercise for the reader

There is a presumed invariant that is violated. Identify what it is, write it as a property in TLA+, and show the spec violates it. Then fix it.

Answer (in rot13): Gur vainevnag vf "ab gjb crbcyr ner va gur rknpg fnzr ybpngvba". Zbir thnenagrrf guvf ohg Whxr qbrf abg.

More TLA+ Exercises

I've started work on an exercises repo. There's only a handful of specific problems now but I'm planning on adding more over the summer.


  1. learntla is still on the toolbox, but I'm hoping to get it all moved over this summer. ↩

Write the most clever code you possibly can

8 May 2025 at 15:04

I started writing this early last week but Real Life Stuff happened and now you're getting the first-draft late this week. Warning, unedited thoughts ahead!

New Logic for Programmers release!

v0.9 is out! This is a big release, with a new cover design, several rewritten chapters, online code samples and much more. See the full release notes at the changelog page, and get the book here!

The new cover! It's a lot nicer

Write the cleverest code you possibly can

There are millions of articles online about how programmers should not write "clever" code, and instead write simple, maintainable code that everybody understands. Sometimes the example of "clever" code looks like this (src):

# Python

p=n=1
exec("p*=n*n;n+=1;"*~-int(input()))
print(p%n)

This is code-golfing, the sport of writing the most concise code possible. Obviously you shouldn't run this in production for the same reason you shouldn't eat dinner off a Rembrandt.

Other times the example looks like this:

def is_prime(x):
    if x == 1:
        return False
    return all([x%n != 0 for n in range(2, x)])

This is "clever" because it uses a single list comprehension, as opposed to a "simple" for loop. Yes, "list comprehensions are too clever" is something I've read in one of these articles.

I've also talked to people who think that datatypes besides lists and hashmaps are too clever to use, that most optimizations are too clever to bother with, and even that functions and classes are too clever and code should be a linear script.1. Clever code is anything using features or domain concepts we don't understand. Something that seems unbearably clever to me might be utterly mundane for you, and vice versa.

How do we make something utterly mundane? By using it and working at the boundaries of our skills. Almost everything I'm "good at" comes from banging my head against it more than is healthy. That suggests a really good reason to write clever code: it's an excellent form of purposeful practice. Writing clever code forces us to code outside of our comfort zone, developing our skills as software engineers.

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you [will get excellent debugging practice at exactly the right level required to push your skills as a software engineer] β€” Brian Kernighan, probably

There are other benefits, too, but first let's kill the elephant in the room:2

Don't commit clever code

I am proposing writing clever code as a means of practice. Being at work is a job with coworkers who will not appreciate if your code is too clever. Similarly, don't use too many innovative technologies. Don't put anything in production you are uncomfortable with.

We can still responsibly write clever code at work, though:

  1. Solve a problem in both a simple and a clever way, and then only commit the simple way. This works well for small scale problems where trying the "clever way" only takes a few minutes.
  2. Write our personal tools cleverly. I'm a big believer of the idea that most programmers would benefit from writing more scripts and support code customized to their particular work environment. This is a great place to practice new techniques, languages, etc.
  3. If clever code is absolutely the best way to solve a problem, then commit it with extensive documentation explaining how it works and why it's preferable to simpler solutions. Bonus: this potentially helps the whole team upskill.

Writing clever code...

...teaches simple solutions

Usually, code that's called too clever composes several powerful features together β€” the "not a single list comprehension or function" people are the exception. Josh Comeau's "don't write clever code" article gives this example of "too clever":

const extractDataFromResponse = (response) => {
  const [Component, props] = response;

  const resultsEntries = Object.entries({ Component, props });
  const assignIfValueTruthy = (o, [k, v]) => (v
    ? { ...o, [k]: v }
    : o
  );

  return resultsEntries.reduce(assignIfValueTruthy, {});
}

What makes this "clever"? I count eight language features composed together: entries, argument unpacking, implicit objects, splats, ternaries, higher-order functions, and reductions. Would code that used only one or two of these features still be "clever"? I don't think so. These features exist for a reason, and oftentimes they make code simpler than not using them.

We can, of course, learn these features one at a time. Writing the clever version (but not committing it) gives us practice with all eight at once and also with how they compose together. That knowledge comes in handy when we want to apply a single one of the ideas.

I've recently had to do a bit of pandas for a project. Whenever I have to do a new analysis, I try to write it as a single chain of transformations, and then as a more balanced set of updates.

...helps us master concepts

Even if the composite parts of a "clever" solution aren't by themselves useful, it still makes us better at the overall language, and that's inherently valuable. A few years ago I wrote Crimes with Python's Pattern Matching. It involves writing horrible code like this:

from abc import ABC

class NotIterable(ABC):

    @classmethod
    def __subclasshook__(cls, C):
        return not hasattr(C, "__iter__")

def f(x):
    match x:
        case NotIterable():
            print(f"{x} is not iterable")
        case _:
            print(f"{x} is iterable")

if __name__ == "__main__":
    f(10)
    f("string")
    f([1, 2, 3])

This composes Python match statements, which are broadly useful, and abstract base classes, which are incredibly niche. But even if I never use ABCs in real production code, it helped me understand Python's match semantics and Method Resolution Order better.

...prepares us for necessity

Sometimes the clever way is the only way. Maybe we need something faster than the simplest solution. Maybe we are working with constrained tools or frameworks that demand cleverness. Peter Norvig argued that design patterns compensate for missing language features. I'd argue that cleverness is another means of compensating: if our tools don't have an easy way to do something, we need to find a clever way.

You see this a lot in formal methods like TLA+. Need to check a hyperproperty? Cast your state space to a directed graph. Need to compose ten specifications together? Combine refinements with state machines. Most difficult problems have a "clever" solution. The real problem is that clever solutions have a skill floor. If normal use of the tool is at difficult 3 out of 10, then basic clever solutions are at 5 out of 10, and it's hard to jump those two steps in the moment you need the cleverness.

But if you've practiced with writing overly clever code, you're used to working at a 7 out of 10 level in short bursts, and then you can "drop down" to 5/10. I don't know if that makes too much sense, but I see it happen a lot in practice.

...builds comradery

On a few occasions, after getting a pull request merged, I pulled the reviewer over and said "check out this horrible way of doing the same thing". I find that as long as people know they're not going to be subjected to a clever solution in production, they enjoy seeing it!

Next week's newsletter will probably also be late, after that we should be back to a regular schedule for the rest of the summer.


  1. Mostly grad students outside of CS who have to write scripts to do research. And in more than one data scientist. I think it's correlated with using Jupyter. ↩

  2. If I don't put this at the beginning, I'll get a bajillion responses like "your team will hate you" ↩

Before yesterdayUncategorized

The evasive evitability of enshittification

15 June 2025 at 02:52

Our company recently announced a fundraise. We were grateful for all the community support, but the Internet also raised a few of its collective eyebrows, wondering whether this meant the dreaded β€œenshittification” was coming next.

That word describes a very real pattern we’ve all seen before: products start great, grow fast, and then slowly become worse as the people running them trade user love for short-term revenue.

It’s a topic I find genuinely fascinating, and I've seen the downward spiral firsthand at companies I once admired. So I want to talk about why this happens, and more importantly, why it won't happen to us. That's big talk, I know. But it's a promise I'm happy for people to hold us to.

What is enshittification?

The term "enshittification" was first popularized in a blog post by Corey Doctorow, who put a catchy name to an effect we've all experienced. Software starts off good, then goes bad. How? Why?

Enshittification proposes not just a name, but a mechanism. First, a product is well loved and gains in popularity, market share, and revenue. In fact, it gets so popular that it starts to defeat competitors. Eventually, it's the primary product in the space: a monopoly, or as close as you can get. And then, suddenly, the owners, who are Capitalists, have their evil nature finally revealed and they exploit that monopoly to raise prices and make the product worse, so the captive customers all have to pay more. Quality doesn't matter anymore, only exploitation.

I agree with most of that thesis. I think Doctorow has that mechanism mostly right. But, there's one thing that doesn't add up for me:

Enshittification is not a success mechanism.

I can't think of any examples of companies that, in real life, enshittified because they were successful. What I've seen is companies that made their product worse because they were... scared.

A company that's growing fast can afford to be optimistic. They create a positive feedback loop: more user love, more word of mouth, more users, more money, more product improvements, more user love, and so on. Everyone in the company can align around that positive feedback loop. It's a beautiful thing. It's also fragile: miss a beat and it flattens out, and soon it's a downward spiral instead of an upward one.

So, if I were, hypothetically, running a company, I think I would be pretty hesitant to deliberately sacrifice any part of that positive feedback loop, the loop I and the whole company spent so much time and energy building, to see if I can grow faster. User love? Nah, I'm sure we'll be fine, look how much money and how many users we have! Time to switch strategies!

Why would I do that? Switching strategies is always a tremendous risk. When you switch strategies, it's triggered by passing a threshold, where something fundamental changes, and your old strategy becomes wrong.

Threshold moments and control

In Saint John, New Brunswick, there's a river that flows one direction at high tide, and the other way at low tide. Four times a day, gravity equalizes, then crosses a threshold to gently start pulling the other way, then accelerates. What doesn't happen is a rapidly flowing river in one direction "suddenly" shifts to rapidly flowing the other way. Yes, there's an instant where the limit from the left is positive and the limit from the right is negative. But you can see that threshold coming. It's predictable.

In my experience, for a company or a product, there are two kinds of thresholds like this, that build up slowly and then when crossed, create a sudden flow change.

The first one is control: if the visionaries in charge lose control, chances are high that their replacements won't "get it."

The new people didn't build the underlying feedback loop, and so they don't realize how fragile it is. There are lots of reasons for a change in control: financial mismanagement, boards of directors, hostile takeovers.

The worst one is temptation. Being a founder is, well, it actually sucks. It's oddly like being repeatedly punched in the face. When I look back at my career, I guess I'm surprised by how few times per day it feels like I was punched in the face. But, the constant face punching gets to you after a while. Once you've established a great product, and amazing customer love, and lots of money, and an upward spiral, isn't your creation strong enough yet? Can't you step back and let the professionals just run it, confident that they won't kill the golden goose?

Empirically, mostly no, you can't. Actually the success rate of control changes, for well loved products, is abysmal.

The saturation trap

The second trigger of a flow change is comes from outside: saturation. Every successful product, at some point, reaches approximately all the users it's ever going to reach. Before that, you can watch its exponential growth rate slow down: the infamous S-curve of product adoption.

Saturation can lead us back to control change: the founders get frustrated and back out, or the board ousts them and puts in "real business people" who know how to get growth going again. Generally that doesn't work. Modern VCs consider founder replacement a truly desperate move. Maybe a last-ditch effort to boost short term numbers in preparation for an acquisition, if you're lucky.

But sometimes the leaders stay on despite saturation, and they try on their own to make things better. Sometimes that does work. Actually, it's kind of amazing how often it seems to work. Among successful companies, it's rare to find one that sustained hypergrowth, nonstop, without suffering through one of these dangerous periods.

(That's called survivorship bias. All companies have dangerous periods. The successful ones surivived them. But of those survivors, suspiciously few are ones that replaced their founders.)

If you saturate and can't recover - either by growing more in a big-enough current market, or by finding new markets to expand into - then the best you can hope for is for your upward spiral to mature gently into decelerating growth. If so, and you're a buddhist, then you hire less, you optimize margins a bit, you resign yourself to being About This Rich And I Guess That's All But It's Not So Bad.

The devil's bargain

Alas, very few people reach that state of zen. Especially the kind of ambitious people who were able to get that far in the first place. If you can't accept saturation and you can't beat saturation, then you're down to two choices: step away and let the new owners enshittify it, hopefully slowly. Or take the devil's bargain: enshittify it yourself.

I would not recommend the latter. If you're a founder and you find yourself in that position, honestly, you won't enjoy doing it and you probably aren't even good at it and it's getting enshittified either way. Let someone else do the job.

Defenses against enshittification

Okay, maybe that section was not as uplifting as we might have hoped. I've gotta be honest with you here. Doctorow is, after all, mostly right. This does happen all the time.

Most founders aren't perfect for every stage of growth. Most product owners stumble. Most markets saturate. Most VCs get board control pretty early on and want hypergrowth or bust. In tech, a lot of the time, if you're choosing a product or company to join, that kind of company is all you can get.

As a founder, maybe you're okay with growing slowly. Then some copycat shows up, steals your idea, grows super fast, squeezes you out along with your moral high ground, and then runs headlong into all the same saturation problems as everyone else. Tech incentives are awful.

But, it's not a lost cause. There are companies (and open source projects) that keep a good thing going, for decades or more. What do they have in common?

  • An expansive vision that's not about money, and which opens you up to lots of users. A big addressable market means you don't have to worry about saturation for a long time, even at hypergrowth speeds. Google certainly never had an incentive to make Google Search worse.

    (Update 2025-06-14: A few people disputed that last bit. Okay. Perhaps Google has ccasionally responded to what they thought were incentives to make search worse -- I wasn't there, I don't know -- but it seems clear in retrospect that when search gets worse, Google does worse. So I'll stick to my claim that their true incentives are to keep improving.)

  • Keep control. It's easy to lose control of a project or company at any point. If you stumble, and you don't have a backup plan, and there's someone waiting to jump on your mistake, then it's over. Too many companies "bet it all" on nonstop hypergrowth and don't have any way back have no room in the budget, if results slow down even temporarily.

    Stories abound of companies that scraped close to bankruptcy before finally pulling through. But far more companies scraped close to bankruptcy and then went bankrupt. Those companies are forgotten. Avoid it.

  • Track your data. Part of control is predictability. If you know how big your market is, and you monitor your growth carefully, you can detect incoming saturation years before it happens. Knowing the telltale shape of each part of that S-curve is a superpower. If you can see the future, you can prevent your own future mistakes.

  • Believe in competition. Google used to have this saying they lived by: "the competition is only a click away." That was excellent framing, because it was true, and it will remain true even if Google captures 99% of the search market. The key is to cultivate a healthy fear of competing products, not of your investors or the end of hypergrowth. Enshittification helps your competitors. That would be dumb.

    (And don't cheat by using lock-in to make competitors not, anymore, "only a click away." That's missing the whole point!)

  • Inoculate yourself. If you have to, create your own competition. Linus Torvalds, the creator of the Linux kernel, famously also created Git, the greatest tool for forking (and maybe merging) open source projects that has ever existed. And then he said, this is my fork, the Linus fork; use it if you want; use someone else's if you want; and now if I want to win, I have to make mine the best. Git was created back in 2005, twenty years ago. To this day, Linus's fork is still the central one.

If you combine these defenses, you can be safe from the decline that others tell you is inevitable. If you look around for examples, you'll find that this does actually work. You won't be the first. You'll just be rare.

Side note: Things that aren't enshittification

I often see people worry about enshittification that isn't. They might be good or bad, wise or unwise, but that's a different topic. Tools aren't inherently good or evil. They're just tools.

  1. "Helpfulness." There's a fine line between "telling users about this cool new feature we built" in the spirit of helping them, and "pestering users about this cool new feature we built" (typically a misguided AI implementation) to improve some quarterly KPI. Sometimes it's hard to see where that line is. But when you've crossed it, you know.

    Are you trying to help a user do what they want to do, or are you trying to get them to do what you want them to do?

    Look into your heart. Avoid the second one. I know you know how. Or you knew how, once. Remember what that feels like.

  2. Charging money for your product. Charging money is okay. Get serious. Companies have to stay in business.

    That said, I personally really revile the "we'll make it free for now and we'll start charging for the exact same thing later" strategy. Keep your promises.

    I'm pretty sure nobody but drug dealers breaks those promises on purpose. But, again, desperation is a powerful motivator. Growth slowing down? Costs way higher than expected? Time to capture some of that value we were giving away for free!

    In retrospect, that's a bait-and-switch, but most founders never planned it that way. They just didn't do the math up front, or they were too naive to know they would have to. And then they had to.

    Famously, Dropbox had a "free forever" plan that provided a certain amount of free storage. What they didn't count on was abandoned accounts, accumulating every year, with stored stuff they could never delete. Even if a very good fixed fraction of users each year upgraded to a paid plan, all the ones that didn't, kept piling up... year after year... after year... until they had to start deleting old free accounts and the data in them. A similar story happened with Docker, which used to host unlimited container downloads for free. In hindsight that was mathematically unsustainable. Success guaranteed failure.

    Do the math up front. If you're not sure, find someone who can.

  3. Value pricing. (ie. charging different prices to different people.) It's okay to charge money. It's even okay to charge money to some kinds of people (say, corporate users) and not others. It's also okay to charge money for an almost-the-same-but-slightly-better product. It's okay to charge money for support for your open source tool (though I stay away from that; it incentivizes you to make the product worse).

    It's even okay to charge immense amounts of money for a commercial product that's barely better than your open source one! Or for a part of your product that costs you almost nothing.

    But, you have to do the rest of the work. Make sure the reason your users don't switch away is that you're the best, not that you have the best lock-in. Yeah, I'm talking to you, cloud egress fees.

  4. Copying competitors. It's okay to copy features from competitors. It's okay to position yourself against competitors. It's okay to win customers away from competitors. But it's not okay to lie.

  5. Bugs. It's okay to fix bugs. It's okay to decide not to fix bugs; you'll have to sometimes, anyway. It's okay to take out technical debt. It's okay to pay off technical debt. It's okay to let technical debt languish forever.

  6. Backward incompatible changes. It's dumb to release a new version that breaks backward compatibility with your old version. It's tempting. It annoys your users. But it's not enshittification for the simple reason that it's phenomenally ineffective at maintaining or exploiting a monopoly, which is what enshittification is supposed to be about. You know who's good at monopolies? Intel and Microsoft. They don't break old versions.

Enshittification is real, and tragic. But let's protect a useful term and its definition! Those things aren't it.

Epilogue: a special note to founders

If you're a founder or a product owner, I hope all this helps. I'm sad to say, you have a lot of potential pitfalls in your future. But, remember that they're only potential pitfalls. Not everyone falls into them.

Plan ahead. Remember where you came from. Keep your integrity. Do your best.

I will too.

APIs as a product: Investing in the current and next generation of technical contributors

12 June 2025 at 16:21

Wikipedia is coming up on its 25th birthday, and that would not have been possible without the Wikimedia technical volunteer community. Supporting technical volunteers is crucial to carrying forward Wikimedia’s free knowledge mission for generations to come. In line with this commitment, the Foundation is turning its attention to an important area of developer supportβ€”the Wikimedia web (HTTP) APIs.Β 

Both Wikimedia and the Internet have changed a lot over the last 25 years. Patterns that are now ubiquitous standards either didn’t exist or were still in their infancy as the first APIs allowing developers to extend features and automate tasks on Wikimedia projects emerged. In fact, the term β€œrepresentational state transfer”, better known today as the REST framework, was first coined in 2000, just months before the very first Wikipedia post was published, and only 6 years before the Action API was introduced. Because we preceded what have since become industry standards, our most powerful and comprehensive API solution, the Action API, sticks out as being unlike other APIs – but for good reason, if you understand the history.

Wikimedia APIs are used within Foundation-authored features and by volunteer developers. A common sentiment surfaced through the recent API Listening Tour conducted with a mix of volunteers and Foundation staff is β€œWikimedia APIs are great, once you know what you’re doing.” New developers first entering the Wikimedia community face a steep learning curve when trying to onboard due to unfamiliar technologies and complex APIs that may require a deep understanding of the underlying Wikimedia systems and processes. While recognizing the power, flexibility, and mission-critical value that developers created using the existing API solutions, we want to make it easier for developers to make more meaningful contributions faster. We have no plans to deprecate the Action API nor treat it as β€˜legacy’. Instead, we hope to make it easier and more approachable for both new and experienced developers to use. We also aim to expand REST coverage to better serve developers who are more comfortable working in those structures.

We are focused on simplifying, modernizing, and standardizing Wikimedia API offerings as part of the Responsible Use of Infrastructure objective in the FY25-26 Annual Plan (see: the WE5.2 key result). Focusing on common infrastructure that encourages responsible use allows us to continue to prioritize reliable, free access to knowledge for the technical volunteer community, as well as the readers and contributors they support. Investing in our APIs and the developer experiences surrounding them will ensure a healthy technical community for years to come. To achieve these objectives, we see three main areas for improving the sustainability of our API offering: simplification, documentation, and communication.

Simplification

To reduce maintenance costs and ensure a seamless developer experience, we are simplifying our API infrastructure and bringing greater consistency across all APIs. Decades of organic growth without centralized API governance led to fragmented, bespoke implementations that now hinder technical agility and standardization. Beyond that, maintaining services is not free; we are paying for duplicative infrastructure costs, some of which are scaling directly with the amount of scraper traffic hitting our services.

In light of the above, we will focus on transitioning at least 70% of our public endpoints to common API infrastructure (see the WE 5.2 key result). Common infrastructure makes it easier to maintain and roll out changes across our APIs, in addition to empowering API authors to move faster. Instead of expecting API authors to build and manage their own solutions for things like routing and rate limiting, we will create centralized tools and processes that make it easier to follow the β€œgolden path” of recommended standards. That will allow centralized governance mechanisms to drive more consistent and sustainable end-user experiences, while enabling flexible, federated API ownership.Β 

An example of simplified internal infrastructure will be introducing a common API Gateway for handling and routing all Wikimedia API requests. Our approach will start as an β€œinvisible gateway” or proxy, with no changes to URL structure or functional behavior for any existing APIs. Centralizing API traffic will make observability across APIs easier, allowing us to make better data-driven decisions. We will use this data to inform endpoint deprecation and versioning, prioritize human and mission-oriented access first, and ultimately provide better support to our developer community.Β Β 

Centralized management and traffic identification will also allow us to have more consistent and transparent enforcement of our API policies. API policy enforcement enables us to protect our infrastructure and ensure continued access for all. Once API traffic is rerouted through a centralized gateway, we will explore simplifying options for developer identification mechanisms and standardizing how rate limits and other API access controls are applied. The goal is to make it easier for all developers to know exactly what is expected and what limitations apply.

As we update our API usage policies and developer requirements, we will avoid breaking existing community tools as much as possible. We will continue offering low-friction entry points for volunteer developers experimenting with new ideas, lightly exploring data, or learning to build in the Wikimedia ecosystem. But we must balance support for community creativity and innovation with the need to reduce abuse, such as scraping, Denial of Service (DoS) attacks, and other harmful activities. While open, unauthenticated API access for everyone will continue, we will need to make adjustments. To reduce the likelihood and impact of abuse, we may apply stricter rate limits to unauthenticated traffic and more consistent authentication requirements to better match our documented API policy, Robot policy, and API etiquette guidelines, as well as consolidate per-API access guidelines to reduce the likelihood and impact of abuse.

To continue supporting Wikimedia’s technical volunteer community and minimize disruption to existing tools, community developers will have simple ways to identify themselves and receive higher limits or other access privileges. In many cases, this won’t require additional steps. For example, instead of universally requiring new access tokens or authentication methods, we plan to use IP ranges from Wikimedia Cloud Services (WMCS) and User-Agent headers to grant elevated privileges to trusted community tools, approved bots, and research projects.Β 

Documentation

It is essential for any API to enable developers to self-serve their use cases through clear, consistent, and modern documentation experiences. However, Wikimedia API documentation is frequently spread across multiple wiki projects, generated sites, and communication channels, which can make it difficult for developers to find the information they need, when they need it.Β 

To address this, we are working towards a top-requested item coming out of the 2024 developer satisfaction survey: OpenAPI specs and interactive sandboxes for all of our APIs (including conducting experiments to see if we can use OpenAPI to describe the Action API). The MediaWiki Interfaces team began addressing this request through the REST Sandbox, which we released to a limited number of small Wikipedia projects on March 31, 2025. Our implementation approach allows us to generate an OpenAPI specification, which we then use to power a SwaggerUI sandbox. We are also using the OpenAPI specs to automatically validate our endpoints as part of our automated deployment testing, which helps ensure that the generated documentation always matches the actual endpoint behavior.Β 

In addition, the generated OpenAPI spec offers translation support (powered by Translatewiki) for critical and contextual information like endpoint and parameter descriptions. We believe this is a more equitable approach to API documentation for developers who don’t have English as their preferred language. In the coming year, we plan to transition from Swagger UI to a custom Codex implementation for our sandbox experiences, which will enable full translation support for sandbox UI labels and navigation, as well as a more consistent look and feel for Wikimedia developers. We will also expand coverage for OpenAPI specs and sandbox experiences by introducing repeatable patterns for API authors to publish their specs to a single location where developers can easily browse, learn, and make test calls across all Wikimedia API offerings.Β 

Communication

When new endpoints are released or breaking changes are required, we need a better way to keep developers informed. As information is shared through different channels, it can become challenging to keep track of the full picture. Over the next year, we will address this on a few fronts.Β 

First, from a technical change management perspective, we will introduce a centralized API changelog. The changelog will summarize new endpoints, as well as new versions, planned deprecations, and minor changes such as new optional parameters. This will help developers with troubleshooting, as well as help them to more easily understand and monitor the changes happening across the Wikimedia APIs.

In addition to the changelog, we remain committed to consistently communicating changes early and often. As another step towards this commitment, we will provide migration guides and, where needed, provide direct communication channels for developers impacted by the changes to help guarantee a smooth transition. Recognizing that the Wikimedia technical community is split across many smaller communities both on and off-wiki, we will share updates in the largest off-wiki communities, but we will need volunteer support in directing questions and feedback to the right on-wiki pages in various languages. We will also work with communities to make their purpose and audience clearer for new developers so they can more easily get support when they need it and join the discussion with fellow technical contributors.Β 

Over the next few months, we will also launch a new API beta program, where developers are invited to interact with new endpoints and provide feedback before the capabilities are locked into a long-term stable version. Introducing new patterns through a beta program will allow developers to directly shape the future of the Wikimedia APIs to better suit their needs. To demonstrate this pattern, we will start with changes to MediaWiki REST APIs, including introducing API modularization and consistent structures.Β 

What’s Next

We are still in the early stages – we are just making the first steps on the journey to a unified API product offering. But we hope that by this time next year, we will be running towards it together. Your involvement and insights can help us shape a future that better serves the technical volunteers behind our knowledge mission. To keep you informed, we will continue to post updates on mailing lists, Diff, TechBlog, and other technical volunteer communication channels. We also invite you to stay actively engaged: share your thoughts on the WE5 objective in the annual plan, ask questions on the related discussion pages, review slides from the Future of Wikimedia APIs session we conducted at the Wikimedia Hackathon, volunteer for upcoming Listening Tour topics, or come talk to us at upcoming events such as Wikimania Nairobi.Β 

Technical volunteers play an essential role in the growth and evolution of Wikipedia, as well as all other Wikimedia projects. Together, we can make a better experience for developers who can’t remember life before Wikipedia, and make sure that the next generation doesn’t have to live without it. Here’s to another 25 years!Β 

Promoting events and WikiProjects

Editatona Mujeres Artistas Mexicanas 2024, Museo Universitario de Arte ContemporΓ‘neo, Mexico City, Mexico
Editatona_Mujeres_Artistas_Mexicanas_2024_10, ProtoplasmaKid

The Campaigns team at WMF has released two features that allow organizers to promote events and WikiProjects on the wikis: Invitation Lists and Collaboration List. These two tools are a part of the CampaignEvents extension, which is available on many wikis.

Invitation Lists

Product overview

Invitation Lists allows organizers to generate a list of people to invite to their WikiProjects, events, or other collaborative activities. It can be accessed by going to Special:GenerateInvitationList, if a wiki has the CampaignEvents extension enabled. You can watch this video demo to see how it works.

It works by looking at a list of articles that an organizer plans to focus on during an activity and then finding users to invite based on the following criteria: the bytes they contributed to the articles, the number of edits they made to the articles, their overall edit count on the wikis, and how recently they have edited the wikis. This makes it easier for organizers to invite people who are already interested in the activity’s topics, hence increasing the likelihood of participation.

With this work, we hope to empower organizers to seek out new audiences. We also hope to highlight the important work done by editors, who may be inspired or touched to receive an invitation to an activity based on their work. However, if someone does not want to receive invitations, they can opt out of being included in Invitation Lists via Preferences.

Technical overview

The β€œInvitation Lists” feature is part of the CampaignEvents extension for MediaWiki, designed to assist event organizers in identifying and reaching out to potential participants based on their editing activity.

Access and Permissions

  • Special Pages: The feature introduces two special pages:
    • Special:GenerateInvitationList: Allows organizers to create new invitation lists.
    • Special:InvitationList: Displays the generated list of recommended invitees.
  • User Rights: Access to these pages is restricted to users with the event-organizer right, ensuring that only authorized individuals can generate and view invitation lists.

Invitation List Generation Process

  1. Input Parameters:
    • List Name: Organizers provide a name for the invitation list.
    • Target Articles: A list of up to 300 articles relevant to the event’s theme.
      • Β The articles will need to be on the wiki of the Invitation List.
    • Event Page Link: Optionally, a link to the event’s registration page can be included.
  2. Data Collection:
    • The system analyzes the specified articles to identify contributors.
    • For each contributor, it gathers metrics such as:
      • Bytes Added: The total number of bytes the user has added to the articles.
      • Edit Count: The number of edits made by the user on the specified articles.
      • Overall Edit Count: The user’s total edit count across the wiki.
      • Recent Activity: The recency of the user’s edits on the wiki.
  3. Scoring and Ranking:
    • Contributors are scored based on the collected metrics.
    • The scoring algorithm assigns weights to each metric to calculate a composite score for each user.
    • Users are then ranked and categorized into:
      • Highly Recommended to Invite: Top contributors with high relevance and recent activity.
      • Recommended to Invite: Contributors with moderate relevance and activity.
  4. Output:
    • The generated invitation list is displayed on the Special:InvitationList page.
    • Each listed user includes a link to their contributions page, facilitating further review by the organizer.

Technical Implementation Details

  • Backend Processing:
    • The extension utilizes MediaWiki’s job queue system to handle the processing of invitation lists asynchronously, ensuring that the generation process does not impact the performance of the wiki.
    • Jobs are queued upon submission of the article list and processed in the background.
    • The articles will need to be on the wiki of the Invitation List, and they can add a maximum of 300 articles.
  • Data Retrieval:
    • The extension interfaces with MediaWiki’s revision and user tables to extract the necessary contribution data.
    • Efficient querying and indexing strategies are employed to handle large datasets and ensure timely processing.
  • User Preferences and Privacy:
    • Users have the option to opt out of being included in invitation lists via their preferences.
    • The extension respects these preferences by excluding opted-out users from the generated lists.
  • Integration with Event Registration:
    • If an event page link is provided, the invitation list can be associated with the event’s registration data. This way, we can link their invitation data to their event registration data.

Collaboration List

Product overview

The Collaboration List is a list of events and WikiProjects. It can be accessed by going to SpecialːAllEvents,  if a wiki has the CampaignEvents extension enabled.

The Collaboration List has two tabs: β€œEvents” and β€œCommunities.” The Events tab is a global, automated list of all events that use Event Registration. It also has search filters, so you can find events by start and end dates, meeting type (i.e., online, in person, or hybrid), event topic, event wikis, and by keyword searches. You can also find events that are both ongoing (i.e., started before but continue within the selected date range) and upcoming (i.e., events that start within the selected date range).

The Communities tab provides a list of WikiProjects on the local wiki. The WikiProject list is generated by using Wikidata, and it includes: WikiProject name, description, a link to the WikiProject page, and a link to the Wikidata item for the WikiProject. We aim to produce a symbiotic relationship with WikiProjects, in which people can find WikiProjects that interest them, and they can also enhance the Wikidata items for those projects, which in turn improves our project.

Additionally, you can embed the Collaboration List on any wiki page, if the CampaignEvents extension is enabled on that wiki. To do this, you transclude the Collaboration List on a wiki page. You can also choose to customize the Collaboration List through URL parameters, if you want. For example, you can choose to only display a certain number of events or to add formatting. You can read more about this on Help:Extension:CampaignEvents/Collaboration list/Transclusion.

With the Collaboration List, we hope to make it easier for people to find events and WikiProjects that interest them, so more people can find community and make impactful contributions on the wikis together.

Screenshot of the Collaboration List
Screenshot of the Collaboration List

Technical Overview: Events Tab of Collaboration List

  • Purpose: Displays a global list of events across all participating wikis.
  • Data Source: Event data stored centrally in Wikimedia’s X1 database cluster.
  • Displayed Information:
    • Event name and description
    • Event dates (start and end)
    • Event type (online, in-person, hybrid)
    • Associated wikis and event topics
  • Search and Filters:
    • Date range (start/end)
    • Meeting type (online, in-person, hybrid)
    • Event topics and wikis
    • Keyword search
    • Ongoing and upcoming event filtering
  • Technical Implementation:
    • The CampaignEvents extension retrieves event data directly from centralized tables within the X1 cluster.
    • Efficient SQL queries and indexing optimize performance for cross-wiki data retrieval.

This implementation ensures quick access and easy discoverability of events from across Wikimedia projects.

Technical Overview: Communities Tab of Collaboration List

  • Purpose: Displays a list of local WikiProjects on the wiki.
  • Data Source: Dynamically retrieved from Wikidata via the Wikidata Query Service (WDQS).
  • Displayed Information:
    • WikiProject name
    • Description from Wikidata
    • Link to the local WikiProject page
    • Link to the Wikidata item
  • Performance Optimization:
    • Query results from WDQS are cached locally using MediaWiki’s caching mechanisms (WANObjectCache).
    • Cache reduces repeated queries and ensures quick loading times.
  • Technical Implementation:
    • The WikimediaCampaignEvents extension retrieves data via SPARQL from WDQS.
    • The CampaignEvents extension renders the data on Special:AllEvents under the Communities tab.
  • Extension Communication:
    • The extensions communicate using MediaWiki’s hook system. The WikimediaCampaignEvents extension provides WikiProject data to the CampaignEvents extension through hook implementations.

This structure enables efficient collaboration between extensions, ensuring clear responsibilities, optimized performance, and simplified discoverability of WikiProjects.

Wikimedia Cloud VPS: IPv6 support

6 May 2025 at 17:58
Dietmar Rabich, Cape Town (ZA), Sea Point, Nachtansicht β€” 2024 β€” 1867-70 – 2, CC BY-SA 4.0

Wikimedia Cloud VPS is a service offered by the Wikimedia Foundation, built using OpenStack and managed by the Wikimedia Cloud Services team. It provides cloud computing resources for projects related to the Wikimedia movement, including virtual machines, databases, storage, Kubernetes, and DNS.

A few weeks ago, in April 2025, we were finally able to introduce IPv6 to the cloud virtual network, enhancing the platform’s scalability, security, and future-readiness. This is a major milestone, many years in the making, and serves as an excellent point to take a moment to reflect on the road that got us here. There were definitely a number of challenges that needed to be addressed before we could get into IPv6. This post covers the journey to this implementation.

The Wikimedia Foundation was an early adopter of the OpenStack technology, and the original OpenStack deployment in the organization dates back to 2011. At that time, IPv6 support was still nascent and had limited implementation across various OpenStack components. In 2012, the Wikimedia cloud users formally requested IPv6 support.

When Cloud VPS was originally deployed, we had set up the network following some of the upstream-recommended patterns:

  • nova-networks as the engine in charge of the software-defined virtual network
  • using a flat network topology – all virtual machines would share the same network
  • using a physical VLAN in the datacenter
  • using Linux bridges to make this physical datacenter VLAN available to virtual machines
  • using a single virtual router as the edge network gateway, also executing a global egress NAT – barring some exceptions, using what was called β€œdmz_cidr” mechanism

In order for us to be able to implement IPv6 in a way that aligned with our architectural goals and operational requirements, pretty much all the elements in this list would need to change. First of all, we needed to migrate from nova-networks into Neutron, a migration effort that started in 2017. Neutron was the more modern component to implement software-defined networks in OpenStack. To facilitate this transition, we made the strategic decision to backport certain functionalities from nova-networks into Neutron, specifically the β€œdmz_cidr” mechanism and some egress NAT capabilities.

Once in Neutron, we started to think about IPv6. In 2018 there was an initial attempt to decide on the network CIDR allocations that Wikimedia Cloud Services would have. This initiative encountered unforeseen challenges and was subsequently put on hold. We focused on removing the previously backported nova-networks patches from Neutron.

Between 2020 and 2021, we initiated another significant network refresh. We were able to introduce the cloudgw project, as part of a larger effort to rework the Cloud VPS edge network. The new edge routers allowed us to drop all the custom backported patches we had in Neutron from the nova-networks era, unblocking further progress. Worth mentioning that the cloudgw router would use nftables as firewalling and NAT engine.

A pivotal decision in 2022 was to expose the OpenStack APIs to the internet, which crucially enabled infrastructure management via OpenTofu. This was key in the IPv6 rollout as will be explained later. Before this, management was limited to Horizon – the OpenStack graphical interface – or the command-line interface accessible only from internal control servers.

Later, in 2023, following the OpenStack project’s announcement of the deprecation of the neutron-linuxbridge-agent, we began to seriously consider migrating to the neutron-openvswitch-agent. This transition would, in turn, simplify the enablement of β€œtenant networks” – a feature allowing each OpenStack project to define its own isolated network, rather than all virtual machines sharing a single flat network.

Once we replaced neutron-linuxbridge-agent with neutron-openvswitch-agent, we were ready to migrate virtual machines to VXLAN. Demonstrating perseverance, we decided to execute the VXLAN migration in conjunction with the IPv6 rollout.

We prepared and tested several things, including the rework of the edge routing to be based on BGP/OSPF instead of static routing. In 2024 we were ready for the initial attempt to deploy IPv6, which failed for unknown reasons. There was a full network outage and we immediately reverted the changes. This quick rollback was feasible due to our adoption of OpenTofu: deploying IPv6 had been reduced to a single code change within our repository.

We started an investigation, corrected a few issues, and increased our network functional testing coverage before trying again. One of the problems we discovered was that Neutron would enable the β€œenable_snat” configuration flag for our main router when adding the new external IPv6 address.

Finally, in April 2025, after many years in the making, IPv6 was successfully deployed.

Compared to the network from 2011, we would have:

  • Neutron as the engine in charge of the software-defined virtual network
  • Ready to use tenant-networks
  • Using a VXLAN-based overlay network
  • Using neutron-openvswitch-agent to provide networking to virtual machines
  • A modern and robust edge network setup

Over time, the WMCS team has skillfully navigated numerous challenges to ensure our service offerings consistently meet high standards of quality and operational efficiency. Often engaging in multi-year planning strategies, we have enabled ourselves to set and achieve significant milestones.

The successful IPv6 deployment stands as further testament to the team’s dedication and hard work over the years. I believe we can confidently say that the 2025 Cloud VPS represents its most advanced and capable iteration to date.

2025-06-08 Omnimax

8 June 2025 at 00:00

In a previous life, I worked for a location-based entertainment company, part of a huge team of people developing a location for Las Vegas, Nevada. It was COVID, a rough time for location-based anything, and things were delayed more than usual. Coworkers paid a lot of attention to another upcoming Las Vegas attraction, one with a vastly larger budget but still struggling to make schedule: the MSG (Madison Square Garden) Sphere.

I will set aside jokes about it being a square sphere, but they were perhaps one of the reasons that it underwent a pre-launch rebranding to merely the Sphere. If you are not familiar, the Sphere is a theater and venue in Las Vegas. While it's know mostly for the video display on the outside, that's just marketing for the inside: a digital dome theater, with seating at a roughly 45 degree stadium layout facing a near hemisphere of video displays.

It is a "near" hemisphere because the lower section is truncated to allow a flat floor, which serves as a stage for events but is also a practical architectural decision to avoid completely unsalable front rows. It might seem a little bit deceptive that an attraction called the Sphere does not quite pull off even a hemisphere of "payload," but the same compromise has been reached by most dome theaters. While the use of digital display technology is flashy, especially on the exterior, the Sphere is not quite the innovation that it presents itself as. It is just a continuation of a long tradition of dome theaters. Only time will tell, but the financial difficulties of the Sphere suggest that it follows the tradition faithfully: towards commercial failure.

You could make an argument that the dome theater is hundreds of years old, but I will omit it. Things really started developing, at least in our modern tradition of domes, with the 1923 introduction of the Zeiss planetarium projector. Zeiss projectors and their siblings used a complex optical and mechanical design to project accurate representations of the night sky. Many auxiliary projectors, incorporated into the chassis and giving these projectors famously eccentric shapes, rendered planets and other celestial bodies. Rather than digital light modulators, the images from these projectors were formed by purely optical means: perforated metal plates, glass plates with etched metalized layers, and fiber optics. The large, precisely manufactured image elements and specialized optics created breathtaking images.

While these projectors had considerable entertainment value, especially in the mid-century when they represented some of the most sophisticated projection technology yet developed, their greatest potential was obviously in education. Planetarium projectors were fantastically expensive (being hand-built in Germany with incredible component counts) [1], they were widely installed in science museums around the world. Most of us probably remember a dogbone-shaped Zeiss, or one of their later competitors like Spitz or Minolta, from our youths. Unfortunately, these marvels of artistic engineering were mostly retired as digital projection of near comparable quality became similarly priced in the 2000s.

But we aren't talking about projectors, we're talking about theaters. Planetarium projectors were highly specialized to rendering the night sky, and everything about them was intrinsically spherical. For both a reasonable viewing experience, and for the projector to produce a geometrically correct image, the screen had to be a spherical section. Thus the planetarium itself: in its most traditional form, rings of heavily reclined seats below a hemispherical dome. The dome was rarely a full hemisphere, but was usually truncated at the horizon. This was mostly a practical decision but integrated well into the planetarium experience, given that sky viewing is usually poor near the horizon anyway. Many planetaria painted a city skyline or forest silhouette around the lower edge to make the transition from screen to wall more natural. Later, theatrical lighting often replaced the silhouette, reproducing twilight or the haze of city lights.

Unsurprisingly, the application-specific design of these theaters also limits their potential. Despite many attempts, the collective science museum industry has struggled to find entertainment programming for planetaria much beyond Pink Floyd laser shows [2]. There just aren't that many things that you look up at. Over time, planetarium shows moved in more narrative directions. Film projection promised new flexibility---many planetaria with optical star projectors were also equipped with film projectors, which gave show producers exciting new options. Documentary video of space launches and animations of physical principles became natural parts of most science museum programs, but were a bit awkward on the traditional dome. You might project four copies of the image just above the horizon in the four cardinal directions, for example. It was very much a compromise.

With time, the theater adapted to the projection once again: the domes began to tilt. By shifting the dome in one direction, and orienting the seating towards that direction, you could create a sort of compromise point between the traditional dome and traditional movie theater. The lower central area of the screen was a reasonable place to show conventional film, while the full size of the dome allowed the starfield to almost fill the audience's vision. The experience of the tilted dome is compared to "floating in space," as opposed to looking up at the sky.

In true Cold War fashion, it was a pair of weapons engineers (one nuclear weapons, the other missiles) who designed the first tilted planetarium. In 1973, the planetarium of what is now called the Fleet Science Center in San Diego, California opened to the public. Its dome was tilted 25 degrees to the horizon, with the seating installed on a similar plane and facing in one direction. It featured a novel type of planetarium projector developed by Spitz and called the Space Transit Simulator. The STS was not the first, but still an early mechanical projector to be controlled by a computer---a computer that also had simultaneous control of other projectors and lighting in the theater, what we now call a show control system.

Even better, the STS's innovative optical design allowed it to warp or bend the starfield to simulate its appearance from locations other than earth. This was the "transit" feature: with a joystick connected to the control computer, the planetarium presenter could "fly" the theater through space in real time. The STS was installed in a well in the center of the seating area, and its compact chassis kept it low in the seating area, preserving the spherical geometry (with the projector at the center of the sphere) without blocking the view of audience members sitting behind it and facing forward.

And yet my main reason for discussing the Fleet planetarium is not the the planetarium projector at all. It is a second projector, an "auxiliary" one, installed in a second well behind the STS. The designers of the planetarium intended to show film as part of their presentations, but they were not content with a small image at the center viewpoint. The planetarium commissioned a few of the industry's leading film projection experts to design a film projection system that could fill the entire dome, just as the planetarium projector did.

They knew that such a large dome would require an exceptionally sharp image. Planetarium projectors, with their large lithographed slides, offered excellent spatial resolution. They made stars appear as point sources, the same as in the night sky. 35mm film, spread across such a large screen, would be obviously blurred in comparison. They would need a very large film format.

Omnimax dome with work lights on at Chicago Museum of Science and Industry

Fortuitously, almost simultaneously the Multiscreen Corporation was developing a "sideways" 70mm format. This 15-perf format used 70mm film but fed it through the projector sideways, making each frame much larger than typical 70mm film. In its debut, at a temporary installation in the 1970 Expo Osaka, it was dubbed IMAX. IMAX made an obvious basis for a high-resolution projection system, and so the then-named IMAX Corporation was added to the planetarium project. The Fleet's film projector ultimately consisted of an IMAX film transport with a custom-built compact, liquid-cooled lamphouse and spherical fisheye lens system.

The large size of the projector, the complex IMAX framing system and cooling equipment, made it difficult to conceal in the theater's projector well. Threading film into IMAX projectors is quite complex, with several checks the projectionist must make during a pre-show inspection. The projectionist needed room to handle the large film, and to route it to and from the enormous reels. The projector's position in the middle of the seating area left no room for any of this. We can speculate that it was, perhaps, one of the designer's missile experience that lead to the solution: the projector was serviced in a large projection room beneath the theater's seating. Once it was prepared for each show, it rose on near-vertical rails until just the top emerged in the theater. Rollers guided the film as it ran from a platter, up the shaft to the projector, and back down to another platter. Cables and hoses hung below the projector, following it up and down like the traveling cable of an elevator.

To advertise this system, probably the greatest advance in film projection since the IMAX format itself, the planetarium coined the term Omnimax.

Omnimax was not an easy or economical format. Ideally, footage had to be taken in the same format, using a 70mm camera with a spherical lens system. These cameras were exceptionally large and heavy, and the huge film format limited cinematographers to short takes. The practical problems with Omnimax filming were big enough that the first Omnimax films faked it, projecting to the larger spherical format from much smaller conventional negatives. This was the case for "Voyage to the Outer Planets" and "Garden Isle," the premier films at the Fleet planetarium. The history of both is somewhat obscure, the latter especially.

"Voyage to the Outer Planets" was executive-produced by Preston Fleet, a founder of the Fleet center (which was ultimately named for his father, a WWII aviator). We have Fleet's sense of showmanship to thank for the invention of Omnimax: He was an accomplished business executive, particularly in the photography industry, and an aviation enthusiast who had his hands in more than one museum. Most tellingly, though, he had an eccentric hobby. He was a theater organist. I can't help but think that his passion for the theater organ, an instrument almost defined by the combination of many gizmos under electromechanical control, inspired "Voyage." The film, often called a "multimedia experience," used multiple projectors throughout the planetarium to depict a far-future journey of exploration. The Omnimax film depicted travel through space, with slide projectors filling in artist's renderings of the many wonders of space.

The ten-minute Omnimax film was produced by Graphic Films Corporation, a brand that would become closely associated with Omnimax in the following decades. Graphic was founded in the midst of the Second World War by Lester Novros, a former Disney animator who found a niche creating training films for the military. Novros's fascination with motion and expertise in presenting complicated 3D scenes drew him to aerospace, and after the war he found much of his business in the newly formed Air Force and NASA. He was also an enthusiast of niche film formats, and Omnimax was not his first dome.

For the 1964 New York World's Fair, Novros and Graphic Films had produced "To the Moon and Beyond," a speculative science film with thematic similarities to "Voyage" and more than just a little mechanical similarity. It was presented in Cinerama 360, a semi-spherical, dome-theater 70mm format presented in a special theater called the Moon Dome. "To the Moon and Beyond" was influential in many ways, leading to Graphic Films' involvement in "2001: A Space Odyssey" and its enduring expertise in domes.

The Fleet planetarium would not remain the only Omnimax for long. In 1975, the city of Spokane, Washington struggled to find a new application for the pavilion built for Expo '74 [3]. A top contender: an Omnimax theater, in some ways a replacement for the temporary IMAX theater that had been constructed for the actual Expo. Alas, this project was not to be, but others came along: in 1978, the Detroit Science Center opened the second Omnimax theater ("the machine itself looks like and is the size of a front loader," the Detroit Free Press wrote). The Science Museum of Minnesota, in St. Paul, followed shortly after.

Omnimax hit prime time the next year, with the 1979 announcement of an Omnimax theater at Caesars Palace in Las Vegas, Nevada. Unlike the previous installations, this 380-seat theater was purely commercial. It opened with the 1976 IMAX film "To Fly!," which had been optically modified to fit the Omnimax format. This choice of first film is illuminating. "To Fly!" is a 27 minute documentary on the history of aviation in the United States, originally produced for the IMAX theater at the National Air and Space Museum [4]. It doesn't exactly seem like casino fare.

The IMAX format, the flat-screen one, was born of world's fairs. It premiered at an Expo, reappeared a couple of years later at another one, and for the first years of the format most of the IMAX theaters built were associated with either a major festival or an educational institution. This noncommercial history is a bit hard to square with the modern IMAX brand, closely associated with major theater chains and the Marvel Cinematic Universe.

Well, IMAX took off, and in many ways it sold out. Over the decades since the 1970 Expo, IMAX has met widespread success with commercial films and theater owners. Simultaneously, the definition or criteria for IMAX theaters have relaxed, with smaller screens made permissible until, ultimately, the transition to digital projection eliminated the 70mm film and more or less reduce IMAX to just another ticket surcharge brand. It competes directly with Cinemark xD, for example. To the theater enthusiast, this is a pretty sad turn of events, a Westinghouse-esque zombification of a brand that once heralded the field's most impressive technical achievements.

The same never happened to Omnimax. The Caesar's Omnimax theater was an odd exception; the vast majority of Omnimax theaters were built by science museums and the vast majority of Omnimax films were science documentaries. Quite a few of those films had been specifically commissioned by science museums, often on the occasion of their Omnimax theater opening. The Omnimax community was fairly tight, and so the same names recur.

The Graphic Films Corporation, which had been around since the beginning, remained so closely tied to the IMAX brand that they practically shared identities. Most Omnimax theaters, and some IMAX theaters, used to open with a vanity card often known as "the wormhole." It might be hard to describe beyond "if you know you know," it certainly made an impression on everyone I know that grew up near a theater that used it. There are some videos, although unfortunately none of them are very good.

I have spent more hours of my life than I am proud to admit trying to untangle the history of this clip. Over time, it has appeared in many theaters with many different logos at the end, and several variations of the audio track. This is in part informed speculation, but here is what I believe to be true: the "wormhole" was originally created by Graphic Films for the Fleet planetarium specifically, and ran before "Voyage to the Outer Planets" and its double-feature companion "Garden Isle," both of which Graphic Films had worked on. This original version ended with the name Graphic Films, accompanied by an odd sketchy drawing that was also used as an early logo of the IMAX Corporation. Later, the same animation was re-edited to end with an IMAX logo.

This version ran in both Omnimax and conventional IMAX theaters, probably as a result of the extensive "cross-pollination" of films between the two formats. Many Omnimax films through the life of the format had actually been filmed for IMAX, with conventional lenses, and then optically modified to fit the Omnimax dome after the fact. You could usually tell: the reprojection process created an unusual warp in the image, and more tellingly, these pseudo-Omnimax films almost always centered the action at the middle of the IMAX frame, which was too high to be quite comfortable in an Omnimax theater (where the "frame center" was well above the "front center" point of the theater). Graphic Films had been involved in a lot of these as well, perhaps explaining the animation reuse, but it's just as likely that they had sold it outright to the IMAX corporation which used it as they pleased.

For some reason, this version also received new audio that is mostly the same but slightly different. I don't have a definitive explanation, but I think there may have been an audio format change between the very early Omnimax theaters and later IMAX/Omnimax systems, which might have required remastering.

Later, as Omnimax domes proliferated at science museums, the IMAX Corporation (which very actively promoted Omnimax to education) gave many of these theaters custom versions of the vanity card that ended with the science museum's own logo. I have personally seen two of these, so I feel pretty confident that they exist and weren't all that rare (basically 2 out of 2 Omnimax theaters I've visited used one), but I cannot find any preserved copies.

Another recurring name in the world of IMAX and Omnimax is MacGillivray Freeman Films. MacGillivray and Freeman were a pair of teenage friends from Laguna Beach who dropped out of school in the '60s to make skateboard and surf films. This is, of course, a rather clichΓ© start for documentary filmmakers but we must allow that it was the '60s and they were pretty much the ones creating the clichΓ©. Their early films are hard to find in anything better than VHS rip quality, but worth watching: Wikipedia notes their significance in pioneering "action cameras," mounting 16mm cinema cameras to skateboards and surfboards, but I would say that their cinematography was innovative in more ways than just one. The 1970 "Catch the Joy," about sandrails, has some incredible shots that I struggle to explain. There's at least one where they definitely cut the shot just a couple of frames before a drifting sandrail flung their camera all the way down the dune.

For some reason, I would speculate due to their reputation for exciting cinematography, the National Air and Space Museum chose MacGillivray and Freeman for "To Fly!". While not the first science museum IMAX documentary by any means (that was, presumably, "Voyage to the Outer Planets" given the different subject matter of the various Expo films), "To Fly!" might be called the first modern one. It set the pattern that decades of science museum films followed: a film initially written by science educators, punched up by producers, and filmed with the very best technology of the time. Fearing that the film's history content would be dry, they pivoted more towards entertainment, adding jokes and action sequences. "To Fly!" was a hit, running in just about every science museum with an IMAX theater, including Omnimax.

Sadly, Jim Freeman died in a helicopter crash shortly after production. Nonetheless, MacGillivray Freeman Films went on. Over the following decades, few IMAX science documentaries were made that didn't involve them somehow. Besides the films they produced, the company consulted on action sequences in most of the format's popular features.

Omnimax projection room at OMSI

I had hoped to present here a thorough history of the films that were actually produced in the Omnimax format. Unfortunately, this has proven very difficult: the fact that most of them were distributed only to science museums means that they are very spottily remembered, and besides, so many of the films that ran in Omnimax theaters were converted from IMAX presentations that it's hard to tell the two apart. I'm disappointed that this part of cinema history isn't better recorded, and I'll continue to put time into the effort. Science museum documentaries don't get a lot of attention, but many of the have involved formidable technical efforts.

Consider, for example, the cameras: befitting the large film, IMAX cameras themselves are very large. When filming "To Fly!", MacGillivray and Freeman complained that the technically very basic 80 pound cameras required a lot of maintenance, were complex to operate, and wouldn't fit into the "action cam" mounting positions they were used to. The cameras were so expensive, and so rare, that they had to be far more conservative than their usual approach out of fear of damaging a camera they would not be able to replace. It turns out that they had it easy. Later IMAX science documentaries would be filmed in space ("The Dream is Alive" among others) and deep underwater ("Deep Sea 3D" among others). These IMAX cameras, modified for simpler operation and housed for such difficult environments, weighed over 1,000 pounds. Astronauts had to be trained to operate the cameras; mission specialists on Hubble service missions had wrangling a 70-pound handheld IMAX camera around the cabin and developing its film in a darkroom bag among their duties. There was a lot of film to handle: as a rule of thumb, one mile of IMAX film is good for eight and a half minutes.

I grew up in Portland, Oregon, and so we will make things a bit more approachable by focusing on one example: The Omnimax theater of the Oregon Museum of Science and Industry, which opened as part of the museum's new waterfront location in 1992. This 330-seat boasted a 10,000 sq ft dome and 15 kW of sound. The premier feature was "Ring of Fire," a volcano documentary originally commissioned by the Fleet, the Fort Worth Museum of Science and Industry, and the Science Museum of Minnesota. By the 1990s, the later era of Omnimax, the dome format was all but abandoned as a commercial concept. There were, an announcement article notes, around 90 total IMAX theaters (including Omnimax) and 80 Omnimax films (including those converted from IMAX) in '92. Considering the heavy bias towards science museums among these theaters, it was very common for the films to be funded by consortia of those museums.

Considering the high cost of filming in IMAX, a lot of the documentaries had a sort of "mashup" feel. They would combine footage taken in different times and places, often originally for other projects, into a new narrative. "Ring of Fire" was no exception, consisting of a series of sections that were sometimes more loosely connected to the theme. The 1982 Loma Prieta earthquake was a focus, and the eruption of Mt. St. Helens, and lava flows in Hawaii. Perhaps one of the reasons it's hard to catalog IMAX films is this mashup quality, many of the titles carried at science museums were something along the lines of "another ocean one." I don't mean this as a criticism, many of the IMAX documentaries were excellent, but they were necessarily composed from painstakingly gathered fragments and had to cover wide topics.

Given that I have an announcement feature piece in front of me, let's also use the example of OMSI to discuss the technical aspects. OMSI's projector cost about $2 million and weighted about two tons. To avoid dust damaging the expensive prints, the "projection room" under the seating was a positive-pressure cleanroom. This was especially important since the paucity of Omnimax content meant that many films ran regularly for years. The 15 kW water-cooled lamp required replacement at 800 to 1,000 hours, but unfortunately, the price is not noted.

By the 1990s, Omnimax had become a rare enough system that the projection technology was a major part of the appeal. OMSI's installation, like most later Omnimax theaters, had the audience queue below the seating, separated from the projection room by a glass wall. The high cost of these theaters meant that they operated on high turnovers, so patrons would wait in line to enter immediately after the previous showing had exited. While they waited, they could watch the projectionist prepare the next show while a museum docent explained the equipment.

I have written before about multi-channel audio formats, and Omnimax gives us some more to consider. The conventional audio format for much of Omnimax's life was six-channel: left rear, left screen, center screen, right screen, right rear, and top. Each channel had an independent bass cabinet (in one theater, a "caravan-sized" enclosure with eight JBL 2245H 46cm woofers), and a crossover network fed the lowest end of all six channels to a "sub-bass" array at screen bottom. The original Fleet installation also had sub-bass speakers located beneath the audience seating, although that doesn't seem to have become common.

IMAX titles of the '70s and '80s delivered audio on eight-track magnetic tape, with the additional tracks used for synchronization to the film. By the '90s, IMAX had switched to distributing digital audio on three CDs (one for each two channels). OMSI's theater was equipped for both, and the announcement amusingly notes the availability of cassette decks. A semi-custom audio processor made for IMAX, the Sonics TAC-86, managed synchronization with film playback and applied equalization curves individually calibrated to the theater.

IMAX domes used perforated aluminum screens (also the norm in later planetaria), so the speakers were placed behind the screen in the scaffold-like superstructure that supported it. When I was young, OMSI used to start presentations with a demo program that explained the large size of IMAX film before illuminating work lights behind the screen to make the speakers visible. Much of this was the work of the surprisingly sophisticated show control system employed by Omnimax theaters, a descendent of the PDP-15 originally installed in the Fleet.

Despite Omnimax's almost complete consignment to science museums, there were some efforts at bringing commercial films. Titles like Disney's "Fantasia" and "Star Wars: Episode III" were distributed to Omnimax theaters via optical reprojection, sometimes even from 35mm originals. Unfortunately, the quality of these adaptations was rarely satisfactory, and the short runtimes (and marketing and exclusivity deals) typical of major commercial releases did not always work well with science museum schedules. Still, the cost of converting an existing film to dome format is pretty low, so the practice continues today. "Star Wars: The Force Awakens," for example, ran on at least one science museum dome. This trickle of blockbusters was not enough to make commercial Omnimax theaters viable.

Caesars Palace closed, and then demolished, their Omnimax theater in 2000. The turn of the 21st century was very much the beginning of the end for the dome theater. IMAX was moving away from their film system and towards digital projection, but digital projection systems suitable for large domes were still a nascent technology and extremely expensive. The end of aggressive support from IMAX meant that filming costs became impractical for documentaries, so while some significant IMAX science museum films were made in the 2000s, the volume definitely began to lull and the overall industry moved away from IMAX in general and Omnimax especially.

It's surprising how unforeseen this was, at least to some. A ten-screen commercial theater in Duluth opened an Omnimax theater in 1996! Perhaps due to the sunk cost, it ran until 2010, not a bad closing date for an Omnimax theater. Science museums, with their relatively tight budgets and less competitive nature, did tend to hold over existing Omnimax installations well past their prime. Unfortunately, many didn't: OMSI, for example, closed its Omnimax theater in 2013 for replacement with a conventional digital theater that has a large screen but is not IMAX branded.

Fortunately, some operators hung onto their increasingly costly Omnimax domes long enough for modernization to become practical. The IMAX Corporation abandoned the Omnimax name as more of the theaters closed, but continued to support "IMAX Dome" with the introduction of a digital laser projector with spherical optics. There are only ten examples of this system. Others, including Omnimax's flagship at the Fleet Science Center, have been replaced by custom dome projection systems built by competitors like Sony.

Few Omnimax projectors remain. The Fleet, to their credit, installed the modern laser projectors in front of the projector well so that the original film projector could remain in place. It's still functional and used for reprisals of Omnimax-era documentaries. IMAX projectors in general are a dying breed, a number of them have been preserved but their complex, specialized design and the end of vendor support means that it may become infeasible to keep them operating.

We are, of course, well into the digital era. While far from inexpensive, digital projection systems are now able to match the quality of Omnimax projection. The newest dome theaters, like the Sphere, dispense with projection entirely. Instead, they use LED display panels capable of far brighter and more vivid images than projection, and with none of the complexity of water-cooled arc lamps.

Still, something has been lost. There was once a parallel theater industry, a world with none of the glamor of Hollywood but for whom James Cameron hauled a camera to the depths of the ocean and Leonardo DiCaprio narrated repairs to the Hubble. In a good few dozen science museums, two-ton behemoths rose from beneath the seats, the zenith of film projection technology. After decades of documentaries, I think people forgot how remarkable these theaters were.

Science museums stopped promoting them as aggressively, and much of the showmanship faded away. Sometime in the 2000s, OMSI stopped running the pre-show demonstration, instead starting the film directly. They stopped explaining the projectionist's work in preparing the show, and as they shifted their schedule towards direct repetition of one feature, there was less for the projectionist to do anyway. It became just another museum theater, so it's no wonder that they replaced it with just another museum theater: a generic big-screen setup with the exceptionally dull name of "Empirical Theater."

From time to time, there have been whispers of a resurgence of 70mm film. Oppenheimer, for example, was distributed to a small number of theaters in this giant of film formats: 53 reels, 11 miles, 600 pounds of film. Even conventional IMAX is too costly for the modern theater industry, though. Omnimax has fallen completely by the wayside, with the few remaining dome operators doomed to recycling the same films with a sprinkling of newer reformatted features. It is hard to imagine a collective of science museums sending another film camera to space.

Omnimax poses a preservation challenge in more ways than one. Besides the lack of documentation on Omnimax theaters and films, there are precious few photographs of Omnimax theaters and even fewer videos of their presentations. Of course, the historian suffers where Madison Square Garden hopes to succeed: the dome theater is perhaps the ultimate in location-based entertainment. Photos and videos, represented on a flat screen, cannot reproduce the experience of the Omnimax theater. The 180 horizontal degrees of screen, the sound that was always a little too loud, in no small part to mask the sound of the projector that made its own racket in the middle of the seating. You had to be there.

Omnimax projector at St. Louis Science Center

IMAGES: Omnimax projection room at OMSI, Flickr user truk. Omnimax dome with work lights on at MSI Chicago, Wikimedia Commons user GualdimG. Omnimax projector at St. Louis Science Center, Flickr user pasa47.

[1] I don't have extensive information on pricing, but I know that in the 1960s an "economy" Spitz came in over $30,000 (~10x that much today).

[2] Pink Floyd's landmark album Dark Side of The Moon debuted in a release event held at the London Planetarium. This connection between Pink Floyd and planetaria, apparently much disliked by the band itself, has persisted to the present day. Several generations of Pink Floyd laser shows have been licensed by science museums around the world, and must represent by far the largest success of fixed-installation laser projection.

[3] Are you starting to detect a theme with these Expos? the World's Fairs, including in their various forms as Expos, were long one of the main markets for niche film formats. Any given weird projection format you run into, there's a decent chance that it was originally developed for some short film for an Expo. Keep in mind that it's the nature of niche projection formats that they cannot easily be shown in conventional theaters, so they end up coupled to these crowd events where a custom venue can be built.

[4] The Smithsonian Institution started looking for an exciting new theater in 1970. As an example of the various niche film formats at the time, the Smithsonian considered a dome (presumably Omnimax), Cinerama (a three-projector ultrawide system), and Circle-Vision 360 (known mostly for the few surviving Expo films at Disney World's EPCOT) before settling on IMAX. The Smithsonian theater, first planned for the Smithsonian Museum of Natural History before being integrated into the new National Air and Space Museum, was tremendously influential on the broader world of science museum films. That is perhaps an understatement, it is sometimes credited with popularizing IMAX in general, and the newspaper coverage the new theater received throughout North America lends credence to the idea. It is interesting, then, to imagine how different our world would be if they had chosen Circle-Vision. "Captain America: Brave New World" in Cinemark 360.

2025-05-27 the first smart homes

27 May 2025 at 00:00

Sometimes I think I should pivot my career to home automation critic, because I have many opinions on the state of the home automation industry---and they're pretty much all critical. Virtually every time I bring up home automation, someone says something about the superiority of the light switch. Controlling lights is one of the most obvious applications of home automation, and there is a roughly century long history of developments in light control---yet, paradoxically, it is an area where consumer home automation continues to struggle.

An analysis of how and why billion-dollar tech companies fail to master the simple toggling of lights in response to human input will have to wait for a future article, because I will have a hard time writing one without descending into incoherent sobbing about the principles of scene control and the interests of capital. Instead, I want to just dip a toe into the troubled waters of "smart lighting" by looking at one of its earliest precedents: low-voltage lighting control.

A source I generally trust, the venerable "old internet" website Inspectapedia, says that low-voltage lighting control systems date back to about 1946. The earliest conclusive evidence I can find of these systems is a newspaper ad from 1948, but let's be honest, it's a holiday and I'm only making a half effort on the research. In any case, the post-war timing is not a coincidence. The late 1940s were a period of both rapid (sub)urban expansion and high copper prices, and the original impetus for relay systems seems to have been the confluence of these two.

But let's step back and explain what a relay or low-voltage lighting control system is. First, I am not referring to "low voltage lighting" meaning lights that run on 12 or 24 volts DC or AC, as was common in landscape lighting and is increasingly common today for integrated LED lighting. Low-voltage lighting control systems are used for conventional 120VAC lights. In the most traditional construction, e.g. in the 1940s, lights would be served by a "hot" wire that passed through a wall box containing a switch. In many cases the neutral (likely shared with other fixtures) went directly from the light back to the panel, bypassing the switch... running both the hot and neutral through the switch box did not become conventional until fairly recently, to the chagrin of anyone installing switches that require a neutral for their own power, like timers or "smart" switches.

The problem with this is that it lengthens the wiring runs. If you have a ceiling fixture with two different switches in a three-way arrangement, say in a hallway in a larger house, you could be adding nearly 100' in additional wire to get the hot to the switches and the runner between them. The cost of that wiring, in the mid-century, was quite substantial. Considering how difficult it is to find an employee to unlock the Romex cage at Lowes these days, I'm not sure that's changed that much.

There are different ways of dealing with this. In the UK, the "ring main" served in part to reduce the gauge (and thus cost) of outlet wiring, but we never picked up that particular eccentricity in the US (for good reason). In commercial buildings, it's not unusual for lighting to run on 240v for similar reasons, but 240v is discouraged in US residential wiring. Besides, the mid-century was an age of optimism and ambition in electrical technology, the days of Total Electric Living. Perhaps the technology of the relay, refined by so many innovations of WWII, could offer a solution.

Switch wiring also had to run through wall cavities, an irritating requirement in single-floor houses where much of the lighting wiring could be contained to the attic. The wiring of four-way and other multi-switch arrangements could become complex and require a lot more wall runs, discouraging builders providing switches in the most convenient places. What if relays also made multiple switches significantly easier to install and relocate?

You probably get the idea. In a typical low-voltage lighting control system, a transformer provides a low voltage like 24VAC, much the same as used by doorbells. The light switches simply toggle the 24VAC control power to the coils of relays. Some (generally older) systems powered the relay continuously, but most used latching relays. In this case, all light switches are momentary, with an "on" side and an "off" side. This could be a paddle that you push up or down (much like a conventional light switch), a bar that you push the left or right sides of, or a pair of two push buttons.

In most installations, all of the relays were installed together in a single enclosure, usually in the attic where the high-voltage wiring to the actual lights would be fairly short. The 24VAC cabling to the switches was much smaller gauge, and depending on the jurisdiction might not require any sort of license to install.

Many systems had enclosures with separate high voltage and low voltage components, or mounted the relays on the outside of an enclosure such that the high voltage wiring was inside and low voltage outside. Both arrangements helped to meet code requirements for isolating high and low voltage systems and provided a margin of safety in the low voltage wiring. That provided additional cost savings as well; low voltage wiring was usually installed without any kind of conduit or sheathed cable.

By 1950, relay lighting controls were making common appearances in real estate listings. A feature piece on the "Melody House," a builder's model home, in the Tacoma News Tribune reads thus:

Newest features in the house are the low voltage touch plate and relay system lighting controls, with wide plates instead of snap buttons---operated like the stops of a pipe organ, with the merest flick of a finger.

The comparison to a pipe organ is interesting, first in its assumption that many readers were familiar with typical organ stops. Pipe organs were, increasingly, one of the technological marvels of the era: while the concept of the pipe organ is very old, this same era saw electrical control systems (replete with relays!) significantly reduce the cost and complexity of organ consoles. What's more, the tonewheel electric organ had become well-developed and started to find its way into homes.

The comparison is also interesting because of its deficiencies. The Touch-Plate system described used wide bars, which you pressed the left or right side of---you could call them momentary SPDT rocker switches if you wanted. There were organs with similar rocker stops but I do not think they were common in 1950. My experience is that such rocker switch stops usually indicate a fully digital control system, where they make momentary action unobtrusive and avoid state synchronization problems. I am far from an expert on organs, though, which is why I haven't yet written about them. If you have a guess at which type of pipe organ console our journalist was familiar with, do let me know.

Touch-Plate seems to have been one of the first manufacturers of these systems, although I can't say for sure that they invented them. Interestingly, Touch-Plate is still around today, but their badly broken WordPress site ("Welcome to the new touch-plate.com" despite it actually being touchplate.com) suggests they may not do much business. After a few pageloads their WordPress plugin WAF blocked me for "exceed[ing] the maximum number of page not found errors per minute for humans." This might be related to my frustration that none of the product images load. It seems that the Touch-Plate company has mostly pivoted to reselling imported LED lighting (touchplateled.com), so I suppose the controls business is withering on the vine.

The 1950s saw a proliferation of relay lighting control brands, with GE introducing a particularly popular system with several generations of fixtures. Kyle Switch Plates, who sell replacement switch plates (what else?), list options for Remcon, Sierra, Bryant, Pyramid, Douglas, and Enercon systems in addition to the two brands we have met so far. As someone who pays a little too much attention to light switches, I have personally seen four of these brands, three of them still in use and one apparently abandoned in place.

Now, you might be thinking that simply economizing wiring by relocating the switches does not constitute "home automation," but there are other features to consider. For one, low-voltage light control systems made it feasible to install a lot more switches. Houses originally built with them often go a little wild with the n-way switching, every room providing lightswitches at every door. But there is also the possibility of relay logic. From the same article:

The necessary switches are found in every room, but in the master bedroom there is a master control panel above the bed, from where the house and yard may be flooded with instant light in case of night emergency.

Such "master control panels" were a big attraction for relay lighting, and the finest homes of the 1950s and 1960s often displayed either a grid of buttons near the head of the master bed, or even better, a GE "Master Selector" with a curious system of rotary switches. On later systems, timers often served as auxiliary switches, so you could schedule exterior lights. With a creative installer, "scenes" were even possible by wiring switches to arbitrary sets of relays (this required DC or half-wave rectified control power and diodes to isolate the switches from each other).

Many of these relay control systems are still in use today. While they are quite outdated in a certain sense, the design is robust and the simple components mean that it's usually not difficult to find replacement parts when something does fail. The most popular system is the one offered by GE, using their RR series relays (RR3, RR4, etc., to the modern RR9). That said, GE suggests a modernization path to their LightSweep system, which is really a 0-10v analog dimming controller that has the add-on ability to operate relays.

The failure modes are mostly what you would expect: low voltage wiring can chafe and short, or the switches can become stuck. This tends to cause the lights to stick on or off, and the continuous current through the relay coil often burns it out. The fix requires finding the stuck switch or short and correcting it, and then replacing the relay.

One upside of these systems that persists today is density: the low voltage switches are small, so with most systems you can fit 3 per gang. Another is that they still make N-way switching easier. There is arguably a safety benefit, considering the reduction in mains-voltage wire runs.

Yet we rarely see such a thing installed in homes newer than around the '80s. I don't know that I can give a definitive explanation of the decline of relay lighting control, but reduced prices for copper wiring were probably a main factor. The relays added a failure point, which might lead to a perception of unreliability, and the declining familiarity of electricians means that installing a relay system could be expensive and frustrating today.

What really interests me about relay systems is that they weren't really replaced... the idea just went away. It's not like modern homes are providing a master control panel in the bedroom using some alternative technology. I mean, some do, those with prices in the eight digits, but you'll hardly ever see it.

That gets us to the tension between residential lighting and architectural lighting control systems. In higher-end commercial buildings, and in environments like conference rooms and lecture halls, there's a well established industry building digital lighting control systems. Today, DALI is a common standard for the actual lighting control, but if you look at a range of existing buildings you will find everything from completely proprietary digital distributed dimming to 0-10v analog dimming to central dimmer racks (similar to traditional theatrical lighting).

Relay lighting systems were, in a way, a nascent version of residential architectural lighting control. And the architectural lighting control industry continues to evolve. If there is a modern equivalent to relay lighting, it's something like Lutron QSX. That's a proprietary digital lighting (and shade) control system, marketed for both residential and commercial use. QSX offers a wide range of attractive wall controls, tight integration to Lutron's HomeSense home automation platform, and a price tag that'll make your eyes water. Lutron has produced many generations of these systems, and you could make an argument that they trace their heritage back to the relay systems of the 1940s. But they're just priced way beyond the middle-class home.

And, well, I suppose that requires an argument based on economics. Prices have gone up. Despite tract construction being a much older idea than people often realize, it seems clear that today's new construction homes have been "value engineered" to significantly lower feature and quality levels than those of the mid-century---but they're a lot bigger. There is a sort of maxim that today's home buyers don't care about anything but square footage, and if you've seen what Pulte or D. R. Horton are putting up... well, I never knew that 3,000 sq ft could come so cheap, and look it too.

Modern new-construction homes just don't come with the gizmos that older ones did, especially in the '60s and '70s. Looking at the sales brochure for a new development in my own Albuquerque ("Estates at La Cuentista"), besides 21st century suburbanization (Gated Community! "East Access to Paseo del Norte" as if that's a good thing!) most of the advertised features are "big." I'm serious! If you look at the "More Innovation Built In" section, the "innovations" are a home office (more square footage), storage (more square footage), indoor and outdoor gathering spaces (to be fair, only the indoor ones are square footage), "dedicated learning areas" for kids (more square footage), and a "basement or bigger garage" for a home gym (more square footage). The only thing in the entire innovation section that I would call a "technical" feature is water filtration. You can scroll down for more details, and you get to things like "space for a movie room" and a finished basement described eight different ways.

Things were different during the peak of relay lighting in the '60s. A house might only be 1,600 sq ft, but the builder would deck it out with an intercom (including multi-room audio of a primitive sort), burglar alarm, and yes, relay lighting. All of these technologies were a lot newer and people were more excited about them; I bring up Total Electric Living a lot because of an aesthetic obsession but it was a large-scale advertising and partnership campaign by the electrical industry (particularly Westinghouse) that gave builders additional cross-promotion if they included all of these bells and whistles.

Remember, that was when people were watching those old videos about the "kitchen of the future." What would a 2025 "Kitchen of the Future" promotional film emphasize? An island bigger than my living room and a nook for every meal, I assume. Features like intercoms and even burglar alarms have become far less common in new construction, and even if they were present I don't think most buyers would use them.

But that might seem a little odd, right, given the push towards home automation? Well, built-in home automation options have existed for longer than any of today's consumer solutions, but "built in" is a liability for a technology product. There are practical reasons, in that built-in equipment is harder to replace, but there's also a lamer commercial reason. Consumer technology companies want to sell their products like consumer technology, so they've recontextualized lighting control as "IoT" and "smart" and "AI" rather than something an electrician would hook up.

While I was looking into relay lighting control systems, I ran into an interesting example. The Lutron Lu Master Lumi 5. What a name! Lutron loves naming things like this. The Lumi 5 is a 1980s era product with essentially the same features as a relay system, but architected in a much stranger way. It is, essentially, five three way switches in a box with remote controls. That means that each of the actual light switches in the house (which could also be dimmers) need mains-voltage wiring, including runner, back to the Lumi 5 "interface."

Pressing a button on one of the Lutron wall panels toggles the state of the relay in the "interface" cabinet, toggling the light. But, since it's all wired as a three-way switch, toggling the physical switch at the light does the same thing. As is typical when combining n-way switches and dimming, the Lumi 5 has no control over dimmers. You can only dim a light up or down at the actual local control, the Lumi 5 can just toggle the dimmer on and off using the 3-way runner. The architecture also means that you have two fundamentally different types of wall panels in your house: local switches or dimmers wired to each light, and the Lu Master panels with their five buttons for the five circuits, along with "all on" and "all off."

The Lumi 5 "interface" uses simple relay logic to implement a few more features. Five mains-voltage-level inputs can be wired to time clocks, so that you can schedule any combination(s) of the circuits to turn on and off. The manual recommends models including one with an astronomical clock for sunrise/sunset. An additional input causes all five circuits to turn on; it's suggested for connection to an auxiliary relay on a burglar alarm to turn all of the lights on should the alarm be triggered.

The whole thing is strange and fascinating. It is basically a relay lighting control system, like so many before it, but using a distinctly different wiring convention. I think the main reason for the odd wiring was to accommodate dimmers, an increasingly popular option in the 1980s that relay systems could never really contend with. It doesn't have the cost advantages of relay systems at all, it will definitely be more expensive! But it adds some features over the fancy Lutron switches and dimmers you were going to install anyway.

The Lu Master is the transitional stage between relay lighting systems and later architectural lighting controls, and it straddled too the end of relay light control in homes. It gives an idea of where relay light control in homes would have evolved, had the whole technology not been doomed to the niche zone of conference centers and universities.

If you think about it, the Lu Master fills the most fundamental roles of home automation in lighting: control over multiple lights in a convenient place, scheduling and triggers, and an emergency function. It only lacks scenes, which I think we can excuse considering that the simple technology it uses does not allow it to adjust dimmers. And all of that with no Node-RED in sight!

Maybe that conveys what most frustrates me about the "home automation" industry: it is constantly reinventing the wheel, an oligopoly of tech companies trying to drag people's homes into their "ecosystem." They do so by leveraging the buzzword of the moment, IoT to voice assistants to, I guess now AI?, to solve a basic set of problems that were pretty well solved at least as early as 1948.

That's not to deny that modern home automation platforms have features that old ones don't. They are capable of incredibly sophisticated things! But realistically, most of their users want only very basic functionality: control in convenient places, basic automation, scenes. It wouldn't sting so much if all these whiz-bang general purpose computers were good at those tasks, but they aren't. For the very most basic tasks, things like turning on and off a group of lights, major tech ecosystems like HomeKit provide a user experience that is significantly worse than the model home of 1950.

You could install a Lutron system, and it would solve those fundamental tasks much better... for a much higher price. But it's not like Lutron uses all that money to be an absolute technical powerhouse, a center of innovation at the cutting edge. No, even the latest Lutron products are really very simple, technically. The technical leaders here, Google, Apple, are the companies that can't figure out how to make a damn light switch.

The problem with modern home automation platforms is that they are too ambitious. They are trying to apply enormously complex systems to very simple tasks, and thus contaminating the simplest of electrical systems with all the convenience and ease of a Smart TV.

Sometimes that's what it feels like this whole industry is doing: adding complexity while the core decays. From automatic programming to AI coding agents, video terminals to Electron, the scope of the possible expands while the fundamentals become more and more irritating.

But back to the real point, I hope you learned about some cool light switches. Check out the Kyle Switch Plates reference and you'll start seeing these buildings and homes, at least if you live in an area that built up during the era that they were common (1950s to the 1970s).

2025-05-11 air traffic control

11 May 2025 at 00:00

Air traffic control has been in the news lately, on account of my country's declining ability to do it. Well, that's a long-term trend, resulting from decades of under-investment, severe capture by our increasingly incompetent defense-industrial complex, no small degree of management incompetence in the FAA, and long-lasting effects of Reagan crushing the PATCO strike. But that's just my opinion, you know, maybe airplanes got too woke. In any case, it's an interesting time to consider how weird parts of air traffic control are. The technical, administrative, and social aspects of ATC all seem two notches more complicated than you would expect. ATC is heavily influenced by its peculiar and often accidental development, a product of necessity that perpetually trails behind the need, and a beneficiary of hand-me-down military practices and technology.

Aviation Radio

In the early days of aviation, there was little need for ATC---there just weren't many planes, and technology didn't allow ground-based controllers to do much of value. There was some use of flags and signal lights to clear aircraft to land, but for the most part ATC had to wait for the development of aviation radio. The impetus for that work came mostly from the First World War.

Here we have to note that the history of aviation is very closely intertwined with the history of warfare. Aviation technology has always rapidly advanced during major conflicts, and as we will see, ATC is no exception.

By 1913, the US Army Signal Corps was experimenting with the use of radio to communicate with aircraft. This was pretty early in radio technology, and the aircraft radios were huge and awkward to operate, but it was also early in aviation and "huge and awkward to operate" could be similarly applied to the aircraft of the day. Even so, radio had obvious potential in aviation. The first military application for aircraft was reconnaissance. Pilots could fly past the front to find artillery positions and otherwise provide useful information, and then return with maps. Well, even better than returning with a map was providing the information in real-time, and by the end of the war medium-frequency AM radios were well developed for aircraft.

Radios in aircraft led naturally to another wartime innovation: ground control. Military personnel on the ground used radio to coordinate the schedules and routes of reconnaissance planes, and later to inform on the positions of fighters and other enemy assets. Without any real way to know where the planes were, this was all pretty primitive, but it set the basic pattern that people on the ground could keep track of aircraft and provide useful information.

Post-war, civil aviation rapidly advanced. The early 1920s saw numerous commercial airlines adopting radio, mostly for business purposes like schedule coordination. Once you were in contact with someone on the ground, though, it was only logical to ask about weather and conditions. Many of our modern practices like weather briefings, flight plans, and route clearances originated as more or less formal practices within individual airlines.

Air Mail

The government was not left out of the action. The Post Office operated what may have been the largest commercial aviation operation in the world during the early 1920s, in the form of Air Mail. The Post Office itself did not have any aircraft; all of the flying was contracted out---initially to the Army Air Service, and later to a long list of regional airlines. Air Mail was considered a high priority by the Post Office and proved very popular with the public. When the transcontinental route began proper operation in 1920, it became possible to get a letter from New York City to San Francisco in just 33 hours by transferring it between airplanes in a nearly non-stop relay race.

The Post Office's largesse in contracting the service to private operators provided not only the funding but the very motivation for much of our modern aviation industry. Air travel was not very popular at the time, being loud and uncomfortable, but the mail didn't complain. The many contract mail carriers of the 1920s grew and consolidated into what are now some of the United States' largest companies. For around a decade, the Post Office almost singlehandedly bankrolled civil aviation, and passengers were a side hustle [1].

Air mail ambition was not only of economic benefit. Air mail routes were often longer and more challenging than commercial passenger routes. Transcontinental service required regular flights through sparsely populated parts of the interior, challenging the navigation technology of the time and making rescue of downed pilots a major concern. Notably, air mail operators did far more nighttime flying than any other commercial aviation in the 1920s. The post office became the government's de facto technical leader in civil aviation. Besides the network of beacons and markers built to guide air mail between cities, the post office built 17 Air Mail Radio Stations along the transcontinental route.

The Air Mail Radio Stations were the company radio system for the entire air mail enterprise, and the closest thing to a nationwide, public air traffic control service to then exist. They did not, however, provide what we would now call control. Their role was mainly to provide pilots with information (including, critically, weather reports) and to keep loose tabs on air mail flights so that a disappearance would be noticed in time to send search and rescue.

In 1926, the Watres Act created the Aeronautic Branch of the Department of Commerce. The Aeronautic Branch assumed a number of responsibilities, but one of them was the maintenance of the Air Mail routes. Similarly, the Air Mail Radio Stations became Aeronautics Branch facilities, and took on the new name of Flight Service Stations. No longer just for the contract mail carriers, the Flight Service Stations made up a nationwide network of government-provided services to aviators. They were the first edifices in what we now call the National Airspace System (NAS): a complex combination of physical facilities, technologies, and operating practices that enable safe aviation.

In 1935, the first en-route air traffic control center opened, a facility in Newark owned by a group of airlines. The Aeronautic Branch, since renamed the Bureau of Air Commerce, supported the airlines in developing this new concept of en-route control that used radio communications and paperwork to track which aircraft were in which airways. The rising number of commercial aircraft made in-air collisions a bigger problem, so the Newark control center was quickly followed by more facilities built on the same pattern. In 1936, the Bureau of Air Commerce took ownership of these centers, and ATC became a government function alongside the advisory and safety services provided by the flight service stations.

En route center controllers worked off of position reports from pilots via radio, but needed a way to visualize and track aircraft's positions and their intended flight paths. Several techniques helped: first, airlines shared their flight planning paperwork with the control centers, establishing "flight plans" that corresponded to each aircraft in the sky. Controllers adopted a work aid called a "flight strip," a small piece of paper with the key information about an aircraft's identity and flight plan that could easily be handed between stations. By arranging the flight strips on display boards full of slots, controllers could visualize the ordering of aircraft in terms of altitude and airway.

Second, each center was equipped with a large plotting table map where controllers pushed markers around to correspond to the position reports from aircraft. A small flag on each marker gave the flight number, so it could easily be correlated to a flight strip on one of the boards mounted around the plotting table. This basic concept of air traffic control, of a flight strip and a position marker, is still in use today.

Radar

The Second World War changed aviation more than any other event of history. Among the many advancements were two British inventions of particular significance: first, the jet engine, which would make modern passenger airliners practical. Second, the radar, and more specifically the magnetron. This was a development of such significance that the British government treated it as a secret akin to nuclear weapons; indeed, the UK effectively traded radar technology to the US in exchange for participation in US nuclear weapons research.

Radar created radical new possibilities for air defense, and complimented previous air defense development in Britain. During WWI, the organization tasked with defending London from aerial attack had developed a method called "ground-controlled interception" or GCI. Under GCI, ground-based observers identify possible targets and then direct attack aircraft towards them via radio. The advent of radar made GCI tremendously more powerful, allowing a relatively small number of radar-assisted air defense centers to monitor for inbound attack and then direct defenders with real-time vectors.

In the first implementation, radar stations reported contacts via telephone to "filter centers" that correlated tracks from separate radars to create a unified view of the airspace---drawn in grease pencil on a preprinted map. Filter center staff took radar and visual reports and updated the map by moving the marks. This consolidated information was then provided to air defense bases, once again by telephone.

Later technical developments in the UK made the process more automated. The invention of the "plan position indicator" or PPI, the type of radar scope we are all familiar with today, made the radar far easier to operate and interpret. Radar sets that automatically swept over 360 degrees allowed each radar station to see all activity in its area, rather than just aircraft passing through a defensive line. These new capabilities eliminated the need for much of the manual work: radar stations could see attacking aircraft and defending aircraft on one PPI, and communicated directly with defenders by radio.

It became routine for a radar operator to give a pilot navigation vectors by radio, based on real-time observation of the pilot's position and heading. A controller took strategic command of the airspace, effectively steering the aircraft from a top-down view. The ease and efficiency of this workflow was a significant factor in the end of the Battle of Britain, and its remarkable efficacy was noticed in the US as well.

At the same time, changes were afoot in the US. WWII was tremendously disruptive to civil aviation; while aviation technology rapidly advanced due to wartime needs those same pressing demands lead to a slowdown in nonmilitary activity. A heavy volume of military logistics flights and flight training, as well as growing concerns about defending the US from an invasion, meant that ATC was still a priority. A reorganization of the Bureau of Air Commerce replaced it with the Civil Aeronautics Authority, or CAA. The CAA's role greatly expanded as it assumed responsibility for airport control towers and commissioned new en route centers.

As WWII came to a close, CAA en route control centers began to adopt GCI techniques. By 1955, the name Air Route Traffic Control Center (ARTCC) had been adopted for en route centers and the first air surveillance radars were installed. In a radar-equipped ARTCC, the map where controllers pushed markers around was replaced with a large tabletop PPI built to a Navy design. The controllers still pushed markers around to track the identities of aircraft, but they moved them based on their corresponding radar "blips" instead of radio position reports.

Air Defense

After WWII, post-war prosperity and wartime technology like the jet engine lead to huge growth in commercial aviation. During the 1950s, radar was adopted by more and more ATC facilities (both "terminal" at airports and "en route" at ARTCCs), but there were few major changes in ATC procedure. With more and more planes in the air, tracking flight plans and their corresponding positions became labor intensive and error-prone. A particular problem was the increasing range and speed of aircraft, and corresponding longer passenger flights, that meant that many aircraft passed from the territory of one ARTCC into another. This required that controllers "hand off" the aircraft, informing the "next" ARTCC of the flight plan and position at which the aircraft would enter their airspace.

In 1956, 128 people died in a mid-air collision of two commercial airliners over the Grand Canyon. In 1958, 49 people died when a military fighter struck a commercial airliner over Nevada. These were not the only such incidents in the mid-1950s, and public trust in aviation started to decline. Something had to be done. First, in 1958 the CAA gave way to the Federal Aviation Administration. This was more than just a name change: the FAA's authority was greatly increased compared to the CAA, most notably by granting it authority over military aviation.

This is a difficult topic to explain succinctly, so I will only give broad strokes. Prior to 1958, military aviation was completely distinct from civil aviation, with no coordination and often no communication at all between the two. This was, of course, a factor in the 1958 collision. Further, the 1956 collision, while it did not involve the military, did result in part from communications issues between separate distinct CAA facilities and the airline's own control facilities. After 1958, ATC was completely unified into one organization, the FAA, which assumed the work of the military controllers of the time and some of the role of the airlines. The military continues to have its own air controllers to this day, and military aircraft continue to include privileges such as (practical but not legal) exemption from transponder requirements, but military flights over the US are still beholden to the same ATC as civil flights. Some exceptions apply, void where prohibited, etc.

The FAA's suddenly increased scope only made the practical challenges of ATC more difficult, and commercial aviation numbers continued to rise. As soon as the FAA was formed, it was understood that there needed to be major investments in improving the National Airspace System. While the first couple of years were dominated by the transition, the FAA's second director (Najeeb Halaby) prepared two lengthy reports examining the situation and recommending improvements. One of these, the Beacon report (also called Project Beacon), specifically addressed ATC. The Beacon report's recommendations included massive expansion of radar-based control (called "positive control" because of the controller's access to real-time feedback on aircraft movements) and new control procedures for airways and airports. Even better, for our purposes, it recommended the adoption of general-purpose computers and software to automate ATC functions.

Meanwhile, the Cold War was heating up. US air defense, a minor concern in the few short years after WWII, became a higher priority than ever before. The Soviet Union had long-range aircraft capable of reaching the United States, and nuclear weapons meant that only a few such aircraft had to make it to cause massive destruction. Considering the vast size of the United States (and, considering the new unified air defense command between the United States and Canada, all of North America) made this a formidable challenge.

During the 1950s, the newly minted Air Force worked closely with MIT's Lincoln Laboratory (an important center of radar research) and IBM to design a computerized, integrated, networked system for GCI. When the Air Force committed to purchasing the system, it was christened the Semi-Automated Ground Environment, or SAGE. SAGE is a critical juncture in the history of the computer and computer communications, the first system to demonstrate many parts of modern computer technology and, moreover, perhaps the first large-scale computer system of any kind.

SAGE is an expansive topic that I will not take on here; I'm sure it will be the focus of a future article but it's a pretty well-known and well-covered topic. I have not so far felt like I had much new to contribute, despite it being the first item on my "list of topics" for the last five years. But one of the things I want to tell you about SAGE, that is perhaps not so well known, is that SAGE was not used for ATC. SAGE was a purely military system. It was commissioned by the Air Force, and its numerous operating facilities (called "direction centers") were located on Air Force bases along with the interceptor forces they would direct.

However, there was obvious overlap between the functionality of SAGE and the needs of ATC. SAGE direction centers continuously received tracks from remote data sites using modems over leased telephone lines, and automatically correlated multiple radar tracks to a single aircraft. Once an operator entered information about an aircraft, SAGE stored that information for retrieval by other radar operators. When an aircraft with associated data passed from the territory of one direction center to another, the aircraft's position and related information were automatically transmitted to the next direction center by modem.

One of the key demands of air defense is the identification of aircraft---any unknown track might be routine commercial activity, or it could be an inbound attack. The air defense command received flight plan data on commercial flights (and more broadly all flights entering North America) from the FAA and entered them into SAGE, allowing radar operators to retrieve "flight strip" data on any aircraft on their scope.

Recognizing this interconnection with ATC, as soon as SAGE direction centers were being installed the Air Force started work on an upgrade called SAGE Air Traffic Integration, or SATIN. SATIN would extend SAGE to serve the ATC use-case as well, providing SAGE consoles directly in ARTCCs and enhancing SAGE to perform non-military safety functions like conflict warning and forward projection of flight plans for scheduling. Flight strips would be replaced by teletype output, and in general made less necessary by the computer's ability to filter the radar scope.

Experimental trial installations were made, and the FAA participated readily in the research efforts. Enhancement of SAGE to meet ATC requirements seemed likely to meet the Beacon report's recommendations and radically improve ARTCC operations, sooner and cheaper than development of an FAA-specific system.

As it happened, well, it didn't happen. SATIN became interconnected with another planned SAGE upgrade to the Super Combat Centers (SCC), deep underground combat command centers with greatly enhanced SAGE computer equipment. SATIN and SCC planners were so confident that the last three Air Defense Sectors scheduled for SAGE installation, including my own Albuquerque, were delayed under the assumption that the improved SATIN/SCC equipment should be installed instead of the soon-obsolete original system. SCC cost estimates ballooned, and the program's ambitions were reduced month by month until it was canceled entirely in 1960. Albuquerque never got a SAGE installation, and the Albuquerque air defense sector was eliminated by reorganization later in 1960 anyway.

Flight Service Stations

Remember those Flight Service Stations, the ones that were originally built by the Post Office? One of the oddities of ATC is that they never went away. FSS were transferred to the CAB, to the CAA, and then to the FAA. During the 1930s and 1940s many more were built, expanding coverage across much of the country.

Throughout the development of ATC, the FSS remained responsible for non-control functions like weather briefing and flight plan management. Because aircraft operating under instrument flight rules must closely comply with ATC, the involvement of FSS in IFR flights is very limited, and FSS mostly serve VFR traffic.

As ATC became common, the FSS gained a new and somewhat odd role: playing go-between for ATC. FSS were more numerous and often located in sparser areas between cities (while ATC facilities tended to be in cities), so especially in the mid-century, pilots were more likely to be able to reach an FSS than ATC. It was, for a time, routine for FSS to relay instructions between pilots and controllers. This is still done today, although improved communications have made the need much less common.

As weather dissemination improved (another topic for a future post), FSS gained access to extensive weather conditions and forecasting information from the Weather Service. This connectivity is bidirectional; during the midcentury FSS not only received weather forecasts by teletype but transmitted pilot reports of weather conditions back to the Weather Service. Today these communications have, of course, been computerized, although the legacy teletype format doggedly persists.

There has always been an odd schism between the FSS and ATC: they are operated by different departments, out of different facilities, with different functions and operating practices. In 2005, the FAA cut costs by privatizing the FSS function entirely. Flight service is now operated by Leidos, one of the largest government contractors. All FSS operations have been centralized to one facility that communicates via remote radio sites.

While flight service is still available, increasing automation has made the stations far less important, and the general perception is that flight service is in its last years. Last I looked, Leidos was not hiring for flight service and the expectation was that they would never hire again, retiring the service along with its staff.

Flight service does maintain one of my favorite internet phenomenon, the phone number domain name: 1800wxbrief.com. One of the odd manifestations of the FSS/ATC schism and the FAA's very partial privatization is that Leidos maintains an online aviation weather portal that is separate from, and competes with, the Weather Service's aviationweather.gov. Since Flight Service traditionally has the responsibility for weather briefings, it is honestly unclear to what extent Leidos vs. the National Weather Service should be investing in aviation weather information services. For its part, the FAA seems to consider aviationweather.gov the official source, while it pays for 1800wxbrief.com. There's also weathercams.faa.gov, which duplicates a very large portion (maybe all?) of the weather information on Leidos's portal and some of the NWS's. It's just one of those things. Or three of those things, rather. Speaking of duplication due to poor planning...

The National Airspace System

Left in the lurch by the Air Force, the FAA launched its own program for ATC automation. While the Air Force was deploying SAGE, the FAA had mostly been waiting, and various ARTCCs had adopted a hodgepodge of methods ranging from one-off computer systems to completely paper-based tracking. By 1960 radar was ubiquitous, but different radar systems were used at different facilities, and correlation between radar contacts and flight plans was completely manual. The FAA needed something better, and with growing congressional support for ATC modernization, they had the money to fund what they called National Airspace System En Route Stage A.

Further bolstering historical confusion between SAGE and ATC, the FAA decided on a practical, if ironic, solution: buy their own SAGE.

In an upcoming article, we'll learn about the FAA's first fully integrated computerized air traffic control system. While the failed detour through SATIN delayed the development of this system, the nearly decade-long delay between the design of SAGE and the FAA's contract allowed significant technical improvements. This "New SAGE," while directly based on SAGE at a functional level, used later off-the-shelf computer equipment including the IBM System/360, giving it far more resemblance to our modern world of computing than SAGE with its enormous, bespoke AN/FSQ-7.

And we're still dealing with the consequences today!

[1] It also laid the groundwork for the consolidation of the industry, with a 1930 decision that took air mail contracts away from most of the smaller companies and awarded them instead to the precursors of United, TWA, and American Airlines.

2025-05-04 iBeacons

4 May 2025 at 00:00

You know sometimes a technology just sort of... comes and goes? Without leaving much of an impression? And then gets lodged in your brain for the next decade? Let's talk about one of those: the iBeacon.

I think the reason that iBeacons loom so large in my memory is that the technology was announced at WWDC in 2013. Picture yourself in 2013: Steve Jobs had only died a couple of years ago, Apple was still widely viewed as a visionary leader in consumer technology, and WWDC was still happening. Back then, pretty much anything announced at an Apple event was a Big Deal that got Big Coverage. Even, it turns out, if it was a minor development for a niche application. That's the iBeacon, a specific solution to a specific problem. It's not really that interesting, but the valance of it's Apple origin makes it seem cool?

iBeacon Technology

Let's start out with what iBeacon is, as it's so simple as to be underwhelming. Way back in the '00s, a group of vendors developed a sort of "Diet Bluetooth": a wireless protocol that was directly based on Bluetooth but simplified and optimized for low-power, low-data-rate devices. This went through an unfortunate series of names, including the delightful Wibree, but eventually settled on Bluetooth Low Energy (BLE). BLE is not just lower-power, but also easier to implement, so it shows up in all kinds of smart devices today. Back in 2011, it was quite new, and Apple was one of the first vendors to adopt it.

BLE is far less connection-oriented than regular Bluetooth; you may have noticed that BLE devices are often used entirely without conventional "pairing." A lot of typical BLE profiles involve just broadcasting some data into the void for any device that cares (and is in short range) to receive, which is pretty similar to ANT+ and unsurprisingly appears in ANT+-like applications of fitness monitors and other sensors. Of course, despite the simpler association model, BLE applications need some way to find devices, so BLE provides an advertising mechanism in which devices transmit their identifying info at regular intervals.

And that's all iBeacon really is: a standard for very simple BLE devices that do nothing but transmit advertisements with a unique ID as the payload. Add a type field on the advertising packet to specify that the device is trying to be an iBeacon and you're done. You interact with an iBeacon by receiving its advertisements, so you know that you are near it. Any BLE device with advertisements enabled could be used this way, but iBeacons are built only for this purpose.

The applications for iBeacon are pretty much defined by its implementation in iOS; there's not much of a standard even if only for the reason that there's not much to put in a standard. It's all obvious. iOS provides two principle APIs for working with iBeacons: the region monitoring API allows an app to determine if it is near an iBeacon, including registering the region so that the app will be started when the iBeacon enters range. This is useful for apps that want to do something in response to the user being in a specific location.

The ranging API allows an app to get a list of all of the nearby iBeacons and a rough range from the device to the iBeacon. iBeacons can actually operate at substantial ranges---up to hundreds of meters for more powerful beacons with external power, so ranging mode can potentially be used as sort of a lightweight local positioning system to estimate the location of the user within a larger space.

iBeacon IDs are in the format of a UUID, followed by a "major" number and a "minor" number. There are different ways that these get used, especially if you are buying cheap iBeacons and not reconfiguring them, but the general idea is roughly that the UUID identifies the operator, the major a deployment, and the minor a beacon within the deployment. In practice this might be less common than just every beacon having its own UUID due to how they're sourced. It would be interesting to survey iBeacon applications to see which they do.

Promoted Applications

So where do you actually use these? Retail! Apple seems to have designed the iBeacon pretty much exclusively for "proximity marketing" applications in the retail environment. It goes something like this: when you're in a store and open that store's app, the app will know what beacons you are nearby and display relevant content. For example, in a grocery store, the grocer's app might offer e-coupons for cosmetics when you are in the cosmetics section.

That's, uhh, kind of the whole thing? The imagined universe of applications around the launch of iBeacon was pretty underwhelming to me, even at the time, and it still seems that way. That's presumably why iBeacon had so little success in consumer-facing applications. You might wonder, who actually used iBeacons?

Well, Apple did, obviously. During 2013 and into 2014 iBeacons were installed in all US Apple stores, and prompted the Apple Store app to send notifications about upgrade offers and other in-store deals. Unsurprisingly, this Apple Store implementation was considered the flagship deployment. It generated a fair amount of press, including speculation as to whether or not it would prove the concept for other buyers.

Around the same time, Apple penned a deal with Major League Baseball that would see iBeacons installed in MLB stadiums. For the 2014 season, MLB Advanced Marketing, a joint venture of team owners, had installed iBeacon technology in 20 stadiums.

Baseball fans will be able to utilize iBeacon technology within MLB.com At The Ballpark when the award-winning app's 2014 update is released for Opening Day. Complete details on new features being developed by MLBAM for At The Ballpark, including iBeacon capabilities, will be available in March.

What's the point? the iBeacons "enable the At The Ballpark app to play specific videos or offer coupons."

This exact story repeats for other retail companies that have picked the technology up at various points, including giants like Target and WalMart. The iBeacons are simply a way to target advertising based on location, with better indoor precision and lower power consumption than GPS. Aiding these applications along, Apple integrated iBeacon support into the iOS location framework and further blurred the lines between iBeacon and other positioning services by introducing location-based-advertising features that operated on geofencing alone.

Some creative thinkers did develop more complex applications for the iBeacon. One of the early adopters was a company called Exact Editions, which prepared the Apple Newsstand version of a number of major magazines back when "readable on iPad" was thought to be the future of print media. Exact Editions explored a "read for free" feature where partner magazines would be freely accessible to users at partnering locations like coffee shops and book stores. This does not seem to have been a success, but using the proximity of an iBeacon to unlock some paywalled media is at least a little creative, if probably ill-advised considering security considerations we'll discuss later.

The world of applications raises interesting questions about the other half of the mobile ecosystem: how did this all work on Android? iOS has built-in support for iBeacons. An operating system service scans for iBeacons and dispatches notifications to apps as appropriate. On Android, there has never been this type of OS-level support, but Android apps have access to relatively rich low-level Bluetooth functionality and can easily scan for iBeacons themselves. Several popular libraries exist for this purpose, and it's not unusual for them to be used to give ported cross-platform apps more or less equivalent functionality. These apps do need to run in the background if they're to notify the user proactively, but especially back in 2013 Android was far more generous about background work than iOS.

iBeacons found expanded success through ShopKick, a retail loyalty platform that installed iBeacons in locations of some major retailers like American Eagle. These powered location-based advertising and offers in the ShopKick app as well as retailer-specific apps, which is kind of the start of a larger, more seamless network, but it doesn't seem to have caught on. Honestly, consumers just don't seem to want location-based advertising that much. Maybe because, when you're standing in an American Eagle, getting ads for products carried in the American Eagle is inane and irritating. iBeacons sort of foresaw cooler screens in this regard.

To be completely honest, I'm skeptical that anyone ever really believed in the location-based advertising thing. I mean, I don't know, the advertising industry is pretty good at self-deception, but I don't think there were ever any real signs of hyper-local smartphone-based advertising taking off. I think the play was always data collection, and advertising and special offers just provided a convenient cover story.

Real Applications

iBeacons are one of those technologies that feels like a flop from a consumer perspective but has, in actuality, enjoyed surprisingly widespread deployments. The reason, of course, is data mining.

To Apple's credit, they took a set of precautions in the design of the iBeacon iOS features that probably felt sufficient in 2013. Despite the fact that a lot of journalists described iBeacons as being used to "notify a user to install an app," that was never actually a capability (a very similar-seeming iOS feature attached to Siri actually used conventional geofencing rather than iBeacons). iBeacons only did anything if the user already had an app installed that either scanned for iBeacons when in the foreground or registered for region notifications.

In theory, this limited iBeacons to companies with which consumers already had some kind of relationship. What Apple may not have foreseen, or perhaps simply accepted, is the incredible willingness of your typical consumer brand to sell that relationship to anyone who would pay.

iBeacons became, in practice, just another major advancement in pervasive consumer surveillance. The New York Times reported in 2019 that popular applications were including SDKs that reported iBeacon contacts to third-party consumer data brokers. This data became one of several streams that was used to sell consumer location history to advertisers.

It's a little difficult to assign blame and credit, here. Apple, to their credit, kept iBeacon features in iOS relatively locked down. This suggests that they weren't trying to facilitate massive location surveillance. That said, Apple always marketed iBeacon to developers based on exactly this kind of consumer tracking and micro-targeting, they just intended for it to be done under the auspices of a single brand. That industry would obviously form data exchanges and recruit random apps into reporting everything in your proximity isn't surprising, but maybe Apple failed to foresee it.

They certainly weren't the worst offender. Apple's promotion of iBeacon opened the floodgates for everyone else to do the same thing. During 2014 and 2015, Facebook started offering bluetooth beacons to businesses that were ostensibly supposed to facilitate in-app special offers (though I'm not sure that those ever really materialized) but were pretty transparently just a location data collection play.

Google jumped into the fray in their Signature Google style, with an offering that was confusing, semi-secret, incoherently marketed, and short lived. Google's Project Beacon, or Google My Business, also shipped free Bluetooth beacons out to businesses to give Android location services a boost. Google My Business seems to have been the source of a fair amount of confusion even at the time, and we can virtually guarantee that (as reporters speculated at the time) Google was intentionally vague and evasive about the system to avoid negative attention from privacy advocates.

In the case of Facebook, well, they don't have the level of opsec that Google does so things are a little better documented:

Leaked documents show that Facebook worried that users would 'freak out' and spread 'negative memes' about the program. The company recently removed the Facebook Bluetooth beacons section from their website.

The real deployment of iBeacons and closely related third-party iBeacon-like products [1] occurred at massive scale but largely in secret. It became yet another dark project of the advertising-industrial complex, perhaps the most successful yet of a long-running series of retail consumer surveillance systems.

Payments

One interesting thing about iBeacon is how it was compared to NFC. The two really aren't that similar, especially considering the vast difference in usable ranges, but NFC was the first radio technology to be adopted for "location marketing" applications. "Tap your phone to see our menu," kinds of things. Back in 2013, Apple had rather notably not implemented NFC in its products, despite its increasing adoption on Android.

But, there is much more to this story than learning about new iPads and getting a surprise notification that you are eligible for a subsidized iPhone upgrade. What we're seeing is Apple pioneering the way mobile devices can be utilized to make shopping a better experience for consumers. What we're seeing is Apple putting its money where its mouth is when it decided not to support NFC. (MacObserver)

Some commentators viewed iBeacon as Apple's response to NFC, and I think there's more to that than you might think. In early marketing, Apple kept positioning iBeacon for payments. That's a little weird, right, because iBeacons are a purely one-way broadcast system.

Still, part of Apple's flagship iBeacon implementation was a payment system:

Here's how he describes the purchase he made there, using his iPhone and the EasyPay system: "We started by using the iPhone to scan the product barcode and then we had to enter our Apple ID, pretty much the way we would for any online Apple purchase [using the credit card data on file with one's Apple account]. The one key difference was that this transaction ended with a digital receipt, one that we could show to a clerk if anyone stopped us on the way out."

Apple Wallet only kinda-sorta existed at the time, although Apple was clearly already midway into a project to expand into consumer payments. It says a lot about this point in time in phone-based payments that several reporters talk about iBeacon payments as a feature of iTunes, since Apple was mostly implementing general-purpose billing by bolting it onto iTunes accounts.

It seems like what happened is that Apple committed to developing a pay-by-phone solution, but decided against NFC. To be competitive with other entrants in the pay-by-phone market, they had to come up with some kind of technical solution to interact with retail POS, and iBeacon was their choice. From a modern perspective this seems outright insane; like, Bluetooth broadcasts are obviously not the right way to initiate a payment flow, and besides, there's a whole industry-standard stack dedicated to that purpose... built on NFC.

But remember, this was 2013! EMV was not yet in meaningful use in the US; several major banks and payment networks had just committed to rolling it out in 2012 and every American can tell you that the process was long and torturous. Because of the stringent security standards around EMV, Android devices did not implement EMV until ARM secure enclaves became widely available. EMVCo, the industry body behind EMV, did not have a certification program for smartphones until 2016.

Android phones offered several "tap-to-pay" solutions, from Google's frequently rebranded Google Wallet^w^wAndroid Pay^w^wGoogle Wallet to Verizon's embarrassingly rebranded ISIS^wSoftcard and Samsung Pay. All of these initially relied on proprietary NFC protocols with bespoke payment terminal implementations. This was sketchy enough, and few enough phones actually had NFC, that the most successful US pay-by-phone implementations like Walmart's and Starbucks' used barcodes for communication. It would take almost a decade before things really settled down and smartphones all just implemented EMV.

So, in that context, Apple's decision isn't so odd. They must have figured that iBeacon could solve the same "initial handshake" problem as Walmart's QR codes, but more conveniently and using radio hardware that they already included in their phones. iBeacon-based payment flows used the iBeacon only to inform the phone of what payment devices were nearby, everything else happened via interaction with a cloud service or whatever mechanism the payment vendor chose to implement. Apple used their proprietary payments system through what would become your Apple Account, PayPal slapped together an iBeacon-based fast path to PayPal transfers, etc.

I don't think that Apple's iBeacon-based payments solution ever really shipped. It did get some use, most notably by Apple, but these all seem to have been early-stage implementations, and the complete end-to-end SDK that a lot of developers expected never landed.

You might remember that this was a very chaotic time in phone-based payments, solutions were coming and going. When Apple Pay was properly announced a year after iBeacons, there was little mention of Bluetooth. By the time in-store Apple Pay became common, Apple had given up and adopted NFC.

Limitations

One of the great weaknesses of iBeacon was the security design, or lack thereof. iBeacon advertisements were sent in plaintext with no authentication of any type. This did, of course, radically simplify implementation, but it also made iBeacon untrustworthy for any important purpose. It is quite trivial, with a device like an Android phone, to "clone" any iBeacon and transmit its identifiers wherever you want. This problem might have killed off the whole location-based-paywall-unlocking concept had market forces not already done so. It also opens the door to a lot of nuisance attacks on iBeacon-based location marketing, which may have limited the depth of iBeacon features in major apps.

iBeacon was also positioned as a sort of local positioning system, but it really wasn't. iBeacon offers no actual time-of-flight measurements, only RSSI-based estimation of range. Even with correct on-site calibration (which can be aided by adjusting a fixed RSSI-range bias value included in some iBeacon advertisements) this type of estimation is very inaccurate, and in my little experiments with a Bluetooth beacon location library I can see swings from 30m to 70m estimated range based only on how I hold my phone. iBeacon positioning has never been accurate enough to do more than assert whether or not a phone is "near" the beacon, and "near" can take on different values depending on the beacon's transmit power.

Developers have long looked towards Bluetooth as a potential local positioning solution, and it's never quite delivered. The industry is now turning towards Ultra-Wideband or UWB technology, which combines a high-rate, high-bandwidth radio signal with a time-of-flight radio ranging protocol to provide very accurate distance measurements. Apple is, once again, a technical leader in this field and UWB radios have been integrated into the iPhone 11 and later.

Senescence

iBeacon arrived to some fanfare, quietly proliferated in the shadows of the advertising industry, and then faded away. The Wikipedia article on iBeacons hasn't really been updated since support on Windows Phone was relevant. Apple doesn't much talk about iBeacons any more, and their compatriots Facebook and Google both sunset their beacon programs years ago.

Part of the problem is, well, the pervasive surveillance thing. The idea of Bluetooth beacons cooperating with your phone to track your every move proved unpopular with the public, and so progressively tighter privacy restrictions in mobile operating systems and app stores have clamped down on every grocery store app selling location data to whatever broker bids the most. I mean, they still do, but it's gotten harder to use Bluetooth as an aid. Even Android, the platform of "do whatever you want in the background, battery be damned," strongly discourages Bluetooth scanning by non-foreground apps.

Still, the basic technology remains in widespread use. BLE beacons have absolutely proliferated, there are plenty of apps you can use to list nearby beacons and there almost certainly are nearby beacons. One of my cars has, like, four separate BLE beacons going on all the time, related to a phone-based keyless entry system that I don't think the automaker even supports any more. Bluetooth beacons, as a basic primitive, are so useful that they get thrown into all kinds of applications. My earbuds are a BLE beacon, which the (terrible, miserable, no-good) Bose app uses to detect their proximity when they're paired to another device. A lot of smart home devices like light bulbs are beacons. The irony, perhaps, of iBeacon-based location tracking is that it's a victim of its own success. There is so much "background" BLE beacon activity that you scarcely need to add purpose-built beacons to track users, and only privacy measures in mobile operating systems and the beacons themselves (some of which rotate IDs) save us.

Apple is no exception to the widespread use of Bluetooth beacons: iBeacon lives on in virtually every apple device. If you do try out a Bluetooth beacon scanning app, you'll discover pretty much every Apple product in a 30 meter radius. From MacBooks Pro to Airpods, almost all Apple products transmit iBeacon advertisements to their surroundings. These are used for the initial handshake process of peer-to-peer features like Airdrop, and Find My/AirTag technology seems to be derived from the iBeacon protocol (in the sense that anything can be derived from such a straightforward design). Of course, pretty much all of these applications now randomize identifiers to prevent passive use of device advertisements for long-term tracking.

Here's some good news: iBeacons are readily available in a variety of form factors, and they are very cheap. Lots of libraries exist for working with them. If you've ever wanted some sort of location-based behavior for something like home automation, iBeacons might offer a good solution. They're neat, in an old technology way. Retrotech from the different world of 2013.

It's retro in more ways than one. It's funny, and a bit quaint, to read the contemporary privacy concerns around iBeacon. If only they had known how bad things would get! Bluetooth beacons were the least of our concerns.

[1] Things can be a little confusing here because the iBeacon is such a straightforward concept, and Apple's implementation is so simple. We could define "iBeacon" as including only officially endorsed products from Apple affiliates, or as including any device that behaves the same as official products (e.g. by using the iBeacon BLE advertisement type codes), or as any device that is performing substantially the same function (but using a different advertising format). I usually mean the latter of these three as there isn't really much difference between an iBeacon and ten million other BLE beacons that are doing the same thing with a slightly different identifier format. Facebook and Google's efforts fall into this camp.

2025-04-18 white alice

18 April 2025 at 00:00

When we last talked about Troposcatter, it was Pole Vault. Pole Vault was the first troposcatter communications network, on the east coast of Canada. It would not be alone for long. By the time the first Pole Vault stations were complete, work was already underway on a similar network for Alaska: the White Alice Communication System, WACS.

USACE illustration of troposcatter antennas

Alaska has long posed a challenge for communications. In the 1860s, Western Union wanted to extend their telegraph network from the United States into Europe. Although the technology would be demonstrated shortly after, undersea telegraph cables were still notional and it seemed that a route that minimized the ocean crossing would be preferable---of course, that route maximized the length on land, stretching through present-day Alaska and Siberia on each side of the Bering Strait. This task proved more formidable than Western Union had imagined, and the first transatlantic telegraph cable (on a much further south crossing) was completed before the arctic segments of the overland route. The "Western Union Telegraph Expedition" abandoned its work, leaving a telegraph line well into British Columbia that would serve as one of the principle communications assets in the region for decades after.

This ill-fated telegraph line failed to link San Francisco to Moscow, but its aftermath included a much larger impact on Russian interests in North America: the purchase of Alaska in 1867. Shortly after, the US military began its expansion into the new frontier. The Army Signal Corps, mostly to fulfill its function in observing the weather, built and staffed small installations that stretched further and further west. Later, in the 1890s, a gold rush brought a sudden influx of American settlers to Alaska's rugged terrain. The sudden economic importance of Klondike, and the rather colorful personalities of the prospectors looking to exploit it, created a much larger need for military presence. Fortuitously, many of the forts present had been built by the Signal Corps, which had already started on lines of communication. Construction was difficult, though, and without Alaskan communications as major priority there was only minimal coverage.

Things changed in 1900, when Congress appropriated a substantial budget to the Washington-Alaska Military Cable and Telegraph System. The Signal Corps set on Alaska like, well, like an army, and extensive telegraph and later telephone lines were built to link the various military outposts. Later renamed the Alaska Communications System, these cables brought the first telecommunication to much of Alaska. The arrival of the telegraph was quite revolutionary for remote towns, who could now receive news in real-time that had previously been delayed by as much as a year [1]. Telegraphy was important to civilians as well, something that Congress had anticipated: The original act authorizing the Alaska Communications System dictated that it would carry commercial traffic as well. The military had an unusual role in Alaska, and one aspect of it was telecommunications provider.

In 1925, an outbreak of diphtheria began to claim the lives of children in Nome, a town in far western Alaska on the Seward Peninsula. The daring winter delivery of antidiphtheria serum by dog sled is widely remembered due to its tangential connection to the Iditarod, but there were two sides of the "serum run." The message from Nome's sole doctor requesting the urgent shipment was transmitted from Nome to the Public Health Service in DC over the Alaska Communications System. It gives us some perspective on the importance of the telegraph in Alaska that the 600 mile route to Nome took five days and many feats of heroism---but at the same time could be crossed instantaneously by telegrams.

The Alaska Communications System included some use of radio from the beginning. A pair of HF radio stations specifically handled traffic for Nome, covering a 100-mile stretch too difficult for even the intrepid Signal Corps. While not a totally new technology to the military, radio was quite new to the telegraph business, and the ACS to Nome was probably the first commercial radiotelegraph system on the continent. By the 1930s, the condition of the Alaskan telegraph cables had decayed while demand for telephony had increased. Much of ACS was upgraded and modernized to medium-frequency radiotelephone links. In towns small and large, even in Anchorage itself, the sole telephone connection to the contiguous United States was an ACS telephone installed in the general store.

Alaskan communications became an even greater focus of the military with the onset of the Second World War. A few weeks after Pearl Harbor, the Japanese attacked Fort Mears in the Aleutian Islands. Fort Mears had no telecommunications connections, so despite the proximity of other airbases support was slow to come. The lack of a telegraph or telephone line contributed to 43 deaths and focused attention on the ACS. By 1944, the Army Signal Corps had a workforce of 2,000 dedicated to Alaska.

WWII brought more than one kind of attention to Alaska. Several Japanese assaults on the Aleutian islands represented the largest threats to American soil outside of Pearl Harbor, showing both Alaska's vulnerability and the strategic importance given to it by its relative proximity to Eurasia. WWII ended but, in 1949, the USSR demonstrated an atomic weapon. A combination of Soviet expansionism and the new specter of nuclear war turned military planners towards air defense. Like the Canadian Maritimes in the East, Alaska covered a huge swath of the airspace through which Soviet bombers might approach the US. Alaska was, once again, a battleground.

USAF photo

The early Cold War military buildup of Alaska was particularly heavy on air defense. During the late '40s and early '50s, more than a dozen new radar and control sites were built. The doctrine of ground-controlled interception requires real-time communication between radar centers, stressing the limited number of voice channels available on the ACS. As early as 1948, the Signal Corps had begun experiments to choose an upgrade path. Canadian early-warning radar networks, including the Distant Early Warning Line, were on the drawing board and would require many communications channels in particularly remote parts of Alaska.

Initially, point-to-point microwave was used in relatively favorable terrain (where the construction of relay stations about every 50 miles was practical). For the more difficult segments, the Signal Corps found that VHF radio could provide useful communications at ranges over 100 miles. VHF radiotelephones were installed at air defense radar stations, but there was a big problem: the airspace surveillance radar of the 1950s also operated in the VHF band, and caused so much interference with the radiotelephones that they were difficult to use. The radar stations were probably the most important users of the network, so VHF would have to be abandoned.

In 1954, a military study group was formed to evaluate options for the ACS. That group, in turn, requested a proposal from AT&T. Bell Laboratories had been involved in the design and evaluation of Pole Vault, the first sites of which had been completed two years before, so they naturally positioned troposcatter as the best option.

It is worth mentioning the unusual relationship AT&T had with Alaska, or rather, the lack of one. While the Bell System enjoyed a monopoly on telephony in most of the United States [2], they had never expanded into Alaska. Alaska was only a territory, after all, and a very sparsely populated one at that. The paucity of long-distance leads to or from Alaska (only one connected to Anchorage, for example) limited the potential for integration of Alaska into the broader Bell System anyway. Long-distance telecommunications in Alaska were a military project, and AT&T was involved only as a vendor.

Because of the high cost of troposcatter stations, proven during Pole Vault construction, a hybrid was proposed: microwave stations could be spaced every 50 miles along the road network, while troposcatter would cover the long stretches without roads.

In 1955, the Signal Corps awarded Western Electric a contract for the White Alice Communications System. The Corps of Engineers surveyed the locations of 31 sites, verifying each by constructing a temporary antenna tower. The Corps of Engineers led construction of the first 11 sites, and the final 20 were built on contract by Western Electric itself. All sites used radio equipment furnished by Western Electric and were built to Western Electric designs.

Construction was far from straightforward. Difficult conditions delayed completion of the original network until 1959, two years later than intended. A much larger issue, though, was the budget. The original WACS was expected to cost $38 million. By the time the first 31 sites were complete, the bill totaled $113 million---equivalent to over a billion dollars today. Western Electric had underestimated not only the complexity of the sites but the difficulty of their construction. A WECo report read:

On numerous occasions, the men were forced to surrender before the onslaught of cold, wind and snow and were immobilized for days, even weeks . This ordeal of waiting was of times made doubly galling by the knowledge that supplies and parts needed for the job were only a few miles distant but inaccessible because the white wall of winter had become impenetrable

WACS initial capability included 31 stations, of which 22 were troposcatter and the remainder only microwave (using Western Electric's TD-2). A few stations were equipped with both troposcatter and microwave, serving as relays between the two carriers.

In 1958, construction started on the Ballistic Missile Early Warning System or BMEWS. BMEWS was an over-the-horizon radar system intended to provide early warning of a Soviet attack. BMEWS would provide as little as 15 minutes warning, requiring that alerts reach NORAD in Colorado as quickly as possible. One BMEWS set was installed in Greenland, where the Pole Vault system was expanded to provide communications. Similarly, the BMEWS set at Clear Missile Early Warning Station in central Alaska relied on White Alice. Planners were concerned about the ability of the Soviet Union to suppress an alert by destroying infrastructure, so two redundant chains of microwave sites were added to White Alice. One stretched from Clear to Ketchikan where it connected to an undersea cable to Seattle. The other went east, towards Canada, where it met existing telephone cables on the Alaska Highway.

A further expansion of White Alice started the next year, in 1959. Troposcatter sites were extended through the Aleutian islands in "Project Stretchout" to serve new DEW Line stations. During the 1960s, existing WACS sites were expanded and new antennas were installed at Air Force installations. These were generally microwave links connecting the airbases to existing troposcatter stations.

In total, WACS reached 71 sites. Four large sites served as key switching points with multiple radio links and telephone exchanges. Pedro Dome, for example, had a 15,000 square foot communications building with dormitories, a power plant, and extensive equipment rooms. Support facilities included a vehicle maintenance building, storage warehouse, and extensive fuel tanks. A few WACS sites even had tramways for access between the "lower camp" (where equipment and personnel were housed) and the "upper camp" (where the antennas were located)... although they apparently did not fare well in the Alaskan conditions.

While Western Electric had initially planned for six people and 25 KW of power at each station, the final requirements were typically 20 people and 120-180 KW of generator capacity. Some sites stored over half a million gallons of fuel---conditions often meant that resupply was only possible during the summer.

Besides troposcatter and microwave radios, the equipment included tandem telephone exchanges. These are described in a couple of documents as "ATSS-4A," ATSS standing for Alaska Telephone Switching System. Based on the naming and some circumstantial evidence, I believe these were Western Electric 4A crossbar exchanges. They were later incorporated into AUTOVON, but also handled commercial long-distance traffic between Alaskan towns.

With troposcatter comes large antennas, and depending on connection lengths, WACS troposcatter antennas ranged from 30' dishes to 120' "billboard" antennas similar to those seen at Pole Vault sites. The larger antennas handled up to 50kW of transmit power. Some 60' and 120' antennas included their own fuel tanks and steam plants that heated the antennas through winter to minimize snow accumulation.

Nearly all of the equipment used by WACS was manufactured by Western Electric, with a lot of reuse of standard telephone equipment. For example, muxing on the troposcatter links used standard K-carrier (originally for telephone cables) and L-carrier (originally for coaxial cables). Troposcatter links operated at about 900 MHz with a wide bandwidth, and carrier two L-carrier supergroups (60 channels) and one K-carrier (12 channels) for a nominal capacity of 132 channels, although most stations did not have fully-populated L-carrier groups so actual capacity varied based on need. This was standard telephone carrier equipment in widespread use on the long-distance network, but some output modifications were made to suit the radio application.

The exception to the Western Electric rule was the radio sets themselves. They were manufactured by Radio Engineering Laboratories, the same company that built the radios for Pole Vault. REL pulled out all of the tricks they had developed for Pole Vault, and the longer WACS links used two antennas at different positions for space diversity. Each antenna had two feed horns, of orthoganal polarization, matching similar dual-transmitters for further diversity. REL equipment selected the best signal of the four available receiver options.

USAF photo

WACS represented an enormous improvement in Alaskan communications. The entire system was multi-channel with redundancy in many key parts of the network. Outside of the larger cities, WACS often brought the first usable long-distance telephone service. Even in Anchorage, WACS provided the only multi-channel connection. Despite these achievements, WACS was set for much the same fate as other troposcatter systems: obsolescence after the invention of communications satellites.

The experimental satellites Telstar 1 and 2 launched in the early 1960s, and the military began a shift towards satellite communications shortly after. Besides, the formidable cost of WACS had become a political issue. Maintenance of the system overran estimates by just as much as construction, and placing this cost on taxpayers was controversial since much of the traffic carried by the system consisted of regular commercial telephone calls. Besides, a general reticence to allocate money to WACS had lead to a general decay of the system. WACS capacity was insufficient for the rapidly increasing long-distance telephone traffic of the '60s, and due to decreased maintenance funding reliability was beginning to decline.

The retirement of a Cold War communications system is not unusual, but the particular fate of WACS is. It entered a long second life.

After acting as the sole long-distance provider for 60 years, the military began its retreat. In 1969, Congress passed the Alaska Communications Disposal Act. It called for complete divestment of the Alaska Communications System and WACS, to a private owner determined by a bidding process. Several large independent communications companies bid, but the winner was RCA. Committing to a $28.5 million purchase price followed by $30 million in upgrades, RCA reorganized the Alaska Communications System as RCA Alascom.

Transfer of the many ACS assets from the military to RCA took 13 years, involving both outright transfer of property and complex lease agreements on sites colocated with military installations. RCA's interest in Alaskan communications was closely connected to the coming satellite revolution: RCA had just built the Bartlett Earth Station, the first satellite ground station in Alaska. While Bartlett was originally an ACS asset owned by the Signal Corps, it became just the first of multiple ground stations that RCA would build for Alascom. Several of the new ground stations were colocated with WACS sites, establishing satellite as an alternative to the troposcatter links.

Alascom appears to have been the first domestic satellite voice network in commercial use, initially relying on a Canadian communications satellite [3]. In 1974, SATCOM 1 and 2 launched. These were not the first commercial communications satellites, but they represented a significant increase in capacity over previous commercial designs and are sometimes thought of as the true beginning of the satellite communications era. Both were built and owned by RCA, and Alascom took advantage of the new transponders.

At the same time, Alascom launched a modernization effort. 22 of the former WACS stations were converted to satellite ground stations, a project that took much of the '70s as Alascom struggled with the same conditions that had made WACS so challenging to begin with. Modernization also included the installation of DMS-10 telephone switches and conversion of some connections to digital.

A series of regulatory and business changes in the 1970s lead RCA to step away from the domestic communications industry. In 1979, Alascom sold to Pacific Power and Light, now for $200 million and $90 million in debt. PP&L continued on much the same trajectory, expanding the Alascom system to over 200 ground stations and launching the satellite Aurora I---the first of a small series of satellites that gave Alaska the distinction of being the only state with its own satellite communications network. For much of the '70s to the '00s, large parts of Alaska relied on satellite relay for calls between towns.

In a slight twist of irony considering its long lack of interest in the state, AT&T purchased parts of Alascom from PP&L in 1995, forming AT&T Alascom which has faded away as an independent brand. Other parts of the former ACS network, generally non-toll (or non-long-distance) operations, were split off into then PP&L subsidiary CenturyTel. While CenturyTel has since merged into CenturyLink, the Alaskan assets were first sold to Alaska Communications. Alaska Communications considers itself the successor of the ACS heritage, giving them a claim to over 100 years of communications history.

As electronics technology has continued to improve, penetration of microwave relays into inland Alaska has increased. Fewer towns rely on satellite today than in the 1970s, and the half-second latency to geosynchronous orbit is probably not missed. Alaska communications have also become more competitive, with long-distance connectivity available from General Communications (GCI) as well as AT&T and Alaska Communications.

Still, the legacy of Alaska's complex and expensive long-distance infrastructure still echoes in our telephone bills. State and federal regulators have allowed for extra fees on telephone service in Alaska and calls into Alaska, both intended to offset the high cost of infrastructure. Alaska is generally the most expensive long-distance calling destination in the United States, even when considering the territories.

But what of White Alice?

The history of the Alaska Communications System's transition to private ownership is complex and not especially well documented. While RCA's winning bid following the Alaska Communications Disposal Act set the big picture, the actual details of the transition were established by many individual negotiations spanning over a decade. Depending on the station, WACS troposcatter sites generally conveyed to RCA in 1973 or 1974. Some, colocated with active military installations, were leased rather than included in the sale. RCA generally decommissioned each WACS site once a satellite ground station was ready to replace it, either on-site or nearby.

For some WACS sites, this meant the troposcatter equipment was shut down in 1973. Others remained in use later. The Boswell Bay troposcatter station seems to have been the last turned down, in 1985. The 1980s were decidedly the end of WACS. Alascom's sale to PP&L cemented the plan to shut down all troposcatter operations, and the 1980 Comprehensive Environmental Response, Compensation, and Liability Act lead to the establishment of the Formerly Used Defense Sites (FUDS) program within DoD. Under FUDS, the Corps of Engineers surveyed the disused WACS sites and found nearly all had significant contamination by asbestos (used in seemingly every building material in the '50s and '60s) and leaked fuel oil.

As a result, most White Alice sites were demolished between 1986 and 1999. The cost of demolition and remediation in such remote locations was sometimes greater than the original construction. No WACS sites remain intact today.

USAF photo

Postscript:

A 1988 Corps of Engineers historical inventory of WACS, prepared due to the demolition of many of the stations, mentions that meteor burst communications might replace troposcatter. Meteor burst is a fascinating communications mode, similar in many ways to troposcatter but with the twist that the reflecting surface is not the troposphere but the ionized trail of meteors entering the atmosphere. Meteor burst connections only work when there is a meteor actively vaporizing in the upper atmosphere, but atmospheric entry of small meteors is common enough that meteor burst communications are practical for low-rate packetized communications. For example, meteor burst has been used for large weather and agricultural telemetry systems.

The Alaska Meteor Burst Communications System was implemented in 1977 by several federal agencies, and was used primarily for automated environmental telemetry. Unlike most meteor burst systems, though, it seems to have been used for real-time communications by the BLM and FAA. I can't find much information, but they seem to have built portable teleprinter terminals for this use.

Even more interesting, the Air Force's Alaskan Air Command built its own meteor burst network around the same time. This network was entirely for real-time use, and demonstrated the successful transmission of radar track data from radar stations across the state to Elmendorf Air Force base. Even better, the Air Force experimented with the use of meteor burst for intercept control by fitting aircraft with a small speech synthesizer that translated coded messages into short phrases. The Air Force experimented with several meteor burst systems during the Cold War, anticipating that it might be a survivable communications system in wartime. More details on these will have to fill a future article.

[1] Crews of the Western Union Telegraph Expedition reportedly continued work for a full year after the completion of the transatlantic telephone cable, because news of it hadn't reached them yet.

[2] Eliding here some complexities like GTE and their relationship to the Bell System.

[3] Perhaps owing to the large size of the country and many geographical challenges to cable laying, Canada has often led North America in satellite communications technology.

Note: I have edited this post to add more information, a couple of hours after originally publishing it. I forgot about a source I had open in a tab. Sorry.

Error'd: Lucky Penny

30 May 2025 at 06:30

High-roller Matthew D. fears Finance. "This is from our corporate expense system. Will they flag my expenses in the April-December quarter as too high? And do we really need a search function for a list of 12 items?"

0

Β 

Tightfisted Adam R. begrudges a trifling sum. "The tipping culture is getting out of hand. After I chose 'Custom Tip' for some takeout, they filled out the default tip with a few extra femtocents. What a rip!"

1

Β 

Cool Customer Reinier B. sums this up: "I got some free B&J icecream a while back. Since one of them was priced at €0.01, the other one obviously had to cost zero point minus 1 euros to make a total of zero euro. Makes sense. Or probably not."

3

Β 

An anonymous browniedad is ready to pack his poptart off for the summer. "I know {First Name} is really excited for camp..." Kudos on getting Mom to agree to that name choice!

2

Β 

Finally, another anonymous assembler's retrospective visualisation. "CoPilot rendering a graphical answer of the semantics of a pointer. Point taken. " There's no error'd here really, but I'm wondering how long before this kind of wtf illustration lands somewhere "serious".

4

Β 

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

CodeSOD: Recasting the Team

29 May 2025 at 06:30

Nina's team has a new developer on the team. They're not a junior developer, though Nina wishes they could replace this developer with a junior. Inexperience is better than whatever this Java code is.

Object[] test = (Object[]) options;
List<SchedulePlatform> schedulePlatformList = (List<SchedulePlatform>)((Object[])options)[0];
List<TableColumn> visibleTableCols = (List<TableColumn>)((Object[])options)[1];

We start by casting options into an array of Objects. That's already a code stench, but we actually don't even use the test variable and instead just redo the cast multiple times.

But worse than that, we cast to an array of object, access an element, and then cast that element to a collection type. I do not know what is in the options variable, but based on how it gets used, I don't like it. What it seems to be is a class (holding different options as fields) rendered as an array (holding different options as elements).

The new developer (ab)uses this pattern everywhere.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

CodeSOD: Format Identified

28 May 2025 at 06:30

Many nations have some form of national identification number, especially around taxes. Argentina is no exception.

Their "CUIT" (Clave Única de Identificación Tributaria) and "CUIL" (Código Único de Identificación Laboral) are formatted as "##-########-#".

Now, as datasets often don't store things in their canonical representation, Nick's co-worker was given a task: "given a list of numbers, reformat them to look like CUIT/CUIL. That co-worker went off for five days, and produced this Java function.

public String normalizarCuitCuil(String cuitCuilOrigen){
	String valorNormalizado = new String();
	
	if (cuitCuilOrigen == null || "".equals(cuitCuilOrigen) || cuitCuilOrigen.length() < MINIMA_CANTIDAD_ACEPTADA_DE_CARACTERES_PARA_NORMALIZAR){
		valorNormalizado = "";
	}else{
		StringBuilder numerosDelCuitCuil = new StringBuilder(13);
		cuitCuilOrigen = cuitCuilOrigen.trim();
		
		// Se obtienen solo los nΓΊmeros:
		Matcher buscadorDePatron =  patternNumeros.matcher(cuitCuilOrigen);
		while (buscadorDePatron.find()){
			numerosDelCuitCuil.append(buscadorDePatron.group());
		}
		
		// Se le agregan los guiones:
		valorNormalizado = numerosDelCuitCuil.toString().substring(0,2) 
							+ "-"
							+ numerosDelCuitCuil.toString().substring(2,numerosDelCuitCuil.toString().length()-1) 
							+ "-"
							+ numerosDelCuitCuil.toString().substring(numerosDelCuitCuil.toString().length()-1, numerosDelCuitCuil.toString().length());
		
	}
	return valorNormalizado;
}

We start with a basic sanity check that the string exists and is long enough. If it isn't, we return an empty string, which already annoys me, because an empty result is not a good way to communicate "I failed to parse".

But assuming we have data, we construct a string builder and trim whitespace. And already we have a problem: we already validated that the string was long enough, but if the string contained more trailing whitespace than a newline, we're looking at a problem. Now, maybe we can assume the data is good, but the next line implies that we can't rely on that- they create a regex matcher to identify numeric values, and for each numeric value they find, they append it to our StringBuilder. This implies that the string may contain non-numeric values which need to be rejected, which means our length validation was still wrong.

So either the data is clean and we're overvalidating, or the data is dirty and we're validating in the wrong order.

But all of that's a preamble to a terrible abuse of string builders, where they discard all the advantages of using a StringBuilder by calling toString again and again and again. Now, maybe the function caches results or the compiler can optimize it, but the result is a particularly unreadable blob of slicing code.

Now, this is ugly, but at least it works, assuming the input data is good. It definitely should never pass a code review, but it's not the kind of bad code that leaves one waking up in the middle of the night in a cold sweat.

No, what gets me about this is that it took five days to write. And according to Nick, the responsible developer wasn't just slacking off or going to meetings the whole time, they were at their desk poking at their Java IDE and looking confused for all five days.

And of course, because it took so long to write the feature, management didn't want to waste more time on kicking it back via a code review. So voila: it got forced through and released to production since it passed testing.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

The Missing Link of Ignorance

27 May 2025 at 06:30

Our anonymous submitter, whom we'll call Craig, worked for GlobalCon. GlobalCon relied on an offshore team on the other side of the world for adding/removing users from the system, support calls, ticket tracking, and other client services. One day at work, an urgent escalated ticket from Martin, the offshore support team lead, fell into Craig's queue. Seated before his cubicle workstation, Craig opened the ticket right away:

A fictional example of a parcel delivery SMS phishing message

The new GlobalCon support website is not working. Appears to have been taken over by ChatGPT. The entire support team is blocked by this.

Instead of feeling any sense of urgency, Craig snorted out loud from perverse amusement.

"What was that now?" The voice of Nellie, his coworker, wafted over the cubicle wall that separated them.

"Urgent ticket from the offshore team," Craig replied.

"What is it this time?" Nellie couldn't suppress her glee.

"They're dead in the water because the new support page was, quote, taken over by ChatGPT."

Nellie laughed out loud.

"Hey! I know humor is important to surviving this job." A level, more mature voice piped up behind Craig from the cube across from his. It belonged to Dana, his manager. "But it really is urgent if they're all blocked. Do your best to help, escalate to me if you get stuck."

"OK, thanks. I got this," Craig assured her.

He was already 99.999% certain that no part of their web domain had gone down or been conquered by a belligerent AI, or else he would've heard of it by now. To make sure, Craig opened support.globalcon.com in a browser tab: sure enough, it worked. Martin had supplied no further detail, no logs or screenshots or videos, and no steps to reproduce, which was sadly typical of most of these escalations. At a loss, Craig took a screenshot of the webpage, opened the ticket, and posted the following: Everything's fine on this end. If it's still not working for you, let's do a screenshare.

Granted, a screensharing session was less than ideal given the 12-hour time difference. Craig hoped that whatever nefarious shenanigans ChatGPT had allegedly committed were resolved by now.

The next day, Craig received an update. Still not working. The entire team is still blocked. We're too busy to do a screenshare, please resolve ASAP.

Craig checked the website again with both laptop and phone. He had other people visit the website for him, trying different operating systems and web browsers. Every combination worked. Two things mystified him: how was the entire offshore team having this issue, and how were they "too busy" for anything if they were all dead in the water? At a loss, Craig attached an updated screenshot to the ticket and typed out the best CYA response he could muster. The new support website is up and has never experienced any issues. With no further proof or steps to reproduce this, I don't know what to tell you. I think a screensharing session would be the best thing at this point.

The next day, Martin parroted his last message almost word for word, except this time he assented to a screensharing session, suggesting the next morning for himself.

It was deep into the evening when Craig set up his work laptop on his kitchen counter and started a call and session for Martin to join. "OK. Can you show me what you guys are trying to do?"

To his surprise, he watched Martin open up Microsoft Teams first thing. From there, Martin accessed a chat to the entire offshore support team from the CPO of GlobalCon. The message proudly introduced the new support website and outlined the steps for accessing it. One of those steps was to visit support.globalcon.com.

The web address was rendered as blue outlined text, a hyperlink. Craig observed Martin clicking the link. A web browser opened up. Lo and behold, the page that finally appeared was www.chatgpt.com.

Craig blinked with surprise. "Hang on! I'm gonna take over for a second."

Upon taking control of the session, Craig switched back to Teams and accessed the link's details. The link text was correct, but the link destination was ChatGPT. It seemed like a copy/paste error that the CPO had tried to fix, not realizing that they'd needed to do more than simply update the link text.

"This looks like a bad link," Craig said. "It got sent to your entire team. And all of you have been trying to access the support site with this link?"

"Correct," Martin replied.

Craig was glad he couldn't be seen frowning and shaking his head. "Lemme show you what I've been doing. Then you can show everyone else, OK?"

After surrendering control of the session, Craig patiently walked Martin through the steps of opening a web browser, typing support.globalcon.com into the header, and hitting Return. The site opened without any issue. From there, Craig taught Martin how to create a bookmark for it.

"Just click on that from now on, and it'll always take you to the right place," Craig said. "In the future, before you click on any hyperlink, make sure you hover your mouse over it to see where it actually goes. Links can be labeled one thing when they actually take you somewhere else. That's how phishing works."

"Oh," Martin said. "Thanks!"

The call ended on a positive note, but left Craig marveling at the irony of lecturing the tech support lead on Internet 101 in the dead of night.

[Advertisement] Picking up NuGet is easy. Getting good at it takes time. Download our guide to learn the best practice of NuGet for the Enterprise.

Classic WTF: Superhero Wanted

26 May 2025 at 06:30
It's a holiday in the US today, so we're taking a long weekend. We flip back to a classic story of a company wanting to fill 15 different positions by hiring only one person. It's okay, Martin handles the database. Original - Remy

A curious email arrived in Phil's Inbox. "Windows Support Engineer required. Must have experience of the following:" and then a long list of Microsoft products.

Phil frowned. The location was convenient; the salary was fine, just the list of software seemed somewhat intimidating. Nevertheless, he replied to the agency saying that he was interested in applying for the position.

A few days later, Phil met Jason, the guy from the recruitment agency, in a hotel foyer. "It's a young, dynamic company", the recruiter explained,"They're growing really fast. They've got tons of funding and their BI Analysis Suite is positioning them to be a leading player in their field."

Phil nodded. "Ummm, I'm a bit worried about this list of products", referring to the job description. "I've never dealt with Microsoft Proxy Server 1.0, and I haven't dealt with Windows 95 OSR2 for a long while."

"Don't worry," Jason assured, "The Director is more an idea man. He just made a list of everything he's ever heard of. You'll just be supporting Windows Server 2003 and their flagship application."

Phil winced. He was a vanilla network administrator – supporting a custom app wasn't quite what he was looking for, but he desperately wanted to get out of his current job.

A few days later, Phil arrived for his interview. The company had rented smart offices on a new business park on the edge of town. He was ushered into the conference room, where he was joined by The Director and The Manager.

"So", said The Manager. "You've seen our brochure?"

"Yeah", said Phil, glancing at the glossy brochure in front of him with bright, Barbie-pink lettering all over it.

"You've seen a demo version of our application – what do you think?"

"Well, I think that it's great!", said Phil. He'd done his research – there were over 115 companies offering something very similar, and theirs wasn't anything special. "I particularly like the icons."

"Wonderful!" The Director cheered while firing up PowerPoint. "These are our servers. We rent some rack space in a data center 100 miles away." Phil looked at the projected picture. It showed a rack of a dozen servers.

"They certainly look nice." said Phil. They did look nice – brand new with green lights.

"Now, we also rent space in another data center on the other side of the country," The Manager added.

"This one is in a former cold-war bunker!" he said proudly. "It's very secure!" Phil looked up at another photo of some more servers.

"What we want the successful applicant to do is to take care of the servers on a day to day basis, but we also need to move those servers to the other data center", said The Director. "Without any interruption of service."

"Also, we need someone to set up the IT for the entire office. You know, email, file & print, internet access – that kind of thing. We've got a dozen salespeople starting next week, they'll all need email."

"And we need it to be secure."

"And we need it to be documented."

Phil was scribbled notes as best he could while the interviewing duo tag teamed him with questions.

"You'll also provide second line support to end users of the application."

"And day-to-day IT support to our own staff. Any questions?"

Phil looked up. "Ah… which back-end database does the application use?" he asked, expecting the answer would be SQL Server or perhaps Oracle, but The Director's reply surprised him.

"Oh, we wrote our own database from scratch. Martin wrote it." Phil realized his mouth was open, and shut it. The Director saw his expression, and explained. "You see, off the shelf databases have several disadvantages – the data gets fragmented, they're not quick enough, and so on. But don't have to worry about that – Martin takes care of the database. Do you have any more questions?"

Phil frowned. "So, to summarize: you want a data center guy to take care of your servers. You want someone to migrate the application from one data center to another, without any outage. You want a network administrator to set up, document and maintain an entire network from scratch. You want someone to provide internal support to the staff. And you want a second line support person to support the our flagship application."

"Exactly", beamed The Director paternally. "We want one person who can do all those things. Can you do that?"

Phil took a deep breath. "I don't know," he replied, and that was the honest answer.

"Right", The Manager said. "Well, if you have any questions, just give either of us a call, okay?"

Moments later, Phil was standing outside, clutching the garish brochure with the pink letters. His head was spinning. Could he do all that stuff? Did he want to? Was Martin a genius or a madman to reinvent the wheel with the celebrated database?

In the end, Phil was not offered the job and decided it might be best to stick it out at his old job for a while longer. After all, compared to Martin, maybe his job wasn't so bad after all.

[Advertisement] Plan Your .NET 9 Migration with Confidence
Your journey to .NET 9 is more than just one decision.Avoid migration migraines with the advice in this free guide. Download Free Guide Now!

Error'd: Mike's Job Search Job

23 May 2025 at 06:30

Underqualified Mike S. is suffering a job hunt. "I could handle uD83D and uDC77 well enough, but I am a little short of uD83C and the all important uDFFE requirement."

0

Β 

Frank forecasts frustration. "The weather app I'm using seems to be a bit confused on my location as I'm on vacation right now." It would be a simple matter for the app to simply identify each location, if it can't meaningfully choose only one.

1

Β 

Marc WΓΌrth is making me hungry. Says Marc "I was looking through my Evernote notes for "transactional" (email service). It didn't find anything. Evernote, though, tried to be helpful and thought I was looking for some basil (German "Basilikum")."

3

Β 

"To be from or not from be," muses Michael R. Indeed, that is the question at Stansted Shakespeare airport.

4

Β 

That is not the King," Brendan commented. "I posted this on Discord, and my friend responded with "They have succeeded in alignment. Their AI is truly gender blind." Not only gender-blind but apparently also existence-blind as well. I think the Bard might have something quotable here as well but it escapes me. Comment section is open.

2

Β 

...and angels sing thee to thy rest.
[Advertisement] Picking up NuGet is easy. Getting good at it takes time. Download our guide to learn the best practice of NuGet for the Enterprise.

CodeSOD: A Trying Block

22 May 2025 at 06:30

Mark sends us a very simple Java function which has the job of parsing an integer from a string. Now, you might say, "But Java has a built in for that, Integer.parseInt," and have I got good news for you: they actually used it. It's just everything else they did wrong.

private int makeInteger(String s)
{
  int i=0;
  try
  {
    Integer.parseInt(s);
  }
  catch (NumberFormatException e)
  {
    i=0;
    return i;
  }
  i=Integer.parseInt(s);
  return i;
}

This function is really the story of variable i, the most useless variable ever. It's doing its best, but there's just nothing for it to do here.

We start by setting i to zero. Then we attempt to parse the integer, and do nothing with the result. If it fails, we set i to zero again, just for fun, and then return i. Why not just return 0? Because then what would poor i get to do?

Assuming we didn't throw an exception, we parse the input again, storing its result in i, and then return i. Again, we treat i like a child who wants to help paint the living room: we give it a dry brush and a section of wall we're not planning to paint and let it go to town. Nothing it does matters, but it feels like a participant.

Now, Mark went ahead and refactored this function basically right away, into a more terse and clear version:

private int makeInteger(String s)
{
  try
  {
    return Integer.parseInt(s);
  }
  catch (NumberFormatException e)
  {
    return 0;
  }
}

He went about his development work, and then a few days later came across makeInteger reverted back to its original version. For a moment, he wanted to be mad at someone for reverting his change, but no- this was in an entirely different class. With that information, Mark went and did a search for makeInteger in the code, only to find 39 copies of this function, with minor variations.

There are an unknown number of copies of the function where the name is slightly different than makeInteger, but a search for Integer.parseInt implies that there may be many more.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

CodeSOD: Buff Reading

21 May 2025 at 06:30

Frank inherited some code that reads URLs from a file, and puts them into a collection. This is a delightfully simple task. What could go wrong?

static String[]  readFile(String filename) {
    String record = null;
    Vector vURLs = new Vector();
    int recCnt = 0;

    try {
        FileReader fr = new FileReader(filename);
        BufferedReader br = new BufferedReader(fr);

        record = new String();

        while ((record = br.readLine()) != null) {
            vURLs.add(new String(record));
            //System.out.println(recCnt + ": " + vURLs.get(recCnt));
            recCnt++;
        }
    } catch (IOException e) {
        // catch possible io errors from readLine()
        System.out.println("IOException error reading " + filename + " in readURLs()!\n");
        e.printStackTrace();
    }

    System.out.println("Reading URLs ...\n");

    int arrCnt = 0;
    String[] sURLs = new String[vURLs.size()];
    Enumeration eURLs = vURLs.elements();

    for (Enumeration e = vURLs.elements() ; e.hasMoreElements() ;) {
        sURLs[arrCnt] = (String)e.nextElement();
        System.out.println(arrCnt + ": " + sURLs[arrCnt]);
        arrCnt++;
    }

    if (recCnt != arrCnt++) {
        System.out.println("WARNING: The number of URLs in the input file does not match the number of URLs in the array!\n\n");
    }

    return sURLs;
} // end of readFile()

So, we start by using a FileReader and a BufferedReader, which is the basic pattern any Java tutorial on file handling will tell you to do.

What I see here is that the developer responsible didn't fully understand how strings work in Java. They initialize record to a new String() only to immediately discard that reference in their while loop. They also copy the record by doing a new String which is utterly unnecessary.

As they load the Vector of strings, they also increment a recCount variable, which is superfluous since the collection can tell you how many elements are in it.

Once the Vector is populated, they need to copy all this data into a String[]. Instead of using the toArray function, which is built in and does that, they iterate across the Vector and put each element into the array.

As they build the array, they increment an arrCnt variable. Then, they do a check: if (recCnt != arrCnt++). Look at that line. Look at the post-increment on arrCnt, despite never using arrCnt again. Why is that there? Just for fun, apparently. Why is this check even there?

The only way it's possible for the counts to not match is if somehow an exception was thrown after vURLs.add(new String(record)); but before recCount++, which doesn't seem likely. Certainly, if it happens, there's something worse going on.

Now, I'm going to be generous and assume that this code predates Java 8- it just looks old. But it's worth noting that in Java 8, the BufferedReader class got a lines() function which returns a Stream<String> that can be converted directly toArray, making all of this code superfluous, but also, so much of this code is just superfluous anyway.

Anyway, for a fun game, start making the last use of every variable be a post-increment before it goes out of scope. See how many code reviews you can sneak it through!

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Representative Line: What the FFFFFFFF

20 May 2025 at 06:30

Combining Java with lower-level bit manipulations is asking for trouble- not because the language is inadequate to the task, but because so many of the developers who work in Java are so used to working at a high level they might not quite "get" what they need to do.

Victor inherited one such project, which used bitmasks and bitwise operations a great deal, based on the network protocol it implemented. Here's how the developers responsible created their bitmasks:

private static long FFFFFFFF = Long.parseLong("FFFFFFFF", 16);

So, the first thing that's important to note, is that Java does support hex literals, so 0xFFFFFFFF is a perfectly valid literal. So we don't need to create a string and parse it. But we also don't need to make a constant simply named FFFFFFFF, which is just the old twenty = 20 constant pattern: technically you've made a constant but you haven't actually made the magic number go away.

Of course, this also isn't actually a constant, so it's entirely possible that FFFFFFFF could hold a value which isn't 0xFFFFFFFF.

[Advertisement] Picking up NuGet is easy. Getting good at it takes time. Download our guide to learn the best practice of NuGet for the Enterprise.

Representative Line: Identifying the Representative

19 May 2025 at 06:30

Kate inherited a system where Java code generates JavaScript (by good old fashioned string concatenation) and embeds it into an output template. The Java code was written by someone who didn't fully understand Java, but JavaScript was also a language they didn't understand, and the resulting unholy mess was buggy and difficult to maintain.

Why trying to debug the JavaScript, Kate had to dig through the generated code, which led to this little representative line:

dojo.byId('html;------sites------fileadmin------en------fileadmin------index.html;;12').setAttribute('isLocked','true');

The byId function is an alias to the browser's document.getElementById function. The ID on display here is clearly generated by the Java code, resulting in an absolutely cursed ID for an element in the page. The semicolons are field separators, which means you can parse the ID to get other information about it. I have no idea what the 12 means, but it clearly means something. Then there's that long kebab-looking string. It seems like maybe some sort of hierarchy information? But maybe not, because fileadmin appears twice? Why are there so many dashes? If I got an answer to that question, would I survive it? Would I be able to navigate the world if I understood the dark secret of those dashes? Or would I have to give myself over to our Dark Lords and dedicate my life to bringing about the end of all things?

Like all good representative lines, this one hints at darker, deeper evils in the codebase- the code that generates (or parses) this ID must be especially cursed.

The only element which needs to have its isLocked attribute set to true is the developer responsible for this: they must be locked away before they harm the rest of us.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Error'd: Teamwork

16 May 2025 at 06:30

Whatever would we do without teamwork.

David doesn't know. "Microsoft Teams seems to have lost count (it wasn't a very big copy/paste)"

4

Β 

A follow-up from an anonymous doesn't know either. "Teams doing its best impression of a ransom note just to say you signed out. At least it still remembers how to suggest closing your browser. Small victories."

1

Β 

Bob F. just wants to make memes. "I've been setting my picture widths in this document to 7.5" for weeks, and suddenly after the latest MS Word update, Microsoft thinks 7.5 is not between -22.0 and 22.0. They must be using AI math to determine this."

2

Β 

Ewan W. wonders "a social life: priceless...?". Ewan has some brand confusion but after the Boom Battle Bar I bet I know why.

0

Β 

Big spender Bob B. maybe misunderstands NaN. He gleefully exclaims "I'm very happy to get 15% off - Here's hoping the total ends up as NaN and I get it all free." Yikes. 191.78-NaN is indeed NaN, but that just means you're going to end up owing them NaN. Don't put that on a credit card!

3

Β 

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

CodeSOD: A Jammed Up Session

15 May 2025 at 06:30

Andre has inherited a rather antique ASP .Net WebForms application. It's a large one, with many pages in it, but they all follow a certain pattern. Let's see if you can spot it.

protected void btnSearch_Click(object sender, EventArgs e)
{
    ArrayList paramsRel = new ArrayList();
    paramsRel["Name"] = txtNome.Text;
    paramsRel["Date"] = txtDate.Text;
    Session["paramsRel"] = paramsRel;
   
    List<Client> clients = Controller.FindClients();
    //Some other code
}

Now, at first glance, this doesn't look terrible. Using an ArrayList as a dictionary and frankly, storing a dictionary in the Session object is weird, but it's not an automatic red flag. But wait, why is it called paramsRel? They couldn't be… no, they wouldn't…

public List<Client> FindClients()
{
    ArrayList paramsRel = (ArrayList)Session["paramsRel"];
    string name = (string)paramsRel["Name"];
    string dateStr = (string)paramsRel["Date"];
    DateTime date = DateTime.Parse(dateStr);
   
   //More code...
}

Now there's the red flag. paramsRel is how they pass parameters to functions. They stuff it into the Session, then call a function which retrieves it from that Session.

This pattern is used everywhere in the application. You can see that there's a vague gesture in the direction of trying to implement some kind of Model-View-Controller pattern (as FindClients is a member of the Controller object), but that modularization gets undercut by everything depending on Session as a pseudoglobal for passing state information around.

The only good news is that the Session object is synchronized so there's no thread safety issue here, though not for want of trying.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

CodeSOD: itouhhh…

14 May 2025 at 06:30

Frequently in programming, we can make a tradeoff: use less (or more) CPU in exchange for using more (or less) memory. Lookup tables are a great example: use a big pile of memory to turn complicated calculations into O(1) operations.

So, for example, implementing itoa, the C library function for turning an integer into a character array (aka, a string), you could maybe make it more efficient using a lookup table.

I say "maybe", because Helen inherited some C code that, well, even if it were more efficient, it doesn't help because it's wrong.

Let's start with the lookup table:

char an[1000][3] = 
{
	{'0','0','0'},{'0','0','1'},{'0','0','2'},{'0','0','3'},{'0','0','4'},{'0','0','5'},{'0','0','6'},{'0','0','7'},{'0','0','8'},{'0','0','9'},
	{'0','1','0'},{'0','1','1'},{'0','1','2'},{'0','1','3'},{'0','1','4'},{'0','1','5'},{'0','1','6'},{'0','1','7'},{'0','1','8'},{'0','1','9'},
    …

I'm abbreviating the lookup table for now. This lookup table is meant to be use to convert every number from 0…999 into a string representation.

Let's take a look at how it's used.

int ll = f->cfg.len_len;
long dl = f->data_len;
// Prepare length
if ( NULL == dst )
{
    dst_len = f->data_len + ll + 1 ;
    dst = (char*) malloc ( dst_len );
}
else
//if( dst_len < ll + dl )
if( dst_len < (unsigned) (ll + dl) )
{
    // TO DOO - error should be processed
    break;
}
long i2;
switch ( f->cfg.len_fmt)
{
    case ASCII_FORM:
    {
        if ( ll < 2 )
        {
            dst[0]=an[dl][2];
        }
        else if ( ll < 3 )
        {
            dst[0]=an[dl][1];
            dst[1]=an[dl][2];
        }
        else if ( ll < 4 )
        {
            dst[0]=an[dl][0];
            dst[1]=an[dl][1];
            dst[2]=an[dl][2];
        }
        else if ( ll < 5 )
        {
            i2 = dl / 1000;
            dst[0]=an[i2][2];
            i2 = dl % 1000;
            dst[3]=an[i2][2];
            dst[2]=an[i2][1];
            dst[1]=an[i2][0];
        }
        else if ( ll < 6 )
        {
            i2 = dl / 1000;
            dst[0]=an[i2][1];
            dst[1]=an[i2][2];
            i2 = dl % 1000;
            dst[4]=an[i2][2];
            dst[3]=an[i2][1];
            dst[2]=an[i2][0];
        }
        else
        {
            // General case
            for ( int k = ll  ; k > 0  ; k-- )
            {
                dst[k-1] ='0' + dl % 10;
                dl/=10;
            }
        }

        dst[dl]=0;

        break;
    }
}

Okay, we start with some reasonable bounds checking. I have no idea what to make of a struct member called len_len- the length of the length? I'm lacking some context here.

Then we get into the switch statement. For all values less than 4 digits, everything makes sense, more or less. I'm not sure what the point of using a 2D array for you lookup table is if you're also copying one character at a time, but for such a small number of copies I'm sure it's fine.

But then we get into the len_lens longer than 3, and we start dividing by 1000 so that our lookup table continues to work. Which, again, I guess is fine, but I'm still left wondering why we're doing this, why this specific chain of optimizations is what we need to do. And frankly, why we couldn't just use itoa or a similar library function which already does this and is probably more optimized than anything I'm going to write.

When we have an output longer than 5 characters, we just use a naive for-loop and some modulus as our "general" case.

So no, I don't like this code. It reeks of premature optimization, and it also has the vibe of someone starting to optimize without fully understanding the problem they were optimizing, and trying to change course midstream without changing their solution.

But there's a punchline to all of this. Because, you see, I skipped most of the lookup table. Would you like to see how it ends? Of course you do:

{'9','8','0'},{'9','8','1'},{'9','8','2'},{'9','8','3'},{'9','8','4'},{'9','8','5'},{'9','8','6'},{'9','8','7'},{'9','8','8'},{'9','8','9'}
};

The lookup table doesn't work for values from 990 to 999. There are just no entries there. All this effort to optimize converting integers to text and we end up here: with a function that doesn't work for 1% of the possible values it could receive. And, given that the result is an out-of-bounds array access, it fails with everyone's favorite problem: undefined behavior. Usually it'll segfault, but who knows! Maybe it returns whatever bytes it finds? Maybe it sends the nasal demons after you. The compiler is allowed to do anything.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

CodeSOD: Exactly a Date

13 May 2025 at 06:30

Alexandar sends us some C# date handling code. The best thing one can say is that they didn't reinvent any wheels, but that might be worse, because they used the existing wheels to drive right off a cliff.

try
{
    var date = DateTime.ParseExact(member.PubDate.ToString(), "M/d/yyyy h:mm:ss tt", null); 
    objCustomResult.PublishedDate = date;
}
catch (Exception datEx)
{
}

member.PubDate is a Nullable<DateTime>. So its ToString will return one of two things. If there is a value there, it'll return the DateTimes value. If it's null, it'll just return an empty string. Attempting to parse the empty string will throw an exception, which we helpfully swallow, do nothing about, and leave objCustomResult.PublishedDate in whatever state it was in- I'm going to guess null, but I have no idea.

Part of this WTF is that they break the advantages of using nullable types- the entire point is to be able to handle null values without having to worry about exceptions getting tossed around. But that's just a small part.

The real WTF is taking a DateTime value, turning it into a string, only to parse it back out. But because this is in .NET, it's more subtle than just the generation of useless strings, because member.PubDate.ToString()'s return value may change depending on your culture info settings.

Which sure, this is almost certainly server-side code running on a single server with a well known locale configured. So this probably won't ever blow up on them, but it's 100% the kind of thing everyone thinks is fine until the day it's not.

The punchline is that ToString allows you to specify the format you want the date formatted in, which means they could have written this:

var date = DateTime.ParseExact(member.PubDate.ToString("M/d/yyyy h:mm:ss tt"), "M/d/yyyy h:mm:ss tt", null);

But if they did that, I suppose that would have possibly tickled their little grey cells and made them realize how stupid this entire block of code was?

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

CodeSOD: Would a Function by Any Other Name Still be WTF?

12 May 2025 at 06:30

"Don't use exception handling for normal flow control," is generally good advice. But Andy's lead had a PhD in computer science, and with that kind of education, wasn't about to let good advice or best practices tell them what to do. That's why, when they needed to validate inputs, they wrote code C# like this:


    public static bool IsDecimal(string theValue)
    {
        try
        {
            Convert.ToDouble(theValue);
            return true;
        }
        catch
        {
            return false;
        }
    } 

They attempt to convert, and if they succeed, great, return true. If they fail, an exception gets caught, and they return false. What could be simpler?

Well, using the built in TryParse function would be simpler. Despite its name, actually avoids throwing an exception, even internally, because exceptions are expensive in .NET. And it is already implemented, so you don't have to do this.

Also, Decimal is a type in C#- a 16-byte floating point value. Now, I know they didn't actually mean Decimal, just "a value with 0 or more digits behind the decimal point", but pedantry is the root of clarity, and the naming convention makes this bad code unclear about its intent and purpose. Per the docs there are Single and Double values which can't be represented as Decimal and trigger an OverflowException. And conversely, Decimal loses precision if converted to Double. This means a value that would be represented as Decimal might not pass this function, and a value that can't be represented as Decimal might, and none of this actually matters but the name of the function is bad.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Sky Extends A.I. Automation to Your Entire Mac

By: Nick Heer
30 May 2025 at 04:18

Federico Viticci, MacStories:

For the past two weeks, I’ve been able to use Sky, the new app from the people behind Shortcuts who left Apple two years ago. As soon as I saw a demo, I felt the same way I did about Editorial, Workflow, and Shortcuts: I knew Sky was going to fundamentally change how I think about my macOS workflow and the role of automation in my everyday tasks.

Only this time, because of AI and LLMs, Sky is more intuitive than all those apps and requires a different approach, as I will explain in this exclusive preview story ahead of a full review of the app later this year.

Matthew Cassinelli has also been using an early version of Sky:

Sky bridges the gap between old-school scripting, modern automation, and new-age LLM technology, built with a deep love for working on the Mac as a platform.

This feels like the so-far-unfulfilled promise of Apple Intelligence β€” but more. The ways I want to automate iOS are limited. But the kinds of things I want help with on my Mac are boundless. Viticci shares the example of automatically sorting a disorganized folder in Finder, and that is absolutely something I want to do easier than I currently can. Yes, I could cobble together something with AppleScript or an Automator workflow, but it would be so much nicer if I could just tell my computer to do something in the most natural language I understand. This is fascinating.

βŒ₯ Permalink

Judge Dismisses 2021 Rumble Antitrust Suit Against Google on Statute of Limitations Grounds

By: Nick Heer
28 May 2025 at 18:16

Mike Scarcella, Reuters:

Alphabet’s Google has persuaded a federal judge in California to reject a lawsuit from video platform Rumble accusing the technology giant of illegally monopolizing the online video-sharing market.

In a ruling on Wednesday, U.S. District Judge Haywood Gilliam Jr said Rumble’s 2021 lawsuit seeking more than $2 billion in damages was untimely filed outside the four-year statute of limitations for antitrust claims.

Rumble is dishonest and irritating, but I thought its case in which it argued Google engages in self-preferencing could be interesting. It seems to rank YouTube videos more highly than those from other sources. This can be explained by YouTube’s overwhelming popularity β€” it consistently ranks in the top ten web services according to Cloudflare β€” yet I can see anyone’s discomfort in taking Google’s word for it, since it has misrepresented its ranking criteria.

This is an unsatisfying outcome, but it seems Rumble has another suit it is still litigating.

βŒ₯ Permalink

Google is Burying the Web Alive

By: Nick Heer
28 May 2025 at 04:28

John Herrman, New York magazine:

But I also don’t want to assume Google knows exactly how this stuff will play out for Google, much less what it will actually mean for millions of websites, and their visitors, if Google stops sending as many people beyond its results pages. Google’s push into productizing generative AI is substantially fear-driven, faith-based, and informed by the actions of competitors that are far less invested in and dependent on the vast collection of behaviors β€” websites full of content authentic and inauthentic, volunteer and commercial, social and antisocial, archival and up-to-date β€” that make up what’s left of the web and have far less to lose. […]

Very nearly since it launched, Google has attempted to answer users’ questions as immediately as possible. It had the β€œI’m Feeling Lucky” button since it was still a stanford.edu subdomain, and it has since steadily changed the results page to more directly respond to queries. But this seems entirely different β€” a way to benefit from Google’s decades-long ingestion of the web and giving almost nothing back. Or, perhaps, giving back something ultimately worse: invented answers users cannot trust, and will struggle to check because sources are intermingled and buried.

βŒ₯ Permalink

The CIA’s 2010s Covert Communication Websites

By: Nick Heer
27 May 2025 at 23:28

Ciro Santilli:

This article is about covert agent communication channel websites used by the CIA in many countries from the late 2000s until the early 2010s, when they were uncovered by counter intelligence of the targeted countries circa 2010-2013.

This is a pretty clever scheme in theory, but seems to have been pretty sloppy in practice. That is, many of the sites seem to share enough elements allowing an enterprising person to link the seemingly unrelated sites β€” even, as it turns out, years later and after they have been pulled offline. That apparently resulted in the deaths of, according to Foreign Policy, dozens of people.

βŒ₯ Permalink

Apple Gets Its Annual Fraud Prevention Headlines

By: Nick Heer
27 May 2025 at 18:04

Apple issued a news release today touting the safety of the App Store, dutifully covered without context by outlets like 9to5Mac, AppleInsider, and MacRumors. This has become an annual tradition in trying to convince people β€” specifically, developers and regulators β€” of the wisdom of allowing native software to be distributed for iOS only through the App Store. Apple published similar stats in 2021, 2022, 2023, and 2024, reflecting the company’s efforts in each preceding year. Each contains similar figures; for example:

  • In its new report, Apple says it β€œterminated more than 146,000 developer accounts over fraud concerns” in 2024, an increase from 118,000 in 2023, which itself was a decrease from 428,000 in 2022. Apple said the decrease between 2022 and 2023 was β€œthanks to continued improvements to prevent the creation of potentially fraudulent accounts in the first place”. Does the increase in 2024 reflect poorer initial anti-fraud controls, or an increase in fraud attempts? Is it possible to know either way?

  • Apple says it deactivated β€œnearly 129 million customer accounts” in 2024, a significant decrease from deactivating 374 million the year prior. However, it blocked 711 million account creations in 2024, which is several times greater than the 153 million blocked in the year before. Compare to 2022, when it disabled 282 million accounts and prevented the creation of 198 million potentially fraudulent accounts. In 2021, the same numbers were 170 million and 118 million; in 2020, 244 million and 424 million. These numbers are all over the place.

  • A new statistic Apple is publishing this year is β€œillicit app distribution”. It says that, in the past month, it β€œstopped nearly 4.6 million attempts to install or launch apps distributed illicitly outside the App Store or approved third-party marketplaces”. These are not necessarily fraudulent, pirated, or otherwise untoward apps. This statistic is basically a reflection of the control maintained by Apple over iOS regardless of user intentions.

There are plenty of numbers just like these in Apple’s press release. They all look impressive in large part because just about any statistic would be at Apple’s scale. Apple is also undeniably using the App Store to act as a fraud reduction filter, with mixed results. I do not expect a 100% success rate, but I still do not know how much can be gleaned from context-free numbers.

βŒ₯ Permalink

Lawyers Keep Failing Clients By Relying on A.I.

By: Nick Heer
26 May 2025 at 18:32

Nicholas Chrastil, the Guardian:

State officials have praised Butler Snow for its experience in defending prison cases – and specifically William Lunsford, head of the constitutional and civil rights litigation practice group at the firm. But now the firm is facing sanctions by the federal judge overseeing Johnson’s case after an attorney at the firm, working with Lunsford, cited cases generated by artificial intelligence – which turned out not to exist.

It is one of a growing number of instances in which attorneys around the country have faced consequences for including false, AI-generated information in official legal filings. A database attempting to track the prevalence of the cases has identified 106 instances around the globe in which courts have found β€œAI hallucinations” in court documents.

The database is now up to 120 cases, including some fairly high-profile ones like that against Timothy Burke.

Here is a little behind-the-scenes from this weekend’s piece about β€œnimble fingers” and Apple’s supply chain. The claim, as framed by Tripp Mickle, in the New York Times, is that β€œ[y]oung Chinese women have small fingers, and that has made them a valuable contributor to iPhone production because they are more nimble at installing screws and other miniature parts”. This sounded suspicious to me because I thought about it for five seconds. There are other countries where small objects are carefully assembled by hand, for example, and attributing a characteristic like β€œsmall fingers” to hundreds of millions of β€œyoung Chinese women” seems reductive, to put it mildly. But this assumption had to come from somewhere, especially since Patrick McGee also mentioned it.

So I used both DuckDuckGo and Google to search for relevant keywords within a date range of the last fifteen years and excluding the past month or so. I could not quickly find anything of relevance; both thought I was looking for smartphones for use with small hands. So I thought this might be a good time to try ChatGPT. It immediately returned a quote from a 2014 report from an international labour organization, but did not tell me the title of the report or give me a link. I asked it for the title. ChatGPT responded it was actually a 2012 report that mentioned β€œnimble fingers” of young women being valuable, and gave me the title. But when I found copies of the report, there was no such quote or anything remotely relevant. I did, however, get the phrase β€œnimble fingers”, which sent me down the correct search path to finding articles documenting this longstanding prejudice.

Whether because of time crunch or laziness, it baffles me how law firms charging as much as they do have repeatedly failed to verify the claims generated by artificial intelligence tools.

βŒ₯ Permalink

βŒ₯ β€˜Nimble Fingers’ Racism and iPhone Manufacturing

By: Nick Heer
24 May 2025 at 21:19

Tripp Mickle, of the New York Times, wrote another one of those articles exploring the feasibility of iPhone manufacturing in the United States. There is basically nothing new here; the only reason it seems to have been published is because the U.S. president farted out yet another tariff idea, this time one targeted specifically at the iPhone at a rate of 25%.1

Anyway, there is one thing in this article β€” bizarrely arranged in a question-and-answer format β€” that is notable:

What does China offer that the United States doesn’t?

Small hands, a massive, seasonal work force and millions of engineers.

Young Chinese women have small fingers, and that has made them a valuable contributor to iPhone production because they are more nimble at installing screws and other miniature parts in the small device, supply chain experts said. In a recent analysis the company did to explore the feasibility of moving production to the United States, the company determined that it couldn’t find people with those skills in the United States, said two people familiar with the analysis who spoke on the condition of anonymity.

I will get to the racial component of this in a moment, but this answer has no internal logic. There are two sentences in that larger paragraph. The second posits that people in the U.S. do not have the β€œskills” needed to carefully assemble iPhones, but the skills as defined in the first sentence are small fingers β€” which is not a skill. I need someone from the Times to please explain to me how someone can be trained to shrink their fingers.

Anyway, this is racist trash. In response to a question from Julia Carrie Wong of the Guardian, Times communications director Charlie Stadtlander disputed the story was furthering β€œracial or genetic generalizations”, and linked to a podcast segment clipped by Mickle. In it, Patrick McGee, author of β€œApple in China”, says:

The tasks that are often being done to make iPhones require little fingers. So the fact that it’s young Chinese women with little fingers β€” that actually matters. Like, Apple engineers will talk about this.

The podcast in question is, unsurprisingly, Bari Weiss’; McGee did not mention any of this when he appeared on, for example, the Daily Show.

Maybe some Apple engineers actually believe this, and maybe some supply chain experts do, too. But it is a longstanding sexist stereotype. (Thanks to Kat for the Feminist Review link.) It is ridiculous to see this published in a paper of record as though it is just one fact among many, instead of something which ought to be debunked.

The Times has previously reported why iPhones cannot really be made in the U.S. in any significant quantity. It has nothing to do with finger size, and everything to do with a supply chain the company has helped build for decades, as McGee talks about extensively in that Daily Show interview and, presumably, writes about in his book. (I do not yet have a copy.) Wages play a role, but it is the sheer concentration of manufacturing capability that explains why iPhones are made in China, and why it has been so difficult for Apple to extricate itself from the country.


  1. About which the funniest comment comes from Anuj Ahooja on Threads.Β β†₯︎

Dara Khosrowshahi Knows Uber Is Just Reinventing the Bus

By: Nick Heer
24 May 2025 at 16:16

Uber CEO Dara Khosrowshahi was on the Verge’s β€œDecoder” podcast with Nilay Patel, and was asked about Route Share:

I read this press release announcing Route Share, and I had this very mid-2010s reaction, which was what if Uber just invented a bus. Did you just invent a bus?

I think to some extent it’s inspired by the bus. If you step back a little bit, a part of us looking to expand and grow is about making Uber more affordable to more people. I think one of the things that makes tech companies different from most companies out there is that our goal is to lower prices. If we lower the price, then we can extend the audience.

There is more to Khosrowshahi’s answer, but I am going to interject with three objections. First, the idea that Route Share is β€œinspired” β€œto some extent” by a bus is patently ridiculous β€” it is a vehicle with multiple passengers who embark and disembark at fixed points along a fixed route. It is a bus. A bad one, but a bus.

Second, tech companies are not the only kinds of companies that want to lower prices. Basically every consumer business is routinely marketed on lowering prices and saving customers money. This is the whole entire concept of big box stores like Costco and Walmart. Whether they are actually saving people money is a whole different point.

Which brings me to my third objection, which is that Uber has been raising prices, not reducing them. In the past year, according to a Gridwise report, Uber’s fares increased by 7.2% in the United States, even though driver pay fell 3.4%. Uber has been steadily increasing its average fare since 2018, probably to set the groundwork for its 2019 initial public offering.

Patel does not raise any similar objections.

Anyway, back to Khosrowshahi:

There are two ways of lowering price as it relates to Route Share. One is you get more than one person to share a car because cars cost money, drivers’ time costs money, etc., or you reduce the size or price of the vehicle. And we’re doing that actively. For example, with two-wheelers and three-wheelers in a lot of countries. We’ve been going after this shared concept, which is a bus, for many, many years. We started with UberX Share, for example, which is on-demand sharing.

But this concept takes it to the next level. If you schedule and create consistency among routes, then I think we can up the matching quotient, so to speak, and then essentially pass the savings on to the consumer. So, call it a next-gen bus, but the goal is just to reduce prices to the consumer and then help with congestion and the environment. That’s all good as well.

Given the premise of β€œyou get more than one person to share a car because cars cost money”, you might think Khosrowshahi would discuss the advantageous economics of increasing vehicle capacity. Instead, he cleverly pivots to smaller vehicles, despite Khosrowshahi and Patel discussing earlier how often their Uber ride occurs in a Toyota Highlander β€” a β€œmid-size” but still large SUV. This is an obviously inefficient way of moving one driver and one passenger around a city.

We just need better public transit. We should have an adequate supply of taxis, yes, but it is vastly better for everyone if we improve our existing infrastructure of trains and buses. Part of the magic of living in a city is the viability of shared public services like these.

βŒ₯ Permalink

An Elixir of Production, Not of Craft

By: Nick Heer
24 May 2025 at 03:53

Greg Storey begins this piece with a well-known quote from Plato’s β€œPhaedrus”, in which the invention of writing is decried as β€œan elixir not of memory, but of reminding”. Storey compares this to a criticism of large language models, and writes:

Even though Plato thought writing might kill memory, he still wrote it down.

But this was not Plato’s thought β€” it was the opinion of Socrates expressed through Thamus. Socrates was too dismissive of the written word for a reason he believed worthwhile β€” that memory alone is a sufficient marker of intelligence and wisdom.

If anything, I think Storey’s error in attribution actually reinforces the lesson we can draw from it. If we relied on the pessimism of Socrates, we might not know what he said today; after all, human memory is faulty. Because Plato bothered to write it down, we can learn from it. But the ability to interpret it remains ours.

What struck me most about this article, though, is this part:

The real threat to creativity isn’t a language model. It’s a workplace that rewards speed over depth, scale over care, automation over meaning. If we’re going to talk about what robs people of agency, let’s start there. […]

Thanks to new technologies β€” from writing to large language models, from bicycles to jets β€” we are able to dramatically increase the volume of work done in our waking hours and that, in turn, increases the pressure to produce even more. The economic term for this is β€œproductivity”, which I have always disliked. It distills everything down to the ratio of input effort compared to output value. In its most raw terms, it rewards the simplistic view of what a workplace ought to be, as Storey expresses well.

βŒ₯ Permalink

Reflecting on Tom Cruise’s Stunt Work

By: Nick Heer
23 May 2025 at 21:45

Ryan Francis Bradley, New York Times Magazine:

Only β€” what if we did know exactly how he did the thing, and why? Before the previous installment of the franchise, β€œDead Reckoning,” Paramount released a nine-minute featurette titled β€œThe Biggest Stunt in Cinema History.” It was a behind-the-scenes look at that midair-motorbike moment, tracking how Cruise and his crew pulled it off. We saw a huge ramp running off the edge of a Norwegian fjord. We heard about Cruise doing endless motocross jumps as preparation (13,000 of them, the featurette claims) and skydiving repeatedly (more than 500 dives). We saw him touching down from a jump, his parachute still airborne above him, and giving the director Christopher McQuarrie a dap and a casual β€œHey, McQ.” We heard a chorus of stunt trainers telling us how fantastic Cruise is (β€œan amazing individual,” his base-jumping coach says). And we hear from Cruise himself, asking his driving question: β€œHow can we involve the audience?”

The featurette was an excellent bit of Tom Cruise propaganda and a compelling look at his dedication to (or obsession with) his own mythology (or pathology). But for the movie itself, the advance release of this featurette was completely undermining. When the jump scene finally arrived, it was impossible to ignore what you already knew about it. […]

Not only was the stunt compromised by the featurette, the way it was shot and edited did not help matters. Something about it does not look quite right β€” maybe it is the perpetual late afternoon light β€” and the whole sequence feels unbelievable. That is, I know Cruise is the one performing the stunt, but if I found out each shot contained a computer-generated replacement for Cruise, it would not surprise me.

I am as excited for this instalment as anyone. I hope it looks as good as a $300 million blockbuster should. But the way this franchise has been shot since β€œFallout” has been a sore spot for me and, with the same director, cinematographer, and editor as β€œDead Reckoning”, I cannot imagine why it would be much different.

βŒ₯ Permalink

Tim Cook Called Texas Governor to Stop App Store Age Checking Legislation

By: Nick Heer
23 May 2025 at 21:34

Rolfe Winkler, Amrith Ramkumar, and Meghan Bobrowsky, Wall Street Journal:

Apple stepped up efforts in recent weeks to fight Texas legislation that would require the iPhone-maker to verify ages of device users, even drafting Chief Executive Tim Cook into the fight.

The CEO called Texas Gov. Greg Abbott last week to ask for changes to the legislation or, failing that, for a veto, according to people familiar with the call. These people said that the conversation was cordial and that it made clear the extent of Apple’s interest in stopping the bill.

Abbott has yet to say whether he will sign it, though it passed the Texas legislature with veto-proof majorities.

This comes just a few months after Apple announced it would be introducing age range APIs in iOS later this year. Earlier this month, U.S. lawmakers announced federal bills with the same intent. This is clearly the direction things are going. Is there something specific in Texas’ bill that makes it particularly objectionable? Or is it simply the case Apple and Google would prefer a single federal law instead of individual state laws?

βŒ₯ Permalink

Sponsor: Magic Lasso Adblock: 2.0Γ— Faster Web Browsing in Safari

By: Nick Heer
23 May 2025 at 17:30

Want to experience twice as fast load times in Safari on your iPhone, iPad, and Mac?

Then download Magic Lasso Adblock β€” the ad blocker designed for you.

Magic Lasso Adblock: 2.0Γ— Faster Web Browsing in Safari

As an efficient, high performance and native Safari ad blocker, Magic Lasso blocks all intrusive ads, trackers, and annoyances – delivering a faster, cleaner, and more secure web browsing experience.

By cutting down on ads and trackers, common news websites load 2Γ— faster and browsing uses less data while saving energy and battery life.

Rely on Magic Lasso Adblock to:

  • Improve your privacy and security by removing ad trackers

  • Block all YouTube ads, including pre-roll video ads

  • Block annoying cookie notices and privacy prompts

  • Double battery life during heavy web browsing

  • Lower data usage when on the go

With over 5,000 five star reviews, it’s simply the best ad blocker for your iPhone, iPad, and Mac.

And unlike some other ad blockers, Magic Lasso Adblock respects your privacy, doesn’t accept payment from advertisers, and is 100% supported by its community of users.

So, join over 350,000 users and download Magic Lasso Adblock today.

βŒ₯ Permalink

U.S. Spy Agencies Get One-Stop Shop to Buy Personal Data

By: Nick Heer
22 May 2025 at 23:24

Remember how, in 2023, the U.S. Office of the Director of National Intelligence published a report acknowledging mass stockpiling of third-party data it had purchased? It turns out there is so much private information about people it is creating a big headache for the intelligence agencies β€” not because of any laws or ethical qualms, but simply because of the sheer volume.

Sam Biddle, the Intercept:

The Office of the Director of National Intelligence is working on a system to centralize and β€œstreamline” the use of commercially available information, or CAI, like location data derived from mobile ads, by American spy agencies, according to contract documents reviewed by The Intercept. The data portal will include information deemed by the ODNI as highly sensitive, that which can be β€œmisused to cause substantial harm, embarrassment, and inconvenience to U.S. persons.” The documents state spy agencies will use the web portal not just to search through reams of private data, but also run them through artificial intelligence tools for further analysis.

Apparently, the plan is to feed all this data purchased from brokers and digital advertising companies into artificial intelligence systems. The DNI says it has rules about purchasing and using this data, so there is nothing to worry about.

By the way, the DNI’s Freedom of Information Act page was recently updated to remove links to released records and FOIA logs. They were live on May 5 but, as of May 16, those pages have been removed, and direct links no longer resolve either. Strange.

Update: The ODNI told me its β€œwebsite is currently under construction”.

βŒ₯ Permalink

Speculating About the Hardware Ambitions of OpenAI

By: Nick Heer
22 May 2025 at 16:48

Berber Jin, Wall Street Journal:

Altman and Ive offered a few hints at the secret project they have been working on [at a staff meeting]. The product will be capable of being fully aware of a user’s surroundings and life, will be unobtrusive, able to rest in one’s pocket or on one’s desk, and will be a third core device a person would put on a desk after a MacBook Pro and an iPhone.

Ambitious, albeit marginally less hubristic than considering it a replacement for either of those two device categories.

Stephen Hackett:

If OpenAI’s future product is meant to work with the iPhone and Android phones, then the company is opening a whole other set of worms, from the integration itself to the fact that most people will still prefer to simply pull their phone out of their pockets for basically any task.

I am reminded of an April 2024 article by Jason Snell at Six Colors:

The problem is that I’m dismissing the Ai Pin and looking forward to the Apple Watch specifically because of the control Apple has over its platforms. Yes, the company’s entire business model is based on tightly integrating its hardware and software, and it allows devices like the Apple Watch to exist. But that focus on tight integration comes at a cost (to everyone but Apple, anyway): Nobody else can have the access Apple has.

A problem OpenAI could have with this device is the same as was faced by Humane, which is that Apple treats third-party hardware and software as second-class citizens in its post-P.C. ecosystem. OpenAI is laying the groundwork for better individual context. But this is a significant limitation, and it is one I am curious to see how it is overcome.

Whatever this thing is, it is undeniably interesting to me. OpenAI has become a household name on a foundation of an academic-sounding product that has changed the world. Jony Ive has been the name attached to entire eras of design. There is plenty to criticize about both. Yet the combination of these things is surely intriguing, inviting the kind of speculation that used to be commonplace in tech before it all became rote. I have little faith our world will become meaningfully better with another gadget in it. Yet I hope the result is captivating, at least, because we could use some of that.

βŒ₯ Permalink

GeoGuessr Community Maps Go Dark in Protest of EWC Ties to Human Rights Abuses

By: Nick Heer
22 May 2025 at 03:34

Jessica Conditt, Engadget:

A group of GeoGuessr map creators have pulled their contributions from the game to protest its participation in the Esports World Cup 2025, calling the tournament β€œa sportswashing tool used by the government of Saudi Arabia to distract from and conceal its horrific human rights record.” The protestors say the blackout will hold until the game’s publisher, GeoGuessr AB, cancels its planned Last Chance Wildcard tournament at the EWC in Riyadh, Saudi Arabia, from July 21 to 27.

Those participating in this blackout created some of the most popular and notable maps in the game. Good for them.

Update: GeoGuessr says it is withdrawing from the EWC.

βŒ₯ Permalink

The Carbon Footprint Sham

By: Nick Heer
22 May 2025 at 03:05

Thinking about the energy β€œfootprint” of artificial intelligence products makes it a good time to re-link to Mark Kaufman’s excellent 2020 Mashable article in which he explores the idea of a carbon footprint:

The genius of the β€œcarbon footprint” is that it gives us something to ostensibly do about the climate problem. No ordinary person can slash 1 billion tons of carbon dioxide emissions. But we can toss a plastic bottle into a recycling bin, carpool to work, or eat fewer cheeseburgers. β€œPsychologically we’re not built for big global transformations,” said John Cook, a cognitive scientist at the Center for Climate Change Communication at George Mason University. β€œIt’s hard to wrap our head around it.”

Ogilvy & Mather, the marketers hired by British Petroleum, wove the overwhelming challenges inherent in transforming the dominant global energy system with manipulative tactics that made something intangible (carbon dioxide and methane β€” both potent greenhouse gases β€” are invisible), tangible. A footprint. Your footprint.

The framing of most of the A.I. articles I have seen thankfully shies away from ascribing individual blame; instead, they point to systemic flaws. This is preferable, but it still does little at the scale of electricity generation worldwide.

βŒ₯ Permalink

The Energy Footprint of A.I.

By: Nick Heer
21 May 2025 at 23:49

Casey Crownhart, MIT Technology Review:

Today, new analysis by MIT Technology Review provides an unprecedented and comprehensive look at how much energy the AI industry uses β€” down to a single query β€” to trace where its carbon footprint stands now, and where it’s headed, as AI barrels towards billions of daily users.

We spoke to two dozen experts measuring AI’s energy demands, evaluated different AI models and prompts, pored over hundreds of pages of projections and reports, and questioned top AI model makers about their plans. Ultimately, we found that the common understanding of AI’s energy consumption is full of holes.

This robust story comes on the heels of a series of other discussions about how much energy is used by A.I. products and services. Last month, for example, Andy Masley published a comparison of using ChatGPT against other common activities. The Economist ran another, and similar articles have been published before. As far as I can tell, they all come down to the same general conclusion: training A.I. models is energy-intensive, using A.I. products is not, lots of things we do online and offline have a greater impact on the environment, and the current energy use of A.I. is the lowest it will be from now on.

There are lots of good reasons to critique artificial intelligence. I am not sure its environmental impact is a particularly strong one; I think the true energy footprint of tech companies, of which A.I. is one part, is more relevant. Even more pressing, however, is our need to electrify our world as much as we can, and that will require a better and cleaner grid.

βŒ₯ Permalink

Jony Ive’s β€˜io’ Acquired by OpenAI; Ive to Remain as Designer

By: Nick Heer
21 May 2025 at 18:06

Last month, the Information reported OpenAI was considering buying io Products β€” unfortunate capitalization theirs β€” for around $500 million. The company, founded by Jony Ive and employing several ex-Apple designers and engineers, was already known to be working with OpenAI, but it was still an external entity. Now, it is not, to the tune of over $6 billion in equity.

OpenAI today published a press release and video β€” set in LoveFrom’s distinctive proprietary serif face β€” featuring Ive and Sam Altman in conversation. There is barely a hint of what they are working on but, whether because of honesty or just clever packaging, it comes across as an earnest attempt to think about the new technologies OpenAI has successfully brought to the world as part of our broader cultural fabric. Of course, it will be expressed in something that can be assembled in a factory and sold for money, so let us not get too teary-eyed. We have heard a similar tune before.

The video promises revealing something β€œnext year”.

βŒ₯ Permalink

Two Major Newspapers Published an A.I.-Generated Guide to Summer Books That Do Not Exist

By: Nick Heer
21 May 2025 at 00:10

Albert Burneko, Defector:

Over this past weekend, the Chicago Sun-Times and Philadelphia Inquirer’s weekend editions included identical huge β€œBest of Summer” inserts; in the Inquirer’s digital edition the insert runs 54 pages, while the entire rest of the paper occupies 36. Before long, readers began noticing something strange about the β€œSummer reading list for 2025” section of the insert. Namely, that while the list includes some very well-known authors, most of the books listed in it do not exist.

This is the kind of fluffy insert long purchased by publishers to pad newspapers. In this case, it appears to be produced by Hearst Communications, which feels about right for something with Hearst’s name on it. I cannot imagine most publishers read these things very carefully; adding more work or responsibility is not the point of buying a guide like this.

What I found very funny today was watching the real-time reporting of this story in parallel with Google’s I/O presentation, at which it announced one artificial intelligence feature after another. On the one hand, A.I. features can help you buy event tickets or generate emails offering travel advice based on photos from trips you have taken. On the other, it is inventing books, experts, and diet advice.

βŒ₯ Permalink

My blocking of some crawlers is an editorial decision unrelated to crawl volume

By: cks
30 May 2025 at 02:33

Recently I read a lobste.rs comment on one of my recent entries that said, in part:

Repeat after me everyone: the problem with these scrapers is not that they scrape for LLM’s, it’s that they are ill-mannered to the point of being abusive. LLM’s have nothing to do with it.

This may be some people's view but it is not mine. For me, blocking web scrapers here on Wandering Thoughts is partly an editorial decision of whether I want any of my resources or my writing to be fed into whatever they're doing. I will certainly block scrapers for doing what I consider an abusive level of crawling, and in practice most of the scrapers that I block come to my attention due to their volume, but I will block low-volume scrapers because I simply don't like what they're doing it for.

Are you a 'brand intelligence' firm that scrapes the web and sells your services to brands and advertisers? Blocked. In general, do you charge for access to whatever you're generating from scraping me? Probably blocked. Are you building a free search site for a cause (and with a point of view) that I don't particularly like? Almost certainly blocked. All of this is an editorial decision on my part on what I want to be even vaguely associated with and what I don't, not a technical decision based on the scraping's effects on my site.

I am not going to even bother trying to 'justify' this decision. It's a decision that needs no justification to some and to others, it's one that can never be justified. My view is that ethics matter. Technology and our decisions of what to do with technology are not politically neutral. We can make choices, and passively not doing anything is a choice too.

(I could say a lot of things here, probably badly, but ethics and politics are in part about what sort of a society we want, and there's no such thing as a neutral stance on that. See also.)

I would block LLM scrapers regardless of how polite they are. The only difference them being politer would make is that I would be less likely to notice (and then block) them. I'm probably not alone in this view.

Our Grafana and Loki installs have quietly become 'legacy software' here

By: cks
29 May 2025 at 03:00

At this point we've been running Grafana for quite some time (since late 2018), and (Grafana) Loki for rather less time and on a more ad-hoc and experimental basis. However, over time both have become 'legacy software' here, by which I mean that we (I) have frozen their versions and don't update them any more, and we (I) mostly or entirely don't touch their configurations any more (including, with Grafana, building or changing dashboards).

We froze our Grafana version due to backward compatibility issues. With Loki I could say that I ran out of enthusiasm for going through updates, but part of it was that Loki explicitly deprecated 'promtail' in favour of a more complex solution ('Alloy') that seemed to mostly neglect the one promtail feature we seriously cared about, namely reading logs from the systemd/journald complex. Another factor was it became increasingly obvious that Loki was not intended for our simple setup and future versions of Loki might well work even worse in it than our current version does.

Part of Grafana and Loki going without updates and becoming 'legacy' is that any future changes in them would be big changes. If we ever have to update our Grafana version, we'll likely have to rebuild a significant number of our current dashboards, because they use panels that aren't supported any more and the replacements have a quite different look and effect, requiring substantial dashboard changes for the dashboards to stay decently usable. With Loki, if the current version stopped working I'd probably either discard the idea entirely (which would make me a bit sad, as I've done useful things through Loki) or switch to something else that had similar functionality. Trying to navigate the rapids of updating to a current Loki is probably roughly as much work (and has roughly as much chance of requiring me to restart our log collection from scratch) as moving to another project.

(People keep mentioning VictoriaLogs (and I know people have had good experiences with it), but my motivation for touching any part of our Loki environment is very low. It works, it hasn't eaten the server it's on and shows no sign of doing that any time soon, and I'm disinclined to do any more work with smart log collection until a clear need shows up. Our canonical source of history for logs continues to be our central syslog server.)

Intel versus AMD is currently an emotional decision for me

By: cks
28 May 2025 at 02:40

I recently read Michael Stapelberg's My 2025 high-end Linux PC. One of the decisions Stapelberg made was choosing an Intel (desktop) CPU because of better (ie lower) idle power draw. This is a perfectly rational decision to make, one with good reasoning behind it, and also as I read the article I realized that it was one I wouldn't have made. Not because I don't value idle power draw; like Stapelberg's machine but more so, my desktops spend most of their time essentially idle. Instead, it was because I realized (or confirmed my opinion) that right now, I can't stand to buy Intel CPUs.

I am tired of all sorts of aspects of Intel. I'm tired of their relentless CPU product micro-segmentation across desktops and servers, with things like ECC allowed in some but not all models. I'm tired of their whole dance of P-cores and E-cores, and also of having to carefully read spec sheets to understand the P-core and E-core tradeoffs for a particular model. I'm tired of Intel just generally being behind AMD and repeatedly falling on its face with desperate warmed over CPU refreshes that try to make up for its process node failings. I'm tired of Intel's hardware design failure with their 13th and 14th generation CPUs (see eg here). I'm sure AMD Ryzens have CPU errata too that would horrify me if I knew, but they're not getting rubbed in my face the way the Intel issue is.

At this point Intel has very little going for its desktop CPUs as compared to the current generation AMD Ryzens. Intel CPUs have better idle power levels, and may have better single-core burst performance. In absolute performance I probably won't notice much difference, and unlike Stapelberg I don't do the kind of work where I really care about build speed (and if I do, I have access to much more powerful machines). As far as the idle power goes, I likely will notice the better idle power level (some of the time), but my system is likely to idle at lower power in general than Stapelberg's will, especially at home where I'll try to use the onboard graphics if at all possible (so I won't have the (idle) power price of a GPU card).

(At work I need to drive two 4K displays at 60Hz and I don't think there are many motherboards that will do that with onboard graphics, even if the CPU's built in graphics system is up to it in general.)

But I don't care about the idle power issue. If or when I build a new home desktop, I'll eat the extra 20 watts or so of idle power usage for an AMD CPU (although this may vary in practice, especially with screens blanked). And I'll do it because right now I simply don't want to give Intel my money.

My GNU Emacs settings for the vertico package (as of mid 2025)

By: cks
27 May 2025 at 02:38

As covered in my Emacs packages, vertico is one of the third party Emacs packages that I have installed to modify how minibuffer completion works for me, or at least how it looks. In my experience, vertico took a significant amount of customization before I really liked it (eventually including some custom code), so I'm going to write down some notes about why I made various settings.

Vertico itself is there to always show me a number of the completion targets, as a help to narrowing in on what I want; I'm willing to trade vertical space during completion for a better view of what I'm navigating around. It's not the only way to do this (there's fido-vertical-mode in standard GNU Emacs, for example), but it's what I started with and it has a number of settings that let me control both how densely the completions are presented (and so how many of them I get to see at once) and how they're presented.

The first thing I do with vertico is override its key binding for TAB, because I want standard Emacs minibuffer tab completion, not vertico's default behavior of inserting the current thing completion is currently on. Specifically, my key bindings are:

 :bind (:map vertico-map
             ("TAB" . minibuffer-complete)
             ;; M-v is taken by vertico
             ("M-g M-c" . switch-to-completions)
             ;; Original tab binding, which we want sometimes when
             ;; using orderless completion.
             ("M-TAB" . vertico-insert))

I normally work by using regular tab completion and orderless's completion until I'm happy, then hitting M-TAB if necessary and then RET. I use M-g M-c so rarely that I'd forgotten it until writing this entry. Using M-TAB is especially likely for a long filename completion, where I might use the cursor keys (or theoretically the mouse) to move vertico's selection to a directory and then hit M-TAB to fill it in so I can then tab-complete within it.

Normally, vertico displays a single column of completion candidates, which potentially leaves a lot of wasted space on the right; I use marginalia to add information some sorts of completion targets (such as Emacs Lisp function names) in this space. For other sorts of completions where there's no particular additional information, such as MH-E mail folder names, I use vertico's vertico-multiform-mode to switch to a vertico-grid so I fill the space with several columns of completion candidates and reduce the number of vertical lines that vertico uses (both are part of vertico's extensions).

(I also have vertico-mouse enabled when I'm using Emacs under X, but in practice I mostly don't use it.)

Another important change (for me) is to turn off vertico's default behavior of remembering the history of your completions and putting recently used entries first in the list. This sounds like a fine idea, but in practice I want my completion order to be completely predictable and I'm rarely completing the same thing over and over again. The one exception is my custom MH-E folder completion, where I do enable history because I may be, for example, refiling messages into one of a few folders. This is done through another extension, vertico-sort, or at least I think it is.

(When vertico is installed as an ELPA or MELPA package and then use-package'd, you apparently get all of the extensions without necessarily having to specifically enable them and can just use bits from them.)

My feeling is that effective use of vertico probably requires this sort of customization if you regularly use minibuffer completion for anything beyond standard things where vertico (and possibly marginalia) can make good use of all of your horizontal space. Beyond what key bindings and other vertico behavior you can stand and what behavior you have to change, you want to figure out how to tune vertico so that it's significantly useful for each thing you regularly complete, instead of mostly showing you a lot of empty space and useless results. This is intrinsically a relatively personal thing.

PS: One area where vertico's completion history is not as useful as it looks is filename completion or anything that looks like it (such as standard MH-E folder completion). This is because Emacs filename completion and thus vertico's history happens component by component, while you probably want your history to give you the full path that you wound up completing.

PPS: I experimented with setting vertico-resize, but found that the resulting jumping around was too visually distracting.

A thought on JavaScript "proof of work" anti-scraper systems

By: cks
26 May 2025 at 02:50

One of the things that people are increasingly using these days to deal with the issue of aggressive LLM and other web scrapers is JavaScript based "proof of work" systems, where your web server requires visiting clients to run some JavaScript to solve a challenge; one such system (increasingly widely used) is Xe Iaso's Anubis. One of the things that people say about these systems is that LLM scrapers will just start spending the CPU time to run this challenge JavaScript, and LLM scrapers may well have lots of CPU time available through means such as compromised machines. One of my thoughts is that things are not quite as simple for the LLM scrapers as they look.

An LLM scraper is operating in a hostile environment (although its operator may not realize this). In a hostile environment, dealing with JavaScript proof of work systems is not as simple as simply running it, because you can't particularly tell a JavaScript proof of work system from JavaScript that does other things. Letting your scraper run JavaScript means that it can also run JavaScript for other purposes, for example for people who would like to exploit your scraper's CPU to do some cryptocurrency mining, or simply have you run JavaScript for as long as you'll let it keep going (perhaps because they've recognized you as a LLM scraper and want to waste as much of your CPU as possible).

An LLM scraper can try to recognize a JavaScript proof of work system but this is a losing game. The other parties have every reason to make themselves look like a proof of work system, and the proof of work systems don't necessarily have an interest in being recognized (partly because this might allow LLM scrapers to short-cut their JavaScript with optimized host implementations of the challenges). And as both spammers and cryptocurrency miners have demonstrated, there is no honor among thieves. If LLM scrapers dangle free computation in front of people, someone will spring up to take advantage of it. This leaves LLM scrapers trying to pick a JavaScript runtime limit that doesn't cut them off from too many sites, while sites can try to recognize LLM scrapers and increase their proof of work difficulty if they see a suspect.

(This is probably not an original thought, but it's been floating around my head for a while.)

PS: JavaScript proof of work systems aren't the greatest thing, but they're going to happen unless someone convincingly demonstrates a better alternative.

The length of file names in early Unix

By: cks
25 May 2025 at 01:33

If you use Unix today, you can enjoy relatively long file names on more or less any filesystem that you care to name. But it wasn't always this way. Research V7 had 14-byte filenames, and the System III/System V lineage continued this restriction until it merged with BSD Unix, which had significantly increased this limit as part of moving to a new filesystem (initially called the 'Fast File System', for good reasons). You might wonder where this unusual number came from, and for that matter, what the file name limit was on very early Unixes (it was 8 bytes, which surprised me; I vaguely assumed that it had been 14 from the start).

I've mentioned before that the early versions of Unix had a quite simple format for directory entries. In V7, we can find the directory structure specified in sys/dir.h (dir(5) helpfully directs you to sys/dir.h), which is so short that I will quote it in full:

#ifndef	DIRSIZ
#define	DIRSIZ	14
#endif
struct	direct
{
    ino_t    d_ino;
    char     d_name[DIRSIZ];
};

To fill in the last blank, ino_t was a 16-bit (two byte) unsigned integer (and field alignment on PDP-11s meant that this structure required no padding), for a total of 16 bytes. This directory structure goes back to V4 Unix. In V3 Unix and before, directory entries were only ten bytes long, with 8 byte file names.

(Unix V4 (the Fourth Edition) was when the kernel was rewritten in C, so that may have been considered a good time to do this change. I do have to wonder how they handled the move from the old directory format to the new one, since Unix at this time didn't have multiple filesystem types inside the kernel; you just had the filesystem, plus all of your user tools knew the directory structure.)

One benefit of the change in filename size is that 16-byte directory entries fit evenly in 512-byte disk blocks (or other powers-of-two buffer sizes). You never have a directory entry that spans two disk blocks, so you can deal with directories a block at a time. Ten byte directory entries don't have this property; eight-byte ones would, but then that would leave space for only six character file names, and presumably that was considered too small even in Unix V1.

PS: That inode numbers in V7 (and earlier) were 16-bit unsigned integers does mean what you think it means; there could only be at most 65,536 inodes in a single classical V7 filesystem. If you needed more files, you had better make more filesystems. Early Unix had a lot of low limits like that, some of them quite hard-coded.

What keeps Wandering Thoughts more or less free of comment spam (2025 edition)

By: cks
24 May 2025 at 02:50

Like everywhere else, Wandering Thoughts (this blog) gets a certain amount of automated comment spam attempts. Over the years I've fiddled around with a variety of anti-spam precautions, although not all of them have worked out over time. It's been a long time since I've written anything about this, because one particular trick has been extremely effective ever since I introduced it.

That one trick is a honeypot text field in my 'write a comment' form. This field is normally hidden by CSS, and in any case the label for the field says not to put anything in it. However, for a very long time now, automated comment spam systems seem to operate by stuffing some text into every (text) form field that they find before they submit the form, which always trips over this. I log the form field's text out of curiosity; sometimes it's garbage and sometimes it's (probably) meaningful for the spam comment that the system is trying to submit.

Obviously this doesn't stop human-submitted spam, which I get a small amount of every so often. In general I don't expect anything I can reasonably do to stop humans who do the work themselves; we've seen this play out in email and I don't have any expectations that I can do better. It also probably wouldn't work if I was using a popular platform that had this as a general standard feature, because then it would be worth the time of the people writing automated comment spam systems to automatically recognize it and work around it.

Making comments on Wandering Thoughts also has an additional small obstacle in the way of automated comment spammers, which is that you must initially preview your comment before you can submit it (although you don't have to submit the comment that you previewed, you can edit it after the first preview). Based on a quick look at my server logs, I don't think this matters to the current automated comment spam systems that try things here, as they only appear to try submitting once. I consider requiring people to preview their comment before posting it to be a good idea in general, especially since Wandering Thoughts uses a custom wiki-syntax and a forced preview gives people some chance of noticing any mistakes.

(I think some amount of people trying to write comments here do miss this requirement and wind up not actually posting their comment in the end. Or maybe they decide not to after writing one version of it; server logs give me only so much information.)

In a world that is increasingly introducing various sorts of aggressive precautions against LLM crawlers, including 'proof of work' challenges, all of this may become increasingly irrelevant. This could go either way; either the automated comment spammers die off as more and more systems have protections that are too aggressive for them to deal with, or the automated systems become increasingly browser-based and sidestep my major precaution because they no longer 'see' the honeypot field.

Fedora's DNF 5 and the curse of mandatory too-smart output

By: cks
23 May 2025 at 02:49

DNF is Fedora's high(er) level package management system, which pretty much any system administrator is going to have to use to install and upgrade packages. Fedora 41 and later have switched from DNF 4 to DNF 5 as their normal (and probably almost mandatory) version of DNF. I ran into some problems with this switch, and since then I've found other issues, all of which boil down to a simple issue: DNF 5 insists on doing too-smart output.

Regardless of what you set your $TERM to and what else you do, if DNF 5 is connected to a terminal (and perhaps if it isn't), it will pretty-print its output in an assortment of ways. As far as I can tell it simply assumes ANSI cursor addressability, among other things, and will always fit its output to the width of your terminal window, truncating output as necessary. This includes output from RPM package scripts that are running as part of the update. Did one of them print a line longer than your current terminal width? Tough, it was probably truncated. Are you using script so that you can capture and review all of the output from DNF and RPM package scripts? Again, tough, you can't turn off the progress bars and other things that will make a complete mess of the typescript.

(It's possible that you can find the information you want in /var/log/dnf5.log in un-truncated and readable form, but if so it's buried in debug output and I'm not sure I trust dnf5.log in general.)

DNF 5 is far from the only offender these days. An increasing number of command line programs simply assume that they should always produce 'smart' output (ideally only if they're connected to a terminal). They have no command line option to turn this off and since they always use 'ANSI' escape sequences, they ignore the tradition of '$TERM' and especially 'TERM=dumb' to turn that off. Some of them can specifically disable colour output (typically with one of a number of environment variables, which may or may not be documented, and sometimes with a command line option), but that's usually the limits of their willingness to stop doing things. The idea of printing one whole line at a time as you do things and not printing progress bars, interleaving output, and so on has increasingly become a non-starter for modern command line tools.

(Another semi-offender is Debian's 'apt' and also 'apt-get' to some extent, although apt-get's progress bars can be turned off and 'apt' is explicitly a more user friendly front end to apt-get and friends.)

PS: I can't run DNF with its output directed into a file because it wants you to interact with it to approve things, and I don't feel like letting it run freely without that.

Thinking about what you'd want in a modern simple web server

By: cks
22 May 2025 at 02:14

Over on the Fediverse, I said:

I'm currently thinking about what you'd want in a simple modern web server that made life easy for sites that weren't purely static. I think you want CGI, FastCGI, and HTTP reverse proxying, plus process supervision. Automatic HTTPS of course. Rate limiting support, and who knows what you'd want to make it easier to deal with the LLM crawler problem.

(This is where I imagine a 'stick a third party proxy in the middle' mode of operation.)

What I left out of my Fediverse post is that this would be aimed at small scale sites. Larger, more complex sites can and should invest in the power, performance, and so on of headline choices like Apache, Nginx, and so on. And yes, one obvious candidate in this area is Caddy, but at the same time something that has "more scalable" (than alternatives) as a headline features is not really targeting the same area as I'm thinking of.

This goal of simplicity of operation is why I put "process supervision" into the list of features. In a traditional reverse proxy situation (whether this is FastCGI or HTTP), you manage the reverse proxy process separately from the main webserver, but that requires more work from you. Putting process supervision into the web server has the goal of making all of that more transparent to you. Ideally, in common configurations you wouldn't even really care that there was a separate process handling FastCGI, PHP, or whatever; you could just put things into a directory or add some simple configuration to the web server and restart it, and everything would work. Ideally this would extend to automatically supporting PHP by just putting PHP files somewhere in the directory tree, just like CGI; internally the web server would start a FastCGI process to handle them or something.

(Possibly you'd implement CGI through a FastCGI gateway, but if so this would be more or less pre-configured into the web server and it'd ship with a FastCGI gateway for this (and for PHP).)

This is also the goal for making it easy to stick a third party filtering proxy in the middle of processing requests. Rather than having to explicitly set up two web servers (a frontend and a backend) with an anti-LLM filtering proxy in the middle, you would write some web server configuration bits and then your one web server would split itself into a frontend and a backend with the filtering proxy in the middle. There's no technical reason you can't do this, and even control what's run through the filtering proxy and what's served directly by the front end web server.

This simple web server should probably include support for HTTP Basic Authentication, so that you can easily create access restricted areas within your website. I'm not sure if it should include support for any other sort of authentication, but if it did it would probably be OpenID Connect (OIDC), since that would let you (and other people) authenticate through external identity providers.

It would be nice if the web server included some degree of support for more or less automatic smart in-memory (or on-disk) caching, so that if some popular site linked to your little server, things wouldn't explode (or these days, if a link to your site was shared on the Fediverse and all of the Fediverse servers that it propagated to immediately descended on your server). At the very least there should be enough rate limiting that your little server wouldn't fall over, and perhaps some degree of bandwidth limits you could set so that you wouldn't wake up to discover you had run over your outgoing bandwidth limits and were facing large charges.

I doubt anyone is going to write such a web server, since this isn't likely to be the kind of web server that sets the world on fire, and probably something like Caddy is more or less good enough.

(Doing a good job of writing such a server would also involve a fair amount of research to learn what people want to run at a small scale, how much they know, what sort of server resources they have or want to use, what server side languages they wind up using, what features they need, and so on. I certainly don't know enough about the small scale web today.)

PS: One reason I'm interested in this is that I'd sort of like such a server myself. These days I use Apache and I'm quite familiar with it, but at the same time I know it's a big beast and sometimes it has entirely too many configuration options and special settings and so on.

The five platforms we have to cover when planning systems

By: cks
21 May 2025 at 03:33

Suppose, not entirely hypothetically, that you're going to need a 'VPN' system that authenticates through OIDC. What platforms do you need this VPN system to support? In our environment, the answer is that we have five platforms that we need to care about, and they're the obvious four plus one more: Windows, macOS, iOS, Android, and Linux.

We need to cover these five platforms because people here use our services from all of those platforms. Both Windows and macOS are popular on laptops (and desktops, which still linger around), and there's enough people who use Linux to be something we need to care about. On mobile devices (phones and tablets), obviously iOS and Android are the two big options, with people using either or both. We don't usually worry about the versions of Windows and macOS and suggest that people to stick to supported ones, but that may need to change with Windows 10.

Needing to support mobile devices unquestionably narrows our options for what we can use, at least in theory, because there are certain sorts of things you can semi-reasonably do on Linux, macOS, and Windows that are infeasible to do (at least for us) on mobile devices. But we have to support access to various of our services even on iOS and Android, which constrains us to certain sorts of solutions, and ideally ones that can deal with network interruptions (which are quite common on mobile devices in Toronto, as anyone who takes our subways is familiar with).

(And obviously it's easier for open source systems to support Linux, macOS, and Windows than it is for them to extend this support to Android and especially iOS. This extends to us patching and rebuilding them for local needs; with various modern languages, we can produce Windows or macOS binaries from modified open source projects. Not so much for mobile devices.)

In an ideal world it would be easy to find out the support matrix of platforms (and features) for any given project. In this world, the information can sometimes be obscure, especially for what features are supported on what platforms. One of my resolutions to myself is that when I find interesting projects but they seem to have platform limitations, I should note down where in their documentation they discuss this, so I can find it later to see if things have changed (or to discuss with people why certain projects might be troublesome).

Python, type hints, and feeling like they create a different language

By: cks
20 May 2025 at 02:31

At this point I've only written a few, relatively small programs with type hints. At times when doing this, I've wound up feeling that I was writing programs in a language that wasn't quite exactly Python (but obviously was closely related to it). What was idiomatic in one language was non-idiomatic in the other, and I wanted to write code differently. This feeling of difference is one reason I've kept going back and forth over whether I should use type hints (well, in personal programs).

Looking back, I suspect that this is partly a product of a style where I tried to use typing.NewType a lot. As I found out, this may not really be what I want to do. Using type aliases (or just structural descriptions of the types) seems like it's going to be easier, since it's mostly just a matter of marking up things. I also suspect that this feeling that typed Python is a somewhat different language from plain Python is a product of my lack of experience with typed Python (which I can fix by doing more with types in my own code, perhaps revising existing programs to add type annotations).

However, I suspect some of this feeling of difference is that you (I) want to structure 'typed' Python code differently than untyped code. In untyped Python, duck typing is fine, including things like returning None or some meaningful type, and you can to a certain extent pass things around without caring what type they are. In this sort of situation, typed Python has pushed me toward narrowing the types involved in my code (although typing.Optional can help here). Sometimes this is a good thing; at other times, I wind up using '0.0' to mean 'this float value is not set' when in untyped Python I would use 'None' (because propagating the type difference of the second way through the code is too annoying). Or to put it another way, typed Python feels less casual, and there are good and bad aspects to this.

Unfortunately, one significant source of Python code that I work on is effectively off limits for type hints, and that's the Python code I write for work. For that code, I need to stick to the subset of Python that my co-workers know and can readily understand, and that subset doesn't include Python's type hints. I could try to teach my co-workers about type hints, but my view is that if I'm wrestling with whether it's worth it, my co-workers will be even less receptive to the idea of trying to learn and remember them (especially when they look at my Python code only infrequently). If we were constantly working with medium to large Python programs where type hints were valuable for documenting things and avoiding irritating errors it would be one thing, but as it is our programs are small and we can go months between touching any Python code. I care about Python type hints and have active exposure to them, and even I have to refresh my memory on them from time to time.

(Perhaps some day type hints will be pervasive enough in third party Python code and code examples that my co-workers will absorb and remember them through osmosis, but that day isn't today.)

The lack of a good command line way to sort IPv6 addresses

By: cks
19 May 2025 at 02:54

A few years ago, I wrote about how 'sort -V' can sort IPv4 addresses into their natural order for you. Even back then I was smart enough to put in that 'IPv4' qualification and note that this didn't work with IPv6 addresses, and said that I didn't know of any way to handle IPv6 addresses with existing command line tools. As far as I know, that remains the case today, although you can probably build a Perl, Python, or other language program that does such sorting for you if you need to do this regularly.

Unix tools like 'sort' are pretty flexible, so you might innocently wonder why it can't be coerced into sorting IPv6 addresses. The first problem is that IPv6 addresses are written in hex without leading 0s, not decimal. Conventional sort will correctly sort hex numbers if all of the numbers are the same length, but IPv6 addresses are written in hex groups that conventionally drop leading zeros, so you will have 'ff' instead of '00ff' in common output (or '0' instead of '0000'). The second and bigger problem is the IPv6 '::' notation, which stands for the longest run of all-zero fields, ie some number of '0000' fields.

(I'm ignoring IPv6 scopes and zones for this, let's assume we have public IPv6 addresses.)

If IPv6 addresses were written out in full, with leading 0s on fields and all their 0000 fields, you could handle them as a simple conventional sort (you wouldn't even need to tell sort that the field separator was ':'). Unfortunately they almost never are, so you need to either transform them to that form, print them out, sort the output, and perhaps transform them back, or read them into a program as 128-bit numbers, sort the numbers, and print them back out as IPv6 addresses. Ideally your language of choice for this has a way to sort a collection of IPv6 addresses.

The very determined can probably do this with awk with enough work (people have done amazing things in awk). But my feeling is that doing this in conventional Unix command line tools is a Turing tarpit; you might as well use a language where there's a type of IPv6 addresses that exposes the functionality that you need.

(And because IPv6 addresses are so complex, I suspect that GNU Sort will never support them directly. If you need GNU Sort to deal with them, the best option is a program that turns them into their full form.)

PS: People have probably written programs to sort IPv6 addresses, but with the state of the Internet today, the challenge is finding them.

It's not obvious how to verify TLS client certificates issued for domains

By: cks
18 May 2025 at 02:24

TLS server certificate verification has two parts; you first verify that the TLS certificate is valid, CA-signed certificate, and then you verify that the TLS certificate is for the host you're connecting to. One of the practical issues with TLS 'Client Authentication' certificates for host and domain names (which are on the way out) is that there's no standard meaning for how you do the second part of this verification, and if you even should. In particular, what host name are you validating the TLS client certificate against?

Some existing protocols provide the 'client host name' to the server; for example, SMTP has the EHLO command. However, existing protocols tend not to have explicitly standardized using this name (or any specific approach) for verifying a TLS client certificate if one is presented to the server, and large mail providers vary in what they send as a TLS client certificate in SMTP conversations. For example, Google's use of 'smtp.gmail.com' doesn't match any of the other names available, so its only meaning is 'this connection comes from a machine that has access to private keys for a TLS certificate for smtp.gmail.com', which hopefully means that it belongs to GMail and is supposed to be used for this purpose.

If there is no validation of the TLS client certificate host name, that is all that a validly signed TLS client certificate means; the connecting host has access to the private keys and so can be presumed to be 'part of' that domain or host. This isn't nothing, but it doesn't authenticate what exactly the client host is. If you want to validate the host name, you have to decide what to validate against and there are multiple answers. If you design the protocol you can have the protocol send a client host name and then validate the TLS certificate against the hostname; this is slightly better than using the TLS certificate's hostname as is in the rest of your processing, since the TLS certificate might have a wildcard host name. Otherwise, you might validate the TLS certificate host name against its reverse DNS, which is more complicated than you might expect and which will fail if DNS isn't working. If the TLS client certificate doesn't have a wildcard, you could also try to look up the IP addresses associated with the host names in the TLS certificate and see if any of the IP addresses match, but again you're depending on DNS.

(You can require non-wildcard TLS certificate names in your protocol, but people may not like it for various reasons.)

This dependency on DNS for TLS client certificates is different from the DNS dependency for TLS server certificates. If DNS doesn't work for the server case, you're not connecting at all since you have no target IPs; if you can connect, you have a target hostname to validate against (in the straightforward case of using a hostname instead of an IP address). In the TLS client certificate case, the client can connect but then the TLS server may deny it access for apparently arbitrary reasons.

That your protocol has to specifically decide what verifying TLS client certificates means (and there are multiple possible answers) is, I suspect, one reason that TLS client certificates aren't used more in general Internet protocols. In turn this is a disincentive for servers implementing TLS-based protocols (including SMTP) from telling TLS clients that they can send a TLS client certificate, since it's not clear what you should do with it if one is sent.

Let's Encrypt drops "Client Authentication" from its TLS certificates

By: cks
17 May 2025 at 02:53

The TLS news of the time interval is that Let's Encrypt certificates will no longer be usable to authenticate your client to a TLS server (via a number of people on the Fediverse). This is driven by a change in Chrome's "Root Program", covered in section 3.2, with a further discussion of this in Chrome's charmingly named Moving Forward, Together in the "Understanding dedicated hierarchies" section; apparently only half of the current root Certificate Authorities actually issue TLS server certificates. As far as I know this is not yet a CA/Browser Forum requirement, so this is all driven by Chrome.

In TLS client authentication, a TLS client (the thing connecting to a TLS server) can present its own TLS certificate to the TLS server, just as the TLS server presents its certificate to the client. The server can then authenticate the client certificate however it wants to, although how to do this is not as clear as when you're authenticating a TLS server's certificate. To enable this usage, a TLS certificate and the entire certificate chain must be marked as 'you can use these TLS certificates for client authentication' (and similarly, a TLS certificate that will be used to authenticate a server to clients must be marked as such). That marking is what Let's Encrypt is removing.

This doesn't affect public web PKI, which basically never used conventional CA-issued host and domain TLS certificates as TLS client certificates (websites that used TLS client certificates used other sorts of TLS certificates). It does potentially affect some non-web public TLS, where domain TLS certificates have seen small usage in adding more authentication to SMTP connections between mail systems. I run some spam trap SMTP servers that advertise that sending mail systems can include a TLS client certificate if the sender wants to, and some senders (including GMail and Outlook) do send proper public TLS certificates (and somewhat more SMTP senders include bad TLS certificates). Most mail servers don't, though, and given that one of the best sources of free TLS certificates has just dropped support for this usage, that's unlikely to change. Let's Encrypt's TLS certificates can still be used by your SMTP server for receiving email, but you'll no longer be able to use them for sending it.

On the one hand, I don't think this is going to have material effects on much public Internet traffic and TLS usage. On the other hand, it does cut off some possibilities in non-web public TLS, at least until someone starts up a free, ACME-enabled Certificate Authority that will issue TLS client certificates. And probably some number of mail servers will keep sending their TLS certificates to people as client certificates even though they're no longer valid for that purpose.

PS: If you're building your own system and you want to, there's nothing stopping you from accepting public TLS server certificates from TLS clients (although you'll have to tell your TLS library to validate them as TLS server certificates, not client certificates, since they won't be marked as valid for TLS client usage). Doing the security analysis is up to you but I don't think it's a fatally flawed idea.

Classical "Single user computers" were a flawed or at least limited idea

By: cks
16 May 2025 at 02:33

Every so often people yearn for a lost (1980s or so) era of 'single user computers', whether these are simple personal computers or high end things like Lisp machines and Smalltalk workstations. It's my view that the whole idea of a 1980s style "single user computer" is not what we actually want and has some significant flaws in practice.

The platonic image of a single user computer in this style was one where everything about the computer (or at least its software) was open to your inspection and modification, from the very lowest level of the 'operating system' (which was more of a runtime environment than an OS as such) to the highest things you interacted with (both Lisp machines and Smalltalk environments often touted this as a significant attraction, and it's often repeated in stories about them). In personal computers this was a simple machine that you had full control over from system boot onward.

The problem is that this unitary, open environment is (or was) complex and often lacked resilience. Famously, in the case of early personal computers, you could crash the entire system with programming mistakes, and if there's one thing people do all the time, it's make mistakes. Most personal computers mitigated this by only doing one thing at once, but even then it was unpleasant, and the Amiga would let you blow multiple processes up at once if you could fit them all into RAM. Even on better protected systems, like Lisp and Smalltalk, you still had the complexity and connectedness of a unitary environment.

One of the things that we've learned from computing over the past N decades is that separation, isolation, and abstraction are good ideas. People can only keep track of so many things in their heads at once, and modularity (in the broad sense) is one large way we keep things within that limit (or at least closer to it). Single user computers were quite personal but usually not very modular. There are reasons that people moved to computers with things like memory protection, multiple processes, and various sorts of privilege separation.

(Let us not forget the great power of just having things in separate objects, where you can move around or manipulate or revert just one object instead of 'your entire world'.)

I think that there is a role for computers that are unapologetically designed to be used by only a single person who is in full control of everything and able to change it if they want to. But I don't think any of the classical "single user computer" designs are how we want to realize a modern version of the idea.

(As a practical matter I think that a usable modern computer system has to be beyond the understanding of any single person. There is just too much complexity involved in anything except very restricted computing, even if you start from complete scratch. This implies that an 'understandable' system really needs strong boundaries between its modules so that you can focus on the bits that are of interest to you without having to learn lots of things about the rest of the system or risk changing things you don't intend to.)

Two broad approaches to having Multi-Factor Authentication everywhere

By: cks
15 May 2025 at 03:05

In this modern age, more and more people are facing more and more pressure to have pervasive Multi-Factor Authentication, with every authentication your people perform protected by MFA in some way. I've come to feel that there are two broad approaches to achieving this and one of them is more realistic than the other, although it's also less appealing in some ways and less neat (and arguably less secure).

The 'proper' way to protect everything with MFA is to separately and individually add MFA to everything you have that does authentication. Ideally you will have a central 'single sign on' system, perhaps using OIDC, and certainly your people will want you to have only one form of MFA even if it's not all run through your SSO. What this implies is that you need to add MFA to every service and protocol you have, which ranges from generally easy (websites) through being annoying to people or requiring odd things (SSH) to almost impossible at the moment (IMAP, authenticated SMTP, and POP3). If you opt to set it up with no exemptions for internal access, this approach to MFA insures that absolutely everything is MFA protected without any holes through which an un-MFA'd authentication can be done.

The other way is to create some form of MFA-protected network access (a VPN, a mesh network, a MFA-authenticated SSH jumphost, there are many options) and then restrict all non-MFA access to coming through this MFA-protected network access. For services where it's easy enough, you might support additional MFA authenticated access from outside your special network. For other services where MFA isn't easy or isn't feasible, they're only accessible from the MFA-protected environment and a necessary step for getting access to them is to bring up your MFA-protected connection. This approach to MFA has the obvious problem that if someone gets access to your MFA-protected network, they have non-MFA access to everything else, and the not as obvious problem that attackers might be able to MFA as one person to the network access and then do non-MFA authentication as another person on your systems and services.

The proper way is quite appealing to system administrators. It gives us an array of interesting challenges to solve, neat technology to poke at, and appealingly strong security guarantees. Unfortunately the proper way has two downsides; there's essentially no chance of it covering your IMAP and authenticated SMTP services any time soon (unless you're willing to accept some significant restrictions), and it requires your people to learn and use a bewildering variety of special purpose, one-off interfaces and sometimes software (and when it needs software, there may be restrictions on what platforms the software is readily available on). Although it's less neat and less nominally secure, the practical advantage of the MFA protected network access approach is that it's universal and it's one single thing for people to deal with (and by extension, as long as the network system itself covers all platforms you care about, your services are fully accessible from all platforms).

(In practice the MFA protected network approach will probably be two things for people to deal with, not one, since if you have websites the natural way to protect them is with OIDC (or if you have to, SAML) through your single sign on system. Hopefully your SSO system is also what's being used for the MFA network access, so people only have to sign on to it once a day or whatever.)

Using awk to check your script's configuration file

By: cks
14 May 2025 at 02:39

Suppose, not hypothetically, that you have a shell script with a relatively simple configuration file format that people can still accidentally get wrong. You'd like to check the configuration file for problems before you use it in the rest of your script, for example by using it with 'join' (where things like the wrong number or type of fields will be a problem). Recently on the Fediverse I shared how I was doing this with awk, so here's a slightly more elaborate and filled out version:

errs=$(awk '
         $1 ~ "^#" { next }
         NF != 3 {
            printf " line %d: wrong number of fields\n", NR;
            next }
         [...]
         ' "$cfg_file"
       )

if [ -n "$errs" ]; then
   echo "$prog: Errors found in '$cfg_file'. Stopping." 1>&2
   echo "$errs" 1>&2
   exit 1
fi

(Here I've chosen to have awk's diagnostic messages indented by one space when the script prints them out, hence the space before 'line %d: ...'.)

The advantage of having awk simply print out the errors it detects and letting the script deal with them later is that you don't need to mess around with awk's exit status; your awk program can simply print what it finds and be done. Using awk for the syntax checks is handy because it lets you express a fair amount of logic and checks relatively simply (you can even check for duplicate entries and so on), and it also gives you line numbers for free.

One trick with using awk in this way is to progressively filter things in your checks (by skipping further processing of the current line with 'next'). We start out by skipping all comments, then we report and otherwise skip every line with the wrong number of fields, and then every check after this can assume that at least we have the right number of fields so it can confidently check what should be in each one. If the number of fields in a line is wrong there's no point in complaining about how one of them has the wrong sort of value, and the early check and 'next' to skip the rest of this line's processing is the simple way.

If you're also having awk process the configuration file later you might be tempted to have it check for errors at the same time, in an all-in-one awk program, but my view is that it's simpler to split the error checking from the processing. That way you don't have to worry about stopping the processing if you detect errors or intermingle processing logic with checking logic. You do have to make sure the two versions have the same handling of comments and so on, but in simple configuration file formats this is usually easy.

(Speaking from personal experience, you don't want to use '$1 == "#"' as your comment definition, because then you can't just stick a '#' in front of an existing configuration file line to comment it out. Instead you have to remember to make it '# ', and someday you'll forget.)

PS: If your awk program is big and complex enough, it might make more sense to use a here document to create a shell variable containing it, which will let you avoid certain sorts of annoying quoting problems.

Our need for re-provisioning support in mesh networks (and elsewhere)

By: cks
13 May 2025 at 03:00

In a comment on my entry on how WireGuard mesh networks need a provisioning system, vcarceler pointed me to Innernet (also), an interesting but opinionated provisioning system for WireGuard. However, two bits of it combined made me twitch a bit; Innernet only allows you to provision a given node once, and once a node is assigned an internal IP, that IP is never reused. This lack of support for re-provisioning machines would be a problem for us and we'd likely have to do something about it, one way or another. Nor is this an issue unique to Innernet, as a number of mesh network systems have it.

Our important servers have fixed, durable identities, and in practice these identities are both DNS names and IP addresses (we have some generic machines, but they aren't as important). We also regularly re-provision these servers, which is to say that we reinstall them from scratch, usually on new hardware. In the usual course of events this happens roughly every two years or every four years, depending on whether we're upgrading the machine for every Ubuntu LTS release or every other one. Over time this is a lot of re-provisionings, and we need the re-provisioned servers to keep their 'identity' when this happens.

We especially need to be able to rebuild a dead server as an identical replacement if its hardware completely breaks and eats its system disks. We're already in a crisis, we don't want to have a worse crisis because other things need to be updated because we can't exactly replace the server but instead have to build a new server that fills the same role, or will once DNS is updated, configurations are updated, etc etc.

This is relatively straightforward for regular Linux servers with regular networking; there's the issue of SSH host keys, but there's several solutions. But obviously there's a problem if the server is also a mesh network node and the mesh network system will not let it be re-provisioned under the same name or the same internal IP address. Accepting this limitation would make it difficult to use the mesh network for some things, especially things where we don't want to depend on DNS working (for example, sending system logs via syslog). Working around the limitation requires reverse engineering where the mesh network system stores local state and hopefully being able to save a copy elsewhere and restore it; among other things, this has implications for the mesh network system's security model.

For us, it would be better if mesh networking systems explicitly allowed this re-provisioning. They could make it a non-default setting that took explicit manual action on the part of the network administrator (and possibly required nodes to cooperate and extend more trust than normal to the central provisioning system). Or a system like Innernet could have a separate class of IP addresses, call them 'service addresses', that could be assigned and reassigned to nodes by administrators. A node would always have its unique identity but could also be assigned one or more service addresses.

(Of course our other option is to not use a mesh network system that imposes this restriction, even if it would otherwise make our lives easier. Unless we really need the system for some other reason or its local state management is explicitly documented, this is our more likely choice.)

PS: The other problem with permanently 'consuming' IP addresses as machines are re-provisioned is that you run out of them sooner or later unless you use gigantic network blocks that are many times larger than the number of servers you'll ever have (well, in IPv4, but we're not going to switch to IPv6 just to enable a mesh network provisioning system).

How and why typical (SaaS) pricing is too high for university departments

By: cks
12 May 2025 at 02:48

One thing I've seen repeatedly is that companies that sell SaaS or SaaS like things and offer educational pricing (because they want to sell to universities too) are setting (initial) educational pricing that is in practice much too high. Today I'm going to work through a schematic example to explain what I mean. All of this is based on how it works in Canadian and I believe US universities; other university systems may be somewhat different.

Let's suppose that you're a SaaS vendor and like many vendors, you price your product at $X per person per month; I'll pick $5 (US, because most of the time the prices are in USD). Since you want to sell to universities and other educational institutions and you understand they don't have as much money to spend as regular companies, you offer a generous academic discount; they pay only $3 USD per person per month.

(If these numbers seem low, I'm deliberately stacking the deck in the favour of the SaaS company. Things get worse for your pricing as the numbers go up.)

The research and graduate student side of a large but not enormous university department is considering your software. They have 100 professors 'in' the department, 50 technical and administrative support staff (this is a low ratio), and professors have an average of 10 graduate students, research assistants, postdocs, outside collaborators, undergraduate students helping out with research projects, and so on around them, for a total of 1,000 additional people 'in' the department who will also have to be covered. These 1,150 people will cost the department $3,450 USD a month for your software, a total of $41,400 USD a year, which is a significant saving over what a commercial company would pay for the same number of people.

Unfortunately, unless your software is extremely compelling or absolutely necessary, this cost is likely to be a very tough sell. In many departments, that's enough money to fund (or mostly fund) an additional low-level staff position, and it's certainly enough money to hire more TAs, supplement more graduate student stipends (these are often the same thing, since hiring graduate students as TAs is one of the ways that you support them), or pay for summer projects, all of which are likely to be more useful and meaningful to the department than a year of your service. It's also more than enough money to cause people in the department to ask awkward questions like 'how much technical staff time will it take to put together an inferior but functional enough alternate to this', which may well not be $41,000 worth of time (especially not every year).

(Of course putting together a complete equivalent of your SaaS will cost much more than that, since you have multiple full time programmers working on it and you've invested years in your software at this point. But university departments are already used to not having nice things, and staff time is often considered almost free.)

If you decide to make your pricing nicer by only charging based on the actual number of people who wind up using your stuff, unfortunately you've probably made the situation worse for the university department. One thing that's worse than a large predictable bill is an uncertain but possibly large bill; the department will have to reserve and allocate the money in its budget to cover the full cost, and then figure out what to do with the unused budget at the end of the year (or the end of every month, or whatever). Among other things, this may lead to awkward conversations with higher powers about how the department's initial budget and actual spending don't necessarily match up.

As we can see from the numbers, one big part of the issue is those 1,000 non-professor, non-staff people. These people aren't really "employees" the way they would be in a conventional organization (and mostly don't think of themselves as employees), and the university isn't set up to support their work and spend money on them in the way it is for the people it considers actual employees. The university cares if a staff member or a professor can't get their work done, and having them work faster or better is potentially valuable to the university. This is mostly not true for graduate students and many other additional people around a department (and almost entirely not true if the person is an outside collaborator, an undergraduate doing extra work to prepare for graduate studies elsewhere, and so on).

In practice, most of those 1,000 extra people will and must be supported on a shoestring basis (for everything, not just for your SaaS). The university as a whole and their department in particular will probably only pay a meaningful per-person price for them for things that are either absolutely necessary or extremely compelling. At the same time, often the software that the department is considering is something that those people should be using too, and they may need a substitute if the department can't afford the software for them. And once the department has the substitute, it becomes budgetarily tempting and perhaps politically better if everyone uses the substitute and the department doesn't get your software at all.

(It's probably okay to charge a very low price for such people, as opposed to just throwing them in for free, but it has to be low enough that the department or the university doesn't have to think hard about it. One way to look at it is that regardless of the numbers, the collective group of those extra people is 'less important' to provide services to than the technical staff, the administrative staff, and the professors, and the costs probably should work out accordingly. Certainly the collective group of extra people isn't more important than the other groups, despite having a lot more people in it.)

Incidentally, all of this applies just as much (if not more so) when the 'vendor' is the university's central organizations and they decide to charge (back) people within the university for something on a per-person basis. If this is truly cost recovery and accurately represents the actual costs to provide the service, then it's not going to be something that most graduate students get (unless the university opts to explicitly subsidize it for them).

PS: All of this is much worse if undergraduate students need to be covered too, because there are even more of them. But often the department or the university can get away with not covering them, partly because their interactions with the university are often much narrower than those of graduate students.

Using WireGuard seriously as a mesh network needs a provisioning system

By: cks
11 May 2025 at 02:45

One thing that my recent experience expanding our WireGuard mesh network has driven home to me is how (and why) WireGuard needs a provisioning system, especially if you're using it as a mesh networking system. In fact I think that if you use a mesh WireGuard setup at any real scale, you're going to wind up either adopting or building such a provisioning system.

In a 'VPN' WireGuard setup with a bunch of clients and one or a small number of gateway servers, adding a new client is mostly a matter of generating and giving it some critical information. However, it's possible to more or less automate this and make it relatively easy for people who want to connect to you to do this. You'll still need to update your WireGuard VPN server too, but at least you only have one of them (probably), and it may well be the host where you generate the client configuration and provide it to the client's owner.

The extra problem with adding a new client to a WireGuard mesh network is that there's many more WireGuard nodes that need to be updated (and also the new client needs a lot more information; it needs to know about all of the other nodes it's supposed to talk to). More broadly, every time you change the mesh network configuration, every node needs to update with the new information. If you add a client, remove a client, a client changes its keys for some reason (perhaps it had to be re-provisioned because the hardware died), all of these means nodes need updates (or at least the nodes that talk to the changed node). In the VPN model, only the VPN server node (and the new client) needed updates.

Our little WireGuard mesh is operating at a small scale, so we can afford to do this by hand. As you have more WireGuard nodes and more changes in nodes, you're not going to want to manually update things one by one, any more than you want to do that for other system administration work. Thus, you're going to want some sort of a provisioning system, where at a minimum you can say 'this is a new node' or 'this node has been removed' and all of your WireGuard configurations are regenerated, propagated to WireGuard nodes, trigger WireGuard configuration reloads, and so on. Some amount of this can be relatively generic in your configuration management system, but not all of it.

(Many configuration systems can propagate client-specific files to clients on changes and then trigger client side actions when the files are updated. But you have to build the per-client WireGuard configuration.)

PS: I haven't looked into systems that will do this for you, either as pure WireGuard provisioning systems or as bigger 'mesh networking using WireGuard' software, so I don't have any opinions on how you want to handle this. I don't even know if people have built and published things that are just WireGuard provisioning systems, or if everything out there is a 'mesh networking based on WireGuard' complex system.

Some notes on using 'join' to supplement one file with data from another

By: cks
10 May 2025 at 02:45

Recently I said something vaguely grumpy about the venerable Unix 'join' tool. As the POSIX specification page for join will unhelpfully tell you, join is a 'relational database operator', which means that it implements the rough equivalent of SQL joins. One way to use join is to add additional information for some lines in your input data.

Suppose, not entirely hypothetically, that we have an input file (or data stream) that starts with a login name and contains some additional information, and that for some logins (but not all of them) we have useful additional data about them in another file. Using join, the simple case of this is easy, if the 'master' and 'suppl' files are already sorted:

join -1 1 -2 1 -a 1 master suppl

(I'm sticking to POSIX syntax here. Some versions of join accept '-j 1' as an alternative to '-1 1 -2 1'.)

Our specific options tell join to join each line of 'master' and 'suppl' on the first field in each (the login) and print them, and also print all of the lines from 'master' that didn't have a login in 'suppl' (that's the '-a 1' argument). For lines with matching logins, we get all of the fields from 'master' and then all of the extra fields from 'suppl'; for lines from 'master' that don't match, we just get the fields from 'master'. Generally you'll tell apart which lines got supplemented and which ones didn't by how many fields they have.

If we want something other than all of the fields in the order that they are in the existing data source, in theory we have the '-o <list>' option to tell join what fields from each source to output. However, this option has a little problem, which I will show you by quoting the important bit from the POSIX standard (emphasis mine):

The fields specified by list shall be written for all selected output lines. Fields selected by list that do not appear in the input shall be treated as empty output fields.

What that means is that if we're also printing non-joined lines from our 'master' file, our '-o' still applies and any fields we specified from 'suppl' will be blank and empty (unless you use '-e'). This can be inconvenient if you were re-ordering fields so that, for example, a field from 'suppl' was listed before some fields from 'master'. It also means that you want to use '1.1' to get the login from 'master', which is always going to be there, not '2.1', the login from 'suppl', which is only there some of the time.

(All of this assumes that your supplementary file is listed second and the master file first.)

On the other hand, using '-e' we can simplify life in some situations. Suppose that 'suppl' contains only one additional interesting piece of information, and it has a default value that you'll use if 'suppl' doesn't contain a line for the login. Then if 'master' has three fields and 'suppl' two, we can write:

join -1 1 -2 1 -a 1 -e "$DEFVALUE" -o '1.1,1.2,1.3,2.2' master suppl

Now we don't have to try to tell whether or not a line from 'master' was supplemented by counting how many fields it has; everything has the same number of fields, it's just sometimes the last (supplementary) field is the default value.

(This is harder to apply if you have multiple fields from the 'suppl' file, but possibly you can find a 'there is nothing here' value that works for the rest of your processing.)

In Apache, using OIDC instead of SAML makes for easier testing

By: cks
9 May 2025 at 02:56

In my earlier installment, I wrote about my views on the common Apache modules for SAML and OIDC authentication, where I concluded that OpenIDC was generally easier to use than Mellon (for SAML). Recently I came up with another reason to prefer OIDC, one sufficiently strong enough that we converted one of our remaining Mellon uses over to OIDC. The advantage is that OIDC is easier to test if you're building a new version of your web server under another name.

Suppose that you're (re)building a version of your Apache based web server with authentication on, for example, a new version of Ubuntu, using a test server name. You want to test that everything still works before you deploy it, including your authentication. If you're using Mellon, as far as I can see you have to generate an entirely new SP configuration using your test server's name and then load it into your SAML IdP. You can't use your existing SAML SP configuration from your existing web server, because it specifies the exact URL the SAML IdP needs to use for various parts of the SAML protocol, and of course those URLs point to your production web server under its production name. As far as I know, to get another set of URLs that point to your test server, you need to set up an entirely new SP configuration.

OIDC has an equivalent thing in its redirect URI, but the OIDC redirect URL works somewhat differently. OIDC identity providers typically allow you to list multiple allowed redirect URIs for a given OIDC client, and it's the client that tells the server what redirect URI to use during authentication. So when you need to test your new server build under a different name, you don't need to register a new OIDC client; you can just add some more redirect URIs to your existing production OIDC client registration to allow your new test server to provide its own redirect URI. In the OpenIDC module, this will typically require no Apache configuration changes at all (from the production version), as the module automatically uses the current virtual host as the host for the redirect URI. This makes testing rather easier in practice, and it also generally tests the Apache OIDC configuration you'll use in production, instead of a changed version of it.

(You can put a hostname in the Apache OIDCRedirectURI directive, but it's simpler to not do so. Even if you did use a full URL in this, that's a single change in a text file.)

Chosing between "it works for now" and "it works in the long term"

By: cks
8 May 2025 at 02:50

A comment on my entry about how Netplan can only have WireGuard peers in one file made me realize one of my implicit system administration views (it's the first one by Jon). That is the tradeoff between something that works now and something that not only works now but is likely to keep working in the long term. In system administration this is a tradeoff, not an obvious choice, because what you want is different depending on the circumstances.

Something that works now is, for example, something that works because of how Netplan's code is currently written, where you can hack around an issue by structuring your code, your configuration files, or your system in a particular way. As a system administrator I do a surprisingly large amount of these, for example to fix or work around issues in systemd units that people have written in less than ideal or simply mistaken ways.

Something that's going to keep working in the longer term is doing things 'correctly', which is to say in whatever way that the software wants you to do and supports. Sometimes this means doing things the hard way when the software doesn't actually implement some feature that would make your life better, even if you could work around it with something that works now but isn't necessarily guaranteed to keep working in the future.

When you need something to work and there's no other way to do it, you have to take a solution that (only) works now. Sometimes you take a 'works now' solution even if there's an alternative because you expect your works-now version to be good enough for the lifetime of this system, this OS release, or whatever; you'll revisit things for the next version (at least in theory, workarounds to get things going can last a surprisingly long time if they don't break anything). You can't always insist on a 'works now and in the future' solution.

On the other hand, sometimes you don't want to do a works-now thing even if you could. A works-now thing is in some sense technical debt, with all that that implies, and this particular situation isn't important enough to justify taking on such debt. You may solve the problem properly, or you may decide that the problem isn't big and important enough to solve at all and you'll leave things in their imperfect state. One of the things I think about when making this decision is how annoying it would be and how much would have to change if my works-now solution broke because of some update.

(Another is how ugly the works-now solution is, including how big of a note we're going to want to write for our future selves so we can understand what this peculiar load bearing thing is. The longer the note, the more I generally wind up questioning the decision.)

It can feel bad to not deal with a problem by taking a works-now solution. After all, it works, and otherwise you're stuck with the problem (or with less pleasant solutions). But sometimes it's the right option and the works-now solution is simply 'too clever'.

(I've undoubtedly made this decision many times over my career. But Jon's comment and my reply to it crystalized the distinction between a 'works now' and a 'works for the long term' solution in my mind in a way that I think I can sort of articulate.)

Netplan can only have WireGuard peers in one file

By: cks
7 May 2025 at 02:43

We have started using WireGuard to build a small mesh network so that machines outside of our network can securely get at some services inside it (for example, to send syslog entries to our central syslog server). Since this is all on Ubuntu, we set it up through Netplan, which works but which I said 'has warts' in my first entry about it. Today I discovered another wart due to what I'll call the WireGuard provisioning problem:

Current status: provisioning WireGuard endpoints is exhausting, at least in Ubuntu 22.04 and 24.04 with netplan. So many netplan files to update. I wonder if Netplan will accept files that just define a single peer for a WG network, but I suspect not.

The core WireGuard provisioning problem is that when you add a new WireGuard peer, you have to tell all of the other peers about it (or at least all of the other peers you want to be able to talk to the new peer). When you're using iNetplan, it would be convenient if you could put each peer in a separate file in /etc/netplan; then when you add a new peer, you just propagate the new Netplan file for the peer to everything (and do the special Netplan dance required to update peers).

(Apparently I should now call it 'Canonical Netplan', as that's what its front page calls it. At least that makes it clear exactly who is responsible for Netplan's state and how it's not going to be widely used.)

Unfortunately this doesn't work, and it doesn't work in a dangerous way, which is that Netplan only notices one set of WireGuard peers in one netplan file (at least on servers, using systemd-networkd as the backend). If you put each peer in its own file, only the first peer is picked up. If you define some peers in the file where you define your WireGuard private key, local address, and so on, and some peers in another file, only peers from whichever is first will be used (even if the first file only defines peers, which isn't enough to bring up a WireGuard device by itself). As far as I can see, Netplan doesn't report any errors or warnings to the system logs on boot about this situation; instead, you silently get incomplete WireGuard configurations.

This is visibly and clearly a Netplan issue, because on servers you can inspect the systemd-networkd files written by Netplan (in /run/systemd/network). When I do this, the WireGuard .netdev file has only the peers from one file defined in it (and the .netdev file matches the state of the WireGuard interface). This is especially striking when the netplan file with the private key and listening port (and some peers) is second; since the .netdev file contains the private key and so on, Netplan is clearly merging data from more than one netplan file, not completely ignoring everything except the first one. It's just ignoring any peers encountered after the first set of them.

My overall conclusion is that in Netplan, you need to put all configuration for a given WireGuard interface into a single file, however tempting it might be to try splitting it up (for example, to put core WireGuard configuration stuff in one file and then list all peers in another one).

I don't know if this is an already filed Netplan bug and I don't plan on bothering to file one for it, partly because I don't expect Canonical to fix Netplan issues any more than I expect them to fix anything else and partly for other reasons.

PS: I'm aware that we could build a system to generate the Netplan WireGuard file, or maybe find a YAML manipulating program that could insert and delete blocks that matched some criteria. I'm not interested in building yet another bespoke custom system to deal with what is (for us) a minor problem, since we don't expect to be constantly deploying or removing WireGuard peers.

I moved my local Firefox changes between Git trees the easy way

By: cks
6 May 2025 at 03:20

Firefox recently officially switched to Git, in a completely different Git tree than their old mirror. This presented me a little bit of a problem because I have a collection of local changes I make to my own Firefox builds, which I carry as constantly-rebased commits on top of the upstream Firefox tree. The change in upstream trees meant that I was going to have to move my commits to the new tree. When I wrote my first entry I thought I might try to do this in some clever way similar to rebasing my own changes on top of something that was rebased, but in the end I decided to do it the simple and brute force way that I was confident would either work or would leave me in a situation I could back out from easily.

This simple and brute force way was to get both my old tree and my new 'firefox' tree up to date, then export my changes with 'git format-patch' from the old tree and import them into the new tree with 'git am'. There were a few irritations along the way, of course. First I (re)discovered that 'git am' can't directly consume the directory of patches you create with 'git format-patch'. Git-am will consume a Maildir of patches, but git-format-patch will only give you a directory full of files with names like '00NN-<author>-<title>.patch', which is not a proper Maildir. The solution is to cat all of the .patch files together in order to some other file, which is now a mailbox that git-am will handle. The other minor thing is that git-am unsurprisingly has no 'dry-run' option (which would probably be hard to implement). Of course in my situation, I can always reset 'main' back to 'origin/main', which was one reason I was willing to try this.

(Looking at the 'git format-patch' manual page suggests that what I might have wanted was the '--stdout' option, which would have automatically created the mbox format version for me. On the other hand it was sort of nice to be able to look at the list of patches and see that they were exactly what I expected.)

On the one hand, moving my changes in this brute force way (and to a completely separate new tree) feels like giving in to my unfamiliarity with git. There are probably clever git ways to do this move in a single tree without having to turn everything into patches and then apply them (even if most of that is automated). On the other hand, this got the job done with minimal hassles and time consumed, and sometimes I need to put a stop to my programmer's urge to be clever.

LLMs ('AI') are coming for our jobs whether or not they work

By: cks
5 May 2025 at 02:58

Over on the Fediverse, I said something about this:

Hot take: I don't really know what vibe coding is but I can confidently predict that it's 'coming for', if not your job, then definitely the jobs of the people who work in internal development at medium to large non-tech companies. I can predict this because management at such companies has *always* wanted to get rid of programmers, and has consistently seized on every excuse presented by the industry to do so. COBOL, report generators, rule based systems, etc etc etc at length.

(The story I heard is that at one point COBOL's English language basis was at least said to enable non-programmers to understand COBOL programs and maybe even write them, and this was seen as a feature by organizations adopting it.)

The current LLM craze is also coming for the jobs of system administrators for the same reason; we're overhead, just like internal development at (most) non-tech companies. In most non-tech organizations, both internal development and system administration is something similar to janitorial services; you have to have it because otherwise your organization falls over, but you don't like it and you're happy to spend as little on it as possible. And, unfortunately, we have a long history in technology that shows the long term results don't matter for the people making short term decisions about how many people to hire and who.

(Are they eating their seed corn? Well, they probably don't think it matters to them, and anyway that's a collective problem, which 'the market' is generally bad at solving.)

As I sort of suggested by using 'excuse' in my Fediverse post, it doesn't really matter if LLMs truly work, especially if they work over the long run. All they need to do in order to get senior management enthused about 'cutting costs' is appear to work well enough over the short term, and appearing to work is not necessarily a matter of substance. In sort of a flipside of how part of computer security is convincing people, sometimes it's enough to simply convince (senior) people and not have obvious failures.

(I have other thoughts about the LLM craze and 'vibe coding', as I understand it, but they don't fit within the margins of this entry.)

PS: I know it's picky of me to call this an 'LLM craze' instead of an 'AI craze', but I feel I have to both as someone who works in a computer science department that does all sorts of AI research beyond LLMs and as someone who was around for a much, much earlier 'AI' craze (that wasn't all of AI either, cf).

These days, Linux audio seems to just work (at least for me)

By: cks
4 May 2025 at 02:29

For a long time, the common perception was that 'Linux audio' was the punchline for a not particularly funny joke. I sort of shared that belief; although audio had basically worked for me for a long time, I had a simple configuration and dreaded having to make more complex audio work in my unusual desktop environment. But these days, audio seems to just work for me, even in systems that have somewhat complex audio options.

On my office desktop, I've wound up with three potential audio outputs and two audio inputs: the motherboard's standard sound system, a USB headset with a microphone that I use for online meetings, the microphone on my USB webcam, and (to my surprise) a HDMI audio output because my LCD displays do in fact have tiny little speakers built in. In PulseAudio (or whatever is emulating it today), I have the program I use for online meetings set to use the USB headset and everything else plays sound through the motherboard's sound system (which I have basic desktop speakers plugged into). All of this works sufficiently seamlessly that I don't think about it, although I do keep a script around to reset the default audio destination.

On my home desktop, for a long time I had a simple single-output audio system that played through the motherboard's sound system (plus a microphone on a USB webcam that was mostly not connected). Recently I got an outboard USB DAC and, contrary to my fears, it basically plugged in and just worked. It was easy to set the USB DAC as the default output in pavucontrol and all of the settings related to it stick around even when I put it to sleep overnight and it drops off the USB bus. I was quite pleased by how painless the USB DAC was to get working, since I'd been expecting much more hassles.

(Normally I wouldn't bother meticulously switching the USB DAC to standby mode when I'm not using it for an extended time, but I noticed that the case is clearly cooler when it rests in standby mode.)

This is still a relatively simple audio configuration because it's basically static. I can imagine more complex ones, where you have audio outputs that aren't always present and that you want some programs (or more generally audio sources) to use when they are present, perhaps even with priorities. I don't know if the Linux audio systems that Linux distributions are using these days could cope with that, or if they did would give you any easy way to configure it.

(I'm aware that PulseAudio and so on can be fearsomely complex under the hood. As far as the current actual audio system goes, I believe that what my Fedora 41 machines are using for audio is PipeWire (also) with WirePlumber, based on what processes seem to be running. I think this is the current Fedora 41 audio configuration in general, but I'm not sure.)

The HTTP status codes of responses from about 22 hours of traffic to here (part 2)

By: cks
3 May 2025 at 03:09

A few months ago, I wrote an entry about this topic, because I'd started putting in some blocks against crawlers, including things that claimed to be old versions of browsers, and I'd also started rate-limiting syndication feed fetching. Unfortunately, my rules at the time were flawed, rejecting a lot of people that I actually wanted to accept. So here are some revised numbers from today, a day when my logs suggest that I've seen what I'd call broadly typical traffic and traffic levels.

I'll start with the overall numbers (for HTTP status codes) for all requests:

  10592 403		[26.6%]
   9872 304		[24.8%]
   9388 429		[23.6%]
   8037 200		[20.2%]
   1629 302		[ 4.1%]
    114 301
     47 404
      2 400
      2 206

This is a much more balanced picture of activity than the last time around, with a lot less of the overall traffic being HTTP 403s. The HTTP 403s are from aggressive blocks, the HTTP 304s and HTTP 429s are mostly from syndication feed fetchers, and the HTTP 302s are mostly from things with various flaws that I redirect to informative static pages instead of giving HTTP 403s. The two HTTP 206s were from Facebook's 'externalhit' agent on a recent entry. A disturbing amount of the HTTP 403s were from Bing's crawler and almost 500 of them were from something claiming to be an Akkoma Fediverse server. 8.5% of the HTTP 403s were from something using Go's default User-Agent string.

The most popular User-Agent strings today for successful requests (of anything) were for versions of NetNewsWire, FreshRSS, and Miniflux, then Googlebot and Applebot, and then Chrome 130 on 'Windows NT 10'. Although I haven't checked, I assume that all of the first three were for syndication feeds specifically, with few or no fetches of other things. Meanwhile, Googlebot and Applebot can only fetch regular pages; they're blocked from syndication feeds.

The picture for syndication feeds looks like this:

   9923 304		[42%]
   9535 429		[40%]
   1984 403		[ 8.5%]
   1600 200		[ 6.8%]
    301 302
     34 301
      1 404

On the one hand it's nice that 42% of syndication feed fetches successfully did a conditional GET. On the other hand, it's not nice that 40% of them got rate-limited, or that there were clearly more explicitly blocked requests that there were HTTP 200 responses. On the sort of good side, 37% of the blocked feed fetches were from one IP that's using "Go-http-client/1.1" as its User-Agent (and which accounts for 80% of the blocks of that). This time around, about 58% of the requests were for my syndication feed, which is better than it was before but still not great.

These days, if certain problems are detected in a request I redirect the request to a static page about the problem. This gives me some indication of how often these issues are detected, although crawlers may be re-visiting the pages on their own (I can't tell). Today's breakdown of this is roughly:

   78%  too-old browser
   13%  too generic a User-Agent
    9%  unexpectedly using HTTP/1.0

There were slightly more HTTP 302 responses from requests to here than there were requests for these static pages, so I suspect that not everything that gets these redirects follows them (or at least doesn't bother re-fetching the static page).

I hope that the better balance in HTTP status codes here is a sign that I have my blocks in a better state than I did a couple of months ago. It would be even better if the bad crawlers would go away, but there's little sign of that happening any time soon.

The complexity of mixing mesh networking and routes to subnets

By: cks
2 May 2025 at 02:51

One of the in things these days is encrypted (overlay) mesh networks, where you have a bunch of nodes and the nodes have encrypted connections to each other that they use for (at least) internal IP traffic. WireGuard is one of the things that can be used for this. A popular thing to add to such mesh network solutions is 'subnet routes', where nodes will act as gateways to specific subnets, not just endpoints in themselves. This way, if you have an internal network of servers at your cloud provider, you can establish a single node on your mesh network and route to the internal network through that node, rather than having to enroll every machine in the internal network.

(There are various reasons not to enroll every machine, including that on some of them it would be a security or stability risk.)

In simple configurations this is easy to reason about and easy to set up through the tools that these systems tend to give you. Unfortunately, our network configuration isn't simple. We have an environment with multiple internal networks, some of which are partially firewalled off from each other, and where people would want to enroll various internal machines in any mesh networking setup (partly so they can be reached directly). This creates problems for a simple 'every node can advertise some routes and you accept the whole bundle' model.

The first problem is what I'll call the direct subnet problem. Suppose that you have a subnet with a bunch of machines on it and two of them are nodes (call them A and B), with one of them (call it A) advertising a route to the subnet so that other machines in the mesh can reach it. The direct subnet problem is that you don't want B to ever send its traffic for the subnet to A; since it's directly connected to the subnet, it should send the traffic directly. Whether or not this happens automatically depends on various implementation choices the setup makes.

The second problem is the indirect subnet problem. Suppose that you have a collection of internal networks that can all talk to each other (perhaps through firewalls and somewhat selectively). Not all of the machines on all of the internal networks are part of the mesh, and you want people who are outside of your networks to be able to reach all of the internal machines, so you have a mesh node that advertises routes to all of your internal networks. However, if a mesh node is already inside your perimeter and can reach your internal networks, you don't want it to go through your mesh gateway; you want it to send its traffic directly.

(You especially want this if mesh nodes have different mesh IPs from their normal IPs, because you probably want the traffic to come from the normal IP, not the mesh IP.)

You can handle the direct subnet case with a general rule like 'if you're directly attached to this network, ignore a mesh subnet route to it', or by some automatic system like route priorities. The indirect subnet case can't be handled automatically because it requires knowledge about your specific network configuration and what can reach what without the mesh (and what you want to reach what without the mesh, since some traffic you want to go over the mesh even if there's a non-mesh route between the two nodes). As far as I can see, to deal with this you need the ability to selectively configure or accept (subnet) routes on a mesh node by mesh node basis.

(In a simple topology you can get away with accepting or not accepting all subnet routes, but in a more complex one you can't. You might have two separate locations, each with their own set of internal subnets. Mesh nodes in each location want the other location's subnet routes, but not their own location's subnet routes.)

❌
❌