Picking uncontested private IP subnets with usage data

If the device you are reading this on has an IPv4 address, it is very likely not a publicly routeable one. This is because the wide scale dep

Mumble: A referral was returned from the server.

You may get A referral was returned from the server. when launching Digitally signed applications like Mumble on windows.

This is normally caused by the Digital Signature expiring. You can check

Lessons from APL, a “lost” language

A few weeks back the forum that I go on often held a obfuscation challenge, and people joked around about submitting a entry that was coded in APL.

APL (A Programming Language) is a programming language that w

NPM build error

NPM has a bunch of useful stuff on it, however you could in life while using NPM get this:

stack Error: "pre" versions of node cannot be installed, use the --node dir flag instead

This error basically says “Give me the node

Dealing with MySQL resource shortage

You may in life while working with mysql get the following errors:

ERROR 1016 (HY000) at line 1: Can't open file: './blah/table.frm' (errno: 24)

or

``` SQL Error (23): Out of resources when o

Super Easy Twitter Bots

I often get really quite mad ideas on writing twitter bots, But I often get pretty bored of doing all of the boiler plate that is required when wanting to achieve these things.

The typical process to making a twitter bot

Running around HTTP firewalls

When I was in 6th form and I wanted to get SSH access to my own systems I had quite a few issues doing so, Since port 22 was blocked and just about every other port was, For the first year that was fine however since

I guess I’ll cover the context first so that we can move on to the good stuff.

Man, everything is going terribly. It’s hard to overstate how bad things are for America right now. We’re just doing the thing: we’ve elected fascists and they’re funding an unaccountable military that will do their bidding. They’re doing the worst possible economic policy and attacking elected officials. The New York Times is actively fighting against New York’s Democratic candidate for mayor by publishing weak hit pieces day after day, and doing their weak-ass quasi-endorsement of Cuomo even after they publicly stated they wouldn’t endorse in local campaigns. Shame on them. As mtsw says, they like Trump and want him to rule like a king. You probably know all this, and every day you’re feeling the fascism. If you aren’t, how? Or, alternatively, what’s wrong with you?

Now on to the entertaining content (staring into the void, copy & pasting links from my Instapaper ‘Liked’ stories)

Listening

The big new listen this month was S. G. Goodman, who I learned from Hearing Things:

In very broad strokes she’s next to Waxahatchee in my MP3 library in Swinsian: country-inflected folk rock. It’s a solid album with some hits. And I usually don’t notice this, but she has really really nailed her visual style and merch. A+.

Watching

I’ve been watching Murderbot, which is very watchable, and extremely brief: each episode is about 17 minutes of actual video. So far I’m happy that things happen in it, but it isn’t very funny or meaningful. Watching it is kind of like nicotine gum, a way to take a small dose of television and wean yourself off of it.

I’m a sucker for ‘how it works’ videos and this one is an all-timer. It’s at the perfect level of detail and fascinating end-to-end. I just love these physical effects, the creativity involved in the shots, the commitment to the craft by actors, designers, even a guy making remote-controlled robots. Watch it.

Other than that, I watched Walk Hard this month and it was so incredibly dumb and I loved every moment.

Reading

Finally got some momentum with books! Glass Century and Things Become Other Things are both hard to describe but great in their own ways. And similar also in that their authors have been on my radar for a long time: I regularly read Ross Barkan’s blog for its local political coverage, and admire Craig Mod’s photos and online writing.

Chill out for a moment listening to the cool guy play the nice piano before I rant about the cops

But if history is any indication, any attempts to limit or even question the infinite expansion of the NYPD’s size, budget, power, and impunity will turn police against a candidate.

Matthew Guariglia’s Zohran Mamdani Better Get Ready To Fight The NYPD was a riveting read. The more I know about the NYPD and its history, the more I am convinced that it is a government all to itself, unaccountable and malevolent. I recommend reading about the Patrolmen’s Benevolent Association Riot as a starting point. Then read The City’s coverage of the NYPD. Every new piece of information paints a worse picture.

Why, exactly, are we meant to show these people respect? Because they run a company that provides a continually-disintegrating service? Because that service has such a powerful monopoly that it’s difficult to leave it if you’re interacting with other people or businesses?

Frankly Ed Zitron’s writing can be too much for me, but Make Fun of Them has a great central dynamic: that major tech companies are simultaneously posing as champions of the next generation of ‘AI’ technology while failing to maintain the basic software that is their main actual product.

And so it was that a lifetime of doing, gradually, then suddenly gave way to a remainder of lacking—lacking the tools that had enabled her longer decades of productivity than most. My grandmother was a testament to the term “workhorse.” I often joke that they simply don’t make people like they used to.

I really like reading netigen, a semi-anonymous blog.

There are seconds here and there when I feel a deep gratitude about getting older. They’re really random; I’ve sometimes held dishes while cleaning them, thinking “I might hate doing this right now, but I’ll miss it once I’m no longer on this earth. I’ll miss everything, no matter how mundane.”.

And also ava’s blog. There are more like this that I read - blogs that are nostalgic, personal, affecting. I can’t write like that, because this blog has my real all over it, and from day one I’ve drawn some boundaries around what I write about: no personal stories, no real-life drama, no “journaling” on this website. I keep a paper journal for that stuff. But I appreciate people who are able to write honestly about life, death, family, love, and all of it.

It’s not lost on me that filling time with an endless card game is a way to escape the cold clutch of grief. There are no counters, no tricky cards you can pull out of your hand. When that bastard attacks, it sails right through all defenses.

Similarly, from Dan Sinker, who I admire, and whose writing I admire.

New York

Knowing New York will take a lifetime, but I’m trying.

I went to The New York Earth Room, which is a second-floor apartment filled with dirt. You can’t walk on the dirt. There’s someone who works there, nodding at the visitors, probably answering questions about the dirt. The same gallery network, Dia, houses The Broken Kilometer, a kilometer of brass rods arranged on the floor of another space. Dia is its own story, starting with an oil fortune and carrying on to the present day leader converting to Sufism and founding a Sufi lodge downtown. I haven’t been to the lodge yet. The MELA foundation is the obvious next place to go.

epanet-js is a new web application that combines modern web maps with the industry-standard EPANET hydraulic simulation algorithm. It’s for people planning and updating water utility systems: connecting pipes and pressures and figuring out what will happen. It’s a problem area that I’m totally fascinated by and know very little about. It’s made by the folks from Iterating - Luke Butler and Sam Payá, who are experts in the field.

If you’ve been following along with my blog and projects, you might notice something familiar about this screenshot of epanet-js:

Yep! A lifetime ago, I built a company and product called Placemark, which was a tool for creating and editing map data. When the business part didn’t work out, I published a free-to-use version of Placemark and made the code open source. I chose a very permissive license for the code: the MIT license. I wanted to let anything happen, including - especially - people creating paid products with the code.

It’s hard to explain to people outside of the world of open source that folks building a business with the help of my code is the dream.

Partly it’s that ‘being helpful’ is one of the goals in life that never gets old. The other part is that most software becomes obsolete and abandoned quickly. Creating software with any legacy is rare and something to celebrate.

Plus, Placemark was a general purpose tool in search of a successful niche: I never figured out what that niche was. I had a lot of learning to do about product-market-fit. But hydraulic simulation is a real market, and these folks are real experts.

There are other markets like this, I’m sure, and using Placemark code as a base could be a good way to jumpstart those companies - it’s not perfect, as I’ve documented extensively, but it could be good as a first step.

And the folks at Iterating have even contributed changes upstream to the open source Placemark codebase, but that isn’t required by the MIT license at all: that’s just because they’re cool folks.

They’ve also open sourced the core epanet-js library, and released the web application under the Functional Source License, which makes new code contributions open source after two years.

And epanet-js is a tool that you can run in a browser - full simulations with a WASM-based engine. It’s competing with expensive old-school software that costs $16,000 a year, runs exclusively on Windows, is priced by “pipes”, and uses the same engine, EPANET. This is so much better in comparison. A radical improvement.

Congrats to Luke and Sam for creating epanet-js. It’s super cool. If you speak hydraulic simulation or know someone who does, try it out!

A little late on this one, but I got around to it!

Reading

I got stuck on two books: books that I want to enjoy but can’t get any momentum on. So my reading “stats” are suffering and this is a light year for books so far. But I switched gears to read Glass Century by Ross Barkan, whose newsletter is my primary news source for local political intrigue. It’s great so far.

As the culture of the Who Cares Era grinds towards the lowest common denominator, support those that are making real things. Listen to something with your full attention. Watch something with your phone in the other room. Read an actual paper magazine or a book.

Dan Sinker’s Who Cares Era is great writing and a message that I believe in in my bones. Caring a lot kind of comes naturally to me, and nihilism of all kinds my natural enemy.

Speaking of which: if you’re in New York, there’s an election coming up. Please care about it, vote, and don’t rank Cuomo. It really matters.

…I quoted Hannah Arendt’s observation that the Germans who refused to participate in Nazi atrocities were those who possessed “the disposition to live together explicitly with oneself, to have intercourse with oneself, that is, to be engaged in that silent dialogue between me and myself which, since Socrates and Plato, we usually call thinking.” To engage in that “silent dialogue,” you need to be capable of thinking alone — or not precisely alone, but in solitude. Because as Arendt notes elsewhere, in The Human Condition: “To be in solitude means to be with one’s self, and thinking, therefore though it may be the most solitary of all activities, it is never altogether without a partner and without company.”

From you’ll never think alone by Ned Resnikoff. Not to brag, but even as I’ve become a more social person, my comfort with solitude has stayed the same. When I rode from Brooklyn to Brewster, I did so alone, with no music, no podcasts, few breaks. It felt good: I’ve done the same before, and I like it, the silence is comfortable. Thinking in solitude is the default state. But I can see how it isn’t for everyone, how consulting a chatbot for personal questions is where people end up. It’s just sad, though.

I think there’s a role for AI in generating marketing assets. In Buzz, you can generate images and you can write text. What I have not seen yet is a world where the models can generate content that a brand team would be really proud of. Maybe that’s coming, but it seems further away than you might expect.

I was kind of impressed by this interview with Dylan Field, Figma co-founder, on their new AI-flavored tools. For the CEO role, there’s only one accepted stance toward AI and it is wild-eyed enthusiasm. Even hedging a little bit, like he does here, is unusual and good.

Listening

What a banner month for albums!

The biggest hit was Slow Mass’s On Foot. It’s such a solid album. Almost every track has 5 stars in my Swinsian library.

I saw Nils Frahm perform at Kings Theatre and instantly became a fan. His albums are so handcrafted and lovely.

And then it dropped that Forth Wanderers are returning with another album. I started to listen to them right around the time that they took a very long break in 2018, and about seven years later they’re back. A lot like Slow Mass, they’re just a great combination of instrument and voice, and they really capture a mood.

Finally, I really like this Julian Cubillos track, which I found via Hearing Things.

Technology Optimism Hour: Fediverse edition

Last week I got the idea to write an update of my Technology Optimism Hour from 2022. What’s the cool new stuff in the technology industry that’s sprouted up since then? And then I hit some writers block, or rather I couldn’t think of much.

So I asked around on Mastodon and got some interesting answers! Refer over there for some good ideas from the folks on the ‘fediverse.’ A lot of the technologies listed are technically pre-2022, but are hitting their stride now. A few of the answers:

• 3D printing
• Contactless payments
• CSS improvements
• jujutsu
• uv
• DuckDB
• Apple Silicon and other ARM chips

Good answers! There’s a bit of overlap with my last writing and a lot of these are pre-2022, but the path of technology is complicated and it’s very true that something like 3D printing has had a lot of eras and every new step toward mainstreaming is exciting.

I’m probably not buying a 3D printer though, for reasons of Brooklyn apartment sizes, so the main thing on this list that I need to dive into is jj. A lot of smart people love it and I trust that it’s great.

Elsewhere

I wrote quite a lot of micro posts this month, on chatbots, New York elections, Blog micro-optimization, LeaderKey, expertise, and a ThinkUp clone that I built on Val Town.

Also I wrote a little AMA app on Val Town too, so you can ask something, don’t be weird. I do have a slight addendum to my answer to one thing to change the world. I think it’s Direct File. Direct File - the free tax-filing service that has been a longtime dream of good politicians and enemy of horrifying monopolies and bad politicians: that’s it. The slow, hard work of building government services that work. Not carelessly thrown-together code that doesn’t work and only aims to achieve a vacuous libertarian budget-cutting ideology. I don’t believe in “one weird trick” anymore, if I ever did. I want the expansive vision of city owned grocery stores and functioning government services.

Journalism

This month one of my new music ideas came from Hearing Things, an independent music publication that I recently subscribed to. It’s worker owned and consistently high-quality. What else is like this?

In New York, you have to subscribe to Hell Gate and donate to The City, both of which are doing awesome work. The aforementioned Political Currents by Ross Barkan is an invaluable source for understanding the finer points of city politics, like where the Working Families Party fits alongside New York’s Democrats.

404 media does amazing coverage on technology-adjacent topics: it’s great to listen to their podcast and hear folks with more than enough knowledge to get the technical specifics right, as well as really high standards for validating information.

Colossal is great independent art writing. My friend David writes an extremely good newsletter about the alcohol industry as well as a local outlet for Richmond, VA.

Right now, Shubhanshu Shukla is on his way back to Earth from the International Space Station. Am I proud he’s been the first Indian up there? I don’t know. It’s not clear.

The whole thing seemed to be stage-managed. Shukla didn’t say anything surprising, nothing that popped. In fact he said exactly what we expected him to say. Nothing more, nothing less.

Fuck controversy. It’s possible to be interesting in new ways all the time without edging into the objectionable. It’s not hard to beat predictability — but there it was for two weeks straight. I wonder if Shukla was fed all his lines. It could’ve been a monumental thing but it feels… droll.

“India’s short on cash.” “India’s short on skills.” “India’s short on liberties.” We’ve heard these refrains as we’ve covered science and space journalism. But it’s been clear for some time now that “India’s short on cash” is a myth.

We’ve written and spoken over and over that Gaganyaan needs better accountability and more proactive communication from ISRO’s Human Space Flight Centre. But it’s also true that it needs even more money than the Rs 20,000 crore it’s already been allocated.

One thing I’ve learnt about the Narendra Modi government is that if it puts its mind to it, if it believes it can extract political mileage from a particular commitment, it will find a way to go all in. So when it doesn’t, the fact that it doesn’t sticks out. It’s a signal that The Thing isn’t a priority.

Looking at the Indian space programme through the same lens can be revealing. Shukla’s whole trip and back was carefully choreographed. There’s been no sense of adventure. Grit is nowhere to be seen.

But between Prime Minister Modi announcing his name in the list of four astronaut-candidates for Gaganyaan’s first crewed flight (currently set for 2027) and today, I know marginally more about Shukla, much less about the other three, and nothing really personal to boot. Just banal stuff.

This isn’t some military campaign we’re talking about, is it? Just checking.

Chethan Kumar at ToI and Jatan Mehta have done everyone a favour: one by reporting extensively on Shukla’s and ISRO’s activities and the other by collecting even the most deeply buried scraps of information from across the internet in one place. The point, however, is that it shouldn’t have come to this. Their work is laborious, made possible by the fact that it’s by far their primary responsibility.

It needed to be much easier than this to find out more about India’s first homegrown astronauts. ISRO itself has been mum, so much so that every new ISRO story is turning out to be an investigative story. The details of Shukla’s exploits needed to be interesting, too. The haven’t been.

So now, Shukla’s returning from the International Space Station. It’s really not clear what one’s expected to be excited about…

Featured image credit: Ray Hennessy/Unsplash.

I hadn’t checked my notifications on X.com in a while. When I did yesterday, I found Pradx had tagged me in a blog post called “a challenge of blog questions” in March. The point is to answer a short list of questions about my blogging history, then tag other bloggers to carry the enterprise forward. With thanks to Pradx, here goes.

Why did you start blogging in the first place?

I started blogging for two reasons in 2008. I started writing itself when I realised it helps me clarify my thoughts, then I started publishing my writing on the web so I could share those thoughts with my friends in different parts of the world. My blog soon gave me a kind of third space on the internet, a separate world I could escape to as I laboured through four years of engineering school, which I didn’t like at the time.

What platform are you using to manage your blog and why did you choose it? Have you blogged on other platforms before?

I’ve blogged on Xanga, Blogspot, Typed, Movable Type, various static site generators, Svbtle, Geocities, Grav, October, Mataroa, Ghost, and WordPress. And I’ve always found myself returning to WordPress, which — despite its flaws — allows me to have just the kind of blog I’d like to in terms of look, feel, spirit, and community. The last two are particularly important. Ghost comes a close second to WordPress but it’s too magaziney. The options to host Ghost are also (relatively) more expensive.

Earlier this year, Matt Mullenweg of Automattic tested my support for WordPress.com with his words and actions vis-à-vis his vendetta against WP Engine but the sentiments and conversations in the wider WordPress community encouraged me to keep going.

How do you write your posts? For example, in a local editing tool, or in a panel/dashboard that’s part of your blog?

I used to love WordPress’s Calypso interface and its WYSIWYG editor both on desktop and mobile and used to use that to compose posts. But then WordPress ‘upgraded’ to the blocks-based Gutenberg interface, which made composing a jerky, clunky, glitchy process. At that point I tried a combination of different local editors, including Visual Studio Code, iA Writer, and Obsidian.md. Each editor provided an idiosyncratic environment: e.g. VS Code seemed like a good environment in which to compose technical posts, Obsidian (with its dark UI) for angry/moody ones, and iA writer for opinionated ones with long sentences and complex thoughts.

Then about three years ago I discovered MarsEdit and have been using it for all kinds of posts since. I particularly appreciate its old-school-like interface, that it’s built to work with WordPress, and the fact that it maintains an offline archive of all the posts on the blog.

When do you feel most inspired to write?

I’ve answered this question before in conversations with friends and every time my answer has prompted them to wonder if I’m lying or mocking them.

When I feel most inspired to write is not in my control. I’ve been writing for so long that it’s become a part of the way I think. If I have a thought and I’m not able to articulate it clearly in writing, it’s a sign for me that the thought is still inchoate. In this paradigm, whenever I have a fully formed thought that I think could help someone else think about or through something, I enter a half-trance-like state, where my entire brain is seized of the need to write and I’m only conscious enough to open MarsEdit and start typing.

In these circumstance my ability to multi-task even minor activities, like typing with one hand while sipping from a mug of tea in the other, vanishes.

Do you publish immediately after writing, or do you let it simmer a bit as a draft?

That depends on what I’m writing about. When I draft posts in the ‘Op-eds’ or ‘Science’ categories, I’m usually more clear-headed and confident about my post’s contents, and publish as soon as the post is ready. For ‘Analysis’ and ‘Scicomm’ posts, however, I distract myself for about 30 minutes after finishing a draft and read it again to make sure there aren’t any holes in my arguments.

I also have a few friends who peer-review my posts if I’m not sure I’ve articulated myself well or if I’m not able to think through the soundness of my own arguments by myself (usually because I suspect there’s something I don’t know). Four of the most frequent reviewers are Thomas Manuel, Srividya Tadepalli, Mahima Jain, and Chitralekha Manohar.

In all these cases, however, I do read the post a couple times more after it’s finished to fix grammar and clumsy sentence constructions.

What’s your favorite post on your blog?

No such thing.

Any future plans for your blog? Maybe a redesign, a move to another platform, or adding a new feature?

I’m not keen on major redesigns. There are too many WordPress themes available off the shelf and for free these days. I change my blog’s theme depending on my mood. I don’t think it makes a difference to whether or how people read my posts. I think those that have been reading will continue to read. The text is paramount.

I don’t see myself moving to another platform either. If anything, I might move from WordPress.com to a self-hosted setup in future but it’s not something I’m thinking of right now.

I am currently in the process of removing duplicated posts in the archives — at last count I spotted about 20. Many posts are also missing images I’d added at the time of publishing, mostly because they were associated with a domain that I no longer use. I need to fix that.

A few years ago I lost around 120 posts after someone managed to hack my account when the blog was hosted with a provider of cPanel hosting services. I maintain a long-term backup of all my posts on a Backblaze dump. I’m still in the process of identifying which posts I lost and retrieving them from the archive.

So yeah, focusing on this clean-up right now.

Who’s next?

This is embarrassing: I only know a few other bloggers. I stopped keeping track after many bloggers I’d been following in the early years just stopped at some point. Right now, of those blogs I still follow, Jatan and Pradx have already been nominated for this ‘challenge’. So let me nominate Suvrat Kher and Dhiya Gerber next, both of whom I think will have interesting answers.

Featured image credit: Chris Briggs/Unsplash.

From ’Sharks hunt via Lévy flights’, Physics World, June 11, 2010:

They were menacing enough before, but how would you feel if you knew sharks were employing advanced mathematical concepts in their hunt for the kill? Well, this is the case, according to new research, which has tracked the movement of these marine predators along with a number of other species as they foraged for prey in the Pacific and Atlantic oceans. The results showed that these animals hunt for food by alternating between Brownian motion and Lévy flights, depending on the scarcity of prey.

Animals don’t use advanced mathematical concepts. This statement encompasses many humans as well because it’s not a statement about intelligence but one about language and reality. You see a shark foraging in a particular pattern. You invent a language to efficiently describe such patterns. And in that language your name for the shark’s pattern is a Lévy flight. This doesn’t mean the shark is using a Lévy flight. The shark is simply doing what makes sense to it, but which we — in our own description of the world — call a Lévy flight.

The Lévy flight isn’t an advanced concept either. It’s a subset of a broader concept called the random walk. Say you’re on a square grid, like a chessboard. You’re standing on one square. You can move only one step at a time. You roll a four-sided die. Depending on the side it lands on, you step one square forwards, backwards, to the right or to the left. The path you trace over time is called a random walk because its shape is determined by the die roll, which is random.

There are different kinds of walks depending on the rule that determines the choice of your next step. A Lévy flight is a random walk that varies both the direction of the next step and the length of the step. In the random walk on the chessboard, you took steps of fixed lengths: to the adjacent squares. In a Lévy flight, the direction of the next step is random and the length is picked at random from a Lévy distribution. This is what the distribution looks like:

Notice how a small part of each curve (for different values of c in the distribution’s function) has high values and the majority has smaller values. When you pick your step length at random from, say, the red curve, you have higher odds of of picking a smaller step length than a longer one. This means in a Lévy flight, most of the step lengths will be short but a small number of steps will be long. Thus the ‘flight’ looks like this:

Sharks and many other animals have been known to follow a Lévy flight when foraging. To quote from an older post:

Research has shown that the foraging path of animals looking for food that is scarce can be modelled as a Lévy flight: the large steps correspond to the long distances towards food sources that are located far apart and the short steps to finding food spread in a small area at each source.

Brownian motion is a more famous kind of random walk. It’s the name for the movement of an object that’s following the Wiener process. This means the object’s path needs to obey the following five rules (from the same post):

(i) Each increment of the process is independent of other (non-overlapping) increments;

(ii) How much the process changes over a period of time depends only on the duration of the period;

(iii) Increments in the process are randomly sampled from a Gaussian distribution;

(iv) The process has a statistical mean equal to zero;

(v) The process’s covariance between any two time points is equal to the lower variance at those two points (variance denotes how quickly the value of a variable is spreading out over time).

Thus Brownian motion models the movement of pollen grains in water, dust particles in the air, electrons in a conductor, and colloidal particles in a fluid, and the fluctuation of stock prices, the diffusion of molecules in liquids, and population dynamics in biology. That is, all these processes in disparate domains evolve at least in part according to the rules of the Wiener process.

Still doesn’t mean a shark understands what a Lévy flight is. By saying “sharks use a Lévy flight”, we also discard in the process how the shark makes its decisions — something worth learning about in order to make more complete sense of the world around us rather than force the world to make sense only in those ways we’ve already dreamt up. (This is all the more relevant now with #sharkweek just a week away.)

I care so much because metaphors are bridges between language and reality. Even if the statement “sharks employ advanced mathematical concepts” doesn’t feature a metaphor, the risk it represents hews close to one that stalks the use of metaphors in science journalism: the creation of false knowledge.

Depending on the topic, it’s not uncommon for science journalists to use metaphors liberally, yet scientists have not infrequently upbraided them for using the wrong metaphors in some narratives or for not alerting readers to the metaphors’ limits. This is not unfair: while I disagree with some critiques along these lines for being too pedantic, in most cases it’s warranted. As science philosopher Daniel Sarewitz put it in that 2012 article:

Most people, including most scientists, can acquire knowledge of the Higgs only through the metaphors and analogies that physicists and science writers use to try to explain phenomena that can only truly be characterized mathematically.

Here’s The New York Times: “The Higgs boson is the only manifestation of an invisible force field, a cosmic molasses that permeates space and imbues elementary particles with mass … Without the Higgs field, as it is known, or something like it, all elementary forms of matter would zoom around at the speed of light, flowing through our hands like moonlight.” Fair enough. But why “a cosmic molasses” and not, say, a “sea of milk”? The latter is the common translation of an episode in Hindu cosmology, represented on a spectacular bas-relief panel at Angkor Wat showing armies of gods and demons churning the “sea of milk” to producean elixir of immortality.

For those who cannot follow the mathematics, belief in the Higgs is an act of faith, not of rationality.

A metaphor is not the thing itself and shouldn’t be allow to masquerade as such.

Just as well, there are important differences between becoming aware of something and learning it, and a journalist may require metaphors only to facilitate the former. Toeing this line also helps journalists tame the publics’ expectations of them.

Featured image credit: David Clode/Unsplash.

A heatwave is like the COVID-19 virus. During the pandemic, the virus infected and killed many people. When vaccines became available, the mortality rate dropped even though the virus continued to spread. But vaccines weren’t the only way to keep people from dying. The COVID-19 virus killed more people if the people were already unhealthy

In India, an important cause for people being unhealthy is the state itself. In many places, the roads are poorly laid, kicking dust exposed by traffic use up into the air, where it joins the PM2.5 particles emitted by industrial facilities allowed to set up shop near residential and commercial areas without proper emission controls. If this is one extreme, becauses these experiences are so common for so many Indians, at the other is the state’s apathy towards public health. India’s doctor-to-patient ratio is dismal; hospitals are understaffed and under-equipped; drug quality is so uneven as to be a gamble; insurance coverage is iffy and unclear; privatisation is increasing; and the national government’s financial contribution towards public health is in free fall.

For these reasons as well, and not just because of vaccine availability or coverage, the COVID-19 virus killed more people than it should have been able to. A person’s vulnerability to this or any other infection is thus determined by their well-being — which is affected both by explicit factors like a new pathogen in the population and implicit factors like the quality of healthcare they have been able to access.

A heatwave resembles the virus for the same reason: a person’s vulnerability to high heat is determined by their well-being — which in turn is affected by the amount of ambient heat and relative humidity as well as the extent to which they are able to evade the effects of that combination. This weekend, a new investigative effort by a team of journalists at The Hindu (including me) has reported just this fact, but for the first time with ground-zero details that people in general, and perhaps even the Tamil Nadu government itself, have thus far only presumed to be the case. Read it online, in the e-paper or in today’s newspaper.

The fundamental issues are two-pronged. First, Tamil Nadu’s policies on protecting people during heatwaves require the weather department to have declared a heatwave to apply. Second, even when there is no heatwave, many people but especially the poorer consistently suffer heatwave conditions. (Note: I’m criticising Tamil Nadu here because it’s my state of residence and equally because it’s one of a few states actually paying as much attention to economic growth as it is to public health, of which heat safety is an important part.)

The net effect is for people to suffer their private but nonetheless very real heatwave conditions without enjoying the support the state has promised for people in these conditions. The criticism also indicts the state for falling short on enforcing other heat-related policies that leave the vulnerable even more stranded.

The corresponding measures include (i) access to clean toilets, a lack of which forces people — but especially women, who can’t urinate in public the way men are known to — to drink less water and suppress their urges to urinate, risking urinary tract infections; (ii) access to clean and cool drinking water, a paucity of which forces people to pay out of their pockets to buy chilled water or beverages, reducing the amount of money they have left for medical expenses as well as risking the ill health that comes with consuming aerated and/or sugary beverages; and (ii) state-built quarters that pay meaningful attention to ventilating living spaces, which when skipped exposes people to humidity levels that prevent their bodies from cooling by sweating, rendering them more susceptible to heat-related illnesses.

And as The Hindu team revealed, these forms of suffering are already playing out.

The India Meteorological Department defines a heatwave based on how much the temperature deviates from a historical average. But this is a strictly meteorological definition that doesn’t account for the way class differences create heatwave-like conditions. These conditions kick in as a combination of temperature and humidity, and as the report shows, even normal temperature can induce them if the relative humidity is higher and/or if an individual is unable to cool themselves. The state has a significant role to play in the latter. Right now, it needs to abandon the strictly meteorological definition of heatwaves in its policy framework and instead develop a more holistic sociological definition.

Featured image credit: Austin Curtis/Unsplash.

In physics, the second law of thermodynamics says that a closed system tends to become more disordered over time. This disorder is captured in an entity called entropy. Many devices, especially clocks, are affected by this law because they need to tick regularly to measure time. But every tick creates a bit of disorder, i.e. increases the entropy, and physicists have believed for a long time now that this places a fundamental limit on how precise a clock can be. The more precise you want your clock, the more entropy (and thus more energy) you’ll have to expend.

A study published in Nature Physics on June 2 challenges this wisdom. In it, researchers from Austria, Malta, and Sweden asked if the second law of thermodynamics really set a limit on a clock’s precision and came away, surprisingly, with a design of a new kind of quantum clock that’s too precise scientists once believed possible for the amount of energy it spends to achieve that precision.

The researchers designed this clock using a spin chain. Imagine a ring made of several quantum sites, like minuscule cups. Each cup can hold an excitation — say, a marble that can hop from cup to cup. This excitation moves around the ring and every time it completes a full circle, the clock ticks once. A spin chain is, broadly speaking, a series of connected quantum systems (the sites) arranged in a ring and the excitation is a subatomic particle or packet of energy that moves from site to site.

In most clocks, every tick is accompanied by the dissipation of some energy and a small increase in entropy. But in the model in the new study, only the last link in the circle, where the last quantum system was linked to the first one, dissipated energy. Everywhere else, the excitation moved without losing energy, like a wave gliding smoothly around the ring. The movement of the excitation in this lossless way through most of the ring is called coherent transport.

The researchers used computer simulations to help them adjust the hopping rates — or how easily the excitation moved between sites — and thus to make the clock as precise as possible. They found that the best setup involved dividing the ring into three regions: (i) in the preparation ramp, the excitation was shaped into a wave packet; (ii) in the bulk propagation phase, the wave packet moved steadily through the ring; and (iii) in the boundary matching phase, the wave packet was reset for the next tick.

The team measured the clock’s precision as the number of ticks it completed before it was one tick ahead or behind a perfect clock. Likewise, team members defined the entropy per tick to be the amount of energy dissipated per tick. Finally, the team compared this quantum clock to classical clocks and other quantum models, which typically show a linear relationship between precision and entropy: e.g. if the precision doubled, the entropy doubled as well.

The researchers, however, found that the precision of their quantum clock grew exponentially with entropy. In other words, if the amount of entropy per tick increased only slightly, the precision increased by a big leap. It was proof that, at least in principle, it’s possible to build a clock to be arbitrarily precise while keeping the system’s entropy down, all without falling afoul of the second law.

That is, contrary to what many physicists thought, the second law of thermodynamics doesn’t strictly limit a clock precision, at least not for quantum clocks like this one. The clock’s design allowed it to sidestep the otherwise usual trade-off between precision and entropy.

During coherent transport, the process is governed only by the system’s Hamiltonian, i.e. the rules for how energy moves in a closed quantum system. In this regime, the excitation acts like a wave that spreads smoothly and reversibly, without losing any energy or creating any disorder. Imagine a ball rolling on a perfectly smooth, frictionless track. It keeps moving without slowing down or heating up the track. Such a thing is impossible in classical mechanics, like in the ball example, but it’s possible in quantum systems. The tradeoff of course is that the latter are very small and very fragile and thus harder to manipulate.

In the present study, the researchers have proved that it’s possible to build a quantum clock that takes advantage of coherent transport to tick while dissipating very little energy. Their model, the spin chain, uses a Hamiltonian that only allows the excitation to coherently hop to its nearest neighbour. The researchers engineered the couplings between the sites in the preparation ramp part of the ring to shape the excitation into a traveling wave packet that moves predominantly in the forward direction.

This tendency to move in only direction is further bolstered at the last link, where the last site is coupled to the first. Here, the researchers installed a thermal gradient — a small temperature difference that encouraged the wave to restart its journey rather than be reflected and move backwards through the ring. When the excitation crossed this thermodynamic bias, the clock ticked once and also dissipated some energy.

Three points here. First, remember that this is a quantum system. The researchers are dealing with energy (almost) at its barest, manipulating it directly without having to bother with an accoutrement of matter covering it. In the classical regime, such accoutrements are unavoidable. For example, if you have a series of cups and you want to make an excitation hop through it, you do so with a marble. But while the marble contains the (potential) energy that you want to move through the cups, it also has mass and it dissipates energy whenever it hops into a cup, e.g. it might bounce when it lands and it will release sound when it strikes the cup’s material. So while the marble metaphor earlier might have helped you visualise the quantum clock, remember that the metaphor has limitations.

Second, for the quantum clock to work as a clock, it needs to break time-reversal symmetry (a concept I recently discussed in the context of quasicrystals). Say you remove the thermodynamic bias at the last link of the ring and replace it with a regular link. In this case the excitation will move randomly — i.e. at each step it will randomly pick the cup to move to, forward or backward, and keep going. If you reversed time, the excitation’s path will still be random and just evolve in reverse.

However, the final thermodynamically biased link causes the excitation to acquire a preference for moving in one direction. The system thus breaks time-reversal symmetry because even if you reverse the flow of time, the system will encourage the excitation to move in one direction and one direction only. This in turn is essential for the quantum system to function like a clock. That is, the excitation needs to traverse a fixed number of cups in the spin chain and then start from the first cup. Only between these two stages will the system count off a ‘tick’. Breaking time-reversal symmetry thus turns the device into a clock.

Three, the thermodynamic bias ensures that the jump from the last site to the first is more likely than the reverse, and the entropy is the cost the system pays in order to ensure the jump. Equally, the greater the thermodynamic bias, the more likely the excitation is to move in one direction through spin chain as well as make the jump in the right direction at the final step. Thus, the greater the thermodynamic bias, the more precise the clock will be.

The new study excelled by creating a sufficiently precise clock while minimising the entropy cost.

According to the researchers, its design design could help build better quantum clocks, which are important for quantum computers, quantum communication, and to make ultra-precise precise measurements of the kind demanded by atomic clocks. The clock’s ticks could also be used to emit single photons at regular intervals — a technology increasingly in demand for its use in quantum networks of the sort China, the US, and India are trying to build.

But more fundamentally, the clock’s design — which confines energy dissipation to a single link and uses coherent transport everywhere else — and that design’s ability to evade the precision-entropy trade-off challenges a longstanding belief that the second law of thermodynamics strictly limits precision.

Featured image credit: Meier, F., Minoguchi, Y., Sundelin, S. et al. Nat. Phys. (2025).

Scientists have made a new material that is both a quasicrystal and antiferromagnetic — a combination never seen before.

Quasicrystals are a special kind of solid. Unlike normal crystals, whose atoms are arranged in repeating patterns, quasicrystals have patterns that never exactly repeat but which still have an overall order. While regular crystals have left-right symmetries, quasicrystals have unusual rotational ones.

For decades, scientists wondered if certain kinds of magnetism, but especially antiferromagnetism, could exist in these strange materials. In all materials the electrons have a property called spin. It’s as if a small magnet is embedded inside each electron. The spin denotes the direction of this magnet’s magnetic field. In ferromagnets, the spins are aligned in a common direction, so the materials are attracted to magnets. In antiferromagnetic materials, the electron spins line up in alternating directions, so their effects cancel out.

While antiferromagnetism is common in regular crystals, it’s thus far never been observed in a true quasicrystal.

The new study is the first to show clear evidence of antiferromagnetic order in a real, three-dimensional quasicrystal — one made of gold, indium, and europium. The findings were published in Nature Physics on April 14.

The team confirmed such a material is real by carefully measuring how its atoms and spins are arranged and by observing how it behaves at low temperatures. Their work shows that even in the weird world of quasicrystals, complex magnetic order is possible, opening the door to new discoveries and technologies.

The scientists created a new alloy with the formula Au<sub>56</sub>In<sub>28.5</sub>Eu<sub>15.5</sub>. This means in 1,000 atoms’ worth of the material, 560 will be gold, 285 will be indium, and 155 will be europium. The composition tells us that the scientists were going for a particularly precise combination of these elements — which they could have known in one of two ways. It might have been trial-and-error*, but that makes research very expensive, or the scientists had reasons to expect antiferromagnetic order would appear in this material.

They did. Specifically, the team focused on Au<sub>56</sub>In<sub>28.5</sub>Eu<sub>15.5</sub> because of its (i) unique positive Curie-Weiss temperature and (ii) rare-earth content, and (iii) because its structural features matched the theoretical criteria for stable antiferromagnetic order. Previous studies focused on quasicrystals containing rare-earth elements because they often have strong magnetic interactions. However, these compounds typically displayed a negative Curie-Weiss temperature, indicating dominant antiferromagnetic interactions but resulting only in disordered magnetic states.

A positive Curie-Weiss temperature indicates dominant ferromagnetic interactions. In this case, however, it also suggested a unique balance of magnetic forces that could potentially stabilise antiferromagnetic order rather than spin-glass behaviour. Studies on approximant crystals — periodic structures closely related to quasicrystals — had also shown that both ferromagnetic and antiferromagnetic orders are stabilised only when the Curie-Weiss temperature is positive. In contrast, a negative temperature led to spin-glass states.

The scientists of the new study noticed that the Au-In-Eu quasicrystal fit into the positive Curie-Weiss temperature category, making it a promising candidate to have antiferromagnetic order.

For added measure, by slightly altering the composition, e.g. adding an impurity to increase the electron-per-atom ratio, the scientists could make the antiferromagnetic phase disappear, to be replaced by spin-glass behaviour. This sensitivity to electron concentration further hinted that the composition of the alloy was at a sweet spot for stabilising antiferromagnetism.

Finally, the team had also recently discovered ferromagnetic order in some similar gold-based quasicrystals with rare-earth elements. The success encouraged them to explore the magnetic properties of new compositions, especially those with unusual Curie-Weiss temperatures.

The Au-In-Eu quasicrystal is also a Tsai-type icosahedral quasicrystal, meaning it features a highly symmetric atomic arrangement. Theoretical work has suggested that such structures could support antiferromagnetic order in the right conditions, especially if the atoms occupied specific sites in the lattice.

To make the alloy, the scientists used a technique called arc-melting, where highly pure metals are melted together using an electric arc, then quickly cooled to form the solid quasicrystal. To ensure the mixture was even, the team melted and flipped the sample several times.

Then they used X-ray and electron diffraction to check the atomic arrangement. These techniques passed X-rays and electrons through the material. A detector on the other side picked up the radiation scattered by the material’s atoms and used it to recreate their arrangement. The patterns showed the material was a primitive icosahedral quasicrystal, a structure with 20-sided symmetry and no repeating units.

The team also confirmed special arrangement of atoms by the way the diffraction patterns followed mathematical rules that are special to quasicrystals. Team members also used a magnetometer to track how much the material was magnetised when exposed to a magnetic field, from temperatures as low as 0.4 K to up to 300 K. Finally they also measured the material’s specific heat, i.e. the amount of heat energy it took to raise its temperature by 1º C. This reading can show signs of magnetic transitions.

To confirm how the spins inside the material were arranged, the team used neutron diffraction. Neutrons are adept at passing through materials and are sensitive to both atoms’ positions and magnetic order. By comparing patterns at temperatures above and below the suspected transition point, they could spot the appearance of new peaks that signal magnetic order.

This way, the team reported that at 6.5 K, the magnetisation curve showed a sharp change, known as a cusp. This is a classic sign of an antiferromagnetic transition, where the material suddenly changes from being unordered to having a regular up-and-down pattern of spins. The specific heat also showed a sharp peak at this temperature, confirming something dramatic was happening inside the material.

The scientists also reported that there was no sign of spin-glass behaviour — where the spins are pointing in random directions but unchanging — which is common in other magnetic quasicrystals.

Below 6.5 K, new peaks appeared in the neutron diffraction data, evidence that the spins inside the material were lining up in the regular but alternating pattern characteristic of antiferromagnetic order. The peaks were also sharp and well-defined, showing the order was long-range, meaning they were there throughout the material and not confined to small patches.

The team also experimented by adding a small amount of tin to the alloy, which changed the balance of electrons. This little change caused the material to lose its antiferromagnetic order and become a spin glass instead, showing how delicate the balance is between different magnetic states in quasicrystals.

The findings are important because this is the first time scientists have observed antiferromagnetic order in a real, three-dimensional quasicrystal, settling a long-standing debate. They also open up a new field of study, of quasiperiodic antiferromagnets, and suggest that by carefully tuning the composition, scientists may be able to find yet other types of magnetic order in quasicrystals.

“The present discovery will stimulate both experimental and theoretical efforts to elucidate not only its unique magnetic structure but also the intrinsic properties of the quasiperiodic order parameter,” the scientists wrote in their paper. “Another exciting aspect of magnetically ordered quasicrystals is their potential for new applications such as functional materials in spintronics” — which use electron spins to store and process information in ultra-fast computers of the future.

* Which is not the same as serendipity.

Featured image credit: Nature Physics volume 21, pages 974–979 (2025).

In the last two or three years, groups of scientists from around the world have made several claims that they had discovered a room-temperature superconductor. Many of these claims concerned high-pressure superconductors — materials that superconduct electricity at room temperature but only if they are placed under extreme pressure (a million atmospheres’ worth). Yet other scientists had challenged these claims on many grounds, but one in particular was whether these materials really exhibited the Meissner effect.

Room-temperature superconductors are often called the ‘holy grail’ of materials science. I abhor clichés but in this case the idiom fits perfectly. If such a material is invented or discovered, it could revolutionise many industries. To quote at length from an article by electrical engineer Massoud Pedram in The Conversation:

Room-temperature superconductors would enable ultra high-speed digital interconnects for next-generation computers and low-latency broadband wireless communications. They would also enable high-resolution imaging techniques and emerging sensors for biomedical and security applications, materials and structure analyses, and deep-space radio astrophysics.

Room-temperature superconductors would mean MRIs could become much less expensive to operate because they would not require liquid helium coolant, which is expensive and in short supply. Electrical power grids would be at least 20% more power efficient than today’s grids, resulting in billions of dollars saved per year, according to my estimates. Maglev trains could operate over longer distances at lower costs. Computers would run faster with orders of magnitude lower power consumption. And quantum computers could be built with many more qubits, enabling them to solve problems that are far beyond the reach of today’s most powerful supercomputers.

However, this surfeit of economic opportunities could also lure scientists into not thoroughly double-checking their results, cherry-picking from their data or jumping to conclusions if they believe they have found a room-temperature superconductor. Many papers written by scientists claiming they had found a room-temperature superconductor have in fact been published in and subsequently retracted from peer-reviewed journals with prestigious reputations, including Nature and Science, after independent experts found the papers to contain flawed data. Whatever the reasons for these mistakes, independent scrutiny of such reports has become very important.

If a material is a superconductor, it needs to meet two conditions*. The first of course is that it needs conduct a direct electric current with zero resistance. Second, the material should display the Meissner effect. Place a magnet over a superconducting material. Then, gradually cool the material to lower and lower temperatures, until you cross the critical temperature. Just as you cross this threshold, the magnet will start to float above the material. You’ve just physically observed the Meissner effect. It happens because when the material transitions to its superconducting state, it will expel all magnetic fields within its bulk to its surface. This results in any magnets already sitting nearby to be pushed away. In fact, the Meissner effect is considered to be the hallmark sign of a superconductor because it’s difficult to fake.

The problem with acquiring evidence of the Meissner effect is the setup in which many of these materials become superconductors. In order to apply the tens to hundreds of gigapascals (GPa) of pressure, a small sample of the material — a few grams or less — is placed between a pair of high-quality diamond crystals and squeezed. This diamond anvil cell apparatus leaves no room for a conventional magnetic field sensor to be placed inside the cell. Measuring the magnetic properties of the sample is also complicated because of the fields from other sources in the apparatus, which will have to be accurately measured and then subtracted from the final data.

To tackle this problem, some scientists have of late suggested measuring the sample’s magnetic properties using the only entity that can still enter and leave the diamond anvil cell: light.

In technical terms, such a technique is called optical magnetometry. Magnetometry in general is any technique that converts some physical signal into data about a magnetic field. In this case the signal is in the form of light, thus the ‘optical’ prefix. To deploy optical magnetometry in the context of verifying whether a material is a high-pressure superconductor, scientists have suggested using nitrogen vacancy (NV) centres.

Say you have a good crystal of diamond with you. The crystal consists of carbon atoms bound to each other in sets of four in the shape of a pyramid. Millions of copies of such pyramids together make up the diamond. Now, say you substitute one of the carbon atoms in the gem with a nitrogen atom and also knock out an adjacent carbon atom. Physicists have found that this vacancy in the lattice, called an NV centre, has interesting, useful properties. For example, an NV centre can fluoresce, i.e. absorb light of a higher frequency and emit light of a lower frequency.

Because each NV centre is surrounded by three carbon atoms and one nitrogen atom, the vacancy hosts six electrons, two of which are unpaired. All electrons have a property called quantum spin. The quantum spin is the constitutive entity of magnetism the same way the electric charge is the constitutive entity of electricity. For example, if a block of iron is to be turned into a magnet, the spins of all the electrons inside have to be made point in the same direction. Each spin can point in one of two directions, which for a magnet are called ‘north’ and ‘south’. Planet earth has a magnetic north and a magnetic south because the spins of the trillions upon trillions of electrons in its core have come to point in roughly the same direction.

The alignment of the spins of different electrons also affects what energy they have. For example, in the right conditions, an atom with two electrons will have more energy if the electrons’ spins are aligned (↑↑) than when the electrons’ spins are anti-aligned (↑↓). This fundamental attribute of the electrons in the NV centres allows the centres to operate as a super-sensitive detector of magnetic fields — and which is what scientists from institutions around France have reported doing in a June 30 paper in Physical Review Applied.

The scientists implanted a layer of 10,000 to 100,000 NV centres a few nanometres under the surface of one of the diamond anvils. These centres had electrons with energies precisely 2.87 GHz apart.** When the centres were then exposed to microwave laser of some frequency, every NV centre could absorb green laser light and re-emit red light.

As the diamond anvils squeezed the sample past 4 GPa, the pressure at which it would have become a superconductor, the sample displayed the Meissner effect, expelling magnetic fields from within its bulk to the surface. As a result, the NV centres were exposed to a magnetic field in their midst that wasn’t there before. This field affected the electrons’ collective spin and thus their energy levels, which in turn caused the red light being emitted from the centres to dim.

The researchers could easily track the levels and patterns of dimming in the NV centres with a microscopy, and based on that were able to ascertain whether the sample had displayed the Meissner effect. As Physical Review Letters associate editor Martin Rodriguez-Vega wrote in Physics magazine: “A statistical analysis of the [optical] dataset revealed information about the magnetic-field strength and orientation across the sample. Mapping these quantities produced a visualisation of the Meissner effect and revealed the existence of defects in the superconductor.”

Because the NV centres were less than 1 micrometre away from the sample, they were extremely sensitive to changes in the magnetic field. In fact the researchers reported that the various centres were able to reveal the critical temperature for different parts of the sample separately than for the sample as a whole — a resolution not possible with conventional techniques. The pristine diamond matrix also conferred the electrons’ spins inside the NV centres with a long lifetime. And because there were so many NV centres, the researchers were able to ‘scan’ them with the microwave laser en masse instead of having to maintain focus on a single point on the diamond anvil, when looking for evidence of changes in the sample’s magnetic field. Finally, while the sample in the study became superconducting at a critical temperature of around 140 K, the centres were stable to under 4 K.

Another major advantage of the technique is that it can be used with type II superconductors as well. Type I superconductors are materials that transition to their superconducting state in a single step, under the critical temperature. Type II superconductors transition to their superconducting states in more than one step and display a combination of flux-pinning and the Meissner effect. From my piece in The Hindu in August 2023: “When a flux-pinned superconductor is taken away from a particular part of the magnetic field and put back in, it will snap back to its original relative position.” This happens because type II materials, while they don’t expel magnetic fields from within their bulk, also prevent the fields from moving around inside. Thus the magnetic field lines are pinned in place.

Because of the spatial distribution of the NV centres and their sensitivity, they can reveal flux-pinning in the sample by ‘sensing’ the magnetic fields at different distances.

* The material can make a stronger case for itself if it displays two more properties. (i) The heat energy required to raise the material’s electrons by 1º C has to change drastically at the critical temperature, which is the temperature below which the material becomes a superconductor. (ii) The material’s electrons shouldn’t be able to have certain energy readings. (That is, a map of the energies of all the electrons should show some gaps.) These properties are however considered optional.

** While 2.87 GHz is a frequency figure, recall Planck’s equation from high school: E = hv. Energy is equal to frequency times Planck’s constant, h. Since h is a constant (6.62 × 10<sup>-34</sup> m<sup>2</sup>kg/s), energy figures are frequently denoted in terms of frequency in physics. An interested party can calculate the energy by themselves.

There’s no denying that there currently prevails a public culture in India that equates criticism, even well-reasoned, with pooh-poohing. It’s especially pronounced in certain geographies where the Bharatiya Janata Party (BJP) enjoys majority support as well as vis-à-vis institutions that the subscribers of Hindu politics consider to be ripe for international renown, especially in the eyes of the country’s former colonial masters. The other side of the same cultural coin is the passive encouragement it offers to those who’d play up the feats of Indian enterprises even if they lack substantive evidence to back their claims up. While these tendencies are pronounced in many enterprises, I have encountered them most often in the spaceflight domain.

Through its feats of engineering and administration over the years, the Indian Space Research Organisation (ISRO) has cultivated a deserved reputation of setting a high bar for itself and meeting them. Its achievements are the reason why India is one of a few countries today with a functionally complete space programme. It operates launch vehicles, conducts spaceflight-related R&D, has facilities to develop as well as track satellites, and maintains data-processing pipeliness to turn the data it collects from space into products usable for industry and academia. It is now embarking on a human spaceflight programme as well. ISRO has also launched interplanetary missions to the moon and Mars, with one destined for Venus in the works. In and of itself the organisation has an enviable legacy. Thus, unsurprisingly, many sections of the Hindutva brigade have latched onto ISRO’s achievements to animate their own propaganda of India’s greatness, both real and imagined.

The surest signs of this adoption are most visible when ISRO missions fail or succeed in unclear ways. The Chandrayaan 2 mission and the Axiom-4 mission respectively are illustrative examples. As if to forestall any allegations that the Chandrayaan 2 mission failed, then ISRO chairman K. Sivam said right after its Vikram lander crashed on the moon that it had been a “98% success”. Chandrayaan 2 was a technology demonstrator and it did successfully demonstrate most of those onboard very well. The “98%” figure, however, was so disproportionate as to suggest Sivan was defending the mission less on its merits than on its ability to fit into reductive narratives of how good ISRO was. (Recall, similarly, when former DCGI V.G. Somani claimed the homegrown Covaxin vaccine was “110% safe” when safety data from its phase III clinical trials weren’t even available.)

On the other hand, even as the Axiom-4 mission was about to kick off, neither ISRO nor the Department of Space (DoS) had articulated what Indian astronaut Shubhanshu Shukla’s presence onboard the mission was expected to achieve. If these details didn’t actually exist before the mission, to participate in which ISRO had paid Axiom Space more than Rs 500 crore, both ISRO and the DoS were effectively keeping the door open to picking a goalpost of their choosing to kick the ball through as the mission progressed. If they did have these details but had elected to not share them, their (in)actions raised — or ought to have — difficult questions about the terms on which these organisations believed they were accountable in a democratic country. Either way, the success of the Axiom-4 mission vis-à-vis Shukla’s participation was something of an empty vessel: a ready receptacle for any narrative that could be placed inside ex post facto.

At the same time, raising this question has often been construed in the public domain, but especially on social media platforms, in response to arguments presented in the news, and in conversations among people interested in Indian spaceflight, as naysaying Shukla’s activities altogether. By all means let’s celebrate Shukla’s and by extension India’s ‘citius, altius, fortius’ moment in human spaceflight; the question is: what didn’t ISRO/DoS share before Axiom-4 lifted off and why? (Note that what journalists have been reporting since liftoff, while valuable, isn’t the answer to the question posed here.) While it’s tempting to think this pinched communication is a strategy developed by the powers that be to cope with insensitive reporting in the press, doing so would also ignore the political capture institutions like ISRO have already suffered and which ISRO arguably has as well, during and after Sivan’s term as chairman.

For just two examples of institutions that have historically enjoyed a popularity comparable in both scope and flavour to that of ISRO, consider India’s cricket administration and the Election Commission. During the 2024 men’s T20 World Cup that India eventually won, the Indian team had the least amount of travel and the most foreknowledge on the ground it was to play its semifinal game on. At the 2023 men’s ODI World Cup, too, India played all its matches on Sundays, ensuring the highest attendance for its own contests rather than be able to share that opportunity with all teams. The tournament is intended to be a celebration of the sport after all. For added measure, police personnel were also deployed at various stadia to take away spectators’ placards and flags in support of Pakistan in matches featuring the Pakistani team. The stage management of both World Cups only lessened, rather than heightened, the Indian team’s victories.

It’s been a similar story with the Election Commission of India, which has of late come under repeated attack from the Indian National Congress party and some of its allies for allegedly rigging their electronic voting machines and subsequently entire elections in favour of the BJP. While the Congress has failed to submit the extraordinary evidence required to support these extraordinary claims, doubts about the ECI’s integrity have spread anyway because there are other, more overt ways in which the once-independent institution of Indian democracy favours the BJP — including scheduling elections according to the availability of party supremo Narendra Modi to speak at rallies.

Recently, a more obscure but nonetheless pertinent controversy erupted in some circles when in an NDTV report incumbent ISRO chairman V. Narayanan seemed to suggest that SpaceX called one of the attempts to launch Axiom-4 off because his team at ISRO had insisted that the company thoroughly check its rocket for bugs. The incident followed SpaceX engineers spotting a leak on the rocket. The point of egregiousness here is that while SpaceX had built and flown that very type of rocket hundreds of times, Narayanan and ambiguous wording in the NDTV report made it out to be that SpaceX would have flown the rocket if not for ISRO’s insistence. What’s more likely to have happened is NASA and SpaceX engineers would have consulted ISRO as they would have consulted the other agencies involved in the flight — ESA, HUNOR, and Axiom Space — about their stand, and the ISRO team on its turn would have clarified its position: that SpaceX recheck the rocket before the next launch attempt. However, the narrative “if not for ISRO, SpaceX would’ve flown a bad rocket” took flight anyway.

Evidently these are not isolated incidents. The last three ISRO chairmen — Sivan, Somanath, and now Narayanan — have progressively curtailed the flow of information from the organisation to the press even as they have maintained a steady pro-Hindutva, pro-establishment rhetoric. All three leaders have also only served at ISRO’s helm when the BJP was in power at the Centre, wielding its tendency to centralise power by, among others, centralising the permissions to speak freely. Some enterprising journalists like Chethan Kumar and T.S. Subramanian and activists like r/Ohsin and X.com/@SolidBoosters have thus far kept the space establishment from resembling a black hole. But the overarching strategy is as simple as it is devious: while critical arguments become preoccupied by whataboutery and fending off misguided accusations of neocolonialist thinking (“why should we measure an ISRO mission’s success the way NASA measures its missions’ successes?”), unconditional expressions of support and adulation spread freely through our shared communication networks. This can only keep up a false veil of greatness that crumbles the moment it brooks legitimate criticism, becoming desperate for yet another veil to replace itself.

But even that is beside the point: to echo the philosopher Bruno Latour, when criticism is blocked from attending to something we have all laboured to build, that something is deprived of the “care and caution” it needs to grow, to no longer be fragile. Yet that’s exactly what the Indian space programme risks becoming today. Here’s a brand new case in point, from the tweets that prompted this post: according to an RTI query filed by @SolidBoosters, India’s homegrown NavIC satellite navigation constellation is just one clock failure away from “complete operational collapse”. The issue appears to be ISRO’s subpar launch cadence and the consequently sluggish replacement of clocks that have already failed.

Granted, rushed critiques and critiques designed to sting more than guide can only be expected to elicit defensive posturing. But to minimise one’s exposure to all criticism altogether, especially those from learned quarters and conveyed in respectful language, is to deprive oneself of the pressure and the drive to solve the right problems in the right ways, both drawing from and adding to India’s democratic fabric. The end results are public speeches and commentary that are increasingly removed from reality as well as, more importantly, thicker walls between criticism and The Thing it strives to nurture.

From ‘What is next for Iran’s nuclear programme?’, The Hindu, June 28, 2025:

As things stand, Iran has amassed both the technical knowhow and the materials required to make a nuclear weapon. Second, the Israelis and the Americans have failed to deprive Iran of these resources in their latest salvo. In fact the airstrikes against Iran from June 13 cast Tehran as the victim of foreign aggression and increased the premium on its option to withdraw from the Non-Proliferation Treaty (NPT) without significant international censure.

…

While Tehran’s refusal to cooperate with the IAEA is suggestive, it hasn’t explicitly articulated that it will pursue nuclear weapons. … But the presence of large quantities of HEU in the stockpile is intriguing. From a purely technical standpoint, the HEU can still be diverted for non-military applications…

… such as R&D for naval applications and downconversion to less enriched reactor fuel. But these are niche use cases. In fact while it’s possible to downconvert a stockpile of uranium enriched to 60% to that enriched to 19.75%, 5% or 3% without using centrifuges, it’s also possible to do this by mixing uranium enriched to 20% with natural or depleted feedstock.

If anything, the highly enriched uranium stockpile [which Iran went to some lengths to protect from American bombing], the technical knowhow in the country, the absence of a nuclear warhead per se, and the sympathy created by the bombing allow Tehran a perfect bargaining chip: to simultaneously be in a state of pre-breakout readiness while being able to claim in earnest that it is interested in nuclear energy for peace.

Read more.

Featured image: An example of zellij tilework in the Al Attarine Madrasa in Fes, Morocco (2012), with complex geometric patterns on the lower walls and a band of calligraphy above. Caption and credit: just_a_cheeseburger (CC BY)

‘Quasi’ means almost. It’s an unfair name for quasicrystals. These crystals exist in their own right. Their name comes from the internal arrangement of their atoms. A crystal is made up of a repeating group of some atoms arranged in a fixed way. The smallest arrangement that repeats to make up the whole crystal is called the unit cell. In diamond, a convenient unit cell is four carbon atoms bonded to each other in a tetrahedral (pyramid-like) arrangement. Millions of copies of this unit cell together make up a diamond crystal. The unit cell of sodium chloride has a cubical shape: the chloride ions (Cl<sup>–</sup>) occupy the corners and face centres while the sodium ions (Na<sup>+</sup>) occupy the middle of the edges and centre of the cube. As this cube repeats itself, you get table salt.

The structure of all crystals thus follows two simple rules: have a unit cell and repeat it. Thus the internal structure of crystals is periodic. For example if a unit cell is 5 nanometres wide, it stands to reason you’ll see the same arrangement of atoms after every 5 nm. And because it’s the same unit cell in all directions and they don’t have any gaps between them, the unit cells fill the space available. It’s thus an exercise in tiling. For example, you can cover a floor of any shape completely with square or triangular tiles (you’ll just need to trim those at the edges). But you can’t do this with pentagonal tiles. If you do, the tiles will have gaps between them that other pentagonal tiles can’t fill.

Quasicrystals buck this pattern in a simple way: their unit cells are like pentagonal tiles. They repeat themselves but the resulting tiling isn’t periodic. There are no gaps in the crystal either because instead of each unit cell just like the one on its left or right, the tiles sometimes slot themselves in by rotating by an angle. Thus rather than the crystal structure following a grid-like pattern, the unit cells seem to be ordered along curves. As a result, even though the structure may have an ordered set of atoms, it’s impossible to find a unit cell that by repeating itself in a straight line gives rise to the overall crystal. In technical parlance, the crystal is said to lack translational symmetry.

Such structures are called quasicrystals. They’re obviously not crystalline, because they lack a periodic arrangement of atoms. They aren’t amorphous either, like the haphazardly arranged atoms of glass. Quasicrystals are somewhere in between: their atoms are arranged in a fixed way, with different combinations of pentagonal, octagonal, and other tile shapes that are disallowed in regular crystals, and with the substance lacking a unit cell. Instead the tiles twist and turn within the structure to form mosaic patterns like the ones featured in Islamic architecture (see image at the top).

The discovery of quasicrystals in the early 1980s was a revolutionary moment in the history of science. It shook up what chemists believed a crystal should look like and what rules the unit cell ought to follow. The first quasicrystals that scientists studied were made in the lab, in particular aluminium-manganese alloys, and there was a sense that these unusual crystals didn’t occur in nature. That changed in the 1990s and 2000s when expeditions to Siberia uncovered natural quasicrystals in meteorites that had smashed into the earth millions of years ago. But even this discovery kept one particular question about quasicrystals alive: why do they exist? Both Al-Mn alloys and the minerals in meteorites form in high temperatures and extreme pressures. The question of their existence, more than just because they can, is a question about whether the atoms involved are forced to adopt a quasicrystal rather than a crystal structure. In other words, it asks if the atoms would rather adopt a crystal structure but don’t because their external conditions force them not to.

This post benefited from feedback from Adhip Agarwala.

Often a good way to understand the effects of extreme conditions on a substance is using the tools of thermodynamics — the science of the conditions in which heat moves from place to another. And in thermodynamics, the existential question can be framed like this, to quote from a June paper in Nature Physics: “Are quasicrystals enthalpy-stabilised or entropy-stabilised?” Enthalpy-stabilised means the atoms of a quasicrystal are arranged in a way where they collectively have the lowest energy possible for that group. It means the atoms aren’t arranged in a less-than-ideal way forced by their external conditions but because the quasicrystal structure in fact is better than a crystal structure. It answers “why do quasicrystals exist?” with “because they want to, not just because they can”. Entropy-stabilised goes the other way. That is: at 0 K (-273.15º C), the atoms would rather come together as a crystal because a crystal structure has lower energy at absolute zero. But as the temperature increases, the energy in the crystal builds up and forces the atoms to adjust where they’re sitting so that they can accommodate new forces. At some higher temperature, the structure becomes entropy-stabilised. That is, there’s enough disorder in the structure — like sound passing through the grid of atoms and atoms momentarily shifting their positions — that allows it to hold the ‘excess’ energy but at the same time deviate from the orderliness of a crystal structure. Entropy stabilisation answers “why do quasicrystals exist?” with “because they’re forced to, not because they want to”.

In materials science, the go-to tool to judge whether a crystal structure is energetically favourable is density functional theory (DFT). It estimates the total energy of a solid and from there scientists can compare competing phases and decide which one is most stable. If four atoms will have less energy arranged as a cuboid than as a pyramid at a certain temperature and pressure, then the cuboidal phase is said to be more favoured. The problem is DFT can’t be directly applied to quasicrystals because the technique assumes that a given mineral has a periodic internal structure. Quasicrystals are aperiodic. But because scientists are already comfortable with using DFT, they have tried to surmount this problem by considering a superunit cell that’s made up of a large number of atoms or by assuming that a quasicrystal’s structure, while being aperiodic in three dimensions, could be periodic in say four dimensions. But the resulting estimates of the solid’s energy have not been very good.

In the new Nature Physics paper, scientists from the University of Michigan, Ann Arbor, have reported a way around the no-unit-cell problem to apply DFT to estimate the energy of two quasicrystals. And they found that these quasicrystals are enthalpy-stabilised. The finding answer is a chemistry breakthrough because it raises the possibility of performing DFT in crystals without translational symmetry. Further, by showing that two real quasicrystals are enthalpy-stabilised, chemists may be forced to rethink why almost every other inorganic material does adopt a repeating structure. Crystals are no longer at the centre of the orderliness universe.

The team started by studying the internal structure of two quasicrystals using X-rays, then ‘scooped’ out five random parts for further analysis. Each of these scoops had 24 to 740 atoms. Second, the team used a modified version of DFT called DFT-FE. The computational cost of running DFT scales increases according to the cube of the number of atoms being studied. If studying four atoms with DFT requires X amount of computing power, 24 atoms would require 8,000 times X and 740 atoms would require 400 million times X. Instead the computational cost of DFT-FE scales as the square of the number of atoms, which makes a big difference. Continuing from the previous example, 24 atoms would require 400 times X and 740 atoms would require half a million times X. But the lower computational cost of DFT-FE is still considerable. The researchers’ solution was to use GPUs — the processors originally developed to run complicated video games and today used to run artificial intelligence (AI) apps like ChatGPT.

The team was able to calculate that the resulting energy estimates for a quasicrystal was off by no more than 0.3 milli-electron-volt (meV) per atom, considered acceptable. They also applied their technique to a known crystal, ScZn<sub>6</sub>, and confirmed that its estimate of the energy matched the known value (5-9 meV per atom). They were ready to go now.

When they applied DFT-FE to scandium-zinc and ytterbium-cadmium quasicrystals, they found clear evidence that they were enthalpy-stabilised. Each atom in the scandium-zinc quasicrystal had 23 meV less energy than if it had been part of a crystal structure. Similarly atoms in the ytterbium-cadmium quasicrystal had roughly 7 meV less each. The verdict was obvious: translational symmetry is not required for the most stable form of an inorganic solid.

The researchers also explored why the ytterbium-cadmium quasicrystal is so much easier to make than the scandium-zinc quasicrystal. In fact the former was the world’s first two-element quasicrystal to be discovered, 25 years ago this year. The team broke down the total energy as the energy in the bulk plus energy on the surface, and found that the scandium-zinc quasicrystal has high surface energy.

This is important because in thermodynamics, energy is like cost. If you’re hungry and go to a department store, you buy the pack of biscuits that you can afford rather than wait until you have enough money to buy the most expensive one. Similarly, when there’s a hot mass of scandium-zinc as a liquid and scientists are slowly cooling it, the atoms will form the first solid phase they can access rather than wait until they have accumulated enough surface energy to access the quasicrystal phase. And the first phase they can access will be crystalline. On the other hand scientists discovered the ytterbium-cadmium quasicrystal so quickly because it has a modest amount of energy across its surface and thus when cooled from liquid to solid, the first solid phase the atoms can access is also the quasicrystal phase.

This is an important discovery: the researchers found that a phase diagram alone can’t be used to say which phase will actually form. Understanding the surface-energy barrier is also important, and could pave the way to a practical roadmap for scientists trying to grow crystals for specific applications.

The big question now is: what special bonding or electronic effects allow atoms to have order without periodicity? After Israeli scientist Dan Shechtman discovered quasicrystals in 1982, he didn’t publish his findings until two years later, after including some authors on his submission to improve its chances with a journal, because he thought he wouldn’t be taken seriously. This wasn’t a silly concern: Linus Pauling, one of the greatest chemists in the history of subject, dismissed Shechtman’s work and called him a “quasi-scientist”. The blowback was so sharp and swift because chemists like Pauling, who had helped establish the science of crystal structures, were certain there was a way crystals could look and a way they couldn’t — and quasicrystals didn’t have the right look. But now, the new study has found that quasicrystals look perfect. Perhaps it’s crystals that need to explain themselves…

Paged Out! was always intended as a PDF+print zine, but the "print" part turned out to be pretty elusive. We actually did an initial test print of 500 copies in 2019 for a conference I've co-organized (Security PWNing), but that's it. Until last month that is, when we pretty much got back on track with prints — both free prints for events, and — additionally — print on demand if someone wants to buy a copy. We actually also updated the website with a lot of print-related information.

So let's cut to the chase — how to get printed Paged Out!?

• You can buy it in the first print-on-demand online bookstore we onboarded: lulu.com/spotlight/pagedout
  Note: there's a normal edition, and there are sponsorship editions — there's no difference between them apart from the price and the back cover; get a sponsorship one if you want to show additional love for the zine :)
• You can collect one for free at one of these events: pagedout.institute/?page=event-prints.php
  Note: Revisit this list from time to time — it keeps growing.
• And finally, you can print it yourself: pagedout.institute/?page=personal-prints.php

At the same time if you or your company would like to sponsor some Paged Out! prints for a specific event or in general, please let us know (prints AT pagedout DOT institute).

So far only issue #6 is available, but we're working on getting all of them out there, including older ones. We're basically going one by one, first #5, then #4, and so on.

Speaking of issues — Call For Articles for issue #7 now has a soft deadline: 30 June 2025

As usual, we're accepting technical 1-page articles about everything interesting related to computers, electronics, radio, and so on. See pagedout.institute/?page=cfp.php for details.

Note: We're having problems getting articles about retro computers, speedrunning, and movement techniques in games (e.g. Apex Legends), so if you can write about that, please do; and if you know someone who could write something about this, please ping them. Of course all the usual topics are welcomed too, as always.

It's the 20-year anniversary of the CONFidence conference! And it's happening next week (2-3 June) in Kraków, so don't miss out.

Furthermore, we've shipped 500 Paged Out! #6 issues there, so – if you're fast enough – you can grab one for free there :)

Enjoy CONFidence!

P.S. If you don't have a ticket my code might still work: GYN10
P.P.S. Huh, time flies, doesn't it. I think the first CONFidence I attended was in 2008. It's great to see this conference is still going strong and amazing to be a part of its program committee.

I get questions from people who read my posts and sometimes I answer them in a post. This is one of those times.

...

Someone read my "war room" post and picked up on the part where I spent several weeks trying to get to the bottom of what turned out to be "kill -9 -1" nuking the world on a bunch of FB machines. They asked how I keep track of things during that time, what my working memory is, and is it paper, text files, IRC messages, or just remembering things.

The answer is: back when I used to do that kind of thing, I found it very useful to have a "MMDD" (hey, I'm in the US, so just pretend it's the back half of an 8601 number... you'll see why...) directory, and inside of it, I'd have some short names for whatever I was dealing with that day.

That means today would be 0629, and then something inside of it might be "fbar" or "rsw" or "webi" or something. It was just something to keep all of the crap together and yet away from other things I was doing that day, while also distinguishing it from other times I might've done a "fbar" or "webi" or whatever project, if that makes sense.

These would just live off my home directory, and yes, it would fill up with crap, but I'd batch them up when a year ended. I'd take 01xx through 12xx and move them into "2013" when 2014 began, for instance. So, as you can see, they *are* ISO-8601 dates, sorta, but it's ~/YYYY/MMDD/foobar once it's old, and it's merely ~/MMDD/foobar until then. This is a balancing act between speed and having my homedir fill up with tons of ancient crap.

Any time I needed scratch space for output from things, that's what I used. That might be the output from a bunch of sweeps over the fleet to look for anomalies. Let's say I ran a command that sshed into a few hundred thousand hosts to look for common items in the logs. The stdout/stderr from that would probably be there so I could hit it up multiple times without asking the job runner system for another copy. It's a lot faster that way.

But in terms of troubleshooting things and dealing with permutations, it's hard to beat paper, and I usually end up with some kind of "lab notebook" at any job I work. I can think of examples of those going back over 20 years. The only problem is that the stuff in them really properly belongs to the company so I don't tend to have them after the fact. A great many pages of context have gone in the shredders over time, and not because I didn't ask. I've asked if anyone wanted them, and the answer has always been no.

There were also the internal posts about things, and then running commentary on IRC channels, but those tend to be useful after the fact, or for getting help from other people. My own state-keeping tends to stay on something close at hand and (usually) physically tangible.

As for those <date>/<term> directories, some of them turned out to be rather handy after the fact. A lot of times, something would happen, and then time would pass, and it would break *again* in exactly the same way, and I could think "didn't we deal with these people already?", dig around, find something from six months earlier, and go "AHA!". Now armed with the date, I could pull up the right posts, IRC logs, group messages, graphs, or whatever else.

Considering how much random crap I dealt with in any given day during my time "in the barrel" (kids, ask your parents), that was the only way to make any sense of it later. Without stuff like that, by the end of the week, I'd have no idea what I had been doing on Monday or Tuesday. That's how bad the load got at points.

...

Someone else wrote in and asked if I could do something to improve the overflow calculator thing I hacked up last week. It wasn't particularly usable on mobile, and that's definitely true. I wrote it on a laptop and paid exactly zero attention to what it would look like on a weirdly small screen that's usually taller than it is wide.

It was a fair point, so I took a whack at making it suck slightly less. It's probably still terrible, but in the specific case of me holding my phone upright, it looks halfway usable now. You're no longer forced into "flyspeck-3" mode with the fonts, at least.

CSS is such a mess.

...

I occasionally hear that the site is unreachable from one spot or another. This almost certainly comes down to IP filtering on my part. Perhaps you've read about the influx of web weasels who are scraping the living shit out of everything remotely URL-shaped they can get their hands on. My stuff is certainly in that space, and they show up here regularly.

Besides that, there are also a number of networks which send nothing but straight-up abusive traffic. This is where you look at the logs and see stuff like them using every single IP address in a (v4) /24 to scan for random webshit vulnerabilities. It's like, nice job, fucko, but I don't run PHP here. Anyway, that's pretty solid evidence that a given network is not worth hearing from ever again, and so into the filter it goes.

A whole lot of this happens automatically just based on whatever traffic is sent this way first. Send bad traffic and meet the bit bucket. I don't even find about most of it since it's constant and utterly uninteresting.

Then there are the people who run feed readers which don't play nicely. As previously described, they will get a handful of chances to slow down with 429s, and then if the web server feels like those aren't having an effect, it'll just ignore the traffic for however long it feels like. Again, I'm also not in the loop on this sort of thing. It's all automatic.

Finally, there is a new rub. I taught myself how to ingest BGP data, and so can now trivially go from an IP address to the autonomous system number of whoever's advertising it, including overlapping advertisements (like a /24 out of a bigger /20). Then I wrote something that will dump out an entire AS, and it's not hard to imagine what I do with that.

Enough bad behavior from a host -> filter the host.

Enough bad hosts in a netblock -> filter the netblock.

Enough bad netblocks in an AS -> filter the AS. Think of it as an "AS death penalty", if you like.

As long as clueless network operators continue to let abusive customers bounce around thousands of dynamic IP addresses with not so much as a hint of SWIP data, those entire net blocks will find themselves unable to get to large swaths of the net. That's just how it is, and it's nothing new. Send crap, meet /dev/null.

Dealing with what the web has become is exhausting.

I've long had a list of "magic numbers" which show up in a bunch of places, and even made a post about it back in November of 2020. You ever wonder about certain permutations, like 497 days, or 19.6 years, or 5184 hours, and what they actually mean?

I've been doing that stuff by hand in a calculator and finally decided to just do it in Javascript and put it online for anyone to try.

So, here's my latest waste of CPU cycles:

My rollover calculator.

I still haven't figured out the Crucial SSD 5184 hour thing, so it's not in there. 5124 hours, sure, I can understand that one, but 5184? 60 more?

Anyway, have fun while the world burns.

rsync is one of those tools which is rather useful. It saves you from spending the time and effort on copying data which you already have. It's the backbone of many a mirror site, and it also gets used for any number of backup solutions.

There's just one problem: in the name of efficiency, it can miss certain changes. rsync normally looks at the size and modification time of a candidate file, and if they are the same at both ends, that's the end of any consideration. It won't get any further attention and it moves on to something else.

"So what", you might think. "All files change at least their mtime when someone writes to them. That's the whole point of a mtime."

And yet... I'm writing this post, and here we are.

The keen-eyed observers out there are probably already thinking "ooh, bit rot" and other things where one of the files has actually become corrupted while "at rest" for whatever reason. Those observers are right! That's totally a problem that you have to worry about, especially if you're using SSDs to hold your bits and those SSDs aren't always being powered.

But no, this is something you have to worry about *beyond* that. This is about a "sneak path" that you probably didn't consider. I didn't.

Here, let's run a little experiment. If you have a x86_64 Debian box that's relatively current and you've been backing up the whole thing via rsync for a year or two, go do something for me.

Go run your favorite file-hasher tool on /usr/lib/x86_64-linux-gnu/libfribidi.so.0.4.0 for me. Give it a sha256sum or whatever, or even md5sum if you're feeling brash. Then note the modification time on the file.

Now mount one of your backups and do the same thing on the version of the file that's on the backup device. See anything ... odd? Unusual?

Identical mtimes, identical sizes... and different hashes, right? I spotted this on a bunch of my machines after going "hmmm..." about the whole SSD-data-loss thing.

Clearly, something unusual happened somewhere, and it's been escaping the notice of your rsync runs ever since. I haven't gone digging into the package history for this thing to find out just when and where it happened, and (more importantly) how. It's rather unusual.

If you're freaking out right now, there is some hope. rsync has both -I and -c which promise to not use the quick method and instead will run a checksum on the files. It's slower so you won't want to do this normally, but it's not a bad idea to add this to the mix of things that you do every so many rotations.

I should point out that the first time you do a forced-checksum run, --dry-run will let you see the changes before it blows anything away, so you can make the call as to which version is the right one! In theory, your *source* files can get corrupted, and if you just copy one of those across, you have now corrupted your backup.

Isn't entropy FUN?

At the start of 2023, I wrote a post talking about why I still had an "old-school cert" on my https site. Well, things have shifted, and it's time to talk about why.

I've been aware of the ACME protocol for a while. I have tech notes going back as far as 2018, and every time I looked at it, I recoiled in horror. The whole thing amounts to "throw in every little bit of webshit tech that we can", and it makes for a real problem to try to implement this in a safe and thorough way.

Many of the existing clients are also scary code, and I was not about to run any of them on my machines. They haven't earned the right to run with privileges for my private keys and/or ability to frob the web server (as root!) with their careless ways.

That meant I was stuck: unwilling to bring myself to deal with the protocol while simultaneously unwilling to budge on allowing the cruft code of existing projects into my life.

Well, time passed, and I managed to crack some of my own barriers. It wasn't by using the other projects, though. I started ripping into them to figure out just how the spec really worked, and started biting off really really small pieces of the problem. It took a particular forcing function to get me off my butt and into motion.

About six months ago, I realized that it was probably time to get away from Gandi as a registrar and also SSL provider (reseller). They had been eaten by private equity some years before, and the rot has been setting in. Their "no bullshit" tagline is gone, and their prices have been creeping up. I happened to renew my domains for multiple years and have been insulated for a while, but it was going to be a problem in 2025.

Giving them the "yeet" was no big deal, but the damn rbtb certificate was going to be a problem. Was I going to start paying even more for the stupid thing every year, or was I going to finally suck it up and deal with ACME?

That still left the problem of overcoming my inherent disgust for the entire protocol and having to deal with all of these encodings they force upon you. My first steps towards the solution involved writing really small and stupid utility functions and libraries that would come in handy later. I'm talking about wrapping jansson (a C library that handles JSON) so that it made sense in my C++ world and I could import JSON (something I use as little as possible). That kind of thing.

This also meant going down some dead-ends, like noticing that libraries existed which would allegedly create certain things (like a JWK) for you, and then realizing that they were not going to make my life any easier. I'd poke at it, reach my limit, and then swear and walk away for another couple of days.

This went on for some time. I have a series of notes where I'd grab a piece of the problem, wrangle it around for a while, get grossed out, and then set it down and go do something else. This just kept happening but I slowly made progress with small pieces that would Do Stuff, and then they'd connect to each other, and so on like this.

One positive development during all of this was discovering this "pebble" test server I could run on an isolated fake system. It would act as an ACME server and would let me harass it with my feeble attempts at implementing a client instead of bothering the real CAs. Even "staging" servers deserve better treatment than active development, after all.

And, well, after a whole lot of mangling and dead-ends and rewrites and other terrible crap, I had an awful little tool that would take a CSR, do all of the idiot dances and would plop out a certificate. I pointed it at Let's Encrypt staging, and it worked. Then I pointed it at their prod site, and _that_ worked. So I did it for the real thing ([www.]rachelbythebay.com), and *that* worked, and I dropped it into place.

Thus, for the past couple of weeks, if you've been hitting the https version of my site, you've been doing it across the new setup.

Now, I took notes about this, and I wanted to share some of my original off-the-cuff thoughts about implementing this for anyone who's similarly broken in the head and wants to see how bad it can be. I will note that I wrote this based on the first thing that worked, and it does not necessarily reflect the implementation I'm on a few weeks later.

...

Make an RSA key for your web site. Then make a CSR for it, setting the CN and adding a matching altname as an extension. No other fields matter. Nobody looks at those, anyway, and none of them will influence your final certificate no matter how prosaic or precise you get in there.

Make an RSA key of 4096 bits. Call it your personal key.

Write something that'll read a CSR. It needs to extract the CN and the SANs - the DNS: ones, at least. Ensure there's actually a CN [*] and actually SANs, and that the CN occurs within the SANs. So, yes, you have to have at least one SAN.

[* - I now know you can run with just SANs. I did not at the time.]

Write something that'll do a HTTP GET to <directory URL> which is given to you by the ACME service operator. It then needs to parse the body as JSON (or die) and extract some strings from the top-level object: "newNonce", "newAccount" and "newOrder" at the very least.

Write something that'll read an RSA key file on disk. It needs to extract the publicExponent (probably 65537, but you never know...) and the modulus. Make it read your personal key from earlier.

If you end up using "openssl rsa -in foo -noout -text" to do this, the modulus is a bunch of printed hex digits, like "00:ff:11:ab:cd:ef:22:33". It hard-wraps the output and also indents the lines, so you get to clean all of that up first.

Skip the first 00 for some inexplicable reason. Take the other bytes of the modulus and turn them into the actual character values, so a literal 0xff, 0x11, 0xab, 0xcd and so on down the line in the same order you find them in the file. Hang on to these values for later.

Write something that'll turn an ordinary integer into its equivalent big-endian bag of bytes, but don't pad it out to any particular alignment. You need to take that "65537" from your publicExponent and turn it into the equivalent bytes, so 0x01, 0x00, 0x01. Yes, it's a number you just turned back into a binary representation.

Write something that will do base64 *style* encoding, but not quite. The last two characters in the encoding set are usually + and /, but that won't do for webshit, so you need to make it use - and _ instead.

Take that publicExponent (65537), pump it through your big-endian bag-of-bytes encoder to get 0x01 0x00 0x01, then put it through your "base64web" encoder to get "AQAB".

Start a new JSON object. Add a string called "e" and set it to the output of the above step. So, yes, instead of saying that "e" equals "65537", you're saying that "e" equals "AQAB". Aren't you glad you did those extra steps?

Add another string called "kty" and set it to "RSA".

Add another string called "n" and set it to the "base64web" version of your modulus bag-of-bytes from earlier.

Take this JSON object and make it into a sorted compact string representation. This means it goes "e, kty, n" and it also has all of the usual padding (spaces) squished out. Call this a JWK string and save it for later.

Create a second JSON object. Add a boolean to it named "termsOfServiceAgreed" that's set to true. (Guess you'd better agree...)

Look up the URL for "newNonce" from the directory JSON you got earlier.

Make a HTTP HEAD request to that URL. Dig around in the headers (not the body, since there is no body on a HEAD) until you find "Replay-Nonce". Extract the value of that header. Hang onto it for later.

Look up the URL for "newAccount" from that directory JSON for before.

Create a third JSON object. Add a string to it called "url". Set it to that (newAccount) URL. Add a string called "alg". Set it to "RS256". Add a string called "nonce" and set it to the value from the *header* in that last HTTP HEAD request.

Add an object to this third object called "jwk". Within it, add "e", "kty" and "n" in a manner that matches what you did earlier (you know, from the "JWK string" you're still holding for later).

Dump this third JSON object to a sorted compact string representation. Call it "protected".

Dump the second JSON object to a sorted compact string representation. Call it "payload".

Create a string where you literally concatenate those two prior strings, such that it's the value of protected, then an actual period (as in 0x2e, a full stop, whatever), then the value of payload.

Write something that'll create a SHA256 digest of an arbitrary string and will sign it with an arbitrary RSA key (like "openssl dgst -sha256 -sign <key>"). The key in question is your personal key.

Pump that "<protected>.<payload>" string through the digest function. Then run it through your "base64web" encoder. Call this the signature.

Create a fourth JSON object. Add a string called "protected" and set it to whatever you built a few steps earlier. Add another string called "payload" and set it likewise. Then add one called "signature" and set it, too.

Take this fourth JSON object and dump it to a sorted compact string representation. Call this your post body.

Make a HTTP POST request to the "newAccount" URL from the directory. Set the content-type to "application/jose+json". Set the post data to the post body string from the previous step.

Dig around in the headers of the response, looking for one named "Location". Don't follow it like a redirection. Why would you ever follow a Location header in a HTTP header, right? Nope, that's your user account's identifier! Yes, you are a URL now. Hang on to that URL for later.

You now have an account.

You still have much to do.

...

That's about where I stopped writing my take on the protocol.

Again, my program no longer works quite like this, but this is where it started after having observed a bunch of other stuff that already existed.

So far, we have (at least): RSA keys, SHA256 digests, RSA signing, base64 but not really base64, string concatenation, JSON inside JSON, Location headers used as identities instead of a target with a 301 response, HEAD requests to get a single value buried as a header, making one request (nonce) to make ANY OTHER request, and there's more to come.

We haven't even scratched the surface of creating an order, dealing with authorizations and challenges, the whole "key thumbprint" thing, what actually goes into those TXT records, and all of that other fun stuff.

...

Random side note: while looking at existing ACME clients, I found that at least one of them screws up their encoding of the publicExponent and ends up interpreting it as hex instead of decimal. That is, instead of 65537, aka 0x10001, it reads it as 0x65537, aka 415031!

Somehow, this anomaly exists and apparently doesn't break anything? I haven't actually run the client in question, but I imagine people are using it since it's in apt.

...

This complexity must be job security for somebody. Maybe multiple somebodies.

During the Apollo flights to the Moon, the astronauts observed the spacecraft's orientation on a special instrument called the FDAI (Flight Director / Attitude Indicator). This instrument showed the spacecraft's attitude—its orientation—by rotating a ball. This ball was nicknamed the "8-ball" because it was black (albeit only on one side). The instrument also acted as a flight director, using three yellow needles to indicate how the astronauts should maneuver the spacecraft. Three more pointers showed how fast the spacecraft was rotating.

Since the spacecraft rotates along three axes (roll, pitch, and yaw), the ball also rotates along three axes. It's not obvious how the ball can rotate to an arbitrary orientation while remaining attached. In this article, I look inside an FDAI from Apollo that was repurposed for a Space Shuttle simulator1 and explain how it operates. (Spoiler: the ball mechanism is firmly attached at the "equator" and rotates in two axes. What you see is two hollow shells around the ball mechanism that spin around the third axis.)

The FDAI in Apollo

For the missions to the Moon, the Lunar Module had two FDAIs, as shown below: one on the left for the Commander (Neil Armstrong in Apollo 11) and one on the right for the Lunar Module Pilot (Buzz Aldrin in Apollo 11). With their size and central positions, the FDAIs dominate the instrument panel, a sign of their importance. (The Command Module for Apollo also had two FDAIs, but with a different design; I won't discuss them here.2)

Each Lunar Module FDAI could display inputs from multiple sources, selected by switches on the panel.3 The ball could display attitude from either the Inertial Measurement Unit or from the backup Abort Guidance System, selected by the "ATTITUDE MON" toggle switch next to either FDAI. The pitch attitude could also be supplied by an electromechanical unit called ORDEAL (Orbital Rate Display Earth And Lunar) that simulates a circular orbit. The error indications came from the Apollo Guidance Computer, the Abort Guidance System, the landing radar, or the rendezvous radar (controlled by the "RATE/ERROR MON" switches). The pitch, roll, and yaw rate displays were driven by the Rate Gyro Assembly (RGA). The rate indications were scaled by a switch below the FDAI, selecting 25°/sec or 5°/sec.

The FDAI mechanism

The ball inside the indicator shows rotation around three axes. I'll first explain these axes in the context of an aircraft, since the axes of a spacecraft are more arbitrary.4 The roll axis indicates the aircraft's angle if it rolls side-to-side along its axis of flight, raising one wing and lowering the other. Thus, the indicator shows the tilt of the horizon as the aircraft rolls. The pitch axis indicates the aircraft's angle if it pitches up or down, with the indicator showing the horizon moving down or up in response. Finally, the yaw axis indicates the compass direction that the aircraft is heading, changing as the aircraft turns left or right. (A typical aircraft attitude indicator omits yaw.)

I'll illustrate how the FDAI rotates the ball in three axes, using an orange as an example. Imagine pinching the horizontal axis between two fingers with your arm extended. Rotating your arm will roll the ball counter-clockwise or clockwise (red arrow). In the FDAI, this rotation is accomplished by a motor turning the frame that holds the ball. For pitch, the ball rotates forward or backward around the horizontal axis (yellow arrow). The FDAI has a motor inside the ball to produce this rotation. Yaw is a bit more difficult to envision: imagine hemisphere-shaped shells attached to the top and bottom shafts. When a motor rotates these shells (green arrow), the hemispheres will rotate, even though the ball mechanism (the orange) remains stationary.

The diagram below shows the mechanism inside the FDAI. The indicator uses three motors to move the ball. The roll motor is attached to the FDAI's frame, while the pitch and yaw motors are inside the ball. The roll motor rotates the roll gimbal through gears, causing the ball to rotate clockwise or counterclockwise. The roll gimbal is attached to the ball mechanism at two points along the "equator"; these two points define the pitch axis. Numerous wires on the roll gimbal enter the ball along the pitch axis. The roll control transformer provides position feedback, as will be explained below.

Removing the hemispherical shells reveals the mechanism inside the ball. When the roll gimbal is rotated, this mechanism rotates with it. The pitch motor causes the ball mechanism to rotate around the pitch axis. The yaw motor and control transformer are not visible in this photo; they are behind the pitch components, oriented perpendicularly. The yaw motor turns the vertical shaft, with the two hemisphere shells attached to the top and bottom of the shaft. Thus, the yaw motor rotates the ball shells around the yaw axis, while the mechanism itself remains stationary. The control transformers for pitch and yaw provide position feedback.

Why doesn't the wiring get tangled up as the ball rotates? The solution is two sets of slip rings to implement the electrical connections. The photo below shows the first slip ring assembly, which handles rotation around the roll axis. These slip rings connect the stationary part of the FDAI to the rotating roll gimbal. The vertical metal brushes are stationary; there are 23 pairs of brushes, one for each connection to the ball mechanism. Each pair of brushes contacts one metal ring on the striped shaft, maintaining contact as the shaft rotates. Inside the shaft, 23 wires connect the circular metal contacts to the roll gimbal.

A second set of slip rings inside the ball handles rotation around the pitch axis. These rings provide the electrical connection between the wiring on the roll gimbal and the ball mechanism. The yaw axis does not use slip rings since only the hemisphere shells rotate around the yaw axis; no wires are involved.

Synchros and the servo loop

In this section, I'll explain how the FDAI is controlled by synchros and servo loops. In the 1950s and 1960s, the standard technique for transmitting a rotational signal electrically was through a synchro. Synchros were used for everything from rotating an instrument indicator in avionics to rotating the gun on a navy battleship. A synchro produces an output that depends on the shaft's rotational position, and transmits this output signal on three wires. If you connect these wires to a second synchro, you can use the first synchro to control the second one: the shaft of the second synchro will rotate to the same angle as the first shaft. Thus, synchros are a convenient way to send a control signal electrically.

The photo below shows a typical synchro, with the input shaft on the top and five wires at the bottom: two for power and three for the output.

Internally, the synchro has a rotating winding called the rotor that is driven with 400 Hz AC. Three fixed stator windings provide the three AC output signals. As the shaft rotates, the voltages of the output signals change, indicating the angle. (A synchro resembles a transformer with three variable secondary windings.) If two connected synchros have different angles, the magnetic fields create a torque that rotates the shafts into alignment.

The downside of synchros is that they don't produce a lot of torque. The solution is to use a more powerful motor, controlled by the synchro and a feedback loop called a servo loop. The servo loop drives the motor in the appropriate direction to eliminate the error between the desired position and the current position.

The diagram below shows how the servo loop is constructed from a combination of electronics and mechanical components. The goal is to rotate the output shaft to an angle that exactly matches the input angle, specified by the three synchro wires. The control transformer compares the input angle and the output shaft position, producing an error signal. The amplifier uses this error signal to drive the motor in the appropriate direction until the error signal drops to zero. To improve the dynamic response of the servo loop, the tachometer signal is used as a negative feedback voltage. The feedback slows the motor as the system gets closer to the right position, so the motor doesn't overshoot the position and oscillate. (This is sort of like a PID controller.)

A control transformer is similar to a synchro in appearance and construction, but the rotating shaft operates as an input, not the output. In a control transformer, the three stator windings receive the inputs and the rotor winding provides the error output. If the rotor angle of the synchro transmitter and control transformer are the same, the signals cancel out and there is no error voltage. But as the difference between the two shaft angles increases, the rotor winding produces an error signal. The phase of the error signal indicates the direction of the error.

In the FDAI, the motor is a special motor/tachometer, a device that was often used in avionics servo loops. This motor is more complicated than a regular electric motor. The motor is powered by 115 volts AC at 400 hertz, but this won't spin the motor on its own. The motor also has two low-voltage control windings. Energizing the control windings with the proper phase causes the motor to spin in one direction or the other. The motor/tachometer unit also contains a tachometer to measure its speed for the feedback loop. The tachometer is driven by another 115-volt AC winding and generates a low-voltage AC signal that is proportional to the motor's rotational speed.

The photo above shows a motor/tachometer with the rotor removed. The unit has many wires because of its multiple windings. The rotor has two drums. The drum on the left, with the spiral stripes, is for the motor. This drum is a "squirrel-cage rotor", which spins due to induced currents. (There are no electrical connections to the rotor; the drums interact with the windings through magnetic fields.) The drum on the right is the tachometer rotor; it induces a signal in the output winding proportional to the speed due to eddy currents. The tachometer signal is at 400 Hz like the driving signal, either in phase or 180º out of phase, depending on the direction of rotation. For more information on how a motor/tachometer works, see my teardown.

The amplifiers

The FDAI has three servo loops—one for each axis—and each servo loop has a separate control transformer, motor, and amplifier. The photo below shows one of the three amplifier boards. The construction is unusual and somewhat chaotic, with some components stacked on top of others to save space. Some of the component leads are long and protected with clear plastic sleeves.5 The cylindrical pulse transformer in the middle has five colorful wires coming out of it. At the left are the two transistors that drive the motor's control windings, with two capacitors between them. The transistors are mounted on a heat sink that is screwed down to the case of the amplifier assembly for cooling. Each amplifier is connected to the FDAI through seven wires with pins that plug into the sockets on the right of the board.6

The function of the board is to amplify the error signal so the motor rotates in the appropriate direction. The amplifier also uses the tachometer output from the motor unit to slow the motor as the error signal decreases, preventing overshoot. The inputs to the amplifier are 400 hertz AC signals, with the magnitude indicating the amount of error or speed and the phase indicating the direction. The two outputs from the amplifier drive the two control windings of the motor, determining which direction the motor rotates.

The schematic for the amplifier board is below. 7 The two transistors on the left amplify the error and tachometer signals, driving the pulse transformer. The outputs of the pulse transformer will have opposite phases, driving the output transistors for opposite halves of the 400 Hz cycle. This activates the motor control winding, causing the motor to spin in the desired direction.8

History of the FDAI

Bill Lear, born in 1902, was a prolific inventor with over 150 patents, creating everything from the 8-track tape to the Learjet, the iconic private plane of the 1960s. He created multiple companies in the 1920s as well as inventing one of the first car radios for Motorola before starting Lear Avionics, a company that specialized in aerospace instruments.9 Lear produced innovative aircraft instruments and flight control systems such as the F-5 automatic pilot, which received a trophy as the "greatest aviation achievement in America" for 1950.

Bill Lear went on to solve an indicator problem for the Air Force: the supersonic F-102 Delta Dagger interceptor (1953) could climb at steep angles, but existing attitude indicators could not handle nearly vertical flight. Lear developed a remote two-gyro platform that drove the cockpit indicator while avoiding "gimbal lock" during vertical flight. For the experimental X-15 rocket-powered aircraft (1959), Lear improved this indicator to handle three axes: roll, pitch, and yaw.

Meanwhile, the Siegler Corporation started in 1950 to manufacture space heaters for homes. A few years later, Siegler was acquired by John Brooks, an entrepreneur who was enthusiastic about acquisitions. In 1961, Lear Avionics became his latest acquisition, and the merged company was called Lear Siegler Incorporated, often known as LSI. (Older programmers may know Lear Siegler through the ADM-3A, an inexpensive video display terminal from 1976 that housed the display and keyboard in a stylish white case.)

The X-15's attitude indicator became the basis of the indicator for the F-4 fighter plane (the ARU/11-A). Then, after "a minimum of modification", the attitude-director indicator was used in the Gemini space program. In total, Lear Siegler provided 11 instruments in the Gemini instrument panel, with the attitude director the most important. Next, Gemini's indicator was modified to become the FDAI (flight director-attitude indicator) in the Lunar Module for Apollo.10 Lear Siegler provided numerous components for the Apollo program, from a directional gyro for the Lunar Rover to the electroluminescent display for the Apollo Guidance Computer's Display/Keyboard (DSKY).

In 1974, Lear Siegler obtained a contract to develop the Attitude-Director Indicator (ADI) for the Space Shuttle, producing a dozen ADI units for the Space Shuttle. However, by this time, Lear Siegler was losing enthusiasm for low-volume space avionics. The Instrument Division president said that "the business that we were in was an engineering business and engineers love a challenge." However, manufacturing refused to deal with the special procedures required for space manufacturing, so the Shuttle units were built by the engineering department. Lear Siegler didn't bid on later Space Shuttle avionics and the Shuttle ADI became its last space product. In the early 2000s, the Space Shuttle's instruments were upgraded to a "glass cockpit" with 11 flat-panel displays known as the Multi-function Electronic Display System (MEDS). The MEDS was produced by Lear Siegler's long-term competitor, Honeywell.

Getting back to Bill Lear, he wanted to manufacture aircraft, not just aircraft instruments, so he created the Learjet, the first mass-produced business jet. The first Learjet flew in 1963, with over 3000 eventually delivered. In the early 1970s, Lear designed a steam turbine automobile engine. Rather than water, the turbine used a secret fluorinated hydrocarbon called "Learium". Lear had visions of thousands of low-pollution "Learmobiles", but the engine failed to catch on. Lear had been on the verge of bankruptcy in the 1960s; one of his VPs explained that "the great creative minds can't be bothered with withholding taxes and investment credits and all this crap". But by the time of his death in 1978, Lear had a fortune estimated at $75 million.

Comparing the ARU/11-A and the FDAI

Looking inside our FDAI sheds more details on the evolution of Lear Siegler's attitude directors. The photo below compares the Apollo FDAI (top) to the earlier ARU/11-A used in the F-4 aircraft (bottom). While the basic mechanism and the electronic amplifiers are the same between the two indicators, there are also substantial changes.

The biggest difference between the ARU-11/A indicator and the FDAI is that the electronics for the ARU-11/A are in a separate module that was plugged into the back of the indicator, while the FDAI includes the electronics internally, with boards mounted on the instrument frame. Specifically, the ARU-11/A has a separate unit containing a multi-winding transformer, a power supply board, and three amplifier boards (one for each axis), while the FDAI contains these components internally. The amplifier boards in the ARU-11/A and the FDAI are identical, constructed from germanium transistors rather than silicon.11 The unusual 11-pin transformers are also the same. However, the power supply boards are different, probably because the boards also contain scaling resistors that vary between the units.12 The power supply boards are also different shapes to fit the available space.

The ball assemblies of the ARU/11-A and the FDAI are almost the same, with the same motor assemblies and slip ring mechanism. The gearing has minor changes. In particular, the FDAI has two plastic gears, while the ARU/11-A uses exclusively metal gears.

The ARU/11-A has a patented pitch trim feature that was mostly—but not entirely—removed from the Apollo FDAI. The motivation for this feature is that an aircraft in level flight will be pitched up a few degrees, the "angle of attack". It is desirable for the attitude indicator to show the aircraft as horizontal, so a pitch trim knob allows the angle of attack to be canceled out on the display. The problem is that if you fly your fighter plane vertically, you want the indicator to show precisely vertical flight, rather than applying the pitch trim adjustment. The solution in the ARU-11/A is a special 8-zone potentiometer on the pitch axis that will apply the pitch trim adjustment in level flight but not in vertical flight, while providing a smooth transition between the regions. This special potentiometer is mounted inside the ball of the ARU-11/A. However, this pitch trim adjustment is meaningless for a spacecraft, so it is not implemented in the Apollo or Space Shuttle instruments. Surprisingly, the shell of the potentiometer still exists in our FDAI, but without the potentiometer itself or the wiring. Perhaps it remained to preserve the balance of the ball. In the photo below, the cylindrical potentiometer shell is indicated by an arrow. Note the holes in the front of the shell; in the ARU-11/A, the potentiometer's wiring terminals protrude through these holes, but in the FDAI, the holes are covered with tape.

Finally, the mounting of the ball hemispheres is slightly different. The ARU/11-A uses four screws at the pole of each hemisphere. Our FDAI, however, uses a single screw at each pole; the screw is tightened with a Bristol Key, causing the shaft to expand and hold the hemisphere in place.

To summarize, the Apollo FDAI occupies a middle ground: while it isn't simply a repurposed ARU-11/A, neither is it a complete redesign. Instead, it preserves the old design where possible, while stripping out undesired features such as pitch trim. The separate amplifier and mechanical units of the ARU/11-A were combined to form the larger FDAI.

Differences from Apollo

The FDAI that we examined is a special unit: it was originally built for Apollo but was repurposed for a Space Shuttle simulator. Our FDAI is labeled Model 4068F, which is a Lunar Module part number. Moreover, the FDAI is internally stamped with the date "Apr. 22 1968", over a year before the first Moon landing.

However, a closer look shows that several key components were modified to make the Apollo FDAI work in the Shuttle Simulator.14 The Apollo FDAI (and the Shuttle ADI) used resolvers as inputs to control the ball, while our FDAI uses synchros. (Resolvers and synchros are similar, except resolvers use sine and cosine inputs, 90° apart, on two wire pairs, while synchros use three inputs, 120° apart, on three wires.) NASA must have replaced the three resolver control transformers in the FDAI with synchro control transformers for use in the simulator.

The Apollo FDAI used electroluminescent lighting for the display, while ours uses eight small incandescent bulbs. The metal case of our FDAI has a Dymo embossed tape label "INCANDESCENT LIGHTING", alerting users to the change from Apollo's illumination. Our FDAI also contains a step-down transformer to convert the 115 VAC input into 5 VAC to power the bulbs, while the Shuttle powered its ADI illumination directly from 5 volts.

The dial of our FDAI was repainted to match the dial of the Shuttle FDAI. The Apollo FDAI had red bands on the left and right of the dial. A close examination of our dial shows that black paint was carefully applied over the red paint, but a few specks of red paint are still visible (below). Moreover, the edges of the lines and the lozenge show slight unevenness from the repainting. Second, the Apollo FDAI had the text "ROLL RATE", "PITCH RATE", and "YAW RATE" in white next to the needle scales. In our FDAI, this text has been hidden by black paint to match the Shuttle display.13 Third, the Apollo LM FDAI had a crosshair in the center of the instrument, while our FDAI has a white U-shaped indicator, the same as the Shuttle (and the Command Module's FDAI). Finally, the ball of the Apollo FDAI has red circular regions at the poles to warn of orientations that can cause gimbal lock. Our FDAI (like the Shuttle) does not have these circles. We couldn't see any evidence that these regions were repainted, so we suspect that our FDAI has Shuttle hemispheres on the ball.

Our FDAI has also been modified electrically. Small green connectors (Micro-D MDB1) have been added between the slip rings and the motors, as well as on the gimbal arm. We think these connectors were added post-Apollo, since they are attached somewhat sloppily with glue and don't look flight-worthy. Perhaps these connectors were added to make disassembly and modification easier. Moreover, our FDAI has an elapsed time indicator, also mounted with glue.

The back of our FDAI is completely different from Apollo. First, the connector's pinout is completely different. Second, each of the six indicator needles has a mechanical adjustment as well as a trimpot (details). Finally, each of the three axes has an adjustment potentiometer.

The Shuttle's ADI (Attitude Director Indicator)

Each Space Shuttle had three ADIs (Attitude Director Indicators), which were very similar to the Apollo FDAI, despite the name change. The photo below shows the two octagonal ADIs in the forward flight deck, one on the left in front of the Commander, and one on the right in front of the Pilot. The aft flight deck station had a third ADI.15

Our FDAI appears to have been significantly modified for use in the Shuttle simulator, as described above. However, it is much closer to the Apollo FDAI than the ADI used in the Shuttle, as I'll show in this section. My hypothesis is that the simulator was built before the Shuttle's ADI was created, so the Apollo FDAI was pressed into service.

The Shuttle's ADI was much more complicated electrically than the Apollo FDAI and our FDAI, providing improved functionality.16 For instance, while the Apollo FDAI had a simple "OFF" indicator flag to show that the indicator had lost power, the Shuttle's ADI had extensive error detection. It contained voltage level monitors to check its five power supplies. (The Shuttle ADI used three DC power sources and two AC power sources, compared to the single AC supply for Apollo.) The Shuttle's ADI also monitored the ball servos to detect position errors. Finally, it received an external "Data OK" signal. If a fault was detected by any of these monitors, the "OFF" flag was deployed to indicate that the ADI could not be trusted.

The Shuttle's ADI had six needles, the same as Apollo, but the Shuttle used feedback to make the positions more accurate. Specifically, each Shuttle needle had a feedback sensor, a Linear Variable Differential Transformer (LVDT) that generates a voltage based on the needle position. The LVDT output drove a servo feedback loop to ensure that the needle was in the exact desired position. In the Apollo FDAI, on the other hand, the needle input voltage drove a galvanometer, swinging the needle proportionally, but there was no closed loop to ensure accuracy.

I assume that the Shuttle's ADI had integrated circuit electronics to implement this new functionality, considerably more modern than the germanium transistors in the Apollo FDAI. The Shuttle probably used the same mechanical structures to rotate the ball, but I can't confirm that.

Conclusions

The FDAI was a critical instrument in Apollo, indicating the orientation of the spacecraft in three axes. It wasn't obvious to me how the "8-ball" can rotate in three axes while still being securely connected to the instrument. The trick is that most of the mechanism rotates in two axes, while hollow hemispherical shells provide the third rotational axis.

The FDAI has an interesting evolutionary history, from the experimental X-15 rocket plane and the F-4 fighter to the Gemini, Apollo, and Space Shuttle flights. Our FDAI has an unusual position in this history: since it was modified from Apollo to function in a Space Shuttle simulator, it shows aspects of both Apollo and the Space Shuttle indicators. It would be interesting to compare the design of a Shuttle ADI to the Apollo FDAI, but I haven't been able to find interior photos of a Shuttle ADI (or of an unmodified Apollo FDAI).17

You can see a brief video of the FDAI in motion here. For more, follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS. (I've given up on Twitter.) I worked on this project with CuriousMarc, Mike Stewart, and Eric Schlapfer, so expect a video at some point. Thanks to Richard for providing the FDAI. I wrote about the F-4 fighter plane's attitude indicator here.

Notes and references

# Introduction

After thinking about BusKill product that triggers a command once the USB cord disconnects, I have been thinking at a simple alternative.

=> https://www.buskill.in BusKill official project website

When using a laptop connected to power most of the time, you may want it to power off once it gets disconnected, this can be really useful if you use it in a public area like a bar or a train.  The idea is to protect the laptop if it gets stolen while in use and unlocked.

Here is how to proceed on Linux, using a trigger on an udev rule looking for a change in the power_supply subsystem.

For OpenBSD users, it is possible to use apmd as I explained in this article:

=> https://dataswamp.org/~solene/2024-02-20-rarely-known-openbsd-features.html#_apmd_daemon_hooks => Rarely known OpenBSD features: apmd daemon hooks

In the example, the script will just power off the machine, it is up to you to do whatever you want like destroy the LUKS master key or trigger the coffee machine :D

# Setup

Create a file `/etc/udev/rules.d/disconnect.rules`, you can name it how you want as long as it ends with `.rules`:

```
SUBSYSTEM=="power_supply", ENV{POWER_SUPPLY_ONLINE}=="0", ENV{POWER_SUPPLY_TYPE}=="Mains", RUN+="/usr/local/bin/power_supply_off"
```

Create a file `/usr/local/bin/power_supply_off` that will be executed when you unplug the laptop:

```
#!/bin/sh
echo "Going off because power supply got disconnected" | systemd-cat
systemctl poweroff
```

This simple script will add an entry in journald before triggering the system shutdown.

Mark this script executable with:
```
chmod +x /usr/local/bin/power_supply_off
```

Reload udev rules using the following commands:

```
udevadm control --reload-rules
udevadm trigger
```

# Testing

If you unplug your laptop power, it should power off, you should find an entry in the logs.

If nothing happens, looks at systemd logs to see if something is wrong in udev, like a syntax error in the file you created or an incorrect path for the script.

# Script ideas

Depending on your needs, here is a list of actions the script could do, from gentle to hardcore:

* Lock user sessions
* Hibernate
* Proper shutdown
* Instant power off (through sysrq)
* Destroy LUKS master key to make LUKS volume unrecoverable + Instant power off

# Conclusion

While BusKill is an effective / unusual product that is certainly useful for a niche, protecting a running laptop against thieves is an extra layer when being outside.

Obviously, this use case works only when the laptop is connected to power.

Today, we’re looking at two case studies in how to respond when reactionaries appear in your free software community.

Exhibit A

It is a technical decision.

The technical reason is that the security team does not have the bandwidth to provide lifecycle maintenance for multiple X server implementations. Part of the reason for moving X from main to community was to reduce the burden on the security team for long-term maintenance of X. Additionally, nobody so far on the security team has expressed any interest in collaborating with xxxxxx on security concerns.

We have a working relationship with Freedesktop already, while we would have to start from the beginning with xxxxxx.

Why does nobody on the security team have any interest in collaboration with xxxxxx? Well, speaking for myself only here – when I looked at their official chat linked in their README, I was immediately greeted with alt-right propaganda rather than tactically useful information about xxxxxx development. At least for me, I don’t have any interest in filtering through hyperbolic political discussions to find out about CVEs and other relevant data for managing the security lifecycle of X.

Without relevant security data products from xxxxxx, as well as a professionally-behaving security contact, it is unlikely for xxxxxx to gain traction in any serious distribution, because X is literally one of the more complex stacks of software for a security team to manage already.

At the same time, I sympathize with the need to keep X alive and in good shape, and agree that there hasn’t been much movement from freedesktop in maintaining X in the past few years. There are many desktop environments which will never get ported to Wayland and we do need a viable solution to keep those desktop environments working.

I know the person who wrote this, and I know that she’s a smart cookie, and therefore I know that she probably understood at a glance that the community behind this “project” literally wants to lynch her. In response, she takes the high road, avoids confronting the truth directly, and gives the trolls a bunch of talking points to latch on for counter-arguments. Leaves plenty of room for them to bog everyone down in concern trolling and provides ample material to fuel their attention-driven hate machine.

There’s room for improvement here.

Exhibit B

Concise, speaks the truth, answers ridiculous proposals with ridicule, does not afford the aforementioned reactionary dipshits an opportunity to propose a counter-argument. A+.

Extra credit for the follow-up:

The requirement for a passing grade in this class is a polite but summary dismissal, but additional credit is awarded for anyone who does not indulge far-right agitators as if they were equal partners in maintaining a sense of professional decorum.

If you are a community leader in FOSS, you are not obligated to waste your time coming up with a long-winded technical answer to keep nazis out of your community. They want you to argue with them and give them attention and feed them material for their reactionary blog or whatever. Don’t fall into their trap. Do not answer bad faith with good faith. This is a skill you need to learn in order to be an effective community leader.

If you see nazis 👏👏 you ban nazis 👏👏 — it’s as simple as that.

The name of the project is censored not because it’s particularly hard for you to find, but because all they really want is attention, and you and me are going to do each other a solid by not giving them any of that directly.

To preclude the sorts of reply guys who are going to insist on name-dropping the project and having a thread about the underlying drama in the comments, the short introduction is as follows:

For a few years now, a handful of reactionary trolls have been stoking division in the community by driving a wedge between X11 and Wayland users, pushing a conspiracy theory that paints RedHat as the DEI boogeyman of FOSS and assigning reactionary values to X11 and woke (pejorative) values to Wayland. Recently, reactionary opportunists “forked” Xorg, replaced all of the literature with political manifestos and dog-whistles, then used it as a platform to start shit with downstream Linux distros by petitioning for inclusion and sending concern trolls to waste everyone’s time.

The project itself is of little consequence; they serve our purposes today by providing us with case-studies in dealing with reactionary idiots starting shit in your community.

Tech workers have long resisted the suggestion that we should be organized into unions. The topic is consistently met with a cold reception by tech workers when it is raised, and no big tech workforce is meaningfully organized. This is a fatal mistake – and I don’t mean “fatal” in the figurative sense. Tech workers, it’s time for you to unionize, and strike, or you and your loved ones are literally going to die.

In this article I will justify this statement and show that it is clearly not hyperbolic. I will explain exactly what you need to do, and how organized labor can and will save your life.

Hey – if you want to get involved in labor organizing in the tech sector you should consider joining the new unitelabor.dev forum. Adding a head’s up here in case you don’t make it to the end of this very long blog post.

The imperative to organize is your economic self-interest

Before I talk about the threats to your life and liberty that you must confront through organized labor, let me re-iterate the economic position for unionizing your workplace. It is important to revisit this now, because the power politics of the tech sector has been rapidly changing over the past few years, and those changes are not in your favor.

The tech industry bourgeoisie has been waging a prolonged war on labor for at least a decade. Far from mounting any kind of resistance, most of tech labor doesn’t even understand that this is happening to them. Your boss is obsessed with making you powerless and replaceable. You may not realize how much leverage you have over your boss, but your boss certainly does – and has been doing everything in their power to undermine you before you wizen up. Don’t let yourself believe you’re a part of their club – if your income depends on your salary, you are part of the working class.

Payroll – that’s you – is the single biggest expense for every tech company. When tech capitalists look at their balance sheet and start thinking of strategies for increasing profits, they see an awful lot of pesky zeroes stacked up next to the line item for payroll and benefits. Long-term, what’s their best play?

It starts with funneling cash and influence into educating a bigger, cheaper generation of compsci graduates to flood the labor market – “everyone can code”. Think about strategic investments in cheap(ish), broadly available courses, online schools and coding “bootcamps” – dangling your high salary as the carrot in front of wannabe coders fleeing dwindling prospects in other industries, certain that the carrot won’t be nearly as big when they all eventually step into a crowded labor market.

The next step is rolling, industry-wide mass layoffs – often obscured under the guise of “stack ranking” or some similar nonsense. Big tech has been callously cutting jobs everywhere, leaving workers out in the cold in batches of thousands or tens of thousands. If you don’t count yourself among them yet, maybe you will soon. What are your prospects for re-hire going to look like if this looming recession materializes in the next few years?

Consider what’s happening now – why do you think tech is driving AI mandates down from the top? Have you been ordered to use an LLM assistant to “help” with your programming? Have you even thought about why the executives would push this crap on you? You’re “training” your replacement. Do you really think that, if LLMs really are going to change the way we code, they aren’t going to change the way we’re paid for it? Do you think your boss doesn’t see AI as a chance to take $100M off of their payroll expenses?

Aren’t you worried you could get laid off and this junior compsci grad or an H1B takes your place for half your salary? You should be – it’s happening everywhere. What are you going to do about it? Resent the younger generation of programmers just entering the tech workforce? Or the immigrant whose family pooled their resources to send them abroad to study and work? Or maybe you weren’t laid off yet, and you fancy yourself better than the poor saps down the hall who were. Don’t be a sucker – your enemy isn’t in the cubicle next to you, or on the other side of the open office. Your enemy has an office with a door on it.

Listen: a tech union isn’t just about negotiating higher wages and benefits, although that’s definitely on the table. It’s about protecting yourself, and your colleagues, from the relentless campaign against labor that the tech leadership is waging against us. And more than that, it’s about seizing some of the awesome, society-bending power of the tech giants. Look around you and see what destructive ends this power is being applied to. You have your hands at the levers of this power if only you rise together with your peers and make demands.

And if you don’t, you are responsible for what’s going to happen next.

The imperative to organize is existential

If global warming is limited to 2°C, here’s what Palo Alto looks like in 2100:<sup>1</sup>

Limiting warming to 2° C requires us to cut global emissions in half by 2030 – in 5 years – but emissions haven’t even peaked yet. Present-day climate policies are only expected to limit warming to 2.5° to 2.9° C by 2100.<sup>2</sup> Here’s Palo Alto in 75 years if we stay our current course:

Here’s the Gulf of Mexico in 75 years:

This is what will happen if things don’t improve. Things aren’t improving – they’re getting worse. The US elected an anti-science president who backed out of the Paris agreement, for a start. Your boss is pouring all of our freshwater into datacenters to train these fucking LLMs and expanding into this exciting new market with millions of tons of emissions as the price of investment. Cryptocurrencies still account for a full 1% of global emissions. Datacenters as a whole account for 2%. That’s on us – tech workers. That is our fucking responsibility.

Climate change is accelerating, and faster than we thought, and the rich and powerful are making it happen faster. Climate catastrophe is not in the far future, it’s not our children or our children’s children, it’s us, it’s already happening. You and I will live to see dozens of global catastrophes playing out in our lifetimes, with horrifying results. Even if we started a revolution tomorrow and overthrew the ruling class and implemented aggressive climate policies right now we will still watch tens or hundreds of millions die.

Let’s say you are comfortably living outside of these blue areas, and you’ll be sitting pretty when Louisiana or Bruges or Fiji are flooded. Well, 13 million Americans are expected to have to migrate out of flooded areas – and 216 million globally<sup>3</sup> – within 25 to 30 years. That’s just from the direct causes of climate change – as many as 1 billion could be displaced if we account for the ensuing global conflict and civil unrest.<sup>4</sup> What do you think will happen to non-coastal cities and states when 4% of the American population is forced to flee their homes? You think you won’t be affected by that? What happens when anywhere from 2.5% to 12% of the Earth’s population becomes refugees?

What are you going to eat? Climate change is going to impact fresh water supplies and reduce the world’s agriculturally productive land. Livestock is expected to be reduced by 7-10% in just 25 years.<sup>5</sup> Food prices will skyrocket and people will starve. 7% of all species on Earth may already be extinct because of human activities.<sup>6</sup> You think that’s not going to affect you?

The overwhelming majority of the population supports climate action.<sup>7</sup> The reason it’s not happening is because, under capitalism, capital is power, and the few have it and the many don’t. We live in a global plutocracy.

The plutocracy has an answer to climate change: fascism. When 12% of the world’s population is knocking at the doors of the global north, their answer will be concentration camps and mass murder. They are already working on it today. When the problem is capitalism, the capitalists will go to any lengths necessary to preserve the institutions that give them power – they always have. They have no moral compass or reason besides profit, wealth, and power. The 1% will burn and pillage and murder the 99% without blinking.

They are already murdering us. 1.2 million Americans are rationing their insulin.<sup>8</sup> The healthcare industry, organized around the profit motive, murders 68,000 Americans per year.<sup>9</sup> To the Europeans among my readership, don’t get too comfortable, because I assure you that our leaders are working on destroying our healthcare systems, too.

Someone you love will be laid off, get sick, and die because they can’t afford healthcare. Someone you know, probably many people that you know, will be killed by climate change. It might be someone you love. It might be you.

When you do get laid off mid-recession, your employer replaces you and three of your peers with a fresh bootcamp “graduate” and a GitHub Copilot subscription, and all of the companies you might apply to have done the same… how long can you keep paying rent? What about your friends and family, those who don’t have a cushy tech job or tech worker prospects, what happens when they get laid off or automated away or just priced out of the cost of living? Homelessness is at an all time high and it’s only going to get higher. Being homeless takes 30 years off of your life expectancy.<sup>10</sup> In the United States, there are 28 vacant homes for every homeless person.<sup>11</sup>

Capitalism is going to murder the people you love. Capitalism is going to murder you.

We need a different answer to the crises that we face. Fortunately, the working class can offer a better solution – one with a long history of success.

Organizing is the only answer and it will work

The rich are literally going to kill you and everyone you know and love just because it will make them richer. Because it is making them richer.

Do you want to do something about any of the real, urgent problems you face? Do you want to make meaningful, rapid progress on climate change, take the catastrophic consequences we are already guaranteed to face in stride, and keep your friends and family safe?

Well, tough shit – you can’t. Don’t tell me you’ll refuse the work, or that it’ll get done anyway without you, or that you can just find another job. They’ll replace you, you won’t find another job, and the world will still burn. You can’t vote your way to a solution, either: elections don’t matter, your vote doesn’t matter, and your voice is worthless to politicians.<sup>12</sup> Martin Gilens and Benjamin Page demonstrated this most clearly in their 2014 study, “Testing Theories of American Politics: Elites, Interest Groups, and Average Citizens”.<sup>13</sup>

Gilens and Page plotted a line chart which shows us the relationship between the odds of a policy proposal being adopted (Y axis) charted against public support for the policy (X axis). If policy adoption was entirely driven by public opinion, we would expect a 45° line (Y=X), where broad public support guarantees adoption and broad public opposition prevents adoption. We could also substitute “public opinion” for the opinions of different subsets of the public to see their relative impact on policy. Here’s what they got:

For most of us, we get a flat line: Y, policy adoption, is completely unrelated to X, public support. Our opinion has no influence whatsoever on policy adoption. Public condemnation or widespread support has the same effect on a policy proposal, i.e. none. But for the wealthy, it’s a different story entirely. I’ve never seen it stated so plainly and clearly: the only thing that matters is money, wealth, and capital. Money is power, and the rich have it and you don’t.

Nevertheless, you must solve these problems. You must participate in finding and implementing solutions. You will be fucked if you don’t. But it is an unassailable fact that you can’t solve these problems, because you have no power – at least, not alone.

Together, we do have power. In fact, we can fuck with those bastards’ money and they will step in line if, and only if, we organize. It is the only solution, and it will work.

The ultra-rich possess no morals or ideology or passion or reason. They align with fascists because the fascists promise what they want, namely tax cuts, subsidies, favorable regulation, and cracking the skulls of socialists against the pavement. The rich hoard and pillage and murder with abandon for one reason and one reason only: it’s profitable. The rich always do what makes them richer, and only what makes them richer. Consequently, you need to make this a losing strategy. You need to make it more profitable to do what you want. To control the rich, you must threaten the only thing they care about.

Strikes are so costly for companies that they will do anything to prevent them – and if they fail to prevent them, then shareholders will pressure them to capitulate if only to stop the hemorrhaging of profit. This threat is so powerful that it doesn’t have to stop at negotiating your salary and benefits. You could demand your employer participate in boycotting Israel. You could demand that your employer stops anti-social lobbying efforts, or even adopts a pro-social lobbying program. You could demand that your CEO cannot support causes that threaten the lives and dignity of their queer or PoC employees. You could demand that they don’t bend the knee to fascists. If you get them where it hurts – their wallet – they will fall in line. They are more afraid of you than we are afraid of them. They are terrified of us, and it’s time we used that to our advantage.

We know it works because it has always worked. In 2023, United Auto Workers went on strike and most workers won a 25% raise. In February, teachers in Los Angeles went on strike for just 8 days and secured a 19% raise. Nurses in Oregon won a 22% raise, better working schedules, and more this year – and Hawaiian nurses secured an agreement to improve worker/patient ratios in September. Tech workers could take a page out of the Writer’s Guild’s book – in 2023 they secured a prohibition against the use of their work to train AI models and the use of AI to suppress their wages.

Organized labor is powerful and consistently gets concessions from the rich and powerful in a way that no other strategy has ever been able to. It works, and we have a moral obligation to do it. Unions gets results.

How to organize step by step

I will give you a step-by-step plan for exactly what you need to do to start moving the needle here. The process is as follows:

1. Building solidarity and community with your peers
2. Understanding your rights and how to organize safely
3. Establishing the consensus to unionize, and do it
4. Promoting solidarity with across tech workplaces and labor as a whole

Remember that you will not have to do this alone – in fact, that’s the whole point. Step one is building community with your colleagues. Get to know them personally, establish new friendships and grow the friendships you already have. Learn about each other’s wants, needs, passions, and so on, and find ways to support each other. If someone takes a sick day, organize someone to check on them and make them dinner or pick up their kids from school. Organize a board game night at your home with your colleagues, outside of work hours. Make it a regular event!

Talk to your colleagues about work, and your workplace. Tell each other about your salaries and benefits. When you get a raise, don’t be shy, tell your colleagues how much you got and how you negotiated it. Speak positively about each other at performance reviews and save critical feedback for their ears only. Offer each other advice about how to approach their boss to get their needs met, and be each other’s advocate.

Talk about the power you have to work together to accomplish bigger things. Talk about the advantage of collective action. It can start small – perhaps your team collectively refuses to incorporate LLMs into your workflow. Soon enough you and your colleagues will be thinking about unionizing.

Disclaimer: Knowledge about specific processes and legal considerations in this article is US-specific. Your local laws are likely similar, but you should research the differences with your colleagues.

The process of organizing a union in the US is explained step-by-step at workcenter.gov. More detailed resources, including access to union organizers in your neighborhood, are available from the American Federation of Labor and Congress of Industrial Organizations (AFL-CIO). But your biggest resources will be people already organizing in the tech sector: in particular you should consult CODE-CWA, which works with tech workers to provide mentoring and resources on organizing tech workplaces – and has already helped several tech workplaces organize their unions and start making a difference. They’ve got your back.

This is a good time to make sure that you and your colleagues understand your rights. First of all, you would be wise to pool your resources and hire the attention of a lawyer specializing in labor – consult your local bar association to find one (it’s easy, just google it and they’ll have a web thing). Definitely reach out to AFL-CIO and CODE-CWA to meet experienced union organizers who can help you.

You cannot be lawfully fired or punished for discussing unions, workplace conditions, or your compensation and benefits, with your colleagues. You cannot be punished for distributing literature in support of your cause, especially if you do it off-site (even just outside of the front door). Be careful not to make careless remarks about your boss’s appearance, complain about the quality of your company’s products, make disparaging comments about clients or customers, etc – don’t give them an easy excuse. Hold meetings and discussions outside of work if necessary, and perform your duties as you normally would while organizing.

Once you start getting serious about organizing, your boss will start to work against you, but know that they cannot stop you. Nevertheless, you and/or some of your colleagues may run the risk of unlawful retaliation or termination for organizing – this is why you should have a lawyer on retainer. This is also why it’s important to establish systems of mutual aid, so that if one of your colleagues gets into trouble you can lean on each other to keep supporting your families. And, importantly, remember that HR works for the company, not for you. HR are the front lines that are going to execute the unionbusting mandates from above.

Once you have a consensus among your colleagues to organize – which you will know because they will have signed union cards – you can approach your employer to ask them to voluntarily recognize the union. If they agree to opening an organized dialogue amicably, you do so. If not, you will reach out to the National Labor Relations Board (NLRB) to organize a vote to unionize. Only organize a vote that you know you will win. Once your workplace votes to unionize, your employer is obligated to negotiate with you in good faith. Start making collective decisions about what you want from your employer and bring them to the table.

In this process, you will have established a relationship with more experienced union organizers who will continue to help you with conducting your union’s affairs and start getting results. The next step is to make yourself available for this purpose to the next tech workplace that wants to unionize: to share what you’ve learned and support the rest of the industry in solidarity. Talk to your friends across the industry and build solidarity and power in mass.

Prepare for the general strike on May 1st, 2028

The call has gone out: on Labor Day, 2028 – just under three years from now – there will be a general strike in the United States. The United Auto Workers union, one of the largest in the United States, has arranged for their collective bargaining agreements to end on this date, and has called for other unions to do the same across all industries. The American Federation of Teachers and its 1.2 million members are on board, and other unions are sure to follow. Your new union should be among them.

This is how we collectively challenge not just our own employers, but our political institutions as a whole. This is how we turn this nightmare around.

A mass strike is a difficult thing to organize. It is certain to be met with large-scale, coordinated, and well-funded propaganda and retaliation from the business and political spheres. Moreover, a mass strike depends on careful planning and mass mutual aid. We need to be prepared to support each other to get it done, and to plan and organize seriously. When you and your colleagues get organized, discuss this strike amongst yourselves and be prepared to join in solidarity with the rest of the 99% around the country and the world at large.

To commit yourselves to participate or get involved in the planning of the grassroots movement, see generalstrikeus.com.

Join unitelabor.dev

I’ve set up a Discourse instance for discussion, organizing, Q&A, and solidarity among tech workers at unitelabor.dev. Please check it out!

If you have any questions or feedback on this article, please post about it there.

Unionize or die

You must organize, and you must start now, or the worst will come to pass. Fight like your life depends on it, beause it does. It has never been more urgent. The tech industry needs to stop fucking around and get organized.

We are powerful together. We can change things, and we must. Spread the word, in your workplace and with your friends and online. On the latter, be ready to fight just to speak – especially in our online spaces owned and controlled by the rich (ahem – YCombinator, Reddit, Twitter – etc). But fight all the same, and don’t stop fighting until we’re done.

We can do it, together.

Resources

Tech-specific:

• unitelabor.dev (new)
• Tech Workers Coalition (Global)
• CODE-CWA (US)
• UTAW (UK)
• Prospect (UK)
• Alphabet Workers Union (Google)

General:

• Industrial Workers of the World (Global)
• International Confederation of Labor (Global)
• AFL-CIO (US)
• workcenter.gov (US)
• generalstrikeus.com (US)
• Emergency Workplace Organizing Committee (US)
• Labor Notes (US)
• Organize Now! (UK)

Send me more resources to add here!

My spouse and I are on vacation in Japan, spending half our time seeing the sights and the other half working remotely and enjoying the experience of living in a different place for a while. To get here, we flew on British Airways from London to Tokyo, and I entertained myself on the long flight by browsing the interactive flight map on the back of my neighbor’s seat and trying to figure out how the poor developer who implemented this map solved the thorny problems that displaying a world map implies.

I began my survey by poking through the whole interface of this little in-seat entertainment system<sup>1</sup> to see if I can find out anything about who made it or how it works – I was particularly curious to find a screen listing open source licenses that such such devices often disclose. To my dismay I found nothing at all – no information about who made it or what’s inside. I imagine that there must be some open source software in that thing, but I didn’t find any licenses or copyright statements.

When I turned my attention to the map itself, I did find one copyright statement, the only one I could find in the whole UI. If you zoom in enough, it switches from a satellite view to a street view showing the OpenStreetMap copyright line:

Given that British Airways is the proud flag carrier of the United Kingdom I assume that this is indeed the only off-the-shelf copyrighted material included in this display, and everything else was developed in-house without relying on any open source software that might require a disclosure of license and copyright details. For similar reasons I am going to assume that all of the borders shown in this map are reflective of the official opinion of British Airways on various international disputes.

As I briefly mentioned a moment ago, this map has two views: satellite photography and a very basic street view. Your plane and its route are shown in real-time, and you can touch the screen to pan and zoom the map anywhere you like. You can also rotate the map and change the angle in “3D” if you have enough patience to use complex multitouch gestures on the cheapest touch panel they could find.

The street view is very sparse and only appears when you’re pretty far zoomed in, so it was mostly useless for this investigation. The satellite map, thankfully, includes labels: cities, country names, points of interest, and, importantly, national borders. The latter are very faint, however. Here’s an illustrative example:

We also have our first peek at a border dispute here: look closely between the “Georgia” and “Caucasus Mountains” labels. This ever-so-faint dotted line shows what I believe is the Russian-occupied territory of South Ossetia in Georgia. Disputes implicating Russia are not universally denoted as such – I took a peek at the border with Ukraine and found that Ukraine is shown as whole and undisputed, with its (undotted) border showing Donetsk, Luhansk, and Crimea entirely within Ukraine’s borders.

Of course, I didn’t start at Russian border disputes when I went looking for trouble. I went directly to Palestine. Or rather, I went to Israel, because Palestine doesn’t exist on this map:

I squinted and looked very closely at the screen and I’m fairly certain that both the West Bank and Gaza are outlined in these dotted lines using the borders defined by the 1949 armistice. If you zoom in a bit more to the street view, you can see labels like “West Bank” and the “Area A”, “Area B” labels of the Oslo Accords:

Given that this is British Airways, part of me was surprised not to see the whole area simply labelled Mandatory Palestine, but it is interesting to know that British Airways officially supports the Oslo Accords.

Heading south, let’s take a look at the situation in Sudan:

This one is interesting – three areas within South Sudan’s claimed borders are disputed, and the map only shows two with these dotted lines. The border dispute with Sudan in the northeast is resolved in South Sudan’s favor. Another case where BA takes a stand is Guyana, which has an ongoing dispute with Venezuela – but the map only shows Guyana’s claim, albeit with a dotted line, rather than the usual approach of drawing both claims with dotted lines.

Next, I turned my attention to Taiwan:

The cities of Taipei and Kaohsiung are labelled, but the island as a whole was not labelled “Taiwan”. I zoomed and panned and 3D-zoomed the map all over the place but was unable to get a “Taiwan” label to appear. I also zoomed into the OSM-provided street map and panned that around but couldn’t find “Taiwan” anywhere, either.

The last picture I took is of the Kashmir area:

I find these faint borders difficult to interpret and I admit to not being very familiar with this conflict, but perhaps someone in the know with the patience to look more closely will email me their understanding of the official British Airways position on the Kashmir conflict (here’s the full sized picture).

Here are some other details I noted as I browsed the map:

• The Hala’ib Triangle and Bir Tawil are shown with dotted lines
• The Gulf of Mexico is labelled as such
• Antarctica has no labelled borders or settlements

After this thrilling survey of the official political positions of British Airways, I spent the rest of the flight reading books or trying to sleep.

Hi nerds, I'm back from Systems Distributed! I'd heartily recommend it, wildest conference I've been to in years. I have a lot of work to catch up on, so this will be a short newsletter.

In an earlier version of my talk, I had a gag about unit tests. First I showed the test f([1,2,3]) == 3, then said that this was satisfied by f(l) = 3, f(l) = l[-1], f(l) = len(l), f(l) = (129*l[0]-34*l[1]-617)*l[2] - 443*l[0] + 1148*l[1] - 182. Then I progressively rule them out one by one with more unit tests, except the last polynomial which stubbornly passes every single test.

If you're given some function of f(x: int, y: int, …): int and a set of unit tests asserting specific inputs give specific outputs, then you can find a polynomial that passes every single unit test.

To find the gag, and as SMT practice, I wrote a Python program that finds a polynomial that passes a test suite meant for max. It's hardcoded for three parameters and only finds 2nd-order polynomials but I think it could be generalized with enough effort.

The code

Full code here, breakdown below.

from z3 import *  # type: ignore
s1, s2 = Solver(), Solver()

Z3 is just the particular SMT solver we use, as it has good language bindings and a lot of affordances.

As part of learning SMT I wanted to do this two ways. First by putting the polynomial "outside" of the SMT solver in a python function, second by doing it "natively" in Z3. I created two solvers so I could test both versions in one run.

a0, a, b, c, d, e, f = Consts('a0 a b c d e f', IntSort())
x, y, z = Ints('x y z')
t = "a*x+b*y+c*z+d*x*y+e*x*z+f*y*z+a0"

Both Const('x', IntSort()) and Int('x') do the exact same thing, the latter being syntactic sugar for the former. I did not know this when I wrote the program.

To keep the two versions in sync I represented the equation as a string, which I later eval. This is one of the rare cases where eval is a good idea, to help us experiment more quickly while learning. The polynomial is a "2nd-order polynomial", even though it doesn't have x^2 terms, as it has xy and xz terms.

lambdamax = lambda x, y, z: eval(t)

z3max = Function('z3max', IntSort(), IntSort(), IntSort(),  IntSort())
s1.add(ForAll([x, y, z], z3max(x, y, z) == eval(t)))

lambdamax is pretty straightforward: create a lambda with three parameters and eval the string. The string "a*x" then becomes the python expression a*x, a is an SMT symbol, while the x SMT symbol is shadowed by the lambda parameter. To reiterate, a terrible idea in practice, but a good way to learn faster.

z3max function is a little more complex. Function takes an identifier string and N "sorts" (roughly the same as programming types). The first N-1 sorts define the parameters of the function, while the last becomes the output. So here I assign the string identifier "z3max" to be a function with signature (int, int, int) -> int.

I can load the function into the model by specifying constraints on what z3max could be. This could either be a strict input/output, as will be done later, or a ForAll over all possible inputs. Here I just use that directly to say "for all inputs, the function should match this polynomial." But I could do more complicated constraints, like commutativity (f(x, y) == f(y, x)) or monotonicity (Implies(x < y, f(x) <= f(y))).

Note ForAll takes a list of z3 symbols to quantify over. That's the only reason we need to define x, y, z in the first place. The lambda version doesn't need them.

inputs = [(1,2,3), (4, 2, 2), (1, 1, 1), (3, 5, 4)]

for g in inputs:
    s1.add(z3max(*g) == max(*g))
    s2.add(lambdamax(*g) == max(*g))

This sets up the joke: adding constraints to each solver that the polynomial it finds must, for a fixed list of triplets, return the max of each triplet.

for s, func in [(s1, z3max), (s2, lambdamax)]:
    if s.check() == sat:
        m = s.model()
        for x, y, z in inputs:
            print(f"max([{x}, {y}, {z}]) =", m.evaluate(func(x, y, z)))
        print(f"max([x, y, z]) = {m[a]}x + {m[b]}y",
            f"+ {m[c]}z +", # linebreaks added for newsletter rendering
            f"{m[d]}xy + {m[e]}xz + {m[f]}yz + {m[a0]}\n")

Output:

max([1, 2, 3]) = 3
# etc
max([x, y, z]) = -133x + 130y + -10z + -2xy + 62xz + -46yz + 0

max([1, 2, 3]) = 3
# etc
max([x, y, z]) = -17x + 16y + 0z + 0xy + 8xz + -6yz + 0

I find that z3max (top) consistently finds larger coefficients than lambdamax does. I don't know why.

Practical Applications

Test-Driven Development recommends a strict "red-green refactor" cycle. Write a new failing test, make the new test pass, then go back and refactor. Well, the easiest way to make the new test pass would be to paste in a new polynomial, so that's what you should be doing. You can even do this all automatically: have a script read the set of test cases, pass them to the solver, and write the new polynomial to your code file. All you need to do is write the tests!

Pedagogical Notes

Writing the script took me a couple of hours. I'm sure an LLM could have whipped it all up in five minutes but I really want to learn SMT and LLMs may decrease learning retention.<sup>1</sup> Z3 documentation is not... great for non-academics, though, and most other SMT solvers have even worse docs. One useful trick I use regularly is to use Github code search to find code using the same APIs and study how that works. Turns out reading API-heavy code is a lot easier than writing it!

Anyway, I'm very, very slowly feeling like I'm getting the basics on how to use SMT. I don't have any practical use cases yet, but I wanted to learn this skill for a while and glad I finally did.

No newsletter next week

I’ll be speaking at Systems Distributed. My talk isn't close to done yet, which is why this newsletter is both late and short.

Solving LinkedIn Queens in SMT

The article Modern SAT solvers: fast, neat and underused claims that SAT solvers<sup>1</sup> are "criminally underused by the industry". A while back on the newsletter I asked "why": how come they're so powerful and yet nobody uses them? Many experts responded saying the reason is that encoding SAT kinda sucked and they rather prefer using tools that compile to SAT.

I was reminded of this when I read Ryan Berger's post on solving “LinkedIn Queens” as a SAT problem.

A quick overview of Queens. You’re presented with an NxN grid divided into N regions, and have to place N queens so that there is exactly one queen in each row, column, and region. While queens can be on the same diagonal, they cannot be adjacently diagonal.

(Important note: Linkedin “Queens” is a variation on the puzzle game Star Battle, which is the same except the number of stars you place in each row/column/region varies per puzzle, and is usually two. This is also why 'queens' don’t capture like chess queens.)

Ryan solved this by writing Queens as a SAT problem, expressing properties like "there is exactly one queen in row 3" as a large number of boolean clauses. Go read his post, it's pretty cool. What leapt out to me was that he used CVC5, an SMT solver.<sup>2</sup> SMT solvers are "higher-level" than SAT, capable of handling more data types than just boolean variables. It's a lot easier to solve the problem at the SMT level than at the SAT level. To show this, I whipped up a short demo of solving the same problem in Z3 (via the Python API).

Full code here, which you can compare to Ryan's SAT solution here. I didn't do a whole lot of cleanup on it (again, time crunch!), but short explanation below.

The code

from z3 import * # type: ignore
from itertools import combinations, chain, product
solver = Solver()
size = 9 # N

Initial setup and modules. size is the number of rows/columns/regions in the board, which I'll call N below.

# queens[n] = col of queen on row n
# by construction, not on same row
queens = IntVector('q', size) 

SAT represents the queen positions via N² booleans: q_00 means that a Queen is on row 0 and column 0, !q_05 means a queen isn't on row 0 col 5, etc. In SMT we can instead encode it as N integers: q_0 = 5 means that the queen on row 0 is positioned at column 5. This immediately enforces one class of constraints for us: we don't need any constraints saying "exactly one queen per row", because that's embedded in the definition of queens!

(Incidentally, using 0-based indexing for the board was a mistake on my part, it makes correctly encoding the regions later really painful.)

To actually make the variables [q_0, q_1, …], we use the Z3 affordance IntVector(str, n) for making n variables at once.

solver.add([And(0 <= i, i < size) for i in queens])
# not on same column
solver.add(Distinct(queens))

First we constrain all the integers to [0, N), then use the incredibly handy Distinct constraint to force all the integers to have different values. This guarantees at most one queen per column, which by the pigeonhole principle means there is exactly one queen per column.

# not diagonally adjacent
for i in range(size-1):
    q1, q2 = queens[i], queens[i+1]
    solver.add(Abs(q1 - q2) != 1)

One of the rules is that queens can't be adjacent. We already know that they can't be horizontally or vertically adjacent via other constraints, which leaves the diagonals. We only need to add constraints that, for each queen, there is no queen in the lower-left or lower-right corner, aka q_3 != q_2 ± 1. We don't need to check the top corners because if q_1 is in the upper-left corner of q_2, then q_2 is in the lower-right corner of q_1!

That covers everything except the "one queen per region" constraint. But the regions are the tricky part, which we should expect because we vary the difficulty of queens games by varying the regions.

regions = {
        "purple": [(0, 0), (0, 1), (0, 2), (0, 3), (0, 4), (0, 5), (0, 6), (0, 7), (0, 8),
                   (1, 0), (2, 0), (3, 0), (4, 0), (5, 0), (6, 0), (7, 0), (8, 0),
                   (1, 1), (8, 1)],
        "red": [(1, 2), (2, 2), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (6, 2), (7, 1), (7, 2), (8, 2), (8, 3),],
        # you get the picture
        }

# Some checking code left out, see below

The region has to be manually coded in, which is a huge pain.

(In the link, some validation code follows. Since it breaks up explaining the model I put it in the next section.)

for r in regions.values():
    solver.add(Or(
        *[queens[row] == col for (row, col) in r]
        ))

Finally we have the region constraint. The easiest way I found to say "there is exactly one queen in each region" is to say "there is a queen in region 1 and a queen in region 2 and a queen in region 3" etc." Then to say "there is a queen in region purple" I wrote "q_0 = 0 OR q_0 = 1 OR … OR q_1 = 0 etc."

Why iterate over every position in the region instead of doing something like (0, q[0]) in r? I tried that but it's not an expression that Z3 supports.

if solver.check() == sat:
    m = solver.model()
    print([(l, m[l]) for l in queens])

Finally, we solve and print the positions. Running this gives me:

[(q__0, 0), (q__1, 5), (q__2, 8), 
 (q__3, 2), (q__4, 7), (q__5, 4), 
 (q__6, 1), (q__7, 3), (q__8, 6)]

Which is the correct solution to the queens puzzle. I didn't benchmark the solution times, but I imagine it's considerably slower than a raw SAT solver. Glucose is really, really fast.

But even so, solving the problem with SMT was a lot easier than solving it with SAT. That satisfies me as an explanation for why people prefer it to SAT.

Sanity checks

One bit I glossed over earlier was the sanity checking code. I knew for sure that I was going to make a mistake encoding the region, and the solver wasn't going to provide useful information abut what I did wrong. In cases like these, I like adding small tests and checks to catch mistakes early, because the solver certainly isn't going to catch them!

all_squares = set(product(range(size), repeat=2))
def test_i_set_up_problem_right():
    assert all_squares == set(chain.from_iterable(regions.values()))

    for r1, r2 in combinations(regions.values(), 2):
        assert not set(r1) & set(r2), set(r1) & set(r2)

The first check was a quick test that I didn't leave any squares out, or accidentally put the same square in both regions. Converting the values into sets makes both checks a lot easier. Honestly I don't know why I didn't just use sets from the start, sets are great.

def render_regions():
    colormap = ["purple",  "red", "brown", "white", "green", "yellow", "orange", "blue", "pink"]
    board = [[0 for _ in range(size)] for _ in range(size)] 
    for (row, col) in all_squares:
        for color, region in regions.items():
            if (row, col) in region:
                board[row][col] = colormap.index(color)+1

    for row in board:
        print("".join(map(str, row)))

render_regions()

The second check is something that prints out the regions. It produces something like this:

111111111
112333999
122439999
124437799
124666779
124467799
122467899
122555889
112258899

I can compare this to the picture of the board to make sure I got it right. I guess a more advanced solution would be to print emoji squares like 🟥 instead.

Neither check is quality code but it's throwaway and it gets the job done so eh.

Update for the Internet

This was sent as a weekly newsletter, which is usually on topics like software history, formal methods, unusual technologies, and the theory of software engineering. You can subscribe here.

New Logic for Programmers Release

v0.10 is now available! This is a minor release, mostly focused on logic-based refactoring, with new material on set types and testing refactors are correct. See the full release notes at the changelog page. Due to conference pressure v0.11 will also likely be a minor release.

AI is a gamechanger for TLA+ users

TLA+ is a specification language to model and debug distributed systems. While very powerful, it's also hard for programmers to learn, and there's always questions of connecting specifications with actual code.

That's why The Coming AI Revolution in Distributed Systems caught my interest. In the post, Cheng Huang claims that Azure successfully used LLMs to examine an existing codebase, derive a TLA+ spec, and find a production bug in that spec. "After a decade of manually crafting TLA+ specifications", he wrote, "I must acknowledge that this AI-generated specification rivals human work".

This inspired me to experiment with LLMs in TLA+ myself. My goals are a little less ambitious than Cheng's: I wanted to see how LLMs could help junior specifiers write TLA+, rather than handling the entire spec automatically. Details on what did and didn't work below, but my takeaway is that LLMs are an immense specification force multiplier.

All tests were done with a standard VSCode Copilot subscription, writing Claude 3.7 in Agent mode. Other LLMs or IDEs may be more or less effective, etc.

Things Claude was good at

Fixing syntax errors

TLA+ uses a very different syntax than mainstream programming languages, meaning beginners make a lot of mistakes where they do a "programming syntax" instead of TLA+ syntax:

NotThree(x) = \* should be ==, not =
    x != 3 \* should be #, not !=

The problem is that the TLA+ syntax checker, SANY, is 30 years old and doesn't provide good information. Here's what it says for that snippet:

Was expecting "==== or more Module body"
Encountered "NotThree" at line 6, column 1

That only isolates one error and doesn't tell us what the problem is, only where it is. Experienced TLA+ users get "error eyes" and can quickly see what the problem is, but beginners really struggle with this.

The TLA+ foundation has made LLM integration a priority, so the VSCode extension naturally supports several agents actions. One of these is running SANY, meaning an agent can get an error, fix it, get another error, fix it, etc. Provided the above sample and asked to make it work, Claude successfully fixed both errors. It also fixed many errors in a larger spec, as well as figure out why PlusCal specs weren't compiling to TLA+.

This by itself is already enough to make LLMs a worthwhile tool, as it fixes one of the biggest barriers to entry.

Understanding error traces

When TLA+ finds a violated property, it outputs the sequence of steps that leads to the error. This starts in plaintext, and VSCode parses it into an interactive table:

Learning to read these error traces is a skill in itself. You have to understand what's happening in each step and how it relates back to the actually broken property. It takes a long time for people to learn how to do this well.

Claude was successful here, too, accurately reading 20+ step error traces and giving a high-level explanation of what went wrong. It also could condense error traces: if ten steps of the error trace could be condensed into a one-sentence summary (which can happen if you're modeling a lot of process internals) Claude would do it.

I did have issues here with doing this in agent mode: while the extension does provide a "run model checker" command, the agent would regularly ignore this and prefer to run a terminal command instead. This would be fine except that the LLM consistently hallucinated invalid commands. I had to amend every prompt with "run the model checker via vscode, do not use a terminal command". You can skip this if you're willing to copy and paste the error trace into the prompt.

As with syntax checking, if this was the only thing LLMs could effectively do, that would already be enough<sup>1</sup> to earn a strong recommend. Even as a TLA+ expert I expect I'll be using this trick regularly.

Boilerplate tasks

TLA+ has a lot of boilerplate. One of the most notorious examples is UNCHANGED rules. Specifications are extremely precise — so precise that you have to specify what variables don't change in every step. This takes the form of an UNCHANGED clause at the end of relevant actions:

RemoveObjectFromStore(srv, o, s) ==
  /\ o \in stored[s]
  /\ stored' = [stored EXCEPT ![s] = @ \ {o}]
  /\ UNCHANGED <<capacity, log, objectsize, pc>>

Writing this is really annoying. Updating these whenever you change an action, or add a new variable to the spec, is doubly so. Syntax checking and error analysis are important for beginners, but this is what I wanted for myself. I took a spec and prompted Claude

Add UNCHANGED <> for each variable not changed in an action.

And it worked! It successfully updated the UNCHANGED in every action.

(Note, though, that it was a "well-behaved" spec in this regard: only one "action" happened at a time. In TLA+ you can have two actions happen simultaneously, that each update half of the variables, meaning neither of them should have an UNCHANGED clause. I haven't tested how Claude handles that!)

That's the most obvious win, but Claude was good at handling other tedious work, too. Some examples include updating vars (the conventional collection of all state variables), lifting a hard-coded value into a model parameter, and changing data formats. Most impressive to me, though, was rewriting a spec designed for one process to instead handle multiple processes. This means taking all of the process variables, which originally have types like Int, converting them to types like [Process -> Int], and then updating the uses of all of those variables in the spec. It didn't account for race conditions in the new concurrent behavior, but it was an excellent scaffold to do more work.

Writing properties from an informal description

You have to be pretty precise with your intended property description but it handles converting that precise description into TLA+'s formalized syntax, which is something beginners often struggle with.

Things it is less good at

Generating model config files

To model check TLA+, you need both a specification (.tla) and a model config file (.cfg), which have separate syntaxes. Asking the agent to generate the second often lead to it using TLA+ syntax. It automatically fixed this after getting parsing errors, though.

Fixing specs

Whenever the ran model checking and discovered a bug, it would naturally propose a change to either the invalid property or the spec. Sometimes the changes were good, other times the changes were not physically realizable. For example, if it found that a bug was due to a race condition between processes, it would often suggest fixing it by saying race conditions were okay. I mean yes, if you say bugs are okay, then the spec finds that bugs are okay! Or it would alternatively suggest adding a constraint to the spec saying that race conditions don't happen. But that's a huge mistake in specification, because race conditions happen if we don't have coordination. We need to specify the mechanism that is supposed to prevent them.

Finding properties of the spec

After seeing how capable it was at translating my properties to TLA+, I started prompting Claude to come up with properties on its own. Unfortunately, almost everything I got back was either trivial, uninteresting, or too coupled to implementation details. I haven't tested if it would work better to ask it for "properties that may be violated".

Generating code from specs

I have to be specific here: Claude could sometimes convert Python into a passable spec, an vice versa. It wasn't good at recognizing abstraction. For example, TLA+ specifications often represent sequential operations with a state variable, commonly called pc. If modeling code that nonatomically retrieves a counter value and increments it, we'd have one action that requires pc = "Get" and sets the new value to "Inc", then another that requires it be "Inc" and sets it to "Done".

I found that Claude would try to somehow convert pc into part of the Python program's state, rather than recognize it as a TLA+ abstraction. On the other side, when converting python code to TLA+ it would often try to translate things like sleep into some part of the spec, not recognizing that it is abstractable into a distinct action. I didn't test other possible misconceptions, like converting randomness to nondeterminism.

For the record, when converting TLA+ to Python Claude tended to make simulators of the spec, rather than possible production code implementing the spec. I really wasn't expecting otherwise though.

Unexplored Applications

Things I haven't explored thoroughly but could possibly be effective, based on what I know about TLA+ and AI:

Writing Java Overrides

Most TLA+ operators are resolved via TLA+ interpreters, but you can also implement them in "native" Java. This lets you escape the standard language semantics and add capabilities like executing programs during model-checking or dynamically constrain the depth of the searched state space. There's a lot of cool things I think would be possible with overrides. The problem is there's only a handful of people in the world who know how to write them. But that handful have written quite a few overrides and I think there's enough there for Claude to work with.

Writing specs, given a reference mechanism

In all my experiments, the LLM only had my prompts and the occasional Python script as information. That makes me suspect that some of its problems with writing and fixing specs come down to not having a system model. Maybe it wouldn't suggest fixes like "these processes never race" if it had a design doc saying that the processes can't coordinate.

(Could a Sufficiently Powerful LLM derive some TLA+ specification from a design document?)

Connecting specs and code

This is the holy grail of TLA+: taking a codebase and showing it correctly implements a spec. Currently the best ways to do this are by either using TLA+ to generate a test suite, or by taking logged production traces and matching them to TLA+ behaviors. This blog post discusses both. While I've seen a lot of academic research into these approaches there are no industry-ready tools. So if you want trace validation you have to do a lot of manual labour tailored to your specific product.

If LLMs could do some of this work for us then that'd really amplify the usefulness of TLA+ to many companies.

Thoughts

Right now, agents seem good at the tedious and routine parts of TLA+ and worse at the strategic and abstraction parts. But, since the routine parts are often a huge barrier to beginners, this means that LLMs have the potential to make TLA+ far, far more accessible than it previously was.

I have mixed thoughts on this. As an advocate, this is incredible. I want more people using formal specifications because I believe it leads to cheaper, safer, more reliable software. Anything that gets people comfortable with specs is great for our industry. As a professional TLA+ consultant, I'm worried that this obsoletes me. Most of my income comes from training and coaching, which companies will have far less demand of now. Then again, maybe this an opportunity to pitch "agentic TLA+ training" to companies!

Anyway, if you're interested in TLA+, there has never been a better time to try it. I mean it, these tools handle so much of the hard part now. I've got a free book available online, as does the inventor of TLA+. I like this guide too. Happy modeling!

Systems Distributed

I'll be speaking at Systems Distributed next month! The talk is brand new and will aim to showcase some of the formal methods mental models that would be useful in mainstream software development. It has added some extra stress on my schedule, though, so expect the next two monthly releases of Logic for Programmers to be mostly minor changes.

What does "Undecidable" mean, anyway

Last week I read Against Curry-Howard Mysticism, which is a solid article I recommend reading. But this newsletter is actually about one comment:

I like to see posts like this because I often feel like I can’t tell the difference between BS and a point I’m missing. Can we get one for questions like “Isn’t XYZ (Undecidable|NP-Complete|PSPACE-Complete)?”

I've already written one of these for NP-complete, so let's do one for "undecidable". Step one is to pull a technical definition from the book Automata and Computability:

A property P of strings is said to be decidable if ... there is a total Turing machine that accepts input strings that have property P and rejects those that do not. (pg 220)

Step two is to translate the technical computer science definition into more conventional programmer terms. Warning, because this is a newsletter and not a blog post, I might be a little sloppy with terms.

Machines and Decision Problems

In automata theory, all inputs to a "program" are strings of characters, and all outputs are "true" or "false". A program "accepts" a string if it outputs "true", and "rejects" if it outputs "false". You can think of this as automata studying all pure functions of type f :: string -> boolean. Problems solvable by finding such an f are called "decision problems".

This covers more than you'd think, because we can bootstrap more powerful functions from these. First, as anyone who's programmed in bash knows, strings can represent any other data. Second, we can fake non-boolean outputs by instead checking if a certain computation gives a certain result. For example, I can reframe the function add(x, y) = x + y as a decision problem like this:

IS_SUM(str) {
    x, y, z = split(str, "#")
    return x + y == z
}

Then because IS_SUM("2#3#5") returns true, we know 2 + 3 == 5, while IS_SUM("2#3#6") is false. Since we can bootstrap parameters out of strings, I'll just say it's IS_SUM(x, y, z) going forward.

A big part of automata theory is studying different models of computation with different strengths. One of the weakest is called "DFA". I won't go into any details about what DFA actually can do, but the important thing is that it can't solve IS_SUM. That is, if you give me a DFA that takes inputs of form x#y#z, I can always find an input where the DFA returns true when x + y != z, or an input which returns false when x + y == z.

It's really important to keep this model of "solve" in mind: a program solves a problem if it correctly returns true on all true inputs and correctly returns false on all false inputs.

(total) Turing Machines

A Turing Machine (TM) is a particular type of computation model. It's important for two reasons:

1. By the Church-Turing thesis, a Turing Machine is the "upper bound" of how powerful (physically realizable) computational models can get. This means that if an actual real-world programming language can solve a particular decision problem, so can a TM. Conversely, if the TM can't solve it, neither can the programming language.<sup>1</sup>

2. It's possible to write a Turing machine that takes a textual representation of another Turing machine as input, and then simulates that Turing machine as part of its computations.

Property (1) means that we can move between different computational models of equal strength, proving things about one to learn things about another. That's why I'm able to write IS_SUM in a pseudocode instead of writing it in terms of the TM computational model (and why I was able to use split for convenience).

Property (2) does several interesting things. First of all, it makes it possible to compose Turing machines. Here's how I can roughly ask if a given number is the sum of two primes, with "just" addition and boolean functions:

IS_SUM_TWO_PRIMES(z):
    x := 1
    y := 1
    loop {
        if x > z {return false}
        if IS_PRIME(x) {
            if IS_PRIME(y) {
                if IS_SUM(x, y, z) {
                    return true;
                }
            }
        }
        y := y + 1
        if y > x {
            x := x + 1
            y := 0
        }
    }

Notice that without the if x > z {return false}, the program would loop forever on z=2. A TM that always halts for all inputs is called total.

Property (2) also makes "Turing machines" a possible input to functions, meaning that we can now make decision problems about the behavior of Turing machines. For example, "does the TM M either accept or reject x within ten steps?"<sup>2</sup>

IS_DONE_IN_TEN_STEPS(M, x) {
    for (i = 0; i < 10; i++) {
        `simulate M(x) for one step`
        if(`M accepted or rejected`) {
            return true
        }
    }
    return false
}

Decidability and Undecidability

Now we have all of the pieces to understand our original definition:

A property P of strings is said to be decidable if ... there is a total Turing machine that accepts input strings that have property P and rejects those that do not. (220)

Let IS_P be the decision problem "Does the input satisfy P"? Then IS_P is decidable if it can be solved by a Turing machine, ie, I can provide some IS_P(x) machine that always accepts if x has property P, and always rejects if x doesn't have property P. If I can't do that, then IS_P is undecidable.

IS_SUM(x, y, z) and IS_DONE_IN_TEN_STEPS(M, x) are decidable properties. Is IS_SUM_TWO_PRIMES(z) decidable? Some analysis shows that our corresponding program will either find a solution, or have x>z and return false. So yes, it is decidable.

Notice there's an asymmetry here. To prove some property is decidable, I need just to need to find one program that correctly solves it. To prove some property is undecidable, I need to show that any possible program, no matter what it is, doesn't solve it.

So with that asymmetry in mind, do are there any undecidable problems? Yes, quite a lot. Recall that Turing machines can accept encodings of other TMs as input, meaning we can write a TM that checks properties of Turing machines. And, by Rice's Theorem, almost every nontrivial semantic<sup>3</sup> property of Turing machines is undecidable. The conventional way to prove this is to first find a single undecidable property H, and then use that to bootstrap undecidability of other properties.

The canonical and most famous example of an undecidable problem is the Halting problem: "does machine M halt on input i?" It's pretty easy to prove undecidable, and easy to use it to bootstrap other undecidability properties. But again, any nontrivial property is undecidable. Checking a TM is total is undecidable. Checking a TM accepts any inputs is undecidable. Checking a TM solves IS_SUM is undecidable. Etc etc etc.

What this doesn't mean in practice

I often see the halting problem misconstrued as "it's impossible to tell if a program will halt before running it." This is wrong. The halting problem says that we cannot create an algorithm that, when applied to an arbitrary program, tells us whether the program will halt or not. It is absolutely possible to tell if many programs will halt or not. It's possible to find entire subcategories of programs that are guaranteed to halt. It's possible to say "a program constructed following constraints XYZ is guaranteed to halt."

The actual consequence of undecidability is more subtle. If we want to know if a program has property P, undecidability tells us

1. We will have to spend time and mental effort to determine if it has P
2. We may not be successful.

This is subtle because we're so used to living in a world where everything's undecidable that we don't really consider what the counterfactual would be like. In such a world there might be no need for Rust, because "does this C program guarantee memory-safety" is a decidable property. The entire field of formal verification could be unnecessary, as we could just check properties of arbitrary programs directly. We could automatically check if a change in a program preserves all existing behavior. Lots of famous math problems could be solved overnight.

(This to me is a strong "intuitive" argument for why the halting problem is undecidable: a halt detector can be trivially repurposed as a program optimizer / theorem-prover / bcrypt cracker / chess engine. It's too powerful, so we should expect it to be impossible.)

But because we don't live in that world, all of those things are hard problems that take effort and ingenuity to solve, and even then we often fail.

Update for the Internet

This was sent as a weekly newsletter, which is usually on topics like software history, formal methods, unusual technologies, and the theory of software engineering. You can subscribe here.

Planner programming is a programming technique where you solve problems by providing a goal and actions, and letting the planner find actions that reach the goal. In a previous edition of Logic for Programmers, I demonstrated how this worked by solving the 24 puzzle with planning. For reasons discussed here I replaced that example with something more practical (orchestrating deployments), but left the code online for posterity.

Recently I saw a family member try and fail to vibe code a tool that would find all valid 24 puzzles, and realized I could adapt the puzzle solver to also be a puzzle generator. First I'll explain the puzzle rules, then the original solver, then the generator.<sup>1</sup> For a much longer intro to planning, see here.

The rules of 24

You're given four numbers and have to find some elementary equation (+-*/+groupings) that uses all four numbers and results in 24. Each number must be used exactly once, but do not need to be used in the starting puzzle order. Some examples:

• [6, 6, 6, 6] -> 6+6+6+6=24
• [1, 1, 6, 6] -> (6+6)*(1+1)=24
• [4, 4, 4, 5] -> 4*(5+4/4)=24

Some setups are impossible, like [1, 1, 1, 1]. Others are possible only with non-elementary operations, like [1, 5, 5, 324] (which requires exponentiation).

The solver

We will use the Picat, the only language that I know has a built-in planner module. The current state of our plan with be represented by a single list with all of the numbers.

import planner, math.
import cp.

action(S0, S1, Action, Cost) ?=>
  member(X, S0)
  , S0 := delete(S0, X) % , is `and`
  , member(Y, S0)
  , S0 := delete(S0, Y)
  , (
      A = $(X + Y) 
    ; A = $(X - Y)
    ; A = $(X * Y)
    ; A = $(X / Y), Y > 0
    )
    , S1 = S0 ++ [apply(A)]
  , Action = A
  , Cost = 1
  .

This is our "action", and it works in three steps:

1. Nondeterministically pull two different values out of the input, deleting them
2. Nondeterministically pick one of the basic operations
3. The new state is the remaining elements, appended with that operation applied to our two picks.

Let's walk through this with [1, 6, 1, 7]. There are four choices for X and three four Y. If the planner chooses X=6 and Y=7, A = $(6 + 7). This is an uncomputed term in the same way lisps might use quotation. We can resolve the computation with apply, as in the line S1 = S0 ++ [apply(A)].

final([N]) =>
  N =:= 24. % handle floating point

Our final goal is just a list where the only element is 24. This has to be a little floating point-sensitive to handle floating point divison, done by =:=.

main =>
  Start = [1, 5, 5, 6]
  , best_plan(Start, 4, Plan)
  , printf("%w %w%n", Start, Plan)
  .

For main, we just find the best plan with the maximum cost of 4 and print it. When run from the command line, picat automatically executes whatever is in main.

$ picat 24.pi
[1,5,5,6] [1 + 5,5 * 6,30 - 6]

I don't want to spoil any more 24 puzzles, so let's stop showing the plan:

main =>
- , printf("%w %w%n", Start, Plan)
+ , printf("%w%n", Start)

Generating puzzles

Picat provides a find_all(X, p(X)) function, which ruturns all X for which p(X) is true. In theory, we could write find_all(S, best_plan(S, 4, _). In practice, there are an infinite number of valid puzzles, so we need to bound S somewhat. We also don't want to find any redundant puzzles, such as [6, 6, 6, 4] and [4, 6, 6, 6].

We can solve both issues by writing a helper valid24(S), which will check that S a sorted list of integers within some bounds, like 1..8, and also has a valid solution.

valid24(Start) =>
  Start = new_list(4)
  , Start :: 1..8 % every value in 1..8
  , increasing(Start) % sorted ascending
  , solve(Start) % turn into values
  , best_plan(Start, 4, Plan)
  .

This leans on Picat's constraint solving features to automatically find bounded sorted lists, which is why we need the solve step.<sup>2</sup> Now we can just loop through all of the values in find_all to get all solutions:

main =>
  foreach([S] in find_all(
    [Start],
    valid24(Start)))
    printf("%w%n", S)
  end.

$ picat 24.pi

[1,1,1,8]
[1,1,2,6]
[1,1,2,7]
[1,1,2,8]
# etc

Finding hard puzzles

Last Friday I realized I could do something more interesting with this. Once I have found a plan, I can apply further constraints to the plan, for example to find problems that can be solved with division:

valid24(Start, Plan) =>
  Start = new_list(4)
  , Start :: 1..8
  , increasing(Start)
  , solve(Start)
  , best_plan(Start, 4, Plan)
+ , member($(_ / _), Plan)
  .

In playing with this, though, I noticed something weird: there are some solutions that appear if I sort up but not down. For example, [3,3,4,5] appears in the solution set, but [5, 4, 3, 3] doesn't appear if I replace increasing with decreasing.

As far as I can tell, this is because Picat only finds one best plan, and [5, 4, 3, 3] has two solutions: 4*(5-3/3) and 3*(5+4)-3. best_plan is a deterministic operator, so Picat commits to the first best plan it finds. So if it finds 3*(5+4)-3 first, it sees that the solution doesn't contain a division, throws [5, 4, 3, 3] away as a candidate, and moves on to the next puzzle.

There's a couple ways we can fix this. We could replace best_plan with best_plan_nondet, which can backtrack to find new plans (at the cost of an enormous number of duplicates). Or we could modify our final to only accept plans with a division:

% Hypothetical change
final([N]) =>
+ member($(_ / _), current_plan()),
  N =:= 24.

My favorite "fix" is to ask another question entirely. While I was looking for puzzles that can be solved with division, what I actually want is puzzles that must be solved with division. What if I rejected any puzzle that has a solution without division?

+ plan_with_no_div(S, P) => best_plan_nondet(S, 4, P), not member($(_ / _), P).

valid24(Start, Plan) =>
  Start = new_list(4)
  , Start :: 1..8
  , increasing(Start)
  , solve(Start)
  , best_plan(Start, 4, Plan)
- , member($(_ / _), Plan)
+ , not plan_with_no_div(Start, _)
  .

The new line's a bit tricky. plan_with_div nondeterministically finds a plan, and then fails if the plan contains a division.<sup>3</sup> Since I used best_plan_nondet, it can backtrack from there and find a new plan. This means plan_with_no_div only fails if not such plan exists. And in valid24, we only succeed if plan_with_no_div fails, guaranteeing that the only existing plans use division. Since this doesn't depend on the plan found via best_plan, it doesn't matter how the values in Start are arranged, this will not miss any valid puzzles.

Aside for my logic book readers

The new clause is equivalent to !(some p: Plan(p) && !(div in p)). Applying the simplifications we learned:

1. !(some p: Plan(p) && !(div in p)) (init)
2. all p: !(plan(p) && !(div in p)) (all/some duality)
3. all p: !plan(p) || div in p) (De Morgan's law)
4. all p: plan(p) => div in p (implication definition)

Which more obviously means "if P is a valid plan, then it contains a division".

Back to finding hard puzzles

Anyway, with not plan_with_no_div, we are filtering puzzles on the set of possible solutions, not just specific solutions. And this gives me an idea: what if we find puzzles that have only one solution?

different_plan(S, P) => best_plan_nondet(S, 4, P2), P2 != P.

valid24(Start, Plan) =>
+ , not different_plan(Start, Plan)

I tried this from 1..8 and got:

[1,2,7,7]
[1,3,4,6]
[1,6,6,8]
[3,3,8,8]

These happen to be some of the hardest 24 puzzles known, though not all of them. Note this is assuming that (X + Y) and (Y + X) are different solutions. If we say they're the same (by appending writing A = $(X + Y), X <= Y in our action) then we got a lot more puzzles, many of which are considered "easy". Other "hard" things we can look for include plans that require fractions:

plan_with_no_fractions(S, P) => 
  best_plan_nondet(S, 4, P)
  , not(
    member(X, P),
    round(apply(X)) =\= X
  ).

% insert `not plan...` in valid24 as usual

Finally, we could try seeing if a negative number is required:

plan_with_no_negatives(S, P) => 
  best_plan_nondet(S, 4, P)
  , not(
    member(X, P),
    apply(X) < 0
  ).

Interestingly this one returns no solutions, so you are never required to construct a negative number as part of a standard 24 puzzle.

You're walking down the street and need to pass someone going the opposite way. You take a step left, but they're thinking the same thing and take a step to their right, aka your left. You're still blocking each other. Then you take a step to the right, and they take a step to their left, and you're back to where you started. I've heard this called "walkwarding"

Let's model this in TLA+. TLA+ is a formal methods tool for finding bugs in complex software designs, most often involving concurrency. Two people trying to get past each other just also happens to be a concurrent system. A gentler introduction to TLA+'s capabilities is here, an in-depth guide teaching the language is here.

The spec

---- MODULE walkward ----
EXTENDS Integers

VARIABLES pos
vars == <<pos>>

Double equals defines a new operator, single equals is an equality check. <<pos>> is a sequence, aka array.

you == "you"
me == "me"
People == {you, me}

MaxPlace == 4

left == 0
right == 1

I've gotten into the habit of assigning string "symbols" to operators so that the compiler complains if I misspelled something. left and right are numbers so we can shift position with right - pos.

direction == [you |-> 1, me |-> -1]
goal == [you |-> MaxPlace, me |-> 1]

Init ==
  \* left-right, forward-backward
  pos = [you |-> [lr |-> left, fb |-> 1], me |-> [lr |-> left, fb |-> MaxPlace]]

direction, goal, and pos are "records", or hash tables with string keys. I can get my left-right position with pos.me.lr or pos["me"]["lr"] (or pos[me].lr, as me == "me").

Juke(person) ==
  pos' = [pos EXCEPT ![person].lr = right - @]

TLA+ breaks the world into a sequence of steps. In each step, pos is the value of pos in the current step and pos' is the value in the next step. The main outcome of this semantics is that we "assign" a new value to pos by declaring pos' equal to something. But the semantics also open up lots of cool tricks, like swapping two values with x' = y /\ y' = x.

TLA+ is a little weird about updating functions. To set f[x] = 3, you gotta write f' = [f EXCEPT ![x] = 3]. To make things a little easier, the rhs of a function update can contain @ for the old value. ![me].lr = right - @ is the same as right - pos[me].lr, so it swaps left and right.

("Juke" comes from here)

Move(person) ==
  LET new_pos == [pos[person] EXCEPT !.fb = @ + direction[person]]
  IN
    /\ pos[person].fb # goal[person]
    /\ \A p \in People: pos[p] # new_pos
    /\ pos' = [pos EXCEPT ![person] = new_pos]

The EXCEPT syntax can be used in regular definitions, too. This lets someone move one step in their goal direction unless they are at the goal or someone is already in that space. /\ means "and".

Next ==
  \E p \in People:
    \/ Move(p)
    \/ Juke(p)

I really like how TLA+ represents concurrency: "In each step, there is a person who either moves or jukes." It can take a few uses to really wrap your head around but it can express extraordinarily complicated distributed systems.

Spec == Init /\ [][Next]_vars

Liveness == <>(pos[me].fb = goal[me])
====

Spec is our specification: we start at Init and take a Next step every step.

Liveness is the generic term for "something good is guaranteed to happen", see here for more. <> means "eventually", so Liveness means "eventually my forward-backward position will be my goal". I could extend it to "both of us eventually reach out goal" but I think this is good enough for a demo.

Checking the spec

Four years ago, everybody in TLA+ used the toolbox. Now the community has collectively shifted over to using the VSCode extension.<sup>1</sup> VSCode requires we write a configuration file, which I will call walkward.cfg.

SPECIFICATION Spec
PROPERTY Liveness

I then check the model with the VSCode command TLA+: Check model with TLC. Unsurprisingly, it finds an error:

The reason it fails is "stuttering": I can get one step away from my goal and then just stop moving forever. We say the spec is unfair: it does not guarantee that if progress is always possible, progress will be made. If I want the spec to always make progress, I have to make some of the steps weakly fair.

+ Fairness == WF_vars(Next)

- Spec == Init /\ [][Next]_vars
+ Spec == Init /\ [][Next]_vars /\ Fairness

Now the spec is weakly fair, so someone will always do something. New error:

\* First six steps cut
7: <Move("me")>
pos = [you |-> [lr |-> 0, fb |-> 4], me |-> [lr |-> 1, fb |-> 2]]
8: <Juke("me")>
pos = [you |-> [lr |-> 0, fb |-> 4], me |-> [lr |-> 0, fb |-> 2]]
9: <Juke("me")> (back to state 7)

In this failure, I've successfully gotten past you, and then spend the rest of my life endlessly juking back and forth. The Next step keeps happening, so weak fairness is satisfied. What I actually want is for both my Move and my Juke to both be weakly fair independently of each other.

- Fairness == WF_vars(Next)
+ Fairness == WF_vars(Move(me)) /\ WF_vars(Juke(me))

If my liveness property also specified that you reached your goal, I could instead write \A p \in People: WF_vars(Move(p)) etc. I could also swap the \A with a \E to mean at least one of us is guaranteed to have fair actions, but not necessarily both of us.

New error:

3: <Move("me")>
pos = [you |-> [lr |-> 0, fb |-> 2], me |-> [lr |-> 0, fb |-> 3]]
4: <Juke("you")>
pos = [you |-> [lr |-> 1, fb |-> 2], me |-> [lr |-> 0, fb |-> 3]]
5: <Juke("me")>
pos = [you |-> [lr |-> 1, fb |-> 2], me |-> [lr |-> 1, fb |-> 3]]
6: <Juke("me")>
pos = [you |-> [lr |-> 1, fb |-> 2], me |-> [lr |-> 0, fb |-> 3]]
7: <Juke("you")> (back to state 3)

Now we're getting somewhere! This is the original walkwarding situation we wanted to capture. We're in each others way, then you juke, but before either of us can move you juke, then we both juke back. We can repeat this forever, trapped in a social hell.

Wait, but doesn't WF(Move(me)) guarantee I will eventually move? Yes, but only if a move is permanently available. In this case, it's not permanently available, because every couple of steps it's made temporarily unavailable.

How do I fix this? I can't add a rule saying that we only juke if we're blocked, because the whole point of walkwarding is that we're not coordinated. In the real world, walkwarding can go on for agonizing seconds. What I can do instead is say that Liveness holds as long as Move is strongly fair. Unlike weak fairness, strong fairness guarantees something happens if it keeps becoming possible, even with interruptions.

Liveness == 
+  SF_vars(Move(me)) => 
    <>(pos[me].fb = goal[me])

This makes the spec pass. Even if we weave back and forth for five minutes, as long as we eventually pass each other, I will reach my goal. Note we could also by making Move in Fairness strongly fair, which is preferable if we have a lot of different liveness properties to check.

A small exercise for the reader

There is a presumed invariant that is violated. Identify what it is, write it as a property in TLA+, and show the spec violates it. Then fix it.

Answer (in rot13): Gur vainevnag vf "ab gjb crbcyr ner va gur rknpg fnzr ybpngvba". Zbir thnenagrrf guvf ohg Whxr qbrf abg.

More TLA+ Exercises

I've started work on an exercises repo. There's only a handful of specific problems now but I'm planning on adding more over the summer.

I started writing this early last week but Real Life Stuff happened and now you're getting the first-draft late this week. Warning, unedited thoughts ahead!

New Logic for Programmers release!

v0.9 is out! This is a big release, with a new cover design, several rewritten chapters, online code samples and much more. See the full release notes at the changelog page, and get the book here!

Write the cleverest code you possibly can

There are millions of articles online about how programmers should not write "clever" code, and instead write simple, maintainable code that everybody understands. Sometimes the example of "clever" code looks like this (src):

# Python

p=n=1
exec("p*=n*n;n+=1;"*~-int(input()))
print(p%n)

This is code-golfing, the sport of writing the most concise code possible. Obviously you shouldn't run this in production for the same reason you shouldn't eat dinner off a Rembrandt.

Other times the example looks like this:

def is_prime(x):
    if x == 1:
        return False
    return all([x%n != 0 for n in range(2, x)])

This is "clever" because it uses a single list comprehension, as opposed to a "simple" for loop. Yes, "list comprehensions are too clever" is something I've read in one of these articles.

I've also talked to people who think that datatypes besides lists and hashmaps are too clever to use, that most optimizations are too clever to bother with, and even that functions and classes are too clever and code should be a linear script.<sup>1</sup>. Clever code is anything using features or domain concepts we don't understand. Something that seems unbearably clever to me might be utterly mundane for you, and vice versa.

How do we make something utterly mundane? By using it and working at the boundaries of our skills. Almost everything I'm "good at" comes from banging my head against it more than is healthy. That suggests a really good reason to write clever code: it's an excellent form of purposeful practice. Writing clever code forces us to code outside of our comfort zone, developing our skills as software engineers.

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you [will get excellent debugging practice at exactly the right level required to push your skills as a software engineer] — Brian Kernighan, probably

There are other benefits, too, but first let's kill the elephant in the room:<sup>2</sup>

Don't commit clever code

I am proposing writing clever code as a means of practice. Being at work is a job with coworkers who will not appreciate if your code is too clever. Similarly, don't use too many innovative technologies. Don't put anything in production you are uncomfortable with.

We can still responsibly write clever code at work, though:

1. Solve a problem in both a simple and a clever way, and then only commit the simple way. This works well for small scale problems where trying the "clever way" only takes a few minutes.
2. Write our personal tools cleverly. I'm a big believer of the idea that most programmers would benefit from writing more scripts and support code customized to their particular work environment. This is a great place to practice new techniques, languages, etc.
3. If clever code is absolutely the best way to solve a problem, then commit it with extensive documentation explaining how it works and why it's preferable to simpler solutions. Bonus: this potentially helps the whole team upskill.

Writing clever code...

...teaches simple solutions

Usually, code that's called too clever composes several powerful features together — the "not a single list comprehension or function" people are the exception. Josh Comeau's "don't write clever code" article gives this example of "too clever":

const extractDataFromResponse = (response) => {
  const [Component, props] = response;

  const resultsEntries = Object.entries({ Component, props });
  const assignIfValueTruthy = (o, [k, v]) => (v
    ? { ...o, [k]: v }
    : o
  );

  return resultsEntries.reduce(assignIfValueTruthy, {});
}

What makes this "clever"? I count eight language features composed together: entries, argument unpacking, implicit objects, splats, ternaries, higher-order functions, and reductions. Would code that used only one or two of these features still be "clever"? I don't think so. These features exist for a reason, and oftentimes they make code simpler than not using them.

We can, of course, learn these features one at a time. Writing the clever version (but not committing it) gives us practice with all eight at once and also with how they compose together. That knowledge comes in handy when we want to apply a single one of the ideas.

I've recently had to do a bit of pandas for a project. Whenever I have to do a new analysis, I try to write it as a single chain of transformations, and then as a more balanced set of updates.

...helps us master concepts

Even if the composite parts of a "clever" solution aren't by themselves useful, it still makes us better at the overall language, and that's inherently valuable. A few years ago I wrote Crimes with Python's Pattern Matching. It involves writing horrible code like this:

from abc import ABC

class NotIterable(ABC):

    @classmethod
    def __subclasshook__(cls, C):
        return not hasattr(C, "__iter__")

def f(x):
    match x:
        case NotIterable():
            print(f"{x} is not iterable")
        case _:
            print(f"{x} is iterable")

if __name__ == "__main__":
    f(10)
    f("string")
    f([1, 2, 3])

This composes Python match statements, which are broadly useful, and abstract base classes, which are incredibly niche. But even if I never use ABCs in real production code, it helped me understand Python's match semantics and Method Resolution Order better.

...prepares us for necessity

Sometimes the clever way is the only way. Maybe we need something faster than the simplest solution. Maybe we are working with constrained tools or frameworks that demand cleverness. Peter Norvig argued that design patterns compensate for missing language features. I'd argue that cleverness is another means of compensating: if our tools don't have an easy way to do something, we need to find a clever way.

You see this a lot in formal methods like TLA+. Need to check a hyperproperty? Cast your state space to a directed graph. Need to compose ten specifications together? Combine refinements with state machines. Most difficult problems have a "clever" solution. The real problem is that clever solutions have a skill floor. If normal use of the tool is at difficult 3 out of 10, then basic clever solutions are at 5 out of 10, and it's hard to jump those two steps in the moment you need the cleverness.

But if you've practiced with writing overly clever code, you're used to working at a 7 out of 10 level in short bursts, and then you can "drop down" to 5/10. I don't know if that makes too much sense, but I see it happen a lot in practice.

...builds comradery

On a few occasions, after getting a pull request merged, I pulled the reviewer over and said "check out this horrible way of doing the same thing". I find that as long as people know they're not going to be subjected to a clever solution in production, they enjoy seeing it!

Next week's newsletter will probably also be late, after that we should be back to a regular schedule for the rest of the summer.

Our company recently announced a fundraise. We were grateful for all the community support, but the Internet also raised a few of its collective eyebrows, wondering whether this meant the dreaded “enshittification” was coming next.

That word describes a very real pattern we’ve all seen before: products start great, grow fast, and then slowly become worse as the people running them trade user love for short-term revenue.

It’s a topic I find genuinely fascinating, and I've seen the downward spiral firsthand at companies I once admired. So I want to talk about why this happens, and more importantly, why it won't happen to us. That's big talk, I know. But it's a promise I'm happy for people to hold us to.

What is enshittification?

The term "enshittification" was first popularized in a blog post by Corey Doctorow, who put a catchy name to an effect we've all experienced. Software starts off good, then goes bad. How? Why?

Enshittification proposes not just a name, but a mechanism. First, a product is well loved and gains in popularity, market share, and revenue. In fact, it gets so popular that it starts to defeat competitors. Eventually, it's the primary product in the space: a monopoly, or as close as you can get. And then, suddenly, the owners, who are Capitalists, have their evil nature finally revealed and they exploit that monopoly to raise prices and make the product worse, so the captive customers all have to pay more. Quality doesn't matter anymore, only exploitation.

I agree with most of that thesis. I think Doctorow has that mechanism mostly right. But, there's one thing that doesn't add up for me:

Enshittification is not a success mechanism.

I can't think of any examples of companies that, in real life, enshittified because they were successful. What I've seen is companies that made their product worse because they were... scared.

A company that's growing fast can afford to be optimistic. They create a positive feedback loop: more user love, more word of mouth, more users, more money, more product improvements, more user love, and so on. Everyone in the company can align around that positive feedback loop. It's a beautiful thing. It's also fragile: miss a beat and it flattens out, and soon it's a downward spiral instead of an upward one.

So, if I were, hypothetically, running a company, I think I would be pretty hesitant to deliberately sacrifice any part of that positive feedback loop, the loop I and the whole company spent so much time and energy building, to see if I can grow faster. User love? Nah, I'm sure we'll be fine, look how much money and how many users we have! Time to switch strategies!

Why would I do that? Switching strategies is always a tremendous risk. When you switch strategies, it's triggered by passing a threshold, where something fundamental changes, and your old strategy becomes wrong.

Threshold moments and control

In Saint John, New Brunswick, there's a river that flows one direction at high tide, and the other way at low tide. Four times a day, gravity equalizes, then crosses a threshold to gently start pulling the other way, then accelerates. What doesn't happen is a rapidly flowing river in one direction "suddenly" shifts to rapidly flowing the other way. Yes, there's an instant where the limit from the left is positive and the limit from the right is negative. But you can see that threshold coming. It's predictable.

In my experience, for a company or a product, there are two kinds of thresholds like this, that build up slowly and then when crossed, create a sudden flow change.

The first one is control: if the visionaries in charge lose control, chances are high that their replacements won't "get it."

The new people didn't build the underlying feedback loop, and so they don't realize how fragile it is. There are lots of reasons for a change in control: financial mismanagement, boards of directors, hostile takeovers.

The worst one is temptation. Being a founder is, well, it actually sucks. It's oddly like being repeatedly punched in the face. When I look back at my career, I guess I'm surprised by how few times per day it feels like I was punched in the face. But, the constant face punching gets to you after a while. Once you've established a great product, and amazing customer love, and lots of money, and an upward spiral, isn't your creation strong enough yet? Can't you step back and let the professionals just run it, confident that they won't kill the golden goose?

Empirically, mostly no, you can't. Actually the success rate of control changes, for well loved products, is abysmal.

The saturation trap

The second trigger of a flow change is comes from outside: saturation. Every successful product, at some point, reaches approximately all the users it's ever going to reach. Before that, you can watch its exponential growth rate slow down: the infamous S-curve of product adoption.

Saturation can lead us back to control change: the founders get frustrated and back out, or the board ousts them and puts in "real business people" who know how to get growth going again. Generally that doesn't work. Modern VCs consider founder replacement a truly desperate move. Maybe a last-ditch effort to boost short term numbers in preparation for an acquisition, if you're lucky.

But sometimes the leaders stay on despite saturation, and they try on their own to make things better. Sometimes that does work. Actually, it's kind of amazing how often it seems to work. Among successful companies, it's rare to find one that sustained hypergrowth, nonstop, without suffering through one of these dangerous periods.

(That's called survivorship bias. All companies have dangerous periods. The successful ones surivived them. But of those survivors, suspiciously few are ones that replaced their founders.)

If you saturate and can't recover - either by growing more in a big-enough current market, or by finding new markets to expand into - then the best you can hope for is for your upward spiral to mature gently into decelerating growth. If so, and you're a buddhist, then you hire less, you optimize margins a bit, you resign yourself to being About This Rich And I Guess That's All But It's Not So Bad.

The devil's bargain

Alas, very few people reach that state of zen. Especially the kind of ambitious people who were able to get that far in the first place. If you can't accept saturation and you can't beat saturation, then you're down to two choices: step away and let the new owners enshittify it, hopefully slowly. Or take the devil's bargain: enshittify it yourself.

I would not recommend the latter. If you're a founder and you find yourself in that position, honestly, you won't enjoy doing it and you probably aren't even good at it and it's getting enshittified either way. Let someone else do the job.

Defenses against enshittification

Okay, maybe that section was not as uplifting as we might have hoped. I've gotta be honest with you here. Doctorow is, after all, mostly right. This does happen all the time.

Most founders aren't perfect for every stage of growth. Most product owners stumble. Most markets saturate. Most VCs get board control pretty early on and want hypergrowth or bust. In tech, a lot of the time, if you're choosing a product or company to join, that kind of company is all you can get.

As a founder, maybe you're okay with growing slowly. Then some copycat shows up, steals your idea, grows super fast, squeezes you out along with your moral high ground, and then runs headlong into all the same saturation problems as everyone else. Tech incentives are awful.

But, it's not a lost cause. There are companies (and open source projects) that keep a good thing going, for decades or more. What do they have in common?

• An expansive vision that's not about money, and which opens you up to lots of users. A big addressable market means you don't have to worry about saturation for a long time, even at hypergrowth speeds. Google certainly never had an incentive to make Google Search worse.

  (Update 2025-06-14: A few people disputed that last bit. Okay. Perhaps Google has ccasionally responded to what they thought were incentives to make search worse -- I wasn't there, I don't know -- but it seems clear in retrospect that when search gets worse, Google does worse. So I'll stick to my claim that their true incentives are to keep improving.)

• Keep control. It's easy to lose control of a project or company at any point. If you stumble, and you don't have a backup plan, and there's someone waiting to jump on your mistake, then it's over. Too many companies "bet it all" on nonstop hypergrowth and don't have any way back have no room in the budget, if results slow down even temporarily.

  Stories abound of companies that scraped close to bankruptcy before finally pulling through. But far more companies scraped close to bankruptcy and then went bankrupt. Those companies are forgotten. Avoid it.

• Track your data. Part of control is predictability. If you know how big your market is, and you monitor your growth carefully, you can detect incoming saturation years before it happens. Knowing the telltale shape of each part of that S-curve is a superpower. If you can see the future, you can prevent your own future mistakes.

• Believe in competition. Google used to have this saying they lived by: "the competition is only a click away." That was excellent framing, because it was true, and it will remain true even if Google captures 99% of the search market. The key is to cultivate a healthy fear of competing products, not of your investors or the end of hypergrowth. Enshittification helps your competitors. That would be dumb.

  (And don't cheat by using lock-in to make competitors not, anymore, "only a click away." That's missing the whole point!)

• Inoculate yourself. If you have to, create your own competition. Linus Torvalds, the creator of the Linux kernel, famously also created Git, the greatest tool for forking (and maybe merging) open source projects that has ever existed. And then he said, this is my fork, the Linus fork; use it if you want; use someone else's if you want; and now if I want to win, I have to make mine the best. Git was created back in 2005, twenty years ago. To this day, Linus's fork is still the central one.

If you combine these defenses, you can be safe from the decline that others tell you is inevitable. If you look around for examples, you'll find that this does actually work. You won't be the first. You'll just be rare.

Side note: Things that aren't enshittification

I often see people worry about enshittification that isn't. They might be good or bad, wise or unwise, but that's a different topic. Tools aren't inherently good or evil. They're just tools.

1. "Helpfulness." There's a fine line between "telling users about this cool new feature we built" in the spirit of helping them, and "pestering users about this cool new feature we built" (typically a misguided AI implementation) to improve some quarterly KPI. Sometimes it's hard to see where that line is. But when you've crossed it, you know.

   Are you trying to help a user do what they want to do, or are you trying to get them to do what you want them to do?

   Look into your heart. Avoid the second one. I know you know how. Or you knew how, once. Remember what that feels like.

2. Charging money for your product. Charging money is okay. Get serious. Companies have to stay in business.

   That said, I personally really revile the "we'll make it free for now and we'll start charging for the exact same thing later" strategy. Keep your promises.

   I'm pretty sure nobody but drug dealers breaks those promises on purpose. But, again, desperation is a powerful motivator. Growth slowing down? Costs way higher than expected? Time to capture some of that value we were giving away for free!

   In retrospect, that's a bait-and-switch, but most founders never planned it that way. They just didn't do the math up front, or they were too naive to know they would have to. And then they had to.

   Famously, Dropbox had a "free forever" plan that provided a certain amount of free storage. What they didn't count on was abandoned accounts, accumulating every year, with stored stuff they could never delete. Even if a very good fixed fraction of users each year upgraded to a paid plan, all the ones that didn't, kept piling up... year after year... after year... until they had to start deleting old free accounts and the data in them. A similar story happened with Docker, which used to host unlimited container downloads for free. In hindsight that was mathematically unsustainable. Success guaranteed failure.

   Do the math up front. If you're not sure, find someone who can.

3. Value pricing. (ie. charging different prices to different people.) It's okay to charge money. It's even okay to charge money to some kinds of people (say, corporate users) and not others. It's also okay to charge money for an almost-the-same-but-slightly-better product. It's okay to charge money for support for your open source tool (though I stay away from that; it incentivizes you to make the product worse).

   It's even okay to charge immense amounts of money for a commercial product that's barely better than your open source one! Or for a part of your product that costs you almost nothing.

   But, you have to do the rest of the work. Make sure the reason your users don't switch away is that you're the best, not that you have the best lock-in. Yeah, I'm talking to you, cloud egress fees.

4. Copying competitors. It's okay to copy features from competitors. It's okay to position yourself against competitors. It's okay to win customers away from competitors. But it's not okay to lie.

5. Bugs. It's okay to fix bugs. It's okay to decide not to fix bugs; you'll have to sometimes, anyway. It's okay to take out technical debt. It's okay to pay off technical debt. It's okay to let technical debt languish forever.

6. Backward incompatible changes. It's dumb to release a new version that breaks backward compatibility with your old version. It's tempting. It annoys your users. But it's not enshittification for the simple reason that it's phenomenally ineffective at maintaining or exploiting a monopoly, which is what enshittification is supposed to be about. You know who's good at monopolies? Intel and Microsoft. They don't break old versions.

Enshittification is real, and tragic. But let's protect a useful term and its definition! Those things aren't it.

Epilogue: a special note to founders

If you're a founder or a product owner, I hope all this helps. I'm sad to say, you have a lot of potential pitfalls in your future. But, remember that they're only potential pitfalls. Not everyone falls into them.

Plan ahead. Remember where you came from. Keep your integrity. Do your best.

I will too.

Wikipedia is coming up on its 25th birthday, and that would not have been possible without the Wikimedia technical volunteer community. Supporting technical volunteers is crucial to carrying forward Wikimedia’s free knowledge mission for generations to come. In line with this commitment, the Foundation is turning its attention to an important area of developer support—the Wikimedia web (HTTP) APIs. 

Both Wikimedia and the Internet have changed a lot over the last 25 years. Patterns that are now ubiquitous standards either didn’t exist or were still in their infancy as the first APIs allowing developers to extend features and automate tasks on Wikimedia projects emerged. In fact, the term “representational state transfer”, better known today as the REST framework, was first coined in 2000, just months before the very first Wikipedia post was published, and only 6 years before the Action API was introduced. Because we preceded what have since become industry standards, our most powerful and comprehensive API solution, the Action API, sticks out as being unlike other APIs – but for good reason, if you understand the history.

Wikimedia APIs are used within Foundation-authored features and by volunteer developers. A common sentiment surfaced through the recent API Listening Tour conducted with a mix of volunteers and Foundation staff is “Wikimedia APIs are great, once you know what you’re doing.” New developers first entering the Wikimedia community face a steep learning curve when trying to onboard due to unfamiliar technologies and complex APIs that may require a deep understanding of the underlying Wikimedia systems and processes. While recognizing the power, flexibility, and mission-critical value that developers created using the existing API solutions, we want to make it easier for developers to make more meaningful contributions faster. We have no plans to deprecate the Action API nor treat it as ‘legacy’. Instead, we hope to make it easier and more approachable for both new and experienced developers to use. We also aim to expand REST coverage to better serve developers who are more comfortable working in those structures.

We are focused on simplifying, modernizing, and standardizing Wikimedia API offerings as part of the Responsible Use of Infrastructure objective in the FY25-26 Annual Plan (see: the WE5.2 key result). Focusing on common infrastructure that encourages responsible use allows us to continue to prioritize reliable, free access to knowledge for the technical volunteer community, as well as the readers and contributors they support. Investing in our APIs and the developer experiences surrounding them will ensure a healthy technical community for years to come. To achieve these objectives, we see three main areas for improving the sustainability of our API offering: simplification, documentation, and communication.

Simplification

To reduce maintenance costs and ensure a seamless developer experience, we are simplifying our API infrastructure and bringing greater consistency across all APIs. Decades of organic growth without centralized API governance led to fragmented, bespoke implementations that now hinder technical agility and standardization. Beyond that, maintaining services is not free; we are paying for duplicative infrastructure costs, some of which are scaling directly with the amount of scraper traffic hitting our services.

In light of the above, we will focus on transitioning at least 70% of our public endpoints to common API infrastructure (see the WE 5.2 key result). Common infrastructure makes it easier to maintain and roll out changes across our APIs, in addition to empowering API authors to move faster. Instead of expecting API authors to build and manage their own solutions for things like routing and rate limiting, we will create centralized tools and processes that make it easier to follow the “golden path” of recommended standards. That will allow centralized governance mechanisms to drive more consistent and sustainable end-user experiences, while enabling flexible, federated API ownership. 

An example of simplified internal infrastructure will be introducing a common API Gateway for handling and routing all Wikimedia API requests. Our approach will start as an “invisible gateway” or proxy, with no changes to URL structure or functional behavior for any existing APIs. Centralizing API traffic will make observability across APIs easier, allowing us to make better data-driven decisions. We will use this data to inform endpoint deprecation and versioning, prioritize human and mission-oriented access first, and ultimately provide better support to our developer community.  

Centralized management and traffic identification will also allow us to have more consistent and transparent enforcement of our API policies. API policy enforcement enables us to protect our infrastructure and ensure continued access for all. Once API traffic is rerouted through a centralized gateway, we will explore simplifying options for developer identification mechanisms and standardizing how rate limits and other API access controls are applied. The goal is to make it easier for all developers to know exactly what is expected and what limitations apply.

As we update our API usage policies and developer requirements, we will avoid breaking existing community tools as much as possible. We will continue offering low-friction entry points for volunteer developers experimenting with new ideas, lightly exploring data, or learning to build in the Wikimedia ecosystem. But we must balance support for community creativity and innovation with the need to reduce abuse, such as scraping, Denial of Service (DoS) attacks, and other harmful activities. While open, unauthenticated API access for everyone will continue, we will need to make adjustments. To reduce the likelihood and impact of abuse, we may apply stricter rate limits to unauthenticated traffic and more consistent authentication requirements to better match our documented API policy, Robot policy, and API etiquette guidelines, as well as consolidate per-API access guidelines to reduce the likelihood and impact of abuse.

To continue supporting Wikimedia’s technical volunteer community and minimize disruption to existing tools, community developers will have simple ways to identify themselves and receive higher limits or other access privileges. In many cases, this won’t require additional steps. For example, instead of universally requiring new access tokens or authentication methods, we plan to use IP ranges from Wikimedia Cloud Services (WMCS) and User-Agent headers to grant elevated privileges to trusted community tools, approved bots, and research projects. 

Documentation

It is essential for any API to enable developers to self-serve their use cases through clear, consistent, and modern documentation experiences. However, Wikimedia API documentation is frequently spread across multiple wiki projects, generated sites, and communication channels, which can make it difficult for developers to find the information they need, when they need it. 

To address this, we are working towards a top-requested item coming out of the 2024 developer satisfaction survey: OpenAPI specs and interactive sandboxes for all of our APIs (including conducting experiments to see if we can use OpenAPI to describe the Action API). The MediaWiki Interfaces team began addressing this request through the REST Sandbox, which we released to a limited number of small Wikipedia projects on March 31, 2025. Our implementation approach allows us to generate an OpenAPI specification, which we then use to power a SwaggerUI sandbox. We are also using the OpenAPI specs to automatically validate our endpoints as part of our automated deployment testing, which helps ensure that the generated documentation always matches the actual endpoint behavior. 

In addition, the generated OpenAPI spec offers translation support (powered by Translatewiki) for critical and contextual information like endpoint and parameter descriptions. We believe this is a more equitable approach to API documentation for developers who don’t have English as their preferred language. In the coming year, we plan to transition from Swagger UI to a custom Codex implementation for our sandbox experiences, which will enable full translation support for sandbox UI labels and navigation, as well as a more consistent look and feel for Wikimedia developers. We will also expand coverage for OpenAPI specs and sandbox experiences by introducing repeatable patterns for API authors to publish their specs to a single location where developers can easily browse, learn, and make test calls across all Wikimedia API offerings. 

Communication

When new endpoints are released or breaking changes are required, we need a better way to keep developers informed. As information is shared through different channels, it can become challenging to keep track of the full picture. Over the next year, we will address this on a few fronts. 

First, from a technical change management perspective, we will introduce a centralized API changelog. The changelog will summarize new endpoints, as well as new versions, planned deprecations, and minor changes such as new optional parameters. This will help developers with troubleshooting, as well as help them to more easily understand and monitor the changes happening across the Wikimedia APIs.

In addition to the changelog, we remain committed to consistently communicating changes early and often. As another step towards this commitment, we will provide migration guides and, where needed, provide direct communication channels for developers impacted by the changes to help guarantee a smooth transition. Recognizing that the Wikimedia technical community is split across many smaller communities both on and off-wiki, we will share updates in the largest off-wiki communities, but we will need volunteer support in directing questions and feedback to the right on-wiki pages in various languages. We will also work with communities to make their purpose and audience clearer for new developers so they can more easily get support when they need it and join the discussion with fellow technical contributors. 

Over the next few months, we will also launch a new API beta program, where developers are invited to interact with new endpoints and provide feedback before the capabilities are locked into a long-term stable version. Introducing new patterns through a beta program will allow developers to directly shape the future of the Wikimedia APIs to better suit their needs. To demonstrate this pattern, we will start with changes to MediaWiki REST APIs, including introducing API modularization and consistent structures. 

What’s Next

We are still in the early stages – we are just making the first steps on the journey to a unified API product offering. But we hope that by this time next year, we will be running towards it together. Your involvement and insights can help us shape a future that better serves the technical volunteers behind our knowledge mission. To keep you informed, we will continue to post updates on mailing lists, Diff, TechBlog, and other technical volunteer communication channels. We also invite you to stay actively engaged: share your thoughts on the WE5 objective in the annual plan, ask questions on the related discussion pages, review slides from the Future of Wikimedia APIs session we conducted at the Wikimedia Hackathon, volunteer for upcoming Listening Tour topics, or come talk to us at upcoming events such as Wikimania Nairobi. 

Technical volunteers play an essential role in the growth and evolution of Wikipedia, as well as all other Wikimedia projects. Together, we can make a better experience for developers who can’t remember life before Wikipedia, and make sure that the next generation doesn’t have to live without it. Here’s to another 25 years! 

The Campaigns team at WMF has released two features that allow organizers to promote events and WikiProjects on the wikis: Invitation Lists and Collaboration List. These two tools are a part of the CampaignEvents extension, which is available on many wikis.

Invitation Lists

Product overview

Invitation Lists allows organizers to generate a list of people to invite to their WikiProjects, events, or other collaborative activities. It can be accessed by going to Special:GenerateInvitationList, if a wiki has the CampaignEvents extension enabled. You can watch this video demo to see how it works.

It works by looking at a list of articles that an organizer plans to focus on during an activity and then finding users to invite based on the following criteria: the bytes they contributed to the articles, the number of edits they made to the articles, their overall edit count on the wikis, and how recently they have edited the wikis. This makes it easier for organizers to invite people who are already interested in the activity’s topics, hence increasing the likelihood of participation.

With this work, we hope to empower organizers to seek out new audiences. We also hope to highlight the important work done by editors, who may be inspired or touched to receive an invitation to an activity based on their work. However, if someone does not want to receive invitations, they can opt out of being included in Invitation Lists via Preferences.

Technical overview

The “Invitation Lists” feature is part of the CampaignEvents extension for MediaWiki, designed to assist event organizers in identifying and reaching out to potential participants based on their editing activity.

Access and Permissions

• Special Pages: The feature introduces two special pages:
  • Special:GenerateInvitationList: Allows organizers to create new invitation lists.
  • Special:InvitationList: Displays the generated list of recommended invitees.
• User Rights: Access to these pages is restricted to users with the event-organizer right, ensuring that only authorized individuals can generate and view invitation lists.
  

Invitation List Generation Process

1. Input Parameters:
   • List Name: Organizers provide a name for the invitation list.
   • Target Articles: A list of up to 300 articles relevant to the event’s theme.
     •  The articles will need to be on the wiki of the Invitation List.
   • Event Page Link: Optionally, a link to the event’s registration page can be included.
     
2. Data Collection:
   • The system analyzes the specified articles to identify contributors.
   • For each contributor, it gathers metrics such as:
     • Bytes Added: The total number of bytes the user has added to the articles.
     • Edit Count: The number of edits made by the user on the specified articles.
     • Overall Edit Count: The user’s total edit count across the wiki.
     • Recent Activity: The recency of the user’s edits on the wiki.
       
3. Scoring and Ranking:
   • Contributors are scored based on the collected metrics.
   • The scoring algorithm assigns weights to each metric to calculate a composite score for each user.
   • Users are then ranked and categorized into:
     • Highly Recommended to Invite: Top contributors with high relevance and recent activity.
     • Recommended to Invite: Contributors with moderate relevance and activity.
4. Output:
   • The generated invitation list is displayed on the Special:InvitationList page.
   • Each listed user includes a link to their contributions page, facilitating further review by the organizer.

Technical Implementation Details

• Backend Processing:
  • The extension utilizes MediaWiki’s job queue system to handle the processing of invitation lists asynchronously, ensuring that the generation process does not impact the performance of the wiki.
  • Jobs are queued upon submission of the article list and processed in the background.
  • The articles will need to be on the wiki of the Invitation List, and they can add a maximum of 300 articles.
    
• Data Retrieval:
  • The extension interfaces with MediaWiki’s revision and user tables to extract the necessary contribution data.
  • Efficient querying and indexing strategies are employed to handle large datasets and ensure timely processing.
    
• User Preferences and Privacy:
  • Users have the option to opt out of being included in invitation lists via their preferences.
  • The extension respects these preferences by excluding opted-out users from the generated lists.
    
• Integration with Event Registration:
  • If an event page link is provided, the invitation list can be associated with the event’s registration data. This way, we can link their invitation data to their event registration data.

Collaboration List

Product overview

The Collaboration List is a list of events and WikiProjects. It can be accessed by going to SpecialːAllEvents,  if a wiki has the CampaignEvents extension enabled.

The Collaboration List has two tabs: “Events” and “Communities.” The Events tab is a global, automated list of all events that use Event Registration. It also has search filters, so you can find events by start and end dates, meeting type (i.e., online, in person, or hybrid), event topic, event wikis, and by keyword searches. You can also find events that are both ongoing (i.e., started before but continue within the selected date range) and upcoming (i.e., events that start within the selected date range).

The Communities tab provides a list of WikiProjects on the local wiki. The WikiProject list is generated by using Wikidata, and it includes: WikiProject name, description, a link to the WikiProject page, and a link to the Wikidata item for the WikiProject. We aim to produce a symbiotic relationship with WikiProjects, in which people can find WikiProjects that interest them, and they can also enhance the Wikidata items for those projects, which in turn improves our project.

Additionally, you can embed the Collaboration List on any wiki page, if the CampaignEvents extension is enabled on that wiki. To do this, you transclude the Collaboration List on a wiki page. You can also choose to customize the Collaboration List through URL parameters, if you want. For example, you can choose to only display a certain number of events or to add formatting. You can read more about this on Help:Extension:CampaignEvents/Collaboration list/Transclusion.

With the Collaboration List, we hope to make it easier for people to find events and WikiProjects that interest them, so more people can find community and make impactful contributions on the wikis together.

Technical Overview: Events Tab of Collaboration List

• Purpose: Displays a global list of events across all participating wikis.
• Data Source: Event data stored centrally in Wikimedia’s X1 database cluster.
• Displayed Information:
  • Event name and description
  • Event dates (start and end)
  • Event type (online, in-person, hybrid)
  • Associated wikis and event topics
• Search and Filters:
  • Date range (start/end)
  • Meeting type (online, in-person, hybrid)
  • Event topics and wikis
  • Keyword search
  • Ongoing and upcoming event filtering
    
• Technical Implementation:
  • The CampaignEvents extension retrieves event data directly from centralized tables within the X1 cluster.
  • Efficient SQL queries and indexing optimize performance for cross-wiki data retrieval.

This implementation ensures quick access and easy discoverability of events from across Wikimedia projects.

Technical Overview: Communities Tab of Collaboration List

• Purpose: Displays a list of local WikiProjects on the wiki.
• Data Source: Dynamically retrieved from Wikidata via the Wikidata Query Service (WDQS).
• Displayed Information:
  • WikiProject name
  • Description from Wikidata
  • Link to the local WikiProject page
  • Link to the Wikidata item
    
• Performance Optimization:
  • Query results from WDQS are cached locally using MediaWiki’s caching mechanisms (WANObjectCache).
  • Cache reduces repeated queries and ensures quick loading times.
    
• Technical Implementation:
  • The WikimediaCampaignEvents extension retrieves data via SPARQL from WDQS.
  • The CampaignEvents extension renders the data on Special:AllEvents under the Communities tab.
    
• Extension Communication:
  • The extensions communicate using MediaWiki’s hook system. The WikimediaCampaignEvents extension provides WikiProject data to the CampaignEvents extension through hook implementations.

This structure enables efficient collaboration between extensions, ensuring clear responsibilities, optimized performance, and simplified discoverability of WikiProjects.

Wikimedia Cloud VPS is a service offered by the Wikimedia Foundation, built using OpenStack and managed by the Wikimedia Cloud Services team. It provides cloud computing resources for projects related to the Wikimedia movement, including virtual machines, databases, storage, Kubernetes, and DNS.

A few weeks ago, in April 2025, we were finally able to introduce IPv6 to the cloud virtual network, enhancing the platform’s scalability, security, and future-readiness. This is a major milestone, many years in the making, and serves as an excellent point to take a moment to reflect on the road that got us here. There were definitely a number of challenges that needed to be addressed before we could get into IPv6. This post covers the journey to this implementation.

The Wikimedia Foundation was an early adopter of the OpenStack technology, and the original OpenStack deployment in the organization dates back to 2011. At that time, IPv6 support was still nascent and had limited implementation across various OpenStack components. In 2012, the Wikimedia cloud users formally requested IPv6 support.

When Cloud VPS was originally deployed, we had set up the network following some of the upstream-recommended patterns:

• nova-networks as the engine in charge of the software-defined virtual network
• using a flat network topology – all virtual machines would share the same network
• using a physical VLAN in the datacenter
• using Linux bridges to make this physical datacenter VLAN available to virtual machines
• using a single virtual router as the edge network gateway, also executing a global egress NAT – barring some exceptions, using what was called “dmz_cidr” mechanism

In order for us to be able to implement IPv6 in a way that aligned with our architectural goals and operational requirements, pretty much all the elements in this list would need to change. First of all, we needed to migrate from nova-networks into Neutron, a migration effort that started in 2017. Neutron was the more modern component to implement software-defined networks in OpenStack. To facilitate this transition, we made the strategic decision to backport certain functionalities from nova-networks into Neutron, specifically the “dmz_cidr” mechanism and some egress NAT capabilities.

Once in Neutron, we started to think about IPv6. In 2018 there was an initial attempt to decide on the network CIDR allocations that Wikimedia Cloud Services would have. This initiative encountered unforeseen challenges and was subsequently put on hold. We focused on removing the previously backported nova-networks patches from Neutron.

Between 2020 and 2021, we initiated another significant network refresh. We were able to introduce the cloudgw project, as part of a larger effort to rework the Cloud VPS edge network. The new edge routers allowed us to drop all the custom backported patches we had in Neutron from the nova-networks era, unblocking further progress. Worth mentioning that the cloudgw router would use nftables as firewalling and NAT engine.

A pivotal decision in 2022 was to expose the OpenStack APIs to the internet, which crucially enabled infrastructure management via OpenTofu. This was key in the IPv6 rollout as will be explained later. Before this, management was limited to Horizon – the OpenStack graphical interface – or the command-line interface accessible only from internal control servers.

Later, in 2023, following the OpenStack project’s announcement of the deprecation of the neutron-linuxbridge-agent, we began to seriously consider migrating to the neutron-openvswitch-agent. This transition would, in turn, simplify the enablement of “tenant networks” – a feature allowing each OpenStack project to define its own isolated network, rather than all virtual machines sharing a single flat network.

Once we replaced neutron-linuxbridge-agent with neutron-openvswitch-agent, we were ready to migrate virtual machines to VXLAN. Demonstrating perseverance, we decided to execute the VXLAN migration in conjunction with the IPv6 rollout.

We prepared and tested several things, including the rework of the edge routing to be based on BGP/OSPF instead of static routing. In 2024 we were ready for the initial attempt to deploy IPv6, which failed for unknown reasons. There was a full network outage and we immediately reverted the changes. This quick rollback was feasible due to our adoption of OpenTofu: deploying IPv6 had been reduced to a single code change within our repository.

We started an investigation, corrected a few issues, and increased our network functional testing coverage before trying again. One of the problems we discovered was that Neutron would enable the “enable_snat” configuration flag for our main router when adding the new external IPv6 address.

Finally, in April 2025, after many years in the making, IPv6 was successfully deployed.

Compared to the network from 2011, we would have:

• Neutron as the engine in charge of the software-defined virtual network
• Ready to use tenant-networks
• Using a VXLAN-based overlay network
• Using neutron-openvswitch-agent to provide networking to virtual machines
• A modern and robust edge network setup

Over time, the WMCS team has skillfully navigated numerous challenges to ensure our service offerings consistently meet high standards of quality and operational efficiency. Often engaging in multi-year planning strategies, we have enabled ourselves to set and achieve significant milestones.

The successful IPv6 deployment stands as further testament to the team’s dedication and hard work over the years. I believe we can confidently say that the 2025 Cloud VPS represents its most advanced and capable iteration to date.

In a previous life, I worked for a location-based entertainment company, part of a huge team of people developing a location for Las Vegas, Nevada. It was COVID, a rough time for location-based anything, and things were delayed more than usual. Coworkers paid a lot of attention to another upcoming Las Vegas attraction, one with a vastly larger budget but still struggling to make schedule: the MSG (Madison Square Garden) Sphere.

I will set aside jokes about it being a square sphere, but they were perhaps one of the reasons that it underwent a pre-launch rebranding to merely the Sphere. If you are not familiar, the Sphere is a theater and venue in Las Vegas. While it's know mostly for the video display on the outside, that's just marketing for the inside: a digital dome theater, with seating at a roughly 45 degree stadium layout facing a near hemisphere of video displays.

It is a "near" hemisphere because the lower section is truncated to allow a flat floor, which serves as a stage for events but is also a practical architectural decision to avoid completely unsalable front rows. It might seem a little bit deceptive that an attraction called the Sphere does not quite pull off even a hemisphere of "payload," but the same compromise has been reached by most dome theaters. While the use of digital display technology is flashy, especially on the exterior, the Sphere is not quite the innovation that it presents itself as. It is just a continuation of a long tradition of dome theaters. Only time will tell, but the financial difficulties of the Sphere suggest that it follows the tradition faithfully: towards commercial failure.

You could make an argument that the dome theater is hundreds of years old, but I will omit it. Things really started developing, at least in our modern tradition of domes, with the 1923 introduction of the Zeiss planetarium projector. Zeiss projectors and their siblings used a complex optical and mechanical design to project accurate representations of the night sky. Many auxiliary projectors, incorporated into the chassis and giving these projectors famously eccentric shapes, rendered planets and other celestial bodies. Rather than digital light modulators, the images from these projectors were formed by purely optical means: perforated metal plates, glass plates with etched metalized layers, and fiber optics. The large, precisely manufactured image elements and specialized optics created breathtaking images.

While these projectors had considerable entertainment value, especially in the mid-century when they represented some of the most sophisticated projection technology yet developed, their greatest potential was obviously in education. Planetarium projectors were fantastically expensive (being hand-built in Germany with incredible component counts) [1], they were widely installed in science museums around the world. Most of us probably remember a dogbone-shaped Zeiss, or one of their later competitors like Spitz or Minolta, from our youths. Unfortunately, these marvels of artistic engineering were mostly retired as digital projection of near comparable quality became similarly priced in the 2000s.

But we aren't talking about projectors, we're talking about theaters. Planetarium projectors were highly specialized to rendering the night sky, and everything about them was intrinsically spherical. For both a reasonable viewing experience, and for the projector to produce a geometrically correct image, the screen had to be a spherical section. Thus the planetarium itself: in its most traditional form, rings of heavily reclined seats below a hemispherical dome. The dome was rarely a full hemisphere, but was usually truncated at the horizon. This was mostly a practical decision but integrated well into the planetarium experience, given that sky viewing is usually poor near the horizon anyway. Many planetaria painted a city skyline or forest silhouette around the lower edge to make the transition from screen to wall more natural. Later, theatrical lighting often replaced the silhouette, reproducing twilight or the haze of city lights.

Unsurprisingly, the application-specific design of these theaters also limits their potential. Despite many attempts, the collective science museum industry has struggled to find entertainment programming for planetaria much beyond Pink Floyd laser shows [2]. There just aren't that many things that you look up at. Over time, planetarium shows moved in more narrative directions. Film projection promised new flexibility---many planetaria with optical star projectors were also equipped with film projectors, which gave show producers exciting new options. Documentary video of space launches and animations of physical principles became natural parts of most science museum programs, but were a bit awkward on the traditional dome. You might project four copies of the image just above the horizon in the four cardinal directions, for example. It was very much a compromise.

With time, the theater adapted to the projection once again: the domes began to tilt. By shifting the dome in one direction, and orienting the seating towards that direction, you could create a sort of compromise point between the traditional dome and traditional movie theater. The lower central area of the screen was a reasonable place to show conventional film, while the full size of the dome allowed the starfield to almost fill the audience's vision. The experience of the tilted dome is compared to "floating in space," as opposed to looking up at the sky.

In true Cold War fashion, it was a pair of weapons engineers (one nuclear weapons, the other missiles) who designed the first tilted planetarium. In 1973, the planetarium of what is now called the Fleet Science Center in San Diego, California opened to the public. Its dome was tilted 25 degrees to the horizon, with the seating installed on a similar plane and facing in one direction. It featured a novel type of planetarium projector developed by Spitz and called the Space Transit Simulator. The STS was not the first, but still an early mechanical projector to be controlled by a computer---a computer that also had simultaneous control of other projectors and lighting in the theater, what we now call a show control system.

Even better, the STS's innovative optical design allowed it to warp or bend the starfield to simulate its appearance from locations other than earth. This was the "transit" feature: with a joystick connected to the control computer, the planetarium presenter could "fly" the theater through space in real time. The STS was installed in a well in the center of the seating area, and its compact chassis kept it low in the seating area, preserving the spherical geometry (with the projector at the center of the sphere) without blocking the view of audience members sitting behind it and facing forward.

And yet my main reason for discussing the Fleet planetarium is not the the planetarium projector at all. It is a second projector, an "auxiliary" one, installed in a second well behind the STS. The designers of the planetarium intended to show film as part of their presentations, but they were not content with a small image at the center viewpoint. The planetarium commissioned a few of the industry's leading film projection experts to design a film projection system that could fill the entire dome, just as the planetarium projector did.

They knew that such a large dome would require an exceptionally sharp image. Planetarium projectors, with their large lithographed slides, offered excellent spatial resolution. They made stars appear as point sources, the same as in the night sky. 35mm film, spread across such a large screen, would be obviously blurred in comparison. They would need a very large film format.

Fortuitously, almost simultaneously the Multiscreen Corporation was developing a "sideways" 70mm format. This 15-perf format used 70mm film but fed it through the projector sideways, making each frame much larger than typical 70mm film. In its debut, at a temporary installation in the 1970 Expo Osaka, it was dubbed IMAX. IMAX made an obvious basis for a high-resolution projection system, and so the then-named IMAX Corporation was added to the planetarium project. The Fleet's film projector ultimately consisted of an IMAX film transport with a custom-built compact, liquid-cooled lamphouse and spherical fisheye lens system.

The large size of the projector, the complex IMAX framing system and cooling equipment, made it difficult to conceal in the theater's projector well. Threading film into IMAX projectors is quite complex, with several checks the projectionist must make during a pre-show inspection. The projectionist needed room to handle the large film, and to route it to and from the enormous reels. The projector's position in the middle of the seating area left no room for any of this. We can speculate that it was, perhaps, one of the designer's missile experience that lead to the solution: the projector was serviced in a large projection room beneath the theater's seating. Once it was prepared for each show, it rose on near-vertical rails until just the top emerged in the theater. Rollers guided the film as it ran from a platter, up the shaft to the projector, and back down to another platter. Cables and hoses hung below the projector, following it up and down like the traveling cable of an elevator.

To advertise this system, probably the greatest advance in film projection since the IMAX format itself, the planetarium coined the term Omnimax.

Omnimax was not an easy or economical format. Ideally, footage had to be taken in the same format, using a 70mm camera with a spherical lens system. These cameras were exceptionally large and heavy, and the huge film format limited cinematographers to short takes. The practical problems with Omnimax filming were big enough that the first Omnimax films faked it, projecting to the larger spherical format from much smaller conventional negatives. This was the case for "Voyage to the Outer Planets" and "Garden Isle," the premier films at the Fleet planetarium. The history of both is somewhat obscure, the latter especially.

"Voyage to the Outer Planets" was executive-produced by Preston Fleet, a founder of the Fleet center (which was ultimately named for his father, a WWII aviator). We have Fleet's sense of showmanship to thank for the invention of Omnimax: He was an accomplished business executive, particularly in the photography industry, and an aviation enthusiast who had his hands in more than one museum. Most tellingly, though, he had an eccentric hobby. He was a theater organist. I can't help but think that his passion for the theater organ, an instrument almost defined by the combination of many gizmos under electromechanical control, inspired "Voyage." The film, often called a "multimedia experience," used multiple projectors throughout the planetarium to depict a far-future journey of exploration. The Omnimax film depicted travel through space, with slide projectors filling in artist's renderings of the many wonders of space.

The ten-minute Omnimax film was produced by Graphic Films Corporation, a brand that would become closely associated with Omnimax in the following decades. Graphic was founded in the midst of the Second World War by Lester Novros, a former Disney animator who found a niche creating training films for the military. Novros's fascination with motion and expertise in presenting complicated 3D scenes drew him to aerospace, and after the war he found much of his business in the newly formed Air Force and NASA. He was also an enthusiast of niche film formats, and Omnimax was not his first dome.

For the 1964 New York World's Fair, Novros and Graphic Films had produced "To the Moon and Beyond," a speculative science film with thematic similarities to "Voyage" and more than just a little mechanical similarity. It was presented in Cinerama 360, a semi-spherical, dome-theater 70mm format presented in a special theater called the Moon Dome. "To the Moon and Beyond" was influential in many ways, leading to Graphic Films' involvement in "2001: A Space Odyssey" and its enduring expertise in domes.

The Fleet planetarium would not remain the only Omnimax for long. In 1975, the city of Spokane, Washington struggled to find a new application for the pavilion built for Expo '74 [3]. A top contender: an Omnimax theater, in some ways a replacement for the temporary IMAX theater that had been constructed for the actual Expo. Alas, this project was not to be, but others came along: in 1978, the Detroit Science Center opened the second Omnimax theater ("the machine itself looks like and is the size of a front loader," the Detroit Free Press wrote). The Science Museum of Minnesota, in St. Paul, followed shortly after.

Omnimax hit prime time the next year, with the 1979 announcement of an Omnimax theater at Caesars Palace in Las Vegas, Nevada. Unlike the previous installations, this 380-seat theater was purely commercial. It opened with the 1976 IMAX film "To Fly!," which had been optically modified to fit the Omnimax format. This choice of first film is illuminating. "To Fly!" is a 27 minute documentary on the history of aviation in the United States, originally produced for the IMAX theater at the National Air and Space Museum [4]. It doesn't exactly seem like casino fare.

The IMAX format, the flat-screen one, was born of world's fairs. It premiered at an Expo, reappeared a couple of years later at another one, and for the first years of the format most of the IMAX theaters built were associated with either a major festival or an educational institution. This noncommercial history is a bit hard to square with the modern IMAX brand, closely associated with major theater chains and the Marvel Cinematic Universe.

Well, IMAX took off, and in many ways it sold out. Over the decades since the 1970 Expo, IMAX has met widespread success with commercial films and theater owners. Simultaneously, the definition or criteria for IMAX theaters have relaxed, with smaller screens made permissible until, ultimately, the transition to digital projection eliminated the 70mm film and more or less reduce IMAX to just another ticket surcharge brand. It competes directly with Cinemark xD, for example. To the theater enthusiast, this is a pretty sad turn of events, a Westinghouse-esque zombification of a brand that once heralded the field's most impressive technical achievements.

The same never happened to Omnimax. The Caesar's Omnimax theater was an odd exception; the vast majority of Omnimax theaters were built by science museums and the vast majority of Omnimax films were science documentaries. Quite a few of those films had been specifically commissioned by science museums, often on the occasion of their Omnimax theater opening. The Omnimax community was fairly tight, and so the same names recur.

The Graphic Films Corporation, which had been around since the beginning, remained so closely tied to the IMAX brand that they practically shared identities. Most Omnimax theaters, and some IMAX theaters, used to open with a vanity card often known as "the wormhole." It might be hard to describe beyond "if you know you know," it certainly made an impression on everyone I know that grew up near a theater that used it. There are some videos, although unfortunately none of them are very good.

I have spent more hours of my life than I am proud to admit trying to untangle the history of this clip. Over time, it has appeared in many theaters with many different logos at the end, and several variations of the audio track. This is in part informed speculation, but here is what I believe to be true: the "wormhole" was originally created by Graphic Films for the Fleet planetarium specifically, and ran before "Voyage to the Outer Planets" and its double-feature companion "Garden Isle," both of which Graphic Films had worked on. This original version ended with the name Graphic Films, accompanied by an odd sketchy drawing that was also used as an early logo of the IMAX Corporation. Later, the same animation was re-edited to end with an IMAX logo.

This version ran in both Omnimax and conventional IMAX theaters, probably as a result of the extensive "cross-pollination" of films between the two formats. Many Omnimax films through the life of the format had actually been filmed for IMAX, with conventional lenses, and then optically modified to fit the Omnimax dome after the fact. You could usually tell: the reprojection process created an unusual warp in the image, and more tellingly, these pseudo-Omnimax films almost always centered the action at the middle of the IMAX frame, which was too high to be quite comfortable in an Omnimax theater (where the "frame center" was well above the "front center" point of the theater). Graphic Films had been involved in a lot of these as well, perhaps explaining the animation reuse, but it's just as likely that they had sold it outright to the IMAX corporation which used it as they pleased.

For some reason, this version also received new audio that is mostly the same but slightly different. I don't have a definitive explanation, but I think there may have been an audio format change between the very early Omnimax theaters and later IMAX/Omnimax systems, which might have required remastering.

Later, as Omnimax domes proliferated at science museums, the IMAX Corporation (which very actively promoted Omnimax to education) gave many of these theaters custom versions of the vanity card that ended with the science museum's own logo. I have personally seen two of these, so I feel pretty confident that they exist and weren't all that rare (basically 2 out of 2 Omnimax theaters I've visited used one), but I cannot find any preserved copies.

Another recurring name in the world of IMAX and Omnimax is MacGillivray Freeman Films. MacGillivray and Freeman were a pair of teenage friends from Laguna Beach who dropped out of school in the '60s to make skateboard and surf films. This is, of course, a rather cliché start for documentary filmmakers but we must allow that it was the '60s and they were pretty much the ones creating the cliché. Their early films are hard to find in anything better than VHS rip quality, but worth watching: Wikipedia notes their significance in pioneering "action cameras," mounting 16mm cinema cameras to skateboards and surfboards, but I would say that their cinematography was innovative in more ways than just one. The 1970 "Catch the Joy," about sandrails, has some incredible shots that I struggle to explain. There's at least one where they definitely cut the shot just a couple of frames before a drifting sandrail flung their camera all the way down the dune.

For some reason, I would speculate due to their reputation for exciting cinematography, the National Air and Space Museum chose MacGillivray and Freeman for "To Fly!". While not the first science museum IMAX documentary by any means (that was, presumably, "Voyage to the Outer Planets" given the different subject matter of the various Expo films), "To Fly!" might be called the first modern one. It set the pattern that decades of science museum films followed: a film initially written by science educators, punched up by producers, and filmed with the very best technology of the time. Fearing that the film's history content would be dry, they pivoted more towards entertainment, adding jokes and action sequences. "To Fly!" was a hit, running in just about every science museum with an IMAX theater, including Omnimax.

Sadly, Jim Freeman died in a helicopter crash shortly after production. Nonetheless, MacGillivray Freeman Films went on. Over the following decades, few IMAX science documentaries were made that didn't involve them somehow. Besides the films they produced, the company consulted on action sequences in most of the format's popular features.

I had hoped to present here a thorough history of the films that were actually produced in the Omnimax format. Unfortunately, this has proven very difficult: the fact that most of them were distributed only to science museums means that they are very spottily remembered, and besides, so many of the films that ran in Omnimax theaters were converted from IMAX presentations that it's hard to tell the two apart. I'm disappointed that this part of cinema history isn't better recorded, and I'll continue to put time into the effort. Science museum documentaries don't get a lot of attention, but many of the have involved formidable technical efforts.

Consider, for example, the cameras: befitting the large film, IMAX cameras themselves are very large. When filming "To Fly!", MacGillivray and Freeman complained that the technically very basic 80 pound cameras required a lot of maintenance, were complex to operate, and wouldn't fit into the "action cam" mounting positions they were used to. The cameras were so expensive, and so rare, that they had to be far more conservative than their usual approach out of fear of damaging a camera they would not be able to replace. It turns out that they had it easy. Later IMAX science documentaries would be filmed in space ("The Dream is Alive" among others) and deep underwater ("Deep Sea 3D" among others). These IMAX cameras, modified for simpler operation and housed for such difficult environments, weighed over 1,000 pounds. Astronauts had to be trained to operate the cameras; mission specialists on Hubble service missions had wrangling a 70-pound handheld IMAX camera around the cabin and developing its film in a darkroom bag among their duties. There was a lot of film to handle: as a rule of thumb, one mile of IMAX film is good for eight and a half minutes.

I grew up in Portland, Oregon, and so we will make things a bit more approachable by focusing on one example: The Omnimax theater of the Oregon Museum of Science and Industry, which opened as part of the museum's new waterfront location in 1992. This 330-seat boasted a 10,000 sq ft dome and 15 kW of sound. The premier feature was "Ring of Fire," a volcano documentary originally commissioned by the Fleet, the Fort Worth Museum of Science and Industry, and the Science Museum of Minnesota. By the 1990s, the later era of Omnimax, the dome format was all but abandoned as a commercial concept. There were, an announcement article notes, around 90 total IMAX theaters (including Omnimax) and 80 Omnimax films (including those converted from IMAX) in '92. Considering the heavy bias towards science museums among these theaters, it was very common for the films to be funded by consortia of those museums.

Considering the high cost of filming in IMAX, a lot of the documentaries had a sort of "mashup" feel. They would combine footage taken in different times and places, often originally for other projects, into a new narrative. "Ring of Fire" was no exception, consisting of a series of sections that were sometimes more loosely connected to the theme. The 1982 Loma Prieta earthquake was a focus, and the eruption of Mt. St. Helens, and lava flows in Hawaii. Perhaps one of the reasons it's hard to catalog IMAX films is this mashup quality, many of the titles carried at science museums were something along the lines of "another ocean one." I don't mean this as a criticism, many of the IMAX documentaries were excellent, but they were necessarily composed from painstakingly gathered fragments and had to cover wide topics.

Given that I have an announcement feature piece in front of me, let's also use the example of OMSI to discuss the technical aspects. OMSI's projector cost about $2 million and weighted about two tons. To avoid dust damaging the expensive prints, the "projection room" under the seating was a positive-pressure cleanroom. This was especially important since the paucity of Omnimax content meant that many films ran regularly for years. The 15 kW water-cooled lamp required replacement at 800 to 1,000 hours, but unfortunately, the price is not noted.

By the 1990s, Omnimax had become a rare enough system that the projection technology was a major part of the appeal. OMSI's installation, like most later Omnimax theaters, had the audience queue below the seating, separated from the projection room by a glass wall. The high cost of these theaters meant that they operated on high turnovers, so patrons would wait in line to enter immediately after the previous showing had exited. While they waited, they could watch the projectionist prepare the next show while a museum docent explained the equipment.

I have written before about multi-channel audio formats, and Omnimax gives us some more to consider. The conventional audio format for much of Omnimax's life was six-channel: left rear, left screen, center screen, right screen, right rear, and top. Each channel had an independent bass cabinet (in one theater, a "caravan-sized" enclosure with eight JBL 2245H 46cm woofers), and a crossover network fed the lowest end of all six channels to a "sub-bass" array at screen bottom. The original Fleet installation also had sub-bass speakers located beneath the audience seating, although that doesn't seem to have become common.

IMAX titles of the '70s and '80s delivered audio on eight-track magnetic tape, with the additional tracks used for synchronization to the film. By the '90s, IMAX had switched to distributing digital audio on three CDs (one for each two channels). OMSI's theater was equipped for both, and the announcement amusingly notes the availability of cassette decks. A semi-custom audio processor made for IMAX, the Sonics TAC-86, managed synchronization with film playback and applied equalization curves individually calibrated to the theater.

IMAX domes used perforated aluminum screens (also the norm in later planetaria), so the speakers were placed behind the screen in the scaffold-like superstructure that supported it. When I was young, OMSI used to start presentations with a demo program that explained the large size of IMAX film before illuminating work lights behind the screen to make the speakers visible. Much of this was the work of the surprisingly sophisticated show control system employed by Omnimax theaters, a descendent of the PDP-15 originally installed in the Fleet.

Despite Omnimax's almost complete consignment to science museums, there were some efforts at bringing commercial films. Titles like Disney's "Fantasia" and "Star Wars: Episode III" were distributed to Omnimax theaters via optical reprojection, sometimes even from 35mm originals. Unfortunately, the quality of these adaptations was rarely satisfactory, and the short runtimes (and marketing and exclusivity deals) typical of major commercial releases did not always work well with science museum schedules. Still, the cost of converting an existing film to dome format is pretty low, so the practice continues today. "Star Wars: The Force Awakens," for example, ran on at least one science museum dome. This trickle of blockbusters was not enough to make commercial Omnimax theaters viable.

Caesars Palace closed, and then demolished, their Omnimax theater in 2000. The turn of the 21st century was very much the beginning of the end for the dome theater. IMAX was moving away from their film system and towards digital projection, but digital projection systems suitable for large domes were still a nascent technology and extremely expensive. The end of aggressive support from IMAX meant that filming costs became impractical for documentaries, so while some significant IMAX science museum films were made in the 2000s, the volume definitely began to lull and the overall industry moved away from IMAX in general and Omnimax especially.

It's surprising how unforeseen this was, at least to some. A ten-screen commercial theater in Duluth opened an Omnimax theater in 1996! Perhaps due to the sunk cost, it ran until 2010, not a bad closing date for an Omnimax theater. Science museums, with their relatively tight budgets and less competitive nature, did tend to hold over existing Omnimax installations well past their prime. Unfortunately, many didn't: OMSI, for example, closed its Omnimax theater in 2013 for replacement with a conventional digital theater that has a large screen but is not IMAX branded.

Fortunately, some operators hung onto their increasingly costly Omnimax domes long enough for modernization to become practical. The IMAX Corporation abandoned the Omnimax name as more of the theaters closed, but continued to support "IMAX Dome" with the introduction of a digital laser projector with spherical optics. There are only ten examples of this system. Others, including Omnimax's flagship at the Fleet Science Center, have been replaced by custom dome projection systems built by competitors like Sony.

Few Omnimax projectors remain. The Fleet, to their credit, installed the modern laser projectors in front of the projector well so that the original film projector could remain in place. It's still functional and used for reprisals of Omnimax-era documentaries. IMAX projectors in general are a dying breed, a number of them have been preserved but their complex, specialized design and the end of vendor support means that it may become infeasible to keep them operating.

We are, of course, well into the digital era. While far from inexpensive, digital projection systems are now able to match the quality of Omnimax projection. The newest dome theaters, like the Sphere, dispense with projection entirely. Instead, they use LED display panels capable of far brighter and more vivid images than projection, and with none of the complexity of water-cooled arc lamps.

Still, something has been lost. There was once a parallel theater industry, a world with none of the glamor of Hollywood but for whom James Cameron hauled a camera to the depths of the ocean and Leonardo DiCaprio narrated repairs to the Hubble. In a good few dozen science museums, two-ton behemoths rose from beneath the seats, the zenith of film projection technology. After decades of documentaries, I think people forgot how remarkable these theaters were.

Science museums stopped promoting them as aggressively, and much of the showmanship faded away. Sometime in the 2000s, OMSI stopped running the pre-show demonstration, instead starting the film directly. They stopped explaining the projectionist's work in preparing the show, and as they shifted their schedule towards direct repetition of one feature, there was less for the projectionist to do anyway. It became just another museum theater, so it's no wonder that they replaced it with just another museum theater: a generic big-screen setup with the exceptionally dull name of "Empirical Theater."

From time to time, there have been whispers of a resurgence of 70mm film. Oppenheimer, for example, was distributed to a small number of theaters in this giant of film formats: 53 reels, 11 miles, 600 pounds of film. Even conventional IMAX is too costly for the modern theater industry, though. Omnimax has fallen completely by the wayside, with the few remaining dome operators doomed to recycling the same films with a sprinkling of newer reformatted features. It is hard to imagine a collective of science museums sending another film camera to space.

Omnimax poses a preservation challenge in more ways than one. Besides the lack of documentation on Omnimax theaters and films, there are precious few photographs of Omnimax theaters and even fewer videos of their presentations. Of course, the historian suffers where Madison Square Garden hopes to succeed: the dome theater is perhaps the ultimate in location-based entertainment. Photos and videos, represented on a flat screen, cannot reproduce the experience of the Omnimax theater. The 180 horizontal degrees of screen, the sound that was always a little too loud, in no small part to mask the sound of the projector that made its own racket in the middle of the seating. You had to be there.

IMAGES: Omnimax projection room at OMSI, Flickr user truk. Omnimax dome with work lights on at MSI Chicago, Wikimedia Commons user GualdimG. Omnimax projector at St. Louis Science Center, Flickr user pasa47.

[1] I don't have extensive information on pricing, but I know that in the 1960s an "economy" Spitz came in over $30,000 (~10x that much today).

[2] Pink Floyd's landmark album Dark Side of The Moon debuted in a release event held at the London Planetarium. This connection between Pink Floyd and planetaria, apparently much disliked by the band itself, has persisted to the present day. Several generations of Pink Floyd laser shows have been licensed by science museums around the world, and must represent by far the largest success of fixed-installation laser projection.

[3] Are you starting to detect a theme with these Expos? the World's Fairs, including in their various forms as Expos, were long one of the main markets for niche film formats. Any given weird projection format you run into, there's a decent chance that it was originally developed for some short film for an Expo. Keep in mind that it's the nature of niche projection formats that they cannot easily be shown in conventional theaters, so they end up coupled to these crowd events where a custom venue can be built.

[4] The Smithsonian Institution started looking for an exciting new theater in 1970. As an example of the various niche film formats at the time, the Smithsonian considered a dome (presumably Omnimax), Cinerama (a three-projector ultrawide system), and Circle-Vision 360 (known mostly for the few surviving Expo films at Disney World's EPCOT) before settling on IMAX. The Smithsonian theater, first planned for the Smithsonian Museum of Natural History before being integrated into the new National Air and Space Museum, was tremendously influential on the broader world of science museum films. That is perhaps an understatement, it is sometimes credited with popularizing IMAX in general, and the newspaper coverage the new theater received throughout North America lends credence to the idea. It is interesting, then, to imagine how different our world would be if they had chosen Circle-Vision. "Captain America: Brave New World" in Cinemark 360.

Sometimes I think I should pivot my career to home automation critic, because I have many opinions on the state of the home automation industry---and they're pretty much all critical. Virtually every time I bring up home automation, someone says something about the superiority of the light switch. Controlling lights is one of the most obvious applications of home automation, and there is a roughly century long history of developments in light control---yet, paradoxically, it is an area where consumer home automation continues to struggle.

An analysis of how and why billion-dollar tech companies fail to master the simple toggling of lights in response to human input will have to wait for a future article, because I will have a hard time writing one without descending into incoherent sobbing about the principles of scene control and the interests of capital. Instead, I want to just dip a toe into the troubled waters of "smart lighting" by looking at one of its earliest precedents: low-voltage lighting control.

A source I generally trust, the venerable "old internet" website Inspectapedia, says that low-voltage lighting control systems date back to about 1946. The earliest conclusive evidence I can find of these systems is a newspaper ad from 1948, but let's be honest, it's a holiday and I'm only making a half effort on the research. In any case, the post-war timing is not a coincidence. The late 1940s were a period of both rapid (sub)urban expansion and high copper prices, and the original impetus for relay systems seems to have been the confluence of these two.

But let's step back and explain what a relay or low-voltage lighting control system is. First, I am not referring to "low voltage lighting" meaning lights that run on 12 or 24 volts DC or AC, as was common in landscape lighting and is increasingly common today for integrated LED lighting. Low-voltage lighting control systems are used for conventional 120VAC lights. In the most traditional construction, e.g. in the 1940s, lights would be served by a "hot" wire that passed through a wall box containing a switch. In many cases the neutral (likely shared with other fixtures) went directly from the light back to the panel, bypassing the switch... running both the hot and neutral through the switch box did not become conventional until fairly recently, to the chagrin of anyone installing switches that require a neutral for their own power, like timers or "smart" switches.

The problem with this is that it lengthens the wiring runs. If you have a ceiling fixture with two different switches in a three-way arrangement, say in a hallway in a larger house, you could be adding nearly 100' in additional wire to get the hot to the switches and the runner between them. The cost of that wiring, in the mid-century, was quite substantial. Considering how difficult it is to find an employee to unlock the Romex cage at Lowes these days, I'm not sure that's changed that much.

There are different ways of dealing with this. In the UK, the "ring main" served in part to reduce the gauge (and thus cost) of outlet wiring, but we never picked up that particular eccentricity in the US (for good reason). In commercial buildings, it's not unusual for lighting to run on 240v for similar reasons, but 240v is discouraged in US residential wiring. Besides, the mid-century was an age of optimism and ambition in electrical technology, the days of Total Electric Living. Perhaps the technology of the relay, refined by so many innovations of WWII, could offer a solution.

Switch wiring also had to run through wall cavities, an irritating requirement in single-floor houses where much of the lighting wiring could be contained to the attic. The wiring of four-way and other multi-switch arrangements could become complex and require a lot more wall runs, discouraging builders providing switches in the most convenient places. What if relays also made multiple switches significantly easier to install and relocate?

You probably get the idea. In a typical low-voltage lighting control system, a transformer provides a low voltage like 24VAC, much the same as used by doorbells. The light switches simply toggle the 24VAC control power to the coils of relays. Some (generally older) systems powered the relay continuously, but most used latching relays. In this case, all light switches are momentary, with an "on" side and an "off" side. This could be a paddle that you push up or down (much like a conventional light switch), a bar that you push the left or right sides of, or a pair of two push buttons.

In most installations, all of the relays were installed together in a single enclosure, usually in the attic where the high-voltage wiring to the actual lights would be fairly short. The 24VAC cabling to the switches was much smaller gauge, and depending on the jurisdiction might not require any sort of license to install.

Many systems had enclosures with separate high voltage and low voltage components, or mounted the relays on the outside of an enclosure such that the high voltage wiring was inside and low voltage outside. Both arrangements helped to meet code requirements for isolating high and low voltage systems and provided a margin of safety in the low voltage wiring. That provided additional cost savings as well; low voltage wiring was usually installed without any kind of conduit or sheathed cable.

By 1950, relay lighting controls were making common appearances in real estate listings. A feature piece on the "Melody House," a builder's model home, in the Tacoma News Tribune reads thus:

Newest features in the house are the low voltage touch plate and relay system lighting controls, with wide plates instead of snap buttons---operated like the stops of a pipe organ, with the merest flick of a finger.

The comparison to a pipe organ is interesting, first in its assumption that many readers were familiar with typical organ stops. Pipe organs were, increasingly, one of the technological marvels of the era: while the concept of the pipe organ is very old, this same era saw electrical control systems (replete with relays!) significantly reduce the cost and complexity of organ consoles. What's more, the tonewheel electric organ had become well-developed and started to find its way into homes.

The comparison is also interesting because of its deficiencies. The Touch-Plate system described used wide bars, which you pressed the left or right side of---you could call them momentary SPDT rocker switches if you wanted. There were organs with similar rocker stops but I do not think they were common in 1950. My experience is that such rocker switch stops usually indicate a fully digital control system, where they make momentary action unobtrusive and avoid state synchronization problems. I am far from an expert on organs, though, which is why I haven't yet written about them. If you have a guess at which type of pipe organ console our journalist was familiar with, do let me know.

Touch-Plate seems to have been one of the first manufacturers of these systems, although I can't say for sure that they invented them. Interestingly, Touch-Plate is still around today, but their badly broken WordPress site ("Welcome to the new touch-plate.com" despite it actually being touchplate.com) suggests they may not do much business. After a few pageloads their WordPress plugin WAF blocked me for "exceed[ing] the maximum number of page not found errors per minute for humans." This might be related to my frustration that none of the product images load. It seems that the Touch-Plate company has mostly pivoted to reselling imported LED lighting (touchplateled.com), so I suppose the controls business is withering on the vine.

The 1950s saw a proliferation of relay lighting control brands, with GE introducing a particularly popular system with several generations of fixtures. Kyle Switch Plates, who sell replacement switch plates (what else?), list options for Remcon, Sierra, Bryant, Pyramid, Douglas, and Enercon systems in addition to the two brands we have met so far. As someone who pays a little too much attention to light switches, I have personally seen four of these brands, three of them still in use and one apparently abandoned in place.

Now, you might be thinking that simply economizing wiring by relocating the switches does not constitute "home automation," but there are other features to consider. For one, low-voltage light control systems made it feasible to install a lot more switches. Houses originally built with them often go a little wild with the n-way switching, every room providing lightswitches at every door. But there is also the possibility of relay logic. From the same article:

The necessary switches are found in every room, but in the master bedroom there is a master control panel above the bed, from where the house and yard may be flooded with instant light in case of night emergency.

Such "master control panels" were a big attraction for relay lighting, and the finest homes of the 1950s and 1960s often displayed either a grid of buttons near the head of the master bed, or even better, a GE "Master Selector" with a curious system of rotary switches. On later systems, timers often served as auxiliary switches, so you could schedule exterior lights. With a creative installer, "scenes" were even possible by wiring switches to arbitrary sets of relays (this required DC or half-wave rectified control power and diodes to isolate the switches from each other).

Many of these relay control systems are still in use today. While they are quite outdated in a certain sense, the design is robust and the simple components mean that it's usually not difficult to find replacement parts when something does fail. The most popular system is the one offered by GE, using their RR series relays (RR3, RR4, etc., to the modern RR9). That said, GE suggests a modernization path to their LightSweep system, which is really a 0-10v analog dimming controller that has the add-on ability to operate relays.

The failure modes are mostly what you would expect: low voltage wiring can chafe and short, or the switches can become stuck. This tends to cause the lights to stick on or off, and the continuous current through the relay coil often burns it out. The fix requires finding the stuck switch or short and correcting it, and then replacing the relay.

One upside of these systems that persists today is density: the low voltage switches are small, so with most systems you can fit 3 per gang. Another is that they still make N-way switching easier. There is arguably a safety benefit, considering the reduction in mains-voltage wire runs.

Yet we rarely see such a thing installed in homes newer than around the '80s. I don't know that I can give a definitive explanation of the decline of relay lighting control, but reduced prices for copper wiring were probably a main factor. The relays added a failure point, which might lead to a perception of unreliability, and the declining familiarity of electricians means that installing a relay system could be expensive and frustrating today.

What really interests me about relay systems is that they weren't really replaced... the idea just went away. It's not like modern homes are providing a master control panel in the bedroom using some alternative technology. I mean, some do, those with prices in the eight digits, but you'll hardly ever see it.

That gets us to the tension between residential lighting and architectural lighting control systems. In higher-end commercial buildings, and in environments like conference rooms and lecture halls, there's a well established industry building digital lighting control systems. Today, DALI is a common standard for the actual lighting control, but if you look at a range of existing buildings you will find everything from completely proprietary digital distributed dimming to 0-10v analog dimming to central dimmer racks (similar to traditional theatrical lighting).

Relay lighting systems were, in a way, a nascent version of residential architectural lighting control. And the architectural lighting control industry continues to evolve. If there is a modern equivalent to relay lighting, it's something like Lutron QSX. That's a proprietary digital lighting (and shade) control system, marketed for both residential and commercial use. QSX offers a wide range of attractive wall controls, tight integration to Lutron's HomeSense home automation platform, and a price tag that'll make your eyes water. Lutron has produced many generations of these systems, and you could make an argument that they trace their heritage back to the relay systems of the 1940s. But they're just priced way beyond the middle-class home.

And, well, I suppose that requires an argument based on economics. Prices have gone up. Despite tract construction being a much older idea than people often realize, it seems clear that today's new construction homes have been "value engineered" to significantly lower feature and quality levels than those of the mid-century---but they're a lot bigger. There is a sort of maxim that today's home buyers don't care about anything but square footage, and if you've seen what Pulte or D. R. Horton are putting up... well, I never knew that 3,000 sq ft could come so cheap, and look it too.

Modern new-construction homes just don't come with the gizmos that older ones did, especially in the '60s and '70s. Looking at the sales brochure for a new development in my own Albuquerque ("Estates at La Cuentista"), besides 21st century suburbanization (Gated Community! "East Access to Paseo del Norte" as if that's a good thing!) most of the advertised features are "big." I'm serious! If you look at the "More Innovation Built In" section, the "innovations" are a home office (more square footage), storage (more square footage), indoor and outdoor gathering spaces (to be fair, only the indoor ones are square footage), "dedicated learning areas" for kids (more square footage), and a "basement or bigger garage" for a home gym (more square footage). The only thing in the entire innovation section that I would call a "technical" feature is water filtration. You can scroll down for more details, and you get to things like "space for a movie room" and a finished basement described eight different ways.

Things were different during the peak of relay lighting in the '60s. A house might only be 1,600 sq ft, but the builder would deck it out with an intercom (including multi-room audio of a primitive sort), burglar alarm, and yes, relay lighting. All of these technologies were a lot newer and people were more excited about them; I bring up Total Electric Living a lot because of an aesthetic obsession but it was a large-scale advertising and partnership campaign by the electrical industry (particularly Westinghouse) that gave builders additional cross-promotion if they included all of these bells and whistles.

Remember, that was when people were watching those old videos about the "kitchen of the future." What would a 2025 "Kitchen of the Future" promotional film emphasize? An island bigger than my living room and a nook for every meal, I assume. Features like intercoms and even burglar alarms have become far less common in new construction, and even if they were present I don't think most buyers would use them.

But that might seem a little odd, right, given the push towards home automation? Well, built-in home automation options have existed for longer than any of today's consumer solutions, but "built in" is a liability for a technology product. There are practical reasons, in that built-in equipment is harder to replace, but there's also a lamer commercial reason. Consumer technology companies want to sell their products like consumer technology, so they've recontextualized lighting control as "IoT" and "smart" and "AI" rather than something an electrician would hook up.

While I was looking into relay lighting control systems, I ran into an interesting example. The Lutron Lu Master Lumi 5. What a name! Lutron loves naming things like this. The Lumi 5 is a 1980s era product with essentially the same features as a relay system, but architected in a much stranger way. It is, essentially, five three way switches in a box with remote controls. That means that each of the actual light switches in the house (which could also be dimmers) need mains-voltage wiring, including runner, back to the Lumi 5 "interface."

Pressing a button on one of the Lutron wall panels toggles the state of the relay in the "interface" cabinet, toggling the light. But, since it's all wired as a three-way switch, toggling the physical switch at the light does the same thing. As is typical when combining n-way switches and dimming, the Lumi 5 has no control over dimmers. You can only dim a light up or down at the actual local control, the Lumi 5 can just toggle the dimmer on and off using the 3-way runner. The architecture also means that you have two fundamentally different types of wall panels in your house: local switches or dimmers wired to each light, and the Lu Master panels with their five buttons for the five circuits, along with "all on" and "all off."

The Lumi 5 "interface" uses simple relay logic to implement a few more features. Five mains-voltage-level inputs can be wired to time clocks, so that you can schedule any combination(s) of the circuits to turn on and off. The manual recommends models including one with an astronomical clock for sunrise/sunset. An additional input causes all five circuits to turn on; it's suggested for connection to an auxiliary relay on a burglar alarm to turn all of the lights on should the alarm be triggered.

The whole thing is strange and fascinating. It is basically a relay lighting control system, like so many before it, but using a distinctly different wiring convention. I think the main reason for the odd wiring was to accommodate dimmers, an increasingly popular option in the 1980s that relay systems could never really contend with. It doesn't have the cost advantages of relay systems at all, it will definitely be more expensive! But it adds some features over the fancy Lutron switches and dimmers you were going to install anyway.

The Lu Master is the transitional stage between relay lighting systems and later architectural lighting controls, and it straddled too the end of relay light control in homes. It gives an idea of where relay light control in homes would have evolved, had the whole technology not been doomed to the niche zone of conference centers and universities.

If you think about it, the Lu Master fills the most fundamental roles of home automation in lighting: control over multiple lights in a convenient place, scheduling and triggers, and an emergency function. It only lacks scenes, which I think we can excuse considering that the simple technology it uses does not allow it to adjust dimmers. And all of that with no Node-RED in sight!

Maybe that conveys what most frustrates me about the "home automation" industry: it is constantly reinventing the wheel, an oligopoly of tech companies trying to drag people's homes into their "ecosystem." They do so by leveraging the buzzword of the moment, IoT to voice assistants to, I guess now AI?, to solve a basic set of problems that were pretty well solved at least as early as 1948.

That's not to deny that modern home automation platforms have features that old ones don't. They are capable of incredibly sophisticated things! But realistically, most of their users want only very basic functionality: control in convenient places, basic automation, scenes. It wouldn't sting so much if all these whiz-bang general purpose computers were good at those tasks, but they aren't. For the very most basic tasks, things like turning on and off a group of lights, major tech ecosystems like HomeKit provide a user experience that is significantly worse than the model home of 1950.

You could install a Lutron system, and it would solve those fundamental tasks much better... for a much higher price. But it's not like Lutron uses all that money to be an absolute technical powerhouse, a center of innovation at the cutting edge. No, even the latest Lutron products are really very simple, technically. The technical leaders here, Google, Apple, are the companies that can't figure out how to make a damn light switch.

The problem with modern home automation platforms is that they are too ambitious. They are trying to apply enormously complex systems to very simple tasks, and thus contaminating the simplest of electrical systems with all the convenience and ease of a Smart TV.

Sometimes that's what it feels like this whole industry is doing: adding complexity while the core decays. From automatic programming to AI coding agents, video terminals to Electron, the scope of the possible expands while the fundamentals become more and more irritating.

But back to the real point, I hope you learned about some cool light switches. Check out the Kyle Switch Plates reference and you'll start seeing these buildings and homes, at least if you live in an area that built up during the era that they were common (1950s to the 1970s).

Air traffic control has been in the news lately, on account of my country's declining ability to do it. Well, that's a long-term trend, resulting from decades of under-investment, severe capture by our increasingly incompetent defense-industrial complex, no small degree of management incompetence in the FAA, and long-lasting effects of Reagan crushing the PATCO strike. But that's just my opinion, you know, maybe airplanes got too woke. In any case, it's an interesting time to consider how weird parts of air traffic control are. The technical, administrative, and social aspects of ATC all seem two notches more complicated than you would expect. ATC is heavily influenced by its peculiar and often accidental development, a product of necessity that perpetually trails behind the need, and a beneficiary of hand-me-down military practices and technology.

Aviation Radio

In the early days of aviation, there was little need for ATC---there just weren't many planes, and technology didn't allow ground-based controllers to do much of value. There was some use of flags and signal lights to clear aircraft to land, but for the most part ATC had to wait for the development of aviation radio. The impetus for that work came mostly from the First World War.

Here we have to note that the history of aviation is very closely intertwined with the history of warfare. Aviation technology has always rapidly advanced during major conflicts, and as we will see, ATC is no exception.

By 1913, the US Army Signal Corps was experimenting with the use of radio to communicate with aircraft. This was pretty early in radio technology, and the aircraft radios were huge and awkward to operate, but it was also early in aviation and "huge and awkward to operate" could be similarly applied to the aircraft of the day. Even so, radio had obvious potential in aviation. The first military application for aircraft was reconnaissance. Pilots could fly past the front to find artillery positions and otherwise provide useful information, and then return with maps. Well, even better than returning with a map was providing the information in real-time, and by the end of the war medium-frequency AM radios were well developed for aircraft.

Radios in aircraft led naturally to another wartime innovation: ground control. Military personnel on the ground used radio to coordinate the schedules and routes of reconnaissance planes, and later to inform on the positions of fighters and other enemy assets. Without any real way to know where the planes were, this was all pretty primitive, but it set the basic pattern that people on the ground could keep track of aircraft and provide useful information.

Post-war, civil aviation rapidly advanced. The early 1920s saw numerous commercial airlines adopting radio, mostly for business purposes like schedule coordination. Once you were in contact with someone on the ground, though, it was only logical to ask about weather and conditions. Many of our modern practices like weather briefings, flight plans, and route clearances originated as more or less formal practices within individual airlines.

Air Mail

The government was not left out of the action. The Post Office operated what may have been the largest commercial aviation operation in the world during the early 1920s, in the form of Air Mail. The Post Office itself did not have any aircraft; all of the flying was contracted out---initially to the Army Air Service, and later to a long list of regional airlines. Air Mail was considered a high priority by the Post Office and proved very popular with the public. When the transcontinental route began proper operation in 1920, it became possible to get a letter from New York City to San Francisco in just 33 hours by transferring it between airplanes in a nearly non-stop relay race.

The Post Office's largesse in contracting the service to private operators provided not only the funding but the very motivation for much of our modern aviation industry. Air travel was not very popular at the time, being loud and uncomfortable, but the mail didn't complain. The many contract mail carriers of the 1920s grew and consolidated into what are now some of the United States' largest companies. For around a decade, the Post Office almost singlehandedly bankrolled civil aviation, and passengers were a side hustle [1].

Air mail ambition was not only of economic benefit. Air mail routes were often longer and more challenging than commercial passenger routes. Transcontinental service required regular flights through sparsely populated parts of the interior, challenging the navigation technology of the time and making rescue of downed pilots a major concern. Notably, air mail operators did far more nighttime flying than any other commercial aviation in the 1920s. The post office became the government's de facto technical leader in civil aviation. Besides the network of beacons and markers built to guide air mail between cities, the post office built 17 Air Mail Radio Stations along the transcontinental route.

The Air Mail Radio Stations were the company radio system for the entire air mail enterprise, and the closest thing to a nationwide, public air traffic control service to then exist. They did not, however, provide what we would now call control. Their role was mainly to provide pilots with information (including, critically, weather reports) and to keep loose tabs on air mail flights so that a disappearance would be noticed in time to send search and rescue.

In 1926, the Watres Act created the Aeronautic Branch of the Department of Commerce. The Aeronautic Branch assumed a number of responsibilities, but one of them was the maintenance of the Air Mail routes. Similarly, the Air Mail Radio Stations became Aeronautics Branch facilities, and took on the new name of Flight Service Stations. No longer just for the contract mail carriers, the Flight Service Stations made up a nationwide network of government-provided services to aviators. They were the first edifices in what we now call the National Airspace System (NAS): a complex combination of physical facilities, technologies, and operating practices that enable safe aviation.

In 1935, the first en-route air traffic control center opened, a facility in Newark owned by a group of airlines. The Aeronautic Branch, since renamed the Bureau of Air Commerce, supported the airlines in developing this new concept of en-route control that used radio communications and paperwork to track which aircraft were in which airways. The rising number of commercial aircraft made in-air collisions a bigger problem, so the Newark control center was quickly followed by more facilities built on the same pattern. In 1936, the Bureau of Air Commerce took ownership of these centers, and ATC became a government function alongside the advisory and safety services provided by the flight service stations.

En route center controllers worked off of position reports from pilots via radio, but needed a way to visualize and track aircraft's positions and their intended flight paths. Several techniques helped: first, airlines shared their flight planning paperwork with the control centers, establishing "flight plans" that corresponded to each aircraft in the sky. Controllers adopted a work aid called a "flight strip," a small piece of paper with the key information about an aircraft's identity and flight plan that could easily be handed between stations. By arranging the flight strips on display boards full of slots, controllers could visualize the ordering of aircraft in terms of altitude and airway.

Second, each center was equipped with a large plotting table map where controllers pushed markers around to correspond to the position reports from aircraft. A small flag on each marker gave the flight number, so it could easily be correlated to a flight strip on one of the boards mounted around the plotting table. This basic concept of air traffic control, of a flight strip and a position marker, is still in use today.

Radar

The Second World War changed aviation more than any other event of history. Among the many advancements were two British inventions of particular significance: first, the jet engine, which would make modern passenger airliners practical. Second, the radar, and more specifically the magnetron. This was a development of such significance that the British government treated it as a secret akin to nuclear weapons; indeed, the UK effectively traded radar technology to the US in exchange for participation in US nuclear weapons research.

Radar created radical new possibilities for air defense, and complimented previous air defense development in Britain. During WWI, the organization tasked with defending London from aerial attack had developed a method called "ground-controlled interception" or GCI. Under GCI, ground-based observers identify possible targets and then direct attack aircraft towards them via radio. The advent of radar made GCI tremendously more powerful, allowing a relatively small number of radar-assisted air defense centers to monitor for inbound attack and then direct defenders with real-time vectors.

In the first implementation, radar stations reported contacts via telephone to "filter centers" that correlated tracks from separate radars to create a unified view of the airspace---drawn in grease pencil on a preprinted map. Filter center staff took radar and visual reports and updated the map by moving the marks. This consolidated information was then provided to air defense bases, once again by telephone.

Later technical developments in the UK made the process more automated. The invention of the "plan position indicator" or PPI, the type of radar scope we are all familiar with today, made the radar far easier to operate and interpret. Radar sets that automatically swept over 360 degrees allowed each radar station to see all activity in its area, rather than just aircraft passing through a defensive line. These new capabilities eliminated the need for much of the manual work: radar stations could see attacking aircraft and defending aircraft on one PPI, and communicated directly with defenders by radio.

It became routine for a radar operator to give a pilot navigation vectors by radio, based on real-time observation of the pilot's position and heading. A controller took strategic command of the airspace, effectively steering the aircraft from a top-down view. The ease and efficiency of this workflow was a significant factor in the end of the Battle of Britain, and its remarkable efficacy was noticed in the US as well.

At the same time, changes were afoot in the US. WWII was tremendously disruptive to civil aviation; while aviation technology rapidly advanced due to wartime needs those same pressing demands lead to a slowdown in nonmilitary activity. A heavy volume of military logistics flights and flight training, as well as growing concerns about defending the US from an invasion, meant that ATC was still a priority. A reorganization of the Bureau of Air Commerce replaced it with the Civil Aeronautics Authority, or CAA. The CAA's role greatly expanded as it assumed responsibility for airport control towers and commissioned new en route centers.

As WWII came to a close, CAA en route control centers began to adopt GCI techniques. By 1955, the name Air Route Traffic Control Center (ARTCC) had been adopted for en route centers and the first air surveillance radars were installed. In a radar-equipped ARTCC, the map where controllers pushed markers around was replaced with a large tabletop PPI built to a Navy design. The controllers still pushed markers around to track the identities of aircraft, but they moved them based on their corresponding radar "blips" instead of radio position reports.

Air Defense

After WWII, post-war prosperity and wartime technology like the jet engine lead to huge growth in commercial aviation. During the 1950s, radar was adopted by more and more ATC facilities (both "terminal" at airports and "en route" at ARTCCs), but there were few major changes in ATC procedure. With more and more planes in the air, tracking flight plans and their corresponding positions became labor intensive and error-prone. A particular problem was the increasing range and speed of aircraft, and corresponding longer passenger flights, that meant that many aircraft passed from the territory of one ARTCC into another. This required that controllers "hand off" the aircraft, informing the "next" ARTCC of the flight plan and position at which the aircraft would enter their airspace.

In 1956, 128 people died in a mid-air collision of two commercial airliners over the Grand Canyon. In 1958, 49 people died when a military fighter struck a commercial airliner over Nevada. These were not the only such incidents in the mid-1950s, and public trust in aviation started to decline. Something had to be done. First, in 1958 the CAA gave way to the Federal Aviation Administration. This was more than just a name change: the FAA's authority was greatly increased compared to the CAA, most notably by granting it authority over military aviation.

This is a difficult topic to explain succinctly, so I will only give broad strokes. Prior to 1958, military aviation was completely distinct from civil aviation, with no coordination and often no communication at all between the two. This was, of course, a factor in the 1958 collision. Further, the 1956 collision, while it did not involve the military, did result in part from communications issues between separate distinct CAA facilities and the airline's own control facilities. After 1958, ATC was completely unified into one organization, the FAA, which assumed the work of the military controllers of the time and some of the role of the airlines. The military continues to have its own air controllers to this day, and military aircraft continue to include privileges such as (practical but not legal) exemption from transponder requirements, but military flights over the US are still beholden to the same ATC as civil flights. Some exceptions apply, void where prohibited, etc.

The FAA's suddenly increased scope only made the practical challenges of ATC more difficult, and commercial aviation numbers continued to rise. As soon as the FAA was formed, it was understood that there needed to be major investments in improving the National Airspace System. While the first couple of years were dominated by the transition, the FAA's second director (Najeeb Halaby) prepared two lengthy reports examining the situation and recommending improvements. One of these, the Beacon report (also called Project Beacon), specifically addressed ATC. The Beacon report's recommendations included massive expansion of radar-based control (called "positive control" because of the controller's access to real-time feedback on aircraft movements) and new control procedures for airways and airports. Even better, for our purposes, it recommended the adoption of general-purpose computers and software to automate ATC functions.

Meanwhile, the Cold War was heating up. US air defense, a minor concern in the few short years after WWII, became a higher priority than ever before. The Soviet Union had long-range aircraft capable of reaching the United States, and nuclear weapons meant that only a few such aircraft had to make it to cause massive destruction. Considering the vast size of the United States (and, considering the new unified air defense command between the United States and Canada, all of North America) made this a formidable challenge.

During the 1950s, the newly minted Air Force worked closely with MIT's Lincoln Laboratory (an important center of radar research) and IBM to design a computerized, integrated, networked system for GCI. When the Air Force committed to purchasing the system, it was christened the Semi-Automated Ground Environment, or SAGE. SAGE is a critical juncture in the history of the computer and computer communications, the first system to demonstrate many parts of modern computer technology and, moreover, perhaps the first large-scale computer system of any kind.

SAGE is an expansive topic that I will not take on here; I'm sure it will be the focus of a future article but it's a pretty well-known and well-covered topic. I have not so far felt like I had much new to contribute, despite it being the first item on my "list of topics" for the last five years. But one of the things I want to tell you about SAGE, that is perhaps not so well known, is that SAGE was not used for ATC. SAGE was a purely military system. It was commissioned by the Air Force, and its numerous operating facilities (called "direction centers") were located on Air Force bases along with the interceptor forces they would direct.

However, there was obvious overlap between the functionality of SAGE and the needs of ATC. SAGE direction centers continuously received tracks from remote data sites using modems over leased telephone lines, and automatically correlated multiple radar tracks to a single aircraft. Once an operator entered information about an aircraft, SAGE stored that information for retrieval by other radar operators. When an aircraft with associated data passed from the territory of one direction center to another, the aircraft's position and related information were automatically transmitted to the next direction center by modem.

One of the key demands of air defense is the identification of aircraft---any unknown track might be routine commercial activity, or it could be an inbound attack. The air defense command received flight plan data on commercial flights (and more broadly all flights entering North America) from the FAA and entered them into SAGE, allowing radar operators to retrieve "flight strip" data on any aircraft on their scope.

Recognizing this interconnection with ATC, as soon as SAGE direction centers were being installed the Air Force started work on an upgrade called SAGE Air Traffic Integration, or SATIN. SATIN would extend SAGE to serve the ATC use-case as well, providing SAGE consoles directly in ARTCCs and enhancing SAGE to perform non-military safety functions like conflict warning and forward projection of flight plans for scheduling. Flight strips would be replaced by teletype output, and in general made less necessary by the computer's ability to filter the radar scope.

Experimental trial installations were made, and the FAA participated readily in the research efforts. Enhancement of SAGE to meet ATC requirements seemed likely to meet the Beacon report's recommendations and radically improve ARTCC operations, sooner and cheaper than development of an FAA-specific system.

As it happened, well, it didn't happen. SATIN became interconnected with another planned SAGE upgrade to the Super Combat Centers (SCC), deep underground combat command centers with greatly enhanced SAGE computer equipment. SATIN and SCC planners were so confident that the last three Air Defense Sectors scheduled for SAGE installation, including my own Albuquerque, were delayed under the assumption that the improved SATIN/SCC equipment should be installed instead of the soon-obsolete original system. SCC cost estimates ballooned, and the program's ambitions were reduced month by month until it was canceled entirely in 1960. Albuquerque never got a SAGE installation, and the Albuquerque air defense sector was eliminated by reorganization later in 1960 anyway.

Flight Service Stations

Remember those Flight Service Stations, the ones that were originally built by the Post Office? One of the oddities of ATC is that they never went away. FSS were transferred to the CAB, to the CAA, and then to the FAA. During the 1930s and 1940s many more were built, expanding coverage across much of the country.

Throughout the development of ATC, the FSS remained responsible for non-control functions like weather briefing and flight plan management. Because aircraft operating under instrument flight rules must closely comply with ATC, the involvement of FSS in IFR flights is very limited, and FSS mostly serve VFR traffic.

As ATC became common, the FSS gained a new and somewhat odd role: playing go-between for ATC. FSS were more numerous and often located in sparser areas between cities (while ATC facilities tended to be in cities), so especially in the mid-century, pilots were more likely to be able to reach an FSS than ATC. It was, for a time, routine for FSS to relay instructions between pilots and controllers. This is still done today, although improved communications have made the need much less common.

As weather dissemination improved (another topic for a future post), FSS gained access to extensive weather conditions and forecasting information from the Weather Service. This connectivity is bidirectional; during the midcentury FSS not only received weather forecasts by teletype but transmitted pilot reports of weather conditions back to the Weather Service. Today these communications have, of course, been computerized, although the legacy teletype format doggedly persists.

There has always been an odd schism between the FSS and ATC: they are operated by different departments, out of different facilities, with different functions and operating practices. In 2005, the FAA cut costs by privatizing the FSS function entirely. Flight service is now operated by Leidos, one of the largest government contractors. All FSS operations have been centralized to one facility that communicates via remote radio sites.

While flight service is still available, increasing automation has made the stations far less important, and the general perception is that flight service is in its last years. Last I looked, Leidos was not hiring for flight service and the expectation was that they would never hire again, retiring the service along with its staff.

Flight service does maintain one of my favorite internet phenomenon, the phone number domain name: 1800wxbrief.com. One of the odd manifestations of the FSS/ATC schism and the FAA's very partial privatization is that Leidos maintains an online aviation weather portal that is separate from, and competes with, the Weather Service's aviationweather.gov. Since Flight Service traditionally has the responsibility for weather briefings, it is honestly unclear to what extent Leidos vs. the National Weather Service should be investing in aviation weather information services. For its part, the FAA seems to consider aviationweather.gov the official source, while it pays for 1800wxbrief.com. There's also weathercams.faa.gov, which duplicates a very large portion (maybe all?) of the weather information on Leidos's portal and some of the NWS's. It's just one of those things. Or three of those things, rather. Speaking of duplication due to poor planning...

The National Airspace System

Left in the lurch by the Air Force, the FAA launched its own program for ATC automation. While the Air Force was deploying SAGE, the FAA had mostly been waiting, and various ARTCCs had adopted a hodgepodge of methods ranging from one-off computer systems to completely paper-based tracking. By 1960 radar was ubiquitous, but different radar systems were used at different facilities, and correlation between radar contacts and flight plans was completely manual. The FAA needed something better, and with growing congressional support for ATC modernization, they had the money to fund what they called National Airspace System En Route Stage A.

Further bolstering historical confusion between SAGE and ATC, the FAA decided on a practical, if ironic, solution: buy their own SAGE.

In an upcoming article, we'll learn about the FAA's first fully integrated computerized air traffic control system. While the failed detour through SATIN delayed the development of this system, the nearly decade-long delay between the design of SAGE and the FAA's contract allowed significant technical improvements. This "New SAGE," while directly based on SAGE at a functional level, used later off-the-shelf computer equipment including the IBM System/360, giving it far more resemblance to our modern world of computing than SAGE with its enormous, bespoke AN/FSQ-7.

And we're still dealing with the consequences today!

[1] It also laid the groundwork for the consolidation of the industry, with a 1930 decision that took air mail contracts away from most of the smaller companies and awarded them instead to the precursors of United, TWA, and American Airlines.

You know sometimes a technology just sort of... comes and goes? Without leaving much of an impression? And then gets lodged in your brain for the next decade? Let's talk about one of those: the iBeacon.

I think the reason that iBeacons loom so large in my memory is that the technology was announced at WWDC in 2013. Picture yourself in 2013: Steve Jobs had only died a couple of years ago, Apple was still widely viewed as a visionary leader in consumer technology, and WWDC was still happening. Back then, pretty much anything announced at an Apple event was a Big Deal that got Big Coverage. Even, it turns out, if it was a minor development for a niche application. That's the iBeacon, a specific solution to a specific problem. It's not really that interesting, but the valance of it's Apple origin makes it seem cool?

iBeacon Technology

Let's start out with what iBeacon is, as it's so simple as to be underwhelming. Way back in the '00s, a group of vendors developed a sort of "Diet Bluetooth": a wireless protocol that was directly based on Bluetooth but simplified and optimized for low-power, low-data-rate devices. This went through an unfortunate series of names, including the delightful Wibree, but eventually settled on Bluetooth Low Energy (BLE). BLE is not just lower-power, but also easier to implement, so it shows up in all kinds of smart devices today. Back in 2011, it was quite new, and Apple was one of the first vendors to adopt it.

BLE is far less connection-oriented than regular Bluetooth; you may have noticed that BLE devices are often used entirely without conventional "pairing." A lot of typical BLE profiles involve just broadcasting some data into the void for any device that cares (and is in short range) to receive, which is pretty similar to ANT+ and unsurprisingly appears in ANT+-like applications of fitness monitors and other sensors. Of course, despite the simpler association model, BLE applications need some way to find devices, so BLE provides an advertising mechanism in which devices transmit their identifying info at regular intervals.

And that's all iBeacon really is: a standard for very simple BLE devices that do nothing but transmit advertisements with a unique ID as the payload. Add a type field on the advertising packet to specify that the device is trying to be an iBeacon and you're done. You interact with an iBeacon by receiving its advertisements, so you know that you are near it. Any BLE device with advertisements enabled could be used this way, but iBeacons are built only for this purpose.

The applications for iBeacon are pretty much defined by its implementation in iOS; there's not much of a standard even if only for the reason that there's not much to put in a standard. It's all obvious. iOS provides two principle APIs for working with iBeacons: the region monitoring API allows an app to determine if it is near an iBeacon, including registering the region so that the app will be started when the iBeacon enters range. This is useful for apps that want to do something in response to the user being in a specific location.

The ranging API allows an app to get a list of all of the nearby iBeacons and a rough range from the device to the iBeacon. iBeacons can actually operate at substantial ranges---up to hundreds of meters for more powerful beacons with external power, so ranging mode can potentially be used as sort of a lightweight local positioning system to estimate the location of the user within a larger space.

iBeacon IDs are in the format of a UUID, followed by a "major" number and a "minor" number. There are different ways that these get used, especially if you are buying cheap iBeacons and not reconfiguring them, but the general idea is roughly that the UUID identifies the operator, the major a deployment, and the minor a beacon within the deployment. In practice this might be less common than just every beacon having its own UUID due to how they're sourced. It would be interesting to survey iBeacon applications to see which they do.

Promoted Applications

So where do you actually use these? Retail! Apple seems to have designed the iBeacon pretty much exclusively for "proximity marketing" applications in the retail environment. It goes something like this: when you're in a store and open that store's app, the app will know what beacons you are nearby and display relevant content. For example, in a grocery store, the grocer's app might offer e-coupons for cosmetics when you are in the cosmetics section.

That's, uhh, kind of the whole thing? The imagined universe of applications around the launch of iBeacon was pretty underwhelming to me, even at the time, and it still seems that way. That's presumably why iBeacon had so little success in consumer-facing applications. You might wonder, who actually used iBeacons?

Well, Apple did, obviously. During 2013 and into 2014 iBeacons were installed in all US Apple stores, and prompted the Apple Store app to send notifications about upgrade offers and other in-store deals. Unsurprisingly, this Apple Store implementation was considered the flagship deployment. It generated a fair amount of press, including speculation as to whether or not it would prove the concept for other buyers.

Around the same time, Apple penned a deal with Major League Baseball that would see iBeacons installed in MLB stadiums. For the 2014 season, MLB Advanced Marketing, a joint venture of team owners, had installed iBeacon technology in 20 stadiums.

Baseball fans will be able to utilize iBeacon technology within MLB.com At The Ballpark when the award-winning app's 2014 update is released for Opening Day. Complete details on new features being developed by MLBAM for At The Ballpark, including iBeacon capabilities, will be available in March.

What's the point? the iBeacons "enable the At The Ballpark app to play specific videos or offer coupons."

This exact story repeats for other retail companies that have picked the technology up at various points, including giants like Target and WalMart. The iBeacons are simply a way to target advertising based on location, with better indoor precision and lower power consumption than GPS. Aiding these applications along, Apple integrated iBeacon support into the iOS location framework and further blurred the lines between iBeacon and other positioning services by introducing location-based-advertising features that operated on geofencing alone.

Some creative thinkers did develop more complex applications for the iBeacon. One of the early adopters was a company called Exact Editions, which prepared the Apple Newsstand version of a number of major magazines back when "readable on iPad" was thought to be the future of print media. Exact Editions explored a "read for free" feature where partner magazines would be freely accessible to users at partnering locations like coffee shops and book stores. This does not seem to have been a success, but using the proximity of an iBeacon to unlock some paywalled media is at least a little creative, if probably ill-advised considering security considerations we'll discuss later.

The world of applications raises interesting questions about the other half of the mobile ecosystem: how did this all work on Android? iOS has built-in support for iBeacons. An operating system service scans for iBeacons and dispatches notifications to apps as appropriate. On Android, there has never been this type of OS-level support, but Android apps have access to relatively rich low-level Bluetooth functionality and can easily scan for iBeacons themselves. Several popular libraries exist for this purpose, and it's not unusual for them to be used to give ported cross-platform apps more or less equivalent functionality. These apps do need to run in the background if they're to notify the user proactively, but especially back in 2013 Android was far more generous about background work than iOS.

iBeacons found expanded success through ShopKick, a retail loyalty platform that installed iBeacons in locations of some major retailers like American Eagle. These powered location-based advertising and offers in the ShopKick app as well as retailer-specific apps, which is kind of the start of a larger, more seamless network, but it doesn't seem to have caught on. Honestly, consumers just don't seem to want location-based advertising that much. Maybe because, when you're standing in an American Eagle, getting ads for products carried in the American Eagle is inane and irritating. iBeacons sort of foresaw cooler screens in this regard.

To be completely honest, I'm skeptical that anyone ever really believed in the location-based advertising thing. I mean, I don't know, the advertising industry is pretty good at self-deception, but I don't think there were ever any real signs of hyper-local smartphone-based advertising taking off. I think the play was always data collection, and advertising and special offers just provided a convenient cover story.

Real Applications

iBeacons are one of those technologies that feels like a flop from a consumer perspective but has, in actuality, enjoyed surprisingly widespread deployments. The reason, of course, is data mining.

To Apple's credit, they took a set of precautions in the design of the iBeacon iOS features that probably felt sufficient in 2013. Despite the fact that a lot of journalists described iBeacons as being used to "notify a user to install an app," that was never actually a capability (a very similar-seeming iOS feature attached to Siri actually used conventional geofencing rather than iBeacons). iBeacons only did anything if the user already had an app installed that either scanned for iBeacons when in the foreground or registered for region notifications.

In theory, this limited iBeacons to companies with which consumers already had some kind of relationship. What Apple may not have foreseen, or perhaps simply accepted, is the incredible willingness of your typical consumer brand to sell that relationship to anyone who would pay.

iBeacons became, in practice, just another major advancement in pervasive consumer surveillance. The New York Times reported in 2019 that popular applications were including SDKs that reported iBeacon contacts to third-party consumer data brokers. This data became one of several streams that was used to sell consumer location history to advertisers.

It's a little difficult to assign blame and credit, here. Apple, to their credit, kept iBeacon features in iOS relatively locked down. This suggests that they weren't trying to facilitate massive location surveillance. That said, Apple always marketed iBeacon to developers based on exactly this kind of consumer tracking and micro-targeting, they just intended for it to be done under the auspices of a single brand. That industry would obviously form data exchanges and recruit random apps into reporting everything in your proximity isn't surprising, but maybe Apple failed to foresee it.

They certainly weren't the worst offender. Apple's promotion of iBeacon opened the floodgates for everyone else to do the same thing. During 2014 and 2015, Facebook started offering bluetooth beacons to businesses that were ostensibly supposed to facilitate in-app special offers (though I'm not sure that those ever really materialized) but were pretty transparently just a location data collection play.

Google jumped into the fray in their Signature Google style, with an offering that was confusing, semi-secret, incoherently marketed, and short lived. Google's Project Beacon, or Google My Business, also shipped free Bluetooth beacons out to businesses to give Android location services a boost. Google My Business seems to have been the source of a fair amount of confusion even at the time, and we can virtually guarantee that (as reporters speculated at the time) Google was intentionally vague and evasive about the system to avoid negative attention from privacy advocates.

In the case of Facebook, well, they don't have the level of opsec that Google does so things are a little better documented:

Leaked documents show that Facebook worried that users would 'freak out' and spread 'negative memes' about the program. The company recently removed the Facebook Bluetooth beacons section from their website.

The real deployment of iBeacons and closely related third-party iBeacon-like products [1] occurred at massive scale but largely in secret. It became yet another dark project of the advertising-industrial complex, perhaps the most successful yet of a long-running series of retail consumer surveillance systems.

Payments

One interesting thing about iBeacon is how it was compared to NFC. The two really aren't that similar, especially considering the vast difference in usable ranges, but NFC was the first radio technology to be adopted for "location marketing" applications. "Tap your phone to see our menu," kinds of things. Back in 2013, Apple had rather notably not implemented NFC in its products, despite its increasing adoption on Android.

But, there is much more to this story than learning about new iPads and getting a surprise notification that you are eligible for a subsidized iPhone upgrade. What we're seeing is Apple pioneering the way mobile devices can be utilized to make shopping a better experience for consumers. What we're seeing is Apple putting its money where its mouth is when it decided not to support NFC. (MacObserver)

Some commentators viewed iBeacon as Apple's response to NFC, and I think there's more to that than you might think. In early marketing, Apple kept positioning iBeacon for payments. That's a little weird, right, because iBeacons are a purely one-way broadcast system.

Still, part of Apple's flagship iBeacon implementation was a payment system:

Here's how he describes the purchase he made there, using his iPhone and the EasyPay system: "We started by using the iPhone to scan the product barcode and then we had to enter our Apple ID, pretty much the way we would for any online Apple purchase [using the credit card data on file with one's Apple account]. The one key difference was that this transaction ended with a digital receipt, one that we could show to a clerk if anyone stopped us on the way out."

Apple Wallet only kinda-sorta existed at the time, although Apple was clearly already midway into a project to expand into consumer payments. It says a lot about this point in time in phone-based payments that several reporters talk about iBeacon payments as a feature of iTunes, since Apple was mostly implementing general-purpose billing by bolting it onto iTunes accounts.

It seems like what happened is that Apple committed to developing a pay-by-phone solution, but decided against NFC. To be competitive with other entrants in the pay-by-phone market, they had to come up with some kind of technical solution to interact with retail POS, and iBeacon was their choice. From a modern perspective this seems outright insane; like, Bluetooth broadcasts are obviously not the right way to initiate a payment flow, and besides, there's a whole industry-standard stack dedicated to that purpose... built on NFC.

But remember, this was 2013! EMV was not yet in meaningful use in the US; several major banks and payment networks had just committed to rolling it out in 2012 and every American can tell you that the process was long and torturous. Because of the stringent security standards around EMV, Android devices did not implement EMV until ARM secure enclaves became widely available. EMVCo, the industry body behind EMV, did not have a certification program for smartphones until 2016.

Android phones offered several "tap-to-pay" solutions, from Google's frequently rebranded Google Wallet^w^wAndroid Pay^w^wGoogle Wallet to Verizon's embarrassingly rebranded ISIS^wSoftcard and Samsung Pay. All of these initially relied on proprietary NFC protocols with bespoke payment terminal implementations. This was sketchy enough, and few enough phones actually had NFC, that the most successful US pay-by-phone implementations like Walmart's and Starbucks' used barcodes for communication. It would take almost a decade before things really settled down and smartphones all just implemented EMV.

So, in that context, Apple's decision isn't so odd. They must have figured that iBeacon could solve the same "initial handshake" problem as Walmart's QR codes, but more conveniently and using radio hardware that they already included in their phones. iBeacon-based payment flows used the iBeacon only to inform the phone of what payment devices were nearby, everything else happened via interaction with a cloud service or whatever mechanism the payment vendor chose to implement. Apple used their proprietary payments system through what would become your Apple Account, PayPal slapped together an iBeacon-based fast path to PayPal transfers, etc.

I don't think that Apple's iBeacon-based payments solution ever really shipped. It did get some use, most notably by Apple, but these all seem to have been early-stage implementations, and the complete end-to-end SDK that a lot of developers expected never landed.

You might remember that this was a very chaotic time in phone-based payments, solutions were coming and going. When Apple Pay was properly announced a year after iBeacons, there was little mention of Bluetooth. By the time in-store Apple Pay became common, Apple had given up and adopted NFC.

Limitations

One of the great weaknesses of iBeacon was the security design, or lack thereof. iBeacon advertisements were sent in plaintext with no authentication of any type. This did, of course, radically simplify implementation, but it also made iBeacon untrustworthy for any important purpose. It is quite trivial, with a device like an Android phone, to "clone" any iBeacon and transmit its identifiers wherever you want. This problem might have killed off the whole location-based-paywall-unlocking concept had market forces not already done so. It also opens the door to a lot of nuisance attacks on iBeacon-based location marketing, which may have limited the depth of iBeacon features in major apps.

iBeacon was also positioned as a sort of local positioning system, but it really wasn't. iBeacon offers no actual time-of-flight measurements, only RSSI-based estimation of range. Even with correct on-site calibration (which can be aided by adjusting a fixed RSSI-range bias value included in some iBeacon advertisements) this type of estimation is very inaccurate, and in my little experiments with a Bluetooth beacon location library I can see swings from 30m to 70m estimated range based only on how I hold my phone. iBeacon positioning has never been accurate enough to do more than assert whether or not a phone is "near" the beacon, and "near" can take on different values depending on the beacon's transmit power.

Developers have long looked towards Bluetooth as a potential local positioning solution, and it's never quite delivered. The industry is now turning towards Ultra-Wideband or UWB technology, which combines a high-rate, high-bandwidth radio signal with a time-of-flight radio ranging protocol to provide very accurate distance measurements. Apple is, once again, a technical leader in this field and UWB radios have been integrated into the iPhone 11 and later.

Senescence

iBeacon arrived to some fanfare, quietly proliferated in the shadows of the advertising industry, and then faded away. The Wikipedia article on iBeacons hasn't really been updated since support on Windows Phone was relevant. Apple doesn't much talk about iBeacons any more, and their compatriots Facebook and Google both sunset their beacon programs years ago.

Part of the problem is, well, the pervasive surveillance thing. The idea of Bluetooth beacons cooperating with your phone to track your every move proved unpopular with the public, and so progressively tighter privacy restrictions in mobile operating systems and app stores have clamped down on every grocery store app selling location data to whatever broker bids the most. I mean, they still do, but it's gotten harder to use Bluetooth as an aid. Even Android, the platform of "do whatever you want in the background, battery be damned," strongly discourages Bluetooth scanning by non-foreground apps.

Still, the basic technology remains in widespread use. BLE beacons have absolutely proliferated, there are plenty of apps you can use to list nearby beacons and there almost certainly are nearby beacons. One of my cars has, like, four separate BLE beacons going on all the time, related to a phone-based keyless entry system that I don't think the automaker even supports any more. Bluetooth beacons, as a basic primitive, are so useful that they get thrown into all kinds of applications. My earbuds are a BLE beacon, which the (terrible, miserable, no-good) Bose app uses to detect their proximity when they're paired to another device. A lot of smart home devices like light bulbs are beacons. The irony, perhaps, of iBeacon-based location tracking is that it's a victim of its own success. There is so much "background" BLE beacon activity that you scarcely need to add purpose-built beacons to track users, and only privacy measures in mobile operating systems and the beacons themselves (some of which rotate IDs) save us.

Apple is no exception to the widespread use of Bluetooth beacons: iBeacon lives on in virtually every apple device. If you do try out a Bluetooth beacon scanning app, you'll discover pretty much every Apple product in a 30 meter radius. From MacBooks Pro to Airpods, almost all Apple products transmit iBeacon advertisements to their surroundings. These are used for the initial handshake process of peer-to-peer features like Airdrop, and Find My/AirTag technology seems to be derived from the iBeacon protocol (in the sense that anything can be derived from such a straightforward design). Of course, pretty much all of these applications now randomize identifiers to prevent passive use of device advertisements for long-term tracking.

Here's some good news: iBeacons are readily available in a variety of form factors, and they are very cheap. Lots of libraries exist for working with them. If you've ever wanted some sort of location-based behavior for something like home automation, iBeacons might offer a good solution. They're neat, in an old technology way. Retrotech from the different world of 2013.

It's retro in more ways than one. It's funny, and a bit quaint, to read the contemporary privacy concerns around iBeacon. If only they had known how bad things would get! Bluetooth beacons were the least of our concerns.

[1] Things can be a little confusing here because the iBeacon is such a straightforward concept, and Apple's implementation is so simple. We could define "iBeacon" as including only officially endorsed products from Apple affiliates, or as including any device that behaves the same as official products (e.g. by using the iBeacon BLE advertisement type codes), or as any device that is performing substantially the same function (but using a different advertising format). I usually mean the latter of these three as there isn't really much difference between an iBeacon and ten million other BLE beacons that are doing the same thing with a slightly different identifier format. Facebook and Google's efforts fall into this camp.

When we last talked about Troposcatter, it was Pole Vault. Pole Vault was the first troposcatter communications network, on the east coast of Canada. It would not be alone for long. By the time the first Pole Vault stations were complete, work was already underway on a similar network for Alaska: the White Alice Communication System, WACS.

Alaska has long posed a challenge for communications. In the 1860s, Western Union wanted to extend their telegraph network from the United States into Europe. Although the technology would be demonstrated shortly after, undersea telegraph cables were still notional and it seemed that a route that minimized the ocean crossing would be preferable---of course, that route maximized the length on land, stretching through present-day Alaska and Siberia on each side of the Bering Strait. This task proved more formidable than Western Union had imagined, and the first transatlantic telegraph cable (on a much further south crossing) was completed before the arctic segments of the overland route. The "Western Union Telegraph Expedition" abandoned its work, leaving a telegraph line well into British Columbia that would serve as one of the principle communications assets in the region for decades after.

This ill-fated telegraph line failed to link San Francisco to Moscow, but its aftermath included a much larger impact on Russian interests in North America: the purchase of Alaska in 1867. Shortly after, the US military began its expansion into the new frontier. The Army Signal Corps, mostly to fulfill its function in observing the weather, built and staffed small installations that stretched further and further west. Later, in the 1890s, a gold rush brought a sudden influx of American settlers to Alaska's rugged terrain. The sudden economic importance of Klondike, and the rather colorful personalities of the prospectors looking to exploit it, created a much larger need for military presence. Fortuitously, many of the forts present had been built by the Signal Corps, which had already started on lines of communication. Construction was difficult, though, and without Alaskan communications as major priority there was only minimal coverage.

Things changed in 1900, when Congress appropriated a substantial budget to the Washington-Alaska Military Cable and Telegraph System. The Signal Corps set on Alaska like, well, like an army, and extensive telegraph and later telephone lines were built to link the various military outposts. Later renamed the Alaska Communications System, these cables brought the first telecommunication to much of Alaska. The arrival of the telegraph was quite revolutionary for remote towns, who could now receive news in real-time that had previously been delayed by as much as a year [1]. Telegraphy was important to civilians as well, something that Congress had anticipated: The original act authorizing the Alaska Communications System dictated that it would carry commercial traffic as well. The military had an unusual role in Alaska, and one aspect of it was telecommunications provider.

In 1925, an outbreak of diphtheria began to claim the lives of children in Nome, a town in far western Alaska on the Seward Peninsula. The daring winter delivery of antidiphtheria serum by dog sled is widely remembered due to its tangential connection to the Iditarod, but there were two sides of the "serum run." The message from Nome's sole doctor requesting the urgent shipment was transmitted from Nome to the Public Health Service in DC over the Alaska Communications System. It gives us some perspective on the importance of the telegraph in Alaska that the 600 mile route to Nome took five days and many feats of heroism---but at the same time could be crossed instantaneously by telegrams.

The Alaska Communications System included some use of radio from the beginning. A pair of HF radio stations specifically handled traffic for Nome, covering a 100-mile stretch too difficult for even the intrepid Signal Corps. While not a totally new technology to the military, radio was quite new to the telegraph business, and the ACS to Nome was probably the first commercial radiotelegraph system on the continent. By the 1930s, the condition of the Alaskan telegraph cables had decayed while demand for telephony had increased. Much of ACS was upgraded and modernized to medium-frequency radiotelephone links. In towns small and large, even in Anchorage itself, the sole telephone connection to the contiguous United States was an ACS telephone installed in the general store.

Alaskan communications became an even greater focus of the military with the onset of the Second World War. A few weeks after Pearl Harbor, the Japanese attacked Fort Mears in the Aleutian Islands. Fort Mears had no telecommunications connections, so despite the proximity of other airbases support was slow to come. The lack of a telegraph or telephone line contributed to 43 deaths and focused attention on the ACS. By 1944, the Army Signal Corps had a workforce of 2,000 dedicated to Alaska.

WWII brought more than one kind of attention to Alaska. Several Japanese assaults on the Aleutian islands represented the largest threats to American soil outside of Pearl Harbor, showing both Alaska's vulnerability and the strategic importance given to it by its relative proximity to Eurasia. WWII ended but, in 1949, the USSR demonstrated an atomic weapon. A combination of Soviet expansionism and the new specter of nuclear war turned military planners towards air defense. Like the Canadian Maritimes in the East, Alaska covered a huge swath of the airspace through which Soviet bombers might approach the US. Alaska was, once again, a battleground.

The early Cold War military buildup of Alaska was particularly heavy on air defense. During the late '40s and early '50s, more than a dozen new radar and control sites were built. The doctrine of ground-controlled interception requires real-time communication between radar centers, stressing the limited number of voice channels available on the ACS. As early as 1948, the Signal Corps had begun experiments to choose an upgrade path. Canadian early-warning radar networks, including the Distant Early Warning Line, were on the drawing board and would require many communications channels in particularly remote parts of Alaska.

Initially, point-to-point microwave was used in relatively favorable terrain (where the construction of relay stations about every 50 miles was practical). For the more difficult segments, the Signal Corps found that VHF radio could provide useful communications at ranges over 100 miles. VHF radiotelephones were installed at air defense radar stations, but there was a big problem: the airspace surveillance radar of the 1950s also operated in the VHF band, and caused so much interference with the radiotelephones that they were difficult to use. The radar stations were probably the most important users of the network, so VHF would have to be abandoned.

In 1954, a military study group was formed to evaluate options for the ACS. That group, in turn, requested a proposal from AT&T. Bell Laboratories had been involved in the design and evaluation of Pole Vault, the first sites of which had been completed two years before, so they naturally positioned troposcatter as the best option.

It is worth mentioning the unusual relationship AT&T had with Alaska, or rather, the lack of one. While the Bell System enjoyed a monopoly on telephony in most of the United States [2], they had never expanded into Alaska. Alaska was only a territory, after all, and a very sparsely populated one at that. The paucity of long-distance leads to or from Alaska (only one connected to Anchorage, for example) limited the potential for integration of Alaska into the broader Bell System anyway. Long-distance telecommunications in Alaska were a military project, and AT&T was involved only as a vendor.

Because of the high cost of troposcatter stations, proven during Pole Vault construction, a hybrid was proposed: microwave stations could be spaced every 50 miles along the road network, while troposcatter would cover the long stretches without roads.

In 1955, the Signal Corps awarded Western Electric a contract for the White Alice Communications System. The Corps of Engineers surveyed the locations of 31 sites, verifying each by constructing a temporary antenna tower. The Corps of Engineers led construction of the first 11 sites, and the final 20 were built on contract by Western Electric itself. All sites used radio equipment furnished by Western Electric and were built to Western Electric designs.

Construction was far from straightforward. Difficult conditions delayed completion of the original network until 1959, two years later than intended. A much larger issue, though, was the budget. The original WACS was expected to cost $38 million. By the time the first 31 sites were complete, the bill totaled $113 million---equivalent to over a billion dollars today. Western Electric had underestimated not only the complexity of the sites but the difficulty of their construction. A WECo report read:

On numerous occasions, the men were forced to surrender before the onslaught of cold, wind and snow and were immobilized for days, even weeks . This ordeal of waiting was of times made doubly galling by the knowledge that supplies and parts needed for the job were only a few miles distant but inaccessible because the white wall of winter had become impenetrable

WACS initial capability included 31 stations, of which 22 were troposcatter and the remainder only microwave (using Western Electric's TD-2). A few stations were equipped with both troposcatter and microwave, serving as relays between the two carriers.

In 1958, construction started on the Ballistic Missile Early Warning System or BMEWS. BMEWS was an over-the-horizon radar system intended to provide early warning of a Soviet attack. BMEWS would provide as little as 15 minutes warning, requiring that alerts reach NORAD in Colorado as quickly as possible. One BMEWS set was installed in Greenland, where the Pole Vault system was expanded to provide communications. Similarly, the BMEWS set at Clear Missile Early Warning Station in central Alaska relied on White Alice. Planners were concerned about the ability of the Soviet Union to suppress an alert by destroying infrastructure, so two redundant chains of microwave sites were added to White Alice. One stretched from Clear to Ketchikan where it connected to an undersea cable to Seattle. The other went east, towards Canada, where it met existing telephone cables on the Alaska Highway.

A further expansion of White Alice started the next year, in 1959. Troposcatter sites were extended through the Aleutian islands in "Project Stretchout" to serve new DEW Line stations. During the 1960s, existing WACS sites were expanded and new antennas were installed at Air Force installations. These were generally microwave links connecting the airbases to existing troposcatter stations.

In total, WACS reached 71 sites. Four large sites served as key switching points with multiple radio links and telephone exchanges. Pedro Dome, for example, had a 15,000 square foot communications building with dormitories, a power plant, and extensive equipment rooms. Support facilities included a vehicle maintenance building, storage warehouse, and extensive fuel tanks. A few WACS sites even had tramways for access between the "lower camp" (where equipment and personnel were housed) and the "upper camp" (where the antennas were located)... although they apparently did not fare well in the Alaskan conditions.

While Western Electric had initially planned for six people and 25 KW of power at each station, the final requirements were typically 20 people and 120-180 KW of generator capacity. Some sites stored over half a million gallons of fuel---conditions often meant that resupply was only possible during the summer.

Besides troposcatter and microwave radios, the equipment included tandem telephone exchanges. These are described in a couple of documents as "ATSS-4A," ATSS standing for Alaska Telephone Switching System. Based on the naming and some circumstantial evidence, I believe these were Western Electric 4A crossbar exchanges. They were later incorporated into AUTOVON, but also handled commercial long-distance traffic between Alaskan towns.

With troposcatter comes large antennas, and depending on connection lengths, WACS troposcatter antennas ranged from 30' dishes to 120' "billboard" antennas similar to those seen at Pole Vault sites. The larger antennas handled up to 50kW of transmit power. Some 60' and 120' antennas included their own fuel tanks and steam plants that heated the antennas through winter to minimize snow accumulation.

Nearly all of the equipment used by WACS was manufactured by Western Electric, with a lot of reuse of standard telephone equipment. For example, muxing on the troposcatter links used standard K-carrier (originally for telephone cables) and L-carrier (originally for coaxial cables). Troposcatter links operated at about 900 MHz with a wide bandwidth, and carrier two L-carrier supergroups (60 channels) and one K-carrier (12 channels) for a nominal capacity of 132 channels, although most stations did not have fully-populated L-carrier groups so actual capacity varied based on need. This was standard telephone carrier equipment in widespread use on the long-distance network, but some output modifications were made to suit the radio application.

The exception to the Western Electric rule was the radio sets themselves. They were manufactured by Radio Engineering Laboratories, the same company that built the radios for Pole Vault. REL pulled out all of the tricks they had developed for Pole Vault, and the longer WACS links used two antennas at different positions for space diversity. Each antenna had two feed horns, of orthoganal polarization, matching similar dual-transmitters for further diversity. REL equipment selected the best signal of the four available receiver options.

WACS represented an enormous improvement in Alaskan communications. The entire system was multi-channel with redundancy in many key parts of the network. Outside of the larger cities, WACS often brought the first usable long-distance telephone service. Even in Anchorage, WACS provided the only multi-channel connection. Despite these achievements, WACS was set for much the same fate as other troposcatter systems: obsolescence after the invention of communications satellites.

The experimental satellites Telstar 1 and 2 launched in the early 1960s, and the military began a shift towards satellite communications shortly after. Besides, the formidable cost of WACS had become a political issue. Maintenance of the system overran estimates by just as much as construction, and placing this cost on taxpayers was controversial since much of the traffic carried by the system consisted of regular commercial telephone calls. Besides, a general reticence to allocate money to WACS had lead to a general decay of the system. WACS capacity was insufficient for the rapidly increasing long-distance telephone traffic of the '60s, and due to decreased maintenance funding reliability was beginning to decline.

The retirement of a Cold War communications system is not unusual, but the particular fate of WACS is. It entered a long second life.

After acting as the sole long-distance provider for 60 years, the military began its retreat. In 1969, Congress passed the Alaska Communications Disposal Act. It called for complete divestment of the Alaska Communications System and WACS, to a private owner determined by a bidding process. Several large independent communications companies bid, but the winner was RCA. Committing to a $28.5 million purchase price followed by $30 million in upgrades, RCA reorganized the Alaska Communications System as RCA Alascom.

Transfer of the many ACS assets from the military to RCA took 13 years, involving both outright transfer of property and complex lease agreements on sites colocated with military installations. RCA's interest in Alaskan communications was closely connected to the coming satellite revolution: RCA had just built the Bartlett Earth Station, the first satellite ground station in Alaska. While Bartlett was originally an ACS asset owned by the Signal Corps, it became just the first of multiple ground stations that RCA would build for Alascom. Several of the new ground stations were colocated with WACS sites, establishing satellite as an alternative to the troposcatter links.

Alascom appears to have been the first domestic satellite voice network in commercial use, initially relying on a Canadian communications satellite [3]. In 1974, SATCOM 1 and 2 launched. These were not the first commercial communications satellites, but they represented a significant increase in capacity over previous commercial designs and are sometimes thought of as the true beginning of the satellite communications era. Both were built and owned by RCA, and Alascom took advantage of the new transponders.

At the same time, Alascom launched a modernization effort. 22 of the former WACS stations were converted to satellite ground stations, a project that took much of the '70s as Alascom struggled with the same conditions that had made WACS so challenging to begin with. Modernization also included the installation of DMS-10 telephone switches and conversion of some connections to digital.

A series of regulatory and business changes in the 1970s lead RCA to step away from the domestic communications industry. In 1979, Alascom sold to Pacific Power and Light, now for $200 million and $90 million in debt. PP&L continued on much the same trajectory, expanding the Alascom system to over 200 ground stations and launching the satellite Aurora I---the first of a small series of satellites that gave Alaska the distinction of being the only state with its own satellite communications network. For much of the '70s to the '00s, large parts of Alaska relied on satellite relay for calls between towns.

In a slight twist of irony considering its long lack of interest in the state, AT&T purchased parts of Alascom from PP&L in 1995, forming AT&T Alascom which has faded away as an independent brand. Other parts of the former ACS network, generally non-toll (or non-long-distance) operations, were split off into then PP&L subsidiary CenturyTel. While CenturyTel has since merged into CenturyLink, the Alaskan assets were first sold to Alaska Communications. Alaska Communications considers itself the successor of the ACS heritage, giving them a claim to over 100 years of communications history.

As electronics technology has continued to improve, penetration of microwave relays into inland Alaska has increased. Fewer towns rely on satellite today than in the 1970s, and the half-second latency to geosynchronous orbit is probably not missed. Alaska communications have also become more competitive, with long-distance connectivity available from General Communications (GCI) as well as AT&T and Alaska Communications.

Still, the legacy of Alaska's complex and expensive long-distance infrastructure still echoes in our telephone bills. State and federal regulators have allowed for extra fees on telephone service in Alaska and calls into Alaska, both intended to offset the high cost of infrastructure. Alaska is generally the most expensive long-distance calling destination in the United States, even when considering the territories.

But what of White Alice?

The history of the Alaska Communications System's transition to private ownership is complex and not especially well documented. While RCA's winning bid following the Alaska Communications Disposal Act set the big picture, the actual details of the transition were established by many individual negotiations spanning over a decade. Depending on the station, WACS troposcatter sites generally conveyed to RCA in 1973 or 1974. Some, colocated with active military installations, were leased rather than included in the sale. RCA generally decommissioned each WACS site once a satellite ground station was ready to replace it, either on-site or nearby.

For some WACS sites, this meant the troposcatter equipment was shut down in 1973. Others remained in use later. The Boswell Bay troposcatter station seems to have been the last turned down, in 1985. The 1980s were decidedly the end of WACS. Alascom's sale to PP&L cemented the plan to shut down all troposcatter operations, and the 1980 Comprehensive Environmental Response, Compensation, and Liability Act lead to the establishment of the Formerly Used Defense Sites (FUDS) program within DoD. Under FUDS, the Corps of Engineers surveyed the disused WACS sites and found nearly all had significant contamination by asbestos (used in seemingly every building material in the '50s and '60s) and leaked fuel oil.

As a result, most White Alice sites were demolished between 1986 and 1999. The cost of demolition and remediation in such remote locations was sometimes greater than the original construction. No WACS sites remain intact today.

Postscript:

A 1988 Corps of Engineers historical inventory of WACS, prepared due to the demolition of many of the stations, mentions that meteor burst communications might replace troposcatter. Meteor burst is a fascinating communications mode, similar in many ways to troposcatter but with the twist that the reflecting surface is not the troposphere but the ionized trail of meteors entering the atmosphere. Meteor burst connections only work when there is a meteor actively vaporizing in the upper atmosphere, but atmospheric entry of small meteors is common enough that meteor burst communications are practical for low-rate packetized communications. For example, meteor burst has been used for large weather and agricultural telemetry systems.

The Alaska Meteor Burst Communications System was implemented in 1977 by several federal agencies, and was used primarily for automated environmental telemetry. Unlike most meteor burst systems, though, it seems to have been used for real-time communications by the BLM and FAA. I can't find much information, but they seem to have built portable teleprinter terminals for this use.

Even more interesting, the Air Force's Alaskan Air Command built its own meteor burst network around the same time. This network was entirely for real-time use, and demonstrated the successful transmission of radar track data from radar stations across the state to Elmendorf Air Force base. Even better, the Air Force experimented with the use of meteor burst for intercept control by fitting aircraft with a small speech synthesizer that translated coded messages into short phrases. The Air Force experimented with several meteor burst systems during the Cold War, anticipating that it might be a survivable communications system in wartime. More details on these will have to fill a future article.

[1] Crews of the Western Union Telegraph Expedition reportedly continued work for a full year after the completion of the transatlantic telephone cable, because news of it hadn't reached them yet.

[2] Eliding here some complexities like GTE and their relationship to the Bell System.

[3] Perhaps owing to the large size of the country and many geographical challenges to cable laying, Canada has often led North America in satellite communications technology.

Note: I have edited this post to add more information, a couple of hours after originally publishing it. I forgot about a source I had open in a tab. Sorry.

High-roller Matthew D. fears Finance. "This is from our corporate expense system. Will they flag my expenses in the April-December quarter as too high? And do we really need a search function for a list of 12 items?"

Tightfisted Adam R. begrudges a trifling sum. "The tipping culture is getting out of hand. After I chose 'Custom Tip' for some takeout, they filled out the default tip with a few extra femtocents. What a rip!"

Cool Customer Reinier B. sums this up: "I got some free B&J icecream a while back. Since one of them was priced at €0.01, the other one obviously had to cost zero point minus 1 euros to make a total of zero euro. Makes sense. Or probably not."

An anonymous browniedad is ready to pack his poptart off for the summer. "I know {First Name} is really excited for camp..." Kudos on getting Mom to agree to that name choice!

Finally, another anonymous assembler's retrospective visualisation. "CoPilot rendering a graphical answer of the semantics of a pointer. Point taken. " There's no error'd here really, but I'm wondering how long before this kind of wtf illustration lands somewhere "serious".

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

Nina's team has a new developer on the team. They're not a junior developer, though Nina wishes they could replace this developer with a junior. Inexperience is better than whatever this Java code is.

Object[] test = (Object[]) options;
List<SchedulePlatform> schedulePlatformList = (List<SchedulePlatform>)((Object[])options)[0];
List<TableColumn> visibleTableCols = (List<TableColumn>)((Object[])options)[1];

We start by casting options into an array of Objects. That's already a code stench, but we actually don't even use the test variable and instead just redo the cast multiple times.

But worse than that, we cast to an array of object, access an element, and then cast that element to a collection type. I do not know what is in the options variable, but based on how it gets used, I don't like it. What it seems to be is a class (holding different options as fields) rendered as an array (holding different options as elements).

The new developer (ab)uses this pattern everywhere.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Many nations have some form of national identification number, especially around taxes. Argentina is no exception.

Their "CUIT" (Clave Única de Identificación Tributaria) and "CUIL" (Código Único de Identificación Laboral) are formatted as "##-########-#".

Now, as datasets often don't store things in their canonical representation, Nick's co-worker was given a task: "given a list of numbers, reformat them to look like CUIT/CUIL. That co-worker went off for five days, and produced this Java function.

public String normalizarCuitCuil(String cuitCuilOrigen){
	String valorNormalizado = new String();
	
	if (cuitCuilOrigen == null || "".equals(cuitCuilOrigen) || cuitCuilOrigen.length() < MINIMA_CANTIDAD_ACEPTADA_DE_CARACTERES_PARA_NORMALIZAR){
		valorNormalizado = "";
	}else{
		StringBuilder numerosDelCuitCuil = new StringBuilder(13);
		cuitCuilOrigen = cuitCuilOrigen.trim();
		
		// Se obtienen solo los números:
		Matcher buscadorDePatron =  patternNumeros.matcher(cuitCuilOrigen);
		while (buscadorDePatron.find()){
			numerosDelCuitCuil.append(buscadorDePatron.group());
		}
		
		// Se le agregan los guiones:
		valorNormalizado = numerosDelCuitCuil.toString().substring(0,2) 
							+ "-"
							+ numerosDelCuitCuil.toString().substring(2,numerosDelCuitCuil.toString().length()-1) 
							+ "-"
							+ numerosDelCuitCuil.toString().substring(numerosDelCuitCuil.toString().length()-1, numerosDelCuitCuil.toString().length());
		
	}
	return valorNormalizado;
}

We start with a basic sanity check that the string exists and is long enough. If it isn't, we return an empty string, which already annoys me, because an empty result is not a good way to communicate "I failed to parse".

But assuming we have data, we construct a string builder and trim whitespace. And already we have a problem: we already validated that the string was long enough, but if the string contained more trailing whitespace than a newline, we're looking at a problem. Now, maybe we can assume the data is good, but the next line implies that we can't rely on that- they create a regex matcher to identify numeric values, and for each numeric value they find, they append it to our StringBuilder. This implies that the string may contain non-numeric values which need to be rejected, which means our length validation was still wrong.

So either the data is clean and we're overvalidating, or the data is dirty and we're validating in the wrong order.

But all of that's a preamble to a terrible abuse of string builders, where they discard all the advantages of using a StringBuilder by calling toString again and again and again. Now, maybe the function caches results or the compiler can optimize it, but the result is a particularly unreadable blob of slicing code.

Now, this is ugly, but at least it works, assuming the input data is good. It definitely should never pass a code review, but it's not the kind of bad code that leaves one waking up in the middle of the night in a cold sweat.

No, what gets me about this is that it took five days to write. And according to Nick, the responsible developer wasn't just slacking off or going to meetings the whole time, they were at their desk poking at their Java IDE and looking confused for all five days.

And of course, because it took so long to write the feature, management didn't want to waste more time on kicking it back via a code review. So voila: it got forced through and released to production since it passed testing.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Our anonymous submitter, whom we'll call Craig, worked for GlobalCon. GlobalCon relied on an offshore team on the other side of the world for adding/removing users from the system, support calls, ticket tracking, and other client services. One day at work, an urgent escalated ticket from Martin, the offshore support team lead, fell into Craig's queue. Seated before his cubicle workstation, Craig opened the ticket right away:

The new GlobalCon support website is not working. Appears to have been taken over by ChatGPT. The entire support team is blocked by this.

Instead of feeling any sense of urgency, Craig snorted out loud from perverse amusement.

"What was that now?" The voice of Nellie, his coworker, wafted over the cubicle wall that separated them.

"Urgent ticket from the offshore team," Craig replied.

"What is it this time?" Nellie couldn't suppress her glee.

"They're dead in the water because the new support page was, quote, taken over by ChatGPT."

Nellie laughed out loud.

"Hey! I know humor is important to surviving this job." A level, more mature voice piped up behind Craig from the cube across from his. It belonged to Dana, his manager. "But it really is urgent if they're all blocked. Do your best to help, escalate to me if you get stuck."

"OK, thanks. I got this," Craig assured her.

He was already 99.999% certain that no part of their web domain had gone down or been conquered by a belligerent AI, or else he would've heard of it by now. To make sure, Craig opened support.globalcon.com in a browser tab: sure enough, it worked. Martin had supplied no further detail, no logs or screenshots or videos, and no steps to reproduce, which was sadly typical of most of these escalations. At a loss, Craig took a screenshot of the webpage, opened the ticket, and posted the following: Everything's fine on this end. If it's still not working for you, let's do a screenshare.

Granted, a screensharing session was less than ideal given the 12-hour time difference. Craig hoped that whatever nefarious shenanigans ChatGPT had allegedly committed were resolved by now.

The next day, Craig received an update. Still not working. The entire team is still blocked. We're too busy to do a screenshare, please resolve ASAP.

Craig checked the website again with both laptop and phone. He had other people visit the website for him, trying different operating systems and web browsers. Every combination worked. Two things mystified him: how was the entire offshore team having this issue, and how were they "too busy" for anything if they were all dead in the water? At a loss, Craig attached an updated screenshot to the ticket and typed out the best CYA response he could muster. The new support website is up and has never experienced any issues. With no further proof or steps to reproduce this, I don't know what to tell you. I think a screensharing session would be the best thing at this point.

The next day, Martin parroted his last message almost word for word, except this time he assented to a screensharing session, suggesting the next morning for himself.

It was deep into the evening when Craig set up his work laptop on his kitchen counter and started a call and session for Martin to join. "OK. Can you show me what you guys are trying to do?"

To his surprise, he watched Martin open up Microsoft Teams first thing. From there, Martin accessed a chat to the entire offshore support team from the CPO of GlobalCon. The message proudly introduced the new support website and outlined the steps for accessing it. One of those steps was to visit support.globalcon.com.

The web address was rendered as blue outlined text, a hyperlink. Craig observed Martin clicking the link. A web browser opened up. Lo and behold, the page that finally appeared was www.chatgpt.com.

Craig blinked with surprise. "Hang on! I'm gonna take over for a second."

Upon taking control of the session, Craig switched back to Teams and accessed the link's details. The link text was correct, but the link destination was ChatGPT. It seemed like a copy/paste error that the CPO had tried to fix, not realizing that they'd needed to do more than simply update the link text.

"This looks like a bad link," Craig said. "It got sent to your entire team. And all of you have been trying to access the support site with this link?"

"Correct," Martin replied.

Craig was glad he couldn't be seen frowning and shaking his head. "Lemme show you what I've been doing. Then you can show everyone else, OK?"

After surrendering control of the session, Craig patiently walked Martin through the steps of opening a web browser, typing support.globalcon.com into the header, and hitting Return. The site opened without any issue. From there, Craig taught Martin how to create a bookmark for it.

"Just click on that from now on, and it'll always take you to the right place," Craig said. "In the future, before you click on any hyperlink, make sure you hover your mouse over it to see where it actually goes. Links can be labeled one thing when they actually take you somewhere else. That's how phishing works."

"Oh," Martin said. "Thanks!"

The call ended on a positive note, but left Craig marveling at the irony of lecturing the tech support lead on Internet 101 in the dead of night.

It's a holiday in the US today, so we're taking a long weekend. We flip back to a classic story of a company wanting to fill 15 different positions by hiring only one person. It's okay, Martin handles the database. Original - Remy

A curious email arrived in Phil's Inbox. "Windows Support Engineer required. Must have experience of the following:" and then a long list of Microsoft products.

Phil frowned. The location was convenient; the salary was fine, just the list of software seemed somewhat intimidating. Nevertheless, he replied to the agency saying that he was interested in applying for the position.

A few days later, Phil met Jason, the guy from the recruitment agency, in a hotel foyer. "It's a young, dynamic company", the recruiter explained,"They're growing really fast. They've got tons of funding and their BI Analysis Suite is positioning them to be a leading player in their field."

Phil nodded. "Ummm, I'm a bit worried about this list of products", referring to the job description. "I've never dealt with Microsoft Proxy Server 1.0, and I haven't dealt with Windows 95 OSR2 for a long while."

"Don't worry," Jason assured, "The Director is more an idea man. He just made a list of everything he's ever heard of. You'll just be supporting Windows Server 2003 and their flagship application."

Phil winced. He was a vanilla network administrator – supporting a custom app wasn't quite what he was looking for, but he desperately wanted to get out of his current job.

A few days later, Phil arrived for his interview. The company had rented smart offices on a new business park on the edge of town. He was ushered into the conference room, where he was joined by The Director and The Manager.

"So", said The Manager. "You've seen our brochure?"

"Yeah", said Phil, glancing at the glossy brochure in front of him with bright, Barbie-pink lettering all over it.

"You've seen a demo version of our application – what do you think?"

"Well, I think that it's great!", said Phil. He'd done his research – there were over 115 companies offering something very similar, and theirs wasn't anything special. "I particularly like the icons."

"Wonderful!" The Director cheered while firing up PowerPoint. "These are our servers. We rent some rack space in a data center 100 miles away." Phil looked at the projected picture. It showed a rack of a dozen servers.

"They certainly look nice." said Phil. They did look nice – brand new with green lights.

"Now, we also rent space in another data center on the other side of the country," The Manager added.

"This one is in a former cold-war bunker!" he said proudly. "It's very secure!" Phil looked up at another photo of some more servers.

"What we want the successful applicant to do is to take care of the servers on a day to day basis, but we also need to move those servers to the other data center", said The Director. "Without any interruption of service."

"Also, we need someone to set up the IT for the entire office. You know, email, file & print, internet access – that kind of thing. We've got a dozen salespeople starting next week, they'll all need email."

"And we need it to be secure."

"And we need it to be documented."

Phil was scribbled notes as best he could while the interviewing duo tag teamed him with questions.

"You'll also provide second line support to end users of the application."

"And day-to-day IT support to our own staff. Any questions?"

Phil looked up. "Ah… which back-end database does the application use?" he asked, expecting the answer would be SQL Server or perhaps Oracle, but The Director's reply surprised him.

"Oh, we wrote our own database from scratch. Martin wrote it." Phil realized his mouth was open, and shut it. The Director saw his expression, and explained. "You see, off the shelf databases have several disadvantages – the data gets fragmented, they're not quick enough, and so on. But don't have to worry about that – Martin takes care of the database. Do you have any more questions?"

Phil frowned. "So, to summarize: you want a data center guy to take care of your servers. You want someone to migrate the application from one data center to another, without any outage. You want a network administrator to set up, document and maintain an entire network from scratch. You want someone to provide internal support to the staff. And you want a second line support person to support the our flagship application."

"Exactly", beamed The Director paternally. "We want one person who can do all those things. Can you do that?"

Phil took a deep breath. "I don't know," he replied, and that was the honest answer.

"Right", The Manager said. "Well, if you have any questions, just give either of us a call, okay?"

Moments later, Phil was standing outside, clutching the garish brochure with the pink letters. His head was spinning. Could he do all that stuff? Did he want to? Was Martin a genius or a madman to reinvent the wheel with the celebrated database?

In the end, Phil was not offered the job and decided it might be best to stick it out at his old job for a while longer. After all, compared to Martin, maybe his job wasn't so bad after all.

Underqualified Mike S. is suffering a job hunt. "I could handle uD83D and uDC77 well enough, but I am a little short of uD83C and the all important uDFFE requirement."

Frank forecasts frustration. "The weather app I'm using seems to be a bit confused on my location as I'm on vacation right now." It would be a simple matter for the app to simply identify each location, if it can't meaningfully choose only one.

Marc Würth is making me hungry. Says Marc "I was looking through my Evernote notes for "transactional" (email service). It didn't find anything. Evernote, though, tried to be helpful and thought I was looking for some basil (German "Basilikum")."

"To be from or not from be," muses Michael R. Indeed, that is the question at Stansted Shakespeare airport.

That is not the King," Brendan commented. "I posted this on Discord, and my friend responded with "They have succeeded in alignment. Their AI is truly gender blind." Not only gender-blind but apparently also existence-blind as well. I think the Bard might have something quotable here as well but it escapes me. Comment section is open.

Mark sends us a very simple Java function which has the job of parsing an integer from a string. Now, you might say, "But Java has a built in for that, Integer.parseInt," and have I got good news for you: they actually used it. It's just everything else they did wrong.

private int makeInteger(String s)
{
  int i=0;
  try
  {
    Integer.parseInt(s);
  }
  catch (NumberFormatException e)
  {
    i=0;
    return i;
  }
  i=Integer.parseInt(s);
  return i;
}

This function is really the story of variable i, the most useless variable ever. It's doing its best, but there's just nothing for it to do here.

We start by setting i to zero. Then we attempt to parse the integer, and do nothing with the result. If it fails, we set i to zero again, just for fun, and then return i. Why not just return 0? Because then what would poor i get to do?

Assuming we didn't throw an exception, we parse the input again, storing its result in i, and then return i. Again, we treat i like a child who wants to help paint the living room: we give it a dry brush and a section of wall we're not planning to paint and let it go to town. Nothing it does matters, but it feels like a participant.

Now, Mark went ahead and refactored this function basically right away, into a more terse and clear version:

private int makeInteger(String s)
{
  try
  {
    return Integer.parseInt(s);
  }
  catch (NumberFormatException e)
  {
    return 0;
  }
}

He went about his development work, and then a few days later came across makeInteger reverted back to its original version. For a moment, he wanted to be mad at someone for reverting his change, but no- this was in an entirely different class. With that information, Mark went and did a search for makeInteger in the code, only to find 39 copies of this function, with minor variations.

There are an unknown number of copies of the function where the name is slightly different than makeInteger, but a search for Integer.parseInt implies that there may be many more.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Frank inherited some code that reads URLs from a file, and puts them into a collection. This is a delightfully simple task. What could go wrong?

static String[]  readFile(String filename) {
    String record = null;
    Vector vURLs = new Vector();
    int recCnt = 0;

    try {
        FileReader fr = new FileReader(filename);
        BufferedReader br = new BufferedReader(fr);

        record = new String();

        while ((record = br.readLine()) != null) {
            vURLs.add(new String(record));
            //System.out.println(recCnt + ": " + vURLs.get(recCnt));
            recCnt++;
        }
    } catch (IOException e) {
        // catch possible io errors from readLine()
        System.out.println("IOException error reading " + filename + " in readURLs()!\n");
        e.printStackTrace();
    }

    System.out.println("Reading URLs ...\n");

    int arrCnt = 0;
    String[] sURLs = new String[vURLs.size()];
    Enumeration eURLs = vURLs.elements();

    for (Enumeration e = vURLs.elements() ; e.hasMoreElements() ;) {
        sURLs[arrCnt] = (String)e.nextElement();
        System.out.println(arrCnt + ": " + sURLs[arrCnt]);
        arrCnt++;
    }

    if (recCnt != arrCnt++) {
        System.out.println("WARNING: The number of URLs in the input file does not match the number of URLs in the array!\n\n");
    }

    return sURLs;
} // end of readFile()

So, we start by using a FileReader and a BufferedReader, which is the basic pattern any Java tutorial on file handling will tell you to do.

What I see here is that the developer responsible didn't fully understand how strings work in Java. They initialize record to a new String() only to immediately discard that reference in their while loop. They also copy the record by doing a new String which is utterly unnecessary.

As they load the Vector of strings, they also increment a recCount variable, which is superfluous since the collection can tell you how many elements are in it.

Once the Vector is populated, they need to copy all this data into a String[]. Instead of using the toArray function, which is built in and does that, they iterate across the Vector and put each element into the array.

As they build the array, they increment an arrCnt variable. Then, they do a check: if (recCnt != arrCnt++). Look at that line. Look at the post-increment on arrCnt, despite never using arrCnt again. Why is that there? Just for fun, apparently. Why is this check even there?

The only way it's possible for the counts to not match is if somehow an exception was thrown after vURLs.add(new String(record)); but before recCount++, which doesn't seem likely. Certainly, if it happens, there's something worse going on.

Now, I'm going to be generous and assume that this code predates Java 8- it just looks old. But it's worth noting that in Java 8, the BufferedReader class got a lines() function which returns a Stream<String> that can be converted directly toArray, making all of this code superfluous, but also, so much of this code is just superfluous anyway.

Anyway, for a fun game, start making the last use of every variable be a post-increment before it goes out of scope. See how many code reviews you can sneak it through!

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Combining Java with lower-level bit manipulations is asking for trouble- not because the language is inadequate to the task, but because so many of the developers who work in Java are so used to working at a high level they might not quite "get" what they need to do.

Victor inherited one such project, which used bitmasks and bitwise operations a great deal, based on the network protocol it implemented. Here's how the developers responsible created their bitmasks:

private static long FFFFFFFF = Long.parseLong("FFFFFFFF", 16);

So, the first thing that's important to note, is that Java does support hex literals, so 0xFFFFFFFF is a perfectly valid literal. So we don't need to create a string and parse it. But we also don't need to make a constant simply named FFFFFFFF, which is just the old twenty = 20 constant pattern: technically you've made a constant but you haven't actually made the magic number go away.

Of course, this also isn't actually a constant, so it's entirely possible that FFFFFFFF could hold a value which isn't 0xFFFFFFFF.

Kate inherited a system where Java code generates JavaScript (by good old fashioned string concatenation) and embeds it into an output template. The Java code was written by someone who didn't fully understand Java, but JavaScript was also a language they didn't understand, and the resulting unholy mess was buggy and difficult to maintain.

Why trying to debug the JavaScript, Kate had to dig through the generated code, which led to this little representative line:

dojo.byId('html;------sites------fileadmin------en------fileadmin------index.html;;12').setAttribute('isLocked','true');

The byId function is an alias to the browser's document.getElementById function. The ID on display here is clearly generated by the Java code, resulting in an absolutely cursed ID for an element in the page. The semicolons are field separators, which means you can parse the ID to get other information about it. I have no idea what the 12 means, but it clearly means something. Then there's that long kebab-looking string. It seems like maybe some sort of hierarchy information? But maybe not, because fileadmin appears twice? Why are there so many dashes? If I got an answer to that question, would I survive it? Would I be able to navigate the world if I understood the dark secret of those dashes? Or would I have to give myself over to our Dark Lords and dedicate my life to bringing about the end of all things?

Like all good representative lines, this one hints at darker, deeper evils in the codebase- the code that generates (or parses) this ID must be especially cursed.

The only element which needs to have its isLocked attribute set to true is the developer responsible for this: they must be locked away before they harm the rest of us.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Whatever would we do without teamwork.

David doesn't know. "Microsoft Teams seems to have lost count (it wasn't a very big copy/paste)"

A follow-up from an anonymous doesn't know either. "Teams doing its best impression of a ransom note just to say you signed out. At least it still remembers how to suggest closing your browser. Small victories."

Bob F. just wants to make memes. "I've been setting my picture widths in this document to 7.5" for weeks, and suddenly after the latest MS Word update, Microsoft thinks 7.5 is not between -22.0 and 22.0. They must be using AI math to determine this."

Ewan W. wonders "a social life: priceless...?". Ewan has some brand confusion but after the Boom Battle Bar I bet I know why.

Big spender Bob B. maybe misunderstands NaN. He gleefully exclaims "I'm very happy to get 15% off - Here's hoping the total ends up as NaN and I get it all free." Yikes. 191.78-NaN is indeed NaN, but that just means you're going to end up owing them NaN. Don't put that on a credit card!

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

Andre has inherited a rather antique ASP .Net WebForms application. It's a large one, with many pages in it, but they all follow a certain pattern. Let's see if you can spot it.

protected void btnSearch_Click(object sender, EventArgs e)
{
    ArrayList paramsRel = new ArrayList();
    paramsRel["Name"] = txtNome.Text;
    paramsRel["Date"] = txtDate.Text;
    Session["paramsRel"] = paramsRel;
   
    List<Client> clients = Controller.FindClients();
    //Some other code
}

Now, at first glance, this doesn't look terrible. Using an ArrayList as a dictionary and frankly, storing a dictionary in the Session object is weird, but it's not an automatic red flag. But wait, why is it called paramsRel? They couldn't be… no, they wouldn't…

public List<Client> FindClients()
{
    ArrayList paramsRel = (ArrayList)Session["paramsRel"];
    string name = (string)paramsRel["Name"];
    string dateStr = (string)paramsRel["Date"];
    DateTime date = DateTime.Parse(dateStr);
   
   //More code...
}

Now there's the red flag. paramsRel is how they pass parameters to functions. They stuff it into the Session, then call a function which retrieves it from that Session.

This pattern is used everywhere in the application. You can see that there's a vague gesture in the direction of trying to implement some kind of Model-View-Controller pattern (as FindClients is a member of the Controller object), but that modularization gets undercut by everything depending on Session as a pseudoglobal for passing state information around.

The only good news is that the Session object is synchronized so there's no thread safety issue here, though not for want of trying.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Frequently in programming, we can make a tradeoff: use less (or more) CPU in exchange for using more (or less) memory. Lookup tables are a great example: use a big pile of memory to turn complicated calculations into O(1) operations.

So, for example, implementing itoa, the C library function for turning an integer into a character array (aka, a string), you could maybe make it more efficient using a lookup table.

I say "maybe", because Helen inherited some C code that, well, even if it were more efficient, it doesn't help because it's wrong.

Let's start with the lookup table:

char an[1000][3] = 
{
	{'0','0','0'},{'0','0','1'},{'0','0','2'},{'0','0','3'},{'0','0','4'},{'0','0','5'},{'0','0','6'},{'0','0','7'},{'0','0','8'},{'0','0','9'},
	{'0','1','0'},{'0','1','1'},{'0','1','2'},{'0','1','3'},{'0','1','4'},{'0','1','5'},{'0','1','6'},{'0','1','7'},{'0','1','8'},{'0','1','9'},
    …

I'm abbreviating the lookup table for now. This lookup table is meant to be use to convert every number from 0…999 into a string representation.

Let's take a look at how it's used.

int ll = f->cfg.len_len;
long dl = f->data_len;
// Prepare length
if ( NULL == dst )
{
    dst_len = f->data_len + ll + 1 ;
    dst = (char*) malloc ( dst_len );
}
else
//if( dst_len < ll + dl )
if( dst_len < (unsigned) (ll + dl) )
{
    // TO DOO - error should be processed
    break;
}
long i2;
switch ( f->cfg.len_fmt)
{
    case ASCII_FORM:
    {
        if ( ll < 2 )
        {
            dst[0]=an[dl][2];
        }
        else if ( ll < 3 )
        {
            dst[0]=an[dl][1];
            dst[1]=an[dl][2];
        }
        else if ( ll < 4 )
        {
            dst[0]=an[dl][0];
            dst[1]=an[dl][1];
            dst[2]=an[dl][2];
        }
        else if ( ll < 5 )
        {
            i2 = dl / 1000;
            dst[0]=an[i2][2];
            i2 = dl % 1000;
            dst[3]=an[i2][2];
            dst[2]=an[i2][1];
            dst[1]=an[i2][0];
        }
        else if ( ll < 6 )
        {
            i2 = dl / 1000;
            dst[0]=an[i2][1];
            dst[1]=an[i2][2];
            i2 = dl % 1000;
            dst[4]=an[i2][2];
            dst[3]=an[i2][1];
            dst[2]=an[i2][0];
        }
        else
        {
            // General case
            for ( int k = ll  ; k > 0  ; k-- )
            {
                dst[k-1] ='0' + dl % 10;
                dl/=10;
            }
        }

        dst[dl]=0;

        break;
    }
}

Okay, we start with some reasonable bounds checking. I have no idea what to make of a struct member called len_len- the length of the length? I'm lacking some context here.

Then we get into the switch statement. For all values less than 4 digits, everything makes sense, more or less. I'm not sure what the point of using a 2D array for you lookup table is if you're also copying one character at a time, but for such a small number of copies I'm sure it's fine.

But then we get into the len_lens longer than 3, and we start dividing by 1000 so that our lookup table continues to work. Which, again, I guess is fine, but I'm still left wondering why we're doing this, why this specific chain of optimizations is what we need to do. And frankly, why we couldn't just use itoa or a similar library function which already does this and is probably more optimized than anything I'm going to write.

When we have an output longer than 5 characters, we just use a naive for-loop and some modulus as our "general" case.

So no, I don't like this code. It reeks of premature optimization, and it also has the vibe of someone starting to optimize without fully understanding the problem they were optimizing, and trying to change course midstream without changing their solution.

But there's a punchline to all of this. Because, you see, I skipped most of the lookup table. Would you like to see how it ends? Of course you do:

{'9','8','0'},{'9','8','1'},{'9','8','2'},{'9','8','3'},{'9','8','4'},{'9','8','5'},{'9','8','6'},{'9','8','7'},{'9','8','8'},{'9','8','9'}
};

The lookup table doesn't work for values from 990 to 999. There are just no entries there. All this effort to optimize converting integers to text and we end up here: with a function that doesn't work for 1% of the possible values it could receive. And, given that the result is an out-of-bounds array access, it fails with everyone's favorite problem: undefined behavior. Usually it'll segfault, but who knows! Maybe it returns whatever bytes it finds? Maybe it sends the nasal demons after you. The compiler is allowed to do anything.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Alexandar sends us some C# date handling code. The best thing one can say is that they didn't reinvent any wheels, but that might be worse, because they used the existing wheels to drive right off a cliff.

try
{
    var date = DateTime.ParseExact(member.PubDate.ToString(), "M/d/yyyy h:mm:ss tt", null); 
    objCustomResult.PublishedDate = date;
}
catch (Exception datEx)
{
}

member.PubDate is a Nullable<DateTime>. So its ToString will return one of two things. If there is a value there, it'll return the DateTimes value. If it's null, it'll just return an empty string. Attempting to parse the empty string will throw an exception, which we helpfully swallow, do nothing about, and leave objCustomResult.PublishedDate in whatever state it was in- I'm going to guess null, but I have no idea.

Part of this WTF is that they break the advantages of using nullable types- the entire point is to be able to handle null values without having to worry about exceptions getting tossed around. But that's just a small part.

The real WTF is taking a DateTime value, turning it into a string, only to parse it back out. But because this is in .NET, it's more subtle than just the generation of useless strings, because member.PubDate.ToString()'s return value may change depending on your culture info settings.

Which sure, this is almost certainly server-side code running on a single server with a well known locale configured. So this probably won't ever blow up on them, but it's 100% the kind of thing everyone thinks is fine until the day it's not.

The punchline is that ToString allows you to specify the format you want the date formatted in, which means they could have written this: