Reading view

There are new articles available, click to refresh the page.

New Blog

By: cpldcpu

I relocated my blog to Hugo due to easier maintainance and more control over content and layout. You can find it here.

All articles from this blog have been preserved, although I won’t list some that I found lacking in quality.

Recently

This’ll be the last Recently in 2025. It’s been a decent year for me, a pretty rough year for the rest of the world. I hope, for everyone, that 2026 sees the reversal of some of the current trends.

Watching

This video from Daniel Yang, who makes spectacular bikes of his own, covers a lot of the economics of being a bike-builder, which are all pretty rough. I felt a lot of resonance with Whit: when I ran a business I always felt like it was tough to be commercial about it, and had to fight my own instincts to over engineer parts of it. It’s also heartbreaking to think about how many jobs are so straightforwardly good for the maker and good for the buyer but economically unviable because of the world we live in. I feel like the world would be a lot different if the cost of living was lower.

Yes, the fan video! It’s a solid 48 minutes of learning how fans work. I finally watched it. Man, if every company did their advertising this way it would be so fun. I learned a lot watching this.

I have trouble finding videos about how things work that are actually about how things work. Titles like “how it’s made” or “how it works” or “how we did it” perform well in A/B tests and SEO so they get used for media that actually doesn’t explain how it’s made and how it works, greatly frustrating people like me. But the fan video delivers.

Reading

But then, you realize that the goal post has shifted. As the tech industry has become dramatically more navigable, YC became much less focused on making the world understandable, revolving, instead, around feeding consensus. “Give the ecosystem what they want.”

I have extremely mixed feelings about Build What’s Fundable, this article from Kyle Harrison. Some of it I think is bravely truth-telling in an industry that usually doesn’t do public infighting - Harrison is a General Partner at a big VC firm, and he’s critiquing a lot of firms directly on matters both financial and ethical.

But on the other hand, there’s this section about “Breaking the Normative Chains”:

When you look at successful contrarian examples, many of them have been built by existing billionaires (Tesla, SpaceX, Palantir, Anduril). The lesson from that, I think, isn’t “be a billionaire first then you can have independent thoughts.” It is, instead, to reflect on what other characteristics often lead to those outcomes. And, in my opinion, the other commonality that a lot of those companies have is that they’re led by ideological purists. People that believe in a mission.

And then, in the next section he pulls up an example of a portfolio company that encapsulates the idea of true believers, and he names Base, which has a job ad saying “Don’t tell your grandkids all you did was B2B SaaS.”

Now, Base’s mission is cool: they’re doing power storage at scale. I like the website. But I have to vent here that the founder is Zach Dell. Michael Dell’s son. Of Dell Computer, and a 151 billion dollar fortune.

I just think that if we’re going to talk about how the lesson isn’t that you should be a billionaire first before having independent thoughts and building a big tech company, it should be easy to find someone who is not the son of the 10th wealthiest person in the world to prove that point. I have nothing against Zach in particular: he is probably a talented person. But in the random distribution of talented, hardworking people, very few of them are going to be the son of the 10th wealthiest person in the world.


Like so many other bits of Times coverage, the whole of the piece is structured as an orchestrated encounter. Some people say this; however, others say this. It’s so offhand you can think you’re gazing through a pane of glass. Only when you stand a little closer, or when circumstances make you a little less blinkered, do you notice the fact which then becomes blinding and finally crazymaking, which is just that there is zero, less than zero, stress put on the relation between those two “sides,” or their histories, or their sponsors, or their relative evidentiary authority, or any of it.

I love this article on maybe don’t talk to the New York Times about Zohran Mamdani. It describes the way in which the paper launders its biases, which overlaps with one of my favorite rules from Wikipedia editing about weasel words.

I don’t want you to hate this guy. Yes, he actively promotes poisonous rhetoric – ignore that for now. This is about you. Reflect on all your setbacks, your unmet potential, and the raw unfairness of it all. It sucks, and you mustn’t let that bitterness engulf you. You can forgive history itself; you can practice gratitude towards an unjust world. You need no credentials, nor awards, nor secrets, nor skills to do so. You are allowed to like yourself.

Taylor Troesh on IQ is exactly what I needed that day.

The React team knows this makes React complicated. But the bet is clear: React falls on the sword of complexity so developers don’t have to. That’s admirable, but it asks developers to trust React’s invisible machinery more than ever.

React and Remix Choose Different Futures is perfect tech writing: it unpacks the story and philosophy behind a technical decision without cramming it into a right-versus-wrong framework.

When you consider quitting, try to find a different scoreboard. Score yourself on something else: on how many times you dust yourself off and get up, or how much incremental progress you make. Almost always, in your business or life, there are things you can make daily progress on that can make you feel like you’re still winning. Start compounding.

“Why a language? Because I believe that the core of computing is not based on operating system or processor technologies but on language capability. Language is both a tool of thought and a means of communication. Just as our minds are shaped by human language, so are operating systems shaped by programming languages. We implement what we can express. If it cannot be expressed, it will not be implemented.” – Carl Sassenrath

Alexis Sellier, whose work and aesthetic I’ve admired since the early days of Node.js, is working on a new operating system. A real new operating system, like playb.it or SerenityOS (bad politics warning). I’m totally into it: we need more from-scratch efforts like this!

Yes, the funds available for any good cause are scarce, but that’s not because of some natural law, some implacable truth about human society. It’s because oligarchic power has waged war on benign state spending, leading to the destruction of USAID and drastic cuts to the aid budgets of other countries, including the UK. Austerity is a political choice. The decision to impose it is driven by governments bowing to the wishes of the ultra-rich.

The Guardian on Bill Gates is a good read. I’ve had The Bill Gates Problem on my reading list for a long time. Maybe it’s next after I finish The Fort Bragg Cartel.

Contrast this with the rhetorical shock and awe campaign that has been waged by technology companies for the last fifteen years championing the notion of ephemerality.

Implicit, but unspoken, in this worldview is the idea of transience leading to an understanding of a world awash in ephemeral moments that, if not seized on and immediately capitalized to maximum effect, will be forever lost to the mists of time and people’s distracted lifestyles.

Another incredible article by Aaron Straup Cope about AI, the web, ephemerality, and culture. (via Perfect Sentences)


Also, no exact quote, but I’ve been subscribed to Roma’s Unpolished Posts and they have been pretty incredible: mostly technical articles about CSS, which have been ‘new to me’ almost every day, and the author is producing them once a day. Feels like a cheat code to absorb so much new information so quickly.

Listening

I didn’t add any major new albums to my collection this month. I did see Tortoise play a show, which was something I never expected to do. So in lieu of new albums, here’s a theme.

There are a bunch of songs in my library that use a meter change as a way to add or resolve tension. I’m not a big fan of key changes but I love a good rhythm or production shift.

First off: Kissing the Beehive. Yes, it’s nearly 11 minutes long. Magically feeling “off kilter” and “in the pocket” at the same time. I’m no drummer but I think the first part is something like three measures of 4/4 and one of 6/4. But then at 3:26, brand new connected song, and by the time we get to 7 minutes in, we’re in beautiful breezy easy 4/4!

An Andrew Bird classic: about a minute of smooth 4/4, and then over to 7/4 in the second half or so.

I adore Akron/Family’s Running, Returning. Starts in classic 5/4, then transitions to 4/4, then 6/8. For me it all feels very cohesive. Notably, the band is not from Akron Ohio but formed in Williamsburg and were from other East Coast places. If you’re looking for the band from Akron, it’s The Black Keys.

Slow Mass’s Schemes might be my song of the year. Everything they write just sounds so cool. The switchup happens around 2:40 when the vocals move to 6/4. Astounding.

Predictions

Back in January, I made some predictions about 2025. Let’s see how they turned out!

1: The web becomes adversarial to AI

I am marking this one as a absolute win: more and more websites are using Anubis, which was released in March, to block LLM scrapers. Cloudflare is rolling out more LLM bot protections. I at Val Town have started to turn on those protections to keep LLM bots from eating up all of our bandwidth and CPU. The LLM bots are being assholes and everyone hates them.

2: Copyright nihilism breeds a return to physical-only media

This was at most going to be a moderate win because physical-only media will be niche, but I think there are good signs that this is right. The Internet Phone Book, in which this site is featured, started publishing this year. Gen Z seems to be buying more vinyl and printing out more photos.

3: American tech companies will pull out of Europe because they want to do acquisitions

Middling at best: there are threats and there is speculation, but nothing major to report.

4: The tech industry’s ‘DEI backlash’ will run up against reality

Ugh, probably the opposite has happened. Andreesen Horowitz shut down their fund that focused on women, minorities, and people underepresented in VC funding. We’ll know more about startups themselves when Carta releases their annual report, which looked pretty bad last year.

5: Local-first will have a breakthrough moment

Sadly, no. Lots and lots of promising projects, but the ecosystem really struggles to produce something production-ready that offers good tradeoffs. Tanstack DB might be the next contender.

6: Local, small AI models will be a big deal

Not yet. Big honkin’ models are still grabbing most of the headlines. LLMs still really thrive at accomplishing vague tasks that are achievable with a wide range of acceptance criteria, like chatbots, and are pretty middling at tasks that require specific strict, quantifiable outputs.

For my mini predictions:

  • Substack will re-bundle news. Sort of! Multi-editor newsletters like The Argument are all the rage right now.
  • TypeScript gets a zeitwork equivalent and lots of people use it. Sadly, no.
  • Node.js will fend off its competitors. Mostly! Bun keeps making headway but Node.js keeps implementing the necessary featureset, and doesn’t seem to be losing that much marketshare.
  • Another US city starts seriously considering congestion pricing. There are rumblings from Boston and Chicago!
  • Stripe will IPO. Nope. Unlimited tender offers from Sequoia is just as good as IPOing, so why would they.

Val Town 2023-2025 Retrospective

It’s the end of 2025, which means that I’m closing in on three years at Val Town. I haven’t written much about the company or what it’s really been like. The real story of companies is usually told well after years after the dust has settled. Founders usually tell a heroic story of success while they’re building.

Reading startup news really warps your perspective, especially when you’re building a startup yourself. Everyone else is getting fabulously rich! It makes me less eager to write about anything.

But I’m incurably honest and like working with people who are too. Steve, the first founder of Val Town (I joined shortly after as cofounder/CTO) is a shining example of this. He is a master of saying the truth in situations when other people are afraid to. I’ve seen it defuse tension and clear paths. It’s a big part of ‘the culture’ of the company.

So here’s some of the story so far.

Delivering on existing expectations and promises

Here’s what the Val Town interface looked like fairly early on:

Val Town user interface in mid-2023

When I initially joined, we had a prototype and a bit of hype. The interface was heavily inspired by Twitter - every time that you ran code, it would save a new ‘val’ and add it to an infinite-scrolling list.

Steve and Dan had really noticed the utter exhaustion in the world of JavaScript: runaway complexity. A lot of frameworks and infrastructure was designed for huge enterprises and was really, really bad at scaling down. Just writing a little server that does one thing should be easy, but if you do it with AWS and modern frameworks, it can be a mess of connected services and boilerplate.

Val Town scaled down to 1 + 1. You could type 1 + 1 in the text field and get 2. That’s the way it should work.

It was a breath of fresh air. And a bunch of people who encountered it even in this prototype-phase state were inspired and engaged.

The arrows marketing page

One of the pivotal moments of this stage was creating this graphic for our marketing site: the arrows graphic. It really just tied it all together: look how much power there was in this little val! And no boilerplate either. Where there otherwise is a big ritual of making something public or connecting an email API, there’s just a toggle and a few lines of code.

I kind of call this early stage, for me, the era of delivering on existing expectations and promises. The core cool idea of the product was there, but it was extremely easy to break.

Security was one of the top priorities. We weren’t going to be a SOC2 certified bank-grade platform, but we also couldn’t stay where we were. Basically, it was trivially easy to hack: we were using the vm2 NPM module to run user code. I appreciate that vm2 exists, but it really, truly, is a trap. There are so many ways to get out of its sandbox and access other people’s code and data. We had a series of embarrassing security vulnerabilities.

For example: we supported web handlers so you could easily implement a little server endpoint, and the API for this was based on express, the Node.js server framework. You got a request object and response object, from express, and in this case they were literally the same as our server’s objects. Unfortunately, there’s a method response.download(path: string) which sends an arbitrary file from your server to the internet. You can see how this one ends: not ideal.

So, we had to deliver on a basic level of security. Thankfully, in the way that it sometimes does, the road rose to meet us. The right technology appeared just in time: Deno. Deno’s sandboxing made it possible to run people’s code securely without having to build a mess of Kubernetes and Docker sandbox optimizations. It delivered being secure, fast, and simple to implement: we haven’t identified a single security bug caused by Deno.

That said, the context around JavaScript runtimes has been tough. Node.js is still dominant and Bun has attracted most of the attention as an alternative, with Deno in a distant third place, vibes-wise. The three are frustratingly incompatible - Bun keeps adding built-ins like an S3 client which would have seemed unthinkable in the recent past. Node added an SQLite client in 22. Contrary to what I hoped in 2022, JavaScript has gotten more splintered and inconsistent as an ecosystem.

Stability was the other problem. The application was going down constantly for a number of reasons, but most of all was the database, which was Supabase. I wrote about switching away from Supabase, which they responded to in a pretty classy way, and I think they’ve since improved. But Render has been a huge step up in maintainability and maturity for how we host Val Town.

Adding Max was a big advance in our devops-chops too: he was not only able to but excited to work on the hard server capacity and performance problems. We quietly made a bunch of big improvements like allowing vals to stay alive after serving requests - before that, every run was a cold start.

What to do about AI

Townie

Townie, the Val Town chatbot, in early 2024

Believe it or not, but in early 2023, there were startups that didn’t say “AI” on the front page of their marketing websites. The last few years have been a dizzying shift in priorities and vibes, which I have had mixed feelings about that I’ve written about a lot.

At some point it became imperative to figure out what Val Town was supposed to do about all that. Writing code is undeniably one of the sweet spots of what LLMs can do, and over the last few years the fastest-growing most hyped startups have emerged from that ability.

This is where JP Posma comes in. He was Steve’s cofounder at a previous startup, Zaplib, and was our ‘summer intern’ - the quotes because he’s hilariously overqualified for that title. He injected some AI-abilities into Val Town, both RAG-powered search and he wrote the first version of Townie, a chatbot that is able to write code.

Townie has been really interesting. Basically it lets you write vals (our word for apps) with plain English. This development happened around the same time as a lot of the ‘vibe-coding’ applications, like Bolt and Lovable. But Townie was attached to a platform that runs code and has community elements and a lot more. It’s an entry point to the rest of the product, while a lot of other vibe-coding tools were the core product that would eventually expand to include stuff like what Val Town provides.

Ethan Ding has written a few things about this: it’s maybe preferable to sell compute instead of being the frontend for LLM-vibe-coding. But that’s sort of a long-run prediction about where value accrues rather than an observation about what companies are getting hype and funding in the present.

Vibe coding companies

There are way too many companies providing vibe-coding tools without having a moat or even a pathway to positive margins. But having made a vibe-coding tool, I completely see why: it makes charts look amazing. Townie was a huge growth driver for a while, and a lot of people were hearing about Townie first, and only later realizing that Val Town could run code, act as a lightweight GitHub alternative, and power a community.

Unlike a lot of AI startups, we didn’t burn a ton of money running Townie. We did have negative margins on it, but to the tune of a few thousand dollars a month during the most costly months.

Introducing a pro plan made it profitable pretty quickly and today Townie is pay-as-you-go, so it doesn’t really burn money at all. But on the flip side, we learned a lot about the users of vibe-coding tools. In particular, they use the tools a lot, and they really don’t want to pay for them. This kind of makes sense: vibe-coding actual completed apps without ever dropping down to write or read code is zeno’s paradox: every prompt gets you halfway there, so you inch closer and closer but never really get to your destination.

So you end up chatting for eight hours, typically getting angrier and angrier, and using a lot of tokens. This would be great for business in theory, but in practice it doesn’t work for obvious reasons: people like to pay for results, not the process. Vibe-coding is a tough industry - it’s simultaneously one of the most expensive products to run, and one of the most flighty and cost-sensitive user-bases I’ve encountered.

So AI has been complicated. On one hand, it’s amazing for growth and obviously has spawned wildly successful startups. On the other, it can be a victim of its own expectations: every company seems to promise perfect applications generated from a single prompt and that just isn’t the reality. And that results in practically every tool falling short of those expectations and thus getting the rough end of user sentiment.

We’re about to launch MCP support, which will make it possible to use Val Town via existing LLM interfaces like Claude Code. It’s a lot better than previous efforts - more powerful and flexible, plus it requires us to reinvent less of the wheel. The churn in the ‘state of the art’ feels tremendous: first we had tool-calling, then MCPs, then tool calling writing code to call MCPs: it’s hard to tell if this is fast progress or just churn.

As a business

When is a company supposed to make money? It’s a question that I’ve thought about a lot. When I was running a bootstrapped startup, the answer was obviously as soon as possible, because I’d like to stop paying my rent from my bank account. Venture funding lets you put that off for a while, sometimes a very long while, and then when companies start making real revenue they at best achieve break-even. There are tax and finance reasons for all of this – I don’t make the rules!

Anyway, Val Town is far from break-even. But that’s the goal for 2026, and it’s optimistically possible.

One thing I’ve thought for a long time is that people building startups are building complicated machines. They carry out a bunch of functions, maybe they proofread your documents or produce widgets, or whatever, but the machine also has a button on it that says “make money.” And everything kind of relates to that button as you’re building it, but you don’t really press it.

The nightmare is if the rest of the machine works, you press the button, and it doesn’t do anything. You’ve built something useful but not valuable. This hearkens back to the last section about AI: you can get a lot of people using the platform, but if you ask them for money and they’re mostly teenagers or hobbyists, they’re not going to open their wallets. They might not even have wallets.

So we pressed the button. It kind of works.

But what I’ve learned is that making revenue is a lot like engineering: it requires a lot of attempts, testing, and hard work. It’s not something that just results from a good product. Here’s where I really saw Charmaine and Steve at work, on calls, making it happen.

The angle right now is to sell tools for ‘Go To Market’ - stuff like capturing user signups of your website, figuring out which users are from interesting companies or have interesting use-cases, and forwarding that to Slack, pushing it to dashboards, and generally making the sales pipeline work. It’s something Val Town can do really well: most other tools for this kind of task have some sort of limit in how complicated and custom they can get, and Val Town doesn’t.

Expanding and managing the complexity

Product-wise, the big thing about Val Town that has evolved is that it can do more stuff and it’s more normal. When we started out, a Val was a single JavaScript expression - this was part of what made Val Town scale down so beautifully and be so minimal, but it was painfully limited. Basically people would type into the text box

const x = 10;
function hi() {};
console.log(1);

And we couldn’t handle that at all: if you ran the Val did it run that function? Export the x variable? It was magic but too confusing. The other tricky niche choice was that we had a custom import syntax like this:

@tmcw.helper(10);

In which @tmcw.helper was the name of another val and this would automatically import and use it. Extremely slick but really tricky to build off of because this was non-standard syntax, and it overlapped with the proposed syntax for decorators in JavaScript. Boy, I do not love decorators: they have been under development for basically a decade and haven’t landed, just hogging up this part of the unicode plane.

But regardless this syntax wasn’t worth it. I have some experience with this problem and have landed squarely on the side of normality is good.

So, in October 2023, we ditched it, adopted standard ESM import syntax, and became normal. This is was a big technical undertaking, in large part because we tried to keep all existing code running by migrating it. Thankfully JavaScript has a very rich ecosystem of tools that can parse & produce code and manipulate syntax trees, but it was still a big, dramatic shift.

This is one of the core tensions of Val Town as well as practically every startup: where do you spend your user-facing innovation energy?

I’m a follower of the use boring technology movement when it comes to how products are built: Val Town intentionally uses some boring established parts like Postgres and React Router, but what about when it comes to the product itself? I’ve learned the hard way that most of what people call intuition is really familiarity: it’s good when an interface behaves like other interfaces. A product that has ten new concepts and a bunch of new UI paradigms is going to be hard to learn and probably will lose out to one that follows some familiar patterns.

Moving to standard JavaScript made Val Town more learnable for a lot of people while also removing some of its innovation. Now you can copy code into & out of Val Town without having to adjust it. LLMs can write code that targets Val Town without knowing everything about its quirks. It’s good to go with the flow when it comes to syntax.

Hiring and the team

Office Sign

Val Town has an office. I feel like COVID made everything remote by default and the lower-COVID environment that we now inhabit (it’s still not gone!) has led to a swing-back, but the company was founded in the latter era and has never been remote. So, we work from home roughly every other Friday.

This means that we basically try to hire people in New York. It hasn’t been too hard in the past. About 6% of America lives in the New York City metro area and the Northeast captures about 23% of venture funding, so there are lots of people who live here or want to.

Stuff on the window sill in the office

Here’s something hard to publish: we’re currently at three people. It was five pretty recently. Charmaine got poached by Anthropic where she’ll definitely kick ass, and Max is now at Cloudflare where he’s writing C++, which will be even more intimidating than his chess ranking. The company’s really weirdly good at people leaving: we had parties and everyone exchanges hand-written cards. How people handle hard things says a lot.

But those three are pretty rad: Jackson was a personal hero of mine before we hired him (he still is). He’s one of the best designers I’ve worked with, and an incredibly good engineer to boot. He’s worked at a bunch of startups you’ve heard of, had a DJ career, gotten to the highest echelons of tech without acquiring an ego. He recently beat me to the top spot in our GitHub repo’s lines-changed statistic.

Steve has what it takes for this job: grit, optimism, curiosity. The job of founding a company and being a CEO is a different thing every few months - selling, hiring, managing, promoting. Val Town is a very developer-consumer oriented product and that kind of thing requires a ton of promotion. Steve has done so much, in podcasts, spreading the word in person, writing, talking to customers. He has really put everything into this. A lot of the voice and the attitude of the company flows down from the founder, and Steve is that.

Did I mention that we’re hiring?

In particular, for someone to be a customer-facing technical promoter type - now called a “GTM” hire. Basically, who can write a bit of code but has the attitude of someone in sales. Who can see potential and handle rejection. Not necessarily the world’s best programmer, but who can probably code, and definitely someone who can write. Blogging and writing online is a huge green flag for this position.

And the other role that we really need is an “application engineer.” These terms keep shifting, so if full-stack engineer means more, sure, that too. Basically someone who can write code across boundaries. This is more or less what Jackson and I do - writing queries, frontend code, fixing servers, the whole deal. Yeah, it sounds like a lot but this is how all small companies operate, and I’ve made a lot of decisions to make this possible: we’ve avoided complexity like the plague in Val Town’s stack, so it should all be learnable. I’ve written a bunch of documentation for everything, and constantly tried to keep the codebase clean.

Sidenote, but even though I think that the codebase is kind of messy, I’ve heard from very good engineers (even the aforementioned JP Posma) that it’s one of the neatest and most rational codebases they’ve seen. Maybe it is, maybe it isn’t, see for yourself!

What we’re really looking for in hires

Tech hiring has been broken the whole time I’ve been in the industry, for reasons that would take a whole extra article to ponder. But one thing that makes it hard is vagueness, both on the part of applicants and companies. I get it - cast a wide net, don’t put people off. But I can say that:

  • For the GTM position, you should be able to write for the internet. This can be harder than it looks: there are basically three types of writing: academic, corporate, and internet, and they are incompatible.
  • You should also be kind of entrepreneurial: which means optimistic, resilient, and opportunistic.
  • For the application engineering role, you should be a good engineer who understands the code you write and is good at both writing and reading code. Using LLM tools is great, but relying on them exclusively is a dealbreaker. LLMs are not that good at writing code.

What the job is like

The company’s pretty low drama. Our office is super nice. We work hard but not 996. We haven’t had dinner in the office. But we all do use PagerDuty so when the servers go down, we wake up and it sucks. Thankfully the servers go down less than they used to.

We all get paid the same: $175k. Lower than FAANG, but pretty livable for Brooklyn. Both of the jobs listed - the Product Engineer, and Growth Engineer - are set at 1% equity. $175k is kind of high-average for where we’re at, but 1% in my opinion is pretty damn good. Startups say that equity is “meaningful” at all kinds of numbers but it’s definitely meaningful at that one. If Val Town really succeeds, you can get pretty rich off of that.

Of course, will it succeed? It’s something I think about all the time. I was born to ruminate. We have a lot going for us, and a real runway to make things happen. Some of the charts in our last investor update looked great. Some days felt amazing. Other days were a slog. But it’s a good team, with a real shot of making it.

Recently

Hello! Only a day late this time. October was another busy month but it didn’t yield much content. I ran a second half-marathon, this time with much less training, but only finished a few minutes slower than I did earlier this year. Next year I’m thinking about training at a normal mileage for the kinds of races I’m running - 25 miles or so per week instead of this year’s roughly 15.

And speaking of running, I just wrote up this opinion I have about how fewer people should run marathons.

Reading

I enjoyed reading Why Functor Doesn’t Matter, but I don’t really agree. The problem that I had with functional programming jargon isn’t that the particular terms are strange or uncommon, but that their definitions rely on a series of other jargon terms, and the discipline tends to omit good examples, metaphors, or plain-language explanations. It’s not that the strict definition is bad, but when a function is defined as _a mapping that associates a morphism F: X -> Y in category C to a morphism F(f): F(X) -> F(Y), in category D, you now have to define morphism, categories, and objects, and all of which have domain-specific definitions.

I am loving Sherif’s posting about building a bike from the frame up.

Maximizers are biased to speed, optionality, breadth, momentum, opportunism, parallel bets, hype, luck exposure, momentum, “Why not both?”, “Better to move fast than wait for perfect”. Maximizers want to see concrete examples before they’ll make tradeoffs. They anchor decisions in the tangible. “Stop making things so complicated.” “Stop overthinking.”

Focusers are biased to focus, coherence, depth, meaningful constraints, doing less for more, sequential experiments, intentionality, sustainability, “What matters most?”, compounding clarity. Focusers are comfortable with abstraction. A clear constraint or principle is enough to guide them. “Stop mistaking chaos for progress.” “Stop overdoing.”

John Cutler’s post about maximizers vs. focusers matches my experience in tech. Like many young engineers, I think I started out as a focuser and have tried to drift to the center all the time, but the tension both internally and interpersonally at every job is present.

I recently remarked to a friend that traveling abroad after the advent of the smartphone feels like studying biology after the advent of microplastics. It has touched every aspect of life. No matter where you point your microscope you will see its impact.

Josh Erb’s blog about living in India is great, personal, a classic blog’s blog.

For me, the only reason to keep going is to try and make AI a wonderful technology for the world. Some feel the same. Others are going because they’re locked in on a path to generational wealth. Plenty don’t have either of these alignments, and the wall of effort comes sooner.

This article about AI researchers working all the time and burning out is interesting, in part because I find the intention of AI researchers so confusing. I can see the economic intention: these guys are making bank! Congrats to all of them. But it’s so rare to talk to anyone who has a concrete idea about how they are making the world better by doing what they’re doing, and that’s the reason why they’re working so hard. OpenAI seems to keep getting distracted from that cancer cure, and their restructuring into a for-profit company kind of indicates that there’s more greed than altruism in the mix.

every vc who bet on the modern data stack watched their investments get acquired for pennies or go to zero. the only survivors: the warehouses themselves, or the companies the warehouses bought to strengthen their moats.

It’s niche, but this article about Snowflake, dbt, fivetran, and other ‘data lake’ architecture is really enlightening.

Listening

Totorro’s new album was the only one I picked up this month. It’s pretty good math-rock, very energetic and precise.

Watching

  • One Battle After Another was incredible.
  • eXistenZ is so gloriously weird, I really highly recommend it. It came out the same year as The Matrix and explores similar themes, but the treatment of futuristic technology is something you won’t see anywhere else: instead of retro-steampunk metal or fully dystopian grimness, it’s colorful, slimy, squelchy, organic, and weird.

Speaking of weird, Ben Levin’s gesamtkunstwerk videos are wild and glorious.

OpenAI employees… are you okay?

You might have seen an article making the rounds this week, about a young man who ended his life after ChatGPT encouraged him to do so. The chat logs are really upsetting.

Someone two degrees removed from me took their life a few weeks ago. A close friend related the story to me, about how this person had approached their neighbor one evening to catch up, make small talk, and casually discussed their suicidal ideation at some length. At the end of the conversation, they asked to borrow a rope, and their neighbor agreed without giving the request any critical thought. The neighbor found them the next morning.

I didn’t know the deceased, nor their neighbor, but I’m close friends with someone who knew both. I found their story deeply chilling – ice runs through my veins when I imagine how the neighbor must have felt. I had a similar feeling upon reading this article, wondering how the people behind ChatGPT and tools like it are feeling right now.

Two years ago, someone I knew personally took their life as well. I was not friendly with this person – in fact, we were on very poor terms. I remember at the time, I had called a crisis hotline just to ask an expert for advice on how to break this news to other people in my life, many of whom were also on poor terms with a person whose struggles to cope with their mental health issues caused a lot of harm to others.

None of us had to come to terms with any decisions with the same gravity as what that unfortunate neighbor had to face. None of us were ultimately responsible for this person’s troubles or were the impetus for what happened. Nonetheless, the uncomfortable and confronting feelings I experienced in the wake of that event perhaps give me some basis for empathy and understanding towards the neighbor, or for OpenAI employees, and others who find themselves in similar situations.

If you work on LLMs, well… listen, I’ve made my position as an opponent of this technology clear. I feel that these tools are being developed and deployed recklessly, and I believe tragedy is the inevitable result of that recklessness. If you confide in me, I’m not going to validate your career choice. But maybe that’s not necessarily a bad quality to have in a confidant? I still feel empathy towards you and I recognize your humanity and our need to acknowledge each other as people.

If you feel that I can help, I encourage you to reach out. I will keep our conversation in confidence, and you can reach out anonymously if that makes you feel safer. I’m a good listener and I want to know how you’re doing. Email me.


If you’re experiencing a crisis, 24-hour support is available from real people who are experts in getting you the help you need. Please consider reaching out. All you need to do is follow the link.

More tales about outages and numeric limits

Outages, you say? Of course I have stories about outages, and limits, and some limits causing outages, and other things just screwing life up. Here are some random thoughts which sprang to mind upon reading this morning's popcorn-fest.

...

I was brand new at a company that "everybody knew" had AMAZING infrastructure. They could do things with Linux boxes that nobody else could. As part of the new employee process, I had to get accounts in a bunch of systems, and one of them was this database used to track the states of machines. It was where you could look to see if it was (supposed to be) serving, or under repair, or whatever. You could also see (to some degree) what services were supposed to be running on it, and what servers (that is, actual programs), the port numbers, and whether all of that stuff was synced to the files on the box or not.

My request didn't go through for a while, and I found out that it had something to do with my employee ID being a bit over 32767. And yeah, for those of you who didn't just facepalm at seeing that number, that's one of those "magic numbers" which pops up a bunch when talking about limits. That one is what you get when you try to store numbers as 16 bit values... with a sign to allow negative values. Why you'd want a negative employee number is anyone's guess, but that's how they configured it.

I assume they fixed the database schema at some point to allow more than ~15 bits of employee numbers, but they did an interesting workaround to get me going before then. They just shaved off the last digit and gave me that ID in their system instead. I ended up as 34xx instead of 34xxx, more or less.

This was probably my first hint that their "amazing infra" was in fact the same kind of random crazytown as everywhere else once you got to see behind the curtain.

...

Then there was the time that someone decided that a log storage system that had something like a quarter of a million machines (and growing fast) feeding it needed a static configuration. The situation unfolded like this:

(person 1) Hey, why is this thing crashing so much?

(person 2) Oh yeah, it's dumping cores constantly! Wow!

(person 1) It's running but there's nothing in the log?

(person 2) Huh, "runtime error ... bad id mapping?"

(person 2) It's been doing this for a month... and wait, other machines are doing it, too!

(person 1) Guess I'll dig into this.

(person 2) "range name webserv_log.building1.phase3 range [1-20000]"

(person 2) But this machine is named webserv20680...

(person 2) Yeah, that's enough for me. Bye!

The machines were named with a ratcheting counter: any time they were assigned to be a web server, they got names like "webserv1", "webserv2", ... and so on up the line. That had been the case all along.

Whoever designed this log system years later decided to put a hard-coded limiter into it. I don't know if they did it because they wanted to feel useful every time it broke so they could race in and fix it, or if they didn't care, or if they truly had no idea that numbers could in fact grow beyond 20000.

Incidentally, that particular "building1.phase3" location didn't even have 20000 machines at that specific moment. It had maybe 15000 of them, but as things went away and came back, the ever-incrementing counter just went up and up and up. So, there _had been_ north of 20K machines in that spot overall, and that wasn't even close to a surprising number.

...

There was a single line that would catch obvious badness at a particular gig where we had far too many Apache web servers running on various crusty Linux distributions:

locate access_log | xargs ls -la | grep 2147

It was what I'd send in chat to someone who said "hey, the customer's web server won't stay up". The odds were very good that they had a log file that had grown to 2.1 GB, and had hit a hard limit which was present in that particular system. Apache would try to write to it, that write would fail, and the whole process would abort.

"2147", of course, is the first 4 digits of the expected file size: 2147483647 ... or (2^31)-1.

Yep, that's another one of those "not enough bits" problems like the earlier story, but this one is 32 bits with one of them being for the sign, not 16 like before. It's the same problem, though: the counter maxes out and you're done.

These days, files can get quite a bit bigger... but you should still rotate your damn log files once in a while. You should probably also figure out what's pooping in them so much and try to clean that up, too!

...

As the last one for now, there was an outage where someone reported that something like half of their machines were down. They had tried to do a kernel update, and wound up hitting half of them at once. I suspect they wanted to do a much smaller quantity, but messed up and hit fully half of them somehow. Or, maybe they pointed it at all of them, and only half succeeded at it. Whatever the cause, they now had 1000 freshly-rebooted machines.

The new kernel was fine, and the usual service manager stuff came back up, and it went to start the workload for those systems, and then it would immediately crash. It would try to start it again. It would crash again. Crash crash crash. This is why we call it "crashlooping".

Finally, the person in question showed up in the usual place where we discussed outages, and started talking about what was going on.

(person 1) Our stuff isn't coming back.

(person 2) Oh yeah, that's bad, they're all trying to start.

(person 1) Start, abort, start, abort, ...

(person 2) Yep, aborting... right about here: company::project::client::BlahClient::loadConfig ... which is this code: <paste>

(person 2) It's calling "get or throw" on a map for an ID number...

(person 1) My guess is the config provider service isn't running.

(person 2) It's there... it's been up for 30 minutes...

(person 1) Restarting the jobs.

(person 2) Nooooooooooo...

<time passes>

(person 2) Why is there no entry for number 86 in the map in the config?

(person 1) Oh, I bet it's problems with port takeover.

(person 3) I think entry 86 is missing from <file>.

(person 2) Definitely is missing.

(person 4) Hey everyone, we removed that a while back. Why would it only be failing now?

(person 2) It's only loaded at startup, right?

(person 4) Right.

(person 2) So if they were running for a long time, then it changed, then they're toast after a restart...

(person 3) Hey, this change looks related.

(person 4) I'm going to back that out.

This is a common situation: program A reads config C. When it starts up, config C is on version C1, and everything is fine. While A is running, the config is updated from C1 to C2, but nothing notices. Later, A tries to restart and it chokes on the C2 config, and refuses to start.

Normally, you'd only restart a few things to get started, and you'd notice that your program can't consume the new config at that point. You'd still have a few instances down, but that's it - a *few* instances. Your service should keep running on whatever's left over that you purposely didn't touch.

This is why you strive to release things in increments.

Also, it helps when programs notice config changes while they're running, so this doesn't sneak up on you much later when you're trying to restart. If the programs notice the bad config right after the change is made, it's *far* easier to correlate it to the change just by looking at the timeline.

Tuesday, 11:23:51: someone applies change.

Tuesday, 11:23:55: first 1% of machines which subscribe to the change start complaining.

... easy, right? Now compare it to this:

Tuesday, November 18: someone applies a change

Wednesday, January 7: 50% of machines fail to start "for some reason"

That's a lot harder to nail down.

...

Random aside: restarting the jobs did not help. They were already restarting themselves. "Retry, reboot, reinstall, repeat" is NOT a strategy for success.

It was not the config system being down. It was up the whole time.

It was nothing to do with "port takeover". What does that have to do with a config file being bad?

The evidence was there: the processes were crashing. They were logging a message about WHY they were killing themselves. It included a number they wanted to see, but couldn't find. It also said what part of the code was blowing up.

*That* is where you start looking. You don't just start hammering random things.

Unusual circuits in the Intel 386's standard cell logic

I've been studying the standard cell circuitry in the Intel 386 processor recently. The 386, introduced in 1985, was Intel's most complex processor at the time, containing 285,000 transistors. Intel's existing design techniques couldn't handle this complexity and the chip began to fall behind schedule. To meet the schedule, the 386 team started using a technique called standard cell logic. Instead of laying out each transistor manually, the layout process was performed by a computer.

The idea behind standard cell logic is to create standardized circuits (standard cells) for each type of logic element, such as an inverter, NAND gate, or latch. You feed your circuit description into software that selects the necessary cells, positions these cells into columns, and then routes the wiring between the cells. This "automatic place and route" process creates the chip layout much faster than manual layout. However, switching to standard cells was a risky decision since if the software couldn't create a dense enough layout, the chip couldn't be manufactured. But in the end, the 386 finished ahead of schedule, an almost unheard-of accomplishment.1

The 386's standard cell circuitry contains a few circuits that I didn't expect. In this blog post, I'll take a quick look at some of these circuits: surprisingly large multiplexers, a transistor that doesn't fit into the standard cell layout, and inverters that turned out not to be inverters. (If you want more background on standard cells in the 386, see my earlier post, "Reverse engineering standard cell logic in the Intel 386 processor".)

The photo below shows the 386 die with the automatic-place-and-route regions highlighted; I'm focusing on the red region in the lower right. These blocks of logic have cells arranged in rows, giving them a characteristic striped appearance. The dark stripes are the transistors that make up the logic gates, while the lighter regions between the stripes are the "routing channels" that hold the wiring that connects the cells. In comparison, functional blocks such as the datapath on the left and the microcode ROM in the lower right were designed manually to optimize density and performance, giving them a more solid appearance.

The 386 die with the standard-cell regions highlighted.

The 386 die with the standard-cell regions highlighted.

As for other features on the chip, the black circles around the border are bond wire connections that go to the chip's external pins. The chip has two metal layers, a small number by modern standards, but a jump from the single metal layer of earlier processors such as the 286. (Providing two layers of metal made automated routing practical: one layer can hold horizontal wires while the other layer can hold vertical wires.) The metal appears white in larger areas, but purplish where circuitry underneath roughens its surface. The underlying silicon and the polysilicon wiring are obscured by the metal layers.

The giant multiplexers

The standard cell circuitry that I'm examining (red box above) is part of the control logic that selects registers while executing an instruction. You might think that it is easy to select which registers take part in an instruction, but due to the complexity of the x86 architecture, it is more difficult. One problem is that a 32-bit register such as EAX can also be treated as the 16-bit register AX, or two 8-bit registers AH and AL. A second problem is that some instructions include a "direction" bit that switches the source and destination registers. Moreover, sometimes the register is specified by bits in the instruction, but in other cases, the register is specified by the microcode. Due to these factors, selecting the registers for an operation is a complicated process with many cases, using control bits from the instruction, from the microcode, and from other sources.

Three registers need to be selected for an operation—two source registers and a destination register—and there are about 17 cases that need to be handled. Registers are specified with 7-bit control signals that select one of the 30 registers and control which part of the register is accessed. With three control signals, each 7 bits wide, and about 17 cases for each, you can see that the register control logic is large and complicated. (I wrote more about the 386's registers here.)

I'm still reverse engineering the register control logic, so I won't go into details. Instead, I'll discuss how the register control circuit uses multiplexers, implemented with standard cells. A multiplexer is a circuit that combines multiple input signals into a single output by selecting one of the inputs.2 A multiplexer can be implemented with logic gates, for instance, by ANDing each input with the corresponding control line, and then ORing the results together. However, the 386 uses a different approach—CMOS switches—that avoids a large AND/OR gate.

Schematic of a CMOS switch.

Schematic of a CMOS switch.

The schematic above shows how a CMOS switch is constructed from two MOS transistors. When the two transistors are on, the output is connected to the input, but when the two transistors are off, the output is isolated. An NMOS transistor is turned on when its input is high, but a PMOS transistor is turned on when its input is low. Thus, the switch uses two control inputs, one inverted. The motivation for using two transistors is that an NMOS transistor is better at pulling the output low, while a PMOS transistor is better at pulling the output high, so combining them yields the best performance.3 Unlike a logic gate, the CMOS switch has no amplification, so a signal is weakened as it passes through the switch. As will be seen below, inverters can be used to amplify the signal.

The image below shows how CMOS switches appear under the microscope. This image is very hard to interpret because the two layers of metal on the 386 are packed together densely, but you can see that some wires run horizontally and others run vertically. The bottom layer of metal (called M1) runs vertically in the routing area, as well as providing internal wiring for a cell. The top layer of metal (M2) runs horizontally; unlike M1, the M2 wires can cross a cell. The large circles are vias that connect the M1 and M2 layers, while the small circles are connections between M1 and polysilicon or M1 and silicon. The central third of the image is a column of standard cells with two CMOS switches outlined in green. The cells are bordered by the vertical ground rail and +5V rail that power the cells. The routing areas are on either side of the cells, holding the wiring that connects the cells.

Two CMOS switches, highlighted in green. The lower switch is flipped vertically compared to the upper switch.

Two CMOS switches, highlighted in green. The lower switch is flipped vertically compared to the upper switch.

Removing the metal layers reveals the underlying silicon with a layer of polysilicon wiring on top. The doped silicon regions show up as dark outlines. I've drawn the polysilicon in green; it forms a transistor (brighter green) when it crosses doped silicon. The metal ground and power lines are shown in blue and red, respectively, with other metal wiring in purple. The black dots are vias between layers. Note how metal wiring (purple) and polysilicon wiring (green) are combined to route signals within the cell. Although this standard cell is complicated, the important thing is that it only needs to be designed once. The standard cells for different functions are all designed to have the same width, so the cells can be arranged in columns, snapped together like Lego bricks.

A diagram showing the silicon for a standard-cell switch. The polysilicon is shown in green. The bottom metal is shown in blue, red, and purple.

A diagram showing the silicon for a standard-cell switch. The polysilicon is shown in green. The bottom metal is shown in blue, red, and purple.

To summarize, this switch circuit allows the input to be connected to the output or disconnected, controlled by the select signal. This switch is more complicated than the earlier schematic because it includes two inverters to amplify the signal. The data input and the two select lines are connected to the polysilicon (green); the cell is designed so these connections can be made on either side. At the top, the input goes through a standard two-transistor inverter. The lower left has two transistors, combining the NMOS half of an inverter with the NMOS half of the switch. A similar circuit on the right combines the PMOS part of an inverter and switch. However, because PMOS transistors are weaker, this part of the circuit is duplicated.

A multiplexer is constructed by combining multiple switches, one for each input. Turning on one switch will select the corresponding input. For instance, a four-to-one multiplexer has four switches, so it can select one of the four inputs.

A four-way multiplexer constructed from CMOS switches and individual transistors.

A four-way multiplexer constructed from CMOS switches and individual transistors.

The schematic above shows a hypothetical multiplexer with four inputs. One optimization is that if an input is always 0, the PMOS transistor can be omitted. Likewise, if an input is always 1, the NMOS transistor can be omitted. One set of select lines is activated at a time to select the corresponding input. The pink circuit selects 1, green selects input A, yellow selects input B, and blue selects 0. The multiplexers in the 386 are similar, but have more inputs.

The diagram below shows how much circuitry is devoted to multiplexers in this block of standard cells. The green, purple, and red cells correspond to the multiplexers driving the three register control outputs. The yellow cells are inverters that generate the inverted control signals for the CMOS switches. This diagram also shows how the automatic layout of cells results in a layout that appears random.

A block of standard-cell logic with multiplexers highlighted. The metal and polysilicon layers were removed for this photo, revealing the silicon transistors.

A block of standard-cell logic with multiplexers highlighted. The metal and polysilicon layers were removed for this photo, revealing the silicon transistors.

The misplaced transistor

The idea of standard-cell logic is that standardized cells are arranged in columns. The space between the cells is the "routing channel", holding the wiring that links the cells. The 386 circuitry follows this layout, except for one single transistor, sitting between two columns of cells.

The "misplaced" transistor, indicated by the arrow. The irregular green regions are oxide that was incompletely removed.

The "misplaced" transistor, indicated by the arrow. The irregular green regions are oxide that was incompletely removed.

I wrote some software tools to help me analyze the standard cells. Unfortunately, my tools assumed that all the cells were in columns, so this one wayward transistor caused me considerable inconvenience.

The transistor turns out to be a PMOS transistor, pulling a signal high as part of a multiplexer. But why is this transistor out of place? My hypothesis is that the transistor is a bug fix. Regenerating the cell layout was very costly, taking many hours on an IBM mainframe computer. Presumably, someone found that they could just stick the necessary transistor into an unused spot in the routing channel, manually add the necessary wiring, and avoid the delay of regenerating all the cells.

The fake inverter

The simplest CMOS gate is the inverter, with an NMOS transistor to pull the output low and a PMOS transistor to pull the output high. The standard cell circuitry that I examined contains over a hundred inverters of various sizes. (Performance is improved by using inverters that aren't too small but also aren't larger than necessary for a particular circuit. Thus, the standard cell library includes inverters of multiple sizes.)

The image below shows a medium-sized standard-cell inverter under the microscope. For this image, I removed the two metal layers with acid to show the underlying polysilicon (bright green) and silicon (gray). The quality of this image is poor—it is difficult to remove the metal without destroying the polysilicon—but the diagram below should clarify the circuit. The inverter has two transistors: a PMOS transistor connected to +5 volts to pull the output high when the input is 0, and an NMOS transistor connected to ground to pull the output low when the input is 1. (The PMOS transistor needs to be larger because PMOS transistors don't function as well as NMOS transistors due to silicon physics.)

An inverter as seen on the die. The corresponding standard cell is shown below.

An inverter as seen on the die. The corresponding standard cell is shown below.

The polysilicon input line plays a key role: where it crosses the doped silicon, a transistor gate is formed. To make the standard cell more flexible, the input to the inverter can be connected on either the left or the right; in this case, the input is connected on the right and there is no connection on the left. The inverter's output can be taken from the polysilicon on the upper left or the right, but in this case, it is taken from the upper metal layer (not shown). The power, ground, and output lines are in the lower metal layer, which I have represented by the thin red, blue, and yellow lines. The black circles are connections between the metal layer and the underlying silicon.

This inverter appears dozens of times in the circuitry. However, I came across a few inverters that didn't make sense. The problem was that the inverter's output was connected to the output of a multiplexer. Since an inverter is either on or off, its value would clobber the output of the multiplexer.4 This didn't make any sense. I double- and triple-checked the wiring to make sure I hadn't messed up. After more investigation, I found another problem: the input to a "bad" inverter didn't make sense either. The input consisted of two signals shorted together, which doesn't work.

Finally, I realized what was going on. A "bad inverter" has the exact silicon layout of an inverter, but it wasn't an inverter: it was independent NMOS and PMOS transistors with separate inputs. Now it all made sense. With two inputs, the input signals were independent, not shorted together. And since the transistors were controlled separately, the NMOS transistor could pull the output low in some circumstances, the PMOS transistor could pull the output high in other circumstances, or both transistors could be off, allowing the multiplexer's output to be used undisturbed. In other words, the "inverter" was just two more cases for the multiplexer.

The "bad" inverter. (Image is flipped vertically for comparison with the previous inverter.)

The "bad" inverter. (Image is flipped vertically for comparison with the previous inverter.)

If you compare the "bad inverter" cell below with the previous cell, they look almost the same, but there are subtle differences. First, the gates of the two transistors are connected in the real inverter, but disconnected by a small gap in the transistor pair. I've indicated this gap in the photo above; it is hard to tell if the gap is real or just an imaging artifact, so I didn't spot it. The second difference is that the "fake" inverter has two input connections, one to each transistor, while the inverter has a single input connection. Unfortunately, I assumed that the two connections were just a trick to route the signal across the inverter without requiring an extra wire. In total, this cell was used 32 times as a real inverter and 9 times as independent transistors.

Conclusions

Standard cell logic and automatic place and route have a long history before the 386, back to the early 1970s, so this isn't an Intel invention.5 Nonetheless, the 386 team deserves the credit for deciding to use this technology at a time when it was a risky decision. They needed to develop custom software for their placing and routing needs, so this wasn't a trivial undertaking. This choice paid off and they completed the 386 ahead of schedule. The 386 ended up being a huge success for Intel, moving the x86 architecture to 32 bits and defining the dominant computer architecture for the rest of the 20th century.

If you're interested in standard cell logic, I also wrote about standard cell logic in an IBM chip. I plan to write more about the 386, so follow me on Mastodon, Bluesky, or RSS for updates. Thanks to Pat Gelsinger and Roxanne Koester for providing helpful papers.

For more on the 386 and other chips, follow me on Mastodon (@kenshirriff@oldbytes.space), Bluesky (@righto.com), or RSS. (I've given up on Twitter.) If you want to read more about the 386, I've written about the clock pin, prefetch queue, die versions, packaging, and I/O circuits.

Notes and references

  1. The decision to use automatic place and route is described on page 13 of the Intel 386 Microprocessor Design and Development Oral History Panel, a very interesting document on the 386 with discussion from some of the people involved in its development. 

  2. Multiplexers often take a binary control signal to select the desired input. For instance, an 8-to-1 multiplexer selects one of 8 inputs, so a 3-bit control signal can specify the desired input. The 386's multiplexers use a different approach with one control signal per input. One of the 8 control signals is activated to select the desired input. This approach is called a "one-hot encoding" since one control line is activated (hot) at a time. 

  3. Some chips, such as the MOS Technology 6502 processor, are built with NMOS technology, without PMOS transistors. Multiplexers in the 6502 use a single NMOS transistor, rather than the two transistors in the CMOS switch. However, the performance of the switch is worse. 

  4. One very common circuit in the 386 is a latch constructed from an inverter loop and a switch/multiplexer. The inverter's output and the switch's output are connected together. The trick, however, is that the inverter is constructed from special weak transistors. When the switch is disabled, the inverter's weak output is sufficient to drive the loop. But to write a value into the latch, the switch is enabled and its output overpowers the weak inverter.

    The point of this is that there are circuits where an inverter and a multiplexer have their outputs connected. However, the inverter must be constructed with special weak transistors, which is not the situation that I'm discussing. 

  5. I'll provide more history on standard cells in this footnote. RCA patented a bipolar standard cell in 1971, but this was a fixed arrangement of transistors and resistors, more of a gate array than a modern standard cell. Bell Labs researched standard cell layout techniques in the early 1970s, calling them Polycells, including a 1973 paper by Brian Kernighan. By 1979, A Guide to LSI Implementation discussed the standard cell approach and it was described as well-known in this patent application. Even so, Electronics called these design methods "futuristic" in 1980.

    Standard cells became popular in the mid-1980s as faster computers and improved design software made it practical to produce semi-custom designs that used standard cells. Standard cells made it to the cover of Digital Design in August 1985, and the article inside described numerous vendors and products. Companies like Zymos and VLSI Technology (VTI) focused on standard cells. Traditional companies such as Texas Instruments, NCR, GE/RCA, Fairchild, Harris, ITT, and Thomson introduced lines of standard cell products in the mid-1980s.  

My mental model of the AI race

I left a loose end the other day when I said that AI is about intent and context.

That was when I said "what’s context at inference time is valuable training data if it’s recorded."

But I left it at that, and didn’t really get into why training data is valuable.

I think we often just draw a straight arrow from “collect training data,” like ingest pages from Wikipedia or see what people are saying to the chatbot, to “now the AI model is better and therefore it wins.”

But I think it’s worth thinking about what that arrow actually means. Like, what is the mechanism here?

Now all of this is just my mental model for what’s going on.

With that caveat:

To my mind, the era-defining AI company is the one that is the first to close two self-accelerating loops.

Both are to do with training data. The first is the general theory; the second is specific.


Training data for platform capitalism

When I say era-defining companies, to me there’s an era-defining idea, or at least era-describing, and that’s Nick Srnicek’s concept of Platform Capitalism (Amazon).

It is the logic that underpins the success of Uber, Facebook, Amazon, Google search (and in the future, Waymo).

I’ve gone on about platform capitalism before (2020) but in a nutshell Srnicek describes a process whereby

  • these companies create a marketplace that brings together buyers and sellers
  • they gather data about what buyers want, what sellers have, how they decide on each other (marketing costs) and how decisions are finalised (transaction costs)
  • then use that data to (a) increase the velocity of marketplace activity and (b) grow the marketplace overall
  • thereby gathering data faster, increasing marketplace efficiency and size faster, gathering data faster… and so on, a runaway loop.

Even to the point that in 2012 Amazon filed a patent on anticipatory shipping in 2012 (TechCrunch) in which, if you display a strong intent to buy laundry tabs, they’ll put them on a truck and move them towards your door, only aborting delivery if you end up not hitting the Buy Now button.

And this is also kinda how Uber works right?

Uber has a better matching algorithm than you keeping the local minicab company on speed dial on your phone, which only works when you’re in your home location, and surge pricing moves drivers to hotspots in anticipation of matching with passengers.

And it’s how Google search works.

They see what people click on, and use that to improve the algo which drives marketplace activity, and AdSense keyword cost incentivises new entrants which increases marketplace size.

So how do marketplace efficiency and marketplace size translate to, say, ChatGPT?

ChatGPT can see what success looks like for a “buyer” (a ChatGPT user).

They generate an answer; do users respond well to it or not? (However that is measured.)

So that usage data becomes training data to improve the model to close the gap between user intent and transaction.

Right now, ChatGPT itself is the “seller”. To fully close the loop, they’ll need to open up to other sellers and ChatGPT itself transitions to being the market-maker (and taking a cut of transactions).

And you can see that process with the new OpenAI shopping feature right?

This is the template for all kinds of AI app products: anything that people want, any activity, if there’s a transaction at the end, the model will bring buyers and sellers closer together – marketplace efficiency.

Also there is marketplace size.

Product discovery: OpenAI can see what people type into ChatGPT. Which means they know how to target their research way better than the next company which doesn’t have access to latent user needs like that.

So here, training data for the model mainly comes from usage data. It’s a closed loop.

But how does OpenAI (or whoever) get the loop going in the first place?

With some use cases, like (say) writing a poem, the “seed” training data was in the initial web scrape; with shopping the seed training data came as a result of adding web search to chat and watching users click on links.

But there are more interesting products…

How do product managers triage tickets?

How do plumbers do their work?

You can get seed training data for those products in a couple ways but I think there’s an assumption that what happens is that the AI companies need to trick people out of their data by being present in their file system or adding an AI agent to their SaaS software at work, then hiding something in the terms of service that says the data can be used to train future models.

I just don’t feel like that assumption holds, at least not for the biggest companies.

Alternate access to seed training data method #1: just buy it.

I’ll take one example which is multiplayer chat. OpenAI just launched group chat in ChatGPT:

We’ve also taught ChatGPT new social behaviors for group chats. It follows the flow of the conversation and decides when to respond and when to stay quiet based on the context of the group conversation.

Back in May I did a deep dive into multiplayer AI chat. It’s really complicated. I outlined all the different parts of conversational turn taking theory that you need to account for to have a satisfying multiplayer conversation.

What I didn’t say at the end of that post was that, if I was building it, the whole complicated breakdown that I provided is not what I would do.

Instead I would find a big corpus of group chats for seed data and just train the model against that.

And it wouldn’t be perfect but it would be good enough to launch a product, and then you have actual live usage data coming in and you can iteratively train from there.

Where did that seed data come from for OpenAI? I don’t know. There was that reddit deal last year, maybe it was part of the bundle.

So they can buy data.

Or they can make it.

Alternate access to seed training data #2: cosplay it.

Every so often you hear gossip about how seed training data can be manufactured… I remember seeing a tweet about this a few months ago and now there’s a report:

AI agents are being trained on clones of SaaS products.

According to a new @theinformation report, Anthropic and OpenAI are building internal clones of popular SaaS apps so that they can train AI agents how to use them.

Internal researchers are giving the agents cloned, fake versions of products like Zendesk and Salesforce to teach the agents how to perform the tasks that white collar workers currently do.

The tweet I ran across was from a developer saying that cloning business apps for the purpose of being used in training was a sure-fire path to a quick acquisition, but that it felt maybe not ok.

My point is that AI companies don’t need sneak onto computers to watch product managers triaging tickets in Linear. Instead, given the future value is evident, it’s worth it to simply build a simulation of Linear, stuff it with synthetic data, then pay fake product managers to cosplay managing product inside fake Linear, and train off that.

Incidentally, the reason I keep saying seed training data is that the requirement for it is one-off. Once the product loop has started, the product creates it own. Which is why I don’t believe that revenue from licensing social network data or scientific paper is real. There will be a different pay-per-access model in the future.

I’m interested in whether this model extends to physical AI.

Will they need lanyards around the necks of plumbers in order to observe plumbing and to train the humanoid robots of the future?

Or will it be more straightforward to scrape YouTube plumbing tutorials to get started, and then build a simulation of a house (physical or virtual, in Unreal Engine) and let the AI teach itself?

What I mean is that AI companies need access to seed training data, but where it comes from is product-dependent and there are many ways to skin a cat.


That’s loop #1 – a LLM-mediated marketplace loop that (a) closes on transactions and (b) throws off usage data that improves market efficiency and reveals other products.

Per-product seed training data is a one-off investment for the AI company and can be found in many ways.

This loop produces cash.


Coding is the special loop that changes everything

Loop #2 starts with a specific product from loop #1.

A coding product isn’t just a model which is good at understanding and writing code. It has to be wrapped in an agent for planning, and ultimately needs access to collaboration tools, AI PMs, AI user researchers, and all the rest.

I think it’s pretty clear now that coding with an agent is vastly quicker than a human coding on their own. And not just quicker but, from my own experience, I can achieve goals that were previously beyond my grasp.

The loop closes when coding agents accelerate the engineers who are building the coding agents and also, as a side effect, working on the underlying general purpose large language model.

There’s an interesting kind of paperclip maximisation problem here which is, if you’re choosing where to put your resources, do you build paperclip machines or do you build the machines to build the paperclip machines?

Well it seems like all the big AI companies have made the same call right now which is to pile their efforts into accelerating coding, because doing that accelerates everything else.


So those are the two big loops.

Whoever gets those first will win, that’s how I think about it.

I want to add two notes on this.


On training data feeding the marketplace loop:

Running the platform capitalism/marketplace loop is not the only way for a company to participate in the AI product economy.

Another way is to enable it.

Stripe is doing this. They’re working hard to be the default transaction rails for AI agents.

Apple has done this for the last decade or so of the previous platform capitalism loop. iPhone is the place to reach people for all of Facebook, Google, Amazon, Uber and more.

When I said before that AI companies are trying to get closer to the point of intent, part of what I mean I that they are trying to figure out a way that a single hardware company like Apple can’t insert itself into the loop and take its 30%.

Maybe, in the future, device interactions will be super commoditised. iPhone’s power is that is bundles together an interaction surface, connectivity, compute, identity and payment, and we have one each. It’s interesting to imagine what might break that scarcity.


On coding tools that improve coding tools:

How much do you believe in this accelerating, self-improving loop?

The big AI research labs all believe – or at least, if they don’t believe, they believe that the risk of being wrong is worse.

But, if true, “tools that make better tools that allow grabbing bigger marketplaces” is an Industrial Revolution-like driver: technology went from the steam engine to the transistor in less than 200 years. Who knows what will happen this time around.

Because there’s a third loop to be found, and that’s when the models get so good that they can be used for novel R&D, and the AI labs (who have the cash and access to the cheapest compute) start commercialising wheels with weird new physics or whatever.

Or maybe it’ll stall out. Hard to know where the top of the S-curve is.


Auto-detected kinda similar posts:

Context plumbing

These past few weeks I’ve been deep in code and doing what I think about as context plumbing.

I’ve been building an AI system and that’s what it feels like.

Let me unpack.


Intent

Loosely AI interfaces are about intent and context.

Intent is the user’s goal, big or small, explicit or implicit.

Uniquely for computers, AI can understand intent and respond in a really human way. This is a new capability! Like the user can type "I want to buy a camera" or point at a keylight and subvocalise "I’ve got a call in 20 minutes" or hit a button labeled "remove clouds" and job done.

Companies care about this because computers that are closer to intent tend to win.

e.g. the smartphone displaced the desktop. On a phone, you see something and then you touch it directly. With a desktop that intent is mediated through a pointer – you see something on-screen but to interact you tell your arm to move the mouse that moves the pointer. Although it doesn’t seem like much your monkey brain doesn’t like it.

So the same applies to user interfaces in general: picking commands from menus or navigating and collating web pages to plan a holiday or remembering how the control panel on your HVAC works. All of that is bureaucracy. Figuring out the sequence for yourself is administrative burden between intent and result.

Now as an AI company, you can overcome that burden. And you want to be present at the very millisecond and in the very location where the user’s intent - desire - arises. You don’t want the user to have the burden of even taking a phone out of their pocket, or having to formulate an unconscious intent into words. Being closest to the origin of intent will crowd out their competitor companies.

That explains the push for devices like AI-enabled glasses or lanyards or mics or cameras that read your body language.

This is why I think the future of interfaces is Do What I Mean: it’s not just a new capability enabled by AI, there’s a whole attentional economics imperative to it.


Context

What makes an AI able to handle intent really, really well is context.

Sure there’s the world knowledge in the large language model itself, which it gets from vast amounts of training data.

But let’s say an AI agent is taking some user intent and hill-climbing towards that goal using a sequence of tool calls (which is how agents work) then it’s going to do way better when the prompt is filled with all kinds of useful context:

For example:

  • Background knowledge from sources like Wikipedia or Google about what others have done in this situation.
  • Documentation about the tools the agent will use to satisfy the intent.
  • The user’s context such as what they’ve done before, the time of day, etc.
  • Tacit knowledge and common ground shared between the user and the AI, i.e. what we’re all assuming we’re here to do.
  • The shared “whiteboard”: the document we’re working on.
  • For the agent itself, session context: whether this task is a subtask of a larger goal, what’s worked before and what hasn’t, and so on.

This has given rise to the idea of context engineering (LangChain blog):

Context engineering is building dynamic systems to provide the right information and tools in the right format such that the LLM can plausibly accomplish the task.

btw access to context also explains some behaviour of the big AI companies:

If you want to best answer user intent, then you need to be where the user context is, and that’s why being on a lanyard with an always-on camera is preferred over a regular on-demand camera, and why an AI agent that lives in your email archive is going to be more effective than one that doesn’t. So they really wanna get in there, really cosy up.

(And what’s context at inference time is valuable training data if it’s recorded, so there’s that too.)


Plumbing?

What’s missing in the idea of context engineering is that context is dynamic. It changes, it is timely.

Context appears at disparate sources, by user activity or changes in the user’s environment: what they’re working on changes, emails appear, documents are edited, it’s no longer sunny outside, the available tools have been updated.

This context is not always where the AI runs (and the AI runs as close as possible to the point of user intent).

So the job of making an agent run really well is to move the context to where it needs to be.

Essentially copying data out of one database and putting it into another one – but as a continuous process.

You often don’t want your AI agent to have to look up context every single time it answers intent. That’s slow. If you want an agent to act quickly then you have to plan ahead: build pipes that flow potential context from where it is created to where it’s going to be used.

How can that happen continuously behind the scenes without wasting bandwidth or cycles or the data going stale?

So I’ve been thinking of AI system technical architecture as plumbing the sources and sinks of context.


In the old days of Web 2.0 the go-to technical architecture was a “CRUD” app: a web app wrapping a database where you would have entities and operations to create, read, update, and delete (these are also the HTTP verbs).

This was also the user experience, so the user entity would have a webpage (a profile) and the object entity, say a photo, would have a webpage, and then dynamic webpages would index the entities in different ways (a stream or a feed). And you could decompose webapps like this; the technology and the user understanding aligned.

With AI systems, you want the user to have an intuition about what context is available to it. The plumbing of context flow isn’t just what is technically possible or efficient, but what matches user expectation.


Anyway.

I am aware this is getting - for you, dear reader - impossibly abstract.

But for me, I’m building the platform I’ve been trying to build for the last 2 years only this time it’s working.

I’m building on Cloudflare and I have context flowing between all kinds of entities and AI agents and sub-agents running where they need to run, and none of it feels tangled or confusing because it is plumbed just right.

And I wanted to make a note about that even if I can’t talk specifically, yet, about what it is.


Auto-detected kinda similar posts:

Spinning up a new thing: Inanimate

I’m spinning up something new with a buddy and you can guess what it is by what I’ve been writing about recently.

Big picture, there are two visions for the future of computing: cyborgs and rooms. I’m Team Augmented Environments. Mainly because so much of what I care about happens in small groups in physical space: family time, team collaboration, all the rest.

Then what happens when we’re together with AI in these environments? Interfaces will be intent-first so what’s the room-scale OS for all of this? What are the products you find there?

And where do you start?

Anyway we’ve been designing and coding and planning.

“We” is me and Daniel Fogg. We’ve known each other for ages, both done hardware, and he’s been scaling and running Big Things the last few years.

We’re at the point where it’s more straightforward than not to give this thing a name and a landing page…

Yes early days but we need a logo for renders, software protos and the raise deck haha

(Drop me a note if you’d like to chat.)

So say hello to Inanimate and you can find us over here.


Auto-detected kinda similar posts:

3 books with Samuel Arbesman

I had a look to see when I first mentioned Samuel Arbesman here. It was 2011: the average size of scientific discoveries is getting smaller.

Anyway I’ve been reading his new book, The Magic of Code (official site).

There’s computing history, magic, the simulation hypothesis, and a friendly unpacking of everything from procedural generation to Unix.

And through it all, an enthusiastic appeal to look again at computation, as if to say, look, isn’t it WEIRD! Isn’t it COOL! Because we’ve forgotten that code and computation deserves our wonder. And although this book isn’t an apology for technology ("computing is meant to be for the humans", says Arbesman), it is a reminder - demonstrated chapter by chapter - that wonder, delight and curiosity are there to be found.

(And if we look at computation afresh then we’ll have new ideas about what to do with it.)

Now I’m decently well-read in this kind of stuff.

Yet The Magic of Code is bringing me new-to-me computing lore, which I’m loving.

So, in the spirit of a virtual book tour - an old idea from the internet where book authors would tour blogs instead of book stores, as previously mentioned - I asked Samuel Arbesman for a reading list: 3 books from the Magic of Code bibliography.

(I’ve collected a couple dozen 3 Books reading lists over the years.)

I’ll ask him to introduce himself first…


Samuel! Tell us about yourself?

I’m a scientist and writer playing in the world of venture capital as Lux Capital‘s Scientist in Residence, where I help Lux explore the ever-changing landscape of science and technology, and also host a podcast called The Orthogonal Bet where I get to speak with some of the most interesting thinkers and authors I can find. I also write books about science and tech, most recently The Magic of Code, as well as The Half-Life of Facts and Overcomplicated. The themes in my work are often related to radical interdisciplinarity, intellectual humility in the face of complex technologies and our changing knowledge, and how to use tech to allow us to be the best version of ourselves.

The best way to follow me and what I’m thinking about is my newsletter: Cabinet of Wonders.

I asked for three fave books the bibliography…

#1. Ideas That Created the Future: Classic Papers of Computer Science, edited by Harry R. Lewis

I love the history of computing. It’s weird and full of strange turns and dead ends, things worth rediscovering and understanding. But it’s far too easy to forget the historically contingent reasons why we have the technologies that we have (or simply know the paths not taken), and understanding this history-including the history of the ideas that undergird this world-is vital. More broadly, I want everyone in tech to have a “historical sense” and this book is a good place to start: it’s a handbook to seminal ideas and developments in computing, from the ELIZA chatbot and Licklider’s vision of “man-computer symbiosis” to Dijkstra’s hatred of the “go to” command. Because the ideas we are currently grappling with are not necessarily new and they have a deep intellectual pedigree. Want to know the grand mages of computing history and what they thought about? Read this book.

Ideas That Created the Future: Classic Papers of Computer Science: Amazon

#2. In the Beginning… Was the Command Line, Neal Stephenson

I’m pretty sure that I first read this entire book–it’s short–in a single sitting at the library after stumbling upon it. It’s ornery and opinionated about so many computing ideas, from Linux and GUIs to open source and even the Be operating system (it was written in the 1990’s and is very much of its time). Want to think about these ideas in the context of bizarre metaphors or a comparison to the Epic of Gilgamesh? Stephenson is your guy. This expanded my mind as to what computing is and what it can mean (the image of a demiurge using a command line to generate our universe has long stuck with me).

In the Beginning… Was the Command Line: Amazon / Wikipedia

#3. Building SimCity: How to Put the World in a Machine, Chaim Gingold

Chaim Gingold worked with Will Wright while at Maxis and has thought a lot about the history of SimCity. And when I mean history, I don’t just mean the way that Maxis came about and how SimCity was created and published, though there’s that too; I mean the winding intellectual origins of SimCity: cellular automata, system dynamics, and more. SimCity and its foundation is a window into the smashing-together of so many ideas–analog computers, toys, the nature of simulation–that is indicative of the proper way to view computing: computers are weirder and far more interdisciplinary than we give them credit for and we all need to know that. Computing is a liberal art and this book takes this idea seriously.

Building SimCity: How to Put the World in a Machine: Amazon


Amazing.

Hey here’s a deep cut ref for you: in 2010 Arbesman coined the term mesofact, "facts which we tend to view as fixed, but which shift over the course of a lifetime," or too slowly for us to notice. I think we all carry around a bunch of outdated priors and that means we often don’t see what’s right in-front of us. I use this term a whole bunch in trying to think about and identity what I’m not seeing but should be.

Thank you Sam!


More posts tagged: 3-books (34).

Auto-detected kinda similar posts:

Oedipus is about the act of figuring out what Oedipus is about

Ok spoilers ahead.

But Oedipus Rex a.k.a. Oedipus Tyrannus by Sophocles is almost 2,500 years old at this point so it’s fair game imo.

The Oedipus story in a nutshell:

Oedipus, who was secretly adopted, receives a prophecy that he will kill his dad. So to thwart fate he leaves his dad and winds up in a city with a missing king (btw killing an argumentative guy on the way). Many years after becoming the new king and marrying the widow, he discovers that the dude he long-ago killed on the road was the missing king. Uh oh. And that the missing king was actually his birth dad, prophecy fulfilled. Double uh oh. And that his now-wife is therefore his birth mom. Uh oh for the third time, wife/mom suicides, stabs out own eyes, exiles self. End.

So the Sophocles play is a re-telling of this already well-worn story, at a time when Athenian culture was oriented around annual drama competitions (it came second).

The narrative new spin on the old tale is that it’s told as a whodunnit set over a single day, sunrise to sunset.

In a combination of flashbacks and new action, Oedipus himself acts as detective solving the mystery of the old king’s murder.

We’re already well into Oedipus’ reign of Thebes at the moment of the play, so his arrival is all backstory, then it’s tragic revelation after tragic revelation as his investigations bear fruit, and–

Oedipus discovers the identity of the mysterious murderer, and it’s him.

What a twist!


I mean, this is “he was dead all along” levels of whoa, right?

So I’ve been trying to think of other whodunnits in which the detective finds out that they themselves are the killer.

I can only think of one and a half, plus one I’m not sure about?

SPOILERS

SPOILERS

SPOILERS

So there’s Fight Club (1999) which, if you see it as a whodunnit in which the protagonist is trying to catch up with Tyler Durden, they discover that yes indeed etc

A clearer fit is Angel Heart (1987) in which Mickey Rourke plays PI Harry Angel who is commissioned by Robert De Niro to dig into a murder, and well you can guess who did it by my topic, and also it turns out that De Niro is the devil.

There is also Memento (2000), maybe, because ironically I can’t remember what happened.

You would have thought that detective-catching-up-with-their-quarry-and-it’s-them would be a common twist.

But yeah, 3.5 auto-whodunnits in 2.5 thousand years is not so many.

There must be more?


In literature:

I can’t think of any Agatha Christies that do this but admittedly I’ve not read too many.

There’s a sci-fi time-loop element to the auto-whodunnit - the investigating time traveller from the future turns out to be the instigator in the past - but although the concept feels tantalisingly familiar, no specific stories come to mind.


I enjoy a Straussian reading and I would like to dowse the hidden, esoteric meaning of Oedipus, Angel Heart and the rest. What is the meaning behind the story?

Freud has his famous interpretation of course but although I am taken with his take on Medusa I don’t think he goes deep enough with Oedipus.

BECAUSE:

My go-to razor for deciphering creative works is that all creative works are fundamentally about the act of creation (2017).

That’s true of Star Wars (the Force is narrative), Blade Runner (the replicants are actors), Hamlet (Shakespeare himself played the ghost), and in a follow-up post I added Groundhog Day (the experience of script iteration) and 1984 (the real omniscient Big Brother is the reader).

Many such cases, as they say.

I call it the Narcissist Creator Razor. They can’t help themselves, those creators, it’s all they know.

So I believe that Oedipus Tyrannus, the original auto-whodunnit, is the ur-exemplar of this razor: what Oedipus tells us is that we can search and search and search for the meaning of a story, and search some more, and ultimately what we’ll find is ourselves, searching.

(Even as an author, part of what you do is try to fully understand what you’re saying in your own creation, so both author and reader are engaged in working to interpret the work.)

i.e. when you interpret Oedipus, you learn that what Oedipus is really about is the act of trying to interpret what Oedipus is really about.

Which makes you want to stab your eyes out, perhaps.

Honestly I’m wasted in the technology world, I should be a philosopher working to understand the nature of reality working to understand itself over an overflowing ashtray in a smoke-filled cafe in 50s Paris.


More posts tagged: inner-and-outer-realities (6), the-ancient-world-is-now (16).

Filtered for wobbly tables and other facts

1.

When you sit with friends at a wobbly table,
Simply rotate till it becomes stable.
No need to find a wedge for one of its four feet.
Math will ensure nothing spills while you eat.

The Wobbly Table Theorem (Department of Mathematics, Harvard University).

2.

David Ogilvy changed advertising in 1951.
Shirts sold. Job done.
He used a surprise black eyepatch in the magazine spot:
“story appeal” makes consumers think a lot.

History of advertising: No 110: The Hathaway man’s eyepatch (Campaign, 2014).

3.

Frogs live in ponds.
These massive ones too.
But they dig their own ponds
when nothing else will do.

The world’s biggest frogs build their own ponds (Science, 2019).

4.

Rhyming poems have been going away,
from 70% in 1900 to almost zero today.
You know, I feel like we should all be doing our bit
to reverse the decline. But my poems are terrible.

Can you tell AI from human genius? (James Marriott).


More posts tagged: filtered-for (119).

Some wholesome media

I’m on my hols. Some recommendations.


Watch The Ballad of Wallis Island.

Charming, poignant comedy about a washed-up folk musician and loss. By Tim Key and Tom Basden.

Now I knew Basden can write - the first episode of Party is the tightest wittiest 25 minutes of ensemble radio you’ll hear - and I love everything Tim Key does as a comedian. But Key really is the revelation. Who knew he could act like that.

Watch on streaming and then listen to the soundtrack.


Play A Short Hike (I played it on Switch).

Indie video game about a cartoon bird hiking and climbing. A play-through will take you about 3 hours.

It’s cute and gentle and fun with a dozen subplots, and by the time I achieved the ostensible goal of the game I had forgotten what the purpose was and it totally took me by surprise. (Which made me cry, for personal reasons, another surprise.)

Also my kid just played this, her first self-guided video game experience. A Short Hike is deftly designed to nudge you forward through lo-fi NPC interactions, and invisibly gates levels of difficulty using geography.

Once you’ve played, watch Adam Robinson-Yu discussing A Short Hike’s design (GDC, 2020).


New daily puzzle: Clues by Sam.

A logic game that’ll take you 10 minutes each day. Follow the clues to figure out who is guilty and who is innocent. It’s easiest on Mondays so maybe begin then.


Meanwhile I’m running woodland trails in Ibiza and the scent of wild rosemary and sage fills the air in the morning. Right now I’m on a cove beach listening to the surf and the others are variously exploring, snacking and sunbathing. See you on the other side.

One more week to the Logic for Programmers Food Drive

A couple of weeks ago I started a fundraiser for the Greater Chicago Food Depository: get Logic for Programmers 50% off and all the royalties will go to charity.1 Since then, we've raised a bit over $1600. Y'all are great!

The fundraiser is going on until the end of November, so you still have one more week to get the book real cheap.

I feel a bit weird about doing two newsletter adverts without raw content, so here's a teaser from a old project I really need to get back to. Notes on structured concurrency argues that old languages had a "old-testament fire-and-brimstone goto" that could send control flow anywhere, like from the body of one function into the body of another function. This "wild goto", the article claims, what Dijkstra was railing against in Go To Statement Considered Harmful, and that modern goto statements are much more limited, "tame" if you will, and wouldn't invoke Dijkstra's ire.

I've shared this historical fact about Dijkstra many times, but recently two separate people have told me it doesn't makes sense: Dijkstra used ALGOL-60, which already had tame gotos. All of the problems he raises with goto hold even for tame ones, none are exclusive to wild gotos. So

This got me looking to see which languages, if any, ever had the wild goto. I define this as any goto which lets you jump from outside to into a loop or function scope. Turns out, FORTRAN had tame gotos from the start, BASIC has wild gotos, and COBOL is a nonsense language intentionally designed to horrify me. I mean, look at this:

The COBOL ALTER statement, which redefines a goto target

The COBOL ALTER statement changes a goto's target at runtime.

(Early COBOL has tame gotos but only on a technicality: there are no nested scopes in COBOL so no jumping from outside and into a nested scope.)

Anyway I need to write up the full story (and complain about COBOL more) but this is pretty neat! Reminder, fundraiser here. Let's get it to 2k.


  1. Royalties are 80% so if you already have the book you get a bit more bang for your buck by donating to the GCFD directly 

I'm taking a break

Hi everyone,

I've been getting burnt out on writing a weekly software essay. It's gone from taking me an afternoon to write a post to taking two or three days, and that's made it really difficult to get other writing done. That, plus some short-term work and life priorities, means now feels like a good time for a break.

So I'm taking off from Computer Things for the rest of the year. There might be some announcements and/or one or two short newsletters in the meantime but I won't be attempting a weekly cadence until 2026.

Thanks again for reading!

Hillel

Modal editing is a weird historical contingency we have through sheer happenstance

A while back my friend Pablo Meier was reviewing some 2024 videogames and wrote this:

I feel like some artists, if they didn't exist, would have the resulting void filled in by someone similar (e.g. if Katy Perry didn't exist, someone like her would have). But others don't have successful imitators or comparisons (thinking Jackie Chan, or Weird Al): they are irreplaceable.

He was using it to describe auteurs but I see this as a property of opportunity, in that "replaceable" artists are those who work in bigger markets. Katy Perry's market is large, visible and obviously (but not easily) exploitable, so there are a lot of people who'd compete in her niche. Weird Al's market is unclear: while there were successful parody songs in the past, it wasn't clear there was enough opportunity there to support a superstar.

I think that modal editing is in the latter category. Vim is now very popular and has spawned numerous successors. But its key feature, modes, is not obviously-beneficial, to the point that if Bill Joy didn't make vi (vim's direct predecessor) fifty years ago I don't think we'd have any modal editors today.

A quick overview of "modal editing"

In a non-modal editor, pressing the "u" key adds a "u" to your text, as you'd expect. In a modal editor, pressing "u" does something different depending on the "mode" you are in. In Vim's default "normal" mode, "u" undoes the last change to the text, while in the "visual" mode it lowercases all selected text. It only inserts the character in "insert" mode. All other keys, as well as chorded shortcuts (ctrl-x), work the same way.

The clearest benefit to this is you can densely pack the keyboard with advanced commands. The standard US keyboard has 48ish keys dedicated to inserting characters. With the ctrl and shift modifiers that becomes at least ~150 extra shortcuts for each other mode. This is also what IMO "spiritually" distinguishes modal editing from contextual shortcuts. Even if a unimodal editor lets you change a keyboard shortcut's behavior based on languages or focused panel, without global user-controlled modes it simply can't achieve that density of shortcuts.

Now while modal editing today is widely beloved (the Vim plugin for VSCode has at least eight million downloads), I suspect it was "carried" by the popularity of vi, as opposed to driving vi's popularity.

Modal editing is an unusual idea

Pre-vi editors weren't modal. Some, like EDT/KED, used chorded commands, while others like ed or TECO basically REPLs for text-editing DSLs. Both of these ideas widely reappear in modern editors.

As far as I can tell, the first modal editor was Butler Lampson's Bravo in 1974. Bill Joy admits he used it for inspiration:

A lot of the ideas for the screen editing mode were stolen from a Bravo manual I surreptitiously looked at and copied. Dot is really the double-escape from Bravo, the redo command. Most of the stuff was stolen.

Bill Joy probably took the idea because he was working on dumb terminals that were slow to register keystrokes, which put pressure to minimize the number needed for complex operations.

Why did Bravo have modal editing? Looking at the Alto handbook, I get the impression that Xerox was trying to figure out the best mouse and GUI workflows. Bravo was an experiment with modes, one hand on the mouse and one issuing commands on the keyboard. Other experiments included context menus (the Markup program) and toolbars (Draw).

Xerox very quickly decided against modes, as the successors Gypsy and BravoX were modeless. Commands originally assigned to English letters were moved to graphical menus, special keys, and chords.

It seems to me that modes started as an unsuccessful experiment deal with a specific constraint and then later successfully adopted to deal with a different constraint. It was a specialized feature as opposed to a generally useful feature like chords.

Modal editing didn't popularize vi

While vi was popular at Bill Joy's coworkers, he doesn't attribute its success to its features:

I think the wonderful thing about vi is that it has such a good market share because we gave it away. Everybody has it now. So it actually had a chance to become part of what is perceived as basic UNIX. EMACS is a nice editor too, but because it costs hundreds of dollars, there will always be people who won't buy it.

Vi was distributed for free with the popular BSD Unix and was standardized in POSIX Issue 2, meaning all Unix OSes had to have vi. That arguably is what made it popular, and why so many people ended up learning a modal editor.

Modal editing doesn't really spread outside of vim

I think by the 90s, people started believing that modal editing was a Good Idea, if not an obvious one. That's why we see direct descendants of vi, most famously vim. It's also why extensible editors like Emacs and VSCode have vim-mode extensions, but these are but these are always simple emulation layers on top of a unimodal baselines. This was good for getting people used to the vim keybindings (I learned on Kile) but it means people weren't really doing anything with modal editing. It was always "The Vim Gimmick".

Modes also didn't take off anywhere else. There's no modal word processor, spreadsheet editor, or email client.1 Visidata is an extremely cool modal data exploration tool but it's pretty niche. Firefox used to have vimperator (which was inspired by Vim) but that's defunct now. Modal software means modal editing which means vi.

This has been changing a little, though! Nowadays we do see new modal text editors, like kakoune and Helix, that don't just try to emulate vi but do entirely new things. These were made, though, in response to perceived shortcomings in vi's editing model. I think they are still classifiable as descendants. If vi never existed, would the developers of kak and helix have still made modal editors, or would they have explored different ideas?

People aren't clamouring for more experiments

Not too related to the overall picture, but a gripe of mine. Vi and vim have a set of hardcoded modes, and adding an entirely new mode is impossible. Like if a plugin (like vim's default netrw) adds a file explorer it should be able to add a filesystem mode, right? But it can't, so instead it waits for you to open the filesystem and then adds 60 new mappings to normal mode. There's no way to properly add a "filesystem" mode, a "diff" mode, a "git" mode, etc, so plugin developers have to mimic them.

I don't think people see this as a problem, though! Neovim, which aims to fix all of the baggage in vim's legacy, didn't consider creating modes an important feature. Kak and Helix, which reimagine modal editing from from the ground up, don't support creating modes either.2 People aren't clamouring for new modes!

Modes are a niche power user feature

So far I've been trying to show that vi is, in Pablo's words, "irreplaceable". Editors weren't doing modal editing before Bravo, and even after vi became incredibly popular, unrelated editors did not adapt modal editing. At most, they got a vi emulation layer. Kak and helix complicate this story but I don't think they refute it; they appear much later and arguably count as descendants (so are related).

I think the best explanation is that in a vacuum modal editing sounds like a bad idea. The mode is global state that users always have to know, which makes it dangerous. To use new modes well you have to memorize all of the keybindings, which makes it difficult. Modal editing has a brutal skill floor before it becomes more efficient than a unimodal, chorded editor like VSCode.

That's why it originally appears in very specific circumstances, as early experiments in mouse UX and as a way of dealing with modem latencies. The fact we have vim today is a historical accident.

And I'm glad for it! You can pry Neovim from my cold dead hands, you monsters.


P99 talk this Thursday!

My talk, "Designing Low-Latency Systems with TLA+", is happening 10/23 at 11:40 central time. Tickets are free, the conf is online, and the talk's only 16 minutes, so come check it out!


  1. I guess if you squint gmail kinda counts but it's basically an antifeature 

  2. It looks like Helix supports creating minor modes, but these are only active for one keystroke, making them akin to a better, more ergonomic version of vim multikey mappings. 

HortiRoot scanners

Beste Zeussers en Zeusinnen,

Enkele weken geleden werden we gecontacteerd door de onderzoeksgroep HortiRoot. In hun lab hebben ze een redelijk aantal Epson flatbed foto scanners omgebouwd om er kleine plantjes in petriplaten op te kunnen groeien. Hiermee kunnen ze met korte tijdsintervallen scans nemen, om timelapses te maken van de groei in hoge resolutie.

A metal frame holding 4 scanners in place, sitting in a lab. On the left there is an electrical box where the LED controls and Arduinos are housed.

De UGent workshop WE62 heeft hun geholpen met de hardware mods:

  • Het deksel (lid) werd verwijderd. De elektronica verantwoordelijk voor de “closed” detectie wordt nu nagebootst door een Arduino.
  • Over het bed kan nu een kunststoffen plaat geschoven worden waar de petriplaten in passen.
  • Een frame werd gemaakt om alles op zijn plaats te houden, en zodat de scans verticaal gemaakt kunnen worden.
  • Ook is er nu een ledstrip om semi-natuurlijk licht na te bootsen.

Voor de software merkten de onderzoekers echter dat de UGent geen equivalent van die werkplaats heeft. De onderzoekers van HortiRoot hebben zelf wat geprobeerd om de standaard Windows 10 Epson software te automatiseren met behulp van pyautogui. Uiteindelijk hadden ze wel een werkend systeem, dat momenteel nog steeds gebruikt wordt. Onderhoudbaar is dit helemaal niet: automatisch de muis proberen bedienen schaalt niet bepaald goed… Ze hadden ook de software voor Ubuntu uitgeprobeerd, maar die avonturen waren jammer genoeg van korte duur omdat DICT geen ondersteundend personeel had met voldoende expertise hiermee. Nu recent stak de verplichte update naar Windows 11 ook een spaak in het wiel.

Na wat details van het project te horen wees DICT de onderzoeksgroep naar ons door. Even later werden Xander, Hannes en ik hartelijk ontvangen op campus Coupure. In een labo vol groen konden wij de verschillende iteraties van de scanner mods aanschouwen. Maar vooral van de vele hacks die ze gevonden hadden om toch maar de software te doen werken, waren we onder de indruk. Zo lukte het niet om de scan software te gebruiken met meerdere apparaten aangesloten, dus hadden ze als oplossing om elke scanner op een aparte virtuele machine aan te sluiten. Niet ideaal, dus probeerden ze om maar 1 apparaat tegelijk te tonen aan de software met behulp van programmeerbare USB hubs. Dit werkte niet meer in Windows 11, maar zelfs dan was dit vrij beperkt in aantal apparaten. Ook hadden ze ontdekt dat ze doorheen dinsdagnacht geen scans konden doen, omdat DICT patch tuesdays doet…

Na uitwisselen van ons intern telefoonnummer, werden even later 2 van de scanners geleverd aan de Zeuskelder. Er zijn hier al wat mensen bezig geweest met de scanners om te helpen de opties te verkennen. Wat ons meteen verrastte is dat de Linux software epsonscan2 die Epson publiceert, bijna volledig open-source en vrij is. Bovendien is dit een grote bron aan documentatie van de interne werking van de scanners. De enige drempel is dat die deels in het Japans is 😅.

Om deze software inderdaad werkend te krijgen, is een ander verhaal. Wanneer de software probeerde te verbinden gingen de scanners steeds in een “internal error” state. Door wat instrumentatie aan esponscan2 toe voegen en veel verschillende opstellingen te proberen, vonden we uiteindelijk een oplossing. We merkten dat een scanner slechts éénmaal correct geïnitializeerd moest worden door andere software (zoals de windows 11 versie), waarna de scanner werkt met alle software. Door de communicatie te onderscheppen, konden we een programma maken dat de scanners op dezelfde manier kan initializeren. Waarom exact dit nodig is, is nog niet helemaal duidelijk.

We zijn opgelucht dat dit werkt, omdat dit betekent dat we helemaal geen nood hebben aan Windows, virtuele machines, USB multiplexers of het automatisch navigeren van menu’s. In de plaats kunnen we volledig werken met de betrouwbare en voorspelbare software die we gewend zijn.

Het plan is om scanners per ~4 te groeperen, die samen aangestuurd worden door 1 node (Raspberry Pi). Deze nodes worden ontdekt op het lokale netwerk door de backend van een webapplicatie. Die webapplicatie stuurt dan requests uit om scans te nemen met bepaalde parameters, op regelmatige intervallen. De geproduceerde afbeeldingen worden dan verplaatst van de node naar de machine waar de webapp draait. Gebruikers kunnen dan die timelapses beheren vanuit de webapp op het lokaal netwerk, en eveneens ook de afbeeldingen downloaden.

Wat is reeds geïmplementeerd?

  • Nodes:
    • Er is een script dat nieuw aangesloten scanners detecteert en ze meteen correct initializeert.
      • Dit gebeurt hier door specifieke USB_BULK pakketten te sturen, onder andere met de firmware blob.
      • Het script (sc) is in D geschreven, zogoed als alles wordt gedaan via de libusb C library.
      • Soms faalt het initializeren door een “permission denied”, omdat de udev rule nog niet geactiveerd is op het moment dat libusb het apparaat detecteert. Als root uitvoeren fixt het…
      • Het protocol dat gebruikt wordt bovenop USB_BULK is ESC/I. Enkele links naar documentatie hierover zijn hier te vinden.
      • Dit protocol verder ontdekken kan nog steeds handig zijn: Het lukt momenteel niet om elke scanner uniek te identificeren. Ze dragen geen serial number zoals de meeste USB devices. Misschien is er wel een command om iets gelijkaardig op te slaan? Ik zie ook verwijzingen naar een non-volatile user variables api.
    • De epsonscan2 software build vanuit de git repo met behulp van docker.
      • De build artifact is gemaakt voor Debian Bookworm. Kijken of het mogelijk is te upgraden naar Trixie.
      • Er is een dependency op een non-free tool, die binair gepatcht wordt om de paden ook naar /opt/epsonscan2 te doen wijzen. Deze prefix is gemakkelijker te gebruiken dan .deb packages maken die naar /usr uitpakken.
      • De software heeft een command-line versie, maar die kan nog niet een bepaalde scanner selecteren. We kunnen dit zelf implementeren.
  • Webapp:
    • Nog niets! Bepaal zelf welke taal/framework (of talen/frameworks, meer is beter right?). Requirements in de repo te vinden.

Kijk zeker eens in de git repo en join al vast het ~horitroot-scanners kanaal op Mattermost!

PS: Wat zouden wij kunnen scannen?

Systems design 3: LLMs and the semantic revolution

Long ago in the 1990s when I was in high school, my chemistry+physics teacher pulled me aside. "Avery, you know how the Internet works, right? I have a question."

I now know the correct response to that was, "Does anyone really know how the Internet works?" But as a naive young high schooler I did not have that level of self-awareness. (Decades later, as a CEO, that's my answer to almost everything.)

Anyway, he asked his question, and it was simple but deep. How do they make all the computers connect?

We can't even get the world to agree on 60 Hz vs 50 Hz, 120V vs 240V, or which kind of physical power plug to use. Communications equipment uses way more frequencies, way more voltages, way more plug types. Phone companies managed to federate with each other, eventually, barely, but the ring tones were different everywhere, there was pulse dialing and tone dialing, and some of them still charge $3/minute for international long distance, and connections take a long time to establish and humans seem to be involved in suspiciously many places when things get messy, and every country has a different long-distance dialing standard and phone number format.

So Avery, he said, now they're telling me every computer in the world can connect to every other computer, in milliseconds, for free, between Canada and France and China and Russia. And they all use a single standardized address format, and then you just log in and transfer files and stuff? How? How did they make the whole world cooperate? And who?

When he asked that question, it was a formative moment in my life that I'll never forget, because as an early member of what would be the first Internet generation… I Had Simply Never Thought of That.

I mean, I had to stop and think for a second. Wait, is protocol standardization even a hard problem? Of course it is. Humans can't agree on anything. We can't agree on a unit of length or the size of a pint, or which side of the road to drive on. Humans in two regions of Europe no farther apart than Thunder Bay and Toronto can't understand each other's speech. But this Internet thing just, kinda, worked.

"There's… a layer on top," I uttered, unsatisfyingly. Nobody had taught me yet that the OSI stack model existed, let alone that it was at best a weak explanation of reality.

"When something doesn't talk to something else, someone makes an adapter. Uh, and some of the adapters are just programs rather than physical things. It's not like everyone in the world agrees. But as soon as one person makes an adapter, the two things come together."

I don't think he was impressed with my answer. Why would he be? Surely nothing so comprehensively connected could be engineered with no central architecture, by a loosely-knit cult of mostly-volunteers building an endless series of whimsical half-considered "adapters" in their basements and cramped university tech labs. Such a creation would be a monstrosity, just as likely to topple over as to barely function.

I didn't try to convince him, because honestly, how could I know? But the question has dominated my life ever since.

When things don't connect, why don't they connect? When they do, why? How? …and who?

Postel's Law

The closest clue I've found is this thing called Postel's Law, one of the foundational principles of the Internet. It was best stated by one of the founders of the Internet, Jon Postel. "Be conservative in what you send, and liberal in what you accept."

What it means to me is, if there's a standard, do your best to follow it, when you're sending. And when you're receiving, uh, assume the best intentions of your counterparty and do your best and if that doesn't work, guess.

A rephrasing I use sometimes is, "It takes two to miscommunicate." Communication works best and most smoothly if you have a good listener and a clear speaker, sharing a language and context. But it can still bumble along successfully if you have a poor speaker with a great listener, or even a great speaker with a mediocre listener. Sometimes you have to say the same thing five ways before it gets across (wifi packet retransmits), or ask way too many clarifying questions, but if one side or the other is diligent enough, you can almost always make it work.

This asymmetry is key to all high-level communication. It makes network bugs much less severe. Without Postel's Law, triggering a bug in the sender would break the connection; so would triggering a bug in the receiver. With Postel's Law, we acknowledge from the start that there are always bugs and we have twice as many chances to work around them. Only if you trigger both sets of bugs at once is the flaw fatal.

…So okay, if you've used the Internet, you've probably observed that fatal connection errors are nevertheless pretty common. But that misses how incredibly much more common they would be in a non-Postel world. That world would be the one my physics teacher imagined, where nothing ever works and it all topples over.

And we know that's true because we've tried it. Science! Let us digress.

XML

We had the Internet ("OSI Layer 3") mostly figured out by the time my era began in the late 1900s, but higher layers of the stack still had work to do. It was the early days of the web. We had these newfangled hypertext ("HTML") browsers that would connect to a server, download some stuff, and then try their best to render it.

Web browsers are and have always been an epic instantiation of Postel's Law. From the very beginning, they assumed that the server (content author) had absolutely no clue what they were doing and did their best to apply some kind of meaning on top, despite every indication that this was a lost cause. List items that never end? Sure. Tags you've never heard of? Whatever. Forgot some semicolons in your javascript? I'll interpolate some. Partially overlapping italics and bold? Leave it to me. No indication what language or encoding the page is in? I'll just guess.

The evolution of browsers gives us some insight into why Postel's Law is a law and not just, you know, Postel's Advice. The answer is: competition. It works like this. If your browser interprets someone's mismash subjectively better than another browser, your browser wins.

I think economists call this an iterated prisoner's dilemma. Over and over, people write web pages (defect) and browsers try to render them (defect) and absolutely nobody actually cares what the HTML standard says (stays loyal). Because if there's a popular page that's wrong and you render it "right" and it doesn't work? Straight to jail.

(By now almost all the evolutionary lines of browsers have been sent to jail, one by one, and the HTML standard is effectively whatever Chromium and Safari say it is. Sorry.)

This law offends engineers to the deepness of their soul. We went through a period where loyalists would run their pages through "validators" and proudly add a logo to the bottom of their page saying how valid their HTML was. Browsers, of course, didn't care and continued to try their best.

Another valiant effort was the definition of "quirks mode": a legacy rendering mode meant to document, normalize, and push aside all the legacy wonko interpretations of old web pages. It was paired with a new, standards-compliant rendering mode that everyone was supposed to agree on, starting from scratch with an actual written spec and tests this time, and public shaming if you made a browser that did it wrong. Of course, outside of browser academia, nobody cares about the public shaming and everyone cares if your browser can render the popular web sites, so there are still plenty of quirks outside quirks mode. It's better and it was well worth the effort, but it's not all the way there. It never can be.

We can be sure it's not all the way there because there was another exciting development, HTML Strict (and its fancier twin, XHTML), which was meant to be the same thing, but with a special feature. Instead of sending browsers to jail for rendering wrong pages wrong, we'd send page authors to jail for writing wrong pages!

To mark your web page as HTML Strict was a vote against the iterated prisoner's dilemma and Postel's Law. No, your vote said. No more. We cannot accept this madness. We are going to be Correct. I certify this page is correct. If it is not correct, you must sacrifice me, not all of society. My honour demands it.

Anyway, many page authors were thus sacrificed and now nobody uses HTML Strict. Nobody wants to do tech support for a web page that asks browsers to crash when parsing it, when you can just… not do that.

Excuse me, the above XML section didn't have any XML

Yes, I'm getting to that. (And you're soon going to appreciate that meta joke about schemas.)

In parallel with that dead branch of HTML, a bunch of people had realized that, more generally, HTML-like languages (technically SGML-like languages) had turned out to be a surprisingly effective way to build interconnected data systems.

In retrospect we now know that the reason for HTML's resilience is Postel's Law. It's simply easier to fudge your way through parsing incorrect hypertext, than to fudge your way through parsing a Microsoft Word or Excel file's hairball of binary OLE streams, which famously even Microsoft at one point lost the knowledge of how to parse. But, that Postel's Law connection wasn't really understood at the time.

Instead we had a different hypothesis: "separation of structure and content." Syntax and semantics. Writing software to deal with structure is repetitive overhead, and content is where the money is. Let's automate away the structure so you can spend your time on the content: semantics.

We can standardize the syntax with a single Extensible Markup Language (XML). Write your content, then "mark it up" by adding structure right in the doc, just like we did with plaintext human documents. Data, plus self-describing metadata, all in one place. Never write a parser again!

Of course, with 20/20 hindsight (or now 2025 hindsight), this is laughable. Yes, we now have XML parser libraries. If you've ever tried to use one, you will find they indeed produce parse trees automatically… if you're lucky. If you're not lucky, they produce a stream of "tokens" and leave it to you to figure out how to arrange it in a tree, for reasons involving streaming, performance, memory efficiency, and so on. Basically, if you use XML you now have to deeply care about structure, perhaps more than ever, but you also have to include some giant external parsing library that, left in its normal mode, might spontaneously start making a lot of uncached HTTP requests that can also exploit remote code execution vulnerabilities haha oops.

If you've ever taken a parser class, or even if you've just barely tried to write a parser, you'll know the truth: the value added by outsourcing parsing (or in some cases only tokenization) is not a lot. This is because almost all the trouble of document processing (or compiling) is the semantic layer, the part where you make sense of the parse tree. The part where you just read a stream of characters into a data structure is the trivial, well-understood first step.

Now, semantics is where it gets interesting. XML was all about separating syntax from semantics. And they did some pretty neat stuff with that separation, in a computer science sense. XML is neat because it's such a regular and strict language that you can completely validate the syntax (text and tags) without knowing what any of the tags mean or which tags are intended to be valid at all.

…aha! Did someone say validate?! Like those old HTML validators we talked about? Oh yes. Yes! And this time the validation will be completely strict and baked into every implementation from day 1. And, the language syntax itself will be so easy and consistent to validate (unlike SGML and HTML, which are, in all fairness, bananas) that nobody can possibly screw it up.

A layer on top of this basic, highly validatable XML, was a thing called XML Schemas. These were documents (mysteriously not written in XML) that described which tags were allowed in which places in a certain kind of document. Not only could you parse and validate the basic XML syntax, you could also then validate its XML schema as a separate step, to be totally sure that every tag in the document was allowed where it was used, and present if it was required. And if not? Well, straight to jail. We all agreed on this, everyone. Day one. No exceptions. Every document validates. Straight to jail.

Anyway XML schema validation became an absolute farce. Just parsing or understanding, let alone writing, the awful schema file format is an unpleasant ordeal. To say nothing of complying with the schema, or (heaven forbid) obtaining a copy of someone's custom schema and loading it into the validator at the right time.

The core XML syntax validation was easy enough to do while parsing. Unfortunately, in a second violation of Postel's Law, almost no software that outputs XML runs it through a validator before sending. I mean, why would they, the language is highly regular and easy to generate and thus the output is already perfect. …Yeah, sure.

Anyway we all use JSON now.

JSON

Whoa, wait! I wasn't done!

This is the part where I note, for posterity's sake, that XML became a decade-long fad in the early 2000s that justified billions of dollars of software investment. None of XML's technical promises played out; it is a stain on the history of the computer industry. But, a lot of legacy software got un-stuck because of those billions of dollars, and so we did make progress.

What was that progress? Interconnection.

Before the Internet, we kinda didn't really need to interconnect software together. I mean, we sort of did, like cut-and-pasting between apps on Windows or macOS or X11, all of which were surprisingly difficult little mini-Postel's Law protocol adventures in their own right and remain quite useful when they work (except "paste formatted text," wtf are you people thinking). What makes cut-and-paste possible is top-down standards imposed by each operating system vendor.

If you want the same kind of thing on the open Internet, ie. the ability to "copy" information out of one server and "paste" it into another, you need some kind of standard. XML was a valiant effort to create one. It didn't work, but it was valiant.

Whereas all that money investment did work. Companies spent billions of dollars to update their servers to publish APIs that could serve not just human-formatted HTML, but also something machine-readable. The great innovation was not XML per se, it was serving data over HTTP that wasn't always HTML. That was a big step, and didn't become obvious until afterward.

The most common clients of HTTP were web browsers, and web browsers only knew how to parse two things: HTML and javascript. To a first approximation, valid XML is "valid" (please don't ask the validator) HTML, so we could do that at first, and there were some Microsoft extensions. Later, after a few billions of dollars, true standardized XML parsing arrived in browsers. Similarly, to a first approximation, valid JSON is valid javascript, which woo hoo, that's a story in itself (you could parse it with eval(), tee hee) but that's why we got here.

JSON (minus the rest of javascript) is a vastly simpler language than XML. It's easy to consistently parse (other than that pesky trailing comma); browsers already did. It represents only (a subset of) the data types normal programming languages already have, unlike XML's weird mishmash of single attributes, multiply occurring attributes, text content, and CDATA. It's obviously a tree and everyone knows how that tree will map into their favourite programming language. It inherently works with unicode and only unicode. You don't need cumbersome and duplicative "closing tags" that double the size of every node. And best of all, no guilt about skipping that overcomplicated and impossible-to-get-right schema validator, because, well, nobody liked schemas anyway so nobody added them to JSON (almost).

Today, if you look at APIs you need to call, you can tell which ones were a result of the $billions invested in the 2000s, because it's all XML. And you can tell which came in the 2010s and later after learning some hard lessons, because it's all JSON. But either way, the big achievement is you can call them all from javascript. That's pretty good.

(Google is an interesting exception: they invented and used protobuf during the same time period because they disliked XML's inefficiency, they did like schemas, and they had the automated infrastructure to make schemas actually work (mostly, after more hard lessons). But it mostly didn't spread beyond Google… maybe because it's hard to do from javascript.)

Blockchain

The 2010s were another decade of massive multi-billion dollar tech investment. Once again it was triggered by an overwrought boondoggle technology, and once again we benefited from systems finally getting updated that really needed to be updated.

Let's leave aside cryptocurrencies (which although used primarily for crime, at least demonstrably have a functioning use case, ie. crime) and look at the more general form of the technology.

Blockchains in general make the promise of a "distributed ledger" which allows everyone the ability to make claims and then later validate other people's claims. The claims that "real" companies invested in were meant to be about manufacturing, shipping, assembly, purchases, invoices, receipts, ownership, and so on. What's the pattern? That's the stuff of businesses doing business with other businesses. In other words, data exchange. Data exchange is exactly what XML didn't really solve (although progress was made by virtue of the dollars invested) in the previous decade.

Blockchain tech was a more spectacular boondoggle than XML for a few reasons. First, it didn't even have a purpose you could explain. Why do we even need a purely distributed system for this? Why can't we just trust a third party auditor? Who even wants their entire supply chain (including number of widgets produced and where each one is right now) to be visible to the whole world? What is the problem we're trying to solve with that?

…and you know there really was no purpose, because after all the huge investment to rewrite all that stuff, which was itself valuable work, we simply dropped the useless blockchain part and then we were fine. I don't think even the people working on it felt like they needed a real distributed ledger. They just needed an updated ledger and a budget to create one. If you make the "ledger" module pluggable in your big fancy supply chain system, you can later drop out the useless "distributed" ledger and use a regular old ledger. The protocols, the partnerships, the databases, the supply chain, and all the rest can stay the same.

In XML's defense, at least it was not worth the effort to rip out once the world came to its senses.

Another interesting similarity between XML and blockchains was the computer science appeal. A particular kind of person gets very excited about validation and verifiability. Both times, the whole computer industry followed those people down into the pits of despair and when we finally emerged… still no validation, still no verifiability, still didn't matter. Just some computers communicating with each other a little better than they did before.

LLMs

In the 2020s, our industry fad is LLMs. I'm going to draw some comparisons here to the last two fads, but there are some big differences too.

One similarity is the computer science appeal: so much math! Just the matrix sizes alone are a technological marvel the likes of which we have never seen. Beautiful. Colossal. Monumental. An inspiration to nerds everywhere.

But a big difference is verification and validation. If there is one thing LLMs absolutely are not, it's verifiable. LLMs are the flakiest thing the computer industry has ever produced! So far. And remember, this is the industry that brought you HTML rendering.

LLMs are an almost cartoonishly amplified realization of Postel's Law. They write human grammar perfectly, or almost perfectly, or when they're not perfect it's a bug and we train them harder. And, they can receive just about any kind of gibberish and turn it into a data structure. In other words, they're conservative in what they send and liberal in what they accept.

LLMs also solve the syntax problem, in the sense that they can figure out how to transliterate (convert) basically any file syntax into any other. Modulo flakiness. But if you need a CSV in the form of a limerick or a quarterly financial report formatted as a mysql dump, sure, no problem, make it so.

In theory we already had syntax solved though. XML and JSON did that already. We were even making progress interconnecting old school company supply chain stuff the hard way, thanks to our nominally XML- and blockchain- investment decades. We had to do every interconnection by hand – by writing an adapter – but we could do it.

What's really new is that LLMs address semantics. Semantics are the biggest remaining challenge in connecting one system to another. If XML solved syntax, that was the first 10%. Semantics are the last 90%. When I want to copy from one database to another, how do I map the fields? When I want to scrape a series of uncooperative web pages and turn it into a table of products and prices, how do I turn that HTML into something structured? (Predictably microformats, aka schemas, did not work out.) If I want to query a database (or join a few disparate databases!) using some language that isn't SQL, what options do I have?

LLMs can do it all.

Listen, we can argue forever about whether LLMs "understand" things, or will achieve anything we might call intelligence, or will take over the world and eradicate all humans, or are useful assistants, or just produce lots of text sludge that will certainly clog up the web and social media, or will also be able to filter the sludge, or what it means for capitalism that we willingly invented a machine we pay to produce sludge that we also pay to remove the sludge.

But what we can't argue is that LLMs interconnect things. Anything. To anything. Whether you like it or not. Whether it's bug free or not (spoiler: it's not). Whether it gets the right answer or not (spoiler: erm…).

This is the thing we have gone through at least two decades of hype cycles desperately chasing. (Three, if you count java "write once run anywhere" in the 1990s.) It's application-layer interconnection, the holy grail of the Internet.

And this time, it actually works! (mostly)

The curse of success

LLMs aren't going away. Really we should coin a term for this use case, call it "b2b AI" or something. For this use case, LLMs work. And they're still getting better and the precision will improve with practice. For example, imagine asking an LLM to write a data translator in some conventional programming language, instead of asking it to directly translate a dataset on its own. We're still at the beginning.

But, this use case, which I predict is the big one, isn't what we expected. We expected LLMs to write poetry or give strategic advice or whatever. We didn't expect them to call APIs and immediately turn around and use what it learned to call other APIs.

After 30 years of trying and failing to connect one system to another, we now have a literal universal translator. Plug it into any two things and it'll just go, for better or worse, no matter how confused it becomes. And everyone is doing it, fast, often with a corporate mandate to do it even faster.

This kind of scale and speed of (successful!) rollout is unprecedented, even by the Internet itself, and especially in the glacially slow world of enterprise system interconnections, where progress grinds to a halt once a decade only to be finally dislodged by the next misguided technology wave. Nobody was prepared for it, so nobody was prepared for the consequences.

One of the odd features of Postel's Law is it's irresistible. Big Central Infrastructure projects rise and fall with funding, but Postel's Law projects are powered by love. A little here, a little there, over time. One more person plugging one more thing into one more other thing. We did it once with the Internet, overcoming all the incompatibilities at OSI layers 1 and 2. It subsumed, it is still subsuming, everything.

Now we're doing it again at the application layer, the information layer. And just like we found out when we connected all the computers together the first time, naively hyperconnected networks make it easy for bad actors to spread and disrupt at superhuman speeds. We had to invent firewalls, NATs, TLS, authentication systems, two-factor authentication systems, phishing-resistant two-factor authentication systems, methodical software patching, CVE tracking, sandboxing, antivirus systems, EDR systems, DLP systems, everything. We'll have to do it all again, but faster and different.

Because this time, it's all software.

memories of .us

How much do you remember from elementary school? I remember vinyl tile floors, the playground, the teacher sentencing me to standing in the hallway. I had a teacher who was a chess fanatic; he painted a huge chess board in the paved schoolyard and got someone to fabricate big wooden chess pieces. It was enough of an event to get us on the evening news. I remember Run for the Arts, where I tried to talk people into donating money on the theory that I could run, which I could not. I'm about six months into trying to change that and I'm good for a mediocre 5k now, but I don't think that's going to shift the balance on K-12 art funding.

I also remember a domain name: bridger.pps.k12.or.us

I have quipped before that computer science is a field mostly concerned with assigning numbers to things, which is true, but it only takes us so far. Computer scientists also like to organize those numbers into structures, and one of their favorites has always been the tree. The development of wide-area computer networking surfaced a whole set of problems around naming or addressing computer systems that belong to organizations. A wide-area network consists of a set of institutions that manage their own affairs. Each of those institutions may be made up of departments that manage their own affairs. A tree seemed a natural fit. Even the "low level" IP addresses, in the days of "classful" addressing, were a straightforward hierarchy: each dot separated a different level of the tree, a different step in an organizational hierarchy.

The first large computer networks, including those that would become the Internet, initially relied on manually building lists of machines by name. By the time the Domain Name System was developed, this had already become cumbersome. The rapid growth of the internet was hard to keep up with, and besides, why did any one central entity---Jon Postel or whoever---even care about the names of all of the computers at Georgia Tech? Like IP addressing, DNS was designed as a hierarchy with delegated control. A registrant obtains a name in the hierarchy, say gatech.edu, and everything "under" that name is within the control, and responsibility, of the registrant. This arrangement is convenient for both the DNS administrator, which was a single organization even after the days of Postel, and for registrants.

We still use the same approach today... mostly. The meanings of levels of the hierarchy have ossified. Technically speaking, the top of the DNS tree, the DNS root, is a null label referenced by a trailing dot. It's analogous to the '/' at the beginning of POSIX file paths. "gatech.edu" really should be written as "gatech.edu." to make it absolute rather than relative, but since resolution of relative URLs almost always recurses to the top of the tree, the trailing dot is "optional" enough that it is now almost always omitted. The analogy to POSIX file paths raises an interesting point: domain names are backwards. The 'root' is at the end, rather than at the beginning, or in other words, they run from least significant to most significant, rather than most significant to least significant. That's just... one of those things, you know? In the early days one wasn't obviously better than the other, people wrote hierarchies out both ways, and as the dust settled the left-to-right convention mostly prevailed but right-to-left hung around in some protocols. If you've ever dealt with endianness, this is just one of those things about computers that you have to accept: we cannot agree on which way around to write things.

Anyway, the analogy to file paths also illustrates the way that DNS has ossified. The highest "real" or non-root component of a domain name is called the top-level domain or TLD, while the component below it is called a second-level domain. In the US, it was long the case that top-level domains were fixed while second-level domains were available for registration. There have always been exceptions in other countries and our modern proliferation of TLDs has changed this somewhat, but it's still pretty much true. When you look at "gatech.edu" you know that "edu" is just a fixed name in the hierarchy, used to organize domain names by organization type, while "gatech" is a name that belongs to a registrant.

Under the second-level name, things get a little vague. We are all familiar with the third-level name "www," which emerged as a convention for web servers and became a practical requirement. Web servers having the name "www" under an organization's domain was such a norm for so many years that hosting a webpage directly at a second-level name came to be called a "naked domain" and had some caveats and complications.

Other than www, though, there are few to no standards for the use of third-level and below names. Larger organizations are more likely to use third-level names for departments, infrastructure operators often have complex hierarchies of names for their equipment, and enterprises the world 'round name their load-balanced webservers "www2," "www3" and up. If you think about it, this situation seems like kind of a failure of the original concept of DNS... we do use the hierarchy, but for the most part it is not intended for human consumption. Users are only expected to remember two names, one of which is a TLD that comes from a relatively constrained set.

The issue is more interesting when we consider geography. For a very long time, TLDs have been split into two categories: global TLDs, or gTLDs, and country-code TLDs, or ccTLDs. ccTLDs reflect the ISO country codes of each country, and are intended for use by those countries, while gTLDs are arbitrary and reflect the fact that DNS was designed in the US. The ".gov" gTLD, for example, is for use by the US government, while the UK is stuck with ".gov.uk". This does seem unfair but it's now very much cemented into the system: for the large part, US entities use gTLDs, while entities in other countries use names under their respective ccTLDs. The ".us" ccTLD exists just as much as all the others, but is obscure enough that my choice to put my personal website under .us (not an ideological decision but simply a result of where a nice form of my name was available) sometimes gets my email address rejected.

Also, a common typo for ".us" is ".su" and that's geopolitically amusing. .su is of course the ccTLD for the Soviet Union, which no longer exists, but the ccTLD lives on in a limited way because it became Structurally Important and difficult to remove, as names and addresses tend to do.

We can easily imagine a world where this historical injustice had been fixed: as the internet became more global, all of our US institutions could have moved under the .us ccTLD. In fact, why not go further? Geographers have long organized political boundaries into a hierarchy. The US is made up of states, each of which has been assigned a two-letter code by the federal government. We have ".us", why not "nm.us"?

The answer, of course, is that we do.

In the modern DNS, all TLDs have been delegated to an organization who administers them. The .us TLD is rightfully administered by the National Telecommunications and Information Administration, on the same basis by which all ccTLDs are delegated to their respective national governments. Being the US government, NTIA has naturally privatized the function through a contract to telecom-industrial-complex giant Neustar. Being a US company, Neustar restructured and sold its DNS-related business to GoDaddy. Being a US company, GoDaddy rose to prominence on the back of infamously tasteless television commercials, and its subsidiary Registry Services LLC now operates our nation's corner of the DNS.

But that's the present---around here, we avoid discussing the present so as to hold crushing depression at bay. Let's turn our minds to June 1993, and the publication of RFC 1480 "The US Domain." To wit:

Even though the original intention was that any educational institution anywhere in the world could be registered under the EDU domain, in practice, it has turned out with few exceptions, only those in the United States have registered under EDU, similarly with COM (for commercial). In other countries, everything is registered under the 2-letter country code, often with some subdivision. For example, in Korea (KR) the second level names are AC for academic community, CO for commercial, GO for government, and RE for research. However, each country may go its own way about organizing its domain, and many have.

Oh, so let's sort it out!

There are no current plans of putting all of the organizational domains EDU, GOV, COM, etc., under US. These name tokens are not used in the US Domain to avoid confusion.

Oh. Oh well.

Currently, only four year colleges and universities are being registered in the EDU domain. All other schools are being registered in the US Domain.

Huh?

RFC 1480 is a very interesting read. It makes passing references to so many facets of DNS history that could easily be their own articles. It also defines a strict, geography-based hierarchy for the .us domain that is a completely different universe from the one in which we now live. For example, we learned above that, in 1993, only four-year institutions were being placed under .edu. What about the community colleges? Well, RFC 1480 has an answer. Central New Mexico Community College would, of course, fall under cnm.cc.nm.us. Well, actually, in 1993 it was called the Technical-Vocational Institute, so it would have been tvi.tec.nm.us. That's right, the RFC describes both "cc" for community colleges and "tec" for technical institutes.

Even more surprising, it describes placing entities under a "locality" such as a city. The examples of localities given are "berkeley.ca.us" and "portland.wa.us", the latter of which betrays an ironic geographical confusion. It then specifies "ci" for city and "co" for county, meaning that the city government of our notional Portland, Washington would be ci.portland.wa.us. Agencies could go under the city government component (the RFC gives the example "Fire-Dept.CI.Los-Angeles.CA.US") while private businesses could be placed directly under the city (e.g. "IBM.Amonk.NY.US"). The examples here reinforce that the idea itself is different from how we use DNS today: The DNS of RFC 1480 is far more hierarchical and far more focused on full names, without abbreviations.

Of course, the concept is not limited to local government. RFC 1480 describes "fed.us" as a suffix for the federal government (the example "dod.fed.us" illustrates that this has not at all happened), and even "General Independent Entities" and "Distributed National Institutes" for those trickier cases.

We can draw a few lessons from how this proposal compares to our modern day. Back in the 1990s, .gov was limited to the federal government. The thinking was that all government agencies would move into .us, where the hierarchical structure made it easier to delegate management of state and locality subtrees. What actually happened was the opposite: the .us thing never really caught on, and a more straightforward and automated management process made .gov available to state and local governments. The tree has effectively been flattened.

That's not to say that none of these hierarchical names saw use. GoDaddy continues to maintain what they call the "usTLD Locality-Based Structure". At the decision of the relevant level of the hierarchy (e.g. a state), locality-based subdomains of .us can either be delegated to the state or municipality to operate, or operated by GoDaddy itself as the "Delegated Manager." The latter arrangement is far more common, and it's going to stay that way: RFC 1480 names are not dead, but they are on life support. GoDaddy's contract allows them to stop onboarding any additional delegated managers, and they have.

Few of these locality-based names found wide use, and there are even fewer left today. Multnomah County Library once used "multnomah.lib.or.us," which I believe was actually the very first "library" domain name registered. It now silently redirects to "multcolib.org", which we could consider a graceful name only in that the spelling of "Multnomah" is probably not intuitive to those not from the region. As far as I can tell, the University of Oregon and OGI (part of OHSU) were keeping very close tabs on the goings-on of academic DNS, as Oregon entities are conspicuously over-represented in the very early days of RFC 1480 names---behind only California, although Georgia Tech and Trent Heim of former Colorado company XOR both registered enough names to give their states a run for the money.

"co.bergen.nj.us" works, but just gets you a redirect notice page to bergencountynj.gov. It's interesting that this name is actually longer than the RFC 1480 name, but I think most people would agree that bergencountynj.gov is easier to remember. Some of that just comes down to habit, we all know ".gov", but some of it is more fundamental. I don't think that people often understand the hierarchical structure of DNS, at least not intuitively, and that makes "deeply hierarchical" (as GoDaddy calls them) names confusing.

Certainly the RFC 1480 names for school districts produced complaints. They were also by far the most widely adopted. You can pick and choose examples of libraries (.lib.[state].us) and municipal governments that have used RFC 1480 names, but school districts are another world: most school districts that existed at the time have a legacy of using RFC 1480 naming. As one of its many interesting asides, RFC 1480 explains why: the practice of putting school districts under [district].k12.[state].us actually predates RFC 1480. Indeed, the RFC seems to have been written in part to formalize the existing practice. The idea of the k12.[state].us hierarchy originated within IANA in consultation with InterNIC (newly created at the time) and the Federal Networking Council, a now-defunct advisory committee of federal agencies that made a number of important early decisions about internet architecture.

RFC 1480 is actually a revision on the slightly older RFC 1386, which instead of saying that schools were already using the k12 domains, says that "there ought to be a consistent scheme for naming them." It then says that the k12 branch has been "introduced" for that purpose. RFC 1386 is mostly silent on topics other than schools, so I think it was written to document the decision made about schools with other details about the use of locality-based domains left sketchy until the more thorough RFC 1480.

The decision to place "k12" under the state rather than under a municipality or county might seem odd, but the RFC gives a reason. It's not unusual for school districts, even those named after a municipality, to cover a larger area than the municipality itself. Albuquerque Public Schools operates schools in the East Mountains; Portland Public Schools operates schools across multiple counties and beyond city limits. Actually the RFC gives exactly that second one as an example:

For example, the Portland school district in Oregon, is in three or four counties. Each of those counties also has non-Portland districts.

I include that quote mostly because I think it's funny that the authors now know what state Portland is in. When you hear "DNS" you think Jon Postel, at least if you're me, but RFC 1480 was written by Postel along with a less familiar name, Ann Westine Cooper. Cooper was a coworker of Postel at USC, and RFC 1480 very matter-of-factly names the duo of Postel and Cooper as the administrator of the .US TLD. That's interesting considering that almost five years later Postel would become involved in a notable conflict with the federal government over control of DNS---one of the events that precipitated today's eccentric model of public-private DNS governance.

There are other corners of the RFC 1480 scheme that were not contemplated in 1993, and have managed to outlive many of the names that were. Consider, for example, our indigenous nations: these are an exception to the normal political hierarchy of the US. The Navajo Nation, for example, exists in a state that is often described as parallel to a state, but isn't really. Native nations are sovereign, but are also subject to federal law by statute, and subject to state law by various combinations of statute, jurisprudence, and bilateral agreement. I didn't really give any detail there and I probably still got something wrong, such is the complicated legal history and present of Native America. So where would a native sovereign government put their website? They don't fall under the traditional realm of .gov, federal government, nor do they fall under a state-based hierarchy. Well, naturally, the Navajo Nation is found at navajo-nsn.gov.

We can follow the "navajo" part but the "nsn" is odd, unless they spelled "nation" wrong and then abbreviated it, which I've always thought is what it looks like on first glance. No, this domain name is very much an artifact of history. When the problem of sovereign nations came to Postel and Cooper, the solution they adopted was a new affinity group, like "fed" and "k12" and "lib": "nsn", standing for Native Sovereign Nation. Despite being a late comer, nsn.us probably has the most enduring use of any part of the RFC 1480 concept. Dozens of pueblos, tribes, bands, and confederations still use it. squamishtribe.nsn.us, muckleshoot.nsn.us, ctsi.nsn.us, sandiapueblo.nsn.us.

Yet others have moved away... in a curiously "partial" fashion. navajo-nsn.gov as we have seen, but an even more interesting puzzler is tataviam-nsn.us. It's only one character away from a "standardized" NSN affinity group locality domain, but it's so far away. As best I can tell, most of these governments initially adopted "nsn.us" names, which cemented the use of "nsn" in a similar way to "state" or "city" as they appear in many .gov domains to this day. Policies on .gov registration may be a factor as well, the policies around acceptable .gov names seem to have gone through a long period of informality and then changed a number of times. Without having researched it too deeply, I have seen bits and pieces that make me think that at various points NTIA has preferred that .gov domains for non-federal agencies have some kind of qualifier to indicate their "level" in the political hierarchy. In any case, it's a very interesting situation because "native sovereign nation" is not otherwise a common term in US government. It's not like lawyers or lawmakers broadly refer to tribal governments as NSNs, the term is pretty much unique to the domain names.

So what ever happened to locality-based names? RFC 1480 names have fallen out of favor to such an extent as to be considered legacy by many of their users. Most Americans are probably not aware of this name hierarchy at all, despite it ostensibly being the unified approach for this country. In short, it failed to take off, and those sectors that had widely adopted it (such as schools) have since moved away. But why?

As usual, there seem to be a few reasons. The first is user-friendliness. This is, of course, a matter of opinion---but anecdotally, many people seem to find deeply hierarchical domain names confusing. This may be a self-fulfilling prophecy, since the perception that multi-part DNS names are user-hostile means that no one uses them which means that no users are familiar with them. Maybe, in a different world, we could have broken out of that loop. I'm not convinced, though. In RFC 1480, Postel and Cooper argue that a deeper hierarchy is valuable because it allows for more entities to have their "obviously correct" names. That does make sense to me, splitting the tree up into more branches means that there is less name contention within each branch. But, well, I think it might be the kind of logic that is intuitive only those who work in computing. For the general public, I think long multi-part names quickly become difficult to remember and difficult to type. When you consider the dollar amounts that private companies have put into dictionary word domain names, it's no surprise that government agencies tend to prefer one-level names with full words and simple abbreviations.

I also think that the technology outpaced the need that RFC 1480 was intended to address. The RFC makes it very clear that Postel and Cooper were concerned about the growing size of the internet, and expected the sheer number of organizations going online to make maintenance of the DNS impractical. They correctly predicted the explosion of hosts, but not the corresponding expansion of the DNS bureaucracy. Between the two versions of the .us RFC, DNS operations were contracted to Network Solutions. This began a winding path that lead to delegation of DNS zones to various private organizations, most of which fully automated registration and delegation and then federated it via a common provisioning protocol. The size of, say, the .com zone really did expand beyond what DNS's designers had originally anticipated... but it pretty much worked out okay. The mechanics of DNS's maturation probably had a specifically negative effect on adoption of .us, since it was often under a different operator from the "major" domain names and not all "registrars" initially had access.

Besides, the federal government never seems to have been all that on board with the concept. RFC 1480 could be viewed as a casualty of the DNS wars, a largely unexplored path on the branch of DNS futures that involved IANA becoming completely independent of the federal government. That didn't happen. Instead, in 2003 .gov registration was formally opened to municipal, state, and tribal governments. It became federal policy to encourage use of .gov for trust reasons (DNSSEC has only furthered this), and .us began to fall by the wayside.

That's not to say that RFC 1480 names have ever gone away. You can still find many of them in use. state.nm.us doesn't have an A record, but governor.state.nm.us and a bunch of other examples under it do. The internet is littered with these locality-based names, many of them hiding out in smaller agencies and legacy systems. Names are hard to get right, and one of the reasons is that they're very hard to get rid of.

When things are bigger, names have to be longer. There is an argument that with only 8-character names, and in each position allow a-z, 0-9, and -, you get 37**8 = 3,512,479,453,921 or 3.5 trillion possible names. It is a great argument, but how many of us want names like "xs4gp-7q". It is like license plate numbers, sure some people get the name they want on a vanity plate, but a lot more people who want something specific on a vanity plate can't get it because someone else got it first. Structure and longer names also let more people get their "obviously right" name.

You look at Reddit these days and see all these usernames that are two random words and four random numbers, and you see that Postel and Cooper were right. Flat namespaces create a problem, names must either be complex or long, and people don't like it either. What I think they got wrong, at a usability level, is that deep hierarchies still create names that are complex and long. It's a kind of complexity that computer scientists are more comfortable with, but that's little reassurance when you're staring down the barrel of "bridger.pps.k12.or.us".

the steorn orbo

We think that we're converting time into energy... that's the engineering principle.

In the 1820s, stacks, ovens, and gasometers rose over the docklands of Dublin. The Hibernian Gas Company, one of several gasworks that would eventually occupy the land around the Grand Canal Docks, heated coal to produce town gas. That gas would soon supply thousands of lights on the streets of Dublin, a quiet revolution in municipal development that paved the way for electrification---both conceptually, as it proved the case for public lighting, and literally, as town gas fired the city's first small power plants.

Ireland's supply of coal became scarce during the Second World War; as part of rationing of the town gas supply most street lights were shut off. Town gas would never make a full recovery. By that time, electricity had proven its case for lighting. Although coal became plentiful after the war, imported from England and transported from the docks to the gasworks by horse teams, even into the 1960s---this form of energy had become obsolete. In the 1980s, the gasworks stoked their brick retorts for the last time. Natural gas had arrived. It was cheaper, cleaner, safer.

The Docklands still echo with the legacy of the town gas industry. The former site of the Hibernian gasworks is now the Bord Gáis Energy Theatre, a performing arts center named for the British-owned energy conglomerate that had once run Ireland's entire gas industry as a state monopoly. Metal poles, the Red Sticks, jut into the sky from the Grand Canal Square. They are an homage to the stacks that once loomed over the industrial docks. Today, those docks have turned over to gastropubs, offices, the European headquarters of Google.

Out on the water, a new form of energy once spun to life. In December of 2009, a man named Sean McCarthy booked the Waterways Ireland Visitors Centre for a press event. In the dockside event space, and around the world through a live stream, he invited the public to witness a demonstration of his life's work. The culmination of three years of hype, controversy, and no small amount of ridicule, this was his opportunity to prove what famed physicist Michio Kaku and his own hand-picked jury of scientists had called a sham.

He had invented a perpetual motion machine.


Sean McCarthy had his start in the energy industry. Graduating with an engineering degree in the 1980s, he took a software position with a company that quickly became part of European energy equipment giant ABB. By some route now lost to history, he made is way into sales. 1994 found McCarthy in Bahrain, managing sales for the Middle East.

McCarthy is clearly a strong salesman. In Bahrain, he takes credit for growing ABB's business by orders of magnitude. Next, ABB sent him to Houston, where he worked sales to Central and South America. In 1999, the company called him to Azerbaijan---but he chose otherwise, leaving ABB and returning to Dublin.

Contacts from the energy industry, and no doubt his start in software, brought Graham to the world of the internet. 1999 in Dublin was, like so many places, crowded with dotcom boom startups. McCarthy saw opportunity: not necessarily by joining the fray, but by selling expertise and experience to those who already had. With his brother-in-law, a lawyer, he formed Steorn.

The nature of Steorn's business during these early years is a bit hazy. As McCarthy recounts it, to Barry Whyte for the book The Impossible Dream, Steorn gathered some early support from the Dublin internet industry that connected him with both capital and clients. One of their most notable clients, attested by several sources, was dotcom wonder WorldOfFruit.com. The World of Fruit, a subsidiary of fruit importer Fyffes, was a web-based auction platform for wholesale import and export of fruit. I have not been able to clarify how exactly Steorn was involved in World Of Fruit, although it cannot be ignored that the aforementioned brother-in-law and cofounder, Francis Hackett, had been the general counsel of World Of Fruit before becoming involved in Steorn. As for the future of online fruit trading, it went about as well as most dotcom-era market platforms did, stumbling on for a few years before folding with millions of Euro lost.

Later, by McCarthy's telling, Steorn shifted its business more towards R&D. They worked on a project for a client fighting credit card fraud, which led to later work for Irish and British law enforcement organizations, mostly related to video surveillance and machine vision. For Ireland's DART commuter train, for example, Steorn attempted a system that would automatically detect aggressive behavior in platform surveillance video---though it was canceled before completion. These projects floated Steorn through the worst of the dotcom bust and supported an in-house R&D operation that trialed a variety of ideas, including something like an early version of California's modern electronic license plates.

I am not sure who well this account squares with Steorn's public presence. Steorn was incorporated in July of 2000, and when their website appeared in 2001 it advertised "programme management and technical assessment advice for European companies engaging in e-commerce projects." The main emphasis of the website was in the project management experience of McCarthy and the company's CTO, Sean Menzies, formerly of recruiting and staffing firm Zartis. The primary claim was that Steorn could oversee projects to develop and launch e-commerce websites, reducing the risk and expense of jumping into the world wide web.

Steorn has been formed by individuals with significant experience in this area who have recognised that there is an absence in the market of genuinely impartial and objective advice as to the best way of implementing e-commerce ideas.

A few months later, Steorn's business became a little more general: "Steorn is an expert in the field of technology risk management." In August of 2001, Steorn launched its first concrete product offering, one that is curiously absent from McCarthy's later stories of the company. Not only would Steorn supervise technology projects, it had partnered with an insurance company to underwrite surety bonds on the completion of those projects.

Steorn had a proprietary program management methodology which they called GUIDE. Frustratingly, although GUIDE is always styled in all caps, I have not found any indication that it stood for something. What I do know is that GUIDE "offers predictability, visibility, control and results," and that it has some relation to a project planning methodology called DIRECT.

In all truth, while Steorn's 2001 websites are laden with bespoke buzzwords, the project and risk management services they offer seem unremarkable in a good way. Probabalistic project scheduling models, checkpoints and interim deliverables, "an approach which is systematic yet sufficiently flexible." The only way that Steorn's services seem foolish is the extent to which these methods have been adopted by every technology project---or have become so dated or unfashionable as to appear obsolete.

This was 2001, though, and many established companies found themselves plunging into the deep end of the online economy for the first time. No doubt the guidance of an outside expert with experience in online projects could be valuable. But, then, I am a sympathetic audience: they were hiring technical architects. That's my day job! The benefits package was, they said, best in class. Alas, the conditions that may necessitate my own move to Dublin come more than twenty years too late.

Despite McCarthy's description of their multiple projects in security, surveillance, and machine vision, Steorn never seems to have advertised or publicized any work in those areas. In 2003, their website underwent a redesign that added "services for technology inventors" and advertised a subsidiary web hosting provider called HostsMe. Perhaps my view is too colored by the late-2000s context in which I came to understand the internet industry, but pivoting to the low barrier of entry, low chance of success shared hosting market feels like a move of desperation. In 2004, the website changed again, becoming a single page of broken links with a promise that it is "being redesigned."

For this entire period, from late 2001 after the announcement of their risk underwriting service to the company's complete swerve into energy from nothing, I can find no press mentions of Steorn or any of their projects. McCarthy seems to suggest that Steorn had a staff of engineers and technicians, but I struggle to find any names or details. Well, that was a long time ago, and of course the staff are not all listed on LinkedIn. On the other hand, one of McCarthy's investors, shown a laboratory full of white-coated scientists, later came to suspect they were temps hired only to create an appearance.

This is not to accuse McCarthy of lying. Some details definitely check out. For example, their project with Fraudhalt to detect tampering with ATMs via machine vision is the subject of a patent listing Sean McCarthy's name. But I do think that McCarthy is prone to exaggerating his successes and, in the process, muddling the details. When it comes to Steorn's most famous invention, McCarthy and early employee and project manager Michael Daly agree that it was an accidental discovery. Both tell detailed stories of a project for which Steorn purchased and modified small wind turbines. The trouble is that they are not describing the same project. Two different customers, two different contexts, two different objectives; but both lead to the Orbo.

Other attempts at checking into the particulars of Steorn's early business lead in more whimsical directions. To Whyte, McCarthy described a sort of joke project by Steorn to create an "online excuse exchange" called WhatsMyExcuse.com. I can't find any evidence of this project, although there's no particular reason I would. What I can say is that some ideas are evergreen: four months ago, a Reddit user posted that they "built a tiny excuse generator for fun." It's live right now, at whatsmyexcuse.com.

In any case, the nature of Steorn around 2003 remains mysterious to me. McCarthy's claims to have been a regional expert on credit card fraud are hard to square with the complete lack of any mention of this line of business on Steorn's website. On the other hand, Steorn had a confusing but clearly close relationship with Fraudhalt to such an extent that I think Steorn may have gone largely dormant as McCarthy spent his time with the other company.


After five years of near silence, a period that McCarthy characterized as modest success, Steorn captured the world's attention with what must be one of the boldest pivots of all time. An August 2006 full-page ad in the Economist began with a George Bernard Shaw quote, "all great truths begin as blasphemies." "Imagine," it reads below, "a world with an infinite supply of pure energy. Never having to recharge your phone. Never having to refuel your car."

Steorn's website underwent a complete redesign, now promoting the free energy device they dubbed the Orbo. A too-credulous press went wild with the story, especially in Ireland where Steorn's sudden prominence fit a narrative of a new Irish technology renaissance. The Irish Independent ran a photo of McCarthy kneeling on the floor beside a contraption of aluminum extrusion and wheels and wiring. "Irish firm claims to have solved energy crisis" is the headline. The photo caption reminds us that it "has been treated with some skepticism in the scientific community because it appears to contravene the laws of physics."

I will not devote much time to a detailed history of the development of the Orbo; that topic is much better addressed by Barry Whyte's book, which benefits greatly from interviews with McCarthy. The CliffsNotes 1 go something like this: Steorn was working on some surveillance project and wanted to power the cameras with small wind turbines. The exact motivations here are confusing since, whichever version of the story you go with, the cameras were being installed in locations that already had readily available electrical service. It may have been simple greenwashing. In any case, the project lead Steorn to experimenting with modifications to off-the-shelf compact wind turbines.

At some point, presumably one or two years before 2006, it was discovered that a modified turbine seemed to produce more energy in output than it received as input. I have not been able to find enough detail on these experiments to understand what exactly was being done, so I am limited to this vague description. Steorn at first put it out of mind, but McCarthy later returned to this mystery and built a series of prototypes that further demonstrated the effect. A series of physicists and research teams were brought in to evaluate the device. This part of the story is complicated and details conflict, but an opinionated summary is that entrepreneurially-inclined technology development executives tended to find the Orbo promising while academic physicists found the whole thing so laughable that they declined to look at it. Trinity College contained both, and in 2005 McCarthy's discussions with a technology commercialization team there lead to a confrontation with physicist Michael Coey. From that point on, McCarthy viewed the scientific establishment as resistance to be overcome.

McCarthy determined that the only way to bring Orbo to the world was to definitively prove that it worked, and to do so largely without outside experience. For that, he would need equipment and a team. After long discussions with a venture capital firm, McCarthy seems to have balked at the amount of due diligence involved in such formal investment channels. Instead, he brought on a haphazard series of individuals, mostly farmers, who collectively put up about 20 million euro.


One of the curious things about the Orbo was Steorn's emphasis on cell phone batteries. I suppose it is understandable, in that ubiquitous mobile phones were new at the time and the hassle of keeping them charged was suddenly a familiar problem to all. It also feels like a remarkable failure of ambition. Having invented a source of unlimited free energy, most of what McCarthy had to say was that you wouldn't have to charge your phone any more. Sure, investment documents around Steorn laid out a timeline to portable generators, vehicles, standby generators for telecom applications... a practical set of concerns, sure, but one that is rather underwhelming from the creator of the first over-unity energy device.

It is also, when you think about it, a curious choice. McCarthy later talked about pitching the device to Apple's battery suppliers. The timeline here is a bit confusing, given that the iPhone launched after the period I think he was discussing, but still, it makes a good example. The first-generation iPhone was about a half inch thick, hefty by today's standards but a very far cry from Steorn's few publicized prototypes, which occupied perhaps two feet square. Miniaturization is a challenge for most technologies, with the result that pocket-portable devices are usually a late-stage application, not an early one. That's especially true for an electromechanical generator, one that involves bearings and electromagnets and a spinning rotor. Still, it was all about phones.

Steorn's trouble getting the attention of battery manufacturers lead them to seek independent validation. Turned away, as McCarthy tells it, by university physics departments and industrial laboratories, Steorn came up with an unconventional plan. In the 2006 Economist advertisement, they announced "The Challenge." Steorn was recruiting scientists and engineers to form a panel of independent jurors who would evaluate the technology. You could apply at steorn.com. Nearly 500 people did.

The steorn.com of the Orbo era is a fascinating artifact. It has a daily poll in a sidebar, just for some interactive fun. You could apply for the scientific jury, or just sign up to express your interest in the technology. Most intriguingly, though, there was a forum.

This is some intense mid-2000s internet. There was a time when it just made sense for every website to have a forum; I guess the collective internet industry hadn't yet learned that user-generated content is, on average, about as good as toxic waste, and that giving the general public the ability to publish their opinions under your brand is rarely a good business decision. Still, Steorn did, and what few threads of the steorn.com discussion forum survive in the Internet Archive are a journey through a bygone era. Users have the obvious arguments: is Orbo real, or a scam? They have the less obvious arguments, over each other's writing styles. There is a thread for testing BBCode. A few overactive users appear seemingly everywhere, always hostile and dismissive. An occasional thread will become suddenly, shockingly misogynistic. This is, of course, the soup we all grew up in.

The forum exemplifies one of the things that I find most strange about Steorn. The company was often obsessively secret, apparently working on the Orbo technology in quiet for years, but it was also incredibly bold, choosing a full-page ad for a publicity stunt as their first public announcement. McCarthy was well aware that the scientific establishment viewed him as a fraud, by this time individual investors seemed to be growing skeptical, and yet they had a discussion forum.


The late years of Steorn are best summarized as a period of humiliation. In 2007, the company promised a public demonstration of the Orbo at a science museum in London. I remember following this news in real time: the announcement of the demonstration triggered a new round of press coverage, and then it was canceled before it even began. Steorn claimed that the lighting in the museum had overheated the device and damaged it. The public mostly laughed.

In 2009, Steorn arranged a second public demonstration, the one at the Waterways Visitors Centre in Dublin. This one actually happened, but the device clearly incorporated a battery and Steorn didn't allow any meaningful level of inspection. The demonstration proved nothing but that Steorn could use a battery to make a wheel spin, hardly a world-changing innovation. Not even a way to charge a phone.

Later that same year, The Challenge came to its close. The independent jury of experts, after meetings, demonstrations, experiments, and investigations, returned its verdict. The news was not good. Jurors unanimously approved a statement that Steorn had not shown any evidence of energy production.

When something is as overhyped and underdeveloped as the Orbo, when a company claims the impossible and still attracts millions, you sort of expect a dramatic failure. Surely, you cannot just claim to have created a source of free energy and then wander on through life as usual. Or, perhaps you can. In 2010, steorn.com gained the Developer Community, where you could pay a licensing fee for access to information on the Steorn technology.

Even the mobile phones were not abandoned. In 2015, Steorn mounted another demonstration, of a sort. A small Orbo device, now apparently solid-state, was displayed supposedly charging a phone in a Dublin pub. A lot of big claims are made and tested in pubs, but few with so much financial backing. Steorn had been reduced to a story told over beer and the internet: at the end of 2015, McCarthy announced via Facebook that an Orbo phone and phone charger would soon go on sale. The charger, the Orbo Cube, ran €1,200 and was due to ship within the month.

As far as I can tell, these devices were actually completely unrelated to the original Orbo. The "solid-state Orbo" was completely novel design, to the extent that you would call it a design. Some units did ship, and reverse engineering efforts revealed an eccentric set of components including two alkaline 9v batteries and shrinkwrapped mystery cells that have the appearance of capacitors but are, supposedly, proprietary components relying on some kind of magnetic effect. It's hard to find much solid information about these devices, I think very few were made and they shipped mostly to "friendly" customers. It is said that many of them failed to work at all, the result of an acknowledged quality control problem with the clearly handmade interior. There is a lot of electrical tape involved. Like a lot. And some potting compound, but mostly electrical tape. The two I've found detailed accounts of failed to charge a phone even when new out of the box.

Through this whole period, Steorn was still McCarthy's day job, at least on paper. But he had turned towards a new way to make money: poker. Starting around 2015, McCarthy became a professional poker player, apparently his main form of income to this day. He announced that Steorn would be liquidated in 2017; it was finally dissolved in 2024.


History is replete with perpetual motion machines. There are rumors of some sort of machine in the 8th century, although questions about the historicity of these accounts lead to difficult questions around what makes a perpetual motion machine "real." Of the many putative free energy sources of history, many have involved some configuration of magnets. I can understand why: magnets offer just the right combination of the mysterious and the relatable. They seem to work as if by magic, and yet, they are physical objects that we can hold and manipulate. You can see how someone might develop an intuitive understanding of magnetic behavior, extend it to an impossible idea, and remain quite convinced of their discovery.

If perpetual motion machines are a dime a dozen, why does the story of Steorn so fascinate me? I think that it has many parallels to our situation today. The Orbo is not merely the product of a deluded inventor, it's the product of a period of Irish economic history. Whyte's book is focused primarily on this aspect of the story. He articulates how the overstuffed, overhyped Irish economy of the 2000s, a phenomenon known as the Celtic Tiger, created an environment in which just about any claim could raise millions---no matter how outlandish.

And here we are today, with some of the largest components of our economy run by people who fashion themselves as technical experts but have backgrounds mostly in business. They, too, are collecting money on the premise that they may build a world-changing technology---AGI.

Sure, this analogy is imperfect in many ways. But I think it is instructive. Sean McCarthy wrote of Steorn's early days:

It was just a gold rush: if you weren't doing web, you didn't exist. So, a banana import-export business needs to invest millions in e-commerce. They probably did need to invest, but it was driven, in my opinion, by all of these companies driving their share price up by doing something on the web rather than by any real business cases.

The dotcom boom and bust was an embarrassment to the computer industry, and it was also a pivotal moment in its growth. E-commerce, the graveyard of many '90s businesses, is one of the pillars of modern life. The good days of easy money can be an incubator for important ideas. They can also be an incubator for idiocy.

Whenever I read about some historic scam, I always wonder about the people at the top. Was Sean McCarthy a fraud, or was he deluded? Did he play up the Orbo to keep the money coming in, or did he really believe that he just needed one more demonstration?

Whyte's book takes a strong position for the latter, that McCarthy had the best intentions and got in over his head. He still believes in Orbo, to this day. Of course, Whyte is far more sympathetic than I find myself. The book is almost half written by McCarthy himself in the form of its lengthy excerpts from his statements. I am more cynical, I think that McCarthy must have known things were going sideways well before he shipped out broken power banks with nine volt batteries covered in electrical tape. I don't think he started out that way, though.

I think he started out with the best intentions: the leader of a moderately successful tech company, one that wasn't getting much business but had ready access to capital. He went fishing for ideas, for opportunities for growth, and he hooked the biggest of them all. A world-changing idea, one so profound that he himself struggled to understand its implications. An impossible dream of not having to charge his phone. To sustain the company, he had to bring in money. To bring in money, he had to double down. Perpetual motion was always just around the corner.

I believe that McCarthy is a con man, in that he has too much confidence. It's a trait that makes for a good poker player. McCarthy saw something remarkable, perhaps an error in measurement or an error in understanding. He placed a bet, and spent the next decade pushing more money into the pot, trying to make it work out. I would hope that our industry has learned to be more cautious, but then, we're only human. We watched the wheels spin, faster and faster, and we convince ourselves that they can spin forever.

  1. CliffsNotes was a primitive technology for summarizing large texts, which relied mostly on humans to do the summarization. Since summarizing texts has since become the primary focus of the US economy, the process has become more automated but does not necessarily produce better results.

The Ascent to Sandia Crest II

Where we left off, Albuquerque's boosters, together with the Forest Service, had completed construction of the Ellis Ranch Loop and a spur to the Sandia Crest. It was possible, even easy, to drive from Albuquerque east through Tijeras Pass, north to the present-day location of Sandia Park, and through the mountains to Placitas before reaching Bernalillo to return by the highway. The road provided access to the Ellis Ranch summer resort, now operated by the Cooper family and the First Presbyterian Church, and to the crest itself.

The road situation would remain much the same for decades to come, although not due to a lack of investment. One of the road-building trends of the 1920s and 1930s was the general maturation of the United States' formidable highway construction program. The Federal Aid Highway Act of 1921 established the pattern that much of western road building would follow: the federal government would split costs 50:50 to help western states build highways. This funding would bring about many of the US highways that we use today.

A share of the money, called the forest highway fund, was specifically set aside for highways that were in national forests or connected national forests to existing state highways. By 1926, the Federal Lands Transportation Program had taken its first form, a set of partnerships between the Bureau of Public Roads (later the Federal Highway Administration) and federal land management agencies to develop roads for economic and recreational use of federal land. For the Forest Service of the era, a forest without road access was of limited use. The following years saw a systematic survey of the national forests of New Mexico with an eye towards construction.

The Federal Aid Highway Act presaged the later interstate freeway program in several ways. First, the state-federal cost sharing model would become the norm for new highways and drive the politics of road construction to this day. Second, despite the nominal responsibility of the states for highways, the Act established the pattern of the federal government determining a map of "desirable" or "meritorious" road routes that states would be expected to follow. And finally, the Act enshrined the relationship between the military and road building. The first notional map of an integrated US highway system, developed mostly by the Army for the Bureau of Public Roads, was presented to Congress by esteemed General of the Armies John Pershing. This plan, the Pershing Map, is now over 100 years old but still resembles our contemporary freeway system.

The Great Depression did not generally drive construction, but roads would prove an exception. Roosevelt's New Deal, and the dryer language of the Emergency Relief Acts of the early 1930s, provided substantial federal funding for construction and improvement of highways. The impact on New Mexico was considerable. "Surveys, plans, and estimates of Navajo Canyon Section of State Highway No. 2 within the Carson National Forest, south of Canjilon, and survey, plans, and estimates on State Highway No. 12 between Reserve and Apache Creek on the Datil-Reserve Forest Highway route." The lists of highway projects in newspapers become difficult to track. The forest highway program's budget doubled, and doubled again. "The building of new Forest Highways in the National Forests of New Mexico will be rushed as rapidly as possible... to open up the National Forests to greater use and protection from fires."

The papers were optimistic. The Carlsbad Current-Argus ran a list of Forest Highway efforts in 1932; the headline read simply "Forest Service To Solve Unemployment."


The depression-era building campaign left the Ellis Ranch loop on the same route, but saw much of the route improved to a far higher standard than before. Graveling, oiling, and even paving stabilized the road surface while CCC crews built out embankments on the road's mountainside segments. By 1940, much of the loop had been incorporated into New Mexico Highway 44---a continuous route from Cedar Crest to Aztec.

Former NM 44 exemplifies changes to the loop's northern segment. Much of NM 44 is now known as NM 550, a busy highway from Bernalillo and alongside the far reaches of Los Lunas that eventually becomes one of the major routes in the state's northwest corner. The connection to Cedar Crest, though, is now difficult to fathom. The former route of NM 44 continued east of the freeway through Placitas and along NM 165, over the mountains to become NM 536 at Balsam Glade and then NM 14 at San Antonito. NM 14 is a major route as well, and NM 536 is at least paved---but as discussed in part I, NM 165 is now a rough unpaved road appealing mostly to OHV users and impassable in winter 1.

Sections of NM 165 do seem to have been paved, mostly closer to Placitas, but maintenance as a main highway had ended at least by 1988 when the NM 44 designation was retired.

This is the major reason for my fascination with the Ellis Ranch Loop: today, it is hardly a loop---the whole loop is still there, and you can drive it at least in summer, but the segment of the highway in the north part of the Sandias, what is now 165, doesn't feel at all like a continuation of the same loop as NM 536. On the ground, when driving up 536, it's clear that the "main" route at Balsam Glade is to follow the curve west onto Sandia Crest Scenic Highway. And yet, the Scenic Highway was only a spur of the original loop.

This oddity of history is still reflected in the road designations and, as a result, modern maps. NM 536 and NM 165 are state highways. Sandia Crest Scenic Highway, despite the name, is not. As you zoom out, Google Maps hides the Scenic Highway entirely, depicting the former Ellis Ranch Loop as the main route. This is very different from what you will find if you actually drive it.

This halfway state of change, with NM 165 still designated as a highway but effectively abandoned and the more major Scenic Highway not designated, partly reflects the bureaucratic details of the 1930s. NM 165 was considered part of a proper inter-city route; the Scenic Highway deadends at the crest and thus never really "went anywhere." But there is more to the story: it also reflects the ambitions, errors, and shifting priorities of the mid-century.


The first formal ski area in the Sandia Mountains opened in 1936, developed by Robert Nordhaus's Albuquerque Ski Club. The ski area, then called La Madera for the canyon that tops out at the ski slopes, became a center of southwestern wintersports. In the following decades, the ski area would displace Ellis Ranch as the main destination in the mountains.

World War II disrupted skiing and recreation more broadly; progress on Sandia Mountain development stalled into the 1940s. The decline wouldn't last: WWII raised the possibility of ground combat in Northern Europe or even a need to defend the inland United States against invasion. Military planners considered Denver the best choice for an emergency relocation of the national capital in response to an Atlantic assault. While never a formal policy, the concept of Denver as "national redoubt" survived into the Cold War and directed military attention to the Rocky Mountains.

From 1939 into the 1940s, the National Ski Patrol lobbied for development of ski-mounted military units to the extent that it became a de facto auxiliary of the US Army. Ultimately only one Army "Mountain Division" would be established, but its members---recruited and trained by the Ski Patrol itself---went on to an instrumental role in securing the surrender of German forces in Austria.

While the Alpine Light Infantry were never a large part of the Army, they brought public attention to skiing and mountain recreation. Military support enabled expansion of the Ski Patrol's training programs, and returning soldiers had stories of skiing in the Alps. Much like aviation, skiing surged after the war, elevated from a fairly obscure extreme sport to one of the country's best known forms of recreation.

In 1946, the La Madera Ski Area installed what was then the longest T-bar lift in the country. It built by Ernst Constam, a Swiss engineer who invented the modern ski lift and became trapped in the United States when the war broke out during his business trip. Constam found himself an American by accident, but developed a fierce allegiance to his new home and soon worked for the Army. He participated in the training of the Mountain Division, contributed his understanding of European alpine combat to military intelligence, and even built the first military ski lift: a T-bar at Camp Hale in Colorado, now called Ski Cooper.

During the 1950s, the ski area underwent ownership changes and some financial difficulty. Nordhaus operated the area profitably, but needed cash to build the infrastructure for continued growth. That cash came in the form of Albuquerque businessman and real estate developer Ben Abruzzo, who is remembered first as a famous balloonist (responsible for much of Albuquerque's prominence in hot air ballooning) but was also an avid skier. Nordhaus and Abruzzo formed the Sandia Peak Ski Company and construction was soon underway.

You could access the base of the ski area from the Ellis Ranch Loop, but it was slow going, especially in winter. The trip from Albuquerque reportedly took hours. Ski area expansion required easier access. After negotiations related to the ski area permit expansion, the Forest Service paved the entirety of Ellis Loop from the base of the mountains to the ski area. This was the first time that a large section of the road in the mountains was fully paved, and it appears to be the Forest Service's decision to pave from the ski area southeast to San Antonito, rather than north to Placitas, that sealed the fate of NM 165. From that point on, the "main" way from Albuquerque to the Mountains was via Tijeras, not via Placitas.


The late 1950s were a busy time in the Sandias. The ski area was not the only growing tourism attraction; a post-war resurgence in travel and automobilism renewed the pro-tourism boosterism of the 1920s. Albuquerque business interests were back to lobbying the Forest Service for tourist facilities, and Abruzzo and Nordhaus's efforts at the ski area inspired confidence in a bright future for mountain recreation. All that was needed were better roads.

Sometime during the 1950s, the names "Ellis Ranch Loop" or "Ellis Loop" gave way to "Sandia Loop." Despite the new name, the road kept to its old history: the state and the Forest Service announced their intent to complete a project that, as far as they were concerned, had been in intermittent progress since the 1930s: paving and widening the entire route as a two-lane highway.

The project had never been completed in part because of the difficult conditions, but there were politics at play as well. Federal funding, critical to completing highway projects, was allocated based on the recommendations of committees at each level of government. The development of "secondary highways," a category that included NM 44, fell to county committees for prioritization. Despite the emphasis that Bernalillo County put on the Sandia Loop, Sandoval County was less motivated. Another nail in NM-165's coffin came in 1958 when the Sandoval County Road Committee definitively stated that their budget was quite limited and the Sandia Loop was not a priority. As far as I can tell, serious efforts to pave the existing road between Balsam Glade and Placitas ended at that meeting. A few years later, the state highway department announced that the project had been canceled.

Despite the fate of the loop's northern half, the rest of the Sandia mountain roads enjoyed constant attention. The southern portion of the loop was widened (by four feet, three inches) with new road base laid. The road to the crest itself, despite the involvement of the state highway department in the original construction, was decisively the problem of the Forest Service---at least, this was the point repeatedly made by the state highway department as Albuquerque interests campaigned for its improvement. Still, boosters did not have to wait for long. In 1958 the Forest Service let contracts for improvement of the Sandia Crest road as an "anti-recession project."

While many reports describe the late-1950s Sandia Crest project as widening and resurfacing, some changes to the alignment were made as well. The most severe of its many switchbacks, at the Nine Mile Picnic Area, was reworked for fewer curves. The original alignment is still in use for access to the picnic area.

In the mean time, crews contracted by the Forest Service completed yet another round of improvements to the north section of the loop road. This project spanned from a "preliminary engineering survey" in 1958 to the point where the gravel base was largely complete in 1959, but while paving of the entire route was planned it does not seem to have happened. The Albuquerque business community felt that there was good reason for the paving project: the Albuquerque Tribune reported in 1958 that "upwards of 100,000 cars annually use the Loop Road in making the punishing trip up to La Madera ski run and the Crest." At the same time, though, Albuquerque's once unified push for mountain access started to face political headwinds.


The 1960s were a different world, but some things stayed the same: Bob Cooper advertised cabins for lease at Ellis Ranch, and the business community wanted to see an improved, paved loop highway around the Sandias. The original completion of the Ellis Ranch Loop appeared in the Albuquerque Tribune's "30 Years Ago" feature in 1960, alongside coverage of the growing debate over placement of the city's two upcoming interstate freeways.

Despite efforts by the Chamber of Commerce, Highland Business Men's Association, and Downtown Business Men's Association to restart the Sandia Loop project, the state Highway Department remained silent on the subject. This was the peak era of freeway construction, and the department had many construction problems on the I-25 corridor especially. The Sandia Loop remained a popular facility, though, recommended by the Mobil travel guide starting in 1962. That same year, the La Madera ski area, now called Sandia Peak, opened the "Summit House" restaurant at the top. Summit House would later be known as High Finance before closing for replacement by today's Ten 3.

This might be confusing to today's Burqueños, as the main attraction at Sandia Peak would not begin construction until 1964. The "Spectacular Sandia Peak Chairlift Ride" described in 1962 newspaper advertisements is the ski area's Lift #1, a chairlift that is no longer in service and slated for replacement with a mixed chair/gondola system in coming years. And yet, the restaurant and lift were operated by the "Sandia Peak Ski and Aerial Tram Co." This somewhat contradicts the "official" history of the Sandia Peak Tramway, but one would think that Nordhaus and Abruzzo must have already had a bigger vision for access to the mountaintop.

The full story of the Sandia Peak Tramway could occupy its own article, and perhaps it will, but the short version is this: Bob Nordhaus visited Switzerland, where he learned that ski areas were being made more accessible by means of gondola lifts. He thought that the same scheme would work in Albuquerque, and together with Abruzzo, mounted a long campaign for permitting and funding. The plan was ambitious and more than just a bit quixotic, but Nordhaus and Abruzzo were both big personalities with extensive business connections. In 1966, they opened what was then, and for many decades after, the longest aerial tramway in the world. It runs from the northeast fringe of Albuquerque directly up the western slope of the Sandias, delivering passengers to the very top of the ski area. The 3,819 foot climb, over 2.7 miles and just two towers, takes about fifteen minutes. It is far faster, of course, than the drive around and up the east side. And yet, in the fashion of Albuquerque, one of its biggest impacts on our modern city is a road: Tramway Boulevard, or NM 556, roughly one quarter of what was originally planned as a beltway highway around the city.

While the tramway opened the steep western slope to tourism, attention to the east side was reinvigorated. The Albuquerque business community had continued to lobby the Forest Service for new and better roads, and as the tramway went in the road boosters found success. The Forest Service circulated a draft plan that delighted some and terrified others: a brand new loop¸ the Crest Loop, that would replace the road from Balsam Glade to the crest and extend it straight up the very spine of the mountain, allowing views down both sides, until coming down the northern ridge to meet the existing Sandia Loop Road near Placitas. It would be a first-class skyline road, 24' wide pavement over almost exactly the route of today's crest trail.


Previous efforts at radical Sandia mountain road construction were held off by weather, funding, and war. For the Crest Loop, there was a new enemy: bighorn sheep.

Rocky Mountain Bighorn Sheep don't appear to have ever been common in New Mexico, but a small population around the Rio Grande Valley was completely wiped out by human impacts in the late 19th and early 20th centuries. Efforts to reintroduce bighorns started in the 1930s, with more translocated sheep in the 1940s and 1960s. These populations struggled severely, and despite the attempts there are no bighorn sheep in the Sandia Mountains today. Still, the population in the mid-'60s was likely the largest it would ever be, a hard-won victory that the state was inclined to protect.

Besides, Rachel Caron's Silent Spring was published in 1962, a landmark in the history of American environmentalism. The environmentalist movement wasn't as powerful as it would become, but it was certainly gaining momentum. Not only the state Game and Fish Department opposed, but community groups like the New Mexico Mountain Club stepped up to oppose development along the ridge.

On Monday, August 8th, 1966, the Forest Service called a public hearing to be held at the East Central Branch of Albuquerque National Bank. To an audience of businessmen and community groups, regional forester William D. Hurst presented the official plan for the Sandias. The existing road from Balsam Glade up to the crest, the Crest Scenic Highway, would be significantly reworked. The old alignment had over a dozen switchbacks, the new alignment only four. The wider, straighter road would offer access to a completely new ski area at about the location of today's Capulin snow play area.

The Crest Loop would not make it quite all the way to the crest, instead turning north a bit to the east (and below) the crest where a parking lot and turnout area formed the main attraction. The highway itself no longer followed the ridge, an admission to the Bighorn Sheep interests, but instead paralleled it about 1/4 mile to the east. About 2.5 miles north of the crest, the highway would switch back and start it a descent down the eastern face, through Las Huertas canyon, to meet the Sandia Loop before Placitas.

It was noted that, in the future, an additional road could be planned up the canyon above Sandia Cave for yet a third road all the way up the eastern side. At least you can't say they weren't ambitious.

The 1966 Crest Loop plan was presented by the Forest Service in a rather matter-of-fact way. Hurst told the press that he did not expect much objection, and apparently the presentation itself was well received. Still, Hurst's quip that the plan presented "the greatest good for the largest numbers of people" hinted that it was not uncontroversial, as did the fact that every newspaper article about the presentation made some reference to the "hubbub."

While the 1/4 mile shift away from the ridge had won the support of the state, it was not enough to please the New Mexico Wildlife and Conservation Association and gained only the most tacit support from the New Mexico Mountain Club, which described their position to the papers as "wait and see." They had time: the Forest Service expected the project to take as much as ten years, even with surveying work starting immediately.

The Crest Loop was only part of a broader plan for Sandia recreation, one that responded to Albuquerque's rapid mid-century population growth. There were more than twice as many people in Albuquerque then as there had been before the war, for example, and the Forest Service considered the nine picnic areas on the existing road to be overused. The new crest loop would enable 200 acres of new picnic grounds

For all of the promise of the new Crest Loop, there was a curious omission: the view from the crest. The road's alignment 1/4 mile off the crest meant that the highest point on the road, the turnout with bathrooms and facilities, was on the other side of the ridge from Albuquerque. It would have views to the north and northeast, but not at all to the west or southwest. In other words, you would no longer be able to drive up to the crest and then look down on the city---getting the same view offered by the old road would require a hike.

This is a fascinating decision and one that, if you will allow me to editorialize, seems to reflect the Forest Service's process-focused approach to planning. The original Crest Loop plan would have placed the road directly on the ridge, probably providing one of the finest roadside views in the nation. Well, in actuality, there was always disagreement over the aesthetic merits of the plan. The forest service had always denied any intent to clearcut trees to improve sightlines, so it was likely that even the original ridgetop alignment would have had trees blocking the view along much of the route. The Forest Service even conceded to environmental groups by promising to keep the road off of the large clear bluff just north of the crest, the spot with the very best potential for automobile sightseeing.

The final proposed alignment, downslope from the ridge, had the view completely blocked on the west... and it was away from the windswept, rocky western face, decidedly in the forest. Even to the east, where the terrain dropped away precipitously, you would be unlikely to see anything other than the forest right in front of you.

Proponents said the Crest Loop would be a natural attraction like no other, with a 360 degree perspective over the Rio Grande Valley and the High Plains. Opponents said it would be a "green tunnel," so densely forested on both sides that it might as well have been down in the valley for all the sightseeing it would afford. They seem to have been right: as ultimately proposed, the "skyline" Crest Loop actually offered less of a view than the existing route.

What it would offer is capacity. The road was designed for traffic, intended to support visitors to the new picnic areas, the expanding Sandia Peak Ski Area, and the new ski area yet to be named. The new ski area penciled out as 1,000 acres, larger than Sandia Peak. A new snow play area was expected to attract thousands of non-skiing visitors. And parking---parking was a problem, the Forest Service reported, with just 750 spaces at the bottom of the ski area. The Forest Service intended to provide 800 spaces at the new snow play area, and the bypassed switchbacks of the old crest road would themselves be repurposed as new parking areas.


As far as the Forest Service was concerned, the Crest Loop was a done deal. They had already committed $600,000 a year for the next several years to fund surveys and planning, and expected more federal funding to match state support. And yet, as always, progress was slow. 1967 passed with little mention of the road. In 1968 the Albuquerque Chamber of Commerce must have been getting nervous, as they passed a resolution in support of its completion, for no reason in particular.

In 1969, an interesting new proposal for the Crest Scenic Highway put it back into the papers: a race. Albuquerque advertising firm Eicher and Associates proposed to one-up the Pikes Peak by hosting a Labor Day hill climb. The state Highway Commission approved the plan, but the race stalled when a State Police captain pointed out that state law forbid racing on public roads---and the Attorney General issued an opinion that the Highway Commission had no authority to grant an exception. Nothing ultimately came of the Sandia Peak Climb, or at least, nothing other than a series of headlines and another round of objections to planned improvements.

Central to the opponent's arguments was a disagreement over the nature of forest recreation. The New Mexico Wildlife and Conservation Association advocated for the establishment of wilderness areas covering much of the Sandias, precluding further development. Their proposal actually accommodated the Crest Loop by putting it just off of the edge of the wilderness area covering the western face, but Cibola National Forest Supervisor George Proctor was still hesitant: "Proctor made clear he did not rule out wilderness use for the land, but only wanted time to weigh the proposals against possible other needs---as for future additional tramways or recreation areas." Perhaps one tramway and two highways was not enough; we can't know what the future will hold.

The conservationists said exactly what you would expect:

It would be nice to have complete access for large volumes of people---since so many have seen nothing like the top of the Sandias---said [NMCWA director] McDonald, but when large volumes come, then the beauty of the area often ceases to exist. 2

While the Forest Service had been quiet, it had not been entirely idle. Surveying for the route was nearly complete, and the Forest Service contracted the first phase of the work: clearing trees for the new route, starting in the immediate area of the crest. A bit over three miles of right of way were clearcut, mostly along the "top" of the road a quarter mile from the crest. The new upper alignment of the existing road was cleared as well, wider switchbacks well to the north and south of the current road.

Almost immediately after the clearing, Forest Service funding for the project unexpectedly dried up. Work was discontinued until new funding for the project could be secured, a delay that was not expected to last long but that fell at the worst possible time.

The Crest Loop had been abstract, seen only in the form of Forest Service presentations and hand-drawn newspaper illustrations. In that form, it had attracted considerable support from the business community and only muted objections from conservationists. But then the trees fell. Acres of clearcut forest, so close to the ridge and so visible from the road, turned the highway plan into an eyesore. As they sat, unchanged, month after month, an eyesore turned into a scandal.


By the summer of 1970, the Wildlife and Conservation Association, the New Mexico Conservation Coordination, and the National Wildlife Federation sent letters opposing the project. State legislators became involved. There were accusations of treachery, with Forest Service employees allegedly lobbying business leaders to support the project while on the clock, and the district forester holding secret meetings with state officials to coordinate.

Over the following year, opposition stacked up. US Rep. Manuel Lujan, a prominent New Mexico politician, wrote to the director of the Forest Service with his objections. NM Attorney General David Norvell joined with a press release asking the Forest Service to reconsider the plan. These objections were actually rather weak: Lujan, in particular, mostly just thought the proposed road was too expensive, a waste of money considering that there was already a highway connecting the same places. Even environmental groups tempered their criticism, with at least two clarifying that they objected only to the highway along the ridge, and not to improvements to the road between Balsam Glade and the crest.

The Forest Service's motivations can be difficult to understand. The whole Crest Loop plan had reportedly begun with a presidential directive to identify opportunities for new recreational facilities, and it had certainly had the strong support of the business community in the 1960s. At the turn of the 1970s, the situation had noticeably changed. Most press coverage of the project focused on the opposition, and Albuquerque's business clubs had fallen silent, perhaps averse to the controversy. There's not much to say for the Forest Service's lobbying effort beyond inertia: as far as they were concerned, they had committed to the plan in 1966 and were now simply executing on an old decision.

The federal government, once it gets into motion, does not easily change directions. For Forest Service staff, careers, or at least plans to climb the corporate ladder, must have been staked on the success of the Crest Loop. Even as the budget escalated from $3 million to $3.5, then as high as $5 million, the Forest Service remained committed to finishing what it had started. "[Opponents] claim the Forest Service won't heed their cries of distress," the Albuquerque Journal summarized. "The Forest Service says the decision to build the road was made a long time ago."

In May of 1971, the controversy broke out into demonstrations. This was no Vietnam war; anti-Crest Loop protests had a more whimsical quality. On Sunday the 16th, around two hundred demonstrators went up the Crest Scenic Highway and gathered in the cleared path of the future road. The plan, originally, had been to replant the clearing with new trees, a slow way of walking back the road's likewise slow advance. This threat of minor vandalism was met by a threat of minor reprisal when the Forest Service informed the press that unauthorized planting of trees in a National Forest could be charged as a misdemeanor.

The result, a mountaintop clash between Albuquerque's most motivated environmentalists and the force majeure of the Forest Service, must be one of the stranger moments in Albuquerque's environmental history.

One group---lead by Dr. Gerald Bordin---advocated "peaceful protest." Another, lead partly by Jesse Cole, planted three symbolic trees. Finally, two individuals---Florencio Baca and Marvin Price---began a private discussion that turned into a simulated panel discussion on the negative aspects of the proposed road....

Few persons started the planned three-mile walk down the graded Ellis Loop in the 10,000-foot altitude. Several protesters, disillusioned for the lack of trees to plant, began to leave by noon.

Bordin produced a cardboard replica of a pine tree, dissected by a highway, and placed it on the proposed road. "This type of planting is legal," he said; and another protester, watching two others dig a hole for a tree a few yards away, asked Bordin "Where's your tree, man?" 3


On the first day of 1970, president Richard Nixon signed the National Environmental Policy Act. Nixon was famously a mixed bag on environmental issues; his creation of the Environmental Protection Agency is one of the largest single steps forward in the history of US conservation, but he also pinched pennies to a degree that smothered many environmental programs. As it would turn out, the fate of the Crest Loop hinged on the interplay of these two sides of Nixon.

To satisfy its obligations under NEPA, the Forest Service completed an environmental impact assessment of the proposed Crest Road in 1973. It was now almost five years since the clearing of part of the right of way, but the only progress made had been on paper. Unsurprisingly, the impact assessment found the road to be feasible. "The highway is completely on the east slope and won't even be able to be seen from Albuquerque," the forest supervisor explained.

The environmental analysis spurred another series of exchanges between the Forest Service and environmental interests. The state Environmental Planning Commission filed comment, suggesting that the Crest Loop would better be replaced by a system of two one-way roads at different altitudes. One thing the EPC did agree on was the necessity that something be done. Like the groups who had assured their support for improving the road to the crest in the years before, the EPC recommended against taking no action. The existing crest road was unsustainable, they found, and needed to be improved in some way.

During the plant-in, one organizer suggested letters to the editor. Perhaps he was heard, as letters in opposition stacked up during 1973. Frank Marquart wrote that the highway would be good for General Motors and no one else. A letter from L. Graham got the simple headline "Crest Road Not Necessary." Among the growing public protest, the gears of government were turning: the Bernalillo County Commission made a resolution asking the Forest Service to stop.

Joe Whiton attended a hearing on the matter in Tijeras, leaving a Journal reporter with a memorable quote:

The Crest Road is a juggernaut already aimed, the switch is on and it is going to be fired no matter what we say. I have a feeling it is going to happen anyway and I don't think it should. 4


Despite the Forest Service's momentum, two significant roadblocks had arisen in the Crest Loop plan: first, the Forest Service was having a hard time getting to a final environmental impact statement on the project. Several drafts had been circulated, and from what I can tell one was even published as final, although it only covered a portion of the work rather than the entirety. The National Environmental Policy Act has many problems and often fails to achieve its aims, but it does have its virtues: while NEPA does not necessarily require agencies to listen to public feedback, it does require them to solicit it, and the hearings and comment periods that make up federal environmental policy gave the Crest Loop's opponents a platform on which to organize.

Second, there was President Nixon. Nixon had generally supported the Federal Lands Highway Program, under which the Crest Loop now fell, but his efforts to reduce federal spending through impoundment quickly became controversial. A fight with Congress over changes to the federal highway funding model in 1972 lead to a lapse in highway funding, a situation that became even more chaotic as Nixon specifically refused funding for a long list of highway grants. The funding situation probably didn't kill the Crest Loop, but it delayed further progress during the 1970s, contributing to both the sense of scandal over the lack of progress and to the opposition's developing power.

In 1973, the Forest Service produced new draft environmental documents covering four variant plans, which other than the no action alternative all involved some form of new highway. This was, as it turns out, the last gasp of the Crest Loop: in 1975, over a decade after planning first started, the Forest Service released a new environmental impact statement covering the entire Sandia mountain recreational plan. Its alternatives included new picnic grounds, bathrooms, and campgrounds. It included various improvements to the Crest Scenic Highway. But more important is what it did not include: any mention of the Crest Loop.

In the following years, the Forest Service would release further environmental and planning documents on the remaining scope of work, which was only the improvement of the road between Balsam Glade and the Crest. A 1976 impact statement covered three alternatives: the first would use the area already cleared for realignment, the second would keep the old alignment, and the third would finish realignment of the entire route, as included in the original Crest Loop plan. Based on the Albuquerque Tribune's count of public comments, Burqueños overwhelmingly favored the second option---keeping the existing route. While the road has been widened and resurfaced over the years, it remains on its original, winding route.


In 1927, roads reached Sandia Crest. Today, just shy of one hundred years later, the drive to the crest is essentially the same as this original route. It's all paved now, it's wider, and the curves has softened. It is still, by our modern standards, a rough road. The biggest change from the long-ago Ellis Ranch Loop to today's forest highways is actually a loss: the reduction of NM 165 to a minor, unpaved road. One might say, then, that progress since 1927 has been backwards. We are less ambitious than we once were.

On the flip side, the ambitions of the mid-century are so difficult to square with our modern sense of environmental preservation. The Forest Service dreamed, at times, of thousands of parking spaces, of cabins, a resort. We now value conservation over improvement, wilderness over access. It almost seems like it has always been that way---but it hasn't. This is a modern idea in land management, an innovation of the post-war era and the American environmental movement.

The old tensions between tourist promotion and wilderness preservation have never gone away. In 2023, Mountain Capital Partners signed a joint venture with the Sandia Peak Ski Company. The Sandia Peak Ski Area has barely opened for the last five years; climate change has shortened the season so severely that the Sandias can go all winter without enough snow to ski. Mountain Capital Partners how hopes, through artificial snowmaking, to bring the ski area back to life. Still, it's clear that snow will never be enough: ski areas are, in general, facing tough times.

Sandia Peak Ski Company has developed an ambitious plan for a "four season" recreational destination. It called for a mountain roller coaster, mountainbike trails, a complete replacement of Lift #1. The mountain coaster has already been abandoned, having attracted more environmental controversy than the Forest Service was prepared to handle.

In aerial images, you can still clearly see the path of the Crest Loop, at least to the first switchback where 1968 clearing work ended. Some of the modern trails follow the unused highway rights of way. They are incongruously described by in some Forest Service documents as firebreaks, a function that they do serve but were never intended for. I get the impression that even some of the Forest Service staff have forgotten the entire Crest Loop story. Well, it's one of many things about the Sandias that have been forgotten.

Ellis Ranch is now hardly remembered, up a canyon from a picnic area. It is once again difficult to access, past the part of the former Ellis Loop that is still paved. The Crest Loop was canceled in its early stages, the notional second highway was never even planned.

But Ellis Ranch Loop Road is still there. Millions of people drive up it every year. They look down on the city, and marvel at the view. I wonder how many realize---that the climb started so long ago, and that it has still never been finished. I think that it's better this way. A summit should always be an achievement, so easy to see, but so hard to reach. We get used to living in the shadows of mountains.

  1. Impassable is of course a subjective term. Last time I had it in mind to attempt NM 165 with snow on the ground, about two years ago, I started from the top at Balsam Glade. At least, that was my plan... on arrival at Balsam Glade I found a small crowd of people formed around an F-350 that had only gotten perhaps 50' down 165, a steep hill at that spot, before becoming unable to get back up. I think that a smaller vehicle would have fared better but this was an obstacle that did not seem likely to clear up quickly. I abandoned the attempt.

  2. Albuquerque Journal, 1969-07-20 p. 37

  3. Albuquerque Journal, 1971-05-16 p. 1

  4. Albuquerque Journal, 1974-02-15 p. 45

Error'd: Will You Still Need Me?

... when I'm eight thousand and three? Doesn't quite scan.

Old soul jeffphi hummed "It's comforting to know that I'll have health insurance coverage through my 8,030th birthday!"

1

 

Scififan Steve muttered "I asked Copilot if Tom Baker and Lalla Ward were the same age. Apparently, they have been traveling on different timelines, so Copilot claims they are the same age despite having birthdays 17 years apart. Who knew?" It's a timey-wimey thing.

2

 

An anonymous Aussie announced "I was trying to look up a weather forecast for several weeks in advance as I'm attending an event on Saturday 22nd November. Apparently the Weather Channel has invented its own calendar (or decided there are too many days in the year), as while the 1st of November was a Saturday and 22nd of November occur on a Saturday, the Weather Channel has decided those dates fall on a Friday.
Rosanna is in Melbourne, Australia. Temperatures are displayed in Celsius. " Rosanna is rated the 99th most liveable suburb of Melbourne, probably due to the aberrant calendar.

0

 

Beatrix W. wants to relocate to a more pedestrian-friendly nabe. "I was bored and looked for houses on a german real estate website. When I tried to have the distance to a house calculated I got the lovely result from the screenshot. The 99 minutes are okay when using the car but the rest is -1 minute."

3

 

Taxpayer Marlin D. is making sure he gets everything lined up for year-end. Checking his witholdings, he found that the IRS try very hard to be precise with their language. "This is from the IRS online Tax Withholding Estimator tool. I guess they really do mean *between*."

4

 

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

CodeSOD: Lucky Thirteen

Wolferitza sends us a large chunk of a C# class. We'll take it in chunks because there's a lot here, but let's start with the obvious problem:

    private int iID0;
    private int iID1;
    private int iID2;
    private int iID3;
    private int iID4;
    private int iID5;
    private int iID6;
    private int iID7;
    private int iID8;
    private int iID9;
    private int iID10;
    private int iID11;
    private int iID12;
    private int iID13;

If you say, "Maybe they should use an array," you're missing the real problem here: Hungarian notation. But sure, yes, they should probably use arrays. And you might think, "Hey, they should use arrays," would be an easy fix. Not for this developer, who used an ArrayList.

private void Basculer(DataTable dtFrom, DataTable dtTo)
{
    ArrayList arrRows = new ArrayList();

    int index;

    DataRow drNew1 = dtTo.NewRow();
    DataRow drNew2 = dtTo.NewRow();
    DataRow drNew3 = dtTo.NewRow();
    DataRow drNew4 = dtTo.NewRow();
    DataRow drNew5 = dtTo.NewRow();
    DataRow drNew6 = dtTo.NewRow();
    DataRow drNew7 = dtTo.NewRow();
    DataRow drNew8 = dtTo.NewRow();
    DataRow drNew9 = dtTo.NewRow();
    DataRow drNew10 = dtTo.NewRow();
    DataRow drNew11 = dtTo.NewRow();
    DataRow drNew12 = dtTo.NewRow();
    DataRow drNew13 = dtTo.NewRow();
    DataRow drNew14 = dtTo.NewRow();
    DataRow drNew15 = dtTo.NewRow();

    arrRows.Add(drNew1);
    arrRows.Add(drNew2);
    arrRows.Add(drNew3);
    arrRows.Add(drNew4);
    arrRows.Add(drNew5);
    arrRows.Add(drNew6);
    arrRows.Add(drNew7);
    arrRows.Add(drNew8);
    arrRows.Add(drNew9);
    arrRows.Add(drNew10);
    arrRows.Add(drNew11);
    arrRows.Add(drNew12);
    arrRows.Add(drNew13);
    arrRows.Add(drNew14);
    arrRows.Add(drNew15);
    // more to come…

Someone clearly told them, "Hey, you should use an array or an array list", and they said, "Sure." There's just one problem: arrRows is never used again. So they used an ArrayList, but also, they didn't use an ArrayList.

But don't worry, they do use some arrays in just a moment. Don't say I didn't warn you.

    if (m_MTTC)
    {
        if (m_dtAAfficher.Columns.Contains("MTTCRUB" + dr[0].ToString()))
        {
            arrMappingNames.Add("MTTCRUB" + dr[0].ToString());
            arrHeadersTexte.Add(dr[4]);
            arrColumnsFormat.Add("");
            arrColumnsAlign.Add("1");

Ah, they're splitting up the values in their data table across multiple arrays; the "we have object oriented programming at home" style of building objects.

And that's all the setup. Now we can get into the real WTF here.

            if (iCompt == Convert.ToInt16(0))
            {
                iID0 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(1))
            {
                iID1 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(2))
            {
                iID2 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(3))
            {
                iID3 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(4))
            {
                iID4 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(5))
            {
                iID5 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(6))
            {
                iID6 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(7))
            {
                iID7 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(8))
            {
                iID8 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(9))
            {
                iID9 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(10))
            {
                iID10 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(11))
            {
                iID11 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(12))
            {
                iID12 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(13))
            {
                iID13 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
        }
    }

Remember those private iID* values? Here's how they get populated. We check a member variable called iCompt and pull the first value out of a dr variable (a data reader, also a member variable). You may have looked at the method signature and assumed dtFrom and dtTo would be used, but no- they have to purpose in this method at all.

And if you liked what happened in this branch of the if, you'll love the else:

    else
    {
        if (m_dtAAfficher.Columns.Contains("MTTHRUB" + dr[0].ToString()))
        {
            arrMappingNames.Add("MTTHRUB" + dr[0].ToString());
            arrHeadersTexte.Add(dr[4]);
            arrColumnsFormat.Add("");
            arrColumnsAlign.Add("1");

            if (iCompt == Convert.ToInt16(0))
            {
                iID0 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(1))
            {
                iID1 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(2))
            {
                iID2 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(3))
            {
                iID3 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(4))
            {
                iID4 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(5))
            {
                iID5 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(6))
            {
                iID6 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(7))
            {
                iID7 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(8))
            {
                iID8 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(9))
            {
                iID9 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(10))
            {
                iID10 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(11))
            {
                iID11 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(12))
            {
                iID12 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
            else if (iCompt == Convert.ToInt16(13))
            {
                iID13 = Convert.ToInt32(dr[0]);
                iCompt = iCompt + 1;
            }
        }
    }
}

I can only assume that this function is called inside of a loop, though I have to wonder about how that loop exits? Maybe I'm being too generous, this might not be called inside of a loop, and the whole class gets to read 13 IDs out before it's populated. Does iCompt maybe get reset somewhere? No idea.

Honestly, does this even work? Wolferitza didn't tell us what it's actually supposed to do, but found this code because there's a bug in there somewhere that needed to be fixed. To my mind, "basically working" is the worst case scenario- if the code were fundamentally broken, it could just be thrown away. If it mostly works except for some bugs (and terrible maintainability) no boss is going to be willing to throw it away. It'll just fester in there forever.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

CodeSOD: Historical Dates

Handling non-existent values always presents special challenges. We've (mostly) agreed that NULL is, in some fashion, the right way to do it, though it's still common to see some sort of sentinel value that exists outside of the expected range- like a function returning a negative value when an error occurred, and a zero (or positive) value when the operation completes.

Javier found this function, which has a… very French(?) way of handling invalid dates.

 Private Function CheckOraDate(ByVal sDate As String) As String
        Dim OraOValDate As New DAL.PostGre.DataQuery()
        Dim tdate As Date
        If IsDate(sDate) Then
            Return IIf(OraOValDate.IsOracle, CustomOracleDate(Convert.ToDateTime(sDate).ToString("MM-dd-yyyy")), "'" & sDate & "'")
        Else
            '~~~ No Date Flag of Bastille Day
            Return CustomOracleDate(Convert.ToDateTime("07/14/1789").ToString("MM-dd-yyyy"))
        End If

    End Function

Given a date string, we check if it is a valid date string using IsDate. If it is, we check if our data access layer thinks the IsOracle flag is set, and if it is, we do some sort of conversion to a `CustomOracleDate", otherwise we just return the input wrapped in quotes.

All that is sketchy- any function that takes dates as a string input and then returns the date in a new format as a string always gets my hackles up. It implies loads of stringly typed operations.

But the WTF is how we handle a bad input date: we return Bastille Day.

In practice, this meant that their database system was reporting customers' birthdays as Bastille Day. And let me tell you, those customers don't look a day over 200, let alone 236.

For an extra bonus WTF, while the "happy path" checks if we should use the custom oracle formatting, the Bastille Day path does not, and just does whatever the Oracle step is every time.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

CodeSOD: Losing a Digit

Alicia recently moved to a new country and took a job with a small company willing to pay well and help with relocation costs. Overall, the code base was pretty solid. Despite the overall strong code base, one recurring complaint was that running the test suite was painfully long.

While Alicia doesn't specify what the core business is, but says: "in this company's core business, random numbers were the base of everything."

As such, they did take generating random numbers fairly seriously, and mostly used strong tools for doing that. However, whoever wrote their test suite was maybe a bit less concerned, and wrote this function:

public static Long generateRandomNumberOf(int length) {
    while (true) {
            long numb = (long)(Math.random() * 100000000 * 1000000); // had to use this as int's are to small for a 13 digit number.
            if (String.valueOf(numb).length() == length)
                return numb;		
        }		
}

They want many digits of random number. So they generate a random floating point, and then multiply it a few times to get a large number. If the length of the resulting number, in characters, is the desired length, we return it. Otherwise, we try again.

The joy here, of course, is that this function is never guaranteed to exit. In fact, if you request more than 15 digits, it definitely won't exit. In practice, most of the time, the function is able to hit the target length in a relative handful of iterations, but there's no guarantee for that.

Alicia was tracking down a bug in a test which called this function. So she went ahead and fixed this function so that it use a sane way to generate the appropriate amount of entropy that actually guaranteed a result. She included that change in her pull request, nobody had any comments, and it got merged in.

The unit tests aren't vastly faster than they were, but they are faster. Who knows what other surprises the test suite has in store?

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

CodeSOD: High Temperature

Brian (previously)found himself contracting for an IoT company, shipping thermostats and other home automation tools, along with mobile apps to control them.

Brian was hired because the previous contractor had hung around long enough for the product to launch, cashed the check, and vanished, never to be heard from again.

And let's just say that Brian's predecessor had a unique communication style.

    private class NoOpHandler extends AsyncCharisticHandler {
        public NoOpHandler(Consumer<BluetoothGattCharacteristic> consumer) {
            super(null, consumer);
        }

        @Override
        public void Execute() {
            // Don't do any actual BLE Communication here, and just go straight to the callback, This
            // handler is just to allow people to get a callback after after a bunch of Async IO Operations
            // have happened, without throwing all the completion logic into the "last" async callback of your batch
            // since the "last" one changes.
            InvokeCallback();
            // After this callback has been handled, recheck the queue to run any subsequent Async IO Operations
            OnBLEAsyncOperationCompleted();
            // I'm aware this is recursive. If you get a stack overflow here, you're using it wrong.
            // You're not supposed to queue thousands of NoOp handlers one after the other, Stop doing it!
            // If you need to have code executed sequentially just, er... write a fu*king function there is
            // nothing special about this callback, or the thread it is called on, and you don't need to use
            // it for anything except getting a callback after doing a batch of async IO, and then, it runs 
            // in the context of the last IO Completion callback, which shouldn't take ages. If you use 
            // AsyncRunWhenCompleted() to create more of these within the callback of AsyncRunWhenCompleted
            // it just keeps the IO completion thread busy, which also breaks shit. 
            // Basically, you shouldn't be using AsyncRunWhenCompleted() at all if you're not me.
        }
    }

Who said bad programmers don't write comments? This bad programmer wrote a ton of comments. What's funny about this is that, despite the wealth of comments, I'm not 100% certain I actually know what I'm supposed to do, aside from not use AsyncRunWhenCompleted.

The block where we initialize the Bluetooth system offers more insight into this programmer's style.

    @SuppressLint("MissingPermission")
    private void initializeBluetooth() {
        _bluetoothManager = (BluetoothManager) getSystemService(BLUETOOTH_SERVICE);
        _bluetoothAdapter = _bluetoothManager != null ? _bluetoothManager.getAdapter() : null;
        if (_bluetoothAdapter != null && _bluetoothAdapter.isEnabled()) {
            /* #TODO: I don't know if cancelDiscovery does anything... either good, or bad. It seems to make BLE discovery faster after 
            *         the service is restarted by android, but I don't know if it screws anything else up in the process. Someone should check into that */
            _bluetoothAdapter.cancelDiscovery();
            _bluetoothScanner = _bluetoothAdapter.getBluetoothLeScanner();
            _scanFilters = Collections.singletonList(new ScanFilter.Builder().setServiceUuid(new ParcelUuid(BLE_LIVELINK_UUID)).build());
            CreateScanCallback();
        } else {
            // #TODO: Handle Bluetooth not available or not enabled
            stopSelf();
        }
    }

This is a clear example of "hacked together till it works". What does cancelDiscovery do? No idea, but we call it anyway because it seems to be faster. Should we look it up? Because yes, it sounds like calling it is correct, based on the docs. Which took me 15 seconds to find. "Someone should check into that," and apparently I am that someone.

Similarly, the second TODO seems like an important missing feature. At least a notification which says, "Hey, you need bluetooth on to talk to bluetooth devices," would go a long way.

All this is in service of an IoT control app, which seems to double as a network scanner. It grabs the name of every Bluetooth and WiFi device it finds, and sends them and a location back to a web service. That web service logs them in a database, which nobody at the company ever looks at. No one wants to delete the database, because it's "valuable", though no one can ever specify exactly how they'd get value from it. At best, they claim, "Well, every other app does it." Mind you, I think we all know how they'd get value: sell all this juicy data to someone else. It's just no one at the company is willing to say that out loud.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Error'd: What Goes Up

As I was traveling this week (just home today), conveyances of all sorts were on my mind.

Llarry A. warned "This intersection is right near my house. Looks like it's going to be inconvenient for a while..." Keeping this in mind, I chose to take the train rather than drive.

1

 

Unfortunately US trains are restricted to plodding sublight speeds, but Mate has it better. "I love how the Swiss Federal Railways keep investing in new rolling stock... like this one that can teleport from one side of the country to the other in 0 minutes?! "

0

 

And Michael R. 's TfL seems to operate between parallel universes. "I was happy to see that the "not fitted" Northern Line train actually rolled in 2 mins later."

2

 

Daniel D. 's elevator went up but the ubiquitous screen went down. Daniel even slipped a bit of selfie into his submission. Sneaky. "This display usually features some looping video. This time it featured only desktop with bunch of scripts / *.bat files. I guess it's better when elevator's display crashes than when an actual elevator crashes?"

4

 

Joel C. 's elevator had a different fault. "The screen in my hotel's elevator is not sending quite the message they probably want."

3

 

[Advertisement] Plan Your .NET 9 Migration with Confidence
Your journey to .NET 9 is more than just one decision.Avoid migration migraines with the advice in this free guide. Download Free Guide Now!

Secure to Great Lengths

Our submitter, Gearhead, was embarking on STEM-related research. This required him to pursue funding from a governmental agency that we’ll call the Ministry of Silly Walks. In order to start a grant application and track its status, Gearhead had to create an account on the Ministry website.

The registration page asked for a lot of personal information first. Then Gearhead had to create his own username and password. He used his password generator to create a random string: D\h.|wAi=&:;^t9ZyoO

Silly Walk Gait

Upon clicking Save, he received an error.

Your password must be a minimum eight characters long, with no spaces. It must include at least three of the following character types: uppercase letter, lowercase letter, number, special character (e.g., !, $, % , ?).

Perplexed, Gearhead emailed the Ministry’s web support, asking why his registration failed. The reply:

Hello,
The site rejects password generators as hacking attempts. You will need to manually select a password.
Ex. GHott*01

Thank you,

Support

So a long sequence of random characters was an active threat, but a 1990s-era AOL username was just fine. What developer had this insane idea and convinced other people of it? How on earth did they determine what was a "manually selected" string versus a randomly-generated one?

It seems the deciding factor is nothing more than length. If you go to the Ministry’s registration page now, their password guidelines have changed (emphasis theirs):

Must be 8-10 characters long, must contain at least one special character ( ! @ # $ % ^ & * ( ) + = { } | < > \ _ - [ ] / ? ) and no spaces, may contain numbers (0-9), lower and upper case letters (a-z, A-Z). Please note that your password is case sensitive.

Only good can come of forcing tiny passwords.

The more a company or government needs secure practices, the less good they are at secure practices. Is that a law yet? It should be.

[Advertisement] Plan Your .NET 9 Migration with Confidence
Your journey to .NET 9 is more than just one decision.Avoid migration migraines with the advice in this free guide. Download Free Guide Now!

Future Documentation

Dotan was digging through vendor supplied documentation to understand how to use an API. To his delight, he found a specific function which solved exactly the problem he had, complete with examples of how it was to be used. Fantastic!

He copied one of the examples, and hit compile, and reviewed the list of errors. Mostly, the errors were around "the function you're calling doesn't exist". He went back to the documentation, checked it, went back to the code, didn't find any mistakes, and scratched his head.

Now, it's worth noting the route Dotan took to find the function. He navigated there from a different documentation page, which sent him to an anchor in the middle of a larger documentation page- vendorsite.com/docs/product/specific-api#specific-function.

This meant that as the page loaded, his browser scrolled directly down to the specific-function section of the page. Thus, Dotan missed the gigantic banner at the top of the page for that API, which said this:

/!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!
This doc was written to help flesh out a user API. The features described here are all hypothetical and do not actually exist yet, don't assume anything you see on this page works in any version /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\ NOTE /!\

On one hand, I think providing this kind of documentation is invaluable, both to your end users and for your own development team. It's a great roadmap, a "documentation driven development" process. And I can see that they made an attempt to be extremely clear about it being incomplete and unimplemented- but they didn't think about how people actually used their documentation site. A banner at the top of the page only works if you read the page from top to bottom, but documentation pages you will frequently skip to specific sections of the page.

But there was a deeper issue with the way this particular approach was executed: while the page announced that one shouldn't assume anything works, many of the functions on the page did work. Many did not. There was no rhyme or reason, to version information or other indicators to help a developer understand what was and was not actually implemented.

So while the idea of a documentation-oriented roadmap specifying features that are coming is good, the execution here verged into WTF territory. It was a roadmap, but with all the landmarks erased, so you had no idea where you actually were along the length of that road. And the one warning sign that would help you was hidden behind a bush.

Dotan asks: "WTF is that page doing on the official documentation wiki?"

And I'd say, I understand why it's there, but boy it should have been more clear about what it actually was.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

Undefined Tasks

Years ago, Brian had a problem: their C# application would crash sometimes. What was difficult to understand was why it was crashing, because it wouldn't crash in response to a user action, or really, any easily observable action.

The basic flow was that the users used a desktop application. Many operations that the users wanted to perform were time consuming, so the application spun up background tasks to do them, thus allowing the user to do other things within the application. And sometimes, the application would just crash, both when the user hadn't done anything, and when all background jobs should have been completed.

The way the background task was launched was this:

seeder.RunSeeder();

It didn't take too much head scratching to realize what was running every time the application crashed: the garbage collector.

RunSeeder returned a Task object, but since Brian's application treated the task as "fire and forget", they didn't worry about the value itself. But C# did- the garbage collector had to clean up that memory.

And this was running under .Net 4.0. This particular version of the .Net framework was a special, quantum realm, at least when it came to tasks. You see, if a Task raises an exception, nothing happens. At least, not right away. No one is notified of the exception unless they inspect the Task object directly. There's a cat in the box, and no one knows the state of the cat unless they open the box.

The application wasn't checking the Task result. The cat remained in a superposition of "exception" and "no exception". But the garbage collector looked at the task. And, in .Net 4.0, Microsoft made a choice about what to do there: when they opened the box and saw an exception (instead of a cat), they chose to crash.

Microsoft's logic here wasn't entirely bad; an uncaught exception means something has gone wrong and hasn't been handled. There's no way to be certain the application is in a safe state to continue. Treating it akin to undefined behavior and letting the application crash was a pretty sane choice.

The fix for Brian's team was simple: observe the exception, and choose not to do anything with it. They truly didn't care- these tasks were fire-and-forget, and failure was acceptable.

seeder.RunSeeder().ContinueWith(t => { var e = t.IsFaulted ? t.Exception : null; }); // Observe exceptions to prevent quantum crashes

This code merely opens the box and sees if there's an exception in there. It does nothing with it.

Now, I'd say as a matter of programming practice, Microsoft was right here. Ignoring exceptions blindly is a definite code smell, even for a fire-and-forget task. Writing the tasks in such a way as they catch and handle any exceptions that bubble up is better, as is checking the results.

But I, and Microsoft, were clearly on the outside in this argument. Starting with .Net 4.5 and moving forward, uncaught exceptions in background tasks were no longer considered show-stoppers. Whether there was a cat or an exception in the box, when the garbage collector observed it, it got thrown away either way.

In the end, this reminds me of my own failing using background tasks in .Net.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

CodeSOD: Solve a Captcha to Continue

The first time Z hit the captcha on his company's site, he didn't think much of it. And to be honest, the second time he wasn't paying that much attention. So it wasn't until the third time that he realized that the captcha had showed him the same image every single time- a "5" with lines scribbled all over it.

That led Z to dig out the source and see how the captcha was implemneted.

<Center>Click a number below to proceed to the next page. <br>Some browsers do not like this feature and will try to get around it. If you are having trouble<br> seeing the image, empty your internet cache and temporary internet files may help. <br>Please ensure you have no refresher add-ons installed on your browser.<br />
<table border=1><Tr><td colspan='3' align='center'>
<font style='font-size:36px;'><img width='150' title='5' alt='5' src='valimages/5.gif'> </font></td></tr>
<Tr>
<Td align=center><font size='6'><A href='valid.php?got=crimied&linknum=1'>1</a></font></td>
<Td align=center><font size='6'><A href='valid.php?got=crimied&linknum=2'>2</a></font></td>
<Td align=center><font size='6'><A href='valid.php?got=crimied&linknum=3'>3</a></font></td>
</tr>
<Tr>
<Td align=center><font size='6'><A href='valid.php?got=crimied&linknum=4'>4</a></font></td>
<Td align=center><font size='6'><A href='valid.php?got=crimied&linknum=5'>5</a></font></td>
<Td align=center><font size='6'><A href='valid.php?got=crimied&linknum=6'>6</a></font></td>
</tr>
<tr>
<Td align=center><font size='6'><A href='valid.php?got=crimied&linknum=7'>7</a></font></td>
<Td align=center><font size='6'><A href='valid.php?got=crimied&linknum=8'>8</a></font></td>
<Td align=center><font size='6'><A href='valid.php?got=crimied&linknum=9'>9</a></font></td>
</tr></table>
</Center>

Look, I know there's a joke about how hard it is to center things in CSS, but I think we've gone a little overboard with our attempt here.

Now, the PHP driving this page could have easily been implemented to randomly select an image from the valimages directory, and there was some commented out code to that effect. But it appears that whoever wrote it couldn't quite understand how to link the selected image to the behavior in valid.php, so they just opted to hard code in five as the correct answer.

The bonus, of course, is that the image for five is named 5.gif, which means if anyone really wanted to bypass the captcha, it'd be trivial to do so by scraping the code. I mean, not more trivial than just realizing "it's the same answer every time", but still, trivial.

Of course, out here in the real world, captchas have never been about keeping bots out of sites, and instead are just a way to trick the world into training AI. Pretty soon we'll roll out the Voight-Kampf test, but again, the secret purpose won't be to find the replicants, but instead gather data so that the next generation of replicants can pass the test.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Error'd: Once Is Never Enough

"Getting ready to!" anticipated richard h. but then this happened. "All I want are the CLI options to mark the stupid TOS box so I can install this using our Chef automation. "What are the options" is too much to ask, apparently. But this is Microsoft. Are stupid errors like this really that unexpected?"

1

 

Followed immediately by richard's report: "Following up my error'd submission a few minutes ago, I clicked the "Accept TOS" box, and the "Something unexpected happened" box lit up, so I clicked the button to let the unexpected do what the something wanted to do. Now I have successfully Something unexpected happened. smh Microsoft. "

0

 

An anonymous griper snickered "It's a made up word, but I just wanted to check the spelling before writing it in a slack comment as a joke referencing the show("that's a nice tnetennba"), but the first thing I saw was the AI preview with the first sentence incorrectly claiming it's "basketball" spelled backwards(which it's clearly not, backwards it would be "abnnetent" which is also not a word). " I have to differ, though. Spelled backwards it would be llabteksab.

Silly monkey, backwards it would be ti

 

And a different anonymous griper (I assume they're different, but they're anonymous so who can really know?) needed some help doing a quite trivial computation. "On which planet?" we all wonder together. 3

 

Finally, a recurring theme from a recurring reader, B.J.H. keeps buying stuff. "This screen shot was captured the morning of 26 October. I'm not sure what bothers me more, that the package was picked up twice (once in the future), or that "Standard Transit" (when the package should be expected) is a day before the pick-up. Or maybe they just lie about the pickup to cover for not meeting the standard delivery date. "

4

 

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

The Ghost Cursor

Everyone's got workplace woes. The clueless manager; the disruptive coworker; the cube walls that loom ever higher as the years pass, trapping whatever's left of your soul.

But sometimes, Satan really leaves his mark on a joint. I worked Tech Support there. You may remember The C-Level Ticket. I'm Anonymous. This is my story.


Between 2 Buildings (Montreal) - Flickr - MassiveKontent

Night after night, my dreams are full of me trying and failing at absolutely everything. Catch a bus? I'm already running late and won't make it. Dial a phone number to get help? I can't recall the memorized sequence, and the keypad's busted anyway. Drive outta danger? The car won't start. Run from a threat? My legs are frozen.

Then I wake up in my bed in total darkness, scared out of my skull, and I can't move for real. Not one muscle works. Even if I could move, I'd stay still because I'm convinced the smallest twitch will give me away to the monster lurking nearby, looking to do me in.

The alarm nags me before the sun's even seen fit to show itself. What day is it? Tuesday? An invisible, overwhelming dread pins me in place under the covers. I can't do it. Not again.

The thing is, hunger, thirst, and cold are even more nagging than the alarm. Dead tired, I force myself up anyway to do the whole thing over.


The office joe that morning was so over-brewed as to be sour. I tossed down the last swig in my mug, checking my computer one more time to make sure no Tech Support fires were raging by instant message or email. Then I threw on my coat and hat and quit my cube, taking the stairs to ground level.

I pushed open a heavy fire-escape door and stepped out into the narrow alley between two massive office buildings. Brisk autumn air and the din of urban motor traffic rushed to greet me. The dull gray sky above threatened rain. Leaning against the far brick wall were Toby and Reynaldo, a couple of network admins, hugging themselves as they nursed smoldering cigarettes. They nodded hello.

I tipped my hat in greeting, slipping toward the usual spot, a patch of asphalt I'd all but worn grooves in by that point. I lit my own cigarette and took in a deep, warming draw.

"Make it last another year," Toby spoke in a mocking tone, tapping ash onto the pavement. "I swear, that jerk can squeeze a nickel until Jefferson poops!"

An ambulance siren blared through the alley for a minute. The rig was no doubt racing toward the hospital down the street.

Reynaldo smirked. "You think Morty finally did it?"

Toby smirked as well.

I raised an eyebrow. "Did what?"

"Morty always says he's gonna run out into traffic one of these days so they can take him to the hospital and he won't have to be here," Reynaldo explained.

I frowned at the morbid suggestion. "Hell of a way to catch a break."

"Well, it's not like we can ask for time off," Toby replied bitterly. "They always find some way to rope us back in."

I nodded in sympathy. "You have it worse than we do. But my sleep's still been jacked plenty of times by 3AM escalated nonsense that shoulda been handled by a different part of the globe."

Reynaldo's eyes lit up fiercely. "They have all the same access and training, but it never falls on them! Yeah, been there."

The door swung open again, admitting a young woman with the weight of the world on her shoulders. This was Megan, a junior developer and recent hire. I tipped my hat while helping myself to another drag.

She hastened my way, pulling a pack of cigarettes from her handbag. With shaking hands, she fumbled to select a single coffin nail. "I quit these things!" she lamented. After returning the pack to her bag, she rummaged through it fruitlessly. "Dammit, where are those matches?!" She glanced up at me with a pleading expression.

I pulled the lighter from my coat pocket. "You sure?"

She nodded like she hadn't been more sure about anything in her entire life.

I lit it for her. She took a lung-filling pull, then exhaled a huge cloud of smoke.

"Goin' that well, huh?" I asked.

Megan also hugged herself, her expression pained. "Every major player in the industry uses our platform, and I have no idea how it hasn't all come crashing down. There are thousands of bugs in the code base. Thousands! It breaks all the time. Most of the senior devs have no clue what they're doing. And now we're about to lose the only guy who understands the scheduling algorithm, the most important thing!"

"That's tough." I had no idea what else to say. Maybe it was enough that I listened.

Megan glanced up nervously at the brewing storm overhead. "I just know that algorithm's gonna get dumped in my lap."

"The curse of competence." I'd seen it plenty of times.

"Ain't that the truth!" She focused on me again with a look of apology. "How've you been?"

I shrugged. "Same old, same old." I figured a fresh war story might help. "Had to image and set up the tech for this new manager's onboarding. Her face is stuck in this permanent glare. Every time she opens her mouth, it's to bawl someone out."

"Ugh."

"The crazy thing is, the walls of her office are completely covered with crucifixes, and all these posters plastered with flowers and hearts and sap like Choose Kindness." I leaned in and lowered my voice. "You know what I think? I think she’s an ancient Roman whose spite has kept her alive for over two thousand years. Those crosses are a threat!"

That teased a small laugh out of Megan. For a moment, the amusement reached her eyes. Then it was gone, overwhelmed by worry. She took to pacing through the narrow alley.


Back at my cube, I found a new urgent ticket at the top of my case load. Patricia Dracora, a senior project manager, had put in a call claiming her computer had been hacked. Her mouse cursor was moving around and clicking things all on its own.

It was too early in the morning for a case like this. That old dread began sneaking up on me again. The name put me on edge as well. Over the years, our paths had never crossed, but her nickname throughout Tech Support, Dracula, betrayed what everyone else made of her.

"Make like a leaf and blow!"

The boss barked his stern command over my shoulder. I stood and turned from my computer to find him at my cubicle threshold with arms folded, blocking my egress.

I couldn't blow, so I shrugged. "Can't be as bad as The Crucifier."

"Dracula's worse than The Crucifier," the boss replied under his breath in a warning tone. "For your own good, don't keep her waiting!" He tossed a thumb over his shoulder for good measure.

When he finally backed out of the way, I made tracks outta there. A few of my peers made eye contact as I passed, looking wary on my behalf.

The ticket pegged Dracora's office in a subfloor I'd never set foot in before. Descending the stairs, I had too much time to think. Of course I didn't expect a real hacking attempt. Peripheral hardware on the fritz, some software glitch: there'd be a simple explanation. What fresh hell would I have to endure to reach that point? That was what my tired brain couldn't let go of. The stimulants hadn't kicked in yet. With the strength of a kitten, I was stepping into a lion's den. A lion who might make me wish for crucifixion by the time it was all over.

From the stairwell, I entered a dank, deserted corridor. Old florescent lighting fixtures hummed and flickered overhead. That, combined with the overwhelming stench of paint fumes, set the stage for a ripping headache. There were no numbers on the walls to lead me to the right place. They must've taken them down to paint and never replaced them. I inched down worn, stained carpeting, peeking into each open gap I found to either side of me. Nothing but darkness, dust, and cobwebs at first. Eventually, I spotted light blaring from one of the open doors ahead of me. I jogged the rest of the way, eager to see any living being by that point.

The room I'd stumbled onto was almost closet-sized. It contained a desk and chair, a laptop docking station, and a stack of cardboard boxes on the floor. Behind the desk was a woman of short stature, a large purse slung over one shoulder. Her arms were folded as she paced back and forth in the space behind her chair. When I appeared, she stopped and looked to me wide-eyed, maybe just as relieved as I was. "Are you Tech Support?"

"Yes, ma'am." I entered the room. "What's—?"

"I don't know how it happened!" Dracora returned to pacing, both hands making tight fists around the straps of the purse she was apparently too wired and distracted to set down. "They made me move here from the fourth floor. I just brought everything down and set up my computer, and now someone has control of the mouse. Look, look!" She stopped and pointed at the monitor.

I rounded the desk. By the time I got there, whatever she'd seen had vanished. Onscreen, the mouse cursor sat still against a backdrop of open browsers and folders. Nothing unusual.

"It was moving, I swear!" Anguished, Dracora pleaded with me to believe her.

It seemed like she wasn't hostile at all, just stressed out and scared. I could handle that. "I'm sure we can figure this out, ma'am. Lemme have a look here."

I sat down at the desk and tried the wireless mouse first. It didn't work at all to move the cursor.

"The hacker's locked us out!" Dracora returned to pacing behind me.

As I sat there, not touching a thing, the mouse cursor shuttled across the screen like it was possessed.

"There! You see?"

Suddenly, somehow, my brain smashed everything together. "Ma'am, I have an idea. Could you please stand still?"

Dracora stopped.

I swiveled around in the chair to face her. "Ma'am, you said you were moving in down here. What's in your purse right now?"

Her visible confusion deepened. "What?"

"The mouse cursor only moves around when you do," I explained.

Her eyes widened. She dug deeply into her purse. A moment later, she pulled out a second wireless mouse. Then she looked to me like she couldn't believe it. "That's it?!"

"That's it!" I replied.

"Oh, lord!" Dracora replaced the dud sitting on her mousepad with the mouse that was actually connected to her machine, wilting over the desk as she did so. "I don't know whether to laugh or cry."

I knew the feeling. But the moment of triumph, I gotta admit, felt pretty swell. "Anything else I can help with, ma'am?"

"No, no! I've wasted enough of your time. Thank you so much!"

I had even more questions on the way back upstairs. With this huge, spacious office building, who was forcing Dracora to be in that pit? How had she garnered such a threatening reputation? Why had my experience been so different from everyone else's? I didn't mention it to the boss or my peers. I broke it all down to Megan in the alley a few days later.

"She even put in a good word for me when she closed the ticket," I told her. "The boss says I'm on the fast track for another promotion." I took a drag from my cigarette, full of bemusement. "I'm already as senior as it gets. The only way up from here is management." I shook my head. "That ain't my thing. Look how well it's gone for Dracora."

Megan lowered her gaze, eyes narrowed. "You said it yourself: the only reward for good work is more work."

And then they buried you ... in a basement, or a box.

I remembered being at the start of my career, like Megan. I remembered feeling horrified by all the decades standing between me and the day when I wouldn't or couldn't ever work again. A couple decades in, some part of me that I'd repressed had resurfaced. What the hell is this? What have I been doing?

Stop caring, a different part replied. Just stop caring. Take things day by day, case by case.

I'd obeyed for so long. Where had it gotten me?

Under my breath, I risked airing my wildest wish for the future. "Someday, I wanna break outta this joint."

Megan blinked up at me. I had her attention. "How?"

"I dunno," I admitted. "I gotta figure it out ... before I go nuts."

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

CodeSOD: A Basic Mistake

Way back in 1964, people were starting to recgonize that computers were going to have a large impact on the world. There was not, at the time, very much prepackaged software, which meant if you were going to use a computer to do work, you were likely going to have to write your own programs. The tools to do that weren't friendly to non-mathematicians.

Thus, in 1964, was BASIC created, a language derived from experiments with languages like DOPE (The Dartmouth Oversimplified Programming Experiment). The goal was to be something easy, something that anyone could use.

In 1977, the TRS-80, the Commodore PET, and the Apple II all launched, putting BASIC into the hands of end users. But it's important to note that BASIC had already been seeing wide use for a decade on "big iron" systems, or more hobbyist systems, like the Altair 8800.

Today's submitter, Coyne, was but a humble student in 1977, and despite studying at a decent university, brand spanking new computers were a bit out of reach. Coyne was working with professors to write code to support papers, and using some dialect of BASIC on some minicomputer.

One of Coyne's peers had written a pile of code, and one simple segment didn't work. As it was just a loop to print out a series of numbers, it seemed like it should work, and work quite easily. But the programmer writing it couldn't get it to work. They passed it around to other folks in the department, and those folks also couldn't get it to work. What could possibly be wrong with this code?

3010 O = 45
3020 FOR K = 1 TO O
3030   PRINT K
3040 NEXT K

Now, it's worth noting, this particular dialect of BASIC didn't support long variable names- you could use a single letter, or you could use a letter and a number, and that was it. So the short variable names are not explicitly a problem here- that's just the stone tools which were available to programmers at the time.

For days, people kept staring at this block, trying to figure out what was wrong. Finally, Coyne took a glance, and in a moment was able to spot the problem.

I've done something nasty here, because I posted the correct block first. What the programmer had actually written was this:

3010 O = 45
3020 FOR K = 1 TO 0
3030   PRINT K
3040 NEXT K

The difference is subtle, especially when you're staring at a blurry CRT late at night in the computer lab, with too little coffee and too much overhead lighting. I don't know what device they were using for display; most terminals made sure to make O look different from 0, but I couldn't be bold enough to say all of them did. And, in this era, you'd frequently review code on printed paper, so who knows how it was getting printed out?

But that, in the end, was the problem- the programmer accidentally typed a zero where they meant the letter "O". And that one typo was enough to send an entire computer science department spinning for days when no one could figure it out.

In any case, it's interesting to see how an "easy" to use language once restricted variable names to such deep inscrutability.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

CodeSOD: A Truly Bad Comparison

For C programmers of a certain age (antique), booleans represent a frustrating challenge. But with the addition of stdbool.h, we exited the world of needing to work hard to interact with boolean values. While some gotchas are still in there, your boolean code has the opportunity to be simple.

Mark's predecessor saw how simple it made things, and decided that wouldn't do. So that person went and wrote their own special way of comparing boolean values. It starts with an enum:

typedef enum exop_t {
    EXOP_NONE,
    EXOP_AND,
    EXOP_OR,
    EXOP_EQUAL,
    EXOP_NOTEQUAL,
    EXOP_LT,
    EXOP_GT,
    EXOP_LEQUAL,
    EXOP_GEQUAL,
    EXOP_ADD,
    EXOP_SUBTRACT,
    EXOP_MULTIPLY,
    EXOP_DIV,
    EXOP_MOD,
    EXOP_NEGATE,
    EXOP_UNION,
    EXOP_FILTER1,
    EXOP_FILTER2
};

Yes, they did write an enum to compare booleans. They also wrote not one, but two functions. Let's start with the almost sane one.

static bool compare_booleans (bool bool1,
                              bool bool2,
                              exop_t  exop)
{

    int32_t  cmpresult;

    if ((bool1 && bool2) || (!bool1 && !bool2)) {
        cmpresult = 0;
    } else if (bool1) {
        cmpresult = 1;
    } else {
        cmpresult = -1;
    }

    return convert_compare_result(cmpresult, exop);

}

This function takes two boolean values, and a comparison we wish to perform. Then, we test if they're equal, though the way we do that is by and-ing them together, then or-ing that with the and of their negations. If they're equal, cmpresult is set to zero. If they're not equal, and the first boolean is true, we set cmpresult to one, and finally to negative one.

Thus, we're just invented strcmp for booleans.

But then we call another function, which is super helpful, because it turns that integer into a more normal boolean value.

static boolean
    convert_compare_result (int32_t cmpresult,
                            exop_t exop)
{
    switch (exop) {
    case EXOP_EQUAL:
        return (cmpresult) ? FALSE : TRUE;
    case EXOP_NOTEQUAL:
        return (cmpresult) ? TRUE : FALSE;
    case EXOP_LT:
        return (cmpresult < 0) ? TRUE : FALSE;
    case EXOP_GT:
        return (cmpresult > 0) ? TRUE : FALSE;
    case EXOP_LEQUAL:
        return (cmpresult <= 0) ? TRUE : FALSE;
    case EXOP_GEQUAL:
        return (cmpresult >= 0) ? TRUE : FALSE;
    default:
        printf( "ERR_INTERNAL_VAL\n" );
        return TRUE;
    }
} 

We switch based on the requested operation, and each case is its own little ternary. For equality comparisons, it requires a little bit of backwards logic- if cmpresult is non-zero (thus true), we need to return FALSE. Also note how our expression enum has many more options than convert_compare_result supports, making it very easy to call it wrong- and worse, it returns TRUE if you call it wrong.

At least they made booleans hard again. Who doesn't want to be confused about how to correctly check if two boolean values are the same?

It's worth noting that, for all this code, the rest of the code base never used anything but EXOP_EQUAL and EXOP_NOTEQUAL, because why would you do anything else on booleans? Every instance of compare_booleans could have been replaced with a much clearer == or != operator. Though what should really have been replaced was whoever wrote this code, preferably before they wrote it.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

A Government Data Center

Back in the antediluvian times, when I was in college, people still used floppy disks to work on their papers. This was a pretty untenable arrangement, because floppy disks lost data all the time, and few students had the wherewithal to make multiple copies. Half my time spent working helldesk was breaking out Norton Diskutils to try and rescue people's term papers. To avoid this, the IT department offered network shares where students could store documents. The network share was backed up, tracked versions, and could be accessed from any computer on campus, including the VAX system (in fact, it was stored on the VAX).

I bring this up because we have known for quite some time that companies and governments need to store documents in centrally accessible locations so that you're not reliant on end users correctly managing their files. And if you are a national government, you have to make a choice: either you contract out to a private sector company, or you do it yourself.

South Korea made the choice to do it themselves, with their G-Drive system (short for Government Drive, no relation to Google Drive), a government file store hosted primarily out of a datacenter in Daejeon. Unfortunately, "primarily" is a bit too apropos- last month, a fire in that datacenter destroyed data.

The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for external backups. This vulnerability ultimately left it unprotected.

Someone, somehow, designed a data storage system that was structurally incapable of doing backups? And then told 750,000 government employees that they should put all their files there?

Even outside of that backup failure, while other services had backups, they did not have a failover site, so when the datacenter went down, the government went down with it.

In total, it looks like about 858TB of data got torched. 647 distinct services were knocked out. At least 90 of them were reported to be unrecoverable (that last link is from a company selling Lithium Ion safety products, but is a good recap). A full recovery was, shortly after the accident, predicted to take a month, but as of October 22, only 60% of services had been restored.

Now, any kind of failure of this scale means heads must roll, and police investigations have gone down the path of illegal subcontracting. The claim is that the government hired a contractor broke the law by subcontracting the work, and that those subcontractors were unqualified for the work they were doing- that while they were qualified to install or remove a li-ion battery, they were not qualified to move one, which is what they were doing and resulted in the fire.

I know too little about Korean laws about government contracting and too little about li-ion battery management to weigh in on this. Certainly, high-storage batteries are basically bombs, and need to be handled with great care and protected well. Though, if one knows how to install and uninstall a battery, moving a battery seems covered in those steps.

But if I were doing a root cause analysis here, while that could be the root cause of the fire, it is not the root cause of the outage. If you build a giant datacenter but can't replicate services to another location, you haven't built a reliable cloud storage system- you've just built an expensive floppy disk that is one trip too close to a fridge magnet away from losing all of your work. In this case, the fridge magnet was made of fire, but the result is the same.

I'm not going to say this problem would have be easy to avoid; actually building resilient infrastructure that fails gracefully under extreme stress is hard. But while it's a hard problem, it's also a well-understood problem. There are best practices, and clearly not one of them was followed.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Conservapedia Still Exists

By: Nick Heer

I am not sure it is worth writing at length about Grokipedia, the Elon Musk-funded effort to quite literally rewrite history from the perspective of a robot taught to avoid facts upsetting to the U.S. far right. Perhaps it will be an unfortunate success — the Fox News of encyclopedias, giving ideologues comfortable information as they further isolate themselves.

It is less a Wikipedia competitor than it is a machine-generated alternative to Conservapedia. Founded by Andy Schlafly, an attorney and son of Phyllis Schlafly, the Wikipedia alternative was an attempt to make an online encyclopedia from a decidedly U.S. conservative and American exceptionalism perspective. Seventeen years ago, Schlafly’s effort was briefly profiled by Canadian television and, somehow, the site is still running. Perhaps that is the fate of Grokipedia: a brief curiosity, followed by traffic coming only from a self-selecting mix of weirdos and YouTubers needing material.

⌥ Permalink

A Profile of Setlist.fm

By: Nick Heer

Marc Hogan, New York Times (gift link):

Enter Setlist.fm. The wikilike site, where users document what songs artists play each night on tour, has grown into a vast archive, updated in real time but also reaching back into the historical annals. From the era of Mozart (seriously!) to last night’s Chappell Roan show, Setlist.fm offers reams of statistics — which songs artists play most often, when they last broke out a particular tune. In recent years, the site has begun posting data about average concert start times and set lengths.

Good profile. I had no idea it was owned by Live Nation.

I try to avoid Setlist.fm ahead of a show, but I check it immediately when I get home and for the days following. I might be less familiar with an artist’s catalogue, and this is particularly true of an opener, so it lets me track down particular songs that were played. It is one of the internet’s great resources.

⌥ Permalink

Zoom CEO Eric Yuan Lies About A.I. Leading to Shorter Work Weeks

By: Nick Heer

Sarah Perez, TechCrunch:

Zoom CEO Eric Yuan says AI will shorten our workweek

[…]

“Today, I need to manually focus on all those products to get work done. Eventually, AI will help,” Yuan said.

“By doing that, we do not need to work five days a week anymore, right? … Five years out, three days or four days [a week]. That’s a goal,” he said.

So far, technological advancements have not — in general — produced a shorter work week; that was a product of collective labour action. We have been promised a shorter week before. We do not need to carry water for people who peddle obvious lies. We will always end up being squeezed for greater output.

⌥ Permalink

Colorado Police Officer Caught on Doorbell Camera Talking About Surveillance Powers

By: Nick Heer

Andrew Kenney, Denverite:

It was Sgt. Jamie Milliman [at the door], a police officer with the Columbine Valley Police Department who covers the town of Bow Mar, which begins just south of [Chrisanna] Elser’s home.

[…]

“You know we have cameras in that jurisdiction and you can’t get a breath of fresh air, in or out of that place, without us knowing, correct?” he said.

“OK?” Elser, a financial planner in her 40s, responded in a video captured by her smart doorbell and viewed by Denverite.

“Just as an example,” the sergeant told her, she had “driven through 20 times the last month.”

This story is a civil liberties rollercoaster. Milliman was relying on a nearby town’s use of Flock license plate cameras and Ring doorbells — which may also be connected to the Flock network — to accuse Elser of theft and issue a summons. Elser was able to get the summons dropped by compiling evidence from, in part, the cameras and GPS system on her truck. Milliman’s threats were recorded by a doorbell camera, too. The whole thing is creepy, and all over a $25 package stolen off a doorstep.

I have also had things stolen from me, and I wish the police officers I spoke to had a better answer for me than shrugging their shoulders and saying, in effect, this is not worth our time. But this situation is like a parallel universe ad for Amazon and its Ring subsidiary. Is this the path toward “very close to zero[ing] out crime”? It is not worth it.

⌥ Permalink

Apple’s Tedious and Expensive Procedure for Replacing the Battery in the New MacBook Pro

By: Nick Heer

Carsten Frauenheim and Elizabeth Chamberlain, iFixit:

Apple’s official replacement process requires swapping the entire top case, keyboard and all, just to replace this single consumable component. And it has for a long time. That’s a massive and unreasonable job, requiring complete disassembly and reassembly of the entire device. We’re talking screws, shields, logic board, display, Touch ID, trackpad, everything. In fact, the only thing that doesn’t get transferred are the keyboard and speakers. The keyboard is more or less permanently affixed to this top aluminum, and the speakers are glued in — which, I guess, according to Apple means that the repair is out of the scope of DIY (we disagree).

At least one does not need to send in their laptop for a mere battery replacement. Still, I do not understand why this — the most predictable repair — is so difficult and expensive.

I hate to be that guy, but the battery for a mid-2007 15-inch MacBook Pro used to cost around $150 (about $220 inflation-adjusted) and could be swapped with two fingers. The official DIY solution for replacing the one in my M1 MacBook Pro is over $700, though there is a $124 credit for returning the replaced part. The old battery was, of course, a little bit worse: 60 watt-hours compared to 70 watt-hours in the one I am writing this with. I do not even mind the built-in-ness of this battery. But it should not cost an extra $500 and require swapping the rest of the top case parts.

[…] But for now, this tedious and insanely expensive process is the only offering they make for changing out a dead battery. Is it just a byproduct of this nearly half-a-decade-old chassis design, something that won’t change until the next rethink? We don’t know.

“Nearly half-a-decade-old” is a strange way of writing “four years”, almost like it is attempting to emphasize the age of this design. Four years old does not seem particularly ancient to me. I thought iFixit’s whole vibe was motivating people to avoid the consumerist churn encouraged by rapid redesigns.

⌥ Permalink

Reddit Sues Perplexity and Three Data Scraping Companies Because They Crawled Google

By: Nick Heer

Matt O’Brien, Associated Press:

Social media platform Reddit sued the artificial intelligence company Perplexity AI and three other entities on Wednesday, alleging their involvement in an “industrial-scale, unlawful” economy to “scrape” the comments of millions of Reddit users for commercial gain.

[…]

Also named in the lawsuit are Lithuanian data-scraping company Oxylabs UAB, a web domain called AWMProxy that Reddit describes as a “former Russian botnet,” and Texas-based startup SerpApi, which lists Perplexity as a customer on its website.

Mike Masnick, Techdirt:

Most reporting on this is not actually explaining the nuances, which require a deeper understanding of the law, but fundamentally, Reddit is NOT arguing that these companies are illegally scraping Reddit, but rather that they are illegally scraping… Google (which is not a party to the lawsuit) and in doing so violating the DMCA’s anti-circumvention clause, over content Reddit holds no copyright over. And, then, Perplexity is effectively being sued for linking to Reddit.

This is… bonkers on so many levels. And, incredibly, within their lawsuit, Reddit defends its arguments by claiming it’s filing this lawsuit to protect the open internet. It is not. It is doing the exact opposite.

I am glad Masnick wrote about this despite my disagreement with his views on how much control a website owner ought to have over scraping. This is a necessary dissection of the suit, though I would appreciate views on it from actual intellectual property lawyers. They might be able to explain how a positive outcome of this case for Reddit would have clear rules delineating this conduct from the ways in which artificial intelligence companies have so far benefitted from a generous reading of fair use and terms of service documents.

⌥ Permalink

Apple Threatens to Withdraw App Tracking Transparency in Europe

By: Nick Heer

Andrej Sokolow, Deutsche Presse Agentur:

Apple could switch off a function that prevents users’ apps from tracking their behaviour across various services and websites for advertising purposes in Germany and other European countries.

The iPhone manufacturer on Wednesday complained that it has experienced constant headwinds from the tracking industry.

“Intense lobbying efforts in Germany, Italy and other countries in Europe may force us to withdraw this feature to the detriment of European consumers,” Apple said in a statement.

It is a little rich for Apple to be claiming victimhood in the face of “intense lobbying efforts” by advertising companies when it is the seventh highest spender on lobbying in the European Union. Admittedly, it spends about one-third as much as Meta in Germany, but that is not because Apple cannot afford to spend more. Apple’s argument is weak.

In any case, this is another case where Apple believes it should have a quasi-regulatory role. As I wrote last month:

[…] Apple seems to believe it is its responsibility to implement technical controls to fulfill its definition of privacy and, if that impacts competition and compatibility, too bad. E.U. regulators seem to believe it has policy protections for user privacy, and that users should get to decide how their private data is shared.

I believe there are people within Apple who care deeply about privacy. However, when Apple also gets to define privacy and tracking, it is no coincidence it found an explanation allowing it to use platform activity and in-app purchases for ad targeting. This is hardly as sensitive as the tracking performed by Google and Meta, and Apple does not use third-party data for targeting.

But why would it? Apple owns the platform and, if it wanted, could exploit far more user information without it being considered “tracking” since it is all first-party data. That it does not is a positive reflection of self-policing and, ideally, something it will not change. But it could.

What E.U. authorities are concerned about is this self-serving definition of privacy and the self-policing that results, conflicting with the role of European regulators and privacy laws, and its effects on competition. I think those are reasonable grounds for questioning the validity of App Tracking Transparency. Furthermore, the consequences emanating from violations of privacy law are documented; Meta was penalized €1.2 billion as a result of GDPR violations. Potential violations of App Store policy, on the other hand, are handled differently. If Meta has, as a former employee alleges, circumvented App Tracking Transparency, would the penalties be handled by similar regulatory bodies, or would it — like Uber before — be dealt with privately and rather quietly?

The consequences of previous decisions have been frustrating. They result in poorer on-device privacy controls for users in part because Apple is a self-interested party. It would be able to make its case more convincingly if it walked away from the advertising business altogether.

Sokolow:

Apple argues that it has proposed various solutions to the competition authorities, but has not yet been able to dispel their concerns.

The company wants to continue to offer ATT to European users. However, it argued that the competition authorities have proposed complex solutions that would effectively undermine the function from Apple’s point of view.

Specificity would be nice. It would be better if these kinds of conversations could be had in public instead of in vague statements provided on background to select publications.

⌥ Permalink

The Verge Delivers a Bad Article About Amazon’s Ring

By: Nick Heer

Jennifer Pattison Tuohy, of the Verge, interviewed Ring founder Jamie Siminoff about a new book — which Tuohy has not read — written with Andrew Postman about the success of the company. During this conversation, Tuohy stumbled into Siminoff making a pretty outrageous claim:

While research suggests that today’s video doorbells do little to prevent crime, Siminoff believes that with enough cameras and with AI, Ring could eliminate most of it. Not all crime — “you’ll never stop crime a hundred percent … there’s crimes that are impossible to stop,” he concedes — but close.

“I think that in most normal, average neighborhoods, with the right amount of technology — not too crazy — and with AI, that we can get very close to zero out crime. Get much closer to the mission than I ever thought,” he says. “By the way, I don’t think it’s 10 years away. That’s in 12 to 24 months … maybe even within a year.”

If this sounds ridiculous to you, congratulations, you are thinking harder than whomever wrote the headline on this article:

Ring’s CEO says his cameras can almost ‘zero out crime’ within the next 12 months

The word “almost” and the phrase “very close” are working very hard to keep the core of Siminoff’s claim intact. What he says is that, by this time next year, “normal” communities with enough Ring cameras and a magic dusting of A.I. will have virtually no crime. The caveats are there to imply more nuance, but they are merely an escape hatch for when someone revisits this next year.

The near-complete elimination of crime in “normal” areas — whatever that means — will very obviously not happen. Tuohy cites a 2023 Scientific American story which, in turn, points to articles in MIT Technology Review and CNet. The first debunks a study Ring likes to promote claiming its devices drove a 55% decline in burglaries in Wilshire Park, Los Angeles in 2015, with cameras on about forty homes. Not only does the public data does not support this dramatic reduction, but:

Even if the doorbells had a positive effect, it seemed not to last. In 2017, Wilshire Park suffered more burglaries than in any of the previous seven years.

The CNet article collects a series of reports from other police departments indicating Ring cameras have questionable efficacy at deterring crime on a city-wide level.

This is also something we can know instinctually, since we already have plenty of surveillance cameras. A 2019 meta analysis (PDF) by Eric Piza, et al., found CCTV adoption decreased crime by about 13%. That is not nothing, but it is also a long way from nearly 100%. One could counter that these tests did not factor in Ring’s A.I. features, like summaries of what the camera saw — we have spent so much energy creating summary-making machines — and finding lost dogs.

The counterargument to all of this, however, is that Ring’s vision is a police state enforced by private enterprise. A 2022 paper (PDF) by Dan Calacci, et al., found race was, unsurprisingly, a motivating factor in reports of suspicious behaviour, and that reports within Ring’s Neighbors app was not correlated with the actual frequency of those crimes. Ring recently partnered with Flock, adding a further layer of creepiness.

I will allow that perhaps an article about Siminoff’s book is not the correct place to litigate these claims. By the very same logic, however, the Verge should be more cautious in publishing them, and should not have promoted them in a headline.

⌥ Permalink

App Store Restrictions Face Scrutiny in China, U.K.

By: Nick Heer

Liam Mo and Brenda Goh, Reuters:

A group of 55 Chinese iPhone and iPad users filed a complaint with China’s market regulator on Monday, a lawyer representing the group said, alleging that Apple abuses its market dominance by restricting app distribution and payments to its own platforms while charging high commissions.

[…]

This marks the second complaint against Apple led by Wang. A similar case filed in 2021 was dismissed by a Shanghai court last year.

Imran Rahman-Jones, BBC News:

But the Competition and Markets Authority (CMA) has designated both Apple and Google as having “strategic market status” – effectively saying they have a lot of power over mobile platforms.

The ruling has drawn fury from the tech giants, with Apple saying it risked harming consumers through “weaker privacy” and “delayed access to new features”, while Google called the decision “disappointing, disproportionate and unwarranted”.

The CMA said the two companies “may be limiting innovation and competition”.

Pretty soon it may be easier to list the significant markets in which Apple is still able to exercise complete control over iOS app distribution.

⌥ Permalink

OpenAI Launches ChatGPT Atlas

By: Nick Heer

Maxwell Zeff, TechCrunch:

OpenAI announced Tuesday the launch of its AI-powered browser, ChatGPT Atlas, a major step in the company’s quest to unseat Google as the main way people find information online.

The company says Atlas will first roll out on macOS, with support for Windows, iOS, and Android coming soon. OpenAI says the product will be available to all free users at launch.

Atlas, like Perplexity’s Comet, is a Chromium-based browser. You cannot use it without signing in to ChatGPT. As I was completing the first launch experience, shimmering colours radiated from the setup window and — no joke — it looked like my computer’s screen was failing.

OpenAI:

As you use Atlas, ChatGPT can get smarter and more helpful, too. Browser memories let ChatGPT remember context from the sites you visit and bring that context back when you need it. This means you can ask ChatGPT questions like: “Find all the job postings I was looking at last week and create a summary of industry trends so I can prepare for interviews.” Browser memories in Atlas are completely optional, and you’re always in control: you can view or archive them at any time in settings, and deleting browsing history deletes any associated browser memories.

I love the idea of this. So often, I need to track down something I remember reading, but have only the haziest recollection of what, exactly, it is. I want this in my life. Yet I have zero indication I can trust OpenAI with retaining and synthesizing useful information from my browsing history.

The company says it only retains pages until they have been summarized, and I am sure it thinks it is taking privacy as seriously as it can. But what about down the road? What could it do with all of this data it does retain — information that is tied to your ChatGPT account? OpenAI wants to be everywhere, and it wants to know everything about you to an even greater extent than Google or Meta have been able to accomplish. Why should I trust it? What makes the future of OpenAI look different than the trajectories of the information-hungry businesses before it?

⌥ Permalink

Federico Viticci’s M5 iPad Pro Review

By: Nick Heer

Even if you are not interested in the iPad or Apple product news generally, I recommend making time for Federico Viticci’s review, at MacStories, of the new iPad Pro. Apple claims 3.5× performance gains with A.I. models, so Viticci attempted to verify that number. Unfortunately, he ran into some problems.

Viticci (emphasis his):

This is the paradox of the M5. Theoretically speaking, the new Neural Accelerator architecture should lead to notable gains in token generation and prefill time that may be appreciated on macOS by developers and AI enthusiasts thanks to MLX (more on this below). However, all these improvements amount to very little on iPadOS today because there is no serious app ecosystem for local AI development and tinkering on iPad. That ecosystem absolutely exists on the Mac. On the iPad, we’re left with a handful of non-MLX apps from the App Store, no Terminal, and the untapped potential of the M5.

In case it’s not clear, I’m coming at this from a perspective of disappointment, not anger. […]

Viticci’s frustration with the state of A.I. models on the iPad Pro is palpable. Ideally and hopefully, it is a future-friendly system, but that is not usually the promise of Apple’s products. It usually likes to tell a complete story with the potential for sequels. To get even a glimpse of what that story looks like, Viticci had to go to great lengths, as documented in his review.

In the case of this iPad Pro, it is marketing leaps-and-bounds boosts in A.I. performance — though those claims appear to be optimistic — while still playing catch-up on last year’s Apple intelligence announcements, and offering little news for a user who wants to explore A.I. models directly on their iPad. It feels like a classic iPad story: incredible hardware, restricted by Apple’s software decisions.

Update: I missed a followup post from Viticci in which he points to a review from Max Weinbach of Creative Strategies. Weinbach found the M5 MacBook Pro does, indeed, post A.I. performance gains closer to Apple’s claims.

As an aside, I think it is curious for Apple to be supplying review units to Creative Strategies. It is nominally a research and analysis firm, not a media outlet. While there are concerns about the impartiality of reviewers granted access to prerelease devices, it feels to me like an entirely different thing for a broad-ranging research organization for reasons I cannot quite identify.

⌥ Permalink

Long Lines for Election Day in Alberta

By: Nick Heer

Ken MacGillivray and Karen Bartko, Global News:

“All electors are legislatively required to complete a Statement of Eligibility form (Form 13) at the voting station. This form is a declaration by an elector that they meet the required legislated criteria to receive and cast ballots,” Elections Edmonton said.

[…]

Those casting ballots say confirming voters are on the register or completing the necessary paperwork takes three to five minutes per voter.

I was lucky to be in and out of my polling place in about fifteen minutes, but the longest part was waiting for the person to diligently copy my name, address, and date-of-birth from my driver’s license to a triplicate form, immediately after confirming the same information on the printed voter roll. It is a silly requirement coming down as part of a larger unwanted package from our provincial government for no clear reason. The same legislation also prohibits electronic tabulation, so all the ballots are slowly being counted by hand. These are the kinds of measures that only begin to make sense if you assume someone with influence in our provincial government watches too much Fox News.

I wonder if our Minister of Red Tape Reduction has heard about all the new rules and restrictions implemented by his colleagues.

⌥ Permalink

The Blurry Future of Sora

By: Nick Heer

Jason Parham, Wired:

The uptick in artificial social networks, [Rudy] Fraser tells me, is being driven by the same tech egoists who have eroded public trust and inflamed social isolation through “divisive” algorithms. “[They] are now profiting on that isolation by creating spaces where folks can surround themselves with sycophantic bots.”

I saw this quote circulating on Bluesky over the weekend and it has been rattling around my head since. It cuts to the heart of one reason why A.I.-based “social” networks like Sora and Meta’s Vibes feel so uncomfortable.

Unfortunately, I found the very next paragraph from Parham uncompelling:

In the many conversations I had with experts, similar patterns of thought emerged. The current era of content production prioritizes aesthetics over substance. We are a culture hooked on optimization and exposure; we crave to be seen. We live on our phones and through our screens. We’re endlessly watching and being watched, submerged in a state of looking. With a sort of all-consuming greed, we are transforming into a visual-first society — an infinite form of entertainment for one another to consume, share, fight over, and find meaning through.

Of course our media reflects aesthetic trends and tastes; it always has. I do not know that there was a halcyon era of substance-over-style media, nor do I believe there was a time since celebrity was a feasible achievement in which at least some people did not desire it. In a 1948 British survey of children 10–15 years old, one-sixth to one-third of respondents aspired to “‘romantic’ [career] choices like film acting, sport, and the arts”. An article published in Scouting Magazine in 2000 noted children leaned toward high-profile careers — not necessarily celebrity, but jobs “every child is exposed to”. We love this stuff because we have always loved this stuff.

Among the bits I quibble with in the above, however, this stood out as a new and different thing: “[w]e’re endlessly watching and being watched”. That, I think, is the kind of big change Fraser is quoted as speaking about, and something I think is concerning. We already worried about echo chambers, and platforms like YouTube responded by adjusting recommendations to less frequently send users to dark places. Let us learn something, please.

Cal Newport:

A company that still believes that its technology was imminently going to run large swathes of the economy, and would be so powerful as to reconfigure our experience of the world as we know it, wouldn’t be seeking to make a quick buck selling ads against deep fake videos of historical figures wrestling. They also wouldn’t be entertaining the idea, ​as [Sam] Altman did last week​, that they might soon start offering an age-gated version of ChatGPT so that adults could enjoy AI-generated “erotica.”

To me, these are the acts of a company that poured tens of billions of investment dollars into creating what they hoped would be the most consequential invention in modern history, only to finally realize that what they wrought, although very cool and powerful, isn’t powerful enough on its own to deliver a new world all at once.

I do not think Sora smells of desperation, but I do think it is the product of a company that views unprecedented scale as its primary driver. I think OpenAI wants to be everywhere — and not in the same way that a consumer electronics company wants its smartphones to be the category’s most popular, or anything like that. I wonder if Ben Thompson’s view of OpenAI as “the Windows of A.I.” is sufficient. I think OpenAI is hoping to be a ubiquitous layer in our digital world; or, at least, it is behaving that way.

⌥ Permalink

I Bet Normal Users Will Figure Out Which Power Adapter to Buy

By: Nick Heer

John Gruber, responding to my exploration of the MacBook Pro A.C. adapter non-issue:

The problem I see with the MacBook power adapter situation in Europe is that while power users — like the sort of people who read Daring Fireball and Pixel Envy — will have no problem buying exactly the sort of power adapter they want, or simply re-using a good one they already own, normal users have no idea what makes a “good” power adapter. I suspect there are going to be a lot of Europeans who buy a new M5 MacBook Pro and wind up charging it with inexpensive low-watt power adapters meant for things like phones, and wind up with a shitty, slow charging experience.

Maybe. I think it is fair to be concerned about this being another thing people have to think about when buying a laptop. But, in my experience, less technically adept people still believe they need specific cables and chargers, even when they do not.

When I was in college, a friend forgot to bring the extension cable for their MacBook charger. There was an unused printer in the studio, though, so I was able to use the power cable from that because it is an interchangeable standard plug. I see this kind of thing all the time among friends, family members, and colleagues. It makes sense in a world frequently populated by proprietary adapters.

Maybe some people will end up with underpowered USB-C chargers. I bet a lot of people will just go to the Apple Store and buy the one recommended by staff, though.

⌥ Permalink

Latest Beta of Apple’s Operating Systems Adds Another Translucency Control

By: Nick Heer

Chance Miller, 9to5Mac:

You can find the new option [in 26.1 beta 4] on iPhone and iPad by going to the Settings app and navigating to the Display & Brightness menu. On the Mac, it’s available in the “Appearance” menu in System Settings. Here, you’ll see a new Liquid Glass menu with “Clear” and “Tinted” options.

“Choose your preferred look for Liquid Glass. Clear is more transparent, revealing the content beneath. Tinted increases opacity and adds more contrast,” Apple explains.

After Apple made the menu bar translucent in Mac OS X Leopard, it added a preference to make the bar solid after much pushback. When it refreshed the design of Mac OS X in Yosemite with more frosted glass effects, it added controls to Reduce Transparency and Increase Contrast, which replaced the menu bar-specific setting.

Here we are with yet another theme built around translucency, and more complaints about legibility and contrast — Miller writes “Apple says it heard from users throughout the iOS 26 beta testing period that they’d like a setting to manage the opaqueness of the Liquid Glass design”. Now, as has become traditional, there is another way to moderate the excesses of Apple’s new visual language. I am sure there are some who will claim this undermines the entire premise of Liquid Glass, and I do not know that they are entirely wrong. Some might call it greater personalization and customization, too. I think it feels unfocused. Apple keeps revisiting translucency and finding it needs to add more controls to compensate.

⌥ Permalink

NSO Group Banned From Using or Supplying WhatsApp Exploits

By: Nick Heer

Carly Nairn, Courthouse News Service:

U.S. District Judge Phyllis Hamilton said in a 25-page ruling that there was evidence NSO Group’s flagship spyware could still infiltrate WhatApp users’ devices and granted Meta’s request for a permanent injunction.

However, Hamilton, a Bill Clinton appointee, also determined that any damages would need to follow a ratioed amount of compensation based on a legal framework designed to proportion damages. She ordered that the jury-based award of $167 million should be reduced to a little over $4 million.

Once again, I am mystified by Apple’s decision to drop its suit against NSO Group. What Meta won is protection from WhatsApp being used as an installation vector for NSO’s spyware; importantly, high-value WhatsApp users won a modicum of protection from NSO’s customers. And, as John Scott-Railton of Citizen Lab points out, NSO has “an absolute TON of their business splashed all over the court records”. There are several depositions from which an enterprising journalist could develop a better understanding of this creepy spyware company.

Last week, NSO Group confirmed it had been acquired by U.S. investors. However, according to its spokesperson, its “headquarters and core operations remain in Israel [and] continues to be fully supervised and regulated by the relevant Israeli authorities”.

Lorenzo Franceschi-Bicchierai, TechCrunch:

NSO has long claimed that its spyware is designed to not target U.S. phone numbers, likely to avoid hurting its chances to enter the U.S. market. But the company was caught in 2021 targeting about a dozen U.S. government officials abroad.

Soon after, the U.S. Commerce Department banned American companies from trading with NSO by putting the spyware maker on the U.S. Entities List. Since then, NSO has tried to get off the U.S. government’s blocklist, as recently as May 2025, with the help of a lobbying firm tied to the Trump administration.

I have as many questions about what this change in ownership could mean for its U.S. relationship as I do about how it affects possible targets.

⌥ Permalink

Sponsor: Magic Lasso Adblock: Incredibly Private and Secure Safari Web Browsing

By: Nick Heer

My thanks to Magic Lasso Adblock for sponsoring Pixel Envy this week.

With over 5,000 five star reviews, Magic Lasso Adblock is simply the best ad blocker for your iPhone, iPad, and Mac.

Designed from the ground up to protect your privacy, Magic Lasso blocks all intrusive ads, trackers, and annoyances. It stops you from being followed by ads around the web and, with App Ad Blocking, it stops your app usage being harvested by ad networks.

So, join over 350,000 users and download Magic Lasso Adblock today.

⌥ Permalink

The New MacBook Pro Is €35 Less Expensive in E.U. Countries, Ships Without a Charger

By: Nick Heer

Are you outraged? Have you not heard? Apple updated its entry-level MacBook Pro with a new M5 chip, and across Europe, it does not ship with an A.C. adapter in the box as standard any more. It still comes with a USB-C to MagSafe cable, and you can add an adapter at checkout, but those meddling E.U. regulators have forced Apple to do something stupid and customer-unfriendly again. Right?

William Gallagher, of AppleInsider, gets it wrong:

Don’t blame Apple this time — if you’re in the European Union or the UK, your new M5 14-inch MacBook Pro or iPad Pro may cost you $70 extra because Apple isn’t allowed to bundle a charger.

First of all, the dollar is not the currency in any of these countries. Second, the charger in European countries is €65, which is more like $76 right now. Third, Apple is allowed to bundle an A.C. adapter, it just needs to offer an option to not include it. Fourth, and most important, is that the new MacBook Pro is less expensive in nearly every region in which the A.C. adapter is now a configure-to-order option — even after adding the adapter.

In Ireland, the MacBook Pro used to start at €1,949; it now starts at €1,849; in France, it was €1,899, and it is now €1,799. As mentioned, the adapter is €65, making these new Macs €35 less with a comparable configuration. The same is true in each Euro-currency country I checked: Germany, Italy, and Spain all received a €100 price cut if you do not want an A.C. adapter, and a €35 price cut if you do.

It is not just countries that use the Euro receiving cuts. In Norway, the new MacBook Pro starts at 2,000 krone less than the one it replaces, and a charger is 849 krone. In Hungary, it is 50,000 forint less, with a charger costing about 30,000 forint. There are some exceptions, too. In Switzerland, the new models are 50 francs less, but a charger is 59 francs. And in the U.K., there is no price adjustment, even though the charger is a configure-to-order option there, too.

Countries with a charger in the box, on the other hand, see no such price adjustment, at least for the ones I have checked. The new M5 model starts at the same price as the M4 it replaces in Canada, Japan, Singapore, and the United States. (For the sake of brevity and because not all of these pages have been recently crawled by the Internet Archive, I have not included links to each comparison. I welcome checking my work, however, and would appreciate an email if I missed an interesting price change.)

Maybe Apple was already planning a €100 price cut for these new models. The M4 was €100 less expensive than the M3 it replaced, for example, so it is plausible. That is something we simply cannot know. What we do know for certain is that these new MacBook Pros might not come with an A.C. adapter, but even if someone adds one at checkout, it still costs less in most places with this option.

Gallagher:

It doesn’t appear that Apple has cut prices of the MacBook Pro or iPad Pro to match, either. That can’t be proven, though, because at least with the UK, Apple generally does currency conversion just by swapping symbols.

It can be proven if you bother to put in thirty minutes’ work.

Joe Rossignol, of MacRumors, also gets it a little wrong:

According to the European Union law database, Apple could have let customers in Europe decide whether they wanted to have a charger included in the box or not, but the company has ultimately decided to not include one whatsoever: […]

A customer can, in fact, choose to add an A.C. adapter when they order their Mac.

⌥ Permalink

OpenAI and Nvidia Are at the Centre of a Trillion-Dollar Circular Investment Economy

By: Nick Heer

Tabby Kinder in New York and George Hammond, Financial Times:

OpenAI has signed about $1tn in deals this year for computing power to run its artificial intelligence models, commitments that dwarf its revenue and raise questions about how it can fund them.

Emily Forgash and Agnee Ghosh, Bloomberg:

For much of the AI boom, there have been whispers about Nvidia’s frenzied dealmaking. The chipmaker bolstered the market by pumping money into dozens of AI startups, many of which rely on Nvidia’s graphics processing units to develop and run their models. OpenAI, to a lesser degree, also invested in startups, some of which built services on top of its AI models. But as tech firms have entered a more costly phase of AI development, the scale of the deals involving these two companies has grown substantially, making it harder to ignore.

The day after Nvidia and OpenAI announced their $100 billion investment agreement, OpenAI confirmed it had struck a separate $300 billion deal with Oracle to build out data centers in the US. Oracle, in turn, is spending billions on Nvidia chips for those facilities, sending money back to Nvidia, a company that is emerging as one of OpenAI’s most prominent backers.

I possess none of the skills most useful to understand what all of this means. I am not an economist; I did not have a secret life as an investment banker. As a layperson, however, it is not comforting to read from some People With Specialized Knowledge that this is similar to historically good circular investments, just at an unprecedented scale, while other People With Specialized Knowledge say this has been the force preventing the U.S. from entering a recession. These articles might be like one of those prescient papers from before the Great Recession. Not a great feeling.

⌥ Permalink

The New ‘Foreign Influence’ Scare

By: Nick Heer

Emmanuel Maiberg, 404 Media:

Democratic U.S. Senators Richard Blumenthal and Elizabeth Warren sent letters to the Department of Treasury Secretary Scott Bessent and Electronic Arts CEO Andrew Wilson, raising concerns about the $55 billion acquisition of the giant American video game company in part by Saudi Arabia’s Public Investment Fund (PIF).

Specifically, the Senators worry that EA, which just released Battlefield 6 last week and also publishes The Sims, Madden, and EA Sports FC, “would cease exercising editorial and operational independence under the control of Saudi Arabia’s private majority ownership.”

“The proposed transaction poses a number of significant foreign influence and national security risks, beginning with the PIF’s reputation as a strategic arm of the Saudi government,” the Senators wrote in their letter. […]

In the late 1990s and early 2000s, the assumption was that it would be democratic nations successfully using the web for global influence. But I think the 2016 U.S. presidential election, during which Russian operatives worked to sway voters’ intentions, was a reality check. Fears of foreign influence were then used by U.S. lawmakers to justify banning TikTok, and to strongarm TikTok into allowing Oracle to oversee its U.S. operations. Now, it is Saudi Arabian investment in Electronic Arts raising concerns. Like TikTok, it is not the next election that is, per se, at risk, but the general thoughts and opinions of people in the United States.

U.S. politicians even passed a law intended to address “foreign influence” concerns. However, Saudi Arabia is not one of the four “covered nations” restricted by PAFACA.

Aside from xenophobia, I worry “foreign influence” is becoming a new standard excuse for digital barriers. We usually associate restrictive internet policies with oppressive and authoritarian regimes that do not trust their citizens to be able to think for themselves. This is not to say foreign influence is not a reasonable concern, nor that Saudi Arabia has no red flags, nor still that these worries are a purely U.S. phenomenon. Canadian officials are similarly worried about adversarial government actors covertly manipulating our policies and public opinion. But I think we need to do better if we want to support a vibrant World Wide Web. U.S. adversaries are allowed to have big, successful digital products, too.

⌥ Permalink

My flailing around with Firefox's Multi-Account Containers

By: cks

I have two separate Firefox environments. One of them is quite locked down so that it blocks JavaScript by default, doesn't accept cookies, and so on. Naturally this breaks a lot of things, so I have a second "just make it work" environment that runs all the JavaScript, accepts all the cookies, and so on (although of course I use uBlock Origin, I'm not crazy). This second environment is pretty risky in the sense that it's going to be heavily contaminated with tracking cookies and so on, so to mitigate the risk (and make it a better environment to test things in), I have this Firefox set to discard cookies, caches, local storage, history, and so on when it shuts down.

In theory how I use this Firefox is that I start it when I need to use some annoying site I want to just work, use the site briefly, and then close it down, flushing away all of the cookies and so on. In practice I've drifted into having a number of websites more or less constantly active in this "accept everything" Firefox, which means that I often keep it running all day (or longer at home) and all of those cookies stick around. This is less than ideal, and is a big reason why I wish Firefox had a 'open this site in a specific profile' feature. Yesterday, spurred on by Ben Zanin's Fediverse comment, I decided to make my "accept everything" Firefox environment more complicated in the pursuit of doing better (ie, throwing away at least some cookies more often).

First, I set up a combination of Multi-Account Containers for the basic multi-container support and FoxyTab to assign wildcarded domains to specific containers. My reason to use Multi-Account Containers and to confine specific domains to specific containers is that both M-A C itself and my standard Cookie Quick Manager add-on can purge all of the cookies and so on for a specific container. In theory this lets me manually purge undesired cookies, or all cookies except desired ones (for example, my active Fediverse login). Of course I'm not likely to routinely manually delete cookies, so I also installed Cookie AutoDelete with a relatively long timeout and with its container awareness turned on, and exemptions configured for the (container-confined) sites that I'm going to want to retain cookies from even when I've closed their tab.

(It would be great if Cookie AutoDelete supported different cookie timeouts for different containers. I suspect it's technically possible, along with other container-aware cookie deletion, since Cookie AutoDelete applies different retention policies in different containers.)

In FoxyTab, I've set a number of my containers to 'Limit to Designated Sites'; for example, my 'Fediverse' container is set this way. The intention is that when I click on an external link in a post while reading my Fediverse feed, any cookies that external site sets don't wind up in the Fediverse container; instead they go either in the default 'no container' environment or in any specific container I've set up for them. As part of this I've created a 'Cookie Dump' container that I've assigned as the container for various news sites and so on where I actively want a convenient way to discard all their cookies and data (which is available through Multi-Account Containers).

Of course if you look carefully, much of this doesn't really require Multi-Account Containers and FoxyTab (or containers at all). Instead I could get almost all of this just by using Cookie AutoDelete to clean out cookies from closed sites after a suitable delay. Containers do give me a bit more isolation between the different things I'm using my "just make it work" Firefox for, and maybe that's important enough to justify the complexity.

(I still have this Firefox set to discard everything when it exits. This means that I have to re-log-in every so often even for the sites where I have Cookie AutoDelete keep cookies, but that's fine.)

I wish Firefox Profiles supported assigning websites to profiles

By: cks

One of the things that Firefox is working on these days is improving Firefox's profiles feature so that it's easier to use them. Firefox also has an existing feature that is similar to profiles, in containers and the Multi-Account Containers extension. The reason Firefox is tuning up profiles is that containers only separate some things, while profiles separate pretty much everything. A profile has a separate set of about:config settings, add-ons, add-on settings, memorized logins, and so on. I deliberately use profiles to create two separate and rather different Firefox environments. I'd like to have at least two or three more profiles, but one reason I've been lazy is that the more profiles I have, the more complex getting URLs into the right profile is (even with tooling to help).

This leads me to my wish for profiles, which is for profiles to support the kind of 'assign website to profile' and 'open website in profile' features that you currently have with containers, especially with the Multi-Account Containers extension. Actually I would like a somewhat better version than Multi-Account Containers currently offers, because as far as I can see you can't currently say 'all subdomains under this domain should open in container X' and that's a feature I very much want for one of my use cases.

(Multi-Account Containers may be able to do wildcarded subdomains with an additional add-on, but on the other hand apparently it may have been neglected or abandoned by Mozilla.)

Another way to get much of what I want would be for some of my normal add-ons to be (more) container aware. I could get a lot of the benefit of profiles (although not all of them) by using Multi-Account Containers with container aware cookie management in, say, Cookie AutoDelete (which I believe does support that, although I haven't experimented). Using containers also has the advantage that I wouldn't have to maintain N identical copies of my configuration for core extensions and bookmarklets and so on.

(I'm not sure what you can copy from one profile to a new one, and you currently don't seem to get any assistance from Firefox for it, at least in the old profile interface. This is another reason I haven't gone wild on making new Firefox profiles.)

Modern Linux filesystem mounts are rather complex things

By: cks

Once upon a time, Unix filesystem mounts worked by putting one inode on top of another, and this was also how they worked in very early Linux. It wasn't wrong to say that mounts were really about inodes, with the names only being used to find the inodes. This is no longer how things work in Linux (and perhaps other Unixes, but Linux is what I'm most familiar with for this). Today, I believe that filesystem mounts in Linux are best understood as namespace operations.

Each separate (unmounted) filesystem is a a tree of names (a namespace). At a broad level, filesystem mounts in Linux take some name from that filesystem tree and project it on top of something in an existing namespace, generally with some properties attached to the projection. A regular conventional mount takes the root name of the new filesystem and puts the whole tree somewhere, but for a long time Linux's bind mounts took some other name in the filesystem as their starting point (what we could call the root inode of the mount). In modern Linux, there can also be multiple mount namespaces in existence at one time, with different contents and properties. A filesystem mount does not necessarily appear in all of them, and different things can be mounted at the same spot in the tree of names in different mount namespaces.

(Some mount properties are still global to the filesystem as a whole, while other mount properties are specific to a particular mount. See mount(2) for a discussion of general mount properties. I don't know if there's a mechanism to handle filesystem specific mount properties on a per mount basis.)

This can't really be implemented with an inode-based view of mounts. You can somewhat implement traditional Linux bind mounts with an inode based approach, but mount namespaces have to be separate from the underlying inodes. At a minimum a mount point must be a pair of 'this inode in this namespace has something on top of it', instead of just 'this inode has something on top of it'.

(A pure inode based approach has problems going up the directory tree even in old bind mounts, because the parent directory of a particular directory depends on how you got to the directory. If /usr/share is part of /usr and you bind mounted /usr/share to /a/b, the value of '..' depends on if you're looking at '/usr/share/..' or '/a/b/..', even though /usr/share and /a/b are the same inode in the /usr filesystem.)

If I'm reading manual pages correctly, Linux still normally requires the initial mount of any particular filesystem be of its root name (its true root inode). Only after that initial mount is made can you make bind mounts to pull out some subset of its tree of names and then unmount the original full filesystem mount. I believe that a particular filesystem can provide ways to sidestep this with a filesystem specific mount option, such as btrfs's subvol= mount option that's covered in the btrfs(5) manual page (or 'btrfs subvolume set-default').

You can add arbitrary zones to NSD (without any glue records)

By: cks

Suppose, not hypothetically, that you have a very small DNS server for a captive network situation, where the DNS server exists only to give clients answers for a small set of hosts. One of the ways you can implement this is with an authoritative DNS servers, such as NSD, that simply has an extremely minimal set of DNS data. If you're using NSD for this, you might be curious how minimal you can be and how much you need to mimic ordinary DNS structure.

Here, by 'mimic ordinary DNS structure', I mean inserting various levels of NS records so there is a more or less conventional path of NS delegations from the DNS root ('.') down to your name. If you're providing DNS clients with 'dog.example.org', you might conventionally have a NS record for '.', a NS record for 'org.', and a NS record for 'example.org.', mimicking what you'd see in global DNS. Of course all of your NS records are going to point to your little DNS server, but they're present if anything looks.

Perhaps unsurprisingly, NSD doesn't require this and DNS clients normally don't either. If you say:

zone:
  name: example.org
  zonefile: example-stub

and don't have any other DNS data, NSD won't object and it will answer queries for 'dog.example.org' with your minimal stub data. This works for any zone, including completely made up ones:

zone:
  name: beyond.internal
  zonefile: beyond-stub

The actual NSD stub zone files can be quite minimal. An older OpenBSD NSD appears to be happy with zone files that have only a $ORIGIN, a $TTL, a '@ IN SOA' record, and what records you care about in the zone.

Once I thought about it, I realized I should have expected this. An authoritative DNS server normally only holds data for a small subset of zones and it has to be willing to answer queries about the data it holds. Some authoritative DNS servers (such as Bind) can also be used as resolving name servers so they'd sort of like to have information about at least the root nameservers, but NSD is a pure authoritative server so there's no reason for it to care.

As for clients, they don't normally do DNS resolution starting from the root downward. Instead, they expect to operate by sending the entire query to whatever their configured DNS resolver is, which is going to be your little NSD setup. In a number of configurations, clients either can't talk directly to outside DNS or shouldn't try to do DNS resolution that way because it won't work; they need to send everything to their configured DNS resolver so it can do, for example, "split horizon" DNS.

(Yes, the modern vogue for DNS over HTTPS puts a monkey wrench into split horizon DNS setups. That's DoH's problem, not ours.)

Since this works for a .net zone, you can use it to try to disable DNS over HTTPS resolvers in your stub DNS environment by providing a .net zone with 'use-application-dns CNAME .' or the like, to trigger at least Firefox's canary domain detection.

(I'm not going to address whether you should have such a minimal stub DNS environment or instead count on your firewall to block traffic and have a normal DNS environment, possibly with split horizon or response policy zones to introduce your special names.)

Some of the things that ZFS scrubs will detect

By: cks

Recently I saw a discussion of my entry on how ZFS scrubs don't really check the filesystem structure where someone thought that ZFS scrubs only protected you from the disk corrupting data at rest, for example due to sectors starting to fail (here). While ZFS scrubs have their limits, they do manage to check somewhat more than this.

To start with, ZFS scrubs check the end to end hardware path for reading all your data (and implicitly for writing it). There are a variety of ways that things in the hardware path can be unreliable; for example, you might have slowly failing drive cables that are marginal and sometimes give you errors on data reads (or worse, data writes). A ZFS scrub has some chance to detect this; if a ZFS scrub passes, you know that as of that point in time you can reliably read all your data from all your disks and that all the data was reliably written.

If a scrub passes, you also know that the disks haven't done anything obviously bad with your data. This can be important if you're doing operations that you consider somewhat exotic, such as telling SSDs to discard unused sectors. If you have ZFS send TRIM commands to a SSD and then your scrub passes, you know that the SSD didn't incorrectly discard some sectors that were actually used.

Related to this, if you do a ZFS level TRIM and then the scrub passes, you know that ZFS itself didn't send TRIM commands that told the SSD to discard sectors that were actually used. In general, if ZFS has a serious problem where it writes the wrong thing to the wrong place, a scrub will detect it (although the scrub can't fix it). Similarly, a scrub will detect if a disk itself corrupted the destination of a write (or a read), or if things were corrupted somewhere in the lower level software and hardware path of the write.

There are a variety of ZFS level bugs that could theoretically write the wrong thing to the wrong place, or do something that works out to the same effect. ZFS could have a bug in free space handling (so that it incorrectly thinks some in use sectors are free and overwrites them), or it could write too much or too little, or it could correctly allocate and write data but record the location of the data incorrectly in higher level data structures, or it could accidentally not do a write (for example, if it's supposed to write a duplicate copy of some data but forgets to actually issue the IO). ZFS scrubs can detect all of these issues under the right circumstances.

(To a limited extent a ZFS scrub also checks the high level metadata of filesystems and snapshots. since it has to traverse that metadata to find the object set for each dataset and similar things. Since a scrub just verifies checksums, this won't cross check dataset level metadata like information on how much data was written in each snapshot, or the space usage.)

What little I want out of web "passkeys" in my environment

By: cks

WebAuthn is yet another attempt to do an API for web authentication that doesn't involve passwords but that instead allows browsers, hardware tokens, and so on to do things more securely. "Passkeys" (also) is the marketing term for a "WebAuthn credential", and an increasing number of websites really, really want you to use a passkey for authentication instead of any other form of multi-factor authentication (they may or may not still require your password).

Most everyone that wants you to use passkeys also wants you to specifically use highly secure ones. The theoretically most secure are physical hardware security keys, followed by passkeys that are stored and protected in secure enclaves in various ways by the operating system (provided that the necessary special purpose hardware is available). Of course the flipside of 'secure' is 'locked in', whether locked in to your specific hardware key (or keys, generally you'd better have backups) or locked in to a particular vendor's ecosystem because their devices are the only ones that can possibly use your encrypted passkey vault.

(WebAuthn neither requires nor standardizes passkey export and import operations, and obviously security keys are built to not let anyone export the cryptographic material from them, that's the point.)

I'm extremely not interested in the security versus availability tradeoff that passkeys make in favour of security. I care far more about preserving availability of access to my variety of online accounts than about nominal high security. So if I'm going to use passkeys at all, I have some requirements:

Linux people: is there a passkeys implementation that does not use physical hardware tokens (software only), is open source, works with Firefox, and allows credentials to be backed up and copied to other devices by hand, without going through some cloud service?

I don't think I'm asking for much, but this is what I consider the minimum for me actually using passkeys. I want to be 100% sure of never losing them because I have multiple backups and can use them on multiple machines.

Apparently KeePassXC more or less does what I want (when combined with its Firefox extension), and it can even export passkeys in a plain text format (well, JSON). However, I don't know if anything else can ingest those plain text passkeys, and I don't know if KeePassXC can be told to only do passkeys with the browser and not try to take over passwords.

(But at least a plain text JSON backup of your passkeys can be imported into another KeePassXC instance without having to try to move, copy, or synchronize a KeePassXC database.)

Normally I would ignore passkeys entirely, but an increasing number of websites are clearly going to require me to use some form of multi-factor authentication, no matter how stupid this is (cf), and some of them will probably require passkeys or at least make any non-passkey option very painful. And it's possible that reasonably integrated passkeys will be a better experience than TOTP MFA with my janky minimal setup.

(Of course KeePassXC also supports TOTP, and TOTP has an extremely obvious import process that everyone supports, and I believe KeePassXC will export TOTP secrets if you ask nicely.)

While KeePassXC is okay, what I would really like is for Firefox to support 'memorized passkeys' right along with its memorized passwords (and support some kind of export and import along with it). Should people use them? Perhaps not. But it would put that choice firmly in the hands of the people using Firefox, who could decide on how much security they did or didn't want, not in the hands of websites who want to force everyone to face a real risk of losing their account so that the website can conduct security theater.

(Firefox will never support passkeys this way for an assortment of reasons. At most it may someday directly use passkeys through whatever operating system services expose them, and maybe Linux will get a generic service that works the way I want it to. Nor is Firefox ever going to support 'memorized TOTP codes'.)

Two reasons why Unix traditionally requires mount points to exist

By: cks

Recently on the Fediverse, argv minus one asked a good question:

Why does #Linux require #mount points to exist?

And are there any circumstances where a mount can be done without a pre-existing mount point (i.e. a mount point appears out of thin air)?

I think there is one answer for why this is a good idea in general and otherwise complex to do, although you can argue about it, and then a second historical answer based on how mount points were initially implemented.

The general problem is directory listings. We obviously want and need mount points to appear in readdir() results, but in the kernel, directory listings are historically the responsibility of filesystems and are generated and returned in pieces on the fly (which is clearly necessary if you have a giant directory; the kernel doesn't read the entire thing into memory and then start giving your program slices out of it as you ask). If mount points never appear in the underlying directory, then they must be inserted at some point in this process. If mount points can sometimes exist and sometimes not, it's worse; you need to somehow keep track of which ones actually exist and then add the ones that don't at the end of the directory listing. The simplest way to make sure that mount points always exist in directory listings is to require them to have an existence in the underlying filesystem.

(This was my initial answer.)

The historical answer is that in early versions of Unix, filesystems were actually mounted on top of inodes, not directories (or filesystem objects). When you passed a (directory) path to the mount(2) system call, all it was used for was getting the corresponding inode, which was then flagged as '(this) inode is mounted on' and linked (sort of) to the new mounted filesystem on top of it. All of the things that dealt with mount points and mounted filesystem did so by inode and inode number, with no further use of the paths and the root inode of the mounted filesystem being quietly substituted for the mounted-on inode. All of the mechanics of this needed the inode and directory entry for the name to actually exist (and V7 required the name to be a directory).

I don't think modern kernels (Linux or otherwise) still use this approach to handling mounts, but I believe it lingered on for quite a while. And it's a sufficiently obvious and attractive implementation choice that early versions of Linux also used it (see the Linux 0.96c version of iget() in fs/inode.c).

Sidebar: The details of how mounts worked in V7

When you passed a path to the mount(2) system call (called 'smount()' in sys/sys3.c), it used the name to get the inode and then set the IMOUNT flag from sys/h/inode.h on it (and put the mount details in a fixed size array of mounts, which wasn't very big). When iget() in sys/iget.c was fetching inodes for you and you'd asked for an IMOUNT inode, it gave you the root inode of the filesystem instead, which worked in cooperation with name lookup in a directory (the name lookup in the directory would find the underlying inode number, and then iget() would turn it into the mounted filesystem's root inode). This gave Research Unix a simple, low code approach to finding and checking for mount points, at the cost of pinning a few more inodes into memory (not necessarily a small thing when even a big V7 system only had at most 200 inodes in memory at once, but then a big V7 system was limited to 8 mounts, see h/param.h).

We can't really do progressive rollouts of disruptive things

By: cks

In a comment on my entry on how we reboot our machines right after updating their kernels, Jukka asked a good question:

While I do not know how many machines there are in your fleet, I wonder whether you do incremental rolling, using a small snapshot for verification before rolling out to the whole fleet?

We do this to some extent but we can't really do it very much. The core problem is that the state of almost all of our machines is directly visible and exposed to people. This is because we mostly operate an old fashioned Unix login server environment, where people specifically use particular servers (either directly by logging in to them or implicitly because their home directory is on a particular NFS fileserver). About the only genuinely generic machines we have are the nodes in our SLURM cluster, where we can take specific unused nodes out of service temporarily without anyone noticing.

(Some of these login servers in use all of the time; others we might find idle if we're extremely lucky. But it's hard to predict when someone will show up to try to use a currently empty server.)

This means that progressively rolling out a kernel update (and rebooting things) to our important, visible core servers requires multiple people-visible reboots of machines, instead of one big downtime when everything is rebooted. Generally we feel that repeated disruptions are much more annoying and disruptive overall to people; it's better to get the pain of reboot disruptions over all at once. It's also much easier to explain to people, and we don't have to annoy them with repeated notifications that yet another subset of our servers and services will be down for a bit.

(To make an incremental deployment more painful for us, these will normally have to be after-hours downtimes, which means that we'll be repeatedly staying late, perhaps once a week for three or four weeks as we progressively work through a rollout.)

In addition to the nodes of our SLURM cluster, there are a number of servers that can be rebooted in the background to some degree without people noticing much. We will often try the kernel update out on a few of them in advance, and then update others of them earlier in the day (or the day before) both as a final check and to reduce the number of systems we have to cover at the actual out of hours downtime. But a lot of our servers cannot really be tested much in advance, such as our fileservers or our web server (which is under constant load for reasons outside the scope of this entry). We can (and do) update a test fileserver or a test web server, but neither will see a production load and it's under production loads that problems are most likely to surface.

This is a specific example of how the 'cattle' model doesn't fit all situations. To have a transparent rolling update that involves reboots (or anything else that's disruptive on a single machine), you need to be able to transparently move people off of machines and then back on to them. This is hard to get in any environment where people have long term usage of specific machines, where they have login sessions and running compute jobs and so on, and where you have have non-redundant resources on a single machine (such as NFS fileservers without transparent failover from server to server).

We don't update kernels without immediately rebooting the machine

By: cks

I've mentioned this before in passing (cf, also) but today I feel like saying it explicitly: our habit with all of our machines is to never apply a kernel update without immediately rebooting the machine into the new kernel. On our Ubuntu machines this is done by holding the relevant kernel packages; on my Fedora desktops I normally run 'dnf update --exclude "kernel*"' unless I'm willing to reboot on the spot.

The obvious reason for this is that we want to switch to the new kernel under controlled, attended conditions when we'll be able to take immediate action if something is wrong, rather than possibly have the new kernel activate at some random time without us present and paying attention if there's a power failure, a kernel panic, or whatever. This is especially acute on my desktops, where I use ZFS by building my own OpenZFS packages and kernel modules. If something goes wrong and the kernel modules don't load or don't work right, an unattended reboot can leave my desktops completely unusable and off the network until I can get to them. I'd rather avoid that if possible (sometimes it isn't).

(In general I prefer to reboot my Fedora machines with me present because weird things happen from time to time and sometimes I make mistakes, also.)

The less obvious reason is that when you reboot a machine right after applying a kernel update, it's clear in your mind that the machine has switched to a new kernel. If there are system problems in the days immediately afterward the update, you're relatively likely to remember this and at least consider the possibility that the new kernel is involved. If you apply a kernel update, walk away without rebooting, and the machine reboots a week and a half later for some unrelated reason, you may not remember that one of the things the reboot did was switch to a new kernel.

(Kernels aren't the only thing that this can happen with, since not all system updates and changes take effect immediately when made or applied. Perhaps one should reboot after making them, too.)

I'm assuming here that your Linux distribution's package management system is sensible, so there's no risk of losing old kernels (especially the one you're currently running) merely because you installed some new ones but didn't reboot into them. This is how Debian and Ubuntu behave (if you don't 'apt autoremove' kernels), but not quite how Fedora's dnf does it (as far as I know). Fedora dnf keeps the N most recent kernels around and probably doesn't let you remove the currently running kernel even if it's more than N kernels old, but I don't believe it tracks whether or not you've rebooted into those N kernels and stretches the N out if you haven't (or removes more recent installed kernels that you've never rebooted into, instead of older kernels that you did use at one point).

PS: Of course if kernel updates were perfect this wouldn't matter. However this isn't something you can assume for the Linux kernel (especially as patched by your distribution), as we've sometimes seen. Although big issues like that are relatively uncommon.

We (I) need a long range calendar reminder system

By: cks

About four years ago I wrote an entry about how your SMART drive database of attribute meanings needs regular updates. That entry was written on the occasion of updating the database we use locally on our Ubuntu servers, and at the time we were using a mix of Ubuntu 18.04 and Ubuntu 20.04 servers, both of which had older drive databases that probably dated from early 2018 and early 2020 respectively. It is now late 2025 and we use a mix of Ubuntu 24.04 and 22.04 servers, both of which have drive databases that are from after October of 2021.

Experienced system administrators know where this one is going: today I updated our SMART drive database again, to a version of the SMART database that was more recent than the one shipped with 24.04 instead of older than it.

It's a fact of life that people forget things. People especially forget things that are a long way away, even if they make little notes in their worklog message when recording something that they did (as I did four years ago). It's definitely useful to plan ahead in your documentation and write these notes, but without an external thing to push you or something to explicitly remind you, there's no guarantee that you'll remember.

All of which leads me to the view that it would be useful for us to have a long range calendar reminder system, something that could be used to set reminders for more than a year into the future and ideally allow us to write significant email messages to our future selves to cover all of the details (although there are hacks around that, such as putting the details on a web page and having the calendar mail us a link). Right now the best calendar reminder system we have is the venerable calendar, which we can arrange to email one-line notes to our general address that reaches all sysadmins, but calendar doesn't let you include the year in the reminder date.

(For SMART drive database updates, we could get away with mailing ourselves once a year in, say, mid-June. It doesn't hurt to update the drive database more than every Ubuntu LTS release. But there are situations where a reminder several years in the future is what we want.)

PS: Of course it's not particularly difficult to build an ad-hoc script system to do this, with various levels of features. But every local ad-hoc script that we write is another little bit of overhead, and I'd like to avoid that kind of thing if at all possible in favour of a standard solution (that isn't a shared cloud provider calendar).

We need to start doing web blocking for non-technical reasons

By: cks

My sense is that for a long time, technical people (system administrators, programmers, and so on) have seen the web as something that should be open by default and by extension, a place where we should only block things for 'technical' reasons. Common technical reasons are a harmful volume of requests or clear evidence of malign intentions, such as probing for known vulnerabilities. Otherwise, if it wasn't harming your website and wasn't showing any intention to do so, you should let it pass. I've come to think that in the modern web this is a mistake, and we need to be willing to use blocking and other measures for 'non-technical' reasons.

The core problem is that the modern web seems to be fragile and is kept going in large part by a social consensus, not technical things such as capable software and powerful servers. However, if we only react to technical problems, there's very little that preserves and reinforces this social consensus, as we're busy seeing. With little to no consequences for violating the social consensus, bad actors are incentivized to skate right up to and even over the line of causing technical problems. When we react by taking only narrow technical measures, we tacitly reward the bad actors for their actions; they can always find another technical way. They have no incentive to be nice or to even vaguely respect the social consensus, because we don't punish them for it.

So I've come to feel that if something like the current web is to be preserved, we need to take action not merely when technical problems arise but also when the social consensus is violated. We need to start blocking things for what I called editorial reasons. When software or people do things that merely shows bad manners and doesn't yet cause us technical problems, we should still block it, either soft (temporarily, perhaps with HTTP 429 Too Many Requests) or hard (permanently). We need to take action to create the web that we want to see, or we aren't going to get it or keep it.

To put it another way, if we want to see good, well behaved browsers, feed readers, URL fetchers, crawlers, and so on, we have to create disincentives for ones that are merely bad (as opposed to actively damaging). In its own way, this is another example of the refutation of Postel's Law. If we accept random crap to be friendly, we get random crap (and the quality level will probably trend down over time).

To answer one potential criticism, it's true that in some sense, blocking and so on for social reasons is not good and is in some theoretical sense arguably harmful for the overall web ecology. On the other hand, the current unchecked situation itself is also deeply harmful for the overall web ecology and it's only going to get worse if we do nothing, with more and more things effectively driven off the open web. We only get to pick the poison here.

I wish SSDs gave you CPU performance style metrics about their activity

By: cks

Modern CPUs have an impressive collection of performance counters for detailed, low level information on things like cache misses, branch mispredictions, various sorts of stalls, and so on; on Linux you can use 'perf list' to see them all. Modern SSDs (NVMe, SATA, and SAS) are all internally quite complex, and their behavior under load depends on a lot of internal state. It would be nice to have CPU performance counter style metrics to expose some of those details. For a relevant example that's on my mind (cf), it certainly would be interesting to know how often flash writes had to stall while blocks were hastily erased, or the current erase rate.

Having written this, I checked some of our SSDs (the ones I'm most interested in at the moment) and I see that our SATA SSDs do expose some of this information as (vendor specific) SMART attributes, with things like 'block erase count' and 'NAND GB written' to TLC or SLC (as well as the host write volume and so on stuff you'd expect). NVMe does this in a different way that doesn't have the sort of easy flexibility that SMART attributes do, so a random one of ours that I checked doesn't seem to provide this sort of lower level information.

It's understandable that SSD vendors don't necessarily want to expose this sort of information, but it's quite relevant if you're trying to understand unusual drive performance. For example, for your workload do you need to TRIM your drives more often, or do they have enough pre-erased space available when you need it? Since TRIM has an overhead, you may not want to blindly do it on a frequent basis (and its full effects aren't entirely predictable since they depend on how much the drive decides to actually erase in advance).

(Having looked at SMART 'block erase count' information on one of our servers, it's definitely doing something when the server is under heavy fsync() load, but I need to cross-compare the numbers from it to other systems in order to get a better sense of what's exceptional and what's not.)

I'm currently more focused on write related metrics, but there's probably important information that could be exposed for reads and for other operations. I'd also like it if SSDs provided counters for how many of various sorts of operations they saw, because while your operating system can in theory provide this, it often doesn't (or doesn't provide them at the granularity of, say, how many writes with 'Force Unit Access' or how many 'Flush' operations were done).

(In Linux, I think I'd have to extract this low level operation information in an ad-hoc way with eBPF tracing.)

A (filesystem) journal can be a serialization point for durable writes

By: cks

Suppose that you have a filesystem that uses some form of a journal to provide durability (as many do these days) and you have a bunch of people (or processes) writing and updating things all over the filesystem that they want to be durable, so these processes are all fsync()'ing their work on a regular basis (or the equivalent system call or synchronous write operation). In a number of filesystem designs, this creates a serialization point on the filesystem's journal.

This is related to the traditional journal fsync() problem, but that one is a bit different. In the traditional problem you have a bunch of changes from a bunch of processes, some of which one process wants to fsync() and most of which it doesn't; this can be handled by only flushing necessary things. Here we have a bunch of processes making a bunch of relatively independent changes but approximately all of the processes want to fsync() their changes.

The simple way to get durability (and possibly integrity) for fsync() is to put everything that gets fsync()'d into the journal (either directly or indirectly) and then force the journal to be durably committed to disk. If the filesystem's journal is a linear log, as is usually the case, this means that multiple processes mostly can't be separately writing and flushing journal entries at the same time. Each durable commit of the journal is a bottleneck for anyone who shows up 'too late' to get their change included in the current commit; they have to wait for the current commit to be flushed to disk before they can start adding more entries to the journal (but then everyone can be bundled into the next commit).

In some filesystems, processes can readily make durable writes outside of the journal (for example, overwriting something in place); such processes can avoid serializing on a linear journal. Even if they have to put something in the journal, you can perhaps minimize the direct linear journal contents by having them (durably) write things to various blocks independently, then put only compact pointers to those out of line blocks into the linear journal with its serializing, linear commits. The goal is to avoid having someone show up wanting to write megabytes 'to the journal' and forcing everyone to wait for their fsync(); instead people serialize only on writing a small bit of data at the end, and writing the actual data happens in parallel (assuming the disk allows that).

(I may have made this sound simple but the details are likely fiendishly complex.)

If you have a filesystem in this situation, and I believe one of them is ZFS, you may find you care a bunch about the latency of disks flushing writes to media. Of course you need the workload too, but there are certain sorts of workloads that are prone to this (for example, traditional Unix mail spools).

I believe that you can also see this sort of thing with databases, although they may be more heavily optimized for concurrent durable updates.

Sidebar: Disk handling of durable writes can also be a serialization point

Modern disks (such as NVMe SSDs) broadly have two mechanism to force things to durable storage. You can issue specific writes of specific blocks with 'Force Unit Access' (FUA) set, which causes the disk to write those blocks (and not necessarily any others) to media, or you can issue a general 'Flush' command to the disk and it will write anything it currently has in its write cache to media.

If you issue FUA writes, you don't have to wait for anything else other than your blocks to be written to media. If you issue 'Flush', you get to wait for everyone's blocks to be written out. This means that for speed you want to issue FUA writes when you want things on media, but on the other hand you may have already issued non-FUA writes for some of the blocks before you found out that you wanted them on media (for example, if someone writes a lot of data, so much that you start writeback, and then they issue a fsync()). And in general, the block IO programming model inside your operating system may favour issuing a bunch of regular writes and then inserting a 'force everything before this point to media' fencing operation into the IO stream.

NVMe SSDs and the question of how fast they can flush writes to flash

By: cks

Over on the Fediverse, I had a question I've been wondering about:

Disk drive people, sysadmins, etc: would you expect NVMe SSDs to be appreciably faster than SATA SSDs for a relatively low bandwidth fsync() workload (eg 40 Mbytes/sec + lots of fsyncs)?

My naive thinking is that AFAIK the slow bit is writing to the flash chips to make things actually durable when you ask, and it's basically the same underlying flash chips, so I'd expect NVMe to not be much faster than SATA SSDs on this narrow workload.

This is probably at least somewhat wrong. This 2025 SSD hierarchy article doesn't explicitly cover forced writes to flash (the fsync() case), but it does cover writing 50 GBytes of data in 30,000 files, which is probably enough to run any reasonable consumer NVMe SSD out of fast write buffer storage (either RAM or fast flash). The write speeds they get on this test from good NVMe drives are well over the maximum SATA data rates, so there's clearly a sustained write advantage to NVMe SSDs over SATA SSDs.

In replies on the Fediverse, several people pointed out that NVMe SSDs are likely using newer controllers than SATA SSDs and these newer controllers may well be better at handling writes. This isn't surprising when I thought about it, especially in light of NVMe perhaps overtaking SATA for SSDs, although apparently 'enterprise' SATA/SAS SSDs are still out there and probably seeing improvements (unlike consumer SATA SSDs where price is the name of the game).

Also, apparently the real bottleneck in writing to the actual flash is finding erased blocks or, if you're unlucky, having to wait for blocks to be erased. Actual writes to the flash chips may be able to go at something close to the PCIe 3.0 (or better) bandwidth, which would help explain the Tom's Hardware large write figures (cf).

(If this is the case, then explicitly telling SSDs about discarded blocks is especially important for any write workload that will be limited by flash write speeds, including fsync() heavy workloads.)

PS: The reason I'm interested in this is that we have a SATA SSD based system that seems to have periodic performance issues related enough write IO combined with fsync()s (possibly due to write buffering interactions), and I've been wondering how much moving it to be NVMe based might help. Since this machine uses ZFS, perhaps one thing we should consider is manually doing some ZFS 'TRIM' operations.

The strange case of 'mouse action traps' in GNU Emacs with (slower) remote X

By: cks

Some time back over on the Fediverse, I groused about GNU Emacs tooltips. That grouse was a little imprecise; the situation I usually see problems with is specifically running GNU Emacs in SSH-forwarded X from home, which has a somewhat high latency. This high latency caused me to change how I opened URLs from GNU Emacs, and it seems to be the root of the issues I'm seeing.

The direct experience I was having with tooltips was that being in a situation where Emacs might want to show a GUI tooltip would cause Emacs to stop responding to my keystrokes for a while. If the tooltip was posted and visible it would stay visible, but the stall could happen without that. However, it doesn't seem to be tooltips as such that cause this problem, because even with tooltips disabled as far as I can tell (and certainly not appearing), the cursor and my interaction with Emacs can get 'stuck' in places where there's mouse actions available.

(I tried both setting the tooltip delay times to very large numbers and setting tooltip-functions to do nothing.)

This is especially visible to me because my use of MH-E is prone to this in two cases. First, when composing email flyspell mode will attach a 'correct word' button-2 popup menu to misspelled words, which can then stall things if I move the cursor to them (especially if I use a mouse click to do so, perhaps because I want to make the word into an X selection). Second, when displaying email that has links in it, these links can be clicked on (and have hover tooltips to display what the destination URL is); what I frequently experience is that after I click on a link, when I come back to the GNU Emacs (X) window I can't immediately switch to the next message, scroll the text of the current message, or otherwise do things.

This 'trapping' and stall doesn't usually happen when I'm in the office, which is still using remote X but over a much faster and lower latency 1G network connection. Disabling tooltips themselves isn't ideal because it means I no longer get to see where links go, and anyway it's relatively pointless if it doesn't fix the real problem.

When I thought this was an issue specific to tooltips, it made sense to me because I could imagine that GNU Emacs needed to do a bunch of relatively synchronous X operations to show or clear a tooltip, and those operations could take a while over my home link. Certainly displaying regular GNU Emacs (X) menus isn't particularly fast. Without tooltips displaying it's more mysterious, but it's still possible that Emacs is doing a bunch of X operations when it thinks a mouse or tooltip target is 'active', or perhaps there's something else going on.

(I'm generally happy with GNU Emacs but that doesn't mean it's perfect or that I don't have periodic learning experiences.)

PS: In theory there are tools that can monitor and report on the flow of X events (by interposing themselves into it). In practice it's been a long time since I used any of them, and anyway there's probably nothing I can do about it if GNU Emacs is doing a lot of X operations. Plus it's probably partly the GTK toolkit at work, not GNU Emacs itself.

PPS: Having taken a brief look at the MH-E code, I'm pretty sure that it doesn't even begin to work with GNU Emacs' TRAMP (also) system for working with remote files. TRAMP has some support for running commands remotely, but MH-E has its own low-level command execution and assumes that it can run commands rapidly, whenever it feels like, and then read various results out of the filesystem. Probably the most viable approach would be to use sshfs to mount your entire ~/Mail locally, have a local install of (N)MH, and then put shims in for the very few MH commands that have to run remotely (such as inc and the low level post command that actually sends out messages you've written). I don't know if this would work very well, but it would almost certainly be better than trying to run all those MH commands remotely.

Staring at code can change what I see (a story from long ago)

By: cks

I recently read Hillel Wayne's Sapir-Whorf does not apply to Programming Languages (via, which I will characterize as being about how programming can change how you see things even though the Sapir-Whorf hypothesis doesn't apply (Hillel Wayne points to the Tetris Effect). As it happens, long ago I experienced a particular form of this that still sticks in my memory.

Many years ago, I was recruited to be a TA for the university's upper year Operating Systems course, despite being an undergraduate at the time. One of the jobs of TAs was to mark assignments, which we did entirely by hand back in those days; any sort of automated testing was far in the future, and for these assignments I don't think we even ran the programs by hand. Instead, marking was mostly done by having students hand in printouts of their modifications to the course's toy operating system and we three TAs collectively scoured the result to see if they'd made the necessary changes and spot errors.

Since this was an OS course, some assignments required dealing with concurrency, which meant that students had to properly guard and insulate their changes (in, for example, memory handling) from various concurrency problems. Failure to completely do so would cost marks, so the TAs were on the lookout for such problems. Over the course of the course, I got very good at spotting these concurrency problems entirely by eye in the printed out code. I didn't really have to think about it, I'd be reading the code (or scanning it) and the problem would jump out at me. In the process I formed a firm view that concurrency is very hard for people to deal with, because so many students made so many mistakes (whether obvious or subtle).

(Since students were modifying the toy OS to add or change features, there was no set form that their changes had to follow; people implemented the new features in various different ways. This meant that their concurrency bugs had common patterns but not specific common forms.)

I could have thought that I was spotting these problems because I was a better programmer than these other undergraduate students (some of whom were literally my peers, it was just that I'd taken the OS course a year earlier than they had because it was one of my interests). However, one of the most interesting parts of the whole experience was getting pretty definitive proof that I wasn't, and it was my focused experience that made the difference. One of the people taking this course was a fellow undergraduate who I knew and I knew was a better programmer than I was, but when I was marking his version of one assignment I spotted what I viewed at the time as a reasonably obvious concurrency issue. So I wasn't seeing these issues when the undergraduates doing the assignment missed them because I was a better programmer, since here I wasn't: I was seeing the bugs because I was more immersed in this than they were.

(This also strongly influenced my view of how hard and tricky concurrency is. Here was a very smart programmer, one with at least some familiarity with the whole area, and they'd still made a mistake.)

Uses for DNS server delegation

By: cks

A commentator on my entry on systemd-resolved's new DNS server delegation feature asked:

My memory might fail me here, but: wasn't something like this a feature introduced in ISC's BIND 8, and then considered to be a bad mistake and dropped again in BIND 9 ?

I don't know about Bind, but what I do know is that this feature is present in other DNS resolvers (such as Unbound) and that it has a variety of uses. Some of those uses can be substituted with other features and some can't be, at least not as-is.

The quick version of 'DNS server delegation' is that you can send all queries under some DNS zone name off to some DNS server (or servers) of your choice, rather than have DNS resolution follow any standard NS delegation chain that may or may not exist in global DNS. In Unbound, this is done through, for example, Forward Zones.

DNS server delegation has at least three uses that I know of. First, you can use it to insert entire internal TLD zones into the view that clients have. People use various top level names for these zones, such as .internal, .kvm, .sandbox (our choice), and so on. In all cases you have some authoritative servers for these zones and you need to direct queries to these servers instead of having your queries go to the root nameservers and be rejected.

(Obviously you will be sad if IANA ever assigns your internal TLD to something, but honestly if IANA allows, say, '.internal', we'll have good reason to question their sanity. The usual 'standard DNS environment' replacement for this is to move your internal TLD to be under your organizational domain and then implement split horizon DNS.)

Second, you can use it to splice in internal zones that don't exist in external DNS without going to the full overkill of split horizon authoritative data. If all of your machines live in 'corp.example.org' and you don't expose this to the outside world, you can have your public example.org servers with your public data and your corp.example.org authoritative servers, and you splice in what is effectively a fake set of NS records through DNS server delegation. Related to this, if you want you can override public DNS simply by having an internal and an external DNS server, without split horizon DNS; you use DNS server delegation to point to the internal DNS server for certain zones.

(This can be replaced with split horizon DNS, although maintaining split horizon DNS is its own set of headaches.)

Finally, you can use this to short-cut global DNS resolution for reliability in cases where you might lose external connectivity. For example, there are within-university ('on-campus' in our jargon) authoritative DNS servers for .utoronto.ca and .toronto.edu. We can use DNS server delegation to point these zones at these servers to be sure we can resolve university names even if the university's external Internet connection goes down. We can similarly point our own sub-zone at our authoritative servers, so even if our link to the university backbone goes down we can resolve our own names.

(This isn't how we actually implement this; we have a more complex split horizon DNS setup that causes our resolving DNS servers to have a complete copy of the inside view of our zones, acting as caching secondaries.)

The early Unix history of chown() being restricted to root

By: cks

A few years ago I wrote about the divide in chown() about who got to give away files, where BSD and V7 were on one side, restricting it to root, while System III and System V were on the other, allowing the owner to give them away too. At the time I quoted the V7 chown(2) explanation of this:

[...] Only the super-user may execute this call, because if users were able to give files away, they could defeat the (nonexistent) file-space accounting procedures.

Recently, for reasons, chown(2) and its history was on my mind and so I wondered if the early Research Unixes had always had this, or if a restriction was added at some point.

The answer is that the restriction was added in V6, where the V6 chown(2) manual page has the same wording as V7. In Research Unix V5 and earlier, people can chown(2) away their own files; this is documented in the V4 chown(2) manual page and is what the V5 kernel code for chown() does. This behavior runs all the way back to the V1 chown() manual page, with an extra restriction that you can't chown() setuid files.

(Since I looked it up, the restriction on chown()'ing setuid files was lifted in V4. In V4 and later, a setuid file has its setuid bit removed on chown; in V3 you still can't give away such a file, according to the V3 chown(2) manual page.)

At this point you might wonder where the System III and System V unrestricted chown came from. The surprising to me answer seems to be that System III partly descends from PWB/UNIX, and PWB/UNIX 1.0, although it was theoretically based on V6, has pre-V6 chown(2) behavior (kernel source, manual page). I suspect that there's a story both to why V6 made chown() more restricted and also why PWB/UNIX specifically didn't take that change from V6, but I don't know if it's been documented anywhere (a casual Internet search didn't turn up anything).

(The System III chown(2) manual page says more or less the same thing as the PWB/UNIX manual page, just more formally, and the kernel code is very similar.)

Maybe why OverlayFS had its readdir() inode number issue

By: cks

A while back I wrote about readdir()'s inode numbers versus OverlayFS, which discussed an issue where for efficiency reasons, OverlayFS sometimes returned different inode numbers in readdir() than in stat(). This is not POSIX legal unless you do some pretty perverse interpretations (as covered in my entry), but lots of filesystems deviate from POSIX semantics every so often. A more interesting question is why, and I suspect the answer is related to another issue that's come up, the problem of NFS exports of NFS mounts.

What's common in both cases is that NFS servers and OverlayFS both must create an 'identity' for a file (a NFS filehandle and an inode number, respectively). In the case of NFS servers, this identity has some strict requirements; OverlayFS has a somewhat easier life, but in general it still has to create and track some amount of information. Based on reading the OverlayFS article, I believe that OverlayFS considers this expensive enough to only want to do it when it has to.

OverlayFS definitely needs to go to this effort when people call stat(), because various programs will directly use the inode number (the POSIX 'file serial number') to tell files on the same filesystem apart. POSIX technically requires OverlayFS to do this for readdir(), but in practice almost everyone that uses readdir() isn't going to look at the inode number; they look at the file name and perhaps the d_type field to spot directories without needing to stat() everything.

If there was a special 'not a valid inode number' signal value, OverlayFS might use that, but there isn't one (in either POSIX or Linux, which is actually a problem). Since OverlayFS needs to provide some sort of arguably valid inode number, and since it's reading directories from the underlying filesystems, passing through their inode numbers from their d_ino fields is the simple answer.

(This entry was inspired by Kevin Lyda's comment on my earlier entry.)

Sidebar: Why there should be a 'not a valid inode number' signal value

Because both standards and common Unix usage include a d_ino field in the structure readdir() returns, they embed the idea that the stat()-visible inode number can easily be recovered or generated by filesystems purely by reading directories, without needing to perform additional IO. This is true in traditional Unix filesystems, but it's not obvious that you would do that all of the time in all filesystems. The on disk format of directories might only have some sort of object identifier for each name that's not easily mapped to a relatively small 'inode number' (which is required to be some C integer type), and instead the 'inode number' is an attribute you get by reading file metadata based on that object identifier (which you'll do for stat() but would like to avoid for reading directories).

But in practice if you want to design a Unix filesystem that performs decently well and doesn't just make up inode numbers in readdir(), you must store a potentially duplicate copy of your 'inode numbers' in directory entries.

Keeping notes is for myself too, illustrated (once again)

By: cks

Yesterday I wrote about restarting or redoing something after a systemd service restarts. The non-hypothetical situation that caused me to look into this was that after we applied a package update to one system, systemd-networkd on it restarted and wiped out some critical policy based routing rules. Since I vaguely remembered this happening before, I sighed and arranged to have our rules automatically reapplied on both systems with policy based routing rules, following the pattern I worked out.

Wait, two systems? And one of them didn't seem to have problems after the systemd-networkd restart? Yesterday I ignored that and forged ahead, but really it should have set off alarm bells. The reason the other system wasn't affected was I'd already solved the problem the right way back in March of 2024, when we first hit this networkd behavior and I wrote an entry about it.

However, I hadn't left myself (or my co-workers) any notes about that March 2024 fix; I'd put it into place on the first machine (then the only machine we had that did policy based routing) and forgotten about it. My only theory is that I wanted to wait and be sure it actually fixed the problem before documenting it as 'the fix', but if so, I made a mistake by not leaving myself any notes that I had a fix in testing. When I recently built the second machine with policy based routing I copied things from the first machine, but I didn't copy the true networkd fix because I'd forgotten about it.

(It turns out to have been really useful that I wrote that March 2024 entry because it's the only documentation I have, and I'd probably have missed the real fix if not for it. I rediscovered it in the process of writing yesterday's entry.)

I know (and knew) that keeping notes is good, and that my memory is fallible. And I still let this slip through the cracks for whatever reason. Hopefully the valuable lesson I've learned from this will stick a bit so I don't stub my toe again.

(One obvious lesson is that I should make a note to myself any time I'm testing something that I'm not sure will actually work. Since it may not work I may want to formally document it in our normal system for this, but a personal note will keep me from completely losing track of it. You can see the persistence of things 'in testing' as another example of the aphorism that there's nothing as permanent as a temporary fix.)

Restarting or redoing something after a systemd service restarts

By: cks

Suppose, not hypothetically, that your system is running some systemd based service or daemon that resets or erase your carefully cultivated state when it restarts. One example is systemd-networkd, although you can turn that off (or parts of it off, at least), but there are likely others. To clean up after this happens, you'd like to automatically restart or redo something after a systemd unit is restarted. Systemd supports this, but I found it slightly unclear how you want to do this and today I poked at it, so it's time for notes.

(This is somewhat different from triggering one unit when another unit becomes active, which I think is still not possible in general.)

First, you need to put whatever you want to do into a script and a .service unit that will run the script. The traditional way to run a script through a .service unit is:

[Unit]
....

[Service]
Type=oneshot
RemainAfterExit=True
ExecStart=/your/script/here

[Install]
WantedBy=multi-user.target

(The 'RemainAfterExit' is load-bearing, also.)

To get this unit to run after another unit is started or restarted, what you need is PartOf=, which causes your unit to be stopped and started when the other unit is, along with 'After=' so that your unit starts after the other unit instead of racing it (which could be counterproductive when what you want to do is fix up something from the other unit). So you add:

[Unit]
...
PartOf=systemd-networkd.service
After=systemd-networkd.service

(This is what works for me in light testing. This assumes that the unit you want to re-run after is normally always running, as systemd-networkd is.)

In testing, you don't need to have your unit specifically enabled by itself, although you may want it to be for clarity and other reasons. Even if your unit isn't specifically enabled, systemd will start it after the other unit because of the PartOf=. If the other unit is started all of the time (as is usually the case for systemd-networkd), this effectively makes your unit enabled, although not in an obvious way (which is why I think you should specifically 'systemctl enable' it, to make it obvious). I think you can have your .service unit enabled and active without having the other unit enabled, or even present.

You can declare yourself PartOf a .target unit, and some stock package systemd units do for various services. And a .target unit can be PartOf a .service; on Fedora, 'sshd-keygen.target' is PartOf sshd.service in a surprisingly clever little arrangement to generate only the necessary keys through a templated 'sshd-keygen@.service' unit.

I admit that the whole collection of Wants=, Requires=, Requisite=, BindsTo=, PartOf=, Upholds=, and so on are somewhat confusing to me. In the past, I've used the wrong version and suffered the consequences, and I'm not sure I have them entirely right in this entry.

Note that as far as I know, PartOf= has those Requires= consequences, where if the other unit is stopped, yours will be too. In a simple 'run a script after the other unit starts' situation, stopping your unit does nothing and can be ignored.

(If this seems complicated, well, I think it is, and I think one part of the complication is that we're trying to use systemd as an event-based system when it isn't one.)

Systemd-resolved's new 'DNS Server Delegation' feature (as of systemd 258)

By: cks

A while ago I wrote an entry about things that resolved wasn't for as of systemd 251. One of those things was arbitrary mappings of (DNS) names to DNS servers, for example if you always wanted *.internal.example.org to query a special DNS server. Systemd-resolved didn't have a direct feature for this and attempting to attach your DNS names to DNS server mappings to a network interface could go wrong in various ways. Well, time marches on and as of systemd v258 this is no longer the state of affairs.

Systemd v258 introduces systemd.dns-delegate files, which allow you to map DNS names to DNS servers independently from network interfaces. The release notes describe this as:

A new DNS "delegate zone" concept has been introduced, which are additional lookup scopes (on top of the existing per-interface and the one global scope so far supported in resolved), which carry one or more DNS server addresses and a DNS search/routing domain. It allows routing requests to specific domains to specific servers. Delegate zones can be configured via drop-ins below /etc/systemd/dns-delegate.d/*.dns-delegate.

Since systemd v258 is very new I don't have any machines where I can actually try this out, but based on the systemd.dns-delegate documentation, you can use this both for domains that you merely want diverted to some DNS server and also domains that you also want on your search path. Per resolved.conf's Domains= documentation, the latter is 'Domains=example.org' (example.org will be one of the domains that resolved tries to find single-label hostnames in, a search domain), and the former is 'Domains=~example.org' (where we merely send queries for everything under 'example.org' off to whatever DNS= you set, a route-only domain).

(While resolved.conf's Domains= officially promises to check your search domains in the order you listed them, I believe this is strictly for a single 'Domains=' setting for a single interface. If you have multiple 'Domains=' settings, for example in a global resolved.conf, a network interface, and now in a delegation, I think systemd-resolved makes no promises.)

Right now, these DNS server delegations can only be set through static files, not manipulated through resolvectl. I believe fiddling with them through resolvectl is on the roadmap, but for now I guess we get to restart resolved if we need to change things. In fact resolvectl doesn't expose anything to do with them, although I believe read-only information is available via D-Bus and maybe varlink.

Given the timing of systemd v258's release relative to Fedora releases, I probably won't be able to use this feature until Fedora 44 in the spring (Fedora 42 is current and Fedora 43 is imminent, which won't have systemd v258 given that v258 was released only a couple of weeks ago). My current systemd-resolved setup is okay (if it wasn't I'd be doing something else), but I can probably find uses for these delegations to improve it.

Why I have a GPS bike computer

By: cks

(This is a story about technology. Sort of.)

Many bicyclists with a GPS bike computer probably have it primarily to record their bike rides and then upload them to places like Strava. I'm a bit unusual in that while I do record my rides and make some of them public, and I've come to value this, it's not my primary reason to have a GPS bike computer. Instead, my primary reason is following pre-made routes.

When I started with my recreational bike club, it was well before the era of GPS bike computers. How you followed (or lead) our routes back then was through printed cue sheets, which had all of the turns and so on listed in order, often with additional notes. One of the duties of the leader of the ride was printing out a sufficient number of cue sheets in advance and distributing them to interested parties before the start of the ride. If you were seriously into using cue sheets, you'd use a cue sheet holder (nowadays you can only find these as 'map holders', which is basically the same job); otherwise you might clip the cue sheet to a handlebar brake or gear cable or fold it up and stick it in a back jersey pocket.

Printed cue sheets have a number of nice features, such as giving you a lot of information at a glance. One of them is that a well done cue sheet was and is a lot more than just a list of all of the turns and other things worthy of note; it's an organized, well formatted list of these. The cues would be broken up into sensibly chosen sections, with whitespace between them to make it easier to narrow in on the current one, and you'd lay out the page (or pages) so that the cue or section breaks happened at convenient spots to flip the cue sheet around in cue holders or clips. You'd emphasize important turns, cautions, or other things in various ways. And so on. Some cue sheets even had a map of the route printed on the back.

(You needed to periodically flip the cue sheet around and refold it because many routes had too many turns and other cues to fit in a small amount of printed space, especially if you wanted to use a decently large font size for easy readability.)

Starting in the early 2010s, more and more TBN people started using GPS bike computers or smartphones (cf). People began converting our cue sheet routes to computerized GPS routes, with TBN eventually getting official GPS routes. Over time, more and more members got smartphones and GPS units and there was more and more interest in GPS routes and less and less interest in cue sheets. In 2015 I saw the writing on the wall for cue sheets and the club more or less deprecated them, so in August 2016 I gave in and got a GPS unit (which drove me to finally get a smartphone, because my GPS unit assumed you had one). Cue sheet first routes lingered on for some years afterward, but they're all gone by now; everything is GPS route first.

You can still get cue sheets for club routes (the club's GPS routes typically have turn cues and you can export these into something you can print). But what we don't really have any more is the old school kind of well done, organized cue sheets, and it's basically been a decade since ride leaders would turn up with any printed cue sheets at all. These days it's on you to print your own cue sheet if you need it, and also on you to make a good cue sheet from the basic cue sheet (if you care enough to do so). There are some people who still use cue sheets, but they're a decreasing minority and they probably already had the cue sheet holders and so on (which are now increasingly hard to find). A new rider who wanted to use cue sheets would have an uphill struggle and they might never understand why long time members could be so fond of them.

Cue sheets are still a viable option for route following (and they haven't fundamentally changed). They're just not very well supported any more in TBN because they stopped being popular. If you insist on sticking with them, you still can, but it's not going to be a great experience. I didn't move to a GPS unit because I couldn't possibly use cue sheets any more (I still have my cue sheet holder); I moved because I could see the writing on the wall about which one would be the more convenient, more usable option.

Applications to the (computing) technologies of your choice are left as an exercise for the reader.

PS: As a whole I think GPS bike computers are mostly superior to cue sheets for route following, but that's a different discussion (and it depends on what sort of bicycling you're doing). There are points on both sides.

A Firefox issue and perhaps how handling scaling is hard

By: cks

Over on the Fediverse I shared a fun Firefox issue I've just run into:

Today's fun Firefox bug: if I move my (Nightly) Firefox window left and right across my X display, the text inside the window reflows to change its line wrapping back and forth. I have a HiDPI display with non-integer scaling and some other settings, so I'm assuming that Firefox is now suffering from rounding issues where the exact horizontal pixel position changes its idea of the CSS window width, triggering text reflows as it jumps back and forth by a CSS pixel.

(I've managed to reproduce this in a standard Nightly, although so far only with some of my settings.)

Close inspection says that this isn't quite what's happening, and the underlying problem is happening more often than I thought. What is actually happening is that as I move my Firefox window left and right, a thin vertical black line usually appears and disappears at the right edge of the window (past a scrollbar if there is one). Since I can see it on my HiDPI display, I suspect that this vertical line is at least two screen pixels wide. Under the right circumstances of window width, text size, and specific text content, this vertical black bar takes enough width away from the rest of the window to cause Firefox to re-flow and re-wrap text, creating easily visible changes as the window moves.

A variation of this happens when the vertical black bar isn't drawn but things on the right side of the toolbar and the URL bar area will shift left and right slightly as the window is moved horizontally. If the window is showing a scrollbar, the position of the scroll target in the scrollbar will move left and right, with the right side getting ever so slightly wider or returning back to being symmetrical. It's easiest to see this if I move the window sideways slowly, which is of course not something I do often (usually I move windows rapidly).

(This may be related to how X has a notion of sizing windows in non-pixel units if the window asks for it. Firefox in my configuration definitely asks for this; it asserts that it wants to be resized in units of 2 (display) pixels both horizontally and vertically. However, I can look at the state of a Firefox window in X and see that the window size in pixels doesn't change between the black bar appearing and disappearing.)

All of this is visible partly because under X and my window manager, windows can redisplay themselves even during an active move operation. If the window contents froze while I dragged windows around, I probably wouldn't have noticed this for some time. Text reflowing as I moved a Firefox window sideways created a quite attention-getting shimmer.

It's probably relevant that I need unusual HiDPI settings and I've also set Firefox's layout.css.devPixelsPerPx to 1.7 in about:config. That was part of why I initially assumed this was a scaling and rounding issue, and why I still suspect that area of Firefox a bit.

(I haven't filed this as a Firefox bug yet, partly because I just narrowed down what was happening in the process of writing this entry.)

What (I think) you need to do basic UDP NAT traversal

By: cks

Yesterday I wished for a way to do native "blind" WireGuard relaying, without needing to layer something on top of WireGuard. I wished for this both because it's the simplest approach for getting through NATs and the one you need in general under some circumstances. The classic and excellent work on all of the complexities of NAT traversal is Tailscale's How NAT traversal works, which also winds up covering the situation where you absolutely have to have a relay. But, as I understand things, in a fair number of situations you can sort of do without a relay and have direct UDP NAT traversal, although you need to do some extra work to get it and you need additional pieces.

Following RFC 4787, we can divide NAT into to two categories, endpoint-independent mapping (EIM) and endpoint-dependent mapping (EDM). In EIM, the public IP and port of your outgoing NAT'd traffic depend only on your internal IP and port, not on the destination (IP or port); in EDM they (also) depend on the destination. NAT'ing firewalls normally NAT based on what could be called "flows". For TCP, flows are a real thing; you can specifically tell a single TCP connection and it's difficult to fake one. For UDP, a firewall generally has no idea of what is a valid flow, and the best it can do is accept traffic that comes from the destination IP and port, which in theory is replies from the other end.

This leads to the NAT traffic traversal trick that we can do for UDP specifically. If we have two machines that want to talk to each other on each other's UDP port 51820, the first thing they need is to learn the public IP and port being used by the other machine. This requires some sort of central coordination server as well as the ability to send traffic to somewhere on UDP port 51820 (or whatever port you care about). In the case of WireGuard, you might as well make this a server on a public IP running WireGuard and have an actual WireGuard connection to it, and the discount 'coordination server' can then be basically the WireGuard peer information from 'wg' (the 'endpoint' is the public IP and port you need).

Once the two machines know each other's public IP and port, they start sending UDP port 51820 (or whatever) packets to each other, to the public IP and port they learned through the coordination server. When each of them sends their first outgoing packet, this creates a 'flow' on their respective NAT firewall which will allow the other machine's traffic in. Depending on timing, the first few packets from the other machine may arrive before your firewall has set up its state to allow them in and will get dropped, so each side needs to keep sending until it works or until it's clear that at least one side has an EDM (or some other complication).

(For WireGuard, you'd need something that sets the peer's endpoint to your now-known host and port value and then tries to send it some traffic to trigger the outgoing packets.)

As covered in Tailscale's article, it's possible to make direct NAT traversal work in some additional circumstances with increasing degrees of effort. You may be lucky and have a local EDM firewall that can be asked to stop doing EDM for your UDP port (via a number of protocols for this), and otherwise it may be possible to feel your way around one EDM firewall.

If you can arrange a natural way to send traffic from your UDP port to your coordination server, the basic NAT setup can be done without needing the deep cooperation of the software using the port; all you need is a way to switch what remote IP and port it uses for a particular peer. Your coordination server may need special software to listen to traffic and decode which peer is which, or you may be able to exploit existing features of your software (for example, by making the coordination server a WireGuard peer). Otherwise, I think you need either some cooperation from the software involved or gory hacks.

Wishing for a way to do 'blind' (untrusted) WireGuard relaying

By: cks

Over on the Fediverse, I sort of had a question:

I wonder if there's any way in standard WireGuard to have a zero-trust network relay, so that two WG peers that are isolated from each other (eg both behind NAT) can talk directly. The standard pure-WG approach has a public WG endpoint that everyone talks to and which acts as a router for the internal WG IPs of everyone, but this involves decrypting and re-encrypting the WG traffic.

By 'talk directly' I mean that each of the peers has the WireGuard keys of the other and the traffic between the two of them stays encrypted with those keys all the way through its travels. The traditional approach to the problem of two NAT'd machines that want to talk to each other with WireGuard is to have a WireGuard router that both of them talk to over WireGuard, but this means that the router sees the unencrypted traffic between them. This is less than ideal if you don't want to trust your router machine, for example because you want to make it a low-trust virtual machine rented from some cloud provider.

Since we love indirection in computer science, you can in theory solve this with another layer of traffic encapsulation (with a lot of caveats). The idea is that all of the 'public' endpoint IPs of WireGuard peers are actually on a private network, and you route the private network through your public router. Getting the private network packets to and from the router requires another level of encapsulation and unless you get very clever, all your traffic will go through the router even if two WireGuard peers could talk directly. Since WireGuard automatically keeps track of the current public IPs of peers, it would be ideal to do this with WireGuard, but I'm not sure that WG-in-WG can have the routing maintained the way we want.

This untrusted relay situation is of course one of the things that 'automatic mesh network on top of WireGuard' systems give you, but it would be nice to be able to do this with native features (and perhaps without an explicit control plane server that machines talk to, although that seems unlikely). As far as I know such systems implement this with their own brand of encapsulation, which I believe requires running their WireGuard stack.

(On Linux you might be able to do something clever with redirecting outgoing WireGuard packets to a 'tun' device connected to a user level program, which then wrapped them up, sent them off, received packets back, and injected the received packets into the system.)

Using systems because you know them already

By: cks

Every so often on the Fediverse, people ask for advice on a monitoring system to run on their machine (desktop or server), and some of the time Prometheus, and when it does I wind up making awkward noises. On the one hand, we run Prometheus (and Grafana) and are happy with it, and I run separate Prometheus setups on my work and home desktops. On the other hand, I don't feel I can recommend picking Prometheus for a basic single-machine setup, despite running it that way myself.

Why do I run Prometheus on my own machines if I don't recommend that you do so? I run it because I already know Prometheus (and Grafana), and in fact my desktops (re)use much of our production Prometheus setup (but they scrape different things). This is a specific instance (and example) of a general thing in system administration, which is that not infrequently it's simpler for you to use something you already know even if it's not necessarily an exact fit (or even a great fit) for the problem. For example, if you're quite familiar with operating PostgreSQL databases, it might be simpler to use PostgreSQL for a new system where SQLite could do perfectly well and other people would find SQLite much simpler. Especially if you have canned setups, canned automation, and so on all ready to go for PostgreSQL, and not for SQLite.

(Similarly, our generic web server hammer is Apache, even if we're doing things that don't necessarily need Apache and could be done perfectly well or perhaps better with nginx, Caddy, or whatever.)

This has a flipside, where you use a tool because you know it even if there might be a significantly better option, one that would actually be easier overall even accounting for needing to learn the new option and build up the environment around it. What we could call "familiarity-driven design" is a thing, and it can even be a confining thing, one where you shape your problems to conform to the tools you already know.

(And you may not have chosen your tools with deep care and instead drifted into them.)

I don't think there's any magic way to know which side of the line you're on. Perhaps the best we can do is be a little bit skeptical about our reflexive choices, especially if we seem to be sort of forcing them in a situation that feels like it should have a simpler or better option (such as basic monitoring of a single machine).

(In a way it helps that I know so much about Prometheus because it makes me aware of various warts, even if I'm used to them and I've climbed the learning curves.)

Apache .htaccess files are important because they enable delegation

By: cks

Apache's .htaccess files have a generally bad reputation. For example, lots of people will tell you that they can cause performance problems and you should move everything from .htaccess files into your main Apache configuration, using various pieces of Apache syntax to restrict what configuration directives apply to. The result can even be clearer, since various things can be confusing in .htaccess files (eg rewrites and redirects). Despite all of this, .htaccess files are important and valuable because of one property, which is that they enable delegation of parts of your server configuration to other people.

The Apache .htaccess documentation even spells this out in reverse, in When (not) to use .htaccess files:

In general, you should only use .htaccess files when you don't have access to the main server configuration file. [...]

If you operate the server and would be writing the .htaccess file, you can put the contents of the .htaccess in the main server configuration and make your life easier and Apache faster (and you probably should). But if the web server and its configuration isn't managed as a unitary whole by one group, then .htaccess files allow the people managing the overall Apache configuration to safely delegate things to other people on a per-directory basis, using Unix ownership. This can both enable people to do additional things and reduce the amount of work the central people have to do, letting people things scale better.

(The other thing that .htaccess files allow is dynamic updates without having to restart or reload the whole server. In some contexts this can be useful or important, for example if the updates are automatically generated at unpredictable times.)

I don't think it's an accident that .htaccess files emerged in Apache, because one common environment Apache was initially used in was old fashioned multi-user Unix web servers where, for example, every person with a login on the web server might have their own UserDir directory hierarchy. Hence features like suEXEC, so you could let people run CGIs without those CGIs having to run as the web user (a dangerous thing), and also hence the attraction of .htaccess files. If you have a bunch of (graduate) students with their own web areas, you definitely don't want to let all of them edit your departmental web server's overall configuration.

(Apache doesn't solve all your problems here, at least not in a simple configuration; you're still left with the multiuser PHP problem. Our solution to this problem is somewhat brute force.)

These environments are uncommon today but they're not extinct, at least at universities like mine, and .htaccess files (and Apache's general flexibility) remain valuable to us.

Readdir()'s inode numbers versus OverlayFS

By: cks

Recently I re-read Deep Down the Rabbit Hole: Bash, OverlayFS, and a 30-Year-Old Surprise (via) and this time around, I stumbled over a bit in the writeup that made me raise my eyebrows:

Bash’s fallback getcwd() assumes that the inode [number] from stat() matches one returned by readdir(). OverlayFS breaks that assumption.

I wouldn't call this an 'assumption' so much as 'sane POSIX semantics', although I'm not sure that POSIX absolutely requires this.

As we've seen before, POSIX talks about 'file serial number(s)' instead of inode numbers. The best definition of these is covered in sys/stat.h, where we see that a 'file identity' is uniquely determined by the combination the inode number and the device ID (st_dev), and POSIX says that 'at any given time in a system, distinct files shall have distinct file identities' while hardlinks have the same identity. The POSIX description of readdir() and dirent.h don't caveat the d_ino file serial numbers from readdir(), so they're implicitly covered by the general rules for file serial numbers.

In theory you can claim that the POSIX guarantees don't apply here since readdir() is only supplying d_ino, the file serial number, not the device ID as well. I maintain that this fails due to a POSIX requirement:

[...] The value of the structure's d_ino member shall be set to the file serial number of the file named by the d_name member. [...]

If readdir() gives one file serial number and a fstatat() of the same name gives another, a plain reading of POSIX is that one of them is lying. Files don't have two file serial numbers, they have one. Readdir() can return duplicate d_ino numbers for files that aren't hardlinks to each other (and I think legitimately may do so in some unusual circumstances), but it can't return something different than what fstatat() does for the same name.

The perverse argument here turns on POSIX's 'at any given time'. You can argue that the readdir() is at one time and the stat() is at another time and the system is allowed to entirely change file serial numbers between the two times. This is certainly not the intent of POSIX's language but I'm not sure there's anything in the standard that rules it out, even though it makes file serial numbers fairly useless since there's no POSIX way to get a bunch of them at 'a given time' so they have to be coherent.

So to summarize, OverlayFS has chosen what are effectively non-POSIX semantics for its readdir() inode numbers (under some circumstances, in the interests of performance) and Bash used readdir()'s d_ino in a traditional Unix way that caused it to notice. Unix filesystems can depart from POSIX semantics if they want, but I'd prefer if they were a bit more shamefaced about it. People (ie, programs) count on those semantics.

(The truly traditional getcwd() way wouldn't have been a problem, because it predates readdir() having d_ino and so doesn't use it (it stat()s everything to get inode numbers). I reflexively follow this pre-d_ino algorithm when I'm talking about doing getcwd() by hand (cf), but these days you want to use the dirent d_ino and if possible d_type, because they're much more efficient than stat()'ing everything.)

How part of my email handling drifted into convoluted complexity

By: cks

Once upon a time, my email handling was relatively simple. I wasn't on any big mailing lists, so I had almost everything delivered straight to my inbox (both in the traditional /var/mail mbox sense and then through to MH's own inbox folder directory). I did some mail filtering with procmail, but it was all for things that I basically never looked at, so I had procmail write them to mbox files under $HOME/.mail. I moved email from my Unix /var/mail inbox to MH's inbox with MH's inc command (either running it directly or having exmh run it for me). Rarely, I had a mbox file procmail had written that I wanted to read, and at that point I inc'd it either to my MH +inbox or to some other folder.

Later, prompted by wanting to improve my breaks and vacations, I diverted a bunch of mailing lists away from my inbox. Originally I had procmail write these diverted messages to mbox files, then later I'd inc the files to read the messages. Then I found that outside of vacations, I needed to make this email more readily accessible, so I had procmail put them in MH folder directories under Mail/inbox (one of MH's nice features is that your inbox is a regular folder and can have sub-folders, just like everything else). As I noted at the time, procmail only partially emulates MH when doing this, and one of the things it doesn't do is keep track of new, unread ('unseen') messages.

(MH has a general purpose system for keeping track of 'sequences' of messages in a MH folder, so it tracks unread messages based on what is in the special 'unseen' sequence. Inc and other MH commands update this sequence; procmail doesn't.)

Along with this procmail setup I wrote a basic script, called mlists, to report how many messages each of these 'mailing list' inboxes had in them. After a while I started diverting lower priority status emails and so on through this system (and stopped reading the mailing lists); if I got a type of email in any volume that I didn't want to read right away during work, it probably got shunted to these side inboxes. At some point I made mlists optionally run the MH scan command to show me what was in each inbox folder (well, for the inbox folders where this was potentially useful information). The mlists script was still mostly simple and the whole system still made sense, but it was a bit more complex than before, especially when it also got a feature where it auto-reset the current message number in each folder to the first message.

A couple of years ago, I switched the MH frontend I used from exmh to MH-E in GNU Emacs, which changed how I read my email in practice. One of the changes was that I started using the GNU Emacs Speedbar, which always displays a count of messages in MH folders and especially wants to let you know about folders with unread messages. Since I had the hammer of my mlists script handy, I proceeded to mutate it to be what a comment in the script describes as "a discount maintainer of 'unseen'", so that MH-E's speedbar could draw my attention to inbox folders that had new messages.

This is not the right way to do this. The right way to do this is to have procmail deliver messages through MH's rcvstore, which as a MH command can update the 'unseen' sequence properly. But using rcvstore is annoying, partly because you have to use another program to add the locking it needs, so at every point the path of least resistance was to add a bit more hacks to what I already had. I had procmail, and procmail could deliver to MH folder directories, so I used it (and at the time the limitations were something I considered a feature). I had a script to give me basic information, so it could give me more information, and then it could do one useful thing while it was giving me information, and then the one useful thing grew into updating 'unseen'.

And since I have all of this, it's not even worth the effort of switching to the proper rcvstore approach and throwing a bunch of it away. I'm always going to want the 'tell me stuff' functionality of my mlists script, so part of it has to stay anyway.

Can I see similarities between this and how various of our system tools have evolved, mutated, and become increasingly complex? Of course. I think it's much the same obvious forces involved, because each step seems reasonable in isolation, right up until I've built a discount environment that duplicates much of rcvstore.

Sidebar: an extra bonus bit of complexity

It turns out that part of the time, I want to get some degree of live notification of messages being filed into these inbox folders. I may not look at all or even many of them, but there are some periodic things that I do want to pay attention to. So my discount special hack is basically:

tail -f .mail/procmail-log |
  egrep -B2 --no-group-separator 'Folder: /u/cks/Mail/inbox/'

(This is a script, of course, and I run it in a terminal window.)

This could be improved in various ways but then I'd be sliding down the convoluted complexity slope and I'm not willing to do that. Yet. Give it a few years and I may be back to write an update.

More on the tools I use to read email affecting my email reading

By: cks

About two years ago I wrote an entry about how my switch from reading email with exmh to reading it in GNU Emacs with MH-E had affected my email reading behavior more than I expected. As time has passed and I've made more extensive customizations to my MH-E environment, this has continued. One of the recent ways I've noticed is that I'm slowly making more and more use of the fact that GNU Emacs is a multi-window editor ('multi-frame' in Emacs terminology) and reading email with MH-E inside it still leaves me with all of the basic Emacs facilities. Specifically, I can create several Emacs windows (frames) and use this to be working in multiple MH folders at the same time.

Back when I used exmh extensively, I mostly had MH pull my email into the default 'inbox' folder, where I dealt with it all at once. Sometimes I'd wind up pulling some new email into a separate folder, but exmh only really giving me a view of a single folder at a time combined with a system administrator's need to be regularly responding to email made that a bit awkward. At first my use of MH-E mostly followed that; I had a single Emacs MH-E window (frame) and within that window I switched between folders. But lately I've been creating more new windows when I want to spend time reading a non-inbox folder, and in turn this has made me much more willing to put new email directly into different (MH) folders rather than funnel it all into my inbox.

(I don't always make a new window to visit another folder, because I don't spend long on many of my non-inbox folders for new email. But for various mailing lists and so on, reading through them may take at least a bit of time so it's more likely I'll decide I want to keep my MH inbox folder still available.)

One thing that makes this work is that MH-E itself has reasonably good support for displaying and working on multiple folders at once. There are probably ways to get MH-E to screw this up and run MH commands with the wrong MH folder as the current folder, so I'm careful that I don't try to have MH-E carry out its pending MH operations in two MH-E folders at the same time. There are areas where MH-E is less than ideal when I'm also using command-line MH tools, because MH-E changes MH's global notion of the current folder any time I have it do things like show a message in some folder. But at least MH-E is fine (in normal circumstances) if I use MH commands to change the current folder; MH-E will just switch it back the next time I have it show another message.

PS: On a purely pragmatic basis, another change in my email handling is that I'm no longer as irritated with HTML emails because GNU Emacs is much better at displaying HTML than exmh was. I've actually left my MH-E setup showing HTML by default, instead of forcing multipart/alternative email to always show the text version (my exmh setup). GNU Emacs and MH-E aren't up to the level of, say, Thunderbird, and sometimes this results in confusing emails, but it's better than it was.

(The situation that seems tricky for MH-E is that people sometimes include inlined images, for example screenshots as part of problem reports, and MH-E doesn't always give any indication that it's even omitting something.)

Recently

Some meta-commentary on reading: I’ve been trying to read through my enormous queue of articles saved on Instapaper. In general I love reading stuff from the internet but refuse to do it with a computer or a phone: I don’t want all my stuff to glow!

So, this led me to an abortive trial of the Boox Go 7, an eReader that runs a full Android operating system. Rendering Android interfaces with e-ink was pretty bad, and even though it was possible to run the Instapaper Android app, highlighting text didn’t work and the whole fit & finish was off. I love that most e-ink tablets are really purpose-built computers: this didn’t give me that impression.

So, this month I bought a Kobo Libra Color. Kobo and Instapaper recently announced a partnership and first-class support for Instapaper on the devices.

Overall: it’s better than the Boox experience and definitely better than sending articles to one of my Kindles. My notes so far are:

  • I wish it had highlighting. Pretty confident that it’s on the product roadmap, but for now all it can do is sync, archive, and like articles.
  • The Kobo also integrates directly with Overdrive for local libraries! Amazing and unexpected for me, the vast majority of my books are from the Brooklyn Library or the NYPL.
  • The hardware is pretty decent: the color screen is surprisingly useful because a lot of articles have embedded images. The page turn buttons are a little worse than those on my Kindle Oasis because they’re hinged, so they only work well if you press the top of one button and the bottom of the other. I’ve gotten used to it, but wish they worked via a different mechanism.
  • The first run experience was pretty slick.
  • Annoyingly, it goes to fully ‘off’ mode pretty quickly instead of staying in ‘sleep’ mode, and waking it up from being fully off takes about 15 seconds.

Overall: I think my biggest gripe (no highlighting) will get worked out and this’ll be the perfect internet-reading-without-a-computer device.

Anyway, what’s been good this month?

Reading

In Greece, a homeowner traded ther property development rights to a builder and kept equity in place by receiving finished apartments in the resulting building, often one for themselves and one or more additional units for rental income or adult children, rather than a one-time cash payout. Applied to the U.S., that becomes, swap a house for an apartment (or three), in the same location, with special exemptions to the tax for the home sale.

From Millenial American Dream love a good scheme to fix the housing crisis and densify cities. Definitely seems like a net-positive, but like all other schemes that require big changes to tax codes, would take a miracle to actually implement.

In Zitron’s analysis, it’s always bad. It’s bad when they raise too little money because they’ll run out. It’s bad when they raise too much money because it means they need it.

David Crespo’s critique of Ed Zitron is really strong. Honestly Zitron’s writing, though needed in a certain sense, never hits home for me. In fact a lot of AI critique seems overblown. Maybe I’m warming to it a little.

This Martin Fowler article about how ‘if it hurts, do it more often,’ was good. Summarizes some well-tested wisdom.

I had really mixed feelings about ‘The Autistic Half-Century’ by Byrne Hobart. I probably have some personal stake in this - every test I’ve taken puts me on the spectrum and I have some of the classic features, but like Hobart I’ve never been interested in identifying as autistic. But my discomfort with the subject is all knotted-together and hard to summarize.

One facet I can pick out is my feeling that this era is ending, maybe getting replaced with the ADHD half-century. But the internet I grew up on was pseudonomous, text-oriented, and for me, a calm place. The last ten years have been a slow drift toward real names, and then photograph avatars, and now more and more information being delivered by some person talking into the camera, and that feels really bad, man. Heck, not to drill down on ‘classic traits’, but the number of TikTok videos in which, for some reason the person doing the talking is also eating at the same time, close to the phone, the mouth-sounds? Like, for a long time it was possible to convey information without thinking about whether you were good-looking or were having a good hair day, and that era is ending because everything is becoming oral culture.

If you believed that Trump winning would mean that everyone who supported him was right to have done so, because they had picked the winner; that the mega-rich AI industry buying its way into all corners of American society would mean that critics of the technology and of using it to displace human labors were not just defeated but meaningfully wrong in their criticisms; that some celebrity getting richer from a crypto rug-pull that ripped off hundreds of thousands of less-rich people would actually vindicate the celebrity’s choice to participate in it, because of how much richer it made them. Imagine holding this as an authentic understanding of how the world works: that the simple binary outcome of a contest had the power to reach back through time and adjust the ethical and moral weight of the contestants’ choices along the way. Maybe, in that case, you would feel differently about what to the rest of us looks like straight-up shit eating.

Defector (here, Albert Burneko) is doing some really good work. See also their article on Hailey Welch and ‘bag culture’.

Altman’s claim of a “Cambrian explosion” rings hollow because any tool built on the perverse incentives of social media is not truly designed with creativity in mind, but addiction. Sora may spark a new wave of digital expression, but it’s just as likely to entrench the same attention economy that has warped our online lives already.

From Parmy Olsen writing about OpenAI Sora in Bloomberg. I think this is a really good article: technology is rightfully judged based on what it actually does, not what it could do or is meant to do, and if AI continues to be used for things that make the world worse, it will earn itself a bad reputation.

Watching

Watching Night On Earth was such a joy. It was tender, laugh-out-loud funny, beautiful.

Youth had me crying the whole time, but I’d still recommend it. Got me into Sun Kil Moon from just a few minutes of Mark Kozelek being onscreen as the guitarist at the fancy hotel.

Listening

My old friends at BRNDA have a hit album on their hands. Kind of punk, kind of Sonic Youth, sort of indescribable, lots of good rock flute on this one.

More good ambient-adjacent instrumental rock.

This album feels next door to some Hop Along tracks. It’s a tie between highlighting this track or Holding On which also has a ton of hit quality.


Elsewhere: in September I did a huge bike trip and posted photos of it over in /photos. Might do a full writeup eventually: we rode the Great Allegheny Passage and the C&O canal in four days of pretty hard riding. Everyone survived, we had an amazing time, and I’m extremely glad that we were all able to make the time to do it. It made me feel so zen for about a week after getting back - I have to do that more often!

Porteur bag 2

Back in May, I wrote about a custom porteur bag that I sewed for use on my bike. That bag served me well on two trips - a solo ride up to Brewster and back, and my semi-yearly ride on the Empire State Trail, from Poughkeepsie to Brooklyn in two days.

But I had a longer ride in the plans summer, which I just rode two weeks ago: Pittsburgh to DC, 348 miles in 4 days, with two nights of camping. And at the last minute I decided to make the next version of that bag. Specifically, I wanted to correct three shortcomings of v1:

  • The attachment system was too complex. It had redundant ways to attach to the rack, but the main method was via a shock cord that looped through six grosgrain loops, plus another shock cord that I attached to keep it attached to the back of the rack. Plus hardware that’s used for attachment is better if it’s less flexible. Bungee cords and shock cords are ultimately pretty flawed as ways to attach things on bikes. My frame bag uses non-stretchy paracord, and most bikepacking setups are reliant on voile straps which have a minimal amount of stretch.
  • The bag had no liner, and the material is plasticky and odd-looking. ECOPAK EPLX has a waterproof coating on the outside that makes it look like a plastic bag instead of something from synthetic fabric.
  • The bag had way too many panels: each side was a panel, plus the bottom, plus the flap. These made the shape of the bag messy.

Version 2

Finished bag

It turned out pretty well. Here’s the gist:

Materials

As you can see, there’s only one built-in way to secure this bag to the bike: a webbing strap that attaches to the two buckles and tightens below the rack. This is definitely a superior mechanism to v1: instead of attaching just the front of the bag to the rack and having to deal with the back of the bag separately, this pulls and tensions the whole bag, including its contents, to the rack. It rattles a lot less and is a lot simpler to attach.

On the bike

Construction

This bag is made like a tote bag. The essential ingredient of a tote bag is a piece of fabric cut like this:

Tote bag construction

The lining is simply: the same shape, again, sewed in the same way, sewn to the inside of the bag but with the seams facing in the opposite direction, so that the seams of the liner and the outer shell of the bag face each other.

Techniques for building tote bags are everywhere on the internet, so it’s a really nice place to start. Plus, the bag body can be made with just one cut of fabric. In this case the bottom of the bag is Cordura and the top is ECOPAK, so I just tweaked the tote bag construction by adding panels of ECOPAK on the left and right of the first step above.

The risky part of this bag was its height and the zipper: ideally it could both zip and the top of the bag could fold over for water resistance. I didn’t accomplish both goals and learned something pretty important: if you’re building a waterproof bag with a zipper, once it’s zipped it’ll be hard to compress because the zipper keeps in air.

But the end of the tour included a surprise thunderstorm in Washington, DC and the zipper kept the wet out, so I count that as a win! The zipper also makes the bag very functional off the bike - using the same strap as I use to attach it to the bike but using that as a shoulder strap makes it pretty convenient to carry around. This really came in handy when we were moving bikes onto and off of Amtrak trains.

Plans for version three

Mostly kidding - I’m going to actually use use this bag for the next few trips instead of tinkering with it. But I do have thoughts, from this experience:

  • I have mixed feelings about using Cordura next time. It’s what everyone does, and it helps with the abrasion that the bag experiences from the rack. But I have a feeling that ECOPAK would hold up for many thousands of miles by itself, and is lighter and more waterproof than cordura. It could be cool to make the whole bag body from the same material.
  • I think I’ll make the next bag taller, and skip the zipper. I have more faith in folding over the material rather than relying on a waterproof zipper.
  • A good system for tightening the straps still eludes me. This setup included sliplock slide adjusters, but it’s still kind of annoying to get the right angles to pull the bag tight.
  • Again, I didn’t add any external pockets. I don’t think they’re absolutely necessary, but as it is, like the previous version, it’s not really possible to access the contents of this bag on the move. Which is fine because I have other bags that are easier to access - on this trip this bag carried my clothes, camp kit, and backup spare tires, so nothing essential in the middle of the day.

Cooking with glasses

I’ve been thinking the new Meta Ray-Ban augmented reality glasses. Not because they failed onstage, which they absolutely did. Or that shortly after they received rave reviews from Victoria Song at The Verge and MKBHD, two of the most influential tech reviewers. My impression is that the hardware has improved but the software is pretty bad.

Mostly I keep thinking about the cooking demo. Yeah, it bombed. But what if it worked? What if Meta releases the third iteration of this hardware next year and it worked? This post is just questions.

The demos were deeply weird: both Mark Zuckerberg and Jack Mancuso (the celebrity chef) had their AR glasses in a particular demo mode that broadcasted the audio they were hearing and the video they were seeing to the audience and to the live feed of the event.

Of course they need to square the circle of AR glasses being ‘invisible technology,’ but showing people what it does. According to MKBHD’s review, one of the major breakthroughs of the new edition is that you can’t see when other people are looking at the glasses built-in display.

I should credit Song for mentioning the privacy issues of the glasses in her review for The Verge. MKBHD briefly talks about his concern that people will use the glasses to be even more distracted during face-to-face interactions. It’s not like everyone’s totally ignoring the implications.

But the implications. Like for example: I don’t know, sometimes I’m cooking for my partner. Do they hear a one-sided conversation between me and the “Live AI” about what to do first, how to combine the ingredients? Without the staged demo’s video-casting, this would have been the demo: a celebrity chef talking to himself like a lunatic, asking how to start.

Or what if we’re cooking together – do we both have the glasses, and maybe the Live AI responds to… both of us? Do we hear each other’s audio, or is it a mystery what the AI is telling the sous-chef? Does it broadcast to a bluetooth speaker maybe?

Maybe the glasses are able to do joint attention and know who else is in your personal space and share with them in a permissioned way? Does this lock everyone into a specific brand of the glasses or is there some open protocol? It’s been a long, long time since I’ve seen a new open protocol.

Marques’s video shows an impressively invisible display to others, but surely there’s some reflection on people’s retinas: could you scan that and read people’s screens off of their eyeballs?


I hate to preempt critiques but the obvious counterpoint is that “the glasses will be good for people with accessibility needs.” Maybe the AI is unnecessary for cooking, and a lot of its applications will be creepy or fascist, but being able to translate or caption text in real-time will be really useful to people with hearing issues.

I think that’s true! In the same way as crypto was useful for human rights and commerce in countries with unreliable exchanges and banks - a point most forcefully spoken by Alex Gladstein. I’m not one to do the ethical arithmetic on the benefits of crypto in those situations versus the roughly $79 billion worth of scams that it has generated or its impact on climate change. My outsider view is that there are more people and more volume of crypto involved in the speculative & scam sector than in the human-rights sector, but I’m willing to be wrong.

I’ve always been intrigued by, but not sold on, Dynamicland and its New York cousin, folk.computer. But every AR demo I see sells me on that paradigm more. For all of the kookiness of the Dynamicland paradigm, it seems designed for a social, playful world. Nothing about the technology guarantees that outcome, but it definitely suggests it. Likewise, the AR glasses might usher in a new age of isolation and surveillance, but that isn’t guaranteed - that just seems like a likely outcome.

I’m searching for a vision of the AR future, just like I want to read about what really happens with AI proliferation, or what I wished crypto advocates would have written. I want people who believe in this tech to write about how it’ll change their lives and the lives of others, and what really can be done about the risks.

What's up with FUTO?

Some time ago, I noticed some new organization called FUTO popping up here and there. I’m always interested in seeing new organizations that fund open source popping up, and seeing as they claim several notable projects on their roster, I explored their website with interest and gratitude. I was first confused, and then annoyed by what I found. Confused, because their website is littered with bizzare manifestos,1 and ultimately annoyed because they were playing fast and loose with the term “open source”, using it to describe commercial source-available software.

FUTO eventually clarified their stance on “open source”, first through satire and then somewhat more soberly, perpetuating the self-serving myth that “open source” software can privilege one party over anyone else and still be called open source. I mentally categorized them as problematic but hoped that their donations or grants for genuinely open source projects would do more good than the harm done by this nonsense.

By now I’ve learned better. tl;dr: FUTO is not being honest about their “grant program”, they don’t have permission to pass off these logos or project names as endorsements, and they collaborate with and promote mask-off, self-proclaimed fascists.

An early sign that something is off with FUTO is in that “sober” explanation of their “disdain for OSI approved licenses”, where they make a point of criticizing the Open Source Initiative for banning Eric S. Raymond (aka ESR) from their mailing lists, citing right-wing reactionary conspiracy theorist Bryan Lunduke’s blog post on the incident. Raymond is, as you may know, one of the founders of OSI and a bigoted asshole. He was banned from the mailing lists, not because he’s a bigoted asshole, but because he was being a toxic jerk on the mailing list in question. Healthy institutions outgrow their founders. That said, FUTO’s citation and perspective on the ESR incident could be generously explained as a simple mistake, and we should probably match generosity with generosity given their prolific portfolio of open source grants.

I visited FUTO again quite recently as part of my research on Cloudflare’s donations to fascists, and was pleased to discover that this portfolio of grants had grown immensely since my last visit, and included a number of respectable projects that I admire and depend on (and some projects I don’t especially admire, hence arriving there during my research on FOSS projects run by fascists). But something felt fishy about this list – surely I would have heard about it if someone was going around giving big grants to projects like ffmpeg, VLC, musl libc, Tor, Managarm, Blender, NeoVim – these projects have a lot of overlap with my social group and I hadn’t heard a peep about it.

So I asked Rich Felker, the maintainer of musl libc, about the FUTO grant, and he didn’t know anything about it. Rich and I spoke about this for a while and eventually Rich uncovered a transaction in his GitHub sponsors account from FUTO: a one-time donation of $1,000. This payment circumvents musl’s established process for donations from institutional sponsors. The donation page that FUTO used includes this explanation: “This offer is for individuals, and may be available to small organizations on request. Commercial entities wishing to be listed as sponsors should inquire by email.” It’s pretty clear that there are special instructions for institutional donors who wish to receive musl’s endorsement as thanks for their contribution.

The extent of the FUTO “grant program”, at least in the case of musl libc, involved ignoring musl’s established process for institutional sponsors, quietly sending a modest one-time donation to one maintainer, and then plastering the logo of a well-respected open source project on a list of “grant recipients” on their home page. Rich eventually posted on Mastodon to clarify that the use of the musl name and logo here was unauthorized.

I also asked someone I know on the ffmpeg project about the grant that they had received from FUTO and she didn’t know anything about it, either. Here’s what she said:

I’m sure we did not get a grant from them, since we tear each other to pieces over everything, and that would be enough to start a flame war. Unless some dev independently got money from them to do something, but I’m sure that we as a project got nothing. The only grant we’ve received is from the STF last year.

Neovim is another project FUTO lists as a grant recipient, and they also have a separate process for institutional sponsors. I didn’t reach out to anyone to confirm, but FUTO does not appear on the sponsor list so presumably the M.O. is the same. This is also the case for Wireshark, Conduit, and KiCad. GrapheneOS is listed prominently as well, but that doesn’t seem to have worked out very well for them. Presumably ffmpeg received a similar quiet donation from FUTO, rather than something more easily recognizable as a grant.

So, it seems like FUTO is doing some shady stuff and putting a bunch of notable FOSS projects on their home page without good reason to justify their endorsement. Who’s behind all of this?

As far as I can tell, the important figures are Eron Wolf2 and Louis Rossmann.3 Wolf is the founder of FUTO – a bunch of money fell into his lap from founding Yahoo Games before the bottom fell out of Yahoo, and he made some smart investments to grow his wealth, which he presumably used to fund FUTO. Rossmann is a notable figure in the right to repair movement, with a large following on YouTube, who joined FUTO a year later and ultimately moved to Austin to work more closely with them. His established audience and reputation provides a marketable face for FUTO. I had heard of Rossmann prior to learning about FUTO and held him in generally good regard, despite little specific knowledge of his work, simply because we have a common cause in right to repair.

I hadn’t heard of Wolf before looking into FUTO. However, in the course of my research, several people tipped me off to his association with Curtis Yarvin (aka moldbug), and in particular to the use of FUTO’s platform and the credentials of Wolf and Rossman to platform and promote Yarvin. Curtis Yarvin is a full-blown, mask-off, self-proclaimed fascist. A negligible amount of due diligence is required to verify this, but here’s one source from Politico in January 2025:

I’ve interacted with Vance once since the election. I bumped into him at a party. He said, “Yarvin, you reactionary fascist.” I was like, “Thank you, Mr. Vice President, and I’m glad I didn’t stop you from getting elected.”

Ian Ward for Politico

Vice President Vance and numerous other figures in the American right have cited Yarvin as a friend and source of inspiration in shaping policy.4 Among his many political positions, Yarvin has proclaimed that black people are genetically predisposed to a lower IQ than white people, and moreover suggests that black people are inherently suitable for enslavement.5

Yarvin has appeared on FUTO’s social media channels, in particular in an interview published on PeerTube and Odysee, the latter a platform controversial for its role in spreading hate speech and misinformation.6 Yarvin also appeared on stage to “debate” Louis Rossmann in June 2022, in which Yarvin is permitted to speak at length with minimal interruptions or rebuttals to argue for an authoritarian techno-monarchy to replace democracy.

Rossmann caught some flack for this “debate” and gave a milquetoast response in a YouTube comment on this video, explaining that he agreed to this on very short notice as a favor to Eron, who had donated “a million” to Rossmann’s non-profit prior to bringing Rossmann into the fold at FUTO. Rossmann does rebuke Yarvin’s thesis, albeit buried in this YouTube comment rather than when he had the opportunity to do so on-stage during the debate. Don’t argue with fascists, Louis – they aren’t arguing with you, they are pitching their ideas7 to the audience. Smart fascists are experts at misdirection and bad-faith debate tactics and as a consequence Rossmann just becomes a vehicle for fascist propaganda – consult the YouTube comments to see who this video resonates with the most.

In the end, Rossmann seems to regret agreeing to this debate. I don’t think that Eron Wolf regrets it, though – based on his facilitation of this debate and his own interview with Yarvin on the FUTO channel a month later, I can only assume that Wolf considers Yarvin a close associate. No surprise given that Wolf is precisely the kind of insecure silicon valley techbro Yarvin’s rhetoric is designed to appeal to – moderately wealthy but unknown, and according to Yarvin, fit to be a king. Rossmann probably needs to reflect on why he associates with and lends his reputation to an organization that openly and unapologetically platforms its founder’s fascist friends.

In summary, FUTO is not just the product of some eccentric who founded a grant-making institution that funds open source at the cost of making us read his weird manifestos on free markets and oligopoly. It’s a private, for-profit company that associates with and uses their brand to promote fascists. They push an open-washing narrative and they portray themselves as a grant-making institution when, in truth, they’re passing off a handful of small donations as if they were endorsements from dozens of respectable, high-profile open source projects, in an attempt to legitimize themselves, and, indirectly, legitimize people they platform like Curtis Yarvin.

So, if you read this and discover that your project’s name and logo is being proudly displayed on the front page of a fascist-adjacent, washed-up millionaire’s scummy vanity company, and you don’t like that, maybe you should ask them to knock it off? Eron, Louis – you know that a lot of these logos are trademarked, right?


Updated 2025-10-27:

The FUTO website has been updated to clarify the nature of grants versus donations (before, after) and to reduce the appearance of endorsements from the donation recipients – the site is much better after this change.

I spoke to a representative who spoke for the FUTO leadership, and shared positive feedback regarding the changes to the website. I also asked for a follow-up on the matter of platforming fascist activist Curtis Yarvin on their social media channels, and I was provided this official response:

We prefer to spend our time building great software for people to use, funding interesting projects, and making FUTO better every day. We have no interest in engaging in politics as this is a distraction from our work. We’d like to move past this. We don’t have any further comment that that.

I understand the view that distancing oneself from politics can be a productive approach to your work, and the work FUTO does funding great software like Immich is indeed important work that should be conducted relatively free of distractions. However, FUTO is a fundamentally political organization and it does not distance itself from politics. Consider for example the FUTO statement on Open Source, which takes several political positions:

  • Disapproval of the Open Source Initiative and their legitimacy as an authority
  • Disapproval of OSI’s proposed AI standards
  • Disapproval of the “tech oligopoly”
  • Advocacy for an “open source” which is inclusive of restrictions on commercial use
  • Support for Eric S. Raymond’s side in his conflict with OSI
  • Tacit support for Bryan Lunduke

A truer “apolitical” approach would accept the mainstream definition of open source, would not take positions on conflicts with OSI or Eric Raymond, and would be careful not to cite (or platform) controversial figures such as Lunduke.

It is difficult for FUTO’s work to be apolitical at all. Importantly, there are biases in their grants and donations: their selections have a tendency to privilege projects focused on privacy, decentralization, multimedia, communication, and right to repair, all of which suggest the political priorities of FUTO. The choice to fund “source first” software in addition to open source, or not to fund outright closed source software, or not to vet projects based on their community moderation, are also political factors in their process of selecting funding recipients, or at least are seemingly apolitical decisions which ultimately have political consequences.

This brings us to the political nature of the choice to platform Curtis Yarvin. Yarvin is a self-proclaimed fascist who argues openly for fascist politics. Platforming Yarvin on the FUTO channels legitimizes Yarvin’s ideas and his work, and provides curious listeners a funnel that leads to Yarvin’s more radical ideas and into a far-right rabbit-hole. Platforming Yarvin advances the fascist political program.

It should go without saying that it is political to support fascism or fascists. There is an outrageous moral, intellectual, and political contradiction in claiming that it is apolitical to promote a person whose political program is to dismantle democracy and eject people he disagrees with from the political sphere entirely. FUTO should reflect on their values, acknowledge the political nature of their work, and consider the ways in which their work intersects with politics writ large, then make decisions that align their political actions with their political beliefs.


  1. Here’s a quote from another one: “While the vast majority of elite coders have been neutralized by the Oligopoly, the vast majority of those remaining have become preoccupied with cryptocurrency money-making schemes.” ↩︎

  2. Source: What is FUTO? ↩︎

  3. Source: Announcing the Newest Member of FUTO: Louis Rossmann! ↩︎

  4. Source: Politico: The Seven Thinkers and Groups That Have Shaped JD Vance’s Unusual Worldview ↩︎

  5. Source: Inc.: Controversy Rages Over ‘Pro-Slavery’ Tech Speaker Curtis Yarvin ↩︎

  6. Source: The Guardian: Video platform chief says Nazi posts on white superiority do not merit removal ↩︎

  7. You know, authoritarianism, anti-democracy, genocide, etc. ↩︎

Cloudflare bankrolls fascists

US politics has been pretty fascist lately. The state is filling up concentration camps, engaging in mass state violence against people on the basis of racialized traits, deporting them to random countries without any respect for habeas corpus, exerting state pressure on the free press to censor speech critical of the current administration, and Trump is openly floating the idea of an unconstitutional third term.

Fascism is clearly on the rise, and they’re winning more and more power. None of this is far removed from us in the FOSS community – there are a number of fascists working in FOSS, same as the rest of society. I don’t call them fascists baselessly – someone who speaks out in support of and expresses solidarity with fascists, or who uses fascists dog-whistles or promotes fascist ideology and talking points, or boosts fascist conspiracy theories – well, they’re a fascist.

If one consistently speaks in support of a certain political position and against the opponents of that position then it is correct to identify them with this political position. Facts, as it were, don’t care about feelings, namely the feelings that get hurt when someone is called a fascist. Fascists naturally do not want to be identified as such and will reject the label, but we shouldn’t take their word for it. People should be much more afraid of being called out as fascist than they are afraid of calling someone a fascist. If someone doesn’t want to be called a fascist, they shouldn’t act like one.

It’s in this disturbing political context that I saw an odd post from the Cloudflare blog pop up in my circles this week: Supporting the future of the open web: Cloudflare is sponsoring Ladybird and Omarchy. Based on Ladybird’s sponsorship terms we can assume that these projects received on the order of $100,000 USD from Cloudflare. I find this odd for a few reasons, in particular because one thing that I know these two projects have in common is that they are both run by fascists.

Even at face value this is an unusual pair of projects to fund. I’m all for FOSS projects getting funded, of course, and I won’t complain about a project’s funding on the solitary basis that it’s an odd choice. I will point out that these are odd choices, though, especially Omarchy.

Ladybird makes some sense, given that it’s aligned in principle with Cloudflare’s stated objective to “support the open web”, though I remain bearish that new web browser engines are even possible to make.1 Omarchy is a bizarre choice, though – do we really need another pre-customized Arch Linux distribution? And if we do, do we really need a big corporation like Cloudflare to bankroll it? Everyone on /r/unixporn manages to make Arch Linux look pretty for free.

Omarchy is a very weird project to fund, come to think of it. Making an Arch Linux spin technically requires some work, and work is work, I won’t deny it, but most of the work done here is from Arch Linux and Hyprland. Why not fund those, instead? Well, don’t fund Hyprland, since it’s also run by a bunch of fascists, but you get my point.

Anyway, Omarchy and Ladybird are both run by fascists. Omarchy makes this pretty obvious from the outset – on the home page the big YouTube poster image prominently features SuperGrok, which is a pathetically transparent dog-whistle to signal alliance with Elon Musk’s fascist politics. Omarchy is the pet project of David Heinemeier Hansson, aka DHH, who is well known as a rich fascist weirdo.2 One need only consult his blog to browse his weird, racist views on immigration, fat-shaming objections to diverse representation, vaguely anti-feminist/homophobic/rapey rants on consent, and, recently, tone-policing antifascists who celebrate the death of notable fascist Charlie Kirk.

Speaking of tributes to Charlie Kirk, that brings us to Andreas Kling, the project lead for Ladybird, who tweeted on the occasion of his assassination:

RIP Charlie Kirk

I hope many more debate nerds carry on his quest to engage young people with words, not fists.

@awesomekling

Kling has had a few things to say about Kirk on Twitter lately. Here’s another one – give you three guesses as to which “[group]” he objects to punching. You may also recall that Kling achieved some notoriety for his obnoxious response as the maintainer of SerenityOS when someone proposed gender-neutral language for the documentation:

Screenshot of the interaction on GitHub. Kling responds “This project is not an appropriate arena to advertise your personal politics.”

Replacing “he” with “they” in one sentence of the documentation is the kind of “ideologically motivated change” that Serenity’s CONTRIBUTING.md apparently aims to prevent, a classic case of the sexist “identities that are not men are inherently political” nonsense. Ladybird has a similar, weirdly defensive policy on “neutrality”, and a milquetoast code of conduct, which is based on the Ruby Community Conduct Guideline, which has been itself the subject of many controversies due to its inadequacy leading to real-world incidents of harassment and abuse.

Here’s another one – Kling endorsing white replacement theory in June:

White males are actively discriminated against in tech.

It’s an open secret of Silicon Valley.

One of the last meetings I attended before leaving Apple (in 2017) was management asking us to “keep the corporate diversity targets in mind” when interviewing potential new hires.

The phrasing was careful, but the implication was pretty clear.

I knew in my heart this wasn’t wholesome, but I was too scared to rock the boat at the time.

@awesomekling replying to @danheld3

And in a moment of poetic irony, a few days ago Kling spoke in solidarity with Hansson over his “persecution” for “banal, mainstream positions” on Twitter just a few days ago, in response to Hansson’s tweet signal-boosting another notable reactionary tech fascist, Bryan Lunduke.

So, to sum it up, Kling wears his mask a bit better than Hansson, but as far as I’m concerned it seems clear that both projects are run by fascists. If it walks like a fascist and quacks like a fascist… then why is Cloudflare giving them hundreds of thousands of dollars?


  1. I did think it was cute that Kling used a screenshot of Ladybird rendering my blog post back in 2023 to headline one of his articles on Ladybird, though. ↩︎

  2. And a multi-millionaire who can fund his “Arch Linux for Obnoxious Reactionary Assholes” project his own damn self, in my opinion. He also does not want to be called a fascist, in which case he should probably stop being so fucking faschy. ↩︎

  3. Who is ultimately responding to @shaunmmaguire’s tweet lying about being told he wouldn’t be promoted at Google for being white. Shaun also posts about the threat of “Radical Islamist” “Marxists” to whom Europe “will be lost”, falsely accuse the “far left” of committing “almost all mass murders and political violence” in the last few years, and praise Trump for classifying antifascists as terrorists. ↩︎

A better future for JavaScript that won't happen

In the wake of the largest supply-chain attack in history, the JavaScript community could have a moment of reckoning and decide: never again. As the panic and shame subsides, after compromised developers finish re-provisioning their workstations and rotating their keys, the ecosystem might re-orient itself towards solving the fundamental flaws that allowed this to happen.

After all, people have been sounding the alarm for years that this approach to dependency management is reckless and dangerous and broken by design. Maybe this is the moment when the JavaScript ecosystem begins to understand the importance and urgency of this problem, and begins its course correction. It could leave behind its sprawling dependency trees full of micro-libraries, establish software distribution based on relationships of trust, and incorporate the decades of research and innovation established by more serious dependency management systems.

Perhaps Google and Mozilla, leaders in JavaScript standards and implementations, will start developing a real standard library for JavaScript, which makes micro-dependencies like left-pad a thing of the past. This could be combined with a consolidation of efforts, merging micro-libraries into larger packages with a more coherent and holistic scope and purpose, which prune their own dependency trees in turn.

This could be the moment where npm comes to terms with its broken design, and with a well-funded effort (recall that, ultimately, npm is GitHub is Microsoft, market cap $3 trillion USD), will develop and roll out the next generation of package management for JavaScript. It could incorporate the practices developed and proven in Linux distributions, which rarely suffer from these sorts of attacks, by de-coupling development from packaging and distribution, establishing package maintainers who assemble and distribute curated collections of software libraries. By introducing universal signatures for packages of executable code, smaller channels and webs of trust, reproducible builds, and the many other straightforward, obvious techniques used by responsible package managers.

Maybe other languages that depend on this broken dependency management model, like Cargo, PyPI, RubyGems, and many more, are watching this incident and know that the very same crisis looms in their future. Maybe they will change course, too, before the inevitable.

Imagine if other large corporations who depend on and profit from this massive pile of recklessly organized software committed their money and resources to it, through putting their engineers to the task of fixing these problems, through coming together to establish and implement new standards, through direct funding of their dependencies and by distributing money through institutions like NLNet, ushering in an era of responsible, sustainable, and secure software development.

This would be a good future, but it’s not the future that lies in wait for us. The future will be more of the same. Expect symbolic gestures – mandatory 2FA will be rolled out in more places, certainly, and the big players will write off meager donations in the name of “OSS security and resilience” in their marketing budgets.

No one will learn their lesson. This has been happening for decades and no one has learned anything from it yet. This is the defining hubris of this generation of software development.

LibreWolf

I've hit the last straw with Mozilla's antics in Firefox. I've been a Firefox champion for years and years. But every danged upgrade has some new insane junk I don't want or need and they keep popping up these little "tips" to rub my nose in it. I'm sick of it. I'm done. THANKFULLY we have LibreWolf to solve all of that. I'm a heavy browser user migrating from the fox to the wolf and here are my notes so far...
❌