Reading view

There are new articles available, click to refresh the page.

Conservapedia Still Exists

By: Nick Heer

I am not sure it is worth writing at length about Grokipedia, the Elon Musk-funded effort to quite literally rewrite history from the perspective of a robot taught to avoid facts upsetting to the U.S. far right. Perhaps it will be an unfortunate success — the Fox News of encyclopedias, giving ideologues comfortable information as they further isolate themselves.

It is less a Wikipedia competitor than it is a machine-generated alternative to Conservapedia. Founded by Andy Schlafly, an attorney and son of Phyllis Schlafly, the Wikipedia alternative was an attempt to make an online encyclopedia from a decidedly U.S. conservative and American exceptionalism perspective. Seventeen years ago, Schlafly’s effort was briefly profiled by Canadian television and, somehow, the site is still running. Perhaps that is the fate of Grokipedia: a brief curiosity, followed by traffic coming only from a self-selecting mix of weirdos and YouTubers needing material.

⌥ Permalink

A Profile of Setlist.fm

By: Nick Heer

Marc Hogan, New York Times (gift link):

Enter Setlist.fm. The wikilike site, where users document what songs artists play each night on tour, has grown into a vast archive, updated in real time but also reaching back into the historical annals. From the era of Mozart (seriously!) to last night’s Chappell Roan show, Setlist.fm offers reams of statistics — which songs artists play most often, when they last broke out a particular tune. In recent years, the site has begun posting data about average concert start times and set lengths.

Good profile. I had no idea it was owned by Live Nation.

I try to avoid Setlist.fm ahead of a show, but I check it immediately when I get home and for the days following. I might be less familiar with an artist’s catalogue, and this is particularly true of an opener, so it lets me track down particular songs that were played. It is one of the internet’s great resources.

⌥ Permalink

Zoom CEO Eric Yuan Lies About A.I. Leading to Shorter Work Weeks

By: Nick Heer

Sarah Perez, TechCrunch:

Zoom CEO Eric Yuan says AI will shorten our workweek

[…]

“Today, I need to manually focus on all those products to get work done. Eventually, AI will help,” Yuan said.

“By doing that, we do not need to work five days a week anymore, right? … Five years out, three days or four days [a week]. That’s a goal,” he said.

So far, technological advancements have not — in general — produced a shorter work week; that was a product of collective labour action. We have been promised a shorter week before. We do not need to carry water for people who peddle obvious lies. We will always end up being squeezed for greater output.

⌥ Permalink

Colorado Police Officer Caught on Doorbell Camera Talking About Surveillance Powers

By: Nick Heer

Andrew Kenney, Denverite:

It was Sgt. Jamie Milliman [at the door], a police officer with the Columbine Valley Police Department who covers the town of Bow Mar, which begins just south of [Chrisanna] Elser’s home.

[…]

“You know we have cameras in that jurisdiction and you can’t get a breath of fresh air, in or out of that place, without us knowing, correct?” he said.

“OK?” Elser, a financial planner in her 40s, responded in a video captured by her smart doorbell and viewed by Denverite.

“Just as an example,” the sergeant told her, she had “driven through 20 times the last month.”

This story is a civil liberties rollercoaster. Milliman was relying on a nearby town’s use of Flock license plate cameras and Ring doorbells — which may also be connected to the Flock network — to accuse Elser of theft and issue a summons. Elser was able to get the summons dropped by compiling evidence from, in part, the cameras and GPS system on her truck. Milliman’s threats were recorded by a doorbell camera, too. The whole thing is creepy, and all over a $25 package stolen off a doorstep.

I have also had things stolen from me, and I wish the police officers I spoke to had a better answer for me than shrugging their shoulders and saying, in effect, this is not worth our time. But this situation is like a parallel universe ad for Amazon and its Ring subsidiary. Is this the path toward “very close to zero[ing] out crime”? It is not worth it.

⌥ Permalink

Apple’s Tedious and Expensive Procedure for Replacing the Battery in the New MacBook Pro

By: Nick Heer

Carsten Frauenheim and Elizabeth Chamberlain, iFixit:

Apple’s official replacement process requires swapping the entire top case, keyboard and all, just to replace this single consumable component. And it has for a long time. That’s a massive and unreasonable job, requiring complete disassembly and reassembly of the entire device. We’re talking screws, shields, logic board, display, Touch ID, trackpad, everything. In fact, the only thing that doesn’t get transferred are the keyboard and speakers. The keyboard is more or less permanently affixed to this top aluminum, and the speakers are glued in — which, I guess, according to Apple means that the repair is out of the scope of DIY (we disagree).

At least one does not need to send in their laptop for a mere battery replacement. Still, I do not understand why this — the most predictable repair — is so difficult and expensive.

I hate to be that guy, but the battery for a mid-2007 15-inch MacBook Pro used to cost around $150 (about $220 inflation-adjusted) and could be swapped with two fingers. The official DIY solution for replacing the one in my M1 MacBook Pro is over $700, though there is a $124 credit for returning the replaced part. The old battery was, of course, a little bit worse: 60 watt-hours compared to 70 watt-hours in the one I am writing this with. I do not even mind the built-in-ness of this battery. But it should not cost an extra $500 and require swapping the rest of the top case parts.

[…] But for now, this tedious and insanely expensive process is the only offering they make for changing out a dead battery. Is it just a byproduct of this nearly half-a-decade-old chassis design, something that won’t change until the next rethink? We don’t know.

“Nearly half-a-decade-old” is a strange way of writing “four years”, almost like it is attempting to emphasize the age of this design. Four years old does not seem particularly ancient to me. I thought iFixit’s whole vibe was motivating people to avoid the consumerist churn encouraged by rapid redesigns.

⌥ Permalink

Reddit Sues Perplexity and Three Data Scraping Companies Because They Crawled Google

By: Nick Heer

Matt O’Brien, Associated Press:

Social media platform Reddit sued the artificial intelligence company Perplexity AI and three other entities on Wednesday, alleging their involvement in an “industrial-scale, unlawful” economy to “scrape” the comments of millions of Reddit users for commercial gain.

[…]

Also named in the lawsuit are Lithuanian data-scraping company Oxylabs UAB, a web domain called AWMProxy that Reddit describes as a “former Russian botnet,” and Texas-based startup SerpApi, which lists Perplexity as a customer on its website.

Mike Masnick, Techdirt:

Most reporting on this is not actually explaining the nuances, which require a deeper understanding of the law, but fundamentally, Reddit is NOT arguing that these companies are illegally scraping Reddit, but rather that they are illegally scraping… Google (which is not a party to the lawsuit) and in doing so violating the DMCA’s anti-circumvention clause, over content Reddit holds no copyright over. And, then, Perplexity is effectively being sued for linking to Reddit.

This is… bonkers on so many levels. And, incredibly, within their lawsuit, Reddit defends its arguments by claiming it’s filing this lawsuit to protect the open internet. It is not. It is doing the exact opposite.

I am glad Masnick wrote about this despite my disagreement with his views on how much control a website owner ought to have over scraping. This is a necessary dissection of the suit, though I would appreciate views on it from actual intellectual property lawyers. They might be able to explain how a positive outcome of this case for Reddit would have clear rules delineating this conduct from the ways in which artificial intelligence companies have so far benefitted from a generous reading of fair use and terms of service documents.

⌥ Permalink

Apple Threatens to Withdraw App Tracking Transparency in Europe

By: Nick Heer

Andrej Sokolow, Deutsche Presse Agentur:

Apple could switch off a function that prevents users’ apps from tracking their behaviour across various services and websites for advertising purposes in Germany and other European countries.

The iPhone manufacturer on Wednesday complained that it has experienced constant headwinds from the tracking industry.

“Intense lobbying efforts in Germany, Italy and other countries in Europe may force us to withdraw this feature to the detriment of European consumers,” Apple said in a statement.

It is a little rich for Apple to be claiming victimhood in the face of “intense lobbying efforts” by advertising companies when it is the seventh highest spender on lobbying in the European Union. Admittedly, it spends about one-third as much as Meta in Germany, but that is not because Apple cannot afford to spend more. Apple’s argument is weak.

In any case, this is another case where Apple believes it should have a quasi-regulatory role. As I wrote last month:

[…] Apple seems to believe it is its responsibility to implement technical controls to fulfill its definition of privacy and, if that impacts competition and compatibility, too bad. E.U. regulators seem to believe it has policy protections for user privacy, and that users should get to decide how their private data is shared.

I believe there are people within Apple who care deeply about privacy. However, when Apple also gets to define privacy and tracking, it is no coincidence it found an explanation allowing it to use platform activity and in-app purchases for ad targeting. This is hardly as sensitive as the tracking performed by Google and Meta, and Apple does not use third-party data for targeting.

But why would it? Apple owns the platform and, if it wanted, could exploit far more user information without it being considered “tracking” since it is all first-party data. That it does not is a positive reflection of self-policing and, ideally, something it will not change. But it could.

What E.U. authorities are concerned about is this self-serving definition of privacy and the self-policing that results, conflicting with the role of European regulators and privacy laws, and its effects on competition. I think those are reasonable grounds for questioning the validity of App Tracking Transparency. Furthermore, the consequences emanating from violations of privacy law are documented; Meta was penalized €1.2 billion as a result of GDPR violations. Potential violations of App Store policy, on the other hand, are handled differently. If Meta has, as a former employee alleges, circumvented App Tracking Transparency, would the penalties be handled by similar regulatory bodies, or would it — like Uber before — be dealt with privately and rather quietly?

The consequences of previous decisions have been frustrating. They result in poorer on-device privacy controls for users in part because Apple is a self-interested party. It would be able to make its case more convincingly if it walked away from the advertising business altogether.

Sokolow:

Apple argues that it has proposed various solutions to the competition authorities, but has not yet been able to dispel their concerns.

The company wants to continue to offer ATT to European users. However, it argued that the competition authorities have proposed complex solutions that would effectively undermine the function from Apple’s point of view.

Specificity would be nice. It would be better if these kinds of conversations could be had in public instead of in vague statements provided on background to select publications.

⌥ Permalink

The Verge Delivers a Bad Article About Amazon’s Ring

By: Nick Heer

Jennifer Pattison Tuohy, of the Verge, interviewed Ring founder Jamie Siminoff about a new book — which Tuohy has not read — written with Andrew Postman about the success of the company. During this conversation, Tuohy stumbled into Siminoff making a pretty outrageous claim:

While research suggests that today’s video doorbells do little to prevent crime, Siminoff believes that with enough cameras and with AI, Ring could eliminate most of it. Not all crime — “you’ll never stop crime a hundred percent … there’s crimes that are impossible to stop,” he concedes — but close.

“I think that in most normal, average neighborhoods, with the right amount of technology — not too crazy — and with AI, that we can get very close to zero out crime. Get much closer to the mission than I ever thought,” he says. “By the way, I don’t think it’s 10 years away. That’s in 12 to 24 months … maybe even within a year.”

If this sounds ridiculous to you, congratulations, you are thinking harder than whomever wrote the headline on this article:

Ring’s CEO says his cameras can almost ‘zero out crime’ within the next 12 months

The word “almost” and the phrase “very close” are working very hard to keep the core of Siminoff’s claim intact. What he says is that, by this time next year, “normal” communities with enough Ring cameras and a magic dusting of A.I. will have virtually no crime. The caveats are there to imply more nuance, but they are merely an escape hatch for when someone revisits this next year.

The near-complete elimination of crime in “normal” areas — whatever that means — will very obviously not happen. Tuohy cites a 2023 Scientific American story which, in turn, points to articles in MIT Technology Review and CNet. The first debunks a study Ring likes to promote claiming its devices drove a 55% decline in burglaries in Wilshire Park, Los Angeles in 2015, with cameras on about forty homes. Not only does the public data does not support this dramatic reduction, but:

Even if the doorbells had a positive effect, it seemed not to last. In 2017, Wilshire Park suffered more burglaries than in any of the previous seven years.

The CNet article collects a series of reports from other police departments indicating Ring cameras have questionable efficacy at deterring crime on a city-wide level.

This is also something we can know instinctually, since we already have plenty of surveillance cameras. A 2019 meta analysis (PDF) by Eric Piza, et al., found CCTV adoption decreased crime by about 13%. That is not nothing, but it is also a long way from nearly 100%. One could counter that these tests did not factor in Ring’s A.I. features, like summaries of what the camera saw — we have spent so much energy creating summary-making machines — and finding lost dogs.

The counterargument to all of this, however, is that Ring’s vision is a police state enforced by private enterprise. A 2022 paper (PDF) by Dan Calacci, et al., found race was, unsurprisingly, a motivating factor in reports of suspicious behaviour, and that reports within Ring’s Neighbors app was not correlated with the actual frequency of those crimes. Ring recently partnered with Flock, adding a further layer of creepiness.

I will allow that perhaps an article about Siminoff’s book is not the correct place to litigate these claims. By the very same logic, however, the Verge should be more cautious in publishing them, and should not have promoted them in a headline.

⌥ Permalink

App Store Restrictions Face Scrutiny in China, U.K.

By: Nick Heer

Liam Mo and Brenda Goh, Reuters:

A group of 55 Chinese iPhone and iPad users filed a complaint with China’s market regulator on Monday, a lawyer representing the group said, alleging that Apple abuses its market dominance by restricting app distribution and payments to its own platforms while charging high commissions.

[…]

This marks the second complaint against Apple led by Wang. A similar case filed in 2021 was dismissed by a Shanghai court last year.

Imran Rahman-Jones, BBC News:

But the Competition and Markets Authority (CMA) has designated both Apple and Google as having “strategic market status” – effectively saying they have a lot of power over mobile platforms.

The ruling has drawn fury from the tech giants, with Apple saying it risked harming consumers through “weaker privacy” and “delayed access to new features”, while Google called the decision “disappointing, disproportionate and unwarranted”.

The CMA said the two companies “may be limiting innovation and competition”.

Pretty soon it may be easier to list the significant markets in which Apple is still able to exercise complete control over iOS app distribution.

⌥ Permalink

OpenAI Launches ChatGPT Atlas

By: Nick Heer

Maxwell Zeff, TechCrunch:

OpenAI announced Tuesday the launch of its AI-powered browser, ChatGPT Atlas, a major step in the company’s quest to unseat Google as the main way people find information online.

The company says Atlas will first roll out on macOS, with support for Windows, iOS, and Android coming soon. OpenAI says the product will be available to all free users at launch.

Atlas, like Perplexity’s Comet, is a Chromium-based browser. You cannot use it without signing in to ChatGPT. As I was completing the first launch experience, shimmering colours radiated from the setup window and — no joke — it looked like my computer’s screen was failing.

OpenAI:

As you use Atlas, ChatGPT can get smarter and more helpful, too. Browser memories let ChatGPT remember context from the sites you visit and bring that context back when you need it. This means you can ask ChatGPT questions like: “Find all the job postings I was looking at last week and create a summary of industry trends so I can prepare for interviews.” Browser memories in Atlas are completely optional, and you’re always in control: you can view or archive them at any time in settings, and deleting browsing history deletes any associated browser memories.

I love the idea of this. So often, I need to track down something I remember reading, but have only the haziest recollection of what, exactly, it is. I want this in my life. Yet I have zero indication I can trust OpenAI with retaining and synthesizing useful information from my browsing history.

The company says it only retains pages until they have been summarized, and I am sure it thinks it is taking privacy as seriously as it can. But what about down the road? What could it do with all of this data it does retain — information that is tied to your ChatGPT account? OpenAI wants to be everywhere, and it wants to know everything about you to an even greater extent than Google or Meta have been able to accomplish. Why should I trust it? What makes the future of OpenAI look different than the trajectories of the information-hungry businesses before it?

⌥ Permalink

Federico Viticci’s M5 iPad Pro Review

By: Nick Heer

Even if you are not interested in the iPad or Apple product news generally, I recommend making time for Federico Viticci’s review, at MacStories, of the new iPad Pro. Apple claims 3.5× performance gains with A.I. models, so Viticci attempted to verify that number. Unfortunately, he ran into some problems.

Viticci (emphasis his):

This is the paradox of the M5. Theoretically speaking, the new Neural Accelerator architecture should lead to notable gains in token generation and prefill time that may be appreciated on macOS by developers and AI enthusiasts thanks to MLX (more on this below). However, all these improvements amount to very little on iPadOS today because there is no serious app ecosystem for local AI development and tinkering on iPad. That ecosystem absolutely exists on the Mac. On the iPad, we’re left with a handful of non-MLX apps from the App Store, no Terminal, and the untapped potential of the M5.

In case it’s not clear, I’m coming at this from a perspective of disappointment, not anger. […]

Viticci’s frustration with the state of A.I. models on the iPad Pro is palpable. Ideally and hopefully, it is a future-friendly system, but that is not usually the promise of Apple’s products. It usually likes to tell a complete story with the potential for sequels. To get even a glimpse of what that story looks like, Viticci had to go to great lengths, as documented in his review.

In the case of this iPad Pro, it is marketing leaps-and-bounds boosts in A.I. performance — though those claims appear to be optimistic — while still playing catch-up on last year’s Apple intelligence announcements, and offering little news for a user who wants to explore A.I. models directly on their iPad. It feels like a classic iPad story: incredible hardware, restricted by Apple’s software decisions.

Update: I missed a followup post from Viticci in which he points to a review from Max Weinbach of Creative Strategies. Weinbach found the M5 MacBook Pro does, indeed, post A.I. performance gains closer to Apple’s claims.

As an aside, I think it is curious for Apple to be supplying review units to Creative Strategies. It is nominally a research and analysis firm, not a media outlet. While there are concerns about the impartiality of reviewers granted access to prerelease devices, it feels to me like an entirely different thing for a broad-ranging research organization for reasons I cannot quite identify.

⌥ Permalink

Long Lines for Election Day in Alberta

By: Nick Heer

Ken MacGillivray and Karen Bartko, Global News:

“All electors are legislatively required to complete a Statement of Eligibility form (Form 13) at the voting station. This form is a declaration by an elector that they meet the required legislated criteria to receive and cast ballots,” Elections Edmonton said.

[…]

Those casting ballots say confirming voters are on the register or completing the necessary paperwork takes three to five minutes per voter.

I was lucky to be in and out of my polling place in about fifteen minutes, but the longest part was waiting for the person to diligently copy my name, address, and date-of-birth from my driver’s license to a triplicate form, immediately after confirming the same information on the printed voter roll. It is a silly requirement coming down as part of a larger unwanted package from our provincial government for no clear reason. The same legislation also prohibits electronic tabulation, so all the ballots are slowly being counted by hand. These are the kinds of measures that only begin to make sense if you assume someone with influence in our provincial government watches too much Fox News.

I wonder if our Minister of Red Tape Reduction has heard about all the new rules and restrictions implemented by his colleagues.

⌥ Permalink

The Blurry Future of Sora

By: Nick Heer

Jason Parham, Wired:

The uptick in artificial social networks, [Rudy] Fraser tells me, is being driven by the same tech egoists who have eroded public trust and inflamed social isolation through “divisive” algorithms. “[They] are now profiting on that isolation by creating spaces where folks can surround themselves with sycophantic bots.”

I saw this quote circulating on Bluesky over the weekend and it has been rattling around my head since. It cuts to the heart of one reason why A.I.-based “social” networks like Sora and Meta’s Vibes feel so uncomfortable.

Unfortunately, I found the very next paragraph from Parham uncompelling:

In the many conversations I had with experts, similar patterns of thought emerged. The current era of content production prioritizes aesthetics over substance. We are a culture hooked on optimization and exposure; we crave to be seen. We live on our phones and through our screens. We’re endlessly watching and being watched, submerged in a state of looking. With a sort of all-consuming greed, we are transforming into a visual-first society — an infinite form of entertainment for one another to consume, share, fight over, and find meaning through.

Of course our media reflects aesthetic trends and tastes; it always has. I do not know that there was a halcyon era of substance-over-style media, nor do I believe there was a time since celebrity was a feasible achievement in which at least some people did not desire it. In a 1948 British survey of children 10–15 years old, one-sixth to one-third of respondents aspired to “‘romantic’ [career] choices like film acting, sport, and the arts”. An article published in Scouting Magazine in 2000 noted children leaned toward high-profile careers — not necessarily celebrity, but jobs “every child is exposed to”. We love this stuff because we have always loved this stuff.

Among the bits I quibble with in the above, however, this stood out as a new and different thing: “[w]e’re endlessly watching and being watched”. That, I think, is the kind of big change Fraser is quoted as speaking about, and something I think is concerning. We already worried about echo chambers, and platforms like YouTube responded by adjusting recommendations to less frequently send users to dark places. Let us learn something, please.

Cal Newport:

A company that still believes that its technology was imminently going to run large swathes of the economy, and would be so powerful as to reconfigure our experience of the world as we know it, wouldn’t be seeking to make a quick buck selling ads against deep fake videos of historical figures wrestling. They also wouldn’t be entertaining the idea, ​as [Sam] Altman did last week​, that they might soon start offering an age-gated version of ChatGPT so that adults could enjoy AI-generated “erotica.”

To me, these are the acts of a company that poured tens of billions of investment dollars into creating what they hoped would be the most consequential invention in modern history, only to finally realize that what they wrought, although very cool and powerful, isn’t powerful enough on its own to deliver a new world all at once.

I do not think Sora smells of desperation, but I do think it is the product of a company that views unprecedented scale as its primary driver. I think OpenAI wants to be everywhere — and not in the same way that a consumer electronics company wants its smartphones to be the category’s most popular, or anything like that. I wonder if Ben Thompson’s view of OpenAI as “the Windows of A.I.” is sufficient. I think OpenAI is hoping to be a ubiquitous layer in our digital world; or, at least, it is behaving that way.

⌥ Permalink

I Bet Normal Users Will Figure Out Which Power Adapter to Buy

By: Nick Heer

John Gruber, responding to my exploration of the MacBook Pro A.C. adapter non-issue:

The problem I see with the MacBook power adapter situation in Europe is that while power users — like the sort of people who read Daring Fireball and Pixel Envy — will have no problem buying exactly the sort of power adapter they want, or simply re-using a good one they already own, normal users have no idea what makes a “good” power adapter. I suspect there are going to be a lot of Europeans who buy a new M5 MacBook Pro and wind up charging it with inexpensive low-watt power adapters meant for things like phones, and wind up with a shitty, slow charging experience.

Maybe. I think it is fair to be concerned about this being another thing people have to think about when buying a laptop. But, in my experience, less technically adept people still believe they need specific cables and chargers, even when they do not.

When I was in college, a friend forgot to bring the extension cable for their MacBook charger. There was an unused printer in the studio, though, so I was able to use the power cable from that because it is an interchangeable standard plug. I see this kind of thing all the time among friends, family members, and colleagues. It makes sense in a world frequently populated by proprietary adapters.

Maybe some people will end up with underpowered USB-C chargers. I bet a lot of people will just go to the Apple Store and buy the one recommended by staff, though.

⌥ Permalink

Latest Beta of Apple’s Operating Systems Adds Another Translucency Control

By: Nick Heer

Chance Miller, 9to5Mac:

You can find the new option [in 26.1 beta 4] on iPhone and iPad by going to the Settings app and navigating to the Display & Brightness menu. On the Mac, it’s available in the “Appearance” menu in System Settings. Here, you’ll see a new Liquid Glass menu with “Clear” and “Tinted” options.

“Choose your preferred look for Liquid Glass. Clear is more transparent, revealing the content beneath. Tinted increases opacity and adds more contrast,” Apple explains.

After Apple made the menu bar translucent in Mac OS X Leopard, it added a preference to make the bar solid after much pushback. When it refreshed the design of Mac OS X in Yosemite with more frosted glass effects, it added controls to Reduce Transparency and Increase Contrast, which replaced the menu bar-specific setting.

Here we are with yet another theme built around translucency, and more complaints about legibility and contrast — Miller writes “Apple says it heard from users throughout the iOS 26 beta testing period that they’d like a setting to manage the opaqueness of the Liquid Glass design”. Now, as has become traditional, there is another way to moderate the excesses of Apple’s new visual language. I am sure there are some who will claim this undermines the entire premise of Liquid Glass, and I do not know that they are entirely wrong. Some might call it greater personalization and customization, too. I think it feels unfocused. Apple keeps revisiting translucency and finding it needs to add more controls to compensate.

⌥ Permalink

NSO Group Banned From Using or Supplying WhatsApp Exploits

By: Nick Heer

Carly Nairn, Courthouse News Service:

U.S. District Judge Phyllis Hamilton said in a 25-page ruling that there was evidence NSO Group’s flagship spyware could still infiltrate WhatApp users’ devices and granted Meta’s request for a permanent injunction.

However, Hamilton, a Bill Clinton appointee, also determined that any damages would need to follow a ratioed amount of compensation based on a legal framework designed to proportion damages. She ordered that the jury-based award of $167 million should be reduced to a little over $4 million.

Once again, I am mystified by Apple’s decision to drop its suit against NSO Group. What Meta won is protection from WhatsApp being used as an installation vector for NSO’s spyware; importantly, high-value WhatsApp users won a modicum of protection from NSO’s customers. And, as John Scott-Railton of Citizen Lab points out, NSO has “an absolute TON of their business splashed all over the court records”. There are several depositions from which an enterprising journalist could develop a better understanding of this creepy spyware company.

Last week, NSO Group confirmed it had been acquired by U.S. investors. However, according to its spokesperson, its “headquarters and core operations remain in Israel [and] continues to be fully supervised and regulated by the relevant Israeli authorities”.

Lorenzo Franceschi-Bicchierai, TechCrunch:

NSO has long claimed that its spyware is designed to not target U.S. phone numbers, likely to avoid hurting its chances to enter the U.S. market. But the company was caught in 2021 targeting about a dozen U.S. government officials abroad.

Soon after, the U.S. Commerce Department banned American companies from trading with NSO by putting the spyware maker on the U.S. Entities List. Since then, NSO has tried to get off the U.S. government’s blocklist, as recently as May 2025, with the help of a lobbying firm tied to the Trump administration.

I have as many questions about what this change in ownership could mean for its U.S. relationship as I do about how it affects possible targets.

⌥ Permalink

Sponsor: Magic Lasso Adblock: Incredibly Private and Secure Safari Web Browsing

By: Nick Heer

My thanks to Magic Lasso Adblock for sponsoring Pixel Envy this week.

With over 5,000 five star reviews, Magic Lasso Adblock is simply the best ad blocker for your iPhone, iPad, and Mac.

Designed from the ground up to protect your privacy, Magic Lasso blocks all intrusive ads, trackers, and annoyances. It stops you from being followed by ads around the web and, with App Ad Blocking, it stops your app usage being harvested by ad networks.

So, join over 350,000 users and download Magic Lasso Adblock today.

⌥ Permalink

The New MacBook Pro Is €35 Less Expensive in E.U. Countries, Ships Without a Charger

By: Nick Heer

Are you outraged? Have you not heard? Apple updated its entry-level MacBook Pro with a new M5 chip, and across Europe, it does not ship with an A.C. adapter in the box as standard any more. It still comes with a USB-C to MagSafe cable, and you can add an adapter at checkout, but those meddling E.U. regulators have forced Apple to do something stupid and customer-unfriendly again. Right?

William Gallagher, of AppleInsider, gets it wrong:

Don’t blame Apple this time — if you’re in the European Union or the UK, your new M5 14-inch MacBook Pro or iPad Pro may cost you $70 extra because Apple isn’t allowed to bundle a charger.

First of all, the dollar is not the currency in any of these countries. Second, the charger in European countries is €65, which is more like $76 right now. Third, Apple is allowed to bundle an A.C. adapter, it just needs to offer an option to not include it. Fourth, and most important, is that the new MacBook Pro is less expensive in nearly every region in which the A.C. adapter is now a configure-to-order option — even after adding the adapter.

In Ireland, the MacBook Pro used to start at €1,949; it now starts at €1,849; in France, it was €1,899, and it is now €1,799. As mentioned, the adapter is €65, making these new Macs €35 less with a comparable configuration. The same is true in each Euro-currency country I checked: Germany, Italy, and Spain all received a €100 price cut if you do not want an A.C. adapter, and a €35 price cut if you do.

It is not just countries that use the Euro receiving cuts. In Norway, the new MacBook Pro starts at 2,000 krone less than the one it replaces, and a charger is 849 krone. In Hungary, it is 50,000 forint less, with a charger costing about 30,000 forint. There are some exceptions, too. In Switzerland, the new models are 50 francs less, but a charger is 59 francs. And in the U.K., there is no price adjustment, even though the charger is a configure-to-order option there, too.

Countries with a charger in the box, on the other hand, see no such price adjustment, at least for the ones I have checked. The new M5 model starts at the same price as the M4 it replaces in Canada, Japan, Singapore, and the United States. (For the sake of brevity and because not all of these pages have been recently crawled by the Internet Archive, I have not included links to each comparison. I welcome checking my work, however, and would appreciate an email if I missed an interesting price change.)

Maybe Apple was already planning a €100 price cut for these new models. The M4 was €100 less expensive than the M3 it replaced, for example, so it is plausible. That is something we simply cannot know. What we do know for certain is that these new MacBook Pros might not come with an A.C. adapter, but even if someone adds one at checkout, it still costs less in most places with this option.

Gallagher:

It doesn’t appear that Apple has cut prices of the MacBook Pro or iPad Pro to match, either. That can’t be proven, though, because at least with the UK, Apple generally does currency conversion just by swapping symbols.

It can be proven if you bother to put in thirty minutes’ work.

Joe Rossignol, of MacRumors, also gets it a little wrong:

According to the European Union law database, Apple could have let customers in Europe decide whether they wanted to have a charger included in the box or not, but the company has ultimately decided to not include one whatsoever: […]

A customer can, in fact, choose to add an A.C. adapter when they order their Mac.

⌥ Permalink

OpenAI and Nvidia Are at the Centre of a Trillion-Dollar Circular Investment Economy

By: Nick Heer

Tabby Kinder in New York and George Hammond, Financial Times:

OpenAI has signed about $1tn in deals this year for computing power to run its artificial intelligence models, commitments that dwarf its revenue and raise questions about how it can fund them.

Emily Forgash and Agnee Ghosh, Bloomberg:

For much of the AI boom, there have been whispers about Nvidia’s frenzied dealmaking. The chipmaker bolstered the market by pumping money into dozens of AI startups, many of which rely on Nvidia’s graphics processing units to develop and run their models. OpenAI, to a lesser degree, also invested in startups, some of which built services on top of its AI models. But as tech firms have entered a more costly phase of AI development, the scale of the deals involving these two companies has grown substantially, making it harder to ignore.

The day after Nvidia and OpenAI announced their $100 billion investment agreement, OpenAI confirmed it had struck a separate $300 billion deal with Oracle to build out data centers in the US. Oracle, in turn, is spending billions on Nvidia chips for those facilities, sending money back to Nvidia, a company that is emerging as one of OpenAI’s most prominent backers.

I possess none of the skills most useful to understand what all of this means. I am not an economist; I did not have a secret life as an investment banker. As a layperson, however, it is not comforting to read from some People With Specialized Knowledge that this is similar to historically good circular investments, just at an unprecedented scale, while other People With Specialized Knowledge say this has been the force preventing the U.S. from entering a recession. These articles might be like one of those prescient papers from before the Great Recession. Not a great feeling.

⌥ Permalink

The New ‘Foreign Influence’ Scare

By: Nick Heer

Emmanuel Maiberg, 404 Media:

Democratic U.S. Senators Richard Blumenthal and Elizabeth Warren sent letters to the Department of Treasury Secretary Scott Bessent and Electronic Arts CEO Andrew Wilson, raising concerns about the $55 billion acquisition of the giant American video game company in part by Saudi Arabia’s Public Investment Fund (PIF).

Specifically, the Senators worry that EA, which just released Battlefield 6 last week and also publishes The Sims, Madden, and EA Sports FC, “would cease exercising editorial and operational independence under the control of Saudi Arabia’s private majority ownership.”

“The proposed transaction poses a number of significant foreign influence and national security risks, beginning with the PIF’s reputation as a strategic arm of the Saudi government,” the Senators wrote in their letter. […]

In the late 1990s and early 2000s, the assumption was that it would be democratic nations successfully using the web for global influence. But I think the 2016 U.S. presidential election, during which Russian operatives worked to sway voters’ intentions, was a reality check. Fears of foreign influence were then used by U.S. lawmakers to justify banning TikTok, and to strongarm TikTok into allowing Oracle to oversee its U.S. operations. Now, it is Saudi Arabian investment in Electronic Arts raising concerns. Like TikTok, it is not the next election that is, per se, at risk, but the general thoughts and opinions of people in the United States.

U.S. politicians even passed a law intended to address “foreign influence” concerns. However, Saudi Arabia is not one of the four “covered nations” restricted by PAFACA.

Aside from xenophobia, I worry “foreign influence” is becoming a new standard excuse for digital barriers. We usually associate restrictive internet policies with oppressive and authoritarian regimes that do not trust their citizens to be able to think for themselves. This is not to say foreign influence is not a reasonable concern, nor that Saudi Arabia has no red flags, nor still that these worries are a purely U.S. phenomenon. Canadian officials are similarly worried about adversarial government actors covertly manipulating our policies and public opinion. But I think we need to do better if we want to support a vibrant World Wide Web. U.S. adversaries are allowed to have big, successful digital products, too.

⌥ Permalink

My flailing around with Firefox's Multi-Account Containers

By: cks

I have two separate Firefox environments. One of them is quite locked down so that it blocks JavaScript by default, doesn't accept cookies, and so on. Naturally this breaks a lot of things, so I have a second "just make it work" environment that runs all the JavaScript, accepts all the cookies, and so on (although of course I use uBlock Origin, I'm not crazy). This second environment is pretty risky in the sense that it's going to be heavily contaminated with tracking cookies and so on, so to mitigate the risk (and make it a better environment to test things in), I have this Firefox set to discard cookies, caches, local storage, history, and so on when it shuts down.

In theory how I use this Firefox is that I start it when I need to use some annoying site I want to just work, use the site briefly, and then close it down, flushing away all of the cookies and so on. In practice I've drifted into having a number of websites more or less constantly active in this "accept everything" Firefox, which means that I often keep it running all day (or longer at home) and all of those cookies stick around. This is less than ideal, and is a big reason why I wish Firefox had a 'open this site in a specific profile' feature. Yesterday, spurred on by Ben Zanin's Fediverse comment, I decided to make my "accept everything" Firefox environment more complicated in the pursuit of doing better (ie, throwing away at least some cookies more often).

First, I set up a combination of Multi-Account Containers for the basic multi-container support and FoxyTab to assign wildcarded domains to specific containers. My reason to use Multi-Account Containers and to confine specific domains to specific containers is that both M-A C itself and my standard Cookie Quick Manager add-on can purge all of the cookies and so on for a specific container. In theory this lets me manually purge undesired cookies, or all cookies except desired ones (for example, my active Fediverse login). Of course I'm not likely to routinely manually delete cookies, so I also installed Cookie AutoDelete with a relatively long timeout and with its container awareness turned on, and exemptions configured for the (container-confined) sites that I'm going to want to retain cookies from even when I've closed their tab.

(It would be great if Cookie AutoDelete supported different cookie timeouts for different containers. I suspect it's technically possible, along with other container-aware cookie deletion, since Cookie AutoDelete applies different retention policies in different containers.)

In FoxyTab, I've set a number of my containers to 'Limit to Designated Sites'; for example, my 'Fediverse' container is set this way. The intention is that when I click on an external link in a post while reading my Fediverse feed, any cookies that external site sets don't wind up in the Fediverse container; instead they go either in the default 'no container' environment or in any specific container I've set up for them. As part of this I've created a 'Cookie Dump' container that I've assigned as the container for various news sites and so on where I actively want a convenient way to discard all their cookies and data (which is available through Multi-Account Containers).

Of course if you look carefully, much of this doesn't really require Multi-Account Containers and FoxyTab (or containers at all). Instead I could get almost all of this just by using Cookie AutoDelete to clean out cookies from closed sites after a suitable delay. Containers do give me a bit more isolation between the different things I'm using my "just make it work" Firefox for, and maybe that's important enough to justify the complexity.

(I still have this Firefox set to discard everything when it exits. This means that I have to re-log-in every so often even for the sites where I have Cookie AutoDelete keep cookies, but that's fine.)

I wish Firefox Profiles supported assigning websites to profiles

By: cks

One of the things that Firefox is working on these days is improving Firefox's profiles feature so that it's easier to use them. Firefox also has an existing feature that is similar to profiles, in containers and the Multi-Account Containers extension. The reason Firefox is tuning up profiles is that containers only separate some things, while profiles separate pretty much everything. A profile has a separate set of about:config settings, add-ons, add-on settings, memorized logins, and so on. I deliberately use profiles to create two separate and rather different Firefox environments. I'd like to have at least two or three more profiles, but one reason I've been lazy is that the more profiles I have, the more complex getting URLs into the right profile is (even with tooling to help).

This leads me to my wish for profiles, which is for profiles to support the kind of 'assign website to profile' and 'open website in profile' features that you currently have with containers, especially with the Multi-Account Containers extension. Actually I would like a somewhat better version than Multi-Account Containers currently offers, because as far as I can see you can't currently say 'all subdomains under this domain should open in container X' and that's a feature I very much want for one of my use cases.

(Multi-Account Containers may be able to do wildcarded subdomains with an additional add-on, but on the other hand apparently it may have been neglected or abandoned by Mozilla.)

Another way to get much of what I want would be for some of my normal add-ons to be (more) container aware. I could get a lot of the benefit of profiles (although not all of them) by using Multi-Account Containers with container aware cookie management in, say, Cookie AutoDelete (which I believe does support that, although I haven't experimented). Using containers also has the advantage that I wouldn't have to maintain N identical copies of my configuration for core extensions and bookmarklets and so on.

(I'm not sure what you can copy from one profile to a new one, and you currently don't seem to get any assistance from Firefox for it, at least in the old profile interface. This is another reason I haven't gone wild on making new Firefox profiles.)

Modern Linux filesystem mounts are rather complex things

By: cks

Once upon a time, Unix filesystem mounts worked by putting one inode on top of another, and this was also how they worked in very early Linux. It wasn't wrong to say that mounts were really about inodes, with the names only being used to find the inodes. This is no longer how things work in Linux (and perhaps other Unixes, but Linux is what I'm most familiar with for this). Today, I believe that filesystem mounts in Linux are best understood as namespace operations.

Each separate (unmounted) filesystem is a a tree of names (a namespace). At a broad level, filesystem mounts in Linux take some name from that filesystem tree and project it on top of something in an existing namespace, generally with some properties attached to the projection. A regular conventional mount takes the root name of the new filesystem and puts the whole tree somewhere, but for a long time Linux's bind mounts took some other name in the filesystem as their starting point (what we could call the root inode of the mount). In modern Linux, there can also be multiple mount namespaces in existence at one time, with different contents and properties. A filesystem mount does not necessarily appear in all of them, and different things can be mounted at the same spot in the tree of names in different mount namespaces.

(Some mount properties are still global to the filesystem as a whole, while other mount properties are specific to a particular mount. See mount(2) for a discussion of general mount properties. I don't know if there's a mechanism to handle filesystem specific mount properties on a per mount basis.)

This can't really be implemented with an inode-based view of mounts. You can somewhat implement traditional Linux bind mounts with an inode based approach, but mount namespaces have to be separate from the underlying inodes. At a minimum a mount point must be a pair of 'this inode in this namespace has something on top of it', instead of just 'this inode has something on top of it'.

(A pure inode based approach has problems going up the directory tree even in old bind mounts, because the parent directory of a particular directory depends on how you got to the directory. If /usr/share is part of /usr and you bind mounted /usr/share to /a/b, the value of '..' depends on if you're looking at '/usr/share/..' or '/a/b/..', even though /usr/share and /a/b are the same inode in the /usr filesystem.)

If I'm reading manual pages correctly, Linux still normally requires the initial mount of any particular filesystem be of its root name (its true root inode). Only after that initial mount is made can you make bind mounts to pull out some subset of its tree of names and then unmount the original full filesystem mount. I believe that a particular filesystem can provide ways to sidestep this with a filesystem specific mount option, such as btrfs's subvol= mount option that's covered in the btrfs(5) manual page (or 'btrfs subvolume set-default').

You can add arbitrary zones to NSD (without any glue records)

By: cks

Suppose, not hypothetically, that you have a very small DNS server for a captive network situation, where the DNS server exists only to give clients answers for a small set of hosts. One of the ways you can implement this is with an authoritative DNS servers, such as NSD, that simply has an extremely minimal set of DNS data. If you're using NSD for this, you might be curious how minimal you can be and how much you need to mimic ordinary DNS structure.

Here, by 'mimic ordinary DNS structure', I mean inserting various levels of NS records so there is a more or less conventional path of NS delegations from the DNS root ('.') down to your name. If you're providing DNS clients with 'dog.example.org', you might conventionally have a NS record for '.', a NS record for 'org.', and a NS record for 'example.org.', mimicking what you'd see in global DNS. Of course all of your NS records are going to point to your little DNS server, but they're present if anything looks.

Perhaps unsurprisingly, NSD doesn't require this and DNS clients normally don't either. If you say:

zone:
  name: example.org
  zonefile: example-stub

and don't have any other DNS data, NSD won't object and it will answer queries for 'dog.example.org' with your minimal stub data. This works for any zone, including completely made up ones:

zone:
  name: beyond.internal
  zonefile: beyond-stub

The actual NSD stub zone files can be quite minimal. An older OpenBSD NSD appears to be happy with zone files that have only a $ORIGIN, a $TTL, a '@ IN SOA' record, and what records you care about in the zone.

Once I thought about it, I realized I should have expected this. An authoritative DNS server normally only holds data for a small subset of zones and it has to be willing to answer queries about the data it holds. Some authoritative DNS servers (such as Bind) can also be used as resolving name servers so they'd sort of like to have information about at least the root nameservers, but NSD is a pure authoritative server so there's no reason for it to care.

As for clients, they don't normally do DNS resolution starting from the root downward. Instead, they expect to operate by sending the entire query to whatever their configured DNS resolver is, which is going to be your little NSD setup. In a number of configurations, clients either can't talk directly to outside DNS or shouldn't try to do DNS resolution that way because it won't work; they need to send everything to their configured DNS resolver so it can do, for example, "split horizon" DNS.

(Yes, the modern vogue for DNS over HTTPS puts a monkey wrench into split horizon DNS setups. That's DoH's problem, not ours.)

Since this works for a .net zone, you can use it to try to disable DNS over HTTPS resolvers in your stub DNS environment by providing a .net zone with 'use-application-dns CNAME .' or the like, to trigger at least Firefox's canary domain detection.

(I'm not going to address whether you should have such a minimal stub DNS environment or instead count on your firewall to block traffic and have a normal DNS environment, possibly with split horizon or response policy zones to introduce your special names.)

Some of the things that ZFS scrubs will detect

By: cks

Recently I saw a discussion of my entry on how ZFS scrubs don't really check the filesystem structure where someone thought that ZFS scrubs only protected you from the disk corrupting data at rest, for example due to sectors starting to fail (here). While ZFS scrubs have their limits, they do manage to check somewhat more than this.

To start with, ZFS scrubs check the end to end hardware path for reading all your data (and implicitly for writing it). There are a variety of ways that things in the hardware path can be unreliable; for example, you might have slowly failing drive cables that are marginal and sometimes give you errors on data reads (or worse, data writes). A ZFS scrub has some chance to detect this; if a ZFS scrub passes, you know that as of that point in time you can reliably read all your data from all your disks and that all the data was reliably written.

If a scrub passes, you also know that the disks haven't done anything obviously bad with your data. This can be important if you're doing operations that you consider somewhat exotic, such as telling SSDs to discard unused sectors. If you have ZFS send TRIM commands to a SSD and then your scrub passes, you know that the SSD didn't incorrectly discard some sectors that were actually used.

Related to this, if you do a ZFS level TRIM and then the scrub passes, you know that ZFS itself didn't send TRIM commands that told the SSD to discard sectors that were actually used. In general, if ZFS has a serious problem where it writes the wrong thing to the wrong place, a scrub will detect it (although the scrub can't fix it). Similarly, a scrub will detect if a disk itself corrupted the destination of a write (or a read), or if things were corrupted somewhere in the lower level software and hardware path of the write.

There are a variety of ZFS level bugs that could theoretically write the wrong thing to the wrong place, or do something that works out to the same effect. ZFS could have a bug in free space handling (so that it incorrectly thinks some in use sectors are free and overwrites them), or it could write too much or too little, or it could correctly allocate and write data but record the location of the data incorrectly in higher level data structures, or it could accidentally not do a write (for example, if it's supposed to write a duplicate copy of some data but forgets to actually issue the IO). ZFS scrubs can detect all of these issues under the right circumstances.

(To a limited extent a ZFS scrub also checks the high level metadata of filesystems and snapshots. since it has to traverse that metadata to find the object set for each dataset and similar things. Since a scrub just verifies checksums, this won't cross check dataset level metadata like information on how much data was written in each snapshot, or the space usage.)

What little I want out of web "passkeys" in my environment

By: cks

WebAuthn is yet another attempt to do an API for web authentication that doesn't involve passwords but that instead allows browsers, hardware tokens, and so on to do things more securely. "Passkeys" (also) is the marketing term for a "WebAuthn credential", and an increasing number of websites really, really want you to use a passkey for authentication instead of any other form of multi-factor authentication (they may or may not still require your password).

Most everyone that wants you to use passkeys also wants you to specifically use highly secure ones. The theoretically most secure are physical hardware security keys, followed by passkeys that are stored and protected in secure enclaves in various ways by the operating system (provided that the necessary special purpose hardware is available). Of course the flipside of 'secure' is 'locked in', whether locked in to your specific hardware key (or keys, generally you'd better have backups) or locked in to a particular vendor's ecosystem because their devices are the only ones that can possibly use your encrypted passkey vault.

(WebAuthn neither requires nor standardizes passkey export and import operations, and obviously security keys are built to not let anyone export the cryptographic material from them, that's the point.)

I'm extremely not interested in the security versus availability tradeoff that passkeys make in favour of security. I care far more about preserving availability of access to my variety of online accounts than about nominal high security. So if I'm going to use passkeys at all, I have some requirements:

Linux people: is there a passkeys implementation that does not use physical hardware tokens (software only), is open source, works with Firefox, and allows credentials to be backed up and copied to other devices by hand, without going through some cloud service?

I don't think I'm asking for much, but this is what I consider the minimum for me actually using passkeys. I want to be 100% sure of never losing them because I have multiple backups and can use them on multiple machines.

Apparently KeePassXC more or less does what I want (when combined with its Firefox extension), and it can even export passkeys in a plain text format (well, JSON). However, I don't know if anything else can ingest those plain text passkeys, and I don't know if KeePassXC can be told to only do passkeys with the browser and not try to take over passwords.

(But at least a plain text JSON backup of your passkeys can be imported into another KeePassXC instance without having to try to move, copy, or synchronize a KeePassXC database.)

Normally I would ignore passkeys entirely, but an increasing number of websites are clearly going to require me to use some form of multi-factor authentication, no matter how stupid this is (cf), and some of them will probably require passkeys or at least make any non-passkey option very painful. And it's possible that reasonably integrated passkeys will be a better experience than TOTP MFA with my janky minimal setup.

(Of course KeePassXC also supports TOTP, and TOTP has an extremely obvious import process that everyone supports, and I believe KeePassXC will export TOTP secrets if you ask nicely.)

While KeePassXC is okay, what I would really like is for Firefox to support 'memorized passkeys' right along with its memorized passwords (and support some kind of export and import along with it). Should people use them? Perhaps not. But it would put that choice firmly in the hands of the people using Firefox, who could decide on how much security they did or didn't want, not in the hands of websites who want to force everyone to face a real risk of losing their account so that the website can conduct security theater.

(Firefox will never support passkeys this way for an assortment of reasons. At most it may someday directly use passkeys through whatever operating system services expose them, and maybe Linux will get a generic service that works the way I want it to. Nor is Firefox ever going to support 'memorized TOTP codes'.)

Two reasons why Unix traditionally requires mount points to exist

By: cks

Recently on the Fediverse, argv minus one asked a good question:

Why does #Linux require #mount points to exist?

And are there any circumstances where a mount can be done without a pre-existing mount point (i.e. a mount point appears out of thin air)?

I think there is one answer for why this is a good idea in general and otherwise complex to do, although you can argue about it, and then a second historical answer based on how mount points were initially implemented.

The general problem is directory listings. We obviously want and need mount points to appear in readdir() results, but in the kernel, directory listings are historically the responsibility of filesystems and are generated and returned in pieces on the fly (which is clearly necessary if you have a giant directory; the kernel doesn't read the entire thing into memory and then start giving your program slices out of it as you ask). If mount points never appear in the underlying directory, then they must be inserted at some point in this process. If mount points can sometimes exist and sometimes not, it's worse; you need to somehow keep track of which ones actually exist and then add the ones that don't at the end of the directory listing. The simplest way to make sure that mount points always exist in directory listings is to require them to have an existence in the underlying filesystem.

(This was my initial answer.)

The historical answer is that in early versions of Unix, filesystems were actually mounted on top of inodes, not directories (or filesystem objects). When you passed a (directory) path to the mount(2) system call, all it was used for was getting the corresponding inode, which was then flagged as '(this) inode is mounted on' and linked (sort of) to the new mounted filesystem on top of it. All of the things that dealt with mount points and mounted filesystem did so by inode and inode number, with no further use of the paths and the root inode of the mounted filesystem being quietly substituted for the mounted-on inode. All of the mechanics of this needed the inode and directory entry for the name to actually exist (and V7 required the name to be a directory).

I don't think modern kernels (Linux or otherwise) still use this approach to handling mounts, but I believe it lingered on for quite a while. And it's a sufficiently obvious and attractive implementation choice that early versions of Linux also used it (see the Linux 0.96c version of iget() in fs/inode.c).

Sidebar: The details of how mounts worked in V7

When you passed a path to the mount(2) system call (called 'smount()' in sys/sys3.c), it used the name to get the inode and then set the IMOUNT flag from sys/h/inode.h on it (and put the mount details in a fixed size array of mounts, which wasn't very big). When iget() in sys/iget.c was fetching inodes for you and you'd asked for an IMOUNT inode, it gave you the root inode of the filesystem instead, which worked in cooperation with name lookup in a directory (the name lookup in the directory would find the underlying inode number, and then iget() would turn it into the mounted filesystem's root inode). This gave Research Unix a simple, low code approach to finding and checking for mount points, at the cost of pinning a few more inodes into memory (not necessarily a small thing when even a big V7 system only had at most 200 inodes in memory at once, but then a big V7 system was limited to 8 mounts, see h/param.h).

We can't really do progressive rollouts of disruptive things

By: cks

In a comment on my entry on how we reboot our machines right after updating their kernels, Jukka asked a good question:

While I do not know how many machines there are in your fleet, I wonder whether you do incremental rolling, using a small snapshot for verification before rolling out to the whole fleet?

We do this to some extent but we can't really do it very much. The core problem is that the state of almost all of our machines is directly visible and exposed to people. This is because we mostly operate an old fashioned Unix login server environment, where people specifically use particular servers (either directly by logging in to them or implicitly because their home directory is on a particular NFS fileserver). About the only genuinely generic machines we have are the nodes in our SLURM cluster, where we can take specific unused nodes out of service temporarily without anyone noticing.

(Some of these login servers in use all of the time; others we might find idle if we're extremely lucky. But it's hard to predict when someone will show up to try to use a currently empty server.)

This means that progressively rolling out a kernel update (and rebooting things) to our important, visible core servers requires multiple people-visible reboots of machines, instead of one big downtime when everything is rebooted. Generally we feel that repeated disruptions are much more annoying and disruptive overall to people; it's better to get the pain of reboot disruptions over all at once. It's also much easier to explain to people, and we don't have to annoy them with repeated notifications that yet another subset of our servers and services will be down for a bit.

(To make an incremental deployment more painful for us, these will normally have to be after-hours downtimes, which means that we'll be repeatedly staying late, perhaps once a week for three or four weeks as we progressively work through a rollout.)

In addition to the nodes of our SLURM cluster, there are a number of servers that can be rebooted in the background to some degree without people noticing much. We will often try the kernel update out on a few of them in advance, and then update others of them earlier in the day (or the day before) both as a final check and to reduce the number of systems we have to cover at the actual out of hours downtime. But a lot of our servers cannot really be tested much in advance, such as our fileservers or our web server (which is under constant load for reasons outside the scope of this entry). We can (and do) update a test fileserver or a test web server, but neither will see a production load and it's under production loads that problems are most likely to surface.

This is a specific example of how the 'cattle' model doesn't fit all situations. To have a transparent rolling update that involves reboots (or anything else that's disruptive on a single machine), you need to be able to transparently move people off of machines and then back on to them. This is hard to get in any environment where people have long term usage of specific machines, where they have login sessions and running compute jobs and so on, and where you have have non-redundant resources on a single machine (such as NFS fileservers without transparent failover from server to server).

We don't update kernels without immediately rebooting the machine

By: cks

I've mentioned this before in passing (cf, also) but today I feel like saying it explicitly: our habit with all of our machines is to never apply a kernel update without immediately rebooting the machine into the new kernel. On our Ubuntu machines this is done by holding the relevant kernel packages; on my Fedora desktops I normally run 'dnf update --exclude "kernel*"' unless I'm willing to reboot on the spot.

The obvious reason for this is that we want to switch to the new kernel under controlled, attended conditions when we'll be able to take immediate action if something is wrong, rather than possibly have the new kernel activate at some random time without us present and paying attention if there's a power failure, a kernel panic, or whatever. This is especially acute on my desktops, where I use ZFS by building my own OpenZFS packages and kernel modules. If something goes wrong and the kernel modules don't load or don't work right, an unattended reboot can leave my desktops completely unusable and off the network until I can get to them. I'd rather avoid that if possible (sometimes it isn't).

(In general I prefer to reboot my Fedora machines with me present because weird things happen from time to time and sometimes I make mistakes, also.)

The less obvious reason is that when you reboot a machine right after applying a kernel update, it's clear in your mind that the machine has switched to a new kernel. If there are system problems in the days immediately afterward the update, you're relatively likely to remember this and at least consider the possibility that the new kernel is involved. If you apply a kernel update, walk away without rebooting, and the machine reboots a week and a half later for some unrelated reason, you may not remember that one of the things the reboot did was switch to a new kernel.

(Kernels aren't the only thing that this can happen with, since not all system updates and changes take effect immediately when made or applied. Perhaps one should reboot after making them, too.)

I'm assuming here that your Linux distribution's package management system is sensible, so there's no risk of losing old kernels (especially the one you're currently running) merely because you installed some new ones but didn't reboot into them. This is how Debian and Ubuntu behave (if you don't 'apt autoremove' kernels), but not quite how Fedora's dnf does it (as far as I know). Fedora dnf keeps the N most recent kernels around and probably doesn't let you remove the currently running kernel even if it's more than N kernels old, but I don't believe it tracks whether or not you've rebooted into those N kernels and stretches the N out if you haven't (or removes more recent installed kernels that you've never rebooted into, instead of older kernels that you did use at one point).

PS: Of course if kernel updates were perfect this wouldn't matter. However this isn't something you can assume for the Linux kernel (especially as patched by your distribution), as we've sometimes seen. Although big issues like that are relatively uncommon.

We (I) need a long range calendar reminder system

By: cks

About four years ago I wrote an entry about how your SMART drive database of attribute meanings needs regular updates. That entry was written on the occasion of updating the database we use locally on our Ubuntu servers, and at the time we were using a mix of Ubuntu 18.04 and Ubuntu 20.04 servers, both of which had older drive databases that probably dated from early 2018 and early 2020 respectively. It is now late 2025 and we use a mix of Ubuntu 24.04 and 22.04 servers, both of which have drive databases that are from after October of 2021.

Experienced system administrators know where this one is going: today I updated our SMART drive database again, to a version of the SMART database that was more recent than the one shipped with 24.04 instead of older than it.

It's a fact of life that people forget things. People especially forget things that are a long way away, even if they make little notes in their worklog message when recording something that they did (as I did four years ago). It's definitely useful to plan ahead in your documentation and write these notes, but without an external thing to push you or something to explicitly remind you, there's no guarantee that you'll remember.

All of which leads me to the view that it would be useful for us to have a long range calendar reminder system, something that could be used to set reminders for more than a year into the future and ideally allow us to write significant email messages to our future selves to cover all of the details (although there are hacks around that, such as putting the details on a web page and having the calendar mail us a link). Right now the best calendar reminder system we have is the venerable calendar, which we can arrange to email one-line notes to our general address that reaches all sysadmins, but calendar doesn't let you include the year in the reminder date.

(For SMART drive database updates, we could get away with mailing ourselves once a year in, say, mid-June. It doesn't hurt to update the drive database more than every Ubuntu LTS release. But there are situations where a reminder several years in the future is what we want.)

PS: Of course it's not particularly difficult to build an ad-hoc script system to do this, with various levels of features. But every local ad-hoc script that we write is another little bit of overhead, and I'd like to avoid that kind of thing if at all possible in favour of a standard solution (that isn't a shared cloud provider calendar).

We need to start doing web blocking for non-technical reasons

By: cks

My sense is that for a long time, technical people (system administrators, programmers, and so on) have seen the web as something that should be open by default and by extension, a place where we should only block things for 'technical' reasons. Common technical reasons are a harmful volume of requests or clear evidence of malign intentions, such as probing for known vulnerabilities. Otherwise, if it wasn't harming your website and wasn't showing any intention to do so, you should let it pass. I've come to think that in the modern web this is a mistake, and we need to be willing to use blocking and other measures for 'non-technical' reasons.

The core problem is that the modern web seems to be fragile and is kept going in large part by a social consensus, not technical things such as capable software and powerful servers. However, if we only react to technical problems, there's very little that preserves and reinforces this social consensus, as we're busy seeing. With little to no consequences for violating the social consensus, bad actors are incentivized to skate right up to and even over the line of causing technical problems. When we react by taking only narrow technical measures, we tacitly reward the bad actors for their actions; they can always find another technical way. They have no incentive to be nice or to even vaguely respect the social consensus, because we don't punish them for it.

So I've come to feel that if something like the current web is to be preserved, we need to take action not merely when technical problems arise but also when the social consensus is violated. We need to start blocking things for what I called editorial reasons. When software or people do things that merely shows bad manners and doesn't yet cause us technical problems, we should still block it, either soft (temporarily, perhaps with HTTP 429 Too Many Requests) or hard (permanently). We need to take action to create the web that we want to see, or we aren't going to get it or keep it.

To put it another way, if we want to see good, well behaved browsers, feed readers, URL fetchers, crawlers, and so on, we have to create disincentives for ones that are merely bad (as opposed to actively damaging). In its own way, this is another example of the refutation of Postel's Law. If we accept random crap to be friendly, we get random crap (and the quality level will probably trend down over time).

To answer one potential criticism, it's true that in some sense, blocking and so on for social reasons is not good and is in some theoretical sense arguably harmful for the overall web ecology. On the other hand, the current unchecked situation itself is also deeply harmful for the overall web ecology and it's only going to get worse if we do nothing, with more and more things effectively driven off the open web. We only get to pick the poison here.

I wish SSDs gave you CPU performance style metrics about their activity

By: cks

Modern CPUs have an impressive collection of performance counters for detailed, low level information on things like cache misses, branch mispredictions, various sorts of stalls, and so on; on Linux you can use 'perf list' to see them all. Modern SSDs (NVMe, SATA, and SAS) are all internally quite complex, and their behavior under load depends on a lot of internal state. It would be nice to have CPU performance counter style metrics to expose some of those details. For a relevant example that's on my mind (cf), it certainly would be interesting to know how often flash writes had to stall while blocks were hastily erased, or the current erase rate.

Having written this, I checked some of our SSDs (the ones I'm most interested in at the moment) and I see that our SATA SSDs do expose some of this information as (vendor specific) SMART attributes, with things like 'block erase count' and 'NAND GB written' to TLC or SLC (as well as the host write volume and so on stuff you'd expect). NVMe does this in a different way that doesn't have the sort of easy flexibility that SMART attributes do, so a random one of ours that I checked doesn't seem to provide this sort of lower level information.

It's understandable that SSD vendors don't necessarily want to expose this sort of information, but it's quite relevant if you're trying to understand unusual drive performance. For example, for your workload do you need to TRIM your drives more often, or do they have enough pre-erased space available when you need it? Since TRIM has an overhead, you may not want to blindly do it on a frequent basis (and its full effects aren't entirely predictable since they depend on how much the drive decides to actually erase in advance).

(Having looked at SMART 'block erase count' information on one of our servers, it's definitely doing something when the server is under heavy fsync() load, but I need to cross-compare the numbers from it to other systems in order to get a better sense of what's exceptional and what's not.)

I'm currently more focused on write related metrics, but there's probably important information that could be exposed for reads and for other operations. I'd also like it if SSDs provided counters for how many of various sorts of operations they saw, because while your operating system can in theory provide this, it often doesn't (or doesn't provide them at the granularity of, say, how many writes with 'Force Unit Access' or how many 'Flush' operations were done).

(In Linux, I think I'd have to extract this low level operation information in an ad-hoc way with eBPF tracing.)

A (filesystem) journal can be a serialization point for durable writes

By: cks

Suppose that you have a filesystem that uses some form of a journal to provide durability (as many do these days) and you have a bunch of people (or processes) writing and updating things all over the filesystem that they want to be durable, so these processes are all fsync()'ing their work on a regular basis (or the equivalent system call or synchronous write operation). In a number of filesystem designs, this creates a serialization point on the filesystem's journal.

This is related to the traditional journal fsync() problem, but that one is a bit different. In the traditional problem you have a bunch of changes from a bunch of processes, some of which one process wants to fsync() and most of which it doesn't; this can be handled by only flushing necessary things. Here we have a bunch of processes making a bunch of relatively independent changes but approximately all of the processes want to fsync() their changes.

The simple way to get durability (and possibly integrity) for fsync() is to put everything that gets fsync()'d into the journal (either directly or indirectly) and then force the journal to be durably committed to disk. If the filesystem's journal is a linear log, as is usually the case, this means that multiple processes mostly can't be separately writing and flushing journal entries at the same time. Each durable commit of the journal is a bottleneck for anyone who shows up 'too late' to get their change included in the current commit; they have to wait for the current commit to be flushed to disk before they can start adding more entries to the journal (but then everyone can be bundled into the next commit).

In some filesystems, processes can readily make durable writes outside of the journal (for example, overwriting something in place); such processes can avoid serializing on a linear journal. Even if they have to put something in the journal, you can perhaps minimize the direct linear journal contents by having them (durably) write things to various blocks independently, then put only compact pointers to those out of line blocks into the linear journal with its serializing, linear commits. The goal is to avoid having someone show up wanting to write megabytes 'to the journal' and forcing everyone to wait for their fsync(); instead people serialize only on writing a small bit of data at the end, and writing the actual data happens in parallel (assuming the disk allows that).

(I may have made this sound simple but the details are likely fiendishly complex.)

If you have a filesystem in this situation, and I believe one of them is ZFS, you may find you care a bunch about the latency of disks flushing writes to media. Of course you need the workload too, but there are certain sorts of workloads that are prone to this (for example, traditional Unix mail spools).

I believe that you can also see this sort of thing with databases, although they may be more heavily optimized for concurrent durable updates.

Sidebar: Disk handling of durable writes can also be a serialization point

Modern disks (such as NVMe SSDs) broadly have two mechanism to force things to durable storage. You can issue specific writes of specific blocks with 'Force Unit Access' (FUA) set, which causes the disk to write those blocks (and not necessarily any others) to media, or you can issue a general 'Flush' command to the disk and it will write anything it currently has in its write cache to media.

If you issue FUA writes, you don't have to wait for anything else other than your blocks to be written to media. If you issue 'Flush', you get to wait for everyone's blocks to be written out. This means that for speed you want to issue FUA writes when you want things on media, but on the other hand you may have already issued non-FUA writes for some of the blocks before you found out that you wanted them on media (for example, if someone writes a lot of data, so much that you start writeback, and then they issue a fsync()). And in general, the block IO programming model inside your operating system may favour issuing a bunch of regular writes and then inserting a 'force everything before this point to media' fencing operation into the IO stream.

NVMe SSDs and the question of how fast they can flush writes to flash

By: cks

Over on the Fediverse, I had a question I've been wondering about:

Disk drive people, sysadmins, etc: would you expect NVMe SSDs to be appreciably faster than SATA SSDs for a relatively low bandwidth fsync() workload (eg 40 Mbytes/sec + lots of fsyncs)?

My naive thinking is that AFAIK the slow bit is writing to the flash chips to make things actually durable when you ask, and it's basically the same underlying flash chips, so I'd expect NVMe to not be much faster than SATA SSDs on this narrow workload.

This is probably at least somewhat wrong. This 2025 SSD hierarchy article doesn't explicitly cover forced writes to flash (the fsync() case), but it does cover writing 50 GBytes of data in 30,000 files, which is probably enough to run any reasonable consumer NVMe SSD out of fast write buffer storage (either RAM or fast flash). The write speeds they get on this test from good NVMe drives are well over the maximum SATA data rates, so there's clearly a sustained write advantage to NVMe SSDs over SATA SSDs.

In replies on the Fediverse, several people pointed out that NVMe SSDs are likely using newer controllers than SATA SSDs and these newer controllers may well be better at handling writes. This isn't surprising when I thought about it, especially in light of NVMe perhaps overtaking SATA for SSDs, although apparently 'enterprise' SATA/SAS SSDs are still out there and probably seeing improvements (unlike consumer SATA SSDs where price is the name of the game).

Also, apparently the real bottleneck in writing to the actual flash is finding erased blocks or, if you're unlucky, having to wait for blocks to be erased. Actual writes to the flash chips may be able to go at something close to the PCIe 3.0 (or better) bandwidth, which would help explain the Tom's Hardware large write figures (cf).

(If this is the case, then explicitly telling SSDs about discarded blocks is especially important for any write workload that will be limited by flash write speeds, including fsync() heavy workloads.)

PS: The reason I'm interested in this is that we have a SATA SSD based system that seems to have periodic performance issues related enough write IO combined with fsync()s (possibly due to write buffering interactions), and I've been wondering how much moving it to be NVMe based might help. Since this machine uses ZFS, perhaps one thing we should consider is manually doing some ZFS 'TRIM' operations.

The strange case of 'mouse action traps' in GNU Emacs with (slower) remote X

By: cks

Some time back over on the Fediverse, I groused about GNU Emacs tooltips. That grouse was a little imprecise; the situation I usually see problems with is specifically running GNU Emacs in SSH-forwarded X from home, which has a somewhat high latency. This high latency caused me to change how I opened URLs from GNU Emacs, and it seems to be the root of the issues I'm seeing.

The direct experience I was having with tooltips was that being in a situation where Emacs might want to show a GUI tooltip would cause Emacs to stop responding to my keystrokes for a while. If the tooltip was posted and visible it would stay visible, but the stall could happen without that. However, it doesn't seem to be tooltips as such that cause this problem, because even with tooltips disabled as far as I can tell (and certainly not appearing), the cursor and my interaction with Emacs can get 'stuck' in places where there's mouse actions available.

(I tried both setting the tooltip delay times to very large numbers and setting tooltip-functions to do nothing.)

This is especially visible to me because my use of MH-E is prone to this in two cases. First, when composing email flyspell mode will attach a 'correct word' button-2 popup menu to misspelled words, which can then stall things if I move the cursor to them (especially if I use a mouse click to do so, perhaps because I want to make the word into an X selection). Second, when displaying email that has links in it, these links can be clicked on (and have hover tooltips to display what the destination URL is); what I frequently experience is that after I click on a link, when I come back to the GNU Emacs (X) window I can't immediately switch to the next message, scroll the text of the current message, or otherwise do things.

This 'trapping' and stall doesn't usually happen when I'm in the office, which is still using remote X but over a much faster and lower latency 1G network connection. Disabling tooltips themselves isn't ideal because it means I no longer get to see where links go, and anyway it's relatively pointless if it doesn't fix the real problem.

When I thought this was an issue specific to tooltips, it made sense to me because I could imagine that GNU Emacs needed to do a bunch of relatively synchronous X operations to show or clear a tooltip, and those operations could take a while over my home link. Certainly displaying regular GNU Emacs (X) menus isn't particularly fast. Without tooltips displaying it's more mysterious, but it's still possible that Emacs is doing a bunch of X operations when it thinks a mouse or tooltip target is 'active', or perhaps there's something else going on.

(I'm generally happy with GNU Emacs but that doesn't mean it's perfect or that I don't have periodic learning experiences.)

PS: In theory there are tools that can monitor and report on the flow of X events (by interposing themselves into it). In practice it's been a long time since I used any of them, and anyway there's probably nothing I can do about it if GNU Emacs is doing a lot of X operations. Plus it's probably partly the GTK toolkit at work, not GNU Emacs itself.

PPS: Having taken a brief look at the MH-E code, I'm pretty sure that it doesn't even begin to work with GNU Emacs' TRAMP (also) system for working with remote files. TRAMP has some support for running commands remotely, but MH-E has its own low-level command execution and assumes that it can run commands rapidly, whenever it feels like, and then read various results out of the filesystem. Probably the most viable approach would be to use sshfs to mount your entire ~/Mail locally, have a local install of (N)MH, and then put shims in for the very few MH commands that have to run remotely (such as inc and the low level post command that actually sends out messages you've written). I don't know if this would work very well, but it would almost certainly be better than trying to run all those MH commands remotely.

Staring at code can change what I see (a story from long ago)

By: cks

I recently read Hillel Wayne's Sapir-Whorf does not apply to Programming Languages (via, which I will characterize as being about how programming can change how you see things even though the Sapir-Whorf hypothesis doesn't apply (Hillel Wayne points to the Tetris Effect). As it happens, long ago I experienced a particular form of this that still sticks in my memory.

Many years ago, I was recruited to be a TA for the university's upper year Operating Systems course, despite being an undergraduate at the time. One of the jobs of TAs was to mark assignments, which we did entirely by hand back in those days; any sort of automated testing was far in the future, and for these assignments I don't think we even ran the programs by hand. Instead, marking was mostly done by having students hand in printouts of their modifications to the course's toy operating system and we three TAs collectively scoured the result to see if they'd made the necessary changes and spot errors.

Since this was an OS course, some assignments required dealing with concurrency, which meant that students had to properly guard and insulate their changes (in, for example, memory handling) from various concurrency problems. Failure to completely do so would cost marks, so the TAs were on the lookout for such problems. Over the course of the course, I got very good at spotting these concurrency problems entirely by eye in the printed out code. I didn't really have to think about it, I'd be reading the code (or scanning it) and the problem would jump out at me. In the process I formed a firm view that concurrency is very hard for people to deal with, because so many students made so many mistakes (whether obvious or subtle).

(Since students were modifying the toy OS to add or change features, there was no set form that their changes had to follow; people implemented the new features in various different ways. This meant that their concurrency bugs had common patterns but not specific common forms.)

I could have thought that I was spotting these problems because I was a better programmer than these other undergraduate students (some of whom were literally my peers, it was just that I'd taken the OS course a year earlier than they had because it was one of my interests). However, one of the most interesting parts of the whole experience was getting pretty definitive proof that I wasn't, and it was my focused experience that made the difference. One of the people taking this course was a fellow undergraduate who I knew and I knew was a better programmer than I was, but when I was marking his version of one assignment I spotted what I viewed at the time as a reasonably obvious concurrency issue. So I wasn't seeing these issues when the undergraduates doing the assignment missed them because I was a better programmer, since here I wasn't: I was seeing the bugs because I was more immersed in this than they were.

(This also strongly influenced my view of how hard and tricky concurrency is. Here was a very smart programmer, one with at least some familiarity with the whole area, and they'd still made a mistake.)

Uses for DNS server delegation

By: cks

A commentator on my entry on systemd-resolved's new DNS server delegation feature asked:

My memory might fail me here, but: wasn't something like this a feature introduced in ISC's BIND 8, and then considered to be a bad mistake and dropped again in BIND 9 ?

I don't know about Bind, but what I do know is that this feature is present in other DNS resolvers (such as Unbound) and that it has a variety of uses. Some of those uses can be substituted with other features and some can't be, at least not as-is.

The quick version of 'DNS server delegation' is that you can send all queries under some DNS zone name off to some DNS server (or servers) of your choice, rather than have DNS resolution follow any standard NS delegation chain that may or may not exist in global DNS. In Unbound, this is done through, for example, Forward Zones.

DNS server delegation has at least three uses that I know of. First, you can use it to insert entire internal TLD zones into the view that clients have. People use various top level names for these zones, such as .internal, .kvm, .sandbox (our choice), and so on. In all cases you have some authoritative servers for these zones and you need to direct queries to these servers instead of having your queries go to the root nameservers and be rejected.

(Obviously you will be sad if IANA ever assigns your internal TLD to something, but honestly if IANA allows, say, '.internal', we'll have good reason to question their sanity. The usual 'standard DNS environment' replacement for this is to move your internal TLD to be under your organizational domain and then implement split horizon DNS.)

Second, you can use it to splice in internal zones that don't exist in external DNS without going to the full overkill of split horizon authoritative data. If all of your machines live in 'corp.example.org' and you don't expose this to the outside world, you can have your public example.org servers with your public data and your corp.example.org authoritative servers, and you splice in what is effectively a fake set of NS records through DNS server delegation. Related to this, if you want you can override public DNS simply by having an internal and an external DNS server, without split horizon DNS; you use DNS server delegation to point to the internal DNS server for certain zones.

(This can be replaced with split horizon DNS, although maintaining split horizon DNS is its own set of headaches.)

Finally, you can use this to short-cut global DNS resolution for reliability in cases where you might lose external connectivity. For example, there are within-university ('on-campus' in our jargon) authoritative DNS servers for .utoronto.ca and .toronto.edu. We can use DNS server delegation to point these zones at these servers to be sure we can resolve university names even if the university's external Internet connection goes down. We can similarly point our own sub-zone at our authoritative servers, so even if our link to the university backbone goes down we can resolve our own names.

(This isn't how we actually implement this; we have a more complex split horizon DNS setup that causes our resolving DNS servers to have a complete copy of the inside view of our zones, acting as caching secondaries.)

The early Unix history of chown() being restricted to root

By: cks

A few years ago I wrote about the divide in chown() about who got to give away files, where BSD and V7 were on one side, restricting it to root, while System III and System V were on the other, allowing the owner to give them away too. At the time I quoted the V7 chown(2) explanation of this:

[...] Only the super-user may execute this call, because if users were able to give files away, they could defeat the (nonexistent) file-space accounting procedures.

Recently, for reasons, chown(2) and its history was on my mind and so I wondered if the early Research Unixes had always had this, or if a restriction was added at some point.

The answer is that the restriction was added in V6, where the V6 chown(2) manual page has the same wording as V7. In Research Unix V5 and earlier, people can chown(2) away their own files; this is documented in the V4 chown(2) manual page and is what the V5 kernel code for chown() does. This behavior runs all the way back to the V1 chown() manual page, with an extra restriction that you can't chown() setuid files.

(Since I looked it up, the restriction on chown()'ing setuid files was lifted in V4. In V4 and later, a setuid file has its setuid bit removed on chown; in V3 you still can't give away such a file, according to the V3 chown(2) manual page.)

At this point you might wonder where the System III and System V unrestricted chown came from. The surprising to me answer seems to be that System III partly descends from PWB/UNIX, and PWB/UNIX 1.0, although it was theoretically based on V6, has pre-V6 chown(2) behavior (kernel source, manual page). I suspect that there's a story both to why V6 made chown() more restricted and also why PWB/UNIX specifically didn't take that change from V6, but I don't know if it's been documented anywhere (a casual Internet search didn't turn up anything).

(The System III chown(2) manual page says more or less the same thing as the PWB/UNIX manual page, just more formally, and the kernel code is very similar.)

Maybe why OverlayFS had its readdir() inode number issue

By: cks

A while back I wrote about readdir()'s inode numbers versus OverlayFS, which discussed an issue where for efficiency reasons, OverlayFS sometimes returned different inode numbers in readdir() than in stat(). This is not POSIX legal unless you do some pretty perverse interpretations (as covered in my entry), but lots of filesystems deviate from POSIX semantics every so often. A more interesting question is why, and I suspect the answer is related to another issue that's come up, the problem of NFS exports of NFS mounts.

What's common in both cases is that NFS servers and OverlayFS both must create an 'identity' for a file (a NFS filehandle and an inode number, respectively). In the case of NFS servers, this identity has some strict requirements; OverlayFS has a somewhat easier life, but in general it still has to create and track some amount of information. Based on reading the OverlayFS article, I believe that OverlayFS considers this expensive enough to only want to do it when it has to.

OverlayFS definitely needs to go to this effort when people call stat(), because various programs will directly use the inode number (the POSIX 'file serial number') to tell files on the same filesystem apart. POSIX technically requires OverlayFS to do this for readdir(), but in practice almost everyone that uses readdir() isn't going to look at the inode number; they look at the file name and perhaps the d_type field to spot directories without needing to stat() everything.

If there was a special 'not a valid inode number' signal value, OverlayFS might use that, but there isn't one (in either POSIX or Linux, which is actually a problem). Since OverlayFS needs to provide some sort of arguably valid inode number, and since it's reading directories from the underlying filesystems, passing through their inode numbers from their d_ino fields is the simple answer.

(This entry was inspired by Kevin Lyda's comment on my earlier entry.)

Sidebar: Why there should be a 'not a valid inode number' signal value

Because both standards and common Unix usage include a d_ino field in the structure readdir() returns, they embed the idea that the stat()-visible inode number can easily be recovered or generated by filesystems purely by reading directories, without needing to perform additional IO. This is true in traditional Unix filesystems, but it's not obvious that you would do that all of the time in all filesystems. The on disk format of directories might only have some sort of object identifier for each name that's not easily mapped to a relatively small 'inode number' (which is required to be some C integer type), and instead the 'inode number' is an attribute you get by reading file metadata based on that object identifier (which you'll do for stat() but would like to avoid for reading directories).

But in practice if you want to design a Unix filesystem that performs decently well and doesn't just make up inode numbers in readdir(), you must store a potentially duplicate copy of your 'inode numbers' in directory entries.

Keeping notes is for myself too, illustrated (once again)

By: cks

Yesterday I wrote about restarting or redoing something after a systemd service restarts. The non-hypothetical situation that caused me to look into this was that after we applied a package update to one system, systemd-networkd on it restarted and wiped out some critical policy based routing rules. Since I vaguely remembered this happening before, I sighed and arranged to have our rules automatically reapplied on both systems with policy based routing rules, following the pattern I worked out.

Wait, two systems? And one of them didn't seem to have problems after the systemd-networkd restart? Yesterday I ignored that and forged ahead, but really it should have set off alarm bells. The reason the other system wasn't affected was I'd already solved the problem the right way back in March of 2024, when we first hit this networkd behavior and I wrote an entry about it.

However, I hadn't left myself (or my co-workers) any notes about that March 2024 fix; I'd put it into place on the first machine (then the only machine we had that did policy based routing) and forgotten about it. My only theory is that I wanted to wait and be sure it actually fixed the problem before documenting it as 'the fix', but if so, I made a mistake by not leaving myself any notes that I had a fix in testing. When I recently built the second machine with policy based routing I copied things from the first machine, but I didn't copy the true networkd fix because I'd forgotten about it.

(It turns out to have been really useful that I wrote that March 2024 entry because it's the only documentation I have, and I'd probably have missed the real fix if not for it. I rediscovered it in the process of writing yesterday's entry.)

I know (and knew) that keeping notes is good, and that my memory is fallible. And I still let this slip through the cracks for whatever reason. Hopefully the valuable lesson I've learned from this will stick a bit so I don't stub my toe again.

(One obvious lesson is that I should make a note to myself any time I'm testing something that I'm not sure will actually work. Since it may not work I may want to formally document it in our normal system for this, but a personal note will keep me from completely losing track of it. You can see the persistence of things 'in testing' as another example of the aphorism that there's nothing as permanent as a temporary fix.)

Restarting or redoing something after a systemd service restarts

By: cks

Suppose, not hypothetically, that your system is running some systemd based service or daemon that resets or erase your carefully cultivated state when it restarts. One example is systemd-networkd, although you can turn that off (or parts of it off, at least), but there are likely others. To clean up after this happens, you'd like to automatically restart or redo something after a systemd unit is restarted. Systemd supports this, but I found it slightly unclear how you want to do this and today I poked at it, so it's time for notes.

(This is somewhat different from triggering one unit when another unit becomes active, which I think is still not possible in general.)

First, you need to put whatever you want to do into a script and a .service unit that will run the script. The traditional way to run a script through a .service unit is:

[Unit]
....

[Service]
Type=oneshot
RemainAfterExit=True
ExecStart=/your/script/here

[Install]
WantedBy=multi-user.target

(The 'RemainAfterExit' is load-bearing, also.)

To get this unit to run after another unit is started or restarted, what you need is PartOf=, which causes your unit to be stopped and started when the other unit is, along with 'After=' so that your unit starts after the other unit instead of racing it (which could be counterproductive when what you want to do is fix up something from the other unit). So you add:

[Unit]
...
PartOf=systemd-networkd.service
After=systemd-networkd.service

(This is what works for me in light testing. This assumes that the unit you want to re-run after is normally always running, as systemd-networkd is.)

In testing, you don't need to have your unit specifically enabled by itself, although you may want it to be for clarity and other reasons. Even if your unit isn't specifically enabled, systemd will start it after the other unit because of the PartOf=. If the other unit is started all of the time (as is usually the case for systemd-networkd), this effectively makes your unit enabled, although not in an obvious way (which is why I think you should specifically 'systemctl enable' it, to make it obvious). I think you can have your .service unit enabled and active without having the other unit enabled, or even present.

You can declare yourself PartOf a .target unit, and some stock package systemd units do for various services. And a .target unit can be PartOf a .service; on Fedora, 'sshd-keygen.target' is PartOf sshd.service in a surprisingly clever little arrangement to generate only the necessary keys through a templated 'sshd-keygen@.service' unit.

I admit that the whole collection of Wants=, Requires=, Requisite=, BindsTo=, PartOf=, Upholds=, and so on are somewhat confusing to me. In the past, I've used the wrong version and suffered the consequences, and I'm not sure I have them entirely right in this entry.

Note that as far as I know, PartOf= has those Requires= consequences, where if the other unit is stopped, yours will be too. In a simple 'run a script after the other unit starts' situation, stopping your unit does nothing and can be ignored.

(If this seems complicated, well, I think it is, and I think one part of the complication is that we're trying to use systemd as an event-based system when it isn't one.)

Systemd-resolved's new 'DNS Server Delegation' feature (as of systemd 258)

By: cks

A while ago I wrote an entry about things that resolved wasn't for as of systemd 251. One of those things was arbitrary mappings of (DNS) names to DNS servers, for example if you always wanted *.internal.example.org to query a special DNS server. Systemd-resolved didn't have a direct feature for this and attempting to attach your DNS names to DNS server mappings to a network interface could go wrong in various ways. Well, time marches on and as of systemd v258 this is no longer the state of affairs.

Systemd v258 introduces systemd.dns-delegate files, which allow you to map DNS names to DNS servers independently from network interfaces. The release notes describe this as:

A new DNS "delegate zone" concept has been introduced, which are additional lookup scopes (on top of the existing per-interface and the one global scope so far supported in resolved), which carry one or more DNS server addresses and a DNS search/routing domain. It allows routing requests to specific domains to specific servers. Delegate zones can be configured via drop-ins below /etc/systemd/dns-delegate.d/*.dns-delegate.

Since systemd v258 is very new I don't have any machines where I can actually try this out, but based on the systemd.dns-delegate documentation, you can use this both for domains that you merely want diverted to some DNS server and also domains that you also want on your search path. Per resolved.conf's Domains= documentation, the latter is 'Domains=example.org' (example.org will be one of the domains that resolved tries to find single-label hostnames in, a search domain), and the former is 'Domains=~example.org' (where we merely send queries for everything under 'example.org' off to whatever DNS= you set, a route-only domain).

(While resolved.conf's Domains= officially promises to check your search domains in the order you listed them, I believe this is strictly for a single 'Domains=' setting for a single interface. If you have multiple 'Domains=' settings, for example in a global resolved.conf, a network interface, and now in a delegation, I think systemd-resolved makes no promises.)

Right now, these DNS server delegations can only be set through static files, not manipulated through resolvectl. I believe fiddling with them through resolvectl is on the roadmap, but for now I guess we get to restart resolved if we need to change things. In fact resolvectl doesn't expose anything to do with them, although I believe read-only information is available via D-Bus and maybe varlink.

Given the timing of systemd v258's release relative to Fedora releases, I probably won't be able to use this feature until Fedora 44 in the spring (Fedora 42 is current and Fedora 43 is imminent, which won't have systemd v258 given that v258 was released only a couple of weeks ago). My current systemd-resolved setup is okay (if it wasn't I'd be doing something else), but I can probably find uses for these delegations to improve it.

Why I have a GPS bike computer

By: cks

(This is a story about technology. Sort of.)

Many bicyclists with a GPS bike computer probably have it primarily to record their bike rides and then upload them to places like Strava. I'm a bit unusual in that while I do record my rides and make some of them public, and I've come to value this, it's not my primary reason to have a GPS bike computer. Instead, my primary reason is following pre-made routes.

When I started with my recreational bike club, it was well before the era of GPS bike computers. How you followed (or lead) our routes back then was through printed cue sheets, which had all of the turns and so on listed in order, often with additional notes. One of the duties of the leader of the ride was printing out a sufficient number of cue sheets in advance and distributing them to interested parties before the start of the ride. If you were seriously into using cue sheets, you'd use a cue sheet holder (nowadays you can only find these as 'map holders', which is basically the same job); otherwise you might clip the cue sheet to a handlebar brake or gear cable or fold it up and stick it in a back jersey pocket.

Printed cue sheets have a number of nice features, such as giving you a lot of information at a glance. One of them is that a well done cue sheet was and is a lot more than just a list of all of the turns and other things worthy of note; it's an organized, well formatted list of these. The cues would be broken up into sensibly chosen sections, with whitespace between them to make it easier to narrow in on the current one, and you'd lay out the page (or pages) so that the cue or section breaks happened at convenient spots to flip the cue sheet around in cue holders or clips. You'd emphasize important turns, cautions, or other things in various ways. And so on. Some cue sheets even had a map of the route printed on the back.

(You needed to periodically flip the cue sheet around and refold it because many routes had too many turns and other cues to fit in a small amount of printed space, especially if you wanted to use a decently large font size for easy readability.)

Starting in the early 2010s, more and more TBN people started using GPS bike computers or smartphones (cf). People began converting our cue sheet routes to computerized GPS routes, with TBN eventually getting official GPS routes. Over time, more and more members got smartphones and GPS units and there was more and more interest in GPS routes and less and less interest in cue sheets. In 2015 I saw the writing on the wall for cue sheets and the club more or less deprecated them, so in August 2016 I gave in and got a GPS unit (which drove me to finally get a smartphone, because my GPS unit assumed you had one). Cue sheet first routes lingered on for some years afterward, but they're all gone by now; everything is GPS route first.

You can still get cue sheets for club routes (the club's GPS routes typically have turn cues and you can export these into something you can print). But what we don't really have any more is the old school kind of well done, organized cue sheets, and it's basically been a decade since ride leaders would turn up with any printed cue sheets at all. These days it's on you to print your own cue sheet if you need it, and also on you to make a good cue sheet from the basic cue sheet (if you care enough to do so). There are some people who still use cue sheets, but they're a decreasing minority and they probably already had the cue sheet holders and so on (which are now increasingly hard to find). A new rider who wanted to use cue sheets would have an uphill struggle and they might never understand why long time members could be so fond of them.

Cue sheets are still a viable option for route following (and they haven't fundamentally changed). They're just not very well supported any more in TBN because they stopped being popular. If you insist on sticking with them, you still can, but it's not going to be a great experience. I didn't move to a GPS unit because I couldn't possibly use cue sheets any more (I still have my cue sheet holder); I moved because I could see the writing on the wall about which one would be the more convenient, more usable option.

Applications to the (computing) technologies of your choice are left as an exercise for the reader.

PS: As a whole I think GPS bike computers are mostly superior to cue sheets for route following, but that's a different discussion (and it depends on what sort of bicycling you're doing). There are points on both sides.

A Firefox issue and perhaps how handling scaling is hard

By: cks

Over on the Fediverse I shared a fun Firefox issue I've just run into:

Today's fun Firefox bug: if I move my (Nightly) Firefox window left and right across my X display, the text inside the window reflows to change its line wrapping back and forth. I have a HiDPI display with non-integer scaling and some other settings, so I'm assuming that Firefox is now suffering from rounding issues where the exact horizontal pixel position changes its idea of the CSS window width, triggering text reflows as it jumps back and forth by a CSS pixel.

(I've managed to reproduce this in a standard Nightly, although so far only with some of my settings.)

Close inspection says that this isn't quite what's happening, and the underlying problem is happening more often than I thought. What is actually happening is that as I move my Firefox window left and right, a thin vertical black line usually appears and disappears at the right edge of the window (past a scrollbar if there is one). Since I can see it on my HiDPI display, I suspect that this vertical line is at least two screen pixels wide. Under the right circumstances of window width, text size, and specific text content, this vertical black bar takes enough width away from the rest of the window to cause Firefox to re-flow and re-wrap text, creating easily visible changes as the window moves.

A variation of this happens when the vertical black bar isn't drawn but things on the right side of the toolbar and the URL bar area will shift left and right slightly as the window is moved horizontally. If the window is showing a scrollbar, the position of the scroll target in the scrollbar will move left and right, with the right side getting ever so slightly wider or returning back to being symmetrical. It's easiest to see this if I move the window sideways slowly, which is of course not something I do often (usually I move windows rapidly).

(This may be related to how X has a notion of sizing windows in non-pixel units if the window asks for it. Firefox in my configuration definitely asks for this; it asserts that it wants to be resized in units of 2 (display) pixels both horizontally and vertically. However, I can look at the state of a Firefox window in X and see that the window size in pixels doesn't change between the black bar appearing and disappearing.)

All of this is visible partly because under X and my window manager, windows can redisplay themselves even during an active move operation. If the window contents froze while I dragged windows around, I probably wouldn't have noticed this for some time. Text reflowing as I moved a Firefox window sideways created a quite attention-getting shimmer.

It's probably relevant that I need unusual HiDPI settings and I've also set Firefox's layout.css.devPixelsPerPx to 1.7 in about:config. That was part of why I initially assumed this was a scaling and rounding issue, and why I still suspect that area of Firefox a bit.

(I haven't filed this as a Firefox bug yet, partly because I just narrowed down what was happening in the process of writing this entry.)

What (I think) you need to do basic UDP NAT traversal

By: cks

Yesterday I wished for a way to do native "blind" WireGuard relaying, without needing to layer something on top of WireGuard. I wished for this both because it's the simplest approach for getting through NATs and the one you need in general under some circumstances. The classic and excellent work on all of the complexities of NAT traversal is Tailscale's How NAT traversal works, which also winds up covering the situation where you absolutely have to have a relay. But, as I understand things, in a fair number of situations you can sort of do without a relay and have direct UDP NAT traversal, although you need to do some extra work to get it and you need additional pieces.

Following RFC 4787, we can divide NAT into to two categories, endpoint-independent mapping (EIM) and endpoint-dependent mapping (EDM). In EIM, the public IP and port of your outgoing NAT'd traffic depend only on your internal IP and port, not on the destination (IP or port); in EDM they (also) depend on the destination. NAT'ing firewalls normally NAT based on what could be called "flows". For TCP, flows are a real thing; you can specifically tell a single TCP connection and it's difficult to fake one. For UDP, a firewall generally has no idea of what is a valid flow, and the best it can do is accept traffic that comes from the destination IP and port, which in theory is replies from the other end.

This leads to the NAT traffic traversal trick that we can do for UDP specifically. If we have two machines that want to talk to each other on each other's UDP port 51820, the first thing they need is to learn the public IP and port being used by the other machine. This requires some sort of central coordination server as well as the ability to send traffic to somewhere on UDP port 51820 (or whatever port you care about). In the case of WireGuard, you might as well make this a server on a public IP running WireGuard and have an actual WireGuard connection to it, and the discount 'coordination server' can then be basically the WireGuard peer information from 'wg' (the 'endpoint' is the public IP and port you need).

Once the two machines know each other's public IP and port, they start sending UDP port 51820 (or whatever) packets to each other, to the public IP and port they learned through the coordination server. When each of them sends their first outgoing packet, this creates a 'flow' on their respective NAT firewall which will allow the other machine's traffic in. Depending on timing, the first few packets from the other machine may arrive before your firewall has set up its state to allow them in and will get dropped, so each side needs to keep sending until it works or until it's clear that at least one side has an EDM (or some other complication).

(For WireGuard, you'd need something that sets the peer's endpoint to your now-known host and port value and then tries to send it some traffic to trigger the outgoing packets.)

As covered in Tailscale's article, it's possible to make direct NAT traversal work in some additional circumstances with increasing degrees of effort. You may be lucky and have a local EDM firewall that can be asked to stop doing EDM for your UDP port (via a number of protocols for this), and otherwise it may be possible to feel your way around one EDM firewall.

If you can arrange a natural way to send traffic from your UDP port to your coordination server, the basic NAT setup can be done without needing the deep cooperation of the software using the port; all you need is a way to switch what remote IP and port it uses for a particular peer. Your coordination server may need special software to listen to traffic and decode which peer is which, or you may be able to exploit existing features of your software (for example, by making the coordination server a WireGuard peer). Otherwise, I think you need either some cooperation from the software involved or gory hacks.

Wishing for a way to do 'blind' (untrusted) WireGuard relaying

By: cks

Over on the Fediverse, I sort of had a question:

I wonder if there's any way in standard WireGuard to have a zero-trust network relay, so that two WG peers that are isolated from each other (eg both behind NAT) can talk directly. The standard pure-WG approach has a public WG endpoint that everyone talks to and which acts as a router for the internal WG IPs of everyone, but this involves decrypting and re-encrypting the WG traffic.

By 'talk directly' I mean that each of the peers has the WireGuard keys of the other and the traffic between the two of them stays encrypted with those keys all the way through its travels. The traditional approach to the problem of two NAT'd machines that want to talk to each other with WireGuard is to have a WireGuard router that both of them talk to over WireGuard, but this means that the router sees the unencrypted traffic between them. This is less than ideal if you don't want to trust your router machine, for example because you want to make it a low-trust virtual machine rented from some cloud provider.

Since we love indirection in computer science, you can in theory solve this with another layer of traffic encapsulation (with a lot of caveats). The idea is that all of the 'public' endpoint IPs of WireGuard peers are actually on a private network, and you route the private network through your public router. Getting the private network packets to and from the router requires another level of encapsulation and unless you get very clever, all your traffic will go through the router even if two WireGuard peers could talk directly. Since WireGuard automatically keeps track of the current public IPs of peers, it would be ideal to do this with WireGuard, but I'm not sure that WG-in-WG can have the routing maintained the way we want.

This untrusted relay situation is of course one of the things that 'automatic mesh network on top of WireGuard' systems give you, but it would be nice to be able to do this with native features (and perhaps without an explicit control plane server that machines talk to, although that seems unlikely). As far as I know such systems implement this with their own brand of encapsulation, which I believe requires running their WireGuard stack.

(On Linux you might be able to do something clever with redirecting outgoing WireGuard packets to a 'tun' device connected to a user level program, which then wrapped them up, sent them off, received packets back, and injected the received packets into the system.)

Using systems because you know them already

By: cks

Every so often on the Fediverse, people ask for advice on a monitoring system to run on their machine (desktop or server), and some of the time Prometheus, and when it does I wind up making awkward noises. On the one hand, we run Prometheus (and Grafana) and are happy with it, and I run separate Prometheus setups on my work and home desktops. On the other hand, I don't feel I can recommend picking Prometheus for a basic single-machine setup, despite running it that way myself.

Why do I run Prometheus on my own machines if I don't recommend that you do so? I run it because I already know Prometheus (and Grafana), and in fact my desktops (re)use much of our production Prometheus setup (but they scrape different things). This is a specific instance (and example) of a general thing in system administration, which is that not infrequently it's simpler for you to use something you already know even if it's not necessarily an exact fit (or even a great fit) for the problem. For example, if you're quite familiar with operating PostgreSQL databases, it might be simpler to use PostgreSQL for a new system where SQLite could do perfectly well and other people would find SQLite much simpler. Especially if you have canned setups, canned automation, and so on all ready to go for PostgreSQL, and not for SQLite.

(Similarly, our generic web server hammer is Apache, even if we're doing things that don't necessarily need Apache and could be done perfectly well or perhaps better with nginx, Caddy, or whatever.)

This has a flipside, where you use a tool because you know it even if there might be a significantly better option, one that would actually be easier overall even accounting for needing to learn the new option and build up the environment around it. What we could call "familiarity-driven design" is a thing, and it can even be a confining thing, one where you shape your problems to conform to the tools you already know.

(And you may not have chosen your tools with deep care and instead drifted into them.)

I don't think there's any magic way to know which side of the line you're on. Perhaps the best we can do is be a little bit skeptical about our reflexive choices, especially if we seem to be sort of forcing them in a situation that feels like it should have a simpler or better option (such as basic monitoring of a single machine).

(In a way it helps that I know so much about Prometheus because it makes me aware of various warts, even if I'm used to them and I've climbed the learning curves.)

Apache .htaccess files are important because they enable delegation

By: cks

Apache's .htaccess files have a generally bad reputation. For example, lots of people will tell you that they can cause performance problems and you should move everything from .htaccess files into your main Apache configuration, using various pieces of Apache syntax to restrict what configuration directives apply to. The result can even be clearer, since various things can be confusing in .htaccess files (eg rewrites and redirects). Despite all of this, .htaccess files are important and valuable because of one property, which is that they enable delegation of parts of your server configuration to other people.

The Apache .htaccess documentation even spells this out in reverse, in When (not) to use .htaccess files:

In general, you should only use .htaccess files when you don't have access to the main server configuration file. [...]

If you operate the server and would be writing the .htaccess file, you can put the contents of the .htaccess in the main server configuration and make your life easier and Apache faster (and you probably should). But if the web server and its configuration isn't managed as a unitary whole by one group, then .htaccess files allow the people managing the overall Apache configuration to safely delegate things to other people on a per-directory basis, using Unix ownership. This can both enable people to do additional things and reduce the amount of work the central people have to do, letting people things scale better.

(The other thing that .htaccess files allow is dynamic updates without having to restart or reload the whole server. In some contexts this can be useful or important, for example if the updates are automatically generated at unpredictable times.)

I don't think it's an accident that .htaccess files emerged in Apache, because one common environment Apache was initially used in was old fashioned multi-user Unix web servers where, for example, every person with a login on the web server might have their own UserDir directory hierarchy. Hence features like suEXEC, so you could let people run CGIs without those CGIs having to run as the web user (a dangerous thing), and also hence the attraction of .htaccess files. If you have a bunch of (graduate) students with their own web areas, you definitely don't want to let all of them edit your departmental web server's overall configuration.

(Apache doesn't solve all your problems here, at least not in a simple configuration; you're still left with the multiuser PHP problem. Our solution to this problem is somewhat brute force.)

These environments are uncommon today but they're not extinct, at least at universities like mine, and .htaccess files (and Apache's general flexibility) remain valuable to us.

Readdir()'s inode numbers versus OverlayFS

By: cks

Recently I re-read Deep Down the Rabbit Hole: Bash, OverlayFS, and a 30-Year-Old Surprise (via) and this time around, I stumbled over a bit in the writeup that made me raise my eyebrows:

Bash’s fallback getcwd() assumes that the inode [number] from stat() matches one returned by readdir(). OverlayFS breaks that assumption.

I wouldn't call this an 'assumption' so much as 'sane POSIX semantics', although I'm not sure that POSIX absolutely requires this.

As we've seen before, POSIX talks about 'file serial number(s)' instead of inode numbers. The best definition of these is covered in sys/stat.h, where we see that a 'file identity' is uniquely determined by the combination the inode number and the device ID (st_dev), and POSIX says that 'at any given time in a system, distinct files shall have distinct file identities' while hardlinks have the same identity. The POSIX description of readdir() and dirent.h don't caveat the d_ino file serial numbers from readdir(), so they're implicitly covered by the general rules for file serial numbers.

In theory you can claim that the POSIX guarantees don't apply here since readdir() is only supplying d_ino, the file serial number, not the device ID as well. I maintain that this fails due to a POSIX requirement:

[...] The value of the structure's d_ino member shall be set to the file serial number of the file named by the d_name member. [...]

If readdir() gives one file serial number and a fstatat() of the same name gives another, a plain reading of POSIX is that one of them is lying. Files don't have two file serial numbers, they have one. Readdir() can return duplicate d_ino numbers for files that aren't hardlinks to each other (and I think legitimately may do so in some unusual circumstances), but it can't return something different than what fstatat() does for the same name.

The perverse argument here turns on POSIX's 'at any given time'. You can argue that the readdir() is at one time and the stat() is at another time and the system is allowed to entirely change file serial numbers between the two times. This is certainly not the intent of POSIX's language but I'm not sure there's anything in the standard that rules it out, even though it makes file serial numbers fairly useless since there's no POSIX way to get a bunch of them at 'a given time' so they have to be coherent.

So to summarize, OverlayFS has chosen what are effectively non-POSIX semantics for its readdir() inode numbers (under some circumstances, in the interests of performance) and Bash used readdir()'s d_ino in a traditional Unix way that caused it to notice. Unix filesystems can depart from POSIX semantics if they want, but I'd prefer if they were a bit more shamefaced about it. People (ie, programs) count on those semantics.

(The truly traditional getcwd() way wouldn't have been a problem, because it predates readdir() having d_ino and so doesn't use it (it stat()s everything to get inode numbers). I reflexively follow this pre-d_ino algorithm when I'm talking about doing getcwd() by hand (cf), but these days you want to use the dirent d_ino and if possible d_type, because they're much more efficient than stat()'ing everything.)

How part of my email handling drifted into convoluted complexity

By: cks

Once upon a time, my email handling was relatively simple. I wasn't on any big mailing lists, so I had almost everything delivered straight to my inbox (both in the traditional /var/mail mbox sense and then through to MH's own inbox folder directory). I did some mail filtering with procmail, but it was all for things that I basically never looked at, so I had procmail write them to mbox files under $HOME/.mail. I moved email from my Unix /var/mail inbox to MH's inbox with MH's inc command (either running it directly or having exmh run it for me). Rarely, I had a mbox file procmail had written that I wanted to read, and at that point I inc'd it either to my MH +inbox or to some other folder.

Later, prompted by wanting to improve my breaks and vacations, I diverted a bunch of mailing lists away from my inbox. Originally I had procmail write these diverted messages to mbox files, then later I'd inc the files to read the messages. Then I found that outside of vacations, I needed to make this email more readily accessible, so I had procmail put them in MH folder directories under Mail/inbox (one of MH's nice features is that your inbox is a regular folder and can have sub-folders, just like everything else). As I noted at the time, procmail only partially emulates MH when doing this, and one of the things it doesn't do is keep track of new, unread ('unseen') messages.

(MH has a general purpose system for keeping track of 'sequences' of messages in a MH folder, so it tracks unread messages based on what is in the special 'unseen' sequence. Inc and other MH commands update this sequence; procmail doesn't.)

Along with this procmail setup I wrote a basic script, called mlists, to report how many messages each of these 'mailing list' inboxes had in them. After a while I started diverting lower priority status emails and so on through this system (and stopped reading the mailing lists); if I got a type of email in any volume that I didn't want to read right away during work, it probably got shunted to these side inboxes. At some point I made mlists optionally run the MH scan command to show me what was in each inbox folder (well, for the inbox folders where this was potentially useful information). The mlists script was still mostly simple and the whole system still made sense, but it was a bit more complex than before, especially when it also got a feature where it auto-reset the current message number in each folder to the first message.

A couple of years ago, I switched the MH frontend I used from exmh to MH-E in GNU Emacs, which changed how I read my email in practice. One of the changes was that I started using the GNU Emacs Speedbar, which always displays a count of messages in MH folders and especially wants to let you know about folders with unread messages. Since I had the hammer of my mlists script handy, I proceeded to mutate it to be what a comment in the script describes as "a discount maintainer of 'unseen'", so that MH-E's speedbar could draw my attention to inbox folders that had new messages.

This is not the right way to do this. The right way to do this is to have procmail deliver messages through MH's rcvstore, which as a MH command can update the 'unseen' sequence properly. But using rcvstore is annoying, partly because you have to use another program to add the locking it needs, so at every point the path of least resistance was to add a bit more hacks to what I already had. I had procmail, and procmail could deliver to MH folder directories, so I used it (and at the time the limitations were something I considered a feature). I had a script to give me basic information, so it could give me more information, and then it could do one useful thing while it was giving me information, and then the one useful thing grew into updating 'unseen'.

And since I have all of this, it's not even worth the effort of switching to the proper rcvstore approach and throwing a bunch of it away. I'm always going to want the 'tell me stuff' functionality of my mlists script, so part of it has to stay anyway.

Can I see similarities between this and how various of our system tools have evolved, mutated, and become increasingly complex? Of course. I think it's much the same obvious forces involved, because each step seems reasonable in isolation, right up until I've built a discount environment that duplicates much of rcvstore.

Sidebar: an extra bonus bit of complexity

It turns out that part of the time, I want to get some degree of live notification of messages being filed into these inbox folders. I may not look at all or even many of them, but there are some periodic things that I do want to pay attention to. So my discount special hack is basically:

tail -f .mail/procmail-log |
  egrep -B2 --no-group-separator 'Folder: /u/cks/Mail/inbox/'

(This is a script, of course, and I run it in a terminal window.)

This could be improved in various ways but then I'd be sliding down the convoluted complexity slope and I'm not willing to do that. Yet. Give it a few years and I may be back to write an update.

More on the tools I use to read email affecting my email reading

By: cks

About two years ago I wrote an entry about how my switch from reading email with exmh to reading it in GNU Emacs with MH-E had affected my email reading behavior more than I expected. As time has passed and I've made more extensive customizations to my MH-E environment, this has continued. One of the recent ways I've noticed is that I'm slowly making more and more use of the fact that GNU Emacs is a multi-window editor ('multi-frame' in Emacs terminology) and reading email with MH-E inside it still leaves me with all of the basic Emacs facilities. Specifically, I can create several Emacs windows (frames) and use this to be working in multiple MH folders at the same time.

Back when I used exmh extensively, I mostly had MH pull my email into the default 'inbox' folder, where I dealt with it all at once. Sometimes I'd wind up pulling some new email into a separate folder, but exmh only really giving me a view of a single folder at a time combined with a system administrator's need to be regularly responding to email made that a bit awkward. At first my use of MH-E mostly followed that; I had a single Emacs MH-E window (frame) and within that window I switched between folders. But lately I've been creating more new windows when I want to spend time reading a non-inbox folder, and in turn this has made me much more willing to put new email directly into different (MH) folders rather than funnel it all into my inbox.

(I don't always make a new window to visit another folder, because I don't spend long on many of my non-inbox folders for new email. But for various mailing lists and so on, reading through them may take at least a bit of time so it's more likely I'll decide I want to keep my MH inbox folder still available.)

One thing that makes this work is that MH-E itself has reasonably good support for displaying and working on multiple folders at once. There are probably ways to get MH-E to screw this up and run MH commands with the wrong MH folder as the current folder, so I'm careful that I don't try to have MH-E carry out its pending MH operations in two MH-E folders at the same time. There are areas where MH-E is less than ideal when I'm also using command-line MH tools, because MH-E changes MH's global notion of the current folder any time I have it do things like show a message in some folder. But at least MH-E is fine (in normal circumstances) if I use MH commands to change the current folder; MH-E will just switch it back the next time I have it show another message.

PS: On a purely pragmatic basis, another change in my email handling is that I'm no longer as irritated with HTML emails because GNU Emacs is much better at displaying HTML than exmh was. I've actually left my MH-E setup showing HTML by default, instead of forcing multipart/alternative email to always show the text version (my exmh setup). GNU Emacs and MH-E aren't up to the level of, say, Thunderbird, and sometimes this results in confusing emails, but it's better than it was.

(The situation that seems tricky for MH-E is that people sometimes include inlined images, for example screenshots as part of problem reports, and MH-E doesn't always give any indication that it's even omitting something.)

Recently

Some meta-commentary on reading: I’ve been trying to read through my enormous queue of articles saved on Instapaper. In general I love reading stuff from the internet but refuse to do it with a computer or a phone: I don’t want all my stuff to glow!

So, this led me to an abortive trial of the Boox Go 7, an eReader that runs a full Android operating system. Rendering Android interfaces with e-ink was pretty bad, and even though it was possible to run the Instapaper Android app, highlighting text didn’t work and the whole fit & finish was off. I love that most e-ink tablets are really purpose-built computers: this didn’t give me that impression.

So, this month I bought a Kobo Libra Color. Kobo and Instapaper recently announced a partnership and first-class support for Instapaper on the devices.

Overall: it’s better than the Boox experience and definitely better than sending articles to one of my Kindles. My notes so far are:

  • I wish it had highlighting. Pretty confident that it’s on the product roadmap, but for now all it can do is sync, archive, and like articles.
  • The Kobo also integrates directly with Overdrive for local libraries! Amazing and unexpected for me, the vast majority of my books are from the Brooklyn Library or the NYPL.
  • The hardware is pretty decent: the color screen is surprisingly useful because a lot of articles have embedded images. The page turn buttons are a little worse than those on my Kindle Oasis because they’re hinged, so they only work well if you press the top of one button and the bottom of the other. I’ve gotten used to it, but wish they worked via a different mechanism.
  • The first run experience was pretty slick.
  • Annoyingly, it goes to fully ‘off’ mode pretty quickly instead of staying in ‘sleep’ mode, and waking it up from being fully off takes about 15 seconds.

Overall: I think my biggest gripe (no highlighting) will get worked out and this’ll be the perfect internet-reading-without-a-computer device.

Anyway, what’s been good this month?

Reading

In Greece, a homeowner traded ther property development rights to a builder and kept equity in place by receiving finished apartments in the resulting building, often one for themselves and one or more additional units for rental income or adult children, rather than a one-time cash payout. Applied to the U.S., that becomes, swap a house for an apartment (or three), in the same location, with special exemptions to the tax for the home sale.

From Millenial American Dream love a good scheme to fix the housing crisis and densify cities. Definitely seems like a net-positive, but like all other schemes that require big changes to tax codes, would take a miracle to actually implement.

In Zitron’s analysis, it’s always bad. It’s bad when they raise too little money because they’ll run out. It’s bad when they raise too much money because it means they need it.

David Crespo’s critique of Ed Zitron is really strong. Honestly Zitron’s writing, though needed in a certain sense, never hits home for me. In fact a lot of AI critique seems overblown. Maybe I’m warming to it a little.

This Martin Fowler article about how ‘if it hurts, do it more often,’ was good. Summarizes some well-tested wisdom.

I had really mixed feelings about ‘The Autistic Half-Century’ by Byrne Hobart. I probably have some personal stake in this - every test I’ve taken puts me on the spectrum and I have some of the classic features, but like Hobart I’ve never been interested in identifying as autistic. But my discomfort with the subject is all knotted-together and hard to summarize.

One facet I can pick out is my feeling that this era is ending, maybe getting replaced with the ADHD half-century. But the internet I grew up on was pseudonomous, text-oriented, and for me, a calm place. The last ten years have been a slow drift toward real names, and then photograph avatars, and now more and more information being delivered by some person talking into the camera, and that feels really bad, man. Heck, not to drill down on ‘classic traits’, but the number of TikTok videos in which, for some reason the person doing the talking is also eating at the same time, close to the phone, the mouth-sounds? Like, for a long time it was possible to convey information without thinking about whether you were good-looking or were having a good hair day, and that era is ending because everything is becoming oral culture.

If you believed that Trump winning would mean that everyone who supported him was right to have done so, because they had picked the winner; that the mega-rich AI industry buying its way into all corners of American society would mean that critics of the technology and of using it to displace human labors were not just defeated but meaningfully wrong in their criticisms; that some celebrity getting richer from a crypto rug-pull that ripped off hundreds of thousands of less-rich people would actually vindicate the celebrity’s choice to participate in it, because of how much richer it made them. Imagine holding this as an authentic understanding of how the world works: that the simple binary outcome of a contest had the power to reach back through time and adjust the ethical and moral weight of the contestants’ choices along the way. Maybe, in that case, you would feel differently about what to the rest of us looks like straight-up shit eating.

Defector (here, Albert Burneko) is doing some really good work. See also their article on Hailey Welch and ‘bag culture’.

Altman’s claim of a “Cambrian explosion” rings hollow because any tool built on the perverse incentives of social media is not truly designed with creativity in mind, but addiction. Sora may spark a new wave of digital expression, but it’s just as likely to entrench the same attention economy that has warped our online lives already.

From Parmy Olsen writing about OpenAI Sora in Bloomberg. I think this is a really good article: technology is rightfully judged based on what it actually does, not what it could do or is meant to do, and if AI continues to be used for things that make the world worse, it will earn itself a bad reputation.

Watching

Watching Night On Earth was such a joy. It was tender, laugh-out-loud funny, beautiful.

Youth had me crying the whole time, but I’d still recommend it. Got me into Sun Kil Moon from just a few minutes of Mark Kozelek being onscreen as the guitarist at the fancy hotel.

Listening

My old friends at BRNDA have a hit album on their hands. Kind of punk, kind of Sonic Youth, sort of indescribable, lots of good rock flute on this one.

More good ambient-adjacent instrumental rock.

This album feels next door to some Hop Along tracks. It’s a tie between highlighting this track or Holding On which also has a ton of hit quality.


Elsewhere: in September I did a huge bike trip and posted photos of it over in /photos. Might do a full writeup eventually: we rode the Great Allegheny Passage and the C&O canal in four days of pretty hard riding. Everyone survived, we had an amazing time, and I’m extremely glad that we were all able to make the time to do it. It made me feel so zen for about a week after getting back - I have to do that more often!

Porteur bag 2

Back in May, I wrote about a custom porteur bag that I sewed for use on my bike. That bag served me well on two trips - a solo ride up to Brewster and back, and my semi-yearly ride on the Empire State Trail, from Poughkeepsie to Brooklyn in two days.

But I had a longer ride in the plans summer, which I just rode two weeks ago: Pittsburgh to DC, 348 miles in 4 days, with two nights of camping. And at the last minute I decided to make the next version of that bag. Specifically, I wanted to correct three shortcomings of v1:

  • The attachment system was too complex. It had redundant ways to attach to the rack, but the main method was via a shock cord that looped through six grosgrain loops, plus another shock cord that I attached to keep it attached to the back of the rack. Plus hardware that’s used for attachment is better if it’s less flexible. Bungee cords and shock cords are ultimately pretty flawed as ways to attach things on bikes. My frame bag uses non-stretchy paracord, and most bikepacking setups are reliant on voile straps which have a minimal amount of stretch.
  • The bag had no liner, and the material is plasticky and odd-looking. ECOPAK EPLX has a waterproof coating on the outside that makes it look like a plastic bag instead of something from synthetic fabric.
  • The bag had way too many panels: each side was a panel, plus the bottom, plus the flap. These made the shape of the bag messy.

Version 2

Finished bag

It turned out pretty well. Here’s the gist:

Materials

As you can see, there’s only one built-in way to secure this bag to the bike: a webbing strap that attaches to the two buckles and tightens below the rack. This is definitely a superior mechanism to v1: instead of attaching just the front of the bag to the rack and having to deal with the back of the bag separately, this pulls and tensions the whole bag, including its contents, to the rack. It rattles a lot less and is a lot simpler to attach.

On the bike

Construction

This bag is made like a tote bag. The essential ingredient of a tote bag is a piece of fabric cut like this:

Tote bag construction

The lining is simply: the same shape, again, sewed in the same way, sewn to the inside of the bag but with the seams facing in the opposite direction, so that the seams of the liner and the outer shell of the bag face each other.

Techniques for building tote bags are everywhere on the internet, so it’s a really nice place to start. Plus, the bag body can be made with just one cut of fabric. In this case the bottom of the bag is Cordura and the top is ECOPAK, so I just tweaked the tote bag construction by adding panels of ECOPAK on the left and right of the first step above.

The risky part of this bag was its height and the zipper: ideally it could both zip and the top of the bag could fold over for water resistance. I didn’t accomplish both goals and learned something pretty important: if you’re building a waterproof bag with a zipper, once it’s zipped it’ll be hard to compress because the zipper keeps in air.

But the end of the tour included a surprise thunderstorm in Washington, DC and the zipper kept the wet out, so I count that as a win! The zipper also makes the bag very functional off the bike - using the same strap as I use to attach it to the bike but using that as a shoulder strap makes it pretty convenient to carry around. This really came in handy when we were moving bikes onto and off of Amtrak trains.

Plans for version three

Mostly kidding - I’m going to actually use use this bag for the next few trips instead of tinkering with it. But I do have thoughts, from this experience:

  • I have mixed feelings about using Cordura next time. It’s what everyone does, and it helps with the abrasion that the bag experiences from the rack. But I have a feeling that ECOPAK would hold up for many thousands of miles by itself, and is lighter and more waterproof than cordura. It could be cool to make the whole bag body from the same material.
  • I think I’ll make the next bag taller, and skip the zipper. I have more faith in folding over the material rather than relying on a waterproof zipper.
  • A good system for tightening the straps still eludes me. This setup included sliplock slide adjusters, but it’s still kind of annoying to get the right angles to pull the bag tight.
  • Again, I didn’t add any external pockets. I don’t think they’re absolutely necessary, but as it is, like the previous version, it’s not really possible to access the contents of this bag on the move. Which is fine because I have other bags that are easier to access - on this trip this bag carried my clothes, camp kit, and backup spare tires, so nothing essential in the middle of the day.

Cooking with glasses

I’ve been thinking the new Meta Ray-Ban augmented reality glasses. Not because they failed onstage, which they absolutely did. Or that shortly after they received rave reviews from Victoria Song at The Verge and MKBHD, two of the most influential tech reviewers. My impression is that the hardware has improved but the software is pretty bad.

Mostly I keep thinking about the cooking demo. Yeah, it bombed. But what if it worked? What if Meta releases the third iteration of this hardware next year and it worked? This post is just questions.

The demos were deeply weird: both Mark Zuckerberg and Jack Mancuso (the celebrity chef) had their AR glasses in a particular demo mode that broadcasted the audio they were hearing and the video they were seeing to the audience and to the live feed of the event.

Of course they need to square the circle of AR glasses being ‘invisible technology,’ but showing people what it does. According to MKBHD’s review, one of the major breakthroughs of the new edition is that you can’t see when other people are looking at the glasses built-in display.

I should credit Song for mentioning the privacy issues of the glasses in her review for The Verge. MKBHD briefly talks about his concern that people will use the glasses to be even more distracted during face-to-face interactions. It’s not like everyone’s totally ignoring the implications.

But the implications. Like for example: I don’t know, sometimes I’m cooking for my partner. Do they hear a one-sided conversation between me and the “Live AI” about what to do first, how to combine the ingredients? Without the staged demo’s video-casting, this would have been the demo: a celebrity chef talking to himself like a lunatic, asking how to start.

Or what if we’re cooking together – do we both have the glasses, and maybe the Live AI responds to… both of us? Do we hear each other’s audio, or is it a mystery what the AI is telling the sous-chef? Does it broadcast to a bluetooth speaker maybe?

Maybe the glasses are able to do joint attention and know who else is in your personal space and share with them in a permissioned way? Does this lock everyone into a specific brand of the glasses or is there some open protocol? It’s been a long, long time since I’ve seen a new open protocol.

Marques’s video shows an impressively invisible display to others, but surely there’s some reflection on people’s retinas: could you scan that and read people’s screens off of their eyeballs?


I hate to preempt critiques but the obvious counterpoint is that “the glasses will be good for people with accessibility needs.” Maybe the AI is unnecessary for cooking, and a lot of its applications will be creepy or fascist, but being able to translate or caption text in real-time will be really useful to people with hearing issues.

I think that’s true! In the same way as crypto was useful for human rights and commerce in countries with unreliable exchanges and banks - a point most forcefully spoken by Alex Gladstein. I’m not one to do the ethical arithmetic on the benefits of crypto in those situations versus the roughly $79 billion worth of scams that it has generated or its impact on climate change. My outsider view is that there are more people and more volume of crypto involved in the speculative & scam sector than in the human-rights sector, but I’m willing to be wrong.

I’ve always been intrigued by, but not sold on, Dynamicland and its New York cousin, folk.computer. But every AR demo I see sells me on that paradigm more. For all of the kookiness of the Dynamicland paradigm, it seems designed for a social, playful world. Nothing about the technology guarantees that outcome, but it definitely suggests it. Likewise, the AR glasses might usher in a new age of isolation and surveillance, but that isn’t guaranteed - that just seems like a likely outcome.

I’m searching for a vision of the AR future, just like I want to read about what really happens with AI proliferation, or what I wished crypto advocates would have written. I want people who believe in this tech to write about how it’ll change their lives and the lives of others, and what really can be done about the risks.

What's up with FUTO?

Some time ago, I noticed some new organization called FUTO popping up here and there. I’m always interested in seeing new organizations that fund open source popping up, and seeing as they claim several notable projects on their roster, I explored their website with interest and gratitude. I was first confused, and then annoyed by what I found. Confused, because their website is littered with bizzare manifestos,1 and ultimately annoyed because they were playing fast and loose with the term “open source”, using it to describe commercial source-available software.

FUTO eventually clarified their stance on “open source”, first through satire and then somewhat more soberly, perpetuating the self-serving myth that “open source” software can privilege one party over anyone else and still be called open source. I mentally categorized them as problematic but hoped that their donations or grants for genuinely open source projects would do more good than the harm done by this nonsense.

By now I’ve learned better. tl;dr: FUTO is not being honest about their “grant program”, they don’t have permission to pass off these logos or project names as endorsements, and they collaborate with and promote mask-off, self-proclaimed fascists.

An early sign that something is off with FUTO is in that “sober” explanation of their “disdain for OSI approved licenses”, where they make a point of criticizing the Open Source Initiative for banning Eric S. Raymond (aka ESR) from their mailing lists, citing right-wing reactionary conspiracy theorist Bryan Lunduke’s blog post on the incident. Raymond is, as you may know, one of the founders of OSI and a bigoted asshole. He was banned from the mailing lists, not because he’s a bigoted asshole, but because he was being a toxic jerk on the mailing list in question. Healthy institutions outgrow their founders. That said, FUTO’s citation and perspective on the ESR incident could be generously explained as a simple mistake, and we should probably match generosity with generosity given their prolific portfolio of open source grants.

I visited FUTO again quite recently as part of my research on Cloudflare’s donations to fascists, and was pleased to discover that this portfolio of grants had grown immensely since my last visit, and included a number of respectable projects that I admire and depend on (and some projects I don’t especially admire, hence arriving there during my research on FOSS projects run by fascists). But something felt fishy about this list – surely I would have heard about it if someone was going around giving big grants to projects like ffmpeg, VLC, musl libc, Tor, Managarm, Blender, NeoVim – these projects have a lot of overlap with my social group and I hadn’t heard a peep about it.

So I asked Rich Felker, the maintainer of musl libc, about the FUTO grant, and he didn’t know anything about it. Rich and I spoke about this for a while and eventually Rich uncovered a transaction in his GitHub sponsors account from FUTO: a one-time donation of $1,000. This payment circumvents musl’s established process for donations from institutional sponsors. The donation page that FUTO used includes this explanation: “This offer is for individuals, and may be available to small organizations on request. Commercial entities wishing to be listed as sponsors should inquire by email.” It’s pretty clear that there are special instructions for institutional donors who wish to receive musl’s endorsement as thanks for their contribution.

The extent of the FUTO “grant program”, at least in the case of musl libc, involved ignoring musl’s established process for institutional sponsors, quietly sending a modest one-time donation to one maintainer, and then plastering the logo of a well-respected open source project on a list of “grant recipients” on their home page. Rich eventually posted on Mastodon to clarify that the use of the musl name and logo here was unauthorized.

I also asked someone I know on the ffmpeg project about the grant that they had received from FUTO and she didn’t know anything about it, either. Here’s what she said:

I’m sure we did not get a grant from them, since we tear each other to pieces over everything, and that would be enough to start a flame war. Unless some dev independently got money from them to do something, but I’m sure that we as a project got nothing. The only grant we’ve received is from the STF last year.

Neovim is another project FUTO lists as a grant recipient, and they also have a separate process for institutional sponsors. I didn’t reach out to anyone to confirm, but FUTO does not appear on the sponsor list so presumably the M.O. is the same. This is also the case for Wireshark, Conduit, and KiCad. GrapheneOS is listed prominently as well, but that doesn’t seem to have worked out very well for them. Presumably ffmpeg received a similar quiet donation from FUTO, rather than something more easily recognizable as a grant.

So, it seems like FUTO is doing some shady stuff and putting a bunch of notable FOSS projects on their home page without good reason to justify their endorsement. Who’s behind all of this?

As far as I can tell, the important figures are Eron Wolf2 and Louis Rossmann.3 Wolf is the founder of FUTO – a bunch of money fell into his lap from founding Yahoo Games before the bottom fell out of Yahoo, and he made some smart investments to grow his wealth, which he presumably used to fund FUTO. Rossmann is a notable figure in the right to repair movement, with a large following on YouTube, who joined FUTO a year later and ultimately moved to Austin to work more closely with them. His established audience and reputation provides a marketable face for FUTO. I had heard of Rossmann prior to learning about FUTO and held him in generally good regard, despite little specific knowledge of his work, simply because we have a common cause in right to repair.

I hadn’t heard of Wolf before looking into FUTO. However, in the course of my research, several people tipped me off to his association with Curtis Yarvin (aka moldbug), and in particular to the use of FUTO’s platform and the credentials of Wolf and Rossman to platform and promote Yarvin. Curtis Yarvin is a full-blown, mask-off, self-proclaimed fascist. A negligible amount of due diligence is required to verify this, but here’s one source from Politico in January 2025:

I’ve interacted with Vance once since the election. I bumped into him at a party. He said, “Yarvin, you reactionary fascist.” I was like, “Thank you, Mr. Vice President, and I’m glad I didn’t stop you from getting elected.”

Ian Ward for Politico

Vice President Vance and numerous other figures in the American right have cited Yarvin as a friend and source of inspiration in shaping policy.4 Among his many political positions, Yarvin has proclaimed that black people are genetically predisposed to a lower IQ than white people, and moreover suggests that black people are inherently suitable for enslavement.5

Yarvin has appeared on FUTO’s social media channels, in particular in an interview published on PeerTube and Odysee, the latter a platform controversial for its role in spreading hate speech and misinformation.6 Yarvin also appeared on stage to “debate” Louis Rossmann in June 2022, in which Yarvin is permitted to speak at length with minimal interruptions or rebuttals to argue for an authoritarian techno-monarchy to replace democracy.

Rossmann caught some flack for this “debate” and gave a milquetoast response in a YouTube comment on this video, explaining that he agreed to this on very short notice as a favor to Eron, who had donated “a million” to Rossmann’s non-profit prior to bringing Rossmann into the fold at FUTO. Rossmann does rebuke Yarvin’s thesis, albeit buried in this YouTube comment rather than when he had the opportunity to do so on-stage during the debate. Don’t argue with fascists, Louis – they aren’t arguing with you, they are pitching their ideas7 to the audience. Smart fascists are experts at misdirection and bad-faith debate tactics and as a consequence Rossmann just becomes a vehicle for fascist propaganda – consult the YouTube comments to see who this video resonates with the most.

In the end, Rossmann seems to regret agreeing to this debate. I don’t think that Eron Wolf regrets it, though – based on his facilitation of this debate and his own interview with Yarvin on the FUTO channel a month later, I can only assume that Wolf considers Yarvin a close associate. No surprise given that Wolf is precisely the kind of insecure silicon valley techbro Yarvin’s rhetoric is designed to appeal to – moderately wealthy but unknown, and according to Yarvin, fit to be a king. Rossmann probably needs to reflect on why he associates with and lends his reputation to an organization that openly and unapologetically platforms its founder’s fascist friends.

In summary, FUTO is not just the product of some eccentric who founded a grant-making institution that funds open source at the cost of making us read his weird manifestos on free markets and oligopoly. It’s a private, for-profit company that associates with and uses their brand to promote fascists. They push an open-washing narrative and they portray themselves as a grant-making institution when, in truth, they’re passing off a handful of small donations as if they were endorsements from dozens of respectable, high-profile open source projects, in an attempt to legitimize themselves, and, indirectly, legitimize people they platform like Curtis Yarvin.

So, if you read this and discover that your project’s name and logo is being proudly displayed on the front page of a fascist-adjacent, washed-up millionaire’s scummy vanity company, and you don’t like that, maybe you should ask them to knock it off? Eron, Louis – you know that a lot of these logos are trademarked, right?


Updated 2025-10-27:

The FUTO website has been updated to clarify the nature of grants versus donations (before, after) and to reduce the appearance of endorsements from the donation recipients – the site is much better after this change.

I spoke to a representative who spoke for the FUTO leadership, and shared positive feedback regarding the changes to the website. I also asked for a follow-up on the matter of platforming fascist activist Curtis Yarvin on their social media channels, and I was provided this official response:

We prefer to spend our time building great software for people to use, funding interesting projects, and making FUTO better every day. We have no interest in engaging in politics as this is a distraction from our work. We’d like to move past this. We don’t have any further comment that that.

I understand the view that distancing oneself from politics can be a productive approach to your work, and the work FUTO does funding great software like Immich is indeed important work that should be conducted relatively free of distractions. However, FUTO is a fundamentally political organization and it does not distance itself from politics. Consider for example the FUTO statement on Open Source, which takes several political positions:

  • Disapproval of the Open Source Initiative and their legitimacy as an authority
  • Disapproval of OSI’s proposed AI standards
  • Disapproval of the “tech oligopoly”
  • Advocacy for an “open source” which is inclusive of restrictions on commercial use
  • Support for Eric S. Raymond’s side in his conflict with OSI
  • Tacit support for Bryan Lunduke

A truer “apolitical” approach would accept the mainstream definition of open source, would not take positions on conflicts with OSI or Eric Raymond, and would be careful not to cite (or platform) controversial figures such as Lunduke.

It is difficult for FUTO’s work to be apolitical at all. Importantly, there are biases in their grants and donations: their selections have a tendency to privilege projects focused on privacy, decentralization, multimedia, communication, and right to repair, all of which suggest the political priorities of FUTO. The choice to fund “source first” software in addition to open source, or not to fund outright closed source software, or not to vet projects based on their community moderation, are also political factors in their process of selecting funding recipients, or at least are seemingly apolitical decisions which ultimately have political consequences.

This brings us to the political nature of the choice to platform Curtis Yarvin. Yarvin is a self-proclaimed fascist who argues openly for fascist politics. Platforming Yarvin on the FUTO channels legitimizes Yarvin’s ideas and his work, and provides curious listeners a funnel that leads to Yarvin’s more radical ideas and into a far-right rabbit-hole. Platforming Yarvin advances the fascist political program.

It should go without saying that it is political to support fascism or fascists. There is an outrageous moral, intellectual, and political contradiction in claiming that it is apolitical to promote a person whose political program is to dismantle democracy and eject people he disagrees with from the political sphere entirely. FUTO should reflect on their values, acknowledge the political nature of their work, and consider the ways in which their work intersects with politics writ large, then make decisions that align their political actions with their political beliefs.


  1. Here’s a quote from another one: “While the vast majority of elite coders have been neutralized by the Oligopoly, the vast majority of those remaining have become preoccupied with cryptocurrency money-making schemes.” ↩︎

  2. Source: What is FUTO? ↩︎

  3. Source: Announcing the Newest Member of FUTO: Louis Rossmann! ↩︎

  4. Source: Politico: The Seven Thinkers and Groups That Have Shaped JD Vance’s Unusual Worldview ↩︎

  5. Source: Inc.: Controversy Rages Over ‘Pro-Slavery’ Tech Speaker Curtis Yarvin ↩︎

  6. Source: The Guardian: Video platform chief says Nazi posts on white superiority do not merit removal ↩︎

  7. You know, authoritarianism, anti-democracy, genocide, etc. ↩︎

Cloudflare bankrolls fascists

US politics has been pretty fascist lately. The state is filling up concentration camps, engaging in mass state violence against people on the basis of racialized traits, deporting them to random countries without any respect for habeas corpus, exerting state pressure on the free press to censor speech critical of the current administration, and Trump is openly floating the idea of an unconstitutional third term.

Fascism is clearly on the rise, and they’re winning more and more power. None of this is far removed from us in the FOSS community – there are a number of fascists working in FOSS, same as the rest of society. I don’t call them fascists baselessly – someone who speaks out in support of and expresses solidarity with fascists, or who uses fascists dog-whistles or promotes fascist ideology and talking points, or boosts fascist conspiracy theories – well, they’re a fascist.

If one consistently speaks in support of a certain political position and against the opponents of that position then it is correct to identify them with this political position. Facts, as it were, don’t care about feelings, namely the feelings that get hurt when someone is called a fascist. Fascists naturally do not want to be identified as such and will reject the label, but we shouldn’t take their word for it. People should be much more afraid of being called out as fascist than they are afraid of calling someone a fascist. If someone doesn’t want to be called a fascist, they shouldn’t act like one.

It’s in this disturbing political context that I saw an odd post from the Cloudflare blog pop up in my circles this week: Supporting the future of the open web: Cloudflare is sponsoring Ladybird and Omarchy. Based on Ladybird’s sponsorship terms we can assume that these projects received on the order of $100,000 USD from Cloudflare. I find this odd for a few reasons, in particular because one thing that I know these two projects have in common is that they are both run by fascists.

Even at face value this is an unusual pair of projects to fund. I’m all for FOSS projects getting funded, of course, and I won’t complain about a project’s funding on the solitary basis that it’s an odd choice. I will point out that these are odd choices, though, especially Omarchy.

Ladybird makes some sense, given that it’s aligned in principle with Cloudflare’s stated objective to “support the open web”, though I remain bearish that new web browser engines are even possible to make.1 Omarchy is a bizarre choice, though – do we really need another pre-customized Arch Linux distribution? And if we do, do we really need a big corporation like Cloudflare to bankroll it? Everyone on /r/unixporn manages to make Arch Linux look pretty for free.

Omarchy is a very weird project to fund, come to think of it. Making an Arch Linux spin technically requires some work, and work is work, I won’t deny it, but most of the work done here is from Arch Linux and Hyprland. Why not fund those, instead? Well, don’t fund Hyprland, since it’s also run by a bunch of fascists, but you get my point.

Anyway, Omarchy and Ladybird are both run by fascists. Omarchy makes this pretty obvious from the outset – on the home page the big YouTube poster image prominently features SuperGrok, which is a pathetically transparent dog-whistle to signal alliance with Elon Musk’s fascist politics. Omarchy is the pet project of David Heinemeier Hansson, aka DHH, who is well known as a rich fascist weirdo.2 One need only consult his blog to browse his weird, racist views on immigration, fat-shaming objections to diverse representation, vaguely anti-feminist/homophobic/rapey rants on consent, and, recently, tone-policing antifascists who celebrate the death of notable fascist Charlie Kirk.

Speaking of tributes to Charlie Kirk, that brings us to Andreas Kling, the project lead for Ladybird, who tweeted on the occasion of his assassination:

RIP Charlie Kirk

I hope many more debate nerds carry on his quest to engage young people with words, not fists.

@awesomekling

Kling has had a few things to say about Kirk on Twitter lately. Here’s another one – give you three guesses as to which “[group]” he objects to punching. You may also recall that Kling achieved some notoriety for his obnoxious response as the maintainer of SerenityOS when someone proposed gender-neutral language for the documentation:

Screenshot of the interaction on GitHub. Kling responds “This project is not an appropriate arena to advertise your personal politics.”

Replacing “he” with “they” in one sentence of the documentation is the kind of “ideologically motivated change” that Serenity’s CONTRIBUTING.md apparently aims to prevent, a classic case of the sexist “identities that are not men are inherently political” nonsense. Ladybird has a similar, weirdly defensive policy on “neutrality”, and a milquetoast code of conduct, which is based on the Ruby Community Conduct Guideline, which has been itself the subject of many controversies due to its inadequacy leading to real-world incidents of harassment and abuse.

Here’s another one – Kling endorsing white replacement theory in June:

White males are actively discriminated against in tech.

It’s an open secret of Silicon Valley.

One of the last meetings I attended before leaving Apple (in 2017) was management asking us to “keep the corporate diversity targets in mind” when interviewing potential new hires.

The phrasing was careful, but the implication was pretty clear.

I knew in my heart this wasn’t wholesome, but I was too scared to rock the boat at the time.

@awesomekling replying to @danheld3

And in a moment of poetic irony, a few days ago Kling spoke in solidarity with Hansson over his “persecution” for “banal, mainstream positions” on Twitter just a few days ago, in response to Hansson’s tweet signal-boosting another notable reactionary tech fascist, Bryan Lunduke.

So, to sum it up, Kling wears his mask a bit better than Hansson, but as far as I’m concerned it seems clear that both projects are run by fascists. If it walks like a fascist and quacks like a fascist… then why is Cloudflare giving them hundreds of thousands of dollars?


  1. I did think it was cute that Kling used a screenshot of Ladybird rendering my blog post back in 2023 to headline one of his articles on Ladybird, though. ↩︎

  2. And a multi-millionaire who can fund his “Arch Linux for Obnoxious Reactionary Assholes” project his own damn self, in my opinion. He also does not want to be called a fascist, in which case he should probably stop being so fucking faschy. ↩︎

  3. Who is ultimately responding to @shaunmmaguire’s tweet lying about being told he wouldn’t be promoted at Google for being white. Shaun also posts about the threat of “Radical Islamist” “Marxists” to whom Europe “will be lost”, falsely accuse the “far left” of committing “almost all mass murders and political violence” in the last few years, and praise Trump for classifying antifascists as terrorists. ↩︎

A better future for JavaScript that won't happen

In the wake of the largest supply-chain attack in history, the JavaScript community could have a moment of reckoning and decide: never again. As the panic and shame subsides, after compromised developers finish re-provisioning their workstations and rotating their keys, the ecosystem might re-orient itself towards solving the fundamental flaws that allowed this to happen.

After all, people have been sounding the alarm for years that this approach to dependency management is reckless and dangerous and broken by design. Maybe this is the moment when the JavaScript ecosystem begins to understand the importance and urgency of this problem, and begins its course correction. It could leave behind its sprawling dependency trees full of micro-libraries, establish software distribution based on relationships of trust, and incorporate the decades of research and innovation established by more serious dependency management systems.

Perhaps Google and Mozilla, leaders in JavaScript standards and implementations, will start developing a real standard library for JavaScript, which makes micro-dependencies like left-pad a thing of the past. This could be combined with a consolidation of efforts, merging micro-libraries into larger packages with a more coherent and holistic scope and purpose, which prune their own dependency trees in turn.

This could be the moment where npm comes to terms with its broken design, and with a well-funded effort (recall that, ultimately, npm is GitHub is Microsoft, market cap $3 trillion USD), will develop and roll out the next generation of package management for JavaScript. It could incorporate the practices developed and proven in Linux distributions, which rarely suffer from these sorts of attacks, by de-coupling development from packaging and distribution, establishing package maintainers who assemble and distribute curated collections of software libraries. By introducing universal signatures for packages of executable code, smaller channels and webs of trust, reproducible builds, and the many other straightforward, obvious techniques used by responsible package managers.

Maybe other languages that depend on this broken dependency management model, like Cargo, PyPI, RubyGems, and many more, are watching this incident and know that the very same crisis looms in their future. Maybe they will change course, too, before the inevitable.

Imagine if other large corporations who depend on and profit from this massive pile of recklessly organized software committed their money and resources to it, through putting their engineers to the task of fixing these problems, through coming together to establish and implement new standards, through direct funding of their dependencies and by distributing money through institutions like NLNet, ushering in an era of responsible, sustainable, and secure software development.

This would be a good future, but it’s not the future that lies in wait for us. The future will be more of the same. Expect symbolic gestures – mandatory 2FA will be rolled out in more places, certainly, and the big players will write off meager donations in the name of “OSS security and resilience” in their marketing budgets.

No one will learn their lesson. This has been happening for decades and no one has learned anything from it yet. This is the defining hubris of this generation of software development.

LibreWolf

I've hit the last straw with Mozilla's antics in Firefox. I've been a Firefox champion for years and years. But every danged upgrade has some new insane junk I don't want or need and they keep popping up these little "tips" to rub my nose in it. I'm sick of it. I'm done. THANKFULLY we have LibreWolf to solve all of that. I'm a heavy browser user migrating from the fox to the wolf and here are my notes so far...

Sometimes you want to skip that CGI Status header

I'm on a roll with these "check out what I screwed up" posts, so let's add another one to the mix. This one is about the feed reader score service and a particular foible it had ever since it launched until a few minutes ago when I finally fixed it.

The foible is: no matter what you sent to the project's test server, it would return a full copy of the meaningless placeholder test feed with a 200 response code. Even if you nailed the If-Modified-Since and/or If-None-Match headers, you'd get a full response as if you fired off an unconditional request like the worst clownware.

Well, in this case, the clown is me, and here's how it happened.

I've been writing hacky little CGI scripts for testing purposes for a very long time, and they generally look like this:

#!/bin/bash
set -e -u -o pipefail

echo "Content-type: text/plain"
echo ""
echo "Something or other here"

Simple and stupid, right? You don't need a Status header, and Apache will happily send a 200 for you. You can set one if you're deliberately overriding it, naturally. I'd only do that for weird stuff, like forcing a 404, or a 301 redirect, or something like that.

What I didn't appreciate is that you can provide a body in there, and as long as you emit the right headers, Apache will do the legwork to see if it matches whatever the client sent. It'll actually absorb the body and will send out a nice "304 Not Modified" instead.

The problem comes from the fact that I wrote a dumb little library to handle CGI stuff a while back, and one of the things it does is to make sure that every single request gets some kind of response. It starts out with status set to 500 (as in "server error"), and it's on you and your handler to set it to something appropriate. This way, nothing goes out the door until you do all of the work required to set everything up properly: the fact it's a 200, an appropriate content-type, access controls on who/what can load it, and all of that other good stuff. Then it always sends a Status header which matches its internal state.

As a result, it was happily blasting out "Status: 200 OK" when it would run within the context of the fake testing feed endpoint which is part of this project. I now know that this will make Apache not do its usual conditional test and instead it will ship the body as-is... with the 200 you told it to use.

Why did it take me a while to figure this out? Well, did you notice that my dumb little shell script never sends a Status header? It doesn't try to cover all of the angles because it's "just a dumb little script". In being implicit about the result code, it ends up enabling a little useful bit of behavior that I wish I had all along.

If you're one of the people participating in the test, and your program just started getting 304s from my end, that's why! You're not hallucinating. You're just seeing the outcome of me finally realizing "why this thing is not like that thing".

As for what the fix was, it's "allow this program to suppress Status generation when it's going down the happy path". It's stupid, but it works.

Side note: this is only talking about the feed score test endpoint. My actual feed (i.e., the thing with this post in it) works completely differently and has been doing 304s properly the whole time.

Solving the NYTimes Pips puzzle with a constraint solver

The New York Times recently introduced a new daily puzzle called Pips. You place a set of dominoes on a grid, satisfying various conditions. For instance, in the puzzle below, the pips (dots) in the purple squares must sum to 8, there must be fewer than 5 pips in the red square, and the pips in the three green squares must be equal. (It doesn't take much thought to solve this "easy" puzzle, but the "medium" and "hard" puzzles are more challenging.)

The New York Times Pips puzzle from Oct 5, 2025 (easy). Hint: What value must go in the three green squares?

The New York Times Pips puzzle from Oct 5, 2025 (easy). Hint: What value must go in the three green squares?

I was wondering about how to solve these puzzles with a computer. Recently, I saw an article on Hacker News—"Many hard LeetCode problems are easy constraint problems"—that described the benefits and flexibility of a system called a constraint solver. A constraint solver takes a set of constraints and finds solutions that satisfy the constraints: exactly what Pips requires.

I figured that solving Pips with a constraint solver would be a good way to learn more about these solvers, but I had several questions. Did constraint solvers require incomprehensible mathematics? How hard was it to express a problem? Would the solver quickly solve the problem, or would it get caught in an exponential search?

It turns out that using a constraint solver was straightforward; it took me under two hours from knowing nothing about constraint solvers to solving the problem. The solver found solutions in milliseconds (for the most part). However, there were a few bumps along the way. In this blog post, I'll discuss my experience with the MiniZinc1 constraint modeling system and show how it can solve Pips.

Approaching the problem

Writing a program for a constraint solver is very different from writing a regular program. Instead of telling the computer how to solve the problem, you tell it what you want: the conditions that must be satisfied. The solver then "magically" finds solutions that satisfy the problem.

To solve the problem, I created an array called pips that holds the number of domino pips at each position in the grid. Then, the three constraints for the above problem can be expressed as follows. You can see how the constraints directly express the conditions in the puzzle.

constraint pips[1,1] + pips[2,1] == 8;
constraint pips[2,3] < 5;
constraint all_equal([pips[3,1], pips[3,2], pips[3,3]]);

Next, I needed to specify where dominoes could be placed for the puzzle. To do this, I defined an array called grid that indicated the allowable positions: 1 indicates a valid position and 0 indicates an invalid position. (If you compare with the puzzle at the top of the article, you can see that the grid below matches its shape.)

grid = [|
1,1,0|
1,1,1|
1,1,1|];

I also defined the set of dominoes for the problem above, specifying the number of spots in each half:

spots = [|5,1| 1,4| 4,2| 1,3|];

So far, the constraints directly match the problem. However, I needed to write some more code to specify how these pieces interact. But before I describe that code, I'll show a solution. I wasn't sure what to expect: would the constraint solver give me a solution or would it spin forever? It turned out to find the unique solution in 109 milliseconds, printing out the solution arrays. The pips array shows the number of pips in each position, while the dominogrid array shows which domino (1 through 4) is in each position.

pips = 
[| 4, 2, 0
 | 4, 5, 3
 | 1, 1, 1
 |];
dominogrid = 
[| 3, 3, 0
 | 2, 1, 4
 | 2, 1, 4
 |];

The text-based solution above is a bit ugly. But it is easy to create graphical output. MiniZinc provides a JavaScript API, so you can easily display solutions on a web page. I wrote a few lines of JavaScript to draw the solution, as shown below. (I just display the numbers since I was too lazy to draw the dots.) Solving this puzzle is not too impressive—it's an "easy" puzzle after all—but I'll show below that the solver can also handle considerably more difficult puzzles.

Graphical display of the solution.

Graphical display of the solution.

Details of the code

While the above code specifies a particular puzzle, a bit more code is required to define how dominoes and the grid interact. This code may appear strange because it is implemented as constraints, rather than the procedural operations in a normal program.

My main design decision was how to specify the locations of dominoes. I considered assigning a grid position and orientation to each domino, but it seemed inconvenient to deal with multiple orientations. Instead, I decided to position each half of the domino independently, with an x and y coordinate in the grid.2 I added a constraint that the two halves of each domino had to be in neighboring cells, that is, either the X or Y coordinates had to differ by 1.

constraint forall(i in DOMINO) (abs(x[i, 1] - x[i, 2]) + abs(y[i, 1] - y[i, 2]) == 1);

It took a bit of thought to fill in the pips array with the number of spots on each domino. In a normal programming language, one would loop over the dominoes and store the values into pips. However, here it is done with a constraint so the solver makes sure the values are assigned. Specifically, for each half-domino, the pips array entry at the domino's x/y coordinate must equal the corresponding spots on the domino:

constraint forall(i in DOMINO, j in HALF) (pips[y[i,j], x[i, j]] == spots[i, j]);

I decided to add another array to keep track of which domino is in which position. This array is useful to see the domino locations in the output, but it also keeps dominoes from overlapping. I used a constraint to put each domino's number (1, 2, 3, etc.) into the occupied position of dominogrid:

constraint forall(i in DOMINO, j in HALF) (dominogrid[y[i,j], x[i, j]] == i);

Next, how do we make sure that dominoes only go into positions allowed by grid? I used a constraint that a square in dominogrid must be empty or the corresponding grid must allow a domino.3 This uses the "or" condition, which is expressed as \/, an unusual stylistic choice. (Likewise, "and" is expressed as /\. These correspond to the logical symbols ∨ and ∧.)

constraint forall(i in 1..H, j in 1..W) (dominogrid[i, j] == 0 \/ grid[i, j] != 0);

Honestly, I was worried that I had too many arrays and the solver would end up in a rathole ensuring that the arrays were consistent. But I figured I'd try this brute-force approach and see if it worked. It turns out that it worked for the most part, so I didn't need to do anything more clever.

Finally, the program requires a few lines to define some constants and variables. The constants below define the number of dominoes and the size of the grid for a particular problem:

int: NDOMINO = 4; % Number of dominoes in the puzzle
int: W = 3; % Width of the grid in this puzzle
int: H = 3; % Height of the grid in this puzzle

Next, datatypes are defined to specify the allowable values. This is very important for the solver; it is a "finite domain" solver, so limiting the size of the domains reduces the size of the problem. For this problem, the values are integers in a particular range, called a set:

set of int: DOMINO = 1..NDOMINO; % Dominoes are numbered 1 to NDOMINO
set of int: HALF = 1..2; % The domino half is 1 or 2
set of int: xcoord = 1..W; % Coordinate into the grid
set of int: ycoord = 1..H;

At last, I define the sizes and types of the various arrays that I use. One very important syntax is var, which indicates variables that the solver must determine. Note that the first two arrays, grid and spots do not have var since they are constant, initialized to specify the problem.

array[1..H,1..W] of 0..1: grid; % The grid defining where dominoes can go
array[DOMINO, HALF] of int: spots; % The number of spots on each half of each domino
array[DOMINO, HALF] of var xcoord: x; % X coordinate of each domino half
array[DOMINO, HALF] of var ycoord: y; % Y coordinate of each domino half
array[1..H,1..W] of var 0..6: pips; % The number of pips (0 to 6) at each location.
array[1..H,1..W] of var 0..NDOMINO: dominogrid; % The domino sequence number at each location

You can find all the code on GitHub. One weird thing is that because the code is not procedural, the lines can be in any order. You can use arrays or constants before you use them. You can even move include statements to the end of the file if you want!

Complications

Overall, the solver was much easier to use than I expected. However, there were a few complications.

By changing a setting, the solver can find multiple solutions instead of stopping after the first. However, when I tried this, the solver generated thousands of meaningless solutions. A closer look showed that the problem was that the solver was putting arbitrary numbers into the "empty" cells, creating valid but pointlessly different solutions. It turns out that I didn't explicitly forbid this, so the sneaky constraint solver went ahead and generated tons of solutions that I didn't want. Adding another constraint fixed the problem. The moral is that even if you think your constraints are clear, solvers are very good at finding unwanted solutions that technically satisfy the constraints. 4

A second problem is that if you do something wrong, the solver simply says that the problem is unsatisfiable. Maybe there's a clever way of debugging, but I ended up removing constraints until the problem can be satisfied, and then see what I did wrong with that constraint. (For instance, I got the array indices backward at one point, making the problem insoluble.)

The most concerning issue is the unpredictability of the solver: maybe it will take milliseconds or maybe it will take hours. For instance, the Oct 5 hard Pips puzzle (below) caused the solver to take minutes for no apparent reason. However, the MiniZinc IDE supports different solver backends. I switched from the default Gecode solver to Chuffed, and it immediately found numerous solutions, 384 to be precise. (Sometimes the Pips puzzles sometimes have multiple solutions, which players find controversial.) I suspect that the multiple solutions messed up the Gecode solver somehow, perhaps because it couldn't narrow down a "good" branch in the search tree. For a benchmark of the different solvers, see the footnote.5

Two of the 384 solutions to the NYT Pips puzzle from Oct 5, 2025 (hard difficulty).

Two of the 384 solutions to the NYT Pips puzzle from Oct 5, 2025 (hard difficulty).

How does a constraint solver work?

If you were writing a program to solve Pips from scratch, you'd probably have a loop to try assigning dominoes to positions. The problem is that the problem grows exponentially. If you have 16 dominoes, there are 16 choices for the first domino, 15 choices for the second, and so forth, so about 16! combinations in total, and that's ignoring orientations. You can think of this as a search tree: at the first step, you have 16 branches. For the next step, each branch has 15 sub-branches. Each sub-branch has 14 sub-sub-branches, and so forth.

An easy optimization is to check the constraints after each domino is added. For instance, as soon as the "less than 5" constraint is violated, you can backtrack and skip that entire section of the tree. In this way, only a subset of the tree needs to be searched; the number of branches will be large, but hopefully manageable.

A constraint solver works similarly, but in a more abstract way. The constraint solver assigns values to the variables, backtracking when a conflict is detected. Since the underlying problem is typically NP-complete, the solver uses heuristics to attempt to improve performance. For instance, variables can be assigned in different orders. The solver attempts to generate conflicts as soon as possible so large pieces of the search tree can be pruned sooner rather than later. (In the domino case, this corresponds to placing dominoes in places with the tightest constraints, rather than scattering them around the puzzle in "easy" spots.)

Another technique is constraint propagation. The idea is that you can derive new constraints and catch conflicts earlier. For instance, suppose you have a problem with the constraints "a equals c" and "b equals c". If you assign "a=1" and "b=2", you won't find a conflict until later, when you try to find a value for "c". But with constraint propagation, you can derive a new constraint "a equals b", and the problem will turn up immediately. (Solvers handle more complicated constraint propagation, such as inequalities.) The tradeoff is that generating new constraints takes time and makes the problem larger, so constraint propagation can make the solver slower. Thus, heuristics are used to decide when to apply constraint propagation.

Researchers are actively developing new algorithms, heuristics, and optimizations6 such as backtracking more aggressively (called "backjumping"), keeping track of failing variable assignments (called "nogoods"), and leveraging Boolean SAT (satisfiability) solvers. Solvers compete in annual challenges to test these techniques against each other. The nice thing about a constraint solver is that you don't need to know anything about these techniques; they are applied automatically.

Conclusions

I hope this has convinced you that constraint solvers are interesting, not too scary, and can solve real problems with little effort. Even as a beginner, I was able to get started with MiniZinc quickly. (I read half the tutorial and then jumped into programming.)

One reason to look at constraint solvers is that they are a completely different programming paradigm. Using a constraint solver is like programming on a higher level, not worrying about how the problem gets solved or what algorithm gets used. Moreover, analyzing a problem in terms of constraints is a different way of thinking about algorithms. Some of the time it's frustrating when you can't use familiar constructs such as loops and assignments, but it expands your horizons.

Finally, writing code to solve Pips is more fun than solving the problems by hand, at least in my opinion, so give it a try!

For more, follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), RSS, or subscribe here.

Solution to the Pips puzzle, September 21, 2005 (hard). This puzzle has regions that must all be equal (=) and regions that must all be different (≠). Conveniently, MiniZinc has all_equal and alldifferent constraint functions.

Solution to the Pips puzzle, September 21, 2005 (hard). This puzzle has regions that must all be equal (=) and regions that must all be different (≠). Conveniently, MiniZinc has all_equal and alldifferent constraint functions.

Notes and references

  1. I started by downloading the MiniZinc IDE and reading the MiniZinc tutorial. The MiniZinc IDE is straightforward, with an editor window at the top and an output window at the bottom. Clicking the "Run" button causes it to generate a solution.

    Screenshot of the MiniZinc IDE. Click for a larger view.

    Screenshot of the MiniZinc IDE. Click for a larger view.

     

  2. It might be cleaner to combine the X and Y coordinates into a single Point type, using a MiniZinc record type

  3. I later decided that it made more sense to enforce that dominogrid is empty if and only if grid is 0 at that point, although it doesn't affect the solution. This constraint uses the "if and only if" operator <->.

    constraint forall(i in 1..H, j in 1..W) (dominogrid[i, j] == 0 <-> grid[i, j] == 0);
    
     
  4. To prevent the solver from putting arbitrary numbers in the unused positions of pips, I added a constraint to force these values to be zero:

    constraint forall(i in 1..H, j in 1..W) (grid[i, j] == 0 -> pips[i, j] == 0);
    

    Generating multiple solutions had a second issue, which I expected: A symmetric domino can be placed in two redundant ways. For instance, a double-six domino can be flipped to produce a solution that is technically different but looks the same. I fixed this by adding constraints for each symmetric domino to allow only one of the two redundant positions. The constraint below forces a preferred orientation for symmetric dominoes.

    constraint forall(i in DOMINO) (spots[i,1] != spots[i,2] \/ x[i,1] > x[i,2] \/ (x[i,1] == x[i,2] /\ y[i,1] > y[i,2]));
    

    To enable multiple solutions in MiniZinc, the setting is under Show Configuration Editor > User Defined Behavior > Satisfaction Problems or the --all flag from the command line. 

  5. MiniZinc has five solvers that can solve this sort of integer problem: Chuffed, OR Tools CP-SAT, Gecode, HiGHS, and Coin-OR BC. I measured the performance of the five solvers against 20 different Pips puzzles. Most of the solvers found solutions in under a second, most of the time, but there is a lot of variation.

    Timings for different solvers on 20 Pip puzzles.

    Timings for different solvers on 20 Pip puzzles.

    Overall, Chuffed had the best performance on the puzzles that I tested, taking well under a second. Google's OR-Tools won all the categories in the 2025 MiniZinc challenge, but it was considerably slower than Chuffed for my Pips programs. The default Gecode solver performed very well most of the time, but it did terribly on a few problems, taking over 15 minutes. HiGHs was slower in general, taking a few minutes on the hardest problems, but it didn't fail as badly as Gecode. (Curiously, Gecode and HiGHS sometimes found different problems to be difficult.) Finally, Coin-OR BC was uniformly bad; at best it took a few seconds, but one puzzle took almost two hours and others weren't solved before I gave up after two hours. (I left Coin-OR BC off the graph because it messed up the scale.)

    Don't treat these results too seriously because different solvers are optimized for different purposes. (In particular, Coin-OR BC is designed for linear problems.) But the results demonstrate the unpredictability of solvers: maybe you get a solution in a second and maybe you get a solution in hours. 

  6. If you want to read more about solvers, Constraint Satisfaction Problems is an overview presentation. The Gecode algorithms are described in a nice technical report: Constraint Programming Algorithms used in Gecode. Chuffed is more complicated: "Chuffed is a state of the art lazy clause solver designed from the ground up with lazy clause generation in mind. Lazy clause generation is a hybrid approach to constraint solving that combines features of finite domain propagation and Boolean satisfiability." The Chuffed paper Lazy clause generation reengineered and slides are more of a challenge.  

I love the smell of autopoiesis in the morning

I belatedly joined the 3D printer club and got myself a brand new Bambu P1S. (Days before the next-gen P2S was released it turns out, but I got the P1S on sale so I don’t feel so bad.)

I have wooden floors.

So I’ve been printing printer foot adaptors to make the optional rubber anti-vibration feet (which I also bought) more effective.

Here’s the 3D model for the adaptors. You open it in the app, select your printer and filament, and it prints directly.

Here are the adaptors printing on my insta.

I am very very into this as a concept: using the printer to print parts of itself. It is a glimpse of the old vision of 3D printers as bootstrapping machines. The old vision and maybe the future too.

(Bootstrapping (Wikipedia) in the self-starting sense, i.e. "analogous to one who would lift himself by his own bootstraps," which is where booting/rebooting/etc for computers comes from.)

For instance RepRap:

a free desktop 3D printer capable of printing plastic objects. Since many parts of RepRap are made from plastic and RepRap prints those parts, RepRap self-replicates by making a kit of itself - a kit that anyone can assemble given time and materials. It also means that - if you’ve got a RepRap - you can print lots of useful stuff, and you can print another RepRap for a friend…

RepRap started in 2005. Remember the heady days of maker culture? People really felt like we were on a path to decentralising and democratising manufacturing; putting the means of production in the hands of everyone.

I believe the RepRap project is now dormant.

But it harks back to an older idea, being John von Neumann’s concept of a universal constructor:

the self-replicating machine consists of three parts: a “description” of (‘blueprint’ or program for) itself, a universal constructor mechanism that can read any description and construct the machine (sans description) encoded in that description, and a universal copy machine that can make copies of any description.

Autopoiesis, right? A system that creates and maintains itself. Life!

In 1980 Robert Freitas took the universal constructor idea for a Nasa study and came up with the self-replicating interstellar probe, still the most rapid architecture for reaching every star system in the Milky Way (due to its exponential growth). On reaching a new star system, REPRO would build mines and factories etc in the local asteroid belt or whatever to make >1 copies of itself, which would then travel onward to multiple other star systems, rinse and repeat.

BTW:

As previously discussed, given the effectiveness of the Von Neumann probe approach, I feel that it’s likely that our own civilisation is merely a stepping stone (as previously discussed, 2022) in someone else’s exploration of the galaxy.

So when I’m printing parts for my new printer using my new printer, I am not just taking a shortcut around ordering accessories off Amazon.

Nor am I just being part of a nascent democratic mode of production.

But rather:

I am participating in the first inch of the first step to develop the tools and technologies that will one day be used to build the very first ancestral self-replicating von Neumann probe and so light the fuse on the human exploration of our home galaxy.

In a microscopic way anyhow. The feeling is there, a faint shimmer, the glint of distant stars in the deep dark. And the printer foot adaptors are handy too. Bright orange!

Cyborgs vs rooms, two visions for the future of computing

Loosely I can see two visions for the future of how we interact with computers: cyborgs and rooms.

The first is where the industry is going today; I’m more interested in the latter.

Cyborgs

Near-term, cyborgs means wearables.

The original definition of cyborg by Clynes and Kline in 1960 was of a human adapting its body to fit a new environment (as previously discussed).

Apple AirPods are cyborg enhancements: transparency mode helps you hear better.

Meta AI glasses augment you with better memory and the knowledge of the internet – you mutter your questions and the answer is returned in audio, side-loaded into your working memory. Cognitively this feels just like thinking hard to remember something.

I can see a future being built out where I have a smart watch that gives me a sense of direction, a smart ring for biofeedback, smart earphones and glasses for perfect recall and anticipation… Andy Clark’s Natural Born Cyborgs (2003) lays out why this is perfectly impedance-matched to how our brains work already.

Long term? I’ve joked before about a transcranial magnetic stimulation helmet that would walk my legs to work and this is the cyborg direction of travel: nootropics, CRISPR gene therapy, body modification and slicing open your fingertips to insert magnets for an electric field sixth sense.

But you can see the cyborg paradigm in action with hardware startups today trying to make the AI-native form factor of the future: lapel pins, lanyards, rings, Neuralink and other brain-computer interfaces…

When tech companies think about the Third Device - the mythical device that comes after the PC and the smartphone - this is what they reach for: the future of the personal computer is to turn the person into the computer.

Rooms

Contrast augmented users with augmented environments. Notably:

  • Dynamicland (2018) – Bret Victor’s vision of "a computer that is a place," a programmable room
  • Put-that-there (1980) – MIT research into room-scale, multimodal (voice and gesture) conversational computing
  • Project Cybersyn (1971) – Stafford Beer’s room-sized cybernetic brain for the economy of Chile
  • SAGE (as previously discussed) (1958–) – the pinnacle of computing before the PC, group computing out of the Cold War.

And innumerable other HCI projects…

The vision of room-scale computing has always had factions.

Is it ubiquitous computing (ubicomp), in which computing power is embedded in everything around us, culminating in smart dust? It is ambient computing, which also supposes that computing will be invisible? Or calm computing, which is more of a design stance that computing must mesh appropriately with our cognitive systems instead of chasing attention?

So there’s no good word for this paradigm, which is why I call it simply room-scale, which is the scale that I can act as a user.

I would put smart speakers in the room-scale/augmented environments bucket: Amazon Alexa, Google Home, all the various smart home systems like Matter, and really the whole internet of things movement – ultimately it’s a Star Trek Holodeck/"Computer…" way of seeing the future of computer interaction.

And robotics too. Roomba, humanoid robots that do our washing up, and tabletop paper robots that act as avatars for your mates, all part of this room-scale paradigm.


Rather than “cyborg”, I like sci-fi author Becky Chambers’ concept of somaforming (as previously discussed), the same concept but gentler.

Somaforming vs terraforming, changing ourselves to adapt to a new environment, or changing the environment to adapt to us.


Both cyborgs and rooms are decent North Stars for our collective computing futures, you know?

Both can be done in good ways and ugly ways. Both can make equal use of AI.

Personally I’m more interested in room-scale computing and where that goes. Multi-actor and multi-modal. We live in the real world and together with other people, that’s where computing should be too. Computers you can walk into… and walk away from.

So it’s an interesting question: while everyone else is building glasses, AR, and AI-enabled cyborg prosthetics that hang round your neck, what should we build irl, for the rooms where we live and work? What are the core enabling technologies?

It has been overlooked I think.


More posts tagged: glimpses-of-our-cyborg-future (15).

Auto-detected kinda similar posts:

Parables involving gross bodily functions

When I was at uni I went with some friends into a cellar with a druid to experience his home-made 6 foot tall light-strobing monolith, what I now know was a type of Dreammachine, which was a pair of stacked out-of-sync strobing lights so bright that it makes the receptors in your eyes produce concentric circles of moving colour even with your eyelids tight shut.

His performance of changing and interconnected strobe frequencies, which he had composed specially for my friend’s birthday, was exciting but also pretty overwhelming and long story short it made me puke.


Another story.

When I was a teenager there was a guy in our circle who - and I don’t know how this came up and apologies in advance for the image - when we were discussing what your poo might feel like, because truthfully you don’t know right, it’s there at the bottom of the water in the loo and you see it and can imagine the feeling it of but you don’t know with the actual sensory accuracy of your hands - he said he did know, and when we asked he continued that he had wondered once, so he had reached into the loo and picked one up, it’s just washing your hands after that’s all, he said.

And we were all vocally disgusted but actually I have always admired him, then and today decades later (I remember exactly where we were standing outside in the garden when he told us) for his clarity of purpose and also with jealousy of his secret knowledge. Anyway he is a successful Hollywood producer now.


Here’s some advice that has never left me since I read it: "When stumped by a life choice, choose “enlargement” over happiness" (as previously discussed).

I feel like sometimes I don’t do some enlarging things, big or small, because it never occurs to me because it would involve something vaguely uncomfortable. Not gross necessarily, but uncomfortable like going into a room that I don’t feel like I belong in, or doing something publicly I don’t know to do with automatic unpleasant embarrassment included. An assumed micro-unhappiness.

I’m pretty good at challenging myself when the potential discomfort is visible to me, but the non-obvious ones are insidious.

It’s important to recognise these invisible discomforts because maybe actually they don’t matter, and they’re inhibiting me from doing things.

I’m talking about the opposite of a velleity.

David Chapman’s concept of velleity, as discussed by DRMacIver, which is where I learnt about it:

A “velleity” is a wish so weak that it does not occur to you to act on it. These are desires you instantly dismiss because they do not match your picture of what you think you want.

(But you should identify them and pursue them! That’s the point!)

So these anti-velleities, micro-discomforts, when unrecognised they act as blinkers so I don’t even think of the possibilities I am denying myself.

The future Hollywood producer was able to see past that.

Wise beyond his years!

Not all enlargements matter, of course. Gotta kiss a lot of frogs.

And what else is life for, if not puking in a druid’s cellar on a friend’s birthday? Answer me that.

The Campaign for Vertical Television

TV networks should have an extra channel that shows exclusively portrait mode content.

i.e. BBC Vertical.

Also: Netflix Vertical, National Geographic Vertical, ESPN Vertical, etc.

Why?

  • Because there’s demand
  • To discover how to make good portrait-mode content
  • To promote new consumer electronics
  • To counter slop.

Good for phones and big black bars down the sides on regular TVs, at least until they ship new ones that hand differently.

Strategic play init.


Why don’t networks have vertical channels already?

They should.

It’s how most people consume most video, judging by how I see people watching on the train.

Sometimes I do see people watching in landscape on the train, it’s true. But it’s always something massively popular like live football or Love Island. Anything less, people avoid regular horizontal TV. (I’ll come back to why that is.)

I think there’s an assumption from the commissioners at traditional TV that portrait mode = short form = story mode. So they look down on it and that’s why they don’t produce any long-form vertical content.

That was the mistaken assumption also made by Quibi (Wikipedia) which launched in April 2020 with a ton of traditional content partners and investor backing of $1.75bn. It flamed out by the end of the year.

The premise was portrait mode, mobile-first TV and ALSO in 10 minute segments.

Too much. To me their approach over-determined the format. (Also I feel like it had misaligned incentives: the only way Quibi would have worked long-term is if viewers habitually opened the Quibi app first and what, content partners are going to cannibalise their existing audiences by pointing them at this upstart rival brand?)

The core premise - that portrait mode TV should be a thing - is good.

But forget the “10 minute segments” constraint. Forget story mode.


So why do people avoid watching horizontally?

Well, it’s so distancing to watch footie or regular telly on a horizontal phone, you feel so far away. It’s un-engaging.

But more than that, it’s uncomfortable…

My dumb explanation of portrait mode dominance is that phones are easier to hold that way, and physically uncomfortable to hold in landscape – increasingly so as they get bigger (compensating for that feeling of being far away). It’s just the shape of our hands and our wrists and the way our necks attach to our heads.

And the only screen-filling content available when you have your phone vertical is story mode content.

But that doesn’t intrinsically mean that it’s the best we can do.


It wouldn’t be enough to take regular TV and cut the sides. Vertical formats will be different.

This is the kind of content that you’d want on BBC Vertical:

  • long-form drama
  • long-form video podcasts
  • news
  • live sport

Live sport in particular. The age of blockbuster MCU-style movies is over. Spoilers are on the socials instantly, there are no surprises in cinematic universes anymore; the real time spent is in the fandoms. You can get an AI to churn out wilder special effects than anything Hollywood can do. Or see it on the news. Who cares about a superhero thumping another superhero.

But live sport? Genuine surprise, genuine drama, genuine “appointments to view” and social shared experiences.

But vertical live sport will need to be shot differently. Different zoom, different angles.

Ditto drama. While vertical drama is not shorter necessarily, it is more first person.

e.g. what the kids are watching:

I walked into a room of tweens watching YouTube on the big screen in the dark at a party yesterday and that’s how I learnt about NEN FAM.

The NEN FAM YouTube channel has over 3m subscribers and a ton of vids over 7m views, which is more than a lot of popular TV shows.

(Sidenote: But no Wikipedia page. Wikipedia is stuck with topics which are notable for millennials and older, right? This is a sign that it has ossified?)

afaict NEN FAM is a bunch of 16 kids in Utah who live in the same house and they get up to hi-jinx like sneaking out at midnight but telling their parents first (YouTube, 1 hour) and hanging out at a playground, all filmed like a super vapid Blair Witch Project. It’s v wholesome.

Join me next week for another instalment of Matt Encounters Modern Culture

Anyway formats will be different is what I’m saying.


So NEN FAM and its like is the future of entertainment, given what kids do today is what mainstream culture is tomorrow.

Honestly I think the only reasons that NEN FAM is not vertical-first are

  • The YouTube algo pushes against that
  • Big screens remain obstinately horizontal.

But how would you do vertical Pride & Prejudice?

omg can you even imagine?

House of Cards with its breaking the 3rd wall and mix in characters carrying the phone pov-style. It would be amazing.

All of this needs to be figured out. Producers, directors, writers, actors, camera crew, lighting – this won’t happen automatically. There needs to be distribution (i.e. the new channels I propose) and the pump needs to be primed so everyone involved has room to learn.


None of this will happen without vertical big screens in the front room.

And this is where there’s opportunity.

Samsung, Sony, LG, all the new Chinese consumer electronics manufacturers: every single one of them is looking for the trigger for a refresh super cycle.

So here’s my proposal:

The consumer electronics companies should get together and start an industry consortium, say the Campaign for Vertical Television or something.

C4VTV would include a fund that they all chip in to.

Then any broadcast or streaming TV network can apply for grants to launch and commission content for their new channels.

It’s a long-term play, sure.

But it has a way better chance of shifting consumer behaviour (and the products we buy) than - to take an example - that push for home 3D a decade ago. At least this pump-priming push is in line with a consumer behaviour shift which is already underway.


The imperative here is that vertical video is getting locked in today, and a lot of it is super poisonous.

Story mode is so often engagement farming, attention mining capitalist slop.

And we’ve seen this week two new apps which are AI-video-only TikTok competitors: Sora from OpenAI and Vibes by Meta AI.

Look I love AI slop as previously discussed.

But what is being trained with these apps is not video quality or internal physics model consistency. They’re already good at that. What they’re pointing the AI machine at is learning how to get a stranglehold on your attention.

The antidote? Good old fashioned content from good old fashion program makers.

But: vertical.


Not just television btw.

Samsung – when you establish the Campaign for Vertical Television, or whatever we call it, can you also throw some money at Zoom and Google Meet to get them to go portrait-mode too?

Honestly on those calls we’re talking serious business things and building relationship. I want big faces. Nobody needs to devote two thirds of screen real estate to shoulders.


Auto-detected kinda similar posts:

I like AI slop and I cannot lie

I looked in my home directory in my desktop Mac, which I don’t very often (I run a tidy operation here), and I found a file I didn’t recognise called out.html.

Here is out.html.

For the benefit of the tape: it is a half-baked GeoCities-style homepage complete with favourite poems, broken characters, and a "This page is best viewed with Netscape Navigator 4.0 or higher!" message in the footer.

The creation date of the file is March of this year.

I don’t know how it got there.

Maybe my computer is haunted?


I have a vague memory of trying out local large language models for HTML generation, probably using the llm command-line tool.

out.html is pretty clearly made with AI (the HTML comments, if you View Source, are all very LLM-voice).

But it’s… bad. ChatGPT or Claude in 2025 would never make a fake GeoCities page this bad.

So what I suspect has happened is that I downloaded a model to run on my desktop Mac, prompted it to save its output into my home directory (lazily), then because the model was local it was really slow… then got distracted and forgot about it while it whirred away in a window in the background, only finding the output 6 months down the line.


UPDATE. This is exactly what happened! I just realised I can search my command history and here is what I typed:

llm -m gemma3:27b ‘Build a single page HTML+CSS+JavaScript UI which looks like an old school GeoCities page with poetry and fave books/celebs, and tons and tons of content. Use HTML+CSS really imaginatively because we do not have images. Respond with only the HTML so it can be run immediately’ > out.html

And that will have taken a whole bunch of time so I must have tabbed elsewhere and not even looked at the result.


Because I had forgotten all about it, it was as if I had discovered a file made by someone else. Other footprints on the deserted beach.

I love it.

I try to remain sensitive to New Feelings.

e.g…

The sense of height and scale in VR is a New Feeling: "What do we do now the gamut of interaction can include vertigo and awe? It’s like suddenly being given an extra colour."

And voice: way back I was asked to nominate for Designs of the Year 2016 and one my nominations was Amazon Echo – it was new! Here’s part of my nomination statement:

we’re now moving into a Post PC world: Our photos, social networks, and taxi services live not in physical devices but in the cloud. Computing surrounds us. But how will we interact with it?

So the New Feeling wasn’t voice per se, but that the location of computing/the internet had transitioned from being contained to containing us, and that felt new.

(That year I also nominated Moth Generator and Unmade, both detailed in dezeen.)

I got a New Feeling when I found out.html just now.


Stumbling across littered AI slop, randomly in my workspace!

I love it, I love it.

It’s like having a cat that leaves dead birds in the hall.

Going from living in a house in which nothing changes when nobody is in the house to a house which has a cat and you might walk back into… anything… is going from 0 to 1 with “aliveness.” It’s not much but it’s different.

Suddenly my computer feels more… inhabited??… haunted maybe, but in a good way.


Three references about computers being inhabited:

  1. Every page on my blog has multiplayer cursors and cursor chat because every webpage deserves to be a place (2024) – and once you realise that a webpage can show passers-by then all other webpages feel obstinately lonely.
  2. Little Computer People (1985), the Commodore 64 game that revealed that your computer was really a tiny inhabited house, and I was obsessed at the time. LCP has been excellently written up by Jay Springett (2024).
  3. I wrote about Gordon Brander’s concept for Geists (2022). Geists are/were little bots that meander over your notes directory, "finding connections between notes, remixing notes, issuing oracular provocations and gnomic utterances."

And let’s not forget Steve Jobs’ unrealised vision for Mister Macintosh: "a mysterious little man who lives inside each Macintosh. He pops up every once in a while, when you least expect it, and then winks at you and disappears again."


After encountering out.html I realise that I have an Old Feeling which is totally unrecognised, and the old feeling, which has always been there it turns out, is that being in my personal computer is lonely.

I would love a little geist that runs a local LLM and wanders around my filesystem at night, perpetually out of sight.

I would know its presence only by the slop it left behind, slop as ectoplasm from where the ghost has been,

a collage of smiles cut out of photos from 2013 and dropped in a mysterious jpg,

some doggerel inspired by a note left in a text file in a rarely-visited dusty folder,

if I hit Back one to many times in my web browser it should start hallucinating whole new internets that have never been.


More posts tagged: ghosts (7).

Auto-detected kinda similar posts:

Filtered for bird calls and catnip

1.

With less human noise to compete with, the birds are able to have ‘deeper conversations,’ says biologist (2020):

Researchers studying birdsong in the San Francisco Bay found the sparrows’ mating calls became quieter, more complex, and just generally “sexier” now that they don’t have to compete with the sounds of cars and cellphones, says study co-author Elizabeth Derryberry.

A side-effect of the pandemic:

Without cars, mating calls travel twice the distance, and also more information can be transmitted.

Does this analogy work? It’s the one they give: "As the party winds down and people go home, you get quieter again, right? You don’t keep yelling, and you maybe have your sort of deeper conversations at that point."

I would love to see a follow-up study? For a brief period, long-form discursive song was favoured. So was there a generation of famous sparrow rhetoricians, like the orators of Ancient Greece? Do they look back on the early 2020s as the golden age of sparrow Homer?

PREVIOUSLY:

Just pandemic things (2023).

2.

Parrots taught to video call each other become less lonely, finds research (The Guardian, 2023): "In total the birds made 147 deliberate calls to each other during the study."

Some would sing, some would play around and go upside down, others would want to show another bird their toys.

More at the FT: Scientists pioneer ‘animal internet’ with dog phones and touchscreens for parrots (paywall-busting link).

26 birds involved … would use the system up to three hours a day, with each call lasting up to five minutes.

Why? Because pet parrots:

typically live alone in their owners’ homes though their counterparts in the wild typically socialise within large flocks.

Flocks.

You know, although the 1:1 parrot-phone is interesting, I wonder whether a zoom conference call would be more appropriate? Or, better, an always-on smart speaker that is a window to a virtual forest, collapsing geography.

Another project, mentioned in that same article:

Ilyena Hirskyj-Douglas, who heads the university’s Animal-Computer Interaction Group, started by developing a DogPhone that enables animals to contact their owners when they are left alone.

Ref.

Birds of a Feather Video-Flock Together: Design and Evaluation of an Agency-Based Parrot-to-Parrot Video-Calling System for Interspecies Ethical Enrichment, CHI ‘23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems.

3.

Empty cities, virtual forests… this bird sings across time.

Lubman first became intrigued by reports of a curious echo from the Mayan pyramid of Kukulkan at Chichen Itza, in Mexico’s Yucatan region. The odd “chirped” echo resounds from the pyramid’s staircases in response to hand claps of people standing near its base. To hear for himself, Lubman packed up his recording gear and traveled to Chichen Itza last January.

After studying the staircases and analyzing his recordings and sonograms of the echoes, Lubman came back convinced that this was no architectural freak. In his paper, Lubman argued that the design of the staircases was deliberate and that the echo is an ancient recording, coded in stone, of the call of the Maya’s sacred bird, the quetzal.

omg too sublime, I am a sucker for avian archaeoacoustics

PREVIOUSLY:

Microdosing cathedrals and the synthetic acoustic environment of the ancient world (2022).

4.

You will NEVER GUESS what I found out about CATNIP this week.

You know when cats go crazy about catnip and start rubbing their faces all over it?

Yeah well catnip is a mosquito repellent. So it’s evolutionary behaviour to avoid mosquito-borne parasites.

Catnip, Nepeta cataria, contains nepetalactol:

Rubbing behavior transfers nepetalactol onto the faces and heads of respondents where it repels the mosquito, Aedes albopictus. Thus, self-anointing behavior helps to protect cats against mosquito bites.

Not quite the same but:

Tonic water contains quinine which is used to treat malaria and gin & tonic was invented by the East India Company to keep its colonising army safe in India.

Bet the covid vaccine would have been more popular if it also got you drunk. A lesson for next time around.

Ref.

Uenoyama, R., Miyazaki, T., Hurst, J. L., Beynon, R. J., Adachi, M., Murooka, T., Onoda, I., Miyazawa, Y., Katayama, R., Yamashita, T., Kaneko, S., Nishikawa, T., & Miyazaki, M. (2021). The characteristic response of domestic cats to plant iridoids allows them to gain chemical defense against mosquitoes. Science advances, 7(4), eabd9135.


More posts tagged: filtered-for (118).

Get your Claude Code muzak here

I suggested last week that Claude Code needs elevator music

Like, in the 3 minutes while you’re waiting for your coding agent to write your code, and meanwhile you’re gazing out the window and contemplating the mysteries of life, or your gas bill, maybe some gentle muzak could pass the time?

WELL.

Mike Davidson a.k.a. Mike Industries only went and built it.

Get Claude Muzak here (GitHub):

Elevator music for your Claude Code tasks! Automatically plays background music while Claude is working, because coding is better with a soundtrack.

Easy to install. Includes three pitch-perfect elevator music MP3s to get you started.

Wild. Wonderful.


Although:

Boris Smus adds a wrinkle: "I do my Claude codes at least two at a time. Is there polyphonic elevator music?"

YES PLEASE.

btw I’ve heard this a few times, Claude Code experts do indeed task up multiple claudes in parallel. Either working on different parts of the same app, or entirely different projects.

I would love to see what a custom UI to manage multiple claudes even looks like. Just, zero downtime, giving go-ahead permissions to one instance, pulling the throttle on another, instructing yet another…

I have a buddy who builds UI for traders at a global bank. The way he describes it, the traders are no longer doing individual stock trades. Instead they have a whole bunch of algorithms written by the quants running high-frequency trading on the market.

And the traders’ job is to sit there taking in the various signals and numbers, I imagine like Ozymandius bathing in the screens, "just me and the world," and instinctively steer the ship by ramping different HFT algorithms up and down.

The UI is to facilitate this fugue state crossed with a airplane pilot console situation.

One day driving our claude swarms will work like this.


So polyphonic Claude Codes?

Imagine that each claude instance plays just one track each and they come together…

I’m imagining Steve Reich’s Music for 18 Claudes.

Or to put it another way, something like video game music which is layered and looped and adaptive and I have discussed before (2020). In Red Dead Redemption,

the music reacts in real-time to the player’s actions … a more foreboding sound is required during moments of suspense. All of these loops have to segue into each other as events evolve on screen.

So the experience of playing RDR is that you’re galloping along at sunset in this beautiful American south-west landscape, and you notice the music quietly weave in an ominous new refrain… so you look out for what’s about to happen before it happens.

I want my polyphonic Claude Codes to intensify the minor key rhythms as the code tests fail and fail and fail again, before it is even escalated to me, drumbeats over the horizon, the distant rumble of thunder, approaching…


Also mentioned in that post:

Matthew Irvine Brown’s project Music for Shuffle (2011) which he composed for the wonderful belt-clip-mounted iPod Shuffle:

music specifically for shuffle mode – making use of randomness to make something more than the sum of its parts.

18 tracks sub 10 seconds each that can be played continuously in any order.

Like… Anthropic and OpenAI and Google, they must have arts programs, right? Are they commissioning for projects like this? Because they should.

The Phase Change

Last week I ran my first 10k.

It wasn't a race or anything. I left that evening planning to run a 5k, and then three miles later thought "what if I kept going?"1

I've been running for just over two years now. My goal was to run a mile, then three, then three at a pace faster than a power-walk. I wish I could say that I then found joy in running, but really I was just mad at myself for being so bad at it. Spite has always been my brightest muse.

Looking back, the thing I find most fascinating is what progress looked like. I couldn't tell you if I was physically progressing steadily, but for sure mental progress moved in discrete jumps. For a long time a 5k was me pushing myself, then suddenly a "phase change" happens and it becomes something I can just do on a run. Sometime in the future the 10k will feel the same way.

I've noticed this in a lot of other places. For every skill I know, my sense of myself follows a phase change. In every programming language I've ever learned, I lurch from "bad" to "okay" to "good". There's no "20% bad / 80% normal" in between. Pedagogical experts say that learning is about steadily building a mental model of the topic. It really feels like knowledge grows continuously, and then it suddenly becomes a model.

Now, for all the time I spend writing about software history and software theory and stuff, my actually job boils down to teaching formal methods. So I now have two questions about phase changes.

The first is "can we make phase changes happen faster?" I don't know if this is even possible! I've found lots of ways to teach concepts faster, cover more ground in less time, so that people know the material more quickly. But it doesn't seem to speed up that very first phase change from "this is foreign" to "this is normal". Maybe we can't really do that until we've spent enough effort on understanding.

So the second may be more productive: "can we motivate people to keep going until the phase change?" This is a lot easier to tackle! For example, removing frustration makes a huge difference. Getting a proper pair of running shoes made running so much less unpleasant, and made me more willing to keep putting in the hours. For teaching tech topics like formal methods, this often takes the form of better tooling and troubleshooting info.

We can also reduce the effort of investing time. This is also why I prefer to pair on writing specifications with clients and not just write specs for them. It's more work for them than fobbing it all off on me, but a whole lot less work than writing the spec by themselves, so they'll put in time and gradually develop skills on their own.

Question two seems much more fruitful than question one but also so much less interesting! Speeding up the phase change feels like the kind of dream that empires are built on. I know I'm going to keep obsessing over it, even if that leads nowhere.


  1. For non-running Americans: 5km is about 3.1 miles, and 10km is 6.2. 

Three ways formally verified code can go wrong in practice

New Logic for Programmers Release!

v0.12 is now available! This should be the last major content release. The next few months are going to be technical review, copyediting and polishing, with a hopeful 1.0 release in March. Full release notes here.

Cover of the boooooook

Three ways formally verified code can go wrong in practice

I run this small project called Let's Prove Leftpad, where people submit formally verified proofs of the eponymous meme. Recently I read Breaking “provably correct” Leftpad, which argued that most (if not all) of the provably correct leftpads have bugs! The lean proof, for example, should render leftpad('-', 9, אֳֽ֑) as ---------אֳֽ֑, but actually does ------אֳֽ֑.

You can read the article for a good explanation of why this goes wrong (Unicode). The actual problem is that correct can mean two different things, and this leads to confusion about how much formal methods can actually guarantee us. So I see this as a great opportunity to talk about the nature of proof, correctness, and how "correct" code can still have bugs.

What we talk about when we talk about correctness

In most of the real world, correct means "no bugs". Except "bugs" isn't a very clear category. A bug is anything that causes someone to say "this isn't working right, there's a bug." Being too slow is a bug, a typo is a bug, etc. "correct" is a little fuzzy.

In formal methods, "correct" has a very specific and precise meaning: the code conforms to a specification (or "spec"). The spec is a higher-level description of what is supposed the code's properties, usually something we can't just directly implement. Let's look at the most popular kind of proven specification:

-- Haskell
inc :: Int -&gt; Int
inc x = x + 1

The type signature Int -> Int is a specification! It corresponds to the logical statement all x in Int: inc(x) in Int. The Haskell type checker can automatically verify this for us. It cannot, however, verify properties like all x in Int: inc(x) > x. Formal verification is concerned with verifying arbitrary properties beyond what is (easily) automatically verifiable. Most often, this takes the form of proof. A human manually writes a proof that the code conforms to its specification, and the prover checks that the proof is correct.

Even if we have a proof of "correctness", though, there's a few different ways the code can still have bugs.

1. The proof is invalid

For some reason the proof doesn't actually show the code matches the specification. This is pretty common in pencil-and-paper verification, where the proof is checked by someone saying "yep looks good to me". It's much rarer when doing formal verification but it can still happen in a couple of specific cases:

  1. The theorem prover itself has a bug (in the code or introduced in the compiled binary) that makes it accept an incorrect proof. This is something people are really concerned about but it's so much rarer than every other way verified code goes wrong, so is only included for completeness.

  2. For convenience, most provers and FM languages have an "just accept this statement is true" feature. This helps you work on the big picture proof and fill in the details later. If you leave in a shortcut, and the compiler is configured to allow code-with-proof-assumptions to compile, then you can compile incorrect code that "passes the proof checker". You really should know better, though.

2. The properties are wrong

The horrible bug you had wasn't covered in the specification/came from some other module/etc

Galois

This code is provably correct:

inc :: Int -&gt; Int
inc x = x-1

The only specification I've given is the type signature Int -> Int. At no point did I put the property inc(x) > x in my specification, so it doesn't matter that it doesn't hold, the code is still "correct".

This is what "went wrong" with the leftpad proofs. They do not prove the property "leftpad(c, n, s) will take up either n spaces on the screen or however many characters s takes up (if more than n)". They prove the weaker property "len(leftpad(c, n, s)) == max(n, len(s)), for however you want to define len(string)". The second is a rough proxy for the first that works in most cases, but if someone really needs the former property they are liable to experience a bug.

Why don't we prove the stronger property? Sometimes it's because the code is meant to be used one way and people want to use it another way. This can lead to accusations that the developer is "misusing the provably correct code" but this should more often be seen as the verification expert failing to educate devs on was actually "proven".

Sometimes it's because the property is too hard to prove. "Outputs are visually aligned" is a proof about Unicode inputs, and the core Unicode specification is 1,243 pages long.

Sometimes it's because the property we want is too hard to express. How do you mathematically represent "people will perceive the output as being visually aligned"? Is it OS and font dependent? These two lines are exactly five characters but not visually aligned:

|||||

MMMMM

Or maybe they are aligned for you! I don't know, lots of people read email in a monospace font. "We can't express the property" comes up a lot when dealing with human/business concepts as opposed to mathematical/computational ones.

Finally, there's just the possibility of a brain fart. All of the proofs in Nearly All Binary Searches and Mergesorts are Broken are like this. They (informally) proved the correctness of binary search with unbound integers, forgetting that many programming languages use machine integers, where a large enough sum can overflow.

3. The assumptions are wrong

This is arguably the most important and most subtle source of bugs. Most properties we prove aren't "X is always true". They are "assuming Y is true, X is also true". Then if Y is not true, the proof no longer guarantees X. A good example of this is binary sort search, which only correctly finds elements assuming the input list is sorted. If the list is not sorted, it will not work correctly.

Formal verification adds two more wrinkles. One: sometimes we need assumptions to make the property valid, but we can also add them to make the proof easier. So the code can be bug-free even if the assumptions used to verify it no longer hold! Even if a leftpad implements visual alignment for all Unicode glyphs, it will be a lot easier to prove visual alignment for just ASCII strings and padding.

Two: we need make a lot of environmental assumptions that are outside our control. Does the algorithm return output or use the stack? Need to assume that there's sufficient memory to store stuff. Does it use any variables? Need to assume nothing is concurrently modifying them. Does it use an external service? Need to assume the vendor doesn't change the API or response formats. You need to assume the compiler worked correctly, the hardware isn't faulty, and the OS doesn't mess with things, etc. Any of these could change well after the code is proven and deployed, meaning formal verification can't be a one-and-done thing.

You don't actually have to assume most of these, but each assumption drop makes the proof harder and the properties you can prove more restricted. Remember, the code might still be bug-free even if the environmental assumptions change, so there's a tradeoff in time spent proving vs doing other useful work.

Another common source of "assumptions" is when verified code depends on unverified code. The Rust compiler can prove that safe code doesn't have a memory bug assuming unsafe code does not have one either, but depends on the human to confirm that assumption. Liquid Haskell is verifiable but can also call regular Haskell libraries, which are unverified. We need to assume that code is correct (in the "conforms to spec") sense, and if it's not, our proof can be "correct" and still cause bugs.


These boundaries are fuzzy. I wrote that the "binary search" bug happened because they proved the wrong property, but you can just as well argue that it was a broken assumption (that integers could not overflow). What really matters is having a clear understanding of what "this code is proven correct" actually tells you. Where can you use it safely? When should you worry? How do you communicate all of this to your teammates?

Good lord it's already Friday

Wat heeft Zeus gedaan, de afgelopen maan?

Wat heeft Zeus gedaan, de afgelopen maan?

We kijken even naar wat er is gebeurd tijdens de zomermaanden!

Het is zomer, wat kun je anders doen dan je in een kelder verstoppen voor de hitte en een hoop code schrijven? Dat is in ieder geval gebeurd! De highlights zijn:

Zeus Profile Images

Zeus heeft nu profielfotos!

Er is een centrale site om je profielfoto in te stellen: https://zpi.zeus.gent/

Je foto zal (langzaam maar zeker) zichtbaar zijn in meerdere Zeus-applicaties. Op dit moment kun je je foto al aanschouwen in ZAUTH en EVENTS. We zijn ook van plan om Tap hier naar over te schakelen, dus als je graag je oude Tapfoto wilt gebruiken, is nu het moment om deze over te zetten!

Documentatie

We zijn aan het werken aan Zeus-documentatie! Omdat nieuwe bestuursleden tijdens de eerste vergadering heel wat nieuwe dingen moeten onthouden (en voor de oudere die iets vergeten zijn) zijn we bezig met informatie te verzamelen in 1 centraal punt over hoe Zeus werkt. Ook informatie over sysadmin zaken en over dingen in de kelder zullen hier verzameld worden. Een WIP versie is hier te vinden. PRs, comments en bedenkingen zijn altijd welkom.

Omdat we toch geen docs kunnen schrijven zonder code te schrijven hebben we ook vlug onze eigen SSG geschreven.

EVENTS

Ons eigen event-management platform heeft een hele reeks updates gekregen. Het is bedoeld voor bestuur om events efficienter te plannen, te zorgen dat announcements op tijd worden gemaakt, en heeft een aantal andere superleuke features zoals automatisch powerpoints maken!

Keldermuzieksoftware

Rond muziek werden er ook leuke dingen gedaan: veel informatie over het huidig nummer is nu beschikbaar via onze MQTT broker, je kunt stemmen op nummers (in de kelder) via ZODOM en informatie krijgen over nummers via GUITAR. Ook hebben we nu terug een versterker! Hoezee!

Ledstrip

Er werd gesleuteld aan de ledstrip: er is nu een muziek progress bar en er werden verbeteringen aangebracht in de besturingscode.

Tap

Tap heeft enkele UX verbeteringen gekregen.

Bestuur

Op vlak van bestuurzaken, zullen we binnenkort (9 september) samenzitten om de events van dit semester te plannen en ons voor te bereiden op de start van het academiejaar!

Ook staan we in contact met Pelicano die graag samen met ons hun eigen urenloop willen organiseren. De details (en of het mogelijk is) zijn we nog aan het bekijken, maar in ieder geval binnenkort meer nieuws hierover!

We hebben ook onze eerste sponsor van dit jaar! De details liggen nog niet vast, maar we zullen samen een event organiseren.

We krijgen vaak de vraag van een bedrijf (of andere instantie) of we een job willen verspreiden onder onze leden. Aangezien we geen fan zijn van dat zomaar in ~general to posten, hebben we het ~jobs kanaal gemaakt, zodat dit voor iedereen opt-in is. Bedrijven die ons sponsoren kunnen hier ook enkele jobs plaatsen (maar dit zal steeds door ons gaan).

Hopelijk was het al een fijne zomer voor iedereen!

The Ascent to Sandia Crest

The Rotary Club will take immediate action on the Ellis ranch loop project. The Rotarians reached this decision at their weekly luncheon, held yesterday at the Albarado hotel.

The club's plan is not merely to give Albuquerque a good, short road to the Ellis ranch... They embrace the building of a seventy-mile scenic loop. 1

Many Western cities are defined, in part, by their mountains. Those moving from town to town often comment on the disorientation, the disruption, caused by a change in the city's relation to the peaks. If you have ever lived in a place with the mountains on the west, and then a place with the mountains on the east, you will know what I mean. We get used to living in the shadows of mountains.

One of the appeals of mountains, perhaps the source of their mysticism, is inaccessibility. Despite their close proximity to Albuquerque, steep slopes and difficult terrain kept the Sandias a world apart from the city, even to this day. Yet we have always been driven to climb, to ascent to the summit.

Humans climb mountains not only as a matter of individual achievement, but also as a matter of infrastructure. Whether the inaccessibility of mountain peaks is a good thing or a bad thing depends on the observer; and even the most challenging mountain ascents are sometimes developed to the scale of an industrial tourism operation.

And somewhere, in between, are the Sandias. Not technically part of the Rocky Mountains but roughly aligned with them, the Sandias lack a clear peak. Instead, the range is an elongated ridge, making up the entire eastern boundary of the city of Albuquerque and extending some distance further north into the Sandia Pueblo. The highest point is at 10,679', relatively modest for the mountain states---but still one of the most prominent in New Mexico, moreso even than the higher-elevation Mt. Taylor and Wheeler Peak.

Today, the Sandias are a major site for recreation. Tourists reach the upper parts of the mountain by one of two means: Sandia Crest Scenic Highway, which climbs the gentler eastern slope of the mountain to the crest itself; or the Sandia Peak Aerial Tramway, which makes a daring ascent from the edge of the city up the western side. Either provides access to an extensive network of trails, as do numerous points in the foothills and along the scenic highway.

Like bagging any peak, these access routes were hard-won. Their present contours---engineering achievements, twists and turns, scars on the landscape---reflect over one hundred years of ambition and conflict. Some of Albuquerque's most prominent figures left their marks on the mountain, as did everyday Burqueños, President Richard M. Nixon, and the slow grind of state and federal bureaucracy. The lay of the land today tells a story about the changing relationship of the city with the mountains, and more broadly of the American public with the wilderness.

It also explains why so many older documents and maps refer to the highway to the crest as a "loop," when it does no such thing. Here's the clickbait headline: Where does the Sandia Loop Highway go? The answer will surprise you.


Exploration of the Sandias by Spanish expeditions were motivated in large part by rumors of great mineral wealth. Certainly there are minerals in the mountains, and the Pueblo people had told the Spanish expeditions coming up the valley of gold and other precious metals, both in the mountains and in the plains beyond them. With the full perspective of history, it seems that these reports were less grounded in fact and more in a well-founded desire to get the Spanish to move on elsewhere. Supposedly several mines were established, but they left little mark on the landscape and their locations are now forgotten.

The Sandias, while rich in scenery, were not rich in gold. For centuries, they remained mostly undeveloped, besides Pueblo settlements in areas such as Tijeras canyon which were later abandoned in large part due to the increasing encroachment of settlers. Just a small number of people, mostly hunters, called the mountains their home. A few mining operations made their way into the mountains during the 19th century, perhaps repeating the mistakes of the Spanish several hundred years before.

It was these mining camps that brought the Ellis family to Albuquerque. They had run a ranch in the eastern plains of New Mexico, but were driven into the city by lack of water. Patriarch George Ellis found work for a fruit distributor, delivering produce to the mining camps in the mountains. Perhaps missing their days out in the country, George took his time as he traveled the canyons and ridges between camps. His wanderings were rewarded when he found Las Huertas Canyon, a narrow slot with a creek, protection from the wind, and what George Ellis himself called "extraordinary beauty."

The Ellis family moved into the canyon as early as 1893, but in 1905 George Ellis filed a land patent for what was, as an outcome of the Treaty of Guadalupe Hidalgo, frontier up for the taking. Ellis Ranch, as it was called, was indeed extraordinary: one of the most pleasant places in the mountains. But, then, the Ellis family were rather extraordinary as well. Life on the mountainside at 7,600 feet wasn't easy, but they kept cattle, chickens, and a modest farm.

Charlotte Ellis, George's daughter, had not yet a year of education at UNM when they moved out of town and up the slopes on the east side of the ridge. Still, she built a notable career as an amateur botanist. It is because of her efforts that Ellis ranch is often described in terms of its plant life---Charlotte collected specimens of hundreds of plant species, some of them new discoveries, and the ranch is said to have contained examples of every flowering plant known to exist in the mountains.

Plants were not Charlotte's only interest, though. She was apparently a skier, in a time well before skiing was known as a recreational pastime. A photo taken in 1896 shows her in a snow-covered landscape, facing away from the camera. She's looking down the slope, wearing a full dress and a pair of wooden skis. Her pose suggests a certain resolve, the kind you would need to spend the winter high above the nearest towns. It is often captioned as the first photograph of a person skiing in New Mexico. It may be the first photo of a woman skiing at all.

It also foreshadows the tremendous impact that skiing would have on the mountains and on Albuquerque. We don't know exactly what she was thinking, but what we do know about her suggests that she viewed the mountains much as we do today: for their beauty, their diversity, and for their thrill.

George Ellis had perhaps filed his land patent just in time. The Forest Service had existed for some years but was reorganized under its current name, and as part of the Department of the Interior, in 1905. The forests of the Sandia Mountains have a complicated legal history owing to the interaction of Spanish and Mexican land grants with the Treaty of Guadalupe Hidalgo at the conclusion of the Mexican-American war. You could say that the matter of the forest's legal ownership was not entirely settled until 2004.

None of that stopped President Roosevelt declaring the Sandias part of the Manzano Forest Reserves in 1906. Shortly after, the Forest Reserves became a National Forest and a ranger station was under construction near Tijeras. Paul Ellis, the son of the family, was soon hired on as one of the national forest's first staff; the Ellis Ranch one of the few private parcels within the federal boundaries.


The Forest Service has always served multiple purposes: "Land of Many Uses," as Forest Service signs still proclaim. Principle concerns of the forest service included lumber, cattle and sheep grazing, mining, and hunting. These uses were already widespread at the dawn of the modern Forest Service, so in practice, much of the role of early Forest Rangers was to bring forest users under control. Extensive regulations and rules were backed by a small set of rangers of the "Stetson era," rugged types not unlike the Ellises.

What we don't always picture as a part of that era was recreation. The "crunchy granola" brand of wilderness recreation most widely promoted today is indeed a more modern phenomenon, but forests have been viewed as recreational assets since the dawn of forest management---the form of that recreation has just changed, largely in response to new concepts of environmental stewardship that emphasize preservation over "improvement."

In 1913, George Ellis passed away. After a small ceremony, he was buried in a cemetery established next to his ranch. With George gone, the rest of the Ellis family began to scatter, moving into towns, closer to work. This left the question of the ranch.

By the time of George's death, Ellis Ranch had already become renowned for hospitality. The newspapers of the day recounted some of Albuquerque's finest taking vacations at the Ellis Ranch, a trip that was something of an adventure in the 1900s. It took a few hours, in fine conditions, to make the way up a rough trail to the Ellis homestead. One local businessman, known to newspaper readers as "the bee man," took some pride in having gotten a Ford automobile to within a mile of the site. Even so, the journey was well-agreed to be worthwhile. Las Huertas canyon was beautiful, and the Ellises known to offer fresh produce, spring water, and mountain air---a commodity in increasingly high demand as tuberculosis swept the country.

The reputation of the Ellis ranch for taking visitors explains the Albuquerque Tribune's matter-of-fact statement that the Ellis ranch would likely become a tourist resort, despite the lack of known buyers. At that time, the Forest Service was already in an infrastructure-building mood. La Madera, then one of the larger settlements in the east mountains, had telephone service and the Forest Service was extending the line to the ranch.

Shortly after, the buyers emerged, Raymond Stamm (uncle of the Stamm of Bradbury Stamm Construction) and Jack Sheehan of Albuquerque.

These young men propose to fit up the ranch as a mountain resort, the house being splendidly adapted to a mountain club house, with fine fishing, hunting, and camping facilities all about. 2

One challenge for this ambitious pair was, of course, the road: for the ranch to be the resort they imagined it needed an easier means of access. It was estimated that this effort would only require a few hundred dollars, reworking about three miles of trail between the ranch and a better road to Tijeras.

Even by 1913, an improved road to Ellis Ranch was a recurring theme in the newspapers. George Ellis, before his death, had estimated $500 to improve the road between the ranch and Placitas to passable condition. After his death, the estimate on this longer segment reached the thousands.

Some of the problem was, of course, hubris: the Albuquerque boosters of the day could look at three miles of faint road climbing up and down cabins and call it a few days of work. When these projects went underway, they always took longer and cost more than expected. To be fair, though, they were also fighting a living foe. Thunderstorm downpours and winter snow repeatedly washed out the road in the steeper sections. Reading of road improvement efforts, you get repeated deja vu. They are, in fact, announcing plans to fix the same section of road over and over again---it having always washed out the previous year.

The Stamm-Sheehan resort seems to have been popular, even as the road stayed rough. In September of 1914, the faculty of UNM held what we might now call an "offsite" there. Indeed, during 1914 and 1915 it seemed that practically everybody who was anybody was vacationing at Ellis Ranch. All the while, though, they were still struggling to get there... a situation that the Rotarians resolved to correct.


The road to Ellis Ranch has been known by many names and promoted by many people. The Rotary Club was a powerful force in the Albuquerque of 1916, though, and their proposal seems to have set the pattern for those that followed. The Ellis Ranch Loop Road, as they called it, would span 70 miles from Albuquerque to Albuquerque. The route went through the Tijeras pass, turned north to San Antonito 3 and passed Ellis Ranch on the way to La Madera and then Placitas. From there, it went nearly to Bernalillo, meeting the Santa Fe-Albuquerque road that would soon become part of Route 66---until 1937, at least.

A big argument for the loop, besides the fact that it makes the Ellis summer resort more easily accessible, is its scenery.

At the time of the Rotary Club proposal, the region's road map was in some ways familiar, and in some ways rather different. Many of the today's routes were already known but their condition was widely variable. The trip from Albuquerque to Tijeras, which would become the ultimate Route 66 alignment, was already in place. It was shorter in 1913, though, not extending much further east, and far from a highway. There was a rough road north to San Antonito, and there had once been a road further north from there to La Madera, but by 1916 it had been closed where it entered a private land grant.

This is probably the biggest difference from our modern road network: the route through Tijeras Canyon was not used for much other than access to Tijeras, and for much of the era covered in this article the preferred way to negotiate the Sandias and Manzanos was to go around them entirely. This is a good part of the reason for Route 66's original north-south alignment through the Rio Grande Valley, one that added significant mileage but avoided the mountain pass that was slow going in good weather and could be outright impassable in winter.

Similarly, from the north, there was an existing road from Bernalillo through Placitas and into the East Mountains. It was, in fact, the main route used by many of the residents there. It was not an easy route, though, long and prone to washouts.

And while these roads came close to forming a loop, they didn't. A section of about four miles was missing, right in the middle. Ellis Ranch was, of course, somewhere in that in-between.

So, the Rotary Club project was mostly the improvement of existing roads, along with the construction of new road in a couple of areas required to complete the loop around the Sandia range. The Rotarians estimated the cost at $5,000, and the Forest Service offered to pitch in.

Now, we must consider that $5,000 was a significant amount of money in 1916, something around $150,000 today. This inflation calculation probably underestimates the ask, because extensive investment in highway construction was not yet the pillar of American culture that it is today. Still, public enthusiasm for the project was high, and by the end of 1916 Bernalillo County had enacted a tax assessment to to support the road. The state legislature joined on, and with the Forest Service's share the project was fully funded. Construction seems to have began in earnest in 1917, on improvements to the road through Tijeras Pass.

Efforts on the loop road coincided, of course, with work to develop Ellis Ranch as a summer resort. Starting in 1917, regular newspaper ads appear for vacation cabin rentals at Ellis Ranch, with instructions to telephone Hugh Cooper. This High Cooper was Hugh Cooper Jr., son of the Hugh Cooper who was pastor of the First Presbyterian Church. Much like George Ellis, Rev. Cooper was a well-known figure around Albuquerque. Given that he officiated the wedding of at least one of Ellis's children, he was presumably friendly with the Ellises as well. The newspapers are not exactly clear on how it came to happen, but Rev. Cooper Sr would soon become much more involved in the ranch: in 1923, he and his sister went in together to buy it.

"We hope to keep this beauty spot of nature, so near our city, from falling into the hands of sports, who will advertise dance pavilions and entertainments of a sordid nature. If our plans mature we hope to make it a place where parents can go with their children and find recreation of an exalted character." 4

Rev. Cooper's motivations were, he assured the newspaper, godly rather than pecuniary. The phrase "exalted recreation" was singled out for the headline and is certainly one to remember as you explore the Sandias. Cooper was not, though, immune to the sort of boosterism that so animated the Rotary Club.

"We hope, with the help of the good people of Albuquerque, to make the famous Ellis Ranch a second Estes park."


The 1910s and '20s were boom years in Albuquerque, with a confluence of the railroad shops, the sawmill, and tuberculosis bringing new residents at a fast clip. The population more than doubled from 1900 to 1920, and with the newcomers came urban development. Subdivisions went under construction west of New Town, a skyscraper rose, and car ownership proliferated. The same period, for many of the same reasons, saw the genesis of American tourism. Trains brought visitors from out of town, and cars brought visitors and residents alike out into the surroundings. Boosters, from the city council to the chamber of commerce to a half dozen clubs, all had plans to make the best of it.

There was, to be sure, a certain aspect of keeping up with the joneses. Colorado was building a strong reputation as a tourist destination, and Denver as a beautiful and prosperous city. Arizona and Nevada had not yet gained populous cities, and west Texas was no less sparse than it is now. For the businessmen of Albuquerque, Colorado was the competition... and Colorado was building summer resorts.

The concept of the "summer resort" is mostly forgotten today, but in the 1920s there was little air conditioning and cities were gaining a reputation as crowded and polluted. The spread of Tuberculosis had brought particular attention to "clean, fresh air," and nothing was quite as clean and fresh as a summer day in the mountains. Estes Park, a town in the northern Colorado rockies, had multiple fine hotels by 1910 and attracted the tourism to match.

Further, in 1910, Denver's own boosters (groups like the Real Estate Exchange and the Denver Motor Club) began to acquire land for the Denver Mountain Parks. The beautiful Genesee Park was under construction in 1920, with a lodge and campground. Daniel Park had opened as a scenic drive (or "Auto View"). Plans for Mount Blue Sky Scenic Byway, the highest paved road in North America, had been finalized and announced to great fanfare.

To the modern conservationist it seems odd that these plans were so focused on roads, but the automobile was new and exciting and in the West had almost singlehandedly popularized the idea of "wilderness recreation." The mountains were no longer a destination only for the hardiest adventurers, they were also for city dwellers, out for a drive and a picnic. The ideal of natural preservation was a long, winding road with regular scenic pullouts. If the road could access a summer resort, all the better. Denver was doing it, and so Albuquerque would too.

Behind the scenes, the wheels of government had been turning. The Forest Service approved construction through the mountains in 1919, which would "make accessible some of the most beautiful mountain scenery in the state, in addition to bringing within easy reach sites for summer homes and temporary camping places." At this point in its history, the Forest Service routinely leased space for the construction of private homes, and a small number were established in the national forest as the road progressed inwards.

Bernalillo County had raised $12,500, the Forest Service committed $25,000, and the state was lobbied for another $12,500. A Forest Service engineer, speaking to the County Commission, noted that the road would provide access to 30 million board feet of lumber. The Forest Service said that construction could be finished by the end of the summer, and in September, the section from Placitas to Ellis Ranch opened to drivers. The road from Ellis Ranch to San Antonito had been surveyed, but work had not yet started.


We should consider again how this historic project relates to the modern roads, because there is a surprise. Today, the primary access from Albuquerque to the eastern slopes of the Sandias is via I-40 to Tijeras, and then NM-14 north through Cedar Crest and to then to San Antonito. There, NM-536 branches to the west, becoming Sandia Crest Scenic Highway.

NM-536 turns into Sandia Crest Scenic Highway, which is actually not a numbered highway at all, at a 90-degree turn by the Balsam Glade picnic area. From the picnic area, a much smaller, rougher dirt road descends to the north: NM-165. Today, NM-165 is a rugged and adventurous route to Placitas, impassable in winter and unfriendly to passenger vehicles.

And yet, in the summer of 1919, it was this route that was made a "first class road" to Ellis Ranch. The former site of Ellis Ranch now has a surprisingly well-paved parking lot off of the dirt NM-165, signed as "Las Huertas Canyon," and a trail continues up the creek through much of what was the summer resort. It took about an hour and a half, the Journal reported, to drive from Albuquerque to Ellis Ranch by this route around the north side. The travel time probably isn't all that much faster today, given the slow going on the dirt road.

In 1920, Bernalillo County ran into some trouble on the project funding. The county's plan to fund much of the project through bonds on the tax assessment was, it turns out, in violation of state law. For accounting reasons, the total sum of the bonds would have to be much lower. The county was effectively out of the project, at least as far as money was concerned.

The Forest Service worked according to its own plans. Construction started on a forest road, what is now NM-536, that would later be home to Tinkertown. In summer of 1922, it reached Sulphur Spring, the first of many recreation sites on its long trip up. Progress was brisk: the newspaper reported that 2.5 miles had been completed, then a few more, then eight. The forest road seems to have gone by different names at different times, and perhaps just depending on who was talking, but as 1922 came to a close the Forest Service named it for the canyon that it followed from San Antonito: Tejano Road.

Tejano Road ended about where the bottom of the ski area is found today, some four miles short of Ellis Ranch. Much of the Ellis Ranch Loop was complete, but it wasn't a loop. Plans had been completed for the final stretch, but it was unclear who would pay for the work---and when. The Journal printed an appeals to finish the project, several times mentioning the importance of a "sky line road."

The sky line road was something of a fad in those days. California and Oregon both had them, and plans were underway in many other parts of the west. A skyline road was what we might now call a scenic byway, but with a particular emphasis on views. Many followed a ridge, right down the center, so that drivers could get a view down both sides. Extensive earth-moving and forestry were sometimes involved, modifying the environment to create these views.

The West had a lead on the idea: it wasn't until 1925 that planning started on the most famous, Skyline Drive of Shenandoah National Park. The 100-mile highway was viewed as a wilderness adventure, but was about as far from our modern conception of wilderness as possible. Extensive clearing, of both trees and residents, were required to manicure its vistas. It was seen as one of the finest examples of American recreation, a model to be followed.


In July of 1923, the Ellis Ranch Loop was considered complete except for the four-mile segment between Tejano Road and Ellis Ranch Road and, oddly enough, the reworking of the road from Placitas to Ellis Ranch---the very one that had been completed in 1922. Road construction in the mountains was a fight against both funding and nature. The road seems to have been minimally improved, to keep the fast schedule and tight budget. During winter, snowfall and then snowmelt would wash parts of the road out. Throughout the 1920s and 1930s, the road was constantly being repaired and reworked. This probably explains why NM-165, once the principle route up the Sandias, is now so minimally maintained: the state and the Forest Service gave up.

During 1924, the Forest Service closed two miles of the gap and had the other two underway. In a turn of luck for the Albuquerque boosters, the Forest Service also offered to pick up work on the road to Placitas. The entire Ellis Ranch Loop project had become a Forest Service initiative, and they did what they set out to do.

In September 24th of 1924, an Albuquerque man named Arthur Ford set out through Tijeras Canyon with a friend, Mrs. Karskadon. They left Albuquerque at 10 am, finding that the new section connecting Ellis Ranch was not, officially, open; Ford was little discouraged and simply moved the barriers aside. At 11:45 they reached Ellis Ranch. After lunch, they continued northwards, to Placitas, through Bernalillo, and back into town. Ford's odometer measured the trip at 68 miles. The Ellis Ranch Loop was complete.

Imagine a circle approximately 20 miles in diameter.

Then imagine that this circle encloses some of the most beautiful mountain, valley, and mesa scenery in the world.

The city has always existed in tension with the mountains. The Ellis Family, who most famously opened the mountain to visitors, also oversaw a sort of closure to nature. George Ellis was the last person to see a Grizzly Bear in the Sandias. He came across the bear in 1905; he shot it.

From 1916 to 1924, Albuquerque business leaders admired the mountains for their beauty but lamented their inaccessibility. The Sandias could attract visitors from around the nation, they argued, if only you could get to them. Charlotte Ellis would hoof it on foot and on skis, but, then, she was Charlotte.

Then imagine that this circle is bounded by a highway that is traversible every day in the year.

Imagine, within the 70-mile circumference of this circle, near its eastern rim, a cluster of summer houses, supplied with water, light, and other necessaries. Imagine, at various spots within the circle, fine picnic and camping grounds, where people from the hot city may go for a day's outing.

We have always been driven to climb. As soon as it was possible to drive the full loop, to reach Ellis Ranch on and easy road, to take in the "Million Dollar Playground" of the Sandias, the Kiwanis Club formed a committee of business leaders and Forest Service representatives to consider the future of Sandia development.

It was "not only practicable, but highly necessary," they said, to take the next logical step. Ellis Ranch, at 7,500 feet, was only halfway from the city to the crest. A road all the way to the top---from Ellis Ranch to the crest---would complete their vision of the Sandias. "An effort will be made to have work begun as soon as possible."

You may think you are dreaming. And perhaps you are. But some day you will wake up and find the dream come true. 5

The Forest Service cleared six miles of steep, tight switchbacks, from the Balsam Glade area just above Ellis Ranch to the crest itself, over 10,500 feet. The New Mexico Highway Department, bolstered by the completion of Route 66, laid a gravel roadbed. Automobiles had become more common, driving a more popular pastime. It didn't take the adventurous Arthur Ford and Mrs. Karskadon to inaugurate the crest spur. On October 10th, 1927, Highway Department officials at the crest counted 110 cars.

Albuquerque had summited the mountain---but the greatest controversy was still to come.

  1. Albuquerque Journal, 1916-08-04 p 6

  2. Albuquerque Tribune, 1916-03-23 p 3

  3. The names of the towns at this location are a bit confusing historically, so I am simply using San Antonito to refer to the spot that is currently occupied by Sandia Park and San Antonito. Sandia Park appears to be a renaming of a town that was originally called San Antonio (i.e. the big one), likely due to the presence of another San Antonio, New Mexico in Socorro County.

  4. Albuquerque Journal, 1923-08-11 p 10

  5. Albuquerque Journal, 1923-07-22 p 3

T-carrier

Few aspects of commercial telecommunications have quite the allure of the T-carrier. Well, to me, at least, but then I have very specific interests.

T-carrier has this odd, enduring influence on discussion of internet connections. I remember that for years, some video game installers (perhaps those using Gamespy?) used to ask what kind of internet service you had, with T1 as the "highest" option. The Steam Hardware Survey included T1 among the options for a long time. This was odd, in a lot of ways. It set T1 as sort of the "gold standard" in the minds of gamers, but residential internet service over T1 would have been very rare. Besides, even by the standards of the 2000s T1 service was actually pretty slow.

Still, T1 involved a healthy life as an important "common denominator" in internet connectivity. As a regulated telephone service, it was expensive, but available pretty much anywhere. It also provided a very high standard for reliability and latency, beyond many of the last-mile media we use today.

Telephone Carriers

We think of telephone calls as being carried over a pair of wires. In the early days of the telephone system, it wasn't all that far off to imagine a phone call as a single long circuit of two wires that extended all the way from your telephone to the phone you had called. This was the most naive and straightforward version of circuit switching: connections were established by creating a circuit.

The era of this extremely literal form of circuit switching did not last as long as you might think. First, we have to remember that two-wire telephone circuits don't actually work that well. Low install cost and convenience means that they are the norm between a telephone exchange and its local callers, but for long-distance carriage over the phone network, you get far better results by splitting the "talk" and "listen" sides into two separate pairs. This is called a four-wire telephone circuit, and while you will rarely see four-wire service at a customer's premises, almost all connectivity between telephone exchanges (and even in the internals of the telephone exchange itself) has been four-wire since the dawn of long-distance service.

Four-wire circuits only exacerbated an obvious scaling problem: in the long distance network, you have connections called a toll leads between two exchanges. In a very simple case, two towns might have a toll lead between them. For simple four-wire telephone lines, that toll lead needs four wires for each channel. If it has four wires, only one phone call can take place between the towns at a time. If it has eight wires, two telephone calls. This got very expensive very fast, considering that even heavily-built four-crossarm open wire routes might only have a capacity for eight simultaneous calls.

For obvious reasons, research in the telephone industry focused heavily on ways to combine more calls onto fewer wires. Some simple electrical techniques could be used, like phantoms that combined two underlying pairs into a single additional "phantom" pair for a 50% increase in capacity. You could extend this technique to create more channels, with a noticeable loss in quality.

By the 1920s, the Bell System relied on a technique that we would later call frequency division multiplexing (FDM). By modulating a phone call onto a higher-frequency carrier, you can put it over the same wires as other phone calls modulated onto different frequencies. The devices that actually did this combined multiple channels onto a single line, so they were known as channel banks. The actual formats they used over the wire, since they originally consisted of modulation onto a carrier, came to be known themselves as carriers. AT&T identified the carriers they developed with simple sequential letters. In the 1940s, the state of the art was up to J- and K-carrier, which allowed 16 channels on a four-wire circuit (over open wire and multipair cable, respectively). A four-crossarm open-wire circuit, with sixteen pairs, could support 256 unidirectional circuits for 128 channels---or simultaneous phone calls.

FDM carriers reached their apex with the coaxial-cable based L-carrier and and microwave radio TH and TD carriers 1, which combined channels into groups, groups into supergroups, and supergroups into mastergroups for total capacities that reached into thousands of channels. Such huge FDM groups required very large bandwidths, though, which could not be achieved over copper pairs.

In the 1950s, rapidly developing digital electronics lead to the development of digital carriers. These carriers relied on a technique called Pulse-Code Modulation or PCM. PCM has sort of an odd name due to the history; it dates so far back into the history of digital communications that it was named before the terminology was well-established. "Pulse-code modulation" really just means "quantizing an analog signal to numbers and sending the numbers," which is now intuitive and obvious, but was an exciting new development of the 1940s.

PCM had a lot of potential for carrying phone calls, because the relaxed needs of voice transmission meant that calls could be digitized to fairly low-rate streams (8kHz by 8 bits) and engineers could see that there was a huge variety of possible techniques for combining and transporting digital signals. The best thing, though, was that digital signals could reliably be transmitted as exact copies rather than analog recreations. That meant that PCM telephone calls could pass through a complex network, with many mux/demux steps, without the reduction in quality that analog FDM carriers caused.

Even better, analog channel banks were large systems with a lot of sensitive analog components. They were expensive to build, required controlled environments, and were still subject to drift that required regular maintenance by technicians. Digital technology involved far fewer sensitive analog components, promising cheaper equipment with less maintenance. Digital was clearly the future.

The question, though, was how to best carry these digital signals over the wiring that made up much of the telephone system: copper pairs. In the late 1950s, Bell Laboratories developed T-carrier as the answer.

T-Carrier

T-carrier is a specification for transporting a stream of bits over copper pairs. The plural here is important: because T-carrier supported multiple channels, it was developed as a trunk technology, used for communication between telephone exchanges rather than between a telephone exchange and a customer. So, like other carriers used for trunks, T-carrier was four-wire, requiring two pairs to carry bidirectional signals.

T-carrier operates at 1.544Mbps, and that's about all you can say about it. The logical protocol used over T-carrier, the actual application of those bits, is determined by a separate protocol called the digital signal or DS. You can roughly think of this as a layer model, with DS running on top of T.

Here, we need to address the elephant in the room: the numbers. The numbers follow a convention used throughout AT&T-developed digital standards that is most clearly illustrated with DS. A DS0, by analogy to DS raised to the zeroeth power, is one single telephone channel expressed as a PCM digital signal. Since a telephone call is conveyed as 8-bit samples at 8kHz, a DS0 is 64kbps of data.

A DS1 is a combination of 24 DS0s, for 1.544Mbps.

A DS2 is a combination of 4 DS1s, for 96 channels or 6.312Mbps.

A DS3 is a combination of 7 DS2s, for 672 channels or 44.736Mbps.

Each level of the DS hierarchy is a TDM combination of several instances of the level below. The numbers are kind of odd, though, right? 24, 4, 7, it has the upsetting feeling of a gallon being four quarts each of which is two pints.

The DS system was developed in close parallel with the carriers actually used to convey the signal, so the short explanation for this odd scheme is that a DS1 is the number of DS0s that fit onto a T1 line, and a DS2 is the number of DS1s that fit onto a T2 line. The numbers are thus parallel: DS1 over T1, DS2 over T2, DS3 over T3. The distinction between T and DS is thus not always that important, and the terms do get used interchangeably.

But still, why 24?

Well, it's said that the number of channels on a T1 was just determined empirically. The 64kbps rate of a DS0 was fixed by the 8b x 8kHz digital format. A test T1 installation was built using a typical in-place copper telephone cable, and the number of channels was increased until it no longer functioned reliably. 24 channels was the magic number, the most achieved without unacceptable errors.

T-Carrier Infrastructure

T1 was designed for operation over a "typical" telephone cable trunk. In the 1950s, this meant a twisted-pair telephone cable installed in 6,600 foot sections with a loading coil at the junction of each section. A loading coil was essentially a big inductor hooked up to a telephone line at regular intervals to compensate for the capacitance of the line---long telephone cables, even four-wire, needed loading coils at regular intervals or the higher frequencies of speech would be lost. Loading coils also had disadvantages, though, in that they imposed a pretty sharp maximum frequency cutoff on the line. High-speed digital signaling needed to operate at those high frequencies, so T1 was designed to fit into existing long cables by replacing the loading coils with repeaters.

That means that T1 required a repeater very 6,600 feet. This repeaters were fairly small devices, often enclosed in pressurized cans to keep water out. 6,600 feet might sound pretty frequent, but because of the loading coil (and splice box) requirements trunk lines usually had underground vaults or equipment cabinets at that interval anyway.

Over time, the 6,600 foot interval became increasingly inconvenient. This was especially true as end-users started to order T1 service, requiring that T1 be supported on local loops that were often longer than 6,600 feet. Rather than installing new repeaters out in the field, it became a widespread practice to deliver T1 over a version of DSL called HDSL. HDSL is older and slower than the newer ADSL and VDSL protocols, and requires four wires, but it was fast enough to carry a DS1 signal and could cover a much longer span than traditional T-carrier. HDSL used the voice frequency band and thus could not coexist with voice calls like ADSL or VDSL, but this had the upside that it "fully controlled" the telephone line and could use measures like continuous monitoring (using a mark signal when there was no traffic) to maintain high reliability.

For the era of internet-over-T1, then, it was far more likely that a given customer actually had an HDSL connection that was converted to T1 at the customer premises by a device called a "smart jack." This pattern of the telco providing a regulated T1 service over a different media of their choice, and converting it at the customer premises, is still common today. T1s ordered later on may have actually been delivered via fiber with a similar on-premises media converter, depending on what was most convenient to the telco.

T1 is typically carried over telephone cable with 8P8C modular connectors, much like Ethernet. It requires two pairs, much like the two-line telephone wiring commonly installed in buildings. However, like most digital carriers, T-carrier is more particular about the wiring than conventional telephone. T1 wiring must be twisted-pair, and it is polarity sensitive.

DS1 Protocol

The DS1 protocol defines the carriage of 24 64kbps channels over a T1 interface. This basically amounts to TDM-muxing the 24 channels by looping over them sending one byte at a time, but there are a surprising number of nuances and details.

Early versions of DS1 only actually carried 7 bits for each sample, which was sufficient for a telephone call when companding was used to recover some of the dynamic range. The eighth bit was used for framing. T-carrier is a completely synchronous system, requiring that all of the equipment on the line have perfectly synchronized "frame clocks" to understand what bits belong to which logical channels. The framing mechanism provided a synchronization signal to achieve this perfect coordination. Later improvements in the framing protocol allowed for the use of all eight bits in some or even all of the samples. This gets to be a complicated and tangled story with many caveats, so I am going to leave it out here or this article would get a lot longer and probably contain a bunch more mistakes.

The various combinations of technologies and conventions used at different points get confusing. If you are curious, look into "robbed bit signaling," an artifact of the transition of where framing and control signals in T1 were placed that was, for some reason, a pet topic of one of my college professors. I think we spent more time on robbed-bit signaling than we did on all of MPLS, which is way cooler. Anyway, the point of this is to understand the protocol overhead involved: T1 operates at 1.544Mbps, but at least one 8-bit "sample" must be used for framing purposes, leaving 1.536Mbps of actual payload. The payload may be further reduced by other framing/signaling overhead, depending on exactly how the channel bank is configured. Most of these issue are specific to carrying telephone calls (and their related signaling); "internet service" T1 lines typically used a maximally-efficient configuration.

The Internet

So far we have pretty much only talked about telephone calls, because that's what T-carrier was developed for. By the 1980s, though, the computer industry was producing a lot of new applications for high-speed digital connections. T1 was widely available, and in many cases a tariffed/regulated service, so it was one of the most widely available high-speed data connections. Especially in the very early days of the internet, was often the only option.

Into the 1990s, T1 was one of the dominant options for commercial internet access. It was rarely seen in homes, though, as it was quite expensive. Keep in mind that, from the perspective of the phone company, a T1 line was basically 24 phone lines. They charged accordingly.

To obtain internet service, you would order internet service either from the telco itself or from an independent provider that then ordered a connection from your premises to their point of presence on an open access basis. In this case you were effectively paying two bills, one to the telco for the T1 line and the other to the independent provider for internet connectivity... but the total was still often more affordable than the telco's high rates for internet services.

Because of the popularity of T1 for internet access, routers with T1 interfaces were readily available. Well, really, the wide variety of media used for data connections before Ethernet became such a common standard means that many routers of the era took interchangeable modules, and T1 was one of the modules you could get for them.

In implementation, a T1 line was basically a fast serial line from your equipment to your ISP's equipment. What actually ran over that serial line was up to the ISP, and there were multiple options. The most classical would be frame relay, an X.25-derived protocol mostly associated with ISDN. PPP was also a fairly common option, as with consumer DSL, and more exotic protocols existed for specialized purposes or ISPs that were just a little weird.

When the internet was new, 1.5Mbps T1 was really very fast---the NSFNET backbone was lauded for its speed when it went online as an all-T1 network in 1991. Of course, today, a 1.5Mbps "backbone" internet connection is pretty laughable. Even as the '90s progressed, 1.5Mbps started to feel tight.

One of the things I find odd about the role of T1 in the history of internet access is that the era when a T1 was "blazing fast" was really pretty short. By 2000, when online gaming for example was taking off, both DSL and cable offered significantly better downstream speeds than T1. However, the nature of T1 as a circuit-switched, telephone-engineered TDM protocol made it very reliable and low-latency, properties that most faster internet media performed poorly on (early DSL notoriously so). Multiplayer gaming would likely have been a better experience on T1 than on a DSL connection offering multiples of the theoretical bandwidth.

A faster T1

Of course, there were options to speed up T-carrier. The most obvious is to combine multiple T1 connections, which was indeed a common practice. Later T1 interfaces were often supplied in multiple for that reason. MLPPP is a variant of the PPP protocol intended for combining the bandwidth of multiple links, referred to in the telephone industry as bonding.

But there were also higher levels in the hierarchy. Remember DS2 and DS3? Well, in practice, T2 wasn't really used. It was far more common to bond multiple T1 connections to reach the equivalent speed of a T2. 44.736Mbps T3 did find use, though. The catch is that T3 required specialized cabling (coaxial pairs!) and had a fairly short range, so it was usually not practical between a telephone exchange and a business.

Fortunately, by the time these larger bandwidths were in demand, fiber optic technology had become well-established. The telephone industry primarily used SONET, a fiber media over which Synchronous Digital Hierarchy (SDH) channels were carried. SONET comes in formats identified by OC (Optical Carrier) numbers in a way very similar to T-carrier numbers. An OC-1 is 51.840Mbps, already faster than T3/DS3. So, in practice, DS3 service was pretty much always provided by a media converter from an OC-1 SONET ring. As bandwidth demands further increased, businesses were much more likely to directly order SONET service rather than T-carrier. SONET was available into the multiple Gbps and enjoyed a long life as a popular internet carrier.

Of course, as the internet proliferated, so too did the stack of network media designed specifically for opportunistic packet-switching computer networks. Chief among them was Ethernet. These protocols have now overtaken traditional telephony protocols in most internet applications, so SONET as well is now mostly out of the picture. On the upside, Ethernet networks are generally more cost-effective on a bandwidth basis and allow you to obtain much faster service than you would be able to afford over the telephone network. On the downside, Ethernet networks have no circuit switching or traffic engineering, and so they do not provide the quality of service that the telephone network did. This means more jitter and less predictability of the available bandwidth, an enduring challenge for real-time media applications.

Miscellaneous

One of the confusing things about T1 to modern readers is its interaction with ISDN. ISDN was a later development that introduced a lot more standardization to digital communications over the telephone network, but it incorporated and reused existing digital technology including T-carrier. In the world of ISDN, a "DS0" or single telephone channel is called a basic rate interface (BRI), while the 24-channel T1 bandwidth is called a primary rate interface (PRI). Many T1 connections in the 1990s were actually ISDN PRIs.

The difference between the two is in the fine details: many of the details related to framing and control signals that were, shall we say, "loosey-goosey" with T-carrier are completely standardized under ISDN. An ISDN PRI always consists of 23 bearer channels ("payload" channels) and one control channel, and the framing is always the same. Since there is a dedicated control channel, there's no need to do weird things with the bits in the bearer channels, and so the overhead is standard across all ISDN PRIs.

In practice, the difference between T1 and ISDN PRI was usually just a matter of configuring your router's interface appropriately. Because of the curious details of the regulation and tariff processes, one was sometimes cheaper than the other, and in general the choice of whether to use a "T1" or a "PRI" was often somewhat arbitrary. It's even possible to use some T1 channels in the traditional way and others as ISDN.

While T1 is now mostly forgotten, some parts of its design live on. DSL and T1 have always had a relationship, DSL having originally been developed as basically a "better T-carrier" for ISDN use. In the modern world, DSL pretty much always refers to either ADSL or VDSL, newer protocols designed for consumer service that can coexist with a voice line and provide very high speeds. Many aspects of how DSL works have their roots in T-carrier, including the common use of protocols like ATM (now rare) or PPP (fading out) to encapsulate IP over DSL.

Okay, I realize this article has been sort of dry and factual, but I thought it'd be interesting to share a bit about T-carrier. I think it's something that people my age vaguely remember as important but have never thought about that much. I, personally, am probably just a bit too old to have had much real interaction with T-carrier. When I worked for an MSP for a bit in high school I saw a few customers that still had HDSL-based T1 service, and when I later interned at General Electric we had an OC-3 SONET connection that was in the process of being turned down. Just really catching the trail end... and yet for years later the Steam Hardware Survey was still asking if I had a T1.

Why did T1 get stuck so long in the very specific context of video games? I assume because video game developers frequently had T-carrier connections to their offices and knew that its guaranteed bandwidth provided very good performance for video games. The poor latency of ADSL meant that, despite a theoretical bandwidth several times larger, it was not really a better choice for the specific application of multiplayer games. So the "T1 is god tier" thing hung around for longer than you would have otherwise expected.

  1. I actually don't know why the microwave carriers have multi-letter names starting in T. Something to look into. This convention is older than T-carrier and presumably started with TDX, the experimental microwave carrier that went into use in the late 1940s. I think the naming convention for carriers changed around the mid-century, as T1 is often said to stand for "transmission system one" which is consistent with later AT&T naming conventions but inconsistent with A-carrier through L-carrier, where the letters didn't stand for anything in particular. On the other hand, it is entirely possible that "T" was just the next letter in the sequence, and it standing for "transmission" was a later invention. You will also see people assert that the "T" stands for "trunk," perhaps evidence that the meaning is made up.

the video lunchbox

An opening note: would you believe that I have been at this for five years, now? If I planned ahead better, I would have done this on the five-year anniversary, but I missed it. Computers Are Bad is now five years and four months old.

When I originally launched CAB, it was my second attempt at keeping up a blog. The first, which I had called 12 Bit Word, went nowhere and I stopped keeping it up. One of the reasons, I figured, is that I had put too much effort into it. CAB was a very low-effort affair, which was perhaps best exemplified by the website itself. It was monospace and 80 characters wide, a decision that I found funny (in a shitposty way) and generated constant complaints. To be fair, if you didn't like the font, it was "user error:" I only ever specified "monospace" and I can't be blamed that certain platforms default to Courier. But there were problems beyond the appearance; the tool that generated the website was extremely rough and made new features frustrating to implement.

Over the years, I have not invested much (or really any) effort in promoting CAB or even making it presentable. I figured my readership, interested in vintage computing, would probably put up with it anyway. That is at least partially true, and I am not going to put any more effort into promotion, but some things have changed. Over time I have broadened my topics quite a bit, and I now regularly write about things that I would have dropped as "off topic" three or four years ago. Similarly, my readership has broadened, and probably to a set of people that find 80 characters of monospace text less charming.

I think I've also changed my mind in some ways about what is "special" about CAB. One of the things that I really value about it, that I don't think comes across to readers well, is the extent to which it is what I call artisanal internet. It's like something you'd get at the farmer's market. What I mean by this is that CAB is a website generated by a static site generator that I wrote, and a newsletter sent by a mailing list system that I wrote, and you access them by connecting directly to a VM that I administer, on a VM cluster that I administer, on hardware that I own, in a rack that I lease in a data center in downtown Albuquerque, New Mexico. This is a very old-fashioned way of doing things, now, and one of the ironies is that it is a very expensive way of doing things. It would be radically cheaper and easier to use wordpress.com, and it would probably go down less often and definitely go down for reasons that are my fault less often. But I figure people listen to me in part because I don't use wordpress.com, because I have weird and often impractical opinions about how to best contribute to internet culture.

I spent a week on a cruise ship just recently, and took advantage of the great deal of time I had to look at the sea to also get some work done. Strategically, I decided, I want to keep the things that are important to me (doing everything myself) and move on from the things that are not so important (the website looking, objectively, bad). So this is all a long-winded announcement that I am launching, with this post, a complete rewrite of the site generator and partial rewrite of the mailing list manager.

This comes with several benefits to you. First, computer.rip is now much more readable and, arguably, better looking. Second, it should be generally less buggy (although to be fair I had eliminated most of the problems with the old generator through sheer brute force over the years). Perhaps most importantly, the emails sent to the mailing list are no longer the unrendered Markdown files.

I originally didn't use markup of any kind, so it was natural to just email out the plaintext files. But then I wanted links, and then I wanted pictures, leading me to implement Markdown in generating the webpages... but I just kept emailing out the plaintext files. I strongly considered switching to HTML emails as a solution and mostly finished the effort, but in the end I didn't like it. HTML email is a massive pain in the ass and, I think, distasteful. Instead, I modified a Markdown renderer to create human-readable plaintext output. Things like links and images will still be a little weird in the plaintext emails, but vastly better than they were before.

I expect some problems to surface when I put this all live. It is quite possible that RSS readers will consider the most recent ten posts to all be new again due to a change in how the article IDs are generated. I tried to avoid that happening but, look, I'm only going to put so much time into testing and I've found RSS readers to be surprisingly inconsistent. If anything else goes weird, please let me know.


There has long been a certain connection between the computer industry and the art of animation. The computer, with a frame-oriented raster video output, is intrinsically an animation machine. Animation itself is an exacting, time-consuming process that has always relied on technology to expand the frontier of the possible. Walt Disney, before he was a business magnate, was a technical innovator in animation. He made great advances in cel animation techniques during the 1930s, propelling the Disney Company to fame not only by artistic achievement but also by reducing the cost and time involved in creating feature-length animated films.

Most readers will be familiar with the case of Pixar, a technical division of Lucasfilm that operated primarily as a computer company before its 1986 spinoff under computer executive Steve Jobs---who led the company through a series of creative successes that overshadowed the company's technical work until it was known to most only as a film studio.

Animation is hard. There are several techniques, but most ultimately come down to an animator using experience, judgment, and trial and error to get a series of individually composed frames to combine into fluid motion. Disney worked primarily in cel animation: each element of each frame was hand-drawn, but on independent transparent sheets. Each frame was created by overlaying the sheets like layers in a modern image editor. The use of separate cels made composition and corrections easier, by allowing the animator to move and redraw single elements of the final image, but it still took a great deal of experience to produce a reasonable result.

The biggest challenge was in anticipating how motion would appear. From the era of Disney's first work, problems like registration (consistent positioning of non-moving objects) had been greatly simplified by the use of clear cels and alignment pegs on the animator's desk that held cels in exact registration for tracing. But some things in an animation are supposed to move, I would say that's what makes it animation. There was no simple jig for ensuring that motion would come out smoothly, especially for complex movements like a walking or gesturing character. The animator could flip two cels back and forth, but that was about as good as they could get without dedicating the animation to film.

For much of the mid-century, a typical animation workflow looked like this: a key animator would draw out the key frames in final or near-final quality, establishing the most important moments in the animation, the positions and poses of the characters. The key animator or an assistant would then complete a series of rough pencil sketches for the frames that would need to go in between. These sketches were sent to the photography department for a "pencil test."

In the photography department, a rostrum camera was used: a cinema camera, often 16mm, permanently mounted on an adjustable stand that pointed it down at a flat desk. The rostrom camera looked a bit like a photographic enlarger and worked much the same way, but backwards: the photographer laid out the cels or sketches on the desk, adjusted the position and focus of the camera for the desired framing, and then exposed one frame. This process was repeated, over and over and over, a simple economy that explains the common use of a low 12 FPS frame rate in animation.

Once the pencil test had been photographed, the film went to the lab where it was developed, and then returned to the animation studio where the production team could watch it played on a cinema projector in a viewing room. Ideally, any problems would be identified during this first viewing before the key frames and pencil sketches were sent to the small army of assistant animators. These workers would refine the cels and redraw the pencil sketches in part by tracing, creating the "in between" frames of the final animation. Any needed changes were costly, even when caught at the earliest stage, as it usually took a full day for the photography department to return a new a pencil test (making the pencil test very much analogous to the dailies used in film). What separated the most skilled animators from amateurs, then, was often their ability to visualize the movement of their individual frames by imagination. They wanted to get it right the first time.


Graphics posed a challenge to computers for similar reasons. Even a very basic drawing involves a huge number of line segments, which a computer will need to process individually during rendering. Add properties such as color, consider the practicalities of rasterizing, and then make it all move: just the number of simple arithmetic problems involved in computer graphics becomes enormous. It is not a coincidence that we picture all early computer systems as text-only, although it is a bit unfair. Graphical output is older than many realize, originating with vector-mode CRT displays in the 1950s. Still, early computer graphics were very slow. Vector-mode displays were often paired with high-end scientific computers and you could still watch them draw in real time. Early graphics-intensive computer applications like CAD used specialized ASICs for drawing and yet provided nothing like the interactivity we expect from computers today.

The complexity of computer graphics ran head-first against an intense desire for more capable graphical computers, driven most prominently by the CAD industry. Aerospace and other advanced engineering fields were undergoing huge advancements during the second half of the 20th century. World War II had seen adoption of the jet engine, for example, machines which were extremely powerful but involved complex mathematics and a multitude of 3D parts that made them difficult for a human to reason over. The new field of computer-aided design promised a revolutionary leap in engineering capability, but ironically, the computers were quite bad at drawing. In the first decades, CAD output was still being sent to traditional draftsmen for final drawings. The computers were not only slow, but unskilled at the art of drafting: limitations on the number and complexity of the shapes that computers could render limited them to only very basic drawings, without the extensive annotations that would be needed for manufacturing.

During the 1980s, the "workstation" began to replace the mainframe in engineering applications. Today, "workstation" mostly just identifies PCs that are usually extra big and always extra expensive. Historically, workstations were a different class of machines from PCs that often employed fundamentally different architectures. Many workstations were RISC, an architecture selected for better mathematical performance. They frequently ran UNIX or a derivative, and featured the first examples of what we now call a GPU. Some things don't change: they were also very big, and very expensive.

It was the heady days of the space program and the Concorde, then, that brought us modern computer graphics. The intertied requirements for scientific computing, numerical simulation, and computer graphics that emerged from Cold War aerospace and weapons programs forged a strong bond between high-end computing and graphics. One could perhaps say that the nexus between AI and GPUs today is an extension of this era, although I think it's a bit of a stretch given the text-heavy applications. The echoes of the dawn of computer graphics are much quieter today, but still around. They persist, for example, in the heavy emphasis on computer visualization seen throughout scientific computing but especially in defense-related fields. They persist also in the names of the companies born in that era, names like Silicon Graphics and Mentor Graphics.


The development of video technology, basically the combination of preexisting television technology with new video tape recorders, lead to a lot of optimizations in film. Video was simply not of good enough quality to displace film for editing and distribution, but it was fast and inexpensive. For example, beginning in the 1960s filmmakers began to adopt a system called "video assist." A video camera was coupled to the film camera, either side-by-side with matched lenses or even sharing the same lens via a beam splitter. By running a video tape recorder during filming, the crew could generate something like an "instant daily" and play the tape back on an on-set TV. For the first time, a director could film a scene and then immediately rewatch it. Video assist was a huge step forward, especially in the television industry where it furthered the marriage of film techniques and television techniques for the production of television dramas.

It certainly seems that there should be a similar optimization for animation. It's not easy, though. Video technology was all designed around sequences of frames in a continuous analog signal, not individual images stored discretely. With the practicalities of video cameras and video recorders, it was surprisingly difficult to capture single frames and then play them back to back.

In the 1970s, animators Bruce Lyon and John Lamb developed the Lyon-Lamb Video Animation System (VAS). The original version of the VAS was a large workstation that replaced a rostrum camera with a video camera, monitor, and a custom video tape recorder. Much like the film rostrum camera, the VAS allowed an operator to capture a single frame at a time by composing it on the desk. Unlike the traditional method, the resulting animation could be played back immediately on the included monitor.

The VAS was a major innovation in cel animation, and netted both an Academy Award and an Emmy for technical achievement. While it's difficult to say for sure, it seems like a large portion of the cel-animated features of the '80s had used the VAS for pencil tests. The system was particularly well-suited to rotoscoping, overlaying animation on live-action images. Through a combination of analog mixing techniques and keying, the VAS could directly overlay an animator's work on the video, radically accelerating the process. To demonstrate the capability, John Lamb created a rotoscoped music video for the Tom Waits song "The One That Got Away." The resulting video, titled "Tom Waits for No One," was probably the first rotoscoped music video as well as the first production created with the video rotoscope process. As these landmarks often do, it languished in obscurity until it was quietly uploaded to YouTube in 2006.

The VAS was not without its limitations. It was large, and it was expensive. Even later generations of the system, greatly miniaturized through the use of computerized controls and more modern tape recorders, came in at over $30,000 for a complete system. And the VAS was designed around the traditional rostrom camera workflow, intended for a dedicated operator working at a desk. For many smaller studios the system was out of reach, and for forms of animation that were not amenable to top-down photography on a desk, the VAS wasn't feasible.


There are some forms of animation that are 3D---truly 3D. Disney had produced pseudo-3D scenes by mounting cels under a camera on multiple glass planes, for example, but it was obviously possible to do so in a more complete form by the use of animated sculptures or puppets. Practical challenges seem to have left this kind of animation mostly unexplored until the rise of its greatest producer, Will Vinton.

Vinton grew up in McMinnville, Oregon, but left to study at UC Berkeley. His time in Berkeley left him not only with an architecture degree (although he had studied filmmaking as well), but also a friendship with Bob Gardiner. Gardiner had a prolific and unfortunately short artistic career, in which he embraced many novel media including the hologram. Among his inventions, though, was a novel form of animation using clay: Gardiner was fascinated with sculpting and posing clay figures, and demonstrated the animation potential to Vinton. Vinton, in turn, developed a method of using his student film camera to photograph the clay scenes frame by frame.

Their first full project together, Closed Mondays, took the Academy Award for Best Animated Short Film in 1975. It was notable not only for the moving clay sculptures, but for its camerawork. Vinton had realized that in clay animation, where scenes are composed in real 3D space, the camera can be moved from frame to frame just like the figures. Not long after this project, Vinton and Gardiner split up. Gardiner seems to have been a prolific artist in that way where he could never stick to one thing for very long, and Vinton had a mind towards making a business out of this new animation technology. It was Vinton who christened it Claymation, then a trademark of his new studio.

Vinton returned to his home state and opened Will Vinton Studios in Portland. Vinton Studios released a series of successful animated shorts in the '70s, and picked up work on numerous other projects, contributing for example to the "Wizard of Oz" film sequel "Return to Oz" and the Disney film "Captain EO." By far Vinton Studios most famous contributions to our culture, though, are their advertising projects. Will Vinton Studios brought us the California Raisins, the Noid, and walking, talking M&M's.

Will Vinton Studios struggled with producing claymation at commercial scale. Shooting with film cameras, it took hours to see the result. Claymation scenes were more difficult to rework than cel animation, setting an even larger penalty for reshoots. Most radically, claymation scenes had to be shot on sets, with camera and light rigging. Reshooting sections without continuity errors was as challenging as animating those sections in the first place.

To reduce rework, they used pencil tests: quicker, lower-effort versions of scenes shot to test the lighting, motion, and sound synchronization before photography with a film camera. Their pencil tests were apparently captured on a crude system of customized VCRs, allowing the animator to see the previous frame on a monitor as they composed the next, and then to play back the whole sequence. It was better than working from film, but it was still slow going.


The area from Beaverton to Hillsboro, in Oregon near Portland, is sometimes called "the silicon forest" largely on the influence of Intel and Tektronix. As in the better known silicon valley, these two keystone companies were important not only on their own, but also as the progenitors of dozens of new companies. Tektronix, in particular, had a steady stream of employees leaving to start their own businesses. Among these alumni was Mentor Graphics.

Mentor Graphics was an early player in electronic design automation (EDA), sort of like a field of CAD specialized to electronics. Mentor products assisted not just in the physical design of circuit boards and ICs, but also simulation and validation of their functionality. Among the challenges of EDA are its fundamentally graphical nature: the final outputs of EDA are often images, masks for photolithographic manufacturing processes, and engineers want to see both manufacturing drawings and logical diagrams as they work on complex designs.

When Mentor started out in 1981, EDA was in its infancy and relied mostly on custom hardware. Mentor went a different route, building a suite of software products that ran on Motorola 68000-based workstations from Apollo. The all-software architecture had cost and agility advantages, and Mentor outpaced their competition to become the field's leader.

Corporations want for growth, and by the 1990s Mentor had a commanding position in EDA and went looking for other industries to which their graphics-intensive software could be applied. One route they considered was, apparently, animation: computer animation was starting to take off, and there were very few vendors for not just the animation software but the computer platforms capable of rendering the product. In the end, Mentor shied away: companies like Silicon Graphics and Pixar already had a substantial lead, and animation was an industry that Mentor knew little about. As best I can tell, though, it was this brief investigation of a new market that exposed Mentor engineering managers Howard Mozeico and Arthur Babitz to the animation industry.

I don't know much about their career trajectories in the years shortly after, only that they both decided to leave Mentor for their own reasons. Arthur Babitz went into independent consulting, and found a client reminiscent of his work at Mentor, an established animation studio that was expanding into computer graphics: Will Vinton Studios. Babitz's work at Will Vinton Studios seems to have been largely unrelated to claymation, but it exposed him to the process, and he watched the way they used jury-rigged VCRs and consumer video cameras to preview animations.

Just a couple of years later, Mozeico and Babitz talked about their experience with animation at Mentor, a field in which they were both still interested. Babitz explained the process he had seen at Will Vinton Studios, and his ideas for improving it. Both agreed that they wanted to figure out a sort of retirement enterprise, what we might now call a "lifestyle business": they each wanted to found a company that would keep them busy, but not too busy. The pair incorporated Animation Toolworks, headquartered in Mozeico's Sherwood, Oregon home.


In 1998 Animation Toolworks hit trade shows with the Video Lunchbox. The engineering was mostly by Babitz, the design and marketing by Mozeico, and the manufacturing done on contract by a third party. The device took its name from its form factor, a black crinkle paint box with a handle on top of its barn-roof-shaped lid. It was something like the Lyon Lamb VAS, if it was portable, digital, and relatively inexpensive.

The Lunchbox was essentially a framegrabber, a compact and simplified version of the computer framegrabbers that were coming into use in the animation industry. You plugged a video camera into the input, and a television monitor into the output. You could see the output of the camera, live, on the monitor while you composed a scene. Then, one press of a button captured a single frame and stored it. With a press of another button, you could swap between the stored frame and the live image, helping to compose the next. You could even enable an automatic "flip-flop" mode that alternated the two rapidly, for hands-free adjustment.

Each successive press of the capture button stored another frame to the Lunchbox's memory, and buttons allowed you to play the entire set of stored frames as a loop, or manually step forward or backward through the frames. And that was basically it: there were a couple of other convenience features like an intervalometer (for time lapse) and the ability to record short sections of real-time video, but complete operation of the device was really very simple. That seems to have been one of its great assets. The Lunchbox was much easier to sell after Mozeico gave a brief demonstration and said that that was all there is to it.

To professionals, the Lunchbox was a more convenient, more reliable, and more portable version of the video tape recorder or computer framegrabber systems they were already using for pencil tests. Early customers of Animation Toolworks included Will Vinton Studios alongside other animation giants like Disney, MTV, and Academy Award-winning animator Mark Osborne. Animation Toolworks press quoted animators from these firms commenting on the simplicity and ease of use, saying that it had greatly sped up the animation test process.

In a review for Animation World Magazine, Kellie-Bea Rainey wrote:

In most cases, computers as framegrabbers offer more complications than solutions. Many frustrations stem from the complexity of learning the computer, the software and it's constant upgrades. But one of the things Gary Schwartz likes most about the LunchBox is that the system requires no techno-geeks. "Computers are too complex and the technology upgrades are so frequent that the learning curve keeps you from mastering the tools. It seems that computers are taking the focus off the art. The Video LunchBox has a minimum learning curve with no upgrade manuals. Everything is in the box, just plug it in."

Indeed, the Lunchbox was so simple that it caught on well beyond the context of professional studios. It is remembered most as an educational tool. Disney used the Lunchbox for teaching cel animation in a summer program, but closer to home, the Lunchbox made its way to animation enthusiast and second-grade teacher Carrie Caramella. At Redmond, Oregon's John Tuck Elementary School, Caramella acted as director of a student production team that brought their short film "The Polka Dot Day" to the Northwest Film Center's Young People's Film and Video Festival. During the early 2000s, after-school and summer animation programs proliferated, many using claymation, and almost all using the Video Lunchbox.

At $3,500, the Video Lunchbox was not exactly cheap. It cost more than some of the more affordable computer-based options, but it was so much easier to use, and so much more durable, that it was very much at home in a classroom. Caramella:

"By using the lunchbox, we receive instant feedback because the camera acts > as an eye. It is also child-friendly, and you can manipulate the film a lot more."

Caramella championed animation at John Tuck, finding its uses in other topics. A math teacher worked with students to make a short animation of a chicken. In a unit on compound words, Caramella led students in animating their two words together: a sun and a flower dance; the word is "sunflower." Butter and milk, base and ball.

In Lake Oswego, an independent summer program called Earthlight Studios took up the system.

With the lunchbox, Corey's black-and-white drawings spring to life, two catlike animé characters circling each other with broad edged swords. It's the opening seconds of what he envisions will be an action-adventure film.

We can imagine how cringeworthy these student animations must be to their creators today, but early-'00s education was fascinated with multimedia and it seems rare that technology served the instructional role so well.

It was in this context that I crossed ways with the Lunchbox. As a kid, I went to a summer animation program at OMSI---a claymation program, which I hazily remember was sponsored by a Will Vinton Studios employee. In an old industrial building beside the museum, we made crude clay figures and then made them crudely walk around. The museum's inventory of Lunchboxes already showed their age, but they worked, in a way that was so straightforward that I think hardly any time was spent teaching operation of the equipment. It was a far cry from an elementary school film project in which, as I recall, nearly an entire day of class time was burned trying to get video off of a DV camcorder and into iMovie.


Mozeico and Babitz aimed for modest success, and that was exactly what they found. Animation Toolworks got started on so little capital that it turned a profit the first year, and by the second year the two made a comfortable salary---and that was all the company would ever really do. Mozeico and Babitz continued to improve on the concept. In 2000, they launched the Lunchbox Sync, which added an audio recorder and the ability to cue audio clips at specific frame numbers. In 2006, the Lunchbox DV added digital video.

By the mid-2000s, computer multimedia technology had improved by leaps and bounds. Framegrabbers and real-time video capture devices were affordable, and animation software on commodity PCs overtook the Lunchbox on price and features. Still, the ease of use and portability of the Lunchbox was a huge appeal to educators. By 2005 Animation Toolworks was basically an educational technology company, and in the following years computers overtook them in that market as well.

The era of the Lunchbox is over, in more ways than one. A contentious business maneuver by Phil Knight saw Will Vinton pushed out of Will Vinton Studios. He was replaced by Phil Knight's son, Travis Knight, and the studio rebranded to Laika. The company has struggled under its new management, and Laika has not achieved the renaissance of stop-motion that some thought Coraline might bring about. Educational technology has shifted its focus, as a business, to a sort of lightweight version of corporate productivity platforms that is firmly dominated by Google.

Animation Toolworks was still selling the Lunchbox DV as late as 2014, but by 2016 Mozeico and Babitz had fully retired and offered support on existing units only. Mozeico died in 2017, crushed under a tractor on his own vineyard. There are worse ways to go. Arthur Babitz is a Hood River County Commissioner.

Kellie-Bea Rainey:

I took the two-minute tutorial and taped it to the wall. I cleaned off a work table and set up a stage and a character. Then I put my Sharp Slimcam on a tripod... Once the camera was plugged into the LunchBox, I focused it on my animation set-up. Next, I plugged in my monitor.

All the machines were on and all the lights were green, standing by. It's time to hit the red button on the LunchBox and animate!

Yippee! Look Houston, we have an image! That was quick, easy and most of all, painless. I want to do more, and more, and even more.

The next time you hear from me I'll be having fun, teaching my own animation classes and making my own characters come to life. I think Gary Schwartz says it best, "The LunchBox brings the student back to what animation is all about: art, self-esteem, results and creativity."

I think we're all a little nostalgic for the way technology used to be. I know I am. But there is something to be said for a simple device, from a small company, that does a specific thing well. I'm not sure that I have ever, in my life, used a piece of technology that was as immediately compelling as the Video Lunchbox. There are numerous modern alternatives, replete with USB and Bluetooth and iPad apps. Somehow I am confident that none of them are quite as good.

CodeSOD: Contracting Space

A ticket came in marked urgent. When users were entering data in the header field, the spaces they were putting in kept getting mangled. This was in production, and had been in production for sometime.

Mike P picked up the ticket, and was able to track down the problem to a file called Strings.java. Yes, at some point, someone wrote a bunch of string helper functions and jammed them into a package. Of course, many of the functions were re-implementations of existing functions: reinvented wheels, now available in square.

For example, the trim function.

    /**
     * @param str
     * @return The trimmed string, or null if the string is null or an empty string.
     */
    public static String trim(String str) {
        if (str == null) {
            return null;
        }

        String ret = str.trim();

        int len = ret.length();
        char last = '\u0021';    // choose a character that will not be interpreted as whitespace
        char c;
        StringBuffer sb = new StringBuffer();
        for (int i = 0; i < len; i++) {
            c = ret.charAt(i);
            if (c > '\u0020') {
                if (last <= '\u0020') {
                    sb.append(' ');
                }
                sb.append(c);
            }
            last = c;
        }
        ret = sb.toString();

        if ("".equals(ret)) {
            return null;
        } else {
            return ret;
        }
    }

Now, Mike's complaint is that this function could have been replaced with a regular expression. While that would likely be much smaller, regexes are expensive- in performance and frequently in cognitive overhead- and I actually have no objections to people scanning strings.

But let's dig into what we're doing here.

They start with a null check, which sure. Then they trim the string; never a good sign when your homemade trim method calls the built-in.

Then, they iterate across the string, copying characters into a StringBuffer. If the current character is above unicode character 20- the realm of printable characters- and if the last character was a whitespace character, we copy a whitespace character into the output, and then the printable character into the output.

What this function does is simply replace runs of whitespace with single whitespace characters.

"This        string"
becomes
"This string"

Badly I should add. Because there are plenty of whitespace characters which appear above \u0020- like the non-breaking space (\u00A0), and many other control characters. While you might be willing to believe your users will never figure out how to type those, you can't guarantee that they'll never copy/paste them.

For me, however, this function does something far worse than being bad at removing extraneous whitespace. Because it has that check at the end- if I handed it a perfectly good string that is only whitespace, it hands me back a null.

I can see the argument- it's a bad input, so just give me back an objectively bad result. No IsNullOrEmpty check, just a simple null check. But I still hate it- turning an actual value into a null just bothers me, and seems like an easy way to cause problems.

In any case, the root problem with this bug was simply developer invented requirements: the users never wanted stray spaces to be automatically removed in the middle of the string. Trimmed yes, gutted no.

No one tried to use multiple spaces for most of the history of the application, thus no one noticed the problem. No one expected it to not work. Hence the ticket and the panic by users who didn't understand what was going on.

[Advertisement] Picking up NuGet is easy. Getting good at it takes time. Download our guide to learn the best practice of NuGet for the Enterprise.

Error'd: Pickup Sticklers

An Anonymous quality analyst and audiophile accounted "As a returning customer at napalmrecords.com I was forced to update my Billing Address. Fine. Sure. But what if my *House number* is a very big number? More than 10 "symbols"? Fortunately, 0xDEADBEEF for House number and J****** for First Name both passed validation."

3

And then he proved it, by screenshot:

4

 

Richard P. found a flubstitution failure mocking "I'm always on the lookout for new and interesting Lego sets. I definitely don't have {{product.name}} in my collection!"

2

 

"I guess short-named siblings aren't allowed for this security question," pointed out Mark T.

0

 

Finally, my favorite category of Error'd -- the security snafu. Tim R. reported this one, saying "Sainsbury/Argos in the UK doesn't want just anybody picking up the item I've ordered online and paid for, so they require not one, not two, but 3 pieces of information when I come to collect it. There's surely no way any interloper could possibly find out all 3, unless they were all sent in the same email obviously." Personally, my threat model for my grocery pickups is pretty permissive, but Tim cares.

1

 

[Advertisement] Picking up NuGet is easy. Getting good at it takes time. Download our guide to learn the best practice of NuGet for the Enterprise.

Coded Smorgasbord: High Strung

Most languages these days have some variation of "is string null or empty" as a convenience function. Certainly, C#, the language we're looking at today does. Let's look at a few example of how this can go wrong, from different developers.

We start with an example from Jason, which is useless, but not a true WTF:

/// <summary>
/// Does the given string contain any characters?
/// </summary>
/// <param name="strToCheck">String to check</param>
/// <returns>
/// true - String contains some characters.
/// false - String is null or empty.
/// </returns>
public static bool StringValid(string strToCheck)
{
        if ((strToCheck == null) ||
                (strToCheck == string.Empty))
                return false;

        return true;
}

Obviously, a better solution here would be to simply return the boolean expression instead of using a conditional, but equally obvious, the even better solution would be to use the built-in. But as implementations go, this doesn't completely lose the plot. It's bad, it shouldn't exist, but it's barely a WTF. How can we make this worse?

Well, Derek sends us an example line, which is scattered through the codebase.

if (Port==null || "".Equals(Port)) { /* do stuff */}

Yes, it's frequently done as a one-liner, like this, with the do stuff jammed all together. And yes, the variable is frequently different- it's likely the developer responsible saved this bit of code as a snippet so they could easily drop it in anywhere. And they dropped it in everywhere. Any place a string got touched in the code, this pattern reared its head.

I especially like the "".Equals call, which is certainly valid, but inverted from how most people would think about doing the check. It echos Python's string join function, which is invoked on the join character (and not the string being joined), which makes me wonder if that's where this developer started out?

I'll never know.

Finally, let's poke at one from Malfist. We jump over to Java for this one. Malfist saw a function called checkNull and foolishly assumed that it returned a boolean if a string was null.

public static final String checkNull(String str, String defaultStr)
{
    if (str == null)
        return defaultStr ;
    else
        return str.trim() ;
}

No, it's not actually a check. It's a coalesce function. Okay, misleading names aside, what is wrong with it? Well, for my money, the fact that the non-null input string gets trimmed, but the default string does not. With the bonus points that this does nothing to verify that the default string isn't null, which means this could easily still propagate null reference exceptions in unexpected places.

I've said it before, and I'll say it again: strings were a mistake. We should just abolish them. No more text, everybody, we're done.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

CodeSOD: Across the 4th Dimension

We're going to start with the code, and then talk about it. You've seen it before, you know the chorus: bad date handling:

C_DATE($1)
C_STRING(7;$0)
C_STRING(3;$currentMonth)
C_STRING(2;$currentDay;$currentYear)
C_INTEGER($month)

$currentDay:=String(Day of($1))
$currentDay:=Change string("00";$currentDay;3-Length($currentDay))
$month:=Month of($1)
Case of

: ($month=1)
$currentMonth:="JAN"

: ($month=2)
$currentMonth:="FEB"

: ($month=3)
$currentMonth:="MAR"

: ($month=4)
$currentMonth:="APR"

: ($month=5)
$currentMonth:="MAY"

: ($month=6)
$currentMonth:="JUN"

: ($month=7)
$currentMonth:="JUL"

: ($month=8)
$currentMonth:="AUG"

: ($month=9)
$currentMonth:="SEP"

: ($month=10)
$currentMonth:="OCT"

: ($month=11)
$currentMonth:="NOV"

: ($month=12)
$currentMonth:="DEC"

End case

$currentYear:=Substring(String(Year of($1));3;2)

$0:=$currentDay+$currentMonth+$currentYear

At this point, most of you are asking "what the hell is that?" Well, that's Brewster's contribution to the site, and be ready to be shocked: the code you're looking at isn't the WTF in this story.

Let's rewind to 1984. Every public space was covered with a thin layer of tobacco tar. The Ground Round restaurant chain would sell children's meals based on the weight of the child and have magicians going from table to table during the meal. And nobody quite figured out exactly how relational databases were going to factor into the future, especially because in 1984, the future was on the desktop, not the big iron "server side".

Thus was born "Silver Surfer", which changed its name to "4th Dimension", or 4D. 4D was an RDBMS, an IDE, and a custom programming language. That language is what you see above. Originally, they developed on Apple hardware, and were almost published directly by Apple, but "other vendors" (like FileMaker) were concerned that Apple having a "brand" database would hurt their businesses, and pressured Apple- who at the time was very dependent on its software vendors to keep its ecosystem viable. In 1993, 4D added a server/client deployment. In 1995, it went cross platform and started working on Windows. By 1997 it supported building web applications.

All in all, 4D seems to always have been a step or two behind. It released a few years after FileMaker, which served a similar niche. It moved to Windows a few years after Access was released. It added web support a few years after tools like Cold Fusion (yes, I know) and PHP (I absolutely know) started to make building data-driven web apps more accessible. It started supporting Service Oriented Architectures in 2004, which is probably as close to "on time" as it ever got for shipping a feature based on market demands.

4D still sees infrequent releases. It supports SQL (as of 2008), and PHP (as of 2010). The company behind it still exists. It still ships, and people- like Brewster- still ship applications using it. Which brings us all the way back around to the terrible date handling code.

4D does have a "date display" function, which formats dates. But it only supports a handful of output formats, at least in the version Brewster is using. Which means if you want DD-MMM-YYYY (24-SEP-2025) you have to build it yourself.

Which is what we see above. The rare case where bad date handling isn't inherently the WTF; the absence of good date handling in the available tooling is.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

CodeSOD: One Last ID

Chris's company has an unusual deployment. They had a MySQL database hosted on Cloud Provider A. They hired a web development company, which wanted to host their website on Cloud Provider B. Someone said, "Yeah, this makes sense," and wrote the web dev company a sizable check. They app was built, tested, and released, and everyone was happy.

Everyone was happy until the first bills came in. They expected the data load for the entire month to be in the gigabytes range, based on their userbase and expected workloads. But for some reason, the data transfer was many terabytes, blowing up their operational budget for the year in a single month.

Chris fired up a traffic monitor and saw that, yes, huge piles of data were getting shipped around with every request. Well, not every request. Every insert operation ended up retrieving a huge pile of data. A little more research was able to find the culprit:

SELECT last_insert_id() FROM some_table_name

The last_insert_id function is a useful one- it returns the last autogenerated ID in your transaction. So you can INSERT, and then check what ID was assigned to the inserted record. Great. But the way it's meant to be used is like so: SELECT last_insert_id(). Note the lack of a FROM clause.

By adding the FROM, what the developers were actually saying were "grab all rows from this table, and select the last_insert_id once for each one of them". The value of last_insert_id() just got repeated once for each row, and there were a lot of rows. Many millions. So every time a user inserted a row into most tables, the database sent back a single number, repeated millions and millions of times. Each INSERT operation caused a 30MB reply. And when you have high enough traffic, that adds up quickly.

On a technical level, it was an easy fix. On a practical one, it took six weeks to coordinate with the web dev company and their hosting setup to make the change, test the change, and deploy the change. Two of those weeks were simply spent convincing the company that yes, this was in fact happening, and yes, it was in fact their fault.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

CodeSOD: Identify a Nap

Guy picked up a bug ticket. There was a Hiesenbug; sometimes, saving a new entry in the application resulted in a duplicate primary key error, which should never happen.

The error was in the message-bus implementation someone else at the company had inner-platformed together, and it didn't take long to understand why it failed.

/**
 * This generator is used to generate message ids.
 * This implementation merely returns the current timestamp as long.
 *
 * We are, thus, limited to insert 1000 new messages per second.
 * That throughput seems reasonable in regard with the overall
 * processing of a ticket.
 *
 * Might have to re-consider that if needed.
 *
 */
public class IdGenerator implements IdentifierGenerator
{

        long previousId;
       
        @Override
        public synchronized Long generate (SessionImplementor session, Object parent) throws HibernateException {
                long newId = new Date().getTime();
                if (newId == previousId) {
                        try { Thread.sleep(1); } catch (InterruptedException ignore) {}
                        newId = new Date().getTime();
                }
                return newId;
        }
}

This generates IDs based off of the current timestamp. If too many requests come in and we start seeing repeating IDs, we sleep for a second and then try again.

This… this is just an autoincrementing counter with extra steps. Which most, but I suppose not all databases supply natively. It does save you the trouble of storing the current counter value outside of a running program, I guess, but at the cost of having your application take a break when it's under heavier than average load.

One thing you might note is absent here: generate doesn't update previousId. Which does, at least, mean we won't ever sleep for a second. But it also means we're not doing anything to avoid collisions here. But that, as it turns out, isn't really that much of a problem. Why?

Because this application doesn't just run on a single server. It's distributed across a handful of nodes, both for load balancing and resiliency. Which means even if the code properly updated previousId, this still wouldn't prevent collisions across multiple nodes, unless they suddenly start syncing previousId amongst each other.

I guess the fix might be to combine a timestamp with something unique to each machine, like… I don't know… hmmm… maybe the MAC address on one of their network interfaces? Oh! Or maybe you could use a sufficiently large random number, like really large. 128-bits or something. Or, if you're getting really fancy, combine the timestamp with some randomness. I dunno, something like that really sounds like it could get you to some kind of universally unique value.

Then again, since the throughput is well under 1,000 messages per second, you could probably also just let your database handle it, and maybe not generate the IDs in code.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

Error'd: You Talkin' to Me?

The Beast In Black is back with a simple but silly factual error on the part of the gateway to all (most) human knowledge.

0

 

B.J.H. "The old saying is "if you don't like the weather wait five minutes". Weather.com found a time saver." The trick here is to notice that the "now" temperature is not the same as the headline temperature, also presumably now.

1

 

"That's some funny math you got there. Be a shame if it was right," says Jason . "The S3 bucket has 10 files in it. Picking any two (or more) causes the Download button to go disabled with this message when moused over. All I could think of is that this S3 bucket must be in the same universe as https://thedailywtf.com/articles/free-birds " Alas, we are all in the same universe as https://thedailywtf.com/articles/free-birds .

2

 

"For others, the markets go up and down, but me, I get real dividends!" gloats my new best friend Mr. TA .

3

 

David B. is waiting patiently. "Somewhere in the USPS a package awaits delivery. Either rain, nor snow, nor gloom of night shall prevent the carrier on their appointed rounds. When these rounds will occur are not the USPS's problem." We may not know the day, but we know the hour!

4

 

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

CodeSOD: An Echo In Here in here

Tobbi sends us a true confession: they wrote this code.

The code we're about to look at is the kind of code that mixes JavaScript and PHP together, using PHP to generate JavaScript code. That's already a terrible anti-pattern, but Tobbi adds another layer to the whole thing.


if (AJAX)
{
    <?php
        echo "AJAX.open(\"POST\", '/timesheets/v2/rapports/FactBCDetail/getDateDebutPeriode.php', true);";
            
    ?>
    
    AJAX.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
    AJAX.onreadystatechange = callback_getDateDebutPeriode;
    AJAX.send(strPostRequest);
}

if (AJAX2)
{
    <?php
        echo "AJAX2.open(\"POST\", '/timesheets/v2/rapports/FactBCDetail/getDateFinPeriode.php', true);";
    ?>
    AJAX2.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
    AJAX2.onreadystatechange = callback_getDateFinPeriode;
    AJAX2.send(strPostRequest);
}

So, this uses server side code to… output string literals which could have just been written directly into the JavaScript without the PHP step.

"What was I thinking when I wrote that?" Tobbi wonders. Likely, you weren't thinking, Tobbi. Have another cup of coffee, I think you need it.

All in all, this code is pretty harmless, but is a malodorous brain-fart. As for absolution: this is why we have code reviews. Either your org doesn't do them, or it doesn't do them well. Anyone can make this kind of mistake, but only organizational failures get this code merged.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

Representative Line: Brace Yourself

Today's representative line is almost too short to be a full line. But I haven't got a category for representative characters, so we'll roll with it. First, though, we need the setup.

Brody inherited a massive project for a government organization. It was the kind of code base that had thousands of lines per file, and frequently thousands of lines per function. Almost none of those lines were comments. Almost.

In the middle of one of the shorter functions (closer to 500 lines), Brody found this:

//    }

This was the only comment in the entire file. And it's a beautiful one, because it tells us so much. Specifically, it tells us the developer responsible messed up the brace count (because clearly a long function has loads of braces in it), and discovered their code didn't compile. So they went around commenting out extra braces until they found the offender. Code compiled, and voila- on to the next bug, leaving the comment behind.

Now, I don't know for certain that's why a single closing brace is commented out. But also, I know for certain that's what happened, because I've seen developers do exactly that.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Representative Line: Reduced to a Union

The code Clemens M supported worked just fine for ages. And then one day, it broke. It didn't break after a deployment, which implied some other sort of bug. So Clemens dug in, playing the game of "what specific data rows are breaking the UI, and why?"

One of the organizational elements of their system was the idea of "zones". I don't know the specifics of the application as a whole, but we can broadly describe it thus:

The application oversaw the making of widgets. Widgets could be assigned to one or more zones. A finished product requires a set of widgets. Thus, the finished product has a number of zones that's the union of all of the zones of its component widgets.

Which someone decided to handle this way:

zones.reduce((accumulator, currentValue) => accumulator = _.union(currentValue))

So, we reduce across zones (which is an array of arrays, where the innermost arrays contain zone names, like zone-0, zone-1). In each step we union it with… nothing. The LoDash union function expects an array of arrays, and returns an array that's the union of all its inputs. This isn't how that function is meant to be used, but the behavior from this incorrect usage was that accumulator would end up holding the last element in zones. Which actually worked until recently, because until recently no one was splitting products across zones. When all the inputs were in the same zone, grabbing the last one was just fine.

The code had been like this for years. It was only just recently, as the company expanded, that it became problematic. The fix, at least, was easy- drop the reduce and just union correctly.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

CodeSOD: Functionally, a Date

Dates are messy things, full of complicated edge cases and surprising ways for our assumptions to fail. They lack the pure mathematical beauty of other data types, like integers. But that absence doesn't mean we can't apply the beautiful, concise, and simple tools of functional programming to handling dates.

I mean, you or I could. J Banana's co-worker seems to struggle a bit with it.

/**
 * compare two dates, rounding them to the day
 */
private static int compareDates( LocalDateTime date1, LocalDateTime date2 ) {
    List<BiFunction<LocalDateTime,LocalDateTime,Integer>> criterias = Arrays.asList(
            (d1,d2) -> d1.getYear() - d2.getYear(),
            (d1,d2) -> d1.getMonthValue() - d2.getMonthValue(),
            (d1,d2) -> d1.getDayOfMonth() - d2.getDayOfMonth()
        );
    return criterias.stream()
        .map( f -> f.apply(date1, date2) )
        .filter( r -> r != 0 )
        .findFirst()
        .orElse( 0 );
}

This Java code creates a list containing three Java functions. Each function will take two dates and returns an integer. It then streams that list, applying each function in turn to a pair of dates. It then filters through the list of resulting integers for the first non-zero value, and failing that, returns just zero.

Why three functions? Well, because we have to check the year, the month, and the day. Obviously. The goal here is to return a negative value if date1 preceeds date2, zero if they're equal, and positive if date1 is later. And on this metric… it does work. That it works is what makes me hate it, honestly. This not only shouldn't work, but it should make the compiler so angry that the computer gets up and walks away until you've thought about what you've done.

Our submitter replaced all of this with a simple:

return date1.toLocalDate().compareTo( date2.toLocalDate() );
[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Error'd: Free Birds

"These results are incomprensible," Brian wrote testily. "The developers at SkillCertPro must use math derived from an entirely different universe than ours. I can boast a world record number of answered questions in one hour and fifteen minutes somewhere."

0

 

"How I Reached Inbox -1," Maia titled her Tickity Tock. "Apparently I've read my messages so thoroughly that my email client (Mailspring) time traveled into the future and read a message before it was even sent."

1

 

... which taught Jason how to use Mailspring to order timely tunes. "Apparently, someone invented a time machine and is able to send us vinyls from the future..."

4

 

"Yes, we have no bananas," sang out Peter G. , rapping "... or email addresses or phone numbers, but we're going to block your post just the same (and this is better than the previous result of "Whoops something went wrong", because you'd never be able to tell something had gone wrong without that helpful message)."

2

 

Finally, our favorite cardsharp Adam R. might have unsharp eyes but sharp browser skills. "While reading an online bridge magazine, I tried to zoom out a bit but was dismayed to find I couldn't zoom out. Once it zooms in to NaN%, you're stuck there."

3

 

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

CodeSOD: The Getter Setter Getter

Today's Java snippet comes from Capybara James.

The first sign something was wrong was this:

private Map<String, String> getExtractedDataMap(PayloadDto payload) {
    return setExtractedDataToMap(payload);
}

Java conventions tell us that a get method retrieves a value, and a set method mutates the value. So a getter that calls a setter is… confusing. But neither of these are truly getters nor setters.

setExtractedDataToMap converts the PayloadDto to a Map<String, String>. getExtractedMap just calls that, which is just one extra layer of indirection that nobody needed, but whatever. At its core, this is just two badly named methods where there should be one.

But that distracts from the true WTF in here. Why on Earth are we converting an actual Java object to a Map<String,String>? That is a definite code smell, a sign that someone isn't entirely comfortable with object-oriented programming. You can't even say, "Well, maybe for serialization to JSON or something?" because Java has serializers that just do this transparently. And that's just the purpose of a DTO in the first place- to be a bucket that holds data for easy serialization.

We're left wondering what the point of all of this code is, and we're not alone. James writes:

I found this gem of a code snippet while trying to understand a workflow for data flow documentation purpose. I was not quite sure what the original developer was trying to achieve and at this point I just gave up

[Advertisement] Picking up NuGet is easy. Getting good at it takes time. Download our guide to learn the best practice of NuGet for the Enterprise.

CodeSOD: Upsert Yours

Henrik H sends us a short snippet, for a relative value of short.

We've all seen this method before, but this is a particularly good version of it:

public class CustomerController
{
    public void MyAction(Customer customer)
    {
        // snip 125 lines

        if (customer.someProperty)
            _customerService.UpsertSomething(customer.Id, 
            customer.Code, customer.Name, customer.Address1, 
            customer.Address2, customer.Zip, customer.City, 
            customer.Country, null, null, null, null, null, 
            null, null, null, null, null, null, null, null, 
            null, false, false, null, null, null, null, null, 
            null, null, null, null, null, null, null, false, 
            false, false, false, true, false, null, null, null,
            false, true, false, true, true, 0, false, false, 
            false, false, customer.TemplateId, false, false, false, 
            false, false, string.Empty, true, false, false, false, 
            false, false, false, false, false, true, false, false, 
            true, false, false, MiscEnum.Standard, false, false, 
            false, true, null, null, null);
        else
            _customerService.UpsertSomething(customer.Id, 
            customer.Code, customer.Name, customer.Address1, 
            customer.Address2, customer.Zip, customer.City, 
            customer.Country, null, null, null, null, null, 
            null, null, null, null, null, null, null, null, 
            null, false, false, null, null, null, null, null, 
            null, null, null, null, null, null, null, false, 
            false, false, false, true, false, null, null, null, 
            false, false, false, true, true, 0, false, false, 
            false, false, customer.TemplateId, false, false, false, 
            false, false, string.Empty, true, false, false, false, 
            false, false, false, false, true, true, false, false, 
            true, false, false, MiscEnum.Standard, false, false, 
            false, true, null, null, null);

        // snip 52 lines
    }
}

Welcome to the world's most annoying "spot the difference" puzzle. I've added line breaks (as each UpsertSomething was all on one line in the original) to help you find it. Here's a hint: it's one of the boolean values. I'm sure that narrows it down for you. It means the original developed didn't need the if/else and instead could have simply passed customer.someProperty as a parameter.

Henrick writes:

While on a simple assignment to help a customer migrate from .NET Framework to .NET core, I encountered this code. The 3 lines are unfortunately pretty representative for the codebase

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Myopic Focus

Chops was a developer for Initrode. Early on a Monday, they were summoned to their manager Gary's office before the caffeine had even hit their brain.

Gary glowered up from his office chair as Chops entered. This wasn't looking good. "We need to talk about the latest commit for Taskmaster."

Taskmaster was a large application that'd been around for decades, far longer than Chops had been an employee. Thousands of internal and external customers relied upon it. Refinements over time had led to remarkable stability, its typical uptime now measured in years. However, just last week, their local installation had unexpectedly suffered a significant crash. Chops had been assigned to troubleshooting and repair.

Looker Studio Marketing Dashboard Overview

"What's wrong?" Chops asked.

"Your latest commit decreased the number of unit tests!" Gary replied as if Chops had slashed the tires on his BMW.

Within Taskmaster, some objects that were periodically generated were given a unique ID from a pool. The pool was of limited size and required scanning to find a spare ID. Each time a value was needed, a search began where the last search ended. IDs returned to the pool as objects were destroyed would only be reused when the search wrapped back around to the start.

Chops had discovered a bug in the wrap-around logic that would inevitably produce a crash if Taskmaster ran long enough. They also found that if the number of objects created exceeded the size of the pool, this would trigger an infinite loop.

Rather than attempt to patch any of this, Chops had nuked the whole thing and replaced it with code that assigned each object a universally unique identifier (UUID) from a trusted library UUID generator within its constructor. Gone was the bad code, along with its associated unit tests.

Knowing they would probably only get in a handful of words, Chops wonderered how on earth to explain all this in a way that would appease their manager. "Well—"

"That number must NEVER go down!" Gary snapped.

"But—"

"This is non-negotiable! Roll it back and come up with something better!"

And so Chops had no choice but to remove their solution, put all the janky code back in place, and patch over it with kludge. Every comment left to future engineers contained a tone of apology.

Taskmaster became less stable. Time and expensive developer hours were wasted. Risk to internal and external customers increased. But Gary could rest assured, knowing that his favored metric never faltered on his watch.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

That Secret Service SIM Farm Story Is Bogus

By: Nick Heer

Robert Graham, clarifying the bad reporting of the big SIM farm bust in New York:

The Secret Service is lying to the press. They know it’s just a normal criminal SIM farm and are hyping it into some sort of national security or espionage threat. We know this because they are using the correct technical terms that demonstrate their understanding of typical SIM farm crimes. The claim that they will likely find other such SIM farms in other cities likewise shows they understand this is a normal criminal activity and not any special national security threat.

One of the things we must always keep in mind is that press releases are written to persuade. That is as true for businesses as it is for various government agencies. In this case, the Secret Service wanted attention, so they exaggerated the threat. And one wonders why public trust in institutions is falling.

⌥ Permalink

Google Provides Feedback on the Digital Markets Act

By: Nick Heer

Something I missed in posting about Apple’s critical appraisal of the Digital Markets Act is its timing. Why now? Well, it turns out the European Commission sought feedback beginning in July, and with a deadline of just before midnight on 24 September. That is why it published that statement, and why Google did the same.

Oliver Bethell, Google’s “senior director, competition”, a job title which implies a day spent chuckling to oneself:

Consider the DMA’s impact on Europe’s tourism industry. The DMA requires Google Search to stop showing useful travel results that link directly to airline and hotel sites, and instead show links to intermediary websites that charge for inclusion. This raises prices for consumers, reduces traffic to businesses, and makes it harder for people to quickly find reliable, direct booking information.

Key parts of the European tourism industry have already seen free, direct booking traffic from Google Search plummet by up to 30%. A recent study on the economic impact of the DMA estimates that European businesses across sectors could face revenue losses of up to €114 billion.

The study in question, though published by Copenhagen Business School, was funded by the Computer & Communications Industry Association, a tech industry lobbying firm funded in part by Google. I do not have the background to assess if the paper’s conclusions are well-founded, but it should be noted the low-end of the paper’s estimates was a loss of €8.5 billion, or just 0.05% of total industry revenue (page 45). The same lobbyists also funded a survey (PDF) conducted online by Nextrade Group.

Like Apple, Google clearly wants this law to go away. It might say it “remain[s] committed to complying with the DMA” and that it “appreciate[s] the Commission’s consistent openness to regulatory dialogue”, but nobody is fooled. To its credit, Google posted the full response (PDF) it sent the Commission which, though clearly defensive, has less of a public relations sheen than either of the company’s press releases.

⌥ Permalink

U.S. Federal Trade Commission Settles With Amazon Just Two Days Into Trial

By: Nick Heer

In 2023 Lina Khan, then-chair of the U.S. Federal Trade Commission, sued Amazon over using (PDF) “manipulative, coercive, or deceptive user-interface designs known as ‘dark patterns’ to trick consumers into enrolling in automatically-renewing Prime subscriptions” and “knowingly complicat[ing] the cancellation process”. Some people thought this case was a long-shot, or attempted to use Khan’s scholarship against her.

Earlier this week, the trial began to adjudicate the government’s claims which, in addition to accusing Amazon itself, also involved charges against company executives. It was looking promising for the FTC.

Annie Palmer, CNBC:

The FTC notched an early win in the case last week when U.S. District Court Judge John Chun ruled Amazon and two senior executives violated the Restore Online Shoppers’ Confidence Act by gathering Prime members’ billing information before disclosing the terms of the service.

Chun also said that the two senior Amazon executives would be individually liable if a jury sides with the FTC due to the level of oversight they maintained over the Prime enrollment and cancellation process.

Then, just two days into the trial, the FTC announced it had reached a settlement:

The Federal Trade Commission has secured a historic order with Amazon.com, Inc., as well as Senior Vice President Neil Lindsay and Vice President Jamil Ghani, settling allegations that Amazon enrolled millions of consumers in Prime subscriptions without their consent, and knowingly made it difficult for consumers to cancel. Amazon will be required to pay a $1 billion civil penalty, provide $1.5 billion in refunds back to consumers harmed by their deceptive Prime enrollment practices, and cease unlawful enrollment and cancellation practices for Prime.

As usual for settlements like these, Amazon will admit no wrongdoing. The executives will not face liability, something Adam Kovacevich, head of the Chamber of Progress, a tech industry lobbying group, said today was a “wild … theory” driven by “Khan’s ego”. Nonsense. The judge in the case, after saying Amazon broke the law, gave credence to the concept these executives were personally liable for the harm they were alleged to have caused.

Former FTC commissioner Alvaro Bedoya on X:

Based on my initial read, do the executives need to do anything separate from that? Do they pay any fines? Are they being demoted? Are they subject to extra monitoring? Do they need to admit any guilt whatsoever? The answers, as far as I can tell are no, no, no, no, and no. What’s worse, the order applies to the executives for only three years — seven years less than the company.

Two-and-a-half billion is a lot of dollars in the abstract. CIRP estimates there are 197 million U.S. subscribers to Amazon Prime, which costs anywhere from $7 to $15 per month. For the sake of argument, assume everyone is — on average — on the annual plan of $11.58 per month. It will take barely more than one billing cycle for Amazon to recoup that FTC settlement. The executives previously charged will bear little responsibility for this outcome.

Those million-dollar inauguration “investments”, as CBS News put it, sure are paying off.

⌥ Permalink

Privacy Commissioner of Canada Releases Findings of Investigation Into TikTok

By: Nick Heer

Catharine Tunney, CBC News:

The immensely popular social media app TikTok has been collecting sensitive information from hundreds of thousands of Canadians under 13 years old, a joint investigation by privacy authorities found.

[…]

The privacy commissioners said TikTok agreed to enhance its age verification and provide up-front notices about its wide-ranging collection of data.

Off the top, the Privacy Commissioner’s report was limited in scope and did not examine “perceived risks to national security” since they were not related to “privacy in the context of commercial activity” and have been adjudicated elsewhere. The results of national security reviews by other agencies have not been published. However, the Commissioner’s review of the company’s privacy practices is still comprehensive for what was in scope.

TikTok detects and removes about 500,000 accounts of Canadian children under 13 annually. Yet even though the company has dedicated significant engineering efforts to estimating users’ ages for advertising and to produce recommendations, it has not developed similar capabilities for restricting minors’ access.

Despite my skepticism of the Commissioner’s efficacy in cases like these, this investigation produced a number of results. TikTok made several changes as the investigation progressed, including restricting ad targeting to minors:

As an additional measure, in its response to the Offices’ Preliminary Report of Investigation, TikTok committed to limit ad targeting for users under 18 in Canada. TikTok informed the Offices that it implemented this change on April 1st, 2025. As a result, advertisers can no longer deliver targeted ads to users under 18, other than according to generic data (such as language and approximate location).

This is a restriction TikTok has in place for some regions, but not everywhere. It is not unique to TikTok, either; Meta and Google targeted minors, and Meta reportedly guessed teens’ emotional state for ad targeting purposes. This industry cannot police itself. All of these companies say they have rules against ad targeting to children and have done so for years, yet all of them have been found to ignore those rules when they are inconvenient.

⌥ Permalink

Apple Attempts to Rally Users Against E.U. Digital Markets Act

By: Nick Heer

Apple issued a press release criticizing the E.U.’s Digital Markets Act in a curious mix of countries. It published it on its European sites — of course — and in Australia, Canada, New Zealand, and the United States, all English-speaking. It also issued the same press release in Brazil, China, Guinea-Bissau, Indonesia, and Thailand — and a handful of other places — but not in Argentina, India, Japan, Mexico, or Singapore. Why this mix? Why did Apple bother to translate it into Thai but not Japanese? It is a fine mystery. Read into it what you will.

Anyway, you will be amazed to know how Apple now views the DMA:

It’s been more than a year since the Digital Markets Act was implemented. Over that time, it’s become clear that the DMA is leading to a worse experience for Apple users in the EU. It’s exposing them to new risks, and disrupting the simple, seamless way their Apple products work together. And as new technologies come out, our European users’ Apple products will only fall further behind.

[…]

That’s why we’re urging regulators to take a closer look at how the law is affecting the EU citizens who use Apple products every day. We believe our users in Europe deserve the best experience on our technology, at the same standard we provide in the rest of the world — and that’s what we’ll keep fighting to deliver.

It thinks the DMA should disappear.

Its reasoning is not great; Michael Tsai read the company’s feature delays more closely and is not convinced. One of the delayed features is Live Translation, about which I wrote:

This is kind of a funny limitation because fully half the languages Live Translation works with — French, German, and Spanish — are the versions spoken in their respective E.U. countries and not, for example, Canadian French or Chilean Spanish. […]

Because of its launch languages, I think Apple expects this holdup will not last for long.

I did not account for a cynical option: Apple is launching with these languages as leverage.

The way I read Apple’s press release is as a fundamental disagreement between the role each party believes it should play, particularly when it comes to user privacy. Apple seems to believe it is its responsibility to implement technical controls to fulfill its definition of privacy and, if that impacts competition and compatibility, too bad. E.U. regulators seem to believe it has policy protections for user privacy, and that users should get to decide how their private data is shared.

Adam Engst, TidBits:

Apple’s claim of “the same standard we provide in the rest of the world” rings somewhat hollow, given that it often adjusts its technology and services to comply with local laws. The company has made significant concessions to operate in China, doesn’t offer FaceTime in the United Arab Emirates, and removes apps from the still-functional Russian App Store at the Russian government’s request. Apple likely pushed back in less public ways in those countries, but in the EU, this public statement appears aimed at rallying its users and influencing the regulatory conversation.

I know what Engst is saying here, and I agree with the sentiment, but this is a bad group of countries to be lumped in together with. That does not mean the DMA is equal to the kinds of policies that restrict services in these other countries. It remains noteworthy how strict Apple is in restricting DMA-mandated features only to countries where they are required, but you can just change your region to work around the UAE FaceTime block.

⌥ Permalink

Australian Opposition Parties Encourage Migration to Forthcoming U.S. Version of TikTok

By: Nick Heer

Oscar Godsell, Sky News:

The opposition’s shadow finance minister James Paterson has since urged the Australian Labor government to follow suit.

Mr Paterson told Sky News if the US was able to create a “safer version” of TikTok, then Australia should liaise with the Trump administration to become part of that solution.

“It would be an unfortunate thing if there was a safe version of TikTok in the United States, but a version of TikTok in Australia which was still controlled by a foreign authoritarian government,” he said.

I am not sure people in Australia are asking for yet more of the country’s media to be under the thumb of Rupert Murdoch. Then again, I also do not think the world needs more social media platforms controlled by the United States, though that is very clearly the wedge the U.S. government is creating: countries can accept the existing version of TikTok, adopt the new U.S.-approved one, or ban them both. The U.S. spinoff does not resolve user privacy problems and it raises new concerns about the goals of its government-friendly ownership and management.

⌥ Permalink

Patreon Will Automatically Enable Audience Irritation Features Next Week

By: Nick Heer

Do you manage a Patreon page as a “creator”? I do; it is where you can give me five dollars per month to add to my guilt over not finishing my thoughts about Liquid Glass.1 You do not have to give me five dollars. I feel guilty enough as it is.

Anyway, you might have missed an email Patreon sent today advising you that Autopilot will be switched on beginning October 1 unless you manually turn it off. According to Patreon’s email:

Autopilot is a growth engine that automatically sends your members and potential members strategic, timely offers which encourage them to join, upgrade, or retain your membership — without you having to lift a finger.

As an extremely casual user, I do not love this; I think it is basically spam. I am sympathetic toward those who make their living with Patreon. I turned this off. If you have a Patreon creator page and missed this email, now you know.

And if you are a subscriber to anyone on Patreon and begin receiving begging emails next week, please be gracious. They might not be aware this feature was switched on.


  1. I am most looking forward to reading others’ reviews when I am done, which I have so far avoided so my own piece is not tainted. ↥︎

⌥ Permalink

❌