I have at least a few readers for which the sound of a man's voice saying "government cell phone detected" will elicit a palpable reaction. In Department of Energy facilities across the country, incidences of employees accidentally carrying phones into secure areas are reduced through a sort of automated nagging. A device at the door monitors for the presence of a tag; when the tag is detected it plays an audio clip. Because this is the government, the device in question is highly specialized, fantastically expensive, and says "government cell phone" even though most of the phones in question are personal devices. Look, they already did the recording, they're not changing it now!

One of the things that I love is weird little wireless networks. Long ago I wrote about ANT+, for example, a failed personal area network standard designed mostly around fitness applications. There's tons of these, and they have a lot of similarities---so it's fun to think about the protocols that went down a completely different path. It's even better, of course, if the protocol is obscure outside of an important niche. And a terrible website, too? What more could I ask for.

The DoE's cell-phone nagging boxes, and an array of related but more critical applications, rely on an unusual personal area networking protocol called RuBee.

RuBee is a product of Visible Assets Inc., or VAI, founded in 2004 ¹ by John K. Stevens. Stevens seems a somewhat improbable founder, with a background in biophysics and eye health, but he's a repeat entrepreneur. He's particularly fond of companies called Visible: he founded Visible Assets after his successful tenure as CEO of Visible Genetics. Visible Genetics was an early innovator in DNA sequencing, and still provides a specialty laboratory service that sequences samples of HIV in order to detect vulnerabilities to antiretroviral medications.

Clinical trials in the early 2000s exposed Visible Genetics to one of the more frustrating parts of health care logistics: refrigeration. Samples being shipped to the lab and reagents shipped out to clinics were both temperature sensitive. Providers had to verify that these materials had stayed adequately cold throughout shipping and handling, otherwise laboratory results could be invalid or incorrect. Stevens became interested in technical solutions to these problems; he wanted some way to verify that samples were at acceptable temperatures both in storage and in transit.

Moreover, Stevens imagined that these sensors would be in continuous communication. There's a lot of overlap between this application and personal area networks (PANs), protocols like Bluetooth that provide low-power communications over short ranges. There is also clear overlap with RFID; you can buy RFID temperature sensors. VAI, though, coined the term visibility network to describe RuBee. That's visibility as in asset visibility: somewhat different from Bluetooth or RFID, RuBee as a protocol is explicitly designed for situations where you need to "keep tabs" on a number of different objects. Despite the overlap with other types of wireless communications, the set of requirements on a visibility network have lead RuBee down a very different technical path.

Visibility networks have to be highly reliable. When you are trying to keep track of an asset, a failure to communicate with it represents a fundamental failure of the system. For visibility networks, the ability to actually convey a payload is secondary: the main function is just reliably detecting that endpoints exist. Visibility networks have this in common with RFID, and indeed, despite its similarities to technologies like BLE RuBee is positioned mostly as a competitor to technologies like UHF RFID.

There are several differences between RuBee and RFID; for example, RuBee uses active (battery-powered) tags and the tags are generally powered by a complete 4-bit microcontroller. That doesn't necessarily sound like an advantage, though. While RuBee tags advertise a battery life of "5-25 years", the need for a battery seems mostly like a liability. The real feature is what active tags enable: RuBee operates in the low frequency (LF) band, typically at 131 kHz.

At that low frequency, the wavelength is very long, about 2.5 km. With such a long wavelength, RuBee communications all happen at much less than one wavelength in range. RF engineers refer to this as near-field operation, and it has some properties that are intriguingly different from more typical far-field RF communications. In the near-field, the magnetic field created by the antenna is more significant than the electrical field. RuBee devices are intentionally designed to emit very little electrical RF signal. Communications within a RuBee network are achieved through magnetic, not electrical fields. That's the core of RuBee's magic.

The idea of magnetic coupling is not unique to RuBee. Speaking of the near-field, there's an obvious comparison to NFC which works much the same way. The main difference, besides the very different logical protocols, is that NFC operates at 13.56 MHz. At this higher frequency, the wavelength is only around 20 meters. The requirement that near-field devices be much closer than a full wavelength leads naturally to NFC's very short range, typically specified as 4 cm.

At LF frequencies, RuBee can achieve magnetic coupling at ranges up to about 30 meters. That's a range comparable to, and often much better than, RFID inventory tracking technologies. Improved range isn't RuBee's only benefit over RFID. The properties of magnetic fields also make it a more robust protocol. RuBee promises significantly less vulnerability to shielding by metal or water than RFID.

There are two key scenarios where this comes up: the first is equipment stored in metal containers or on metal shelves, or equipment that is itself metallic. In that scenario, it's difficult to find a location for an RFID tag that won't suffer from shielding by the container. The case of water might seem less important, but keep in mind that people are made mostly of water. RFID reading is often unreliable for objects carried on a person, which are likely to be shielded from the reader by the water content of the body.

These problems are not just theoretical. WalMart is a major adopter of RFID inventory technology, and in early rollouts struggled with low successful read rates. Metal, moisture (including damp cardboard boxes), antenna orientation, and multipath/interference effects could cause read failure rates as high as 33% when scanning a pallet of goods. Low read rates are mostly addressed by using RFID "portals" with multiple antennas. Eight antennas used as an array greatly increase read rate, but at a cost of over ten thousand dollars per portal system. Even so, WalMart seems to now target a success rate of only 95% during bulk scanning.

95% might sound pretty good, but there are a lot of visibility applications where a failure rate of even a couple percent is unacceptable. These mostly go by the euphemism "high value goods," which depending on your career trajectory you may have encountered in corporate expense and property policies. High-value goods tend to be items that are both attractive to theft and where theft has particularly severe consequences. Classically, firearms and explosives. Throw in classified material for good measure.

I wonder if Stevens was surprised by RuBee's market trajectory. He came out of the healthcare industry and, it seems, originally developed RuBee for cold chain visibility... but, at least in retrospect, it's quite obvious that its most compelling application is in the armory.

Because RuBee tags are small and largely immune to shielding by metals, you can embed them directly in the frames of firearms, or as an aftermarket modification you can mill out some space under the grip. RuBee tags in weapons will read reliably when they are stored in metal cases or on metal shelving, as is often the case. They will even read reliably when a weapon is carried holstered, close to a person's body.

Since RuBee tags incorporate an active microcontroller, there are even more possibilities. Temperature logging is one thing, but firearm-embedded RuBee tags can incorporate an accelerometer (NIST-traceable, VAI likes to emphasize) and actually count the rounds fired.

Sidebar time: there is a long history of political hazard around "smart guns." The term "smart gun" is mostly used more specifically for firearms that identify their user, for example by fingerprint authentication or detection of an RFID fob. The idea has become vague enough, though, that mention of a firearm with any type of RFID technology embedded would probably raise the specter of the smart gun to gun-rights advocates.

Further, devices embedded in firearms that count the number of rounds fired have been proposed for decades, if not a century, as a means of accountability. The holder of a weapon could, in theory, be required to positively account for every round fired. That could eliminate incidents of unreported use of force by police, for example. In practice I think this is less compelling than it sounds, simple counting of rounds leaves too many opportunities to fudge the numbers and conceal real-world use of a weapon as range training, for example.

That said, the NRA has long been vehemently opposed to the incorporation of any sort of technology into weapons that could potentially be used as a means of state control or regulation. The concern isn't completely unfounded; the state of New Jersey did, for a time, have legislation that would have made user-identifying "smart guns" mandatory if they were commercially available. The result of the NRA's strident lobbying is that no such gun has ever become commercially available; "smart guns" have been such a political third rail that any firearms manufacturer that dared to introduce one would probably face a boycott by most gun stores. For better or worse, a result of the NRA's powerful political advocacy in this area is that the concept of embedding security or accountability technology into weapons has never been seriously pursued in the US. Even a tentative step in that direction can produce a huge volume of critical press for everyone involved.

I bring this up because I think it explains some of why VAI seems a bit vague and cagey about the round-counting capabilities of their tags. They position it as purely a maintenance feature, allowing the armorer to keep accurate tabs on the preventative maintenance schedule for each individual weapon (in armory environments, firearm users are often expected to report how many rounds they fired for maintenance tracking reasons). The resistance of RuBee tags to concealment is only positioned as a deterrent to theft, although the idea of RuBee-tagged firearms creates obvious potential for security screening. Probably the most profitable option for VAI would be to promote RuBee-tagged firearms as tool for enforcement of gun control laws, but this is a political impossibility and bringing it up at all could cause significant reputational harm, especially with the government as a key customer. The result is marketing copy that is a bit odd, giving a set of capabilities that imply an application that is never mentioned.

VAI found an incredible niche with their arms-tracking application. Institutional users of firearms, like the military, police, and security forces, are relatively price-insensitive and may have strict accounting requirements. By the mid-'00s, VAI was into the long sales cycle of proposing the technology to the military. That wasn't entirely unsuccessful. RuBee shot-counting weapon inventory tags were selected by the Naval Surface Warfare Center in 2010 for installation on SCAR and M4 rifles. That contract had a five-year term, it's unclear to me if it was renewed. Military contracting opened quite a few doors to VAI, though, and created a commercial opportunity that they eagerly pursued.

Perhaps most importantly, weapons applications required an impressive round of safety and compatibility testing. RuBee tags have the fairly unique distinction of military approval for direct attachment to ordnance, something called "zero separation distance" as the tags do not require a minimum separation from high explosives. Central to that certification are findings of intrinsic safety of the tags (that they do not contain enough energy to trigger explosives) and that the magnetic fields involved cannot convey enough energy to heat anything to dangerous temperatures.

That's not the only special certification that RuBee would acquire. The military has a lot of firearms, but military procurement is infamously slow and mercurial. Improved weapon accountability is, almost notoriously, not a priority for the US military which has often had stolen weapons go undetected until their later use in crime. The Navy's interest in RuBee does not seem to have translated to more widespread military applications.

Then you have police departments, probably the largest institutional owners of firearms and a very lucrative market for technology vendors. But here we run into the political hazard: the firearms lobby is very influential on police departments, as are police unions which generally oppose technical accountability measures. Besides, most police departments are fairly cash-poor and are not likely to make a major investment in a firearms inventory system.

That leaves us with institutional security forces. And there is one category of security force that are particularly well-funded, well-equipped, and beholden to highly R&D-driven, almost pedantic standards of performance: the protection forces of atomic energy facilities.

Protection forces at privately-operated atomic energy facilities, such as civilian nuclear power plants, are subject to licensing and scrutiny by the Nuclear Regulatory Commission. Things step up further at the many facilities operated by the National Nuclear Security Administration (NNSA). Protection forces for NNSA facilities are trained at the Department of Energy's National Training Center, at the former Manzano Base here in Albuquerque. Concern over adequate physical protection of NNSA facilities has lead Sandia National Laboratories to become one of the premier centers for R&D in physical security. Teams of scientists and engineers have applied sometimes comical scientific rigor to "guns, gates, and guards," the traditional articulation of physical security in the nuclear world.

That scope includes the evaluation of new technology for the management of protection forces, which is why Oak Ridge National Laboratory launched an evaluation program for the RuBee tagging of firearms in their armory. The white paper on this evaluation is curiously undated, but citations "retrieved 2008" lead me to assume that the evaluation happened right around the middle of the '00s. At the time, VAI seems to have been involved in some ultimately unsuccessful partnership with Oracle, leading to the branding of the RuBee system as Oracle Dot-Tag Server. The term "Dot-Tag" never occurs outside of very limited materials around the Oracle partnership, so I'm not sure if it was Oracle branding for RuBee or just some passing lark. In any case, Oracle's involvement seems to have mainly just been the use of the Oracle database for tracking inventory data---which was naturally replaced by PostgreSQL at Oak Ridge.

The Oak Ridge trial apparently went well enough, and around the same time, the Pantex Plant in Texas launched an evaluation of RuBee for tracking classified tools. Classified tools are a tricky category, as they're often metallic and often stored in metallic cases. During the trial period, Pantex tagged a set of sample classified tools with RuBee tags and then transported them around the property, testing the ability of the RuBee controllers to reliably detect them entering and exiting areas of buildings. Simultaneously, Pantex evaluated the use of RuBee tags to track containers of "chemical products" through the manufacturing lifecycle. Both seem to have produced positive results.

There are quite a few interesting and strange aspects of the RuBee system, a result of its purpose-built Visibility Network nature. A RuBee controller can have multiple antennas that it cycles through. RuBee tags remain in a deep-sleep mode for power savings until they detect a RuBee carrier during their periodic wake cycle. When a carrier is detected, they fully wake and listen for traffic. A RuBee controller can send an interrogate message and any number of tags can respond, with an interesting and novel collision detection algorithm used to ensure reliable reading of a large number of tags.

The actual RuBee protocol is quite simple, and can also be referred to as IEEE 1902.1 since the decision of VAI to put it through the standards process. Packets are small and contain basic addressing info, but they can also contain arbitrary payload in both directions, perfect for data loggers or sensors. RuBee tags are identified by something that VAI oddly refers to as an "IP address," causing some confusion over whether or not VAI uses IP over 1902.1. They don't, I am confident saying after reading a whole lot of documents. RuBee tags, as standard, have three different 4-byte addresses. VAI refers to these as "IP, subnet, and MAC," ² but these names are more like analogies. Really, the "IP address" and "subnet" are both configurable arbitrary addresses, with the former intended for unicast traffic and the latter for broadcast. For example, you would likely give each asset a unique IP address, and use subnet addresses for categories or item types. The subnet address allows a controller to interrogate for every item within that category at once. The MAC address is a fixed, non-configurable address derived from the tag's serial number. They're all written in the formats we associate with IP networks, dotted-quad notation, as a matter of convenience.

And that's about it as far as the protocol specification, besides of course the physical details which are a 131,072 Hz carrier, 1024 Hz data clock, either ASK or BPSK modulation. The specification also describes an interesting mode called "clip," in which a set of multiple controllers interrogate in exact synchronization and all tags then reply in exact synchronization. Somewhat counter-intuitively, because of the ability of RuBee controllers to separate out multiple simultaneous tag transmissions using an anti-collision algorithm based on random phase shifts by each tag, this is ideal. It allows a room, say an armory, full of RuBee controllers to rapidly interrogate the entire contents of the room. I think this feature may have been added after the Oak Ridge trials...

RuBee is quite slow, typically 1,200 baud, so inventorying a large number of assets can take a while (Oak Ridge found that their system could only collect data on 2-7 tags per second per controller). But it's so robust that it an achieve a 100% read rate in some very challenging scenarios. Evaluation by the DoE and the military produced impressive results. You can read, for example, of a military experiment in which a RuBee antenna embedded in a roadway reliably identified rifles secured in steel containers in passing Humvees.

Paradoxically, then, one of the benefits of RuBee in the military/defense context is that it is also difficult to receive. Here is RuBee's most interesting trick: somewhat oversimplified, the strength of an electrical radio signal goes as 1/r, while the strength of a magnetic field goes as 1/r^3. RuBee equipment is optimized, by antenna design, to produce a minimal electrical field. The result is that RuBee tags can very reliably be contacted at short range (say, around ten feet), but are virtually impossible to contact or even detect at ranges over a few hundred feet. To the security-conscious buyer, this is a huge feature. RuBee tags are highly resistant to communications or electronic intelligence collection.

Consider the logical implications of tagging the military's rifles. With conventional RFID, range is limited by the size and sensitivity of the antenna. Particularly when tags are incidentally powered by a nearby reader, an adversary with good equipment can detect RFID tags at very long range. VAI heavily references a 2010 DEFCON presentation, for example, that demonstrated detection of RFID tags at a range of 80 miles. One imagines that opportunistic detection by satellite is feasible for a state intelligence agency. That means that your rifle asset tracking is also revealing the movements of soldiers in the field, or at least providing a way to detect their approach.

Most RuBee tags have their transmit power reduced by configuration, so even the maximum 100' range of the protocol is not achievable. VAI suggests that typical RuBee tags cannot be detected by radio direction finding equipment at ranges beyond 20', and that this range can be made shorter by further reducing transmit power.

Once again, we have caught the attention of the Department of Energy. Because of the short range of RuBee tags, they have generally been approved as not representing a COMSEC or TEMPEST hazard to secure facilities. And that brings us back to the very beginning: why does the DoE use a specialized, technically interesting, and largely unique radio protocol to fulfill such a basic function as nagging people that have their phones? Because RuBee's security properties have allowed it to be approved for use adjacent to and inside of secure facilities. A RuBee tag, it is thought, cannot be turned into a listening device because the intrinsic range limitation of magnetic coupling will make it impossible to communicate with the tag from outside of the building. It's a lot like how infrared microphones still see some use in secure facilities, but so much more interesting!

VAI has built several different product lines around RuBee, with names like Armory 20/20 and Shot Counting Allegro 20/20 and Store 20/20. The founder started his career in eye health, remember. None of them are that interesting, though. They're all pretty basic CRUD applications built around polling multiple RuBee controllers for tags in their presence.

And then there's the "Alert 20/20 DoorGuard:" a metal pedestal with a RuBee controller and audio announcement module, perfect for detecting government cell phones.

One of the strangest things about RuBee is that it's hard to tell if it's still a going concern. VAI's website has a press release section, where nothing has been posted since 2019. The whole website feels like it was last revised even longer ago. When RuBee was newer, back in the '00s, a lot of industry journals covered it with headlines like "the new RFID." I think VAI was optimistic that RuBee could displace all kinds of asset tracking applications, but despite some special certifications in other fields (e.g. approval to use RuBee controllers and tags around pacemakers in surgical suites), I don't think RuBee has found much success outside of military applications.

RuBee's resistance to shielding is impressive, but RFID read rates have improved considerably with new DSP techniques, antenna array designs, and the generally reduced cost of modern RFID equipment. RuBee's unique advantages, its security properties and resistance to even intentional exfiltration, are interesting but not worth much money to buyers other than the military.

So that's the fate of RuBee and VAI: defense contracting. As far as I can tell, RuBee and VAI are about as vital as they have ever been, but RuBee is now installed as just one part of general defense contracts around weapons systems, armory management, and process safety and security. IEEE standardization has opened the door to use of RuBee by federal contractors under license, and indeed, Lockheed Martin is repeatedly named as a licensee, as are firearms manufacturers with military contracts like Sig Sauer.

Besides, RuBee continues to grow closer to the DoE. In 2021, VAI appointed Lisa Gordon-Hagerty to it board of directors. Gordon-Hagerty was undersecretary of Energy and had lead the NNSA until the year before. This year, the New Hampshire Small Business Development Center wrote a glowing profile of VAI. They described it as a 25-employee company with a goal of hitting $30 million in annual revenue in the next two years.

Despite the outdated website, VAI claims over 1,200 RuBee sites in service. I wonder how many of those are Alert 20/20 DoorGuards? Still, I do believe there are military weapons inventory systems currently in use. RuBee probably has a bright future, as a niche technology for a niche industry. If nothing else, they have legacy installations and intellectual property to lean on. A spreadsheet of VAI-owned patents on RuBee, with nearly 200 rows, encourages would-be magnetically coupled visibility network inventors not to go it on their own. I just wish I could get my hands on a controller....

Scared Stanley stammered "I'm afraid of how to explain to the tax authority that I received $NaN."

Our anonymous friend Anon E. Mous wrote "I went to look up some employee benefits stuff up and ... This isn't a good sign."

Regular Michael R. is not actually operating under an alias, but this (allegedly scamming?) site doesn't know.

Graham F. gloated "I'm glad my child 's school have followed our naming convention for their form groups as well!"

Adam R. is taking his anonymous children on a roadtrip to look for America. "I'm planning a trip to St. Louis. While trying to buy tickets for the Gateway Arch, I noticed that their ticketing website apparently doesn't know how to define adults or children (or any of the other categories of tickets, for that matter)."

It feels like ages ago, when document databases like Mongo were all the rage. That isn't to say that they haven't stuck around and don't deliver value, but gone is the faddish "RDBMSes are dead, bro." The "advantage" they offer is that they turn data management problems into serialization problems.

And that's where today's anonymous submission takes us. Our submitter has a long list of bugs around managing lists of usernames. These bugs largely exist because the contract developer who wrote the code didn't write anything, and instead "vibe coded too close to the sun", according to our submitter.

Here's the offending C# code:

   [JsonPropertyName("invitedTraders")]
   [BsonElement("invitedTraders")]
   [BsonIgnoreIfNull]
   public InvitedTradersV2? InvitedTraders { get; set; }

   [JsonPropertyName("invitedTradersV2")]
   [BsonElement("invitedTradersV2")]
   [BsonIgnoreIfNull]
   public List<string>? InvitedTradersV2 { get; set; }

Let's start with the type InvitedTradersV2. This type contains a list of strings which represent usernames. The field InvitedTradersV2 is a list of strings which represent usernames. Half of our submitter's bugs exist simply because these two lists get out of sync- they should contain the same data, but without someone enforcing that correctly, problems accrue.

This is made more frustrating by the MongoDB attribute, BsonIgnoreIfNull, which simply means that the serialized object won't contain the key if the value is null. But that means the consuming application doesn't know which key it should check.

For the final bonus fun, note the use of JsonPropertyName. This comes from the built-in class library, which tells .NET how to serialize the object to JSON. The problem here is that this application doesn't use the built-in serializer, and instead uses Newtonsoft.JSON, a popular third-party library for solving the problem. While Newtonsoft does recognize some built-in attributes for serialization, JsonPropertyName is not among them. This means that property does nothing in this example, aside from add some confusion to the code base.

I suspect the developer responsible, if they even read this code, decided that the duplicated data was okay, because isn't that just a normal consequence of denormalization? And document databases are all about denormalization. It makes your queries faster, bro. Just one more shard, bro.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

On Thanksgiving Day, Ellis had cuddled up with her sleeping cat on the couch to send holiday greetings to friends. There in her inbox, lurking between several well wishes, was an email from an unrecognized sender with the subject line, Final Account Statement. Upon opening it, she read the following:

Dear Ellis,

Your final account statement dated -1 has been sent to you. Please log into your portal and review your balance due totaling #TOTAL_CHARGES#.

Payment must be received within 30 days of this notice to avoid collection. You may submit payment online via [Payment Portal Link] or by mail to:

Chamberlin Apartments
123 Main Street
Anytown US 12345

If you believe there is an error on your account, please contact us immediately at 212-555-1212.

Thank you for your prompt attention to this matter.

Chamberlin Apartments

Ellis had indeed rented an apartment managed by this company, but had moved out 16 years earlier. She'd never been late with a payment for anything in her life. What a time to receive such a thing, at the start of a long holiday weekend when no one would be able to do anything about it for the next 4 days!

She truly had so much to be grateful for that Thanksgiving, and here was yet more for her list: her broad technical knowledge, her experience working in multiple IT domains, and her many years of writing up just these sorts of stories for The Daily WTF. All of this added up to her laughing instead of panicking. She could just imagine the poor intern who'd hit "Send" by mistake. She also imagined she wasn't the only person who'd received this message. Rightfully scared and angry callers would soon be hammering that phone number, and Ellis was further grateful that she wasn't the one who had to pick up.

"I'll wait for the apology email!" she said out loud with a knowing smile on her face, closing out the browser tab.

Ellis moved on physically and mentally, going forward with her planned Thanksgiving festivities without giving it another thought. The next morning, she checked her inbox with curious anticipation. Had there been a retraction, a please disregard?

No. Instead, there were still more emails from the same sender. The second, sent 7 hours after the first, bore the subject line Second Notice - Outstanding Final Balance:

Dear Ellis,

Our records show that your final balance of #TOTAL_CHARGES# from your residency at your previous residence remains unpaid.

This is your second notice. Please remit payment in full or contact us to discuss the balance to prevent your account from being sent to collections.

Failure to resolve the balance within the next 15 days may result in your account being referred to a third-party collections agency, which could impact your credit rating.

To make payment or discuss your account, please contact us at 212-555-1212 or accounting@chamapts.com.

Sincerely,

Chamberlin Apartments

The third, sent 6 and a half hours later, threatened Final Notice - Account Will Be Sent to Collections.

Dear Ellis,

Despite previous notices, your final account balance remains unpaid.

This email serves as final notice before your account is forwarded to a third-party collections agency for recovery. Once transferred, we will no longer be able to accept payment directly or discuss the account.

To prevent this, payment of #TOTAL_CHARGES# must be paid in full by #CRITICALDATE#.

Please submit payment immediately. Please contact 212-555-1212 to confirm your payment.

Sincerely,

Chamberlin Apartments

It was almost certainly a mistake, but still rather spooky to someone who'd never been in such a situation. There was solace in the thought that, if they really did try to force Ellis to pay #TOTAL_CHARGES# on the basis of these messages, anyone would find it absurd that all 3 notices were sent mere hours apart, on a holiday no less. The first two had also mentioned 30 and 15 days to pay up, respectively.

Suddenly remembering that she probably wasn't the only recipient of these obvious form emails, Ellis thought to check her local subreddit. Sure enough, there was already a post revealing the range of panic and bewilderment they had wrought among hundreds, if not thousands. Current and more recent former tenants had actually seen #TOTAL_CHARGES# populated with the correct amount of monthly rent. People feared everything from phishing attempts to security breaches.

It wasn't until later that afternoon that Ellis finally received the anticipated mea culpa:

We are reaching out to sincerely apologize for the incorrect collection emails you received. These messages were sent in error due to a system malfunction that released draft messages to our entire database.

Please be assured of the following:
The recent emails do not reflect your actual account status.
If your account does have an outstanding balance, that status has not changed, and you would have already received direct and accurate communication from our office.
Please disregard all three messages sent in error. They do not require any action from you.

We understand that receiving these messages, especially over a holiday, was upsetting and confusing, and we are truly sorry for the stress this caused. The issue has now been fully resolved, and our team has worked with our software provider to stop all queued messages and ensure this does not happen again.

If you have any questions or concerns, please feel free to email leasing@chamapts.com. Thank you for your patience and understanding.

All's well that ends well. Ellis thanked the software provider's "system malfunction," whoever or whatever it may've been, that had granted the rest of us a bit of holiday magic to take forward for all time.

Darren is supporting a Delphi application in the current decade. Which is certainly a situation to be in. He writes:

I keep trying to get out of doing maintenance on legacy Delphi applications, but they keep pulling me back in.

The bit of code Darren sends us isn't the largest WTF, but it's a funny mistake, and it's a funny mistake that's been sitting in the codebase for decades at this point. And as we all know, jokes only get funnier with age.

FileName := DestDir + ExtractFileName(FileName);
if FileExists(DestDir + ExtractFileName(FileName)) then
begin
  ...
end;

This code is inside of a module that copies a file from a remote server to the local host. It starts by sanitizing the FileName, using ExtractFileName to strip off any path components, and replace them with DestDir, storing the result in the FileName variable.

And they liked doing that so much, they go ahead and do it again in the if statement, repeating the exact same process.

Darren writes:

As Homer Simpson said "Lather, rinse, and repeat. Always repeat."

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Remy's Law of Requirements Gathering states "No matter what the requirements document says, what your users really wanted was Excel." This has a corrolary: "Any sufficiently advanced Excel file is indistingushable from software."

Given enough time, any Excel file whipped up by any user can transition from "useful" to "mission critical software" before anyone notices. That's why Nemecsek was tasked with taking a pile of Excel spreadsheets and converting them into "real" software, which could be maintained and supported by software engineers.

Nemecsek writes:

This is just one of the formulas they asked me to work on, and not the longest one.

Nemecsek says this is a "formula", but I suspect it's a VBA macro. In reality, it doesn't matter.

InitechNeoDTMachineDevice.InitechNeoDTActivePartContainer(0).InitechNeoDTActivePart(0).
InitechNeoDTActivePartPartContainer(0).InitechNeoDTActivePartPart(iPart).Losses = 
calcLossesInPart(InitechNeoDTMachineDevice.InitechNeoDTActivePartContainer(0).
InitechNeoDTActivePart(0).RatedFrequency, InitechNeoDTMachineDevice.
InitechNeoDTActivePartContainer(0).InitechNeoDTActivePart(0).InitechNeoDTActivePartPartContainer(0).
InitechNeoDTActivePartPart(iPart).RadialPositionToMainDuct, InitechNeoDTMachineDevice.
InitechNeoDTActivePartContainer(0).InitechNeoDTActivePart(0).InitechNeoDTActivePartPartContainer(0).
InitechNeoDTActivePartPart(iPart).InitechNeoDTActivePartPartSectionContainer(0).
InitechNeoDTActivePartPartSection(0).InitechNeoDTActivePartPartConductorComposition(0).IsTransposed, 
InitechNeoDTMachineDevice.InitechNeoDTActivePartContainer(0).InitechNeoDTActivePart(0).
InitechNeoDTActivePartPartContainer(0).InitechNeoDTActivePartPart(iPart).
InitechNeoDTActivePartPartSectionContainer(0).InitechNeoDTActivePartPartSection(0).
InitechNeoDTActivePartPartConductorComposition(0).ParallelRadialCount, InitechNeoDTMachineDevice.
InitechNeoDTActivePartContainer(0).InitechNeoDTActivePart(0).InitechNeoDTActivePartPartContainer(0).
InitechNeoDTActivePartPart(iPart).InitechNeoDTActivePartPartSectionContainer(0).
InitechNeoDTActivePartPartSection(0).InitechNeoDTActivePartPartConductorComposition(0).
ParallelAxialCount, InitechNeoDTMachineDevice.InitechNeoDTActivePartContainer(0).
InitechNeoDTActivePart(0).InitechNeoDTActivePartPartContainer(0).InitechNeoDTActivePartPart(iPart).
InitechNeoDTActivePartPartSectionContainer(0).InitechNeoDTActivePartPartSection(0).
InitechNeoDTActivePartPartConductorComposition(0).InitechNeoDTActivePartPartConductor(0).Type, 
InitechNeoDTMachineDevice.InitechNeoDTActivePartContainer(0).InitechNeoDTActivePart(0).
InitechNeoDTActivePartPartContainer(0).InitechNeoDTActivePartPart(iPart).
InitechNeoDTActivePartPartSectionContainer(0).InitechNeoDTActivePartPartSection(0).
InitechNeoDTActivePartPartConductorComposition(0).InitechNeoDTActivePartPartConductor(0).
DimensionRadialElectric, InitechNeoDTMachineDevice.InitechNeoDTActivePartContainer(0).
InitechNeoDTActivePart(0).InitechNeoDTActivePartPartContainer(0).InitechNeoDTActivePartPart(iPart).
InitechNeoDTActivePartPartSectionContainer(0).InitechNeoDTActivePartPartSection(0).
InitechNeoDTActivePartPartConductorComposition(0).InitechNeoDTActivePartPartConductor(0).
DimensionAxialElectric + InitechNeoDTMachineDevice.InitechNeoDTActivePartContainer(0).
InitechNeoDTActivePart(0).InitechNeoDTActivePartPartContainer(0).InitechNeoDTActivePartPart(iPart).
InitechNeoDTActivePartPartSectionContainer(0).InitechNeoDTActivePartPartSection(0).
InitechNeoDTActivePartPartConductorComposition(0).InitechNeoDTActivePartPartConductor(0).InsulThickness, 
getElectricConductivityAtTemperatureT1(InitechNeoDTMachineDevice.InitechNeoDTActivePartContainer(0).
InitechNeoDTActivePart(0).InitechNeoDTActivePartPartContainer(0).InitechNeoDTActivePartPart(iPart).
InitechNeoDTActivePartPartSectionContainer(0).InitechNeoDTActivePartPartSection(0).
InitechNeoDTActivePartPartConductorComposition(0).InitechNeoDTActivePartPartConductor(0).
InitechNeoDTActivePartPartConductorRawMaterial(0).ElectricConductivityT0, InitechNeoDTMachineDevice.
InitechNeoDTActivePartContainer(0).InitechNeoDTActivePart(0).InitechNeoDTActivePartPartContainer(0).
InitechNeoDTActivePartPart(iPart).InitechNeoDTActivePartPartSectionContainer(0).
InitechNeoDTActivePartPartSection(0).InitechNeoDTActivePartPartConductorComposition(0).
InitechNeoDTActivePartPartConductor(0).InitechNeoDTActivePartPartConductorRawMaterial(0).MaterialFactor, 
InitechNeoDTMachineDevice.InitechNeoDTActivePartContainer(0).InitechNeoDTActivePart(0).
InitechNeoDTActivePartPartContainer(0).InitechNeoDTActivePartPart(iPart).
InitechNeoDTActivePartPartSectionContainer(0).InitechNeoDTActivePartPartSection(0).
InitechNeoDTActivePartPartConductorComposition(0).InitechNeoDTActivePartPartConductor(0).
InitechNeoDTActivePartPartConductorRawMaterial(0).ReferenceTemperatureT0, InitechNeoDTMachineDevice.
ReferenceTemperature), LayerNumberRatedVoltage, InitechNeoDTMachineDevice.InitechNeoDTActivePartContainer(0).
InitechNeoDTActivePart(0).InitechNeoDTActivePartPartContainer(0).InitechNeoDTActivePartPart(iPart).
InitechNeoDTActivePartPartLayerContainer(0),InitechNeoDTMachineDevice.InitechNeoDTActivePartContainer(0).
InitechNeoDTActivePart(0).RFactor)

Line breaks added to try and keep horizontal scrolling sane. This arguably hurts readability, in the same way that beating a dead horse arguably hurts the horse.

This may not be the longest one, but it's certainly painful. I do not know exactly what this is doing, and frankly, I do not want to.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

...matter of fact, it's all dark.

Gitter Hubber checks in on the holidays: "This is the spirit of the Black Friday on GitHub. That's because I'm using dark mode. Otherwise, it would have a different name… You know what? Let's just call it Error Friday!"

"Best get typing!" self-admonishes. Jason G. Suffering a surfeit of snark, he proposes "Not sure my battery will last long enough.
Finally, quantum resistant security.
I can't remember my number after the 5000th digit. " Any of those will do just fine.

Don't count Calle L. out. "This is for a calorie tracking app, on Thanksgiving. Offer was so delicious it wasn't even a number any more! Sadly it did not slim the price down more than expected."

"Snow and rain and rain and snow!" exclaims Paul N. "Weather so astounding, they just had to trigger three separate notifications at the same time."

It's not a holiday for everyone though, is it? Certainly not for Michael R. , who is back with a customer service complaint about custom deliveries. "I am unlucky with my deliveries. This time it's DPD. "

It's a holiday in the US today, one where we give thanks. And today, we give thanks to not have this boss. Original. --Remy

Matt works at an accounting firm, as a data engineer. He makes reports for people who don’t read said reports. Accounting firms specialize in different areas of accountancy, and Matt’s firm is a general firm with mid-size clients.

The CEO of the firm is a legacy from the last century. The most advanced technology on his desk is a business calculator and a pencil sharpener. He still doesn’t use a cellphone. But he does have a son, who is “tech savvy”, which gives the CEO a horrible idea of how things work.

Usually, this is pretty light, in that it’s sorting Excel files or sorting the output of an existing report. Sometimes the requests are bizarre or utter nonsense. And, because the boss doesn’t know what the technical folks are doing, some of the IT staff may be a bit lazy about following best practices.

This means that most of Matt’s morning is spent doing what is essentially Tier 1 support before he gets into doing his real job. Recently, there was a worse crunch, as actual support person Lucinda was out for materinity leave, and Jackie, the one other developer, was off on vacation on a foreign island with no Internet. Matt was in the middle of eating a delicious lunch of take-out lo mein when his phone rang. He sighed when he saw the number.

“Matt!” the CEO exclaimed. “Matt! We need to do a build of the flagship app! And a deploy!”

The app was rather large, and a build could take upwards of 45 minutes, depending on the day and how the IT gods were feeling. But the process was automated, the latest changes all got built and deployed each night. Anything approved was released within 24 hours. With everyone out of the office, there hadn’t been any approved changes for a few weeks.

Matt checked the Github to see if something went wrong with the automated build. Everything was fine.

“Okay, so I’m seeing that everything built on GitHub and everything is available in production,” Matt said.

“I want you to do a manual build, like you used to.”

“If I were to compile right now, it could take quite awhile, and redeploying runs the risk of taking our clients offline, and nothing would be any different.”

“Yes, but I want a build that has the changes which Jackie was working on before she left for vacation.”

Matt checked the commit history, and sure enough, Jackie hadn’t committed any changes since two weeks before leaving on vacation. “It doesn’t looked like she pushed those changes to Github.”

“Githoob? I thought everything was automated. You told me the process was automated,” the CEO said.

“It’s kind of like…” Matt paused to think of an analogy that could explain this to a golden retriever. “Your dishwasher, you could put a timer on it to run it every night, but if you don’t load the dishwasher first, nothing gets cleaned.”

There was a long pause as the CEO failed to understand this. “I want Jackie’s front-page changes to be in the demo I’m about to do. This is for Initech, and there’s millions of dollars riding on their account.”

“Well,” Matt said, “Jackie hasn’t pushed- hasn’t loaded her metaphorical dishes into the dishwasher, so I can’t really build them.”

“I don’t understand, it’s on her computer. I thought these computers were on the cloud. Why am I spending all this money on clouds?”

“If Jackie doesn’t put it on the cloud, it’s not there. It’s uh… like a fax machine, and she hasn’t sent us the fax.”

“Can’t you get it off her laptop?”

“I think she took it home with her,” Matt said.

“So?”

“Have you ever seen Star Trek? Unless Scotty can teleport us to Jackie’s laptop, we can’t get at her files.”

The CEO locked up on that metaphor. “Can’t you just hack into it? I thought the NSA could do that.”

“No-” Matt paused. Maybe Matt could try and recreate the changes quickly? “How long before this meeting?” he asked.

“Twenty minutes.”

“Just to be clear, you want me to do a local build with files I don’t have by hacking them from a computer which may or may not be on and connected to the Internet, and then complete a build process which usually takes 45 minutes- at least- deploy to production, so you can do a demo in twenty minutes?”

“Why is that so difficult?” the CEO demanded.

“I can call Jackie, and if she answers, maybe we can figure something out.”

The CEO sighed. “Fine.”

Matt called Jackie. She didn’t answer. Matt left a voicemail and then went back to eating his now-cold lo mein.

As we enter into the latter portion of the year, folks are traveling to visit family, logging off of work in hopes that everything can look after itself for a month, and somewhere, someone, is going to make the choice "yes, I can push to prod on Christmas Eve, and it'll totally work out for me!"

Over the next few weeks, I'm hoping to get a chance to get some holiday support horrors up on the site, in keeping with the season. Whether it's the absurd challenges of providing family tech support, the last minute pushes to production, the five alarm fires caused by a pointy-haired-bosses's incompetence, we want your tales of holiday IT woe.

So hit that submit button on the side bar, and tell us who's on Santa's naughty list this year.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

In today's Tales from the Interview, our Anonymous submitter relates their experience with an anonymous company:

I had made it through the onsite, but along the way I had picked up some toxic work environment red flags. Since I had been laid off a couple months prior, I figured I wasn't in a position to be picky, so I decided I would still give it my best shot and take the job if I got it, but I'd continue looking for something better.

Then they brought me back onsite a second time for one final interview with 2 senior managers. I went in and they were each holding a printout of my resume. They proceeded to go through everything on it. First they asked why I chose the university I went to, then the same for grad school, which was fine.

Then they got to my first internship. I believe the conversation went something like this:

Manager: "How did you like it?"

Me: "Oh, I loved it!"

Manager: "Were there any negatives?"

Me: "No, not that I can think of."

Manager: "So it was 100% positive?"

Me: "Yep!"

And then they got to my first full-time job, where the same manager repeated the same line of questioning but pushed even harder for me to say something negative, at one point saying "Well, you left for (2nd company on my resume), so there must have been something negative."

I knew better than to bad-mouth a previous employer in an interview, it's like going into a first date and talking smack about your ex. But what do you do when your date relentlessly asks you to talk smack about all your exes and refuses to let the subject turn to anything else? This not only confirmed my suspicions of a toxic work environment, I also figured *they* probably knew it was toxic and were relentlessly testing every candidate to make sure they wouldn't blow the whistle on them.

That was the most excruciatingly awkward interview I've ever had. I didn't get the job, but at that point I didn't care anymore, because I was very, very sure I didn't want to work there in the long term.

I'm glad Subby dodged that bullet, and I hope they're in a better place now.

It seems like this might be some stupid new trend. I recently bombed an interview where I could tell I wasn't giving the person the answer on their checklist, no matter how many times I tried. It was a question about how I handled it when someone opposed what I was doing at work or gave me negative feedback. It felt like they wanted me to admit to more fur-flying drama and fireworks than had ever actually occurred.

I actively ask for and welcome critique on my writing, it makes my work so much better. And if my work is incorrect and needs to be redone, or someone has objections to a project I'm part of, I seek clarification and (A) implement the requested changes, (B) explain why things are as they are and offer alternate suggestions/solutions, (C) seek compromise, depending on the situation. I don't get personal about it.

So, why this trend? Subby believed it was a way to test whether the candidate would someday badmouth the employer. That's certainly feasible, though if that were the goal, you'd think Subby would've passed their ordeal with flying colors. I'm not sure myself, but I have a sneaking suspicion that the nefarious combination of AI and techbro startup culture have something to do with it.

So perhaps I also dodged a bullet: one of the many things I'm grateful for this Thanksgiving.

Feel free to share your ideas, and any and all bullets you have dodged, in the comments.

Today, Reginald approaches us for a confession.

He writes:

I've no idea where I "copied" this code from five years ago. The purpose of this code was to filter out Maps and Collections Maybe the intention was to avoid a recursive implementation by an endless loop? I am shocked that I wrote such code.

Well, that doesn't bode well, Reginald. Let's take a look at this Java snippet:

/**
 * 
 * @param input
 * @return
 */
protected Map rearrangeMap(Map input) {
	Map retMap = new HashMap();

	if (input != null && !input.isEmpty()) {

		Iterator it = input.keySet().iterator();
		while (true) {
			String key;
			Object obj;
			do {
				do {
					if (!it.hasNext()) {
					}
					key = (String) it.next();

				} while (input.get(key) instanceof Map);

				obj = input.get(key);

			} while (obj instanceof Boolean && ((Boolean) obj).equals(Boolean.FALSE));

			if (obj != null) {
				retMap.put(key, obj);
				return retMap;
			}
		}
	} else {
		return retMap;
	}
}

The first thing that leaps out is that this is a non-generic Map, which is always a code smell, but I suspect that's the least of our problems.

We start by verifying that the input Map exists and contains data. If the input is null or empty, we return it. In our main branch, we create an iterator across the keys, before ethering a while(true) loop. So far so bad

Then we enter a pair of nested do loops. Which definitely hints that we've gone off the edge of the map here. In the inner most loop, we do a check- if there isn't a next element in the iterator, we… do absolutely nothing. Whether there is or isn't an element, we advance to the next element, risking a NoSuchElementException. We do this while the key points to an instance of Map. As always, an instanceof check is a nauseating code stench.

Okay, so the inner loop skips across any keys that point to maps, and throws an exception when it gets to the end of the list.

The surrounding loop skips over every key that is a boolean value that is also false.

If we find anything which isn't a Map and isn't a false Boolean and isn't null, we put it in our retMap and return it.

This function finds the first key that points to a non-map, non-false value and creates a new map that contains only that key/value. Which it's a hard thing to understand why I'd want that, especially since some Map implementations make no guarantee about order. And even if I did want that, I definitely wouldn't want to do that this way. A single for loop could have solved this problem.

Reginald, I don't think there's any absolution for this. Instead, my advice would be to install a carbon monoxide detector in your office, because I have some serious concerns about whether or not your brain is getting enough oxygen.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Part of the "fun" of JavaScript is dealing with code which comes from before sensible features existed. For example, if you wanted to clone an object in JavaScript, circa 2013, that was a wheel you needed to invent for yourself, as this StackOverflow thread highlights.

There are now better options, and you'd think that people would use them. However, the only thing more "fun" than dealing with code that hasn't caught up with the times is dealing with developers who haven't, and still insist on writing their own versions of standard methods.

  const objectReplace = (oldObject, newObject) => {
    let keys = Object.keys(newObject)
    try {
      for (let key of keys) {
        oldObject[key] = newObject[key]
      }
    } catch (err) {
      console.log(err, oldObject)
    }     

    return oldObject
  }

It's worth noting that Object.entries returns an array containing both the keys and values, which would be a more sensible for this operation, but then again, if we're talking about using correct functions, Object.assign would replace this function.

There's no need to handle errors here, as nothing about this assignment should throw an exception.

The thing that really irks me about this though is that it pretends to be functional (in the programming idiom sense) by returning the newly modified value, but it's also just changing that value in place because it's a reference. So it has side effects, in a technical sense (changing the value of its input parameters) while pretending not to. Now, I probably shouldn't get too hung up on that, because that's also exactly how Object.assign behaves, but dammit, I'm going to be bothered by it anyway. If you're going to reinvent the wheel, either make one that's substantially worse, or fix the problems with the existing wheel.

In any case, the real WTF here is that this function is buried deep in a 15,000 line file, written by an offshore contract team, and there are at least 5 other versions of this function, all with slightly different names, but all basically doing the same thing, because everyone on the team is just copy/pasting until they get enough code to submit a pull request.

Our submitter wonders, "Is there a way to train an AI to not let people type this?"

No, there isn't. You can try rolling that boulder up a hill, but it'll always roll right back down. Always and forever, people are going to write bad code.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Sometimes, it's hard to know just when you are. This morning, I woke up to a Macbook that thinks it's in Paris, four hours ago. Pining for pain chocolate. A bevy of anonyms have had similar difficulties.

First up, an unarabian anonym observes "They say that visiting Oman feels like traveling back in time to before the rapid modernization of the Arab states. I just think their eVisa application system is taking this "time travel" thing a bit too far... "

Snecod, an unretired (anteretired?) anonym finds it hard to plan when the calendar is unfixed. "The company's retirement plan was having a rough time prior to Second June." Looks like the first wtf was second March.

And an unamerican anonym sent us this (uh, back in first March) "Was looking to change the cable package I have from them. Apparently my discounts are all good until 9th October 1930, and a second one looking good until 9th January 2024."

On a different theme, researcher Jennifer E. exclaimed "Those must have been BIG divorces! Guy was so baller Wikipedia couldn’t figure out when he divorced either of these women." Or so awful they divorced him continuously.

Finally, parsimonious Greg L. saved this for us. "I don't remember much about #Error!, but I guess it was an interesting day."

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

Someone wanted to make sure that invalid routes logged an error in their Go web application. Artem found this when looking at production code.

if (requestUriPath != "/config:system") &&
    (requestUriPath != "/config:system/ntp") &&
    (requestUriPath != "/config:system/ntp/servers") &&
    (requestUriPath != "/config:system/ntp/servers/server") &&
    (requestUriPath != "/config:system/ntp/servers/server/config") &&
    (requestUriPath != "/config:system/ntp/servers/server/config/address") &&
    (requestUriPath != "/config:system/ntp/servers/server/config/key-id") &&
    (requestUriPath != "/config:system/ntp/servers/server/config/minpoll") &&
    (requestUriPath != "/config:system/ntp/servers/server/config/maxpoll") &&
    (requestUriPath != "/config:system/ntp/servers/server/config/version") &&
    (requestUriPath != "/config:system/ntp/servers/server/state") &&
    (requestUriPath != "/config:system/ntp/servers/server/state/address") &&
    (requestUriPath != "/config:system/ntp/servers/server/state/key-id") &&
    (requestUriPath != "/config:system/ntp/servers/server/state/minpoll") &&
    (requestUriPath != "/config:system/ntp/servers/server/state/maxpoll") &&
    (requestUriPath != "/config:system/ntp/servers/server/state/version") {
    log.Info("ProcessGetNtpServer: no return of ntp server state for ", requestUriPath)
    return nil
}

The most disturbing part of this, for Artem, isn't that someone wrote this code and pushed it to production. It's that, according to git blame, two people wrote this code, because the first developer didn't include all the cases.

For the record, the application does have an actual router module, which can trigger logging on invalid routes.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

Today's representative line comes from Capybara James (most recently previously). It's representative, not just of the code base, but of Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Or, "you get what you measure".

If, for example, you decide that code coverage metrics are how you're going to judge developers, then your developers are going to ensure that the code coverage looks great. If you measure code coverage, then you will get code coverage- and nothing else.

That's how you get tests like this:

Mockito.verify(exportRequest, VerificationModeFactory.atLeast(0)).failedRequest(any(), any(), any());

This test passes if the function exportRequest.failedRequest is called at least zero times, with any input parameters.

Which, as you might imagine, is a somewhat useless thing to test. But what's important is that there is a test. The standards for code coverage are met, the metric is satisfied, and Goodhart marks up another win on the board.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

One of the things that makes legacy code legacy is that code, over time, rots. Some of that rot comes from the gradual accumulation of fixes, hacks, and kruft. But much of the rot also comes from the tooling going unsupported or entirely out of support.

For example, many years ago, I worked in a Visual Basic 6 shop. The VB6 IDE went out of support in April, 2008, but we continued to use it well into the next decade. This made it challenging to support the existing software, as the IDE frequently broke in response to OS updates. Even when we started running it inside of a VM running an antique version of Windows 2000, we kept running into endless issues getting projects to compile and build.

A fun side effect of that: the VB6 runtime remains supported. So you can run VB6 software on modern Windows. You just can't modify that software.

Greta has inherited an even more antique tech stack. She writes, "I often wonder if I'm the last person on Earth encumbered with this particular stack." She adds, "The IDE is long-deprecated from a vendor that no longer exists- since 2002." Given the project started in the mid 2010s, it may have been a bad choice to use that tech-stack.

It's not as bad as it sounds- while the technology and tooling is crumbling ruins, the team culture is healthy and the C-suite has given Greta wide leeway to solve problems. But that doesn't mean that the tooling isn't a cause of anguish, and even worse than the tooling- the code itself.

"Some things," Greta writes, "are 'typical bad'" and some things "are 'delightfully unique' bad."

For example, the IDE has a concept of "designer" files, for the UI, and "code behind" files, for the logic powering the UI. The IDE frequently corrupts its own internal state, and loses the ability to properly update the designer files. When this happens, if you attempt to open, save, or close a designer file, the IDE pops up a modal dialog box complaining about the corruption, with a "Yes" and "No" option. If you click "No", the modal box goes away- and then reappears because you're seeing this message because you're on a broken designer file. If you click "Yes", the IDE "helpfully" deletes pretty much everything in your designer file.

Nothing about the error message indicates that this might happen.

The language used is a dialect of C++. I say "dialect" because the vendor-supplied compiler implements some cursed feature set between C++98 and C++11 standards, but doesn't fully conform to either. It's only capable of outputting 32-bit x86 code up to a Pentium Pro. Using certain C++ classes, like std::fstream, causes the resulting executable to throw a memory protection fault on exit.

Worse, the vendor supplied class library is C++ wrappers on top of an even more antique Pascal library. The "class" library is less an object-oriented wrapper and more a collection of macros and weird syntax hacks. No source for the Pascal library exists, so forget about ever updating that.

Because the last release of the IDE was circa 2002, running it on any vaguely modern environment is prone to failures, but it also doesn't play nicely inside of a VM. At this point, the IDE works for one session. If you exit it, reboot your computer, or try to close and re-open the project, it breaks. The only fix is to reinstall it. But the reinstall requires you to know which set of magic options actually lets the install proceed. If you make a mistake and accidentally install, say, CORBA support, attempting to open the project in the IDE leads to a cascade of modal error boxes, including one that simply says, "ABSTRACT ERROR" ("My favourite", writes Greta). And these errors don't limit themselves to the IDE; attempting to run the compiler directly also fails.

But, if anything, it's the code that makes the whole thing really challenging to work with. While the UI is made up of many forms, the "main" form is 18,000 lines of code, with absolutely no separation of concerns. Actually, the individual forms don't have a lot of separation of concerns; data is shared between forms via global variables declared in one master file, and then externed into other places. Even better, the various sub-forms are never destroyed, just hidden and shown, which means they remember their state whether you want that or not. And since much of the state is global, you have to be cautious about which parts of the state you reset.

Greta adds:

There are two files called main.cpp, a Station.cpp, and a Station1.cpp. If you were to guess which one owns the software's entry point, you would probably be wrong.

But, as stated, it's not all as bad as it sounds. Greta writes: "I'm genuinely happy to be here, which is perhaps odd given how terrible the software is." It's honestly not that odd; a good culture can go a long way to making wrangling a difficult tech stack happy work.

Finally, Greta has this to say:

We are actively working on a .NET replacement. A nostalgic, perhaps masochistic part of me will miss the old stack and its daily delights.

Kurt Wagner, Bloomberg:

Meta Platforms Inc.’s Mark Zuckerberg is expected to meaningfully cut resources for building the so-called metaverse, an effort that he once framed as the future of the company and the reason for changing its name from Facebook Inc.

Executives are considering potential budget cuts as high as 30% for the metaverse group next year, which includes the virtual worlds product Meta Horizon Worlds and its Quest virtual reality unit, according to people familiar with the talks, who asked not to be named while discussing private company plans. Cuts that high would most likely include layoffs as early as January, according to the people, though a final decision has not yet been made.

Wagner’s reporting was independently confirmed by Mike Isaac, of the New York Times, and Meghan Bobrowsky and Georgia Wells, of the Wall Street Journal, albeit in slightly different ways. While Wagner wrote it “would most likely include layoffs as early as January”, Isaac apparently confirmed the budget cuts are likely large-scale personnel cuts, which makes sense:

The cuts could come as soon as next month and amount to 10 to 30 percent of employees in the Metaverse unit, which works on virtual reality headsets and a V.R.-based social network, the people said. The numbers of potential layoffs are still in flux, they said. Other parts of the Reality Labs division develop smart glasses, wristbands and other wearable devices. The total number of employees in Reality Labs could not be learned.

Alan Dye is just about to join Reality Labs. I wonder if this news comes as a fun surprise for him.

At Meta Connect a few months ago, the company spent basically the entire time on augmented reality glasses, but it swore up and down it was all related to its metaverse initiatives:

We’re hard at work advancing the state of the art in augmented and virtual reality, too, and where those technologies meet AI — that’s where you’ll find the metaverse.

The metaverse is whatever Meta needs it to be in order to justify its 2021 rebrand.

Our vision for the future is a world where anyone anywhere can imagine a character, a scene, or an entire world and create it from scratch. There’s still a lot of work to do, but we’re making progress. In fact, we’re not far off from being able to create compelling 3D content as easily as you can ask Meta AI a question today. And that stands to transform not just the imagery and videos we see on platforms like Instagram and Facebook, but also the possibilities of VR and AR, too.

You know, whenever I am unwinding and chatting with friends after a long day at work, I always get this sudden urge to create compelling 3D content.

⌥ Permalink

Apple:

Apple today announced that Jennifer Newstead will become Apple’s general counsel on March 1, 2026, following a transition of duties from Kate Adams, who has served as Apple’s general counsel since 2017. She will join Apple as senior vice president in January, reporting to CEO Tim Cook and serving on Apple’s executive team.

In addition, Lisa Jackson, vice president for Environment, Policy, and Social Initiatives, will retire in late January 2026. The Government Affairs organization will transition to Adams, who will oversee the team until her retirement late next year, after which it will be led by Newstead. Newstead’s title will become senior vice president, General Counsel and Government Affairs, reflecting the combining of the two organizations. The Environment and Social Initiatives teams will report to Apple chief operating officer Sabih Khan.

What will tomorrow bring, I wonder?

Newstead has spent the past year working closely with Joel Kaplan, and fighting the FTC’s case against Meta — successfully, I should add. Before that, she was a Trump appointee at the U.S. State Department. Well positioned, then, to fight Apple’s U.S. antitrust lawsuit against a second-term Trump government that has successfully solicited Apple’s money.

John Voorhees, MacStories:

Although Apple doesn’t say so in its press release, it’s pretty clear that a few things are playing out among its executive ranks. First, a large number of them are approaching retirement age, and Apple is transitioning and changing roles internally to account for those who are retiring. Second, the company is dealing with departures like Alan Dye’s and what appears to be the less-than-voluntary retirement of John Giannandrea. Finally, the company is reducing the number of Tim Cook’s direct reports, which is undoubtedly to simplify the transition to a new CEO in the relatively near future.

A careful reader will notice Apple’s newsroom page currently has press releases for these departures and, from earlier this week, John Giannandrea’s, but there is nothing about Alan Dye’s. In fact, even in the statement quoted by Bloomberg, Dye is not mentioned. In fairness, Adams, Giannandrea, and Jackson all have bios on Apple’s leadership page. Dye’s was removed between 2017 and 2018.

Starting to think Mark Gurman might be wrong about that FT report.

⌥ Permalink

Jonathan Slotkin, a surgeon and venture capital investor, wrote for the New York Times about data released by Waymo indicating impressive safety improvements over human drivers through June 2025:

If Waymo’s results are indicative of the broader future of autonomous vehicles, we may be on the path to eliminating traffic deaths as a leading cause of mortality in the United States. While many see this as a tech story, I view it as a public health breakthrough.

[…]

There’s a public health imperative to quickly expand the adoption of autonomous vehicles. […]

We should be skeptical of all self-reported stats, but these figures look downright impressive.

Slotkin responsibly notes several caveats, though neglects to mention the specific cities in which Waymo operates: Austin, Los Angeles, Phoenix, and San Francisco. These are warm cities with relatively low annual precipitation, almost none of which is ever snow. Slotkin’s enthusiasm for widespread adoption should be tempered somewhat by this narrow range of climate data. Still, its data is compelling. These cars seem to crash less often than those driven by people in the same cities and, in particular, avoid causing serious injuries at an impressive rate.

It is therefore baffling to me that Waymo appears to be treating this as a cushion for experimentation.

Katherine Bindley, in a Wall Street Journal article published the very same day as Slotkin’s Times piece:

The training wheels are off. Like the rule-following nice guy who’s tired of being taken advantage of, Waymos are putting their own needs first. They’re bending traffic laws, getting impatient with pedestrians and embracing the idea that when it comes to city driving, politeness doesn’t pay: It’s every car for itself.

[…]

Waymo has been trying to make its cars “confidently assertive,” says Chris Ludwick, a senior director of product management with Waymo, which is owned by Google parent Alphabet. “That was really necessary for us to actually scale this up in San Francisco, especially because of how busy it gets.”

A couple years ago, Tesla’s erroneously named “Full Self-Driving” feature began cruising through crosswalks if it judged it could pass a crossing pedestrian in time, and I wrote:

Advocates of autonomous vehicles often say increased safety is one of its biggest advantages over human drivers. Compliance with the law may not be the most accurate proxy for what constitutes safe driving, but not to a disqualifying extent. Right now, it is the best framework we have, and autonomous vehicles should follow the law. That should not be a controversial statement.

I stand by that. A likely reason for Waymo’s impressive data is that its cars behave with caution and deference. Substituting that with “confidently assertive” driving is a move in entirely the wrong direction. It should not roll through stop signs, even if its systems understand nobody is around. It should not mess up the order of an all-way stop intersection. I have problems with the way traffic laws are written, but it is not up to one company in California to develop a proprietary interpretation. Just follow the law.

Slotkin:

This is not a call to replace every vehicle tomorrow. For one thing, self-driving technology is still expensive. Each car’s equipment costs $100,000 beyond the base price, and Waymo doesn’t yet sell cars for personal use. Even once that changes, many Americans love driving; some will resist any change that seems to alter that freedom.

[…]

There is likely to be some initial public trepidation. We do not need everyone to use self-driving cars to realize profound safety gains, however. If 30 percent of cars were fully automated, it might prevent 40 percent of crashes, as autonomous vehicles both avoid causing crashes and respond better when human drivers err. Insurance markets will accelerate this transition, as premiums start to favor autonomous vehicles.

Slotkin is entirely correct in writing that “Americans love driving” — the U.S. National Household Travel Survey, last conducted in 2022, found 90.5% of commuters said they primarily used a car of some kind (table 7-2, page 50). 4.1% said they used public transit, 2.9% said they walked, and just 2.5% said they chose another mode of transportation in which taxicabs are grouped along with bikes and motorcycles. Those figures are about the same in 2017, though with an unfortunate decline in the number of transit commuters. Commuting is not the only reason for travelling, of course, but this suggests to me that even if every taxicab ride was in an autonomous Waymo, there would still be a massive gap to achieve that 30% adoption rate Slotkin wants. And, if insurance companies begin incentivizing autonomous vehicles, it really means rich people will reap the reward of being able to buy a new car.

Any argument about road safety has to be more comprehensive than what Slotkin is presenting in this article. Regardless of how impressive Waymo’s stats are, it is a vision of the future that is an individualized solution to a systemic problem. I have no specialized knowledge in this area, but I am fascinated by it. I read about this stuff obsessively. The things I want to see are things everyone can benefit from: improvements to street design that encourage drivers to travel at lower speeds, wider sidewalks making walking more comfortable, and generous wheeling infrastructure for bicycles, wheelchairs, and scooters. We can encourage the adoption of technological solutions, too; if this data holds up, it would seem welcome. But we can do so much better for everyone, and on a more predictable timeline.

This is, as Slotkin writes, a public health matter. Where I live, record numbers of people are dying, in part because more people than ever are driving bigger and heavier vehicles with taller fronts while they are distracted. Many of those vehicles will still be on the road in twenty years’ time, even if we accelerate the adoption pace of more autonomous vehicles. We do not need to wait for a headline-friendly technological upgrade. There are boring things cities can start doing tomorrow that would save lives.

⌥ Permalink

I have obtained exclusive footage of Alan Dye leaving Apple Park after overseeing visual interface design updates across Apple’s operating systems.

⌥ Permalink

Mark Gurman, Bloomberg:

Meta Platforms Inc. has poached Apple Inc.’s most prominent design executive in a major coup that underscores a push by the social networking giant into AI-equipped consumer devices.

The company is hiring Alan Dye, who has served as the head of Apple’s user interface design team since 2015, according to people with knowledge of the matter. Apple is replacing Dye with longtime designer Stephen Lemay, according to the people, who asked not to be identified because the personnel changes haven’t been announced.

Big week for changes in Apple leadership.

I am sure more will trickle out about this, but one thing notable to me is that Lemay has been a software designer for over 25 years at Apple. Dye, on the other hand, came from marketing and print design. I do not want to put too much weight on that — someone can be a sufficiently talented multidisciplinary designer — but I am curious to see what Lemay might do in a more senior role.

Admittedly I also have some (perhaps morbid) curiosity about what Dye will do at Meta.

One more note from Gurman’s report:

Dye had taken on a more significant role at Apple after Ive left, helping define how the company’s latest operating systems, apps and devices look and feel. The executive informed Apple this week that he’d decided to leave, though top management had already been bracing for his departure, the people said. Dye will join Meta as chief design officer on Dec. 31.

Let me get this straight: Dye personally launches an overhaul of Apple’s entire visual interface language, then leaves. Is that a good sign for its reception, either internally or externally?

⌥ Permalink

Benj Edwards, Ars Technica:

Microsoft has lowered sales growth targets for its AI agent products after many salespeople missed their quotas in the fiscal year ending in June, according to a report Wednesday from The Information. The adjustment is reportedly unusual for Microsoft, and it comes after the company missed a number of ambitious sales goals for its AI offerings.

Based on Edwards’ summary — I still have no interest in paying for the Information — it sounds like this mostly affects sales of A.I. “agents”, a riskier technology proposition for businesses. This sounds to me like more concrete evidence of a plateau in corporate interest than the surveys reported on by the Economist.

⌥ Permalink

Todd Vaziri:

As far as I can tell, Paul Haine was the first to notice something weird going on with HBO Max’ presentation. In one of season one’s most memorable moments, Roger Sterling barfs in front of clients after climbing many flights of stairs. As a surprise to Paul, you can clearly see the pretend puke hose (that is ultimately strapped to the back side of John Slattery’s face) in the background, along with two techs who are modulating the flow. Yeah, you’re not supposed to see that.

It appears as though this represents the original photography, unaltered before digital visual effects got involved. Somehow, this episode (along with many others) do not include all the digital visual effects that were in the original broadcasts and home video releases. It’s a bizarro mistake for Lionsgate and HBO Max to make and not discover until after the show was streaming to customers.

Eric Vilas-Boas, Vulture:

How did this happen? Apparently, this wasn’t actually HBO Max’s fault — the streamer received incorrect files from Lionsgate Television, a source familiar with the exchange tells Vulture. Lionsgate is now in the process of getting HBO Max the correct files, and the episodes will be updated as soon as possible.

It just feels clumsy and silly for Lionsgate to supply the wrong files in the first place, and for nobody at HBO to verify they are the correct work. An amateur mistake, frankly, for an ostensibly premium service costing U.S. $11–$23 per month. If I were king for a day, it would be illegal to sell or stream a remastered version of something — a show, an album, whatever — without the original being available alongside it.

⌥ Permalink

Apple:

Apple today announced John Giannandrea, Apple’s senior vice president for Machine Learning and AI Strategy, is stepping down from his position and will serve as an advisor to the company before retiring in the spring of 2026. Apple also announced that renowned AI researcher Amar Subramanya has joined Apple as vice president of AI, reporting to Craig Federighi. Subramanya will be leading critical areas, including Apple Foundation Models, ML research, and AI Safety and Evaluation. The balance of Giannandrea’s organization will shift to Sabih Khan and Eddy Cue to align closer with similar organizations.

When Apple hired Giannandrea from Google in 2018, the New York Times called it a “major coup”, given that Siri was “less effective than its counterparts at Google and Amazon”. The world changed a lot in the past six-and-a-half years, though: Siri is now also worse than a bunch of A.I. products. Of course, Giannandrea’s role at Apple was not limited to Siri. He spent time on the Project Titan autonomous car, which was cancelled early last year, before moving to generative A.I. projects. The first results of that effort were shown at WWDC last year; the most impressive features have yet to ship.

I feel embarrassed and dumb for hoping Giannandrea would help shake the company out of its bizarre Siri stupor. Alas, he is now on the Graceful Executive Exit Express, where he gets to spend a few more months at Apple in a kind of transitional capacity — you know the drill. Maybe Subramanya will help move the needle. Maybe this ex-Googler will make it so. Maybe I, Charlie Brown, will get to kick that football.

⌥ Permalink

The Economist:

On November 20th American statisticians released the results of a survey. Buried in the data is a trend with implications for trillions of dollars of spending. Researchers at the Census Bureau ask firms if they have used artificial intelligence “in producing goods and services” in the past two weeks. Recently, we estimate, the employment-weighted share of Americans using AI at work has fallen by a percentage point, and now sits at 11% (see chart 1). Adoption has fallen sharply at the largest businesses, those employing over 250 people. Three years into the generative-AI wave, demand for the technology looks surprisingly flimsy.

[…]

Even unofficial surveys point to stagnating corporate adoption. Jon Hartley of Stanford University and colleagues found that in September 37% of Americans used generative AI at work, down from 46% in June. A tracker by Alex Bick of the Federal Reserve Bank of St Louis and colleagues revealed that, in August 2024, 12.1% of working-age adults used generative AI every day at work. A year later 12.6% did. Ramp, a fintech firm, finds that in early 2025 AI use soared at American firms to 40%, before levelling off. The growth in adoption really does seem to be slowing.

I am skeptical of the metrics used by the Economist to produce this summary, in part because they are all over the place, and also because they are mostly surveys. I am not sure people always know they are using a generative A.I. product, especially when those features are increasingly just part of the modern office software stack.

While the Economist has an unfortunate allergy to linking to its sources, I wanted to track them down because a fuller context is sometimes more revealing. I believe the U.S. Census data is the Business Trends and Outlook Survey though I am not certain because its charts are just plain, non-interactive images. In any case, it is the Economist’s own estimate of falling — not stalling — adoption by workers, not an estimate produced by the Census Bureau, which is curious given two of its other sources indicate more of a plateau instead of a decline.

The Hartley, et al. survey is available here and contains some fascinating results other than the specific figures highlighted by the Economist — in particular, that the construction industry has the fourth-highest adoption of generative A.I., that Gemini is shown in Figure 9 as more popular than ChatGPT even though the text on page 7 indicates the opposite, and that the word “Microsoft” does not appear once in the entire document. I have some admittedly uninformed and amateur questions about its validity. At any rate, this is the only source the Economist cites which indicates a decline.

The data point attributed to the tracker operated by the Federal Reserve Bank of St. Louis is curious. The Economist notes “in August 2024, 12.1% of working-age adults used generative A.I. every day at work. A year later 12.6% did”, but I am looking at the dashboard right now, and it says the share using generative A.I. daily at work is 13.8%, not 12.6%. In the same time period, the share of people using it “at least once last week” jumped from 36.1% to 46.9%. I have no idea where that 12.6% number came from.

Finally, Ramp’s data is easy enough to find. Again, I have to wonder about the Economist’s selective presentation. If you switch the chart from an overall view to a sector-based view, you can see adoption of paid subscriptions has more than doubled in many industries compared to October last year. This is true even in “accommodation and food services”, where I have to imagine use cases are few and far between.

After finding the actual source of the Economist’s data, it has left me skeptical of the premise of this article. However, plateauing interest — at least for now — makes sense to me on a gut level. There is a ceiling to work one can entrust to interns or entry-level employees, and that is approximately similar for many of today’s A.I. tools. There are also sector-level limits. Consider Ramp’s data showing high adoption in the tech and finance industries, with considerably less in sectors like healthcare and food services. (Curiously, Ramp says only 29% of the U.S. construction industry has a subscription to generative A.I. products, while Hartley, et al. says over 40% of the construction industry is using it.)

I commend any attempt to figure out how useful generative A.I. is in the real world. One of the problems with this industry right now is that its biggest purveyors are not public companies and, therefore, have fewer disclosure requirements. Like any company, they are incentivized to inflate their importance, but we have little understanding of how much they are exaggerating. If you want to hear some corporate gibberish, OpenAI interviewed executives at companies like Philips and Scania about their use of ChatGPT, but I do not know what I gleaned from either interview — something about experimentation and vague stuff about people being excited to use it, I suppose. It is not very compelling to me. I am not in the C-suite, though.

The biggest public A.I. firm is arguably Microsoft. It has rolled out Copilot to Windows and Office users around the world. Again, however, its press releases leave much to be desired. Levi Strauss employees, Microsoft says, “report the devices and operating system have led to significant improvements in speed, reliability and data handling, with features like the Copilot key helping reduce the time employees spend searching and free up more time for creating”. Sure. In another case study, Microsoft and Pantone brag about the integration of a colour palette generator that you can use with words instead of your eyes.

Microsoft has every incentive to pretend Copilot is a revolutionary technology. For people actually doing the work, however, its ever-nagging presence might be one of many nuisances getting in the way of the job that person actually knows how to do. A few months ago, the company replaced the familiar Office portal with a Copilot prompt box. It is still little more than a thing I need to bypass to get to my work.

All the stats and apparent enthusiasm about A.I. in the workplace are, as far as I can tell, a giant mess. A problem with this technology is that the ways in which it is revolutionary are often not very useful, its practical application in a work context is a mixed bag that depends on industry and role, and its hype encourages otherwise respectable organizations to suggest their proximity to its promised future.

The Economist being what it is, much of this article revolves around the insufficiently realized efficiency and productivity gains, and that is certainly something for business-minded people to think about. But there are more fundamental issues with generative A.I. to struggle with. It is a technology built on a shaky foundation. It shrinks the already-scant field of entry-level jobs. Its results are unpredictable and can validate harm. The list goes on, yet it is being loudly inserted into our SaaS-dominated world as a top-down mandate.

It turns out A.I. is not magic dust you can sprinkle on a workforce to double their productivity. CEOs might be thrilled by having all their email summarized, but the rest of us do not need that. We need things like better balance of work and real life, good benefits, and adequate compensation. Those are things a team leader cannot buy with a $25-per-month-per-seat ChatGPT business license.

Tyler Hall:

Maybe it’s because my eyes are getting old or maybe it’s because the contrast between windows on macOS keeps getting worse. Either way, I built a tiny Mac app last night that draws a border around the active window. I named it “Alan”.

A good, cheeky name. The results are not what I would call beautiful, but that is not the point, is it? It works well. I wish it did not feel understandable for there to be an app that draws a big border around the currently active window. That should be something made sufficiently obvious by the system.

Unfortunately, this is a problem plaguing the latest versions of MacOS and Windows alike, which is baffling to me. The bar for what constitutes acceptable user interface design seems to have fallen low enough that it is tripping everyone at the two major desktop operating system vendors.

⌥ Permalink

Hank Green was not getting a lot of traction on a promotional post on Threads about a sale on his store. He got just over thirty likes, which does not sound awful, until you learn that was over the span of seven hours and across Green’s following of 806,000 accounts on Threads.

So he tried replying to rage bait with basically the same post, and that was far more successful. But, also, it has some pretty crappy implications:

That’s the signal that Threads is taking from this: Threads is like oh, there’s a discussion going on.

It’s 2025! Meta knows that “lots of discussion” is not a surrogate for “good things happening”!

I assume the home feed ranking systems are similar for Threads and Instagram — though they might not be — and I cannot tell you how many times my feed is packed with posts from many days to a week prior. So many businesses I frequent use it as a promotional tool for time-bound things I learn about only afterward. The same thing is true of Stories, since they are sorted based on how frequently you interact with an account.

Everyone is allowed one conspiracy theory, right? Mine is that a primary reason Meta is hostile to reverse-chronological feeds is because it requires businesses to buy advertising. I have no proof to support this, but it seems entirely plausible.

⌥ Permalink

You have seen Moraine Lake. Maybe it was on a postcard or in a travel brochure, or it was on Reddit, or in Windows Vista, or as part of a “Best of California” demo on Apple’s website. Perhaps you were doing laundry in Lucerne. But I am sure you have seen it somewhere.

Moraine Lake is not in California — or Switzerland, for that matter. It is right here in Alberta, between Banff and Lake Louise, and I have been lucky enough to visit many times. One time I was particularly lucky, in a way I only knew in hindsight. I am not sure the confluence of events occurring in October 2019 is likely to be repeated for me.

In 2019, the road up to the lake would be open to the public from May until about mid-October, though the closing day would depend on when it was safe to travel. This is one reason why so many pictures of it have only the faintest hint of snow capping the mountains behind — it is only really accessible in summer.

I am not sure why we decided to head up to Lake Louise and Moraine Lake that Saturday. Perhaps it was just an excuse to get out of the house. It was just a few days before the road was shut for the season.

We visited Lake Louise first and it was, you know, just fine. Then we headed to Moraine.

Walking from the car to the lakeshore, we could see its surface was that familiar blue-turquoise, but it was entirely frozen. I took a few images from the shore. Then we realized we could just walk on it, as did the handful of other people who were there. This is one of several photos I took from the surface of the lake, the glassy ice reflecting that famous mountain range in the background.

I am not sure I would be able to capture a similar image today. Banff and Lake Louise have received more visitors than ever in recent years, to the extent private vehicles are no longer allowed to travel up to Moraine Lake. A shuttle bus is now required. The lake also does not reliably freeze at an accessible time and, when it does, it can be covered in snow or the water line may have receded. I am not arguing this is an impossible image to create going forward. I just do not think I am likely to see it this way again.

I am very glad I remembered to bring my camera.

Winston Cho, the Hollywood Reporter:

To rewind, authors and publishers have gained access to Slack messages between OpenAI’s employees discussing the erasure of the datasets, named “books 1 and books 2.” But the court held off on whether plaintiffs should get other communications that the company argued were protected by attorney-client privilege.

In a controversial decision that was appealed by OpenAI on Wednesday, U.S. District Judge Ona Wang found that OpenAI must hand over documents revealing the company’s motivations for deleting the datasets. OpenAI’s in-house legal team will be deposed.

Wang’s decision (PDF), to the extent I can read it as a layperson, examines OpenAI’s shifting story about why it erased the books 1 and books2 data sets — apparently, the only time possible training materials were deleted.

I am not sure it has yet been proven OpenAI trained its models on pirated books. Anthropic settled a similar suit in September, and Meta and Apple are facing similar accusations. For practical purposes, however, it is trivial to suggest it did use pirated data in general: if you have access to its Sora app, enter any prompt followed by the word “camrip”.

What is a camrip?, a strictly law-abiding person might ask. It is a label added to a movie pirated in the old-fashioned way: by pointing a video camera at the screen in a theatre. As a result, these videos have a distinctive look and sound which is reproduced perfectly by Sora. It is very difficult for me to see a way in which OpenAI could have trained this model to understand what a camrip is without feeding it a bunch of them, and I do not know of a legitimate source for such videos.

⌥ Permalink

The Internet Archive released a WordPress plugin not too long ago:

Internet Archive Wayback Machine Link Fixer is a WordPress plugin designed to combat link rot—the gradual decay of web links as pages are moved, changed, or taken down. It automatically scans your post content — on save and across existing posts — to detect outbound links. For each one, it checks the Internet Archive’s Wayback Machine for an archived version and creates a snapshot if one isn’t available.

Via Michael Tsai:

The part where it replaces broken links with archive links is implemented in JavaScript. I like that it doesn’t modify the post content in your database. It seems safe to install the plug-in without worrying about it messing anything up. However, I had kind of hoped that it would fix the links as part of the PHP rendering process. Doing it in JavaScript means that the fixed links are not available in the actual HTML tags on the page. And the data that the JavaScript uses is stored in an invisible <div> under the attribute data-iawmlf-post-links, which makes the page fail validation.

I love the idea of this plugin, but I do not love this implementation. I think I understand why it works this way: for the nondestructive property mentioned by Tsai, and also to account for its dependence on a third-party service of varying reliability. I would love to see a demo of this plugin in action.

⌥ Permalink

Nicholas Hune-Brown, the Local:

Every media era gets the fabulists it deserves. If Stephen Glass, Jayson Blair and the other late 20th century fakers were looking for the prestige and power that came with journalism in that moment, then this generation’s internet scammers are scavenging in the wreckage of a degraded media environment. They’re taking advantage of an ecosystem uniquely susceptible to fraud—where publications with prestigious names publish rickety journalism under their brands, where fact-checkers have been axed and editors are overworked, where technology has made falsifying pitches and entire articles trivially easy, and where decades of devaluing journalism as simply more “content” have blurred the lines so much it can be difficult to remember where they were to begin with.

This is likely not the first story you have read about a freelancer managing to land bylines in prestigious publications thanks to dependency on A.I. tools, but it is one told very well.

⌥ Permalink

Good tip from Jeff Johnson:

My business website has a number of “Download on the App Store” links for my App Store apps. Here’s an example of what that looks like:

[…]

The problem is that Live Text, “Select text in images to copy or take action,” is enabled by default on iOS devices (Settings → General → Language & Region), which can interfere with the contextual menu in Safari. Pressing down on the above link may select the text inside the image instead of selecting the link URL.

I love the Live Text feature, but it often conflicts with graphics like these. There is a good, simple, two-line CSS trick for web developers that should cover most situations. Also, if you rock a user stylesheet — and I think you should — it seems to work fine as a universal solution. Any issues I have found have been minor and not worth noting. I say give it a shot.

Update: Adding Johnson’s CSS to a user stylesheet mucks up the layout of Techmeme a little bit. You can exclude it by adding div:not(.ii) > before a:has(> img) { display: inline-block; }.

⌥ Permalink

Quinn Nelson:

[…] at a moment when the Mac has roared back to the centre of Apple’s universe, the iPad feels closer than ever to fulfilling its original promise. Except it doesn’t, not really, because while the iPad has gained windowing and external display support, pro apps, all the trappings of a “real computer”, underneath it all, iPadOS is still a fundamentally mobile operating system with mobile constraints baked into its very DNA.

Meanwhile, the Mac is rumoured to be getting everything the iPad does best: touchscreens, OLED displays, thinner designs.

There are things I quibble with in Nelson’s video, including the above-quoted comparison to mere rumours about the Mac. The rest of the video is more compelling as it presents comparisons with the same or similar software on each platform in real-world head-to-head matches.

Via Federico Viticci, MacStories:

I’m so happy that Apple seems to be taking iPadOS more seriously than ever this year. But now I can’t help but wonder if the iPad’s problems run deeper than windowing when it comes to getting serious work done on it.

Apple’s post-iPhone platforms are only as good as Apple will allow them to be. I am not saying it needs to be possible to swap out Bluetooth drivers or monkey around with low-level code, but without more flexibility, platforms like the iPad and Vision Pro are destined to progress only at the rate Apple says is acceptable, and with the third-party apps it says are permissible. These are apparently the operating systems for the future of computers. They are not required to have similar limitations to the iPhone, but they do anyway. Those restrictions are holding back the potential of these platforms.

⌥ Permalink

Marina Dunbar, the Guardian:

Many of the most influential personalities in the “Make America great again” (Maga) movement on X are based outside of the US, including Russia, Nigeria and India, a new transparency feature on the social media site has revealed.

The new tool, called “about this account”, became available on Friday to users of the Elon Musk-owned platform. It allows anyone to see where an account is located, when it joined the platform, how often its username has been changed, and how the X app was downloaded.

This is a similar approach to adding labels or notes to tweets containing misinformation in that it is adding more speech and context. It is more automatic, but the function and intent is comparable, which means Musk’s hobbyist P.R. team must be all worked up. But I checked, and none seem particularly bothered. Maybe they actually care about trust and safety now, or maybe they are lying hacks.

Mike Masnick, Techdirt:

For years, Matt Taibbi, Michael Shellenberger, and their allies have insisted that anyone working on these [trust and safety] problems was part of a “censorship industrial complex” designed to silence political speech. Politicians like Ted Cruz and Jim Jordan repeated these lies. They treated trust & safety work as a threat to democracy itself.

Then Musk rolled out one basic feature, and within hours proved exactly why trust & safety work existed in the first place.

Jason Koebler, 404 Media, has been covering the monetization of social media:

This has created an ecosystem of side hustlers trying to gain access to these programs and YouTube and Instagram creators teaching people how to gain access to them. It is possible to find these guide videos easily if you search for things like “monetized X account” on YouTube. Translating that phrase and searching in other languages (such as Hindi, Portuguese, Vietnamese, etc) will bring up guides in those languages. Within seconds, I was able to find a handful of YouTubers explaining in Hindi how to create monetized X accounts; other videos on the creators’ pages explain how to fill these accounts with AI-generated content. These guides also exist in English, and it is increasingly popular to sell guides to make “AI influencers,” and AI newsletters, Reels accounts, and TikTok accounts regardless of the country that you’re from.

[…]

Americans are being targeted because advertisers pay higher ad rates to reach American internet users, who are among the wealthiest in the world. In turn, social media companies pay more money if the people engaging with the content are American. This has created a system where it makes financial sense for people from the entire world to specifically target Americans with highly engaging, divisive content. It pays more.

The U.S. market is a larger audience, too. But those of us in rich countries outside the U.S. should not get too comfortable; I found plenty of guides similar to the ones shown by Koebler for targeting Australia, Canada, Germany, New Zealand, and more. Worrisome — especially if you, say, are somewhere with an electorate trying to drive the place you live off a cliff.

Update: Several X accounts purporting to be Albertans supporting separatism appear to be from outside Canada, including a “Concerned 🍁 Mum”, “Samantha”, “Canada the Illusion”, and this “Albertan” all from the United States, and a smaller account from Laos. I tried to check more, but X’s fragile servers are aggressively rate-limited.

I do not think people from outside a country are forbidden from offering an opinion on what is happening within it. I would be a pretty staggering hypocrite if I thought that. Nor do I think we should automatically assume people who are stoking hostile politics on social media are necessarily external or bots. It is more like a reflection of who we are now, and how easily that can be exploited.

⌥ Permalink

Jonathan Weil, Wall Street Journal:

It seems like a marvel of financial engineering: Meta Platforms is building a $27 billion data center in Louisiana, financed with debt, and neither the data center nor the debt will be on its own balance sheet.

That outcome looks too good to be true, and it probably is.

The phrase “marvel of financial engineering” does not seem like a compliment. In addition to the evidence from Weil’s article, Meta is taking advantage of a tax exemption created by Louisiana’s state legislature. But, in its argument, it is merely a user of this data centre.

Also, colour me skeptical this data centre will truly be “the size of Manhattan” before the bubble bursts, despite the disruption to life in the area.

Update: Paris Martineau points to Weil’s bio noting he was “the first reporter to challenge Enron’s accounting practices”.

⌥ Permalink

Fred Vogelstein, Crazy Stupid Tech — which, again, is a compliment:

We’re not only in a bubble but one that is arguably the biggest technology mania any of us have ever witnessed. We’re even back reinventing time. Back in 1999 we talked about internet time, where every year in the new economy was like a dog year – equivalent to seven years in the old.

Now VCs, investors and executives are talking about AI dog years – let’s just call them mouse years – which is internet time divided by five? Or is it by 11? Or 12? Sure, things move way faster than they did a generation ago. But by that math one year today now equals 35 years in 1995. Really?

A sobering piece that, unfortunately, is somewhat undercut since it lacks a single mention of layoffs, jobs, employment, or any other indication that this bubble will wreck the lives of people far outside its immediate orbit. In fairness, few of the related articles linked at the bottom mention that, either. Articles in Stratechery, the Brookings Institute, and the New York Times want you to think a bubble is just a sign of building something new and wonderful. A Bloomberg newsletter mentions layoffs only in the context of changing odds in predictions markets — I chuckled — while M.G. Siegler notes all the people who are being laid off while new A.I. hires get multimillion-dollar employment packages. Maybe all the pain and suffering that is likely to result from the implosion of this massive sector is too obvious to mention for the MBA and finance types. I think it is worth stating, though, not least because it acknowledges other people are worth caring about at least as much as innovation and growth and all that stuff.

⌥ Permalink

I relocated my blog to Hugo due to easier maintainance and more control over content and layout. You can find it here.

All articles from this blog have been preserved, although I won’t list some that I found lacking in quality.

Combining a deep-depthwise CNN architecture with variable quantization in BitNetMCU achieves state-of-the-art MNIST accuracy on a low-end 32-bit microcontroller with 4 kB RAM and 16 kB flash.

Read the article at my new blog location.

This’ll be the last Recently in 2025. It’s been a decent year for me, a pretty rough year for the rest of the world. I hope, for everyone, that 2026 sees the reversal of some of the current trends.

Watching

This video from Daniel Yang, who makes spectacular bikes of his own, covers a lot of the economics of being a bike-builder, which are all pretty rough. I felt a lot of resonance with Whit: when I ran a business I always felt like it was tough to be commercial about it, and had to fight my own instincts to over engineer parts of it. It’s also heartbreaking to think about how many jobs are so straightforwardly good for the maker and good for the buyer but economically unviable because of the world we live in. I feel like the world would be a lot different if the cost of living was lower.

Yes, the fan video! It’s a solid 48 minutes of learning how fans work. I finally watched it. Man, if every company did their advertising this way it would be so fun. I learned a lot watching this.

I have trouble finding videos about how things work that are actually about how things work. Titles like “how it’s made” or “how it works” or “how we did it” perform well in A/B tests and SEO so they get used for media that actually doesn’t explain how it’s made and how it works, greatly frustrating people like me. But the fan video delivers.

Reading

But then, you realize that the goal post has shifted. As the tech industry has become dramatically more navigable, YC became much less focused on making the world understandable, revolving, instead, around feeding consensus. “Give the ecosystem what they want.”

I have extremely mixed feelings about Build What’s Fundable, this article from Kyle Harrison. Some of it I think is bravely truth-telling in an industry that usually doesn’t do public infighting - Harrison is a General Partner at a big VC firm, and he’s critiquing a lot of firms directly on matters both financial and ethical.

But on the other hand, there’s this section about “Breaking the Normative Chains”:

When you look at successful contrarian examples, many of them have been built by existing billionaires (Tesla, SpaceX, Palantir, Anduril). The lesson from that, I think, isn’t “be a billionaire first then you can have independent thoughts.” It is, instead, to reflect on what other characteristics often lead to those outcomes. And, in my opinion, the other commonality that a lot of those companies have is that they’re led by ideological purists. People that believe in a mission.

And then, in the next section he pulls up an example of a portfolio company that encapsulates the idea of true believers, and he names Base, which has a job ad saying “Don’t tell your grandkids all you did was B2B SaaS.”

Now, Base’s mission is cool: they’re doing power storage at scale. I like the website. But I have to vent here that the founder is Zach Dell. Michael Dell’s son. Of Dell Computer, and a 151 billion dollar fortune.

I just think that if we’re going to talk about how the lesson isn’t that you should be a billionaire first before having independent thoughts and building a big tech company, it should be easy to find someone who is not the son of the 10th wealthiest person in the world to prove that point. I have nothing against Zach in particular: he is probably a talented person. But in the random distribution of talented, hardworking people, very few of them are going to be the son of the 10th wealthiest person in the world.

Like so many other bits of Times coverage, the whole of the piece is structured as an orchestrated encounter. Some people say this; however, others say this. It’s so offhand you can think you’re gazing through a pane of glass. Only when you stand a little closer, or when circumstances make you a little less blinkered, do you notice the fact which then becomes blinding and finally crazymaking, which is just that there is zero, less than zero, stress put on the relation between those two “sides,” or their histories, or their sponsors, or their relative evidentiary authority, or any of it.

I love this article on maybe don’t talk to the New York Times about Zohran Mamdani. It describes the way in which the paper launders its biases, which overlaps with one of my favorite rules from Wikipedia editing about weasel words.

I don’t want you to hate this guy. Yes, he actively promotes poisonous rhetoric – ignore that for now. This is about you. Reflect on all your setbacks, your unmet potential, and the raw unfairness of it all. It sucks, and you mustn’t let that bitterness engulf you. You can forgive history itself; you can practice gratitude towards an unjust world. You need no credentials, nor awards, nor secrets, nor skills to do so. You are allowed to like yourself.

Taylor Troesh on IQ is exactly what I needed that day.

The React team knows this makes React complicated. But the bet is clear: React falls on the sword of complexity so developers don’t have to. That’s admirable, but it asks developers to trust React’s invisible machinery more than ever.

React and Remix Choose Different Futures is perfect tech writing: it unpacks the story and philosophy behind a technical decision without cramming it into a right-versus-wrong framework.

When you consider quitting, try to find a different scoreboard. Score yourself on something else: on how many times you dust yourself off and get up, or how much incremental progress you make. Almost always, in your business or life, there are things you can make daily progress on that can make you feel like you’re still winning. Start compounding.

“Why a language? Because I believe that the core of computing is not based on operating system or processor technologies but on language capability. Language is both a tool of thought and a means of communication. Just as our minds are shaped by human language, so are operating systems shaped by programming languages. We implement what we can express. If it cannot be expressed, it will not be implemented.” – Carl Sassenrath

Alexis Sellier, whose work and aesthetic I’ve admired since the early days of Node.js, is working on a new operating system. A real new operating system, like playb.it or SerenityOS (bad politics warning). I’m totally into it: we need more from-scratch efforts like this!

Yes, the funds available for any good cause are scarce, but that’s not because of some natural law, some implacable truth about human society. It’s because oligarchic power has waged war on benign state spending, leading to the destruction of USAID and drastic cuts to the aid budgets of other countries, including the UK. Austerity is a political choice. The decision to impose it is driven by governments bowing to the wishes of the ultra-rich.

The Guardian on Bill Gates is a good read. I’ve had The Bill Gates Problem on my reading list for a long time. Maybe it’s next after I finish The Fort Bragg Cartel.

Contrast this with the rhetorical shock and awe campaign that has been waged by technology companies for the last fifteen years championing the notion of ephemerality.

Implicit, but unspoken, in this worldview is the idea of transience leading to an understanding of a world awash in ephemeral moments that, if not seized on and immediately capitalized to maximum effect, will be forever lost to the mists of time and people’s distracted lifestyles.

Another incredible article by Aaron Straup Cope about AI, the web, ephemerality, and culture. (via Perfect Sentences)

Also, no exact quote, but I’ve been subscribed to Roma’s Unpolished Posts and they have been pretty incredible: mostly technical articles about CSS, which have been ‘new to me’ almost every day, and the author is producing them once a day. Feels like a cheat code to absorb so much new information so quickly.

Listening

I didn’t add any major new albums to my collection this month. I did see Tortoise play a show, which was something I never expected to do. So in lieu of new albums, here’s a theme.

There are a bunch of songs in my library that use a meter change as a way to add or resolve tension. I’m not a big fan of key changes but I love a good rhythm or production shift.

First off: Kissing the Beehive. Yes, it’s nearly 11 minutes long. Magically feeling “off kilter” and “in the pocket” at the same time. I’m no drummer but I think the first part is something like three measures of 4/4 and one of 6/4. But then at 3:26, brand new connected song, and by the time we get to 7 minutes in, we’re in beautiful breezy easy 4/4!

An Andrew Bird classic: about a minute of smooth 4/4, and then over to 7/4 in the second half or so.

I adore Akron/Family’s Running, Returning. Starts in classic 5/4, then transitions to 4/4, then 6/8. For me it all feels very cohesive. Notably, the band is not from Akron Ohio but formed in Williamsburg and were from other East Coast places. If you’re looking for the band from Akron, it’s The Black Keys.

Slow Mass’s Schemes might be my song of the year. Everything they write just sounds so cool. The switchup happens around 2:40 when the vocals move to 6/4. Astounding.

Predictions

Back in January, I made some predictions about 2025. Let’s see how they turned out!

1: The web becomes adversarial to AI

I am marking this one as a absolute win: more and more websites are using Anubis, which was released in March, to block LLM scrapers. Cloudflare is rolling out more LLM bot protections. I at Val Town have started to turn on those protections to keep LLM bots from eating up all of our bandwidth and CPU. The LLM bots are being assholes and everyone hates them.

2: Copyright nihilism breeds a return to physical-only media

This was at most going to be a moderate win because physical-only media will be niche, but I think there are good signs that this is right. The Internet Phone Book, in which this site is featured, started publishing this year. Gen Z seems to be buying more vinyl and printing out more photos.

3: American tech companies will pull out of Europe because they want to do acquisitions

Middling at best: there are threats and there is speculation, but nothing major to report.

4: The tech industry’s ‘DEI backlash’ will run up against reality

Ugh, probably the opposite has happened. Andreesen Horowitz shut down their fund that focused on women, minorities, and people underepresented in VC funding. We’ll know more about startups themselves when Carta releases their annual report, which looked pretty bad last year.

5: Local-first will have a breakthrough moment

Sadly, no. Lots and lots of promising projects, but the ecosystem really struggles to produce something production-ready that offers good tradeoffs. Tanstack DB might be the next contender.

6: Local, small AI models will be a big deal

Not yet. Big honkin’ models are still grabbing most of the headlines. LLMs still really thrive at accomplishing vague tasks that are achievable with a wide range of acceptance criteria, like chatbots, and are pretty middling at tasks that require specific strict, quantifiable outputs.

For my mini predictions:

• Substack will re-bundle news. Sort of! Multi-editor newsletters like The Argument are all the rage right now.
• TypeScript gets a zeitwork equivalent and lots of people use it. Sadly, no.
• Node.js will fend off its competitors. Mostly! Bun keeps making headway but Node.js keeps implementing the necessary featureset, and doesn’t seem to be losing that much marketshare.
• Another US city starts seriously considering congestion pricing. There are rumblings from Boston and Chicago!
• Stripe will IPO. Nope. Unlimited tender offers from Sequoia is just as good as IPOing, so why would they.

It’s the end of 2025, which means that I’m closing in on three years at Val Town. I haven’t written much about the company or what it’s really been like. The real story of companies is usually told well after years after the dust has settled. Founders usually tell a heroic story of success while they’re building.

Reading startup news really warps your perspective, especially when you’re building a startup yourself. Everyone else is getting fabulously rich! It makes me less eager to write about anything.

But I’m incurably honest and like working with people who are too. Steve, the first founder of Val Town (I joined shortly after as cofounder/CTO) is a shining example of this. He is a master of saying the truth in situations when other people are afraid to. I’ve seen it defuse tension and clear paths. It’s a big part of ‘the culture’ of the company.

So here’s some of the story so far.

Delivering on existing expectations and promises

Here’s what the Val Town interface looked like fairly early on:

When I initially joined, we had a prototype and a bit of hype. The interface was heavily inspired by Twitter - every time that you ran code, it would save a new ‘val’ and add it to an infinite-scrolling list.

Steve and Dan had really noticed the utter exhaustion in the world of JavaScript: runaway complexity. A lot of frameworks and infrastructure was designed for huge enterprises and was really, really bad at scaling down. Just writing a little server that does one thing should be easy, but if you do it with AWS and modern frameworks, it can be a mess of connected services and boilerplate.

Val Town scaled down to 1 + 1. You could type 1 + 1 in the text field and get 2. That’s the way it should work.

It was a breath of fresh air. And a bunch of people who encountered it even in this prototype-phase state were inspired and engaged.

One of the pivotal moments of this stage was creating this graphic for our marketing site: the arrows graphic. It really just tied it all together: look how much power there was in this little val! And no boilerplate either. Where there otherwise is a big ritual of making something public or connecting an email API, there’s just a toggle and a few lines of code.

I kind of call this early stage, for me, the era of delivering on existing expectations and promises. The core cool idea of the product was there, but it was extremely easy to break.

Security was one of the top priorities. We weren’t going to be a SOC2 certified bank-grade platform, but we also couldn’t stay where we were. Basically, it was trivially easy to hack: we were using the vm2 NPM module to run user code. I appreciate that vm2 exists, but it really, truly, is a trap. There are so many ways to get out of its sandbox and access other people’s code and data. We had a series of embarrassing security vulnerabilities.

For example: we supported web handlers so you could easily implement a little server endpoint, and the API for this was based on express, the Node.js server framework. You got a request object and response object, from express, and in this case they were literally the same as our server’s objects. Unfortunately, there’s a method response.download(path: string) which sends an arbitrary file from your server to the internet. You can see how this one ends: not ideal.

So, we had to deliver on a basic level of security. Thankfully, in the way that it sometimes does, the road rose to meet us. The right technology appeared just in time: Deno. Deno’s sandboxing made it possible to run people’s code securely without having to build a mess of Kubernetes and Docker sandbox optimizations. It delivered being secure, fast, and simple to implement: we haven’t identified a single security bug caused by Deno.

That said, the context around JavaScript runtimes has been tough. Node.js is still dominant and Bun has attracted most of the attention as an alternative, with Deno in a distant third place, vibes-wise. The three are frustratingly incompatible - Bun keeps adding built-ins like an S3 client which would have seemed unthinkable in the recent past. Node added an SQLite client in 22. Contrary to what I hoped in 2022, JavaScript has gotten more splintered and inconsistent as an ecosystem.

Stability was the other problem. The application was going down constantly for a number of reasons, but most of all was the database, which was Supabase. I wrote about switching away from Supabase, which they responded to in a pretty classy way, and I think they’ve since improved. But Render has been a huge step up in maintainability and maturity for how we host Val Town.

Adding Max was a big advance in our devops-chops too: he was not only able to but excited to work on the hard server capacity and performance problems. We quietly made a bunch of big improvements like allowing vals to stay alive after serving requests - before that, every run was a cold start.

What to do about AI

Townie, the Val Town chatbot, in early 2024

Believe it or not, but in early 2023, there were startups that didn’t say “AI” on the front page of their marketing websites. The last few years have been a dizzying shift in priorities and vibes, which I have had mixed feelings about that I’ve written about a lot.

At some point it became imperative to figure out what Val Town was supposed to do about all that. Writing code is undeniably one of the sweet spots of what LLMs can do, and over the last few years the fastest-growing most hyped startups have emerged from that ability.

This is where JP Posma comes in. He was Steve’s cofounder at a previous startup, Zaplib, and was our ‘summer intern’ - the quotes because he’s hilariously overqualified for that title. He injected some AI-abilities into Val Town, both RAG-powered search and he wrote the first version of Townie, a chatbot that is able to write code.

Townie has been really interesting. Basically it lets you write vals (our word for apps) with plain English. This development happened around the same time as a lot of the ‘vibe-coding’ applications, like Bolt and Lovable. But Townie was attached to a platform that runs code and has community elements and a lot more. It’s an entry point to the rest of the product, while a lot of other vibe-coding tools were the core product that would eventually expand to include stuff like what Val Town provides.

Ethan Ding has written a few things about this: it’s maybe preferable to sell compute instead of being the frontend for LLM-vibe-coding. But that’s sort of a long-run prediction about where value accrues rather than an observation about what companies are getting hype and funding in the present.

There are way too many companies providing vibe-coding tools without having a moat or even a pathway to positive margins. But having made a vibe-coding tool, I completely see why: it makes charts look amazing. Townie was a huge growth driver for a while, and a lot of people were hearing about Townie first, and only later realizing that Val Town could run code, act as a lightweight GitHub alternative, and power a community.

Unlike a lot of AI startups, we didn’t burn a ton of money running Townie. We did have negative margins on it, but to the tune of a few thousand dollars a month during the most costly months.

Introducing a pro plan made it profitable pretty quickly and today Townie is pay-as-you-go, so it doesn’t really burn money at all. But on the flip side, we learned a lot about the users of vibe-coding tools. In particular, they use the tools a lot, and they really don’t want to pay for them. This kind of makes sense: vibe-coding actual completed apps without ever dropping down to write or read code is zeno’s paradox: every prompt gets you halfway there, so you inch closer and closer but never really get to your destination.

So you end up chatting for eight hours, typically getting angrier and angrier, and using a lot of tokens. This would be great for business in theory, but in practice it doesn’t work for obvious reasons: people like to pay for results, not the process. Vibe-coding is a tough industry - it’s simultaneously one of the most expensive products to run, and one of the most flighty and cost-sensitive user-bases I’ve encountered.

So AI has been complicated. On one hand, it’s amazing for growth and obviously has spawned wildly successful startups. On the other, it can be a victim of its own expectations: every company seems to promise perfect applications generated from a single prompt and that just isn’t the reality. And that results in practically every tool falling short of those expectations and thus getting the rough end of user sentiment.

We’re about to launch MCP support, which will make it possible to use Val Town via existing LLM interfaces like Claude Code. It’s a lot better than previous efforts - more powerful and flexible, plus it requires us to reinvent less of the wheel. The churn in the ‘state of the art’ feels tremendous: first we had tool-calling, then MCPs, then tool calling writing code to call MCPs: it’s hard to tell if this is fast progress or just churn.

As a business

When is a company supposed to make money? It’s a question that I’ve thought about a lot. When I was running a bootstrapped startup, the answer was obviously as soon as possible, because I’d like to stop paying my rent from my bank account. Venture funding lets you put that off for a while, sometimes a very long while, and then when companies start making real revenue they at best achieve break-even. There are tax and finance reasons for all of this – I don’t make the rules!

Anyway, Val Town is far from break-even. But that’s the goal for 2026, and it’s optimistically possible.

One thing I’ve thought for a long time is that people building startups are building complicated machines. They carry out a bunch of functions, maybe they proofread your documents or produce widgets, or whatever, but the machine also has a button on it that says “make money.” And everything kind of relates to that button as you’re building it, but you don’t really press it.

The nightmare is if the rest of the machine works, you press the button, and it doesn’t do anything. You’ve built something useful but not valuable. This hearkens back to the last section about AI: you can get a lot of people using the platform, but if you ask them for money and they’re mostly teenagers or hobbyists, they’re not going to open their wallets. They might not even have wallets.

So we pressed the button. It kind of works.

But what I’ve learned is that making revenue is a lot like engineering: it requires a lot of attempts, testing, and hard work. It’s not something that just results from a good product. Here’s where I really saw Charmaine and Steve at work, on calls, making it happen.

The angle right now is to sell tools for ‘Go To Market’ - stuff like capturing user signups of your website, figuring out which users are from interesting companies or have interesting use-cases, and forwarding that to Slack, pushing it to dashboards, and generally making the sales pipeline work. It’s something Val Town can do really well: most other tools for this kind of task have some sort of limit in how complicated and custom they can get, and Val Town doesn’t.

Expanding and managing the complexity

Product-wise, the big thing about Val Town that has evolved is that it can do more stuff and it’s more normal. When we started out, a Val was a single JavaScript expression - this was part of what made Val Town scale down so beautifully and be so minimal, but it was painfully limited. Basically people would type into the text box

const x = 10;
function hi() {};
console.log(1);

And we couldn’t handle that at all: if you ran the Val did it run that function? Export the x variable? It was magic but too confusing. The other tricky niche choice was that we had a custom import syntax like this:

@tmcw.helper(10);

In which @tmcw.helper was the name of another val and this would automatically import and use it. Extremely slick but really tricky to build off of because this was non-standard syntax, and it overlapped with the proposed syntax for decorators in JavaScript. Boy, I do not love decorators: they have been under development for basically a decade and haven’t landed, just hogging up this part of the unicode plane.

But regardless this syntax wasn’t worth it. I have some experience with this problem and have landed squarely on the side of normality is good.

So, in October 2023, we ditched it, adopted standard ESM import syntax, and became normal. This is was a big technical undertaking, in large part because we tried to keep all existing code running by migrating it. Thankfully JavaScript has a very rich ecosystem of tools that can parse & produce code and manipulate syntax trees, but it was still a big, dramatic shift.

This is one of the core tensions of Val Town as well as practically every startup: where do you spend your user-facing innovation energy?

I’m a follower of the use boring technology movement when it comes to how products are built: Val Town intentionally uses some boring established parts like Postgres and React Router, but what about when it comes to the product itself? I’ve learned the hard way that most of what people call intuition is really familiarity: it’s good when an interface behaves like other interfaces. A product that has ten new concepts and a bunch of new UI paradigms is going to be hard to learn and probably will lose out to one that follows some familiar patterns.

Moving to standard JavaScript made Val Town more learnable for a lot of people while also removing some of its innovation. Now you can copy code into & out of Val Town without having to adjust it. LLMs can write code that targets Val Town without knowing everything about its quirks. It’s good to go with the flow when it comes to syntax.

Hiring and the team

Val Town has an office. I feel like COVID made everything remote by default and the lower-COVID environment that we now inhabit (it’s still not gone!) has led to a swing-back, but the company was founded in the latter era and has never been remote. So, we work from home roughly every other Friday.

This means that we basically try to hire people in New York. It hasn’t been too hard in the past. About 6% of America lives in the New York City metro area and the Northeast captures about 23% of venture funding, so there are lots of people who live here or want to.

Here’s something hard to publish: we’re currently at three people. It was five pretty recently. Charmaine got poached by Anthropic where she’ll definitely kick ass, and Max is now at Cloudflare where he’s writing C++, which will be even more intimidating than his chess ranking. The company’s really weirdly good at people leaving: we had parties and everyone exchanges hand-written cards. How people handle hard things says a lot.

But those three are pretty rad: Jackson was a personal hero of mine before we hired him (he still is). He’s one of the best designers I’ve worked with, and an incredibly good engineer to boot. He’s worked at a bunch of startups you’ve heard of, had a DJ career, gotten to the highest echelons of tech without acquiring an ego. He recently beat me to the top spot in our GitHub repo’s lines-changed statistic.

Steve has what it takes for this job: grit, optimism, curiosity. The job of founding a company and being a CEO is a different thing every few months - selling, hiring, managing, promoting. Val Town is a very developer-consumer oriented product and that kind of thing requires a ton of promotion. Steve has done so much, in podcasts, spreading the word in person, writing, talking to customers. He has really put everything into this. A lot of the voice and the attitude of the company flows down from the founder, and Steve is that.

Did I mention that we’re hiring?

In particular, for someone to be a customer-facing technical promoter type - now called a “GTM” hire. Basically, who can write a bit of code but has the attitude of someone in sales. Who can see potential and handle rejection. Not necessarily the world’s best programmer, but who can probably code, and definitely someone who can write. Blogging and writing online is a huge green flag for this position.

And the other role that we really need is an “application engineer.” These terms keep shifting, so if full-stack engineer means more, sure, that too. Basically someone who can write code across boundaries. This is more or less what Jackson and I do - writing queries, frontend code, fixing servers, the whole deal. Yeah, it sounds like a lot but this is how all small companies operate, and I’ve made a lot of decisions to make this possible: we’ve avoided complexity like the plague in Val Town’s stack, so it should all be learnable. I’ve written a bunch of documentation for everything, and constantly tried to keep the codebase clean.

Sidenote, but even though I think that the codebase is kind of messy, I’ve heard from very good engineers (even the aforementioned JP Posma) that it’s one of the neatest and most rational codebases they’ve seen. Maybe it is, maybe it isn’t, see for yourself!

What we’re really looking for in hires

Tech hiring has been broken the whole time I’ve been in the industry, for reasons that would take a whole extra article to ponder. But one thing that makes it hard is vagueness, both on the part of applicants and companies. I get it - cast a wide net, don’t put people off. But I can say that:

• For the GTM position, you should be able to write for the internet. This can be harder than it looks: there are basically three types of writing: academic, corporate, and internet, and they are incompatible.
• You should also be kind of entrepreneurial: which means optimistic, resilient, and opportunistic.
• For the application engineering role, you should be a good engineer who understands the code you write and is good at both writing and reading code. Using LLM tools is great, but relying on them exclusively is a dealbreaker. LLMs are not that good at writing code.

What the job is like

The company’s pretty low drama. Our office is super nice. We work hard but not 996. We haven’t had dinner in the office. But we all do use PagerDuty so when the servers go down, we wake up and it sucks. Thankfully the servers go down less than they used to.

We all get paid the same: $175k. Lower than FAANG, but pretty livable for Brooklyn. Both of the jobs listed - the Product Engineer, and Growth Engineer - are set at 1% equity. $175k is kind of high-average for where we’re at, but 1% in my opinion is pretty damn good. Startups say that equity is “meaningful” at all kinds of numbers but it’s definitely meaningful at that one. If Val Town really succeeds, you can get pretty rich off of that.

Of course, will it succeed? It’s something I think about all the time. I was born to ruminate. We have a lot going for us, and a real runway to make things happen. Some of the charts in our last investor update looked great. Some days felt amazing. Other days were a slog. But it’s a good team, with a real shot of making it.

Hello! Only a day late this time. October was another busy month but it didn’t yield much content. I ran a second half-marathon, this time with much less training, but only finished a few minutes slower than I did earlier this year. Next year I’m thinking about training at a normal mileage for the kinds of races I’m running - 25 miles or so per week instead of this year’s roughly 15.

And speaking of running, I just wrote up this opinion I have about how fewer people should run marathons.

Reading

I enjoyed reading Why Functor Doesn’t Matter, but I don’t really agree. The problem that I had with functional programming jargon isn’t that the particular terms are strange or uncommon, but that their definitions rely on a series of other jargon terms, and the discipline tends to omit good examples, metaphors, or plain-language explanations. It’s not that the strict definition is bad, but when a function is defined as _a mapping that associates a morphism F: X -> Y in category C to a morphism F(f): F(X) -> F(Y), in category D, you now have to define morphism, categories, and objects, and all of which have domain-specific definitions.

I am loving Sherif’s posting about building a bike from the frame up.

Maximizers are biased to speed, optionality, breadth, momentum, opportunism, parallel bets, hype, luck exposure, momentum, “Why not both?”, “Better to move fast than wait for perfect”. Maximizers want to see concrete examples before they’ll make tradeoffs. They anchor decisions in the tangible. “Stop making things so complicated.” “Stop overthinking.”

Focusers are biased to focus, coherence, depth, meaningful constraints, doing less for more, sequential experiments, intentionality, sustainability, “What matters most?”, compounding clarity. Focusers are comfortable with abstraction. A clear constraint or principle is enough to guide them. “Stop mistaking chaos for progress.” “Stop overdoing.”

John Cutler’s post about maximizers vs. focusers matches my experience in tech. Like many young engineers, I think I started out as a focuser and have tried to drift to the center all the time, but the tension both internally and interpersonally at every job is present.

I recently remarked to a friend that traveling abroad after the advent of the smartphone feels like studying biology after the advent of microplastics. It has touched every aspect of life. No matter where you point your microscope you will see its impact.

Josh Erb’s blog about living in India is great, personal, a classic blog’s blog.

For me, the only reason to keep going is to try and make AI a wonderful technology for the world. Some feel the same. Others are going because they’re locked in on a path to generational wealth. Plenty don’t have either of these alignments, and the wall of effort comes sooner.

This article about AI researchers working all the time and burning out is interesting, in part because I find the intention of AI researchers so confusing. I can see the economic intention: these guys are making bank! Congrats to all of them. But it’s so rare to talk to anyone who has a concrete idea about how they are making the world better by doing what they’re doing, and that’s the reason why they’re working so hard. OpenAI seems to keep getting distracted from that cancer cure, and their restructuring into a for-profit company kind of indicates that there’s more greed than altruism in the mix.

every vc who bet on the modern data stack watched their investments get acquired for pennies or go to zero. the only survivors: the warehouses themselves, or the companies the warehouses bought to strengthen their moats.

It’s niche, but this article about Snowflake, dbt, fivetran, and other ‘data lake’ architecture is really enlightening.

Listening

Totorro’s new album was the only one I picked up this month. It’s pretty good math-rock, very energetic and precise.

Watching

• One Battle After Another was incredible.
• eXistenZ is so gloriously weird, I really highly recommend it. It came out the same year as The Matrix and explores similar themes, but the treatment of futuristic technology is something you won’t see anywhere else: instead of retro-steampunk metal or fully dystopian grimness, it’s colorful, slimy, squelchy, organic, and weird.

Speaking of weird, Ben Levin’s gesamtkunstwerk videos are wild and glorious.

You might have seen an article making the rounds this week, about a young man who ended his life after ChatGPT encouraged him to do so. The chat logs are really upsetting.

Someone two degrees removed from me took their life a few weeks ago. A close friend related the story to me, about how this person had approached their neighbor one evening to catch up, make small talk, and casually discussed their suicidal ideation at some length. At the end of the conversation, they asked to borrow a rope, and their neighbor agreed without giving the request any critical thought. The neighbor found them the next morning.

I didn’t know the deceased, nor their neighbor, but I’m close friends with someone who knew both. I found their story deeply chilling – ice runs through my veins when I imagine how the neighbor must have felt. I had a similar feeling upon reading this article, wondering how the people behind ChatGPT and tools like it are feeling right now.

Two years ago, someone I knew personally took their life as well. I was not friendly with this person – in fact, we were on very poor terms. I remember at the time, I had called a crisis hotline just to ask an expert for advice on how to break this news to other people in my life, many of whom were also on poor terms with a person whose struggles to cope with their mental health issues caused a lot of harm to others.

None of us had to come to terms with any decisions with the same gravity as what that unfortunate neighbor had to face. None of us were ultimately responsible for this person’s troubles or were the impetus for what happened. Nonetheless, the uncomfortable and confronting feelings I experienced in the wake of that event perhaps give me some basis for empathy and understanding towards the neighbor, or for OpenAI employees, and others who find themselves in similar situations.

If you work on LLMs, well… listen, I’ve made my position as an opponent of this technology clear. I feel that these tools are being developed and deployed recklessly, and I believe tragedy is the inevitable result of that recklessness. If you confide in me, I’m not going to validate your career choice. But maybe that’s not necessarily a bad quality to have in a confidant? I still feel empathy towards you and I recognize your humanity and our need to acknowledge each other as people.

If you feel that I can help, I encourage you to reach out. I will keep our conversation in confidence, and you can reach out anonymously if that makes you feel safer. I’m a good listener and I want to know how you’re doing. Email me.

If you’re experiencing a crisis, 24-hour support is available from real people who are experts in getting you the help you need. Please consider reaching out. All you need to do is follow the link.

Outages, you say? Of course I have stories about outages, and limits, and some limits causing outages, and other things just screwing life up. Here are some random thoughts which sprang to mind upon reading this morning's popcorn-fest.

...

I was brand new at a company that "everybody knew" had AMAZING infrastructure. They could do things with Linux boxes that nobody else could. As part of the new employee process, I had to get accounts in a bunch of systems, and one of them was this database used to track the states of machines. It was where you could look to see if it was (supposed to be) serving, or under repair, or whatever. You could also see (to some degree) what services were supposed to be running on it, and what servers (that is, actual programs), the port numbers, and whether all of that stuff was synced to the files on the box or not.

My request didn't go through for a while, and I found out that it had something to do with my employee ID being a bit over 32767. And yeah, for those of you who didn't just facepalm at seeing that number, that's one of those "magic numbers" which pops up a bunch when talking about limits. That one is what you get when you try to store numbers as 16 bit values... with a sign to allow negative values. Why you'd want a negative employee number is anyone's guess, but that's how they configured it.

I assume they fixed the database schema at some point to allow more than ~15 bits of employee numbers, but they did an interesting workaround to get me going before then. They just shaved off the last digit and gave me that ID in their system instead. I ended up as 34xx instead of 34xxx, more or less.

This was probably my first hint that their "amazing infra" was in fact the same kind of random crazytown as everywhere else once you got to see behind the curtain.

...

Then there was the time that someone decided that a log storage system that had something like a quarter of a million machines (and growing fast) feeding it needed a static configuration. The situation unfolded like this:

(person 1) Hey, why is this thing crashing so much?

(person 2) Oh yeah, it's dumping cores constantly! Wow!

(person 1) It's running but there's nothing in the log?

(person 2) Huh, "runtime error ... bad id mapping?"

(person 2) It's been doing this for a month... and wait, other machines are doing it, too!

(person 1) Guess I'll dig into this.

(person 2) "range name webserv_log.building1.phase3 range [1-20000]"

(person 2) But this machine is named webserv20680...

(person 2) Yeah, that's enough for me. Bye!

The machines were named with a ratcheting counter: any time they were assigned to be a web server, they got names like "webserv1", "webserv2", ... and so on up the line. That had been the case all along.

Whoever designed this log system years later decided to put a hard-coded limiter into it. I don't know if they did it because they wanted to feel useful every time it broke so they could race in and fix it, or if they didn't care, or if they truly had no idea that numbers could in fact grow beyond 20000.

Incidentally, that particular "building1.phase3" location didn't even have 20000 machines at that specific moment. It had maybe 15000 of them, but as things went away and came back, the ever-incrementing counter just went up and up and up. So, there _had been_ north of 20K machines in that spot overall, and that wasn't even close to a surprising number.

...

There was a single line that would catch obvious badness at a particular gig where we had far too many Apache web servers running on various crusty Linux distributions:

locate access_log | xargs ls -la | grep 2147

It was what I'd send in chat to someone who said "hey, the customer's web server won't stay up". The odds were very good that they had a log file that had grown to 2.1 GB, and had hit a hard limit which was present in that particular system. Apache would try to write to it, that write would fail, and the whole process would abort.

"2147", of course, is the first 4 digits of the expected file size: 2147483647 ... or (2^31)-1.

Yep, that's another one of those "not enough bits" problems like the earlier story, but this one is 32 bits with one of them being for the sign, not 16 like before. It's the same problem, though: the counter maxes out and you're done.

These days, files can get quite a bit bigger... but you should still rotate your damn log files once in a while. You should probably also figure out what's pooping in them so much and try to clean that up, too!

...

As the last one for now, there was an outage where someone reported that something like half of their machines were down. They had tried to do a kernel update, and wound up hitting half of them at once. I suspect they wanted to do a much smaller quantity, but messed up and hit fully half of them somehow. Or, maybe they pointed it at all of them, and only half succeeded at it. Whatever the cause, they now had 1000 freshly-rebooted machines.

The new kernel was fine, and the usual service manager stuff came back up, and it went to start the workload for those systems, and then it would immediately crash. It would try to start it again. It would crash again. Crash crash crash. This is why we call it "crashlooping".

Finally, the person in question showed up in the usual place where we discussed outages, and started talking about what was going on.

(person 1) Our stuff isn't coming back.

(person 2) Oh yeah, that's bad, they're all trying to start.

(person 1) Start, abort, start, abort, ...

(person 2) Yep, aborting... right about here: company::project::client::BlahClient::loadConfig ... which is this code: <paste>

(person 2) It's calling "get or throw" on a map for an ID number...

(person 1) My guess is the config provider service isn't running.

(person 2) It's there... it's been up for 30 minutes...

(person 1) Restarting the jobs.

(person 2) Nooooooooooo...

<time passes>

(person 2) Why is there no entry for number 86 in the map in the config?

(person 1) Oh, I bet it's problems with port takeover.

(person 3) I think entry 86 is missing from <file>.

(person 2) Definitely is missing.

(person 4) Hey everyone, we removed that a while back. Why would it only be failing now?

(person 2) It's only loaded at startup, right?

(person 4) Right.

(person 2) So if they were running for a long time, then it changed, then they're toast after a restart...

(person 3) Hey, this change looks related.

(person 4) I'm going to back that out.

This is a common situation: program A reads config C. When it starts up, config C is on version C1, and everything is fine. While A is running, the config is updated from C1 to C2, but nothing notices. Later, A tries to restart and it chokes on the C2 config, and refuses to start.

Normally, you'd only restart a few things to get started, and you'd notice that your program can't consume the new config at that point. You'd still have a few instances down, but that's it - a *few* instances. Your service should keep running on whatever's left over that you purposely didn't touch.

This is why you strive to release things in increments.

Also, it helps when programs notice config changes while they're running, so this doesn't sneak up on you much later when you're trying to restart. If the programs notice the bad config right after the change is made, it's *far* easier to correlate it to the change just by looking at the timeline.

Tuesday, 11:23:51: someone applies change.

Tuesday, 11:23:55: first 1% of machines which subscribe to the change start complaining.

... easy, right? Now compare it to this:

Tuesday, November 18: someone applies a change

Wednesday, January 7: 50% of machines fail to start "for some reason"

That's a lot harder to nail down.

...

Random aside: restarting the jobs did not help. They were already restarting themselves. "Retry, reboot, reinstall, repeat" is NOT a strategy for success.

It was not the config system being down. It was up the whole time.

It was nothing to do with "port takeover". What does that have to do with a config file being bad?

The evidence was there: the processes were crashing. They were logging a message about WHY they were killing themselves. It included a number they wanted to see, but couldn't find. It also said what part of the code was blowing up.

*That* is where you start looking. You don't just start hammering random things.

I've been studying the standard cell circuitry in the Intel 386 processor recently. The 386, introduced in 1985, was Intel's most complex processor at the time, containing 285,000 transistors. Intel's existing design techniques couldn't handle this complexity and the chip began to fall behind schedule. To meet the schedule, the 386 team started using a technique called standard cell logic. Instead of laying out each transistor manually, the layout process was performed by a computer.

The idea behind standard cell logic is to create standardized circuits (standard cells) for each type of logic element, such as an inverter, NAND gate, or latch. You feed your circuit description into software that selects the necessary cells, positions these cells into columns, and then routes the wiring between the cells. This "automatic place and route" process creates the chip layout much faster than manual layout. However, switching to standard cells was a risky decision since if the software couldn't create a dense enough layout, the chip couldn't be manufactured. But in the end, the 386 finished ahead of schedule, an almost unheard-of accomplishment.1

The 386's standard cell circuitry contains a few circuits that I didn't expect. In this blog post, I'll take a quick look at some of these circuits: surprisingly large multiplexers, a transistor that doesn't fit into the standard cell layout, and inverters that turned out not to be inverters. (If you want more background on standard cells in the 386, see my earlier post, "Reverse engineering standard cell logic in the Intel 386 processor".)

The photo below shows the 386 die with the automatic-place-and-route regions highlighted; I'm focusing on the red region in the lower right. These blocks of logic have cells arranged in rows, giving them a characteristic striped appearance. The dark stripes are the transistors that make up the logic gates, while the lighter regions between the stripes are the "routing channels" that hold the wiring that connects the cells. In comparison, functional blocks such as the datapath on the left and the microcode ROM in the lower right were designed manually to optimize density and performance, giving them a more solid appearance.

As for other features on the chip, the black circles around the border are bond wire connections that go to the chip's external pins. The chip has two metal layers, a small number by modern standards, but a jump from the single metal layer of earlier processors such as the 286. (Providing two layers of metal made automated routing practical: one layer can hold horizontal wires while the other layer can hold vertical wires.) The metal appears white in larger areas, but purplish where circuitry underneath roughens its surface. The underlying silicon and the polysilicon wiring are obscured by the metal layers.

The giant multiplexers

The standard cell circuitry that I'm examining (red box above) is part of the control logic that selects registers while executing an instruction. You might think that it is easy to select which registers take part in an instruction, but due to the complexity of the x86 architecture, it is more difficult. One problem is that a 32-bit register such as EAX can also be treated as the 16-bit register AX, or two 8-bit registers AH and AL. A second problem is that some instructions include a "direction" bit that switches the source and destination registers. Moreover, sometimes the register is specified by bits in the instruction, but in other cases, the register is specified by the microcode. Due to these factors, selecting the registers for an operation is a complicated process with many cases, using control bits from the instruction, from the microcode, and from other sources.

Three registers need to be selected for an operation—two source registers and a destination register—and there are about 17 cases that need to be handled. Registers are specified with 7-bit control signals that select one of the 30 registers and control which part of the register is accessed. With three control signals, each 7 bits wide, and about 17 cases for each, you can see that the register control logic is large and complicated. (I wrote more about the 386's registers here.)

I'm still reverse engineering the register control logic, so I won't go into details. Instead, I'll discuss how the register control circuit uses multiplexers, implemented with standard cells. A multiplexer is a circuit that combines multiple input signals into a single output by selecting one of the inputs.2 A multiplexer can be implemented with logic gates, for instance, by ANDing each input with the corresponding control line, and then ORing the results together. However, the 386 uses a different approach—CMOS switches—that avoids a large AND/OR gate.

The schematic above shows how a CMOS switch is constructed from two MOS transistors. When the two transistors are on, the output is connected to the input, but when the two transistors are off, the output is isolated. An NMOS transistor is turned on when its input is high, but a PMOS transistor is turned on when its input is low. Thus, the switch uses two control inputs, one inverted. The motivation for using two transistors is that an NMOS transistor is better at pulling the output low, while a PMOS transistor is better at pulling the output high, so combining them yields the best performance.3 Unlike a logic gate, the CMOS switch has no amplification, so a signal is weakened as it passes through the switch. As will be seen below, inverters can be used to amplify the signal.

The image below shows how CMOS switches appear under the microscope. This image is very hard to interpret because the two layers of metal on the 386 are packed together densely, but you can see that some wires run horizontally and others run vertically. The bottom layer of metal (called M1) runs vertically in the routing area, as well as providing internal wiring for a cell. The top layer of metal (M2) runs horizontally; unlike M1, the M2 wires can cross a cell. The large circles are vias that connect the M1 and M2 layers, while the small circles are connections between M1 and polysilicon or M1 and silicon. The central third of the image is a column of standard cells with two CMOS switches outlined in green. The cells are bordered by the vertical ground rail and +5V rail that power the cells. The routing areas are on either side of the cells, holding the wiring that connects the cells.

Removing the metal layers reveals the underlying silicon with a layer of polysilicon wiring on top. The doped silicon regions show up as dark outlines. I've drawn the polysilicon in green; it forms a transistor (brighter green) when it crosses doped silicon. The metal ground and power lines are shown in blue and red, respectively, with other metal wiring in purple. The black dots are vias between layers. Note how metal wiring (purple) and polysilicon wiring (green) are combined to route signals within the cell. Although this standard cell is complicated, the important thing is that it only needs to be designed once. The standard cells for different functions are all designed to have the same width, so the cells can be arranged in columns, snapped together like Lego bricks.

To summarize, this switch circuit allows the input to be connected to the output or disconnected, controlled by the select signal. This switch is more complicated than the earlier schematic because it includes two inverters to amplify the signal. The data input and the two select lines are connected to the polysilicon (green); the cell is designed so these connections can be made on either side. At the top, the input goes through a standard two-transistor inverter. The lower left has two transistors, combining the NMOS half of an inverter with the NMOS half of the switch. A similar circuit on the right combines the PMOS part of an inverter and switch. However, because PMOS transistors are weaker, this part of the circuit is duplicated.

A multiplexer is constructed by combining multiple switches, one for each input. Turning on one switch will select the corresponding input. For instance, a four-to-one multiplexer has four switches, so it can select one of the four inputs.

The schematic above shows a hypothetical multiplexer with four inputs. One optimization is that if an input is always 0, the PMOS transistor can be omitted. Likewise, if an input is always 1, the NMOS transistor can be omitted. One set of select lines is activated at a time to select the corresponding input. The pink circuit selects 1, green selects input A, yellow selects input B, and blue selects 0. The multiplexers in the 386 are similar, but have more inputs.

The diagram below shows how much circuitry is devoted to multiplexers in this block of standard cells. The green, purple, and red cells correspond to the multiplexers driving the three register control outputs. The yellow cells are inverters that generate the inverted control signals for the CMOS switches. This diagram also shows how the automatic layout of cells results in a layout that appears random.

The misplaced transistor

The idea of standard-cell logic is that standardized cells are arranged in columns. The space between the cells is the "routing channel", holding the wiring that links the cells. The 386 circuitry follows this layout, except for one single transistor, sitting between two columns of cells.

I wrote some software tools to help me analyze the standard cells. Unfortunately, my tools assumed that all the cells were in columns, so this one wayward transistor caused me considerable inconvenience.

The transistor turns out to be a PMOS transistor, pulling a signal high as part of a multiplexer. But why is this transistor out of place? My hypothesis is that the transistor is a bug fix. Regenerating the cell layout was very costly, taking many hours on an IBM mainframe computer. Presumably, someone found that they could just stick the necessary transistor into an unused spot in the routing channel, manually add the necessary wiring, and avoid the delay of regenerating all the cells.

The fake inverter

The simplest CMOS gate is the inverter, with an NMOS transistor to pull the output low and a PMOS transistor to pull the output high. The standard cell circuitry that I examined contains over a hundred inverters of various sizes. (Performance is improved by using inverters that aren't too small but also aren't larger than necessary for a particular circuit. Thus, the standard cell library includes inverters of multiple sizes.)

The image below shows a medium-sized standard-cell inverter under the microscope. For this image, I removed the two metal layers with acid to show the underlying polysilicon (bright green) and silicon (gray). The quality of this image is poor—it is difficult to remove the metal without destroying the polysilicon—but the diagram below should clarify the circuit. The inverter has two transistors: a PMOS transistor connected to +5 volts to pull the output high when the input is 0, and an NMOS transistor connected to ground to pull the output low when the input is 1. (The PMOS transistor needs to be larger because PMOS transistors don't function as well as NMOS transistors due to silicon physics.)

The polysilicon input line plays a key role: where it crosses the doped silicon, a transistor gate is formed. To make the standard cell more flexible, the input to the inverter can be connected on either the left or the right; in this case, the input is connected on the right and there is no connection on the left. The inverter's output can be taken from the polysilicon on the upper left or the right, but in this case, it is taken from the upper metal layer (not shown). The power, ground, and output lines are in the lower metal layer, which I have represented by the thin red, blue, and yellow lines. The black circles are connections between the metal layer and the underlying silicon.

This inverter appears dozens of times in the circuitry. However, I came across a few inverters that didn't make sense. The problem was that the inverter's output was connected to the output of a multiplexer. Since an inverter is either on or off, its value would clobber the output of the multiplexer.4 This didn't make any sense. I double- and triple-checked the wiring to make sure I hadn't messed up. After more investigation, I found another problem: the input to a "bad" inverter didn't make sense either. The input consisted of two signals shorted together, which doesn't work.

Finally, I realized what was going on. A "bad inverter" has the exact silicon layout of an inverter, but it wasn't an inverter: it was independent NMOS and PMOS transistors with separate inputs. Now it all made sense. With two inputs, the input signals were independent, not shorted together. And since the transistors were controlled separately, the NMOS transistor could pull the output low in some circumstances, the PMOS transistor could pull the output high in other circumstances, or both transistors could be off, allowing the multiplexer's output to be used undisturbed. In other words, the "inverter" was just two more cases for the multiplexer.

If you compare the "bad inverter" cell below with the previous cell, they look almost the same, but there are subtle differences. First, the gates of the two transistors are connected in the real inverter, but disconnected by a small gap in the transistor pair. I've indicated this gap in the photo above; it is hard to tell if the gap is real or just an imaging artifact, so I didn't spot it. The second difference is that the "fake" inverter has two input connections, one to each transistor, while the inverter has a single input connection. Unfortunately, I assumed that the two connections were just a trick to route the signal across the inverter without requiring an extra wire. In total, this cell was used 32 times as a real inverter and 9 times as independent transistors.

Conclusions

Standard cell logic and automatic place and route have a long history before the 386, back to the early 1970s, so this isn't an Intel invention.5 Nonetheless, the 386 team deserves the credit for deciding to use this technology at a time when it was a risky decision. They needed to develop custom software for their placing and routing needs, so this wasn't a trivial undertaking. This choice paid off and they completed the 386 ahead of schedule. The 386 ended up being a huge success for Intel, moving the x86 architecture to 32 bits and defining the dominant computer architecture for the rest of the 20th century.

If you're interested in standard cell logic, I also wrote about standard cell logic in an IBM chip. I plan to write more about the 386, so follow me on Mastodon, Bluesky, or RSS for updates. Thanks to Pat Gelsinger and Roxanne Koester for providing helpful papers.

For more on the 386 and other chips, follow me on Mastodon (@kenshirriff@oldbytes.space), Bluesky (@righto.com), or RSS. (I've given up on Twitter.) If you want to read more about the 386, I've written about the clock pin, prefetch queue, die versions, packaging, and I/O circuits.

Notes and references

A couple of weeks ago I started a fundraiser for the Greater Chicago Food Depository: get Logic for Programmers 50% off and all the royalties will go to charity.¹ Since then, we've raised a bit over $1600. Y'all are great!

The fundraiser is going on until the end of November, so you still have one more week to get the book real cheap.

I feel a bit weird about doing two newsletter adverts without raw content, so here's a teaser from a old project I really need to get back to. Notes on structured concurrency argues that old languages had a "old-testament fire-and-brimstone goto" that could send control flow anywhere, like from the body of one function into the body of another function. This "wild goto", the article claims, what Dijkstra was railing against in Go To Statement Considered Harmful, and that modern goto statements are much more limited, "tame" if you will, and wouldn't invoke Dijkstra's ire.

I've shared this historical fact about Dijkstra many times, but recently two separate people have told me it doesn't makes sense: Dijkstra used ALGOL-60, which already had tame gotos. All of the problems he raises with goto hold even for tame ones, none are exclusive to wild gotos. So

This got me looking to see which languages, if any, ever had the wild goto. I define this as any goto which lets you jump from outside to into a loop or function scope. Turns out, FORTRAN had tame gotos from the start, BASIC has wild gotos, and COBOL is a nonsense language intentionally designed to horrify me. I mean, look at this:

The COBOL ALTER statement changes a goto's target at runtime.

(Early COBOL has tame gotos but only on a technicality: there are no nested scopes in COBOL so no jumping from outside and into a nested scope.)

Anyway I need to write up the full story (and complain about COBOL more) but this is pretty neat! Reminder, fundraiser here. Let's get it to 2k.

From now until the end of the month, you can get Logic for Programmers at half price with the coupon feedchicago. All royalties from that coupon will go to the Greater Chicago Food Depository. Thank you!

Hi everyone,

I've been getting burnt out on writing a weekly software essay. It's gone from taking me an afternoon to write a post to taking two or three days, and that's made it really difficult to get other writing done. That, plus some short-term work and life priorities, means now feels like a good time for a break.

So I'm taking off from Computer Things for the rest of the year. There might be some announcements and/or one or two short newsletters in the meantime but I won't be attempting a weekly cadence until 2026.

Thanks again for reading!

Hillel

A while back my friend Pablo Meier was reviewing some 2024 videogames and wrote this:

I feel like some artists, if they didn't exist, would have the resulting void filled in by someone similar (e.g. if Katy Perry didn't exist, someone like her would have). But others don't have successful imitators or comparisons (thinking Jackie Chan, or Weird Al): they are irreplaceable.

He was using it to describe auteurs but I see this as a property of opportunity, in that "replaceable" artists are those who work in bigger markets. Katy Perry's market is large, visible and obviously (but not easily) exploitable, so there are a lot of people who'd compete in her niche. Weird Al's market is unclear: while there were successful parody songs in the past, it wasn't clear there was enough opportunity there to support a superstar.

I think that modal editing is in the latter category. Vim is now very popular and has spawned numerous successors. But its key feature, modes, is not obviously-beneficial, to the point that if Bill Joy didn't make vi (vim's direct predecessor) fifty years ago I don't think we'd have any modal editors today.

A quick overview of "modal editing"

In a non-modal editor, pressing the "u" key adds a "u" to your text, as you'd expect. In a modal editor, pressing "u" does something different depending on the "mode" you are in. In Vim's default "normal" mode, "u" undoes the last change to the text, while in the "visual" mode it lowercases all selected text. It only inserts the character in "insert" mode. All other keys, as well as chorded shortcuts (ctrl-x), work the same way.

The clearest benefit to this is you can densely pack the keyboard with advanced commands. The standard US keyboard has 48ish keys dedicated to inserting characters. With the ctrl and shift modifiers that becomes at least ~150 extra shortcuts for each other mode. This is also what IMO "spiritually" distinguishes modal editing from contextual shortcuts. Even if a unimodal editor lets you change a keyboard shortcut's behavior based on languages or focused panel, without global user-controlled modes it simply can't achieve that density of shortcuts.

Now while modal editing today is widely beloved (the Vim plugin for VSCode has at least eight million downloads), I suspect it was "carried" by the popularity of vi, as opposed to driving vi's popularity.

Modal editing is an unusual idea

Pre-vi editors weren't modal. Some, like EDT/KED, used chorded commands, while others like ed or TECO basically REPLs for text-editing DSLs. Both of these ideas widely reappear in modern editors.

As far as I can tell, the first modal editor was Butler Lampson's Bravo in 1974. Bill Joy admits he used it for inspiration:

A lot of the ideas for the screen editing mode were stolen from a Bravo manual I surreptitiously looked at and copied. Dot is really the double-escape from Bravo, the redo command. Most of the stuff was stolen.

Bill Joy probably took the idea because he was working on dumb terminals that were slow to register keystrokes, which put pressure to minimize the number needed for complex operations.

Why did Bravo have modal editing? Looking at the Alto handbook, I get the impression that Xerox was trying to figure out the best mouse and GUI workflows. Bravo was an experiment with modes, one hand on the mouse and one issuing commands on the keyboard. Other experiments included context menus (the Markup program) and toolbars (Draw).

Xerox very quickly decided against modes, as the successors Gypsy and BravoX were modeless. Commands originally assigned to English letters were moved to graphical menus, special keys, and chords.

It seems to me that modes started as an unsuccessful experiment deal with a specific constraint and then later successfully adopted to deal with a different constraint. It was a specialized feature as opposed to a generally useful feature like chords.

Modal editing didn't popularize vi

While vi was popular at Bill Joy's coworkers, he doesn't attribute its success to its features:

I think the wonderful thing about vi is that it has such a good market share because we gave it away. Everybody has it now. So it actually had a chance to become part of what is perceived as basic UNIX. EMACS is a nice editor too, but because it costs hundreds of dollars, there will always be people who won't buy it.

Vi was distributed for free with the popular BSD Unix and was standardized in POSIX Issue 2, meaning all Unix OSes had to have vi. That arguably is what made it popular, and why so many people ended up learning a modal editor.

Modal editing doesn't really spread outside of vim

I think by the 90s, people started believing that modal editing was a Good Idea, if not an obvious one. That's why we see direct descendants of vi, most famously vim. It's also why extensible editors like Emacs and VSCode have vim-mode extensions, but these are but these are always simple emulation layers on top of a unimodal baselines. This was good for getting people used to the vim keybindings (I learned on Kile) but it means people weren't really doing anything with modal editing. It was always "The Vim Gimmick".

Modes also didn't take off anywhere else. There's no modal word processor, spreadsheet editor, or email client.¹ Visidata is an extremely cool modal data exploration tool but it's pretty niche. Firefox used to have vimperator (which was inspired by Vim) but that's defunct now. Modal software means modal editing which means vi.

This has been changing a little, though! Nowadays we do see new modal text editors, like kakoune and Helix, that don't just try to emulate vi but do entirely new things. These were made, though, in response to perceived shortcomings in vi's editing model. I think they are still classifiable as descendants. If vi never existed, would the developers of kak and helix have still made modal editors, or would they have explored different ideas?

People aren't clamouring for more experiments

Not too related to the overall picture, but a gripe of mine. Vi and vim have a set of hardcoded modes, and adding an entirely new mode is impossible. Like if a plugin (like vim's default netrw) adds a file explorer it should be able to add a filesystem mode, right? But it can't, so instead it waits for you to open the filesystem and then adds 60 new mappings to normal mode. There's no way to properly add a "filesystem" mode, a "diff" mode, a "git" mode, etc, so plugin developers have to mimic them.

I don't think people see this as a problem, though! Neovim, which aims to fix all of the baggage in vim's legacy, didn't consider creating modes an important feature. Kak and Helix, which reimagine modal editing from from the ground up, don't support creating modes either.² People aren't clamouring for new modes!

Modes are a niche power user feature

So far I've been trying to show that vi is, in Pablo's words, "irreplaceable". Editors weren't doing modal editing before Bravo, and even after vi became incredibly popular, unrelated editors did not adapt modal editing. At most, they got a vi emulation layer. Kak and helix complicate this story but I don't think they refute it; they appear much later and arguably count as descendants (so are related).

I think the best explanation is that in a vacuum modal editing sounds like a bad idea. The mode is global state that users always have to know, which makes it dangerous. To use new modes well you have to memorize all of the keybindings, which makes it difficult. Modal editing has a brutal skill floor before it becomes more efficient than a unimodal, chorded editor like VSCode.

That's why it originally appears in very specific circumstances, as early experiments in mouse UX and as a way of dealing with modem latencies. The fact we have vim today is a historical accident.

And I'm glad for it! You can pry Neovim from my cold dead hands, you monsters.

P99 talk this Thursday!

My talk, "Designing Low-Latency Systems with TLA+", is happening 10/23 at 11:40 central time. Tickets are free, the conf is online, and the talk's only 16 minutes, so come check it out!