I have an ongoing fascination with "interactive TV": a series of efforts, starting in the 1990s and continuing today, to drag the humble living room television into the world of the computer. One of the big appeals of interactive TV was adoption, the average household had a TV long before the average household had a computer. So, it seems like interactive TV services should have proliferated before personal computers, at least following the logic that many in the industry did at the time.

This wasn't untrue! In the UK, for example, Ceefax was a widespread success by the 1980s. In general, TV-based teletext systems were pretty common in Europe. In North America, they never had much of an impact---but not for lack of trying. In fact, there were multiple competing efforts at teletext in the US and Canada, and it may very well have been the sheer number of independent efforts that sunk the whole idea. But let's start at the beginning.

The BBC went live with Ceefax in 1974, the culmination of years of prototype development and test broadcasts over the BBC network. Ceefax was quickly joined by other teletext standards in Europe, and the concept enjoyed a high level of adoption. This must have caught the attention of many in the television industry on this side of the ocean, but it was Bonneville International that first bit [1]. Its premier holding, KSL-TV of Salt Lake City, has an influence larger than its name suggests: KSL was carried by an extensive repeater network and reached a large portion of the population throughout the Mountain States. Because of the wide reach of KSL and the even wider reach of the religion that relied on Bonneville for communications, Bonneville was also an early innovator in satellite distribution of television and data. These were ingredients that made for a promising teletext network, one that could quickly reach a large audience and expand to broader television networks through satellite distribution.

KSL applied to the FCC for an experimental license to broadcast teletext in addition to its television signal, and received it in June of 1978. I am finding some confusion in the historical record over whether KSL adopted the BBC's Ceefax protocol or the competing ORACLE, used in the UK by the independent broadcasters. A 1982 paper on KSL's experiment confusingly says they used "the British CEEFAX/Oracle," but then in the next sentence the author gives the first years of service for Ceefax and ORACLE the wrong way around, so I think it's safe to say that they were just generally confused. I think I know the reason why: in the late '70s, the British broadcasters were developing something called World System Teletext (WST), a new common standard based on aspects of both Ceefax and ORACLE. Although WST wasn't quite final in 1978, I believe that what KSL adopted was actually a draft of WST.

That actually hints at an interesting detail which becomes important to these proposals: in Europe, where teletext thrived, there were usually not very many TV channels. The US's highly competitive media landscape lead to a proliferation of different TV networks, and local operations in addition. It was a far cry from the UK, for example, where 1982 saw the introduction of a fourth channel called, well, Channel 4. By contrast, Salt Lake City viewers with cable were picking from over a dozen channels in 1982, and that wasn't an especially crowded media market. This difference in the industry, between a few major nationwide channels and a longer list of often local ones, has widespread ramifications on how UK and US television technology evolved.

One of them is that, in the UK, space in the VBI to transmit data became a hotly contested commodity. By the '80s, obtaining a line of the VBI on any UK network to use for your new datacasting scheme involved a bidding war with your potential competitors, not unlike the way spectrum was allocated in the US. Teletext schemes were made and broken by the outcomes of these auctions. Over here, there was a long list of television channels and on most of them only a single line of the VBI was in use for data (line 21 for closed captions). You might think this would create fertile ground for VBI-based services, but it also posed a challenge: the market was extensively fractured. You could not win a BBC or IBA VBI allocation and then have nationwide coverage, you would have to negotiate such a deal with a long list of TV stations and then likely provide your own infrastructure for injecting the signal.

In short, this seems to be one of the main reasons for the huge difference in teletext adoption between Europe and North America: throughout Europe, broadcasting tended to be quite centralized, which made it difficult to get your foot in the door but very easy to reach a large customer base once you had. In the US, it was easier to get started, but you had to fight for each market area. "Critical mass" was very hard to achieve [2].

Back at KSL, $40,000 (~$200,000 today) bought a General Automation computer and Tektronix NTSC signal generator that made up the broadcast system. The computer could manage as many as 800 pages of 20x32 teletext, but KSL launched with 120. Texas Instruments assisted KSL in modifying thirty television sets with a new decoder board and a wired remote control for page selection. This setup, very similar to teletext sets in Europe, nearly doubled the price of the TV set. This likely would have become a problem later on, but for the pilot stage, KSL provided the modified sets gratis to their 30 test households.

One of the selling points of teletext in Europe was its ability to provide real-time data. Things like sports scores and stock quotations could be quickly updated in teletext, and news headlines could make it to teletext before the next TV news broadcast. Of course, collecting all that data and preparing it as teletext pages required either a substantial investment in automation or a staff of typists. At the pilot stage, KSL opted for neither, so much of the information that KSL provided was out-of-date. It was very much a prototype. Over time, KSL invested more in the system. In 1979, for example, KSL partnered with the National Weather Service to bring real-time weather updates to teletext---all automatically via the NWS's computerized system called AFOS.

At that time, KSL was still operating under an experimental license, one that didn't allow them to onboard customers beyond their 30-set test market. The goal was to demonstrate the technology and its compatibility with the broader ecosystem. In 1980, the FCC granted a similar experimental license to CBS affiliated KMOX in St. Louis, who started a similar pilot effort using a French system called Antiope. Over the following few years, the FCC allowed expansion of this test to other CBS affiliates including KNXT in Los Angeles. To emphasize the educational and practical value of teletext (and no doubt attract another funding source), CBS partnered with Los Angeles PBS affiliate KCET who carried their own Teletext programming with a characteristic slant towards enrichment. Meanwhile, in Chicago, station WFLD introduced a teletext service called Keyfax, built on Ceefax technology as a joint venture with Honeywell and telecom company Centel. Despite the lack of consumer availability, teletext was becoming a crowded field---and for the sake of narrative simplicity I am leaving out a whole set of other North American ventures right now.

In 1983, there were at least a half dozen stations broadcasting teletext based on British or French technology, and yet, there were zero teletext decoders on the US market. Besides their use of an experimental license, the teletext pilot projects were constrained by the need for largely custom prototype decoders integrated into customer's television sets. Broadcast executives promised the price could come down to $25, but the modifications actually available continued to cost in the hundreds. The director of public affairs at KSL, asked about this odd conundrum of a nearly five-year-old service that you could not buy, pointed out that electronics manufacturers were hesitant to mass produce an inexpensive teletext decoder as long as it was unclear which of several standards would prevail. The reason that no one used teletext, then, was in part the sheer number of different teletext efforts underway. And, of course, things were looking pretty evenly split: CBS had fully endorsed the French-derived system, and was a major nationwide TV network. But most non-network stations with teletext projects had gone the British route. In terms of broadcast channels, it was looking about 50/50.

Further complicating things, teletext proper was not the only contender. There was also videotex. The terminology has become somewhat confused, but I will stick to the nomenclature used in the 1980s: teletext services used a continuous one-way broadcast of every page and decoders simply displayed the requested page when it came around in the loop. Videotex systems were two-way, with the customer using a phone line to request a specific page which was then sent on-demand. Videotex systems tended to operate over telephone lines rather than television cable, but were frequently integrated into television sets. Videotex is not as well remembered as teletext because it was a massive commercial failure, with the very notable exception of the French Minitel.

But in the '80s they didn't know that yet, and the UK had its own videotex venture called Prestel. Prestel had the backing of the Post Office, because they ran the telephones and thus stood to make a lot of money off of it. For the exact same reason, US telephone company GTE bought the rights to the system in the US.

Videotex is significantly closer to "the internet" in its concept than teletext, and GTE was entering a competitive market. In 1981, Radio Shack had introduced a videotex terminal for several years already, a machine originally developed as the "AgVision" for use with an experimental Kentucky agricultural videotex service and then offered nationwide. This creates an amusing irony: teletext services existed but it was very difficult to obtain a decoder to use them. Radio Shack was selling a videotex client nationwide, but what service would you use it with? In practice, the "TRS-80 Videotex" as the AgVision came to be known was used mostly as a client for CompuServe and Dow Jones. Neither of these were actually videotex services, using neither the videotex UX model nor the videotex-specific features of the machine. The TRS-80 Videotex was reduced to just a slightly weird terminal with a telephone modem, and never sold well until Radio Shack beefed it up into a complete microcomputer and relaunched it as the TRS-80 Color Computer.

Radio Shack also sold a backend videotex system, and apparently some newspapers bought it in an effort to launch a "digital edition." The only one to achieve long-term success seems to have been StarText, a service of the Fort Worth Star-Telegram. It was popular enough to be remembered by many from the Fort Worth area, but there was little national impact. It was clearly not enough to float sales of the TRS-80 Videotex and the whole thing has been forgotten. Well, with such a promising market, GTE brought its US Prestel service to market in 1982. As the TRS-80 dropped its Videotex ambitions, Zenith launched a US television set with a built-in Prestel client.

Prestel wasn't the only videotex operation, and GTE wasn't the only company marketing videotex in the US. If the British Post Office and GTE thought they could make money off of something, you know AT&T was somewhere around. They were, and in classic AT&T fashion. During the 1970s, the Canadian Communications Research Center developed a vector-based drawing system. Ontario manufacturer Norpak developed a consumer terminal that could request full-color pages from this system using a videotex-like protocol. Based on the model of Ceefax, the CRC designed a system called Telidon that worked over television (in a more teletext-like fashion) or phone lines (like videotex), with the capability of publishing far more detailed graphics than the simple box drawings of teletext.

Telidon had several cool aspects, like the use of a pretty complicated vector-drawing terminal and a flexible protocol designed for interoperability between different communications media. That's the kind of thing AT&T loved, so they joined the effort. With CRC, AT&T developed NABTS, the North American Broadcast Teletext Specification---based on Telidon and intended for one-way broadcast over TV networks.

NABTS was complex and expensive compared to Ceefax/ORACLE/WST based systems. A review of KSL's pilot notes how the $40,000 budget for their origination system compared to the cost quoted by AT&T for an NABTS headend: as much as $2 million. While KSL's estimates of $25 for a teletext decoder had not been achieved, the prototypes were still running cheaper than NABTS clients that ran into the hundreds. Still, the graphical capabilities of NABTS were immediately impressive compared to text-only services. Besides, the extensibility of NABTS onto telephone systems, where pages could be delivered on-demand, made it capable of far larger databases.

When KSL first introduced teletext, they spoke of a scheme where a customer could call a phone number and, via DTMF menus, request an "extended" page beyond the 120 normally transmitted. They could then request that page on their teletext decoder and, at the end of the normal 120 page loop, it would be sent just for them. I'm not sure if that was ever implemented or just a concept. In any case, videotex systems could function this way natively, with pages requested and retrieved entirely by telephone modem, or using hybrid approaches.

NABTS won the support of NBC, who launched a pilot NABTS service (confusingly called NBC Teletext) in 1981 and went into full service in 1983. CBS wasn't going to be left behind, and trialed and launched NABTS (as CBS ExtraVision) at the same time. That was an ignominious end for CBS's actual teletext pilot, which quietly shut down without ever having gone into full service. ExtraVision and NBC Teletext are probably the first US interactive TV services that consumers could actually buy and use.

Teletext was not dead, though. In 1982, Cincinnati station WKRC ran test broadcasts for a WST-based teletext service called Electra. WKRC's parent company, Taft, partnered with Zenith to develop a real US-market consumer WST decoder for use with the Electra service. In 1983, the same year that ExtraVision and CBS Teletext went live, Zenith teletext decoders appeared on the shelves of Cincinnati stores. They were plug-in modules for recent Zenith televisions, meaning that customers would likely also need to buy a whole new TV to use the service... but it was the only option, and seems to have remained that way for the life of US teletext.

I believe that Taft's Electra was the first teletext service to achieve a regular broadcast license. Through the mid 1980s, Electra would expand to more television stations, reaching similar penetration to the videotex services. In 1982, KeyFax (remember KeyFax? it was the one on WFLD in Chicago) had made the pivot from teletext to videotex as well, adopting the Prestel-derived technology from GTE. In 1984, KeyFax gave up on their broadcast television component and became a telephone modem service only. Electra jumped on the now-free VBI lines of WFLD and launched in Chicago. WTBS in Atlanta carried Electra, and then in the biggest expansion of teletext, Electra appeared on SPN---a satellite network that would later become CNBC.

While major networks, and major companies like GTE and AT&T, pushed for the videotex NABTS, teletext continued to have its supporters among independent stations. Los Angeles's KTTV started its own teletext service in 1984, which combined locally-developed pages with national news syndicated from Electra. This seemed like the start of a promising model for teletext across independent stations, but it wasn't often repeated.

Oh, and KSL? at some point, uncertain to me but before 1984, they switched to NABTS.

Let's stop for a moment and recap the situation. Between about 1978 and 1984, over a dozen major US television stations launched interactive TV offerings using four major protocols that fell into two general categories. One of those categories was one-way over television while the other was two-way over telephone or one-way over television with some operators offering both. Several TV stations switched between types. The largest telcos and TV networks favored one option, but it was significantly more expensive than the other, leading smaller operators to choose differently. The hardware situation was surprisingly straightforward in that, within teletext and videotex, consumers only had one option and it was very expensive.

Oh, and that's just the technical aspects. The business arrangements could get even stranger. Teletext services were generally free, but videotex services often charged a service fee. This was universally true for videotex services offered over telephone and often, but not always, true for videotex services over cable. Were the videotex services over cable even videotex? doesn't that contradict the definition I gave earlier? is that why NBC called their videotex service teletext? And isn't videotex over telephone barely differentiated from computer-based services like CompuServe and The Source that were gaining traction at the same time?

I think this all explains the failure of interactive TV in the 1980s. As you've seen, it's not that no one tried. It's that everyone tried, and they were all tripping over each other the entire time. Even in Canada, where the government had sponsored development of the Telidon system ground-up to be a nationwide standard, the influence of US teletext services created similar confusion. For consumers, there were so many options that they didn't know what to buy, and besides, the price of the hardware was difficult to justify with the few stations that offered teletext. The fact that teletext had been hyped as the "next big thing" by newspapers since 1978, and only reached the market in 1983 as a shambled mess, surely did little for consumer confidence.

You might wonder: where was the FCC during this whole thing? In the US, we do not have a state broadcaster, but we do have state regulation of broadcast media that is really quite strict as to content and form. During the late '70s, under those first experimental licenses, the general perception seemed to be that the FCC was waiting for broadcasters to evaluate the different options before selecting a nationwide standard. Given that the FCC had previously dictated standards for television receivers, it didn't seem like that far of a stretch to think that a national-standard teletext decoder might become mandatory equipment on new televisions.

Well, it was political. The long, odd experimental period from 1978 to 1983 was basically a result of FCC indecision. The commission wasn't prepared to approve anything as a national standard, but the lack of approval meant that broadcasters weren't really allowed to use anything outside of limited experimental programs. One assumes that they were being aggressively lobbied by every side of the debate, which no doubt factored into the FCC's 1981 decision that teletext content would be unregulated, and 1982 statements from commissioners suggesting that the FCC would not, in fact, adopt any technical standards for teletext.

There is another factor wrapped up in this whole story, another tumultuous effort to deliver text over television: closed captioning. PBS introduced closed captioning in 1980, transmitting text over line 21 of the VBI for decoding by a set-top box. There are meaningful technical similarities between closed captioning and teletext, to the extent that the two became competitors. Some broadcasters that added NABTS dropped closed captioning because of incompatibility between the equipment in use. This doesn't seem to have been a real technical constraint, and was perhaps more likely cover for a cost-savings decision, but it generated considerable controversy that lead to the National Association for the Deaf organizing for closed captioning and against teletext.

The topic of closed captioning continued to haunt interactive TV. TV networks tended to view teletext or videotex as the obvious replacements for line 21 closed captioning, due to their more sophisticated technical features. Of course, the problems that limited interactive TV adoption in general, high cost and fragmentation, made it unappealing to the deaf. Closed captioning had only just barely become well-standardized in the mid-1980s and its users were not keen to give it up for another decade of frustration. While some deaf groups did support NABTS, the industry still set up a conflict between closed captioning and interactive TV that must have contributed to the FCC's cold feet.

In April of 1983, at the dawn of US broadcast teletext, the FCC voted 6-1 to allow television networks and equipment manufacturers to support any teletext or videotex protocol of their choice. At the same time, they declined to require cable networks to carry teletext content from broadcast television stations, making it more difficult for any TV network to achieve widespread adoption [3]. The FCC adopted what was often termed a "market-based" solution to the question of interactive TV.

The market would not provide that solution. It had already failed.

In November of 1983, Time ended their teletext service. That's right, Time used to have a TV network and it used to have teletext; it was actually one of the first on the market. It was also the first to fall, but they had company. CBS and NBC had significantly scaled back their NABTS programs, which were failing to make any money because of the lack of hardware that could decode the service.

On the WST side of the industry, Taft reported poor adoption of Electra and Zenith reported that they had sold very few decoders, so few that they were considering ending the product line. Taft was having a hard time anyway, going through a rough reorganization in 1986 that seems to have eliminated most of the budget for Electra. Electra actually seems to have still been operational in 1992, an impressive lifespan, but it says something about the level of adoption that we have to speculate as to the time of death. Interactive TV services had so little adoption that they ended unnoticed, and by 1990, almost none remained.

Conflict with closed captioning still haunted teletext. There had been some efforts towards integrating teletext decoders into TV sets, by Zenith for example, but in 1990 line 21 closed caption decoding became mandatory. The added cost of a closed captioning decoder, and the similarity to teletext, seems to have been enough for the manufacturing industry to decide that teletext had lost the fight. Few, possibly no teletext decoders were commercially available after that date.

In Canada, Telidon met a similar fate. Most Telidon services were gone by 1986, and it seems likely that none were ever profitable. On the other hand, the government-sponsored, open-standards nature of Telidon mean that it and descendants like NABTS saw a number of enduring niche uses. Environment Canada distributed weather data via a dedicated Telidon network, and Transport Canada installed Telidon terminals in airports to distribute real-time advisories. Overall, the Telidon project is widely considered a failure, but it has had enduring impact. The original vector drawing language, the idea that had started the whole thing, came to be known as NAPLPS, the North American Presentation Layer Protocol Syntax. NAPLPS had some conceptual similarities to HTML, as Telidon's concept of interlinking did to the World Wide Web. That similarity wasn't just theoretical: Prodigy, the second largest information service after CompuServe and first to introduce a GUI, ran on NAPLPS. Prodigy is now viewed as an important precursor to the internet, but seen in a different light, it was just another videotex---but one that actually found success.

I know that there are entire branches of North American teletext and videotex and interactive TV services that I did not address in this article, and I've become confused enough in the timeline and details that I'm sure at least one thing above is outright wrong. But that kind of makes the point, doesn't it? The thing about teletext here is that we tried, we really tried, but we badly fumbled it. Even if the internet hadn't happened, I'm skeptical that interactive television efforts would have gotten anywhere without a complete fresh start. And the internet did happen, so abruptly that it nearly killed the whole concept while television carriers were still tossing it around.

Nearly killed... but not quite. Even at the beginning of the internet age, televisions were still more widespread than computers. In fact, from a TV point of view, wasn't the internet a tremendous opportunity? Internet technology and more compact computers could enable more sophisticated interactive television services at lower prices. At least, that's what a lot of people thought. I've written before about Cablesoft and it is just one small part of an entire 1990s renaissance of interactive TV. There's a few major 1980s-era services that I didn't get to here either. Stick around and you'll hear more.

You know what's sort of funny? Remember the AgVision, the first form of the TRS-80? It was built as a client for AGTEXT, a joint project of Kentucky Educational Television (who carried it on the VBI of their television network) and the Kentucky College of Agriculture. At some point, AGTEXT switched over to the line 21 closed captioning protocol and operated until 1998. It was almost the first teletext service, and it may very well have been the last.

[1] There's this weird thing going on where I keep tangentially bringing up Bonnevilles. I think it's just a coincidence of what order I picked topics off of my list but maybe it reflects some underlying truth about the way my brain works. This Bonneville, Bonneville International, is a subsidiary of the LDS Church that owns television and radio stations. It is unrelated, except by being indirectly named after the same person, to the Bonneville Power Administration that operated an early large-area microwave communications network.

[2] There were of course large TV networks in the US, and they will factor into the story later, but they still relied on a network of independent but affiliated stations to reach their actual audience---which meant a degree of technical inconsistency that made it hard to rollout nationwide VBI services. Providing another hint at how centralization vs. decentralization affected these datacasting services, adoption of new datacasting technologies in the US has often been highest among PBS and NPR affiliates, our closest equivalents to something like the BBC or ITV.

[3] The regulatory relationship between broadcast TV stations, cable network TV stations, and cable carriers is a complex one. The FCC's role in refereeing the competition between these different parts of the television industry, which are all generally trying to kill each other off, has lead to many odd details of US television regulation and some of the everyday weirdness of the American TV experience. It's also another area where the US television industry stands in contrast to the European television industry, where state-owned or state-chartered broadcasting meant that the slate of channels available to a consumer was generally the same regardless of how they physically received them. Not so in the US! This whole thing will probably get its own article one day.

One of the most significant single advancements in telecommunications technology was the development of microwave radio. Essentially an evolution of radar, the middle of the Second World War saw the first practical microwave telephone system. By the time Japan surrendered, AT&T had largely abandoned their plan to build an extensive nationwide network of coaxial telephone cables. Microwave relay offered greater capacity at a lower cost. When Japan and the US signed their peace treaty in 1951, it was broadcast from coast to coast over what AT&T called the "skyway": the first transcontinental telephone lead made up entirely of radio waves. The fact that live television coverage could be sent over the microwave system demonstrated its core advantage. The bandwidth of microwave links, their capacity, was truly enormous. Within the decade, a single microwave antenna could handle over 1,000 simultaneous calls.

Microwave's great capacity, its chief advantage, comes from the high frequencies and large bandwidths involved. The design of microwave-frequency radio electronics was an engineering challenge that was aggressively attacked during the war because microwave frequency's short wavelengths made them especially suitable for radar. The cavity magnetron, one of the first practical microwave transmitters, was an invention of such import that it was the UK's key contribution to a technical partnership that lead to the UK's access to US nuclear weapons research. Unlike the "peaceful atom," though, the "peaceful microwave" spread fast after the war. By the end of the 1950s, most long-distance telephone calls were carried over microwave. While coaxial long-distance carriers such as L-carrier saw continued use in especially congested areas, the supremacy of microwave for telephone communications would not fall until adoption of fiber optics in the 1980s.

The high frequency, and short wavelength, of microwave radio is a limitation as well as an advantage. Historically, "microwave" was often used to refer to radio bands above VHF, including UHF. As RF technology improved, microwave shifted higher, and microwave telephone links operated mostly between 1 and 9 GHz. These frequencies are well beyond the limits of beyond-line-of-sight propagation mechanisms, and penetrate and reflect only poorly. Microwave signals could be received over 40 or 50 miles in ideal conditions, but the two antennas needed to be within direct line of sight. Further complicating planning, microwave signals are especially vulnerable to interference due to obstacles within the "fresnel zone," the region around the direct line of sight through which most of the received RF energy passes.

Today, these problems have become relatively easy to overcome. Microwave relays, stations that receive signals and rebroadcast them further along a route, are located in positions of geographical advantage. We tend to think of mountain peaks and rocky ridges, but 1950s microwave equipment was large and required significant power and cooling, not to mention frequent attendance by a technician for inspection and adjustment. This was a tube-based technology, with analog and electromechanical control. Microwave stations ran over a thousand square feet, often of thick hardened concrete in the post-war climate and for more consistent temperature regulation, critical to keeping analog equipment on calibration. Where commercial power wasn't available they consumed a constant supply of diesel fuel. It simply wasn't practical to put microwave stations in remote locations.

In the flatter regions of the country, locating microwave stations on hills gave them appreciably better range with few downsides. This strategy often stopped at the Rocky Mountains.

In much of the American West, telephone construction had always been exceptionally difficult. Open-wire telephone leads had been installed through incredible terrain by the dedication and sacrifice of crews of men and horses. Wire strung over telephone poles proved able to handle steep inclines and rocky badlands, so long as the poles could be set---although inclement weather on the route could make calls difficult to understand. When the first transcontinental coaxial lead was installed, the route was carefully planned to follow flat valley floors whenever possible. This was an important requirement since it was installed mostly by mechanized equipment, heavy machines, which were incapable of navigating the obstacles that the old pole and wire crews had on foot.

The first installations of microwave adopted largely the same strategy. Despite the commanding views offered by mountains on both sides of the Rio Grande Valley, AT&T's microwave stations are often found on low mesas or even at the center of the valley floor. Later installations, and those in the especially mountainous states where level ground was scarce, became more ambitious. At Mt. Rose, in Nevada, an aerial tramway carried technicians up the slope to the roof of the microwave station---the only access during winter when snowpack reached high up the building's walls. Expansion in the 1960s involved increasing use of helicopters as the main access to stations, although roads still had to be graded for construction and electrical service.

These special arrangements for mountain locations were expensive, within the reach of the Long Lines department's monopoly-backed budget but difficult for anyone else, even Bell Operating Companies, to sustain. And the West---where these difficult conditions were encountered the most---also contained some of the least profitable telephone territory, areas where there was no interconnected phone service at all until government subsidy under the Rural Electrification Act. Independent telephone companies and telephone cooperatives, many of them scrappy operations that had expanded out from the manager's personal home, could scarcely afford a mountaintop fortress and a helilift operation to sustain it.

For the telephone industry's many small players, and even the more rural Bell Operating Companies, another property of microwave became critical: with a little engineering, you can bounce it off of a mirror.

James Kreitzberg was, at least as the obituary reads, something of a wunderkind. Raised in Missoula, Montana, he earned his pilots license at 15 and joined the Army Air Corps as soon as he was allowed. The Second World War came to a close shortly after, and so, he went on to the University of Washington where he studied aeronautical engineering and then went back home to Montana, taking up work as an engineer at one of the states' largest electrical utilities. His brother, George, had taken a similar path: a stint in the Marine Corps and an aeronautical engineering degree from Oklahoma. While James worked at Montana Power in Butte, George moved to Salem, Oregon, where he started an aviation company that supplemented their cropdusting revenue by modifying Army-surplus aircraft for other uses.

Montana Power operated hydroelectric dams, coal mines, and power plants, a portfolio of facilities across a sparse and mountainous state that must have made communications a difficult problem. During the 1950s, James was involved in an effort to build a new private telephone system connecting the utility's facilities. It required negotiating some type of obstacle, perhaps a mountain pass. James proposed an idea: a reflector.

Because the wavelength of microwaves are so short, say 10cm, it's practical to build a flat metallic panel that spans multiple wavelengths. Such a panel will function like a reflector or mirror, redirecting microwave energy at an angle proportional to the angle on which it arrived. Much like you can redirect a laser using mirrors, you can also redirect a microwave signal. Some early commenters referred to this technique as a "radio mirror," but by the 1950s the use of "active" microwave repeaters with receivers and transmitters had become well established, so by comparison reflectors came to be known as "passive repeaters."

James believed a passive repeater to be a practical solution, but Montana Power lacked the expertise to build one. For a passive repeater to work efficiently, its surface must be very flat and regular, even under varying temperature. Wind loading had to be accounted for, and the face sufficiently rigid to not flex under the wind. Of course, with his education in aeronautics, James knew that similar problems were encountered in aircraft: the need for lightweight metal structures with surfaces that kept an engineered shape. Wasn't he fortunate, then, that his brother owned a shop that repaired and modified aircraft.

I know very little about the original Montana Power installation, which is unfortunate, as it may very well be the first passive microwave repeater ever put into service. What I do know is that in the fall of 1955, James called his brother George and asked if his company, Kreitzberg Aviation, could fabricate a passive repeater for Montana Power. George, he later recounted, said that "I can build anything you can draw." The repeater was made in a hangar on the side of Salem's McNary Field, erected by the flightline as a test, and then shipped in parts to Montana for reassembly in the field. It worked. It worked so well, in fact, that as word of Montana Power's new telephone system spread, other utilities wrote to inquire about obtaining passive repeaters for their own telephone systems.

In 1956, James Kreitzberg moved to Salem and the two brothers formed the Microflect Company. From the sidelines of McNary Field, Microflect built aluminum "billboards" that can still be found on mountain passes and forested slopes throughout the western United States, and in many other parts of the world where mountainous terrain, adverse weather, and limited utilities made the construction of active repeaters impractical.

Passive repeaters can be used in two basic configurations, defined by the angle at which the signal is reflected. In the first case, the reflection angle is around 90 degrees (the closer to this ideal angle, of course, the more efficiently the repeater performs). This situation is often encountered when there is an obstacle that the microwave path needs to "maneuver" around. For example, a ridge or even a large structure like a building in between two sites. In the second case, the microwave signal must travel in something closer to a straight line---over a mountain pass between two towns, for example. When the reflection angle is greater than 135 degrees, the use of a single passive repeater becomes inefficient or impossible, so Microflect recommends the use of two. Arranged like a dogleg or periscope, the two repeaters reflect the signal to the side and then onward in the intended direction.

Microflect published an excellent engineering manual with many examples of passive repeater installations along with the signal calculations. You might think that passive repeaters would be so inefficient as to be impractical, especially when more than one was required, but this is surprisingly untrue. Flat aluminum panels are almost completely efficient reflectors of microwave, and somewhat counterintuitively, passive repeaters can even provide gain.

In an active repeater, it's easy to see how gain is achieved: power is added. A receiver picks up a signal, and then a powered transmitter retransmits it, stronger than it was before. But passive repeaters require no power at all, one of their key advantages. How do they pull off this feat? The design manual explains with an ITU definition of gain that only an engineer could love, but in an article for "Electronics World," Microflect field engineer Ray Thrower provided a more intuitive explanation.

A passive repeater, he writes, functions essentially identically to a parabolic antenna, or a telescope:

Quite probably the difficulty many people have in understanding how the passive repeater, a flat surface, can have gain relates back to the common misconception about parabolic antennas. It is commonly believed that it is the focusing characteristics of the parabolic antenna that gives it its gain. Therefore, goes the faulty conclusion, how can the passive repeater have gain? The truth is, it isn't focusing that gives a parabola its gain; it is its larger projected aperture. The focusing is a convenient means of transition from a large aperture (the dish) to a small aperture (the feed device). And since it is projected aperture that provides gain, rather than focusing, the passive repeater with its larger aperture will provide high gain that can be calculated and measured reliably. A check of the method of determining antenna gain in any antenna engineering handbook will show that focusing does not enter into the basic gain calculation.

We can also think of it this way: the beam of energy emitted by a microwave antenna expands in an arc as it travels, dissipating the "density" of the energy such that a dish antenna of the same size will receive a weaker and weaker signal as it moves further away (this is the major component of path loss, the "dilution" of the energy over space). A passive repeater employs a reflecting surface which is quite large, larger than practical antennas, and so it "collects" a large cross section of that energy for reemission.

Projected aperture is the effective "window" of energy seen by the antenna at the active terminal as it views the passive repeater. The passive repeater also sees the antenna as a "window" of energy. If the two are far enough away from one another, they will appear to each other as essentially point sources.

In practice, a passive repeater functions a bit like an active repeater that collects a signal with a large antenna and then reemits it with a smaller directional antenna. To be quite honest, I still find it a bit challenging to intuit this effect, but the mathematics bear it out as well. Interestingly, the effect only occurs when the passive repeater is far enough from either terminal so as to be usefully approximated as a point source. Microflect refers to this as the far field condition. When the passive repeater is very close to one of the active sites, within the near field, it is more effective to consider the passive repeater as part of the transmitting antenna itself, and disregard it for path loss calculations. This dichotomy between far field and near field behavior is actually quite common in antenna engineering (where an "antenna" is often multiple radiating and nonradiating elements within the near field of each other), but it's yet another of the things that gives antenna design the feeling of a dark art.

One of the most striking things about passive repeaters is their size. As a passive repeater becomes larger, it reflects a larger cross section of the RF energy and thus provides more gain. Much like with dish or horn antennas, the size of a passive repeater can be traded off with transmitter power (and the size of other antennas involved) to design an economical solution. Microflect offered as standard sizes ranging from 8'x10' (gain at around 6.175GHz: 90.95 dB) to 40'x60' (120.48dB, after a "rough estimate" reduction of 1dB due to interference effects possible from such a short wavelength reflecting off of such a large panel as to invoke multipath effects).

By comparison, a typical active microwave repeater site might provide a gain of around 140dB---and we must bear in mind that dB is a logarithmic unit, so the difference between 121 and 140 is bigger than it sounds. Still, there's a reason that logarithms are used when discussing radio paths... in practice, it is orders of magnitude that make the difference in reliable reception. The reduction in gain from an active repeater to a passive repeater can be made up for with higher-gain terminal antennas and more powerful transmitters. Given that the terminal sites are often at far more convenient locations than the passive repeater, that tradeoff can be well worth it.

Keep in mind that, as Microflect emphasizes, passive repeaters require no power and very little ("virtually no") maintenance. Microflect passive repeaters were manufactured in sections that bolted together in the field, and the support structures provided for fine adjustment of the panel alignment after mounting. These features made it possible to install passive repeaters by helicopter onto simple site-built foundations, and many are found on mountainsides that are difficult to reach even on foot. Even in less difficult locations, these advantages made passive repeaters less expensive to install and operate than active repeaters. Even when the repeater side was readily accessible, passives were often selected simply for cost savings.

Let's consider some examples of passive repeater installations. Microflect was born of the power industry, and electrical generators and utilities remained one of their best customers. Even today, you can find passive repeaters at many hydroelectric dams. There is a practical need to communicate by telephone between a dispatch center (often at the utility's city headquarters) and the operators in the dam's powerhouse, but the powerhouse is at the base of the dam, often in a canyon where microwave signals are completely blocked. A passive repeater set on the canyon rim, at an angle downwards, solves the problem by redirecting the signal from horizontal to vertical. Such an installation can be seen, for example, at the Hoover Dam. In some sense, these passive repeaters "relocate" the radio equipment from the canyon rim (where the desirable signal path is located) to a more convenient location with the other powerhouse equipment. Because of the short distance from the powerhouse to the repeater, these passives were usually small.

This idea can be extended to relocating en-route repeaters to a more serviceable site. In Glacier National Park, Mountain States Telephone and Telegraph installed a telephone system to serve various small towns and National Park Service sites. Glacier is incredibly mountainous, with only narrow valleys and passes. The only points with long sight ranges tend to be very inaccessible. Mt. Furlong provided ideal line of sight to East Glacier and Essex along highway 2, but it would have been extremely challenging to install and maintain a microwave site on the steep peak. Instead, two passive repeaters were installed near the mountaintop, redirecting the signals from those two destinations to an active repeater installed downslope near the highway and railroad.

This example raises another advantage of passive repeaters: their reduced environmental impact, something that Microflect emphasized as the environmental movement of the 1970s made agencies like the Forest Service (which controlled many of the most appealing mountaintop radio sites) less willing to grant permits that would lead to extensive environmental disruption. Construction by helicopter and the lack of a need for power meant that passive repeaters could be installed without extensive clearing of trees for roads and power line rights of way. They eliminated the persistent problem of leakage from standby generator fuel tanks. Despite their large size, passive repeaters could be camouflaged. Many in national forests were painted green to make them less conspicuous. And while they did have a large surface area, Microflect argued that since they could be installed on slopes rather than requiring a large leveled area, passive repeaters would often fall below the ridge or treeline behind them. This made them less visually conspicuous than a traditional active repeater site that would require a tower. Indeed, passive repeaters are only rarely found on towers, with most elevated off the ground only far enough for the bottom edge to be free of undergrowth and snow.

Other passive repeater installations were less a result of exceptionally difficult terrain and more a simple cost optimization. In rural Nevada, Nevada Bell and a dozen independents and coops faced the challenge of connecting small towns with ridges between them. The need for an active repeater at the top of each ridge, even for short routes, made these rural lines excessively expensive. Instead, such towns were linked with dual passive repeaters on the ridge in a "straight through" configuration, allowing microwave antennas at the towns' existing telephone exchange buildings to reach each other. This was the case with the installation I photographed above Pioche. I have been frustratingly unable to confirm the original use of these repeaters, but from context they were likely installed by the Lincoln County Telephone System to link their "hub" microwave site at Mt. Wilson (with direct sight to several towns) to their site near Caliente.

The Microflect manual describes, as an example, a very similar installation connecting Elko to Carlin. Two 20'x32' passive repeaters on a ridge between the two (unfortunately since demolished) provided a direct connection between the two telephone exchanges.

As an example of a typical use, it might be interesting to look at the manual's calculations for this route. From Elko to the repeaters is 13.73 miles, the repeaters are close enough to each other as to be in near field (and so considered as a single antenna system), and from the repeaters to Carlin is 6.71 miles. The first repeater reflects the signal at a 68 degree angle, then the second reflects it back at a 45 degree angle, for a net change in direction of 23 degrees---a mostly straight route. The transmitter produces 33.0 dBm, both antennas provide a 34.5 dB gain, and the passive repeater assembly provides 88 dB gain (this calculated basically by consulting a table in the manual). That means there is 190 dB of gain in the total system. The 6.71 and 13.73 mile paths add up to 244 dB of free space path loss, and Microflect throws in a few more dB of loss to account for connectors and cables and the less than ideal performance of the double passive repeater. The net result is a received signal of -58 dBm, which is plenty acceptable for a 72-channel voice carrier system. This is all done at a significantly lower price than the construction of a full radio site on the ridge [1].

The combination of relocating radio equipment to a more convenient location and simply saving money leads to one of the iconic applications of passive repeaters, the "periscope" or "flyswatter" antenna. Microwave antennas of the 1960s were still quite large and heavy, and most were pressurized. You needed a sturdy tower to support one, and then a way to get up the tower for regular maintenance. This lead to most AT&T microwave sites using short, squat square towers, often with surprisingly convenient staircases to access the antenna decks. In areas where a very tall tower was needed, it might just not be practical to build one strong enough. You could often dodge the problem by putting the site up a hill, but that wasn't always possible, and besides, good hilltop sites that weren't already taken became harder to find.

When Western Union built out their microwave network, they widely adopted the flyswatter antenna as an optimization. Here's how it works: the actual microwave antenna is installed directly on the roof of the equipment building facing up. Only short waveguides are needed, weight isn't an issue, and technicians can conveniently service the antenna without even fall protection. Then, at the top of a tall guyed lattice tower similar to an AM mast, a passive repeater is installed at a 45 degree angle to the ground, redirecting the signal from the rooftop antenna to the horizontal. The passive repeater is much lighter than the antenna, allowing for a thinner tower, and will rarely if ever need service. Western Union often employed two side-by-side lattice towers with a "crossbar" between them at the top for convenient mounting of reflectors each direction, and similar towers were used in some other installations such as the FAA's radar data links. Some of these towers are still in use, although generally with modern lightweight drum antennas replacing the reflectors.

Passive microwave repeaters experienced their peak popularity during the 1960s and 1970s, as the technology became mature and communications infrastructure proliferated. Microflect manufactured thousands of units from their new, larger warehouse, across the street from their old hangar on McNary Field. Microflect's customer list grew to just about every entity in the Bell System, from Long Lines to Western Electric to nearly all of the BOCs. The list includes GTE, dozens of smaller independent telephone companies, most of the nation's major railroads, electrical utilities from the original Montana Power to the Tennessee Valley Authority. Microflect repeaters were used by ITT Arctic Services and RCA Alascom in the far north, and overseas by oil companies and telecoms on islands and in mountainous northern Europe.

In Hawaii, a single passive repeater dodged a mountain to connect Lanai City telephones to the Hawaii Telephone Company network at Tantalus on Oahu---nearly 70 miles in one jump. In Nevada, six passive repeaters joined two active sites to connect six substations to the Sierra Pacific Power Company's control center in Reno. Jamaica's first high-capacity telephone network involved 11 passive repeaters, one as large as 40'x60'.

The Rocky Mountains are still dotted with passive repeaters, structures that are sometimes hard to spot but seem to loom over the forest once noticed. In Seligman, AZ, a sun-faded passive repeater looks over the cemetery. BC Telephone installed passive repeaters to phase out active sites that were inaccessible for maintenance during the winter. Passive repeaters were, it turns out, quite common---and yet they are little known today.

First, it cannot be ignored that passive repeaters are most common in areas where communications infrastructure was built post-1960 through difficult terrain. In North America, this means mostly the West [2], far away from the Eastern cities where we think of telephone history being concentrated. Second, the days of passive repeaters were relatively short. After widespread adoption in the '60s, fiber optics began to cut into microwave networks during the '80s and rendered microwave long-distance links largely obsolete by the late '90s. Considerable improvements in cable-laying equipment, not to mention the lighter and more durable cables, made fiber optics easier to install in difficult terrain than coaxial had ever been.

Besides, during the 1990s, more widespread electrical infrastructure, miniaturization of radio equipment, and practical photovoltaic solar systems all combined to make active repeaters easier to install. Today, active repeater systems installed by helicopter with independent power supplies are not that unusual, supporting cellular service in the Mojave Desert, for example. Most passive repeaters have been obsoleted by changes in communications networks and technologies. Satellite communications offer an even more cost effective option for the most difficult installations, and there really aren't that many places left that a small active microwave site can't be installed.

Moreover, little has been done to preserve the history of passive repeaters. In the wake of the 2015 Wired article on the Long Lines network, considerable enthusiasm has been directed towards former AT&T microwave stations, having been mostly preserved by their haphazard transfer to companies like American Tower. Passive repeaters, lacking even the minimal commercial potential of old AT&T sites, were mostly abandoned in place. Often being found in national forests and other resource management areas, many have been demolished for restoration. In 2019, a historic resources report was written on the Bonneville Power Administration's extensive microwave network. It was prepared to address the responsibility that federal agencies have for historical preservation under the National Historic Preservation Act and National Environmental Policy Act, policies intended to ensure that at least the government takes measures to preserve history before demolishing artifacts. The report reads: "Due to their limited features, passive repeaters are not considered historic resources, and are not evaluated as part of this study."

In 1995, Valmont Industries acquired Microflect. Valmont is known mostly for their agricultural products, including center-pivot irrigation systems, but they had expanded their agricultural windmill business into a general infrastructure division that manufactured radio masts and communication towers. For a time, Valmont continued to manufacture passive repeaters as Valmont Microflect, but business seems to have dried up.

Today, Valmont Structures manufactures modular telecom towers from their facility across the street from McNary Field in Salem, Oregon. A Salem local, descended from early Microflect employees, once shared a set of photos on Facebook: a beat-up hangar with a sign reading "Aircraft Repair Center," and in front of it, stacks of aluminum panel sections. Microflect workers erecting a passive repeater in front of a Douglas A-26. Rows of reflector sections beside a Shell aviation fuel station. George Kreitzberg died in 2004, James in 2017. As of 2025, Valmont no longer manufactures passive repeaters.

Postscript

If you are interested in the history of passive repeaters, there are a few useful tips I can give you.

• Nearly all passive repeaters in North America were built by Microflect, so they have a very consistent design. Locals sometimes confuse passive repeaters with old billboards or even drive-in theater screens, the clearest way to differentiate them is that passive repeaters have a face made up of aluminum modules with deep sidewalls for rigidity and flatness. Take a look at the Microflect manual for many photos.
• Because passive repeaters are passive, they do not require a radio license proper. However, for site-based microwave licenses, the FCC does require that passive repeaters be included in paths (i.e. a license will be for an active site but with a passive repeater as the location at the other end of the path). These "other location" entries often have names ending in "PR" and their type set to "Passive Repeater."
• I don't have any straight answer on whether or not any passive repeaters are still in use. It has likely become very rare but there are probably still examples. Two sources suggest that Rachel, NV still relies on a passive repeater for telephone and DSL, but I'm pretty sure this hasn't been true for some years (I can't find any license covering it). I have so far found one active site-based microwave license covering a passive repeater, but it serves a mine that has been closed since the 1980s and I suspect the license has only been renewed due to a second, different path that does not involve a passive. A reader let me know that Industry Canada has some 80 passive repeaters licensed, but I do not know how many (if any) are in active use.
• For the sake of simplicity I have used "passive repeater" here to refer to microwave reflectors only, but the same term is also used for arrangements of two antennas connected back-to-back. These are much more common in VHF/UHF than in the microwave, although microwave passive repeaters of two parabolic antennas have been used in limited cases.
• Microflect dominated the US and European market for passive repeaters, but the technology was also used in the Soviet Union, seemingly around the same time. I do not know where it was developed first, or whether it was a case of independent invention. The Soviet examples I have seen use a noticeably different support structure from Microflect, and seem to have been engineered for helicopter hoisting in complete form rather than in parts. Passive repeaters proved very useful in the arctic and so I would assume that the Soviet Union installed quite a few.
• Most passive repeaters were installed by "classic communications organizations," meaning telephone companies, power utilities, and railroads---industries that used long-distance communications systems since the turn of the century. I have heard of one passive repeater installed by a television studio for an STL link, and there might be others, but I don't think it was common.

[1] If you find these dB gain/loss calculations confusing, you are not alone. It is deceptively simple in a way that was hard for me to learn, and perhaps I will devote an article to it one day.

[2] Although not exclusively, with installations in places like Vermont and Newfoundland where similar constraints applied.

Ellis knew she needed a walk after she hurried off of Zoom at the end of the meeting to avoid sobbing in front of the group.

She'd just been attending a free online seminar regarding safe job hunting on the Internet. Having been searching since the end of January, Ellis had already picked up plenty of first-hand experience with the modern job market, one rejection at a time. She thought she'd attend the seminar just to see if there were any additional things she wasn't aware of. The seminar had gone well, good information presented in a clear and engaging way. But by the end of it, Ellis was feeling bleak. Goodness gracious, she'd already been slogging through months of this. Hundreds of job applications with nothing to show for it. All of the scams out there, all of the bad actors preying on people desperate for their and their loved ones' survival!

Ellis' childhood had been plagued with anxiety and depression. It was only as an adult that she'd learned any tricks for coping with them. These tricks had helped her avoid spiraling into full-on depression for the past several years. One such trick was to stop and notice whenever those first feelings hit. Recognize them, feel them, and then respond constructively.

First, a walk. Going out where there were trees and sunshine: Ellis considered this "garbage collection" for her brain. So she stepped out the front door and started down a tree-lined path near her house, holding on to that bleak feeling. She was well aware that if she didn't address it, it would take root and grow into hopelessness, self-loathing, fear of the future. It would paralyze her, leave her curled up on the couch doing nothing. And it would all happen without any words issuing from her inner voice. That was the most insidious thing. It happened way down deep in a place where there were no words at all.

Once she returned home, Ellis forced herself to sit down with a notebook and pencil and think very hard about what was bothering her. She wrote down each sentiment:

This was the first step of a reframing technique she'd just read about in the book Right Kind of Wrong by Amy Edmonson. With the words out, it was possible to look at each statement and determine whether it was rational or irrational, constructive or harmful. Each statement could be replaced with something better.

Ellis proceeded step by step through the list.

• Yes, this will end. Everything ends.
• There's nothing wrong with me. Most businesses are swamped with applications. There's a good chance mine aren't even being looked at before they're being auto-rejected. Remember the growth mindset you learned from Carol Dweck. Each application and interview is giving me experience and making me a better candidate.
• This job market is a novel context that changes every day. That means failure is not only inevitable, it's the only way forward.

Ellis realized that her job hunt was very much like a search algorithm trying to find a path through a maze. When the algorithm encountered a dead end, did it deserve blame? Was it an occasion for shame, embarrassment, and despair? Of course not. Simply backtrack and keep going with the knowledge gained.

Yes, there was truth to the fact that this was the toughest job market Ellis had ever experienced. Therefore, taking a note from Viktor Frankl, she spent a moment reimagining the struggle in a way that made it meaningful to her. Ellis began viewing her job hunt in this dangerous market, her gradual accumulation of survival information, as an act of resistance against it. She now hoped to write all about her experience once she was on the other side, in case her advice might help even one other person in her situation save time and frustration.

While unemployed, she also had the opportunity to employ the search algorithm against entirely new mazes. Could Ellis expand her freelance writing into a sustainable gig, for instance? That would mean exploring all the different ways to be a freelance writer, something Ellis was now curious and excited to explore.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Today's Labor Day in the US, a day where we celebrate workers. Well, some of us. This story from the archives is one of the exceptions. Original. --Remy

Sales, as everyone knows, is the mortal enemy of Development.

Their goals are opposite, their people are opposite, their tactics are opposite. Even their credos - developers "Make a good product" but sales will "Do anything to get that money" - are at complete odds.

The company Jordan worked for made a pseudo-enterprise product responsible for everything e-commerce: contacts, inventory, website, shipping, payment...everything. His responsibility included the inventory package, overseeing the development team, designing APIs, integration testing, and coordination with the DBAs and sysadmins...you know, everything. One of his team members implemented a website CMS into the product, letting the website design team ignore the content and focus on making it look good.

Care to guess who was responsible for the site content? If you guessed the VP of Sales, congratulations! You win a noprize.

A couple months passed by without incident. Everything's peachy in fact...that is, until one fateful day when the forty-person stock-and-shipping department are clustered in the parking lot when Jordan shows up.

Jordan parked, crossed the asphalt, and asked one of the less threatening looking warehouse guys, "What's the problem?"

The reply was swift as the entire group unanimously shouted "YOUR F***ING WEBSITE!" Another worker added, "You guys in EYE TEE are so far removed from real life out here. We do REAL WORK, what you guys do from behind your desks?"

Jordan was dumbfounded. What brought this on? For a moment he considered defending his and his team's honor but decided it wouldn't accomplish much besides get his face rearranged and instead replied with a meek "Sure, just let me check into this..." before quickly diving into the nearest entry door.

It didn't take much long after for Jordan to ascertain that the issue wasn't that the website was down, but that the content of one page in particular , the "About Us" page, had upset the hardworking staff who accomplished what the company actually promised: stock and ship the products that they sold on their clients' websites.

After an hour of mediation, it was discovered that the VP of Sales, in a strikingly-insensitive-even-for-him moment, had referred to the warehouse staff as "meatbots." The lively folk who staffed the shipping and stocking departments naturally felt disrespected by being reduced to some stupid sci-fi cloning trope nomenclature. The VP's excuse was simply that he had drunk a couple of beers while he wrote the page text for the website. Oops!

Remarkably, the company (which Jordan left some time later for unrelated reasons) eventually caught up to the backlog of orders to go out. It took a complete warehouse staff replacement, but they did catch up. Naturally, the VP of Sales is still there, with an even more impressive title.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Gordon S. is smarter than the machines. "I can only presume the "Fix Now with AI" button adds some mistakes in order to fix the lack of needed fixes."

"Sorry, repost with the link https://www.daybreaker.com/alive/," wrote Michael R.

And yet again from Michael R., following up with a package mistracker. "Poor DHL driver. I hope he will get a break within those 2 days. And why does the van look like he's driving away from me."

Morgan airs some dirty laundry. "After navigating this washing machine app on holiday and validating my credit card against another app I am greeted by this less than helpful message each time. So is OK okay? Or is the Error in error?
Washing machine worked though."

And finally, scamproof Stuart wondered "Maybe the filter saw the word "scam" and immediately filed it into the scam bucket. All scams include the word "scam" in them, right?"

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Optional types are an attempt to patch the "billion dollar mistake". When you don't know if you have a value or not, you wrap it in an Optional, which ensures that there is a value (the Optional itself), thus avoiding null reference exceptions. Then you can query the Optional to see if there is a real value or not.

This is all fine and good, and can cut down on some bugs. Good implementations are loaded with convenience methods which make it easy to work on the optionals.

But then, you get code like Burgers found. Which just leaves us scratching our heads:

private static final Optional<Boolean> TRUE = Optional.of(Boolean.TRUE);
private static final Optional<Boolean> FALSE = Optional.of(Boolean.FALSE);

Look, any time you're making constants for TRUE or FALSE, something has gone wrong, and yes, I'm including pre-1999 versions of C in this. It's especially telling when you do it in a language that already has such constants, though- at its core- these lines are saying TRUE = TRUE. Yes, we're wrapping the whole thing in an Optional here, which potentially is useful, but if it is useful, something else has gone wrong.

Burgers works for a large insurance company, and writes this about the code:

I was trying to track down a certain piece of code in a Spring web API application when I noticed something curious. It looked like there was a chunk of code implementing an application-specific request filter in business logic, totally ignoring the filter functions offered by the framework itself and while it was not related to the task I was working on, I followed the filter apply call to its declaration. While I cannot supply the entire custom request filter implementation, take these two static declarations as a demonstration of how awful the rest of the class is.

Ah, of course- deep down, someone saw a perfectly functional wheel and said, "I could make one of those myself!" and these lines are representative of the result.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Matt was handed a pile of VB .Net code, and told, "This is yours now. I'm sorry."

As often happens, previous company leadership said, "Why should I pay top dollar for experienced software engineers when I can hire three kids out of college for the same price?" The experiment ended poorly, and the result was a pile of bad VB code, which Matt now owned.

Here's a little taste:

// SET IN SESSION AND REDIRECT TO PRINT PAGE
Session["PrintValue"] = GenerateHTMLOfItem();
Response.Redirect("PrintItem.aspx", true);

The function name here is accurate. GenerateHTMLOfItem takes an item ID, generates the HTML output we want to use to render the item, and stores it in a session variable. It then forces the browser to redirect to a different page, where that HTML can then be output.

You may note, of course, that GenerateHTMLOfItem doesn't actually take parameters. That's because the item ID got stored in the session variable elsewhere.

Of course, it's the redirect that gets all the attention here. This is a client side redirect, so we generate all the HTML, shove it into a session object, and then send a message to the web browser: "Go look over here". The browser sends a fresh HTTP request for the new page, at which point we render it for them.

The Microsoft documentation also has this to add about the use of Response.Redirect(String, Boolean), as well:

Calling Redirect(String) is equivalent to calling Redirect(String, Boolean) with the second parameter set to true. Redirect calls End which throws a ThreadAbortException exception upon completion. This exception has a detrimental effect on Web application performance. Therefore, we recommend that instead of this overload you use the HttpResponse.Redirect(String, Boolean) overload and pass false for the endResponse parameter, and then call the CompleteRequest method. For more information, see the End method.

I love it when I see the developers do a bonus wrong.

Matt had enough fires to put out that fixing this particular disaster wasn't highest on his priority list. For the time being, he could only add this comment:

// SET IN SESSION AND REDIRECT TO PRINT PAGE
// FOR THE LOVE OF GOD, WHY?!?
Session["PrintValue"] = GenerateHTMLOfItem();
Response.Redirect("PrintItem.aspx", true);

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Today's awfulness comes from Tim H, and while it's technically more than one line, it's so representative of the code, and so short that I'm going to call this a representative line. Before we get to the code, we need to talk a little history.

Tim's project is roughly three decades old. It's a C++ tool used for a variety of research projects, and this means that 90% of the people who have worked on it are PhD candidates in computer science programs. We all know the rule of CompSci PhDs and programming: they're terrible at it. It's like the old joke about the farmer who, when unable to find an engineer to build him a cow conveyer, asked a physicist. After months of work, the physicist introduced the result: "First, we assume a perfectly spherical cow in a vacuum…"

Now, this particularly function has been anonymized, but it's easy to understand what the intent was:

bool isFooOrBar() {
  return isFoo() && isBar();
}

The obvious problem here is the mismatch between the function name and the actual function behavior- it promises an or operation, but does an and, which the astute reader may note are different things.

I think this offers another problem, though. Even if the function name were correct, given the brevity of the body, I'd argue that it actually makes the code less clear. Maybe it's just me, but isFoo() && isBar() is more clear in its intent than isFooAndBar(). There's a cognitive overhead to adding more symbols that would make me reluctant to add such a function.

There may be an argument about code-reuse, but it's worth noting- this function is only ever called in one place.

This particular function is not itself, all that new. Tim writes:

This was committed as new code in 2010 (i.e., not a refactor). I'm not sure if the author changed their mind in the middle of writing the function or just forgot which buttons on the keyboard to press.

More likely, Tim, is that they initially wrote it as an "or" operation and then discovered that they were wrong and it needed to be an "and". Despite the fact that the function was only called in one place, they opted to change the body without changing the name, because they didn't want to "track down all the places it's used". Besides, isn't the point of a function to encapsulate the behavior?

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Everyone's got workplace woes. The clueless manager; the disruptive coworker; the cube walls that loom ever higher as the years pass, trapping whatever's left of your soul.

But sometimes, Satan really leaves his mark on a joint. I worked Tech Support there. This is my story. Who am I? Just call me Anonymous.

It starts at the top. A call came in from Lawrence Gibbs, the CEO himself, telling us that a conference room printer was, quote, "leaking." He didn't explain it, he just hung up. The boss ordered me out immediately, told me to step on it. I ignored the elevator, racing up the staircase floor after floor until I reached the dizzying summit of C-Town.

There's less oxygen up there, I'm sure of it. My lungs ached and my head spun as I struggled to catch my breath. The fancy tile and high ceilings made a workaday schmuck like me feel daunted, unwelcome. All the same, I gathered myself and pushed on, if only to learn what on earth "leaking" meant in relation to a printer.

I followed the signs on the wall to the specified conference room. In there, the thermostat had been kicked down into the negatives. The cold cut through every layer of mandated business attire, straight to bone. The scene was thick with milling bystanders who hugged themselves and traded the occasional nervous glance. Gibbs was nowhere to be found.

Remembering my duty, I summoned my nerve. "Tech Support. Where's the printer?" I asked.

Several pointing fingers showed me the way. The large printer/scanner was situated against the far wall, flanking an even more enormous conference table. Upon rounding the table, I was greeted with a grim sight: dozens of sheets of paper strewn about the floor like blood spatter. Everyone was keeping their distance; no one paid me any mind as I knelt to gather the pages. There were 30 in all. Each one was blank on one side, and sported some kind of large, blotchy ring on the other. Lord knew I drank enough java to recognize a coffee mug stain when I saw one, but these weren't actual stains. They were printouts of stains.

The printer was plugged in. No sign of foul play. As I knelt there, unseen and unheeded, I clutched the ruined papers to my chest. Someone had wasted a tree and a good bit of toner, and for what? How'd it go down? Surely Gibbs knew more than he'd let on. The thought of seeking him out, demanding answers, set my heart to pounding. It was no good, I knew. He'd play coy all day and hand me my pink slip if I pushed too hard. As much as I wanted the truth, I had a stack of unpaid bills at home almost as thick as the one in my arms. I had to come up with something else.

There had to be witnesses among the bystanders. I stood up and glanced among them, seeking out any who would return eye contact. There: a woman who looked every bit as polished as everyone else. But for once, I got the feeling that what lay beneath the facade wasn't rotten.

With my eyes, I pleaded for answers.

Not here, her gaze pleaded back.

I was getting somewhere, I just had to arrange for some privacy. I hurried around the table again and weaved through bystanders toward the exit, hoping to beat it out of that icebox unnoticed. When I reached the threshold, I spotted Gibbs charging up the corridor, smoldering with entitlement. "Where the hell is Tech Support?!"

I froze a good distance away from the oncoming executive, whose voice I recognized from a thousand corporate presentations. Instead of putting me to sleep this time, it jolted down my spine like lightning. I had to think fast, or I was gonna lose my lead, if not my life.

"I'm right here, sir!" I said. "Be right back! I, uh, just need to find a folder for these papers."

"I've got one in my office."

A woman's voice issued calmly only a few feet behind me. I spun around, and it was her, all right, her demeanor as cool as our surroundings. She nodded my way. "Follow me."

My spirits soared. At that moment, I would've followed her into hell. Turning around, I had the pleasure of seeing Gibbs stop short with a glare of contempt. Then he waved us out of his sight.

Once we were out in the corridor, she took the lead, guiding me through the halls as I marveled at my luck. Eventually, she used her key card on one of the massive oak doors, and in we went.

You could've fit my entire apartment into that office. The place was spotless. Mini-fridge, espresso machine, even couches: none of it looked used. There were a couple of cardboard boxes piled up near her desk, which sat in front of a massive floor-to-ceiling window admitting ample sunlight.

She motioned toward one of the couches, inviting me to sit. I shook my head in reply. I was dying for a cigarette by that point, but I didn't dare light up within this sanctuary. Not sure what to expect next, I played it cautious, hovering close to the exit. "Thanks for the help back there, ma'am."

"Don't mention it." She walked back to her desk, opened up a drawer, and pulled out a brand-new manila folder. Then she returned to conversational distance and proffered it my way. "You're from Tech Support?"

There was pure curiosity in her voice, no disparagement, which was encouraging. I accepted the folder and stuffed the ruined pages inside. "That's right, ma'am."

She shook her head. "Please call me Leila. I started a few weeks ago. I'm the new head of HR."

Human Resources. That acronym, which usually put me on edge, somehow failed to raise my hackles. I'd have to keep vigilant, of course, but so far she seemed surprisingly OK. "Welcome aboard, Leila. I wish we were meeting in better circumstances." Duty beckoned. I hefted the folder. "Printers don't just leak."

"No." Leila glanced askance, grave.

"Tell me what you saw."

"Well ..." She shrugged helplessly. "Whenever Mr. Gibbs gets excited during a meeting, he tends to lean against the printer and rest his coffee mug on top of it. Today, he must've hit the Scan button with his elbow. I saw the scanner go off. It was so bright ..." She trailed off with a pained glance downward.

"I know this is hard," I told her when the silence stretched too long. "Please, continue."

Leila summoned her mettle. "After he leaned on the controls, those pages spilled out of the printer. And then ... then somehow, I have no idea, I swear! Somehow, all those pages were also emailed to me, Mr. Gibbs' assistant, and the entire board of directors!"

The shock hit me first. My eyes went wide and my jaw fell. But then I reminded myself, I'd seen just as crazy and worse as the result of a cat jumping on a keyboard. A feline doesn't know any better. A top-level executive, on the other hand, should know better.

"Sounds to me like the printer's just fine," I spoke with conviction. "What we have here is a CEO who thinks it's OK to treat an expensive piece of office equipment like his own personal fainting couch."

"It's terrible!" Leila's gaze burned with purpose. "I promise, I'll do everything I possibly can to make sure something like this never happens again!"

I smiled a gallows smile. "Not sure what anyone can do to fix this joint, but the offer's appreciated. Thanks again for your help."

Now that I'd seen this glimpse of better things, I selfishly wanted to linger. But it was high time I got outta there. I didn't wanna make her late for some meeting or waste her time. I backed up toward the door on feet that were reluctant to move.

Leila watched me with a look of concern. "Mr. Gibbs was the one who called Tech Support. I can't close your ticket for you; you'll have to get him to do it. What are you going to do?"

She cared. That made leaving even harder. "I dunno yet. I'll think of something."

I turned around, opened the massive door, and put myself on the other side of it in a hurry, using wall signs to backtrack to the conference room. Would our paths ever cross again? Unlikely. Someone like her was sure to get fired, or quit out of frustration, or get corrupted over time.

It was too painful to think about, so I forced myself to focus on the folder of wasted pages in my arms instead. It felt like a mile-long rap sheet. I was dealing with an alleged leader who went so far as to blame the material world around him rather than accept personal responsibility. I'd have to appeal to one or more of the things he actually cared about: himself, his bottom line, his sense of power.

By the time I returned to the conference room to face the CEO, I knew what to tell him. "You're right, sir, there's something very wrong with this printer. We're gonna take it out here and give it a thorough work-up."

That was how I was able to get the printer out of that conference room for good. Once it underwent "inspection" and "testing," it received a new home in a previously unused closet. Whenever Gibbs got to jawing in future meetings, all he could do was lean against the wall. Ticket closed.

Gibbs remained at the top, doing accursed things that trickled down to the roots of his accursed company. But at least from then on, every onboarding slideshow included a photo of one of the coffee ring printouts, with the title Respect the Equipment.

Thanks, Leila. I can live with that.

"What word can spell with the letters housucops?" asks Mark R. "Sometimes AI hallucinations can be hard to find. Other times, they just kind of stand out..."

"Do I need more disks?" wonders Gordon "I'm replacing a machine which has only 2 GB of HDD. New one has 2 TB, but that may not be enough. Unless Thunar is lying." It's being replaced by an LLM.

"Greenmobility UX is a nightmare" complains an anonymous reader. "Just like last week's submission, do you want to cancel? Cancel or Leave?" This is not quite as bad as last week's.

Cinephile jeffphi rated this film two thumbs down. "This was a very boring preview, cannot recommend."

Malingering Manuel H. muses "Who doesn't like long weekends? Sometimes, one Sunday per week is just not enough, so just put a second one right after the first." I don't want to wait until Oktober for a second Sunday; hope we get one søøn.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Once upon a time, when the Web was young, if you wanted to be a cool kid, you absolutely needed two things on your website: a guestbook for people to sign, and a hit counter showing how many people had visited your Geocities page hosting your Star Trek fan fiction.

These days, we don't see them as often, but companies still like to track the information, especially when it comes to counting downloads. So when Justin started on a new team and saw a download count in their analytics, he didn't think much of it at all. Nor did he think much about it when he saw the download count displayed on the download page.

Another thing that Justin didn't think much about was big piles of commits getting merged in overnight, at least not at first. But each morning, Justin needed to pull in a long litany of changes from a user named "MrStinky". For the first few weeks, Justin was too preoccupied with getting his feet under him, so he didn't think about it too much.

But eventually, he couldn't ignore what he saw in the git logs.

docs: update download count to 51741
docs: update download count to 51740
docs: update download count to 51738

And each commit was exactly what the name implied, a diff like:

- 51740
+ 51741

Each time a user clicked the download link, a ping was sent to their analytics system. Throughout the day, the bot "MrStinky" would query the analytics tool, and create new commits that updated the counter. Overnight, it would bundle those commits into a merge request, approve the request, merge the changes, and then redeploy what was at the tip of main.

"But, WHY?" Justin asked his peers.

One of them just shrugged. "It seemed like the easiest and fastest way at the time?"

"I wanted to wire Mr Stinky up to our content management system's database, but just never got around to it. And this works fine," said another.

Much like the rest of the team, Justin found that there were bigger issues to tackle.

Jessica recently started at a company still using Windows Forms.

Well, that was a short article. Oh, you want more WTF than that? Sure, we can do that.

As you might imagine, a company that's still using Windows Forms isn't going to upgrade any time soon; they've been using an API that's been in maintenance mode for a decade, clearly they're happy with it.

But they're not too happy- Jessica was asked to track down a badly performing report. This of course meant wading through a thicket of spaghetti code, pointless singletons, and the general sloppiness that is the code base. Some of the code was written using Entity Framework for database access, much of it is not.

While it wasn't the report that Jessica was sent to debug, this method caught her eye:

private Dictionary<long, decimal> GetReportDiscounts(ReportCriteria criteria)
{
    Dictionary<long, decimal> rows = new Dictionary<long, decimal>();

    string query = @"select  ii.IID,
        SUM(CASE WHEN ii.AdjustedTotal IS NULL THEN 
        (ii.UnitPrice * ii.Units)  ELSE
            ii.AdjustedTotal END) as 'Costs'
            from ii
                where ItemType = 3
            group by ii.IID
            ";

    string connectionString = string.Empty;
    using (DataContext db = DataContextFactory.GetInstance<DataContext>())
    {
        connectionString = db.Database.Connection.ConnectionString;
    }

    using (SqlConnection connection = new SqlConnection(connectionString))
    {
        using (SqlCommand command = new SqlCommand(query, connection))
        {
            command.Parameters.AddWithValue("@DateStart", criteria.Period.Value.Min.Value.Date);
            command.Parameters.AddWithValue("@DateEnd", criteria.Period.Value.Max.Value.Date.AddDays(1));
            command.Connection.Open();

            using (SqlDataReader reader = command.ExecuteReader())
            {
                while (reader.Read())
                {
                    decimal discount = (decimal)reader["Costs"];
                    long IID = (long)reader["IID"];

                    if (rows.ContainsKey(IID))
                    {
                        rows[IID] += discount;
                    }
                    else
                    {
                        rows.Add(IID, discount);
                    }
                }
            }
        }
    }

    return rows;
}

This code constructs a query, opens a connection, runs the query, and iterates across the results, building a dictionary as its result set. The first thing which leaps out is that, in code, they're doing a summary (iterating across the results and grouping by IID), which is also what they did in the query.

It's also notable that the table they're querying is called ii, which is not a result of anonymization, and actually what they called it. Then there's the fact that they set parameters on the query, for DateStart and DateEnd, but the query doesn't use those. And then there's that magic number 3 in the query, which is its own set of questions.

Then, right beneath that method was one called GetReportTotals. I won't share it, because it's identical to what's above, with one difference:

            string query = @"
select   ii.IID,
                SUM(CASE WHEN ii.AdjustedTotal IS NULL THEN 
                (ii.UnitPrice * ii.Units)  ELSE
                 ii.AdjustedTotal END)  as 'Costs' from ii
				  where  itemtype = 0 
				 group by iid
";

The magic number is now zero.

So, clearly we're in the world of copy/paste programming, but this raises the question: which came first, the 0 or the 3? The answer is neither. GetCancelledInvoices came first.

private List<ReportDataRow> GetCancelledInvoices(ReportCriteria criteria, Dictionary<long, string> dictOfInfo)
{
    List<ReportDataRow> rows = new List<ReportDataRow>();

    string fCriteriaName = "All";

    string query = @"select 
        A long query that could easily be done in EF, or at worst a stored procedure or view. Does actually use the associated parameters";


    string connectionString = string.Empty;
    using (DataContext db = DataContextFactory.GetInstance<DataContext>())
    {
        connectionString = db.Database.Connection.ConnectionString;
    }

    using (SqlConnection connection = new SqlConnection(connectionString))
    {
        using (SqlCommand command = new SqlCommand(query, connection))
        {
            command.Parameters.AddWithValue("@DateStart", criteria.Period.Value.Min.Value.Date);
            command.Parameters.AddWithValue("@DateEnd", criteria.Period.Value.Max.Value.Date.AddDays(1));
            command.Connection.Open();

            using (SqlDataReader reader = command.ExecuteReader())
            {
                while (reader.Read())
                {
                    long ID = (long)reader["ID"];
                    decimal costs = (decimal)reader["Costs"];
                    string mNumber = (string)reader["MNumber"];
                    string mName = (string)reader["MName"];
                    DateTime idate = (DateTime)reader["IDate"];
                    DateTime lastUpdatedOn = (DateTime)reader["LastUpdatedOn"];
                    string iNumber = reader["INumber"] is DBNull ? string.Empty : (string)reader["INumber"];
                    long fId = (long)reader["FID"];
                    string empName = (string)reader["EmpName"];
                    string empNumber = reader["EmpNumber"] is DBNull ? string.Empty : (string)reader["empNumber"];
                    long mId = (long)reader["MID"];

                    string cName = dictOfInfo[matterId];

                    if (criteria.EmployeeID.HasValue && fId != criteria.EmployeeID.Value)
                    {
                        continue;
                    }

                    rows.Add(new ReportDataRow()
                    {
                        CName = cName,
                        IID = ID,
                        Costs = costs * -1, //Cancelled i - minus PC
                        TimedValue = 0,
                        MNumber = mNumber,
                        MName = mName,
                        BillDate = lastUpdatedOn,
                        BillNumber = iNumber + "A",
                        FID = fId,
                        EmployeeName = empName,
                        EmployeeNumber = empNumber
                    });
                }
            }
        }
    }


    return rows;
}

This is the original version of the method. We can infer this because it actually uses the parameters of DateStart and DateEnd. Everything else just copy/pasted this method and stripped out bits until it worked. There are more children of this method, each an ugly baby of its own, but all alike in their ugliness.

It's also worth noting, the original version is doing filtering after getting data from the database, instead of putting that criteria in the WHERE clause.

As for Jessica's poor performing report, it wasn't one of these methods. It was, however, another variation on "run a query, then filter, sort, and summarize in C#". By simply rewriting it as a SQL query in a stored procedure that leveraged indexes, performance improved significantly.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

In theory, HTTP status codes should be easy to work with. In the 100s? You're doing some weird stuff and breaking up large requests into multiple sub-requests. 200s? It's all good. 300s? Look over there. 400s? What the hell are you trying to do? 500s? What the hell is the server trying to do?

This doesn't mean people don't endlessly find ways to make it hard. LinkedIn, for example, apparently likes to send 999s if you try and view a page without being logged in. Shopify has invented a few. Apache has added a 218 "This is Fine". And then there's WebDAV, which not only adds new status codes, but adds a whole bunch of new verbs to HTTP requests.

Francesco D sends us a "clever" attempt at handling status codes.

    try {
      HttpRequest.Builder localVarRequestBuilder = {{operationId}}RequestBuilder({{#allParams}}{{paramName}}{{^-last}}, {{/-last}}{{/allParams}}{{#hasParams}}, {{/hasParams}}headers);
      return memberVarHttpClient.sendAsync(
          localVarRequestBuilder.build(),
          HttpResponse.BodyHandlers.ofString()).thenComposeAsync(localVarResponse -> {
            if (localVarResponse.statusCode()/ 100 != 2) {
              return CompletableFuture.failedFuture(getApiException("{{operationId}}", localVarResponse));
            }
            {{#returnType}}
            try {
              String responseBody = localVarResponse.body();
              return CompletableFuture.completedFuture(
                  responseBody == null || responseBody.isBlank() ? null : memberVarObjectMapper.readValue(responseBody, new TypeReference<{{{returnType}}}>() {})
              );
            } catch (IOException e) {
              return CompletableFuture.failedFuture(new ApiException(e));
            }
            {{/returnType}}
            {{^returnType}}
            return CompletableFuture.completedFuture(null);
            {{/returnType}}
      });
    }

Okay, before we get to the status code nonsense, I first have to whine about this templating language. I'm generally of the mind that generated code is a sign of bad abstractions, especially if we're talking about using a text templating engine, like this. I'm fine with hygienic macros, and even C++'s templating system for code generation, because they exist within the language. But fine, that's just my "ok boomer" opinion, so let's get into the real meat of it, which is this line:

localVarResponse.statusCode()/ 100 != 2

"Hey," some developer said, "since success is in the 200 range, I'll just divide by 100, and check if it's a 2, helpfully truncating the details." Which is fine and good, except neither 100s nor 300s represent a true error, especially because if the local client is doing caching, a 304 tells us that we can used the cached version.

For Francesco, treating 300s as an error created a slew of failed requests which shouldn't have failed. It wasn't too difficult to detect- they were at least logging the entire response- but it was frustrating, if only because it seems like someone was more interested in being clever with math than actually writing good software.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

For months, everything at Yusuf's company was fine. Then, suddenly, he comes in to the office to learn that overnight the log exploded with thousands of panic messages. No software changes had been pushed, no major configurations had happened- just a reboot. What had gone wrong?

This particular function was invoked as part of the application startup:

func (a *App) setupDocDBClient(ctx context.Context) error {
	docdbClient, err := docdb.NewClient(
		ctx,
		a.config.MongoConfig.URI,
		a.config.MongoConfig.Database,
		a.config.MongoConfig.EnableTLS,
	)
	if err != nil {
		return nil
	}

	a.DocDBClient = docdbClient
	return nil
}

This is Go, which passes errors as part of the return. You can see an example where docdb.NewClient returns a client and an err object. At one point in the history of this function, it did the same thing- if connecting to the database failed, it returned an error.

But a few months earlier, an engineer changed it to swallow the error- if an error occurred, it would return nil.

As an organization, they did code reviews. Multiple people looked at this and signed off- or, more likely, multiple people clicked a button to say they'd looked at it, but hadn't.

Most of the time, there weren't any connection issues. But sometimes there were. One reboot had a flaky moment with connecting, and the error was ignored. Later on in execution, downstream modules started failing, which eventually led to a log full of panic level messages.

The change was part of a commit tagged merely: "Refactoring". Something got factored, good and hard, all right.

low-case jeffphi found "Yep, all kinds of technical errors."

Michael R. reports an off by 900 error.

"It is often said that news slows down in August," notes Stewart , wondering if "perhaps The Times have just given up? Or perhaps one of the biggest media companies just doesn't care about their paying subscribers?"

"Zero is a dangerous idea!" exclaims Ernie in Berkeley .

Daniel D. found one of my unfavorites, calling it "Another classic case of cancel dialog. This time featuring KDE Partition Manager."

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Andreas found this in a rather large, rather ugly production code base.

private static void LogView(object o)
{
    try
    {
        ArrayList al = (ArrayList)o;
        int pageId = (int)al[0];
        int userId = (int)al[1];

        // ... snipped: Executing a stored procedure that stores the values in the database
    }
    catch (Exception) { }
}

This function accepts an object of any type, except no, it doesn't, it expect that object to be an ArrayList. It then assumes the array list will then store values in a specific order. Note that they're not using a generic ArrayList here, nor could they- it (potentially) needs to hold a mix of types.

What they've done here is replace a parameter list with an ArrayList, giving up compile time type checking for surprising runtime exceptions. And why?

"Well," the culprit explained when Andreas asked about this, "the underlying database may change. And then the function would need to take different parameters. But that could break existing code, so this allows us to add parameters without ever having to change existing code."

"Have you heard of optional arguments?" Andreas asked.

"No, all of our arguments are required. We'll just default the ones that the caller doesn't supply."

And yes, this particular pattern shows up all through the code base. It's "more flexible this way."

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Ronan works with a vibe coder- an LLM addicted developer. This is a type of developer that's showing up with increasing frequency. Their common features include: not reading the code the AI generated, not testing the code the AI generated, not understanding the context of the code or how it integrates into the broader program, and absolutely not bothering to follow the company coding standards.

Here's an example of the kind of Python code they were "writing":

if isinstance(o, Test):
    if o.requirement is None:
        logger.error(f"Invalid 'requirement' in Test: {o.key}")
        try:
            raise ValueError("Missing requirement in Test object.")
        except ValueError:
            pass

    if o.title is None:
        logger.error(f"Invalid 'title' in Test: {o.key}")
        try:
            raise ValueError("Missing title in Test object.")
        except ValueError:
            pass

An isinstance check is already a red flag. Even without proper type annotations and type checking (though you should use them) any sort of sane coding is going to avoid situations where your method isn't sure what input it's getting. isinstance isn't a WTF, but it's a hint at something lurking off screen. (Yes, sometimes you do need it, this may be one of those times, but I doubt it.)

In this case, if the Test object is missing certain fields, we want to log errors about it. That part, honestly, is all fine. There are potentially better ways to express this idea, but the idea is fine.

No, the obvious turd in the punchbowl here is the exception handling. This is pure LLM, in that it's a statistically probable result of telling the LLM "raise an error if the requirement field is missing". The resulting code, however, raises an exception, immediately catches it, and then does nothing with it.

I'd almost think it's a pre-canned snippet that's meant to be filled in, but no- there's no reason a snippet would throw and catch the same error.

Now, in Ronan's case, this has a happy ending: after a few weeks of some pretty miserable collaboration, the new developer got fired. None of "their" code ever got merged in. But they've already got a few thousand AI generated resumes out to new positions…

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Rajpreet Sahota

U.S. Customs and Border Protection (CBP) has released new data showing a sharp rise in electronic device searches at border crossings.

From April to June alone, CBP conducted 14,899 electronic device searches, up more than 21 per cent from the previous quarter (23 per cent over the same period last year). Most of those were basic searches, but 1,075 were “advanced,” allowing officers to copy and analyze device contents.

U.S. border agents have conducted tens of thousands of searches every year for many years, along a generally increasing trajectory, so this is not necessarily specific to this administration. Unfortunately, as the Electronic Frontier Foundation reminds us, people have few rights at ports of entry, regardless of whether they are a U.S. citizen.

There are no great ways to avoid a civil rights violation, either. As a security expert told the CBC, people with burner devices would be subject to scrutiny because it is obviously not their main device. It stands to reason that someone travelling without any electronic devices at all would also be seen as more suspicious. Encryption is your best bet, but then you may need to have a whole conversation about why all of your devices are encrypted.

The EFF has a pocket guide with your best options.

⌥ Permalink

If you, thankfully, missed Google’s Pixel 10 unveiling — and even if you did not — you will surely appreciate PetaPixel’s review of the Pro version of the phone from the perspective of photographers and videographers. This line of phones has long boasted computational photography bonafides over the competition, and I thought this was a good exploration of what is new and not-so-new in this year’s models.

Come for Chris and Jordan; stay for Chris’ “pet” deer.

⌥ Permalink

Typepad:

After September 30, 2025, access to Typepad – including account management, blogs, and all associated content – will no longer be available. Your account and all related services will be permanently deactivated.   

I have not thought about Typepad in years, and I am certain I am not alone. That is not a condemnation; Typepad occupies a particular time and place on the web. As with anything hosted, however, users are unfortunately dependent on someone else’s interest in maintaining it.

If you have anything hosted at Typepad, now is a good time to back it up.

⌥ Permalink

Kelefa Sanneh, the New Yorker:

[…] In 2018, the social-science blog “Data Colada” looked at Metacritic, a review aggregator, and found that more than four out of five albums released that year had received an average rating of at least seventy points out of a hundred — on the site, albums that score sixty-one or above are colored green, for “good.” Even today, music reviews on Metacritic are almost always green, unlike reviews of films, which are more likely to be yellow, for “mixed/average,” or red, for “bad.” The music site Pitchfork, which was once known for its scabrous reviews, hasn’t handed down a perfectly contemptuous score — 0.0 out of 10 — since 2007 (for “This Is Next,” an inoffensive indie-rock compilation). And, in 2022, decades too late for poor Andrew Ridgeley, Rolling Stone abolished its famous five-star system and installed a milder replacement: a pair of merit badges, “Instant Classic” and “Hear This.”

I have quibbles with this article, which I will get to, but I will front-load this with the twist instead of making you wait — this article is, in effect, Sanneh’s response to himself twenty-one years after popularizing the very concept of poptimism in the New York Times. Sanneh in 2004:

In the end, the problem with rockism isn’t that it’s wrong: all critics are wrong sometimes, and some critics (now doesn’t seem like the right time to name names) are wrong almost all the time. The problem with rockism is that it seems increasingly far removed from the way most people actually listen to music.

Are you really pondering the phony distinction between “great art” and a “guilty pleasure” when you’re humming along to the radio? In an era when listeners routinely — and fearlessly — pick music by putting a 40-gig iPod on shuffle, surely we have more interesting things to worry about than that someone might be lip-synching on “Saturday Night Live” or that some rappers gild their phooey. Good critics are good listeners, and the problem with rockism is that it gets in the way of listening. If you’re waiting for some song that conjures up soul or honesty or grit or rebellion, you might miss out on Ciara’s ecstatic electro-pop, or Alan Jackson’s sly country ballads, or Lloyd Banks’s felonious purr.

Here we are in 2025 and a bunch of the best-reviewed records in recent memory are also some of the most popular. They are well-regarded because critics began to review pop records on the genre’s own terms.

Here is one more bonus twist: the New Yorker article is also preoccupied with criticism of Pitchfork, a fellow Condé Nast publication. This is gestured toward twice in the article. Neither one serves to deflate the discomfort, especially since the second mention is in the context of reduced investment in the site by Condé.

Speaking of Pitchfork, though, the numerical scores of its reviews have led to considerable analysis by the statistics obsessed. For example, a 2020 analysis of reviews published between 1999 and early 2017 found the median score was 7.03. This is not bad at all, and it suggests the site is most interested in what it considers decent-to-good music, and cannot be bothered to review bad stuff. The researchers also found a decreasing frequency of very negative reviews beginning in about 2010, which fits Sanneh’s thesis. However, it also found fewer extremely high scores. The difference is more subtle — and you should ignore the dot in the “10.0” column because the source data set appears to also contain Pitchfork’s modern reviews of classic records — but notice how many dots are rated above 8.75 from 2004–2009 compared to later years. A similar analysis of reviews from 1999–2021 found a similar convergence toward mediocre.

As for Metacritic, I had to go and look up the Data Colada article referenced, since the New Yorker does not bother with links. I do not think this piece reinforces Sanneh’s argument very well. What Joe Simmons, its author, attempts to illustrate is that Metacritic skews positive for bands with few aggregated reviews because most music publications are not going to waste time dunking on a nascent band’s early work. I also think Simmons is particularly cruel to a Modern Studies record.

Anecdotally, I do not know that music critics have truly lost their edge. I read and watch a fair amount of music criticism, and I still see a generous number of withering takes. I think music critics, as they become established and busier, recognize they have little time for bad music. Maroon 5 have been a best-selling act for a couple of decades, but Metacritic has aggregated just four reviews of its latest album, because you can just assume it sucks. Your time might be better spent with the great new Water From Your Eyes record.

Even though I am unsure I agree with Sanneh’s conclusion, I think critics should make time and column space for albums they think are bad. Negative reviews are not cruel — or, at least, they should not be — but it is the presence of bad that helps us understand what is good.

⌥ Permalink

Tripp Mickle and Don Clark, New York Times:

Echoing IBM, Microsoft in 1985 built its Windows software to run on Intel processors. The combination created the “Wintel era,” when the majority of the world’s computers featured Windows software and Intel hardware. Microsoft’s and Intel’s profits soared, turning them into two of the world’s most valuable companies by the mid-1990s. Most of the world’s computers soon featured “Intel Inside” stickers, making the chipmaker a household name.

In 2009, the Obama administration was so troubled by Intel’s dominance in computer chips that it filed a broad antitrust case against the Silicon Valley giant. It was settled the next year with concessions that hardly dented the company’s profits.

This is a gift link because I think this one is particularly worth reading. The headline calls it a “long, painful downfall”, but the remarkable thing about it is that it is short, if anything. Revenue is not always the best proxy for this, but the cracks began to show in the early 2010s when its quarterly growth contracted; a few years of modest growth followed before being clobbered since mid-2020. Every similar company in tech seems to have made a fortune off the combined forces of the covid-19 pandemic and artificial intelligence except Intel.

Tobias Mann, the Register:

For better or worse, the US is now a shareholder in the chipmaker’s success, which makes sense given Intel’s strategic importance to national security. Remember, Intel is the only American manufacturer of leading edge silicon. TSMC and Samsung may be setting up shop in the US, but hell will freeze over before the US military lets either of them fab its most sensitive chips. Uncle Sam awarded Intel $3.2 billion to build that secure enclave for a reason.

Put mildly, The US government needs Intel Foundry and Lip Bu Tan needs Uncle Sam’s cash to make the whole thing work. It just so happens that right now Intel isn’t in a great position to negotiate.

Mann’s skeptical analysis is also worth your time. There is good sense in the U.S. government holding an interest in the success of Intel. Under this president, however, it raises entirely unique questions and concerns.

⌥ Permalink

Mary Cunningham, CBS News:

Tesla was found partly liable in a wrongful death case involving the electric vehicle company’s Autopilot system, with a jury awarding the plaintiffs $200 million in punitive damages plus additional money in compensatory damages.

[…]

“What we ultimately learned from that augmented video is that the vehicle 100% knew that it was about to run off the roadway, through a stop sign, through a blinking red light, through a parked car and through a pedestrian, yet did nothing other than shut itself off when the crash was unavoidable,” said Adam Boumel, one of the plaintiffs’ attorneys.

I continue to believe holding manufacturers legally responsible is the correct outcome for failures of autonomous driving technology. Corporations, unlike people, cannot go to jail; the closest thing we have to accountability is punitive damages.

⌥ Permalink

Andy Baio:

This minute-long clip of a Will Smith concert is blowing up online for all the wrong reasons, with people accusing him of using AI to generate fake crowds filled with fake fans carrying fake signs. The story’s blown up a bit, with coverage in Rolling Stone, NME, The Independent, and Consequence of Sound.

[…]

But here’s where things get complicated.

The crowds are real. Every person you see in the video above started out as real footage of real fans, pulled from video of multiple Will Smith concerts during his recent European tour.

The lines, in this case, are definitely blurry. This is unlike any previous is it A.I.? controversy over crowds I can remember because — and I hope this is more teaser than spoiler — note Baio’s careful word choice in that last quoted paragraph.

⌥ Permalink

Joseph Cox, 404 Media:

A man holds an orange and white device in his hand, about the size of his palm, with an antenna sticking out. He enters some commands with the built-in buttons, then walks over to a nearby car. At first, its doors are locked, and the man tugs on one of them unsuccessfully. He then pushes a button on the gadget in his hand, and the door now unlocks.

The tech used here is the popular Flipper Zero, an ethical hacker’s swiss army knife, capable of all sorts of things such as WiFi attacks or emulating NFC tags. Now, 404 Media has found an underground trade where much shadier hackers sell extra software and patches for the Flipper Zero to unlock all manner of cars, including models popular in the U.S. The hackers say the tool can be used against Ford, Audi, Volkswagen, Subaru, Hyundai, Kia, and several other brands, including sometimes dozens of specific vehicle models, with no easy fix from car manufacturers.

The Canadian government made headlines last year when it banned the Flipper Zero, only to roll it back in favour of a narrowed approach a month later. That was probably the right call. However, too many — including Hackaday and Flipper itself — were too confident in saying the device was not able to, or could not, be used to steal cars. This is demonstrably untrue.

⌥ Permalink

The United States government has long had an interest in boosting its high technology sector, with manifold objectives: for soft power, espionage, and financial dominance, at least. It has accomplished this through tax incentives, funding some of the best universities in the world, lax antitrust and privacy enforcement, and — in some cases — direct involvement. The internet began as a Department of Defense project, and the government invests in businesses through firms like In-Q-Tel.

All of this has worked splendidly for them. The world’s technology stack is overwhelmingly U.S.-dependent across the board, from consumers through large businesses and up to governments, even those which are not allies. Apparently, though, it is not enough and the country’s leaders are desperately worried about regulation in Europe and competition from Eastern Asia.

The U.S. Federal Trade Commission:

Federal Trade Commission Chairman Andrew N. Ferguson sent letters today to more than a dozen prominent technology companies reminding them of their obligations to protect the privacy and data security of American consumers despite pressure from foreign governments to weaken such protections. He also warned them that censoring Americans at the behest of foreign powers might violate the law.

[…]

“I am concerned that these actions by foreign powers to impose censorship and weaken end-to-end encryption will erode Americans’ freedoms and subject them to myriad harms, such as surveillance by foreign governments and an increased risk of identity theft and fraud,” Chairman [Andrew] Ferguson wrote.

These letters (PDF) serve as a reminder to, in effect, enforce U.S. digital supremacy around the world. Many of the most popular social networks are U.S.-based and export the country’s interpretation of permissive expression laws around the world, even to countries with different expectations. Occasionally, there will be conflicting policies which may mean country-specific moderation. What Ferguson’s letter appears to be asking is for U.S. companies to be sovereign places for U.S. citizens regardless of where their speech may appear.

The U.S. government is certainly correct to protect the interests of its citizens. But let us not pretend this is not also re-emphasizing the importance to the U.S. government of exporting its speech policy internationally, especially when it fails to adhere to it on its home territory. It is not just the hypocrisy that rankles, it is also the audacity requiring posts by U.S. users to be treated as a special class, to the extent that E.U. officials enforcing their own laws in their own territory could be subjected to sanctions.

As far as encryption, I have yet to see sufficient evidence of a radical departure from previous statements made by this president. When he was running the first time around, he called for an Apple boycott over the company’s refusal to build a special version of iOS to decrypt an iPhone used by a mass shooter. During his first term, Trump demanded Apple decrypt another iPhone in a different mass shooting. After two attempted assassinations last year, Trump once again said Apple should forcibly decrypt the iPhones of those allegedly responsible. It was under his first administration in which Apple was dissuaded from launching Advanced Data Protection in the first place. U.S. companies with European divisions recently confirmed they cannot comply with E.U. privacy and security guarantees as they are subject to the provisions of the CLOUD Act enacted during the first Trump administration.

The closest Trump has gotten to changing his stance is in a February interview with the Spectator’s Ben Domenech:

BD: But the problem is he [the British Prime Minister] runs, your vice president obviously eloquently pointed this out in Munich, he runs a nation now that is removing the security helmets on Apple phones so that they can—

DJT: We told them you can’t do this.

BD: Yeah, Tulsi, I saw—

DJT: We actually told him… that’s incredible. That’s something, you know, that you hear about with China.

The red line, it seems, is not at a principled opposition to “removing the security helmet” of encryption, but in the U.K.’s specific legislation. It is a distinction with little difference. The president and U.S. law enforcement want on-demand decryption just as much as their U.K. counterparts and have attempted to legislate similar requirements.

While the U.S. has been reinforcing the supremacy of its tech companies in Europe, it has also been propping them up at home:

Intel Corporation today announced an agreement with the Trump Administration to support the continued expansion of American technology and manufacturing leadership. Under terms of the agreement, the United States government will make an $8.9 billion investment in Intel common stock, reflecting the confidence the Administration has in Intel to advance key national priorities and the critically important role the company plays in expanding the domestic semiconductor industry.

The government’s equity stake will be funded by the remaining $5.7 billion in grants previously awarded, but not yet paid, to Intel under the U.S. CHIPS and Science Act and $3.2 billion awarded to the company as part of the Secure Enclave program. Intel will continue to deliver on its Secure Enclave obligations and reaffirmed its commitment to delivering trusted and secure semiconductors to the U.S. Department of Defense. The $8.9 billion investment is in addition to the $2.2 billion in CHIPS grants Intel has received to date, making for a total investment of $11.1 billion.

Despite its size — 10% of the company, making it the single largest shareholder — this press release says this investment is “a passive ownership, with no Board representation or other governance or information rights”. Even so, this is the U.S. attempting to reassert the once-vaunted position of Intel.

This deal is not as absurd as it seems. It is entirely antithetical to the claimed free market capitalist principles common to both major U.S. political parties but, in particular, espoused by Republicans. It is probably going to be wielded in terrible ways. But I can see at least one defensible reason for the U.S. to treat the integrity of Intel as an urgent issue: geology.

Near the end of Patrick McGee’s “Apple in China” sits a section that will haunt the corners of my brain for a long time. McGee writes that a huge amount of microprocessors — “at least 80 percent of the world’s most advanced chips” — are made by TSMC in Taiwan. There are political concerns with the way China has threatened Taiwan, which can be contained and controlled by humans, and frequent earthquakes, which cannot. Even setting aside questions about control, competition, and China, it makes a lot of sense for there to be more manufacturers of high-performance chips in places with less earthquake potential. (Silicon Valley is also sitting in a geologically risky place. Why do we do this to ourselves?)

At least Intel gets the shine of a Trump co-sign, and when has that ever gone wrong?

Then there are the deals struck with Nvidia and AMD, whereby the U.S. government gets a kickback in exchange for trade. Lauren Hirsch and Maureen Farrell, New York Times:

But some of Mr. Trump’s recent moves appear to be a strong break with historical precedent. In the cases of Nvidia and AMD, the Trump administration has proposed dictating the global market that these chipmakers can have access to. The two companies have promised to give 15 percent of their revenue from China to the U.S. government in order to have the right to sell chips in that country and bypass any future U.S. restrictions.

These moves add up and are, apparently, just the beginning. The U.S. has been a dominant force in high technology in part because of a flywheel effect created by early investments, some of which came from government sources and public institutions. This additional context does not undermine the entrepreneurship that came after, and which has been a proud industry trait. In fact, it demonstrates a benefit of strong institutions.

The rest of the world should see these massive investments as an instruction to build up our own high technology industries. We should not be too proud in Canada to set up Crown corporations that can take this on, and we ought to work with governments elsewhere. We should also not lose sight of the increasing hostility of the U.S. government making these moves to reassert its dominance in the space. We can stop getting steamrolled if we want to, but we really need to want to. We can start small.

Michelle Bellefontaine, CBC News:

“Any publicly funded immunization in B.C. can be provided at no cost to any Canadian travelling within the province,” a statement from the ministry said.

“This includes providing publicly funded COVID-19 vaccine to people of Alberta.”

[…]

Alberta is the only Canadian province that will not provide free universal access to COVID-19 vaccines this fall.

The dummies running our province opened what they called a “vaccine booking system” earlier this month allowing Albertans to “pre-order” vaccines. However, despite these terms having defined meanings, the system did not allow anyone to book a specific day, time, or location to receive the vaccine, nor did it take payments or even show prices. The government’s rationale for this strategy is that it is “intended [to] help reduce waste”.

Now that pricing has been revealed, it sure seems like these dopes want us to have a nice weekend just over the B.C. border. A hotel room for a couple or a family will probably be about the same as the combined vaccination cost. Sure, a couple of meals would cost extra, but it is also a nice weekend away. Sure, it means people who are poor or otherwise unable will likely need to pay the $100 “administrative fee” to get their booster, and it means a whole bunch of pre-ordered vaccines will go to waste thereby undermining the whole point of this exercise. But at least it plays to the anti-vaccine crowd. That is what counts for these jokers.

⌥ Permalink

Jane Mundy, writing at the imaginatively named Lawyers and Settlements in December:

A former Apple executive has filed a California labor complaint against Apple and Jay Blahnik, the company’s vice president of fitness technologies. Mandana Mofidi accuses Apple of retaliation after she reported sexual harassment and raised concerns about receiving less pay than her male colleagues.

The Superior Court of California for the County of Los Angeles wants nearly seventeen of the finest United States dollars for a copy of the complaint alone.

Tripp Mickle, New York Times:

But along the way, [Jay] Blahnik created a toxic work environment, said nine current and former employees who worked with or for Mr. Blahnik and spoke about personnel issues on the condition of anonymity. They said Mr. Blahnik, 57, who leads a roughly 100-person division as vice president for fitness technologies, could be verbally abusive, manipulative and inappropriate. His behavior contributed to decisions by more than 10 workers to seek extended mental health or medical leaves of absence since 2022, about 10 percent of the team, these people said.

The behaviours described in this article are deeply unprofessional, at best. It is difficult to square the testimony of a sizeable portion of Blahnik’s team with an internal investigation finding no wrongdoing, but that is what Apple’s spokesperson expects us to believe.

⌥ Permalink

Emily Price, Fast Company:

Meta’s Threads is on a roll.

The social networking app is now home to more than 400 million monthly active users, Meta shared with Fast Company on Tuesday. That’s 50 million more than just a few months ago, and a long way from the 175 million it had around its first birthday last summer.

What is even more amazing about this statistic is how non-essential Threads seems to be. I might be in a bubble, but I cannot recall the last time someone sent me a link to a Threads post or mentioned they saw something worthwhile there. I see plenty of screenshots of posts from Bluesky, X, and even Mastodon circulating in various other social networks, but I cannot remember a single one from Threads.

As if to illustrate Threads’ invisibility, Andy Stone, Meta’s communications guy, rebutted a Wall Street Journal story with a couple of posts on X. He has a Threads account, of course, but he posts there only a few times per month.

⌥ Permalink

Adam Engst, TidBits:

I’m certainly aware that many readers venture outside the Apple ecosystem for certain devices, but I’ve always assumed that most people would opt for Apple’s device in any given category. TidBITS does focus on Apple, after all, and Apple works hard to provide an integrated experience for those who go all-in on Apple. That integration disappears if you use a Mac along with a Samsung Galaxy phone and an Amazon Echo smart speaker.

Let’s put my assumption to the test! Or rather, to the poll. […]

It is a good question; you should take this quick poll if you have a couple of minutes.

This will not be bias-free, but I also have a hard time assuming what kind of bias will be found in a sample of an audience reading TidBits. My gut instinct is many people will be wholly immersed in Apple hardware. However, a TidBits reader probably skews a little more technical and particular — or so I read in the comments — so perhaps not? Engst’s poll only asks about primary hardware and not, say, users’ choice in keyboards or music streaming services, so perhaps it will be different than my gut tells me.

Update: On August 25, Engst revealed the results.

⌥ Permalink

Apple:

Apple today announced the expansion of its Self Service Repair and Genuine Parts Distributor programs to Canada, providing individuals and independent repair professionals across the country broader access to the parts, tools, and manuals needed to repair Apple devices.

As with other regions where Self-Service Repair is available, manuals are available on Apple’s website, but none of the listed parts and tools are linked to the still-sketchy-looking Self-Service Repair site.

There does not seem to be a pricing advantage, either. My wife’s iPhone 12 Pro needs a new battery. Apple says that costs $119 with a Genius Bar appointment, or I can pay $119 from the Self-Service store for a battery kit plus $67 for a week-long rental of all the required tools. This does not include a $1,500 hold on the credit card for the toolkit. After returning the spent battery, I would get a $57.12 credit, so it costs about $10 more to repair it myself than to bring it in. Perhaps that is just how much these parts cost; or, perhaps Apple is able to effectively rig the cost of repairs by competing only with itself. It is difficult to know.

One possible advantage of the Self-Service Repair option and the Genuine Parts Program is in making service more accessible to people in remote areas of Canada. I tried a remote address in Baker Lake, Nunavut, and the Self-Service Store still said it would ship free in 5–7 business days. Whether it would is a different story. Someone in a Canadian territory should please test this.

⌥ Permalink

U.S. Director of National Intelligence Tulsi Gabbard, in a tweet that happens to be the only communication of this news so far:

Over the past few months, I’ve been working closely with our partners in the UK, alongside @POTUS and @VP, to ensure Americans’ private data remains private and our Constitutional rights and civil liberties are protected.

As a result, the UK has agreed to drop its mandate for Apple to provide a “back door” that would have enabled access to the protected encrypted data of American citizens and encroached on our civil liberties.

Zoe Kleinman, BBC News:

The BBC understands Apple has not yet received any formal communication from either the US or UK governments.

[…]

In December, the UK issued Apple with a formal notice demanding the right to access encrypted data from its users worldwide.

It is unclear to me whether Gabbard is saying the U.K.’s backdoor requirement is entirely gone, or if it means the U.K. is only retreating from requiring worldwide access (or perhaps even only access to U.S. citizens’ data). The BBC, the New York Times, and the Washington Post are all interpreting this as a worldwide retreat, but Bloomberg, Reuters, and the Guardian say it is only U.S. data. None of them appear to have confirmation beyond Gabbard’s post, thereby illustrating the folly of an administration continuing to make policy decisions and announcements in tweet form. The news section of the Office of the Director of National Intelligence is instead obsessed with relitigating Russian interference in the dumbest possible way.

Because of the secrecy required of Apple and the U.K. government, this confusion cannot be clarified by the parties concerned, so one is entrusting the Trump administration to communicate this accurately. Perhaps the U.K. availability of Advanced Data Protection can be a canary — if it comes back, we can hope Apple is not complicit with weakening end-to-end encryption.

Also, it seems that Google has not faced similar demands.

⌥ Permalink

When I watched Tim Cook, in the White House, carefully assemble a glass-and-gold trophy fit for a king, it felt to me like a natural outcome of the events and actions exhaustively documented by Patrick McGee in “Apple in China”. It was a reflection of the arc of Cook’s career, and of Apple’s turnaround from dire straits to a kind of supranational superpower. It was a consequence of two of the world’s most powerful nations sliding toward the (even more) authoritarian, and a product of appeasement to strongmen on both sides of the Pacific.

At the heart of that media spectacle was an announcement by Apple of $100 billion in domestic manufacturing investment over four years, in addition to its existing $500 billion promise. This is an extraordinary amount of money to spend in the country from which Apple has extricated its manufacturing over the past twenty years. The message from Cook was “we’re going to keep building technologies at the heart of our products right here in America because we’re a proud American company and we believe deeply in the promise of this great nation”. But what becomes clear after digesting McGee’s book is that core Apple manufacturing is assuredly not returning to the United States.

Do not get me wrong: there is much to be admired in the complementary goals of reducing China-based manufacturing and an increasing U.S. role. Strip away for a minute the context of this president and his corrupt priorities. Rich nations have become dependent on people in poorer nations to make our stuff, and no nation is as critical to our global stuff supply than China. One of the benefits of global trade is that it can smooth local rockiness; a bad harvest season no longer has to mean a shortage of food. Yet even if we ignore their unique political environment and their detestable treatment of Uyghur peoples — among many domestic human rights abuses — it makes little sense for us to be so dependent on this one country. This is basically an antitrust problem.

At the same time, it sure would be nice if we made more of the stuff we buy closer to where we live. We have grown accustomed to externalizing the negative consequences of making all this stuff. Factories exist somewhere else, so the resources they consume and the pollution they create is of little concern to us. They are usually not staffed by a brand we know, and tasks may be subcontracted, so there is often sufficient plausible deniability vis a vis working conditions and labour standards. As McGee documents, activist campaigns had a brief period of limited success in pressuring Apple to reform its standards and crack down on misbehaviour before the pressure of product delivery caught up with the company and it stopped reporting its regressing numbers. Also, it is not as though Apple could truly avoid knowing the conditions at these factories when there are so many of its own employees working side-by-side with Foxconn.

All the work done by people in factories far away from where I live is, frankly, astonishing. Some people still erroneously believe the country of origin is an indicator of whether a product is made with any degree of finesse or care. This is simply untrue, and it has been for decades, as McGee emphasizes. This book is worth reading for this perspective alone. The goods made in China today are among the most precise and well-crafted anywhere, on a simply unbelievable scale. In fact, it is this very ability to produce so much great stuff so quickly that has tied Apple ever tighter to China, argues McGee:

Whereas smartphone rivals like Samsung could bolt a bunch of off-the-shelf components together and make a handset, Apple’s strategy required it to become ever more wedded to the industrial clusters forming around its production. As more of that work took place in China, with no other nation developing the same skills, Apple was growing dependent on the very capabilities it had created. (page 176)

Cook’s White House announcement, for all its patriotic fervour, only underscores this dependency. In the book’s introduction, McGee reports “Apple’s investments in China reached $55 billion per year by 2015, an astronomical figure that doesn’t include the costs of components in Apple hardware” (page 7). That sum built out a complete, nimble, and precise supply chain at vast scale. By contrast, Apple says it is contributing a total of $600 billion over four years, or $150 billion per year. In other words, it is investing about three times as much in the U.S. compared to China and getting far less. Important stuff, to be sure, but less. And, yes, Apple is moving some iPhone production out of China, but not to the U.S. — something like 18% of iPhones are now made in India. McGee’s sources are skeptical of the company’s ability to do so at scale given the organization of the supply chain and the political positioning of its contract manufacturers, but nobody involved thinks Apple is going to have a U.S. iPhone factory.

So much of this story is about the iPhone, and it can be difficult to remember Apple makes a lot of other products. To McGee’s credit, he spends the first two-and-a-half sections of this six-part book exploring Apple’s history, the complex production of the G3 and G4 iMacs, and the making of the iPod which laid the groundwork for the iPhone. But a majority of the rest of the book is about the iPhone. That is unsurprising.

First, the iPhone is the product of a staggering amount of manufacturing knowledge. It is also, of course, a sales bonanza.

In fact, among the most riveting stories in the book do not concern manufacturing at all. McGee writes of grey market iPhone sales — a side effect of which was the implementation of parts pairing and activation — and the early frenzy over the iPad. Most notably, McGee spends a couple of chapters — particularly “5 Alarm Fire” — dissecting the sub-par launch sales of the iPhone XR as revealed through executive emails and depositions after Apple was sued for allegedly misleading shareholders. The case was settled last year for $490 million without Apple admitting wrongdoing. Despite some of these documents becoming public in 2022, it seems nobody before McGee took the time to read through them. I am glad he did because it is revealing. Even pointing to the existence of these documents offers a fascinating glimpse of what Apple does when a product is selling poorly.

Frustratingly, McGee does not attribute specific claims or quotations to individual documents in this chapter. Virtually everything in “5 Alarm Fire” is cited simply to the case number, so you have to go poking around yourself if you wish to validate his claims or learn more about the story.¹ It may be worthwhile, however, since it underscores the unique risk Apple takes by releasing just a few new iPhones each year. If a model is not particularly successful, Apple is not going to quietly drop it and replace it with a different SKU. With the 2018 iPhones, Apple was rocked by a bunch of different problems, most notably the decent but uninteresting iPhone XR — 79% fewer preorders (PDF) when compared to the same sales channels as the iPhone 8 and 8 Plus — and the more exciting new phones from Huawei and Xiaomi released around the same time. Apple had hoped the 2018 iPhones would be more interesting to the Chinese market since they supported dual SIMs (PDF) and the iPhone XS came in gold. Apple responded to weak initial demand with targeted promotions, increasing production of the year-old iPhone X, and more marketing, but this was not enough and the company had to lower its revenue expectations for the quarter.

That Cook called this “obviously a disaster” is, of course, a relative term, as is the way I framed this as a “risk” of Apple’s smartphone release strategy. Apple still sold millions of iPhones — even the XR — and it still made a massive amount of money. It is a unique story, however, as it is one of the few times in the book where Apple has a problem of making too many products rather than too few. It is also illustrative of increasing competition from Chinese brands and, as emails reveal (PDF), trade tensions between the U.S. and China.

The fundamental heart of the story of this book is of the tension of a “proud American company” attempting to appease two increasingly nationalist and hostile governments. McGee examines Apple’s billion-dollar investment in Didi Chuxing, and mentions Cook’s appointment to the board of Tsinghua University School of Economics and Management. This is all part of the politicking the company realized it would need to do to appease President Xi. Similarly, its massive spending in China needed to be framed correctly. For example, in 2016, it said it was investing $275 billion in China over the following five years:

As mind-bogglingly large as its $275 billion investment was, it was not really a quid pro quo. The number didn’t represent any concession on Apple’s part. It was just the $55 billion the company estimated it’d invested for 2015, multiplied by five years. […] What was new, in other words, wasn’t Apple’s investment, but its marketing of the investment. China was accumulating reams of specialized knowledge from Apple, but Beijing didn’t know this because Apple had been so secretive. From this meeting forward, the days in which Apple failed to score any political points from its investments in the country were over. It was learning to speak the local language.

One can see a similar dynamic in the press releases for U.S. investments it began publishing one year later, after Donald Trump first took office. Like Xi, Trump was eager to bend Apple to his administration’s priorities. Some of the company’s actions and investments are probably the same as those it would have made anyhow, but it is important to these autocrat types that they believe they are calling the shots.

Among the reasons the U.S. has given for taking a more hostile trade position on China is its alleged and, in some cases, proven theft of intellectual property. McGee spends less time on this — in part, I imagine, because it is a hackneyed theme frequently used only to treat innovation by Chinese companies with suspicion and contempt. This book is a more levelheaded piece of analysis. Instead of having the de rigueur chapter or two dedicated to intellectual property leaving through the back door, McGee examines the less-reported front-door access points. Companies are pressured to participate in “joint ventures” with Chinese businesses to retain access to markets, for example; this is why iCloud in China is operated not by Apple, but by AIPO Cloud (Guizhou) Technology Co. Ltd.

Even though patent and design disputes are not an area of focus for McGee, it is part of the two countries’ disagreements over trade, and one area where Apple is again stuck in the middle. A concluding anecdote in the book references the launch of the Huawei Mate XT, a phone that folds in three which, to McGee, “appears to be a marvel of industrial engineering”:²

It was only in 2014 that Jony Ive complained of cheap Chinese phones and their brazen “theft” of his designs; it was 2018 when Cupertino expressed shock at Chinese brands’ ability to match the newest features; now, a Chinese brand is designing, manufacturing, and shipping more expensive phones with alluring features that, according to analysts, Apple isn’t expected to match until 2027. No wonder the most liked comment on a YouTube unboxing video of the Mate XT is, “Now you know why USA banned Huawei.” (pages 377–378)

The Mate XT was introduced the same day as the iPhone 16 line, and the differences could not have been more stark. The iPhone was a modest evolution of the company’s industrial design language, yet would be familiar to someone who had been asleep for the preceding fifteen years. The Mate XT was anything but. The phones also had something in common: displays made by BOE. The company is one of several suppliers for the iPhone, and it enables the radical design of Huawei’s phone. But according to Samsung, BOE’s ability to make OLED and flexible displays depends on technology stolen from them. The U.S. International Trade Commission agreed and will issue a final ruling in November which is likely to prohibit U.S. imports of BOE-made displays. It seems like this will be yet another point of tension between the U.S. and China, and another thing Cook can mention during his next White House visit.

“Apple in China” is, as you can imagine, dense. I have barely made a dent in exploring it here. It is about four hundred pages and not a single one is wasted. This is not one of those typical books about Apple; there is little in here you have read before. It answers a bunch of questions I have had and serves as a way to decode Apple’s actions for the past ten years and, I think, during this second Trump presidency.

At the same time, it leaves me asking questions I did not fully consider before. I have long assumed Apple’s willingness to comply with the demands of the Chinese government are due to its supply chain and manufacturing role. That is certainly true, but I also imagine the country’s sizeable purchasing power is playing an increasing role. That is, even if Apple decentralizes its supply chain — unlikely, if McGee’s sources are to be believed — it is perhaps too large and too alluring a market for Apple to ignore. Then again, it arguably created this problem itself. Its investments in China have been so large and, McGee argues, so impactful they can be considered in the same context as the U.S.’ post-World War II European recovery efforts. Also, the design of Apple’s ecosystem is such that it can be so deferential. If the Chinese government does not want people in its country using an app, the centralized App Store means it can be yanked away.³

Cook has previously advocated for expressing social values as a corporate principle. In 2017, he said, perhaps paraphrasing his heroes Martin Luther King Jr. and John Lewis, “if you see something going on that’s not right, the most powerful form of consent is to say nothing”. But how does Cook stand firmly for those values while depending on an authoritarian country for Apple’s hardware, and trying to appease a wanna-be dictator for the good standing of his business? In short, he does not. In long, well, it is this book.

It is this tension — ably shown by McGee in specific actions and stories rather than merely written about — that elevates “Apple in China” above the typical books about Apple and its executives. It is part of the story of how Apple became massive, how an operations team became so influential, and how the seemingly dowdy business of supply chains in China applied increasingly brilliant skills and became such a valuable asset in worldwide manufacturing. And it all leads directly to Tim Cook standing between Donald Trump and J.D. Vance in the White House, using the same autocrat handling skills he has practiced for years. Few people or businesses come out of this story looking good. Some look worse than others.

Nice scoop from Eric Schwarz:

Over the past week, I’ve been working to track down the new owner of MacSurfer’s Headline News, a beloved site that shut down in 2020 and has recently had somewhat mysterious revival. Fortunately, after some digging that didn’t really lead anywhere, I received an email from its new owner, Ken Turner, and he graciously took the time to answer a few questions about the new project.

Turner sounds like a great steward to carry on the MacSurfer legacy. Even in an era of well-known aggregators like Techmeme and massive forums like Hacker News and Reddit, I think there is still a role for a smaller and more focused media tracking site.

I am uncertain what the role of BackBeat Media is in all this. I have not heard from Dave Hamilton or anyone there to confirm if they even have a role.

⌥ Permalink

My thanks to Magic Lasso Adblock for sponsoring Pixel Envy this week.

With over 5,000 five star reviews, Magic Lasso Adblock is simply the best ad blocker for your iPhone, iPad, and Mac.

As an efficient, high performance and native Safari ad blocker, Magic Lasso blocks all intrusive ads, trackers and annoyances – delivering a faster, cleaner, and more secure web browsing experience.

And with the new App Ad Blocking feature in v5.0, it extends the powerful Safari and YouTube ad blocking protection to all apps including News apps, Social media, Games, and other browsers like Chrome and Firefox.

So, join over 350,000 users and download Magic Lasso Adblock today.

⌥ Permalink

Joseph Cox, 404 Media:

Members of a law enforcement group chat including Immigration and Customs Enforcement (ICE) and other agencies inadvertently added a random person to the group called “Mass Text” where they exposed highly sensitive information about an active search for a convicted attempted murderer seemingly marked for deportation, 404 Media has learned.

[…]

The person accidentally added to the group chat, which appears to contain six people, said they had no idea why they had received these messages, and shared screenshots of the chat with 404 Media. 404 Media granted the person anonymity to protect them from retaliation.

This is going to keep happening if law enforcement and government agencies keep communicating through ad hoc means instead of official channels. In fact — and I have no evidence to support this — I bet it has happened, but the errant recipients did not contact a journalist.

⌥ Permalink

Five years ago, Apple and tech news aggregator MacSurfer announced it was shutting down. The site was still accessible albeit in a stopped-time state, and it seemed that is how it would sit until the server died.

In June, though, MacSurfer was relaunched. The design has been updated and it is no longer as technically simple as it once was, but — charmingly — the logo appears to be the exact same static GIF as always. I cannot find any official announcement of its return.

Eric Schwarz:

It looks like Macsurfer is coming back, but I can’t find any details or who’s behind it? I really hope it’s not AI slop or someone trying to make a buck off nostalgia like iLounge or TUAW.

I had the same question, so I started digging. MxToolbox reveals a txt record on the domain for validating with Google apps, registered to BackBeat Media. BackBeat’s other properties include the Mac Observer, AppleInsider, and PowerPage. A review of historical MacSurfer txt records using SecurityTrails indicates the site has been with Backbeat Media since at least 2011, even though BackBeat’s site has not listed MacSurfer even when it was actively updated.

I cannot confirm the ownership is the same yet but I have asked Dave Hamilton, of BackBeat, and will update this if I hear back.

⌥ Permalink

Todays candles have been optimized for millenia not to flicker. But it turns out when we bundle three of them together, we can undo all of these optimizations and the resulting triplet will start to naturally oscillate. A fascinating fact is that the oscillation frequency is rather stable at ~9.9Hz as it mainly depends on gravity and diameter of the flame. 

We use a rather unusual approach based on a wire suspended in the flame, that can sense capacitance changes caused by the ionized gases in the flame, to detect this frequency and divide it down to 1Hz.

Introduction

Candlelight is a curious thing. Candles seem to have a life of their own: the brightness wanders, they flicker, and they react to the faintest motion of air.

There has always been an innate curiosity in understanding how candle flames work and behave. In recent years, people have also extensively sought to emulate this behavior with electronic light sources. I have also been fascinated by this and tried to understand real candles and how artificial candles work.

Now, it’s a curious thing that we try to emulate the imperfections of candles. After all, candle makers have worked for centuries (and millennia) on optimizing candles NOT to flicker?

In essence: The trick is that there is a very delicate balance in how much fuel (the molten candle wax) is fed into the flame. If there is too much, the candle starts to flicker even when undisturbed. This is controlled by how the wick is made.

Candle Triplet Oscillations

Now, there is a particularly fascinating effect that has more recently been the subject of publications in scientific journals¹² : When several candles are brought close to each other, they start to “communicate” and their behavior synchronizes. The simplest demonstration is to bundle three candles together; they will behave like a single large flame.

So, what happens with our bundle of three candles? It will basically undo millennia of candle technology optimization to avoid candle flicker. If left alone in motionless air, the flames will suddenly start to rapidly change their height and begin to flicker. The image below shows two states in that cycle.

Two states of the oscillation cycle in bundled candles

We can also record the brightness variation over time to understand this process better. In this case, a high-resolution ambient light sensor was used to sample the flicker over time. (This was part of more comprehensive set experiments of conducted a while ago, which are still unpublished)

Plotting the brightness evolution over time shows that the oscillations are surprisingly stable, as shown in the image below. We can see a very nice sawtooth-like signal: the flame slowly grows larger until it collapses and the cycle begins anew. You can see a video of this behavior here. (Which, unfortunately cannot embed properly due to WordPress…)

Left: Brightness variation over time showing sawtooth pattern.
Right: Power spectral density showing stable 9.9 Hz frequency

On the right side of the image, you can see the power spectral density plot of the brightness signal on the left. The oscillation is remarkably stable at a frequency of 9.9 Hz.

This is very curious. Wouldn’t you expect more chaotic behavior, considering that everything else about flames seems so random?

The phenomenon of flame oscillations has baffled researchers for a long time. Curiously, they found that the oscillation frequency of a candle flame (or rather a “wick-stabilized buoyant diffusion flame”) depends mainly on just two variables: gravity and the dimension of the fuel source. A comprehensive review can be found in Xia et al.³.

Now that is interesting: gravity is rather constant (on Earth) and the dimensions of the fuel source are defined by the size (diameter) of the candles and possibly their proximity. This leaves us with a fairly stable source of oscillation, or timing, at approximately 10Hz. Could we use the 9.9 Hz oscillation to derive a time base?

Sensing Candle Frequencies with a Phototransistor

Now that we have a source of stable oscillations—remind you, FROM FIRE—we need to convert them into an electrical signal.

The previous investigation of candle flicker was based an I²C-based light sensor to sample the light signal. This provides very high SNR, but is comparatively complex and adds latency.

A phototransistor provides a simpler option. Below you can see the setup with a phototransistor in a 3mm wired package (arrow). Since the phototransistor has internal gain, it provides a much higher current than a photodiode and can be easily picked up without additional amplification.

Phototransistor setup with sensing resistor configuration

The phototransistor was connected via a sensing resistor to a constant voltage source, with the oscilloscope connected across the sensing resistor. The output signal was quite stable and showed a nice ~9.9 Hz oscillation.

In the next step, this could be connected to an ADC input of a microcontroller to process the signal further. But curiously, there is also a simpler way of detecting the flame oscillations.

Capacitive Flame Sensing

Capacitive touch peripherals are part of many microcontrollers and can be easily implemented with an integrated ADC by measuring discharge rates versus an integrated pull-up resistor, or by a charge-sharing approach in a capacitive ADC.

While this is not the most obvious way of measuring changes in a flame, it is to be expected to observe some variations. The heated flame with all its combustion products contains ionized molecules to some degree and is likely to have different dielectric properties compared to the surrounding air, which will be observed as either a change of capacitance or increased electrical loss. A quick internet search also revealed publications on capacitance-based flame detectors.

A CH32V003 microcontroller with the CH32fun environment was used for experiments. The set up is shown below: the microcontroller is located on the small PCB to the left. The capacitance is sensed between a wire suspended in the flame (the scorched one) and a ground wire that is wound around the candle. The setup is completed with an LED as an output.

Complete capacitive sensing setup with CH32V003 microcontroller, candle triplet and a LED.

Initial attempts with two wires in the flame did not yield better results and the setup was mechanically much more unstable.

Read out was implemented straightforward using the TouchADC function that is part of CH32fun. This function measures the capacitance on an input pin by charging it to a voltage and measuring voltage decay while it is discharged via a pull-up/pull-down resistor. To reduce noise, it was necessary to average 32 measurements.

// Enable GPIOD, C and ADC
RCC->APB2PCENR |= RCC_APB2Periph_GPIOA | RCC_APB2Periph_GPIOD | RCC_APB2Periph_GPIOC | RCC_APB2Periph_ADC1;

InitTouchADC();
...

int iterations = 32;
sum = ReadTouchPin( GPIOA, 2, 0, iterations );

First attempts confirmed to concept to work. The sample trace below shows sequential measurements of a flickering candle until it was blown out at the end, as signified by the steep drop of the signal.

The signal is noisier than the optical signal and shows more baseline wander and amplitude drift—but we can work with that. Let’s put it all together.

Capacitive sensing trace showing candle oscillations and extinction

Putting everything together

Additional digitial signal processing is necessary to clean up the signal and extract a stable 1 Hz clock reference.

The data traces were recorded with a Python script from the monitor output and saved as csv files. A separate Python script was used to analyze the data and prototype the signal processing chain. The sample rate is limited to around ~90 Hz due to the overhead of printing data via the debug output, but the data rate turned out to be sufficient for this case.

The image above shows an overview of the signal chain. The raw data (after 32x averaging) is shown on the left. The signal is filtered with an IIR filter to extract the baseline (red). The middle figure shows the signal with baseline removed and zero-cross detection. The zero-cross detector will tag the first sample after a negative-to-positive transition with a short dead-time to prevent it from latching to noise. The right plot shows the PSD of the overall and high-pass filtered signal, showing that despite the wandering input signal, we get a sharp ~9.9 Hz peak for the main frequency.

A detailed zoom-in of raw samples with baseline and HP filtered data is shown below.

The inner loop code is shown below, including implementation of IIR filter, HP filter, and zero-crossing detector. Conversion from 9.9 Hz to 1 Hz is implemented using a fractional counter. The output is used to blink the attached LED. Alternatively, an advanced implementation using a software-implemented DPLL might provide a bit more stability in case of excessive noise or missing zero crossings, but this was not attempted for now.

const int32_t led_toggle_threshold = 32768;  // Toggle LED every 32768 time units (0.5 second)
const int32_t interval = (int32_t)(65536 / 9.9); // 9.9Hz flicker rate
...

sum = ReadTouchPin( GPIOA, 2, 0, iterations );

if (avg == 0) { avg = sum;} // initialize avg on first run
avg = avg - (avg>>5) + sum; // IIR low-pass filter for baseline
hp = sum -  (avg>>5); // high-pass filter

// Zero crossing detector with dead time
if (dead_time_counter > 0) {
    dead_time_counter--;  // Count down dead time
    zero_cross = 0;  // No detection during dead time
} else {
    // Check for positive zero crossing (sign change)
    if ((hp_prev < 0 && hp >= 0)) {
        zero_cross = 1;  
        dead_time_counter = 4;  
        time_accumulator += interval;  
        
        // LED blinking logic using time accumulator
        // Check if time accumulator has reached LED toggle threshold
        if (time_accumulator >= led_toggle_threshold) {
            time_accumulator = time_accumulator - led_toggle_threshold;  // Subtract threshold (no modulo)
            led_state = led_state ^ 1;  // Toggle LED state using XOR
            
            // Set or clear PC4 based on LED state
            if (led_state) {
                GPIOC->BSHR = 1<<4;  // Set PC4 high
            } else {
                GPIOC->BSHR = 1<<(16+4);  // Set PC4 low
            }
        }
    } else {
        zero_cross = 0;  // No zero crossing
    }
}

hp_prev = hp;

Finally, let’s marvel at the result again! You can see the candle flickering at 10 Hz and the LED next to it blinking at 1 Hz! The framerate of the GIF is unfortunately limited, which causes some aliasing. You can see a higher framerate version on YouTube or the original file.

That’s all for our journey from undoing millennia of candle-flicker-mitigation work to turning this into a clock source that can be sensed with a bare wire and a microcontroller. Back to the decade-long quest to build a perfect electronic candle emulation…

All data and code is published in this repository.

This is an entry to the HaD.io “One Hertz Challenge”

References

1. Okamoto, K., Kijima, A., Umeno, Y. & Shima, H. “Synchronization in flickering of three-coupled candle flames.”  Scientific Reports 6, 36145 (2016). ︎
2. Chen, T., Guo, X., Jia, J. & Xiao, J. “Frequency and Phase Characteristics of Candle Flame Oscillation.”  Scientific Reports 9, 342 (2019). ︎
3. J. Xia and P. Zhang, “Flickering of buoyant diffusion flames,” Combustion Science and Technology, 2018. ︎

I missed last month’s Recently because I was traveling. I’ll be pretty busy this weekend too, so I’ll publish this now: a solid double-length post to make up for it.

Listening

It’s been a really good time for music: both discovering new albums by bands I’ve followed, and finding new stuff out of the blue. I get the question occasionally of “how do I find music”, and the answer is:

• When you buy an album off of Bandcamp, by default you get notifications when new albums are released.
• I’m an extremely active listener and will eagerly pursue songs that I hear every day: the Shazam app is always at hand, and I’ll pick up music from the background of TV shows or movies. Three of the albums I picked up this month were by this method: Goat’s album was playing at an excellent vegetarian restaurant called Handlebar in Chicago, and the Ezra Collective & Baby Rose songs were played on the speakers in the hotel in Berlin.
• I look up bands and the people in them. The Duffy x Uhlmann album came up when I looked up the members of SML, whose album I mentioned in February: Gregory Uhlmann is the guitarist in both bands. Wikipedia, personal websites, and sometimes reviews are useful in this kind of browsing.
• Hearing Things is still a great source. For me, their recommendations line up maybe 10% of the time, and that’s good: it gives me exposure to genres that I don’t listen to and I’ll probably warm to eventually.

As I’ve mentioned before, having a definable, finite music catalog changes how I feel about and perceive music. Songs can be waypoints, place markers if you let them be. You can recognize the first few notes and remember who you were when you first heard that tune. It’s a wonderful feeling, a sense of digital home in a way that no streaming service can replicate.

So: to the songs

Duffy x Uhlmann sometimes reminds me of The Books. It’s a pairing of guitar & bass that I don’t see that often in avant-jazz-experimental music.

The rhythm on this track. Ezra Collective is danceable futuristic jazz.

I think I’ve listened this song out. It’s one of those songs that I listened to multiple times in a row after I bought the album because I just wanted to hear that hook.

I realized that there are more Do Make Say Think albums than I thought! This one’s great.

It’s a Swedish ‘world music’ band called Goat that came out with an album “World Music (2024)” that has three goat-themed songs on it: Goatman, Goatlord, and Goathead. Nevertheless, this is a jam.

Cassandra Jenkins, who I first found via David Berman’s Purple Mountains, records consistently very comfortable-sounding deep music.

Watching

Elephant Graveyard is a YouTube channel that critiques the right-wing ‘comedy’ scene. It’s a really well-produced, well-written, funny takedown, and the conclusion that Joe Rogan and right-wing tech oligarchs are creating an alternate reality has a lot in common with Adam Curtis’s documentaries. It’s a pretty useful lens through which to view the disaster.

In response to this video, YouTube/Alphabet/Google responded:

We’re running an experiment on select YouTube Shorts that uses traditional machine learning technology to unblur, denoise and improve clarity in videos during processing (similar to what a modern smartphone does when you record a video)

This is the first time I’ve heard traditional machine learning technology used as a term. Sigh.

Honestly, I am not really a connoisseur of video content: any smart thing I can say about films or TV shows is just extrapolating from what I know about photography and retouching, which is something that I have a lot of experience with. But from that perspective, it’s notable how platforms and ‘creators’ have conflicting incentives: a company like YouTube benefits from all of its content looking kind of homogenous in the same way as Amazon benefits from minimizing some forms of brand awareness. And AI is a superweapon of homogenisation, both intentional and incidental.

I still use YouTube but I want to stop, in part because of this nonsense. It’s sad that a decentralized or even non-Google YouTube alternative is so hard to stand up because of the cost of video streaming. The people running YouTube channels are doing good work that I enjoy, but it’s a sad form of platform lock-in that everyone’s experiencing.

As a first step, I’m going to tinker with avoiding the YouTube website experience: thankfully there are a lot of ways to do that, like Invidious.

Reading

Because the oral world finds it difficult to define and discuss why abstract analytical categories like “moral behavior” or “hard work” are good in their own right, moral instruction has to take the form of children’s stories, where good behavior leads to better personal outcomes.

Joe Weisenthal on AI, Orality, and the Golden Age of Grift is really worth reading. It’s behind a Bloomberg paywall, though: is it weird that Bloomberg is one of my primary news sources? I feel it all in my bones: how the ideas of things being moral and worthwhile are being eroded by the same forces. The whole thing becomes a Keynesian beauty contest of trying to crowd into the same popular things because they’re popular. Like Joe, I find it all incredibly tiring and dispiriting, in part because like a true millennial and like a true former Catholic, I actually do think that morality exists and is really important.

A lot of the focus of e-mobility is on increasing comfort, decreasing exertion, and selling utopias — all of which undermine the rewards of cost-effectiveness, sustainability, physicality, interaction with the world, autonomy, community, and fun that cycling offers.

The Radavist’s coverage of Eurobike, by Petor Georgallou has hints of Gonzo journalism in its account of sweating through and generally not enjoying a big bicycle industry event. I have complicated feelings about e-bikes and e-mobility, not distinct enough from the feelings of better writers so they aren’t really worth writing longform but: it’s good that e-bikes encourage people to bike when they would have driven, it’s cool that some people get more exercise on e-bikes because they’re easier to ride for more purposes, and it’s bad that cities crack down on e-bikes instead of cars. But on the other side, e-bikes make their riders less connected to reality, to other people, and to their bodies than regular bikes do, and they have proprietary, electronic, disposable parts - eliminating one of the things that I love most about bicycles, their extremely long lifespans. I have to say that the average e-bike user I see is less cautious, less engaged, and less happy than the average bicyclist. Being connected to base reality is one of my highest priorities right now and bicycles do it, and e-bikes don’t.

Speaking of which: Berm Peak’s new video about ebikes covers a lot of the same notes. The quote about kids learning how to ride ebikes before they learn to ride a non-electric bike is just so sad.

The relentless pull of productivity—that supposed virtue in our society—casts nearly any downtime as wasteful. This harsh judgment taints everything from immersive video games to quieter, seemingly innocuous tasks like tweaking the appearance of a personal website. I never worried about these things when I was younger, because time was an endless commodity; though I often felt limited in the particulars of a moment, I also knew boundlessness in possibility.

Reading through the archives of netigen, finding more gems like this.

windsurf wasn’t a company. it was an accidentally subsidized training program that discovered the most valuable output wasn’t code — it was coders who knew how to build coding models.

This analysis of windsurf is extremely lucid and harsh. I don’t like the writing style at all but it tells the truth.

Another friend commiserated the difficulty of trying to help an engineer contribute at work. “I review the code, ask for changes, and then they immediately hit me with another round of AI slop.”

From catskull’s blog. Congrats on leaving the industry! Thankfully at Val Town the AI usage is mature and moderate, but everything I hear from the rest of the industry sounds dire.

One begins to suspect that a great many students wanted this all along: to make it through college unaltered, unscathed. To be precisely the same person at graduation, and after, as they were on the first day they arrived on campus. As if the whole experience had never really happened at all.

From ‘I Used to Teach Students. Now I Catch ChatGPT Cheats’. More AI doom?

“It is without a doubt the most illegal search I’ve ever seen in my life,” U.S. Magistrate Judge Zia Faruqui said from the bench. “I’m absolutely flabbergasted at what has happened. A high school student would know this was an illegal search.”

“Lawlessness cannot come from the government,” Judge Faruqui added. “The eyes of the world are on this city right now.”

From this NPR article on the extraordinarily bad cases being brought against people in Washington, DC right now. This era has a constant theme of raw power outweighting intelligence or morality, which makes intelligent or principled people like this judge extremely frustrated.

Quantum technologies and the prospect of advanced, next-generation electronic devices have been maturing at an increasingly rapid pace. Both research groups and governments around the world are investing more attention in this domain.

India for example mooted its National Quantum Mission in 2023 with a decade-long outlay of Rs 6,000 crore. One of the Mission’s goals, in the words of IISER Pune physics professor Umakant Rapol, is “to engineer and utilise the delicate quantum features of photons and subatomic particles to build advanced sensors” for applications in “healthcare, security, and environmental monitoring”.

On the science front, as these technologies become better understood, scientists have been paying increasingly more attention to managing and controlling heat in them. These technologies often rely on quantum physical phenomena that appear only at extremely low temperatures and are so fragile that even a small amount of stray heat can destabilise them. In these settings, scientists have found that traditional methods of handling heat — mainly by controlling the vibrations of atoms in the devices’ materials — become ineffective.

Instead, scientists have identified a promising alternative: energy transfer through photons, the particles of light. And in this paradigm, instead of simply moving heat from one place to another, scientists have been trying to control and amplify it, much like how transistors and amplifiers handle electrical signals in everyday electronics.

Playing with fire

Central to this effort is the concept of a thermal transistor. This device resembles an electrical transistor but works with heat instead of electrical current. Electrical transistors amplify or switch currents, allowing the complex logic and computation required to power modern computers. Creating similar thermal devices would represent a major advance, especially for technologies that require very precise temperature control. This is particularly true in the sub-kelvin temperature range where many quantum processors and sensors operate.

Energy transport at such cryogenic temperatures differs significantly from normal conditions. Below roughly 1 kelvin, atomic vibrations no longer carry most of the heat. Instead, electromagnetic fluctuations — ripples of energy carried by photons — dominate the conduction of heat. Scientists channel these photons through specially designed, lossless wires made of superconducting materials. They keep these wires below their superconducting critical temperatures, allowing only photons to transfer energy between the reservoirs. This arrangement enables careful and precise control of heat flow.

One crucial phenomenon that allows scientists to manipulate heat in this way is negative differential thermal conductance (NDTC). NDTC defies common intuition. Normally, decreasing the temperature difference between two bodies reduces the amount of heat they exchange. This is why a glass of water at 50º C in a room at 25º C will cool faster than a glass of water at 30º C. In NDTC, however, reducing the temperature difference between two connected reservoirs can actually increase the heat flow between them.

NDTC arises from a detailed relationship between temperature and the properties of the material that makes up the reservoirs. When physicists harness NDTC, they can amplify heat signals in a manner similar to how negative electrical resistance powers electrical amplifiers.

A ‘circuit’ for heat

In a new study, researchers from Italy have designed and theoretically modelled a new kind of ‘thermal transistor’ that they have said can actively control and amplify how heat flows at extremely low temperatures for quantum technology applications. Their findings were published recently in the journal Physical Review Applied.

To explore NDTC experimentally, the researchers studied reservoirs made of a disordered semiconductor material that exhibited a transport mechanism called variable range hopping (VRH). An example is neutron-transmutation-doped germanium. In VRH materials, the electrical resistance at low temperatures depends very strongly, sometimes exponentially, on temperature.

This attribute makes them ideal to tune their impedance, a property that controls the material’s resistance to energy flow, simply by adjusting temperature. That is, how well two reservoirs made of VRH materials exchange heat can be controlled by tuning the impedance of the materials, which in turn can be controlled by tuning their temperature.

In the new study, the researchers reported that impedance matching played a key role. When the reservoirs’ impedances matched perfectly (when their temperatures became equal), the efficiency with which they transferred photonic heat reached a peak. As the materials’ temperatures diverged, heat flow dropped. In fact, the researchers wrote that there was a temperature range, especially as the colder reservoir’s temperature rose to approach that of the warmer one, within which the heat flow increased even as the temperature difference shrank. This effect forms the core of NDTC.

The research team, associated with the NEST initiative at the Istituto Nanoscienze-CNR and Scuola Normale Superiore, both in Pisa in Italy, have proposed a device they call the photonic heat amplifier. They built it using two VRH reservoirs connected by superconducting, lossless wires. One reservoir was kept at a higher temperature and served as the source of heat energy. The other reservoir, called the central island, received heat by exchanging photons with the warmer reservoir.

The central island was also connected to two additional metallic reservoirs named the “gate” and the “drain”. These points operated with the same purpose as the control and output terminals in an electrical transistor. The drain stayed cold, allowing the amplified heat signal to exit the system from this point. By adjusting the gate temperature, the team could modulate and even amplify the flow of heat between the source and the drain (see image below).

To understand and predict the amplifier’s behaviour, the researchers developed mathematical models for all forms of heat transfer within the device. These included photonic currents between VRH reservoirs, electron tunnelling through the gate and drain contacts, and energy lost as vibrations through the device’s substrate.

(Tunnelling is a quantum mechanical phenomenon where an electron has a small chance of floating through a thin barrier instead of going around it.)

Raring to go

By carefully selecting the device parameters — including the characteristic temperature of the VRH material, the source temperature, resistances at the gate and drain contacts, the volume of the central island, and geometric factors — the researchers said they could tailor the device for different amplification purposes.

They reported two main operating modes. The first was called ‘current modulation amplifier’. In this configuration, the device amplified small variations in thermal input at the gate. In this mode, small oscillations in the gate heat current produced much larger oscillations, up to 15-times greater, in the photon current between the source and the central island and in the drain current, according to the paper. This amplification was efficient down to 20 millikelvin, matching the ultracold conditions required in quantum technologies. The output range of heat current was similarly broad, showing the device’s suitability to amplify heat signals.

The second mode was called ‘temperature modulation amplifier’. Here, slight changes of only a few millikelvin in the gate temperature, the team wrote, caused the output temperature in the central island to swing by as large as 3.3 times the changes in the input. The device could also handle input temperature ranges over 100 millikelvin. This performance reportedly matched or surpassed other temperature amplifiers already reported in the scientific literature. The researchers also noted that this mode could be used to pre-amplify signals in bolometric detectors used in astronomy telescopes.

An important ability relevant for practical use is the relaxation time, i.e. how soon after operating once the device returned to its original state, ready for the next run. The amplifier in both configurations showed relaxation times between microseconds and milliseconds. According to the researchers, this speed resulted from the device’s low thermal mass and efficient heat channels. Such a fast response could make it suitable to detect and amplify thermal signals in real time.

The researchers wrote that the amplifier also maintained good linearity and low distortion across various inputs. In other words, the output heat signal changed proportionally to the input heat signal and the device didn’t add unwanted changes, noise or artifacts to the input signal. Its noise-equivalent power values were also found to rival the best available solid-state thermometers, indicating low noise levels.

Approaching the limits

For these promising results, realising this device involves some significant practical challenges. For instance, NDTC depends heavily on precise impedance matching. Real materials inevitably have imperfections, including those due to imperfect fabrication and environmental fluctuations. Such deviations could lower the device’s heat transfer efficiency and reduce the operational range of NDTC.

The system also banked on lossless superconducting wires being kept well below their critical temperatures. Achieving and maintaining these ultralow temperatures requires sophisticated and expensive refrigeration infrastructure, which adds to the experimental complexity.

Fabrication also demands very precise doping and finely tuned resistances for the gate and drain terminals. Scaling production to create many devices or arrays poses major technical difficulties. Integrating numerous photonic heat amplifiers into larger thermal circuits risks unwanted thermal crosstalk and signal degradation, a risk compounded by the extremely small heat currents involved.

Furthermore, the fully photonic design offers benefits such as electrical isolation and long-distance thermal connections. However, it also approaches fundamental physical limits. Thermal conductance caps the maximum possible heat flow through photonic channels. This limitation could restrict how much power the device is able to handle in some applications.

Then again, many of these challenges are typical of cutting-edge research in quantum devices, and highlight the need for detailed experimental work to realise and integrate photonic heat amplifiers into operational quantum systems.

If they are successfully realised for practical applications, photonic heat amplifiers could transform how scientists manage heat in quantum computing and nanotechnologies that operate near absolute zero. They could pave the way for on-chip heat control, computers to autonomously stabilise the temperature, and perform thermal logic operations. Redirecting or harvesting waste heat could also improve the efficiency and significantly reduce noise — a critical barrier in ultra-sensitive quantum devices like quantum computers.

Featured image credit: Lucas K./Unsplash.

I believe my blog’s subscribers did not receive email notifications of some recent posts. If you’re interested, I’ve listed the links to the last eight posts at the bottom of this edition.

When reading around for my piece yesterday on the wavefunctions of quantum mechanics, I stumbled across an old and fascinating debate about Saturn’s moon Hyperion.

The question of how the smooth, classical world around us emerges from the rules of quantum mechanics has haunted physicists for a century. Most of the time the divide seems easy: quantum laws govern atoms and electrons while planets, chairs, and cats are governed by the laws of Newton and Einstein. Yet there are cases where this distinction is not so easy to draw. One of the most surprising examples comes not from a laboratory experiment but from the cosmos.

In the 1990s, Hyperion became the focus of a deep debate about the nature of classicality, one that quickly snowballed into the so-called Hyperion dispute. It showed how different interpretations of quantum theory could lead to apparently contradictory claims, and how those claims can be settled by making their underlying assumptions clear.

Hyperion is not one of Saturn’s best-known moons but it is among the most unusual. Unlike round bodies such as Titan or Enceladus, Hyperion has an irregular shape, resembling a potato more than a sphere. Its surface is pocked by craters and its interior appears porous, almost like a sponge. But the feature that caught physicists’ attention was its rotation. Hyperion does not spin in a steady, predictable way. Instead, it tumbles chaotically. Its orientation changes in an irregular fashion as it orbits Saturn, influenced by the gravitational pulls of Saturn and Titan, which is a moon larger than Mercury.

In physics, chaos does not mean complete disorder. It means a system is sensitive to its initial conditions. For instance, imagine two weather models that start with almost the same initial data: one says the temperature in your locality at 9:00 am is 20.000º C, the other says it’s 20.001º C. That seems like a meaningless difference. But because the atmosphere is chaotic, this difference can grow rapidly. After a few days, the two models may predict very different outcomes: one may show a sunny afternoon and the other, thunderstorms.

This sensitivity to initial conditions is often called the butterfly effect — it’s the idea that the flap of a butterfly’s wings in Brazil might, through a chain of amplifications, eventually influence the formation of a tornado in Canada.

Hyperion behaves in a similar way. A minuscule difference in its initial spin angle or speed grows exponentially with time, making its future orientation unpredictable beyond a few months. In classical mechanics this is chaos; in quantum mechanics, those tiny initial uncertainties are built in by the uncertainty principle, and chaos amplifies them dramatically. As a result, predicting its orientation more than a few months ahead is impossible, even with precise initial data.

To astronomers, this was a striking case of classical chaos. But to a quantum theorist, it raised a deeper question: how does quantum mechanics describe such a macroscopic, chaotic system?

Why Hyperion interested quantum physicists is rooted in that core feature of quantum theory: the wavefunction. A quantum particle is described by a wavefunction, which encodes the probabilities of finding it in different places or states. A key property of wavefunctions is that they spread over time. A sharply localised particle will gradually smear out, with a nonzero probability of it being found over an expanding region of space.

For microscopic particles such as electrons, this spreading occurs very rapidly. For macroscopic objects, like a chair, an orange or you, the spread is usually negligible. The large mass of everyday objects makes the quantum uncertainty in their motion astronomically small. This is why you don’t have to be worried about your chai mug being in two places at once.

Hyperion is a macroscopic moon, so you might think it falls clearly on the classical side. But this is where chaos changes the picture. In a chaotic system, small uncertainties get amplified exponentially fast. A variable called the Lyapunov exponent measures this sensitivity. If Hyperion begins with an orientation with a minuscule uncertainty, chaos will magnify that uncertainty at an exponential rate. In quantum terms, this means the wavefunction describing Hyperion’s orientation will not spread slowly, as for most macroscopic bodies, but at full tilt.

In 1998, the Polish-American theoretical physicist Wojciech Zurek calculated that within about 20 years, the quantum state of Hyperion should evolve into a superposition of macroscopically distinct orientations. In other words, if you took quantum mechanics seriously, Hyperion would be “pointing this way and that way at once”, just like Schrödinger’s famous cat that is alive and dead at once.

This startling conclusion raised the question: why do we not observe such superpositions in the real Solar System?

Zurek’s answer to this question was decoherence. Say you’re blowing a soap bubble in a dark room. If no light touches it, the bubble is just there, invisible to you. Now shine a torchlight on it. Photons from the bulb will scatter off the bubble and enter your eyes, letting you see its position and color. But here’s the catch: every photon that bounces off the bubble also carries away a little bit of information about it. In quantum terms, the bubble’s wavefunction becomes entangled with all those photons.

If the bubble were treated purely quantum mechanically, you could imagine a strange state where it was simultaneously in many places in the room — a giant superposition. But once trillions of photons have scattered off it, each carrying “which path?” information, the superposition is effectively destroyed. What remains is an apparent mixture of “bubble here” or “bubble there”, and to any observer the bubble looks like a localised classical object. This is decoherence in action: the environment (the sea of photons here) acts like a constant measuring device, preventing large objects from showing quantum weirdness.

For Hyperion, decoherence would be rapid. Interactions with sunlight, Saturn’s magnetospheric particles, and cosmic dust would constantly ‘measure’ Hyperion’s orientation. Any coherent superposition of orientations would be suppressed almost instantly, long before it could ever be observed. Thus, although pure quantum theory predicts Hyperion’s wavefunction would spread into cat-like superpositions, decoherence explains why we only ever see Hyperion in a definite orientation.

Thus Zurek argued that decoherence is essential to understand how the classical world emerges from its quantum substrate. To him, Hyperion provided an astronomical example of how chaotic dynamics could, in principle, generate macroscopic superpositions, and how decoherence ensures these superpositions remain invisible to us.

Not everyone agreed with Zurek’s conclusion, however. In 2005, physicists Nathan Wiebe and Leslie Ballentine revisited the problem. They wanted to know: if we treat Hyperion using the rules of quantum mechanics, do we really need the idea of decoherence to explain why it looks classical? Or would Hyperion look classical even without bringing the environment into the picture?

To answer this, they did something quite concrete. Instead of trying to describe every possible property of Hyperion, they focused on one specific and measurable feature: the part of its spin that pointed along a fixed axis, perpendicular to Hyperion’s orbit. This quantity — essentially the up-and-down component of Hyperion’s tumbling spin — was a natural choice because it can be defined both in classical mechanics and in quantum mechanics. By looking at the same feature in both worlds, they could make a direct comparison.

Wiebe and Ballentine then built a detailed model of Hyperion’s chaotic motion and ran numerical simulations. They asked: if we look at this component of Hyperion’s spin, how does the distribution of outcomes predicted by classical physics compare with the distribution predicted by quantum mechanics?

The result was striking. The two sets of predictions matched extremely well. Even though Hyperion’s quantum state was spreading in complicated ways, the actual probabilities for this chosen feature of its spin lined up with the classical expectations. In other words, for this observable, Hyperion looked just as classical in the quantum description as it did in the classical one.

From this, Wiebe and Ballentine drew a bold conclusion: that Hyperion doesn’t require decoherence to appear classical. The agreement between quantum and classical predictions was already enough. They went further and suggested that this might be true more broadly: perhaps decoherence is not essential to explain why macroscopic bodies, the large objects we see around us, behave classically.

This conclusion went directly against the prevailing view of quantum physics as a whole. By the early 2000s, many physicists believed that decoherence was the central mechanism that bridged the quantum and classical worlds. Zurek and others had spent years showing how environmental interactions suppress the quantum superpositions that would otherwise appear in macroscopic systems. To suggest that decoherence was not essential was to challenge the very foundation of that programme.

The debate quickly gained attention. On one side stood Wiebe and Ballentine, arguing that simple agreement between quantum and classical predictions for certain observables was enough to resolve the issue. On the other stood Zurek and the decoherence community, insisting that the real puzzle was more fundamental: why we never observe interference between large-scale quantum states.

At this time, the Hyperion dispute wasn’t just about a chaotic moon. It was about how we could define ‘classical behavior’ in the first place. For Wiebe and Ballentine, classical meant “quantum predictions match classical ones”. For Zurek et al., classical meant “no detectable superpositions of macroscopically distinct states”. The difference in definitions made the two sides seem to clash.

But then, in 2008, physicist Maximilian Schlosshauer carefully analysed the issue and showed that the two sides were not actually talking about the same problem. The apparent clash arose because Zurek and Wiebe-Ballentine had started from essentially different assumptions.

Specifically, Wiebe and Ballentine had adopted the ensemble interpretation of quantum mechanics. In everyday terms, the ensemble interpretation says, “Don’t take the quantum wavefunction too literally.” That is, it does not describe the “real state” of a single object. Instead, it’s a tool to calculate the probabilities of what we will see if we repeat an experiment many times on many identical systems. It’s like rolling dice. If I say the probability of rolling a 6 is 1/6, that probability does not describe the dice themselves as being in a strange mixture of outcomes. It simply summarises what will happen if I roll a large collection of dice.

Applied to quantum mechanics, the ensemble interpretation works the same way. If an electron is described by a wavefunction that seems to say it is “spread out” over many positions, the ensemble interpretation insists this does not mean the electron is literally smeared across space. Rather, the wavefunction encodes the probabilities for where the electron would be found if we prepared many electrons in the same way and measured them. The apparent superposition is not a weird physical reality, just a statistical recipe.

Wiebe and Ballentine carried this outlook over to Hyperion. When Zurek described Hyperion’s chaotic motion as evolving into a superposition of many distinct orientations, he meant this as a literal statement: without decoherence, the moon’s quantum state really would be in a giant blend of “pointing this way” and “pointing that way”. From his perspective, there was a crisis because no one ever observes moons or chai mugs in such states. Decoherence, he argued, was the missing mechanism that explained why these superpositions never show up.

But under the ensemble interpretation, the situation looks entirely different. For Wiebe and Ballentine, Hyperion’s wavefunction was never a literal “moon in superposition”. It was always just a probability tool, telling us the likelihood of finding Hyperion with one orientation or another if we made a measurement. Their job, then, was simply to check: do these quantum probabilities match the probabilities that classical physics would give us? If they do, then Hyperion behaves classically by definition. There is no puzzle to be solved and no role for decoherence to play.

This explains why Wiebe and Ballentine concentrated on comparing the probability distributions for a single observable, namely the component of Hyperion’s spin along a chosen axis. If the quantum and classical results lined up — as their calculations showed — then from the ensemble point of view Hyperion’s classicality was secured. The apparent superpositions that worried Zurek were never taken as physically real in the first place.

Zurek, on the other hand, was addressing the measurement problem. In standard quantum mechanics, superpositions are physically real. Without decoherence, there is always some observable that could reveal the coherence between different macroscopic orientations. The puzzle is why we never see such observables registering superpositions. Decoherence provided the answer: the environment prevents us from ever detecting those delicate quantum correlations.

In other words, Zurek and Wiebe-Ballentine were tackling different notions of classicality. For Wiebe and Ballentine, classicality meant the match between quantum and classical statistical distributions for certain observables. For Zurek, classicality meant the suppression of interference between macroscopically distinct states.

Once Schlosshauer spotted this difference, the apparent dispute went away. His resolution showed that the clash was less over data than over perspectives. If you adopt the ensemble interpretation, then decoherence indeed seems unnecessary, because you never take the superposition as a real physical state in the first place. If you are interested in solving the measurement problem, then decoherence is crucial, because it explains why macroscopic superpositions never manifest.

The overarching takeaway is that, from the quantum point of view, there is no single definition of what constitutes “classical behaviour”. The Hyperion dispute forced physicists to articulate what they meant by classicality and to recognise the assumptions embedded in different interpretations. Depending on your personal stance, you may emphasise the agreement of statistical distributions or you may emphasise the absence of observable superpositions. Both approaches can be internally consistent — but they  also answer different questions.

For school students that are reading this story, the Hyperion dispute may seem obscure. Why should we care about whether a distant moon’s tumbling motion demands decoherence or not? The reason is that the moon provides a vivid example of a deep issue: how do we reconcile the strange predictions of quantum theory with the ordinary world we see?

In the laboratory, decoherence is an everyday reality. Quantum computers, for example, must be carefully shielded from their environments to prevent decoherence from destroying fragile quantum information. In cosmology, decoherence plays a role in explaining how quantum fluctuations in the early universe influenced the structure of galaxies. Hyperion showed that even an astronomical body can, in principle, highlight the same foundational issues.

Last five posts:

1. The guiding light of KD45

2. What on earth is a wavefunction?

3.  The PixxelSpace constellation conundrum

4. The Zomato ad and India’s hustle since 1947

5. A new kind of quantum engine with ultracold atoms

6. Trade rift today, cryogenic tech yesterday

7. What keeps the red queen running?

8. A limit of ‘show, don’t tell’

On the subject of belief, I’m instinctively drawn to logical systems that demand consistency, closure, and introspection. And the KD45 system among them exerts a special pull. It consists of the following axioms:

• K (closure): If you believe an implication and you believe the antecedent, then you believe the consequent. E.g. if you believe “if X then Y” and you believe X, then you also believe Y.
• D (consistency): If you believe X, you don’t also believe not-X (i.e. X’s negation).
• 4 (positive introspection): If you believe X, then you also believe that you believe X, i.e. you’re aware of your own beliefs.
• 5 (negative introspection): If you don’t believe X, then you believe that you don’t believe X, i.e. you know what you don’t believe.

Thus, KD45 pictures a believer who never embraces contradictions, who always sees the consequences of what they believe, and who is perfectly aware of their own commitments. It’s the portrait of a mind that’s transparent to itself, free from error in structure, and entirely coherent. There’s something admirable in this picture. In moments of near-perfect clarity, it seems to me to describe the kind of believer I’d like to be.

Yet the attraction itself throws up a paradox. KD45 is appealing precisely because it abstracts away from the conditions in which real human beings actually think. In other words, its consistency is pristine because it’s idealised. It eliminates the compromises, distractions, and biases that animate everyday life. To aspire to KD45 is therefore to aspire to something constantly unattainable: a mind that’s rational at every step, free of contradiction, and immune to the fog of human psychology.

My attraction to KD45 is tempered by an equal admiration for Bayesian belief systems. The Bayesian approach allows for degrees of confidence and recognises that belief is often graded rather than binary. To me, this reflects the world as we encounter it — a realm of incomplete evidence, partial understanding, and evolving perspectives.

I admire Bayesianism because it doesn’t demand that we ignore uncertainty. It compels us to face it directly. Where KD45 insists on consistency, Bayesian thinking insists on responsiveness. I update beliefs not because they were previously incoherent but because new evidence has altered the balance of probabilities. This system thus embodies humility, my admission that no matter how strongly I believe today, tomorrow may bring evidence that forces me to change my mind.

The world, however, isn’t simply uncertain: it’s often contradictory. People hold opposing views, traditions preserve inconsistencies, and institutions are riddled with tensions. This is why I’m also drawn to paraconsistent logics, which allow contradictions to exist without collapsing. If I stick to classical logic, I’ll have to accept everything if I also accept a contradiction. One inconsistency causes the entire system to explode. Paraconsistent theories reject that explosion and instead allow me to live with contradictions without being consumed by them.

This isn’t an endorsement of confusion for its own sake but a recognition that practical thought must often proceed even when the data is messy. I can accept, provisionally, both “this practice is harmful” and “this practice is necessary”, and work through the tension without pretending I can neatly resolve the contradiction in advance. To deny myself this capacity is not to be rational — it’s to risk paralysis.

Finally, if Bayesianism teaches humility and paraconsistency teaches tolerance, the AGM theory of belief revision teaches discipline. Its core idea is that beliefs must be revised when confronted by new evidence, and that there are rational ways of choosing what to retract, what to retain, and what to alter. AGM speaks to me because it bridges the gap between the ideal and the real. It allows me to acknowledge that belief systems can be disrupted by facts while also maintaining that I can manage disruptions in a principled way.

That is to say, I don’t aspire to avoid the shock of revision but to absorb it intelligently.

Taken together, my position isn’t a choice of one system over another. It’s an attempt to weave their virtues together while recognising their limits. KD45 represents the ideal that belief should be consistent, closed under reasoning, and introspectively clear. Bayesianism represents the reality that belief is probabilistic and always open to revision. Paraconsistent logic represents the need to live with contradictions without succumbing to incoherence. AGM represents the discipline of revising beliefs rationally when evidence compels change.

A final point about aspiration itself. To aspire to KD45 isn’t to believe I will ever achieve it. In fact, I acknowledge I’m unlikely to desire complete consistency at every turn. There are cases where contradictions are useful, where I’ll need to tolerate ambiguity, and where the cost of absolute closure is too high. If I deny this, I’ll only end up misrepresenting myself.

However, I’m not going to be complacent either. I believe it’s important to aspire even if what I’m trying to achieve is going to be perpetually out of reach. By holding KD45 as a guiding ideal, I hope to give shape to my desire for rationality even as I expect to deviate from it. The value lies in the direction, not the destination.

Therefore, I state plainly (he said pompously):

• I admire the clarity of KD45 and treat it as the horizon of rational belief
• I embrace the flexibility of Bayesianism as the method of navigating uncertainty
• I acknowledge the need for paraconsistency as the condition of living in a world of contradictions
• I uphold the discipline of AGM belief revision as the art of managing disruption
• I aspire to coherence but accept that my path will involve noise, contradiction, and compromise

In the end, the point isn’t to model myself after one system but to recognise the world demands several. KD45 will always represent the perfection of rational belief but I doubt I’ll ever get there in practice — not because I think I can’t but because I know I will choose not to in many matters. To be rational is not to be pure. It is to balance ideals with realities, to aspire without illusion, and to reason without denying the contradictions of life.

If you drop a pebble into a pond, ripples spread outward in gentle circles. We all know this sight, and it feels natural to call them waves. Now imagine being told that everything — from an electron to an atom to a speck of dust — can also behave like a wave, even though they are made of matter and not water or air. That is the bold claim of quantum mechanics. The waves in this case are not ripples in a material substance. Instead, they are mathematical entities known as wavefunctions.

At first, this sounds like nothing more than fancy maths. But the wavefunction is central to how the quantum world works. It carries the information that tells us where a particle might be found, what momentum it might have, and how it might interact. In place of neat certainties, the quantum world offers a blur of possibilities. The wavefunction is the map of that blur. The peculiar thing is, experiments show that this ‘blur’ behaves as though it is real. Electrons fired through two slits make interference patterns as though each one went through both slits at once. Molecules too large to see under a microscope can act the same way, spreading out in space like waves until they are detected.

So what exactly is a wavefunction, and how should we think about it? That question has haunted physicists since the early 20th century and it remains unsettled to this day.

In classical life, you can say with confidence, “The cricket ball is here, moving at this speed.” If you can’t measure it, that’s your problem, not nature’s. In quantum mechanics, it is not so simple. Until a measurement is made, a particle does not have a definite position in the classical sense. Instead, the wavefunction stretches out and describes a range of possibilities. If the wavefunction is sharply peaked, the particle is most likely near a particular spot. If it is wide, the particle is spread out. Squaring the wavefunction’s magnitude gives the probability distribution you would see in many repeated experiments.

If this sounds abstract, remember that the predictions are tangible. Interference patterns, tunnelling, superpositions, entanglement — all of these quantum phenomena flow from the properties of the wavefunction. It is the script that the universe seems to follow at its smallest scales.

To make sense of this, many physicists use analogies. Some compare the wavefunction to a musical chord. A chord is not just one note but several at once. When you play it, the sound is rich and full. Similarly, a particle’s wavefunction contains many possible positions (or momenta) simultaneously. Only when you press down with measurement do you “pick out” a single note from the chord.

Others have compared it to a weather forecast. Meteorologists don’t say, “It will rain here at exactly 3:07 pm.” They say, “There’s a 60% chance of showers in this region.” The wavefunction is like nature’s own forecast, except it is more fundamental: it is not our ignorance that makes it probabilistic, but the way the universe itself behaves.

Mathematically, the wavefunction is found by solving the Schrödinger equation, which is a central law of quantum physics. This equation describes how the wavefunction changes in time. It is to quantum mechanics what Newton’s second law (F = ma) is to classical mechanics. But unlike Newton’s law, which predicts a single trajectory, the Schrödinger equation predicts the evolving shape of probabilities. For example, it can show how a sharply localised wavefunction naturally spreads over time, just like a drop of ink disperses in water. The difference is that the spreading is not caused by random mixing but by the fundamental rules of the quantum world.

But does that mean the wavefunction is real, like a water wave you can touch, or is it just a clever mathematical fiction?

There are two broad camps. One camp, sometimes called the instrumentalists, argues the wavefunction is only a tool for making predictions. In this view, nothing actually waves in space. The particle is simply somewhere, and the wavefunction is our best way to calculate the odds of finding it. When we measure, we discover the position, and the wavefunction ‘collapses’ because our information has been updated, not because the world itself has changed.

The other camp, the realists, argues that the wavefunction is as real as any energy field. If the mathematics says a particle is spread out across two slits, then until you measure it, the particle really is spread out, occupying both paths in a superposed state. Measurement then forces the possibilities into a single outcome, but before that moment, the wavefunction’s broad reach isn’t just bookkeeping: it’s physical.

This isn’t an idle philosophical spat. It has consequences for how we interpret famous paradoxes like Schrödinger’s cat — supposedly “alive and dead at once until observed” — and for how we understand the limits of quantum mechanics itself. If the wavefunction is real, then perhaps macroscopic objects like cats, tables or even ourselves can exist in superpositions in the right conditions. If it is not real, then quantum mechanics is only a calculating device, and the world remains classical at larger scales.

The ability of a wavefunction to remain spread out is tied to what physicists call coherence. A coherent state is one where the different parts of the wavefunction stay in step with each other, like musicians in an orchestra keeping perfect time. If even a few instruments go off-beat, the harmony collapses into noise. In the same way, when coherence is lost, the wavefunction’s delicate correlations vanish.

Physicists measure this ‘togetherness’ with a parameter called the coherence length. You can think of it as the distance over which the wavefunction’s rhythm remains intact. A laser pointer offers a good everyday example: its light is coherent, so the waves line up across long distances, allowing a sharp red dot to appear even all the way across a lecture hall. By contrast, the light from a torch is incoherent: the waves quickly fall out of step, producing only a fuzzy glow. In the quantum world, a longer coherence length means the particle’s wavefunction can stay spread out and in tune across a larger stretch of space, making the object more thoroughly delocalised.

However, coherence is fragile. The world outside — the air, the light, the random hustle of molecules — constantly disturbs the system. Each poke causes the system to ‘leak’ information, collapsing the wavefunction’s delicate superposition. This process is called decoherence, and it explains why we don’t see cats or chairs spread out in superpositions in daily life. The environment ‘measures’ them constantly, destroying their quantum fuzziness.

One frontier of modern physics is to see how far coherence can be pushed before decoherence wins. For electrons and atoms, the answer is “very far”. Physicists have found their wavefunctions can stretch across micrometres or more. They have also demonstrated coherence with molecules with thousands of atoms, but keeping them coherent has been much more difficult. For larger solid objects, it’s harder still.

Physicists often talk about expanding a wavefunction. What they mean is deliberately increasing the spatial extent of the quantum state, making the fuzziness spread wider, while still keeping it coherent. Imagine a violin string: if it vibrates softly, the motion is narrow; if it vibrates with larger amplitude, it spreads. In quantum mechanics, expansion is more subtle but the analogy holds: you want the wavefunction to cover more ground not through noise or randomness but through genuine quantum uncertainty.

Another way to picture it is as a drop of ink released into clear water. At first, the drop is tight and dark. Over time, it spreads outward, thinning and covering more space. Expanding a quantum wavefunction is like speeding up this spreading process, but with a twist: the cloud must remain coherent. The ink can’t become blotchy or disturbed by outside currents. Instead, it must preserve its smooth, wave-like character, where all parts of the spread remain correlated.

How can this be done? One way is to relax the trap that’s being used to hold the particle in place. In physics, the trap is described by a potential, which is just a way of talking about how strong the forces are that pull the particle back towards the centre. Imagine a ball sitting in a bowl. The shape of the bowl represents the potential. A deep, steep bowl means strong restoring forces, which prevent the ball from moving around. A shallow bowl means the forces are weaker. That is, if you suddenly make the bowl shallower, the ball is less tightly confined and can explore more space. In the quantum picture, reducing the stiffness of the potential is like flattening the bowl, which allows the wavefunction to swell outward. If you later return the bowl to its steep form, you can catch the now-broader state and measure its properties.

The challenge is to do this fast and cleanly, before decoherence destroys the quantum character. And you must measure in ways that reveal quantum behaviour rather than just classical blur.

This brings us to an experiment reported on August 19 in Physical Review Letters, conducted by researchers at ETH Zürich and their collaborators. It seems the researchers have achieved something unprecedented: they prepared a small silica sphere, only about 100 nm across, in a nearly pure quantum state and then expanded its wavefunction beyond the natural zero-point limit. This means they coherently stretched the particle’s quantum fuzziness farther than the smallest quantum wiggle that nature usually allows, while still keeping the state coherent.

To appreciate why this matters, let’s consider the numbers. The zero-point motion of their nanoparticle — the smallest possible movement even at absolute zero — is about 17 picometres (one picometre is a trillionth of a meter). Before expansion, the coherence length was about 21 pm. After the expansion protocol, it reached roughly 73 pm, more than tripling the initial reach and surpassing the ground-state value. For something as massive as a nanoparticle, this is a big step.

The team began by levitating a silica nanoparticle in an optical tweezer, created by a tightly focused laser beam. The particle floated in an ultra-high vacuum at a temperature of just 7 K (-266º C). These conditions reduced outside disturbances to almost nothing.

Next, they cooled the particle’s motion close to its ground state using feedback control. By monitoring its position and applying gentle electrical forces through the surrounding electrodes, they damped its jostling until only a fraction of a quantum of motion remained. At this point, the particle was quiet enough for quantum effects to dominate.

The core step was the two-pulse expansion protocol. First, the researchers switched off the cooling and briefly lowered the trap’s stiffness by reducing the laser power. This allowed the wavefunction to spread. Then, after a carefully timed delay, they applied a second softening pulse. This sequence cancelled out unwanted drifts caused by stray forces while letting the wavefunction expand even further.

Finally, they restored the trap to full strength and measured the particle’s motion by studying how they scattered light. Repeating this process hundreds of times gave them a statistical view of the expanded state.

The results showed that the nanoparticle’s wavefunction expanded far beyond its zero-point motion while still remaining coherent. The coherence length grew more than threefold, reaching 73 ± 34 pm. Per the team, this wasn’t just noisy spread but genuine quantum delocalisation.

More strikingly, the momentum of the nanoparticle had become ‘squeezed’ below its zero-point value. In other words, while uncertainty over the particle’s position increased, that over its momentum decreased, in keeping with Heisenberg’s uncertainty principle. This kind of squeezed state is useful because it’s especially sensitive to feeble external forces.

The data matched theoretical models that considered photon recoil to be the main source of decoherence. Each scattered photon gave the nanoparticle a small kick, and this set a fundamental limit. The experiment confirmed that photon recoil was indeed the bottleneck, not hidden technical noise. The researchers have suggested using dark traps in future — trapping methods that use less light, such as radio-frequency fields — to reduce this recoil. With such tools, the coherence lengths can potentially be expanded to scales comparable to the particle’s size. Imagine a nanoparticle existing in a state that spans its own diameter. That would be a true macroscopic quantum object.

This new study pushes quantum mechanics into a new regime. Thus far, large, solid objects like nanoparticles could be cooled and controlled, but their coherence lengths stayed pinned near the zero-point level. Here, the researchers were able to deliberately increase the coherence length beyond that limit, and in doing so showed that quantum fuzziness can be engineered, not just preserved.

The implications are broad. On the practical side, delocalised nanoparticles could become extremely sensitive force sensors, able to detect faint electric or gravitational forces. On the fundamental side, the ability to hold large objects in coherent, expanded states is a step towards probing whether gravity itself has quantum features. Several theoretical proposals suggest that if two massive objects in superposition can become entangled through their mutual gravity, it would prove gravity must be quantum. To reach that stage, experiments must first learn to create and control delocalised states like this one.

The possibilities for sensing in particular are exciting. Imagine a nanoparticle prepared in a squeezed, delocalised state being used to detect the tug of an unseen mass nearby or to measure an electric field too weak for ordinary instruments. Some physicists have speculated that such systems could help search for exotic particles such as certain dark matter candidates, which might nudge the nanoparticle ever so slightly. The extreme sensitivity arises because a delocalised quantum object is like a feather balanced on a pin: the tiniest push shifts it in measurable ways.

There are also parallels with past breakthroughs. The Laser Interferometer Gravitational-wave Observatories, which detect gravitational waves, rely on manipulating quantum noise in light to reach unprecedented sensitivity. The ETH Zürich experiment has extended the same philosophy into the mechanical world of nanoparticles. Both cases show that pushing deeper into quantum control could yield technologies that were once unimaginable.

But beyond the technologies also lies a more interesting philosophical edge. The experiment strengthens the case that the wavefunction behaves like something real. If it were only an abstract formula, could we stretch it, squeeze it, and measure the changes in line with theory? The fact that researchers can engineer the wavefunction of a many-atom object and watch it respond like a physical entity tilts the balance towards reality. At the least, it shows that the wavefunction is not just a mathematical ghost. It’s a structure that researchers can shape with lasers and measure with detectors.

There are also of course the broader human questions. If nature at its core is described not by certainties but by probabilities, then philosophers must rethink determinism, the idea that everything is fixed in advance. Our everyday world looks predictable only because decoherence hides the fuzziness. But under carefully controlled conditions, that fuzziness comes back into view. Experiments like this remind us that the universe is stranger, and more flexible, than classical common sense would suggest.

The experiment also reminds us that the line between the quantum and classical worlds is not a brick wall but a veil — thin, fragile, and possibly removable in the right conditions. And each time we lift it a little further, we don’t just see strange behaviour: we also glimpse sensors more sensitive than ever, tests of gravity’s quantum nature, and perhaps someday, direct encounters with macroscopic superpositions that will force us to rewrite what we mean by reality.

The announcement that a consortium led by PixxelSpace India will design, build, and operate a constellation of 12 earth-observation satellites marks a sharp shift in how India approaches large space projects. The Indian National Space Promotion and Authorisation Centre (IN-SPACe) awarded the project after a competitive process.

What made headlines was that the winning bid asked for no money from the government. Instead, the group — which includes Piersight Space, SatSure Analytics India, and Dhruva Space — has committed to invest more than Rs 1,200 crore of its own resources over the next four to five years. The constellation will carry a mix of advanced sensors, from multispectral and hyperspectral imagers to synthetic aperture radar, and it will be owned and operated entirely by the private side of the partnership.

PixxelSpace has said the zero-rupee bid is a conscious decision to support the vision of building an advanced earth-observation system for India and the world. The companies have also expressed belief they will recover their investment over time by selling high-value geospatial data and services in India and abroad. IN-SPACe’s chairman has called this a major endorsement of the future of India’s space economy.

Of course the benefits for India are clear. Once operational, the constellation should reduce the country’s reliance on foreign sources of satellite imagery. That will matter in areas like disaster management, agriculture planning, and national security, where delays or restrictions on outside data can have serious consequences. Having multiple companies in the consortium brings together strengths in hardware, analytics, and services, which could create a more complete space industry ecosystem. The phased rollout will also mean technology upgrades can be built in as the system grows, without heavy public spending.

Still, the arrangement raises difficult questions. In practice, this is less a public–private partnership than a joint venture. I assume the state will provide its seal of approval, policy support, and access to launch and ground facilities. If it does share policy support, it will have to explain why that’s vouchsafed for the collaboration isn’t of being expanded to the industry as a whole. I also heard IN-SPACe will ‘collate’ demand within the government for the constellation’s products and help meet them.

Without assuming a fiscal stake, however, the government is left with less leverage to set terms or enforce priorities, especially if the consortium’s commercial goals don’t always align with national needs. It’s worth asking why the government issued an official request-for-proposal if didn’t intend to assume a stake, and whether the Rs-350-crore soft loan IN-SPACe originally offered for the project will still be available, repurposed or quietly withdrawn.

I think the pitch will also test public oversight. IN-SPACe will need stronger technical capacity, legal authority, procedural clarity, and better public communication to monitor compliance without frustrating innovation. Regulations on remote sensing and data-sharing will probably have to be updated to cover a fully commercial system that sells services worldwide. Provisions that guarantee government priority access in emergencies and that protect sensitive imagery will have to be written clearly into law and contracts. Infrastructure access, from integration facilities to launch slots, must be managed transparently to avoid bottlenecks or perceived bias.

The government’s minimal financial involvement saves public money but it also reduces long-term control. If India repeats this model, it should put in place new laws and safeguards that define how sovereignty, security, and public interest are to be protected when critical space assets are run by private companies. Without such steps, the promise of cost-free expansion could instead lead to new dependencies that are even harder to manage in future.

Featured image credit: Carl Wang/Unsplash.

In contemporary India, corporate branding has often aligned itself with nationalist sentiment, adopting imagery such as the tricolour, Sanskrit slogans or references to ancient achievements to evoke cultural pride. Marketing narratives frequently frame consumption as a patriotic act, linking the choice of a product with the nation’s progress or “self-reliance”. This fusion of commercial messaging and nationalist symbolism serves both to capitalise on the prevailing political mood and to present companies as partners in the nationalist project. An advertisement in The Times of India on August 15, which describes the work of nation-building as a “hustle”, is a good example.

I remember in engineering college my class had a small-minded and vindictive professor in our second year of undergraduate studies. He repeatedly picked on one particular classmate to the extent that, as resentment between the two people escalated, the professor’s actions in one arguably innocuous matter resulted in the student being suspended for a semester. He eventually didn’t have the number of credits he needed to graduate and had to spend six more months redoing many of the same classes. Today, this student is a successful researcher in Europe, having gone on to acquire a graduate degree followed by a PhD from some of the best research institutes in the world.

When we were chatting a few years ago about our batch’s decadal reunion that was coming up, we thought it would be a good idea to attend and, there, rub my friend’s success in this professor’s face. We really wanted to do it because we wanted him to know how petty he had been. But as we discussed how we’d orchestrate this moment, it dawned on us that we’d also be signalling that our achievements don’t amount to more than those necessary to snub him, as if to say they have no greater meaning or purpose. We eventually dropped the idea. At the reunion itself, my friend simply ignored the professor.

India may appear today to have progressed well past Winston Churchill’s belief, expressed in the early 1930s, but to advertise as Zomato has is to imply that it remains on our minds and animates the purpose of what we’re trying to do. It is a juvenile and frankly resentful attitude that also hints at a more deep-seated lack of contentment. The advertisement’s achievement of choice is the Chandrayaan 3 mission, its Vikram lander lit dramatically by sunlight and earthlight and photographed by the Pragyan rover. The landing was a significant achievement, but to claim that that above all else describes contemporary India is also to dismiss the evident truth that a functional space organisation and a democracy in distress can coexist within the same borders. One neither carries nor excuses the other.

In fact, it’s possible to argue that ISRO’s success is at least partly a product of the unusual circumstances of its creation and its privileged place in the administrative structure. Founded by a scientist who worked directly with Jawaharlal Nehru — bypassing the bureaucratic hurdles faced by most others — ISRO was placed under the purview of the prime minister, ensuring it received the political attention, resources, and exemptions that are not typically available to other ministries or public enterprises. In this view, ISRO’s achievements are insulated from the broader fortunes of the country and can’t be taken as a reliable proxy for India’s overall ‘success’.

The question here is: to whose words do we pay attention? Obviously not those of Churchill: his prediction is nearly a century old. In fact, as Ramachandra Guha sets out in the prologue of India After Gandhi (which I’m currently rereading), they seem in their particular context to be untempered and provocative.

In the 1940s, with Indian independence manifestly round the corner, Churchill grumbled that he had not becoming the King’s first minister in order to preside over the liquidation of the British Empire. A decade previously he had tried to rebuild a fading political career on the plank of opposing self-government for Indians. After Gandhi’s ‘salt satyagraha’ of 1930 in protest against taxes on salt, the British government began speaking with Indian nationalists about the possibility of granting the colony dominion status. This was vaguely defined, with no timetable set for its realization. Even so, Churchill called the idea ‘not only fantastic in itself but criminally mischievous in its effects’. Since Indians were not fit for self-government, it was necessary to marshal ‘the sober and resolute forces of the British Empire’ to stall any such possibility.

In 1930 and 1931 Churchill delivered numerous speeches designed to work up, in most unsober form, the constituency opposed to independence for India. Speaking to an audience at the City of London in December 1930, he claimed that if the British left the subcontinent, then an ‘army of white janissaries, officered if necessary from Germany, will be hired to secure the armed ascendancy of the Hindu’.

This said, Guha continues later in the prologue:

The forces that divide India are many. … But there are also forces that have kept India together, that have helped transcend or contain the cleavages of class and culture, that — so far, at least — have nullified those many predictions that India would not stay united and not stay democratic. These moderating influences are far less visible. … they have included individuals as well as institutions.

Indeed, reading through the history of independent India, through the 1940s and ’50s filled with hope and ambition, the turmoil of the ’60s and the ’70s, the Emergency, followed by economic downturn, liberalisation, finally to the rise of Hindu nationalism, it has been clear that the work of the “forces that have kept India together” is unceasing. Earlier, the Constitution’s framework, with its guarantees of rights and democratic representation, provided a common political anchor. Regular elections, a free press, and an independent judiciary reinforced faith in the system even as the linguistic reorganisation of states reduced separatist tensions. National institutions such as the armed forces, civil services, and railways fostered a sense of shared identity across disparate regions.

Equally, integrative political movements and leaders — including the All India Kisan Sabha, trade union federations like INTUC and AITUC, the Janata Party coalition of 1977, Akali leaders in Punjab in the post-1984 period, the Mazdoor Kisan Shakti Sangathan, and so on, as well as Lal Bahadur Shastri, Govind Ballabh Pant, C. Rajagopalachari, Vinoba Bhave, Jayaprakash Narayan, C.N. Annadurai, Atal Bihari Vajpayee, and so on — operated despite sharp disagreements largely within constitutional boundaries, sustaining the legitimacy of the Union. Today, however, most of these “forces” are directed at a more cynical cause of disunity: a nationalist ideology that has repeatedly defended itself with deceit, evasion, obfuscation, opportunism, pietism, pretence, subterfuge, vindictiveness, and violence.

In this light, to claim we have “just put in the work, year after year”, as if to suggest India has only been growing from strength to strength, rather than lurching from one crisis to the next and of late becoming a little more balkanised as a result, is plainly disingenuous — and yet entirely in keeping with the alignment of corporate branding with nationalist sentiment, which is designed to create a climate in which criticism of corporate conduct is framed as unpatriotic. When companies wrap themselves in the symbols of the nation and position their products or services as contributions to India’s progress, questioning their practices risks being cast as undermining that progress. This can blunt scrutiny of resource over-extraction, environmental degradation, and exploitative labour practices by accusing dissenters of obstructing development.

Aggressively promoting consumption and consumerism (“fuel your hustle”), which drives profits but also deepens social inequalities in the process, is recast as participating in the patriotic project of economic growth. When corporate campaigns subtly or explicitly endorse certain political agendas, their association with national pride can normalise those positions and marginalise alternative views. In this way, the fusion of commerce and nationalism builds market share while fostering a superficial sense of national harmony, even as it sidelines debates on inequality, exclusion, and the varied experiences of different communities within the nation.

In conventional ‘macroscopic’ engines like the ones that guzzle fossil fuels to power cars and motorcycles, the fuels are set ablaze to release heat, which is converted to mechanical energy and transferred to the vehicle’s moving parts. In order to perform these functions over and over in a continuous manner, the engine cycles through four repeating steps. There are different kinds of cycles depending on the engine’s design and needs. A common example is the Otto cycle, where the engine’s four steps are: 

1. Adiabatic compression: The piston compresses the air-fuel mixture, increasing its pressure and temperature without exchanging heat with the surroundings

2. Constant volume heat addition: At the piston’s top position, a spark plug ignites the fuel-air mixture, rapidly increasing pressure and temperature while the volume remains constant

3. Adiabatic expansion: The high-pressure gas pushes the piston down, doing work on the piston, which powers the engine

4. Constant volume heat rejection: At the bottom of the piston stroke, heat is expelled from the gas at constant volume as the engine prepares to clear the exhaust gases

So the engine goes 1-2-3-4-1-2-3-4 and so on. This is useful. If you plot the pressure and volume of the fuel-air mixture in the engine on two axes of a graph, you’ll see that at the end of the ‘constant volume heat rejection’ step (no. 4), the mixture is in the same state as it is at the start of the adiabatic compression step (no. 1). The work that the engine does on the vehicle is equal to the difference between the work done during the expansion and compression steps. Engines are designed to meet the cyclical requirement while increasing the amount of work it does for a given fuel and vehicle design.

It’s easy to understand the value of machines like this. They’re the reason we have vehicles that we can drive in different ways using our hands, legs, and our senses and in relative comfort. As long as we refill the fuel tank once in a while, engines can repeatedly perform mechanical work using their fuel combustion cycles. It’s understandable then why scientists have been trying to build quantum engines. While conventional engines use classical physics to operate, quantum engines are machines that use the ideas of quantum physics. For now, however, these machines are futuristic because scientists have found that they don’t understand the working principles of quantum engines well enough. University of Kaiserslautern-Landau professor Artur Widera told me the following in September 2023 after he and his team published a paper reporting that they had developed a new kind of quantum engine:

Just observing the development and miniaturisation of engines from macroscopic scales to biological machines and further potentially to single- or few-atom engines, it becomes clear that for few particles close to the quantum regime, thermodynamics as we use in classical life will not be sufficient to understand processes or devices. In fact, quantum thermodynamics is just emerging, and some aspects of how to describe the thermodynamical aspects of quantum processes are even theoretically not fully understood.

This said, recent advances in ultracold atomic physics have allowed physicists to control substances called quantum gases in the so-called low-dimensional regimes, laying the ground for them to realise and study quantum engines. Two recent studies exemplify this progress: the study by Widera et al. in 2023 and a new theoretical study reported in Physical Review E. Both studies have explored engines based on ultracold quantum gases but  have approached the concept of quantum energy conversion from complementary perspectives.

The Physical Review E work investigated a ‘quantum thermochemical engine’ operating with a trapped one-dimensional (1D) Bose gas in the quasicondensate regime as the working fluid — just like the fuel-air mixture in in the internal combustion engine of a petrol-powered car. A Bose gas is a quantum system that consists of subatomic particles called bosons. The ‘1D’ simply means they are limited to moving back and forth on a straight line, i.e. a single spatial dimension. This restriction dramatically changes the bosons’ physical and quantum properties.

According to the paper’s single author, University of Queensland theoretical physicist Vijit Nautiyal, the resulting engine can operate on an Otto cycle where the compression and expansion steps — which dictate the work the engine can do — are implemented by tuning how strongly the bosons interact, instead of changing the volume as in a classical engine. In order to do this, the quantum engine needs to exchange not heat with its surroundings but particles. That is, the particles flow from a hot reservoir to the working boson gas, allowing the engine to perform net work.

Nautiyal’s study focused on the engine’s performance in two regimes: one where the strength of interaction between bosons was suddenly quenched in order to maximise the engine’s power at the cost of its efficiency, and another where the quantum engine operates at maximum efficiency but produces negligible power. Nautiyal has reported doing this using advanced numerical simulations.

The simulations showed that if the engine only used heat but didn’t absorb particles from the hot reservoir, it couldn’t really produce useful energy at a finite temperatures. This was because of complicated quantum effects and uneven density in the boson gas. But when the engine was allowed to gain or lose particles from/to the reservoir, it got the extra energy it needed to work properly. Surprisingly, this particle exchange allowed the engine operate very efficiently, even when it ran fast. Usually, engines have to choose between going fast and losing efficiency or go slow and being more efficient. The particle exchange allowed Nautiyal’s quantum thermochemical engine avoid that trade-off. Letting more particles flow in and out also made the engine produce more energy and be even more efficient.

Finally, unlike regular engines where higher temperature usually means better efficiency, increasing the temperature of the quantum thermochemical engine too much actually lowered its efficiency, speaking to the important role chemical work played in this engine design.

In contrast, the 2023 experimental study — which I wrote about in The Hindu — realised a quantum engine that, instead of relying on conventional heating and cooling with thermal reservoirs, operated by cycling a gas of particles between two quantum states, a Bose-Einstein condensate and a Fermi gas. The process was driven by adiabatic changes (i.e. changes that happen while keeping the entropy fixed) that converted the fundamental difference in total energy distribution arising from the two states into usable work. The experiment demonstrated that this energy difference, called the Pauli energy, constituted a significant resource for thermodynamic cycles.

The theoretical 2025 paper and the experimental 2023 work are intimately connected as complementary explorations of quantum engine operation using ultracold atomic gases. Both have taken advantage of the unique quantum effects accessible in such systems while focusing on distinct energy resources and operational principles.

The 2025 work emphasised the role of chemical work arising from particle exchange in a one-dimensional Bose gas, exploring the balance of efficiency and power in finite-time quantum thermochemical engines. It also provided detailed computational frameworks to understand and optimise these engines. Likewise, the 2023 experiment physically realised a related but conceptually different mechanism: the movement of lithium atoms between two states and converting their Pauli energy to work. This approach highlighted how the fundamental differences between the two states could be a direct energy source, rather than conventional heat baths, and one operating with little to no production of entropy.

Together, these studies broaden the scope of quantum engines beyond traditional heat-based cycles by demonstrating the usefulness of intrinsically quantum energy forms such as chemical work and Pauli energy. Such microscopic ‘machines’ also herald a new class of engines that harness the fundamental laws of quantum physics to convert energy between different forms more efficiently than the best conventional engines can manage with classical physics.

Physics World asked Nautiyal about the potential applications of his work:

… Nautiyal referred to “quantum steampunk”. This term, which was coined by the physicist Nicole Yunger Halpern at the US National Institute of Standards and Technology and the University of Maryland, encapsulates the idea that as quantum technologies advance, the field of quantum thermodynamics must also advance in order to make such technologies more efficient. A similar principle, Nautiyal explains, applies to smartphones: “The processor can be made more powerful, but the benefits cannot be appreciated without an efficient battery to meet the increased power demands.” Conducting research on quantum engines and quantum thermodynamics is thus a way to optimize quantum technologies.

US President Donald Trump recently imposed substantial tariffs on Indian goods, explicitly in response to India’s continued purchase of Russian oil during the ongoing Ukraine conflict. These penalties, reaching an unprecedented cumulative rate of 50% on targeted Indian exports, have been described by Trump as a response to what his administration has called an “unusual and extraordinary threat” posed by India’s trade relations with Russia. The official rationale for these measures centres on national security and foreign policy priorities and their design is to coerce India into aligning with US policy goals vis-à-vis the Russia-Ukraine war.

The enforcement of these tariffs is notable among other things for its selectivity. While India faces acute economic repercussions, other major importers of Russian oil such as China and Turkey have thus far not been subjected to equivalent sanctions. The impact is also likely to be immediate and severe since almost half of Indian exports to the US, which is in fact India’s most significant export market, now encounter sharply higher costs, threatening widespread disruption in sectors such as textiles, automobile parts, pharmaceuticals, and electronics. Thus the tariffs have provoked a strong diplomatic response from the Government of India, which has characterised the US’s actions as “unfair, unjustified, and unreasonable,” while also asserting its primary responsibility to protect the country’s energy security.

This fracas is reminiscent of US-India relations in the early 1990s regarding the former’s denial of cryogenic engine technology. In this period, the US government actively intervened to block the transfer of cryogenic rocket engines and associated technologies from Russia’s Glavkosmos to ISRO by invoking the Missile Technology Control Regime (MTCR) as justification. The MTCR was established in 1987 and was intended to prevent the proliferation of missile delivery systems capable of carrying weapons of mass destruction. In 1992, citing non-proliferation concerns, the US imposed sanctions on both ISRO and Glavkosmos, effectively stalling a deal that would have allowed India to acquire not only fully assembled engines but also the vital expertise for indigenous production in a much shorter timeframe than what transpired.

The stated US concern was that cryogenic technology could potentially be adapted for intercontinental ballistic missiles (ICBMs). However experts had been clear that cryogenic engines are unsuitable for ICBMs because they’re complex, difficult to operate, and can’t be deployed on short notice. In fact, critics as well as historical analyses that followed later have said that the US’s strategic objective was less concerned with preventing missile proliferation and more with restricting advances in India’s ability to launch heavy satellites, thus protecting American and allied commercial and strategic interests in the global space sector.

The response in both eras, economic plus technological coercion, suggests a pattern of American policy: punitive action when India’s sovereign decisions diverge from perceived US security or geoeconomic imperatives. The explicit justifications have also shifted from non-proliferation in the 1990s to support for Ukraine in the present, yet in both cases the US has singled India our for selective enforcement while comparable actions by other states have been allowed to proceed largely unchallenged.

Thus, both actions have produced parallel outcomes. India faced immediate setbacks: export disruptions today; delays in its space launch programme three decades ago. There is an opportunity however. The technology denial in the 1990s catalysed an ambitious indigenous cryogenic engine programme, culminating in landmark achievements for ISRO in the following decades. Similarly, the current trade rift could accelerate India’s efforts to diversify its partnerships and supply chains if it proactively forges strategic trade agreements with emerging and established economies, invests in advanced domestic manufacturing capabilities, incentivises innovation across critical sectors, and fortifies logistical infrastructure.

Diplomatically, however, each episode has strained US-India relations even as their mutual interests have at other times fostered rapprochement. Whenever India’s independent strategic choices appear to challenge core US interests, Washington has thus far used the levers of market access and technology transfers as the means of compulsion. But history suggests that these efforts, rather than yield compliance, could prompt adaptive strategies, whether through indigenous technology development or by recalibrating diplomatic and economic alignments.

Featured image: I don’t know which rocket that is. Credit: Perplexity AI.

AI-generated definition based on ‘Quantitative and analytical tools to analyze the spatiotemporal population dynamics of microbial consortia’, Current Opinion in Biotechnology, August 2022:

The Red Queen hypothesis refers to the idea that a constant rate of extinction persists in a community, independent of the duration of a species’ existence, driven by interspecies relationships where beneficial mutations in one species can negatively impact others.

Encyclopedia of Ecology (second ed.), 2008:

The term is derived from Lewis Carroll’s Through the Looking Glass, where the Red Queen informs Alice that “here, you see, it takes all the running you can do to keep in the same place.” Thus, with organisms, it may require multitudes of evolutionary adjustments just to keep from going extinct.

The Red Queen hypothesis serves as a primary explanation for the evolution of sexual reproduction. As parasites (or other selective agents) become specialized on common host genotypes, frequency-dependent selection favors sexual reproduction (i.e., recombination) in host populations (which produces novel genotypes, increasing the rate of adaptation). The Red Queen hypothesis also describes how coevolution can produce extinction probabilities that are relatively constant over millions of years, which is consistent with much of the fossil record.

Also read: ‘Sexual reproduction as an adaptation to resist parasites (a review).’, Proceedings of the National Academy of Sciences, May 1, 1990.

~

In nature, scientists have found that even very similar strains of bacteria constantly appear and disappear even when their environment doesn’t seem to change much. This is called continual turnover. In a new study in PRX Life, Aditya Mahadevan and Daniel Fisher of Stanford University make sense of how this ongoing change happens, even without big differences between species or dramatic changes in the environment. Their jumping-off point is the red queen hypothesis.

While the hypothesis has usually been used to talk about ‘arms races’, like between hosts and parasites, the new study asked: can continuous red queen evolution also happen in communities where different species or strains overlap a lot in what they do and where there aren’t obvious teams fighting each other?

Mahadevan and Fisher built mathematical models to mimic how communities of microbes evolve over time. These models allowed the duo to simulate what would happen if a population started with just one microbial strain and over time new strains appeared due to random changes in their genes (i.e. mutations). Some of these new strains could invade other species’ resources and survive while others are forced to extinction.

The models focused especially on ecological interactions, meaning how strains or species affected each other’s survival based on how they competed for the same food.

When they ran the models, the duo found that even when there were no clear teams (like host v. parasite), communities could enter a red queen phase. The overall number of coexisting strains stayed roughly constant, but which strains were present keeps changing, like a continuous evolutionary game of musical chairs.

The continual turnover happened most robustly when strains interacted in a non-reciprocal way. As ICTS biological physicist Akshit Goyal put it in Physics:

… almost every attempt to model evolving ecological communities ran into the same problem: One organism, dubbed a Darwinian monster, evolves to be good at everything, killing diversity and collapsing the community. Theorists circumvented this outcome by imposing metabolic trade-offs, essentially declaring that no species could excel at everything. But that approach felt like cheating because the trade-offs in the models needed to be unreasonably strict. Moreover, for mathematical convenience, previous models assumed that ecological interactions between species were reciprocal: Species A affects species B in exactly the same way that B affects A. However, when interactions are reciprocal, community evolution ends up resembling the misleading fixed fitness landscape. Evolution is fast at first but eventually slows down and stops instead of going on endlessly.

Mahadevan and Fisher solved this puzzle by focusing on a previously neglected but ubiquitous aspect of ecological interactions: nonreciprocity. This feature occurs when the way species A affects species B differs from the way B affects A—for example, when two species compete for the same nutrient, but the competition harms one species more than the other

Next, despite the continual turnover, there was a cap on the number of strains that could coexist. This depended on the number of different resources available and how strains interacted, but as new strains invaded others, some old ones had to go extinct, keeping diversity within limits.

If some strains started off much better (i.e. with higher fitness), over time the evolving competition narrowed these differences and only strains with similar overall abilities managed to stick around.

Finally, if the system got close to being perfectly reciprocal, the dynamics could shift to an oligarch phase in which a few strains dominated most of the population and continual turnover slowed considerably.

Taken together, the study’s main conclusion is that there doesn’t need to be a constant or elaborate ‘arms race’ between predator and prey or dramatic environmental changes to keep evolution going in bacterial communities. Such evolution can arise naturally when species or strains interact asymmetrically as they compete for resources.

Featured image: “Now, here, you see, it takes all the running you can do, to keep in the same place.” Credit: Public domain.

The virtue of ‘show, don’t tell’ in writing, including in journalism, lies in its power to create a more vivid, immersive, and emotionally engaging reading experience. Instead of simply providing information or summarising events, the technique encourages writers to use evocative imagery, action, dialogue, and sensory details to invite readers into the world of the story.

The idea is that once they’re in there, they’ll be able to do a lot of the task of engaging for you.

However, perhaps this depends on the world the reader is being invited to enter.

There’s an episode in season 10 of ‘Friends’ where a palaeontologist tells Joey she doesn’t own a TV. Joey is confused and asks, “Then what’s all your furniture pointed at?”

Most of the (textual) journalism of physics I’m seeing these days frames narratives around the application of some discovery or concept. For example, here’s the last paragraph of one of the top articles on Physics World today:

The trio hopes that its technique will help us understand polaron behaviours. “The method we developed could also help study strong interactions between light and matter, or even provide the blueprint to efficiently add up Feynman diagrams in entirely different physical theories,” Bernardi says. In turn, it could help to provide deeper insights into a variety of effects where polarons contribute – including electrical transport, spectroscopy, and superconductivity.

I’m not sure if there’s something implicitly bad about this framing but I do believe it gives the impression that the research is in pursuit of those applications, which in my view is often misguided. Scientific research is incremental and theories and data often takes many turns before they can be stitched together cleanly enough for a technological application in the real world.

Yet I’m also aware that, just like pointing all your furniture at the TV can simplify your decisions about arranging your house, drafting narratives in order to convey the relevance of some research for specific applications can help hold readers’ attention better. Yes, this is a populist approach to the extent that it panders to what readers know they want rather than what they may not know, but it’s useful — especially when the communicator or journalist is pressed for time and/or doesn’t have the mental bandwidth to craft a thoughtful narrative.

But this narrative choice may also imply a partial triumph of “tell, don’t show” over “show, don’t tell”. This is because the narrative has an incentive to restrict itself to communicating whatever physics is required to describe the technology and still be considered complete rather than wade into waters that will potentially complicate the narrative.

A closely related issue here is that a lot of physics worth knowing about — if for no reason other than that they’re windows into scientists’ spirit and ingenuity — is quite involved. (It doesn’t help that it’s also mostly mathematical.) The concepts are simply impossible to show, at least not without the liberal use of metaphors and, inevitably, some oversimplification.

Of course, it’s not possible to compare a physics news piece in Physics World with that in The Hindu: the former will be able to show more by telling itself because its target audience is physicists and other scientists, and they will see more detail in the word “polaron” than readers of The Hindu can be expected to. But even if The Hindu’s readers need more showing, I can’t show them the physics without expecting they will be interested in complicated theoretical ideas.

In fact, I’ll be hard-pressed to be a better communicator than if I resorted to telling. Thus my lesson is that ‘show, don’t tell’ isn’t always a virtue. Sometimes what you show can bore or maybe scare readers off, and for reasons that have nothing to do with your skills as a communicator. Obviously the point isn’t to condescend readers here. Instead, we need to acknowledge that telling is virtuous in its own right, and in the proper context may be the more engaging way to communicate science.

I’ve been on the lookout for a scripting language which can be neatly embedded into Hare programs. Perhaps the obvious candidate is Lua – but I’m not particularly enthusiastic about it. When I was evaluating the landscape of tools which are “like Lua, but not Lua”, I found an interesting contender: Wren.

I found that Wren punches far above its weight for such a simple language. It’s object oriented, which, you know, take it or leave it depending on your use-case, but it’s very straightforwardly interesting for what it is. I found a few things to complain about, of course – its scope rules are silly, the C API has some odd limitations here and there, and in my opinion the “standard library” provided by wren CLI is poorly designed. But, surprisingly, my list of complaints more or less ends there, and I was excited to build a nice interface to it from Hare.

The result is hare-wren. Check it out!

The basic Wren C API is relatively straightforwardly exposed to Hare via the wren module, though I elected to mold it into a more idiomatic Hare interface rather than expose the C API directly to Hare. You can use it something like this:

use wren;

export fn main() void = {
	const vm = wren::new(wren::stdio_config);
	defer wren::destroy(vm);
	wren::interpret(vm, "main", `
		System.print("Hello world!")
	`)!;
};

$ hare run -lc main.ha
Hello world!

Calling Hare from Wren and vice-versa is also possible with hare-wren, of course. Here’s another example:

use fmt;
use wren;

export fn main() void = {
	let config = *wren::stdio_config;
	config.bind_foreign_method = &bind_foreign_method;

	const vm = wren::new(&config);
	defer wren::destroy(vm);

	wren::interpret(vm, "main", `
	class Example {
		foreign static greet(user)
	}

	System.print(Example.greet("Harriet"))
	`)!;
};

fn bind_foreign_method(
	vm: *wren::vm,
	module: str,
	class_name: str,
	is_static: bool,
	signature: str,
) nullable *wren::foreign_method_fn = {
	const is_valid = class_name == "Example" &&
		signature == "greet(_)" && is_static;
	if (!is_valid) {
		return null;
	};
	return &greet_user;
};

fn greet_user(vm: *wren::vm) void = {
	const user = wren::get_string(vm, 1)!;
	const greeting = fmt::asprintf("Hello, {}!", user)!;
	defer free(greeting);
	wren::set_string(vm, 0, greeting);
};

$ hare run -lc main.ha
Hello, Harriet!

In addition to exposing the basic Wren virtual machine to Hare, hare-wren has an optional submodule, wren::api, which implements a simple async runtime based on hare-ev and a modest “standard” library, much like Wren CLI. I felt that the Wren CLI libraries had a lot of room for improvement, so I made the call to implement a standard library which is only somewhat compatible with Wren CLI.

On top of the async runtime, Hare’s wren::api runtime provides some basic features for reading and writing files, querying the process arguments and environment, etc. It’s not much but it is, perhaps, an interesting place to begin building out something a bit more interesting. A simple module loader is also included, which introduces some conventions for installing third-party Wren modules that may be of use for future projects to add new libraries and such.

Much like wren-cli, hare-wren also provides the hwren command, which makes this runtime, standard library, and module loader conveniently available from the command line. It does not, however, support a REPL at the moment.

I hope you find it interesting! I have a few projects down the line which might take advantage of hare-wren, and it would be nice to expand the wren::api library a bit more as well. If you have a Hare project which would benefit from embedding Wren, please let me know – and consider sending some patches to improve it!

Last week, Armin and I worked together on the latest release of Himitsu, a “secret storage manager” for Linux. I haven’t blogged about Himitsu since I announced it three years ago, and I thought it would be nice to give you a closer look at the latest release, both for users eager to see the latest features and for those who haven’t been following along.¹

A brief introduction: Himitsu is like a password manager, but more general: it stores any kind of secret in its database, including passwords but also SSH keys, credit card numbers, your full disk encryption key, answers to those annoying “security questions” your bank obliged you to fill in, and so on. It can also enrich your secrets with arbitrary metadata, so instead of just storing, say, your IMAP password, it can also store the host, port, TLS configuration, and username, storing the complete information necessary to establish an IMAP session.

Another important detail: Himitsu is written in Hare and depends on Hare’s native implementations of cryptographic primitives – neither Himitsu nor the cryptography implementation it depends on have been independently audited.

So, what new and exciting features does Himitsu 0.9 bring to the table? Let me summarize the highlights for you.

A new prompter

The face of Himitsu is the prompter. The core Himitsu daemon has no user interface and only communicates with the outside world through its IPC protocols. One of those protocols is the “prompter”, which Himitsu uses to communicate with the user, to ask you for consent to use your secret keys, to enter the master password, and so on. The prompter is decoupled from the daemon so that it is easy to substitute with different versions which accommodate different use-cases, for example by integrating the prompter more deeply into a desktop environment or to build one that fits better on a touch screen UI like a phone.

But, in practice, given Himitsu’s still-narrow adoption, most people use the GTK+ prompter developed upstream. Until recently, the prompter was written in Python for GTK+ 3, and it was a bit janky and stale. The new hiprompt-gtk changes that, replacing it with a new GTK4 prompter implemented in Hare.

I’m excited to share this one with you – it was personally my main contribution to this release. The prompter is based on Alexey Yerin’s hare-gi, which is a (currently only prototype-quality) code generator which processes GObject Introspection documents into Hare modules that bind to libraries like GTK+. The prompter uses Adwaita for its aesthetic and controls and GTK layer shell for smoother integration on supported Wayland compositors like Sway.

Secret service integration

Armin has been hard at work on a new package, himitsu-secret-service, which provides the long-awaited support for integrating Himitsu with the dbus Secret Service API used by many Linux applications to manage secret keys. This makes it possible for Himitsu to be used as a secure replacement for, say, gnome-keyring.

Editing secret keys

Prior to this release, the only way to edit a secret key was to remove it and re-add it with the desired edits applied manually. This was a tedious and error-prone process, especially when bulk-editing keys. This release includes some work from Armin to improve the process, by adding a “change” request to the IPC protocol and implementing it in the command line hiq client.

For example, if you changed your email address, you could update all of your logins like so:

$ hiq -c email=newemail@example.org email=oldemail@example.org

Don’t worry about typos or mistakes – the new prompter will give you a summary of the changes for your approval before the changes are applied.

You can also do more complex edits with the -e flag – check out the hiq(1) man page for details.

Secret reuse notifications

Since version 0.8, Himitsu has supported “remembering” your choice, for supported clients, to consent to the use of your secrets. This allows you, for example, to remember that you agreed for the SSH agent to use your SSH keys for an hour, or for the duration or your login session, etc. Version 0.9 adds a minor improvement to this feature – you can add a command to himitsu.ini, such as notify-send, which will be executed whenever a client takes advantage of this “remembered” consent, so that you can be notified whenever your secrets are used again, ensuring that any unexpected use of your secrets will get your attention.

himitsu-firefox improvements

There are also some minor improvements landed for himitsu-firefox that I’d like to note. tiosgz sent us a nice patch which makes the identification of login fields in forms more reliable – thanks! And I’ve added a couple of useful programs, himitsu-firefox-import and himitsu-firefox-export, which will help you move logins between Himitsu and Firefox’s native password manager, should that be useful to you.

And the rest

Check out the changelog for the rest of the improvements. Enjoy!

I screwed something up the other day and figured it had enough meat on its bones to turn into a story. So, okay, here we go.

For a while now, I've been doing some "wrapping" of return values in my code. It's C++ stuff, but it's something that's been inspired by what some of my friends have been doing with Rust. It's where instead of just returning a string from a function that might fail, I return something else that enforces some checks.

Basically, I'm not allowed to call .value() or .error() on it until I've checked to see if it succeeded or not. If I do one of those things out of sequence, it will hit a CHECK and will nuke the program. This normally catches me fat-fingering something in development and never ships out.

Some of this code looks like this:

auto ua = req.util->UserAgent();

if (ua()) {
  req.set_user_agent(ua.value());
}

In that case, it's wrapping a string. It's wrapped because it can fail! Sometimes there's no value available because someone decided they didn't want to send that header in their request for some strange reason. I don't "do" "sentinel values", nulls, or other stuff like that, because I have my little "result string" thing going on here.

Easy enough, right? Well, I found myself making some mistakes when dealing with a series of calls to things that could pass or fail which worked in a similar fashion. They don't have a .value() but they can have an .error() and they need to be checked.

Sometimes, in my editor, I'd do a "delete 2-3 lines, then undelete twice, then twiddle the second set" thing for a spot where I had to make two very similar calls in a row. It might look like this:

auto ai = app_->Init();

if (!ai()) {
  log_something("blahblah failed: " + ai.error());
  // return something or other...
}

auto ni = net_->Init();

if (!ni()) {
  log_something("no shrubbery: " + ai.error());
  // return something blahblah...
}

But, do you see the bug? I'm using ai.error in the second spot instead of ni.error. ai is still available since it exists from that "auto ai = ..." line to the bottom of the block, and there's no way to say "hey, compiler, throw a fit if anyone looks at this thing after this point".

I'd have to do something odd like sticking the whole mess into another { ... } block just so ai would disappear, and while that would work, it also gets ugly.

Not too long ago, I came up with something else based on some newer syntax that can be wrangled up in C++. It's apparently called "if scope", where you can define a variable in the course of doing a branch on some condition, and then it only exists right there.

It looks like this:

if (auto ai = app_->Init(); !ai()) {
  log_something("blahblah failed: " + ai.error());
  // return something or other...
}

It looks a little awkward at first, but it's pretty close to the original code, and it also has a nifty side-effect: "ai" doesn't live beyond that one tiny little block where I report the error and then bail out.

With that in place, you can't make that "ai instead of ni" mistake from before. That's a clear win and I've been converting my code to it in chunks all over the place.

A couple of days ago, I did a change like that on some user-agent handling code, but screwed up and did it like this:

if (auto ua = req.util->UserAgent(); !ua()) {
  req.set_user_agent(ua.value());
}

That's basically saying: "if they *didn't* send a user-agent, then add its value to the request we're building up". Now, had that code ever run, it would have CHECKed and blown up right there, since calling .value() after it's returned false on the pass-fail check is not allowed. But, nobody is doing that at the moment, so it never happened.

The other effect it had was that it never added the user-agent value to the outgoing request when clients _did_ present one, and that's been the case all of the time.

So, a few days ago, someone reported that their feed score reporting page said that they apparently didn't send that header with their requests but they're sure that they did. They started chasing a bug on their side. I went "hmmm, oh no...", looked, and found it.

It's supposed to look like this:

if (auto ua = req.util->UserAgent(); ua()) {
  req.set_user_agent(ua.value());
}

So, why did I put the ! in front? Easy: most of the time, I'm handling errors with this stuff and bailing out by returning early. This is one of those relatively infrequent inverted situations where I want the value and jam it in there only if it exists.

It was a quick fix, but the damage was done. A few hundred rows in the database table picked up NULLs for that column while the bad version was deployed on the web server.

So now let's talk about what I'm doing about it. One thing I've been doing all this time when logging hits to the feed score project is that I also log the git commit hash from the source tree at the time it was built by my automatic processes. It's just one more column in the table, and it changes any time I push a new binary out there.

With that, it was possible to see that only this one build had the bug, and I didn't need to fix any other rows. The other rows without UA data are that way because some goofball program is actually not sending the header for whatever reason.

Next, I changed the report page to add a colorful (and very strange-looking) "fingerprint" of the build hash which had been logged all along but not exposed to users previously. Every row in the results table now sports an extra column which has a bunch of wacky Unicode box-drawing characters around U+2580 all in different colors. I use the build hash to set the colors and pick which of the 30 or so weird characters can go in each spot.

If this technique sounds familiar, you might be thinking of a post of mine from August 2011 where it was using MD5 sums of argv strings to render color bars.

This time around, since other people are the intended audience, I can't rely on full-color vision, so that's why there's also a mash-up of characters. Even if all you can see are shades of grey, you can still see the groupings at a glance.

So now, whenever something seems strange, the fsr users can see if I changed something and maybe save themselves from chasing a bug that's on my end and not theirs.

To those people: sorry! I still have to sit down and manually replace the data in the table from the actual web server logs from that time period. It'll fill in and then it'll look like nothing bad ever happened.

Until then, well, just know that one particular version blob has my "brown paper bag" bug associated with it.

Bugs, bugs, bugs...

And finally, yes, a test on this code would have caught this pre-shipping. Obviously. You saw the part where I'm doing this for free, right?

Sometimes, you just need to write down what you're willing to do and what you're not. I have a short tale about doing that at a job, and then bringing that same line of thinking forward to my current concerns.

I used to be on a team that was responsible for the care and feeding of a great many Linux boxes which together constituted the "web tier" for a giant social network. You know, the one with all of the cat pictures... and later the whole genocide thing and enabling fascism. Yeah, them.

Anyway, given that we had a six-digit number of machines that was steadily climbing and people were always experimenting with stuff on them, with them, and under them, it was necessary to apply some balance to keep things from breaking too often. There was a fine line between "everything's broken" and "it's impossible to roll anything out so the business dies".

At some point, I realized that if I wrote a wiki page and documented the things that we were willing to support, I could wait about six months and then it would be like it had always been there. Enough people went through the revolving doors of that place such that six months' worth of employee turnover was sufficient to make it look like a whole other company. All I had to do was write it, wait a bit, then start citing it when needed.

One thing that used to happen is that our "hostprefix" - that is, the first few letters of the hostname - was a dumping ground. It was kind of the default place for testing stuff, trying things, or putting machines when you were "done" with them, whatever that meant. We had picked up all kinds of broken hardware that wasn't really ready to serve production traffic. Sometimes this was developmental hardware that was missing certain key aspects that we depended on, like having several hundred gigs of disk space to have a few days of local logging on board.

My page became a list of things that wouldn't be particularly surprising to anyone who had been paying attention. It must be a box with at least this much memory, this much disk space, this much network bandwidth, this version of CentOS, with the company production Chef environment installed and running properly... and it went on and on like this. It was fairly clear that merely having a thing installed wasn't enough. It had to be running to completion. That means successful runs!

I wish I had saved a copy of it, since it would be interesting to look back on it after over a decade to see what all I had noted back then. Oh well.

Anyway, after it had aged a bit, I was able to point people at it and go "this is what we will do and this is what we will reject". While it wasn't a hard-and-fast ruleset, it was pretty clear about our expectations. Or, well, let's face it - *my* expectations. I had some strong opinions about what's worth supporting and what's just plain broken and a waste of time.

One section of the page had to do with "non-compliant host handling". I forget the specifics (again, operating on memory here...), but it probably included things like "we disable it and it stops receiving production traffic", "it gets reinstalled to remove out-of-spec customizations", and "it is removed from the hostprefix entirely". That last one was mostly for hardware mismatches, since there was no amount of "reinstall to remove your bullshit" that would fix a lack of disk space (or whatever).

One near-quote from that page did escape into the outside world. It has to do with the "non-compliant host" actions:

"Note: any of these many happen *without prior notification* to experiment owners in the interest of keeping the site healthy. Drain first, investigate second."

"Drain" in this case actually referred to a command that we could run to disable a host in the load balancers so they stopped receiving traffic. When a host is gobbling up traffic and making a mess for users, disable it, THEN figure out what to do about it. Don't make people suffer while you debate what's going to happen with the wayward web server.

Given all this, it shouldn't be particularly surprising that I've finally come up with a list of feed reader behaviors. I wrote it like a bunch of items you might see in one of these big tech company performance reviews. You know the ones that are like "$name consistently delivers foo and bar on time"? Imagine that, but for feed readers.

The idea is that I'll be able to point at it and go "that, right there, see, I'm not being capricious or picking on you in particular... this represents a common problem which has existed since well before you showed up". The items are short and sweet and have unique identifiers so it's possible to point at one and say "do it like that".

I've been sharing this with a few other people who also work in this space and have to deal with lots of traffic from feed reader software. If you're one of those people and want to see it, send me a note.

At some point, I'll open it up to the world and then we'll see what happens with that.

When researching networking devices, I frequently encounter a particular vulnerability: the ability to inject arbitrary options into dnsmasq's config files. These devices often delegate functionality to dnsmasq, and when they allow users to set configuration options, they might perform basic templating to generate configuration files that are then fed to dnsmasq. If the device fails to properly encode user input, it may allow users to insert newline characters and inject arbitrary options into the config file.

Continue reading →

I was reading the latest issue of the journal Science, and a paper mentioned the compound Cr₂Gr₂Te₆. For a moment, I thought my knowledge of the periodic table was slipping, since I couldn't remember the element Gr. It turns out that Gr was supposed to be Ge, germanium, but that raises two issues. First, shouldn't the peer reviewers and proofreaders at a top journal catch this error? But more curiously, it appears that Cr₂Gr₂Te₆ is a mistake that has been copied around several times.

The Science paper [1] states, "Intrinsic ferromagnetism in these materials was discovered in Cr₂Gr₂Te₆ and CrI₃ down to the bilayer and monolayer thickness limit in 2017." I checked the referenced paper [2] and verified that the correct compound is Cr₂Ge₂Te₆, with Ge for germanium.

But in the process, I found more publications that specifically mention the 2017 discovery of intrinsic ferromagnetism in both Cr₂Gr₂Te₆ and CrI₃. A 2021 paper in Nanoscale [3] says, "Since the discovery of intrinsic ferromagnetism in atomically thin Cr₂Gr₂Te₆ and CrI₃ in 2017, research on two-dimensional (2D) magnetic materials has become a highlighted topic." Then, a 2023 book chaper [4] opens with the abstract: "Since the discovery of intrinsic long-range magnetic order in two-dimensional (2D) layered magnets, e.g., Cr₂Gr₂Te₆ and CrI₃ in 2017, [...]"

This illustrates how easy it is for a random phrase to get copied around with nobody checking it. (Earlier, I found a bogus computer definition that has persisted for over 50 years.) To be sure, these could all be independent typos—it's an easy typo to make since Ge and Gr are neighbors on the keyboard and Cr2Gr2 scans better than Cr2Ge2. A few other papers [5, 6, 7] have the same typo, but in different contexts. My bigger concern is that once AI picks up the erroneous formula Cr₂Gr₂Te₆, it will propagate as misinformation forever. I hope that by calling out this error, I can bring an end to it. In any case, if anyone ends up here after a web search, I can at least confirm that there isn't a new element Gr and the real compound is Cr₂Ge₂Te₆, chromium germanium telluride.

References

[1] He, B. et al. (2025) ‘Strain-coupled, crystalline polymer-inorganic interfaces for efficient magnetoelectric sensing’, Science, 389(6760), pp 623-631. (link)

[2] Gong, C. et al. (2017) ‘Discovery of intrinsic ferromagnetism in two-dimensional van der Waals crystals’, Nature, 546(7657), pp. 265–269. (link)

[3] Zhang, S. et al. (2021) ‘Two-dimensional magnetic materials: structures, properties and external controls’, Nanoscale, 13(3), pp. 1398–1424. (link)

[4] Yin, T. (2024) ‘Novel Light-Matter Interactions in 2D Magnets’, in D. Ranjan Sahu (ed.) Modern Permanent Magnets - Fundamentals and Applications. (link)

[5] Zhao, B. et al. (2023) ‘Strong perpendicular anisotropic ferromagnet Fe₃GeTe₂/graphene van der Waals heterostructure’, Journal of Physics D: Applied Physics, 56(9) 094001. (link)

[6] Ren, H. and Lan, M. (2023) ‘Progress and Prospects in Metallic Fe_xGeTe₂ (3≤x≤7) Ferromagnets’, Molecules, 28(21), p. 7244. (link)

[7] Hu, S. et al. (2019) 'Anomalous Hall effect in Cr₂Gr₂Te₆/Pt hybride structure', Taiwan-Japan Joint Workshop on Condensed Matter Physics for Young Researchers, Saga, Japan. (link)

I've been reverse-engineering the Intel 386 processor (from 1985), and I've come across some interesting circuits for the chip's input/output (I/O) pins. Since these pins communicate with the outside world, they face special dangers: static electricity and latchup can destroy the chip, while metastability can cause serious malfunctions. These I/O circuits are completely different from the logic circuits in the 386, and I've come across a previously-undescribed flip-flop circuit, so I'm venturing into uncharted territory. In this article, I take a close look at how the I/O circuitry protects the 386 from the "dragons" that can destroy it.

The photo above shows the die of the 386 under a microscope. The dark, complex patterns arranged in rectangular regions arise from the two layers of metal that connect the circuits on the 386 chip. Not visible are the transistors, formed from silicon and polysilicon and hidden beneath the metal. Around the perimeter of this fingernail-sized silicon die, 141 square bond pads provide the connections between the chip and the outside world; tiny gold bond wires connect the bond pads to the package. Next to each I/O pad, specialized circuitry provides the electrical interface between the chip and the external components while protecting the chip. I've zoomed in on three groups of these bond pads along with the associated I/O circuits. The circuits at the top (for data pins) and the left (for address pins) are completely different from the control pin circuits at the bottom, showing how the circuitry varies with the pin's function.

Static electricity

The first dragon that threatens the 386 is static electricity, able to burn a hole in the chip. MOS transistors are constructed with a thin insulating oxide layer underneath the transistor's gate. In the 386, this fragile, glass-like oxide layer is just 250 nm thick, the thickness of a virus. Static electricity, even a small amount, can blow a hole through this oxide layer and destroy the chip. If you've ever walked across a carpet and felt a spark when you touch a doorknob, you've generated at least 3000 volts of chip-destroying static electricity. Intel recommends an anti-static mat and a grounding wrist strap when installing a processor to avoid the danger of static electricity, also known as Electrostatic Discharge or ESD.1

To reduce the risk of ESD damage, chips have protection diodes and other components in their I/O circuitry. The schematic below shows the circuit for a typical 386 input. The goal is to prevent static discharge from reaching the inverter, where it could destroy the inverter's transistors. The diodes next to the pad provide the first layer of protection; they redirect excess voltage to the +5 rail or ground. Next, the resistor reduces the current that can reach the inverter. The third diode provides a final layer of protection. (One unusual feature of this input—unrelated to ESD—is that the input has a pull-up, which is implemented with a transistor that acts like a 20kΩ resistor.2)

The image below shows how this circuit appears on the die. For this photo, I dissolved the metal layers with acids, stripping the die down to the silicon to make the transistors visible. The diodes and pull-up resistor are implemented with transistors.3 Large grids of transistors form the pad-side diodes, while the third diode is above. The current-limiting protection resistor is implemented with polysilicon, which provides higher resistance than metal wiring. The capacitor is implemented with a plate of polysilicon over silicon, separated by a thin oxide layer. As you can see, the protection circuitry occupies much more area than the inverters that process the signal.

Latchup

The transistors in the 386 are created by doping silicon with impurities to change its properties, creating regions of "N-type" and "P-type" silicon. The 386 chip, like most processors, is built from CMOS technology, so it uses two types of transistors: NMOS and PMOS. The 386 starts from a wafer of N-type silicon and PMOS transistors are formed by doping tiny regions to form P-type silicon embedded in the underlying N-type silicon. NMOS transistors are the opposite, with N-type silicon embedded in P-type silicon. To hold the NMOS transistors, "wells" of P-type silicon are formed, as shown in the cross-section diagram below. Thus, the 386 chip contains complex patterns of P-type and N-type silicon that form its 285,000 transistors.

But something dangerous lurks below the surface, the fire-breathing dragon of latchup waiting to burn up the chip. The problem is that these regions of N-type and P-type silicon form unwanted, "parasitic" transistors underneath the desired transistors. In normal circumstances, these parasitic NPN and PNP transistors are inactive and can be ignored. But if a current flows beneath the surface, through the silicon substrate, it can turn on a parasitic transistor and awaken the dreaded latchup.4 The parasitic transistors form a feedback loop, so if one transistor starts to turn on, it turns on the other transistor, and so forth, until both transistors are fully on, a state called latchup.5 Moreover, the feedback loop will maintain latchup until the chip's power is removed.6 During latchup, the chip's power and ground are shorted through the parasitic transistors, causing high current flow that can destroy the chip by overheating it or even melting bond wires.

Latchup can be triggered in many ways, from power supply overvoltage to radiation, but a chip's I/O pins are the primary risk because signals from the outside world are unpredictable. For instance, suppose a floppy drive is connected to the 386 and the drive sends a signal with a voltage higher than the 386's 5-volt supply. (This could happen due to a voltage surge in the drive, reflection in a signal line, or even connecting a cable.) Current will flow through the 386's protection diodes, the diodes that were described in the previous section.7 If this current flows through the chip's silicon substrate, it can trigger latchup and destroy the processor.

Because of this danger, the 386's I/O pads are designed to prevent latchup. One solution is to block the unwanted currents through the substrate, essentially putting fences around the transistors to keep malicious currents from escaping into the substrate. In the 386, this fence consists of "guard rings" around the I/O transistors and diodes. These rings prevent latchup by blocking unwanted current flow and safely redirecting it to power or ground.

The diagram above shows the double guard rings for a typical I/O pad.8 Separate guard rings protect the NMOS transistors and the PMOS transistors. The NMOS transistors have an inner guard ring of P-type silicon connected to ground (blue) and an outer guard ring of N-type silicon connected to +5 (red). The rings are reversed for the PMOS transistors. The guard rings take up significant space on the die, but this space isn't wasted since the rings protect the chip from latchup.

Metastability

The final dragon is metastability: it (probably) won't destroy the chip, but it can cause serious malfunctions.9 Metastability is a peculiar problem where a digital signal can take an unbounded amount of time to settle into a zero or a one. In other words, the circuit temporarily refuses to act digitally and shows its underlying analog nature.10 Metastability was controversial in the 1960s and the 1970s, with many electrical engineers not believing it existed or considering it irrelevant. Nowadays, metastability is well understood, with special circuits to prevent it, but metastability can never be completely eliminated.

In a processor, everything is synchronized to its clock. While a modern processor has a clock speed of several gigahertz, the 386's clock ran at 12 to 33 megahertz. Inside the processor, signals are carefully organized to change according to the clock—that's why your computer runs faster with a higher clock speed. The problem is that external signals may be independent of the CPU's clock. For instance, a disk drive could send an interrupt to the computer when data is ready, which depends on the timing of the spinning disk. If this interrupt arrives at just the wrong time, it can trigger metastability.

In more detail, processors use flip-flops to hold signals under the control of the clock. An "edge-triggered" flip-flop grabs its input at the moment the clock goes high (the "rising edge") and holds this value until the next clock cycle. Everything is fine if the value is stable when the clock changes: if the input signal switches from low to high before the clock edge, the flip-flop will hold this high value. And if the input signal switches from low to high after the clock edge, the flip-flop will hold the low value, since the input was low at the clock edge. But what happens if the input changes from low to high at the exact time that the clock switches? Usually, the flip-flop will pick either low or high. But very rarely, maybe a few times out of a billion, the flip-flop will hesitate in between, neither low nor high. The flip-flop may take a few nanoseconds before it "decides" on a low or high value, and the value will be intermediate until then.

The photo above illustrates a metastable signal, spending an unpredictable time between zero and one before settling on a value. The situation is similar to a ball balanced on top of a hill, a point of unstable equilibrium.11 The smallest perturbation will knock the ball down one of the two stable positions at the bottom of the hill, but you don't know which way it will go or how long it will take.

Metastability is serious because if a digital signal has a value that is neither 0 nor 1 then downstream circuitry may get confused. For instance, if part of the processor thinks that it received an interrupt and other parts of the processor think that no interrupt happened, chaos will reign as the processor takes contradictory actions. Moreover, waiting a few nanoseconds isn't a cure because the duration of metastability can be arbitrarily long. Waiting helps, since the chance of metastability decreases exponentially with time, but there is no guarantee.12

The obvious solution is to never change an input exactly when the clock changes. The processor is designed so that internal signals are stable when the clock changes, avoiding metastability. Specifically, the designer of a flip-flop specifies the setup time—how long the signal must be stable before the clock edge—and the hold time—how long the signal must be stable after the clock edge. As long as the input satisfies these conditions, typically a few picoseconds long, the flip-flop will function without metastability.

Unfortunately, the setup and hold times can't be guaranteed when the processor receives an external signal that isn't synchronized to its clock, known as an asynchronous signal. For instance, a processor receives interrupt signals when an I/O device has data, but the timing is unpredictable because it depends on mechanical factors such as a keypress or a spinning floppy disk. Most of the time, everything will work fine, but what about the one-in-a-billion case where the timing of the signal is unlucky? (Since modern processors run at multi-gigahertz, one-in-a-billion events are not rare; they can happen multiple times per second.)

One solution is a circuit called a synchronizer that takes an asynchronous signal and synchronizes it to the clock. A synchronizer can be implemented with two flip-flops in series: even if the first flip-flop has a metastable output, chances are that it will resolve to 0 or 1 before the second flip-flop stores the value. Each flip-flop provides an exponential reduction in the chance of metastability, so using two flip-flops drastically reduces the risk. In other words, the circuit will still fail occasionally, but if the mean time between failures (MTBF) is long enough (say, decades instead of seconds), then the risk is acceptable.

The schematic above shows how the 386 uses two flip-flops to minimize metastability. The first flip-flop is a special flip-flop that is based on a sense amplifier. It is much more complicated than a regular flip-flop, but it responds faster, reducing the chance of metastability. It is built from two of the sense-amplifier latches below, which I haven't seen described anywhere. In a DRAM memory chip, a sense amplifier takes a weak signal from a memory cell and rapidly amplifies it into a solid 0 or 1. In this flip-flop, the sense amplifier takes a potentially ambiguous signal and rapidly amplifies it into a 0 or 1. By amplifying the signal quickly, the flip-flop reduces metastability. (See the footnote for details.14)

The die photo below shows how this circuitry looks on the die. Each flip-flop is built from two latches; note that the sense-amp latches are larger than the standard latches. As before, the pad has protection diodes inside guard rings. For some reason, however, these diodes have a different structure from the transistor-based diodes described earlier. The 386 has five inputs that use this circuitry to protect against metastability.13 These inputs are all located together at the bottom of the die—it probably makes the layout more compact when neighboring pad circuits are all the same size.

In summary, the 386's I/O circuits are interesting because they are completely different from the chip's regular logic circuitry. In these circuits, the border between digital and analog breaks down; these circuits handle binary signals, but analog issues dominate the design. Moreover, hidden parasitic transistors play key roles; what you don't see can be more important than what you see. These circuits defend against three dangerous "dragons": static electricity, latchup, and metastability. Intel succeeded in warding off these dragons and the 386 was a success.

For more on the 386 and other chips, follow me on Mastodon (@kenshirriff@oldbytes.space), Bluesky (@righto.com), or RSS. (I've given up on Twitter.) If you want to read more about 386 input circuits, I wrote about the clock pin here

Notes and references

Intel released the 386 processor in 1985, the first 32-bit chip in the x86 line. This chip was packaged in a ceramic square with 132 gold-plated pins protruding from the underside, fitting into a socket on the motherboard. While this package may seem boring, a lot more is going on inside it than you might expect. Lumafield performed a 3-D CT scan of the chip for me, revealing six layers of complex wiring hidden inside the ceramic package. Moreover, the chip has nearly invisible metal wires connected to the sides of the package, the spikes below. The scan also revealed that the 386 has two separate power and ground networks: one for I/O and one for the CPU's logic.

The package, below, provides no hint of the complex wiring embedded inside the ceramic. The silicon die is normally not visible, but I removed the square metal lid that covers it.1 As a result, you can also see the two tiers of gold contacts that surround the silicon die.

Intel selected the 132-pin ceramic package to meet the requirements of a high pin count, good thermal characteristics, and low-noise power to the die.2: However, standard packages didn't provide sufficient power, so Intel designed a custom package with "single-row double shelf bonding to two signal layers and four power and ground planes." In other words, the die's bond wires are connected to the two shelves (or tiers) of pads surrounding the die. Internally, the package is like a 6-layer printed-circuit board made from ceramic.

The photo below shows the two tiers of pads with tiny gold bond wires attached: I measured the bond wires at 35 µm in diameter, thinner than a typical human hair. Some pads have up to five wires attached to support more current for the power and ground pads. You can consider the package to be a hierarchical interface from the tiny circuits on the die to the much larger features of the computer's motherboard. Specifically, the die has a feature size of 1 µm, while the metal wiring on top of the die has 6 µm spacing. The chip's wiring connects to the chip's bond pads, which have 0.01" spacing (.25 mm). The bond wires connect to the package's pads, which have 0.02" spacing (.5 mm); double the spacing because there are two tiers. The package connects these pads to the pin grid with 0.1" spacing (2.54 mm). Thus, the scale expands by about a factor of 2500 from the die's microscopic circuitry to the chip's pins. `

The ceramic package is manufactured through a complicated process.4 The process starts with flexible ceramic "green sheets", consisting of ceramic powder mixed with a binding agent. After holes for vias are created in the sheet, tungsten paste is silk-screened onto the sheet to form the wiring. The sheets are stacked, laminated under pressure, and then sintered at high temperature (1500ºC to 1600ºC) to create the rigid ceramic. The pins are brazed onto the bottom of the chip. Next, the pins and the inner contacts for the die are electroplated with gold.3 The die is mounted, gold bond wires are attached, and a metal cap is soldered over the die to encapsulate it. Finally, the packaged chip is tested, the package is labeled, and the chip is ready to be sold.

The diagram below shows a close-up of a signal layer inside the package. The pins are connected to the package's shelf pads through metal traces, spectacularly colored in the CT scan. (These traces are surprisingly wide and free-form; I expected narrower traces to reduce capacitance.) Bond wires connect the shelf pads to the bond pads on the silicon die. (The die image is added to the diagram; it is not part of the CT scan.) The large red circles are vias from the pins. Some vias connect to this signal layer, while other vias pass through to other layers. The smaller red circles are connections to a power layer; because the shelf pads are only on the two signal layers, the six power planes have connections to the signal layers for bonding. Since bond wires are only connected on the signal layers, the power layers need connections to pads on the signal layers.

The diagram below shows the corresponding portion of a power layer. A power layer looks completely different from a signal layer; it is a single conductive plane with holes. The grid of smaller holes allows the ceramic above and below this layer to bond, forming a solid piece of ceramic. The larger holes surround pin vias (red dots), allowing pin connections to pass through to a different layer. The red dots that contact the sheet are where power pins connect to this layer. Because the only connections to the die are from the signal layers, the power layers have connections to the signal layers; these are the smaller dots near the bond wires, either power vias passing through or vias connected to this layer.

With the JavaScript tool below, you can look at the package, layer by layer. Click on a radio button to select a layer. By observing the path of a pin through the layers, you can see where it ends up. For instance, the upper left pin passes through multiple layers until the upper signals layer connects it to the die. The pin to its right passes through all the layers until it reaches the logic Vcc plane on top. (Vcc is the 5-volt supply that powers the chip, called Vcc for historical reasons.)

Pins I/O Vcc Signals I/O gnd Signals Logic gnd Logic Vcc

If you select the logic Vcc plane above, you'll see a bright blotchy square in the center. This is not the die itself, I think, but the adhesive that attaches the die to the package, epoxy filled with silver to provide thermal and electrical conductivity. Since silver blocks X-rays, it is highly visible in the image.

Side contacts for electroplating

What surprised me most about the scans was seeing wires that stick out to the sides of the package. These wires are used during manufacturing when the pins are electroplated with gold.5 In order to electroplate the pins, each pin must be connected to a negative voltage so it can function as a cathode. This is accomplished by giving each pin a separate wire that goes to the edge of the package.

This diagram below compares the CT scan (above) to a visual side view of the package (below). The wires are almost invisible, but can be seen as darker spots. The arrows show how three of these spots match with the CT scan; you can match up the other spots.6

Two power networks

According to the datasheet, the 386 has 20 pins connected to +5V power (Vcc) and 21 pins connected to ground (Vss). Studying the die, I noticed that the I/O circuitry in the 386 has separate power and ground connections from the logic circuitry. The motivation is that the output pins require high-current driver circuits. When a pin switches from 0 to 1 or vice versa, this can cause a spike on the power and ground wiring. If this spike is too large, it can interfere with the processor's logic, causing malfunctions. The solution is to use separate power wiring inside the chip for the I/O circuitry and for the logic circuitry, connected to separate pins. On the motherboard, these pins are all connected to the same power and ground, but decoupling capacitors absorb the I/O spikes before they can flow into the chip's logic.

The diagram below shows how the two power and ground networks look on the die, with separate pads and wiring. The square bond pads are at the top, with dark bond wires attached. The white lines are the two layers of metal wiring, and the darker regions are circuitry. Each I/O pin has a driver circuit below it, consisting of relatively large transistors to pull the pin high or low. This circuitry is powered by the horizontal lines for I/O Vcc (light red) and I/O ground (Vss, light blue). Underneath each I/O driver is a small logic circuit, powered by thinner Vcc (dark red) and Vss (dark blue). Thicker Vss and Vcc wiring goes to the logic in the rest of the chip. Thus, if the I/O circuitry causes power fluctuations, the logic circuit remains undisturbed, protected by its separate power wiring.

The datasheet doesn't mention the separate I/O and logic power networks, but by using the CT scans, I determined which pins power I/O, and which pins power logic. In the diagram below, the light red and blue pins are power and ground for I/O, while the dark red and blue pins are power and ground for logic. The pins are scattered across the package, allowing power to be supplied to all four sides of the die.

"No Connect" pins

As the diagram above shows, the 386 has eight pins labeled "NC" (No Connect)—when the chip is installed in a computer, the motherboard must leave these pins unconnected. You might think that the 132-pin package simply has eight extra, unneeded pins, but it's more complicated than that. The photo below shows five bond pads at the bottom of the 386 die. Three of these pads have bond wires attached, but two have no bond wires: these correspond to No Connect pins. Note the black marks in the middle of the pads: the marks are from test probes that were applied to the die during testing.7 The No Connect pads presumably have a function during this testing process, providing access to an important internal signal.

Seven of the eight No Connect pads are almost connected: the package has a spot for a bond wire in the die cavity and the package has internal wiring to a No Connect pin. The only thing missing is the bond wire between the pad and the die cavity. Thus, by adding bond wires, Intel could easily create special chips with these pins connected, perhaps for debugging the test process itself.

The surprising thing is that one of the No Connect pads does have the bond wire in place, completing the connection to the external pin. (I marked this pin in green in the pinout diagram earlier.) From the circuitry on the die, this pin appears to be an output. If someone with a 386 chip hooks this pin to an oscilloscope, maybe they will see something interesting.

Labeling the pads on the die

The earlier 8086 processor, for example, is packaged in a DIP (Dual-Inline Package) with two rows of pins. This makes it straightforward to figure out which pin (and thus which function) is connected to each pad on the die. However, since the 386 has a two-dimensional grid of pins, the mapping to the pads is unclear. You can guess that pins are connected to a nearby pad, but ambiguity remains. Without knowing the function of each pad, I have a harder time reverse-engineering the die.

In fact, my primary motivation for scanning the 386 package was to determine the pin-to-pad mapping and thus the function of each pad.8 Once I had the CT data, I was able to trace out each hidden connection between the pad and the external pin. The image below shows some of the labels; click here for the full, completely labeled image. As far as I know, this information hasn't been available outside Intel until now.

Conclusions

Intel's early processors were hampered by inferior packages, but by the time of the 386, Intel had realized the importance of packaging. In Intel's early days, management held the bizarre belief that chips should never have more than 16 pins, even though other companies used 40-pin packages. Thus, Intel's first microprocessor, the 4004 (1971), was crammed into a 16-pin package, limiting its performance. By 1972, larger memory chips forced Intel to move to 18-pin packages, extremely reluctantly.9 The eight-bit 8008 processor (1972) took advantage of this slightly larger package, but performance still suffered because signals were forced to share pins. Finally, Intel moved to the standard 40-pin package for the 8080 processor (1974), contributing to the chip's success. In the 1980s, pin-grid arrays became popular in the industry as chips required more and more pins. Intel used a ceramic pin grid array (PGA) with 68 pins for the 186 and 286 processors (1982), followed by the 132-pin package for the 386 (1985).

The main drawback of the ceramic package was its cost. According to the 386 oral history, the cost of the 386 die decreased over time to the point where the chip's package cost as much as the die. To counteract this, Intel introduced a low-cost plastic package for the 386 that cost just a dollar to manufacture, the Plastic Quad Flat Package (PQFP) (details).

In later Intel processors, the number of connections exponentially increased. A typical modern laptop processor uses a Ball Grid Array with 2049 solder balls; the chip is soldered directly onto the circuit board. Other Intel processors use a Land Grid Array (LGA): the chip has flat contacts called lands, while the socket has the pins. Some Xeon processors have 7529 contacts, a remarkable growth from the 16 pins of the Intel 4004.

From the outside, the 386's package looks like a plain chunk of ceramic. But the CT scan revealed surprising complexity inside, from numerous contacts for electroplating to six layers of wiring. Perhaps even more secrets lurk in the packages of modern processors.

Follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS. (I've given up on Twitter.) Thanks to Jon Bruner and Lumafield for scanning the chip. Lumafield's interactive CT scan of the 386 package is available here if you to want to examine it yourself. Lumafield also scanned a 1960s cordwood flip-flop and the Soviet Globus spacecraft navigation instrument for us. Thanks to John McMaster for taking 2D X-rays.

Notes and references

Have you ever wanted to reverse engineer an analog chip from a die photo? Wanted to understand what's inside the "black box" of an integrated circuit? In this article, I explain my reverse engineering process, using the Philips TDA7000 FM radio receiver chip as an example. This chip was the first FM radio receiver on a chip.1 It was designed in 1977—an era of large transistors and a single layer of metal—so it is much easier to examine than modern chips. Nonetheless, the TDA7000 is a non-trivial chip with over 100 transistors. It includes common analog circuits such as differential amplifiers and current mirrors, along with more obscure circuits such as Gilbert cell mixers.

The die photo above shows the silicon die of the TDA7000; I've labeled the main functional blocks and some interesting components. Arranged around the border of the chip are 18 bond pads: the pads are connected by thin gold bond wires to the pins of the integrated circuit package. In this chip, the silicon appears greenish, with slightly different colors—gray, pink, and yellow-green—where the silicon has been "doped" with impurities to change its properties. Carefully examining the doping patterns will reveal the transistors, resistors, and other microscopic components that make up the chip.

The most visible part of the die is the metal wiring, the speckled white lines that connect the silicon structures. The metal layer is separated from the silicon underneath by an insulating oxide layer, allowing metal lines to pass over other circuitry without problem. Where a metal wire connects to the underlying silicon, a small white square is visible; this square is a hole in the oxide layer, allowing the metal to contact the silicon.

This chip has a single layer of metal, so it is much easier to examine than modern chips with a dozen or more layers of metal. However, the single layer of metal made it much more difficult for the designers to route the wiring while avoiding crossing wires. In the die photo above, you can see how the wiring meanders around the circuitry in the middle, going the long way since the direct route is blocked. Later, I'll discuss some of the tricks that the designers used to make the layout successful.

NPN transistors

Transistors are the key components in a chip, acting as switches, amplifiers, and other active devices. While modern integrated circuits are fabricated from MOS transistors, earlier chips such as the TDA7000 were constructed from bipolar transistors: NPN and PNP transistors. The photo below shows an NPN transistor in the TDA7000 as it appears on the chip. The different shades are regions of silicon that have been doped with various impurities, forming N and P regions with different electrical properties. The white lines are the metal wiring connected to the transistor's collector (C), emitter (E), and base (B). Below the die photo, the cross-section diagram shows how the transistor is constructed. The region underneath the emitter forms the N-P-N sandwich that defines the NPN transistor.

The parts of an NPN transistor can be identified by their appearance. The emitter is a compact spot, surrounded by the gray silicon of the base region. The collector is larger and separated from the emitter and base, sometimes separated by a significant distance. The colors may appear different in other chips, but the physical structures are similar. Note that although the base is in the middle conceptually, it is often not in the middle of the physical layout.

The transistor is surrounded by a yellowish-green border of P+ silicon; this border is an important part of the structure because it isolates the transistor from neighboring transistors.2 The isolation border is helpful for reverse-engineering because it indicates the boundaries between transistors.

PNP transistors

You might expect PNP transistors to be similar to NPN transistors, just swapping the roles of N and P silicon. But for a variety of reasons, PNP transistors have an entirely different construction. They consist of a circular emitter (P), surrounded by a ring-shaped base (N), which is surrounded by the collector (P). This forms a P-N-P sandwich horizontally (laterally), unlike the vertical structure of an NPN transistor. In most chips, distinguishing NPN and PNP transistors is straightforward because NPN transistors are rectangular while PNP transistors are circular.

The diagram above shows one of the PNP transistors in the TDA7000. As with the NPN transistor, the emitter is a compact spot. The collector consists of gray P-type silicon; in contrast, the base of an NPN transistor consists of gray P-type silicon. Moreover, unlike the NPN transistor, the base contact of the PNP transistor is at a distance, while the collector contact is closer. (This is because most of the silicon inside the isolation boundary is N-type silicon. In a PNP transistor, this region is connected to the base, while in an NPN transistor, this region is connected to the collector.)

It turns out that PNP transistors have poorer performance than NPN transistors for semiconductor reasons3, so most analog circuits use NPN transistors except when PNP transistors are necessary. For instance, the TDA7000 has over 100 NPN transistors but just nine PNP transistors. Accordingly, I'll focus my discussion on NPN transistors.

Resistors

Resistors are a key component of analog chips. The photo below shows a zig-zagging resistor in the TDA7000, formed from gray P-type silicon. The resistance is proportional to the length,4 so large-valued resistors snake back and forth to fit into the available space. The two red arrows indicate the contacts between the ends of the resistor and the metal wiring. Note the isolation region around the resistor, the yellowish border. Without this isolation, two resistors (formed of P-silicon) embedded in N-silicon could form an unintentional PNP transistor.

Unfortunately, resistors in ICs are very inaccurate; the resistances can vary by 50% from chip to chip. As a result, analog circuits are typically designed to depend on the ratio of resistor values, which is fairly constant within a chip. Moreover, high-value resistors are inconveniently large. We'll see below some techniques to reduce the need for large resistances.

Capacitors

Capacitors are another important component in analog circuits. The capacitor below is a "junction capacitor", which uses a very large reverse-biased diode as a capacitor. The pink "fingers" are N-doped regions, embedded in the gray P-doped silicon. The fingers form a "comb capacitor"; this layout maximizes the perimeter area and thus increases the capacitance. To produce the reverse bias, the N-silicon fingers are connected to the positive voltage supply through the upper metal strip. The P silicon is connected to the circuit through the lower metal strip.

How does a diode junction form a capacitor? When a diode is reverse-biased, the contact region between N and P silicon becomes "depleted", forming a thin insulating region between the two conductive silicon regions. Since an insulator between two conducting surfaces forms a capacitor, the diode acts as a capacitor. One problem with a diode capacitor is that the capacitance varies with the voltage because the thickness of the depletion region changes with voltage. But as we'll see later, the TDA7000's tuning circuit turns this disadvantage into a feature.

Other chips often create a capacitor with a plate of metal over silicon, separated by a thin layer of oxide or other dielectric. However, the manufacturing process for bipolar chips generally doesn't provide thin oxide, so junction capacitors are a common alternative.5 On-chip capacitors take up a lot of space and have relatively small capacitance, so IC designers try to avoid capacitors. The TDA7000 has seven on-chip capacitors but most of the capacitors in this design are larger, external capacitors: the chip uses 12 of its 18 pins just to connect external capacitors to the necessary points in the internal circuitry.

Important analog circuits

A few circuits are very common in analog chips. In this section, I'll explain some of these circuits, but first, I'll give a highly simplified explanation of an NPN transistor, the minimum you should know for reverse engineering. (PNP transistors are similar, except the polarities of the voltages and currents are reversed. Since PNP transistors are rare in the TDA7000, I won't go into details.)

In a transistor, the base controls the current between the collector and the emitter, allowing the transistor to operate as a switch or an amplifier. Specifically, if a small current flows from the base of an NPN transistor to the emitter, a much larger current can flow from the collector to the emitter, larger, perhaps, by a factor of 100.6 To get a current to flow, the base must be about 0.6 volts higher than the emitter. As the base voltage continues to increase, the base-emitter current increases exponentially, causing the collector-emitter current to increase. (Normally, a resistor will ensure that the base doesn't get much more than 0.6V above the emitter, so the currents stay reasonable.)

NPN transistor circuits have some general characteristics. When there is no base current, the transistor is off: the collector is high and the emitter is low. When the transistor turns on, the current through the transistor pulls the collector voltage lower and the emitter voltage higher. Thus, in a rough sense, the emitter is the non-inverting output and the collector is the inverting output.

The complete behavior of transistors is much more complicated. The nice thing about reverse engineering is that I can assume that the circuit works: the designers needed to consider factors such as the Early effect, capacitance, and beta, but I can ignore them.

Emitter follower

One of the simplest transistor circuits is the emitter follower. In this circuit, the emitter voltage follows the base voltage, staying about 0.6 volts below the base. (The 0.6 volt drop is also called a "diode drop" because the base-emitter junction acts like a diode.)

This behavior can be explained by a feedback loop. If the emitter voltage is too high, the current from the base to the emitter drops, so the current through the collector drops due to the transistor's amplification. Less current through the resistor reduces the voltage across the resistor (from Ohm's Law), so the emitter voltage goes down. Conversely, if the emitter voltage is too low, the base-emitter current increases, increasing the collector current. This increases the voltage across the resistor, and the emitter voltage goes up. Thus, the emitter voltage adjusts until the circuit is stable; at this point, the emitter is 0.6 volts below the base.

You might wonder why an emitter follower is useful. Although the output voltage is lower, the transistor can supply a much higher current. That is, the emitter follower amplifies a weak input current into a stronger output current. Moreover, the circuitry on the input side is isolated from the circuitry on the output side, preventing distortion or feedback.

Current mirror

Most analog chips make extensive use of a circuit called a current mirror. The idea is you start with one known current, and then you can "clone" multiple copies of the current with a simple transistor circuit, the current mirror.

In the following circuit, a current mirror is implemented with two identical PNP transistors. A reference current passes through the transistor on the right. (In this case, the current is set by the resistor.) Since both transistors have the same emitter voltage and base voltage, they source the same current, so the current on the left matches the reference current (more or less).7

A common use of a current mirror is to replace resistors. As mentioned earlier, resistors inside ICs are inconveniently large. It saves space to use a current mirror instead of multiple resistors whenever possible. Moreover, the current mirror is relatively insensitive to the voltages on the different branches, unlike resistors. Finally, by changing the size of the transistors (or using multiple collectors of different sizes), a current mirror can provide different currents.

The TDA7000 doesn't use current mirrors as much as I'd expect, but it has a few. The die photo above shows one of its current mirrors, constructed from PNP transistors with their distinctive round appearance. Two important features will help you recognize a current mirror. First, one transistor has its base and collector connected; this is the transistor that controls the current. In the photo, the transistor on the right has this connection. Second, the bases of the two transistors are connected. This isn't obvious above because the connection is through the silicon, rather than in the metal. The trick is that these PNP transistors are inside the same isolation region. If you look at the earlier cross-section of a PNP transistor, the whole N-silicon region is connected to the base. Thus, two PNP transistors in the same isolation region have their bases invisibly linked, even though there is just one base contact from the metal layer.

Current sources and sinks

Analog circuits frequently need a constant current. A straightforward approach is to use a resistor; if a constant voltage is applied, the resistor will produce a constant current. One disadvantage is that circuits can cause the voltage to vary, generating unwanted current fluctuations. Moreover, to produce a small current (and minimize power consumption), the resistor may need to be inconveniently large. Instead, chips often use a simple circuit to control the current: this circuit is called a "current sink" if the current flows into it and a "current source" if the current flows out of it.

Many chips use a current mirror as a current source or sink instead. However, the TDA7000 uses a different approach: a transistor, a resistor, and a reference voltage.8 The transistor acts like an emitter follower, causing a fixed voltage across the resistor. By Ohm's Law, this yields a fixed current. Thus, the circuit sinks a fixed current, controlled by the reference voltage and the size of the resistor. By using a low reference voltage, the resistor can be kept small.

Differential pair amplifier

If you see two transistors with the emitters connected, chances are that it is a differential amplifier: the most common two-transistor subcircuit used in analog ICs.9 The idea of a differential amplifier is that it takes the difference of two inputs and amplifies the result. The differential amplifier is the basis of the operational amplifier (op amp), the comparator, and other circuits. The TDA7000 uses multiple differential pairs for amplification. For filtering, the TDA7000 uses op-amps, formed from differential amplifiers.10

The schematic below shows a simple differential pair. The current sink at the bottom provides a fixed current I, which is split between the two input transistors. If the input voltages are equal, the current will be split equally into the two branches (I1 and I2). But if one of the input voltages is a bit higher than the other, the corresponding transistor will conduct more current, so that branch gets more current and the other branch gets less. The resistors in each branch convert the current to a voltage; either side can provide the output. A small difference in the input voltages results in a large output voltage, providing the amplification. (Alternatively, both sides can be used as a differential output, which can be fed into a second differential amplifier stage to provide more amplification. Note that the two branches have opposite polarity: when one goes up, the other goes down.)

The diagram below shows the locations of differential amps, voltage references, mixers, and current mirrors. As you can see, these circuits are extensively used in the TDA7000.

Tips on tracing out circuitry

Over the years, I've found various techniques helpful for tracing out the circuitry in an IC. In this section, I'll describe some of those techniques.

First, take a look at the datasheet if available. In the case of the TDA7000, the datasheet and application note provide a detailed block diagram and a description of the functionality.21 Sometimes datasheets include a schematic of the chip, but don't be too trusting: datasheet schematics are often simplified. Moreover, different manufacturers may use wildly different implementations for the same part number. Patents can also be helpful, but they may be significantly different from the product.

Mapping the pinout in the datasheet to the pads on the die will make reverse engineering much easier. The power and ground pads are usually distinctive, with thick traces that go to all parts of the chip, as shown in the photo below. Once you have identified the power and ground pads, you can assign the other pads in sequence from the datasheet. Make sure that these pad assignments make sense. For instance, the TDA7000 datasheet shows special circuitry between pads 5 and 6 and between pads 13 and 14; the corresponding tuning diodes and RF transistors are visible on the die. In most chips, you can distinguish output pins by the large driver transistors next to the pad, but this turns out not to help with the TDA7000. Finally, note that chips sometimes have test pads that don't show up in the datasheet. For instance, the TDA7000 has a test pad, shown below; you can tell that it is a test pad because it doesn't have a bond wire.

Once I've determined the power and ground pads, I trace out all the power and ground connections on the die. This makes it much easier to understand the circuits and also avoids the annoyance of following a highly-used signal around the chip only to discover that it is simply ground. Note that NPN transistors will have many collectors connected to power and emitters connected to ground, perhaps through resistors. If you find the opposite situation, you probably have power and ground reversed.

For a small chip, a sheet of paper works fine for sketching out the transistors and their connections. But with a larger chip, I find that more structure is necessary to avoid getting mixed up in a maze of twisty little wires, all alike. My solution is to number each component and color each wire as I trace it out, as shown below. I use the program KiCad to draw the schematic, using the same transistor numbering. (The big advantage of KiCad over paper is that I can move circuits around to get a nicer layout.)

It works better to trace out the circuitry one area at a time, rather than chasing signals all over the chip. Chips are usually designed with locality, so try to avoid following signals for long distances until you've finished up one block. A transistor circuit normally needs to be connected to power (if you follow the collectors) and ground (if you follow the emitters).11 Completing the circuit between power and ground is more likely to give you a useful functional block than randomly tracing out a chain of transistors. (In other words, follow the bases last.)

Finally, I find that a circuit simulator such as LTspice is handy when trying to understand the behavior of mysterious transistor circuits. I'll often whip up a simulation of a small sub-circuit if its behavior is unclear.

How FM radio and the TDA7000 work

Before I explain how the TDA7000 chip works, I'll give some background on FM (Frequency Modulation). Suppose you're listening to a rock song on 97.3 FM. The number means that the radio station is transmitting at a carrier frequency of 97.3 megahertz. The signal, perhaps a Beyoncé song, is encoded by slightly varying the frequency, increasing the frequency when the signal is positive and decreasing the frequency when the signal is negative. The diagram below illustrates frequency modulation; the input signal (red) modulates the output. Keep in mind that the modulation is highly exaggerated in the diagram; the modulation would be invisible in an accurate diagram since a radio broadcast changes the frequency by at most ±75 kHz, less than 0.1% of the carrier frequency.

FM radio's historical competitor is AM (Amplitude Modulation), which varies the height of the signal (the amplitude) rather than the frequency.12 One advantage of FM is that it is more resistant to noise than AM; an event such as lightning will interfere with the signal amplitude but will not change the frequency. Moreover, FM radio provides stereo, while AM radio is mono, but this is due to the implementation of radio stations, not a fundamental characteristic of FM versus AM. (The TDA7000 chip doesn't implement stereo.13) Due to various factors, FM stations require more bandwidth than AM, so FM stations are spaced 200 kHz apart while AM stations are just 10 kHz apart.

An FM receiver such as the TDA7000 must demodulate the radio signal to recover the transmitted audio, converting the changing frequency into a changing signal level. FM is more difficult to demodulate than AM, which can literally be done with a piece of rock: lead sulfide in a crystal detector. There are several ways to implement an FM demodulator; this chip uses a technique called a quadrature detector. The key to a quadrature detector is a circuit that shifts the phase, with the amount of phase shift depending on the frequency. The detector shifts the signal by approximately 90º, multiplies it by the original signal, and then smooths it out with a low-pass filter. If you do this with a sine wave and a 90º phase shift, the result turns out to be 0. But since the phase shift depends on the frequency, a higher frequency gets shifted by more than 90º while a lower frequency gets shifted by less than 90º. The final result turns out to be approximately linear with the frequency, positive for higher frequencies and negative for lower frequencies. Thus, the FM signal is converted into the desired audio signal.

Like most radios, the TDA7000 uses a technique called superheterodyning that was invented around 1917. The problem is that FM radio stations use frequencies from 88.0 MHz to 108.0 MHz. These frequencies are too high to conveniently handle on a chip. Moreover, it is difficult to design a system that can process a wide range of frequencies. The solution is to shift the desired radio station's signal to a frequency that is fixed and much lower. This frequency is called the intermediate frequency. Although FM radios commonly use an intermediate frequency of 10.7 MHz, this was still too high for the TDA7000, so the designers used an intermediate frequency of just 70 kilohertz. This frequency shift is accomplished through superheterodyning.

For example, suppose you want to listen to the radio station at 97.3 MHz. When you tune to this station, you are actually tuning the local oscillator to a frequency that is 70 kHz lower, 97.23 MHz in this case. The local oscillator signal and the radio signal are mixed by multiplying them. If you multiply two sine waves, you get one sine wave at the difference of the frequencies and another sine wave at the sum of the frequencies. In this case, the two signals are at 70 kHz and 194.53 MHz. A low-pass filter (the IF filter) discards everything above 70 kHz, leaving just the desired radio station, now at a fixed and conveniently low frequency. The rest of the radio can then be optimized to work at 70 kHz.

The Gilbert cell multiplier

But how do you multiply two signals? This is accomplished with a circuit called a Gilbert cell.14 This circuit takes two differential inputs, multiplies them, and produces a differential output. The Gilbert cell is a bit tricky to understand,15 but you can think of it as a stack of differential amplifiers, with the current directed along one of four paths, depending on which transistors turn on. For instance, if the A and B inputs are both positive, current will flow through the leftmost transistor, labeled "pos×pos". Likewise, if the A and B inputs are both negative, current flows through the rightmost transistor, labeled "neg×neg". The outputs from both transistors are connected, so both cases produce a positive output. Conversely, if one input is positive and the other is negative, current flows through one of the middle transistors, producing a negative output. Since the multiplier handles all four cases of positive and negative inputs, it is called a "four-quadrant" multiplier.

Although the Gilbert cell is an uncommon circuit in general, the TDA7000 uses it in multiple places. The first mixer implements the superheterodyning. A second mixer provides the FM demodulation, multiplying signals in the quadrature detector described earlier. The TDA7000 also uses a mixer for its correlator, which determines if the chip is tuned to a station or not.16 Finally, a Gilbert cell switches the audio off when the radio is not properly tuned. On the die, the Gilbert cell has a nice symmetry that reflects the schematic.

The voltage-controlled oscillator

One of the trickiest parts of the TDA7000 design is how it manages to use an intermediate frequency of just 70 kilohertz. The problem is that broadcast FM has a "modulation frequency deviation" of 75 kHz, which means that the broadcast frequency varies by up to ±75 kHz. The mixer shifts the broadcast frequency down to 70 kHz, but the shifted frequency will vary by the same amount as the received signal. How can you have a 70 kilohertz signal that varies by 75 kilohertz? What happens when the frequency goes negative?

The solution is that the local oscillator frequency (i.e., the frequency that the radio is tuned to) is continuously modified to track the variation in the broadcast frequency. Specifically, a change in the received frequency causes the local oscillator frequency to change, but only by 80% as much. For instance, if the received frequency decreases by 5 hertz, the local oscillator frequency is decreased by 4 hertz. Recall that the intermediate frequency is the difference between the two frequencies, generated by the mixer, so the intermediate frequency will decrease by just 1 hertz, not 5 hertz. The result is that as the broadcast frequency changes by ±75 kHz, the local oscillator frequency changes by just ±15 kHz, so it never goes negative.

How does the radio constantly adjust the frequency? The fundamental idea of FM is that the frequency shift corresponds to the output audio signal. Since the output signal tracks the frequency change, the output signal can be used to modify the local oscillator's frequency, using a voltage-controlled oscillator.17 Specifically, the circuit uses special "varicap" diodes that vary their capacitance based on the voltage that is applied. As described earlier, the thickness of a diode's "depletion region" depends on the voltage applied, so the diode's capacitance will vary with voltage. It's not a great capacitor, but it is good enough to adjust the frequency.

The image above shows how these diodes appear on the die. The diodes are relatively large and located between two bond pads. The two diodes have interdigitated "fingers"; this increases the capacitance as described earlier with the "comb capacitor". The slightly grayish "background" region is the P-type silicon, with a silicon control line extending to the right. (Changing the voltage on this line changes the capacitance.) Regions of N-type silicon are underneath the metal fingers, forming the PN junctions of the diodes.

Keep in mind that most of the radio tuning is performed with a variable capacitor that is external to the chip and adjusts the frequency from 88 MHz to 108 MHz. The capacitance of the diodes provides the much smaller adjustment of ±60 kHz. Thus, the diodes only need to provide a small capacitance shift.

The VCO and diodes will also adjust the frequency to lock onto the station if the tuning is off by a moderate amount, say, 100 kHz. However, if the tuning is off by a large amount, say, 200 kHz, the FM detector has a "sideband" and the VCO can erroneously lock onto this sideband. This is a problem because the sideband is weak and nonlinear so reception will be bad and will have harmonic distortion. To avoid this problem, the correlator will detect that the tuning is too far off (i.e. the local oscillator is way off from 70 kHz) and will replace the audio with white noise. Thus, the user will realize that they aren't on the station and adjust the tuning, rather than listening to distorted audio and blaming the radio.

Noise source

Where does the radio get the noise signal to replace distorted audio? The noise is generated from the circuit below, which uses the thermal noise from diodes, amplified by a differential amplifier. Specifically, each side of the differential amplifier is connected to two transistors that are wired as diodes (using the base-emitter junction). Random thermal fluctuations in the transistors will produce small voltage changes on either side of the amplifier. The amplifier boosts these fluctuations, creating the white noise output.

Layout tricks and unusual transistors

Because this chip has just one layer of metal, the designers had to go to considerable effort to connect all the components without wires crossing. One common technique to make routing easier is to separate a transistor's emitter, collector, and base, allowing wires to pass over the transistor. The transistor below is an example. Note that the collector, base, and emitter have been stretched apart, allowing one wire to pass between the collector and the base, while two more pass between the base and the emitter. Moreover, the transistor layout is flexible: this one has the base in the middle, while many others have the emitter in the middle. (Putting the collector in the middle won't work since the base needs to be next to the emitter.)

The die photo below illustrates a few more routing tricks. This photo shows one collector, three emitters, and four bases, but there are three transistors. How does that work? First, these three transistors are in the same isolation region, so they share the same "tub" of N-silicon. If you look back at the cross-section of an NPN transistor, you'll see that this tub is connected to the collector contact. Thus, all three transistors share the same collector.18 Next, the two bases on the left are connected to the same gray P-silicon. Thus, the two base contacts are connected and function as a single base. In other words, this is a trick to connect the two base wires together through the silicon, passing under the four other metal wires in the way. Finally, the two transistors on the right have the emitter and base slightly separated so a wire can pass between them. When reverse-engineering a chip, be on the lookout for unusual transistor layouts such as these.

When all else failed, the designers could use a "cross-under" to let a wire pass under other wires. The cross-under is essentially a resistor with a relatively low resistance, formed from N-type silicon (pink in the die photo below). Because silicon has much higher resistance than metal, cross-unders are avoided unless necessary. I see just two cross-unders in the TDA7000.

The circuit that caused me the most difficulty is the noise generator below. The transistor highlighted in red below looks straightforward: a resistor is connected to the collector, which is connected to the base. However, the transistor turned out to be completely different: the collector (red arrow) is on the other side of the circuit and this collector is shared with five other transistors. The structure that I thought was the collector is simply the contact at the end of the resistor, connected to the base.

Conclusions

The TDA7000 almost didn't become a product. It was invented in 1977 by two engineers at the Philips research labs in the Netherlands. Although Philips was an innovative consumer electronics company in the 1970s, the Philips radio group wasn't interested in an FM radio chip. However, a rogue factory manager built a few radios with the chips and sent them to Japanese companies. The Japanese companies loved the chip and ordered a million of them, convincing Philips to sell the chips.

The TDA7000 became a product in 1983—six years after its creation—and reportedly more than 5 billion have now been sold.19 Among other things, the chip allowed an FM radio to be built into a wristwatch, with the headphone serving as an antenna. Since the TDA7000 vastly simplified the construction of a radio, the chip was also popular with electronics hobbyists. Hobbyist magazines provided plans and the chip could be obtained from Radio Shack.20

Why reverse engineer a chip such as the TDA7000? In this case, I was answering some questions for the IEEE microchips exhibit, but even when reverse engineering isn't particularly useful, I enjoy discovering the logic behind the mysterious patterns on the die. Moreover, the TDA7000 is a nice chip for reverse engineering because it has large features that are easy to follow, but it also has many different circuits. Since the chip has over 100 transistors, you might want to start with a simpler chip, but the TDA7000 is a good exercise if you want to increase your reverse-engineering skills. If you want to check your results, my schematic of the TDA7000 is here; I don't guarantee 100% accuracy :-) In any case, I hope you have enjoyed this look at reverse engineering.

Follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS. (I've given up on Twitter.) Thanks to Daniel Mitchell for asking me about the TDA7000 and providing the die photo; be sure to check out the IEEE Chip Hall of Fame's TDA7000 article.

Notes and references

Back in 2021, a collector friend of ours was visiting a dusty warehouse in search of Apollo-era communications equipment. A box with NASA-style lights caught his eye—the "AGC Confirm" light suggested a connection with the Apollo Guidance Computer. Disappointingly, the box was just an empty chassis and the circuit boards were all missing. He continued to poke around the warehouse when, to his surprise, he found a bag on the other side of the warehouse that contained the missing boards! After reuniting the box with its wayward circuit cards, he brought it to us: could we make this undocumented unit work?

A label on the back indicated that it is an "Up-Data Link Confidence Test Set", built by Motorola. As the name suggests, the box was designed to test Apollo's Up-Data Link (UDL), a system that allowed digital commands to be sent up to the spacecraft. As I'll explain in detail below, these commands allowed ground stations to switch spacecraft circuits on or off, interact with the Apollo Guidance Computer, or set the spacecraft's clock. The Up-Data Link needed to be tested on the ground to ensure that its functions operated correctly. Generating the test signals for the Up-Data Link and verifying its outputs was the responsibility of the Up-Data Link Confidence Test Set (which I'll call the Test Set for short)

The Test Set illustrates how, before integrated circuits, complicated devices could be constructed from thumb-sized encapsulated modules. Since I couldn't uncover any documentation on these modules, I had to reverse-engineer them, discovering that different modules implemented everything from flip-flops and logic gates to opto-isolators and analog circuits. With the help of a Lumafield 3-dimensional X-ray scanner, we looked inside the modules and examined the discrete transistors, resistors, diodes, and other components mounted inside.

Reverse-engineering this system—from the undocumented modules to the mess of wiring—was a challenge. Mike found one NASA document that mentioned the Test Set, but the document was remarkably uninformative.1 Moreover, key components of the box were missing, probably removed for salvage years ago. In this article, I'll describe how we learned the system's functionality, uncovered the secrets of the encapsulated modules, built a system to automatically trace the wiring, and used the UDL Test Set in a large-scale re-creation of the Apollo communications system.

The Apollo Up-Data Link

Before describing the Up-Data Link Test Set, I'll explain the Up-Data Link (UDL) itself. The Up-Data Link provided a mechanism for the Apollo spacecraft to receive digital commands from ground stations. These commands allowed ground stations to control the Apollo Guidance Computer, turn equipment on or off, or update the spacecraft's clock. Physically, the Up-Data Link is a light blue metal box with an irregular L shape, weighing almost 20 pounds.

The Apollo Command Module was crammed with boxes of electronics, from communication and navigation to power and sequencing. The Up-Data Link was mounted above the AC power inverters, below the Apollo Guidance Computer, and to the left of the waste management system and urine bags.

Up-Data Link Messages

The Up-Data Link supported four types of messages:

• Mission Control had direct access to the Apollo Guidance Computer (AGC) through the UDL, controlling the computer, keypress by keypress. That is, each message caused the UDL to simulate a keypress on the Display/Keyboard (DSKY), the astronaut's interface to the computer.

• The spacecraft had a clock, called the Central Timing Equipment or CTE, that tracked the elapsed time of the mission, from days to seconds. A CTE message could set the clock to a specified time.

• A system called Real Time Control (RTC) allowed the UDL to turn relays on or off, so some spacecraft systems to be controlled from the ground.2 These 32 relays, mounted inside the Up-Data Link box, could do everything from illuminating an Abort light—indicating that Mission Control says to abort—to controlling the data tape recorder or the S-band radio.

• Finally, the UDL supported two test messages to "exercise all process, transfer and program control logic" in the UDL.

The diagram below shows the format of messages to the Up-Data Link. Each message consisted of 12 to 30 bits, depending on the message type. The first three bits, the Vehicle Address, selected which spacecraft should receive the message. (This allowed messages to be directed to the Saturn V booster, the Command Module, or the Lunar Module.3) Next, three System Address bits specified the spacecraft system to receive the message, corresponding to the four message types above. The remaining bits supplied the message text.

The contents of the message text depended on the message type. A Real Time Control (RTC) message had a six-bit value specifying the relay number as well as whether it should be turned off or on. An Apollo Guidance Computer (AGC) message had a five-bit value specifying a key on the Display/Keyboard (DSKY). For reliability, the message was encoded in 16 bits: the message, the message inverted, the message again, and a padding bit; any mismatching bits would trigger an error. A CTE message set the clock using four 6-bit values indicating seconds, minutes, hours, and days. The UDL processed the message by resetting the clock and then advancing the time by issuing the specified number of pulses to the CTE to advance the seconds, minutes, hours, and days. (This is similar to setting a digital alarm clock by advancing the digits one at a time.) Finally, the two self test messages consisted of 24-bit patterns that would exercise the UDL's internal circuitry. The results of the test were sent back to Earth via Apollo's telemetry system.

For reliability, each bit transmitted to the UDL was replaced by five "sub-bits": each "1" bit was replaced with the sub-bit sequence "01011", and each "0" bit was replaced with the complement, "10100".4 The purpose of the sub-bits was that any corrupted data would result in an invalid sub-bit code so corrupted messages could be rejected. The Up-Data Link performed this validation by matching the input data stream against "01011" or "10100". (The vehicle address at the start of a message used a different sub-bit code, ensuring that the start of the message was properly identified.) By modern standards, sub-bits are an inefficient way of providing redundancy, since the message becomes five times larger. As a consequence, the effective transmission rate was low: 200 bits per second.

There was no security in the Up-Data Link messages, apart from the need for a large transmitter. Of the systems on Apollo, only the rocket destruct system—euphemistically called the Propellant Dispersion System—was cryptographically secure.5

Since the Apollo radio system was analog, the digital sub-bits couldn't be transmitted from ground to space directly. Instead, a technique called phase-shift keying (PSK) converted the data into an audio signal. This audio signal consists of a sine wave that is inverted to indicate a 0 bit versus a 1 bit; in other words, its phase is shifted by 180 degrees for a 0 bit. The Up-Data Link box takes this audio signal as input and demodulates it to extract the digital message data. (Transmitting this audio signal from ground to the Up-Data Link required more steps that aren't relevant to the Test Set, so I'll describe them in a footnote.6)

The Up-Data Link Test Set

Now that I've explained the Up-Data Link, I can describe the Test Set in more detail. The purpose of the UDL Test Set is to test the Up-Data Link system. It sends a message—as an audio signal—to the Up-Data Link box, implementing the message formatting, sub-bit encoding, and phase shift keying described above. Then it verifies the outputs from the UDL to ensure that the UDL performed the correct action.

Perhaps the most visible feature of the Test Set is the paper tape reader on the front panel: this reader is how the Test Set obtains messages to transmit. Messages are punched onto strips of paper tape, encoded as a sequence of 13 octal digits.7 After a message is read from paper tape, it is shown on the 13-digit display. The first three digits are an arbitrary message number, while the remaining 10 octal digits denote the 30-bit message to send to the UDL. Based on the type of message, specified by the System Address digit, the Test Set validates the UDL's response and indicates success or errors on the panel lights.

I created the block diagram below to explain the architecture and construction of the Test Set (click for a larger view). The system has 25 circuit boards, labeled A1 through A25;8 for the most part, they correspond to functional blocks in the diagram.

The Test Set's front panel is dominated by its display of 13 large digits. It turns out that the storage of these digits is the heart of the Test Set. This storage (A3-A9) assembles the digits as they are read from the paper tape, circulates the bits for transmission, and provides digits to the other circuits to select the message type and validate the results. To accomplish this, the 13 digit circuits are configured as a 39-bit shift register. As the message is read from the paper tape, its bits are shifted into the digit storage, right to left, and the message is shown on the display. To send the message, the shift register is reconfigured so the 10 digits form a loop, excluding the message number. As the bits cycle through the loop, the leftmost bit is encoded and transmitted. At the end of the transmission, the digits have cycled back to their original positions, so the message can be transmitted again if desired. Thus, the shift-register mechanism both deserializes the message when it is read and serializes the message for transmission.

The Test Set uses three boards (A15, A2, and A1) to expand the message with sub-bits and to encode the message into audio. The first board converts each bit into five sub-bits. The second board applies phase-shift keying (PSK) modulation, and the third board has filters to produce clean sine waves from the digital signals.

On the input side, the Test Set receives signals from the Up-Data Link (UDL) box through round military-style connectors. These input signals are buffered by boards A25, A22, A23, A10, and A24. Board 15 verifies the input sub-bits by comparing them with the transmitted sub-bits. For an AGC message, the computer signals are verified by board A14. The timing (CTE) signals are verified by boards A20 and A21. The UDL status (validity) signals are processed by board A12. Board A11 implements a switching power supply to power the interface boards.

You can see from the block diagram that the Test Set is complex and implements multiple functions. On the other hand, the block diagram also shows that it takes a lot of 1960s circuitry to implement anything. For instance, one board can only handle two digits, so the digit display alone requires seven boards. Another example is the inputs, requiring a full board for two or three input bits.

Encapsulated modules

The box is built from modules that are somewhat like integrated circuits but contain discrete components. Modules like these were used in the early 1960s before ICs caught on. Each module implements a simple function such as a flip-flop or buffer. They were more convenient than individual components, since a module provided a ready-made function. They were also compact, since the components were tightly packaged inside the module.

Physically, each module has 13 pins: a row of 7 on one side and a row of 6 offset on the other side. This arrangement ensures that a module cannot be plugged in backward.

Reverse engineering these modules was difficult since they were encapsulated in plastic and the components were inaccessible. The text printed on each module hinted at its function. For example, the J-K flip-flop module above is labeled "LP FF". The "2/2G & 2/1G" module turned out to contain two NAND gates and two inverters (the 2G and 1G gates). A "2P/3G" module contains two pull-up resistors and two three-input NAND gates. Other modules provided special-purpose analog functions for the PSK modulation.

I reverse-engineered the functions of the modules by applying signals and observing the results. Conveniently, the pins are on 0.200" spacing so I could plug modules into a standard breadboard. The functions of the logic modules were generally straightforward to determine. The analog modules were more difficult; for instance, the "-3.9V" module contains a -3.9-volt Zener diode, six resistors, and three capacitors in complicated arrangements.

To determine how the modules are constructed internally, we had a module X-rayed by John McMaster and another module X-rayed in three dimensions by Lumafield. The X-rays revealed that modules were built with "cordwood construction", a common technique in the 1960s. That is, cylindrical components were mounted between two boards, stacked parallel similar to a pile of wood logs. Instead of using printed-circuit boards, the leads of the components were welded to metal strips to provide the interconnections.

For more information on these modules, see my articles Reverse-engineering a 1960s cordwood flip-flop module with X-ray CT scans and X-ray reverse-engineering a hybrid module. You can interact with the scan here.

The boards

In this section, I'll describe some of the circuit boards and point out their interesting features. A typical board has up to 15 modules, arranged as five rows of three. The modules are carefully spaced so that two boards can be meshed with the components on one board fitting into the gaps on the other board. Thus, a pair of boards forms a dense block.

Each pair of boards is attached to side rails and a mounting bracket, forming a unit.8 The bracket has ejectors to remove the board unit, since the backplane connectors grip the boards tightly. Finally, each bracket is labeled with the board numbers, the test point numbers, and the Motorola logo. The complexity of this mechanical assembly suggests that Motorola had developed an integrated prototyping system around the circuit modules, prior to the Test Set.

Digit driver boards

The photo below shows a typical board, the digit driver board. At the left, a 47-pin plug provides the connection between the board and the Test Set's backplane. At the right, 15 test connections allow the board to be probed and tested while it is installed. The board itself is a two-sided printed circuit board with gold plating. Boards are powered with +6V, -6V, and ground; the two red capacitors in the lower left filter the two voltages.

The digit driver is the most common board in the system, appearing six times.9 Each board stores two octal digits in a shift register and drives two digit displays on the front panel. Since the digits are octal, each digit requires three bits of storage, implemented with three flip-flop modules connected as a shift register. If you look closely, you can spot the six flip-flop modules, labeled "LP FF".

The digits are displayed through an unusual technology: an edge-lit lightguide display.10 From a distance, it resembles a Nixie tube, but it uses 10 lightbulbs, one for each number value, with a plastic layer for each digit. Each plastic sheet has numerous dots etched in the shape of the corresponding number. One sheet is illuminated from the edge, causing the dots in the sheet to light up and display that number. In the photo below, you can see both the illuminated and the unilluminated dots. The displays take 14 volts, but the box runs at 28 volts, so a board full of resistors on the front panel drops the voltage from 28 to 14, giving off noticeable heat in the process.

For each digit position, the driver board provides eight drive signals, one for each bulb. The drivers are implemented in "LD" modules. Since each LD module contains two drive transistors controlled by 4-input AND gates, a module supports two bulbs. Thus, a driver board holds eight LD modules in total. The LD modules are also used on other boards to drive the lights on the front panel.

Ring counters

The Test Set contains multiple counters to count bits, sub-bits, digits, states, and so forth. While a modern design would use binary counters, the Test Set is implemented with a circuit called a ring counter that optimizes the hardware.

For instance, to count to ten, five flip-flops are arranged as a shift register so each flip-flop sends its output to the next one. However, the last flip-flop sends its inverted output to the first. The result is that the counter will proceed: 10000, 11000, 11100, 11110, 11111 as 1 bits are shifted in at the left. But after a 1 reaches the last bit, 0 bits will be shifted in at the left: 01111, 00111, 00011, 00001, and finally 0000. Thus, the counter moves through ten states.

Why not use a 4-bit binary counter and save a flip-flop? First, the binary counter requires additional logic to go from 9 back to 0. Moreover, acting on a particular binary value requires a 4-input gate to check the four bits. But a particular value of a ring counter can be detected with a smaller 2-input gate by checking the bits on either side of the 0/1 boundary. For instance, to detect a count of 3 (11100), only the two highlighted bits need to be tested. Thus, the decoding logic is much simpler for a ring counter, which is important when each gate comes in an expensive module.

Another use of the ring counter is in the sub-state generator, counting out the five states. Since this ring counter uses three flip-flops, you might expect it to count to six. However, the first flip-flop gets one of its inputs from the second flip-flop, resulting in five states: 000, 100, 110, 011, and 001, with the 111 state skipped.11 This illustrates the flexibility of ring counters to generate arbitrary numbers of states.

The PSK boards

Digital data could not be broadcast directly to the spacecraft, so the data was turned into an audio signal using phase-shift keying (PSK). The Test Set uses two boards (A1 and A2) to produce this signal. These boards are interesting and unusual because they are analog, unlike the other boards in the Test Set.

The idea behind phase-shift keying is to change the phase of a sine wave depending on the bit (i.e., sub-bit) value. Specifically, a 2 kHz sine wave indicated a one bit, while the sine wave was inverted for a zero bit. That is, a phase shift of 180º indicated a 0 bit. But how do you tell which sine wave is original and which is flipped? The solution was to combine the information signal with a 1 kHz reference signal that indicates the start and phase of each bit. The diagram below shows how the bits 1-0-1 are encoded into the composite audio signal that is decoded by the Up-Data Link box.

The core of the PSK modulation circuit is a transformer with a split input winding. The 2 kHz sine wave is applied to the winding's center tap. One side of the winding is grounded (by the "ø DET" module) for a 0 bit, but the other side of the winding is grounded for a 1 bit. This causes the signal to go through the winding in one direction for a 1 bit and the opposite direction for a 0 bit. The transformer's output winding thus receives an inverted signal for a 0 bit, giving the 180º phase shift seen in the second waveform above. Finally, the board produces the composite audio signal by mixing in the reference signal through a potentiometer and the "SUM" module.12

Inconveniently, some key components of the Test Set were missing; probably the most valuable components were salvaged when the box was scrapped. The missing components included the power supplies and amplifiers on the back of the box, as well as parts from PSK board A1. This board had ten white wires that had been cut, going to missing components labeled MP1, R2, L1, and L2. By studying the circuitry, I determined that MP1 had been a 4-kHz oscillator that provided the master clock for the Test Set. R2 was simply a potentiometer to adjust signal levels.

But L1 and L2 were more difficult. It took a lot of reverse-engineering before we determined that L1 and L2 were resonant filters to convert the digital waveforms to the sine waves needed for the PSK output. Marc used a combination of theory and trial-and-error to determine the inductor and capacitor values that produced a clean signal. The photo above shows our substitute filters, along with a replacement oscillator.

Input boards

The Test Set receives signals from the Up-Data Link box under test and verifies that these signals are correct. The Test Set has five input boards (A22 through A25) to buffer the input signals and convert them to digital levels. The input boards also provide electrical isolation between the input signals and the Test Set, avoiding problems caused by ground loops or different voltage levels.

A typical input board is A22, which receives two input signals, supplied through coaxial cables. The board buffers the signals with op-amps, and then produces a digital signal for use by the box. The op-amp outputs go into "1 SS" isolation modules that pass the signal through to the box while ensuring isolation. These modules are optocouplers, using an LED and a phototransistor to provide isolation.13 The op-amps are powered by an isolated power supply.

Each op-amp module is a Burr-Brown Model 1506 module,14 encapsulating a transistorized op-amp into a convenient 8-pin module. The module is similar to an integrated-circuit op-amp, except it has discrete components inside and is considerably larger than an integrated circuit. Burr-Brown is said to have created the first solid-state op-amp in 1957, and started making op-amp modules around 1962.

Board A24 is also an isolated input board, but uses different circuitry. It has two modules that each contain four Schmitt triggers, circuits to sharpen up a noisy input. These modules have the puzzling label "-12+6LC". Each output goes through a "1 SS" isolation module, as with the previous input boards. This board receives the 8-bit "validity" signal from the Up-Data Link.

The switching power supply board

Board A11 is interesting: instead of sealed modules, it has a large green cube with numerous wires attached. This board turned out to be a switching power supply that implements six dual-voltage power supplies. The green cube is a transformer with 14 center-tapped windings connected to 42 pins. The transformer ensures that the power supply's outputs are isolated. This allows the op-amps on the input boards to remain electrically isolated from the rest of the Test Set.

The power supply uses a design known as a Royer Converter; the two transistors drive the transformer in a push-pull configuration. The transistors are turned on alternately at high frequency, driven by a feedback winding. The transformer has multiple windings, one for each output. Each center-tapped winding uses two diodes to produce a DC output, filtered by the large capacitors. In total, the power supply has four ±7V outputs and two ±14V outputs to supply the input boards.

This switching power supply is independent from the power supplies for the rest of the Test Set. On the back of the box, we could see where power supplies and amplifiers had been removed. Determining the voltages of the missing power supplies would have been a challenge. Fortunately, the front of the box had test points with labels for the various voltages: -6, +6, and +28, so we knew what voltages were required.

The front panel

The front panel reveals many of the features of the Test Set. At the top, lights indicate the success or failure of various tests. "Sub-bit agree/error" indicates if the sub-bits read back into the Test Set match the values sent. "AGC confirm/error" shows the results of an Apollo Guidance Computer message, while "CTE confirm/error" shows the results of a Central Timing Equipment message. "Verif confirm/error" indicates if the verification message from the UDL matches the expected value for a test message. At the right, lights indicate the status of the UDL: standby, active, or powered off.

In the middle, toggle switches control the UDL operation. The "Sub-bit spoil" switch causes sub-bits to be occasionally corrupted for testing purposes. "Sub-bit compare/override" enables or disables sub-bit verification. The four switches on the right control the paper tape reader. The "Program start" switch is the important one: it causes the UDL to send one message (in "Single" mode) or multiple messages (in "Serial" mode). The Test Set can stop or continue when an error occurs ("Stop on error" / "Bypass error"). Finally, "Tape advance" causes messages to be read from paper tape, while "Tape stop" causes the UDL to re-use the current message rather than loading a new one.

The UDL provides a verification code that indicates its status. The "Verification Return" knob selects the source of this verification code: the "Direct" position uses a 4-bit verification code, while "Remote" uses an 8-bit verification code.15

At the bottom, "PSK high/low" selects the output level for the PSK signal from the Test Set. (Since the amplifier was removed from our Test Set, this switch has no effect. Likewise, the "Power On / Off" switch has no effect since the power supplies were removed. We power the Test Set with an external lab supply.) In the middle, 15 test points allow access to various signals inside the Test Set. The round elapsed time indicator shows how many hours the Test Set has been running (apparently over 12 months of continuous operation).

Reverse-engineering the backplane

Once I figured out the circuitry on each board, the next problem was determining how the boards were connected. The backplane consists of rows of 47-pin sockets, one for each board. Dense white wiring runs between the sockets as well as to switches, displays, and connectors. I started beeping out the connections with a multimeter, picking a wire and then trying to find the other end. Some wires were easy since I could see both ends, but many wires disappeared into a bundle. I soon realized that manually tracing the wiring was impractically slow: with 25 boards and 47 connections per board, brute-force testing of every pair of connections would require hundreds of thousands of checks.

To automate the beeping-out of connections, I built a system that I call Beep-o-matic. The idea behind Beep-o-matic is to automatically find all the connections between two motherboard slots by plugging two special boards into the slots. By energizing all the pins on the first board in sequence, a microcontroller can detect connected pins on the second board, revealing the wiring between the two slots.

This system worked better than I expected, rapidly generating a list of connections. I still had to plug the Beep-o-matic boards into each pair of slots (about 300 combinations in total), but each scan took just a few seconds, so a full scan was practical. To find the wiring to the switches and connectors, I used a variant of the process. I plugged a board into a slot and used a program to continuously monitor the pins for changes. I went through the various switch positions and applied signals to the connectors to find the associated connections.

Conclusions

I started reverse-engineering the Test Set out of curiosity: given an undocumented box made from mystery modules and missing key components, could we understand it? Could we at least get the paper tape reader to run and the lights to flash? It was a tricky puzzle to figure out the modules and the circuitry, but eventually we could read a paper tape and see the results on the display.

But the box turned out to be useful. Marc has amassed a large and operational collection of Apollo communications hardware. We use the UDL Test Set to generate realistic signals that we feed into Apollo's S-band communication system. We haven't transmitted these signals to the Moon, but we have transmitted signals between antennas a few feet apart, receiving them with a box called the S-band Transponder. Moreover, we have used the Test Set to control an Up-Data Link box, a CTE clock, and a simulated Apollo Guidance Computer, reading commands from the paper tape and sending them through the complete communication path. Ironically, the one thing we haven't done with the Test Set is use it to test the Up-Data Link in the way it is intended: connecting the UDL's outputs to the Test Set and checking the panel lights.

From a wider perspective, the Test Set provides a glimpse of the vast scope of the Apollo program. This complicated box was just one part of the test apparatus for one small part of Apollo's electronics. Think of the many different electronic systems in the Apollo spacecraft, and consider the enormous effort to test them all. And electronics was just a small part of Apollo alongside the engines, mechanical structures, fuel cells, and life support systems. With all this complexity, it's not surprising that the Apollo program employed 400,000 people.

For more information, the footnotes include a list of UDL documentation16 and CuriousMarc's videos17. Follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS. (I've given up on Twitter.) I worked on this project with CuriousMarc, Mike Stewart, and Eric Schlapfer. Thanks to John McMaster for X-rays, thanks to Lumafield for the CT scans, and thanks to Marcel for providing the box.

Notes and references

# Introduction

Qubes OS can appear as something weird and hard to figure for people that never used it.  By this article, I would like to help other understanding what it is, and when it is useful.

=> https://www.qubes-os.org/ Qubes OS official project page

Two years ago, I wrote something that was mostly a list of Qubes OS features, but this was not really helping readers to understand what is Qubes OS except it does XYZ stuff.

While Qubes OS is often tagged as a security operating system, it only offers a canvas to handling compartmentalized systems to work as a whole.

Qubes OS gives its user the ability to do cyber risk management the way they want, which is unique.  A quick word about it if you are not familiar with risk management: for instance, when running software at different level, you should ask "can I trust this?", can you trust the packager?  The signing key?  The original developer?  The transitive dependencies involved?  It is not possible to entirely trust the whole chain, so you might want to take actions like handling sensitive data only when disconnected.  Or you might want to ensure that if your web browser is compromised, the data leak and damage will be reduced to a minimum.  This can go pretty far and is complementary to in-depth defense or security hardening of operating systems.

=> https://dataswamp.org/~solene/2023-06-17-qubes-os-why.html 2023-06-17 Why one would use Qubes OS?

In the article, I will pass on some features that I do not think are interesting for introducing Qubes OS to people or that could be too confusing, so no need to tell me I forgot to talk about XYZ feature :-)

# Meta operating system

I like to call Qubes OS a meta operating system, because it is not a Linux / BSD / Windows based OS: its core is Xen (some kind of virtualization enabled kernel).  Not only it's Xen based, but by design it is meant to run virtual machines, hence the name "meta operating system" which is an OS meant to run many OSes make sense to me. 

Qubes OS comes with a few virtual machines templates that are managed by the development team:

* debian
* fedora
* whonix (debian based distribution hardened for privacy)

There are also community templates for arch linux, gentoo, alpine, kali, kicksecure and certainly other you can find within the community.

Templates are not just templates, they are a ready to work, one-click/command install systems that integrate well within Qubes OS.  It is time to explain how virtual machines interact together, as it is what makes Qubes OS great compared to any Linux system running KVM.

A virtual machine is named a "qube", it is a set of information and integration (template, firewall rules, resources, services, icons, ...).

# Virtual machines synergy and integration

The host system which has some kind of "admin" powers with regard to virtualization is named dom0 in Xen jargon.  On Qubes OS, dom0 is a Fedora system (using a Xen kernel) with very few things installed, no networking and no USB access.  Those two devices classes are assigned to two qubes, respectively named "sys-net" and "sys"usb".  It is so to reduce the surface attack of dom0.

When running a graphical program within a qube, it will show a dedicated window in dom0 window manager, there are no big windows for each virtual machine, so running programs feels like a unified experience.  The seamless windows feature works through a specific graphics driver within the qube, official templates support it and there is a Windows driver for it too.

Each qube has its own X11 server running, its own clipboard, kernel and memory.  There are features to copy the clipboard of one qube, and transfer it to the clipboard of another qube.  This can be configured to prevent clipboards to be used where you should not.  This is rather practical if you store all your passwords in a qube, and you want to copy/paste them.

There are also file copy capabilities between qubes, which goes through Xen channels (some interconnection between Xen virtual machines allowing to transfer data), so no network is involved for data transfer.  Data copy can also be configured, like one qube may be able to receive files from any, but never allow file to be transferred out.

In operations involving RPC features like file copy, a GUI in dom0 is shown to ask confirmation by the user (with a tiny delay to prevent hitting Enter before being able to understand what was going on).

As mentioned above, USB devices are assigned to a qube named "sys-usb", it provides a program to pass a device to a given qube (still through Xen channels), so it is easy to dispatch devices where you need them.

# Networking

Qubes OS offer a tree like networking with sys-net (holding the hardware networking devices) at the root and a sys-firewall qube below, from there, you can attach qubes to sys-firewall to get network.

Firewall rules can be configured per qube, and will be applied on the qube providing network to the one configured, this prevents the qube from removing its own rules because it is done at a level higher in the tree.

A tree like networking system also allow running multiple VPN in parallel, and assign qubes to each VPNs as you need.  In my case, when I work for multiple clients they all have their own VPN, so I dedicate them a qube connecting to their VPN, then I attach qubes I use to work for this client to the according VPN.  With the firewall rule set on the VPN qube to prevent any connection except to the endpoint, I have the guarantee that all traffic of that client work will go through their VPN.

It is also possible to not use any network in a qube, so it is offline and unable to connect to network.

Qubes OS come out of the box (except if you uncheck the box) with a qube encapsulating all traffic network through Tor network (incompatible traffic like UDP is discarded).

# Templates (in Qubes OS jargon)

I talked about templates earlier, in the sense of "ready to be installed and used", but a "Template VM" in Qubes OS has a special meaning.  In order to make things manageable when you have a few dozen qubes, like handling updates or installing software, Qubes OS introduced Templates VMs.

A Template VM is a qube that you almost never use, except when you need to install a software or make a system change within it.  Qubes OS updater will also make sure, from time to time, that installed packages are up-to-date.

So, what are them if there are not used?  They are templates for a type of qubes named "AppVM".  An AppVM is what you work the most with.  It is an instance of the template it is configured to use, always reset from pristine state when starting, with a few directories persistent across reboot for this AppVM.  The directories are all in `/rw/` and symlinked where useful: `/home` and `/usr/local/` by default.  You can have a single Template VM of Debian 13 and a dozen AppVM with each their own data in it, if you want to install "vim", you do it in the template and then all AppVM using Debian 13 Template VM will have "vim" installed (after a reboot after the change). Note that is also work for emacs :)

With this mechanism, it is easy to switch an AppVM from a Linux distribution to another, just switch the qube template to use Fedora instead of Debian, reboot, done.  This is also useful when switching to a new major release of the distribution in the template: Debian 13 is bugged?  Let's switch back to Debian 12 until it is fixed and continue working (do not forget writing a bug report to Debian).

# Disposables templates

You learned about Templates VM and how a AppVM inherits all the template, reset in fresh state every time.  What about an AppVM that could be run from its pristine state the same way?  They did it, it is called a disposable qube.

Basically, a disposable qube is a temporary copy of an AppVM with all its storage discarded on shutdown.  It is the default for the sys-usb qube handling USB, if it gets infected by a device, it will be reset from a fresh state next boot.

Disposables have many use case:

* running a command on non-trusted file, to view or try to convert it to something more trustable (a PDF into BMP?)
* running a known to work system for a specific task, and be sure it will work exactly the same every time, like when using a printer
* as a playground to try stuff in an environment identical to another

# Automatic snapshot

Last but not least, a pretty nice but hidden feature is the ability to revert the storage of a qube to a previous state.

=> https://www.qubes-os.org/doc/volume-backup-revert/ Qubes OS documentation: volume backup and revert

qubes are using virtual storage that can stack multiple changes, from a base image with different layers of changes over time stacked on top of it.  Once the number of revisions to keep is reached, the oldest layer above the base image is merged.  This is a simple mechanism that allows to revert to any given checkpoint between the base image and the last checkpoint.

Did you delete important files, and restoring a backup is way too much effort?  Revert the last volume.  Did a package update break an important software in a template? Revert the last volume.

Obviously, it comes as an extra storage cost, deleted files are only freed from the storage once they do not exist in a checkpoint.

# Downsides of running Qubes OS


Qubes OS has some drawbacks:

* it is slower than running a vanilla system, because all virtualization involved as a cost, most notably all 3D rendering is done on CPU within qubes, which is terrible for eye candy effects or video decoding.  It is possible, with a lot of efforts, to assign second GPU when you have one, to a single qube at a time, to use it, but as the sentence already long enough is telling out loud, it is not practical.
* it requires effort to get into as it is different from your usual operating system, you will need to learn how to use it (this sounds rather logical when using a tool)
* hardware compatibility is a bit limited due Xen kernel, there is compatibility list curated by the community

=> https://www.qubes-os.org/hcl/ Qubes OS hardware compatibility list

# Conclusion

I tried to give a simple overview of major Qubes OS features.  The goal was not to make you reader an expert or be aware of every single feature, but to allow you to understand what Qubes OS can offer.

This one is a hot mess but it's too late in the week to start over. Oh well!

Someone recognized me at last week's Chipy and asked for my opinion on Sapir-Whorf hypothesis in programming languages. I thought this was interesting enough to make a newsletter. First what it is, then why it looks like it applies, and then why it doesn't apply after all.

The Sapir-Whorf Hypothesis

We dissect nature along lines laid down by our native language. — Whorf

To quote from a Linguistics book I've read, the hypothesis is that "an individual's fundamental perception of reality is moulded by the language they speak." As a massive oversimplification, if English did not have a word for "rebellion", we would not be able to conceive of rebellion. This view, now called Linguistic Determinism, is mostly rejected by modern linguists.

The "weak" form of SWH is that the language we speak influences, but does not decide our cognition. For example, Russian has distinct words for "light blue" and "dark blue", so can discriminate between "light blue" and "dark blue" shades faster than they can discriminate two "light blue" shades. English does not have distinct words, so we discriminate those at the same speed. This linguistic relativism seems to have lots of empirical support in studies, but mostly with "small indicators". I don't think there's anything that convincingly shows linguistic relativism having effects on a societal level.¹

The weak form of SWH for software would then be the "the programming languages you know affects how you think about programs."

SWH in software

This seems like a natural fit, as different paradigms solve problems in different ways. Consider the hardest interview question ever, "given a list of integers, sum the even numbers". Here it is in four paradigms:

• Procedural: total = 0; foreach x in list {if IsEven(x) total += x}. You iterate over data with an algorithm.
• Functional: reduce(+, filter(IsEven, list), 0). You apply transformations to data to get a result.
• Array: + fold L * iseven L.² In English: replace every element in L with 0 if odd and 1 if even, multiple the new array elementwise against L, and then sum the resulting array. It's like functional except everything is in terms of whole-array transformations.
• Logical: Somethingish like sumeven(0, []). sumeven(X, [Y|L]) :- iseven(Y) -> sumeven(Z, L), X is Y + Z ; sumeven(X, L). You write a set of equations that express what it means for X to be the sum of events of L.

There's some similarities between how these paradigms approach the problem, but each is also unique, too. It's plausible that where a procedural programmer "sees" a for loop, a functional programmer "sees" a map and an array programmer "sees" a singular operator.

I also have a personal experience with how a language changed the way I think. I use TLA+ to detect concurrency bugs in software designs. After doing this for several years, I've gotten much better at intuitively seeing race conditions in things even without writing a TLA+ spec. It's even leaked out into my day-to-day life. I see concurrency bugs everywhere. Phone tag is a race condition.

But I still don't think SWH is the right mental model to use, for one big reason: language is special. We think in language, we dream in language, there are huge parts of our brain dedicated to processing language. We don't use those parts of our brain to read code.

SWH is so intriguing because it seems so unnatural, that the way we express thoughts changes the way we think thoughts. That I would be a different person if I was bilingual in Spanish, not because the life experiences it would open up but because grammatical gender would change my brain.

Compared to that, the idea that programming languages affect our brain is more natural and has a simpler explanation:

It's the goddamned Tetris Effect.

The Goddamned Tetris Effect

The Tetris effect occurs when someone dedicates vast amounts of time, effort and concentration on an activity which thereby alters their thoughts, dreams, and other experiences not directly linked to said activity. — Wikipedia

Every skill does this. I'm a juggler, so every item I can see right now has a tiny metadata field of "how would this tumble if I threw it up". I teach professionally, so I'm always noticing good teaching examples everywhere. I spent years writing specs in TLA+ and watching the model checker throw concurrency errors in my face, so now race conditions have visceral presence. Every skill does this.

And to really develop a skill, you gotta practice. This is where I think programming paradigms do something especially interesting that make them feel more like Sapir-Whorfy than, like, juggling. Some languages mix lots of different paradigms, like Javascript or Rust. Others like Haskell really focus on excluding paradigms. If something is easy for you in procedural and hard in FP, in JS you could just lean on the procedural bits. In Haskell, too bad, you're learning how to do it the functional way.³

And that forces you to practice, which makes you see functional patterns everywhere. Tetris effect!

Anyway this may all seem like quibbling— why does it matter whether we call it "Tetris effect" or "Sapir-Whorf", if our brains is get rewired either way? For me, personally, it's because SWH sounds really special and unique, while Tetris effect sounds mundane and commonplace. Which it is. But also because TE suggests it's not just programming languages that affect how we think about software, it's everything. Spending lots of time debugging, profiling, writing exploits, whatever will change what you notice, what you think a program "is". And that's a way useful idea that shouldn't be restricted to just PLs.

(Then again, the Tetris Effect might also be a bad analogy to what's going on here, because I think part of it is that it wears off after a while. Maybe it's just "building a mental model is good".)

I just realized all of this might have missed the point

Wait are people actually using SWH to mean the weak form or the strong form? Like that if a language doesn't make something possible, its users can't conceive of it being possible. I've been arguing against the weaker form in software but I think I've seen strong form often too. Dammit.

Well, it's already Thursday and far too late to rewrite the whole newsletter, so I'll just outline the problem with the strong form: we describe the capabilities of our programming languages with human language. In college I wrote a lot of crappy physics lab C++ and one of my projects was filled with comments like "man I hate copying this triply-nested loop in 10 places with one-line changes, I wish I could put it in one function and just take the changing line as a parameter". Even if I hadn't encountered higher-order functions, I was still perfectly capable of expressing the idea. So if the strong SWH isn't true for human language, it's not true for programming languages either.

Systems Distributed talk now up!

Link here! Original abstract:

Building correct distributed systems takes thinking outside the box, and the fastest way to do that is to think inside a different box. One different box is "formal methods", the discipline of mathematically verifying software and systems. Formal methods encourages unusual perspectives on systems, models that are also broadly useful to all software developers. In this talk we will learn two of the most important FM perspectives: the abstract specifications behind software systems, and the property they are and aren't supposed to have.

The talk ended up evolving away from that abstract but I like how it turned out!

New Logic for Programmers Release!

v0.11 is now available! This is over 20% longer than v0.10, with a new chapter on code proofs, three chapter overhauls, and more! Full release notes here.

Software books I wish I could read

I'm writing Logic for Programmers because it's a book I wanted to have ten years ago. I had to learn everything in it the hard way, which is why I'm ensuring that everybody else can learn it the easy way.

Books occupy a sort of weird niche in software. We're great at sharing information via blogs and git repos and entire websites. These have many benefits over books: they're free, they're easily accessible, they can be updated quickly, they can even be interactive. But no blog post has influenced me as profoundly as Data and Reality or Making Software. There is no blog or talk about debugging as good as the Debugging book.

It might not be anything deeper than "people spend more time per word on writing books than blog posts". I dunno.

So here are some other books I wish I could read. I don't think any of them exist yet but it's a big world out there. Also while they're probably best as books, a website or a series of blog posts would be ok too.

Everything about Configurations

The whole topic of how we configure software, whether by CLI flags, environmental vars, or JSON/YAML/XML/Dhall files. What causes the configuration complexity clock? How do we distinguish between basic, advanced, and developer-only configuration options? When should we disallow configuration? How do we test all possible configurations for correctness? Why do so many widespread outages trace back to misconfiguration, and how do we prevent them?

I also want the same for plugin systems. Manifests, permissions, common APIs and architectures, etc. Configuration management is more universal, though, since everybody either uses software with configuration or has made software with configuration.

The Big Book of Complicated Data Schemas

I guess this would kind of be like Schema.org, except with a lot more on the "why" and not the what. Why is important for the Volcano model to have a "smokingAllowed" field?¹

I'd see this less as "here's your guide to putting Volcanos in your database" and more "here's recurring motifs in modeling interesting domains", to help a person see sources of complexity in their own domain. Does something crop up if the references can form a cycle? If a relationship needs to be strictly temporary, or a reference can change type? Bonus: path dependence in data models, where an additional requirement leads to a vastly different ideal data model that a company couldn't do because they made the old model.

(This has got to exist, right? Business modeling is a big enough domain that this must exist. Maybe The Essence of Software touches on this? Man I feel bad I haven't read that yet.)

Computer Science for Software Engineers

Yes, I checked, this book does not exist (though maybe this is the same thing). I don't have any formal software education; everything I know was either self-taught or learned on the job. But it's way easier to learn software engineering that way than computer science. And I bet there's a lot of other engineers in the same boat.

This book wouldn't have to be comprehensive or instructive: just enough about each topic to understand why it's an area of study and appreciate how research in it eventually finds its way into practice.

MISU Patterns

MISU, or "Make Illegal States Unrepresentable", is the idea of designing system invariants in the structure of your data. For example, if a Contact needs at least one of email or phone to be non-null, make it a sum type over EmailContact, PhoneContact, EmailPhoneContact (from this post). MISU is great.

Most MISU in the wild look very different than that, though, because the concept of MISU is so broad there's lots of different ways to achieve it. And that means there are "patterns": smart constructors, product types, properly using sets, newtypes to some degree, etc. Some of them are specific to typed FP, while others can be used in even untyped languages. Someone oughta make a pattern book.

My one request would be to not give them cutesy names. Do something like the Aarne–Thompson–Uther Index, where items are given names like "Recognition by manner of throwing cakes of different weights into faces of old uncles". Names can come later.

The Tools of '25

Not something I'd read, but something to recommend to junior engineers. Starting out it's easy to think the only bit that matters is the language or framework and not realize the enormous amount of surrounding tooling you'll have to learn. This book would cover the basics of tools that enough developers will probably use at some point: git, VSCode, very basic Unix and bash, curl. Maybe the general concepts of tools that appear in every ecosystem, like package managers, build tools, task runners. That might be easier if we specialize this to one particular domain, like webdev or data science.

Ideally the book would only have to be updated every five years or so. No LLM stuff because I don't expect the tooling will be stable through 2026, to say nothing of 2030.

A History of Obsolete Optimizations

Probably better as a really long blog series. Each chapter would be broken up into two parts:

1. A deep dive into a brilliant, elegant, insightful historical optimization designed to work within the constraints of that era's computing technology
2. What we started doing instead, once we had more compute/network/storage available.

c.f. A Spellchecker Used to Be a Major Feat of Software Engineering. Bonus topics would be brilliance obsoleted by standardization (like what people did before git and json were universal), optimizations we do today that may not stand the test of time, and optimizations from the past that did.

Sphinx Internals

I need this. I've spent so much goddamn time digging around in Sphinx and docutils source code I'm gonna throw up.

Systems Distributed Talk Today!

Online premier's at noon central / 5 PM UTC, here! I'll be hanging out to answer questions and be awkward. You ever watch a recording of your own talk? It's real uncomfortable!

I'm way too discombobulated from getting next month's release of Logic for Programmers ready, so I'm pulling a idea from the slush pile. Basically I wanted to come up with a mental model of arrays as a concept that explained APL-style multidimensional arrays and tables but also why there weren't multitables.

So, arrays. In all languages they are basically the same: they map a sequence of numbers (I'll use 1..N)¹ to homogeneous values (values of a single type). This is in contrast to the other two foundational types, associative arrays (which map an arbitrary type to homogeneous values) and structs (which map a fixed set of keys to heterogeneous values). Arrays appear in PLs earlier than the other two, possibly because they have the simplest implementation and the most obvious application to scientific computing. The OG FORTRAN had arrays.

I'm interested in two structural extensions to arrays. The first, found in languages like nushell and frameworks like Pandas, is the table. Tables have string keys like a struct and indexes like an array. Each row is a struct, so you can get "all values in this column" or "all values for this row". They're heavily used in databases and data science.

The other extension is the N-dimensional array, mostly seen in APLs like Dyalog and J. Think of this like arrays-of-arrays(-of-arrays), except all arrays at the same depth have the same length. So [[1,2,3],[4]] is not a 2D array, but [[1,2,3],[4,5,6]] is. This means that N-arrays can be queried on any axis.

 ]x =: i. 3 3
0 1 2
3 4 5
6 7 8
   0 { x NB. first row
0 1 2
   0 {"1 x NB. first column
0 3 6

So, I've had some ideas on a conceptual model of arrays that explains all of these variations and possibly predicts new variations. I wrote up my notes and did the bare minimum of editing and polishing. Somehow it ended up being 2000 words.

1-dimensional arrays

A one-dimensional array is a function over 1..N for some N.

To be clear this is math functions, not programming functions. Programming functions take values of a type and perform computations on them. Math functions take values of a fixed set and return values of another set. So the array [a, b, c, d] can be represented by the function (1 -> a ++ 2 -> b ++ 3 -> c ++ 4 -> d). Let's write the set of all four element character arrays as 1..4 -> char. 1..4 is the function's domain.

The set of all character arrays is the empty array + the functions with domain 1..1 + the functions with domain 1..2 + ... Let's call this set Array[Char]. Our compilers can enforce that a type belongs to Array[Char], but some operations care about the more specific type, like matrix multiplication. This is either checked with the runtime type or, in exotic enough languages, with static dependent types.

(This is actually how TLA+ does things: the basic collection types are functions and sets, and a function with domain 1..N is a sequence.)

2-dimensional arrays

Now take the 3x4 matrix

   i. 3 4
0 1  2  3
4 5  6  7
8 9 10 11

There are two equally valid ways to represent the array function:

1. A function that takes a row and a column and returns the value at that index, so it would look like f(r: 1..3, c: 1..4) -> Int.
2. A function that takes a row and returns that column as an array, aka another function: f(r: 1..3) -> g(c: 1..4) -> Int.²

Man, (2) looks a lot like currying! In Haskell, functions can only have one parameter. If you write (+) 6 10, (+) 6 first returns a new function f y = y + 6, and then applies f 10 to get 16. So (+) has the type signature Int -> Int -> Int: it's a function that takes an Int and returns a function of type Int -> Int.³

Similarly, our 2D array can be represented as an array function that returns array functions: it has type 1..3 -> 1..4 -> Int, meaning it takes a row index and returns 1..4 -> Int, aka a single array.

(This differs from conventional array-of-arrays because it forces all of the subarrays to have the same domain, aka the same length. If we wanted to permit ragged arrays, we would instead have the type 1..3 -> Array[Int].)

Why is this useful? A couple of reasons. First of all, we can apply function transformations to arrays, like "combinators". For example, we can flip any function of type a -> b -> c into a function of type b -> a -> c. So given a function that takes rows and returns columns, we can produce one that takes columns and returns rows. That's just a matrix transposition!

Second, we can extend this to any number of dimensions: a three-dimensional array is one with type 1..M -> 1..N -> 1..O -> V. We can still use function transformations to rearrange the array along any ordering of axes.

Speaking of dimensions:

What are dimensions, anyway

Okay, so now imagine we have a Row × Col grid of pixels, where each pixel is a struct of type Pixel(R: int, G: int, B: int). So the array is

Row -> Col -> Pixel

But we can also represent the Pixel struct with a function: Pixel(R: 0, G: 0, B: 255) is the function where f(R) = 0, f(G) = 0, f(B) = 255, making it a function of type {R, G, B} -> Int. So the array is actually the function

Row -> Col -> {R, G, B} -> Int

And then we can rearrange the parameters of the function like this:

{R, G, B} -> Row -> Col -> Int

Even though the set {R, G, B} is not of form 1..N, this clearly has a real meaning: f[R] is the function mapping each coordinate to that coordinate's red value. What about Row -> {R, G, B} -> Col -> Int? That's for each row, the 3 × Col array mapping each color to that row's intensities.

Really any finite set can be a "dimension". Recording the monitor over a span of time? Frame -> Row -> Col -> Color -> Int. Recording a bunch of computers over some time? Computer -> Frame -> Row ….

This is pretty common in constraint satisfaction! Like if you're conference trying to assign talks to talk slots, your array might be type (Day, Time, Room) -> Talk, where Day/Time/Room are enumerations.

An implementation constraint is that most programming languages only allow integer indexes, so we have to replace Rooms and Colors with numerical enumerations over the set. As long as the set is finite, this is always possible, and for struct-functions, we can always choose the indexing on the lexicographic ordering of the keys. But we lose type safety.

Why tables are different

One more example: Day -> Hour -> Airport(name: str, flights: int, revenue: USD). Can we turn the struct into a dimension like before?

In this case, no. We were able to make Color an axis because we could turn Pixel into a Color -> Int function, and we could only do that because all of the fields of the struct had the same type. This time, the fields are different types. So we can't convert {name, flights, revenue} into an axis. ⁴ One thing we can do is convert it to three separate functions:

airport: Day -> Hour -> Str
flights: Day -> Hour -> Int
revenue: Day -> Hour -> USD

But we want to keep all of the data in one place. That's where tables come in: an array-of-structs is isomorphic to a struct-of-arrays:

AirportColumns(
    airport: Day -> Hour -> Str,
    flights: Day -> Hour -> Int,
    revenue: Day -> Hour -> USD,
)

The table is a sort of both representations simultaneously. If this was a pandas dataframe, df["airport"] would get the airport column, while df.loc[day1] would get the first day's data. I don't think many table implementations support more than one axis dimension but there's no reason they couldn't.

These are also possible transforms:

Hour -> NamesAreHard(
    airport: Day -> Str,
    flights: Day -> Int,
    revenue: Day -> USD,
)

Day -> Whatever(
    airport: Hour -> Str,
    flights: Hour -> Int,
    revenue: Hour -> USD,
)

In my mental model, the heterogeneous struct acts as a "block" in the array. We can't remove it, we can only push an index into the fields or pull a shared column out. But there's no way to convert a heterogeneous table into an array.

Actually there is a terrible way

Most languages have unions or product sum types that let us say "this is a string OR integer". So we can make our airport data Day -> Hour -> AirportKey -> Int | Str | USD. Heck, might as well just say it's Day -> Hour -> AirportKey -> Any. But would anybody really be mad enough to use that in practice?

Oh wait J does exactly that. J has an opaque datatype called a "box". A "table" is a function Dim1 -> Dim2 -> Box. You can see some examples of what that looks like here

Misc Thoughts and Questions

The heterogeneity barrier seems like it explains why we don't see multiple axes of table columns, while we do see multiple axes of array dimensions. But is that actually why? Is there a system out there that does have multiple columnar axes?

The array x = [[a, b, a], [b, b, b]] has type 1..2 -> 1..3 -> {a, b}. Can we rearrange it to 1..2 -> {a, b} -> 1..3? No. But we can rearrange it to 1..2 -> {a, b} -> PowerSet(1..3), which maps rows and characters to columns with that character. [(a -> {1, 3} ++ b -> {2}), (a -> {} ++ b -> {1, 2, 3}].

We can also transform Row -> PowerSet(Col) into Row -> Col -> Bool, aka a boolean matrix. This makes sense to me as both forms are means of representing directed graphs.

Are other function combinators useful for thinking about arrays?

Does this model cover pivot tables? Can we extend it to relational data with multiple tables?

Systems Distributed Talk (will be) Online

The premier will be August 6 at 12 CST, here! I'll be there to answer questions / mock my own performance / generally make a fool of myself.

The excellent-but-defunct blog Programming in the 21st Century defines "puzzle languages" as languages were part of the appeal is in figuring out how to express a program idiomatically, like a puzzle. As examples, he lists Haskell, Erlang, and J. All puzzle languages, the author says, have an "escape" out of the puzzle model that is pragmatic but stigmatized.

But many mainstream languages have escape hatches, too.

Languages have a lot of properties. One of these properties is the language's capabilities, roughly the set of things you can do in the language. Capability is desirable but comes into conflicts with a lot of other desirable properties, like simplicity or efficiency. In particular, reducing the capability of a language means that all remaining programs share more in common, meaning there's more assumptions the compiler and programmer can make ("tractability"). Assumptions are generally used to reason about correctness, but can also be about things like optimization: J's assumption that everything is an array leads to high-performance "special combinations".

Rust is the most famous example of mainstream language that trades capability for tractability.¹ Rust has a lot of rules designed to prevent common memory errors, like keeping a reference to deallocated memory or modifying memory while something else is reading it. As a consequence, there's a lot of things that cannot be done in (safe) Rust, like interface with an external C function (as it doesn't have these guarantees).

To do this, you need to use unsafe Rust, which lets you do additional things forbidden by safe Rust, such as deference a raw pointer. Everybody tells you not to use unsafe unless you absolutely 100% know what you're doing, and possibly not even then.

Sounds like an escape hatch to me!

To extrapolate, an escape hatch is a feature (either in the language itself or a particular implementation) that deliberately breaks core assumptions about the language in order to add capabilities. This explains both Rust and most of the so-called "puzzle languages": they need escape hatches because they have very strong conceptual models of the language which leads to lots of assumptions about programs. But plenty of "kitchen sink" mainstream languages have escape hatches, too:

• Some compilers let C++ code embed inline assembly.
• Languages built on .NET or the JVM has some sort of interop with C# or Java, and many of those languages make assumptions about programs that C#/Java do not.
• The SQL language has stored procedures as an escape hatch and vendors create a second escape hatch of user-defined functions.
• Ruby lets you bypass any form of encapsulation with send.
• Frameworks have escape hatches, too! React has an entire page on them.

(Does eval in interpreted languages count as an escape hatch? It feels different, but it does add a lot of capability. Maybe they don't "break assumptions" in the same way?)

The problem with escape hatches

In all languages with escape hatches, the rule is "use this as carefully and sparingly as possible", to the point where a messy solution without an escape hatch is preferable to a clean solution with one. Breaking a core assumption is a big deal! If the language is operating as if its still true, it's going to do incorrect things.

I recently had this problem in a TLA+ contract. TLA+ is a language for modeling complicated systems, and assumes that the model is a self-contained universe. The client wanted to use the TLA+ to test a real system. The model checker should send commands to a test device and check the next states were the same. This is straightforward to set up with the IOExec escape hatch.² But the model checker assumed that state exploration was pure and it could skip around the state randomly, meaning it would do things like set x = 10, then skip to set x = 1, then skip back to inc x; assert x == 11. Oops!

We eventually found workarounds but it took a lot of clever tricks to pull off. I'll probably write up the technique when I'm less busy with The Book.

The other problem with escape hatches is the rest of the language is designed around not having said capabilities, meaning it can't support the feature as well as a language designed for them from the start. Even if your escape hatch code is clean, it might not cleanly integrate with the rest of your code. This is why people complain about unsafe Rust so often.

I'm a big (neo)vim buff. My config is over 1500 lines and I regularly write new scripts. I recently ported my neovim config to a new laptop. Before then, I was using VSCode to write, and when I switched back I immediately saw a big gain in productivity.

People often pooh-pooh vim (and other assistive writing technologies) by saying that writing code isn't the bottleneck in software development. Reading, understanding, and thinking through code is!

Now I don't know how true this actually is in practice, because empirical studies of time spent coding are all over the place. Most of them, like this study, track time spent in the editor but don't distinguish between time spent reading code and time spent writing code. The only one I found that separates them was this study. It finds that developers spend only 5% of their time editing. It also finds they spend 14% of their time moving or resizing editor windows, so I don't know how clean their data is.

But I have a bigger problem with "writing is not the bottleneck": when I think of a bottleneck, I imagine that no amount of improvement will lead to productivity gains. Like if a program is bottlenecked on the network, it isn't going to get noticeably faster with 100x more ram or compute.

But being able to type code 100x faster, even with without corresponding improvements to reading and imagining code, would be huge.

We'll assume the average developer writes at 80 words per minute, at five characters a word, for 400 characters a minute.What could we do if we instead wrote at 8,000 words/40k characters a minute?

Writing fast

Boilerplate is trivial

Why do people like type inference? Because writing all of the types manually is annoying. Why don't people like boilerplate? Because it's annoying to write every damn time. Programmers like features that help them write less! That's not a problem if you can write all of the boilerplate in 0.1 seconds.

You still have the problem of reading boilerplate heavy code, but you can use the remaining 0.9 seconds to churn out an extension that parses the file and presents the boilerplate in a more legible fashion.

We can write more tooling

This is something I've noticed with LLMs: when I can churn out crappy code as a free action, I use that to write lots of tools that assist me in writing good code. Even if I'm bottlenecked on a large program, I can still quickly write a script that helps me with something. Most of these aren't things I would have written because they'd take too long to write!

Again, not the best comparison, because LLMs also shortcut learning the relevant APIs, so also optimize the "understanding code" part. Then again, if I could type real fast I could more quickly whip up experiments on new apis to learn them faster.

We can do practices that slow us down in the short-term

Something like test-driven development significantly slows down how fast you write production code, because you have to spend a lot more time writing test code. Pair programming trades speed of writing code for speed of understanding code. A two-order-of-magnitude writing speedup makes both of them effectively free. Or, if you're not an eXtreme Programming fan, you can more easily follow the The Power of Ten Rules and blanket your code with contracts and assertions.

We could do more speculative editing

This is probably the biggest difference in how we'd work if we could write 100x faster: it'd be much easier to try changes to the code to see if they're good ideas in the first place.

How often have I tried optimizing something, only to find out it didn't make a difference? How often have I done a refactoring only to end up with lower-quality code overall? Too often. Over time it makes me prefer to try things that I know will work, and only "speculatively edit" when I think it be a fast change. If I could code 100x faster it would absolutely lead to me trying more speculative edits.

This is especially big because I believe that lots of speculative edits are high-risk, high-reward: given 50 things we could do to the code, 49 won't make a difference and one will be a major improvement. If I only have time to try five things, I have a 10% chance of hitting the jackpot. If I can try 500 things I will get that reward every single time.

Processes are built off constraints

There are just a few ideas I came up with; there are probably others. Most of them, I suspect, will share the same property in common: they change the process of writing code to leverage the speedup. I can totally believe that a large speedup would not remove a bottleneck in the processes we currently use to write code. But that's because those processes are developed work within our existing constraints. Remove a constraint and new processes become possible.

The way I see it, if our current process produces 1 Utils of Software / day, a 100x writing speedup might lead to only 1.5 UoS/day. But there are other processes that produce only 0.5 UoS/d because they are bottlenecked on writing speed. A 100x speedup would lead to 10 UoS/day.

The problem with all of this that 100x speedup isn't realistic, and it's not obvious whether a 2x improvement would lead to better processes. Then again, one of the first custom vim function scripts I wrote was an aid to writing unit tests in a particular codebase, and it lead to me writing a lot more tests. So maybe even a 2x speedup is going to be speed things up, too.

Patreon Stuff

I wrote a couple of TLA+ specs to show how to model fork-join algorithms. I'm planning on eventually writing them up for my blog/learntla but it'll be a while, so if you want to see them in the meantime I put them up on Patreon.

I released Logic for Programmers exactly one year ago today. It feels weird to celebrate the anniversary of something that isn't 1.0 yet, but software projects have a proud tradition of celebrating a dozen anniversaries before 1.0. I wanted to share about what's changed in the past year and the work for the next six+ months.

The Road to 0.1

I had been noodling on the idea of a logic book since the pandemic. The first time I wrote about it on the newsletter was in 2021! Then I said that it would be done by June and would be "under 50 pages". The idea was to cover logic as a "soft skill" that helped you think about things like requirements and stuff.

That version sucked. If you want to see how much it sucked, I put it up on Patreon. Then I slept on the next draft for three years. Then in 2024 a lot of business fell through and I had a lot of free time, so with the help of Saul Pwanson I rewrote the book. This time I emphasized breadth over depth, trying to cover a lot more techniques.

I also decided to self-publish it instead of pitching it to a publisher. Not going the traditional route would mean I would be responsible for paying for editing, advertising, graphic design etc, but I hoped that would be compensated by much higher royalties. It also meant I could release the book in early access and use early sales to fund further improvements. So I wrote up a draft in Sphinx, compiled it to LaTeX, and uploaded the PDF to leanpub. That was in June 2024.

Since then I kept to a monthly cadence of updates, missing once in November (short-notice contract) and once last month (Systems Distributed). The book's now on v0.10. What's changed?

A LOT

v0.1 was very obviously an alpha, and I have made a lot of improvements since then. For one, the book no longer looks like a Sphinx manual. Compare!

Also, the content is very, very different. v0.1 was 19,000 words, v.10 is 31,000.¹ This comes from new chapters on TLA+, constraint/SMT solving, logic programming, and major expansions to the existing chapters. Originally, "Simplifying Conditionals" was 600 words. Six hundred words! It almost fit in two pages!

The chapter is now 2600 words, now covering condition lifting, quantifier manipulation, helper predicates, and set optimizations. All the other chapters have either gotten similar facelifts or are scheduled to get facelifts.

The last big change is the addition of book assets. Originally you had to manually copy over all of the code to try it out, which is a problem when there are samples in eight distinct languages! Now there are ready-to-go examples for each chapter, with instructions on how to set up each programming environment. This is also nice because it gives me breaks from writing to code instead.

How did the book do?

Leanpub's all-time visualizations are terrible, so I'll just give the summary: 1180 copies sold, $18,241 in royalties. That's a lot of money for something that isn't fully out yet! By comparison, Practical TLA+ has made me less than half of that, despite selling over 5x as many books. Self-publishing was the right choice!

In that time I've paid about $400 for the book cover (worth it) and maybe $800 in Leanpub's advertising service (probably not worth it).

Right now that doesn't come close to making back the time investment, but I think it can get there post-release. I believe there's a lot more potential customers via marketing. I think post-release 10k copies sold is within reach.

Where is the book going?

The main content work is rewrites: many of the chapters have not meaningfully changed since 1.0, so I am going through and rewriting them from scratch. So far four of the ten chapters have been rewritten. My (admittedly ambitious) goal is to rewrite three of them by the end of this month and another three by the end of next. I also want to do final passes on the rewritten chapters; as most of them have a few TODOs left lying around.

(Also somehow in starting this newsletter and publishing it I realized that one of the chapters might be better split into two chapters, so there could well-be a tenth technique in v0.11 or v0.12!)

After that, I will pass it to a copy editor while I work on improving the layout, making images, and indexing. I want to have something worthy of printing on a dead tree by 1.0.

In terms of timelines, I am very roughly estimating something like this:

• Summer: final big changes and rewrites
• Early Autumn: graphic design and copy editing
• Late Autumn: proofing, figuring out printing stuff
• Winter: final ebook and initial print releases of 1.0.

(If you know a service that helps get self-published books "past the finish line", I'd love to hear about it! Preferably something that works for a fee, not part of royalties.)

This timeline may be disrupted by official client work, like a new TLA+ contract or a conference invitation.

Needless to say, I am incredibly excited to complete this book and share the final version with you all. This is a book I wished for years ago, a book I wrote because nobody else would. It fills a critical gap in software educational material, and someday soon I'll be able to put a copy on my bookshelf. It's exhilarating and terrifying and above all, satisfying.

I realize that for all I've talked about Logic for Programmers in this newsletter, I never once explained basic logical quantifiers. They're both simple and incredibly useful, so let's do that this week!

Sets and quantifiers

A set is a collection of unordered, unique elements. {1, 2, 3, …} is a set, as are "every programming language", "every programming language's Wikipedia page", and "every function ever defined in any programming language's standard library". You can put whatever you want in a set, with some very specific limitations to avoid certain paradoxes.²

Once we have a set, we can ask "is something true for all elements of the set" and "is something true for at least one element of the set?" IE, is it true that every programming language has a set collection type in the core language? We would write it like this:

# all of them
all l in ProgrammingLanguages: HasSetType(l)

# at least one
some l in ProgrammingLanguages: HasSetType(l)

This is the notation I use in the book because it's easy to read, type, and search for. Mathematicians historically had a few different formats; the one I grew up with was ∀x ∈ set: P(x) to mean all x in set, and ∃ to mean some. I use these when writing for just myself, but find them confusing to programmers when communicating.

"All" and "some" are respectively referred to as "universal" and "existential" quantifiers.

Some cool properties

We can simplify expressions with quantifiers, in the same way that we can simplify !(x && y) to !x || !y.

First of all, quantifiers are commutative with themselves. some x: some y: P(x,y) is the same as some y: some x: P(x, y). For this reason we can write some x, y: P(x,y) as shorthand. We can even do this when quantifying over different sets, writing some x, x' in X, y in Y instead of some x, x' in X: some y in Y. We can not do this with "alternating quantifiers":

• all p in Person: some m in Person: Mother(m, p) says that every person has a mother.
• some m in Person: all p in Person: Mother(m, p) says that someone is every person's mother.

Second, existentials distribute over || while universals distribute over &&. "There is some url which returns a 403 or 404" is the same as "there is some url which returns a 403 or some url that returns a 404", and "all PRs pass the linter and the test suites" is the same as "all PRs pass the linter and all PRs pass the test suites".

Finally, some and all are duals: some x: P(x) == !(all x: !P(x)), and vice-versa. Intuitively: if some file is malicious, it's not true that all files are benign.

All these rules together mean we can manipulate quantifiers almost as easily as we can manipulate regular booleans, putting them in whatever form is easiest to use in programming.

Speaking of which, how do we use this in in programming?

How we use this in programming

First of all, people clearly have a need for directly using quantifiers in code. If we have something of the form:

for x in list:
    if P(x):
        return true
return false

That's just some x in list: P(x). And this is a prevalent pattern, as you can see by using GitHub code search. It finds over 500k examples of this pattern in Python alone! That can be simplified via using the language's built-in quantifiers: the Python would be any(P(x) for x in list).

(Note this is not quantifying over sets but iterables. But the idea translates cleanly enough.)

More generally, quantifiers are a key way we express higher-level properties of software. What does it mean for a list to be sorted in ascending order? That all i, j in 0..<len(l): if i < j then l[i] <= l[j]. When should a ratchet test fail? When some f in functions - exceptions: Uses(f, bad_function). Should the image classifier work upside down? all i in images: classify(i) == classify(rotate(i, 180)). These are the properties we verify with tests and types and MISU and whatnot;¹ it helps to be able to make them explicit!

One cool use case that'll be in the book's next version: database invariants are universal statements over the set of all records, like all a in accounts: a.balance > 0. That's enforceable with a CHECK constraint. But what about something like all i, i' in intervals: NoOverlap(i, i')? That isn't covered by CHECK, since it spans two rows.

Quantifier duality to the rescue! The invariant is equivalent to !(some i, i' in intervals: Overlap(i, i')), so is preserved if the query SELECT COUNT(*) FROM intervals CROSS JOIN intervals … returns 0 rows. This means we can test it via a database trigger.³

There are a lot more use cases for quantifiers, but this is enough to introduce the ideas! Next week's the one year anniversary of the book entering early access, so I'll be writing a bit about that experience and how the book changed. It's crazy how crude v0.1 was compared to the current version.

I have a friend who exited his startup a few years ago and is now rich. How rich is unclear. One day, we were discussing ways to expedite the delivery of his superyacht and I suggested paying extra. His response, as to so many of my suggestions, was, “Avery, I’m not that rich.”

Everyone has their limit.

I, too, am not that rich. I have shares in a startup that has not exited, and they seem to be gracefully ticking up in value as the years pass. But I have to come to work each day, and if I make a few wrong medium-quality choices (not even bad ones!), it could all be vaporized in an instant. Meanwhile, I can’t spend it. So what I have is my accumulated savings from a long career of writing software and modest tastes (I like hot dogs).

Those accumulated savings and modest tastes are enough to retire indefinitely. Is that bragging? It was true even before I started my startup. Back in 2018, I calculated my “personal runway” to see how long I could last if I started a company and we didn’t get funded, before I had to go back to work. My conclusion was I should move from New York City back to Montreal and then stop worrying about it forever.

Of course, being in that position means I’m lucky and special. But I’m not that lucky and special. My numbers aren’t that different from the average Canadian or (especially) American software developer nowadays. We all talk a lot about how the “top 1%” are screwing up society, but software developers nowadays fall mostly in the top 1-2%[1] of income earners in the US or Canada. It doesn’t feel like we’re that rich, because we’re surrounded by people who are about equally rich. And we occasionally bump into a few who are much more rich, who in turn surround themselves with people who are about equally rich, so they don’t feel that rich either.

But, we’re rich.

Based on my readership demographics, if you’re reading this, you’re probably a software developer. Do you feel rich?

It’s all your fault

So let’s trace this through. By the numbers, you’re probably a software developer. So you’re probably in the top 1-2% of wage earners in your country, and even better globally. So you’re one of those 1%ers ruining society.

I’m not the first person to notice this. When I read other posts about it, they usually stop at this point and say, ha ha. Okay, obviously that’s not what we meant. Most 1%ers are nice people who pay their taxes. Actually it’s the top 0.1% screwing up society!

No.

I’m not letting us off that easily. Okay, the 0.1%ers are probably worse (with apologies to my friend and his chronically delayed superyacht). But, there aren’t that many of them[2] which means they aren’t as powerful as they think. No one person has very much capacity to do bad things. They only have the capacity to pay other people to do bad things.

Some people have no choice but to take that money and do some bad things so they can feed their families or whatever. But that’s not you. That’s not us. We’re rich. If we do bad things, that’s entirely on us, no matter who’s paying our bills.

What does the top 1% spend their money on?

Mostly real estate, food, and junk. If they have kids, maybe they spend a few hundred $k on overpriced university education (which in sensible countries is free or cheap).

What they don’t spend their money on is making the world a better place. Because they are convinced they are not that rich and the world’s problems are caused by somebody else.

When I worked at a megacorp, I spoke to highly paid software engineers who were torn up about their declined promotion to L4 or L5 or L6, because they needed to earn more money, because without more money they wouldn’t be able to afford the mortgage payments on an overpriced $1M+ run-down Bay Area townhome which is a prerequisite to starting a family and thus living a meaningful life. This treadmill started the day after graduation.[3]

I tried to tell some of these L3 and L4 engineers that they were already in the top 5%, probably top 2% of wage earners, and their earning potential was only going up. They didn’t believe me until I showed them the arithmetic and the economic stats. And even then, facts didn’t help, because it didn’t make their fears about money go away. They needed more money before they could feel safe, and in the meantime, they had no disposable income. Sort of. Well, for the sort of definition of disposable income that rich people use.[4]

Anyway there are psychology studies about this phenomenon. “What people consider rich is about three times what they currently make.” No matter what they make. So, I’ll forgive you for falling into this trap. I’ll even forgive me for falling into this trap.

But it’s time to fall out of it.

The meaning of life

My rich friend is a fountain of wisdom. Part of this wisdom came from the shock effect of going from normal-software-developer rich to founder-successful-exit rich, all at once. He described his existential crisis: “Maybe you do find something you want to spend your money on. But, I'd bet you never will. It’s a rare problem. Money, which is the driver for everyone, is no longer a thing in my life.”

Growing up, I really liked the saying, “Money is just a way of keeping score.” I think that metaphor goes deeper than most people give it credit for. Remember old Super Mario Brothers, which had a vestigial score counter? Do you know anybody who rated their Super Mario Brothers performance based on the score? I don’t. I’m sure those people exist. They probably have Twitch channels and are probably competitive to the point of being annoying. Most normal people get some other enjoyment out of Mario that is not from the score. Eventually, Nintendo stopped including a score system in Mario games altogether. Most people have never noticed. The games are still fun.

Back in the world of capitalism, we’re still keeping score, and we’re still weirdly competitive about it. We programmers, we 1%ers, are in the top percentile of capitalism high scores in the entire world - that’s the literal definition - but we keep fighting with each other to get closer to top place. Why?

Because we forgot there’s anything else. Because someone convinced us that the score even matters.

The saying isn’t, “Money is the way of keeping score.” Money is just one way of keeping score.

It’s mostly a pretty good way. Capitalism, for all its flaws, mostly aligns incentives so we’re motivated to work together and produce more stuff, and more valuable stuff, than otherwise. Then it automatically gives more power to people who empirically[5] seem to be good at organizing others to make money. Rinse and repeat. Number goes up.

But there are limits. And in the ever-accelerating feedback loop of modern capitalism, more people reach those limits faster than ever. They might realize, like my friend, that money is no longer a thing in their life. You might realize that. We might.

There’s nothing more dangerous than a powerful person with nothing to prove

Billionaires run into this existential crisis, that they obviously have to have something to live for, and money just isn’t it. Once you can buy anything you want, you quickly realize that what you want was not very expensive all along. And then what?

Some people, the less dangerous ones, retire to their superyacht (if it ever finally gets delivered, come on already). The dangerous ones pick ever loftier goals (colonize Mars) and then bet everything on it. Everything. Their time, their reputation, their relationships, their fortune, their companies, their morals, everything they’ve ever built. Because if there’s nothing on the line, there’s no reason to wake up in the morning. And they really need to want to wake up in the morning. Even if the reason to wake up is to deal with today’s unnecessary emergency. As long as, you know, the emergency requires them to do something.

Dear reader, statistically speaking, you are not a billionaire. But you have this problem.

So what then

Good question. We live at a moment in history when society is richer and more productive than it has ever been, with opportunities for even more of us to become even more rich and productive even more quickly than ever. And yet, we live in existential fear: the fear that nothing we do matters.[6][7]

I have bad news for you. This blog post is not going to solve that.

I have worse news. 98% of society gets to wake up each day and go to work because they have no choice, so at worst, for them this is a background philosophical question, like the trolley problem.

Not you.

For you this unsolved philosophy problem is urgent right now. There are people tied to the tracks. You’re driving the metaphorical trolley. Maybe nobody told you you’re driving the trolley. Maybe they lied to you and said someone else is driving. Maybe you have no idea there are people on the tracks. Maybe you do know, but you’ll get promoted to L6 if you pull the right lever. Maybe you’re blind. Maybe you’re asleep. Maybe there are no people on the tracks after all and you’re just destined to go around and around in circles, forever.

But whatever happens next: you chose it.

We chose it.

Footnotes

[1] Beware of estimates of the “average income of the top 1%.” That average includes all the richest people in the world. You only need to earn the very bottom of the 1% bucket in order to be in the top 1%.

[2] If the population of the US is 340 million, there are actually 340,000 people in the top 0.1%.

[3] I’m Canadian so I’m disconnected from this phenomenon, but if TV and movies are to be believed, in America the treadmill starts all the way back in high school where you stress over getting into an elite university so that you can land the megacorp job after graduation so that you can stress about getting promoted. If that’s so, I send my sympathies. That’s not how it was where I grew up.

[4] Rich people like us methodically put money into savings accounts, investments, life insurance, home equity, and so on, and only what’s left counts as “disposable income.” This is not the definition normal people use.

[5] Such an interesting double entendre.

[6] This is what AI doomerism is about. A few people have worked themselves into a terror that if AI becomes too smart, it will realize that humans are not actually that useful, and eliminate us in the name of efficiency. That’s not a story about AI. It’s a story about what we already worry is true.

[7] I’m in favour of Universal Basic Income (UBI), but it has a big problem: it reduces your need to wake up in the morning. If the alternative is bullshit jobs or suffering then yeah, UBI is obviously better. And the people who think that if you don’t work hard, you don’t deserve to live, are nuts. But it’s horribly dystopian to imagine a society where lots of people wake up and have nothing that motivates them. The utopian version is to wake up and be able to spend all your time doing what gives your life meaning. Alas, so far science has produced no evidence that anything gives your life meaning.

Alcatraz first operated as a prison in 1859, when the military fort first held convicted soldiers. The prison technology of the time was simple, consisting of little more than a basement room with a trap-door entrance. Only small numbers of prisoners were held in this period, but it established Alcatraz as a center of incarceration. Later, the Civil War triggered construction of a "political prison," a term with fewer negative connotations at the time, for confederate sympathizers.

This prison was more purpose-built (although actually a modification of an existing shop), but it was small and not designed for an especially high security level. It presaged, though, a much larger construction project to come.

Alcatraz had several properties that made it an attractive prison. First, it had seen heavy military construction as a Civil War defensive facility, but just decades later improvements in artillery made its fortifications obsolete. That left Alcatraz surplus property, a complete military installation available for new use. Second, Alcatraz was formidable. The small island was made up of steep rock walls, and it was miles from shore in a bay known for its strong currents. Escape, even for prisoners who had seized control of the island, would be exceptionally difficult.

These advantages were also limitations. Alcatraz was isolated and difficult to support, requiring a substantial roster of military personnel to ferry supplies back and forth. There were no connections to the mainland, requiring on-site power and water plants. Corrosive sea spray, sent over the island by the Bay's strong winds, lay perpetual siege on the island. Buildings needed constant maintenance, rust covered everything. Alcatraz was not just a famous prison, it was a particularly complicated one.

In 1909, Alcatraz lost its previous defensive role and pivoted entirely to military prison. The Citadel, a hardened barracks building dating to the original fortifications, was partially demolished. On top of it, a new cellblock was built. This was a purpose-built prison, designed to house several hundred inmates under high security conditions.

Unfortunately, few records seem to survive from the construction and operation of the cellblock as a disciplinary barracks. At some point, a manual telephone exchange was installed to provide service between buildings on the island. I only really know that because it was recorded as being removed later on. Communications to and from Alcatraz were a challenge. Radio and even light signals were used to convey messages between the island and other military installations on the bay. There was a constant struggle to maintain cables.

Early efforts to lay cables in the bay were less about communications and more about triggering. Starting in 1883, the Army Corps of Engineers began the installation of "torpedoes" in the San Francisco bay. These were different from what we think of as torpedoes today, they were essentially remotely-operated mines. Each device floated in the water by its own buoyancy, anchored to the bottom by a cable that then ran to shore. An electrical signal sent down the cable detonated the torpedo. The system was intended primarily to protect the bay from submarines, a new threat that often required technically complex defenses.

Submarines are, of course, difficult to spot. To make the torpedoes effective, the Army had to devise a targeting system. Observation posts on each side of the Golden Gate made sightings of possible submarines and reported them to a control post, where they were plotted on the map. With a threat confirmed, the control post would begin to detonate nearby torpedoes. A second set of observation posts, and a second line of torpedoes, were located further into the bay to address any submarines that made it through the first barrage.

By 1891, there were three such control points in total: Fort Mason, Angel Island, and Yerba Buena. The rather florid San Francisco Examiner of the day described the control point at Fort Mason, a "chamber of death and destruction" in a tunnel twenty feet underground. The Army "death-dealers" that manned the plotting table in that bunker had access to a board that "greatly resemble[d] the switch board in the great operating rooms of the telephone companies." By cords and buttons, they could select chains of mines and send the signal to fire.

NPS historians found that a torpedo control point had been planned at Alcatraz, and one of the fortifications modified to accommodate it, but never seems to have been used. The 1891 article gives a hint of the reason, noting that the line from Alcatraz to Fort Mason was "favorable for a line of torpedoes" but that currents were so strong that it was difficult to keep them anchored. Perhaps this problem was discovered after construction was already underway.

Somewhere around 1887-1888, the Army Signal Corps had joined the cable-laying fray. A telegraph cable was constructed from the Presidio to Alcatraz, and provided good service except for the many times that it was drug up by anchors and severed. This was a tremendous problem: in 1898, Gen. A. W. Greely of the Signal Corps called San Francisco the "worst bay in the country" for cable laying and said that no cable across the Golden Gate had lasted more than three years. The General attributed the problem mainly to the heavy shipping traffic, but I suspect that the notorious currents must have been a factor in just how many anchors were dragged through cables [1].

In 1889, a brand new Army telegraph cable was announced, one that would run from Alcatraz to Angel Island, and then from Angel Island to Marin County. An existing commercial cable crossed the Golden Gate, providing a connection all the way to the Presidio.

The many failures of Alcatraz cables makes it difficult to keep track. For example, a cable from Fort Mason to Alcatraz Island was apparently laid in 1891---but a few years later, it was lamented that Alcatraz's only cable connection to Fort Mason was indirect, via the 1889 Angel Island cable. Presumably the 1891 cable was damaged at some point and not replaced, but that event doesn't seem to have made the papers (or at least my search results!).

In 1900, a Signal Corps officer on Angel Island made a routine check of the cable to Alcatraz, finding it in good working order---but noticing that a "four masted schooner... in direct line with the cable" seemed to be in trouble just off the island and was being assisted by a tug. That evening, the officer returned to the cable landing box to find the ship gone... along with the cable. A French ship, "Lamoriciere," had drifted from anchor overnight. A Signal Corps sergeant, apparently having spoken with harbor officials, reported that the ship would have run completely aground had the anchor not caught the Alcatraz cable and pulled it taught. Of course the efforts of the tug to free Lamoriciere seems to have freed a little more than intended, and the cable was broken away from its landing. "Its end has been carried into the bay and probably quite a distance from land," the Signal Corps reported.

This ongoing struggle, of laying new cables to Alcatraz and then seeing them dragged away a few years later, has dogged the island basically to the modern day---when we have finally just given up. Today, as during many points in its history, Alcatraz must generate its own power and communicate with the mainland via radio.

When the Bureau of Prisons took control of Alcatraz in 1933, they installed entirely new radio systems. A marine AM radio was used to reach the Coast Guard, their main point of contact in any emergency. Another radio was used to contact "Alcatraz Landing" from which BOP ferries sailed, and over the years several radios were installed to permit direct communications with military installations and police departments around the Bay Area.

At some point, equipment was made available to connect telephone calls to the island. I'm not sure if this was manual patching by BOP or Coast Guard radio operators, or if a contract was made with PT&T to provide telephone service by radio. Such an arrangement seems to have been in place by 1937, when an unexplained distress call from the island made the warden impossible to contact (by the press or Bureau of Prisons) because "all lines [were] tied up."

Unfortunately I have not been able to find much on the radiotelephone arrangements. The BOP, no doubt concerned about security, did not follow the Army's habit of announcing new construction projects to the press. Fortunately, the BOP-era history of Alcatraz is much better covered by modern NPS documentation than the Army era (presumably because the more recent closure of the BOP prison meant that much of the original documentation was archived). Unfortunately, the NPS reports are mostly concerned with the history of the structures on the island and do not pay much attention to outside communications or the infrastructure that supported it.

Internal arrangements on the island almost completely changed when the BOP took over. The Army had left Alcatraz in a degree of disrepair (discussions about closing it having started by at least 1913), and besides, the BOP intended to provide a much higher level of security than the Army had. Extensive renovations were made of the main cellblock and many supporting buildings from 1933 to about 1939.

The 1930s had seen a great deal of innovation in technical security. Technologies like electrostatic and microwave motion sensors were available in early forms. On Alcatraz, though, the island was small and buildings tightly spaced. The prison staff, and in some cases their families, would be housed on the island just a stones throw from the cellblock. That meant there would be quite a few people moving around exterior to the prison, ruling out motion sensors as a means of escape detection. Exterior security would instead be provided by guard and dog patrols.

There was still some cutting-edge technical security when Alcatraz opened, including early metal detectors. At first, the BOP contracted the Teletouch Corporation of New York City. Teletouch, a manufactured burglar alarms and other electronic devices, was owned by or at least affiliated with famed electromagnetics inventor and Soviet spy Leon Theremin. Besides the instrument we remember him for today, Theremin had invented a number of devices for security applications, and the metal detectors were probably of his design. In practice, the Teletouch machines proved unsatisfactory. They were later replaced with machines made by Forewarn. I believe the metal detector on display today is one of the Forewarn products, although the NPS documents are a little unclear on this.

Sensitive common areas like the mess hall, kitchen, and sallyport wre fitted with electrically-activated teargas canisters. Originally, the mess hall teargas was controlled by a set of toggle switches in a corner gun gallery, while the sallyport teargas was controlled from the armory. While the teargas system was never used, it was probably the most radical of Alcatraz's technical security measures. As more electronic systems were installed, the armory, with its hardened vault entrance and gun issue window, served as a de facto control center for Alcatraz's initial security systems.

The Army's small manual telephone switchboard was considered unsuitable for the prison's use. The telephone system provided communication between the guards, making it a critical part of the overall security measures, and the BOP specified that all equipment and cabling needed to be better secured from any access by prisoners. Modifications to the cellblock building's entrance created a new room, just to the side of the sallyport, that housed a 100-line automatic exchange. Automatic Electric telephones that appear throughout historic photos of the prison would suggest that this exchange had been built by AE.

Besides providing dial service between prison offices and the many other structures on the island, the exchange was equipped with a conference circuit that included annunciator panels in each of the prison's main offices. Assuming this was the type provided by Automatic Electric, it provided an emergency communications system in which the guard telephones could ring all of the office and guard phones simultaneously, even interrupting calls already in progress. Annunciator panels in the armory and offices showed which phone had started the emergency conference, and which phones had picked up. From the armory, a siren on the building roof could be sounded to alert the entire island to any attempted escape.

Some locations, including the armory and the warden's office, were also fitted with fire annunciators. I am less clear on this system. Fire circuits similar to the previously described conference circuit (and sometimes called "crash alarms" after their use on airfields) were an optional feature on telephone exchanges of the time. Crash alarms were usually activated by dedicated "hotline" phones, and mentions of "emergency phones" in various prison locations support that this system worked the same way. Indeed, 1950s and 60s photos show a red phone alongside other telephones in several prison locations. The fire annunciator panels probably would have indicated which of the emergency phones had been lifted to initiate the alarm.

One of the most fascinating parts of Alcatraz, to a person like me, is the prison doors. Prison doors have a long history, one that is interrelated with but largely distinct from other forms of physical security. Take a look, for example, at the keys used in prisons. Prisons of the era, and even many today, rely on lever locks manufactured by specialty companies like Folger Adams and Sargent and Greenleaf. These locks are prized for their durability, and that extends to the keys, huge brass plates that could hold up to daily wear well beyond most locks.

At Alcatraz, the first warden adopted a "sterile area" model in which areas accessible to prisoners should be kept as clear as possible of dangerous items like guns and keys. Guards on the cellblock carried no keys, and cell doors lacked traditional locks. Instead, the cell doors were operated by a central mechanical system designed by Stewart Iron Works.

To let prisoners out of cells in the morning, a guard in the elevated gun gallery passed keys to a cellblock guard in a bucket or on a string. The guard unlocked the cabinet of a cell row's control system, revealing a set of large levers. The design is quite ingenious: by purely mechanical means, the guard could select individual cells or the entire row to be unlocked, and then by throwing the largest lever the guard could pull the cell doors open---after returning the necessary key to the gun gallery above. This 1934 system represents a major innovation in centralized access control, designed specifically for Alcatraz.

Stewart Iron Works is still in business, although not building prison doors. Some years ago, the company assisted NPS's work to restore the locking system to its original function. The present-day CEO provided replicas of the original Stewart logo plate for the restored locking cabinets. Interviewing him about the restoration work, the San Francisco Chronicle wrote that "Alcatraz, he believes, is part of the American experience."

The Stewart mechanical system seems to have remained in use on the B and C blocks until the prison closed, but the D block was either originally fitted, or later upgraded, with electrically locked cell doors. These were controlled from a set of switches in the gun gallery.

In 1960, the BOP launched another wave of renovations on Alcatraz, mostly to modernize its access and security arrangements to modern standards. The telephone exchange was moved away from the sallyport to an upper floor of the administration building, freeing up its original space for a new control center. This is the modern sallyport control area that visitors look into through the ballistic windows; the old service windows and viewports into the armory anteroom that had been the de facto control center are now removed.

This control center is more typical of what you will see in modern prisons. Through large windows, guards observed the sallyport and visitor areas and controlled the electrically operated main gates. An electrical interlock prevented opening the full path from the cellblock to the outside, creating a mantrap in the visitor area through which the guards in the control room could identify everyone entering and leaving.

Photos from the 1960 control room, and other parts of the prison around the same time, clearly show consoles for a Western Electric 507B PBX. The 507B is really a manual exchange, although it used keys rather than the more traditional plugboard for a more modern look. It dates back to about 1929---so I assume the 507B had been installed well before the 1960 renovation, and its appearance then is just a bias of more and better photos available from the prison's later days.

Fortunately, the NPS Historic Furnishings Report for the cellblock building includes a complete copy of a 1960s memo describing the layout and requirements for the control center. We're fortunate to get such a detailed listing of the equipment:

• Four phones (these are Automatic Electric instruments, based on the photo). One is a fire reporting phone (presumably on the exchange's "crash alarm" circuit), one is the watch call reporting phone (detailed in a moment), a regular outgoing call telephone, and an "executive right of way" phone that I assume will disconnect other calls from the outgoing trunks.
• The 507B PBX switchboard
• An intercom for communication with each of the guard towers
• Controls for five electrically operated doors
• Intercoms to each of the electrically operated doors (many of these are right outside of the control center, but the glass is very thick and you would not otherwise be able to converse)
• An "annunciator panel for the interior telephone system" which presumably combines the conference circuit, fire circuit, and watch call annunciators.
• An intercom to the visitor registration area
• A "paging intercom for group control purposes." I don't really know what that is, possibly it is for the public address speakers installed in many parts of the cellblock.
• Monitor speaker for the inmate radio system. This presumably allowed the control center to check the operation of the two-channel wired radio system installed in the cells.
• The "watch call answering device," discussed later.
• An indicator panel that shows any open doors in the D cell block (which is the higher security unit and the only one equipped with electrically locking cell doors).
• Two-way radio remote console
• Tear gas controls

Many of these are things we are already familiar with, but the watch call telephone system deserves some more discussion. It was clearly present back in the 1930s, but it wasn't clear to me what it actually did. Fortunately this memo gives some details on the operation.

Guards calling in to report their watch call extension 3331. This connects to the watch call answering device in the control center, which when enabled, automatically answers the call during the first ring. The answering device then allows a guard anywhere in the control center to converse with the caller via a loudspeaker and microphone. So, the watch call system is essentially just a speaker phone. This approach is probably a holdover from the 1930s system (older documents mention a watch call phone as well), and that would have been the early days for speakerphones, making it a somewhat specialized device. Clearly it made these routine watch calls a lot more convenient for the control center, especially since the guard there didn't even have to do anything to answer.

It might be useful to mention why this kind of system was used: I have never found any mention of two-way radios used on Alcatraz, and that's not surprising. Portable two-way radios were a nascent technology even in the 1960s---the handheld radio had basically been invented for the Second World War, and it took years for them to come down in size and price. If Alcatraz ever did issue radios to guards, it probably would have been in the last decade of operation. Instead, telephones were provided at enough places in the facility that guards could report their watch tour and any important events by finding a phone and calling the control center.

Guards were probably required to report their location at various points as they patrolled, so the control center would receive quite a few calls that were just a guard saying where they were---to be written down in a log by a control room guard, who no doubt appreciated not having to walk to a phone to hear these reports. This provided both the functions of a "guard tour" system, ensuring that guards were actually performing their rounds, and improved the safety of guards by making it likely that the control center would notice fairly promptly that they had stopped reporting in.

Alcatraz closed as a BOP prison in 1963, and after a surprising number of twists and turns ranging from plans to develop a shopping center to occupation by the Indians of All Tribes, Alcatraz opened to tourists. Most technology past this point might not be considered "historic," having been installed by NPS for operational purposes. I can't help but mention, though, that there were more attempts at a cable. For the NPS, operating the power plant at Alcatraz was a significant expense that they would much rather save.

The idea of a buried power cable isn't new. I have seen references, although no solid documentation, that the BOP laid a power cable in 1934. They built a new power plant in 1939 and operated it for the rest of the life of the prison, so either that cable failed and was never replaced, or it never existed at all...

I should take a moment here to mention that LLM-generated "AI slop" has become a pervasive and unavoidable problem around any "hot SEO topic" like tourism. Unfortunately the history of tourist sites like Alcatraz has become more and more difficult to learn as websites with well-researched history are displaced in search results by SEO spam---articles that often contains confident but unsourced and often incorrect information. This has always been a problem but it has increased by orders of magnitude over the last couple of years, and it seems that the LLM-generated articles are more likely to contain details that are outright made up than the older human-generated kind. It's really depressing. That's basically all I have to say about it.

It seems that a power cable was installed to Alcatraz sometime in the 1960s but failed by about 1971. I'm a little skeptical of that because that was the era in which it was surplus GSA property, making such a large investment an odd choice, so maybe the 1980s article with that detail is wrong or confusing power with one of the several telephone cables that seem to have been laid (and failed) during BOP operations). In any case, in late 1980 or early 1981, Paul F. Pugh and Associates of Oakland designed a novel type of underwater power cable for the NPS. It was expected to provide power to Alcatraz at much reduced cost compared to more traditional underwater power cable technologies. It never even made it to day 1: after the cable was laid, but before commissioning, some failure caused a large span of it to float to the surface. The cable was evidently not repairable, and it was pulled back to shore.

'I don't know where we go from here,' William J. Whalen, superintendent of the Golden Gate National Recreation Area, said after the broken cable was hauled in.

We do know now: where the NPS went from there was decades of operating two diesel generators on the island, until a 2017 DoE-sponsored project that installed solar panels on the cellblock building roof. The panels were intentionally installed such that they are not visible anywhere from the ground, preserving the historic integrity of the site. In aerial photos, though, they give Alcatraz a curiously modern look. The DoE calls the project, which incorporates battery storage and backup diesel generators, as "one of the largest microgrids in the United States." That is an interesting framing, one that emphasizes the modern valance of "microgrid," since Alcatraz had been a self-sufficient electrical system since the island's first electric lights. But what's old is, apparently, new again.

I originally wrote much of this as part of a larger travelogue on my most recent trip to Alcatraz, which was coincidentally the same day as a visit by Pam Bondi and Doug Burgum to "survey" the prison for potential reopening. That piece became long and unwieldy, so I am breaking it up into more focused articles---this one on the technical history, a travelogue about the experience of visiting the island in this political context and its history as a symbol of justice and retribution, and probably a third piece on the way that the NPS interprets the site today. I am pitching the travelogue itself to other publications so it may not have a clear fate for a while, but if it doesn't appear here I'll let you know where. In any case there probably will be a loose part two to look forward to.

[1] Greely had a rather illustrious Army career. His term as chief of the Signal Corps was something of a retirement after he led several arctic expeditions, the topic of his numerous popular books and articles. He received the Medal of Honor shortly before his death in 1935.

A long time ago I wrote about secret government telephone numbers, and before that, secret military telephone buttons. I suppose this is becoming a series. To be clear, the "secret" here is a joke, but more charitably I could say that it refers to obscurity rather than any real effort to keep them secret. Actually, today's examples really make this point: they're specifically intended to be well known, but are still pretty obscure in practice.

If you've been around for a while, you know how much I love telephone numbers. Here in North America, we have a system called the North American Numbering Plan (NANP) that has rigidly standardized telephone dialing practices since the middle of the 20th century. The US, Canada, and a number of Central American countries benefit from a very orderly system of area codes (more formally numbering plan areas or NPAs) followed by a subscriber number written in the format NXX-XXXX (this is a largely NANP-centric notation for describing phone number patterns, N represents the digits 2-9 and X any digit). All of these NANP numbers reside under the country code 1, allowing at least theoretically seamless international dialing within the NANP community. It's really a pretty elegant system.

NANP is the way it is for many reasons, but it mostly reflects technical requirements of the telephone exchanges of the 1940s. This is more thoroughly explained in the link above, but one of the goals of NANP is to ensure that step-by-step (SxS) exchanges can process phone numbers digit by digit as they are dialed. In other words, it needs to be possible to navigate the decision tree of telephone routing using only the digits dialed so far.

Readers with a computer science education might have some tidy way to describe this in terms of Chompsky or something, but I do not have a computer science education; I have an Information Technology education. That means I prefer flow charts to automata, and we can visualize a basic SxS exchange as a big tree. When you pick up your phone, you start at the root of the tree, and each digit dialed chooses the edge to follow. Eventually you get to a leaf that is hopefully someone's telephone, but at no point in the process does any node benefit from the context of digits you dial before, after, or how many total digits you dial. This creates all kinds of practical constraints, and is the reason, for example, that we tend to write ten-digit phone numbers with a "1" before them.

That requirement was in some ways long-lived (The last SxS exchange on the public telephone network was retired in 1999), and in other ways not so long lived... "common control" telephone exchanges, which did store the entire number in electromechanical memory before making a routing decision, were already in use by the time the NANP scheme was adopted. They just weren't universal, and a common nationwide numbering scheme had to be designed to accommodate the lowest common denominator.

This discussion so far is all applicable to the land-line telephone. There is a whole telephone network that is, these days, almost completely separate but interconnected: cellular phones. Early cellular phones (where "early" extends into CDMA and early GSM deployments) were much more closely attached to the "POTS" (Plain Old Telephone System). AT&T and Verizon both operated traditional telephone exchanges, for example 5ESS, that routed calls to and from their customers. These telephone exchanges have become increasingly irrelevant to mobile telephony, and you won't find a T-Mobile ESS or DMS anywhere. All US cellular carriers have adopted the GSM technology stack, and GSM has its own definition of the switching element that can be, and often is, fulfilled by an AWS EC2 instance running RHEL 8. Calls between cell phones today, even between different carriers, are often connected completely over IP and never touch a traditional telephone exchange.

The point is that not only is telephone number parsing less constrained on today's telephone network, in the case of cellular phones, it is outright required to be more flexible. GSM also defines the properties of phone numbers, and it is a very loose definition. Keep in mind that GSM is deeply European, and was built from the start to accommodate the wide variety of dialing practices found in Europe. This manifests in ways big and small; one of the notable small ways is that the European emergency number 112 works just as well as 911 on US cell phones because GSM dictates special handling for emergency numbers and dictates that 112 is one of those numbers. In fact, the definition of an "emergency call" on modern GSM networks is requesting a SIP URI of "urn:service:sos". This reveals that dialed number handling on cellular networks is fundamentally different.

When you dial a number on your cellular phone, the phone collects the entire number and then applies a series of rules to determine what to do, often leading to a GSM call setup process where the entire number, along with various flags, is sent to the network. This is all software-defined. In the immortal words of our present predicament, "everything's computer."

The bottom line is that, within certain regulatory boundaries and requirements set by GSM, cellular carriers can do pretty much whatever they want with phone numbers. Obviously numbers need to be NANP-compliant to be carried by the POTS, but many modern cellular calls aren't carried by the POTS, they are completed entirely within cellular carrier systems through their own interconnection agreements. This freedom allows all kinds of things like "HD voice" (cellular calls connected without the narrow filtering and companding used by the traditional network), and a lot of flexibility in dialing.

Most people already know about some weird cellular phone numbers. For example, you can dial *#06# to display your phone's various serial numbers. This is an example of a GSM MMI (man-machine interface) code, phone numbers that are handled entirely within your device but nonetheless defined as dialable numbers by GSM for compatibility with even the most basic flip phones. GSM also defined numbers called USSD for unstructured supplementary service data, which set up connections to the network that can be used in any arbitrary way the network pleases. Older prepaid phone services used to implement balance check and top-up operations using USSD numbers, and they're also often used in ways similar to Vertical Service Codes (VSCs) on the landline network to control carrier features. USSDs also enabled the first forms of mobile data, which involved a "special telephone call" to a USSD in order to download a cut-down form of ESPN in a weird mobile-specific markup language.

Now, put yourself in the shoes of an enterprising cellular network. The flexibility of processing phone numbers as you please opens up all kinds of possibilities. Innovative services! Customer convenience! Sell them for money! Oh my god, sell them for money!

It seems like this started with customer service. It is an old practice, dating to the Bell operating companies, to have special short phone numbers to reach the telephone company itself. The details varied by company (often based on technical constraints in their switching system), but a common early setup was that dialing 114 got you the repair service operator to report a problem with your phone line. These numbers were usually listed in the front of the phone book, and for the phone company the fact that they were "special" or nonstandard was sort of a feature, since they could ensure that they were always routed within the same switch. The selection of "911" as the US emergency number seems rooted in this practice, as later on several major telcos used the "N11" numbers for their service lines. This became immortalized in the form of 611, which will get you customer service for most phone carriers.

So cellular companies did the same, allocating themselves "special" numbers for various service lines. Verizon offers #PMT to make a payment. Naturally, there's also room for upsell services: #ROAD for roadside assistance on Verizon.

The odd thing about these phone numbers is that there's really no standard involved, they're just the arbitrary practices of specific cellular companies. The term "mobile dial code" (MDC) is usually used to refer to them, although that term seems to have arisen organically rather than by intent. Remember, these aren't a real thing! The carriers just make them up, all on their own.

The only real constraint on MDCs is that they need to not collide with any POTS number, which is most easily achieved by prefixing them with some combination of * and #, and usually not "*#" because it's referenced by the GSM standard for MMI.

MDCs are available for purchase, but the terms don't seem to be public and you have to negotiate separately with each carrier. That's because there is no centralization. This is where MDCs stand in clear contrast to the better known SMS Short Code, or SMSSC. Those are the five or six-digit numbers widely used in advertising campaigns.

SMSSCs are centrally managed by the SMS Short Code Registry, which is a function of industry association CTIA but contracted to iConectiv. iConectiv is sort of like the SAIC of the communications industry, a huge company that dates back to the Bell System (where it became Bellcore after divestiture) and that no one has heard of but nonetheless is a critically important part of the telephone system.

Providers that want to have an SMSSC (typically on behalf of one of their customers) pay a fee, and usually recoup it from the end user. That fee is not cheap, typical end-user rates for an SMSSC run over $10k a year. But at least it's straightforward, and your SMS A2P or marketing company can make it happen for you.

MDCs have no such centralization, no standardized registration process. You negotiate with each carrier individually. That means it's pretty difficult to put together "complete coverage" on an MDC by getting the same one assigned by every major carrier. And this is one of those areas where "good enough" is seldom good enough; people get pissed off when something you advertise doesn't work. Putting a phone number that only works for some people on a billboard can quickly turn into an expensive embarrassment, so companies will be wary of using an MDC in marketing if they don't feel really confident that it works for the vast majority of cellphone users.

Because of this fragmentation, adoption of MDCs for marketing purposes has been very low. The only going concern I know of is #250, operated by a company called Mobile Direct Response. The premise of #250 is very simple: users call #250 and are greeted by a simple IVR. They say a keyword, and they're either forwarded to the phone number of the business that paid for the keyword or they receive a text message response with more information. #250 is specifically oriented towards radio advertising, where asking people to remember a ten-digit phone number is, well, asking a lot. It's also made the jump to podcast advertising. #250 is priced in a very radio-centric way, by the keyword and the size of the market area in which the advertisement that gives the keyword is played.

#250 was founded by Dave Robinett, who used to work on marketing at Sprint, presumably where he became aware that these MDCs were a possibility. He has negotiated for #250 to work across a substantial list of cellular carriers in the US and Canada, providing almost complete coverage. That wasn't easy, Robinett said in an interview that it took five years to get AT&T, T-Mobile, Verizon, and Sprint on board.

#250 does not appear to be especially widely used. For one, the website is a little junky, with some broken links and other indications that it is not backed by a large communications department. Dave Robinett may be the entire company. They've been operating since at least 2017, and I've only ever heard it in an ad once---a podcast ad that ended with "Call #250 and say I need a dentist." One thing you quickly notice when you look into telephone marketing is that dentists are apparently about 80% of the market. He does mention success with shows like "Rush, Hannity, and Levin," so it's safe to say that my radio habits are a little different from Robinett's.

That's not to say that #250 is a failure. In the same interview Robinett says that the company pays his mortgage and, well, that ain't too bad. But it's also nothing like the widespread adoption of SMSSCs. One wonders if the limitation of MDCs to one company that is so focused on radio marketing limits their potential. It might really open things up if some company created a registration service, and prenegotiated terms with carriers so that companies could pick up their own MDCs to use as they please.

Well, yeah, someone's trying. Around 2006, a recently-founded mobile marketing company called Zoove announced StarStar dialing. I'm a little unclear on Zoove's history. It seems that they were originally founded as Teleractive in Rhode Island as an SMS short code keyword response service, and after an infusion of VC cash moved to Palo Alto and started looking for something bigger. In 2016, they were acquired by a call center technology company called Mindful. Or maybe Zoove sold the StarStar business to Mindful? Stick a pin in that.

I don't love the name StarStar, which has shades of Spacestar Ordering. But it refers to their chosen MDC prefix, two stars. Well, that point is a little odd, according to their marketing material you can also get numbers with a # prefix or * prefix, but all of the examples use **. I would say that, in general, StarStar has it a little less together than #250. Their website is kind of broken, it only loads intermittently and some of the images are missing. At one point it uses the term "CADC" to describe these numbers but I can't find that expanded anywhere. Plus the "About" page refers repeatedly to Virtual Hold Technologies, which renamed to VHT in 2018 and Mindful 2022. It really feels like the vestigial website of a dead company.

I know about StarStar because, for a time, trucks from moving franchise All My Sons prominently bore the number **MOVE on the side. Indeed, this is still one of the headline examples on the StarStar website, but it doesn't work. I just get a loud click and then the call ends. And it's not that StarStar doesn't work with my mobile carrier, because StarStar's own number **MOBILE does connect to their IVR. That IVR promises that a representative will speak with me shortly, plays about five seconds of hold music, and then dumps me on a voicemail system. Despite StarStar numbers apparently basically working, I'm finding that most of the examples they give on their website won't even connect. Perhaps results will vary depending on the mobile network.

Well, perhaps not that much is lost. StarStar was founded by Steve Doumar, a serial telephone marketing entrepreneur with a colorful past founding various inbound call center companies. Perhaps his most famous venture is R360, a "lead acquisition" service memorialized by headlines like "Drug treatment referral service took advantage of addictions to make a quick buck" from the Federal Trade Commission. He's one of those guys whose bio involves founding a new company every two years, which he has to spin as entrepreneurial dynamism rather than some combination of fleeing dissatisfied investors and fleeing angered regulators.

Today he runs whisp.io, a "customer activation platform" that appears to be a glorified SMS advertising service featuring something ominously called "simplified opt-in." Whisp has a YouTube channel which features the 48-second gem "Fun Fact We Absolutely Love About Steve Doumar". Description:

Our very own CEO, Steve Doumar is a kind and generous person who has given back to the community in many ways; this man is absolutely a man with a heart of gold.

Do you want to know the fun fact? Yes you do! Here it is: "He is an incredible philanthropist. He loves helping other people. Every time I'm with him he comes up with new ways and new ideas to help other people. Which I think is amazing. And he doesn't brag about it, he doesn't talk about it a lot." Except he's got his CMO making a YouTube video about it?

From Steve Doumar's blog:

American entrepreneur Ray Kroc expressed the importance of persisting in a busy world where everyone wants a bite of success.

This man is no exception.

An entrepreneur. A family man. A visionary.

These are the many names of a man that has made it possible for opt-ins to be safe, secure, and accurate; Steve Doumar.

I love this stuff, you just can't make it up. I'm pretty sure what's going on here is just an SEO effort to outrank the FTC releases and other articles about the R360 case when you search for his name. It's only partially working, "FTC Hits R360 and its Owner With $3.8 Million Civil ..." still comes in at Google result #4 for "Steve Doumar," at least for me. But hey, #4 is better than #1.

Well, to be fair to StarStar, I don't think Steve Doumar has been involved for some years, but also to be fair, some of their current situation clearly dates to past behavior that is maybe less than savory.

Zoove originally styled itself as "The National StarStar Registry," clearly trying to draw parallels to CTIA/iConectiv's SMSSC registry. Their largest customer was evidently a company called Sumotext, which leased a number of StarStar numbers to offer an SMS and telephone marketing service. In 2016, Sumotext sued StarStar, Zoove, VHT (now Mindful), and a healthy list of other entities all involved in StarStar including the intriguingly named StarSteve LLC. I'm not alone in finding the corporate history a little baffling; in a footnote on one ruling the court expressed confusion about all the different names and opted to call them all Zoove.

In any case, Sumotext alleged that Zoove, StarSteve, and VHT all merged as part of a scheme to illegally monopolize the StarStar market by undercutting the companies that had been leasing the numbers and effectively giving VHT (Mindful) an exclusive ability to offer marketing services with StarStar numbers. The case didn't end up going anywhere for Sumotext, the jury found that Sumotext hadn't established a relevant market which is a key part of a Sherman act case. An appeal was made all the way to the Supreme Court, but they didn't take it up. What the case did do was publicize some pretty sketchy sounding details, like the seemingly uncontested accusation that VHT got Sumotext's customer list from the registry database and used it to convert them all into StarSteve customers.

And yes, the Steve in StarSteve is Steve Doumar. As best I can tell, the story here is that Steve Doumar founded Zoove (or bought Teleractive and renamed it or something?) to establish the National StarStar Registry, then founded a marketing company called StarSteve that resold StarStar numbers, then merged StarSteve and the National StarStar Registry together and cut off all of the other resellers. Apparently not a Sherman act violation but it sure is a bad look, and I wonder how much it contributed to the lack of adoption of the whole StarStar idea---especially given that Sumotext seems to have been responsible for most of that adoption, including the All My Sons deal for **MOVE. I wonder if All My Sons had to take **MOVE off of their trucks because of the whole StarSteve maneuver? That seems to be what happened.

Look, ten-digit phone numbers are had to remember, that much is true. But as is, the "MDC" industry doesn't seem stable enough for advertising applications where the number needs to continue to work into the future. I think the #250 service is probably here to stay, but confined to the niche of audio advertising. StarStar raised at least $30 million in capital in the 2010s, but seems to have shot itself in the foot. StarStar owner VHT/Mindful, now acquired by Medallia, doesn't even mention StarStar as a product offering.

Hey, remember how Steve Doumar is such a great philanthropist? There are a lot of vestiges around of StarStar Inc., a nonprofit that made StarStar numbers available to charitable organizations. Their website, starstar.org, is now a Wix error page. You can find old articles about StarStar Me, also written **me, which sounds lewd but was a $3/mo offering that allowed customers to get a vanity short code (such as ** followed by their name)---the original form of StarStar, dating back to 2012 and the beginning of Zoove.

In a press release announcing the StarStar Me, Zoove CEO Joe Gillespie said:

With two-thirds of smartphone users having downloaded social networking apps to their phones, there’s a rapidly growing trend in today's on-the-go lifestyle to extend our personal communications and identity into the digital realm via our mobile phones.

And somehow this leads to paying $3 for to get StarStarred? I love it! It's so meaningless! And years later it would be StarStar Mobile formerly Zoove by VHT now known as Mindful a Medallia company. Truly an inspiring story of industry, and just one little corner of the vast tapestry of phone numbers.

Some time ago, via a certain orange website, I came across a report about a mission to recover nuclear material from a former Soviet test site. I don't know what you're doing here, go read that instead. But it brought up a topic that I have only known very little about: Hydronuclear testing.

One of the key reasons for the nonproliferation concern at Semipalatinsk was the presence of a large quantity of weapons grade material. This created a substantial risk that someone would recover the material and either use it directly or sell it---either way giving a significant leg up on the construction of a nuclear weapon. That's a bit odd, though, isn't it? Material refined for use in weapons in scarce and valuable, and besides that rather dangerous. It's uncommon to just leave it lying around, especially not hundreds of kilograms of it.

This material was abandoned in place because the nature of the testing performed required that a lot of weapons-grade material be present, and made it very difficult to remove. As the Semipalatinsk document mentions in brief, similar tests were conducted in the US and led to a similar abandonment of special nuclear material at Los Alamos's TA-49. Today, I would like to give the background on hydronuclear testing---the what and why. Then we'll look specifically at LANL's TA-49 and the impact of the testing performed there.

First we have to discuss the boosted fission weapon. Especially in the 21st century, we tend to talk about "nuclear weapons" as one big category. The distinction between an "A-bomb" and an "H-bomb," for example, or between a conventional nuclear weapon and a thermonuclear weapon, is mostly forgotten. That's no big surprise: thermonuclear weapons have been around since the 1950s, so it's no longer a great innovation or escalation in weapons design.

The thermonuclear weapon was not the only post-WWII design innovation. At around the same time, Los Alamos developed a related concept: the boosted weapon. Boosted weapons were essentially an improvement in the efficiency of nuclear weapons. When the core of a weapon goes supercritical, the fission produces a powerful pulse of neutrons. Those neutrons cause more fission, the chain reaction that makes up the basic principle of the atomic bomb. The problem is that the whole process isn't fast enough: the energy produced blows the core apart before it's been sufficiently "saturated" with neutrons to completely fission. That leads to a lot of the fuel in the core being scattered, rather than actually contributing to the explosive energy.

In boosted weapons, a material that will fusion is added to the mix, typically tritium and deuterium gas. The immense heat of the beginning of the supercritical stage causes the gas to undergo fusion, and it emits far more neutrons than the fissioning fuel does alone. The additional neutrons cause more fission to occur, improving the efficiency of the weapon. Even better, despite the theoretical complexity of driving a gas into fusion¸ the mechanics of this mechanism are actually simpler than the techniques used to improve yield in non-boosted weapons (pushers and tampers).

The result is that boosted weapons produce a more powerful yield in comparison to the amount of fuel, and the non-nuclear components can be made simpler and more compact as well. This was a pretty big advance in weapons design and boosting is now a ubiquitous technique.

It came with some downsides, though. The big one is that whole property of making supercriticality easier to achieve. Early implosion weapons were remarkably difficult to detonate, requiring an extremely precisely timed detonation of the high explosive shell. While an inconvenience from an engineering perspective, the inherent difficulty of achieving a nuclear yield also provided a safety factor. If the high explosives detonated for some unintended reason, like being struck by canon fire as a bomber was intercepted, or impacting the ground following an accidental release, it wouldn't "work right." Uneven detonation of the shell would scatter the core, rather than driving it into supercriticality.

This property was referred to as "one point safety:" a detonation at one point on the high explosive assembly should not produce a nuclear yield. While it has its limitations, it became one of the key safety principles of weapon design.

The design of boosted weapons complicated this story. Just a small fission yield, from a small fragment of the core, could potentially start the fusion process and trigger the rest of the core to detonate as well. In other words, weapon designers became concerned that boosted weapons would not have one point safety. As it turns out, two-stage thermonuclear weapons, which were being fielded around the same time, posed a similar set of problems.

The safety problems around more advanced weapon designs came to a head in the late '50s. Incidentally, so did something else: shifts in Soviet politics had given Khrushchev extensive power over Soviet military planning, and he was no fan of nuclear weapons. After some on-again, off-again dialog between the time's nuclear powers, the US and UK agreed to a voluntary moratorium on nuclear testing which began in late 1958.

For weapons designers this was, of course, a problem. They had planned to address the safety of advanced weapon designs through a testing campaign, and that was now off the table for the indefinite future. An alternative had to be developed, and quickly.

In 1959, the Hydronuclear Safety Program was initiated. By reducing the amount of material in otherwise real weapon cores, physicists realized they could run a complete test of the high explosive system and observe its effects on the core without producing a meaningful nuclear yield. These tests were dubbed "hydronuclear," because of the desire to observe the behavior of the core as it flowed like water under the immense explosive force. While the test devices were in some ways real nuclear weapons, the nuclear yield would be vastly smaller than the high explosive yield, practically nill.

Weapons designers seemed to agree that these experiments complied with the spirit of the moratorium, being far from actual nuclear tests, but there was enough concern that Los Alamos went to the AEC and President Eisenhower for approval. They evidently agreed, and work started immediately to identify a suitable site for hydronuclear testing.

While hydronuclear tests do not create a nuclear yield, they do involve a lot of high explosives and radioactive material. The plan was to conduct the tests underground, where the materials cast off by the explosion would be trapped. This would solve the immediate problem of scattering nuclear material, but it would obviously be impractical to recover the dangerous material once it was mixed with unstable soil deep below the surface. The material would stay, and it had to stay put!

The US Army Corps of Engineers, a center of expertise in hydrology because of their reclamation work, arrived in October 1959 to begin an extensive set of studies on the Frijoles Mesa site. This was an unused area near a good road but far on the east edge of the laboratory, well separated from the town of Los Alamos and pretty much anything else. More importantly, it was a classic example of northern New Mexican geology: high up on a mesa built of tuff and volcanic sediments, well-drained and extremely dry soil in an area that received little rain.

One of the main migration paths for underground contaminants is their interaction with water, and specifically the tendency of many materials to dissolve into groundwater and flow with it towards aquifers. The Corps of Engineers drilled test wells, about 1,500' deep, and a series of 400' core samples. They found that on the Frijoles Mesa, ground water was over 1,000' below the surface, and that everything above was far from saturation. That means no mobility of the water, which is trapped in the soil. It's just about the ideal situation for putting something underground and having it stay.

Incidentally, this study would lead to the development of a series of new water wells for Los Alamos's domestic water supply. It also gave the green light for hydronuclear testing, and Frijoles Mesa was dubbed Technical Area 49 and subdivided into a set of test areas. Over the following three years, these test areas would see about 35 hydronuclear detonations carried out in the bottom of shafts that were about 200' deep and 3-6' wide.

It seems that for most tests, the hole was excavated and lined with a ladder installed to reach the bottom. Technicians worked at the bottom of the hole to prepare the test device, which was connected by extensive cabling to instrumentation trailers on the surface. When the "shot" was ready, the hole was backfilled with sand and sealed at the top with a heavy plate. The material on top of the device held everything down, preventing migration of nuclear material to the surface. The high explosives did, of course, destroy the test device and the cabling, but not before the instrumentation trailers had recorded a vast amount of data.

If you read these kinds of articles, you must know that the 1958 moratorium did not last. Soviet politics shifted again, France began nuclear testing, negotiations over a more formal test ban faltered. US intelligence suspected that the Soviet Union had operated their nuclear weapons program at full tilt during the test ban, and the military suspected clandestine tests, although there was no evidence they had violated the treaty. Of course, that they continued their research efforts is guaranteed, we did as well. Physicist Edward Teller, ever the nuclear weapons hawk, opposed the moratorium and pushed to resume testing.

In 1961, the Soviet Union resumed testing, culminating in the test of the record-holding "Tsar Bomba," a 50 megaton device. The US resumed testing as well. The arms race was back on.

US hydronuclear testing largely ended with the resumption of full-scale testing. The same safety studies could be completed on real weapons, and those tests would serve other purposes in weapons development as well. Although post-moratorium testing included atmospheric detonations, the focus had shifted towards underground tests and the 1963 Partial Test Ban Treaty restricted the US and USSR to underground tests only.

One wonders about the relationship between hydronuclear testing at TA-49 and the full-scale underground tests extensively performed at the NTS. Underground testing began in 1951 with Buster-Jangle Uncle, a test to determine how big of a crater could be produced by a ground-penetrating weapon. Uncle wasn't really an underground test in the modern sense, the device was emplaced only 17 feet deep and still produced a huge cloud of fallout. It started a trend, though: a similar 1955 test was set 67 feet deep, producing a spectacular crater, before the 1957 Plumbbob Pascal-A was detonated at 486 feet and produced radically less fallout.

1957's Plumbbob Rainier was the first fully-contained underground test, set at the end of a tunnel excavated far into a hillside. This test emitted no fallout at all, proving the possibility of containment. Thus both the idea of emplacing a test device in a deep hole, and the fact that testing underground could contain all of the fallout, were known when the moratorium began in 1959.

What's very interesting about the hydronuclear tests is the fact that technicians actually worked "downhole," at the bottom of the excavation. Later underground tests were prepared by assembling the test device at the surface, as part of a rocket-like "rack," and then lowering it to the bottom just before detonation. These techniques hadn't yet been developed in the '50s, thus the use of a horizontal tunnel for the first fully-contained test.

Many of the racks used for underground testing were designed and built by LANL, but others (called "canisters" in an example of the tendency of the labs to not totally agree on things) were built by Lawrence Livermore. I'm not actually sure which of the two labs started building them first, a question for future research. It does seem likely that the hydronuclear testing at LANL advanced the state of the art in remote instrumentation and underground test design, facilitating the adoption of fully-contained underground tests in the following years.

During the three years of hydronuclear testing, shafts were excavated in four testing areas. It's estimated that the test program at TA-49 left about 40kg of plutonium and 93kg of enriched uranium underground, along with 92kg of depleted uranium and 13kg of beryllium (both toxic contaminants). Because of the lack of a nuclear yield, these tests did not create the caverns associated with underground testing. Material from the weapons likely spread within just a 10-20' area, as holes were drilled on a 25' grid and contamination from previous neighboring tests was encountered only once.

The tests also produced quite a bit of ancillary waste: things like laboratory equipment, handling gear, cables and tubing, that are not directly radioactive but were contaminated with radioactive or toxic materials. In the fashion typical of the time, this waste was buried on site, often as part of the backfilling of the test shafts.

During the excavation of one of the test shafts, 2-M in December 1960, contamination was detected at the surface. It seems that the geology allowed plutonium from a previous test to spread through cracks into the area where 2-M was being drilled. The surface soil contaminated by drill cuttings was buried back in hole 2-M, but this incident made area 2 the most heavily contaminated part of TA-49. When hydronuclear testing ended in 1961, area 2 was covered by a 6' of gravel and 4-6" of asphalt to better contain any contaminated soil.

Several support buildings on the surface were also contaminated, most notably a building used as a radiochemistry laboratory to support the tests. An underground calibration facility that allowed for exposure of test equipment to a contained source in an underground chamber was also built at TA-49 and similarly contaminated by use with radioisotopes.

The Corps of Engineers continued to monitor the hydrology of the site from 1961 to 1970, and test wells and soil samples showed no indication that any contamination was spreading. In 1971, LANL established a new environmental surveillance department that assumed responsibility for legacy sites like TA-49. That department continued to sample wells, soil, and added air sampling. Monitoring of stream sediment downhill from the site was added in the '70s, as many of the contaminants involved can bind to silt and travel with surface water. This monitoring has not found any spread either.

That's not to say that everything is perfect. In 1975, a section of the asphalt pad over Area 2 collapsed, leaving a three foot deep depression. Rainwater pooled in the depression and then flowed through the gravel into hole 2-M itself, collecting in the bottom of the lining of the former experimental shaft. In 1976, the asphalt cover was replaced, but concerns remained about the water that had already entered 2-M. It could potentially travel out of the hole, continue downwards, and carry contamination into the aquifer around 800' below. Worse, a nearby core sample hole had picked up some water too, suggesting that the water was flowing out of 2-M through cracks and into nearby features. Since the core hole had a slotted liner, it would be easier for water to leave it and soak into the ground below.

In 1980, the water that had accumulated in 2-M was removed by lifting about 24 gallons to the surface. While the water was plutonium contaminated, it fell within acceptable levels for controlled laboratory areas. Further inspections through 1986 did not find additional water in the hole, suggesting that the asphalt pad was continuing to function correctly. Several other investigations were conducted, including the drilling of some additional sample wells and examination of other shafts in the area, to determine if there were other routes for water to enter the Area 2 shafts. Fortunately no evidence of ongoing water ingress was found.

In 1986, TA-49 was designated a hazardous waste site under the Resource Conservation and Recovery Act. Shortly after, the site was evaluated under CERCLA to prioritize remediation. Scoring using the Hazard Ranking System determined a fairly low risk for the site, due to the lack of spread of the contamination and evidence suggesting that it was well contained by the geology.

Still, TA-49 remains an environmental remediation site and now falls under a license granted by the New Mexico Environment Department. This license requires ongoing monitoring and remediation of any problems with the containment. For example, in 1991 the asphalt cover of Area 2 was found to have cracked and allowed more water to enter the sample wells. The covering was repaired once again, and investigations made every few years from 1991 to 2015 to check for further contamination. Ongoing monitoring continues today. So far, Area 2 has not been found to pose an unacceptable risk to human health or a risk to the environment.

NMED permitting also covers the former radiological laboratory and calibration facility, and infrastructure related to them like a leach field from drains. Sampling found some surface contamination, so the affected soil was removed and disposed of at a hazardous waste landfill where it will be better contained.

TA-49 was reused for other purposes after hydronuclear testing. These activities included high explosive experiments contained in metal "bottles," carried out in a metal-lined pit under a small structure called the "bottle house." Part of the bottle house site was later reused to build a huge hydraulic ram used to test steel cables at their failure strength. I am not sure of the exact purpose of this "Cable Test Facility," but given the timeline of its use during the peak of underground testing and the design I suspect LANL used it as a quality control measure for the cable assemblies used in lowering underground test racks into their shafts. No radioactive materials were involved in either of these activities, but high explosives and hydraulic oil can both be toxic, so both were investigated and received some surface soil cleanup.

Finally, the NMED permit covers the actual test shafts. These have received numerous investigations over the sixty years since the original tests, and significant contamination is present as expected. However, that contamination does not seem to be spreading, and modeling suggests that it will stay that way.

In 2022, the NMED issued Certificates of Completion releasing most of the TA-49 remediation sites without further environmental controls. The test shafts themselves, known to NMED by the punchy name of Solid Waste Management Unit 49-001(e), received a certificate of completion that requires ongoing controls to ensure that the land is used only for industrial purposes. Environmental monitoring of the TA-49 site continues under LANL's environmental management program and federal regulation, but TA-49 is no longer an active remediation project. The plutonium and uranium is just down there, and it'll have to stay.

To ensure that several services could only be invoked by trusted parties, someone at Ricardo P's employer had the brilliant idea of requiring a token along with each request. Before servicing a request, they added this check:

private bool IsValidToken(string? token)
{
    if (string.Equals("xxxxxxxx-xxxxxx+xxxxxxx+xxxxxx-xxxxxx-xxxxxx+xxxxx", token)) return true;
    return false;
}

The token is anonymized here, but it's hard-coded into the code, because checking security tokens into source control, and having tokens that never expire has never caused anyone any trouble.

Which, in the company's defense, they did want the token to expire. The problem there is that they wanted to be able to roll out the new token to all of their services over time, which meant the system had to be able to support both the old and new token for a period of time. And you know exactly how they handled that.

private bool IsValidToken(string? token)
{
    if (string.Equals("xxxxxxxx-xxxxxx+xxxxxxx+xxxxxx-xxxxxx-xxxxxx+xxxxx", token)) return true;
    else if (string.Equals("yyyyyyy-yyyyyy+yyyyy+yyyyy-yyyyy-yyyyy+yyyy", token)) return true;
    return false;
}

For a change, I'm more mad about this insecurity than the if(cond) return true pattern, but boy, I hate that pattern.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

The Standard Template Library for C++ is… interesting. A generic set of data structures and algorithms was a pretty potent idea. In practice, early implementations left a lot to be desired. Because the STL is a core part of C++ at this point, and widely used, it also means that it's slow to change, and each change needs to go through a long approval process.

Which is why the STL didn't have a std::map::containsfunction until the C++20 standard. There were other options. For example, one could usestd::map::count, to count how many times a key appear. Or you could use std::map::findto search for a key. One argument against adding astd::map::containsfunction is thatstd::map::count basically does the same job and has the same performance.

None of this stopped people from adding their own. Which brings us to Gaetan's submission. Absent a std::map::contains method, someone wrote a whole slew of fieldExists methods, where field is one of many possible keys they might expect in the map.

bool DataManager::thingyExists (string name)
{
    THINGY* l_pTHINGY = (*m_pTHINGY)[name];
    if(l_pTHINGY == NULL)
    {
        m_pTHINGY->erase(name);
        return false;
    }
        else
    {
        return true;
    }
    return false;
}

I've head of upsert operations- an update and insert as the same operation, but this is the first exert- an existence check and an insert in the same operation.

"thingy" here is anonymization. The DataManager contained several of these methods, which did the same thing, but checked a different member variable. Other classes, similar to DataManager had their own implementations. In truth, the original developer did a lot of "it's a class, but everything inside of it is stored in a map, that's more flexible!"

In any case, this code starts by using the [] accessor on a member variable m_pTHINGY. This operator returns a reference to what's stored at that key, or if the key doesn't exist inserts a default-constructed instance of whatever the map contains.

What the map contains, in this case, is a pointer to a THINGY, so the default construction of a pointer would be null- and that's what they check. If the value is null, then we erase the key we just inserted and return false. Otherwise, we return true. Otherotherwise, we return false.

As a fun bonus, if someone intentionally stored a null in the map, this will think the key doesn't exist and as a side effect, remove it.

Gaetan writes:

What bugs me most is the final, useless return.

I'll be honest, what bugs me most is the Hungarian notation on local variables. But I'm long established as a Hungarian notation hater.

This code at least works, which compared to some bad C++, puts it on a pretty high level of quality. And it even has some upshots, according to Gaetan:

On the bright side: I have obtained easy performance boosts by performing that kind of cleanup lately in that particular codebase.

Or cold. It's getting hot and cold. But on average... no. It's absolutely unbelievable.

"There's been a physics breakthrough!" Mate exclaimed. "Looking at meteoblue, I should probably reconsider that hike on Monday." Yes, you should blow it off, but you won't need to.

An anonymous fryfan frets "The yellow arches app (at least in the UK) is a buggy mess, and I'm amazed it works at all when it does. Whilst I've heard of null, it would appear that they have another version of null, called ullnullf! Comments sent to their technical team over the years, including those with good reproduceable bugs, tend to go unanswered, unfortunately."

Llarry A. whipped out his wallet but baffled "I tried to pay in cash, but I wasn't sure how much."

"Github goes gonzo!" groused Gwenn Le Bihan. "Seems like Github's LLM model broke containment and error'd all over the website layout. crawling out of its grouped button." Gross.

Peter G. gripes "The text in the image really says it all." He just needs to rate his experience above 7 in order to enable the submit button.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Mads introduces today's code sample with this line: " this was before they used git to track changes".

Note, this is not to say that they were using SVN, or Mercurial, or even Visual Source Safe. They were not using anything. How do I know?

/**
  * Converts HTML to PDF using HTMLDOC.
  * 
  * @param printlogEntry
  ** @param inBytes
  *            html.
  * @param outPDF
  *            pdf.
  * @throws IOException
  *             when error.
  * @throws ParseException
*/
public void fromHtmlToPdfOld(PrintlogEntry printlogEntry, byte[] inBytes, final OutputStream outPDF) throws IOException, ParseException
	{...}

/**
 * Converts HTML to PDF using HTMLDOC.
 * 
 * @param printlogEntry
 ** @param inBytes
 *            html.
 * @param outPDF
 *            pdf.
 * @throws IOException
 *             when error.
 * @throws ParseException
 */
public void fromHtmlToPdfNew(PrintlogEntry printlogEntry, byte[] inBytes, final OutputStream outPDF) throws IOException, ParseException
	{...}

Originally, the function was just called fromHtmlToPdf. Instead of updating the implementation, or using it as a wrapper to call the correct implementation, they renamed it to Old, added one named New, then let the compiler tell them where they needed to update the code to use the new implementation.

Mads adds: "And this is just one example in this code. This far, I have found 5 of these."

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

I am on record as not particularly loving JSON as a serialization format. It's fine, and I'm certainly not going to die on any hills over it, but I think that as we stripped down the complexity of XML we threw away too much.

On the flip side, the simplicity means that it's harder to use it wrong. It's absent many footguns.

Well, one might think. But then Hootentoot ran into a problem. You see, an internal partner needed to send them a JSON document which contains a JSON document. Now, one might say, "isn't any JSON object a valid sub-document? Can't you just nest JSON inside of JSON all day? What could go wrong here?"

"value":"[{\"value\":\"1245\",\"begin_datum\":\"2025-05-19\",\"eind_datum\":null},{\"value\":\"1204\",\"begin_datum\":\"2025-05-19\",\"eind_datum\":\"2025-05-19\"}]",

This. This could go wrong. They embedded JSON inside of JSON… as a string.

Hootentoot references the hottest memes of a decade and a half ago to describe this Xzibit:

Yo dawg, i heard you like JSON, so i've put some JSON in your JSON

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

"This keeps giving me a primary key violation!" complained one of Nancy's co-workers. "Screw it, I'm dropping the primary key constraint!"

That was a terrifying thing to hear someone say out loud. Nancy decided to take a look at the table before anyone did anything they'd regret.

CREATE TYPE record_enum AS ENUM('parts');
CREATE TABLE IF NOT EXISTS parts (
    part_uuid VARCHAR(40) NOT NULL,
    record record_enum NOT NULL,
    ...
    ...
    ...
    PRIMARY KEY (part_uuid, record)
);

This table has a composite primary key. The first is a UUID, and the second is an enum with only one option in it- the name of the table. The latter column seems, well, useless, and certainly isn't going to make the primary key any more unique. But the UUID column should be unique. Universally unique, even.

Nancy writes:

Was the UUID not unique enough, or perhaps it was too unique?! They weren't able to explain why they had designed the table this way.

Nor were they able to explain why they kept violating the primary key constraint. It kept happening to them, for some reason until eventually it stopped happening, also for some reason.

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

Adam's organization was going through a period of rapid growth. Part of this growth was spinning up new backend services to support new functionality. The growth would have been extremely fast, except for one thing applying back pressure: for some reason, spinning up a new service meant recompiling and redeploying all the other services.

Adam didn't understand why, but it seemed like an obvious place to start poking at something for improvement. All of the services depended on a library called "ServiceLib"- though not all of them actually used the library. The library was a set of utilities for administering, detecting, and interacting with services in their environment- essentially a homegrown fabric/bus architecture.

It didn't take long, looking at the source control history, to understand why there was a rebuild after the release of every service. Each service triggered a one line change in this:

enum class Services
{
    IniTechBase = 103,
    IniTechAdvanced = 99,
    IniTechFooServer = 102,
    …
}

Each service had a unique, numerical identifier, and this mapped them into an enumerated type.

Adam went to the tech lead, Raymond. "Hey, I've got an idea for speeding up our release process- we should stop hard coding the service IDs in ServiceLib."

Raymond looked at Adam like one might examine an over-enthusiastic lemur. "They're not hard-coded. We store them in an enum."

Eventually Raymond got promoted- for all of their heroic work on managing this rapidly expanding library of services. The new tech lead who came on was much more amenable to "not storing rapidly changing service IDs in an enum", and "not making every service depend on a library they often don't need", and "putting admin functionality in every service because they're linked to that library whether they like it or not."

Eventually, ServiceLib became its own service, and actually helped- instead of hindered- delivering new functionality.

Unfortunately, with no more highly visible heroics to deliver functionality, the entire department became a career dead end. Sure, they delivered on time and under budget consistently, but there were no rockstar developers like Raymond on the team anymore, the real up-and-comers who were pushing themselves.

Michael R. is back with receipts. "I have been going to Tayyabs for >20 years. In the past they only accepted cash tips. Good to see they are testing a new way now."

An anonymous murmers of Outlook 365: "I appreciate being explicit about the timezone for the appointments, but I am wondering how those \" got there. (And the calender in german should start on Monday not Sunday)"

"Only my friends call me {0}," complains Alejandro D. "But wait! I haven't logged in yet, how does DHL know my name?"

"Prices per square foot are through the roof," puns Mr. TA "In fact, I'm guessing 298 sq ft is the area of the kitchen cabinets alone." The price isn't so bad, it's the condo fees that will kill you.

TheRealSteveJudge writes "Have a look at the cheapest ticket price which is available for a ride of 45 km from Duisburg to Xanten -- Günstiger Ticketpreis in German. That's really affordable!" If you've just purchased a 298 ft^2 condo at the Ritz.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Misha has a co-worker who has unusual ideas about how database performance works. This co-worker, Ted, has a vague understanding that a SQL query optimizer will attempt to find the best execution path for a given query. Unfortunately, Ted has just enough knowledge to be dangerous; he believes that the job of a developer is to write SQL queries that will "trick" the optimizer into doing an even better job, somehow.

This means that Ted loves subqueries.

For example, let's say you had a table called tbl_updater, which is used to store pending changes for a batch operation that will later get applied. Each change in updater has a unique change key that identifies it. For reasons best not looked into too deeply, at some point in the lifecycle of a record in this table, the application needs to null out several key fields based on the change value.

If you or I were writing this, we might do something like this:

update tbl_updater set id = null, date = null, location = null, type = null, type_id = null
where change = @change

And this is how you know that you and I are fools, because we didn't use a single subquery.

update tbl_updater set id = null where updater in
        (select updater from tbl_updater where change = @change)

update tbl_updater set date = null where updater in
        (select updater from tbl_updater where change = @change)

update tbl_updater set location = null where updater in
        (select updater from tbl_updater where change = @change)
       
update tbl_updater set type = null where updater in
        (select updater from tbl_updater where change = @change)
       
update tbl_updater set date = null where updater in
        (select updater from tbl_updater where change = @change)
       
update tbl_updater set type_id = null where updater in
        (select updater from tbl_updater where change = @change)

So here, Ted uses where updater in (subquery) which is certainly annoying and awkward, given that we know that change is a unique key. Maybe Ted didn't know that? Of course, one of the great powers of relational databases is that they offer data dictionaries so you can review the structure of tables before writing queries, so it's very easy to find out that the key is unique.

But that simple ignorance doesn't explain why Ted broke it out into multiple updates. If insanity is doing the same thing again and again expecting different results, what does it mean when you actually do get different results but also could have just done all this once?

Misha asked Ted why he took this approach. "It's faster," he replied. When Misha showed benchmarks that proved it emphatically wasn't faster, he just shook his head. "It's still faster this way."

Faster than what? Misha wondered.

Carlos and Claire found themselves supporting a 3rd party logistics package, called IniFreight. Like most "enterprise" software, it was expensive, unreliable, and incredibly complicated. It had also been owned by four different companies during the time Carlos had supported it, as its various owners underwent a series of acquisitions. It kept them busy, which is better than being bored.

One day, Claire asked Carlos, "In SQL, what does an exclamation point mean?"

"Like, as a negation? I don't think most SQL dialects support that."

"No, like-" and Claire showed him the query.

select * from valuation where origin_country < '!'

"IniFreight, I presume?" Carlos asked.

"Yeah. I assume this means, 'where origin country isn't blank?' But why not just check for NOT NULL?"

The why was easy to answer: origin_country had a constraint which prohibited nulls. But the input field didn't do a trim, so the field did allow whitespace only strings. The ! is the first printable, non-whitespace character in ASCII (which is what their database was using, because it was built before "support wide character sets" was a common desire).

Unfortunately, this means that my micronation, which is simply spelled with the ASCII character 0x07 will never show up in their database. You might not think you're familiar with my country, but trust me- it'll ring a bell.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

Alistair sends us a pretty big blob of code, but it's a blob which touches upon everyone's favorite design pattern: the singleton. It's a lot of Java code, so we're going to take this as chunks. Let's start with the two methods responsible for constructing the object.

The purpose of this code is to parse an XML file, and construct a mapping from a "name" field in the XML to a "batch descriptor".

	/**
	 * Instantiates a new batch manager.
	 */
	private BatchManager() {
		try {
			final XMLReader xmlReader = XMLReaderFactory.createXMLReader();
			xmlReader.setContentHandler(this);
			xmlReader.parse(new InputSource(this.getClass().getClassLoader().getResourceAsStream("templates/" + DOCUMENT)));
		} catch (final Exception e) {
			logger.error("Error parsing Batch XML.", e);
		}
	}

	/*
	 * (non-Javadoc)
	 * 
	 * @see nz.this.is.absolute.crap.sax.XMLEntity#initChild(java.lang.String,
	 * java.lang.String, java.lang.String, org.xml.sax.Attributes)
	 */
	@Override
	protected ContentHandler initChild(String uri, String localName,
			String qName, Attributes attributes) throws SAXException {
		final BatchDescriptor batchDescriptor = new BatchDescriptor();
		// put it in the map
		batchMap.put(attributes.getValue("name"), batchDescriptor);
		return batchDescriptor;
	}

Here we see a private constructor, which is reasonable for a singleton. It creates a SAX based reader. SAX is event driven- instead of loading the whole document into a DOM, it emits an event as it encounters each new key element in the XML document. It's cumbersome to use, but far more memory efficient, and I'd hardly say this.is.absolute.crap, but whatever.

This code is perfectly reasonable. But do you know what's unreasonable? There's a lot more code, and these are the only things not marked as static. So let's keep going.

	// singleton instance so that static batch map can be initialised using
	// xml
	/** The Constant singleton. */
	@SuppressWarnings("unused")
	private static final Object singleton = new BatchManager();

Wait… why is the singleton object throwing warnings about being unused? And wait a second, what is that comment saying, "so the static batch map can be initalalised"? I saw a batchMap up in the initChild method above, but it can't be…

	private static Map<String, BatchDescriptor> batchMap = new HashMap<String, BatchDescriptor>();

Oh. Oh no.

	/**
	 * Gets the.
	 * 
	 * @param batchName
	 *            the batch name
	 * 
	 * @return the batch descriptor
	 */
	public static BatchDescriptor get(String batchName) {
		return batchMap.get(batchName);
	}

	/**
	 * Gets the post to selector name.
	 * 
	 * @param batchName
	 *            the batch name
	 * 
	 * @return the post to selector name
	 */
	public static String getPostToSelectorName(String batchName) {
		final BatchDescriptor batchDescriptor = batchMap.get(batchName);
		if (batchDescriptor == null) {
			return null;
		}
		return batchDescriptor.getPostTo();
	}

There are more methods, and I'll share the whole code at the end, but this gives us a taste. Here's what this code is actually doing.

It creates a static Map. static, in this context, means that this instance is shared across all instances of BatchManager.They also create a static instance of BatchManager inside of itself. The constructor of that instance then executes, populating that static Map. Now, when anyone invokes BatchManager.get it will use that static Map to resolve that.

This certainly works, and it offers a certain degree of cleanness in its implementation. A more conventional singleton would have the Map being owned by an instance, and it's just using the singleton convention to ensure there's only a single instance. This version's calling convention is certainly nicer than doing something like BatchManager.getInstance().get(…), but there's just something unholy about this that sticks into me.

I can't say for certain if it's because I just hate Singletons, or if it's this specific abuse of constructors and static members.

This is certainly one of the cases of misusing a singleton- it does not represent something there can be only one of, it's ensuring that an expensive computation is only allowed to be done once. There are better ways to handle that lifecycle. This approach also forces that expensive operation to happen at application startup, instead of being something flexible that can be evaluated lazily. It's not wrong to do this eagerly, but building something that can only do it eagerly is a mistake.

In any case, the full code submission follows:

package nz.this.is.absolute.crap.server.template;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.ResourceBundle;

import nz.this.is.absolute.crap.KupengaException;
import nz.this.is.absolute.crap.SafeComparator;
import nz.this.is.absolute.crap.sax.XMLEntity;
import nz.this.is.absolute.crap.selector.Selector;
import nz.this.is.absolute.crap.selector.SelectorItem;
import nz.this.is.absolute.crap.server.BatchValidator;
import nz.this.is.absolute.crap.server.Validatable;
import nz.this.is.absolute.crap.server.ValidationException;
import nz.this.is.absolute.crap.server.business.BusinessObject;
import nz.this.is.absolute.crap.server.database.EntityHandler;
import nz.this.is.absolute.crap.server.database.SQLEntityHandler;

import org.apache.log4j.Logger;
import org.xml.sax.Attributes;
import org.xml.sax.ContentHandler;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.xml.sax.XMLReader;
import org.xml.sax.helpers.XMLReaderFactory;

/**
 * The Class BatchManager.
 */
public class BatchManager extends XMLEntity {

	private static final Logger logger = Logger.getLogger(BatchManager.class);
	
	/** The Constant DOCUMENT. */
	private final static String DOCUMENT = "Batches.xml";

	/**
	 * The Class BatchDescriptor.
	 */
	public class BatchDescriptor extends XMLEntity {

		/** The batchSelectors. */
		private final Collection<String> batchSelectors = new ArrayList<String>();

		/** The dependentCollections. */
		private final Collection<String> dependentCollections = new ArrayList<String>();

		/** The directSelectors. */
		private final Collection<String> directSelectors = new ArrayList<String>();

		/** The postTo. */
		private String postTo;

		/** The properties. */
		private final Collection<String> properties = new ArrayList<String>();

		/**
		 * Gets the batch selectors iterator.
		 * 
		 * @return the batch selectors iterator
		 */
		public Iterator<String> getBatchSelectorsIterator() {
			return this.batchSelectors.iterator();
		}

		/**
		 * Gets the dependent collections iterator.
		 * 
		 * @return the dependent collections iterator
		 */
		public Iterator<String> getDependentCollectionsIterator() {
			return this.dependentCollections.iterator();
		}

		/**
		 * Gets the post to.
		 * 
		 * @return the post to
		 */
		public String getPostTo() {
			return this.postTo;
		}

		/**
		 * Gets the post to business object.
		 * 
		 * @param businessObject
		 *            the business object
		 * @param postHandler
		 *            the post handler
		 * 
		 * @return the post to business object
		 * 
		 * @throws ValidationException
		 *             the validation exception
		 */
		private BusinessObject getPostToBusinessObject(
				BusinessObject businessObject, EntityHandler postHandler)
				throws ValidationException {
			if (this.postTo == null) {
				return null;
			}
			final BusinessObject postToBusinessObject = businessObject
					.getBusinessObjectFromMap(this.postTo, postHandler);
			// copy properties
			for (final String propertyName : this.properties) {
				String postToPropertyName;
				if ("postToStatus".equals(propertyName)) {
					// status field on batch entity refers to the batch entity
					// itself
					// so postToStatus is used for updating the status property
					// of the postToBusinessObject itself
					postToPropertyName = "status";
				} else {
					postToPropertyName = propertyName;
				}
				final SelectorItem destinationItem = postToBusinessObject
						.find(postToPropertyName);
				if (destinationItem != null) {
					final Object oldValue = destinationItem.getValue();
					final Object newValue = businessObject.get(propertyName);
					if (SafeComparator.areDifferent(oldValue, newValue)) {
						destinationItem.setValue(newValue);
					}
				}
			}
			// copy direct selectors
			for (final String selectorName : this.directSelectors) {
				final SelectorItem destinationItem = postToBusinessObject
						.find(selectorName);
				if (destinationItem != null) {
					// get the old and new values for the selectors
					Selector oldSelector = (Selector) destinationItem
							.getValue();
					Selector newSelector = (Selector) businessObject
							.get(selectorName);
					// strip them down to bare identifiers for comparison
					if (oldSelector != null) {
						oldSelector = oldSelector.getAsIdentifier();
					}
					if (newSelector != null) {
						newSelector = newSelector.getAsIdentifier();
					}
					// if they're different then update
					if (SafeComparator.areDifferent(oldSelector, newSelector)) {
						destinationItem.setValue(newSelector);
					}
				}
			}
			// copy batch selectors
			for (final String batchSelectorName : this.batchSelectors) {
				final Selector batchSelector = (Selector) businessObject
						.get(batchSelectorName);
				if (batchSelector == null) {
					throw new ValidationException(
							"\"PostTo\" selector missing.");
				}
				final BusinessObject batchObject = postHandler
						.find(batchSelector);
				if (batchObject != null) {
					// get the postTo selector for the batch object we depend on
					final BatchDescriptor batchDescriptor = batchMap
							.get(batchObject.getName());
					if (batchDescriptor.postTo != null
							&& postToBusinessObject
									.containsKey(batchDescriptor.postTo)) {
						final Selector realSelector = batchObject
								.getBusinessObjectFromMap(
										batchDescriptor.postTo, postHandler);
						postToBusinessObject.put(batchDescriptor.postTo,
								realSelector);
					}
				}
			}
			businessObject.put(this.postTo, postToBusinessObject);
			return postToBusinessObject;
		}

		/*
		 * (non-Javadoc)
		 * 
		 * @see
		 * nz.this.is.absolute.crap.sax.XMLEntity#initChild(java.lang.String,
		 * java.lang.String, java.lang.String, org.xml.sax.Attributes)
		 */
		@Override
		protected ContentHandler initChild(String uri, String localName,
				String qName, Attributes attributes) throws SAXException {
			if ("Properties".equals(qName)) {
				return new XMLEntity() {
					@Override
					protected ContentHandler initChild(String uri,
							String localName, String qName,
							Attributes attributes) throws SAXException {
						BatchDescriptor.this.properties.add(attributes
								.getValue("name"));
						return null;
					}
				};
			} else if ("DirectSelectors".equals(qName)) {
				return new XMLEntity() {
					@Override
					protected ContentHandler initChild(String uri,
							String localName, String qName,
							Attributes attributes) throws SAXException {
						BatchDescriptor.this.directSelectors.add(attributes
								.getValue("name"));
						return null;
					}
				};
			} else if ("BatchSelectors".equals(qName)) {
				return new XMLEntity() {
					@Override
					protected ContentHandler initChild(String uri,
							String localName, String qName,
							Attributes attributes) throws SAXException {
						BatchDescriptor.this.batchSelectors.add(attributes
								.getValue("name"));
						return null;
					}
				};
			} else if ("PostTo".equals(qName)) {
				return new XMLEntity() {
					@Override
					protected ContentHandler initChild(String uri,
							String localName, String qName,
							Attributes attributes) throws SAXException {
						BatchDescriptor.this.postTo = attributes
								.getValue("name");
						return null;
					}
				};
			} else if ("DependentCollections".equals(qName)) {
				return new XMLEntity() {
					@Override
					protected ContentHandler initChild(String uri,
							String localName, String qName,
							Attributes attributes) throws SAXException {
						BatchDescriptor.this.dependentCollections
								.add(attributes.getValue("name"));
						return null;
					}
				};
			}
			return null;
		}
	}

	/** The batchMap. */
	private static Map<String, BatchDescriptor> batchMap = new HashMap<String, BatchDescriptor>();

	/**
	 * Gets the.
	 * 
	 * @param batchName
	 *            the batch name
	 * 
	 * @return the batch descriptor
	 */
	public static BatchDescriptor get(String batchName) {
		return batchMap.get(batchName);
	}

	/**
	 * Gets the post to selector name.
	 * 
	 * @param batchName
	 *            the batch name
	 * 
	 * @return the post to selector name
	 */
	public static String getPostToSelectorName(String batchName) {
		final BatchDescriptor batchDescriptor = batchMap.get(batchName);
		if (batchDescriptor == null) {
			return null;
		}
		return batchDescriptor.getPostTo();
	}

	// singleton instance so that static batch map can be initialised using
	// xml
	/** The Constant singleton. */
	@SuppressWarnings("unused")
	private static final Object singleton = new BatchManager();

	/**
	 * Post.
	 * 
	 * @param businessObject
	 *            the business object
	 * 
	 * @throws Exception
	 *             the exception
	 */
	public static void post(BusinessObject businessObject) throws Exception {
		// validate the batch root object only - it can validate the rest if it
		// needs to
		
		if (businessObject instanceof Validatable) {
			if (!BatchValidator.validate(businessObject)) {
				logger.warn(String.format("Validating %s failed", businessObject.getClass().getSimpleName()));
				throw new ValidationException(
						"Batch did not validate - it was not posted");
			}
		
			((Validatable) businessObject).validator().prepareToPost();
		}
		final SQLEntityHandler postHandler = new SQLEntityHandler(true);
		final Iterator<BusinessObject> batchIterator = new BatchIterator(
				businessObject, null, postHandler);
		// iterate through batch again posting each object
		try {
			while (batchIterator.hasNext()) {
				post(batchIterator.next(), postHandler);
			}
			postHandler.commit();
		} catch (final Exception e) {
			logger.error("Exception occurred while posting batches", e);
			// something went wrong
			postHandler.rollback();
			throw e;
		}
		return;
	}

	/**
	 * Post.
	 * 
	 * @param businessObject
	 *            the business object
	 * @param postHandler
	 *            the post handler
	 * 
	 * @throws KupengaException
	 *             the kupenga exception
	 */
	private static void post(BusinessObject businessObject,
			EntityHandler postHandler) throws KupengaException {
		if (businessObject == null) {
			return;
		}
		if (Boolean.TRUE.equals(businessObject.get("posted"))) {
			return;
		}
		final BatchDescriptor batchDescriptor = batchMap.get(businessObject
				.getName());
		final BusinessObject postToBusinessObject = batchDescriptor
				.getPostToBusinessObject(businessObject, postHandler);
		if (postToBusinessObject != null) {
			postToBusinessObject.save(postHandler);
		}
		businessObject.setItemValue("posted", Boolean.TRUE);
		businessObject.save(postHandler);
	}

	/**
	 * Instantiates a new batch manager.
	 */
	private BatchManager() {
		try {
			final XMLReader xmlReader = XMLReaderFactory.createXMLReader();
			xmlReader.setContentHandler(this);
			xmlReader.parse(new InputSource(this.getClass().getClassLoader().getResourceAsStream("templates/" + DOCUMENT)));
		} catch (final Exception e) {
			logger.error("Error parsing Batch XML.", e);
		}
	}

	/*
	 * (non-Javadoc)
	 * 
	 * @see nz.this.is.absolute.crap.sax.XMLEntity#initChild(java.lang.String,
	 * java.lang.String, java.lang.String, org.xml.sax.Attributes)
	 */
	@Override
	protected ContentHandler initChild(String uri, String localName,
			String qName, Attributes attributes) throws SAXException {
		final BatchDescriptor batchDescriptor = new BatchDescriptor();
		// put it in the map
		batchMap.put(attributes.getValue("name"), batchDescriptor);
		return batchDescriptor;
	}
}

[Advertisement] Keep all your packages and Docker containers in one place, scan for vulnerabilities, and control who can access different feeds. ProGet installs in minutes and has a powerful free version with a lot of great features that you can upgrade when ready.Learn more.

James's team has a pretty complicated deployment process implemented as a series of bash scripts. The deployment is complicated, the scripts doing the deployment are complicated, and failures mid-deployment are common. That means they need to gracefully roll back, and they way they do that is by making backup copies of the modified files.

This is how they do that.

DATE=`date '+%Y%m%d'`
BACKUPDIR=`dirname ${DESTINATION}`/backup
if [ ! -d $BACKUPDIR ]
then
        echo "Creating backup directory ..."
        mkdir -p $BACKUPDIR
fi
FILENAME=`basename ${DESTINATION}`
BACKUPFILETYPE=${BACKUPDIR}/${FILENAME}.${DATE}
BACKUPFILE=${BACKUPFILETYPE}-1
if [ -f ${BACKUPFILE} ] || [ -f ${BACKUPFILE}.gz ] ; then BACKUPFILE=${BACKUPFILETYPE}-2 ; fi
if [ -f ${BACKUPFILE} ] || [ -f ${BACKUPFILE}.gz ] ; then BACKUPFILE=${BACKUPFILETYPE}-3 ; fi
if [ -f ${BACKUPFILE} ] || [ -f ${BACKUPFILE}.gz ] ; then BACKUPFILE=${BACKUPFILETYPE}-4 ; fi
if [ -f ${BACKUPFILE} ] || [ -f ${BACKUPFILE}.gz ] ; then BACKUPFILE=${BACKUPFILETYPE}-5 ; fi
if [ -f ${BACKUPFILE} ] || [ -f ${BACKUPFILE}.gz ] ; then BACKUPFILE=${BACKUPFILETYPE}-6 ; fi
if [ -f ${BACKUPFILE} ] || [ -f ${BACKUPFILE}.gz ] ; then BACKUPFILE=${BACKUPFILETYPE}-7 ; fi
if [ -f ${BACKUPFILE} ] || [ -f ${BACKUPFILE}.gz ] ; then BACKUPFILE=${BACKUPFILETYPE}-8 ; fi
if [ -f ${BACKUPFILE} ] || [ -f ${BACKUPFILE}.gz ] ; then BACKUPFILE=${BACKUPFILETYPE}-9 ; fi
if [ -f ${BACKUPFILE} ] || [ -f ${BACKUPFILE}.gz ] ; then
cat <<EOF
You have already had 9 rates releases in one day.

${BACKUPFILE} already exists, do it manually !!!
EOF
exit 2
fi

Look, I know that loops in bash can be annoying, but they're not that annoying.

This code creates a backup directory (if it doesn't already exist), and then creates a file name for the file we're about to backup, in the form OriginalName.Ymd-n.gz. It tests to see if this file exists, and if it does, it increments n by one. It does this until either it finds a file name that doesn't exist, or it hits 9, at which point it gives you a delightfully passive aggressive message:

You have already had 9 rates releases in one day. ${BACKUPFILE} already exists, do it manually !!!

Yeah, do it manually. Now, admittedly, I don't think a lot of folks want to do more than 9 releases in a given day, but there's no reason why they couldn't just keep trying until they find a good filename. Or even better, require each release to have an identifier (like the commit or build number or whatever) and then use that for the filenames.

Of course, just fixing this copy doesn't address the real WTF, because we laid out the real WTF in the first paragraph: deployment is a series of complicated bash scripts doing complicated steps that can fail all the time. I've worked in places like that, and it's always a nightmare. There are better tools! Our very own Alex has his product, of course, but there are a million ways to get your builds repeatable and reliable that don't involve BuildMaster but also don't involve fragile scripts. Please, please use one of those.

"Toledo is on Earth, Adrian must be on Venus," remarks Russell M. , explaining "This one's from weather.gov. Note that Adrian is 28 million miles away from Toledo. Being raised in Toledo, Michigan did feel like another world sometimes, but this is something else." Even Toledo itself is a good bit distant from Toledo. Definitely a long walk.

"TDSTF", reports regular Michael R. from London, well distant from Toledo OH and Toledo ES.

Also on the bus, astounded Ivan muses "It's been a long while since I've seen a computer embedded in a piece of public infrastructure (here: a bus payment terminal) literally snow crash. They are usually better at listening to Reason..."

From Warsaw, Jaroslaw time travels twice. First with this entry "Busses at the bus terminus often display time left till departure, on the front display and on the screens inside. So one day I entered the bus - front display stating "Departure in 5 minutes". Inside I saw this (upper image)... After two minutes the numbers changed to the ones on the lower image. I'm pretty sure I was not sitting there for six hours..."

And again with an entry we dug out of the way back bin while I was looking for more bus-related items. Was it a total concidence this bus bit also came from Jaroslaw? who just wanted to know "Is bus sharing virtualised that much?" I won't apologize, any kind of bus will do when we're searching hard to match a theme.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Once upon a time, there was a bank whose business relied on a mainframe. As the decades passed and the 21st century dawned, the bank's bigwigs realized they had to upgrade their frontline systems to applications built in Java and .NET, but—for myriad reasons that boiled down to cost, fear, and stubbornness—they didn't want to migrate away from the mainframe entirely. They also didn't want the new frontline systems to talk directly to the mainframe or vice-versa. So they tasked old-timer Edgar with writing some middleware. Edgar's brainchild was a Windows service that took care of receiving frontline requests, passing them to the mainframe, and sending the responses back.

Edgar's middleware worked well, so well that it was largely forgotten about. It outlasted Edgar himself, who, after another solid decade of service, moved on to another company.

A few years later, our submitter John F. joined the bank's C# team. By this point, the poor middleware seemed to be showing its age. A strange problem had arisen: between 8:00AM and 5:00PM, every 45 minutes or so, it would lock up and have to be restarted. Outside of those hours, there was no issue. The problem was mitigated by automatic restarts, but it continued to inflict pain and aggravation upon internal users and external customers. A true solution had to be found.

Unfortunately, Edgar was long gone. The new "owner" of the middleware was an infrastructure team containing zero developers. Had Edgar left them any documentation? No. Source code? Sort of. Edgar had given a copy of the code to his friend Bob prior to leaving. Unfortunately, Bob's copy was a few point releases behind the version of middleware running in production. It was also in C, and there were no C developers to be found anywhere in the company.

And so, the bank's bigwigs cobbled together a diverse team of experts. There were operating system people, network people, and software people ... including the new guy, John. Poor John had the unenviable task of sifting through Edgar's source code. Just as the C# key sits right next to the C key on a piano, reasoned the bigwigs, C# couldn't be that different from C.

John toiled in an unfamiliar language with no build server or test environment to aid him. It should be no great surprise that he got nowhere. A senior coworker suggested that he check what Windows' Process Monitor registered when the middleware was running. John allowed a full day to pass, then looked at the results: it was now clear that the middleware was constantly creating and destroying threads. John wrote a Python script to analyze the threads, and found that most of them lived for only seconds. However, every 5 minutes, a thread was created but never destroyed.

This only happened during the hours of 8:00AM to 5:00PM.

At the next cross-functional team meeting behind closed doors, John finally had something of substance to report to the large group seated around the conference room table. There was still a huge mystery to solve: where were these middleware-killing threads coming from?

"Wait a minute! Wasn't Frank doing something like that?" one of the other team members piped up.

"Frank!" A department manager with no technical expertise, who insisted on attending every meeting regardless, darted up straight in his chair. For once, he wasn't haranguing them for their lack of progress. He resembled a wolf who'd sniffed blood in the air. "You mean Frank from Accounting?!"

This was the corporate equivalent of an arrest warrant. Frank from Accounting was duly called forth.

"That's my program." Frank stood before the table, laid back and blithe despite the obvious frayed nerves of several individuals within the room. "It queries the middleware every 5 minutes."

They were finally getting somewhere. Galvanized, John's heart pounded. "How?" he asked.

"Well, it could be that the middleware is down, so first, my program opens a connection just to make sure it's working," Frank explained. "If that works, it opens another connection and sends the query."

John's confusion mirrored the multiple frowns that filled the room. He forced himself to carefully parse what he'd just heard. "What happens to the first connection?"

"What do you mean?" Frank asked.

"You said your program opens two connections. What do you do with the first one?"

"Oh! I just use that one to test whether the middleware is up."

"You don't need to do that!" one of the networking experts snarled. "For Pete's sake, take that out of your code! Don't you realize you're tanking this thing for everyone else?"

Frank's expression made clear that he was entirely oblivious to the chaos wrought by his program. Somehow, he survived the collective venting of frustration that followed within that conference room. After one small update to Frank's program, the middleware stabilized—for the time being. And while Frank became a scapegoat and villain to some, he was a hero to many, many more. After all, he single-handedly convinced the bank's bigwigs that the status quo was too precarious. They began to plan out a full migration away from mainframe, a move that would free them from their dependence upon aging, orphaned middleware.

Now that the mystery had been solved, John knew where to look in Edgar's source code. The thread pool had a limit of 10, and every thread began by waiting for input. The middleware could handle bad input well enough, but it hadn't been written to handle the case of no input at all.

[Advertisement] Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!

One of the endless struggles in writing reusable API endpoints is creating useful schemas to describe them. Each new serialization format comes up with new ways to express your constraints, each with their own quirks and footguns and absolute trainwrecks.

Maarten has the "pleasure" of consuming an XML-based API, provided by a third party. It comes with an XML schema, for validation. Now, the XML Schema Language has a large number of validators built in. For example, if you want to restrict a field to being a date, you can mark it's type as xsd:date. This will enforce a YYYY-MM-DD format on the data.

If you want to ruin that validation, you can do what the vendor did:

<xsd:simpleType name="DatumType">
  <xsd:annotation>
    <xsd:documentation>YYYY-MM-DD</xsd:documentation>
  </xsd:annotation>
  <xsd:restriction base="xsd:date">
    <xsd:pattern value="(1|2)[0-9]{3}-(0|1)[0-9]-[0-3][0-9]" />
  </xsd:restriction>
</xsd:simpleType>

You can see the xsd:pattern element, which applies a regular expression to validation. And this regex will "validate" dates, excluding things which are definitely not dates, and allowing very valid dates, like February 31st, November 39th, and the 5th of Bureaucracy (the 18th month of the year), as 2025-02-31, 2025-11-39 and 2025-18-05 are all valid strings according to the regex.

Now, an astute reader will note that this is a xsd:restriction on a date; this means that it's applied in addition to ensuring the value is a valid date. So this idiocy is harmless. If you removed the xsd:pattern element, the behavior would remain unchanged.

That leads us to a series of possible conclusions: either they don't understand how XML schema restrictions work, or they don't understand how dates work. As to which one applies, well, I'd say 1/3 chance they don't understand XML, 1/3 chance they don't understand dates, and a 1/3 chance they don't understand both.