One of the downsides of having written here for six years now is that my messy, ever-growing text file of topics for future articles contains a bunch of ideas that have been there for about six years. Part of the problem is that I just have to scroll so far up to even see them now, part of it is that some of them are on topics where I do not feel equipped to write a good formal treatment. So let's consider this an easy, breezy episode of Computers Are Bad as I knock out one of those items.

One of these O. G. topics, line three, is "avalanche technology." I will admit that I am a little fuzzy on what this originally meant, but I have a few good guesses. First, though, let's just take a step back and talk a little bit about the practicalities of the avalanche. For a lot of you, avalanches are probably a pretty abstract issue.

For me, as well—I snowboard but I'm not that good at it, and the kind of mid-tier ski area where you will find me tumbling down the hill (Ski Santa Fe, towards Totemoff's) manages their slopes to avoid the potential for avalanche as much as possible. This requires some expertise in avalanche risks and it's never quite perfect, but it's pretty good. Snow that is regularly groomed by machines like PistenBullys will become coherently packed such that avalanches can't really propagate. The bigger risk on groomed ski slopes generally comes from up above, and here in New Mexico our mountains are of such a height that we're usually starting down from the very top anyway.

Still, avalanches pose a real hazard. The Department of Homeland Security, when it can be distracted from terrorizing well-meaning migrants, tells us that avalanches claim about 28 lives each year in the United States. Globally, the number is much larger, and avalanche fatalities seem to be generally more common in Europe. I'm not sure if that reflects European weather, geography, or recreation habits more, but it's the way things pan out and means that Northern Europe tends to be the center of avalanche technology (as it is in alpine technology in general).

We tend to associate avalanche hazard most of all with recreation, and that's definitely fair: the highest risk of avalanche probably comes about in cross-country skiing, where it's fairly common to traverse the lower part of slopes that have otherwise received very little disturbance. Most avalanches occur during storms, as the snow falls and piles up, which is of course not when recreationists are most likely to be present. You can't rely on this, though. One of the ways that an avalanche can initiate is when lots of snow is piled up on a slope on a nice sunny day—good recreation conditions! Various things like water following the ground or stratigraphic layers of different types of snow from different storms can create a condition of uneven melting. Snow that is under the surface might liquefy faster than snow that is closer to the surface. This "detaches" the upper level of snow from the ground, and when it happens over a wide area, that upper level of snow can let loose all at once. This type of avalanche is relatively rare, but they happen without warning in otherwise good weather, so it's much more likely that people will be caught. Further complicating the situation, the proximity of people can be a factor in initiating these avalanches.

Here in the Mountain States, though, avalanche risk often manifests in another context: transportation. As Americans, cars and highways are among our most esteemed achievements, and decades of engineering accomplishment allows us to drive through, over, and even under much of the Rocky Mountains even during the winter. Landscape modifications, chemical measures, and the brute force of snowplows keep roadways clear—but not the slopes above them. Avalanches cascading down onto highways are a major concern in the Mountain States, especially Colorado which has the most impressive set of mountains and mountainous highways.

Forecasting

Some of the things, then, that we might call "avalanche technology" are the techniques used to prevent, mitigate, and warn of potential avalanche. Much of the work is institutional. On the recreational front, this falls mostly with the US Forest Service's National Avalanche Center and a loose confederation of regional organizations such as our own Taos Avalanche Center, here in New Mexico. One of the main functions of these organizations is education: posting warning signs, spreading awareness, offering training courses and materials, and so on. They also tend to employ avalanche specialists, who use a combination of meteorology and specialized snow observations to determine when avalanches are most likely. This can lead to posting warnings or closing areas entirely.

Avalanche specialists in the recreational field are, in this area, joined by a substantial cadre of avalanche specialists employed by state Departments of Transportation. They apply the same techniques with an eye towards avalanches that could affect mountain pass highways. Those highways may be closed in extreme circumstances, but in transportation even more than in recreation it is common to use, shall we say, "active measures" to prevent avalanches. We will discuss this later, but first, let's briefly talk about how avalanche hazards are forecast.

I am not going to go into much detail here, because I frankly do not know much about it, but I do want to touch on a couple of things that are solidly in my wheelhouse. One of these is SNOTEL. Forecasting avalanche risk requires a solid understanding of the general amount of snowpack in the mountains, which can be very tricky to acquire since when snow is heavy getting up there to take measurements becomes a very time-consuming and potentially hazardous venture.

Avalanche forecasting is just one of many functions served by the SNOTEL or Snow Telemetry system, a 1960s-era network of hundreds of sensor sites throughout mountainous parts of the United States. SNOTEL was originally designed to help with forecasting spring meltwater flows, which is why it falls under the aegis of the Department of Agriculture, but as with any good sensor network there are now a lot of different stakeholders.

The tricky thing about SNOTEL, and remember here that it dates back 60 years, is the snow. SNOTEL sites are way up there, inaccessible during winter, so they are designed to be installed and serviced during summer and to operate autonomously until the next summer. SNOTEL instruments are solar powered, but more interesting are the communications arrangements. The initiation of the SNOTEL project predates a robust set of earth observation satellites, to say nothing of cellular or other digital radio networks. But the whole purpose of SNOTEL is to provide snowpack data when the snow is still there, so real-time reporting is critical.

SNOTEL was thus one of the first applications of meteor scatter, a fascinating form of radio communication that will probably get its whole own article one day. Here is the summary: SNOTEL data collection sites, in convenient locations like towns, transmit a VHF carrier or "pilot" signal up towards the sky. Occasionally, but more often than you would think, a meteor enters the Earth's atmosphere and burns up, leaving a trail of ionized gas behind it. This ionized gas is, if you remember previous discussion of troposcatter, reflective to RF energy. Some of the "pilot" signal is thus scattered back down towards earth. When a SNOTEL site receives the pilot signal, it knows that a meteor has just entered the atmosphere and this scattering is happening, so it quickly transmits a data packet towards the sky. Ideally, this data packet is reflected by the same meteor trail such that the data collection site can receive it. If this all sounds rather stochastic and chancey, it is, but enough meteors enter the atmosphere and data packets can be sent quickly enough (and tried multiple times) that it works pretty reliably for low-bandwidth telemetry data.

Unfortunately, as interesting as meteor scatter is, it's mostly been obsoleted by general technological advances in radio communications. SNOTEL was probably the largest meteor scatter network in the world¹, but is no longer: all SNOTEL sites now use other means of reporting data. The most common is probably the generic data collection capability of GOES, NOAA's main fleet of "weather satellites." GOES satellites actually serve many purposes related to earth observation, and one of them is telemetry collection from instruments on the ground. If you've ever come across a USGS stream gauging station or another mysterious hut in the middle of the woods with a cylindrical or conical antenna pointed at the sky, it's transmitting data upwards for reception by a GOES satellite. The GOES earth segment collects all of the little data packets the satellites receive into files which are distributed to the various organizations that report data this way. If you are so inclined, you can also receive this data yourself as the satellites rebroadcast it in batches, but most users don't bother since NOAA will do that part for you.

Even that might be fading away: newer SNOTEL sites apparently use the cellular network for data reporting, unless it's not possible to get a signal, but that's becoming less common over time even in the Colorado mountains. The major benefit here is that the cellular network is low latency and very high bandwidth, which makes it feasible for example to view cameras installed at the SNOTEL sites.

With the data part discussed, let's take a quick second look at the data collected. Temperature, humidity, pressure, wind, liquid precipitation are all measured as usual, but SNOTEL adds a couple of specialty features. Soil moisture and evapotranspiration (basically evaporation rate of water from soil) sensors provide more data on how saturated the ground is, an ultrasonic or optical instrument pointed down from atop a post measures how deep the snowpack is, and a funny looking contraption called a snow pillow takes perhaps the most important measurement. The snow pillow is basically a big rubber balloon that gets inflated on the ground, and then covered with snow. By measuring the air pressure from the snow pillow, the SNOTEL station effectively weighs the snowpack, which combined with its height gives the density or water content of the snow. This varies widely depending on the specific weather when the snow falls, and remember that stratigraphic differences in density are one of the factors that's important in the likelihood of a surprise avalanche, so collecting this water content information as snow falls allows forecasters to remotely estimate whether or not the snow conditions are hazardous.

That is, of course, supplemented by good old fashioned techniques like sending out avalanche specialists with shovels and meter sticks to take a look at the snow themselves. DOTs often have technicians doing a regular tour of mountain pass sites, where they curate plastic tubs and boxes of snow from various storms so that they can get a hands-on impression of what the different layers under the snowpack are like. They also take samples to weigh, perform some tests to measure mechanical properties of the snow, there's basically a whole bag of tricks, but ultimately there is also a subjective element of how "good" or "bad" things look. This is born of experience and is one of many reasons to value our state employees.

Mitigation

Within the recreational context, one common approach to mitigation is simply to avoid the area. Especially for backcountry skiing areas, avalanche centers might post warnings or work with the operator to close the area entirely when forecasting shows that conditions are favorable for an avalanche. Warnings might be more useful than you think, since besides reducing the number of people in the area they should also prime people to recognize particularly risky slopes or signs of snow that is becoming unstable. Avalanches can occur without any surface signs of instability, but in practice there often are some signs, so of course keeping an eye out is better than nothing. Warnings also make people more likely to carry safety equipment, discussed later, which improves the chance of self-rescue if things really do go wrong.

But there are also, shall we say, kinetic options. Many of the methods of modern alpinism evolved during the Second World War, and so did a good portion of the mitigation equipment. Once snow becomes unstable enough that an avalanche could occur, it is basically waiting for something to set it off. Popular media concepts that loud noises will trigger avalanches are, well, fictional, but the point stands that people moving around or especially operating vehicles can trigger avalanches, as can basically random happenstance. You can mitigate the risk of an avalanche by going to unstable areas and intentionally triggering them, or at least trying to, basically the same way that a bomb squad might handle an unstable explosive by exploding it. Better that it go off when you expect it than when you don't.

A classic method, still in some use today, is repurposed artillery. State DOTs and ski area operators own a surprisingly large inventory of M101 Howitzers, a 105mm artillery gun that dates more or less to a 1910s German design but was widely fielded by the US during the Second World War. The avalanche application is simple: when you find a slope with unstable conditions that could pose a hazard, you shoot at it. The concussion from the impact, and often from an explosive round, puts a lot of pressure on the snow that will likely break it loose if it's possible to do so. You just have to make sure you set up outside of the resulting avalanche.

The main problem with this method, which everyone agrees is a whole lot of fun, is that the M101 has not been manufactured since 1953 and, in general, the supply of cheap Army-surplus artillery has dried up. The Howitzers are aging out, and other methods are replacing them. Some of these are similarly dynamic: dropping small improvised bombs from helicopters, for example, which is a very common method in Colorado today. Even just hovering a helicopter over a slope can sometimes trigger an avalanche, although a bit of dynamite makes this a lot more reliable. If you have noticed the surfeit of high-explosives bunkers and unexploded ordnance warnings in some parts of the Colorado mountains, well, now you know why. It can feel a little bit like a very slow-running war.

Even these methods are being replaced by more modern and cost-effective ones, though. A French alpine engineering company called MND manufactures a system called Gazex, which is a bit like an airport bird canon on steroids. Gazex units are pre-installed in avalanche risk areas (by helicopter, for example), and then, under remote control, they mix oxygen and a combustible gas and set it off in a big tube pointed at the ground. The force created is similar to an explosion, but the whole thing is easier to handle (since it uses gas cylinders rather than high explosives) and remote control makes everything cheaper for the operators. MND recently introduced a new version of the system called O'bellx (I assume this reads better in French), which is even more portable, albeit comically egg-shaped. This seems to be catching on with ski areas, and MND also emphasizes that the portable design and low-impact installation makes it suitable for areas like national parks with acute natural preservation concerns. Basically, you can remove them at the end of every winter, so that you aren't leaving equipment scattered around your beautiful mountains.

There are also a number of other mitigation approaches that are less fun, so I will devote less attention to them, but you can find them done around here. Structures similar to snow fences, built high up on a slope, provide an "anchor" for the snow that prevents it coming loose enough for an avalanche. There are many variations on this idea, for example using steel cables supporting nets. Another approach is to permit the avalanches to start but prevent them from reaching areas where they cause harm, either by building wall or dam-like structures (I do not believe this is common in the US as I am not aware of any examples) or by putting corridors like highways inside of "snow sheds" that are sturdy enough to allow the avalanche to simply flow over. This latter method is much more common in Europe than here, but you can find some examples on US railroads, which are often in more difficult terrain than highways.

Warning

One of the more interesting developments in avalanche technology are warning systems. Avalanches wouldn't pose much hazard if people knew that they were coming; what makes melt-triggered avalanches so dangerous is that they can be very unexpected. One of the methods that avalanche safety courses teach is to recognize signs of snow masses that have already started to slide, shift, or collapse on themselves. This would indicate that conditions are favorable for an avalanche and that disturbing or going below the snow is quite dangerous. The usual surface signs are cracks, caused by the snow starting to slide apart, but these cracks can be tricky to see even when they are present.

Another indication of avalanche potential is subsidence, or the snow surface starting to drop in level, or any motion of the snow surface downslope. These effects can start minutes or hours before the real avalanche occurs, but they are subtle and often slow, so humans are unlikely to notice them. Fortunately, machines are well adapted to this task. Several vendors offer radar systems that monitor snow slopes for any shifting, and can sound a siren or another alert if things start to move. Like earthquake warning, the actual lead time provided by these systems is not very well known and could be quite short, but something is better than nothing.

Other technologies, like LIDAR and acoustics (e.g. geophones), can be applied to a similar task. I have even heard of older systems that used thin wires or brittle metal bars embedded in the snow, as part of an electrical circuit that would break if the snow started to move ². Such systems can be quite sophisticated, and MND's sales materials suggest things like closing gates on highways when an avalanche is detected. The current state of the art in radar detection uses pulsed Doppler operation, which can be very sensitive to movement in all kinds of directions. All that said, these don't seem to be popular in the US. I couldn't readily find any installations in this country.

As a final note on this topic, national alerting systems like NAWAS and the Emergency Alert System have defined message types for avalanche warnings. In practice, these are very rare, as it doesn't usually make sense from a time or area perspective to put out an avalanche warning through mass media channels. That said, they do happen, usually on the (fortunately rare) occasion that forecasters predict a high risk of avalanche in a town or other populated area. In that case, the procedure is to evacuate the town, and that's exactly the kind of situation that NAWAS/EAS/NWS All Hazards/etc. are intended for.

Rescue

And finally, we reach the topic that I believe led me to put "avalanche technology" on the list to begin with: technical aids to rescue. Avalanches are, in a certain sense, not that dangerous. They can move at formidable speeds (60+ miles per hour often reported), but snow is relatively soft and low-density. Injury from being crushed or striking against something is certainly a possibility, but the bigger hazard of avalanche is usually when you become buried.

If you are buried in snow, and there are no rescuers or they cannot find you, it can be extremely difficult to work your way out and you are likely to succumb to some combination of asphyiation, exhaustion, and hypothermia in the process. That might sound silly but keep in mind that you have just been buried by an avalanche, probably have a variety of minor injuries, have become disoriented, etc., and you try to dig yourself out of feet of snow in that situation. You really need someone on the surface to come help you out. That's why standard avalanche safety equipment includes a probe and a shovel—a probe for finding people (just a metal rod you use to poke under the snow, basically), and a shovel for getting to them.

The problem is that probing a large area of snow is a very slow process, and even if you were right next to someone before an avalanche, you probably don't know where they ended up afterwards. This is an area where radio technology can be extremely helpful.

The concept of avalanche beacons or avalanche transponders dates back to at least 1968. Dr. John Lawton was an engineer at an aerospace research lab now known as the Calspan Corporation, where he worked on a variety of interesting aviation problems ranging from weapons targeting to meteorological research. Lawton was quite an aviation enthusiast, making the news in 2013 for celebrating his 90th birthday by flying his Cessna Skyhawk over the US-Canada border 90 times. That's a bit of a peculiar way to entertain yourself, but anyone who is still flying themselves around at 90 has clearly found a way to stay vital. His lifelong investment in aviation is emphasized by the fact that he started his winding cross-border journey at the airport he owned.

What is not so well documented is his interest in alpinism, but he must have had some affection for the mountains. He did some kind of research work in Europe related to triggering avalanches by dropping explosives from light aircraft, and whether that put the topic on his mind or something else, he also came up with the first commercial avalanche beacon. Commercialized by his company Lawtronics, the "Skadi" transmitted a tone at 2.275 kHz. Groups of people traveling in the mountains could each carry a Skadi unit, and if anyone was lost in an avalanche, any of the other units could be switched to a receive mode and the person located by simple amplitude search. That is, walking around the snow until you find the spot where the tone comes in the loudest, at which you point you start probing and digging.

This same method is still used today, although the radio system has been standardized at 457 kHz. Modern avalanche rescue systems like the "Barryvox" from Mammut or the Black Diamond Recon X are direct descendants of Lawton's Skadi device, using 457 kHz and digital direction finding techniques (like the use of multiple perpendicular directional antennas to compute a likely heading to target) to speed things up and make radiolocation more user friendly. Because the 457 kHz system is an international standard, they ought to work about the same regardless of manufacturer. That's important in real-world scenarios where the people doing the rescuing might be from a whole different group than the one affected by the avalanche.

Modern avalanche beacons are probably one of the closest real-world products to the way that "radio beacons" tend to work in movies and videogames. The reality is of course messier, especially if there are multiple victims, but in principle a digital avalanche beacon in receive mode can show an arrow on screen that you follow until you are standing on top of the buried person. You know if you're getting closer when they beep more. Neat.

Technological development continues apace, and some newer avalanche beacons also support a second protocol called W-Link. W-Link is actually surprisingly poorly documented, presumably because it appears to be proprietary to Mammut (the only non-Mammut beacons that support it seem to have been joint venture projects with Mammut). W-Link is a digital protocol that operates in the ISM band, introducing the downside that ISM bands are not internationally standardized. This means that US and European-market beacons will always be compatible for conventional 457 kHz operation, but two beacons with W-Link support may not be able to use that protocol if they were sold for different regulatory markets.

The upside of W-Link is that the digital protocol lets the beacon send significantly more information. This includes a unique serial number for the beacon, which allows the receiver to discriminate between pulses being received from two or more beacons at the same time—a radical improvement in the ease of locating multiple people, as conventional direction finding techniques really struggle when you cannot discriminate between multiple targets. You can end up with a "swinging needle," while the W-Link devices are capable of locking to a single target at a time. As this implies, you can also determine how many people are buried, and ignore beacons that turn out to be unimportant (because they are discarded on the surface, or carried by a rescuer, etc).

Perhaps the most interesting feature of W-Link is that some W-Link beacons will use an accelerometer to detect motion, and report whether or not any motion is present. In theory, this allows rescuers to prioritize their efforts by focusing on beacons that report motion, meaning that they are carried by people who are still alive. These enhanced capabilities of W-Link seem to have led to enough controversy that the Wikipedia article devotes much of its length to ethics discussions, and manufacturers have put out position papers on their decisions to implement or not implement certain W-Link features. The concern is that W-Link may give rescuers more information on which to prioritize rescue in a discriminatory way, perhaps by searching for victims with (more expensive) W-Link beacons before less expensive conventional beacons, or by using knowledge of W-Link serial numbers to prioritize the search for specific people.

In practice, I think these ethical concerns are more theoretical than actual, because penetration of W-Link technology doesn't seem all that high and most actual beacons only implement the very most basic features. The ability to display the serial numbers of individual beacons, for example, is pretty much limited to a USB dongle/software package that Mammut sells to organizations like ski patrols that loan out avalanche beacons and need to keep track of their inventory, check battery levels, etc.

Avalanche beacons are not the only radio-based approach to avalanche rescue. Around a decade after Lawton developed Skadi, a Swiss engineer designed a system called RECCO. If you have read my articles on loss prevention, I can succinctly explain RECCO by saying that it is just RF electronic article surveillance applied to the outdoors. If you have not, let's get into a little bit more detail.

If you make an antenna that is a small loop, and then put a nonlinear device like a p-n diode in it, you end up with an antenna that behaves asymmetrically with regards to the phase of the radio wave passing over it. Intuitively, we can say that energy freely flows one way around the antenna, but not the other. The energy going that way becomes trapped, and gets re-emitted at at a harmonic frequency to the one that excited the device. This general principle is known as non-linear junction detection, and it has all kinds of interesting applications in fields like technical security, but its most common application is as one of the common types of retail anti-theft tagging.

RECCO is basically a retail loss prevention system aimed at the ground. RECCO tags, called "reflectors" in RECCO parlance, are a simple foil antenna and diode, which makes them cheap and durable. They are so cheap, and compact, that some brands of alpine gear (especially in Europe) build RECCO tags right into products like snow pants and ski boots. Dainese even offers a bicycle helmet with one. The RECCO detector emits a powerful and very directional RF field in an ISM band, and then monitors for the characteristic harmonics emitted by an excited tag.

This technology is not RFID, although it is conceptually similar, and has some of the same downsides such as a tendency for tags to be undetectable if they fall in the wrong point in the phase of the exciting field. For that reason, RECCO recommends that you have two reflectors at different locations on your person. The other main downside of RECCO is that it shifts most of the complexity into the detector, so they are relatively large and expensive. That's not so bad when you consider that detectors only need to be carried by ski patrol and SAR organizations. Rescuers basically aim the detector around until it picks up a signal, and then walk that way until they are aiming the detector straight down. That's where they dig.

RECCO has even designed a detector that can be underslung by a helicopter, although its 100 meter range will require some sporty flying. The RECCO system is generally much more popular in Europe than in the US, but penetration is increasing here, and RECCO publishes a list of SAR organizations with detectors that now includes a number of helicopters in the US and many major ski areas.

I view RECCO as having a bit of a belt-and-suspenders relationship with traditional avalanche beacons. Avalanche beacons are more common, and all beacons also function as receivers, so "buddy rescue" by a bystander is far more likely with beacon technology. On the other hand, RECCO is usable over a longer range (although not radically longer) and the detectors are more directional and potentially faster to use. Most avalanche centers seem to recommend that you carry both, although the traditional avalanche beacon is more important.

Finally, let's consider a couple of other pieces of avalanche safety equipment, which will lead us to consider efficacy.

The core avalanche PPE consists of a probe and a shovel, and both are available in collapsible form for portability. Besides locating technologies, though, the major innovation in avalanche safety is the avalanche airbag. These are backpacks that you wear with a parachute-like ripcord. When you pull it, the bag inflates into a big airbag behind you. There are two ways this is helpful: first, the airbag will tend to float on moving snow, making it less likely that you are buried. Second, if you do become buried, you can deflate the airbag to leave you with a void in the snow. This gives you more air to breath, and room to move, making it more likely that you can dig yourself out.

Avalanche airbags remind me a bit of the motorcycle airbags, as far as being very cutting-edge but also, you know, kind of questionable when it comes to efficacy. Manufacturers of airbags often advertise a 50% reduction in fatalities, but we have to take those numbers with a grain of salt.

One of the problems with safety technology for extreme risks is that the actual risk event just doesn't happen very often. In less abstract terms, people are not caught in avalanches all that often, so the total number of people who have been caught in an avalanche while wearing an avalanche airbag just isn't that big. This limits the confidence we can have in efficacy findings.

Well, I tracked down the paper that the 50% number comes from, and it's not actually that bad. They identified 424 people caught in severe avalanches, roughly half of which had avalanche airbags, and found that the airbags reduced mortality by 50% for people who successfully deployed them, and 41% overall given that there were various situations where they were not triggered or failed. The sample size is not huge, but it's bigger than I expected, and even at the bottom of the 95% confidence interval you still see a meaningful mortality reduction of around 10%.

So, avalanche airbags seem to work, but they do cost around $1,500, so make of that what you will. Another interesting number from that paper is actually a bit of an aside: of the sample of people considered, 99% had avalanche beacons! A basic avalanche beacon can be had for about $250, which is pretty similar to cheap satellite messenger or very basic PLB ³.

On the other hand, the passive nature of RECCO tags translates to a price advantage: a standalone reflector that you can clip to your pack only costs $30, which frankly still feels like a ripoff considering the simplicity of the design. I wonder if a RECCO detector would pick up enough RF EAS tags sewn into your clothing. Or shoplifted. One of the amusing things, though, is that despite being the cheap option (or perhaps beacuse of it), RECCO is relatively poorly backed by data. The number of actual rescues using RECCO remains small, although the development of the helicopter-based detector will probably lead to more successful uses.

Or, you know, just stay out of the mountains. They may be calling, but you don't have to go.

Brigadier Martin Hotine is not quite the image of a decorated officer. His name is styled with trailing acronyms that make it no surprise that there is an official portrait, yet in that painting he appears disheveled, his tie far off center in the collar of his jacket. He leans off to one side, not quite like he is sitting for a portrait, but more like he was caught in the middle of something. Photos of the man are often similar: he's distracted, looking down at his desk or staring into space. His mind seems to be elsewhere. Hotine had a lot to think about. His duties in the First and Second World Wars had only been a distraction from the real work of his career: the precise measurement of the whole British Empire.

Late in Hotine's career, he was honored not only by his own country (as a Commander of the Order of the British Empire) but by the United States as well (named an Officer of the Legion of Merit). Most of his awards, though, reflected the technical nature of his work: the Founder's Medal of the Royal Geographical Society, and shortly after his death in 1968, the Gold Medal of the United States Department of Commerce.

Ribbons and medals, though, do not quite capture the breadth of Hotine's work. His greatest memorial is an artifact of his work: squat concrete pillars surmounted by a triangular brass plate. Found atop mountains and hills throughout the United Kingdom, these "trig points," designed by Hotine himself, are the physical references of the Retriangulation of Great Britain. This effort, spanning from 1935 to 1962 with the interruption of WWII, revised an original triangulation (initiated in the 18th century) as the basis for British surveying. Through the course of this effort, Hotine developed methods that would revolutionize the field of geodesy. His collaboration with mapmakers from the United States, a continuation of his wartime surveying for the Allied Forces, set the stage for one of geodesy's most ambitious projects: a measurement taken across the Atlantic Ocean.

---

Geodesy is the field concerned with the measurement of the Earth. It is perhaps one of the greatest examples of the subtle complexity of the real world: superficially, the measurement of distances and areas is a simple problem. In practice, it is extremely complex, subject to a web of complications that mean that even the most modern efforts should be viewed only as close approximations.

To begin, we have to consider the shape of our planet. This question, "what shape is the Earth?," is a central topic in geodesy known as the "figure of the Earth," and it has occupied mathematicians, cartographers, and astronomers for centuries. Of course we know what shape the Earth is: it is a sphere. Well, that's true to a level of approximation, but one that isn't even close enough for highway construction.

Geodesy's foundations are in the measurement of angles and distances, taken from the Earth's surface—of course, for most of human history, where else would we take them? By measuring the angles between three points and performing some trigonometry, the relative positions of the three points can be determined. This is known as triangulation. The same is true if you measure the distance between three points, known as trilateration, but up until the development of electronics the measurement of very long distances was a far more difficult problem than the measurement of angles.

So, the first triangulation of Great Britain, conducted over some 60 years starting in 1791, measured the angles between mountain and hill peaks. These measurements were taken very precisely using a then-new instrument called a theodolite, which is essentially a telescope coupled to a protractor. By taking enough measurements between enough hilltops, surveyors formed a sort of mesh or web that slowly spread across the country. Eventually, this network of reference points was dense enough that locations of buildings, property claims, and enemy encampments could be stated accurately by their relative position to fixed reference points.

This explanation of geodetic triangulation has omitted a major problem. Three angles or three measurements can precisely define a triangle, but the solution depends on the surface over which the triangle is formed. The math is simplest in a flat plane. Over a sphere, it becomes more complicated but is still well understood. The Earth, though, is not a sphere. It's not even that close.

One of the reasons that it is difficult to define the shape of the Earth is that it is unclear exactly what shape you would refer to. The physical surface of the Earth, its topography, is extremely messy. There are mountains, there are valleys, and the whole thing is the result of long, stochastic processes that left behind something that is not amenable to a mathematical description. Besides, surveyors are often trying to establish where exactly the topography is (e.g. the altitude of a given point), so referencing those measurements against the topography itself would be tautological. The field of geodesy, as separate from mere measurement, is perhaps defined by this realization that measurements of the Earth must be taken in reference to an abstract plane.

The most tempting of these planes is sea level, more or less. Geodesists refer to this as the "geoid:" roughly speaking, the geoid is the shape that the Earth would take if it were covered entirely by water and undisturbed by tides, weather, geotectonics, etc. It is not a sphere because the Earth is rotating, and the centrifugal force pushes things outward at the equator, causing a slight "flattening" of the sphere into an ellipsoid. It is not an ellipsoid either, because the geoid is ultimately defined by gravity, and gravity is controlled by the density of the Earth nearby. For example, where the Earth's crust is very dense, gravity is locally greater and pulls the "water" of the geoid closer, creating a sort of "hill." Where the Earth's crust is less dense, water can flow away. Ultimately, the geoid is irregular, describable only by measurement.

Instead, geodesists rely on a simplified form of the geoid called the ellipsoid. This is an ellipsoid that best approximates the geoid, for the surveyor's purposes. British mathematician and geodesist Alexander Ross Clarke, in the course of the 19th century Ordnance Survey, determined that an ellipsoid with a major axis of 6,378,306 meters and an inverse flattening ratio of 293.465 was the best approximation of the geoid based on the areas that the British Empire had surveyed so far.

Ellipsoids are not useful on their own, though, as they must be anchored to a location and orientation on the surface. In other words, an ellipsoid is just a shape, and you must have at least one reference point and angle, or otherwise multiple reference points, to define how the surface of the Earth is mapped to that shape. When you add a convention for how surface measurements are expressed (and relative to what, typically the same reference point at which the ellipsoid is defined), you have the basis of a geodetic system that can convert between abstract coordinates and physical locations in three dimensions.

---

One of the driving forces of cartography, both as an enabling factor and as a source of demand, is aviation. Aircraft cover large distances, and pilots can make close observations of the ground underneath them. As technology advanced, aerial photography offered a way to record the pilot's observations for later analysis.

Still, as with so many things in cartography, there is a chicken and egg problem. An aircraft can be used to survey a map, but the aircraft needs to know where it is—usually by comparison to a map. In a certain sense, a pilot setting out to map new territory is flying blind. Well, some pilots are literally flying blind, in fog or clouds or under the cover of night. Even before the war, aviation was venturing into more difficult missions and pilots found themselves caught in the dark.

A German company, Lorenz, developed the first solution for blind navigation: an early form of space-modulated radio beacon usually called a Lorenz beam. By emitting a precisely directional radio signal down the approach path to a runway, the Lorenz beam created an invisible path that pilots could follow. An instrument in the aircraft gave the pilot an indication of how far to the left or right they were, and when they were centered, they knew exactly where they were—at least on a line.

The first Lorenz beam was installed in 1932, and it opened the era of radionavigation. Other events in Germany would soon establish radionavigation as one of the period's most profound developments: a blessing to pilots and a curse to the people caught underneath.

---

This combination of an ellipsoid, reference points, and a system of measurement is roughly what geodesists call a datum. Clarke's ellipsoid, while most suitable in the UK where the majority of the measurements on which it was based were taken, was well-known in the very early 20th century when the United States Coast and Geodetic Survey set out to triangulate the United States. Almost 30 years of work led to revisions to the ellipsoid and the selection of a new reference point, Meades Ranch in Kansas. This became known as NAD27, the North American Datum of 1927, and remained the basis of most surveying in the US, Canada, and Mexico until a major revision adopted in 1983.

I explain this background to illustrate a core problem in geodesy: precise measurements require a datum, and a datum requires an ellipsoid, which is an approximation of the geoid calculated by fitting to a set of precise measurements. By the mid-20th century, major triangulations had been completed in the US, in the UK, and in many European countries.

The onset of the Second World War revealed a problem with this state of geodesy: the datums were all different. Military planners laid out maps surveyed by different countries and found that they did not meet up at the edges. Not just due to the projection, or due to the illustration, but because these mapping efforts fundamentally disagreed on the plane over which the maps were made. Different countries had used different ellipsoids (and different basic methods of determining the ellipsoids), they anchored them at different references, and they adopted different values for the correction of measurements. National surveys tended to be anchored somewhere in the interior, for centrality, which meant that datum-related inconsistencies were most acute at the borders of countries where different datums met.

An Army officer planning artillery fire would find that maps of the German-French border, produced by the mapping authorities of the two formerly separate countries, disagreed on the locations of towns by hundreds of meters. This was enough to become a dominant source of error in artillery aiming, so the military knew what it had to do: unify the maps of Europe onto a single datum. Soldiers on the front were joined by tactical surveyors, military units that traded rifles for theodolites as they completed one of the war's most important intelligence missions. This effort, to establish a uniform European datum, was not completed until after the war, when Allied intelligence units located the archives of the German cartographic authority and hauled them to the US for calculations.

The result became ED50, the European Datum of 1950.

---

The problem of flying blind was all the more acute for bombers. Early in the Second World War, Allied Forces had low expectations of German air defense and thought that bombers would be able to operate over Europe mostly unchallenged. Things did not work out so conveniently: German fighters were better than expected, ground-based defenses more extensive, and ultimately, bomber losses were much greater. This forced a change in strategy to night-time bombing runs. With the absent to primitive radar technology of the time, bombers were extremely difficult to intercept at night.

This sword cut both ways: bombing accuracy was also very poor at night as bombers struggled to find their targets, especially during defensive blackouts. German bombers headed towards England encountered the same conditions, forcing the same shift towards night-time bombing. Both sides put to work developing technology for improved night-time bombing. The Germans extended the Lorenz beam into a two-dimensional system that could "designate a target" by radio. On the Allied side, several systems were in simultaneous development in the UK and US. The rapid iteration of radionavigation technology and electronic countermeasures during the bombing of Great Britain became known as the "Battle of the Beams," and set the groundwork for much of our modern navigation technology.

Among the combatants in this ethereal battle was RCA, the Radio Corporation of America, which had unintentionally discovered a means of measuring distances by radio and then greatly improved it by incorporating features of a similar British system. RCA called it SHORAN, the Short Range Navigation system. SHORAN used an aircraft-mounted receiver that emitted pulses and measured the time until "reply" pulses arrived from fixed ground stations. By maintaining a constant delay time to a given ground station, an aircraft could fly a perfect orbit around it. By flying that orbit until a specified delay to a second ground station was achieved, an aircraft could position itself exactly on a bombing target. SHORAN saw extensive use in both WWII and the Korean War.

---

Let us consider the state of geodesy after the Second World War. By this time, the United States had been thoroughly surveyed under NAD27, the Retriangulation of Great Britain was underway using a datum called OSGB36 (Ordinance Survey of Great Britain, 1936), and European maps were being completed under ED50. Combined with new efforts in Japan, this put most of the Allied or Allied-occupied world into the same condition: accurate datums existed that covered large contiguous land areas, but the exact relationship between those land areas was difficult to determine.

In other words, the distance between two locations in the United States or Europe, even very far apart, could be fairly confidently stated because of the common datum. The distance between a location in the United States and a location in Europe, though, came up question marks. Cartographers, especially of the nautical charting variety, had methods of approximating distances over the ocean but they were of very poor accuracy compared to true triangulated surveys. Ocean charts were made mostly by celestial observation, a field that was struggling to develop into "astro-geodesy" because of the limited precision of telescope measurements. To put a scale on the problem, analysis of mid-century nautical charts against later methods found that many Pacific islands had mapped locations that were several kilometers off, and that even major land masses varied by hundreds of meters.

There is a simple reason for this problem: the methods of surveying. While experiments were conducted with intriguing methods like celestial photogrammetry, photographing the same balloon or flare against the starfield from multiple locations and calculating vectors to determine the distance between these locations, the only truly accurate method was still triangulation by theodolites. This was easy where survey stations could be set up on mountains and hills, but it was already a challenge in forested country where you simply couldn't see that far. It was impossible over the ocean, where there was no way of seeing from one landmass to another.

Up to the beginning of the Cold War, no one had paid that much effort towards surveying over the oceans. Geodesy was fundamentally a local concern: knowing where something was within its country was enough. The numerous countries of Europe and their extensive land borders had made the unified European datum a necessity, but even between Western Europe and Eastern Europe there was little need for standardization.

Geodesy did not become a global concern until the invention of the intercontinental ballistic missile.

---

By the end of the war, it was already obvious that SHORAN had potential beyond targeting bombers. SHORAN equipment had been tested off of the US coast, during which RCA engineers noted that a small Bahamian island must be several hundred yards away from its charted position. Later measurements with precision astronomical equipment confirmed the SHORAN-derived measurements as more accurate than the charts. A Coast and Geodetic Survey officer seconded to the Army Air Force for the war effort took note of this result, and reported back to the Survey as it was struggling to map the Aleutian Islands.

Many of the Aleutian Islands were close enough to be surveyed by conventional means, at least in theory. In practice, terrible weather and frequent fog meant that surveyors repeatedly tried and failed to spot one island from the next. The cost of flying people around, waiting for good weather, and getting a few measurements through each break in the fog had put the project well behind schedule. After the delivery of SHORAN equipment, a few months of experimentation developed a technique that completed the survey by radio.

At the same time, the oil industry was pushing into two frontiers: the South American jungle and the ocean offshore. Both were formidable challenges to surveyors, the jungle due to the lack of clear sightlines and the ocean due to the lack of anywhere to set up fixed points. Oil exploration companies purchased secondhand SHORAN equipment from the military and ran their own experiments, leading to refined solid-state radio equipment and another set of well-tested operating practices.

Early experiments in SHORAN surveying came up with a frustrating error: SHORAN measurements, as compared to reference points established by traditional surveying, were always a bit too short. It took careful comparisons of SHORAN networks to Coast and Geodetic Survey networks in Colorado and Florida to confirm the reason: the accepted value of the speed of light, up to that point, was incorrect by 16 ppm. In 1949, the same Coast and Geodetic Survey officer, Colonel Aslakson, published a new value for the speed of light derived from SHORAN. That measurement has held up to the modern day, within 1 ppm.

The British Ordnance Survey had maintained close contact with the Coast and Geodetic Survey, in part through Hotine, and eagerly observed these tests. The Retriangulation of Great Britain was back underway after the war, now with the benefit of improved technology, but it was facing similar challenges to those in the Aleutians. Some experimental use of SHORAN was attempted in Scotland. SHORAN surveying was widely adopted in Canada as well, where in the late 1940s Parliament authorized the first precise mapping of the entire country.

With all of these efforts in mind, and its own objectives as well, the Air Force continued research on SHORAN. Among other developments, this led to HIRAN, a high precision version of SHORAN that used improved receiver electronics to obtain much more accurate travel time measurements. HIRAN reduced the typical error of SHORAN distance measurements from hundreds of yards to around ten feet.

---

The Cold War was a fundamentally different type of conflict from the Second World War. Besides the somewhat speculative nature of the actual conflict, the distances involved were incredible: for the first time in military history, strategists planned out attacks that would start on one continent and end on another.

WWII experience had already shown the importance of accurate mapping for calculating SHORAN bombing targets, but this was less of an issue in the Cold War context since accurate geodesic data for the Soviet Union was a closely held secret and even the best of spies would struggle to operate a SHORAN ground station undetected. Bomber aircraft would have to navigate by celestial observations and then spot their targets more accurately by conventional means, which of course raised the same problems of night-time bombing. This was a contradiction in nuclear reprisal plans that was never really addressed, since by the time research was underway to achieve accurate global navigation the role of the bomber had been supplanted by the ICBM.

ICBMs used primarily inertial navigation, with corrections while underway from celestial measurements. Both inertial and celestial navigation were technologies that saw revolutionary advances due to the ICBM, reaching accuracies that were unimaginable during the Battle of the Beams.

There was a problem: the targeting capabilities of ICBMs were soon superior to the maps that were used to designate the targets.

In the era of the ICBM, the exact distance from, say, North Dakota to Moscow became a question of great significance. Military intelligence set about answering it, and a combination of espionage and information from the previous war allowed for the extension of ED50 into the Soviet Union with some degree of confidence, as well as the anchoring of Soviet maps to ED50. Even before this work was complete, though, it was obvious that with missile silos surveyed against NAD27 and targets surveyed against ED50, the actual misalignment between the two datums would become larger than the probable error of the missiles.

Post-facto, the Department of Defense presents a laundry list of motivations for a World Geodetic System. I'm sure there's some truth to all of it, we know now that accurate geodesy is useful in so many ways. In 1950, though, most of the applications hadn't yet been invented. Older materials tell a plainer story: the military needed to aim missiles, and to aim missiles it needed a single map that contained both the origin and the destination.

NAD27, OSGB36, and ED50 were all sufficient within their scope. What the Cold War demanded was a connection between the two. The Air Force determined that it would have to trilaterate the Atlantic Ocean.

---

Efforts started from both sides. Canadian survey efforts, already routinely using SHORAN, provided NAD27 references in the Maritimes. ED50 provided references in Norway, and the United Kingdom was well surveyed under OSGB36. If trilateration measurements could be made between Canada and the UK, and between the UK and Norway, that would be sufficient to relate all three datums to each other. This came to be known as the North Atlantic Tie.

To complete the tie, the Air Force would have to island-hop its way across: from Canada to Greenland, Greenland to Iceland, Iceland to Faroe Islands, then to Scotland and on to Norway. The first part of this project was completed between Scotland and Norway with the cooperation of the Ordnance Survey, but observations between Scotland and the Faroe Islands were the greatest challenge.

Here we will finally consider the details of SHORAN. SHORAN is a straightforward time of flight system in which ground stations support a transceiver on the aircraft. The aircraft emitted a pulse on some UHF frequency, and a ground station tuned to that frequency received it and immediately emitted another pulse. The aircraft equipment measured the time between transmitting the pulse and receiving the reply which was, much like radar, an indirect measurement of the distance between the aircraft and the ground station. By using two different ground stations on two different frequencies, and doing some math, it was of course possible to establish the position of the aircraft on a map.

More importantly to surveyors, though, it was realized that SHORAN could be used to accurately measure the distance between two ground stations. If you knew the altitude of the aircraft, and found the shortest possible SHORAN distance to the two stations, it was simple trigonometry to find the straight-line distance between stations. This method became known as "line-crossing:" surveyors would mark on a map the estimated straight-line between the two ground stations, and then draw a perpendicular line that crossed it in the middle. An aircraft would then fly back and forth on that line, taking repeated SHORAN distance measurements until the minimum sum was found. That point—where the total distance to the two stations was smallest—must be at the center point of the cross line. Those values were used, with the aircraft's altitude, to calculate the final measurement.

HIRAN, the enhanced accuracy of SHORAN, used the same basic principal but added a process called "gain riding" to zero out error due to the rise time of the transmitted pulses (which led to ambiguity in the "start" of the pulse). The details of gain riding have become obscure, but it was a manual process requiring a dedicated operator. I believe that the operator was simply adjusting the gain on the HIRAN receiving equipment to find the minimum point at which the pulse was detected at all—at which point the pulse must be tripping the detector at its maximum strength, and thus at the end of the rise time.

Combined with calculations to offset atmospheric effects and the electronic properties of the equipment, all of which had to be extensively researched during the refinement of SHORAN surveying, very fine accuracies could be achieved.

Still, the realities of measuring long survey lines in the North Atlantic were a challenge. One veteran of an Air Force geodetic squadron tells of hours spent flying a modified B-50 Superfortress out of Florida to make line crossings along the North Atlantic Tie. The B-50 was used because it already had SHORAN equipment installed and plenty of room for the operators, but the distances involved in the island hops challenged the range of SHORAN and they had to fly as high as possible to improve their chances of receiving the reply pulses.

The B-50 was a heavy aircraft not designed for high altitudes, and so they thundered high over the Atlantic, engines at maximum power, losing altitude with every turn and then waiting to slowly regain it. A film camera, called a reconnaissance recorder, took repeated photographs of the SHORAN instrument's display as operators adjusted the receiver and logged weather and signal conditions for later use in making corrections. The aircraft was essentially war surplus, not in its top condition, and the heat went out.

You can imagine this group of unlucky airmen, flying back and forth over the ocean in an aircraft that was coming to match the outside air at 60 below Fahrenheit. With difficult reception conditions, atmospheric effects became more significant, leading to error. It took repeated crossings of every line to reach a target statistical confidence level in the result. Back and forth, colder and colder, freezing hands struggling to write out logs, until they got the okay to come back home. They had measured the distance between a ground station in Scotland, I believe in the outer Hebrides, and Iceland—over 550 miles. This was, at the time, the longest distance measurement ever taken by such direct means.

---

The North Atlantic Tie was completed in 1955. These are hard to judge, in part due to the intrinsic complexity of geodesy. Many measurements were taken of each line, multiple lines were measured between different ground stations, and multiple ground stations were set up on each island or continent. Some measurements were almost certainly accurate to within ten feet. On the other hand, the final trilateration network, after least-squares fitting, left a troubling large inconsistency in the distance between Norwegian reference stations that suggested some uncaught error.

The North Atlantic Tie was one part of a variety of data used for the computation of the World Geodetic System of 1960, or WGS60, the first global datum. WGS60 is a direct ancestor of WGS84, the datum used by the Global Positioning System and thus the de facto datum of global geodesy today.

We now know so much more of the figure of the Earth. WGS84 models the planet as an ellipsoid with semi-major and semi-minor axes of 6,378,137 and 6,356,752.314245 meters and an inverse flattening factor of 298.257223563. These three precise figures describe only the abstract reference plane, a utilitarian simplification of a geoid that we know to be so complex in its shape that the polynomial approximations used today, if fully expanded from the functions that generate them, would have millions of terms. These approximations are accurate enough to be their undoing; they become outdated as the shape of the planet physically changes from year to year.

WGS60 was the beginning of global geodesy, but it was also the end of global trilateration. By the time the computation of WGS60 was complete, satellite Vanguard 1 had reached orbit. Phase and Doppler measurements of its radio transponder could be used to indirectly determine the position of a ground station relative to the satellite, and then by extension to another ground station. Some of these measurements were incorporated into WGS60, and its fast-following revision WGS66 was heavily corrected based on satellite measurements. The space age had come, and with it, the era of long-distance surveying had gone.

The old school of geodesy had one last hurrah: WGS84 incorporated data prepared for NAD83, which heavily used the Transcontinental Traverse, a 1961-1976 effort that physically measured the United States by headings and distances. This was likely the last large-scale physical survey in the world. By the time it was completed, networks of satellite ground stations had been incorporated into various national geodetic systems, grounding the datums ironically in space.

---

Geodesy is a complicated field with many fascinating stories. For the sake of simplicity, I have left so many details out of this sketch of the first trilateration of the globe. Today, we take accurate geospatial information for granted, and military secrecy around geodesy is mostly a thing of the past. Our maps were hard-won, though, the result of decades of ingenuity and hard effort to answer questions that have troubled scientific thinkers for centuries. Magellan sailed around the globe, but he didn't know how far he'd gone. Hotine retired from the army to help found the Directorate of Overseas Surveys, for which he explored the world again... this time, counting every mile.

During the heyday of AT&T, it was often said that the telephone system was the largest machine ever built. The "hello machine" certainly was vast, but whether or not you consider it to be a single machine raises challenging questions of definition. The Internet, I think we can all agree, is not a single machine but a system of interconnected ones. The telephone network, though, felt a lot more like one device. "One Policy, One System, Universal Service" was once the slogan of AT&T, a message that the telephone network is more than just a sum of parts. The third of these principles, "Universal Service," is a key point around which telecom policy pivots even today. As the telephone system was at its apex, Universal Service became its undoing.

We know the Internet to be a network of independent devices in part because of the wide variety of ways that we access it. Computer networks, almost to their origin, have emphasized independent implementations of standardized interfaces. Computers, as network nodes, are interchangeable. As a result, much of the complexity must be pushed to the edge, where end-user systems are most able to adapt to the unique needs of, well, the end-user. The telephone system was much different: few types of telephone instrument existed, largely from a single manufacturer. Complexity was drawn into the center where telephone offices could house the huge machinery required by mid-century automation. All of it was, for a very long time, hard-wired: central office switches and customer telephones were designed and installed to suit each other. This core difference, between the flexibility of computer networks and the central caretakership of telephone networks, was a core issue in the series of changes that rocked the telephone business in the early 1980s.

1984 is the K/T line of telecommunications, the year the sea peoples came. It is difficult to overstate the extent to which telephone technology, the communications industry, and the basic concept of what a telephone is changed between the 1970s and the 1980s. This was not a single, well-planned, carefully executed reform the way some accounts of divestiture can make it out to be. In practice, it was chaotic, messy, and often drawn out.

The deregulation, re-regulation, and fundamental reshaping of the American telecommunications industry is a complicated story. There are different actors, different agencies, different motives, and different outcomes. Nothing went quite as expected, and some of the consequences still remain to be seen. It is a hard story to tell. We can begin to understand it, though, by focusing in on just one part: the part in your house, the Customer Premises Equipment. Like the "Standard Oil" gas stations that you can still find here and there, the breakup of the Bell System left behind marks. More than anything else, it left behind telephones.

---

From the genesis of the Bell System, telephones were considered just as much part of the telephone system as the central office equipment that supported them. Despite some early uncertainty, AT&T quickly settled on a pattern in which most telephone users paid a single monthly rate that included local calling, maintenance of the local loop to their premises (which we might call "access"), and the telephone itself. When you signed up for service, Ma Bell sent someone to install your phone. When you had trouble, they sent someone again to fix it. When you canceled, or fell delinquent on the bill, they sent someone by to take the phone away.

From our modern perspective of the telephone as "consumer electronics," it's hard to picture paying rent on one. There were a number of reasons for leasing, which varied in prominence over time. Some of the uglier parts of the early 1980s involved disputes over which of these motives really mattered. Phones, we will see, were leased to consumers for many of the same reasons that early computers were mostly leased to businesses. This reflects a parallel shift in the computer industry, from "computer centers" to "personal computing," that happened around the same time and in a similar context. Before we get there, though, we need to understand how it came to be that telephones were "Bell System Property—Not For Sale."

First, we need to understand the nature of telephones and telephone wiring when the practice of telephone leasing was established. Consider, for example, that the first phones in common use were not yet "common battery"—they had local batteries, a wooden box full of big dry cells that required regular changes. Telephones were much more maintenance-intensive, and those maintenance requirements produced a troubling question of responsibility. Most people, if they attempted a telephone call and couldn't understand the other party, would blame the phone company. They'd blame the phone company even if it was the fault of the caller or callee, for not changing their phone batteries. Consider these two facts: telephones required regular service, and if they didn't receive it, the result would reflect poorly on the reputation of the phone company. For both consumer convenience and the integrity of the telephone system, it made sense for the phone company to take responsibility for the end-user's equipment.

Consider also the issue of compatibility. The type of telephone to be installed, the wiring conventions, and the configuration of the "network" of electronics in the phone could all depend on the type of exchange office it was connected to. Up until around 1970, party lines were very common (the majority of all telephones in the mid-century). They remained in use here and there into the 2000s. The problem was even more acute for party line customers: there were a half dozen fundamentally different selective ringing systems for party-line phones ¹, and different basic wiring schemes to match. One of the dominant discussion topics in telephone collecting circles is how to wire and re-wire vintage phones to work with specific equipment. Bell System Practices told technicians which wires to connect to which terminals, and which terminals to strap to which other terminals, to get a standard phone model to work in a given situation. Some scenarios required add-on boards, external cabinets, or modifications. This could all be true even of simple single-line phones, but the moment a business or large residence wanted a more complex system (e.g. with intercom calling), the complexity multiplied.

What was even worse, especially in the case of party lines, is that a single malfunctioning or incorrectly installed phone could impact service along the entire phone line. In the worst case, even a humble single-party line could sap the capacity of the exchange's switching equipment if it stuck off hook.

I should also make sure to emphasize the way that telephones were connected. Today, all telephones use "modular connectors" that simply plug into a wall socket. We shouldn't take modular wiring for granted; telephones were usually hard-wired to screw terminals until the 1960s, and even once connectorized the connectors were not standardized until 1974. In 1980, as a dark storm rose over the horizon, many telephones were still hard-wired, some still to complicated wall ringers or terminal boxes.

Considering all these factors, it makes sense that telephones were professionally installed: it would have been difficult to instruct most telephone users on correct installation, and if they did it wrong you might have to send someone to urgently fix it anyway. Since telephone wiring of the era could physically pass through multiple residences, and telephones could impact the function of other parts of the system, the telephone company viewed access to customer premises and their equipment as a necessity to ensure good service. When it came to third-party equipment, well, nothing made Ma Bell more nervous. Western Electric telephones were carefully manufactured, thoroughly tested, and precisely adjusted for reliable and compatible operation. At the first sign of trouble with a telephone, the company would swap it out for a good one. If consumers connected equipment that didn't belong to the phone company, they would lose the ability to maintain and repair a key part of their product.

Besides, people liked it, or at least so the phone companies argued. Technology was quickly improving. If your local exchange went from pulse dialing to touch-tone, for example, it was convenient that the phone was leased and would be changed out for a new one at the telephone company's expense ². When Western Electric introduced new, compact designs like the Princess and Trimline, customers had the option of upgrading their bulky phone to a slick modern one, for only a modest increase in their monthly rate (the Princess phone, introduced 1959, is also a late example of a phone that required a special wiring arrangement to power the lighted dial). When a customer redecorated, they could contact the phone company and ask that their phone be changed out for one of a different color—usually at no cost at all. Up until the 1970s, when telephone leasing started to show weakness, most of these services were accomplished by sending a technician straight to the customer's home.

As you can imagine, this level of service wasn't exactly cheap. The cost of leasing the phone was incorporated into your phone bill. If you wanted more phones, called extensions, you paid more—even if they were on a single line. If you wanted a fancier phone, you paid more. By the 1960s, leasing had clear downsides. Telephones were manufactured by Western Electric, a subsidiary of AT&T just like the operating companies, so any need to send a technician to service a phone was a hit to the shared bottom line. Western Electric emphasized reliability above all else, leading to phones that were fairly expensive to manufacture but could easily last 25 years in service. Bell operating companies, looking to cut costs, shifted towards a retail model in the 1970s in which customers were highly encouraged to take their phone to a PhoneCenter for exchange, rather than waiting for a technician to visit their home. The adoption of modular connectors was motivated in good part by the development of PhoneCenters.

If a telephone was something that you could pick up at a store, and then take home and plug in yourself, did it really make sense to lease? Did consumers have any reason to lease phones besides Ma Bell giving them no other option?

---

AT&T was never a stranger to antitrust action. The Department of Justice filed a suit in 1949, alleging illegal monopolization under the Sherman act. It was generally acknowledged that telephone service was a "natural monopoly," meaning that the cost of installing and maintaining the outside plant made it inevitable (and, on grounds of economic efficiency, preferable) that a single company would dominate each market. The Sherman act and the Department of Justice did not necessarily object to this form of monopoly. The problem, what Justice considered anti-competitive behavior, was when the phone companies used their natural monopoly on phone service to force customer's hands in markets that might otherwise be competitive.

One of the tricky problems, even in 1949, was interconnection of equipment. A vast and increasingly global electronics industry had come up with all kinds of new ways to use communications lines, but telephone customers could only use the equipment their phone company provided, and at rates set by the phone company. A fervent argument ended in the conclusion that AT&T had valid reasons to require customers to lease telephones, but that the anti-competitive impacts needed to be mitigated. This case, the first of a series of challenges to the Bell System, ended in a consent decree that shaped the future of both AT&T and, I do not think it is unfair to say, the entire American electronics and communications industry. AT&T would continue as a monopoly telephone carrier under state regulation (the principle of "common carrier"), but it would be prohibited from entering any industries in which it was not regulated.

In other words: telephone service was a regulated industry, with rates and service terms set by state public utility commissions, and the system of regulation was felt to be sufficient to temper AT&T's desire for domination. On the other hand, if AT&T entered an unregulated industry, there would be no public commission to slow them down—and they would inevitably use their foothold as a regulated monopoly to promote, perhaps even mandate, use of their unregulated service. I have so far described the settlement in abstract terms, but even at the time it had an obvious practical implication: "computers," in the modern sense, had only just been conceived. But they were rapidly advancing, and AT&T's considerable leadership in electronics, massive R&D center (Bell Laboratories), and integrated manufacturing (Western Electric) made the telephone company an obvious contender for the nation's leading manufacturer of computers.

Too bad: computers were an unregulated industry, so AT&T had to keep their hands off. The consent decree forbade AT&T selling general-purpose computers, broadly limiting their involvement in the electronics industry to telephone switching equipment and government contracts. AT&T mainframes never happened; instead we got IBM. The capability was still there, though, and complex needs of the telephone kept AT&T a leader in computer technology even with zero sales. Consider, for example, UNIX: perhaps the single most influential operating system ever developed, but a product of a company that was expressly prohibited from selling computers. In an era where software was virtually always bundled with hardware sales, that put AT&T in a very odd position, and forced the odd licensing terms of UNIX that allowed it to proliferate and organically grow.

The 1949 consent decree held for decades, but it was always a bit uncomfortable. By the 1970s, AT&T's key regulator, the FCC, came to feel that it no longer made sense. The computer industry was much larger than before, interconnection of computers via telephone was becoming a common practice, and it was apparent that computers could both provide a communications service and be users of a communications service. AT&T built and used computers extensively, and one could argue that a telephone on an Electronic Switching System (ESS) was akin to a terminal on a mainframe computer. The first time-sharing services, such as BBN's introduced in 1962, might use the telephone network for communications but could also offer communications themselves in the form of electronic mail. Was the telephone system a computer service? Was time sharing a communications service? These were important questions that the existing regulatory and legal framework did not comfortably address, and they were only becoming more important as both AT&T and its competitors signaled a desire to try out the other side of the fence.

Moreover, the FCC was concerned that the regulatory environment was holding the industry back. AT&T's complete control over the phone system had already started to crumble, first when an appeals court overruled the FCC to allow the use of non-electrical third-party telephone accessories in Hush-A-Phone v. United States in 1956, and then again when the FCC's own Carterfone decision extended the privilege to even electrical equipment in 1968. While the Carterfone itself was a special-purpose device, the decision set a precedent that was immediately leveraged by third-party telephone manufacturers.

---

Given the dominance that AT&T has held in this discussion so far, and that it holds in telephone history in general, "third-party telephone manufacturers" might come as a surprise. Of course there were telephones made overseas, for example by Ericsson, and some were imported to the US. But the US had its own competitive telephone industry: Automatic Electric was the Western Electric of GTE, AT&T's main competitor, and manufactured the telephones they leased. GTE looked at the possibility of selling telephones to AT&T's much larger customer base and saw green. Railroads and independent telephone companies needed equipment, and while they did buy from Western Electric, they also supported other manufacturers like ITT.

Enterprising retailers started selling telephones from these companies outright (it is hard to tell who was first, but new department store chain Target was early to the game). The Bell System, in response, introduced harsh rules: customers could use third-party equipment, but only by connecting it to a device called a Protective Connecting Arrangement (PCA) leased from AT&T. The PCA was about as expensive as a telephone and installed the same way, requiring a separate PCA for each extension and preventing customers (or their chosen contractors) doing any work on the inside wiring, which remained Bell System property. This in-between state technically opened the telephone set market to competition, but in practice the rules and fees around PCAs were so onerous that few consumers were better off owning their own equipment.

Pressure from telephone manufacturers and consumer groups, and no doubt some fear of another appeals court loss after Carterfone's clear precedent, the FCC promulgated new rules for third-party telephones in 1975. These are basically the same rules we follow today: the customer owns inside wiring past the network interface device (often called a demarcation point for this reason), and can do whatever they want with it, subject to a requirement that interconnected electrical equipment be certified as compliant with Bell System technical rules and registered with the FCC. This was the true beginning of the customer-owned telephone era: many customers would sign up for telephone service, buy a phone, and then simply report the FCC registration number of their own telephone to the phone company. That was all that was required to have service connected.

Yet it creates a strange situation. You would have two options: you could pay the rate advertised by the phone company and receive a phone on lease, or you could buy your own phone and pay a somewhat lower rate. The latter was almost certainly a better deal, especially as the sudden creation of a retail telephone market drove prices radically lower. The former could look like a better deal, though, since the phone was "free" and its cost partially concealed in the phone bill. It could easily become even more confusing: let's say that you leased a phone from the phone company for years, and then went to check out this newfangled Target and set your eyes on a slick new ITT telephone. You buy it, take it home, and if you are lucky enough to have modularized connectors, you plug it right in (post-1975 wiring conversion kits were also readily available). Now you have a phone that you own, but you're still paying for the one that you leased, perhaps without even knowing. Hopefully you kept it: when you notice the overpayment and call the phone company to cancel the lease, they will expect you to return the phone, or otherwise pay them back for it.

And thus came the first blow to telephone leasing: state regulators increasingly required telephone companies to separately itemize the cost of a telephone lease on bills.

---

At the same time that Carterfone and its consequences shook up the rules around third-party equipment, the FCC initiated a more radical rework of telephone regulation. The computer had raised many questions, and especially the rise of time-sharing services and the nascent field of "information services" seemed to create a second, shadow communications industry that was unregulated (and thus competitive) but incredibly vulnerable to the whims of the regulated (and thus monopolized) telephone industry that provided the actual wiring. In keeping with the political winds, the FCC strongly favored market competition. It feared that AT&T had become "too big to regulate" and would use its chokehold on the nation's primary communications infrastructure to stamp out computer-based competition before it could bloom.

To consider this thorny problem, the Commission had opened a broad docket called the Computer inquiry in 1966. Among the conclusions of that multi-year series of working papers and hearings was a new regulatory framework that differentiated between "communications" and "data processing" as industries. "Communications" was a natural monopoly like the phone system, and would carry on under regulation. "Data processing" was an increasingly competitive industry with lower barriers to entry, so it would remain unregulated.

The FCC's largest concern, as in the 1949 Sherman act case, was what would happen when one company drove in both lanes—as AT&T still wanted to do. Such a company could probably use its regulated monopoly services to subsidize its competitive service (even if regulators tried to stop it, since accounting for what investment went where would be very difficult), unfairly blocking the efforts of competitors and driving up prices for consumers that had no other choice.

On the other hand, completely prohibiting regulated common carriers from participating in competitive markets was, by this time, clearly having a negative impact on the development of computer technology. AT&T was a major center of expertise in computers that had to keep most of that expertise to itself, and innovative computer companies like Tymshare were nervous about offering communications features (and sometimes prohibited or arbitrarily limited them) for fear of straying into common carrier territory and, ironically, being forced to abandon their original unregulated services.

The FCC attempted to address this conundrum through a rule of separation. One company would be allowed to participate in both regulated and unregulated markets if, and only if, the two activities were strictly separated into two different entities (one presumably a subsidiary of the other) with accounting and policy firewalls between them.

In practice, the Computer inquiry was a mess. The information services industry was still in its infancy in the 1960s, so the FCC was writing rules without knowing very much at all about the companies they would apply to. The FCC acknowledged that there remained a clear gray area between "communications" and "data processing" (with, once again, electronic mail as the clearest example) but still wrote the rules around "pure communications" and "pure data processing" as two separate domains. The existence of a "hybrid" category was acknowledged but little discussed. One of the key problems related to the actual technical implementation of computer networks: telephone networks all used circuit-switching of analog signals and so there was, at the very least, a logical fiction that all information passed through unmodified. Computer networks, on the other hand, were adopting more sophisticated architectures that often involved rewriting, modifying, or converting messages along the way. Indeed, the lack of standardization of computer communications prior to their domination by the Internet meant that some amount of protocol conversion was practically required for widely-available services. And yet, this type of manipulation of messages appeared to be data processing, not communications. In practice, almost everything turned out to be "hybrid."

This is why the Computer inquiry is now widely known as Computer I: it was almost completely replaced by the second Computer inquiry, or Computer II, in in 1976. Computer II kept the same basic principles as Computer I except that it completely redefined the two categories in terms of functionality, rather than technology. The Computer II categories are "basic services" and "enhanced services." Enhanced services were those that involved manipulation or storage of data, but the FCC made that determination from the perspective of a user rather than an engineer. If the purpose of a service, or the activity in the FCC's words, was manipulation, storage, or retrieval of data, it was an enhanced service. If not, it was a basic service, even if the technical implementation involved some degree of manipulation, storage, and retrieval. This had the effect of making the telephone network a basic service, and computer networks an enhanced service ³.

The separation rule remained the same: one company could offer both basic and enhanced services, but only under terms of strict separation.

To the FCC, Computer II seemed a relatively elegant solution to one of their major problems. Under the definitions adopted, basic services were limited to the actual communications network. Just about everything else would be an enhanced service, firewalled away from any benefit from the regulated monopoly.

Simultaneously with Computer II, the advent of retail telephone sales (remember, despite appearances, this is actually an article about telephone leasing) triggered several lawsuits and petitions to the FCC. The general theory was the telephone companies used their monopoly status to do everything possible to steer customers towards leasing phones, even requiring at least one leased phone in some cases. The FCC recognized that this issue was not totally unrelated to the Computer inquiries: Computer II in particular had led to consideration of very similar problems around modems and the ability of basic service providers to restrict enhanced services over their network (during this period, for example, you could either lease a modem from the telephone company or buy one from IBM—they were functionally identical, but it was not always clear which option was more legally appropriate).

More than one court ruled that, to settle the antitrust problem, the telephone monopoly needed to get out of the leasing business. The FCC agreed in principle and, even better, in practice: it was a simple and obvious step to sweep "provision of terminal equipment" under the same rug as "enhanced services." Telephones, like enhanced services, were a competitive industry with lower barriers to entry. Therefore, like enhanced services, the FCC ruled that a regulated telephone company could only lease terminal equipment through a strictly separated subsidiary.

---

Finally, the two threads of computer-related antitrust action and telephone leasing converge, in the curious form of the short-lived American Bell.

Immediately following the conclusion of Computer II in 1980, AT&T disclosed to the FCC that it intended to take advantage of the new rules by forming a strictly separated subsidiary just as the FCC had contemplated (this was no surprise, as AT&T had been extensively involved in the inquiries). The mechanics were tricky, as AT&T had to come up with the capital to start a completely new subsidiary, with its own offices, support functions, equipment, etc., all without the appearance of subsidizing it with funds from their regulated business. Negotiations between the FCC and AT&T over these details delayed the incorporation of the subsidiary until 1982, when it emerged under the brand name American Bell.

American Bell had two different divisions, in a way that seems quite incongruous without the lengthy regulatory context I have just unloaded on you: Advanced Information Systems, a computer and information services company widely expected to rival the likes of IBM, and Consumer Products, which manufactured and sold telephones.

As a result of the strict separation rules, American Bell would have to replace the telephone leasing functions of the Bell Operating Companies (which were, in 1982, still part of AT&T). The storefronts the BOCs had opened to handle phones, the PhoneCenters, were quickly rebranded from the BOC logos to American Bell. AT&T also launched an expansion campaign that saw, for example, a PhoneCenter as a department in every Sears. They even sold phones wholesale to retailers, where erstwhile Western Electric products sat alongside those from ITT, AE, and the new bevy of imported Asian products ⁴. Most consumers, when they signed up for telephone service, would just go to a PhoneCenter or any other store and buy a phone.

In fact, starting in 1982, leasing might not even be an option: the FCC had agreed to soften the blow to AT&T's balance sheet by allowing the BOCs to continue to sell or lease the telephones they already had in inventory. It was clear that this was a temporary accommodation only, though, as the FCC ordered that the BOCs could not purchase any additional inventory of customer premises equipment, and ordered state regulators to begin the process of removing telephone leases from the tariffs. Telephone company representatives told newspapers across the country that, starting in 1982, they would still "provide" a phone if they had one—but they might not, and customers couldn't be picky about the color. Some predicted their inventory would run out during 1983, and then customers would truly be on their own.

Of course, customers didn't take this as hard as the phone companies did: retail phones were quite popular, and leasing numbers had been in decline since the late 1970s. We should finally talk about prices. From New York Telephone in 1983, as an example, you could lease a 500-style rotary dial phone for $3.03 a month. Purchasing the same phone at a PhoneCenter cost $45—so if you expected to have the phone for two years, you were better off buying. And we must keep in mind that Western Electric phones, while not totally immune to cost engineering, were still built to very high standards and came with a price tag to match. A more inexpensive phone, say one imported from Asia, was available at Radio Shack for $15. AT&T representatives discouraged the use of these cheap and ostensibly lower-quality phones, but struggled to say anything negative about a phone from, say, ITT, given that they were built to the same standards and sometimes to the exact same designs.

The phone was no longer part of the telephone system. It was merely a consumer device, subject to the same forces as every other.

But what of the many leased telephones already in service? The FCC referred to these as "embedded" phones, and had a very hard time deciding what to do about them. Because local telephone companies were prohibited from the leasing business, and the phones had always been handled under the auspices of Western Electric anyway, all of the leased phones were property of AT&T. The local telephone companies installed them, but the rent went to the parent company.

The FCC could require the telephone companies to stop leasing phones and return them all to Western Electric, but that would leave AT&T with a huge financial loss as they wrote off millions of phones they were no longer allowed to sell. They could require the telephone companies to offer customers a buy-out offer, but that created most of the same problems as having the companies lease them in the first place. Who would set the price? presumably the regulator, but how? Would they come with a warranty? For the time being, the FCC settled on a temporary solution: the embedded leased telephones would be left as-is.

AT&T stridently objected. They were being told three things: that they needed to come up with the money to fund a new independent subsidiary to sell telephones, that they had to keep servicing the existing fleet of leased phones, and that in no way could money from the latter be used for the former. AT&T suggested that the FCC should instead let them offer a buy-out on all leased phones, and then use the revenue to bankroll American Bell Consumer Products. The FCC brought back the question of how to set the price, especially given that some of these leased phones were very old. Would they need to be appraised? This stalemate lasted long enough for the ground to once again shake under the feet of the telephone industry.

In 1974, as Computer II was just about to begin, the Department of Justice had decided to take action for the same reasons as the FCC. The monopoly questions around the telephone market, mitigated but not totally addressed by the consent decree in the 1949 case, had become clearer and more urgent. Here is where the story of telephone deregulation falls apart, and where I introduce the character we all know best: United States v. AT&T, the second one, a Sherman act antitrust case that ran in parallel to, but almost completely independent from, Computer II. Judge Harold Greene would have to answer the same questions as the FCC, and he did not answer them the same way.

---

The breakup of the Bell System is a fairly well known topic, so I will keep my explanation brief. The main outcome of the later United States v. AT&T, set in a 1982 settlement called the Modified Final Judgment (MFJ), was a requirement that AT&T divest itself of the local operating companies.

This was a very different outcome from Computer II, but the reasoning along the way was similar: the two inquiries were just looking at different aspects of the problem. Computer II was motivated mostly by the up-and-coming world of computerized information services, so the FCC focused on AT&T's use of the telephone network to bolster its information services business. The MFJ, on the other hand, was mostly motivated by the rise of competitive long-distance carriers like Sprint and MCI.

That problem caused the court to draw a different distinction: local telephone service was deemed a natural monopoly, because of the massive outside plant required. On the other hand, the development of microwave and fiber optic technology had radically reduced the cost of long-distance connections. Through products like Execunet, AT&T's competitors were already proving the viability of competitive long-distance services that interconnected with the Bell System at the local exchange.

While the story of United States v. AT&T is a complex one, it goes roughly like this: the court initially favored a solution that seems more similar to that of Computer II, by ordering AT&T to separate itself from its vertically integrated but non-regulated components. That would mean separating AT&T's local and long-distance telephone operations from Western Electric and likely Bell Laboratories, probably by spin-out into a completely independent company. Ironically, this was the ultimate (and perhaps foreseeable) outcome, but nonetheless AT&T fervently opposed the loss of the division that would lead it into the computer future.

Instead, AT&T got a very different outcome: if local service was the real problem, the antitrust problem could be addressed by breaking out local service instead. AT&T seems to have preferred this route, but it was no less traumatic. The MFJ set a timeline of two years to implement the judgment: On January 1, 1984, the Bell Operating Companies became independent of their former corporate parent. AT&T was prohibited from local telephone service, but in exchange, was permitted to continue its long-distance business with few compromises—besides the big one, that it would now face competition.

For the parallel story of leased telephones, this was a huge wrench in the gears. Leased telephones had always fallen under the purview of Western Electric (which operated the service centers, for example), and local telephone companies were now prohibited from leasing phones anyway, so embedded leased telephones had been moved under the American Bell Consumer Products brand. But American Bell would now be not just a separate subsidiary from the telephone company, but a separate subsidiary of another company that the telephone company just paid for long distance service. Closely following state regulator decisions about separating phone lease billing, they became even more separate: starting in 1984, consumers with leased telephones would receive two completely separate bills, one from the bell operating company and one from American Bell.

But there was another problem: The "Bell System" as it was known was the AT&T monopoly, and the court made sure to take it from AT&T with the rest of the monopolized service. Among the terms of the 1982 MFJ was a prohibition on AT&T using the Bell trademark and name (except in limited circumstances), a rule that the still-new American Bell directly violated. Beginning in 1984, American Bell was rebranded once again to AT&T Technologies, with AT&T Information Systems as one subsidiary and AT&T Consumer Products as another. The signs of over a thousand PhoneCenters were swapped out yet again ⁵.

Ironically, the local operating companies, prohibited from leasing telephones since 1982, actually regained that ability post-divestiture. Since they were completely separate companies from AT&T in 1984, they were free to form their own "strictly separated" subsidiaries to offer unregulated services. You will notice the overlapping timelines: the Computer II-era prohibition on leasing took effect in 1982, the same year that the MFJ was signed. When the BOCs were telling newspapers that they didn't know how long their phones would last, they already knew that they would be permitted to resume leasing operations in 1984, they just weren't sure if they would. This did little for consumer confidence in leasing telephones. In practice, as a business decision, most of the BOCs never leased telephones again. The market was already going downhill, and they would have to try to sign up customers against the competition of both cheap retail phones and existing lease arrangements with AT&T Consumer Products.

---

Along the way, state regulators had slowly come up to speed with the FCC's rulings on leasing. The details here are very hard to tell, in part because they seem to have been different across both jurisdictions (different state regulators took different approaches) and markets (different Bell operating companies lobbied their regulators for different approaches). Further confusing things, some states seem to have adopted an approach to embedded leased telephones before 1984, and others after. These decisions were made in very different contexts and led to different outcomes.

It shook out along these lines: starting in 1982, the FCC required AT&T to keep supporting leased phones for at least 18 months. AT&T offered to continue leasing services indefinitely, which was ultimately permitted. Still, the FCC was making (halting) efforts to implement Computer II, and pushed states to disposition the embedded leased phones. Most states did so by requiring, or perhaps more accurately allowing (since it had been their preference to begin with) AT&T to offer buy-outs to existing lease customers. Most of these buy-outs had to settle by 1984 as divestiture radically complicated the servicing of leased phones—sending a technician to repair or even retrieve one now required a complicated cost-sharing arrangement between the local operating company (which employed the technicians) and AT&T, all closely scrutinized by the FCC.

Quite a few customers took the buy-out offers, but many did not, and by around 1985 the buy-out programs had faded away.

For those customers who had leased telephones, and did not respond to mailings about the buy-out option, they simply remained lease customers of AT&T Consumer Products which issued a separate quarterly bill. And this is where the long twilight of the leased telephone begins.

The 1982 Computer II restrictions and the 1984 United States v. AT&T restrictions came in rapid succession and, it was quickly observed, seemed to contradict each other in goals. You could be either a local telephone company or a long-distance telephone company but not both... or you could be a regulated company or an unregulated company but not both. The original goal of separating monopoly telephone service from "enhanced services" had effectively happened twice, by requiring AT&T to form a separate subsidiary for unregulated services and then two years later requiring them to divest most of the regulate services that had prompted the separate subsidiary to begin with.

AT&T was mad, of course, but even the FCC came to agree that their attempts to promote free market competition had probably actually undermined it by taking away AT&T's ability to meaningfully compete. In 1986, a series of comparatively minor FCC decisions rolled back some of Computer II's implications, including the strict separation rule for AT&T. Free of the regulatory requirement for the business complexity (which had led to frustrations like the famous exclusion of AT&T Technologies employees from the Bell Laboratories library), AT&T reabsorbed its separate subsidiary. AT&T Consumer Products was once again just another division of the faltering giant.

---

AT&T's computer ambitions were ill-fated. Despite the introduction of the 3B line of UNIX-powered minicomputers, and even a line of PCs, AT&T never achieved meaningful success as a computer company. AT&T entered a broad financial decline after the breakup, and the faltering computer business wasn't helping. Still, AT&T Technologies—the division made up mostly of the former Western Electric and Bell Laboratories business units—had potential, or at least they liked to think so.

In 1996, AT&T Technologies once again spun out, this time into a completely independent company called Lucent Technologies. Lucent became the home of the manufacturing and research operations, but because of the circuitous history of AT&T Consumer Products, the embedded leased telephones came along for the ride. There were, as best I can find, approximately one million telephones still under lease when Lucent became the lessor. As an accommodation for billing and to reduce consumer confusion, AT&T granted Lucent a license to continue to use the name "AT&T Consumer Products" for leasing.

Lucent didn't do all that much better than AT&T had, and it was more or less stripped for parts over the following decade, leading to a 2006 acquisition by French communications technology company Alcatel—creating Alcatel-Lucent, later part of Nokia. The leased phones took a different path, though. The consumer telephone business was not profitable for Lucent, and the PhoneCenters closed in 1995. In 1996, the division took on the name Lucent Consumer Products and shifted its focus to wholesale, although it continued to use AT&T branding in connection to leasing.

In 2000, it all had to go. Lucent eliminated almost the entirety of its manufacturing capability, mostly as Avaya which produced networking products and business telephone systems. The consumer products division mostly went to the well-established Hong Kong manufacturer VTech, which continued Lucent's consumer products under the name Advanced American Telephones. But there was an exception: the lease contracts. In the year 2000, there were still several hundred thousand.

That is where the story becomes especially difficult to follow. Lucent sold the phone leasing business to what Wikipedia calls "Consumer Phone Services," although I am now quite confident this company was properly called North Street Consumer Phone Services LLC. I do not know the meaning of "North Street," and I have found conflicting information on whether the company was based in New York, Miami, or somewhere in New Jersey. It probably moved around a bit. North Street seems to have incorporated in 1999, suggesting it may have been formed for this purpose. Further confusing things, as late as 2004 the leasing program was still "managed by" Lucent. I am unclear on what "managed" meant, but newspaper articles about phone leases in the era often feature statements from Lucent spokespeople despite the sale. At the least, Lucent was allowing North Street Consumer Phone Services to continue to use the AT&T name under license.

Around 2006, perhaps in connection to Lucent's sale to Alcatel, the "managed by" relationship apparently ended. In 2008, North Street Consumer Phone Services was replaced by QLT Consumer Lease Services. I know very little about this company, I am not even sure if it is a descendant of North Street Consumer Phone Services or a result of another sale of the lease contracts. There can't be very many lease contracts left. Considering that most marketing for phone leases ended in 1982, anyone still leasing a phone today has most likely been making payments for the last 45 years.

---

Are leased telephones a scam? Perhaps, even probably. By the mid-1990s, the FCC and Federal Trade Commission realized that an inadvertent effect of the 1980s regulatory chaos was that consumers who had dutifully paid their phone bills but not otherwise paid much attention were likely still making lease payments. That probably described most telephone customers who had started service before '82, but by the mid-'90s leased telephones were already becoming a distant memory for most Americans. How many people had a leased phone that they had forgotten about? How many people had a leased phone that they had replaced with one they bought, but never returned?

In theory the separate billing rules mitigated the problem, but it was clear that not everyone read their bills closely. The fact that the Bell Operating Companies and American Bell Consumer Products had switched to billing on separate days was intended to reduce confusion by making sure the bills arrived completely separately, but especially after phone leases switched to quarterly billing (due to their small amounts) it probably made it more likely that people would think both were the phone bill and simply not notice that they were getting two different sets.

In 1996, the FCC and FTC issued a press release warning consumers to check their phone bills for forgotten lease payments. An accusation was made that Lucent Consumer Products was functionally exploiting the elderly, since older people were more likely to have had the same telephone service for decades and probably less likely to critically consider a $15 quarterly lease bill. When you did the math, it was easy to find customers who had had the same leased phone for the last 20 years and paid over one thousand dollars for it. As time passed, more and more of those customers didn't even have the phone any more, but they still got the bills, and they still paid. When asked, many thought the prominently AT&T-branded lease bill was for long-distance calling (it didn't help that Lucent seems to have used some very non-specific language on the bill like "equipment fee").

In 2002, lawsuits in several states were combined into a national class action case that alleged that AT&T and its various successors had scammed vulnerable Americans out of many millions of dollars by simply continuing to bill them. While no one involved admitted fault, they did settle by creating a fund of up to $300 million to reimburse customers who filed claims that they were paying a lease for a phone they did not use. The reimbursement program ultimately paid out less than $10 million due to lack of claims, which consumer advocates quickly criticized on the grounds that people who did not realize they were paying a lease bill probably weren't following developments closely enough to find out about a class action settlement they needed to apply to.

Interestingly, even the composition of the settlement fund was complex. The billing at issue had happened under multiple companies, and many of the rounds of divestiture and acquisitions had come with contractual allocations of liability. AT&T (the original parent company) and Lucent (the spinoff) ended up bearing most of it, but amusingly, National Cash Register was hit for a few million dollars because of liability assignments during the brief period that it was part of AT&T Technologies (the spinoff of Lucent included NCR, and assigned part of AT&T Technologies' liability to NCR that conveyed with the company when Lucent spun it out again). NCR basically caught a stray for a business they were never involved in.

And yet this is still going—in 2004, 2005, 2008, it's easy to find newspaper articles, cable news segments, and AARP columns recounting mostly elderly people who had paid thousands of dollars in lease fees for telephones they no longer had. AT&T, Lucent, and QLT have never had much to say in their defense. In the late '90s, Lucent pointed out that phone lessees got all kinds of fringe benefits, not only the ability to exchange their phone for a different color but a "lease rewards" discount program. QLT offers the same discount program today, the same type of pharmacy discount and coupon card that you can get dozens of other ways.

It's very hard to find much information on QLT, although they're clearly still alive and issuing a charming thrice-yearly newsletter called Lease News & Views, presumably as a bill insert. Well, I say still alive, but the Spring 2026 issue is late and it seems like Summer 2026 ought to come out any day now. I have a lot of basic questions about QLT. For one, what does QLT mean? Well, I figured that one out: it doesn't mean anything, it's just supposed to sound like "quality" when said out loud. I can't find this clearly stated anywhere, but my assumption is that either AT&T revoked the license agreement to the AT&T brand or QLT independently decided that it was better to stop using it—either way, in response to the press coverage and lawsuits that attributed consumer confusion to the phone company branding on bills.

I also wonder how many customers they have. In 2012, QLT responded to a CBS article on yet another elderly person paying for a leased phone for 30 years by noting that they have over 300,000 customers. How many of them are still paying, 14 years further on? QLT has "11-50 employees" per LinkedIn, several of which have been with the operation since divestiture.

In response to a 2019 article on elderly people still paying phone leases, QLT shared that "Our customer research data shows that approximately three quarters of our surveyed leasing customers also have at least one purchased phone in their home." To me, this seems to undermine QLT by suggesting that many of their customers are making lease payments for no reason, but they spin it by positing that these customers demand the reliability of a leased phone as a backup option.

What I can tell you is this: If you want a Western Electric 500, or really a Cortelco 2500, they're only $5.95 a month from QLT in touch-tone. Or $45 to buy outright.

Email, one of the most pervasive and enduring technologies of computer networking, was invented in about a dozen places by dozens of people in the 1960s. It's hard to lay out a clear history of the technology because it's just so obvious—pretty much as soon as more than one person could use a computer, there was some kind of mail facility. These ranged from mainframe-centric systems where all of the users of a single computer could write messages to each other, to PC-centric systems where workstations would mount a network share to store and retrieve messages. Pretty much any scheme you can come up with for moving messages was probably in use somewhere from roughly the 1960s to the 1990s, by which time the ARPANET-derived family of email implementations had taken hold.

This form of email has a clearer heritage, to Ray Tomlinson, who came up with the core idea that addresses could identify both a user and a host, and that some kind of open protocol could be used to send messages to another host when necessary. Over time, and with many revisions, Tomlinson's design became SMTP and was joined by protocols like IMAP that built out the form of email we use today. This is a form of email that is in some ways decentralized (any user is free to choose a host) and in other ways centralized (each host assumed to be continuously online to store-and-forward messages for its users). Tomlinson's design was flexible enough that we have not had to totally get rid of it, but enough has changed about the modern Internet that we have had to take a new approach.

Email has many vexing limitations, artifacts of its age. For example, email handling should not be assumed to be "8-bit clean"—email protocols were originally defined over 7-bit ASCII and ran on many machines that used the eighth bit as a checksum. These machines were prone to changing the last bit of each byte, or otherwise mishandling email with 8-bit content. That wasn't a problem when text was completely limited to that 7-bit plane, but both Unicode and the desire to send binary files made 7-bit email unworkable. MIME was developed as a workaround, an encoding technique that solves a few problems in one go by encoding all non-ASCII content of email in the form of ASCII characters.

This is rather inelegant. MIME's ASCII encoding, whether quoted-printable style or base64 style, is annoying for humans to read and inefficient with regard to message size. As with all base64-like encodings, it counts as a suspicious smell, an indication that we are papering over past mistakes. Email is full of them. Let's consider another: security.

"Internet email," a term we can use to try to explicitly specify the SMTP-type email system inherited from ARPANET, uses a simple design that dates to an era when the operators of networked computers all knew each other. While there were other possibilities, most email was delivered directly to the computer of the recipient, and in a lot of cases the host-level security of the era wasn't very good either. The possibility that someone else could read your private email was known and there wasn't a whole lot you could do about it. The internet existed within a context of trust.

We don't extend that same trust today. Email is usually handled by third-party mail services on either end, which are not necessarily trustworthy, and to get between the two third-party mail services it has to pass through a series of internet links that are also of questionable security. Email became one of the first, probably the first, compelling application of large-scale computer networking. It may have also been the first of the field's enduring security failures. The system was designed with no concern for message confidentiality or integrity, and much of it still operates that way today.

Still, there's a long history of efforts to improve email security, and we should probably start the story with the greatest of all: not just an email protocol, but an entire stack of protocols that promised a better internet in general: the OSI suite.

X.400

The modern internet was born of an era often called the "protocol wars." When we tell the story of the internet today, we tend to simplify it to a nice linear tale in which ARPANET was created, everyone thought it was a good idea, and it proliferated. That's not totally wrong, but it leaves out a lot of the other actors. The telecom industry famously had its objections to the packet-switched model, which had originated from the computer industry rather than the communications industry and showed it. During the 1970s and 1980s, the telecom industry defined its own set of network protocols, and fundamental approach to computer networking, which came to be known as the OSI protocol suite.

Today, the relevance of the OSI protocols is limited to the frustrating insistence of modern networking textbooks on trying to explain the internet in terms of the "OSI model," a now more speculative than actual layer model that describes a different system incompatible with the internet. The reason that academia has this weird fixation on using the design documentation of a failed internet to explain the fundamentally different architecture of the other internet that succeeded... well, that itself is fallout of the protocol wars. History may be written by the victors, but the losers still get to dictate theory.

Despite my ongoing project to discourage the use of the OSI model for learning, we can indeed learn from the OSI protocols. The reasons that the TCP/IP stack succeeded over the OSI stack are many, one of them being that OSI's "design by committee" approach led to a set of protocols that are very thoroughly thought out but quite complex. This is a big difference from the internet protocols, whose design philosophy was closer to "good enough for now."

The difference in philosophy is especially stark when we look at the application-layer protocols. The "internet protocol suite" as it is conventionally defined does not include application protocols at all, itself a telling fact. The OSI suite does. Internet email started out as a special-case use of FTP, before moving over to a loose set of commands over Telnet that eventually formalized (or ossified, depending on who you ask) into SMTP.

By the time the OSI standards were published in 1984, it was known that email was one of the killer applications—the first revision of SMTP was published in 1982, a post-facto response to the popularity of email on the internet, and besides, email had been a key feature of most proprietary networking systems since the 1970s. So, OSI had a solution: X.400.

X.400 is the "Message Handling System" part of the OSI suite, and provides features which are recognizable as email but also a bit more generalized. X.400 was, in most regards, a failure. It was unsuccessful in displacing internet email in almost all environments. Still, X.400 was decidedly influential. There was a period, albeit brief, in which it was thought that major governments would require X.400 compatibility in future purchasing. This was similar, and had a similar effect, to the period where POSIX-compatibility was a government requirement: a lot of vendors designed their products to meet the requirement, the government lost interest, and it became a curious legacy. In email, this is most obvious in the form of Microsoft Exchange: Exchange started out as an X.400 implementation, mostly for government reasons, and still provides a lot of X.400 features.

X.400 is vastly more complex than internet email. For example, the encoding of X.400 messages is ASN.1, a binary serialization format that is more efficient and capable than MIME but also a lot more complicated. That X.400 has complexity stacked on top of complexity is useful in understanding why one of X.400's most compelling features didn't go very far: encryption.

If you are sending an email to someone, and you don't want anyone else to read it, the most obvious approach is end-to-end encryption. For many practical reasons, you'll want to use asymmetric encryption. You find a public key for the recipient, encrypt the message to that key, and then send it over the wire as usual. The recipient then decrypts it with their private key. Simple enough, except that every part of this process is laden with complexity.

To underscore that point, let's take a look at X.400 itself—the 1988 "Blue Book" version, which seems most appropriate to when email was ossifying.

Aspects of an asymmetric key management scheme to support the above features are provided by the directory system authentication framework, described in Recommendation X.509. The directory stores certified copies of public keys for MHS users which can be used to provide authentication and to facilitate key exchange for use in data confidentiality and data integrity mechanisms. The certificates can be read from the directory using the directory access protocol described in Recommendation X.519.

Recommendations for other types of key management schemes, including symmetric encryption, to support the security features are for further study.

End-to-end encrypted email requires two basic parts:

First, there must be a technical format for moving encrypted messages through the message transfer system. X.400 handily resolves this by the use of ASN.1 definitions that allow for encrypted messages bodies, thus making most of that someone else's problem. X.400 actually contemplated a lot more when it comes to security features, things like non-repudiation of delivery, that required more complexity in message transfer but didn't make it to the modern day. We'll just ignore those for now.

Second, there has to be a key infrastructure. You need some way of obtaining a public key for the person you want to send a message, and of being sure that it really belongs to them. This is important both for encrypting outbound email and for authenticating inbound email, since end-to-end encryption is usually deployed alongside end-to-end authentication using the same cryptographic infrastructure. In the case of X.400, that whole problem is deferred to the X.500 family of protocols.

What is X.500? It's our old friend, directory access protocol. The X.500 directory standards are the comparison by which LDAP is "lightweight," and there is a parallel story: LDAP is the scrappy internet protocol that won against the X.500 Goliath.

So the gist of X.400 email security is this: use the directory to look up the recipient's public key, and then encrypt the body of the ASN.1 message object to that key. In actuality, the details are different and more complicated, but that goes for everything OSI and the simplified sketch here is enough to get the gist. It's straightforward.

There are just two problems: ASN.1 as a format for messages did not succeed, and the concept of a global directory never materialized.

To the extent that X.400 succeeded, its security features were actually a big motivation. The ISODE Consortium, which had created something like a reference implementation of much of the OSI application suite, is still around today as Isode Ltd and still sells a message handling system. There are a few other maintained X.400 descendants as well. One of the largest customer bases is military: while less common in the US (keep in mind that internet email was originally developed under the aegis of the US military), a number of European countries adopted X.400 as their secure messaging solution and still have extensive reliance on it in military and intelligence organizations. Other enduring strongholds of X.400 include global aviation (ICAO's messaging system is X.400-based) and electronic data interchange, or EDI, a standardized system of financial and supply chain messaging between ERPs.

It is no coincidence that these are the same kinds of applications where protocols like X.25 survived unusually late (indicating a high affinity to long-running, legacy systems) and that they are contexts where the participants and messaging are governed by a central entity. That means that a directory exists, which means that the key distribution problem is vastly simplified. It also means that the software in use, both transfer agents (e.g. servers) and clients, are standardized and usually purchased from vendors that have specifically designed them to work well in an X.400 environment.

In other words, they are contexts that are very different from email, in which people exchange messages between a huge number of organizations and many users are simply on "mail providers" that are not an organization in a traditional sense. This means that there is no meaningful directory capability in the email system. Email is also used with a hugely diverse landscape of clients, imposing a steep penalty on the "complicated message encoding that requires interaction with a complicated directory system" part of the design.

Since X.400 proved a nonstarter in most of the email world, what about encryption for the rest of us? Well, this situation gets complicated and sad.

MIME

First, it's helpful to expand a bit on the involvement of MIME, the Multipurpose Internet Mail Extensions. MIME originated as a solution to the 8-bit problem with internet email, as a way to encode non-ASCII text and binary files in a consistent, reliable way. Along the way to achieving this aim, it introduces the concept of a "multipart" message which has multiple distinct objects as a body instead of just one string of text. We use this multipart feature heavily today, both for obvious applications like file attachments and for more subtle ones like HTML emails that provide a plaintext variant. I don't think it's totally wrong to summarize MIME as "email 2.0," because while it is limited in scope to the message format (not changing the transport protocols), MIME adds a lot of new functionality to email that we now consider basic requirements for a messaging system.

The history of MIME has a lot more twists and turns than you might expect, which reflects the difficulty of making major changes to a system as distributed and heterogeneous as internet email. Still, the MIME authors seem to have had a similar view that MIME was a significant evolution of email by standardizing message bodies and paving the way for future extensions to that standard, adding new functionality. This quote from RFC 1521 (1993), the first MIME standard, is informative:

STD 11, RFC 822 defines a message representation protocol which specifies considerable detail about message headers, but which leaves the message content, or message body, as flat ASCII text. This document redefines the format of message bodies to allow multi-part textual and non-textual message bodies to be represented and exchanged without loss of information. This is based on earlier work documented in RFC 934 and STD 11, RFC 1049, but extends and revises that work. Because RFC 822 said so little about message bodies, this document is largely orthogonal to (rather than a revision of) RFC 822.

In other words, existing email standards addressed how messages are conveyed between hosts, but did little to address what would actually go into those messages. MIME addresses that problem by creating a proper standard for message bodies, one that adds features while contending with the unfortunate limitations of the many pre-MIME email implementations. The many RFCs around the general topic of MIME (and other proposals for email body formats) spend a lot of time contemplating how these new message types would be handled by email clients that did not understand them. Email technology is generally easy, email standards are hard.

PGP

As these standards efforts marched on, the fast-growing field of computer cryptography had its own ideas. The first major event was cryptographer Phil Zimmerman's unceremonious 1991 release of a tool called PGP, or Pretty Good Privacy. The story of PGP is a long one, jumping from a casual release to an activist Usenet group to criminal charges to the foundation of PGP Inc to commercialize the system.

The funny thing is that PGP itself is not that important to the story of PGP. Despite the corporate ambitions of the company bearing that name, PGP started as a loosely academic or activist project, and it ended up that way in the long term as the open-source clone GnuPG (GPG) displaced PGP in the marketplace. PGP Inc became part of Symantec and faded into the obscurity that most Symantec products do. Along the way, PGP and GPG implementations rode the roller coaster that was 1990s cryptography, swerving from algorithm to algorithm and growing a complex cipher selection system along the way.

PGP was a general encryption tool and could be used for files or messages of any kind, but email was a key application. So how is a PGP-encrypted email actually conveyed? Well, it's not exactly pretty, but it aligns with the other messes we've already seen. PGP typically outputs ciphertext as either binary data or in a format called "ASCII-armored," which is base64 except with an optional CRC checksum trailer (which was deprecated decades ago). This is very similar to the PEM format widely used with modern cryptographic applications, and not by coincidence: PEM stands for Privacy-Enhanced Mail and is itself an artifact of a failed email encryption standard that produced quite a few RFCs. PGP's email application and PEM were under development just about simultaneously, and there was definitely cross-pollination of ideas between the two. PGP won, but not without taking parts of PEM along for the ride.

In practice, ASCII-armored PGP payloads are sent one of two ways: first is "inline" format in which each part of a MIME-encoded email is separately encrypted and have otherwise usual MIME types (e.g. application/octet-stream), so that non-PGP-capable email clients will treat them as opaque files that the user can save off and process with a separate tool. The second approach is the "MIME" format in which the entire MIME-encoded email is encrypted whole and dropped into a new MIME document with the type "application/pgp-encrypted." The latter has better security properties (less metadata leakage) and tends to be more user-friendly and elegant in clients that are designed with PGP in mind, but fails very badly in non-PGP-capable clients which are not even able to separate out attachments. You can still process the email with an external PGP tool by saving off the whole body, MIME-decoding, decrypting, and MIME-decoding again, but you get a little frustrated just reading that, don't you?

Today, this mess has become one of the two main criticisms of PGP. PGP is not the result of a huge, multi-party standards effort like X.400, but rather the product of a small group of people taking a pragmatic approach. Nonetheless, it grew up in the same climate and so survived with most of the same scars. From bottom to top, PGP is very complicated. Almost every part of PGP-encrypted email has multiple options, some of which are known to be insecure. Modern tools mostly follow sane defaults, but plenty of older tools remain in use. Key tools like gpg itself show their age with some of the least user-friendly interfaces ever designed. The immense difficulty faced by PGP users was the subject of seminal 1999 "usable security" paper "Why Johnny Can't Encrypt" (the paper that more or less invented the field of "usable security"), and most of the criticisms raised in that paper are still true of PGP implementations 27 years later.

And that brings us to the second aspect of PGP, key distribution. We've seen, at least in sketch form, how PGP email is conveyed, the first challenge of encrypted email. But what of the second challenge—is PGP for use with a directory system? Well, the earliest examples of PGP de facto were, because they did not address key distribution at all. Very early on, though, PGP gained a novel concept called a "web of trust." The web of trust, or WOT, is simultaneously one of computer security's most intriguing and appealing concepts and one of its most inane and stupid. I'll try to explain it succinctly.

You are Alice, and you want to send an email to Bob. The "why" is not important, these are timeless characters of cryptographic explanations and they have a timeless need to securely exchange messages. If you know Bob in person, say because you are colleagues at an institution, the key distribution problem is simple. You can perform calculations on a key, like a checksum, to produce a "fingerprint" which is unique to the key and short enough to not be completely impractical to read by eye. You, Alice, walk over to Bob's office and ask to see the fingerprint of his key. You write that down on a sticky note, and then in the future, you make sure to encrypt your emails to Bob using a public key that has that correct fingerprint. Easy as pie so far, but that doesn't scale well to situations where Bob is not within walking distance.

And so the "web" enters: Bob may not be conveniently accessible to Alice because, say, he works somewhere else, but Alice's coworker Charlie might have been at a conference with Bob a few months ago. If Charlie took down Bob's key fingerprint, he can give Alice a copy when he gets back. Since Alice knows Charlie and believes him to be trustworthy, this is a pretty good substitute for getting the fingerprint straight from the horse's mouth.

PGP's WOT model encodes this sneakernet approach to key distribution into a software implementation. A PGP user can "sign" another user's key, a way of cryptographically attesting that they believe that to be the correct key. These signatures are uploaded to public "keyservers" that allow anyone to query for a key and for the set of signatures it has received. If there are enough keys with enough signatures, you can treat the whole thing as a graph and try to find "trust paths" between Alice and Bob, even if doing so requires six degrees of separation¹. The tangled interconnection of keys and signatures verifying them forms a web: the web of trust.

In practice, the tide seems to have turned against the WOT model by the 2010s, and today it has been completely abandoned. It didn't scale as well as its proponents had hoped, usability problems with the tooling discouraged participation, and both the implementation and concept were complex enough that poor user understanding undermined the benefits. Today the WOT has fundamental problems (mostly related to the open keyservers) that have made it practically unusable, and I think it is fair to call it a failed experiment. Further extending that fairness, we should note that the WOT was surely an inspiration for the vastly simplified key verification schemes that appear in most modern end-to-end encrypted messengers.

From its messy academic origins to its transformation into an open project, PGP enjoyed a lot of prominence in academic and "cypherpunk" communities during the 1990s and 2000s, but was less common in corporate environments. The WOT model only made sense in the absence of a centrally maintained directory, and while PGP could be used with a directory it was not the path espoused by its advocates. PGP was mostly used among researchers and hobbyists, the cryptographic establishment. In the Enterprise, there was another player: S/MIME.

S/MIME

S/MIME originated with RSA Security, the company founded to commercialize the RSA encryption algorithm. RSA Security survives today as a subsidiary of teacher's pension funds, although its prominence in the industry has generally declined. Back in the 1990s, RSA was on the cutting edge, so 1996's introduction of an RSA solution for email encryption got quick attention from industry.

Over the following two years, RSA's product evolved into an IETF standard called S/MIME, for Secure MIME. S/MIME saw a series of new versions up to the present day, and is exceptionally rare in the academic and hobbyist world but fairly common in corporate and government environments, especially with Microsoft Exchange (having adopted S/MIME as its secure email option of choice after the failure of X.400 to take over inter-organization mail). Let's see how S/MIME handles the first challenge of encrypted email, encoding the message for transport.

Well, as the name suggests, MIME is involved. The simple sketch is like this: an email is prepared as a MIME document, which is then encrypted into the form of a PKCS#7 document. PKCS, the Public Key Cryptography Standards, refer to a series of open standards memos published by RSA Security. PKCS#7 describes "Cryptographic Message Syntax" or CMS, a standard container for an encrypted payload along with various metadata. In the terminology of the time, S/MIME is said to "enhance" messages with the "security services" of PKCS#7. We still use the term PKCS#7 today, mostly to refer to cryptographic certificates serialized in that format. For email, on the other hand, PKCS#7 has been superseded by an IETF standard called simply "CMS."

The CMS version of the email, which remember is an encrypted MIME document plus cryptosystem-related metadata, is then encoded as PEM (functionally Base64) and made the body of a MIME email with a type of "application/pkcs7-mime" (yes, I just said that PKCS#7 is gone from this application, but naturally the name has stayed). This is pretty much equivalent to the MIME version of PGP emails, and has basically the same upsides and downsides: it's relatively elegant and avoids a lot of weird edge cases in clients that support it, but it's quite hostile to clients without S/MIME support, which will be unable to make anything of the message. It also keeps many of the general downsides of MIME, and for example the RFC has to spend some paragraphs justifying why binary data is base64d and then converted to binary and then base64d again. Not having the extra layers of base64 involved would break interoperability scenarios with non-S/MIME tools!

Overall, S/MIME isn't that dissimilar from PGP in this regard, although I would say that it is overall simpler and better standardized. That is not to say that it is "simple" or "well-standardized," these are only relative observations. S/MIME implementations are still quite complex, and interoperability problems can appear even between organizations running the same software, due to the many configuration knobs involved. RFC 2311 includes this gem, to give some of the flavor of it:

S/MIME provides one format for enveloped-only data, several formats for signed-only data, and several formats for signed and enveloped data. Several formats are required to accommodate several environments, in particular for signed messages. The criteria for choosing among these formats are also described.

"Several formats are required to accommodate several environments" is the story of email in a nutshell... if not all of computing.

Let's consider the second aspect of the encrypted email problem, then: key distribution. Here, S/MIME takes a very X.400 style dodge. The original S/MIME standards documents, as far as I can tell, do not address key distribution in any way except for oblique and incidental references like "validating the certificate to a trust anchor." This seems odd except in the context of S/MIME's development by RSA Security and, as it became an IETF standard, Netscape—the birthplace of SSL. S/MIME was intended for use with the PKI, the Public Key Infrastructure, which has a nuanced and sometimes vague history but fell out of a combination of X.500 concepts and implementation work by companies including RSA Security and Netscape. In other words, S/MIME grew up alongside SSL/TLS, and uses the same key distribution architecture.

S/MIME emails are expected to provide the certificates that go along with the keys used (one of the reasons for the CMS container format), and those certificates are supposed to be signed by a Certificate Authority.

This is not exactly the same as saying that keys will be retrieved from a directory, but if you remember that the X.500 that gave us these concepts was itself supposed to be the directory, you can see that they are closely related ideas. In practice, they are often pretty much the same: a classic real-world application of S/MIME is in a Microsoft Active Directory environment, where the Domain Controller acts both as a directory (LDAP) and as a Certificate Authority, signing certificates for the users that populate that directory. This means that key retrieval from LDAP vs. certificate verification against the (AD) CA are pretty much the same thing, and you have the same assumptions around there being some sort of central authority to maintain the directory.

This means that S/MIME thrives in the same places that X.400 was more successful. The US military, birthplace of internet email, opts for S/MIME for encrypting it. They have a truly enormous Exchange and Active Directory environment to this end. Once those emails leave the military, the whole thing becomes a huge hassle, because properly using S/MIME requires establishing some form of federation with the military's Active Directory domain... or at least importing their CA as trusted, which is not all that easy with most mail clients.

Today

So here we have the classical email encryption landscape: there are two major options, PGP and S/MIME, each of which is unsuitable for most users for its own special reasons. Out of this frustrating stalemate, several other phenomenon have emerged.

First, we should briefly touch on "modern" email encryption, where "modern" can become a bit of a euphemism for "proprietary." There is a new set of email providers that offer some kind of built-in, managed E2E. Protonmail is the most prominent. Protonmail's dominant E2E encryption implementation is a proprietary, simplified system that relies on Protonmail as key directory and cryptographic implementation. This is not interoperable with other email providers, so I do not consider it a true form of "email encryption," but rather a proprietary encrypted messaging product offered alongside email. Protonmail does have two features for sending secure email to non-Protonmail recipients: the first is essentially a compatibility layer to PGP that allows Protonmail's encryption to get along with PGP tools. The second is a traditional compliance-driven "secure email" product where the recipient is actually sent a link to a website where they can enter a password to retrieve the message—once again, not really email but a separate product that loosely works with email.

Second, let's talk a bit about how email is really encrypted, in real reality, because there is this huge, grand-canyon-esque disconnect between the way that people discuss email encryption and what actually happens in the real world. Let's start taking that on by differentiating some terms: so far I have discussed end-to-end or E2E encryption, which is not exactly a well-defined technical term but generally means that a single encrypted payload travels unmodified and undecrypted from a human user to a human user. This can be distinguished from non-E2E encryption, where a message might be encrypted and decrypted several times, and using keys controlled by people other than the sender and recipient. We should also note the distinction between "encryption at rest," whether or not data is encrypted when stored (e.g. on disk), and "encryption in transit," or whether or not data is encrypted when sent over the network.

Email is exchanged between email servers using SMTP, and is exchanged between email servers and clients using IMAP ². Both of these protocols are now widely used over encryption, either using the STARTTLS method or "real" TLS, a distinction that I will not explain right now. While I lack solid data, it seems like a safe bet that the vast majority of real-world email is encrypted in transit, because at least the mail server logs that I have access to show the vast majority of mail servers using STARTTLS to secure their transfer connections. Client configuration is a more confusing topic but everything other than IMAP over TLS is now generally regarded as a big mistake, and modern clients should aggressively steer the user towards that route.

This is a confusing contradiction: we have had this big, long, boring discussion of how email encryption has failed to go anywhere. Then I tell you that almost all email is encrypted. What gives?

To start, remember our "in transit vs. at rest, and E2E vs. not-E2E, distinctions. The fact that the network connections between email participants are generally encrypted gives us a lot of security advantages, but it does not address the full scope of requirements that PGP and S/MIME do. Your email is not secure from your email provider, or the recipient's email provider for that matter, since email is usually not encrypted at rest and the provider has all the keys anyway. It also does not provide integrity protection, or non-repudiation, or any of a number of other useful security properties.

What's even worse is the technical reality. I pointed out that one of the big criticisms of PGP is that it is simply too complicated and provides too many options. This resulted in a litany of defects in PGP implementations over the years, and no small amount of insecure user behavior due to lack of usability. In-transit email encryption has very similar problems. STARTTLS, for example, is a fundamentally opportunistic mechanism. In its original form, an on-path attacker attempting to intercept an SMTP session can simply hijack it and suppress the STARTTLS offers from the two sides, preventing them ever negotiating a switch to encryption. This simple "downgrade attack" is a pervasive problem, and it's the main reason that I say that anything other than true TLS is a mistake. Unfortunately, STARTTLS is the most widespread option available for inter-provider email transfer.

That said, there are ways to address this problem. For example, DANE, the DNS-based scheme for TLS certificate distribution, has failed in the HTTP arena but has decent adoption (although very far from universal) in email transfer. It does a good job of providing mail servers with a way to discover that their peers are capable of encryption, without doing so over the SMTP channel itself. Even without DNSSEC to fully protect the DANE channel, this is an improvement that prevents many common attack scenarios. There is an even more popular approach that, while clumsy, avoids the dependence on DNSSEC to avoid a second on-path attack on the key channel: MTA-STS. MTA-STS is similar to DANE in that it provides an independent channel for a mail server to find if a peer supports TLS, but it uses HTTPS for that second channel, thus inheriting the protections of CA-rooted TLS.

Both DANE and MTA-STS are ways of "tipping" a mail server that a peer has STARTTLS support, at which point the mail server knows that it should only connect via STARTTLS, even if the mail server on the actual SMTP channel claims not to support it. While the extra moving parts are unfortunate, this does a lot to close the downgrade attack window, and email transferred between mail providers with DANE or MTA-STS support is really pretty well secured, at least on par with most instant messaging solutions.

This also implies that you can improve your email security a lot by just manually pinning major peers—e.g. configuring your mail server that, regardless of everything else, connections to major providers like gmail.com must be encrypted. This is actually a pretty common practice in some industries, including internal processes to maintain the list of "secure email destinations" so that it includes business partners and other frequent recipients.

In some industries, this is taken to an extreme: within the healthcare sector, for example, there are healthcare information exchanges that operate secure messaging platforms that are mostly just conventional email servers configured to refuse any non-TLS SMTP exchanges. This is sufficient to meet NIST 800-43 recommendations which is sufficient to meet the NIST security architecture for healthcare information. That might be surprising to you but it illustrates how this topic gets confusing: we all know that "email is not secure," but that's really a configuration problem with the "public" email system rather than a technical limitation. Few compliance standards actually require E2E, and securing email in transit and at rest, once you drop the E2E requirement, is so easy that it sometimes happens just by default when SMTP servers have STARTTLS support and disk encryption.

If you look at healthcare marketing materials, you'll sometimes see STARTTLS and S/MIME described as alternatives to each other. This doesn't make much technical sense, but you have to think about the context. If you have to communicate over mail servers whose configuration you do not control (or at least have insight into), you don't have a way of ensuring that your messages will be encrypted in transit unless you use E2E encryption such as S/MIME. If you are sending mail between servers that you do control, though, you can ensure encryption in transit by configuring the mail servers that way, and you do not need E2E encryption to meet that specific aim. Taken in this view, E2E encryption is almost a workaround for a defect of the design of email, and if you have taken anything from this longer than expected journey I hope it will be that: the history of encrypted email is so odd, and full of so many false starts, because it is basically the history of E2E encryption as an overlay "fix" for email's lack of the level of security provided by newer messaging solutions out of the box.

This also helps to explain the lack of adoption: it's not really that we can't figure out how to encrypt email, we do encrypt email. But as with most communications technology, the "security layer" has been pushed onto the service providers rather than the users. This has significant downsides, since it requires trust in those service providers, but one huge, conspicuous upside: if encryption is transparent to the user, people will actually use it.

In 1914 the Department of the Interior, through the Bureau of Reclamation, investigated the possibilities of developing the Columbia River. Thousands of arid but potentially fertile acres needed only water to become the Imperial Valley of the Northwest. Locked in the mountain ranges were valuable ores awaiting electricity to turn them into needed metals.

Two years later the State engineer of Oregon urged the development of the Bonneville site as a national-defense measure: he saw in the proposed power project a source of fertilizer in time of peace and nitrates in time of war. The dam also would completely drown out the Cascade Rapids and extend slack-water navigation some 40 miles eastward to The Dalles.

The Rivers and Harbors Act of 1925 directed the Secretary of War, through the Corps of Engineers, United States Army, to prepare and submit to the Congress an estimate of the cost of surveys, examinations, and investigations of all navigable streams and their tributaries where power development appeared feasible. (Q1)

It is difficult to succinctly explain why, exactly, the United States Army has spent much of its history involved in the construction of dams. It is partly an accident of history, partly the result of interagency federal politics, and entirely a product of American culture. In his book "Cadillac Desert," Marc Reisner examines the history of the American West's water control projects as a religious project, one animated less by practical needs than by a sense that domination of the West's rivers was destiny.

The Bureau of Reclamation, part of the Department of the Interior, was formed for that purpose. At the time, though, the Army had already been used to survey and improve rivers for nearly 100 years. They were not content to give it up. The result was a rivalry, one with several feints and blows before the two settled into their modern areas of control. For the Bureau of Reclamation, the Hoover Dam was their signature project. For the Corps of Engineers, the battle that would go down in history was the Columbia River Project.

The motivations for damming the Columbia were various. The Columbia was prone to flooding, which had caused damage and limited use of land along it. There was a great deal of land surrounding the Columbia that could be farmed, if the Columbia could be tapped for irrigation. Electricity, too, was a reason, although initially a somewhat secondary one. Perhaps the greatest reason, though, was simply economic: by the time that the major parts of the Columbia River Project were truly underway, the nation was in the throes of the Great Depression.

President Franklin D. Roosevelt was already a fan of hydroelectricity. As governor of New York, he was exposed to the pioneering Niagara Falls power plant and pushed for other similar projects in that state. As President, his "New Deal" naturally incorporated hydropower as well. By 1934, he had formed a Regional Planning Commission that sketched out a series of dams along the Columbia, two of which would become the Grand Coulee and the Bonneville. These dams would produce a tremendous amount of electricity, and unlike in other similar Corps of Engineers projects to date, that power would not all be consumed by irrigation pumping. There was power to spare. To distribute that power, the Regional Planning Commission suggested an independent government agency on the model of the Panama Canal or the recently chartered Tennessee Valley Authority.

As an interim measure, the loosely defined Bonneville Project coordinated the civilian side of the Corps of Engineers project until 1938, when the Bonneville Dam was complete and the Grand Coulee was much of the way there. The Bonneville Dam captures little water in its reservoir, so while it does have flood control value, electrical production is its primary purpose. The dam's two powerhouses produce up to 1.2 GW, an impressive number for the 1930s but one that pales in comparison to the Grand Coulee's eventual (1970s) full capacity of nearly 7 GW. The Columbia River dams increased the electrical capacity of the Pacific Northwest by orders of magnitude; the numbers were significant even at a nationwide scale.

The bumper crop of electricity triggered a predictable controversy: what to do with government power? One camp favored public control of the resource, with the government marketing the power on some sort of equitable basis. The other favored private control, arguing that the output of the dams should be contracted entirely to private utilities like Portland General Electric (itself the scion of an important early hydroelectric project at Willamette Falls). In the New Deal political climate, the first camp won: the Columbia did not quite get a TVA, but Congress did charter the Bonneville Power Administration (BPA), the first of what would come to be known as Power Marketing Agencies. Over the following decades, the BPA became part of the Department of Energy (DoE)—uncharacteristically, for the DoE, a part of it that actually generated and sold electricity. Well, technically, the Corps of Engineers generates it, and the BPA markets and distributes it.

In any case, starting in the late 1930s, the BPA was tasked with the construction of a network that could distribute power from Columbia River dams throughout the region—on an equitable, equal-rate basis often called the "postage stamp rate" that allowed rural coops to buy government-generated power at the same rate as the big city private utilities. The sudden bevy of power along the Columbia and the fair rates at which it could be obtained in great quantity led to an industrial revolution for the region, one that saw it as the seat of the American aluminum industry (with the Columbia Gorge producing something like 1/3rd of the nation's aluminum through to the 1970s) and that boosted the fate of hundreds of related industries (aerospace and, specifically, Boeing not least among them). BPA power has enduring influence today, with many towns on the Gorge (The Dalles, Boardman, Umatilla) disproportionately prominent on a map of the nation's data centers. AWS's us-west-2, for example, is a beneficiary of Columbia River dams and located near many of them—not just Bonneville, but the Dalles (1.8 GW), John Day (2.2 GW), McNary (1.1 GW), and more.

In marketing this power, the BPA faced a challenge: the dams are spread across a large area, as are the customers. Industrial customers, such as the Alcoa (Aluminum Company of America) smelter that opened in 1940 at Vancouver, Washington ¹ were opening in rural areas where land was readily available, and an explicit goal of the Columbia River Project had been the extension of electricity to farmers and other rural industries. The concept of long-distance power transmission had been pioneered by an 1889 transmission line, the nation's first, between Willamette Falls at Oregon City and downtown Portland. Beginning in 1938, the BPA was tasked with expanding that concept across a region that would eventually span eight states.

The Master Grid

BPA's first administrator, J. D. Ross, presented a plan he called the BPA Master Grid. This ring-shaped network, made up of 230 kV long-distance transmission lines, would connect the dams not only to Portland and Seattle but to Pasco, Yakima, Spokane, Ellensburg, the Willamette Valley through to California, and the Oregon Coast. By 1945, the Master Grid covered three thousand "Circuit Miles" of transmission lines. It was the first integrated regional power grid in the United States, and would come to pioneer the market-based electricity pricing and distribution, independent system operators (ISOs), and pooling and wheeling agreements that form the modern US electrical infrastructure. The entire Western Interconnection, the unified power system that serves the US and Canada from the Rocky Mountains west, can be said to have crystallized outwards from the seed of the Bonneville Dam's switch yard.

Getting there required that the BPA solve formidable technical problems and develop many new technologies in power distribution. BPA transmission lines operated at higher voltages than any before them and, in the 1960s, introduced high voltage DC transmission to the Americas, connecting the Columbia system to the major demand centers of Southern California at 800 kV DC. BPA was only slightly behind the TVA on the installation of a remarkable analog computer called a Network Analyzer, in 1939, which simulated the behavior of the transmission network like a scale model. The rural nature of the BPA network put substations in remote areas, where they were minimally staffed, and the long stretches of high-voltage transmission line meant there was ample potential for damage by wind, weather, and trees, phenomena that the BPA came to better understand through research laboratories and experimental field sites.

This is not an article about the history of electrical distribution, or at least it wasn't supposed to be, so here we must exercise some discipline and narrow in on a topic. Telecommunications ought to do.

By 1940, as the Master Grid entered operation, its numerous substations already caused administrators a headache. Each had a small staff of technicians, but communicating with them was difficult. Coordinating changes across large areas, or quickly responding to faults, involved a flurry of telephone and radio calls. When Portland General Electric built the transmission line from Willamette Falls to Portland, they encountered the same problem, and by the 1910s had implemented a very early form of its solution: telemetry and teleoperation. Through a set of control wires strung along the transmission line, operators in Portland could see certain measurements from the Oregon City powerhouse and remotely throw switches to bring turbines on and offline in response to load. As the BPA built the Master Grid, they invested in the same technology.

Around 1939, the BPA commissioned a study of communications technology that could be used along the Master Grid. There were three main contenders: commercial telephone networks (which BPA called "land telephone" to differentiate it from the other two), "carrier current telephone" technology that superimposed telephone signals onto the electrical conductors of the transmission lines themselves, and radiotelephone equipment. A working agreement was reached with Pacific Telephone & Telegraph, the Bell System company that would later become US West, to share network information and analyze the cost tradeoffs between purchasing carrier current and radio equipment and leasing telephone lines. Ultimately, the diversity of the BPA network required some of all three.

Each of the BPA's substations had a building, called the control house, that contained control and monitoring equipment along with office facilities for the substation's operators. A room of each control house was dedicated to carrier equipment, devices that modulated multiple telephone circuits using frequency division multiplexing, and to a set of carrier frequencies that could be coupled onto the transmission lines to be received at the next substation. This equipment is similar to carrier equipment used in the telephone network, although specialized to power distribution applications by the choices of carrier frequency. I cannot say for certain, but it is very likely that BPA purchased their system from Lenkurt, a San Francisco-based communications equipment manufacturer that specialized in carrier current systems at the time ².

The BPA's carrier system incorporated selective calling, meaning that users interacted with telephones that looked and felt much like conventional telephones, including a dial. An operator at one substation could dial the number for another and that phone would ring. The main difference from the telephones we use today is that these carrier current systems were interphones, more similar to intercoms or party lines than single-user telephone service. If you picked up a phone on a circuit, you would hear any conversations already underway. Of course, in industrial control applications, this built-in conferencing capability was generally considered a feature, and telephone circuits were assigned to shared use by departments or operating regions.

Radio was installed as well, primarily so that construction and maintenance crews in the field could get messages back to the administrative offices. Some substations, in strategic locations for coverage, had a radio site about a half mile from the substation for isolation from the powerful electromagnetic interference created by the high-voltage transformers. These radio sites were wired to remote heads located in the substation control house office, where substation operators relayed messages between mobile radios in the field and the carrier current telephone system. Bear in mind that these were still early days for mobile radios, and the HF units used by the BPA were proudly described as using only 1.5 cubic feet of space in the trunk of the vehicle, plus the microphone, speaker, and control head in the cab. Finally, while not the main purpose, it was already noted in the 1940 Annual Report that the substations could use the radio stations to substitute for the carrier current system in an emergency.

Complementing all of the above, the BPA leased telephone lines between major substations, agency headquarters in Portland, and the switch yard in Vancouver that was becoming the closest thing to a "main substation" in the network's distributed, ring-shaped design.

Immediately after the BPA's first Annual Report discusses the selection of communications equipment, it moves on to Protection. Here I must introduce a topic in electrical engineering that I have only a loose understanding of, even having spent the last few weeks in part on YouTube engineering tutorials. It's important that we get comfortable with the field of "protective relaying" because, as we will see, it became the most widespread application of private telecommunications networks after the railroads.

Protective Relaying

In your home, you are protected against certain dangerous scenarios by the over-current protection device that we Americans call a circuit breaker. The electromechanical contraptions in your service panel use a combination of methods to monitor the current that passes through them, and if they detect excessive current they open the circuit. The electrical transmission system has similar protections on a larger scale: on the distribution wires strung on poles outside of your house, for example, there are various fuses and circuit-breaker-like devices known as reclosers. High-tension transmission lines ³, like the 230 kV system built by the BPA, need similar protection for similar reasons—except that it is much more complex.

Electrical terminology can be complicated on a good day, and this situation is even more complex because of the historic bifurcation between terms and practices in building electrical wiring versus electrical distribution (which are governed, for example, by two separate electrical codes) and the fact that we are talking about a system that is nearly 100 years old. Relays were a newer technology in the 1930s, as was large-scale over-current protection, so transmission engineers viewed circuit breakers as just an application of the relay and over-current protection on power distribution is still referred to as "relaying" today. Since the whole broader field of supervising transmission lines for safety and reliability is called "protection," relays that open to protect generators, lines, or loads from dangerous conditions are called "protective relays."

Some of the protective relays used in transmission are very much like the circuit breakers in your home. Directional over-current relays, for example, monitor the current passing through them in one direction ("towards" the load) and open if it is excessive. Ground fault relays open when they detect, via various current transformers, that an excessive amount of power is leaking to the ground—just like the GFCI outlets or breakers installed in wet areas of homes.

Some of them, though, are much trickier. The first major problem is directionality. In your home electrical wiring, there is a clear sense of where power flows "from" (the service panel) and "to" (an outlet or fixture). Wiring thus only needs over-current protection in one direction. In a wide-area transmission network, this isn't true. The Master Grid was designed to incorporate a ring for much the same reason that SONET and other communications technologies favored rings: with a ring topology, you can lose the connection between two points and still be able to serve all points by sending power the "other direction." In general, it is common that electrical transmission lines can be "fed" from both ends, and have "load" on both ends. This flexibility to reach the same places by different routes makes the grid more reliable and responsive to changes. It also makes over-current protection more challenging.

Say that you have a span of transmission line, and that somewhere along it a tree falls and pushes one conductor against another. You now have a short-circuit fault at 230 kV (or more in later lines), a dramatic and dangerous condition. You also have thousands, if not hundreds of thousands, of customers that are depending on the power provided by that line. Current transformers can measure the enormous fault current, and indeed it will be detected at many points along the line. But what do you do?

Ideally, protective relays should open on both sides of the fault, and as close to the fault as possible. This cuts power to the dangerous situation while minimizing the number of customers who experience an outage. It's also difficult to achieve in practice: you cannot simply open a protective relay on over-current, or every relay on the line will open at the same time. Instead, analog circuits were used to measure the impedance to the fault as a proxy for its distance. This way protective relays could be carefully tuned to open only when a fault was near them.

Now, consider that a transmission line may be "tapped" and connected to load centers or generation at multiple points along its length. There may also be multiple parallel routes that current can take, with varying capacities. Both of these situations mean that it is often necessary to measure the current (to detect faults) at locations remote from the actual protective relays, which are large devices that needed to be located at substations. Further, the complications of parallel routes and possible ground faults on long lines required the use of a "differential protective relay" or "balanced current relay" that took a much higher level approach to the problem. In a differential system, you measure the current at every connection point to a given protection zone and sum them. The sum should be zero: the same amount of current goes into the line as comes out. If it's not, something has gone wrong somewhere, likely current that is escaping to ground or traveling on a parallel route not engineered for such abuse. But the points where you measure current may be many miles apart, and you still need to make real-time comparisons between them.

In 1940, the BPA was already planning the installation of "pilot relaying" over their carrier current telephone system. In an engineering context, a "pilot" is usually something small that controls something big. A pilot operated relief valve (PORV), for example, is an arrangement where dangerous pressure levels (of a liquid or gas) cause a small pilot valve to open which then triggers a pressure differential in a much bigger valve that causes it to open. There is a similar idea in protective relaying: a pilot-operated relay is a relay that disconnects a very big wire under the control of a smaller wire carrying a pilot signal. The simplest scheme works like this: a device, like a current transformer, monitors a safety parameter and produces a tone whenever it is acceptable. Elsewhere, a protective relay monitors that tone. If the tone ever goes away, the relay opens. The tone might go away because the pilot device detected an unsafe condition, but it might also go away because the line carrying the pilot signal was damaged, making it a fail-safe design. This is good for safety, but bad for reliability, and requires that the communications infrastructure used for protective relaying be very reliable.

Unfortunately, it was not: the carrier current systems installed by BPA through the 1940s were typical of the technology used in the industry at the time, but it was ill-equipped for the scale of the Master Grid. In part to alleviate concern of federal competition wiping out private utilities, and to improve general efficiency, the BPA formed the Northwest Power Pool in 1941. Initially made of about a dozen electrical utilities in the Pacific Northwest, the Power Pool formalized a set of arrangements by which utilities would buy and sell power among themselves, carried over the BPA's transmission network for a small "wheeling" fee. The Northwestern Power Pool would eventually become the Western Power Pool and a template for much of the nation's electrical industry. It significantly increased reliability and efficiency in the region by allowing utilities to sell their overproduction to utilities with high demand, and vice versa. It also brought the carrier communications system to its knees: a practical requirement of the power pool arrangement with wheeling over the BPA's transmission lines was that those transmission lines had to carry protective relaying pilot signals for all of the utilities involved.

A severe fault in a utility taking power off of a BPA line, for example, might require opening protective relays at a power plant operated by a different utility somewhere else on that line. New techniques for communications-aided protective relaying were under development, things like "permissive underreaching transfer trip" and "permissive overreaching transfer trip" that are difficult to explain briefly. These required that protective equipment at each substation have information about the state of protective equipment at all of the other substations, in order to make decisions that are not just based on local conditions but are globally optimal for the health of the whole line.

For example, avoiding a dangerous "islanding" condition on a transmission line might require opening relays at three or more locations along the line, but opening any relays beyond those required would simply cause unnecessary outages. Further, reliability is key and some types of faults on high-voltage lines are self-clearing (this is sort of a euphemism for the fact that a tree branch bridging sufficiently high-tension conductors will often be knocked off of them, if not vaporized entirely, by the resulting arc). Some protective relays are "reclosing" (and are often referred to in brief as "reclosers"), meaning that they will automatically reset (close) after a brief wait period. Some of the time, the fault will be gone and service is restored. Reclosing is potentially dangerous, though, and especially in transmission networks reclosing is only desirable in certain circumstances. Further pilot signals can be used to enable and disable reclosers based on the nature of the fault or the other locations at which it was detected, so that for example a fault that has taken a power plant offline does not lead to a recloser elsewhere "flapping" and stressing the remaining production capacity.

By the end of the 1940s, the BPA's carrier telephone system was overstressed to the point of failure. Power Pool utilities had connected their own carrier equipment, butting frequency bands so closely against each other that they began to interfere and degrade the quality of connections that could already be difficult to make out over hundreds of miles of high voltage infrastructure. The fact that a fault on a transmission line would generally prevent carrier current communications over that line working was a feature for simple fail-safe pilot relaying systems, but as protective relaying technology advanced it became the source of cascading failures. For example, by the early 1950s the BPA had found that lightning strikes on major transmission lines would cause enough interference to the carrier current system that protective relays all along the line would lose their pilot signals, open as fail-safe, and escalate what should have been a localized problem into a systemic one. Besides, many sections of the network had by that time become so congested by various carrier current circuits that there was simply no room in the spectrum for more—and so no capability to install protective equipment for new service lines.

Microwave

Meanwhile, the Second World War had had a profound impact in two ways: first, wartime demand for weapons, aircraft, and aluminum had driven Pacific Northwest industry to new heights. In 1949, power consumption in the Pacific Northwest had more than tripled, and population increased by 45%, compared to 1939. The Corps of Engineers built more dams, so the BPA's network carried more power. Engineers at the Administration's laboratory developed the techniques to operate transmission lines at record-setting voltages, 300 kV and beyond—making protection systems all the more critical. Second, wartime radar research had produced technological byproducts including high-bandwidth microwave transmitters and receivers.

So, the electrical industry adopted microwave communications technology alongside the telephone industry, and for much the same reasons. The high bandwidth of microwave links allowed for multiplexing huge numbers of channels, and the lack of wireline infrastructure (especially shared with the actual transmission lines) promised increased reliability. In my previous article on passive repeaters, I noted that they were especially popular with electrical utilities. Now, we learn why: utilities throughout the country built microwave communication systems that carried a fraction of the traffic of the Bell System, but often carried it to more remote locations and with higher reliability requirements.

The BPA's situation was different from that of more traditional utilities. Most private electrical utilities had started in a city and grown outwards, with a denser service network and fewer long-distance lines. They had mostly built private networks for communications, stringing their own open-wire telephone lines between stations. The BPA, with its far-flung network and huge distances, couldn't afford that kind of investment in stringing wires. In some ways, this proved an advantage: they could start from a clean slate. This network would be architected from the beginning for efficiency and performance, rather than to accommodate existing infrastructure.

While the BPA had initially built substations for a larger staff, the control technology was rapidly evolving and some of the more intensive activity BPA had expected at substations proved unnecessary ⁴. Despite the generously sized control houses, by 1950 most substations had only a single operator on staff, who would often be "out in the field" tending to the transmission lines. This created a problem if some sort of sudden problem required reconfiguring the network to restore service—a system operator might be left ringing a substation phone with no one there to answer.

That year, three substations had been equipped with "supervisory control" technology. This new system combined telemetering and teleoperation, so that a system operator in Portland could monitor voltage and current measurements at the substation and remotely operate the switchgear. The benefits of supervisory control were obvious, but the limitations of the carrier telephone system meant that those three substations were as far as the system could reach. Based on its promising experience with supervisory control of those three stations, and the clear need to continue expanding an already-stressed communications system, the BPA in 1949 committed to the construction of a completely new, completely modern control system for the Northwest Power Pool.

The BPA's substation in Vancouver, already one of the largest, had a combination of ample space and proximity to Portland that made it a convenient location for support facilities. Construction yards, maintenance shops, and research laboratories had all been added onto the facility. The 1949 project launched with a symbolic gesture: the site was renamed, from simply the North Vancouver Substation to the J. D. Ross Complex in honor of the BPA's first administrator. A new building at the Ross Complex, the Control Center, became the nerve center of the system and the Rome to which all microwave routes led.

Keeping with our communications theme, one of the main features of what was then called the Ross Control Center was a "three position turret" (yes, a turret!) by which operators could call any line on the microwave, carrier current, or leased line telephone networks. The plan was to obsolete the turret, though. In 1950, the BPA awarded a half-million-dollar contract to the Philco corporation (originally Philadelphia Battery and later part of Ford, then GTE, then Philips, amusingly providing another expansion of the name) for its new microwave network. Philco had done extensive military work on microwave radar during the war, and was at the time one of the leaders in microwave technology. Philco's expertise would be needed, because the microwave network ordered by the BPA would be the largest electrical utility communications network in the world. While more difficult to conclusively state, I think it is likely that it was either the second largest microwave network of any kind after AT&T's, or the third largest after those of AT&T and the Santa Fe Railroad (both of which were also Philco customers).

At this point, the BPA's transmission network had extended to the Hungry Horse Dam in northwestern Montana, adding customers along the way. There were some 14,000 circuit miles of transmission lines and 400 substations in the system, and Philco's contract specified that they would build out the initial operating network in just 360 days. Work got underway: BPA signed the contract in February of 1950, in September testing and demonstrations were conducted, and in October the BPA officially activated the first leg: a 200 mile route from the Ross Complex to Snohomish, Washington.

Snohomish

This first route, built at a cost just under a million dollars, used repeaters at Mt. Rainier, Chehalis, Olympia, Squak Mountain. The longest single jump was Olympia to Squak Mountain, about 55 miles, an unusually long distance for microwave facilitated by Squak Mountain's prominence and a 150' tower at Olympia. With multiplexing equipment, this route carried 23 channels from the control center to the Puget Sound area.

This first microwave link was quickly put to work for one of the most interesting new applications of utility telecommunications: fault locating. When a fault occurred on a long transmission line, the BPA's first step was to search the whole line for the problem. Many lines were in difficult terrain, so a helicopter or airplane was used to speed up the process. The faults were sometimes minor and not easy to see from the air (say, a broken insulator), so the survey aircraft would take photos for processing and analysis on the ground. This process was expensive and, moreover, it was time consuming. BPA engineers realized that sudden open circuits or shorts in transmission lines created electrical signals that propagated back through the line and could be observed on test equipment—so you could presumably locate a fault by calculating its time of flight to the substations at each end. The problem was obtaining a measurement of when the fault signal arrived at two different locations, in precise synchronization.

Requiring high speed and, more importantly, consistent latency, this was exactly the kind of problem that microwave lines were well suited for. ITT ⁵ designed the system to BPA specifications, including devices that detected fault waves and reported them over a microwave channel, and a machine that compared the timing of the received reports and calculated a likely fault position (in miles) relative to each substation. FTL seems to have estimated that the system was accurate to 600', BPA to 1,000'.

The Ross-Snohomish route carried other important traffic as well: as part of its inaugural celebration, Washington State Representative Henry M. Jackson used the new internal telephone at Snohomish to call Ross over the microwave link, congratulating BPA's administrator and chief engineer on the accomplishment. Although I am unclear on the exact criteria being used, newspaper reports consistently identify the link as the "first of its kind in the world" ⁶.

1950 was still early for microwave technology; AT&T's first commercial microwave link had only gone into service in 1948, and that was experimental. The transcontinental telephone "skyway" wouldn't be completed until a year later. As a result, microwave technology was unfamiliar to the public, and the appearance of parabolic antennas on BPA facilities—and at repeater stations on mountaintops and out in the woods—was conspicuous. One reporter called them the BPA's flying saucers, another explained the repeaters in terms of "pitcher" and "catcher." Every paper ran photos.

Over the next two years, Philco completed a second microwave route up the Columbia Gorge connecting each of the dams through to Spokane, and a third that linked Beverly, on the route to Spokane, to Snohomish—forming a ring like the original Master Grid that provided redundancy and direct protection channels for transmission lines on that route. The 1952 microwave network included fourteen primary terminals and 21 repeaters; it connected the dispatch telephone system at Ross with dams and substations along the microwave routes as well as seven mountaintop HF stations to reach field crews.

As was typical at the time, the microwave equipment at each repeater and terminal ran directly from battery power. The batteries were charged (normally floated) from two different power supplies, one from the utility and the other from an on-site propane generator with a two week fuel supply. BPA initially used prefabricated aluminum shelters for the equipment at repeaters, although many were originally built or later rebuilt as cinderblock. This was, in part, due to the weather: mountaintop repeater stations in Washington coped with severe winters, and BPA went through several rounds of modifications to their building and tower designs. Towers were reinforced against ice accumulation, and repeater stations in particularly snow-prone parts of the Gorge and the Snoqualmie Pass were made two story. These buildings had a "balcony" entrance on the second floor with a ladder to reach it, allowing access even when the first floor was completely buried in snow. Towers were rated for 100 MPH winds, and some for thousands of pounds of ice. BPA designed ice shields for the antennas, and a heated cover to keep ice from covering the reflector surface.

BPA also took advantage of passive repeaters to relocate microwave sites to more accessible locations. At Rockdale, on the edge of the Cascade Mountains, a repeater was installed just next to the highway. Its antennas aimed more upwards than sideways, at reflectors at the top of the mountain ridge. Microflect passive repeaters were manufactured in Oregon, and the BPA was one of Microflect's first large customers, contributing design changes for mountainous service.

By 1955, the BPA microwave network had reached Hungry Horse Dam in Montana and connected all of the major substations of the system. When a decision was made, in 1955, to relocate the power dispatch office from the Ross Complex to the new Portland headquarters, the capacity and expandability of the microwave system made the process much easier. The BPA's new HQ building looked more like a telephone exchange than a federal building: one of its most prominent features was a rooftop microwave tower. The 1957 annual report inventoried 61 microwave radio sites covering 1,300 miles of route, by then using a mix of Philco, ITT, and Motorola equipment.

BPA's network had, by this time, achieved many feats of rural service. One of the most impressive was service to the southern Oregon Coast: this area was very remote and faced terrible weather throughout the winter. BPA's 115 kV South Coast line was the only electrical service into the region, and it was regularly disabled by ice storms and flooding. Because of the area's mountainous terrain, line crews working in the area were only infrequently able to reach a substation near Eugene on their mobile radios, and that was their only way of talking to dispatchers to coordinate repairs. In an effort to improve service reliability, the South Coast became the pilot for a new model of field communications. Relatively closed spaced microwave repeaters, each with a VHF radio, would bridge radio channels onto the microwave telephone system. To stand up to the Oregon Coast, each of these stations used an aluminum enclosure with heating, fire suppression, and a generator. Some of the enclosures were cabled to the ground, to better hold them down against the wind.

During the 1960s, the value of the microwave system had been proven but it was once again facing the limits of its capacity. Microwave technology had improved tremendously in the post-war decade and BPA's 23 and 24-channel multiplexers were obsolete. A $1.6 million contract was let to Lenkurt to upgrade much of the microwave network to the modern Lenkurt 76C microwave radio and 46A or 34A multiplexers, capable of up to 600 channels at around 8 GHz (the previous system varied from site to site but operated at around 2 GHz).

This was equipment designed and built under GTE ownership, and was also typical of the long-distance microwave links in the GTE telephone network. As part of the project, Lenkurt also installed VHF radio relays throughout the system. At some sites, Lenkurt installed dual-polarized antennas to allow simultaneous operation of the old and new multiplex systems and, later, increased capacity. Further contracts expanded the microwave network to Bellingham, Washington and to Corps of Engineers projects on the Snake River. In 1966, BPA dispatchers in Portland could monitor production at 21 dams, receive alarms from 250 substations, and completely remote control fifteen of the network's key switchyards.

When the BPA built the Pacific Intertie, a combination of two 500 kv AC circuits and one 800 kV DC circuit stretching 900 miles from the Columbia River to near Los Angeles, it was the largest transmission line project in US history. Under construction from 1965 to 1970, the line's standard bearer came in microwave form. Collins Radio built microwave routes parallel to the Intertie transmission lines, a $2 million project with 22 new radio stations on the 600-channel Lenkurt system. Most of the work was finished by 1967, a prerequisite for some construction the transmission line itself, since the microwave channels were used for testing and commissioning.

The Computer Age

Microwave was not the only arena in which the mid-century had brought new technology. The BPA was not new to computers; various forms of electromechanical computation had been part of their engineering works since the 1930s. By the 1960s, though, projects like SAGE (a military air defense system) and SABRE (a commercial airline reservation system) demonstrated the potential of combining computers with telecommunications for real-time control. The BPA had a telecommunications network, and it had a real-time control system... and they decided to add a computer.

In 1966, the BPA announced that its control center would once again move, from the Portland headquarters building back to the Ross Complex, where a new building would be designed from the ground up for centralized, computerized control of the power system. Named the Dittmer Control Center after a previous BPA power manager, the low-slung building had a prominent concrete microwave tower and was partially sunk below ground level for hardening against attack (it was, after all, the Cold War).

Much of the Dittmer center's lower-level floorspace was devoted to equipment rooms, which would soon be the home of the computer system that BPA contracted to Rockwell. Among other equipment, Rockwell installed a PDP-10 computer that received, analyzed, and logged telemetering data from throughout the system for display to operators. From its commissioning in the early 1970s, BPA continued to enhance the computer center into an integrated power dispatching system that supported operators in monitoring the network, predicting future demand, switching transmission lines, and ordering changes in production at power plants throughout the Pacific Northwest.

The ultimate manifestation of the PDP-10's software was called RODS, the Real-Time Operations, Dispatch, and Scheduling System. RODS was one of the first systems of its kind, initially contracted to Rockwell in part due to their experience with control computers for the Apollo program. Features of RODS included a time-synchronized data acquisition system to support differential current monitoring (ensuring that the computer compared current measurements taken at precisely the same time), which used an atomic time standard at the Dittmer Control Center to distribute high-precision timecode through the microwave network. As RODS matured, it established the 15-minute scheduling loop used by dispatchers to configure power plants and transmission lines for a constantly changing electrical load. DEC themselves, in an internal sales meeting whose minutes fortunately made it into the historic record, noted the Dittmer PDP-10 as a critical early sale in their efforts to break into the utility industry.

Despite the technical firsts of the Dittmer Control Center and RODS, this chapter of BPA history is best known for its incidental brush with the Pacific Northwest's most famous chapter of computer history: as temporary employees of TRW, the aerospace contractor brought on for RODS software development, Bill Gates and Paul Allen both spent time at Dittmer. They were still in high school, it would be years before they moved to Albuquerque to found Microsoft.

Many parts of the BPA's microwave network are still in use, although improving radio technology and the adoption of fiber optics (including fiber embedded into the neutral conductors of transmission lines) have allowed for elimination of some repeaters. As microwave technology continues to fall obsolete (in comparison to fiber, commercial terrestrial radio networks, and satellite), many of the remaining sites are likely to be demolished. Some of them—sites like Chehalis, Squak Mountain, and Rainier—have been in service for 76 years.

The BPA microwave network is not unusual. It was the first of its type, but the transmission and power marketing concepts pioneered by the BPA are now used nationwide. Wherever power goes, protective relaying follows, and the telecommunications networks that shadow the electrical grid from coast to coast. Few enterprises outside of the communications industry itself have ever operated communications networks on the scale of electrical utilities. They find themselves in the company of railroads and oil pipelines, ventures that must span deserts, climb mountains, and cross rivers.

Unlike many of those operations, though, the BPA is a federal agency, subject to the National Environmental Policy Act and National Historic Preservation Act. BPA's extensive Section 106 NRHP compliance program has produced extensive historic documentation of its transmission lines, communications infrastructure, and research facilities. While sometimes onerous, these federal policies have created the best-documented historic electrical utility communications system in the nation.

An inventory of historic microwave stations still under BPA ownership turned up 28 in Washington, 22 in Oregon, one in Idaho, and two in Montana, and many are eligible for nomination to the Historic Register. From an aluminum shed at the Ross Complex to a squat concrete building and utility pole at Mary's Peak, they are monuments to American history—a singular, almost megalomaniacal vision of the Columbia River tamed; the rise of Pacific Northwest industry under wartime demands; a technical approach to environmental preservation and economic equality. Infrastructure for the public benefit: a great American achievement. We might, one day, find the will to do it again.

Through decades of consolidation, reorganization, and divestiture, AT&T left a famously complicated corporate history. One of the greatest enterprises in American history, arguably the greatest enterprise, AT&T has often rivaled the federal government in the size of its budget and workforce. One of the reasons, as we well know today, was monopolization and its close relative vertical integration. AT&T was the telephone system, or at least aspired to be, and for decades the meaning of "Universal Service" was that the service was designed, built, and operated by AT&T—universally.

While AT&T's tangled origins are fertile ground for the historian, they also obscure many of the early stories of telephone history. Much of the work of the early independent telephone industry has been lost in the voluminous achievements of AT&T. Even very basic facts become obscure. For example, who invented the telephone? Well, we all know the answer: Alexander Graham Bell. We have mostly forgotten that, at the time, this was a hotly contested question. One of the most prominent alternate claimants to the title was a man named Elisha Gray, today immortalized as the "Gray" in electrical distributor "Graybar," but better known in his time as an inventor of telegraph and telephone equipment. Gray contracted prototyping of some of his inventions to an upstart manufacturer and de facto Western Union spinoff, founded by Enos M. Barton (the "bar" in Graybar) and George Shawk. Impressed by Barton's operation, and at odds with Shawk on its future direction, Gray put together the money to buy out Shawk and became half-owner of the company that would reincorporate, in 1872, as Western Electric (WE).

It is ironic, of course, that a man who might fairly be called one of the top enemies of Bell helped to found the company that would become one of the most important parts of the Bell System. It's not a coincidence: Gray's involvement in WE included plans to manufacture his own telephone design, for which he had filed a provisional patent. Like many of the late 20th century's telephone inventors, Gray's greatest challenge in commercializing his invention was not technical but legal. His provisional patent on a telephone transmitter, substantially similar to the one invented by Bell and possibly older, led Western Union to take take part ownership in WE to advance their own plan to compete with AT&T as a telephone company. That set off a protracted legal battle, whose end result included the termination of Gray's patent claim and Western Union's abandonment of telephony.

AT&T was not the kind of company to leave things to chance, though, and least of all when it came to competition. In 1881, AT&T acquired WE. From that point on, WE was no longer a competitor, it was a core part of the Bell System: the manufacturing and supply arm of AT&T. A few decades later, WE had become the primary maker, and often sole supplier, of every piece of equipment used in the Bell telephone network. Everything from telephones to cables to central office switches were made at WE's various works. What few components WE didn't make, it sourced, through an expansive purchasing arm that negotiated orders on the behalf of the entire AT&T family. In 1925, WE had become so dedicated to the Bell System that its remaining non-telephone business, mostly local distributorships, was spun out into a separate company (Graybar). From that point forward, the Bell System was not only WE's sole shareholder but its sole customer as well.

As part of the 1925 reorganization, WE's research and development arm became a new organization, jointly owned by WE and its patron AT&T: Bell Laboratories. This new organization consolidated AT&T's expanding basic science efforts with WE's manufacturing expertise, setting the stage for decades of equipment that was conceived, designed, manufactured, and used within the AT&T empire. Bell operating companies got everything they needed, from tools to the telephones themselves, via requisition to their local WE supply warehouse. Such were the needs of the growing telephone system that WE started manufacturing telephone cable in 1925, quickly became the world's largest manufacturer of wire and cable, and likely held that title continuously until the turn-down of much of its manufacturing capacity in the late 1970s. AT&T was the nation's largest private employer for much of this period, and WE accounted for about 1/6th of that workforce.

Until the Carterfone decision and, for the most part, until the divestiture of AT&T in 1984, telephones were born at Western Electric. All of the phones leased by Bell Operating Companies, ranging from the classic WE 500 to explosion-proof phones for coal mine applications, were made at WE facilities like the Indianapolis Works. There were nearly 10,000 employees there, making 35,000 phones a day—and Indianapolis was not remarkable. It was just one plant of many.

AT&T built its empire through innovation, but also through domination. The acquisition of WE was one of its biggest steps towards complete integration, a goal that WE would pursue through the middle of the 20th century. For example, when the Morton Salt empire indirectly led to the Teletype Corporation and the development of commercial teletypewriter networks, they bought it. Teletype was a WE company from 1930 to its end.

Telephones were not only born at WE; they went there to die. During the 1920s, telephones were expensive instruments that required regular maintenance. Besides the commercial advantage, which would become more significant in later years, this aspect of telephones encouraged a full service lease model. Customers leased their phones from their telephone company in part because (prior to Carterfone) they had to, in part because the arrangement made the telephone company responsible for the phone's care. At the same time, Bell Operating Companies carefully controlled their expenses by reusing equipment as much as possible.

So, when a customer signed up for telephone service, they were issued a phone. When they canceled service, or had trouble with the phone, or quite simply wanted a phone that was a different color (or an upgrade to a Trimline or a Princess), the telephone company took the phone back. It would join hundreds of other phones on a trip to the nearest WE Service Center. The same truck would likely make the return journey loaded with phones ready for customers: the service center refurbished them.

Millions of telephones come back to Bell System service centers each year, many with their housings, handsets, and other molded plastic components bruised and battered. Some can be put back in shape by buffing, solvent polishing, or painting. Others wind up in piles. (Q1)

The scale of WE's phone refurbishing program was remarkable. Huge workshops of WE employees inspected, cleaned, repaired, and tested each phone. Refurbished units visited a test desk for a thorough electrical checkout before they received the service center's stamp or label that they had been remanufactured for use. In the Mountain States and west, WE service centers were found in Denver, Phoenix, Los Angeles, San Francisco, and Seattle. By the 1960s, Portland and Salt Lake City had joined.

Of course, despite the best efforts of all of WE's horses and all of WE's men, not all telephones can be put back together again. Much of the equipment returned to WE could not be satisfactorily refurbished. Besides, it wasn't just phones that telephone companies returned to WE, it was everything. Upgrading a crossbar exchange to an ESS? The ESS came from Western Electric, and the crossbar exchange went back to them. WE supplied telephone poles to the operating companies, and at the end of their life it took them back.

Western Electric has manufactured millions of telephones, millions of miles of wire and cable, tens of thousands of manual and dial switching units, and the thousand-and-one other kinds of apparatus that go into the plant of the Bell System. It has purchased from thousands of other manufacturers the great variety of supplies that are used by the Bell System. (Q2)

The majority of that output—at least what wasn't still in service during WE's decline—went back to WE for disposal, as well. That included the wire: from simple drop wires to heavy multipair cables, old wiring was routinely cut into sections and shipped back to WE—specifically, to the WE Salvage Works on Staten Island.

In 1883, as the component elements of a telephone industry were swirling around New York and accreting by gravity into the shape of the Bell System, Benjamin Lowenstein arrived from Germany. Settling in New York City, he took up a business that he must have learned back in Europe: metal refining. Within a year of his arrival, the B. Lowenstein & Bro. company was smelting scrap metal from a shop in Manhattan (the brother, Moses Lowenstein, was a constant second fiddle in Benjamin's ventures until he sold his share and retired to go his own way in 1900).

Lowenstein had a way of maneuvering his metals businesses into the path of technological progress. His first such success was lead, or rather an alloy of lead with tin and antimony. This specialized alloy was eutectic, meaning that it melted and solidified at a single, well-defined temperature, and a low one at that. These were exactly the requirements for feeding the newly-invented Merganthaler hot-metal typesetting machines, later known as Linotype—much as the metal came to be known as Linotype alloy. By 1890, B. Lowenstein & Bro. was the major supplier of feedstock for hot-metal typesetting in the US.

Linotype metal brought in a lot of money, enough that Lowenstein looked to expand. Manhattan was already dense enough that it was hard to find a site for a large industrial operation. Instead, Lowenstein found land in the southern end of Staten Island, near the neighborhood of Tottenville. There, he founded the Tottenville Copper Company. Tottenville Copper grew quickly, well positioned for the new demand for copper brought about by the electrical revolution. During the 1900s, Lowenstein rebranded B. Lowenstein & Bro. as the Nassau Smelting and Refining Company and moved to consolidate it with Tottenville Copper. In 1914, as the US entered the First World War, Nassau Smelting and Refining was noted as one of four companies responsible for 90% of the country's copper exports.

It's said that war is good for business, and it certainly was for Lowenstein. The war brought a pressing need for copper, and the Tottenville plant was pressed into military service. This part of the company's history is, unfortunately, well-documented due to a scandal all too familiar to our present times: on March 27th, 1918, police officers seconded to Naval Intelligence raided the Tottenville copper plant and arrested sixteen laborers—Germans and Austrians, many of them crew members of German merchant ships who had become trapped in the United States by wartime turmoil. In finding productive employment, a way to support themselves, they had made the critical mistake of taking jobs that supported the war effort.

The intelligence officers spent most of the day at the plant in Tottenville, which employs about 500 workers. Practically all of them were questioned, but most of them were found to be either native Americans or naturalized citizens. The sixteen who were unable to show either that they had registered or had obtained zone permits were placed under arrest at the plant. (Q3)

Because of its role in supplying copper components of artillery shells, the Tottenville smelter was considered a munitions plant, and was thus off limits to any enemy aliens who did not possess a specific movement permit issued under police supervision. A few of the sixteen arrested had not registered as enemy aliens, but most had registered and had the wrong work permit. The newspapers do not suggest that these sixteen had committed any offense other than a lapse in paperwork, but they were nonetheless "turned over to Federal Authorities for internment." The authorities were also, reportedly, investigating an allegation that a manager at the plant had made "seditious remarks." These included criticism of the Liberty Bonds used to fund the war effort, and a suggestion that American forces in France would not prevail. Fortunately for the plant manager, he was able to produce paperwork proving his citizenship, and officers deemed the evidence of his "seditious" opinions to be insufficient for charges.

If the war brought good fortune to Lowenstein, peace took it away. The end of steady military contracts complicated the finances of Nassau Smelting and Refining, requiring a retooling of the plant towards other products in the difficult context of the post-war recession. A major fire at the plant, in 1923, racked up a huge repair bill and cut into production. There were personal problems, too: in the mid-1920s, Lowenstein divorced, starting a bitter multi-year legal battle over custody of his children—a question ultimately resolved in his favor, but not without the involvement of the appellate courts and an axe-wielding deputy sheriff. The financial condition of his company continued to decline as the country slid into the Great Depression. Lowenstein must have been looking for an exit.

By this time, Nassau Smelting and Refining was consolidated into a 45-acre property on Staten Island straddling Mill Creek, just south of State Route 440 and between Arthur Kill Road and Page Avenue. There were two primary operations: the "red metals" complex which processed copper, and the "white metals" complex for lead and tin. Both ran primarily on reclaimed scrap, refining it into ingots ready for reuse. Conveniently, these were two categories of metals in great demand to the Bell System: copper, for wiring, and lead and tin, extensively used to coat cables and splices and as key ingredients in solder. The post-war period brought not only general economic decline, but also an increase in metals prices, stressing AT&T's supply chain. Western Electric turned its mind towards consolidation. Given the Nassau plant's proximity to WE's headquarters in New York City and plants throughout the region, WE must have already done quite a bit of business with Lowenstein's company. In 1931, they bought it.

Likely because Nassau Smelting and Refining was already a well-established business, WE left it to operate as an independent subsidiary, alongside the Teletype Corporation (which was acquired at nearly the same time) and, later, the Sandia Corporation in Albuquerque—the three independent subsidiaries of Western Electric through the 1980s.

By the 1940s, Nassau Smelting and Refining was processing thousands of tons of scrap each year. Most of this was disused telephone equipment and cable that had been broken down at other WE plants and then delivered to Staten Island for smelting. WE reported that about 3/5 of the nonferrous metal content of this scrap was returned to WE as high quality metal stock for manufacturing use. In the mid-1950s, Nassau Smelting and Refining provided about 16% of the Bell System's copper supply and 20% of its lead. During the Second World War, when copper became exceptionally scarce, the Nassau works provided a critical in-house metal recycling capability that not only supplied the military as a contractor but also allowed AT&T a reliable source of metal for its wartime telephone projects. During several such periods of disruption in the metal market, Nassau Smelting and Refining provided most of AT&T's supply. For AT&T, vertically integrating metal smelting thus had two key advantages: cost savings from owning its own supplier, and a degree of protection from the whims of the market. "Nassau Smelting and Refining Company is a further extension of Western Electric's constant effort to do its Bell System job better and more economically" (Q2).

A 1946 newspaper article, announcing an open house at the plant with tours open to the public, gives a sense of the scale of the operation. Each day, an average of five railroad cars of scrap arrived at the plant. Stripping machines separated lead sheathing from telephone cables "like a child would peel a banana" before the remains were fed into the furnaces—enough oil to heat a house for a year kept each furnace at 2,000 degrees for one day. Lead was transferred from furnaces to kettles, 30 tons at a time, and poured into ingot molds. Much of the lead then went to the plant's on-site solder mill. Each of the presses there formed enough rosin-core solder to reach "from Tottenville to St. George [at the far end of Staten Island] and back," each day.

During the mid-century, WE expanded the Nassau company's remit beyond just nonferrous metals. Nassau Smelting and Refining became WE's general broker of scrap and secondary materials, brokering cinders from telephone company power plants as a soil amendment, and iridium recovered from telephone relay contact points as a precious metal. As a subsidiary, rather than a mere component of Western Electric, Nassau was somewhat more independent of AT&T than the rest of WE. The metal operation wasn't restricted to the telephone industry, and both purchased scrap on the open market and sold metals to any buyer. Nassau was one of two bidders, for example, on an enormous post-war Naval copper supply contract.

Metal refining is not a clean operation. In 1947, Nassau faced charges of "smoke annoyance" and "noxious conditions" that might have led to a criminal prosecution, were the case not forestalled the company's agreement to install a $350,000 bag house to filter furnace emissions. By this time, Nassau was called one of the nation's largest "above-ground mines." Metal recycling, while economical nearly from the beginning of metallurgy, received a surge of interest in a post-war nation that keenly remembered the shortages of the previous decade. Unlike our modern association between recycling and environmental protection, in the 1940s it was styled mostly as a new form of extractive industry: "The Nassau Smelting and Refining Company... mines the vast Bell network for valuable metals.... in 1948,.. Nassau reclaimed more copper than was produced in six of the nation's 14 major-producing states. And of 22 major lead-producing states, only four produced more than was reclaimed by Nassau."

The president of Nassau at the time, William Scheuch, described "American homes, factories, and cities" as the "mines upon which modern industry depends." He noted as well that, by that time, rubber, plastics, rope, and "scores of other materials" had fallen under Nassau's responsibility. "So thrifty are Nassau's experts that even floor sweepings are cooked by incandescent heat to reclaim the last drop of metal content" (Q4). Later that year, Nassau hosted a delegation of European metallurgists discussing techniques for recycling aluminum, then a major focus of Nassau's research department.

In 1956, the Staten Island Advance ran a puff piece on Nassau's production (162 million pounds of scrap converted into 138 million pounds of saleable metal in 1955) immediately next to the headline "Smog Growing Problem." In 1958, the same newspaper carried an editorial by Nassau's new president, Arthur Fegel, under the headline "Nassau Battles Air Pollution." In some ways, the piece is a celebration of the company's 75th anniversary, but the headline betrays an underlying political struggle. "Just as good families are good neighbors,.. an industrial concern like Nassau devotes much attention to being a good citizen in its own neighborhood" (Q5). This was the preface to an announcement, at the end of the article, that Nassau was about to build a new furnace that would burn the insulation off of wire. The exhaust, he promised, would be "clean as a whistle."

In the early 1960s, Nassau underwent a wave of expansion, including new office and warehouse facilities on the north side of the property. The plant had become one of the major employers of Staten Island, with a payroll including "12 fathers and their 14 sons." The Staten Island Railroad operated a train station for the plant's employees, called Nassau, and expanded it in the early 1970s. The plant was one of their largest freight customers as well, with an industrial siding just off of the station. In 1971, the Deputy Commissioner of the New York Department of Air Resources paid a visit, or rather a "sniff," in part to confirm that Nassau had complied with an order to stop burning lead away in furnaces. "In terms of other smelters in New York City, it is fairly clean; but that doesn't mean it is in compliance with all the standards" (Q6).

The 1970s were, for the Bell System, the beginning of the end. An upstart subsidiary of the Southern Pacific Railroad waged an intense legal battle against AT&T's monopoly, won the right to compete on long-distance service, and renamed itself to Sprint before merging with principal AT&T competitor GTE. Corning demonstrated a new communications technology based on light trapped in glass fibers; by 1980 the manufacturing technique for these new cables had become refined enough that they presented a lower-cost option compared to AT&T's coaxial and microwave network (Bell Laboratory's alternate plan for the future of communications, long-distance microwave waveguide, was stillborn).

At the same time, the American environmental movement hit its stride. The Clean Air Act, the Clean Water Act, the establishment of the Environmental Protection Agency; each of these steps imposed new requirements on an aging metal plant that had long been considered a major polluter. Wastewater from metal separation processes, a slurry of heavy metals and petrochemicals and God only knows what else, was redirected from Mill Creek to the site's first water treatment plant in 1973. That plant separated the contaminants into a dried sludge, which for years was simply piled up under the approach ramp of the Page Avenue bridge. Later, this sludge was processed to extract precious metals, but that modest revenue didn't make up for the capital investment. It wasn't a good time to be an expensive part of Western Electric: facing declining revenues, labor unrest, and the weakening state of its sole benefactor AT&T, WE entered the late 1970s as a company in decline. WE started backing away from its integrated salvage operation: sometime in the late 1970s, the remaining metal recycling operations at Nassau were contracted to a company called C&D Recycling, which assumed management of part of the Nassau plant.

Copper smelting operations at Nassau ended in 1981, beginning a multi-year decommissioning and demolition process for the Red Metals complex. The main copper processing building was razed by 1985, but in the mean time, the entire Bell System had met a worse fate: divestiture. Antitrust lawsuits against AT&T, brewing throughout the 1970s, leading to a 1982 settlement agreement that required the dismantling of the Bell System over the following years. On the first day of 1984, the Bell Operating Companies became independent corporations, over 2/3rds of AT&T gone overnight. Western Electric went along with them: one of the key findings of the antitrust case was that AT&T had built and maintained a monopoly through vertical integration so extensive that it deprived their competition of equipment and supplies.

AT&T was able to partially mitigate the unwinding of their vertical integration: by agreeing to completely divest the operating companies, it won terms on which Western Electric could remain an AT&T company, under a new name. AT&T had agreed to end use of the trademarks most associated with their nationwide monopoly, including not just "Bell System" itself but the Western Electric name and logo. WE was reorganized, with some divisions meeting other fates, but most of the company became AT&T Technologies. As part of the settlement, AT&T had gotten relief on various restrictions imposed on them by previous antitrust cases, including a key prohibition on Western Electric marketing general-purpose computers. While there was some initial optimism that that small victory would allow the new AT&T Technologies to take on the likes of IBM, it didn't work out that way. A series of poor decisions, several fundamental missteps, and no doubt some plain bad luck had Western Electric, and its close partner Bell Laboratories, on the downhill.

Post-divestiture, Nassau Smelting and Refining took on a new identity: AT&T Nassau Metals. As part of the settlement agreement, Bell Operating Companies could no longer be required to purchase equipment and supplies from WE. The Bell telephone market was suddenly open to competitive manufacturers such as the Canadian company Northern Electric (later Nortel), itself a fragment of Western Electric that had broken away when a 1950s antitrust case led to a settlement agreement that WE would divest its foreign operations. On top of the Carterfone decision and divestiture making consumer telephones a competitive market, WE found its business seriously undermined. The 1980s saw closure of many of WE's largest plants, Nassau not excepted.

The White Metals complex at Nassau continued longer, shutting down in 1991 with demolition starting in 1996. The last manufacturing operations at AT&T Nassau Metals, by then limited only to electroplating, ended in 2001. Nassau had actually outlived its parent, WE, which was renamed to Lucent Technologies and made independent in 1996. By that time, AT&T Nassau Metals had been renamed to simply the Nassau Metals Corporation, and existed primarily to manage the closure and remediation of the Tottenville site.

All of the original, 1930s-era manufacturing buildings were found south of Mill Creek and had been demolished by 2000. The newer 1960s era buildings, an office building and a warehouse north of Mill Creek, were leased to a developer who found various commercial tenants. Most of the site remained abandoned in its post-demolition state for the next twenty years, though: a formidable environmental recovery was required before reuse.

The Nassau Metals site was investigated by the EPA for inclusion on the National Priorities List as a superfund site during the 1990s, but was ultimately not nominated. The main reason was simple: the Potentially Responsible Party was the Nassau Metals Corporation, a subsidiary of AT&T, a company that was still very much alive. With some cajoling by environmental authorities, AT&T agreed to avoid the federal CERCLA process by entering the New York Department of Environmental Conservation's Voluntary Cleanup Program (VCP). Many environmental authorities offer something like the VCP: one of the reasons that contaminated industrial sites, often called brownfields, tend to stay that way is the uncertainty and liability involved in environmental contamination. Real estate developers are understandably hesitant to commit to a property that may require an enormously expensive remediation in the future.

The VCP provides an alternative: when a company participates in the VCP, they develop a comprehensive plan for site remediation—at their expense—that is mutually agreed with and supervised by the Department of Environmental Conservation. In exchange for the land owner completing the approved remediation plan, the Department of Environmental Conservation makes a binding agreement not to impose further requirements in the future. In other words, it's a legal arrangement to settle on what level of cleanup is "good enough," so that future owners of the property are protected from expensive surprises.

For remediation purposes, the Nassau site was divided into three Operable Units. The largest, OU1, includes the primary industrial area where both the Red Metals and White Metals facilities had been located. From a 1991 Site Investigation Report to the 2011 Final Engineering Report, remediation contractors identified extensive contamination of the soil throughout the site, and downstream on Mill Creek, with lead and other heavy metals. It was found, for example, that a substantial portion of OU1 was built on artificial infill of a former wetland. The fill material, in line with standard practice in the 1930s, is best described as "assorted trash." Everything from construction debris to domestic garbage to old telephones had been piled up and compacted, and then factory buildings put on top of it all. Every time there was a storm, Mill Creek surged against its south bank and washed some of it away, downstream, into Arthur Kill. Particulate contamination from these materials could be identified out into the ocean.

During the 2000s, contractors dredged Mill Creek and parts of Arthur Kill, temporarily dammed Mill Creek to facilitate further excavation, stabilized the banks of Mill Creek by geotechnical methods, developed new wetland areas on other parts of Arthur Kill at a 3:1 ratio to the area permanently disturbed by the site, replanted vegetation, and cleaned out storm sewers that had been contaminated by runoff from Nassau.

These were all secondary efforts, though, in comparison to the largest remediation activity. The majority of the site, on both sides of Mill Creek, were covered with an engineered barrier of soil, stone, geosynthetic clay liner, and asphalt, intended to ensure that the contaminated soil will remain on site. It is simply too large of a volume to practically be removed, and even if it was, the degree of the contamination is so severe that it would be difficult to find a facility licensed to dispose of it. We tend to think of nuclear waste in the most severe terms, a contaminant that we can never be rid of, pretending that this is somehow an unusual outcome. The reality is much worse: permanent on-site entombment is one of the most common fates of industrial contamination, and there are tens of thousands of sites across the United States in which it is forbidden to dig.

Nassau is one of those. Nassau Metals no longer owns the land, it has all been sold to private developers. On the north side, where the office building and warehouse remain (now a restaurant and a complex of fitness facilities respectively), the parking lot and foundation slabs of two new fast food restaurants make up the permanent barrier. Whenever any paving or concrete work is required, a qualified environmental engineer must be on-site to supervise the work and ensure that the integrity of the barrier is maintained. On the south side of the property, which was unpaved, the clay barrier must remain in place under the new construction or a new barrier must be approved by the Department of Environmental Conservation. No digging can be done without arranging for disposal of the disturbed soil at a licensed facility. Covenants on the property require that these institutional controls be observed in perpetuity:

the owner of the property shall prohibit the Property from ever being used for purposes other than for Commercial or Industrial use without the express written waiver of such prohibition from the Department

Because of the particular standards to which the site was remediated, it is considered suitable only for limited occupation within the context of the institutional controls. Healthcare facilities, elder care homes, child care facilities, agriculture or gardening of any kind (even at hobby scale), and residential construction are all perpetually prohibited. The groundwater cannot be used without construction of a treatment facility approved by the state. The barrier system containing the contaminated soil must be inspected and recertified by engineers on a regular basis.

This, then, is the story of Western Electric—it calls for many things and many activities, blended together, to create the miracle of telephonic communications. It takes people and equipment, inventiveness and technical skill. It takes experience, and, above all, the fundamental desire to be of service to the public and the nation. (Q2)

For some 75 years, AT&T was the telephone company, and Western Electric made the telephones. It destroyed them, as well: every phone inspected, tested, and if found wanting, condemned to the furnaces of Nassau. Telephone wires stretched across the nation by 1915; and by 1984 that very same wire had likely been cut, stripped, melted, and refined on Staten Island; quite possibly, it had even been returned to Western Electric, just a few molecules in each of the 35,000 telephones rolling off the line of the Indianapolis Works. Likely also, a few molecules of the soil permanently interred under Premiere Pickleball of Staten Island, a few molecules on the banks of Mill Creek, a few molecules of the ocean.

Metal is highly recyclable; it's sometimes estimated that the majority of the copper ever mined is still in use today. Much of that metal passed through Nassau at some point, there might be a bit of a Western Electric 500 in your smartphone today. There certainly is under the YoYo Chicken: a tombstone for the telephone era.

It has been an unfortunate turn in the software industry, one of many as of late, that gambling is once again one of its primary engines. With the rise of almost nationwide online sports betting, not to mention prediction markets, making odds on real-world events and extracting the money of suckers is no longer limited to island nations. It is a great American pursuit, or at least, that's what modern television sports coverage leads you to believe.

There has always been an uncomfortable relationship between software and the manipulation of marks. Techniques developed by casinos became a fundamental part of consumer software, while the software industry wholeheartedly embraced "gaming" as a market (the older meaning of the term here, meaning gambling). We can readily point to a couple of reasons: first, gambling is profitable, and technology is first and foremost a means of accumulation. Second, gambling is mathematical, or at least arithmetical, in nature. Most forms of gambling involve some sort of complex calculation with real-world stakes.

Gambling predates history, or it might be better to say that gambling has been around for as long as recorded history has been able to observe it. Most early gambling seems to have been based on card or dice games, but humans have been betting on animal fights for more than a thousand years. As sensibilities and resources changed, animal fighting has mostly given way to animal competition. The most famous of these wagering opportunities is horse racing, a form of gambling with such a long and pervasive history that it has often achieved a unique regulatory status as one of the only legal sports betting venues in the US. Well, at least, before Murphy v. National Collegiate Athletic Association.

The earliest recorded horse races were held in England in 1539, and bets were placed. By 1666, horse racing had reached such prominence that King Charles II—himself a jockey—commissioned and then won the "Newmarket Town Plate." That event's eccentric history gave way to the King's Plate, a broader 17th-century racing series whose royal remit made up the first formal rules for the sport. Queen Anne founded the racetrack at Ascot in 1711; while it took decades for permanent facilities to be built at the track, only stands for the royal family came before a betting office. As British empire expanded around the world, horse racing spread with it. Likewise, horse racing spread throughout Europe. By the 19th century, horse racing could be found almost anywhere.

For most of the history of horse racing, betting was based on the setting of odds. A bookmaker would apply historical knowledge, experience, and no small amount of "guesstimation" to set odds of various events—say of a specific horse winning. A bookie might set odds of 7/1 on a given horse, read as "seven to one against." A bettor placing $1 on this horse will make $7 if it wins the race, which of course implies that the bookmaker thinks the horse's chance of claiming first is around 14% or less ¹.

This method of betting is known as "fixed odds," and the fact that it relies on the bookmaker to set those odds is a significant limitation. First, there is the constant possibility of error. A bookmaker who accidentally sets odds too favorably could ruin themselves, which creates a natural pressure towards odds that are less favorable to bettors. At the same time, competition between bookies drove odds the other direction. This tension between bookmakers and bettors, and between the bookmakers themselves, was a constant source of conflict. Besides, despite the upper-class connotations of British horse racing, gambling has always touched on the unsavory. Bookmakers were not always known to be honest. A host of possibilities, from collusion to ignorance, could leave bettors with no good options. Every bet offered might be a bad one—even more so than the spread collected by bookmakers would suggest.

Of course, despite their prominence, horses did not originally inspire the revolution in gambling that would soon transform racetracks—roosters did.

Josep Oller was born in Catalonia in 1839, but his parents emigrated to France when he was a child. His parents seem to have found some regret that he grew up without Spanish, and perhaps the opportunities in Spain were better. In any case, as a young adult, he moved to Bilbao to study. Accounts vary on what happened next: depending on who you trust, he either merely observed cockfighting, or he was enthusiastically involved in promoting it. I tend to suspect the latter, as Oller was an enthusiastic promoter of many things and not too concerned with vice. Decades later, after his return to France, he co-founded Moulin Rouge. That wasn't even his first entertainment venue, just the most famous. But that's a later story, in another industry. In Bilbao, in the 1860s, Oller watched the cockfights. And then, he watched the bettors and bookmakers haggle, argue, and fight.

The vagaries of fixed odds, the questionable motives of the bookmakers, and the general atmosphere of debauchery made it all rather ugly. Oller thought there must be a better way: the calculation of payouts according to fixed rules, with the impartiality and precision of mathematics. When he returned to France and began to promote his new betting system to racetracks, he named it Parimutuel.

Here's how it works: parimutuel betting is not against the bookmaker or house; it is against the other bettors on the same event. Take an event with a list of possible outcomes, like a horse race and the set of horses that might take first place. Each bettor wagers a sum of money on a specific horse, and a parimutuel teller records the details of each of these wagers.

After the race, when the outcome is known, the parimutuel teller sums the total money wagered on the question, which is called the "pool." Some amount, usually a percentage, is subtracted from the pool to cover taxes and a house take (profit margin). The remaining majority of the pool is then distributed to all of the people who bet on the winning horse, proportionally to their wagers. For clarity, let's work an example: Secretariat, Shecky Greene, and Warbucks are running a race. Being a clever person well educated on Kentucky Derby outcomes, you put $100 on Secretariat. Your friend, similarly informed but cash poor, bets $50 on the same outcome. At the completion of the race, the betting office adds up all of the wagers to $900 (I guess it wasn't a popular race day). There is a state racing commission tax to keep in mind, and of course the owner of the race track wants a share, so they remove 20% from the pool leaving $720. That pool needs to be distributed among the winning bettors. Let's say that you and your friend were the only two Secretariat fans present. Having bet $100 and $50 for a total of $150, a parimutuel teller works out that you receive 2/3 of the pool and your friend 1/3, proportional to the original wagers. You walk away with $475 and your friend with $237, give or take some change. A good day at the races.

There are a few things to observe about this system. First, there is no estimation of odds involved (properly called handicapping). The payout on bets is calculated based on the bets placed, regardless of what anyone expected the outcome to be. But, like the prediction markets with which Silicon Valley is so infatuated today, the general expectation is that bettors will place bets proportional to the likelihood of the horses winning... this means that if a popular horse, widely expected to win, does indeed take the podium (a big podium that horses fit on), the wagers on that horse will have made up a large portion of the total pool and the payouts will be proportionally lower. In the most extreme cases, it is possible to place a bet, win, and still lose money: if everyone bet on the same outcome, they all just get their money back, but minus the house take. Exactly how these situations are handled varies by jurisdiction, but you might be relieved to know that in some rule systems there are scenarios where the house can similarly lose money on edge-case outcomes.

An implication of this fact is that the returns on a wager, assuming that it wins, are not exactly known until after the race has begun, since the tellers continue to take wagers (that change the size and distribution of the pool) up to that point. It's also, just, a little bit complicated? Picture yourself at the racetrack, decadent and depraved, and no doubt several beers deep. You place a bet on your favorite horse—if it wins, what do you get? Parimutuel payouts are relatively difficult for gamblers to understand, and that could limit sales and satisfaction. That's an especially big problem when parimutuel coexists with simpler fixed odds options.

Oller came up with a neat solution: advertise parimutuel betting as if it were fixed odds. Oller worked mostly out of carts, portable parimutuel offices, with big display boards over the teller window. As bets were placed, Oller would calculate the expected payouts on each option and convert them to odds. The equivalent odds were displayed on the board—just like the fixed-odds bookie had. Of course, these odds are only estimates rather than exact, and they need to be updated as the race approaches so that everyone knows what they're getting into.

The exact design of these boards changed over time, Oller himself was actively experimenting for the first decade. While later versions would to display the equivalent odds and perhaps a bet count, Oller's first scheme made the math easier (for the clerk) by displaying the bet count and totals on each outcome instead (conversion to payouts was left as an exercise for the customer). Since the main point of the calculation was to determine the totals, Oller called the process a totalisator. The board, where the totals could be read, was often shortened to the tote board ².

We should also discuss what people are actually betting on. So far, we have stuck to the simplest case: betting on which horse will take first place. That's called a bet to win, or a straight bet. It is also common to bet on a horse placing either first or second ("place"), or first, second, or third ("show"). There are a lot of other scenarios you can bet on as well, like the composition or order of the placing/showing horses or the place that a specific horse finishes, but these more complex bets also get to be less common and may not be offered as parimutuel (one property of parimutuel is that, as a practical matter, you need to be able to attract a certain minimum size of betting pool for every option you offer). The point is that there can be a lot in the air for any given race, with multiple separate betting pools—and on top of that, a busy track might be taking bets for multiple races.

Within a few decades after Oller's invention, parimutuel was taking over as the norm at horse tracks. Along with the betting system came the boards: huge tote boards were a main feature of "modern" race tracks by the 1880s. Oller's carts gave way to "tote houses," dedicated offices with teller windows at the front, a business office (for totalisators) at the back, and the tote board on top. As clerks updated the totals, they sent the new numbers "upstairs" to a clerk who worked behind the tote board, changing out number cards.

It's hard to tell where the first mechanical calculators for parimutuel emerged. There were probably multiple parallel inventions, since the idea is obvious, and it appears that many different ideas were pursued. Oller himself probably experimented with means of automation, since the first devices probably came from France. By 1880, at least one German company advertised totalisator machines. Not very much is known about these first devices, but they mostly fell into two broad categories.

First, there were techniques and devices for larger operations to coordinate multiple tellers and speed up payout determination. Most of these were not exactly calculating machines as we think of them today, but more like derivatives of other 19th-century office technology like cash ball systems. For example, in one system that seems to have found use in several countries, parimutuel tellers sold "tickets" of a fixed face value. For each ticket sold, they took a ball like a steel bearing or marble and set it on a rail for the corresponding horse. The ball rolled down the rail to the end of the tote house, and fell into a bucket. At the end of the race, the winning horse's bucket was weighed to determine the number of tickets sold and, thus, the payout on each winning ticket.

On the other end of the spectrum were small mechanical machines intended to speed the work of a single parimutuel teller. Once again, the history of these devices is not well documented, but during the late 19th century and at least in Europe there seem to have been several parimutuel machines marketed. These were probably more like adding machines, dispensing tickets and keeping the sum of tickets sold. Their major limitation was that, so far as I can tell, none of this generation of devices tried to suit situations where multiple tellers were involved.

This highlights a dichotomy between two schools of parimutuel technology: devices sold to single-person operations, running out of carts like Oller's, and devices sold to large parimutuel offices at race tracks. The former did the math but left it to the user to combine sales between tellers. The latter combined the sales between tellers, but left most of the math as a manual exercise. Neither family of devices updated the tote board—at best, they made it a faster manual process (e.g. by regularly weighing the buckets). It should also be said that none of these solutions were reportedly that good. Most were short-lived, and bettors don't seem to have had much trust in them. Marble-bucket-type solutions were known in particular for having a certain margin of error... a margin of error that, to a slightly less scrupulous bookie, became a margin of profit.

Of most interest to us, considering later events, is the early history of parimutuel automation in New Zealand. Parimutuel, as a system of gambling, landed in New Zealand around 1880. The first parimutuel devices and machines soon followed, imported from European manufacturers. An 1884 issue of Southland, New Zealand's Western Star gives some of the flavor of this new pastime, describing the happenings at an amateur steeplechase meet:

The British love for backing one's fancy was specially provided for in a manner the fairest possible to the public, in the shape of a pari mutuel. This machine was worked by two gentlemen whose suave manner was inimitable, to the satisfaction of all parties, who volunteered their services, and have handed over legitimate commission for presentation to the Southland Hospital.

This is a fascinating passage that reveals a few things. First, in New Zealand, parimutuel betting as a concept had been popularized mainly by people who were importing and selling machines—as a result, New Zealanders thought of parimutuel betting as closely coupled to mechanized calculation ³. The perceived impartiality of the machines (and, as it turns out, the suave manner of its operators) did much to reinforce the idea of parimutuel as a fundamentally more trustworthy form of gambling.

Second, the mention of a "legitimate commission" to the hospital reflects the legal situation in New Zealand at the time. The history of gambling and its legality is a fascinating topic that I am trying very hard not to digress too far into, but many of the fits and starts in development of gambling technology result from widespread recognition of gambling as a vice and subsequent efforts to regulate, restrict, or outright ban it. This might feel like a modern issue, but in the 19th century it was already at a steady boil, and some of the parimutuel industry's emphasis on "fairness" was clearly an effort to maintain legal status. Much the same interplay (between regulation or banning of gambling and claims that mechanization mitigates the harm) is visible in our more traditional gaming industry today (e.g. slot and pachinko), although the online sports books are evidently free of the need to defend themselves.

Despite legal complexities, parimutuel was a hit in New Zealand. In 1908, the Trentham Racecourse had a large tote house with eleven mechanical calculators, presumably operated by eleven tellers with additional clerks to calculate sums and update the manual tote board that made up much of the building's facade. Mechanical calculators were indeed available but expediency, and the chaos of a busy race, meant that much of the odds calculation was done in the minds of the clerks updating the board. There was a margin of error involved, and by newspaper accounts a lot of stress and conflict. Still, this was the dominant model for the first decade of the 20th century: New Zealand racetracks replaced an ad-hoc system of bookmakers and parimutuel carts with large, central tote houses where a single, track-wide pool was maintained. The staff of a busy tote house could be 30 or 40 people, all running around, copying numbers between slates, and shouting updates to tote board's loft. Add another challenge of racetrack betting: the tote houses, even as large as they were, struggled to keep up. One can imagine the impatience of a gambler who was just won it big, running up against a harried teller whose mind has already moved on to the next event.

This problem was all the worse because of a problem clearly illustrated by our modern prediction markets: the matter of closing bets. At a roulette wheel, the croupier ritually declares bets closed before the ball can touch the wheel. Otherwise, the last to place their bets have more information on the outcome. Similarly, in a horse race, any bets placed after the race has begun are better than those placed before. The takeaway is that, under most rules, the tote house had to close bets and complete final odds calculations before the race could begin. A busy tote house meant that races started late—and then later, and later. This was, apparently, a very serious problem in the nineteen-oughts.

Relief would come not quite from New Zealand, but certainly nearby. George Julius was born in England in 1873, moved to New Zealand in 1889, and soon enrolled at Canterbury College to study railway engineering. After his studies, Julius took a job with the Western Australia Government Railways, the start of a career that found him in Sydney in 1907. By this time, he had two sons, and a clear mechanical inclination. While he set up a practice as quite possibly Australia's first consulting engineer, he spent evenings with his sons on a bit of a side hustle: an automatic vote tabulator.

This part of the history has become obscure, and the order of events is not completely clear to me, but it went something like this: while Julius had been working in Western Australia, a friend had complained about the slow process of elections and suggested that some kind of machine could do most of the work. The idea stuck with Julius, and in Sydney, he built a prototype and perhaps even applied for patent on a "foolproof mechanical voting machine." Australia has a long history of interesting and innovative election practices, so Julius's invention might have started a new chapter in the history of election administration had it not been a total failure. The prototype must have worked, but Australian authorities were unimpressed and declined to purchase the system.

It might have moldered among so many other not-quite-revolutions had it not been for yet another friend, this one in Sydney, who was familiar with parimutuel betting and the current state of the art in machines. This friend told Julius that his election machine met similar ends (adding up numbers and displaying the results) and explained the operational details of the large tote houses typical of New Zealand racetracks (but, as far as I can tell, not yet popular in Australia—at least not in Sydney, as Julius later wrote that this was the first he had heard of parimutuel). What followed was another series of late nights, but one that ended with more success: prototype in hand, Julius incorporated as Totalling Mechanisms Ltd and made his first sale.

In 1913, the first Parallel Automatic Totalisator was installed at the Ellerslie Racecourse in Auckland, New Zealand. This machine was called "parallel" because it was a single, large mechanism that all of the parimutuel tellers operated simultaneously. It was "automatic" in that it performed all of the calculations end to end. Not only did it record the tickets sold by each teller, it recorded the totals (in currency) for each horse and a grand total. All of these numbers were presented to bettors by a brand new kind of tote board: a mechanical one, made of chain-driven number wheels showing through windows.

Julius's machine was installed in the existing large tote house at Ellerslie, and fitted with 30 teller stations. Different from today's approach, Ellerslie's parimutuel operated on fixed-value tickets sold in different denominations at different windows. A person wishing to place a 20 shilling bet would queue for one of the 20 shilling tellers, but once at the window could buy any number of tickets on any number of horses. As the teller dispensed the tickets, the tote board above updated in real time.

While quite trivial today, it was a tremendous accomplishment for 1913. Conceptually, it was closely based on the established type of mechanical counter that had been sold, for example, by Hengstler. The difference was one of scale: these wheel counters were huge, as they were directly read by the public. The tote board actually was the machine, in other words, and the number wheels behind it were not just displays but the actual mechanism by which the totals were calculated and stored.

At the back of the tote house, an array of concrete weights hung from chains over sprocket wheels. Gravity pulled the weights, which turned the sprockets, which rotated drive shafts for each horse. Or, at least, tried. The drive shafts were normally blocked by pawls. Each of the parimutuel tellers sat at a station with a set of levers, one for each horse. When a bet was placed, the teller took a ticket for the corresponding horse out of their drawer. They inserted it into their machine and pulled the corresponding lever, which both caused a pattern of holes to be punched into the ticket (validating it as properly issued) and pulled on a long metal wire that ran up through the ceiling into the tote board. That wire pulled on the levers of the clockwork-like escapements that locked the driveshafts, so that each pull of the wire caused the counter for the selected horse to advance by one.

Each of the individual counters was connected by chain to a "shaft adder," basically a series of sprockets coupled to the shaft by one-way clutches. When each individual counter advanced by one, it also pulled the shaft adder along with it, which drove the grand total display at the top of the board. As the machine ran, the weights descended, with those representing the most popular horses falling the fastest. A catwalk rail, up in the tote board, accommodated a worker who carried a crank handle from driveshaft to driveshaft. They would crank the weights back up as needed, and reset all of them to the top between races.

The first Julius machine was big and complicated, with the attendant inertia and friction. My description of its workings is more than just a bit simplified, but most of the complications have less to do with parimutuel than the practical realities of machinery. There were several extra layers of mechanisms like escapements and shaft adders, required so that all 30 of the teller stations could advance the machine with a reasonable degree of effort (although the size of the teller's levers suggests that they still required some heft) and without the machine jamming, miscounting, or otherwise misbehaving when two tellers happened to pull levers at the same time. Given his background in railroad engineering, I suspect that Julius drew from techniques used in railroad interlocking machines, which at that time also made great use of levers that pulled cables to drive chains.

Despite later patent applications, the history of this machine is not as well documented as you might hope. An excellent description from the University of Auckland notes that the actual design of the counters is unknown, particularly how they performed carries. This isn't all that big of a mystery as similar counters (then often called totalisers) existed at the time and Julius must have used one of several known techniques, but still, it is always disappointing to have mysteries in the details of such a notable invention.

This first parallel automatic totalisator operated until 1918, although it was modified and reworked several times in that span. At least early on, the tote house continued to keep manual totals for payout calculations, mistrusting the machine where it really mattered. There are indications that it was out of service during some races, and it may not have been all that reliable in general. Still, it worked. After its first proper operation during a major race, the New Zealand Herald published a report that conveyed its success and its complexity:

A small knowledge of the extraordinary demands that the work of a totalisator imposes upon its parts shows clearly how [the] mechanism might have failed in a dozen ways. But in the meantime alterations have been made, the machine has become more familiar to its operators, and it worked on Saturday and yesterday with fine regularity, and to everybody’s satisfaction....

[An] obstacle of importance was raised by the fact that, in view of the facility with which many investments could be put upon a single horse at a time, the machinery was apt to be run at times so fast that the starting and stopping of the rotating wheels, which, in spite of their light construction, have considerable inertia, set up almost destructive strains and shocks. This trouble has been overcome with an ingenuity altogether admirable. The counter releases a spring-driven gear, which can go as fast as it will irrespective of the motion of the wheel on which the figures are shown to the public, and the wheel simply runs leisurely and smoothly ahead until it overtakes the gear. Even then the wheel, free to make several rapid revolutions, would be difficult to stop at the right place if its speed were not controlled by a governor.

The Ellerslie totalisator might not be called a complete success. Julius charged the racing club £4,000, but spent over £11,000 on construction and modifications. The result was so complex, and required so much maintenance, that the staff of the tote house actually expanded after its installation. Still, the immediate display of totals was a huge draw for bettors, and it succeeded as well in speeding up the closing process. I am always hesitant to apply superlatives, but the Ellerslie totalisator has been called the largest calculating machine to date. I believe it to also represent a major step in the history of displays: a very early, if not first, implementation of a billboard-sized data display that automatically updated in real time. In any case, the machine brought in money. On the totalisator's first full race day, the grand total counter went through £41,514. An estimated 83,000 bets, 10,000 on some single races.

On the back of the Ellerslie machine's success, Julius restyled Totalling Machanisms Ltd as Automatic Totalisators Ltd, or ATL. The First World War caused a great deal of disruption to, well, just about everything, including the racetracks. It halted progress in totalisators for several years. The lull didn't last long, and by 1922 ATL had built improved totalisators for a half dozen New Zealand racetracks. These used a simplified design, benefiting from newer technologies such as electricity.

While the totalisator would remain mechanical for decades to come, the Ellerslie machine is the only one to have been completely mechanical. It seems that Julius began to contemplate electrical designs before he even finished the Ellserslie installation, and all later machines (including an "upgrade" installed at Ellerslie after 1918) used an electric drive system instead of the concrete weights and their crank-wielding attendant. More significantly, though, they also electrified the teller machines: instead of pulling cranks, the tellers pressed buttons, which energized circuits that operated the escapements in the tote board via solenoids. This made it possible, for the first time, to locate tellers for a single parimutuel pool at different locations throughout the racetrack.

The greatest of ATL's electromechanical totalisators was installed at Longchamp, Paris, in 1928. This machine, occupying a building that one could easily mistake for a grand hotel were it not for the numbers behind each window, supported at least 270 tellers (some sources say more). Sales numbers don't seem to have made it to the modern age, but with so many tellers working it must have hit six-figure ticket volumes for popular races. It remained in use until 1973.

Later ATL totalisators were reduced in size and price (truck-mounted portable versions for small racing clubs, for example), and improved in features. We learned early on that some manual tote boards had displayed equivalent odds, while the early Julius machines showed only totals. That limitation wasn't permanent, and ATL machines switched over to estimated payout odds in the 1930s. These took the form of "barometer" displays, or vertical bar graphs, that rose as the odds approached 1/1 (a horse favored to win) and fell down to 100/1 (decidedly long odds). Many of the early ATL systems were upgraded to this later design, and some of the "win/lose barometer" machines saw service, like Longchamp, into the 1970s when computers presented a cheaper alternative.

Totalisator technology became somewhat stagnant after the 1930s, perhaps a result of saturation. Many large markets like the UK had legal restrictions on race betting that prohibited large totalisators (although an oddity of UK law meant that they were allowed for dog races, where the Julius totalisator is best known in that country). In countries with laxer gambling rules, like New Zealand and France, there simply weren't that many large racetracks and by the end of the '30s most of them already had ATL machines that would operate for decades more.

The more innovative designs were seen in portable systems, which were a big market for the many racing clubs that held seasonal racing at different venues. They could not afford to build a large tote house, and didn't have the attendees to keep one busy anyway, but a semi-trailer machine could be towed around with the race days, to be set up and run a small crew in the tradition of carnival rides. These portable systems demanded miniaturization, which was difficult with chain-driven gear shafts but much easier with the relays, and relay logic, that had emerged from the telephone industry. Similarly, the transition to more electrical totalisators made them easier to assemble and disassemble, since it was a matter of hooking up wires rather than aligning sheaves and tensioning cables

The burgeoning telephone industry also meant that relay systems and all manner of electric devices were better understood and easier to manufacture than ever before, inviting new manufacturers into the market. British firm Bell Punch, founded to manufacture machines for validating (punching) railroad tickets, saw ATL's products and responded with their own line of totalisators that were almost entirely electrical. These were particularly popular at smaller tracks, since they were much more compact and easier to install.

In the United States, electrical engineer and race horse breeder Harry L. Straus founded the American Totalisator Company, or AmTote, in 1928. It took several years for AmTote to get off the ground, but their 1933 system at Arlington Park in Illinois established them as the dominant US manufacturer. Their machines were entirely based on telephone relays, and appear to have been the genesis of incandescent numerical displays. Straus's design used a grid of light bulbs, 4 wide by 6 high, with a cabinet of relays that would activate the correct bulbs to show the digits 0 through 9. AmTote numerical displays were an innovation in their own right and found widespread use outside of totalisators (the prices on the original set of The Price is Right are a notable example), but the totalisator business was good, and AmTote installed hundreds of machines around the world.

AmTote also brought other innovations. Despite the historical practice of bets on "win" or "place," ATL machines had only handled a single pool of bets for the race. AmTote's more compact machines could afford the extra logic, almost a duplicate machine, to run a second pool. That meant that parimutuel tellers at AmTote ticket machines could select either a "win" or "place" ticket for each horse, and odds and payouts were calculated appropriately. Instead of preprinted tickets, the AmTote machines operated a mechanical stamp to mark the ticket with the bet information including a "code word" that changed for each race to discourage modification or forging of tickets. A central control panel, in the tote house, "locked" the ticket machines to close bets at the start of each race. Later generations of AmTote machines even used modular (connectorized) electrical wiring and connected all ticket machines to a shared bus, which made portable systems much easier to set up and encouraged owners of fixed systems to add more teller stations at locations convenient to bettors.

The totalisers used in AmTote machines are similar to, and presumably derived from, the Strowger rotary or step-by-step switches used in telephone exchanges. They were fitted with solenoids that would advance them not just in single steps, but in units of 5 or 10 steps as well, for atomic issuance of tickets at different denominations. Much else was taken from the telephone industry: the "brains" of an AmTote machine were installed on relay racks, and extensive diagnostic features were fitted, including controls to produce void "test tickets" at each ticket station and a self-test routine that should advance the counters to known values. AmTote even used a distinctly telco power arrangement: motor-generators converted mains power to DC, which floated the lead-acid batteries that actually powered the machine.

ATL was not to be left out of these innovations, and in particular designed an impressive relay-logic system that fit into a truck called the "Totemobile." A number of these systems were built, and ATL operated them as a service for smaller racetracks. For a commission, ATL would send its own staff to your club's track, and wire the Totemobile to ticket machines installed wherever convenient. They took bets, paid out winnings, and at the end of the day they drove away to the next job.

George Julius lived until 1946, by which time he was Sir George Julius and better known for his influential tenure as chairman of Australia's government R&D organization CSIRO. At CSIRO, he turned towards electronics, as did the company he had founded. After the Second World War, ATL started on a major rework of their product line that introduced relay logic and led to the company's expansion into relay and analog computers. The post-war global economy led to new totalisator installations around the world, and with that surge of new interest came new features. "Quinella" bets, that two horses will place first and second but in either order, were newly popular in the 1950s and ATL designed the first machines that could efficiently manage them and display the odds. Also during the 1950s, ATL designed a new generation of ticket machines that issued tickets as small segments of punched paper tape.

In 1966, ATL designed a totalisator for the New York Racing Association with 550 ticket issuing machines, two primary incandescent light bulb odds displays, and twenty smaller odds displays located around the stands. Installed at the Aqueduct racetrack in NYC, the machine handled over $700 million in a season. Here, for the first time, there were no chains, no cables, no rotary selectors. Instead, the tote house was home to two Honeywell 200 computers. A partnership of ATL, Honeywell, and a software firm called Data Trends had reinvented the totalisator as an early online computer application, the computers polling the ticket machines and updating the displays. Over the following years, ATL would adopt the PDP-8 as the core of their totalisators. From 1970 on, totalisators were almost universally computers.

North London's Harringray Stadium, a greyhound track, operated an ATL mechanical totalisator until its closure in 1987. There was, reportedly, a mechanical totalisator operating in Caracas as late as 2005. In the UK, where many smaller greyhound tracks had totalisators installed, there are several examples that are disused but still standing. At many older racetracks in New Zealand, the UK, and Europe, bets are still placed at teller windows under the enormous tote board that once housed an ATL machine.

George Julius changed the "totalisator" from a room full of people to a machine. His company, ATL, would later change it from a machine to an application. The era of electronic totalisators is its own story, with its own ups and downs and anecdotes, one that I plan to cover in better depth one day. In particular, electronic totalisators were a major impetus of improvements in numerical displays. Display systems designed for totalisators went on to be used in all kinds of applications, from scoreboards to transit destination signage. Starting in the 1980s, off-track betting led to nationwide networked totalisator systems and all of the fascinating asides that that entails. AmTote is still around, a major developer of gambling technology. ATL has scattered to the winds, but makes up part of the modern Light & Wonder (formerly Scientific Games) and Sportech.

Still, I don't find these later machines quite as romantic as Julius's cruder invention. There's something about the fusion of the totalisator as a system, a building, a process, and a machine. The tellers at their windows, pulling on wires to influence the workings of a great machine hidden away in the ceiling.

The lever pulls a long wire cord; the cord releases the counter. The gearing revolves a little, and the counter wheels turn. Sometimes the wheels move at long intervals; and then, as a horse becomes popular, its set of wheels will commence to spin frantically, so that one might imagine the count terribly apt to fail. But the ‘tens’ tot up the units, the ‘hundreds’ go up in their turn; and grand total wheels turn phlegmatic somersaults, and perform mathematical prodigies. (New Zealand Herald)

It is funny, then, that one of the faults of the earliest electronic totalisators was their struggle with arithmetic. But that's a story for later.

The submarine is a surprisingly ancient technology—at least in its early, primitive forms. The idea is quite simple, that a well-enough-sealed boat ought to be able to submerge and resurface. It's the practicalities that make the whole thing difficult. It is generally considered that the US Civil War was the first use of submarines in combat; these were primitive machines with very limited operating endurance and navigational capabilities. These submarines were more like torpedoes: you pointed them in the right direction and hoped they went straight.

The First World War benefited from tremendous advances in submarine technology. A number of experimental designs during the 19th century had built practical experience, especially in Germany, and the Germans apt use of the first modern "U-boats" had a significant military impact. British and US designs made similar advances, and submarine warfare was born.

The chief advantage of the submarine is its ability to submerge and maneuver while hidden. WW1 submarines were diesel-electric or gasoline, so their submerged endurance was limited by the power supply stored onboard. Still, these submarines could operate underwater longer than any before, long enough to establish the submarine sneak attack as a key part of naval warfare.

It was also long enough to expose one of the trickiest challenges of underwater defense: communications. Water, especially seawater, is dense and conductive. This is very bad for radio wave propagation: by the first world war it had already been discovered that seawater effectively blocked radio communications. HF radio, the main form of communications at sea (and, in the WW1 era, in general) might only penetrate seawater for a few meters in real-world That meant that submarines had to surface in order to communicate, another de facto limitation on their endurance while submerged.

The Navy had been evaluating electronic communication aboard ships since 1887, when they demonstrated a simple and "radio-adjacent" technology using conduction of waves through the seawater itself. This scheme never worked very well, but was saved by the development of modern wireless transmitters late in that century. Marconi himself demonstrated radio to the Navy in 1899, and in 1903 the Navy bought its first radio sets. Tactical reports from conflicts elsewhere on the globe, like the Russo-Japanese war, reinforced the idea that radio would serve a key role in naval combat.

When C-class submarines Stingray and Tarpon, and D-class Narwhal, launched in 1909, they were immediately given duties including the evaluation of radio equipment. In a classic tale of early technology, the evaluations went poorly. Tarpon ran into mechanical trouble that prevented its planned trial voyage, so the radio set was never installed. Stingray received a cutting-edge quenched spark gap transmitter and receiver set, but the transmitter turned out to be DOA. Still, Stingray was able to demonstrate its receivers, copying a message from the nearby Boston Navy Yard while surfaced.

Narwhal's mission was more ambitious: underwater communication. A test was made on the same direct conduction technology, using brass plates suspended below the ships, demonstrated in 1887. It similarly failed to perform. A repetition of those experiments, done the next year and with improved equipment aboard Narwhal's sister ship Grayling, produced better results. The system provided reliable communications with the "antenna" plates submerged as much as two feet below the water... and no deeper. Frustrated Navy engineers concluded that it was possible to get radio signals through seawater, but not practical.

Through the First World War and following decades, engineers focused on ways to get the antenna to the surface without having to bring up the entire submarine. Around 1915, the Navy adopted a floating antenna buoy that a submarine could "winch up" towards the surface on a cable. Putting anything at the surface was less than ideal, but the anti-submarine technology of the era the small antenna buoy was still very difficult to detect at long range. Submarines just had to make sure it was retracted back to the submarine's deck before attempting anything where stealth was key. These floating buoys were not reliable during WW1, but they could work, and the technology has continued to develop to this day.

Still, there were other ideas about underwater communications. The most important development came from two engineers of the National Bureau of Standards (NBS), or at least, that's what a court ruled after a patent dispute between two sets of supposed inventors. John Willoughby was employed by the NBS, which would later be known as the National Institute of Standards and Technology (NIST), to investigate new types of radio receivers. In the summer of 1917, he was arranging various types of coil antennas at a receiver test site on the Chesapeake Bay when he accidentally dropped one of the antennas into the water. Strangely enough, the radio receiver connected to the antenna continued to provide good reception even as it sank into the bay.

NBS management was not especially enthusiastic about this accident, but Willoughby was. He knew that the Navy was investigating means of communication with submarines, and that seawater seemed to block radio waves, all of which suggested that he might have stumbled on an important discovery. Lacking NBS support for further research, he took the idea to gifted radio inventor and NBS colleague Percival Lowell ¹. In a fine tradition of innovation, the two took to Willoughby's basement for a series of experiments that illuminated the underlying phenomenon: Willoughby had been experimenting with unusually low radio frequencies, below 30kHz where wavelengths become too long for most antenna designs and coils become the best receivers. These lower frequencies were significantly less affected by water than higher, more conventional frequencies, and Willoughby and Lowell built a successful prototype for what they called "long-wave" radio between two coils.

The NBS remained surprisingly uninterested, but Willoughby had a contact in the Navy who felt quite differently. In 1918, Willoughby and Percival joined LtCmd H. P. LeClair, then running the Navy's experimental radio program, at submarine base New London (so named after New London, Connecticut, across the Thames River (Connecticut) from the base). They made a hurried and rough installation of their equipment on submarine D-1 and a surface support vessel. Not everything went perfectly, but they proved the idea: Willoughby, Lowell, and LeClair listened attentively to their radio sets as the D-1 submerged and continued to come in loud and clear.

Within a matter of a few years, the Navy accepted long-wave radio as a standard technology for submarine communications. The various jury-rigged installations at New London showed that coil antennas could easily be integrated into a submarine's rigging, and even better, the Navy had found that long-wave radio propagated over the surface as well as under it. Long-wave communications would serve the entire Navy, and a transmitter site was already underway.

Long-range communications had become a top concern throughout the military in the early 20th century, and a series of meetings between US military branches and between the US and UK led to a scheme of "High Power" radio stations. The first of these, NAA, went up near Arlington, Virginia in 1913. Over the following years, similar stations were built in the US and Europe, facilitating the first direct communications between the two and the first transatlantic voice communication in 1915. The construction and operation of these stations also led to considerable advances in radio technology generally, especially powerful transmitters. NAA was one of the early stations to be equipped with Poulson arc transmitters, almost two times more efficient than earlier designs and well-suited to long-wave operation.

Around the same time as the Willoughby/Lowell experiments, Navy engineer LtCdr Albert Taylor found similar results with long-wire antennas shallowly under the water. These experiments offered another design for concealed submarine antennas (which could be stored onboard in reels and let out with floats that kept them just under the surface), and also demonstrated that long-wire antennas could be buried for transmit use.

Five years later, in 1918, construction was underway on NSS—a new high power station in Annapolis, Maryland. Unlike those before, NSS was specifically designed for long-wave signals. Two 500 kW Poulson arc transmitters driving an antenna 400' square and suspended between four 500' tall towers ². The long-wave capability at Annapolis was not originally intended for submarine communications, but it quickly fell into that niche. During the 1920s, NSS became a key station for submarine command and control of submarines.

NSS itself remained in service until 1996, and it was joined by VLF transmitters at Cutler, Maine; Jim Creek, Washington; Lualualei, Hawaii; LaMoure, North Dakota; and Aguada, Puerto Rico; besides sites in Europe operated with allied militaries. Each of these stations is its own interesting story. The 1,205' VLF antenna tower at Aguada remains the tallest structure in the Caribbean. LaMoure was originally built in the 1960s for a long-wave navigation system called Omega, and was repurposed for submarine C2. Jim Creek went into service in 1952 as the most powerful radio transmitter in the world, using a fascinating antenna that draped from one ridge to another across a mountain valley.

Let's focus, though, on Cutler. VLF Transmitter Cutler is the spiritual descendant of the Navy's original High Power program, symbolized in its inheritance of the callsign NAA. Cutler was part of a Cold War expansion of the VLF system, going into service in 1961. Many other VLF sites received upgrades around the same period, but Cutler was a completely new design. Cutler's two antennas, for redundancy, are each supported by 13 towers. The center tower is about 1,000' tall, and the other 12 make up two concentric rings of about 900' height. The complete antenna is over 6,000' across, or nearly 2 km. Between the tower tops stretches a web of tight horizontal wires, each 1" copper, that form an enormous capacitor. The capacitor's other plate is the ground, electrically reinforced by many miles of buried groundplane wires. The radiating elements are vertical wires, hanging down from the upper horizontal mesh.

In Maine's harsh winters, the wires accumulate ice until their weight threatens the towers. Each antenna is alternately switched into a deicing mode in which it is turned into a 3 MW heating element... just for long enough that the ice melts off. Outer towers are supplemented by short, stout structures that allow the 220 ton tension weights to move up and down on tracks. "Helix houses" at the feedlines of the two antennas sheltered enormous inductors; walls lined with copper served as insulation and to ground the occasional arcs that made the helix houses and transmitter rooms unsafe to enter during operation.

The two antennas were driven by a transmitter complex designed and built by Continental Electronics. The 11 MW on-site power plant supplied the AN/FRT-31 transmitter, custom to this installation, consisting of four parallel units of eight ML-6697 transmitter tubes. The transmitter's control room rivaled that of many power plants, as did its output: the military required at least 1 MW, Continental rated the transmitter for just over 2 MW, and it still operates today at powers as high as 1.8 MW. There are several reasons that the "most powerful radio station in the world" is now difficult to pin down, but NAA Cutler is certainly in the running.

That is the end of the VLF story, in that it hasn't ended. The original 1910s and 1920s VLF sites are mostly decommissioned, but only because they have been replaced by more modern equipment, sometimes on the same site. Cutler, Jim Creek, Lualualei, and Aguada are all still in service. LaMoure may be in some kind of mothballs state but is definitely capable of operating, it has recently seen some use for propagation experiments. VLF is still a key technology in the Navy's C2 and nuclear reprisal plans. So, we can say that VLF has achieved one of the great feats of technical history: it has outlived its replacement.

First, though, we should spend some more time on the theory. In modern parlance, "VLF" describes the band from 3-30 kHz. Most Naval VLF stations operate at around 24 kHz, but some stations support lower frequencies as well and other stations have operated as high as 40 kHz (still considered VLF by the Navy for practical purposes). These wavelengths pass through seawater well because of a basic trait of radio waves that was becoming experimentally apparent in the 1920s and received a thorough theoretical underpinning later. Radio waves attenuate as they pass through materials in proportion to the number of wavelengths in the material. In other words, as a rule of thumb, a radio wave with a 12 m wavelength (~24 MHz) will experience about 1,000 times the attenuation of a signal with a 12,000 m wavelength (~24 kHz). This is true of water or air or any other material, but the attenuation rate in saltwater is so high that the effect is extremely apparent in the sea.

This brings us to our first property of VLF: because of the long wavelength of VLF signals, they pass through water with relatively little attenuation. Still, there is a limit. The details of submarine communications are mostly classified, but from open materials it is realistic for a submarine to receive a VLF transmission up to about 100' below the surface. This depth is already far better than what's achievable with HF, and far superior to deploying a floating buoy. Still, intuition dictates that even lower frequencies could be even better, and the Navy did not go without noticing that possibility.

Second, we should revisit the antennas. One of the key insights of early experimenters like Willoughby and Lowell is that coil antennas create an asymmetry in radio communications. Antennas become more efficient as they reach the wavelength of the signal, or multiples thereof. That means that lower frequencies, and longer wavelengths, require larger antennas—thus the 6,000' wide cobwebs at Cutler and more than one regional height record set by VLF antenna towers. On the other hand, coil antennas, or more specifically magnetic loop antennas, can be very small compared to the wavelength they receive.

Unfortunately, the physics trick that makes magnetic loop antennas work so well (magnetic coupling) is basically one-way. Magnetic loop antennas are relatively inefficient but usable for reception; they're completely useless for transmitting. VLF is effectively a one-way technology, and some of the traffic carried by the Navy's VLF network consists simply of orders for submarines to surface or deploy a buoy for more advanced communications.

Finally, we should observe that the capacity of a radio channel to carry information is proportional to its bandwidth, and that the use of lower frequencies and longer wavelengths makes the usable bandwidth of given radio equipment much smaller (we can intuitively understand this by noting that larger antennas are, simply due to scaling, more precisely tuned to their intended wavelength than smaller antennas). VLF transmitters are only capable of very narrow transmissions, functionally limiting them to continuous wave (Morse code) operation or simple digital schemes at very low speeds.

We probably all realize, as did the Navy, that pushing to yet lower frequencies and longer wavelengths would produce better penetration of the seawater, at the cost of basically every other property becoming worse: larger antennas, less efficient transmitters and receivers, narrower bandwidths. The possibility of going even further—from Very Low Frequency to Extremely* Low Frequency—was just a solution in wait of a problem. The military had a lot of those, and the Cold War was one huge problem.

The idea of a nuclear-powered submarine is almost as old as the nuclear program, and a collaboration between the Navy, the Atomic Energy Commission, and famed admiral Hyman Rickover led to the 1954 launch of nuclear-powered submarine Nautilus. The next decade gave the Electric Boat Company new meaning, as nuclear propulsion displaced diesel in the US submarine fleet and fundamentally changed the strategy of submarine warfare. Nuclear submarines, unlike those using diesel-electric or gasoline propulsion, can be set up to remain submerged almost indefinitely. The reactor does not require air, and provides plentiful power for life support equipment that mitigates the fresh air requirement for everything else. This created a generational change: by some definitions, all pre-nuclear submarines were merely submersibles, ships designed to submerge only temporarily. The nuclear submarine was a new kind of creature, one that not only visited the depths but could live there.

Add in the development of submarine-launched ballistic missiles (SLBMs), which enabled a submarine to direct nuclear weapons at targets on shore with shorter travel time than any other means of delivery. Every submarine became a portable missile silo, one that could not only hide but actively evade detection. Their ideal mission was to lurk, undetected, for extended periods of time.

Of course, this new potential for submarines further stressed communications infrastructure. A nuclear submarine might spend weeks submerged in water that is ostensibly controlled by another nation, making stealth critical. Such a submarine doesn't want to remain close to the surface, which makes detection by all means easier, and also doesn't want to deploy floating buoys or antennas that are easily detected by modern radar. On the other hand, for it to have any value as a nuclear deterrent, the Navy needs some way to deliver a launch order without having to wait for the next duty rotation.

The military spent the early Cold War developing a dozen different systems for survivable delivery of nuclear war orders, things like the High Frequency Global Communications System (HFGCS) and TACAMO that solidified the concept of short, simple, one-way Emergency Action Messages to direct nuclear forces. The Navy needed a way to deliver EAMs to submerged submarines, and that provided the impetus to investigate lower frequencies than ever before.

The lowest generally recognized radio band, ITU band 1, is Extremely Low Frequency or ELF. There is some historic complexity around the definition of ELF, and the modern range of 3-30 Hz does not exactly match the way the Navy has used the term. In general, though, we can consider ELF to refer to the very bottom end of the usable radio spectrum. The extreme lower edge could be said to fall around 7 Hz, where the wavelength of a radio signal matches the circumference of the earth. This leads not only to complex interference problems due to constructive and destructive interactions, it also produces a very high noise floor as global lightning storms trigger perturbances that resonate on and on. Balancing the desire for the lowest possible frequency against the practical challenges of ELF, the Navy settled on the range of 72-80 Hz as the most promising window for submerged submarines.

The history of Naval ELF development is not simple to research. First, the Navy conducted much of its ELF research in secrecy, a result of typical Cold War paranoia and an awareness that the Soviet Union was pursuing a similar idea. Second, much like GWEN, ELF became the locus of fervent public opposition grounded in general anti-war sentiment, demands for nuclear disarmament, and the safety of electromagnetic radiation. Many of the readily available sources on ELF history today come from "electrosensitive" advocates or newsletters, a still-strong movement founded on the mostly unscientific premise that EM fields pose a danger to human health. While mostly factually accurate, these sources require some caution since they tend to mix their historical narrative with observations about EM and RF safety that are now broadly considered pseudoscientific. Still, this frustration leads to two positive outcomes: first, it helps to place the development of ELF radio within a broader cultural context of uncertainty about both war and new technology, emphasizes the unknowns involved in the push to ELF, and makes the ELF stations an interesting focus of the anti-war movement. Second, it leads to a personal connection that likely contributed a great deal to my interest in military communications.

There are rumors, even scant evidence, that the Navy initiated classified experiments with ELF in the late 1950s. There is very little that I can say about this first part of ELF history, besides that the experiments must have had promising results. In 1968, the Navy adopted a full-scale ELF communications plan called Project Sanguine.

The original Sanguine proposal was truly an artifact of the Cold War, remarkable in its scale and doomed to obsolescence before construction even began. The Sanguine ELF station would actually be over one hundred independent transmitting stations, operating in synchronization as a form of hardening. The loss of a subset of those stations, say due to nuclear attack, would only reduce power rather than disabling the entire facility. Of course, to maximize survivability of the individual transmitters, they would all be installed in hardened underground bunkers, each with a set of 2" antenna cables extending 40 or more miles in four directions. The overall layout of stations and antennas created a grid with antenna elements spaced every 3-5 miles, covering a total of some 6,500 square miles. That's larger than Connecticut, but smaller than New Jersey. Perhaps more apropos, it is about 1/10 the area of Wisconsin, the state where the Navy planned to install the system ³.

This underscores a fundamental problem with ELF: antenna sizes. At 80 Hz, the wavelength of a radio wave is 2,300 miles, or about one quarter of the diameter of the earth. Take, for example, a half-wave dipole antenna—a very common antenna design in most bands. For ELF, the antenna would need to stretch from Albuquerque to Portland. Clearly, then, any practical ELF antenna needs to be "electrically short" or, in the relative sense of RF engineering, a small antenna. Small antennas are inefficient, and the smaller they get the less efficient they are. Complicating things further, practical ELF propagation over the surface of the earth requires vertically polarized waves. That means a vertically polarized antenna, and there is simply no way to construct a tower that is hundreds of miles tall.

Sanguine proposed, and most later ELF projects adopted, a style of antenna called a ground dipole. A ground dipole is basically two different electrodes, or grounding rods, driven into the ground a great distance apart and connected by feedlines. The power from the transmitter goes through the electrodes into the ground, where it flows as ground current from one end of the antenna to the other. The ground dipole thus forms a loop, with the feedlines as one side and the ground as the other. The actual RF emission results from the magnetic field between the feedlines above ground and the current flowing beneath, somewhat like the VLF antenna at Annapolis if half of it was buried beneath the ground.

Ground dipoles, like a typical dipole antenna, are directional. They emit RF most strongly in the same axis as the antenna, with strong lobes extending away from the ends of the two feedlines. By installing a second antenna on a perpendicular axis and shifting the phase between the two, you can create a steerable antenna with its strongest lobes pointed in the direction of your choice. That's why the Sanguine proposal, and most ELF transmitters after, have used two ground dipoles in a crosswise layout.

During the 1960s, the Navy performed a series of poorly documented experiments to establish the feasibility of Sanguine. These included a Wyoming power transmission line that was temporarily disconnected for use as an ad-hoc 40 mile antenna, and a power-line-like 110 mile antenna built by RCA in North Carolina and Virginia. The details of this RCA experiment, part of Project Pangloss, have become obscure. It appears that RCA was contracted to evaluate a number of different communications options for the Navy, including the use of other planets in the solar system as passive repeaters, but most of them didn't work out. The VLF transmitter for the project was located at Ararat, North Carolina, and the two two electrodes at Algoma, Virginia and Lake Lookout, North Carolina. A 1963 test successfully got a message from the test antenna to a submarine submerged 150' deep and 520 miles from the transmitter.

Like most of the military's ambitious plans in the late 1960s, Project Sanguine didn't happen. The reasons are complex, or at least several. Sanguine was unpopular with the public: besides specific concerns around safety, the late '60s saw a rising anti-nuclear campaign and a general lack of interest in enormously expensive military undertakings. The fact that Sanguine needed a massive amount of land meant that it was pretty much impossible to site it somewhere that wouldn't generate local opposition, so like ICBM fields, Sanguine was kicked around like a football. Originally planned for Wisconsin, it later shifted to Texas, and Texas didn't like it that much either (although by that point the antenna field had been downsized to just 1,600 to 3,200 square miles). And, of course, the technology was struggling to keep up with the threat landscape. The hardened design of Sanguine relied mostly on the idea that the Soviet Union couldn't possibly nuke most of the transmitters distributed over 6,500 square miles, a reassurance that the development of multiple independent reentry vehicles (MIRVs) seriously undermined.

As public opposition formed, a health and safety review commissioned by the Navy resulted in a noncommittal report that did little to reassure the public (and lawmakers) that the plan was safe. Last of all, but certainly not least, the budget projections for Sanguine were formidable, and Congress did not have the appetite for the spending.

Sanguine made it far enough that, during 1968, the Navy and RCA built a scaled down transmitter and antenna in the Chequamegon National Forest of Wisconsin. This came to be known as the Wisconsin Test Facility, and it was used as a transmitter for a series of jamming tests in the late '60s and early '70s. During this period, the Navy also considered the use of a BPA transmission line from The Dalles, Oregon to Los Angeles as an ELF transmitter—the plan being to actually modulate messages onto the 60 Hz AC power carried by the line, which was incidentally radiated due to the line's largely straight 850 mile span. This plan was called PISCES, and it is unclear if it ever went anywhere, although an interesting rumor holds that it was operational for a short period and used as the "jammer" transmitter for jamming susceptibility testing of the Wisconsin transmitter.

The results of these tests were mostly positive, but that wasn't enough to save an unpopular plan. Sanguine faded away, perhaps replaced by a scaled-down system called Super Hard ELF or SHELF. There is very little information on SHELF today. The idea seems to have been to install an ELF antenna in deep underground shafts (potentially over a mile below the surface) using hard-rock mining techniques. Work on SHELF apparently continued through the 1970s, but it probably never got beyond the feasibility stage.

Instead, the Navy shifted its focus to Project Seafarer. Seafarer was clearly a direct descendant of Sanguine, but addressed many of its biggest problems through a stripped down design. Seafarer transmitters, for example, would be located in surface buildings instead of underground. Still, the same basic antenna design remained, a grid on 3.5 mile spacing requiring about 4,700 square miles. The Nevada Test Site was considered as a location, as was White Sands Missile Range and forestland in the Upper Peninsula of Michigan. Michigan was ultimately selected, a result of favorable ground conditions and the lack of frequent large explosions. Seafarer construction was expected to begin in 1977, but instead it ended. The governor of Michigan shot the idea down, Congress didn't like it all that much, and President Carter signed the order ending work on not only Seafarer but ELF in general. In 1977, after roughly two decades of R&D work across multiple experimental sites, the ELF Program was in mothballs.

The Navy was not so easily dissuaded. Later in 1977, they proposed "Austere ELF," a plan to throw together an ELF transmit site more or less from spare parts. A transmitter at Sawyer AFB in Michigan's Upper Peninsula would feed 32, 45, and 53-mile-long antenna elements, and via a leased telephone line the AFB would also control the inactive Wisconsin Test Facility transmitter. Even this basic, partially spare parts plan fell afoul of the public and congress. It failed to address most of the original health and environmental concerns, and still cost too much.

Serious resumption of the ELF program would have to wait for President Ronald Reagan. Reagan was a fan of big, expensive, technically sophisticated solutions to Cold War programs, and ELF sure was one of those. Reagan approved "Project ELF," itself a scaled down version of Austere ELF. Project ELF used the existing Wisconsin Test Facility, supplemented by an identical 56-mile antenna in Michigan's Escanaba State Forest. Both would be operated by Sawyer AFB.

The Wisconsin Test Facility from Project Sanguine, after 20 years, came to be known as Navy Radio Transmitter Clam Lake: the first operational ELF transmitter. The Michigan site, known as Navy Radio Transmitter Republic, quickly joined it.

It's amusing that a temporary test facility ultimately became the final product, but the Navy had already invested a huge amount of effort in the Wisconsin transmitter. Everything from the strength of the EM field produced by the transmitter to its location in a National Forest had posed complications.

Although Sanguine was intended as a hardened, underground system, burying antennas was a lot of work and the Wisconsin Test Facility had originally been temporary. Instead of buried cables, it used 1/2" aluminum wires strung above ground on utility poles for the two antennas. The voltages on the antenna wires required isolation from the surrounding environment, so as with power lines, trees were cleared to make a right of way for the antenna cables. The Forest Service, concerned about aesthetic impact on the forest's recreational areas, required that the antenna routes avoid some parts of the forest and take right-angle jogs near roads so that it was not possible to see a considerable distance down the antenna ROW when driving past (which would make the existence of the cleared ROW much more obvious). The transmitter site and antenna ROWs are still clearly visible today. At each of the four ends, about seven miles from the transmitter building, around 10,000 feet of buried copper wire make up the electrode.

Trickier were the electrical problems. The ELF antennas could induce a significant potential in parallel electrical lines, and the use of ground return meant a lot of interference on telephone lines. When transmitting, which was ultimately the case 24/7, the 2.6 MW transmitter induced a current of about 300 A in the cables and ground. Understanding these impacts of ELF transmitters was actually one of the original purposes of the Wisconsin Test Facility, and the Navy had built model power and telephone lines parallel to the antenna elements. The ELF system was found to cause problems ranging from flickering light bulbs to phantom telephone ringing, and the Navy installed additional grounding and filtering on public utilities throughout the area at its own expense—even reimbursing the utilities for administrative costs related to customer complaints. Still, the interference problems were not fully solved during the test operations and no doubt contributed to the public's less than enthusiastic support.

The former Wisconsin Test Facility, as Clam Lake, became operational in 1985. Its sister site, Republic in Michigan, went online in 1980. Republic was new construction, not an old experimental facility, but for cost and expediency reasons it was a virtually identical design to Clam Lake with above ground wires to buried electrode screens. Because of geographical constraints, the Republic antenna is not in a straightforward cross configuration. Instead, it's more of an "F" shape, electrically equivalent but with the feedlines placed differently. From 1989 on, the two sites operated in synchronization, with their total 2.6 MW operational transmitter power producing a radiated power of about eight watts.

Yes, even at 14 miles in length, ELF ground dipoles are extremely inefficient. This remained a key problem with ELF. Early Navy ELF plans, like Project Sanguine, had assumed the use of extremely high transmit powers to produce a usable signal. ELF propagates very well, but at the paltry 8 W achieved by the Project ELF transmitters, practical reception still required extracting the transmitted signal from a noise floor that was just about as loud. That meant reducing the practical bandwidth of the system even further, and thus its speed. Project Sanguine would probably have been able to transmit EAMs directly to submarines; Project ELF was not. Even the compact format of EAMs was too long for a system with an effective symbol rate of about one letter per five minutes, or fifteen minutes to transmit the three-letter code groups used by the Navy.

This reduced ELF capability was basically a very fancy pager network. The Navy has not disclosed the details of the scheme, but it's probably something like this: each submarine has a three-letter code group assigned to it. When its ELF receiver detects that specific code group, the submarine crew know that there is a message waiting for them, and they have to move at least close enough to the surface for VLF in order to find out what that message is. The Navy often referred to this as "bell ringing:" ELF messages were like the ringing of a telephone. As a means of supervision, so that submarines knew they were capable of receiving a message, "idle" code groups were transmitted 24/7.

For how hard the Navy had fought to build it, Project ELF did not have a long life. The Navy's ELF submarine communications system was conceived around 1958, became operational over 30 years later in 1989, and shut down in 2004 after just 15 years of service. "The Nuclear Register," an anti-nuclear-weapons newsletter, put it like this: "A surprise Navy announcement signaled the end of 36 years of first local, then global, opposition to the Navy's giant transmitter system."

ELF overcame formidable political odds. Besides Congress's lack of interest in the expense and federal policy concerns around health and the environment, a statewide ballot referendum in Michigan had attempted to prohibit construction and legislation prohibiting ELF transmitters was perennially introduced in the federal congress. Activist groups opposed to the transmitters staged regular demonstrations and, as Project ELF proceeded despite their objections, protests gave way to civil disobedience. Utility poles supporting the ELF cables were cut on numerous occasions, and the transmitter buildings vandalized. "The Nuclear Register" wrote:

Nukewatch said the Navy's closure announcement, while welcome, raises more questions than it answers. The Navy said "improved technologies" and "changing requirements of today's Navy" made ELF obsolete. However, "very-low-frequency (ELF) [sic] alternatives to ELF have been around for 30 years and the 'changing requirements' refer to the end of the cold war that happened 14 years ago," LaForge said.

Indeed, it is hard for me to see the undignified closure of the Navy's ELF program as anything other than an admission of failure. The basic technical concept of ELF appears sound, but the transmitters are large, disruptive, and costly to operate. It is not clear that the advantages of ELF, namely the greater depth at which it can be received, outweigh its downsides or compare well to VLF.

VLF is still used by the US Navy today. ELF is not: the US has had no ELF capability since the 2004 closure of Clam Lake and Republic. China, India, and Russia are the only other nations to have constructed ELF transmitters. The Russian system, ZEVS, operates at 82 Hz from ground dipole antenna in place since at least the early 1990s. It is a candidate for the most powerful radio transmitter in the world, although the exact specifications have not been made public. India's INS Kattabomman gained an ELF transmitter in the 2010s, and while few details are known, China is believed to have constructed an enormous ELF transmitter in Huazhong during the 2010s.

It is, of course, interesting that China and India have both built an ELF capability after the US abandoned the technology. One wonders what made an ELF capability so hard to sustain here, even after the Clam Lake and Republic sites were built. Well, there is an inertia to politics: the organized opposition to ELF, once energized, didn't go away. Area residents and politicians continued to organize for the closure of the Wisconsin and Michigan transmitters until their final days.

Opponents of the ELF sites got plenty of help from both science and popular culture. Preliminary research linking ELF radiation to leukemia has not held up to modern scrutiny, but as with broader EM/RF cancer links this is an area of ongoing controversy. Extensive research by the Navy, mostly on the Clam Lake Site, hasn't found evidence of ecological disruption due to the ELF transmitter. Still, there is ongoing controversy, and one of the reasons for Project ELF's long and tortuous construction process was a series of lawsuits and appeals under the National Environmental Policy Act, contesting the thoroughness of the environmental research.

As usual, these possible connections to health and environmental impacts have given way to conspiracy theories. In the more shadowy corners of the internet, ELF is associated with everything from strange sensations to mind control. And that is where I first became involved.

The X-Files episode "Drive" (S06E02) sees Fox Mulder cornered, practically carjacked, by a man who insists that if he does not drive West then his head will explode. The episode aired four years after the release of Speed and no doubt owes inspiration to that film (Mulder even makes a joke about it in the episode), but it attributes the bizarre scenario to a very different cause. The hapless victim, portrayed by Bryan Cranston, gained his head-exploding illness as a result of some sort of military experiment involving long antennas secretly buried beneath his house. Vince Gilligan wrote the episode, and while there were several influences, the final episode is a direct reference to Project ELF and the surrounding controversy. Years later, because of their collaboration on "Drive," Vince Gilligan cast Cranston as the lead in his show Breaking Bad.

In the episode, Cranston doesn't make it to the West Coast. Mulder and Scully hatch a plan to puncture his inner ear and relieve the pressure building in his brain somewhere on the California coast, but Mulder just can't drive fast enough. Cranston's head explodes.

Over the lifespan of the Project ELF facilities, police issued 636 trespass citations to demonstrators. Congressional representatives introduced legislation and amendments to end the ELF program multiple times. At least a half dozen ELF transmitter concepts were canceled, each one less ambitious than the ones before it. ELF is an interesting technology, but in a way, it's more interesting as a case study in military acquisition.

Take a concept that is expensive, politically unpopular, and questionably superior to systems already in service—but if the military wants it, they tend to eventually get it. After thirty years, the military wears resistance down and gets something pushed through. Fifteen years later, the Navy shrugs, calls it obsolete, and shuts it down. What's left is a 14-mile-across "X" in the forests of Wisconsin, a legacy of controversy that still echoes, and a pretty good episode of The X-Files.

If you've done much with modern cellphones, you've probably noticed just how odd the architecture can be around audio. Specifically, I mean call audio: modern smartphones have made call audio less of a special case (mostly by just becoming more complicated in general), but in older phones you would often find arrangements where the cellular modem ¹ had direct analog audio to the microphone and speaker, perhaps via some switching to share amplifiers. That design meant that the cellular modem functioned basically as a completely independent device, a fully-capable "cellular phone" with the ability to make and receive voice calls. The role of the rest of the smartphone, and its operating system, was just to provide control messages for starting and ending calls.

In modern phones the audio path to and from the modem is digital and it's more integrated into the operating system audio service, but still not fully. You might have noticed, for example, that it is excessively difficult to record call audio on most phones. Regulatory and liability pressures are one reason for this, but another is that it's actually kind of difficult: there may not be any physical way for software running on the main processor to receive audio from the cellular modem. The designer has to put in explicit effort to make that work, effort that only became common more recently to facilitate automatic transcription—and VoLTE, a whole complication that I will simply ignore for the sake of a cleaner historical narrative. You come here to read about old phones, not new ones.

You've probably read enough of my writing to know where this is going: the design of cellular radios, which assume call audio to be part of Their Exclusive Domain, is a legacy of an age-old architectural decision traceable to the original Hayes Smartmodem. It relates to a feature of modems that was widely available, but sparsely used, for much of the PC revolution. The details are odd!

First, for context, let's recede into our mind palaces and travel back to the 1980s. AT&T-designed modems like the Bell 103 had created a standardized family of protocols for data over voice lines, and a company called Hayes introduced a Bell 103-like implementation called the Smartmodem. The Smartmodem was quite successful on its own, but it was more significant for having introduced a common control interface between the modem and the computer. Previous modems had acted as transparent devices that expected Something Else to perform call setup tasks, while the Hayes Smartmodem could pick up the line and dial all on its own. That required that the computer send commands to the modem to configure and start a call.

Hayes designed a simple scheme for sending commands to the modem and switching it in and out of transparent data mode, and that protocol was then widely copied by other modem manufacturers. You could call it the "Hayes command set," and older documents often do, but these days it's more commonly known by the two characters that prefix most commands: the AT protocol.

From its origin in 1981, AT has shown remarkable staying power. Virtually all computer-connected modems, to this very day, continue to use AT commands for basic configuration. Likewise, the basic architecture of the Smartmodem persists: the Smartmodem connected to the host computer using a single RS-232 link that switched between carrying control messages and data. The very latest 5G modems still work the same way, complicated by the addition of multiple separate UART serial channels (so that, for example, control commands, data, and GNSS data can each have their own separate channel) and the adoption of the USB communications device class "Abstract Control Model," a standard UART-over-USB implementation mostly intended to simplify modems. Plug a modern 5G modem into a Linux machine and you can easily observe this: virtually all cellular modems are USB-attached and will appear as a USB composite device with multiple serial adapters, usually attached as /dev/ttyACM* due to the USB-CDC ACM class.

Courtesy of the V.250 standard (a formalization of AT commands) and considerable effort by driver implementers, USB-attached modems "Just Work" as network interfaces on modern Linux—but under the hood, the kernel is communicating with the modem over separate serial interfaces. Back in the olden days, it was common to run PPP (point-to-point protocol) over one of the serial interfaces to use the actual data (bearer) channel, but now PPP has mostly given way to "Direct IP" where you just push packets over the serial link.

Just to complicate things a touch more, there are vendor-specific standards like QMI (Qualcomm) that completely replace AT and find use in modern smartphones, but they're messy with regards to Linux support. If you are personally interacting at this layer, messing with modems or writing communications software or whatever, you are almost certainly going to stick to AT commands. Modem vendors continue to build on AT. If you look at LTE modems made for IoT applications, for example, it's common for them to provide a complete HTTP implementation (and sometimes MQTT, and sometimes some kind of proprietary message broker protocol) accessible via AT commands. That means you can implement an IoT device without a network stack at all, deferring all network operations to the modem itself. With a JSON-over-HTTP backend, for example, you might send AT commands with JSON payloads over the serial control channel and then get JSON back. You never interact with the network at all, the modem is a completely self-contained system. At the extreme, you might implement your entire device using exclusively the modem. This is a common approach for telematics devices like GPS trackers: they consist of nothing but a cellular modem, the telemetry application is built for the modem using an SDK from its vendor, and you interact with it using AT commands. IoT-class modems frequently provide GPIO and user flash for just this purpose.

None of that is actually what this article is about, but I want to make clear how profound the implications of the Smartmodem heritage are. In 1981, the Smartmodem was a standalone device controlled over serial because the limitations of the era's computer made that a practical necessity. Processors weren't fast enough to run the modem DSP alongside other workloads, certification requirements for telephone-connected devices were stricter, etc. Despite the late-'90s detour into "winmodems," most of those constraints still exist, just in the different forms of the cellular network. Today's modems are less v.54 and more 5G, but they still act as standalone devices controlled over serial channels.

Most telephone modems of the 1980s were exclusively data modems. You could use AT commands to make a call, switch into data mode, and then you basically had a very long serial cable from your device to the computer on the other end of the call. That was all these modems did; their only interaction with "The Telephone System" besides as a pair of wires was for basic call control like detecting dial tone and sending DTMF dialing. That was quite natural considering their evolution from acoustic coupler modems (where you dialed the phone yourself and then set the handset on the modem), but by the late '80s, as devices like the Smartmodem with their own call control were common, it started to feel primitive. With Carterfone and the breakup of the AT&T monopoly, computers were starting to feel like first-class citizens on the telephone system. Shouldn't they have more complete support for, well, telephone things?

From a modern perspective, it might seem odd that fax came to modems before voice, but it makes technical sense. Fax machines use a digital protocol that is loosely derived from Bell 103 and belongs to the same extended family as other telephone modems, so modems already had the hardware. Implementing fax support was just a matter of software. With some extensions to the AT command set, your computer became a fax machine. By the late 1980s, fax support was common in modems, usually distinguished by marketing the modem as "data/fax."

For example, the command AT+FCLASS=1.0 changed the modem to T.31 fax mode (fax class 1.0). T.31/EIA-578 is a standard for sending and receiving faxes using a serial connection to a telephone modem, and it was widely implemented by commercial software packages. There were so many "PC fax" packages available in the 1990s that you could stock half an aisle of an OfficeMax with them, and indeed that's what happened. The legacy of this industry is that there are still dozens of "fax server" products built around data/fax modems, like the open source Hylafax.

Fax modems also made a more general contribution to the modem state of the art: the concept of distinct modes. "Fax class 0" was data mode, while values like 1 and 2 and, oddly, 1.0 and 2.0 were used for different fax implementations. There was an obvious, and tantalizing, opportunity: more modes. Maybe, even, a modem mode for that most classic application of the telephone: voice. Could you use your computer for telephone calls?

The idea is obvious, so it's no surprise that several vendors were working on it all at once. Early efforts at telephone-on-computer could be quite comical, consisting of a telephone that was more or less glued to a computer, no electrical connectivity between them. The IBM Palm Top PC 110 is my favorite example of this form, a Japanese-market miniature laptop with a speaker and microphone on the front edge so that you could hold it up to your face to make a call. Besides amusement, it illustrates a fundamental challenge of merging computers with telephony: real-time media is hard.

It seems very funny to build a telephone into a computer because computers are general-purpose devices defined by software. Putting a phone in the computer should not mean physically putting a phone in the computer; the phone should obviously be a software application. Well, obviously from our modern perspective, but real-time media has always been difficult for computers (which, for architectural reasons, are mostly seen today as fundamentally asynchronous, non-real-time devices). Modern computers get away with it by brute force; they're just so fast that they can be wildly inefficient with media and still keep up to real-time. But things were different in the 1990s. Real-time audio processing was a fairly demanding application and most of the computer industry preferred to leave it to hardware.

Still, the voice modem was an inevitability. In 1991, the Los Angeles Times reported that at least three companies were working on some form of "modem with voice support" for 1992. They focused mainly on Rockwell International, which proved the right call. We don't remember Rockwell as a semiconductor company today, but in the 1990s they very much were—Rockwell Semiconductor later spun out into Conexant, now part of Synaptics. At the time, Rockwell was a major player in semiconductors, especially for communications.

Rockwell had particular expertise in answering telephones. During the 1970s, the Rockwell Galaxy Automatic Call Distributor just about invented the modern call center. It was the first digitally-controlled system that answered calls on a pool of telephone lines, placed them on hold, and distributed them to a pool of operators. The flexibility and efficiency of Rockwell's computer-controlled system, which was specifically designed to cut costs by presenting calls to operators as rapidly as possible, displaced AT&T's contact center systems (like turrets) and Rockwell had almost complete dominance in the new world of 1-800 customer call centers during the 1980s.

Rockwell did not manufacture complete modems, but instead chipsets that were integrated into modems by other manufacturers. That makes it a little tricky to figure out the first model that Rockwell shipped with voice support, but it was sometime in 1992. Rockwell's chips quickly lead to a generation of "data/fax/voice" modems from all the usual manufacturers. Despite competition from other chipset vendors like Cirrus, Rockwell's voice modems became the Hayes of voice. By the mid-'90s, data/fax/voice modems were widespread and the majority either used a Rockwell chipset or a chipset that matched Rockwell's control protocol.

Let's talk a bit about that protocol, and how voice modems actually worked. Although the v.250 standard for many AT commands was in place, there was no standard for voice control. To oversimplify a bit, the "core" AT commands are generally AT followed by one or two letters. "AT+" came to be used as a prefix for standardized "extension" commands, like those added for fax support. Manufacturer-specific extension commands used AT followed by some other character, and Rockwell chose AT#.

To start, voice mode was presented as just another case of fax classes. Specifically, Rockwell voice mode is fax class 8, so to put a modem into voice mode you sent AT#CLS=8 (it seems like the properly standardized AT+FCLASS=8 also worked on many later chipsets, but I'm not sure about the early examples) to enter Fax Class 8. In most cases, you will also want to use the commands AT+VLS=?, which retrieves the modem's voice feature support, and AT+VSM=?, which returns the list of audio codecs supported by the modem.

Once in voice mode, you can use a set of voice-specific AT# commands to dial an outgoing call or answer an incoming one. The modem provides messages about call state, so once the call is up, you can issue the command AT#VTX to begin transmitting voice. This puts the modem into a mode much like a data or fax connection, in which everything sent over the serial connection is interpreted as audio data to be played onto the telephone line.

This gets into one of the ugly details of voice modems. Early voice modems had no audio connection to the computer, only serial, so audio data had to be sent over the serial connection. In keeping with telephony conventions, 8-bit PCM was widely supported, but practically required a fast 115200 baud serial connection. That's basically the limit for RS-232 and it was desirable to provide lower-bandwidth options, which mostly appeared as a proprietary form of ADPCM (adaptive differential PCM).

During transmission of voice data, the modem would indicate events ranging from hang-up to received DTMF digits using escape codes prefixed with DLE (ASCII 0x10). Similarly, the computer could send the modem an escape code to indicate the end of voice data. Things worked much the same in reverse: AT#VRX put the modem in recording mode, and the modem sent audio data back to the computer over the same serial connection. The computer would use an escape code to stop recording, and could also receive a number of escape codes indicating various state changes from the modem during recording.

The design of voice modems is pretty simple, just about the minimum viable product for sending and receiving voice data over a conventional telephone modem, and that's probably why it stayed. Voice support appeared in most Rockwell chipsets by 1995, and many other chipsets had a similar (but often not completely compatible) proprietary voice command set. Still, due to what I suspect may have been some ugly industry politics, it was not until 1998 that voice modem behavior was standardized by V.253. V.253 voice modem commands are mostly the same as the Rockwell proprietary scheme, but using the AT+ prefix. For example, AT+VRX and AT#VRX do the same thing in the V.253 and Rockwell command sets respectively, and in practice many post-V.253 modems seem to have just recognized both.

I suspect that Rockwell's interest in voice modems has some direct heritage to their contact center solutions, because the Rockwell voice command set provides more or less exactly what you need to build a computer-based interactive voice response (IVR) system... IVRs being a concept that Rockwell could fairly claim to have invented. And indeed, that was one of the main applications.

The voice modem almost immediately spawned a whole little industry of PC-based small office IVRs. As The Internet came into popularity, many businesses would already have a computer modem connected to their phone line for internet use, so why not let the computer answer the phone as well? Widespread addition of voice support to otherwise ordinary modems meant that some people already owned all the hardware necessary for an automatic attendant, and for those that didn't the cost of telephone modems (with voice support!) was crashing in the mid '90s.

The Internet Archive has preserved two such products from the early days of voice modems, Super Voice 2.0 from 1994 and Phone Secretary v1.01 of the same year. Phone Secretary (by Unique Software) has become quite obscure, and while there is still a company called Unique Software in the telephony market, I think they might be unrelated. Super Voice, on the other hand, made it to SuperVoice Pro 11 in 2016 and you can still buy a license today for just $99. Both of these products, and many more like them, had features like multi-mailbox voicemail and menu trees that used to require far more expensive equipment. Much of the original marketing for these products emphasizes the "Big Business" feeling of a phone tree, and Super Voice's description still says "Ideal for individuals, professionals and businesses that want to grow or appear bigger!"

Given how much people hate phone menus today, it's funny to think of them as a feature for your business's reputation, but small businesses are often trying to appear larger and having a menu of departments behind your single inbound phone line was a great way to do it. Besides, plain digital answering machines were very new in the '90s, so PC-based solutions promised more features at a similar price.

Voice modems got a big boost in 1996, when Microsoft introduced the Voice Modem Extensions for Windows 95. That package extended Windows Telephony API (TAPI) with standardized support for voice modems. TAPI fully abstracted voice modems, which proved surprisingly complex and quite important. While most all voice modems used either the Rockwell commands or a very similar protocol, the actual details of audio handling were surprisingly varied. Poor handling of audio by the computers of the era meant that a lot of external voice modems had their own headset and microphone connections, and sometimes built-in speakerphone, for voice use.

External modems almost universally used a single RS-232 interface that switched in and out of audio data mode, as described previously, but by 1996, internal (ISA or PCI) modems were starting to use a separate UART channel for payload, including audio codec data ². Even better, some manufacturers of internal modems were starting to integrate them with audio cards on a single expansion board. This allowed the modem to "cheat" off of the sound card's ADC and DAC components, and the OS's audio stack, reducing hardware cost with the downside that TAPI had to support a bunch of different cases for how the control and audio channels related to each other, and whether the audio channel was "Rockwell style" or "Soundblaster style." And note well that V.253 wasn't yet published, so while most voice modems were at least Rockwell-adjacent there were considerable inconsistencies from chipset to chipset when it came to the fine details of escape codes and behavior. A lot of software for voice modems from the era have noncommittal language about supporting "most" voice modems, and from 1996 forward that usually meant that they supported whatever voice modems Microsoft had special-cased in TAPI.

The latter type, of integrated modem/sound cards, is particularly important as the Creative Phone Blaster and Creative Modem Blaster were popular examples of consumer data/fax/voice modems. They were indeed Sound Blasters with modems added on, usually just stereo audio out, mono in, and a telephone jack. I am not clear on the difference between the Phone Blaster and Modem Blaster product lines, and I think those might just be older and newer names (respectively) for the same lineup. They are not special, lots of manufacturers had similar products, but they are good examples of just how strange consumer voice modems became.

The appeal of the voice modem to businesses is pretty obvious, since they were a far lower cost option for business phone systems. Unlike many telephony products, though, computer modems of the 1990s had little differentiation between consumer and business or "enterprise" products. Many large modem banks ran on the exact same modem chipsets as consumers used for dial-up connections, sometimes packaged in rack-mount modem banks but sometimes just as a whole lot of external modems taped to shelves. That meant that consumer modems tended to have the feature of business systems, including the infrastructure for an IVR system.

The way that this manifested in commerce was... odd. Take the Creative Modem Blaster DI5660, described on the box as a "56kbps internal data fax modem with voice." While not exactly bargain basement, this also wasn't special, it was a pretty standard modem for the late '90s and received revisions until at least 2000. In the manual, after some mind-numbing complexity around integration with other sound cards ³, and the customary 20 pages of screenshots of the driver installer, the manual gives a feature matrix that promises:

Able to record and play voice messages over the telephone line.

Multiple mailboxes using included communications software.

High performance speakerphone.

So, the Creative Modem Blaster is not just a modem. It's not just a sound card. It's an answering machine. Wow! Nothing else is said about voice features. The install CD includes Cheyenne Bitware for sending and receiving faxes, but I'm not sure that this modem even had voicemail software on the CD. Earlier Creative modems do seem to have shipped with a software voicemail implementation, but only a very simple one integrated into the driver. The short AT command reference on the CD doesn't mention the voice-related commands at all, and at no point in the documentation is anything about IVRs or business applications mentioned.

This situation is really common with consumer modems. Find the manuals or driver CDs from 1990s modems (the Internet Archive has tons). Flip through them, and you will often find mention of multi-box voicemail as a feature and then absolutely nothing else about voice capability. "Answering machine" was the exclusive consumer application to such an extent that some of the manufacturers, including Creative, seem to have adopted the terms "voice mail modem" or "message modem" as replacements for "voice modem."

This narrowing of vision doesn't reflect any technical limitation; these modems all supported the full Rockwell (and later V.253) standards. You can find them on compatibility lists for business IVR packages, for example. Instead, it looks like voice mail was just the only consumer application anyone could come up with. The rest of the potential of the voice modem, for complete custom applications, was complicated and difficult to support, so it was easier to just not bring it up.

This is a general problem with more advanced features in modems. Telephone modems for internet access had all these downsides, like hogging a phone line that they arguably still did not fully utilize, that manufacturers kept trying to solve. Consumers, on the other hand, seem to have simply not cared. Computers of the 1990s were relatively big, loud, costly, and unstable, so people didn't tend to leave them running 24/7. That severely cut into the practicality of a modem-based voicemail package, especially for consumers, for whom the value of a more sophisticated inbound calling experience didn't justify much extra computer maintenance.

Other inroads towards unified telephone/modem features met similar adoption challenges. Consider "modem-on-hold," a common feature of late-'90s modems that allowed for answering incoming voice calls while the modem was in use. It seemed like it solved one of the #1 complaints about telephone modems, but in practice it was tricky enough to use that hardly anyone did. Efforts like ASVD and DSVD provided actual multiplexing of Simultaneous Voice and Data (SVD, in Analog and Digital forms), meaning you could talk on the phone and exchange data at the very same time, but hardly anyone did.

There are a few reasons these didn't succeed. One is compatibility; voice modems had a rough start with the long period between the Rockwell implementation and the actual V.253 standard, but at least most modems were fairly similar to Rockwell and the incompatibilities were isolated to the computer/modem interface where they were easier to handle. Protocols like SVD required support in the modems on both ends and were seldom compatible between vendors, which became an even bigger issue as independent BBSs gave way ISP modem banks (which were fundamentally incompatible with the concepts behind SVD).

More time might have solved the compatibility problems, but consumer telephone modems were also proving to be a dead end. ISDN did not sweep away the modem by 1995 as the telcos had expected, but a few years later DSL did. DSL could do simultaneous voice and data much more seamlessly than even the most sophisticated modem SVD implementations, and with more bandwidth for both channels to boot.

To the modern computer user, the idea that a common telephone modem would casually incorporate a voicemail system is silly. Despite the obvious technical overlap, those are very different domains of consumer products. The window in which it made sense to combine them was fairly short, and the practical realities of the Windows 95 and 98 era severely limited the promised convenience. At least in the consumer realms, voice modems went pretty much nowhere.

Still, it is a technology with remarkable staying power. For example, you might assume that the late-1990s reshaping of the modem industry around thin "winmodems" (that eliminated most hardware in favor of software on the host CPU) wiped out voice support, but it didn't. By moving most of the logic to host software, winmodems actually tended to make voice features simpler to implement. V.253 voice support was quite common on winmodems.

In fact, it's still common on modems today. Consider the StarTech USB 2.0 v.92 modem, a very common choice for anyone in need of a telephone modem today. StarTech titles it as "Computer/Laptop Fax Modem" and "USB Data Modem," never quite invoking the traditional Data/Fax/Voice, but if you look at the feature list you'll find: "VOICE AND CALLER ID SUPPORT: Make and receive phone calls through your desktop / laptop computer." Despite the callout in the feature list, the spec sheet makes no mention of any voice command set. It does tell us that the modem is built around a Conexant CX93010-21Z (recall that Conexant is the descendant of Rockwell), and the Conexant reference manual for that chipset describes a voice command set that is "based on V.253."

This gets at one of the other fatal flaws of the voice modem. It's peculiar that a modem manual would specifically claim voice support and then list all of the supported data and fax modes, but not list any voice modes at all. Every single modem I've found after the mid-1990s is like this: they tell you that they have voice support, but absolutely nothing about it. You walk away with no idea how you would use a voice modem, or what a "voice modem" even is, outside of the one feature that the manufacturer may have deemed fit to ship on the CD, which is invariably voicemail.

I think this is the manifestation of an underlying problem: the compatibility and standardization problems with voice modems go beyond the fact that it took six years for Rockwell's command set to turn into an ITU standard. It doesn't appear that voice modems were ever that well standardized. Despite the wide range of voice-compatible modems on the '90s market, and the wide range of software for using them, most all of those software packages come with a manual full of warnings, caveats, limitations, and workarounds. Despite V.253, different manufacturers had different opinions about the meanings of escape codes. Several common chipsets had outright bugs, which applications had to work around. As we found today, it was often the case that understanding voice modem compatibility required identifying the underlying chipset and getting the manufacturer's command set reference manual.

Modem manufacturers, the end-user brands, seldom made their own chipsets. They were at the mercy of companies like Conexant, who seemed uninterested in further evolution of voice features. No wonder that they found it easier to just not talk about it.

For how little it ultimately mattered, voice modems got a surprising amount of technical investment from the software world. I suppose that in a world before widespread DSL and VoIP, they seemed like a solution to two problems: the disconnect between computer and telephone operation on the phone line, and the desire to use computers to automate telephone call handling. In 1997, reviewer Mark Spiwak wrote for Windows Magazine:

Thanks to the Hayes Accura 288 Internal Fax Modem with Voice, small offices seem large to outside callers. This single expansion card from a trusted modem manufacturer equips a PC with a fast modem and fax capabilities, and provides speakerphone and voice-mail functions. The modem distinguishes among incoming voice, data and fax calls, and routes them accordingly. With the modem's speaker, you don't need a sound card (although it will work with one, and a cable is included). There's also a microphone in the box.

There are ups and downs in this review. First, consider the degree of complexity hidden in the modem "distinguishing" voice, data, and fax calls. That's a longstanding hard problem in telephony that engendered a lot of hacky solutions. I think that what Hayes is referring to here is actually more of a Windows/TAPI feature than a modem feature; as part of TAPI Microsoft shipped something called the Microsoft Operator Agent that would answer phone calls on a voice-capable modem, detect modem tones or prompt the user to select a call mode as an IVR menu, and then invoke the specified application to handle the rest of the call. There was a whole generalized, registry-based routing mechanism to allow multiple applications to accept phone calls based on caller intent. It feels like a rather sophisticated piece of operating system functionality to have almost completely withered away.

On the other hand, there's "a microphone in the box" and the promise that the modem will work with a sound card even if one isn't required. These are hints at the kind of complexity that a typical consumer would have found fairly impenetrable, the ugly tradeoffs involved in running a real-time media application on a Pentium system that would use AC'97 audio if you were lucky.

During the early '00s, Microsoft invested in fax as an OS feature in Windows, although I don't think it ever received that much use. It was already too late for voice modems, though, and Windows support for voice modems (and TAPI) generally declined after XP. On the Windows Server side, NT4's TAPI could provide a fairly complete IVR system with Windows alone, but the potential of COM-based TAPI business phone systems got wrapped up in Microsoft's Unified Communications craze and then died along with it.

Voice modems surely lasted longer in business applications, particularly since a number of business products like PABXs actually implemented V.253 as a bridge for TAPI applications. Quite a few voice-modem-based IVR products are still for sale, suggesting that they have enduring users. Of course, the more actively developed of them now all offer VoIP capabilities as well. Practically speaking, SIP has completely replaced the role of the voice modem, and it says something about voice modems that even with its many idiosyncrasies SIP is the simpler and better supported path.

And now, we come back to our modern era. In the course of researching voice modems, I ran into a 2017 blog post, The sad state of voice support in cellular modems. The author, Harald Welte, laments that USB-attached modems do not present a USB audio device, but "instead, what some modem vendors have implemented as an ugly hack is the transport of 8kHz 16bit PCM samples over one of the UARTs."

In the full light of history, we realize that what Welte describes as an "ugly hack" is, in fact, exactly the way it has always worked. In 1981, Hayes introduced the Smartmodem, a standalone device that abstracted every part of telephony into a serial channel and a simple command format. In 1992, Rockwell made the modem speak, by putting PCM audio over that same serial channel. In 2017, very little had changed—Welte's post has aged poorly only in that many modern phones have the modem more fully integrated into the SOC and use a completely proprietary interface instead of the traditional serial channel.

Of course, that's for smartphone manufacturers, who have the extensive engineering department and the close relationship with their chipmakers to implement such proprietary interfaces. For the rest of us, with our USB modems (now universally USB, even when packaged in m.2 form), the modem is still a black box with a serial port. The computer does not participate in a phone call, that's the modem's job. If you want to say something, you better PCM encode it and send it to the TTY.

Light: it's the radiation we can see. The communications potential of light is obvious, and indeed, many of the earliest forms of long-distance communication relied on it: signal fires, semaphore, heliographs. You could say that we still make extensive use of light for communications today, in the form of fiber optics. Early on, some fiber users (such as AT&T) even preferred the term "lightguide," a nice analogy to the long-distance waveguides that Bell Laboratories had experimented with.

The comparison between lightguide and waveguide illuminates (heh) an important dichotomy in radiation-based communication. We make wide use of radio frequency in both free-space applications ("radio" as we think of it) and confined applications (like cable television). We also make wide use of light in confined fiber optic systems. That leaves us to wonder about the less-considered fourth option: free-space optical (FSO) communications, the use of modulated light without a confined guide.

Well, if I had written this two or three years ago, free-space optical might have counted as quite obscure. The idea of using a modulated laser or LED light source for communications over a distance is actually quite old. Commercial products for Ethernet-over-laser have been available since the late 1990s and achieved multi-gigabit speeds by 2010. Motivated mostly by Strategic Defense Initiative and Ballistic Missile Defense Organization requirements for hardened communications within satellite constellations, experiments on a gigabit laser satellite-to-ground link were underway in 1998, although the system ultimately only provided satisfactory performance at a rate of around 300 Mbps. As it turns out, FSO computer networking is nearly as old as computer networking itself, with a 1973 experimental system briefly put into use at Xerox PARC.

Despite the fact that FSO systems have been generally available and even quite functional for decades, they remained a niche technology with very little public profile until the phenomenon of low-orbit communications constellations (namely Starlink) put the concept of intra-satellite laser communication into the spotlight. Despite various experimental satellite-to-satellite systems dating back to the early 2000s, and more or less clandestine military applications over the same period, the first real production system is probably the EU's EDRS, which went live in 2016. Starlink didn't really get the laser technology working until 2022. That's one of the interesting things about FSO: it seems intuitively like it should work, it does work, but it's a technology that has often sat dormant for many years at a time.

Well, thinking about satellites, we all know that space is hard. There are formidable technical challenges around aiming and detecting lasers in space, and the rate of iteration is slowed by the long timelines of aerospace projects. But what about down here on earth? Where everything is so much easier? Well, we got Li-Fi. Li-Fi is a largely stillborn technology, and not the topic of this article, so I will resist the urge to explain it too thoroughly. As the name suggests, it's intended to provide a capability similar to Wi-Fi (short-range networking between multiple devices) but using visible light. Despite various demonstrations of gigabit or faster systems, Li-Fi has next to zero commercial adoption, with most uptake in military applications. There's just something about the military and light, which we'll get to later. But here's what I want to discuss today: the golden age of FSO communications, a brief period where the cutting-edge technology behind the television remote control appeared to be the future of short-range computer networking.

During the 1980s, Hewlett-Packard manufactured scientific and graphing calculators with a feature set that increasingly overlapped with personal computers. This era can be hard to understand for people around my age or younger, who associate graphing calculators exclusively with the few Texas Instruments models blessed (and demanded) by common high school math textbooks. These are a holdover, a specter, of earlier years in which scientific and graphing calculators were serious technical instruments and some of the most sophisticated computing devices available to many of their owners. Features like BASIC programmability, still widespread in graphing calculators but increasingly divorced from actual applications (besides ignoring the math teacher and writing primitive CRPGs), used to be an important part of business computing.

Engineers would obtain (even buy!) calculator BASIC programs that automated common calculations. Life insurance salespeople might quote rates using a calculator BASIC program. Calculator manufacturers often sold ROM cartridges that added domain-specific functions, and these modules now represent the many applications of the programmable calculator: financial modeling. chemical engineering. statistics. Not that many years later, the whole field was virtually wiped out by portable computers, but not before calculators and computers underwent an awkward near-convergence (best exemplified by the TI-95, a calculator with computer characteristics, and the TI-99, a computer with calculator characteristics).

The point is that calculators might fairly be called the first practical portable computers, and people increasingly used calculators as part of business and engineering workflows. The challenge here is that computer applications tend to involve storage and processing of large numbers of records, a task that the small (and often volatile) memory of calculators didn't encourage. A bookkeeper might use a calculator to total the day's transactions, an engineer might use a calculator to compute the capacity of a beam. Both of these tasks involve math, the forté of the calculator, but they also require documentation. The accountant and the engineer both need to record the results of their work for later review.

An interesting early approach by HP reflects the tradition of accounting: bookkeepers tended to use "adding machines" rather than calculators, a distinction that is mostly forgotten today but still apparent to anyone who buys an adding machine and finds it to be a little odd compared to a typical calculator. Besides the keypad layout (with an oversize dual-function +/= key), adding machines usually include a printer. Turn on the printer, total your transactions, and you now have a slip of paper that you can use to check your work, and even retain as part of your records.

These machines were big and bulky, though, and they still are today. What if you could have the convenience of a pocket scientific calculator and a printing adding machine in the same product family? Well, you could: by the end of the 1980s, many HP scientific and graphing calculators supported the 82240B accessory printer. It was even wireless.

HP calculators sent data to the 82240B printer using infrared light. For this purpose, HP developed a simple unidirectional protocol based on a UART hooked up to an LED (and, on the other end, a UART hooked up to a PIN diode). Called "RedEye," the calculator-printer application seems to have evolved over a short span into a more general-purpose, bidirectional protocol called HP SIR, for Serial InfraRed. As the name implies, SIR provided an interface very much like an RS-232 serial port, using a signaling scheme that was even fairly similar to RS-232, if you put the whole thing through an LED-to-photodiode step each way.

There were, naturally, a few adaptations to the nature of FSO communication. HP SIR was bidirectional but only half-duplex, because the realities of optics make it very difficult to build an IR transceiver whose receiving detector will not be completely blinded whenever the transmitting LED is active. Power was also a major concern: the infrared LEDs of the late '80s were not very efficient, and portable devices like calculators were expected to achieve a decent runtime on a few AAs. To cut down on power consumption, HP SIR replaced RS-232's bipolar non-return-to-zero signaling with a return-to-zero scheme in which the actual pulses (i.e. the periods when the LED is active) were much shorter than the bit interval, resulting in a low duty cycle. Since you can only really turn an LED on one way, HP SIR replaced bipolar signaling (e.g. positive for 1 and negative for 0) with a system in which the presence of a pulse in a bit interval indicated 0, and the absence of a pulse indicated 1.

If you paid much attention in your college data communications course, you might wonder about clock recovery when there are a lot of 1s in a row (and thus no pulses at all). We'll get to that later. I had a very painful data communications class that I am trying to forget, and I haven't quite braced myself to discuss line coding yet.

HP SIR was extended to numerous applications, including what we would definitely recognize as portable computers today. HP's "palmtop" computers, like the x00LX series, were just about the size of pocket calculators but ran a full-on DOS. These were a transitional stage between early portables and later PDAs, but they introduced a need that would only become bigger in the PDA era: a quick, convenient way of transferring data between the portable computer and other devices. HP SIR was the perfect answer. Start some software, point the palmtop at the desktop, and press send... infrared provided a surprisingly cost-effective way to implement these local connections without the need for cables. HP didn't forget the printers—this was HP, after all—and palmtops could wirelessly print to select HP printers by "point and shoot."

HP wasn't the only company with a short-range infrared protocol. Japan's Sharp had developed a similar protocol, also for calculators, that might not be much remembered in the United States except for its adoption on Apple's Newton series of PDAs. The Newton's awkward sibling, General Magic's Magic Cap, had a similar (but of course incompatible) infrared capability called MagicBeam. The early '90s brought the PDA and the PDA brought an obvious demand for a consumer-friendly, short-range wireless network protocol... and that's why we ended up with at least a half dozen of them, virtually all infrared-based. Everything from the relative cost of components to the regulatory landscape meant that infrared was easier and cheaper to productize than radio frequency protocols, so infrared was the direction that almost everyone went.

While just about everyone in consumer electronics eventually got involved, it seems to have been Hewlett-Packard that stepped up to drive standardization. I don't have conclusive evidence, but I think there's a fairly obvious reason: HP was one of several companies making portable devices, an area where they weren't unsuccessful but never enjoyed total market dominance. Printers, though... printers were a different matter. HP enjoyed clear leadership in printers from the late '80s and perhaps to our present day. People might own a PDA from one of many brands, but when it came time to print, they'd be pointing that PDA at an HP product.

In 1993, Hewlett-Packard hosted an industry meeting that kicked off the Infrared Data Association, or IrDA. As a group of HP employees recounted the event, it was a smash hit: there were far more attendees than expected, representing more than fifty companies in both consumer and industrial electronics. Within a few years, IrDA's membership grew to 150 companies—including IBM, Microsoft, and Apple. Commercial adoption was similarly impressive: in the late 1990s, IrDA transceivers were a ubiquitous feature of phones, printers, and computers. You may have never used IrDA, but if you were old enough in the 1990s, you almost certainly owned devices with IrDA support.

During the early meetings of IrDA, various candidates were considered before, unsurprisingly, HP SIR was selected as the basis of the new industry-standard protocol. IrDA 1.0 is essentially a rebrand of HP SIR, and "SIR" persisted as the common name for IrDA, although the "S" was changed to "Standard" or, as later versions introduced higher speeds, "Slow." Slow it was, at least by modern standards. IrDA ran at 115 kbps, but for the then-typical purpose of replacing an RS-232 serial connection, 115 kbps was plenty (the same as the maximum speed supported by common serial controllers at the time).

Early versions of IrDA suffered from a lack of standardization. Adopting HP SIR as the basis for IrDA brought in an already mature technology, a simple RS-232 based signaling scheme that was even easy to implement with UARTs. But that low-level standard was basically all IrDA standardized. The application layer, and even error detection and reliable delivery, were left to implementers. As you can imagine, everyone did things slightly differently and interoperability fell apart.

This is an old story in technology standards: you align on the low level, and then the next level up becomes the problem. Reliable interoperability ends up requiring standardized application protocols and, more than likely, peer and service discovery protocols. Well, guess what the IrDA spent the mid-'90s on?

The full IrDA protocol stack, mostly created over the first few years of IrDA's existence, can be a little bit confusing because of the way that it reflects the history. The first IrDA standards were limited to the physical layer, initially SIR, and the IrLAP Link Access Protocol. Besides some basic link control functions, IrLAP handles discovery. When a device wishes to initiate IrDA communications, it repeatedly transmits a random 32-bit ID. The frame timing of the beacon establishes the baseline for a time-slot-based access control mechanism; other IrDA devices that detect the beacons select a random time-slot (in between beacon transmissions) in which to respond with their own ID. This discovery process happens at low speed with small packets, but it also includes capability information on the devices used to negotiate the highest speed supported on both ends.

Once the IrLAP discovery process completes, two IrDA devices will know each other's IDs and have an agreed upon set of basic parameters for ongoing communications. You will note that I say two devices. One of the interesting things about FSO networking is that the nature of light, that most things are opaque to it, creates significant limitations. While IrDA always incorporated features to enable multi-point connections (such as the time-slot contention management procedure during discovery), the physical specifications for IrDA only accommodate connections of up to one meter with the two transceivers contained within a 30-degree-wide cone originating at each of the devices. In practice, IrDA devices had to be so close to each other, and so exactly oriented towards each other, that it was rarely practical for more than two devices to participate in an IrDA connection. This de facto limitation to point-to-point applications became solidified by later development on IrDA standards, which (for the most part) ignored the possibility of multi-point applications. HP application notes describe point-to-multi-point connections as possible but not yet implemented in the higher layers, and it pretty much stayed that way.

Let's take a look at the higher layers, because SIR and IrLAP alone did not specify enough functionality to meet real use-cases. Initially, IrLAP was used as a basic transport layer for various proprietary protocols, but IrDA quickly developed a few open standards that went higher up. IrLMP, the Link Management Protocol, is the most important.

IrLMP can be divided into two sub-layers, although they're less layers and more parallel features that operate at the same level. LM-IAS, Link Management Information Access Services, is a discovery protocol for high-level applications. LM-MUX, Link Management Multiplexing, does what it says on the tin: multiplexes multiple logical connections over a single IrLAP interface.

First, we'll discuss LM-MUX, because it will make LM-IAS clearer. By the time IrDA was in development, TCP/IP was gaining ground as the industry standard for computer networking, and IrDA employed similar ideas about layering and logical connections. LM-MUX specifies a simple frame header that includes the 32-bit addresses of the two IrDA devices and a seven-bit value known as a "selector" or LSAP-SEL. LSAP stands for Link Service Access Point, and LSAPs are the main abstraction for application connections over IrDA. The seven-bit selector is analogous to a TCP/IP port number, so we could compare an LSAP (consisting of a source address, destination address, and selector) to a TCP/IP address/port 4-tuple. If you're not familiar with this part of networking theory, it goes like this: in the world of TCP/IP, the combination of source and destination IP addresses and TCP port numbers uniquely identifies a TCP connection. Similarly, an IrDA connection is uniquely identified by the 3-tuple of source and destination addresses and selector. Like some of TCP/IP's competition, IrDA uses a single selector value for both sides of a connection. If you have read enough about networking to realize the implications of this fact, well, they are indeed true limitations of IrDA. In fact, as we will see later, there are some even odder limitations that emerge from the particular choices behind LSAPs.

Seven bits is not really that many bits, it only allows for 128 or so options. This made it impractical to statically assign selectors to applications, and IrDA never tried. Instead, IrDA relies on a port-mapping approach in which selectors are arbitrarily assigned to applications as needed. The mechanics rely on LM-IAS, so let's explain that.

LM-IAS is the exception to the rule, using a statically assigned selector (0). LM-IAS is based on a data structure (called an "information base" because this was networking in the '90s) consisting of a set of objects. Each object has a "class," identified by name, and an arbitrary number of key-value pairs called attributes. Both class names and attribute names are arbitrary strings, but IrDA encouraged (and to some extent mandated) a colon-separated hierarchical format. For both class names and attribute names, values starting with "IrDA:" were standardized while vendors were free to adopt their own prefixes for internal use (HP papers use the example of "Hewlett-Packard:").

The most important parts of LM-IAS were the "Device" class, which provided basic information on the device itself including a human-readable name, and the list of other classes which represented the applications that the IrDA device supported. For example, the class "Email" described the capability to transfer email messages, and contained a standardized attribute "IrDA:IrLMP:InstanceName" which contained a human-readable name for the email capability (useful if the device, for whatever reason, exposed more than one object of the Email class—which IrDA fully supported).

So, let's go back to the discovery scenario and add in LM-IAS. You point one device at another, it sends beacons, the second device responds, and a connection is established at the IrLAP layer. Now, IrLMP kicks in: an exchange of LM-IAS data allows the first device to present the user with a list of discovered devices (including their names) and a list of applications supported by those devices.

The attributes on LM-IAS objects can be anything, but one was particularly important and, in fact, required by the standard: IrDA:IrLMP:LSapSel (I do not know why it is capitalized this way!). That attribute provided the LSAP selector to be used to communicate with that specific application.

Well, that all sounds simple enough, but let's complicate things further. IrLMP went further than IrLAP alone, but still curiously lacked one of the properties we would expect: flow control. It's actually a little odder than that, IrLMP did provide basic flow control in an exclusive mode with only one logical connection, but when multiplexing was in use, it didn't attempt to address the many flow control problems that emerge with multiple connections (deadlocks, contention, etc). We need another protocol!

At this point, you can tell that internet influence is becoming significant, as IrDA introduced the "Tiny Transport Protocol" or TinyTP. TinyTP is, as the name suggests, much more comparable to TCP in the internet stack. TinyTP added flow control (at the individual connection level) and a robust mechanism for segmenting large payloads, a problem that IrLMP left as an exercise for applications.

What makes TinyTP confusing is its relation to the other protocols. TinyTP is mostly a layer on top of IrLAP, but not quite. TinyTP relies on the exact same selectors as IrLMP, LSAPs, and it relies on LM-IAS for devices to negotiate which applications will run on which LSAPs. We can add a new standard attribute for LM-IAS objects: IrDA:TinyTP:LSapSel, which indicates that a service is available over TinyTP, and the LSAP selector to use. The result is that TinyTP and IrLMP are two parallel alternatives, providing similar interfaces and running over the same lower layer, but TinyTP also relies on IrLMP for initial connection setup.

Now, let's loop back to the implications of LSAPs. In the TCP/IP world, a connection is uniquely identified by the 4-tuple of addresses and ports. This means that one host can open two connections to the same service on another host, differentiated by the choices of source ports (which are ephemeral ports in the TCP/IP design). With IrDA, connections are identified by a 3-tuple, which means that you actually can't do this. A given host can only have one connection to a given service on another host, because there's nothing in the headers to differentiate two connections with the same LSAP selector. This is mostly just of academic interest, since the protocols defined on top of IrDA were designed with this in mind, but it's always interesting to see these differences between network architectures. So, here's another: since IrLMP and TinyTP use the same LSAP selector format, and run under IrLAP with the same addresses, you cannot differentiate between IrLMP and TinyTP connections to the same service. Once again, not a big problem in practice because everyone knew about this limitation and would not attempt to connect to the same service with both protocols, but it's still interesting that you can't. If we compared the difference between IrLMP and TinyTP to that between UDP and TCP (which is an imprecise but still useful comparison), the difference stands out, because UDP and TCP connections are differentiated in several different ways. In practice, IrDA applications addressed the problem by assigning different LSAP selectors to the same application for TinyTP and IrLMP, for those applications that supported both.

IrLMP and TinyTP provided a pretty robust capability that met most needs for basic network connections, especially since TinyTP was similar enough to TCP/IP in its semantics that TinyTP connections could be treated as Berkeley sockets. IrDA applications could thus be written a lot like internet applications, using some of the same libraries and techniques. Of course, mentioning this must make you wonder: TCP/IP over IrDA? Yes, that's an application!

But first, we'll revisit the physical layer, because IrDA received several revisions of its bottom layer. SIR, the 115Kbps mode taken directly from HP, gave way with IrDA 1.1 to MIR, the Medium speed Infrared physical protocol. MIR operated at 1 Mbps, and it's particularly interesting to me because it reflects one of the major trends of IrDA development. IrDA 1.0 was published in 1994, basically as a formalization of existing HP SIR devices (which generally became IrDA devices with just software changes). IrDA 1.1 was published in 1995, just a year later, but is far more reflective of IBM than HP.

In 1995, IBM's ThinkPad line of portable computers was massively successful and basically defined the "business laptop." ThinkPads sold like hotcakes and found widespread use in the kind of applications that used to have people reaching for programmable calculators... and many more. But wireless networking was still a problem, and Wi-Fi wouldn't achieve widespread adoption for several more years. IBM leaned heavily into IrDA, and for many years it was a feature of every ThinkPad model.

MIR was fairly similar to SIR in terms of line coding, but faster. To address clock synchronization problems at the higher speeds, MIR used a bit-stuffing scheme taken from HDLC, part of the ISO network stack that was directly derived from SDLC, which was the data link protocol for IBM's SNA network stack. So, just as TCP/IP took over, IrDA went the SNA/ISO path, at least in a small way.

Subsequent revisions of IrDA introduced FIR (Fast Infrared) at 4 Mbps (1998), and VFIR (Very Fast Infrared) at 16 Mbps in 2001. By the time IrDA got to 16 Mbps, the "basically RS-232 over an LED" scheme had been replaced with a more sophisticated non-return-to-zero run-length-limited line coding, much more similar to what radio protocols used (which were both more sophisticated and more common by 2001). The fact that radio modulation methods could be applied to IR meant that the sky was the limit, and subsequent work introduced a 96 Mbps protocol (UFIR) and a 0.5 or 1Gbps protocol called GigaIR. Unfortunately, GigaIR came much too late in the limited lifespan of IrDA. As far as I can tell, it never made it to any widely available products. Even 16 Mbps VFIR is rare in practice, so for most purposes we can consider 4 Mbps the fastest speed achieved by IrDA.

Aided by the simple and internet-like interface of IrLMP and TinyTP, IrDA was applied to quite a few tasks. One of the more common application protocols used over IrLMP was IrCOMM. I suppose that IrCOMM stands for Infrared Communications, but that doesn't mean that much, does it? IrCOM might be better, because IrCOMM provided emulation of traditional serial and parallel ports over IrDA. It sounds silly to run a serial port emulation protocol over a network protocol over a data link protocol over a physical layer that was originally designed to emulate a serial port, but the scope of IrCOMM's support for physical serial ports goes beyond just providing a character-oriented communications channel. IrCOMM provides a set of control messages and behavioral standards to replicate the full feature set of RS-232, including the control signals. It also provides integrity and reliable delivery, so that traditional serial port applications will reliably work over the unreliable IR link. IrDA as a drop-in replacement for serial ports proved popular, especially in industrial diagnostics and programming. A number of early digital cameras also supported IrDA for transferring images to computers or printers (or, in Japan, sending images over select IrDA-equipped payphones, because Japan is like that)—and they used IrCOMM to convey the same vendor-specific protocols they had used over serial cables.

Considering IrDA's history, it's no surprise that printing was another popular application. There were a few ways that an IrDA device could send a job to the printer (fragmentation remained a problem, to some extent, for the entire lifespan of the protocol), but the Cadillac option was IrJetSend. JetSend is a topic of its own, a surprisingly complicated and highly generalized protocol that could probably be used for just about anything. But HP developed it, so it was used for printers and scanners. IrJetSend let the printer describe complex user interfaces to the client, so you could print from your PDA with access to all of the capabilities of your workgroup printer. Living the dream.

My personal favorite IrDA application protocol is OBEX, and it's also the apex of IrDA's goal of interoperability. Many descriptions of OBEX compare it to HTTP, which is fair, and there are several tells (besides timeline) that suggest that OBEX's designers had the world wide web on their minds. OBEX operates over TinyTP, and once establishing a TinyTP connection to the OBEX object from the LM-IAS, an OBEX client sends a "CONNECT" message that includes a service name to indicate what the client wants to do.

Like HTTP, OBEX was designed for very general document-moving purposes and can thus be used for a lot of different purposes. If we stretch the HTTP analogy, we could say that the service name provided in the CONNECT message is a bit like the Content-Type header, as it tells the server what kind of thing the client intends to interact with. This analogy isn't great, because different OBEX services are likely to be handled by different applications. Well, let's just consider examples. One OBEX service is "file transfer," which is just for generic file copy operations. Others are IrMC, which performs offline PIM synchronization in vCard-family formats, and SyncML, which performs offline PIM synchronization in SyncML (an XML format). If you have no idea what I'm talking about when I say "offline PIM synchronization," well, I should probably write an article about it. In the meantime, look up "Microsoft ActiveSync" and try to imagine that it is 2003 and you own a PDA.

Once an OBEX connection is established, it proceeds using the verbs GET and PUT in basically the same way (and for the same purposes) as HTTP. You can get files, you can put files. The only real difference between the behavior of HTTP and OBEX here, besides that OBEX is vastly simpler, is that OBEX is stateful about the working directory (similar to FTP) so it has a SETPATH verb for changing the working directory.

So I guess OBEX is more like FTP? But it has headers like HTTP and HTTP verbs. I don't know, pick your favorite comparison.

Here's where all this matters: one of the showcase applications of IrDA is that you could send things between phones, like AirDrop or something. You flip open your phone (because this is an era in which it does indeed flip open), push a couple of buttons, point it at your new acquaintance's phone, and it is as if you provided a business card. You aren't just using infrared networking, you are Networking using infrared. Similarly, you can send files, but considering IrDA speeds you probably want to keep them slim. Fortunately the most popular use-case here was sharing photos, and any camera on a phone with IrDA was probably just getting into the megapixels.

In the early days of IrDA, this didn't actually work reliably, because of different application-layer implementations (such as IrMC vs. SyncML). When it did start working reliably, it's because your respective Nokia phones were exchanging vCard files using OBEX over TinyTP over IRLAP over FIR, the IrDA stack in its full glory.

Wait—stop—hold on. When I say that you can, you know, send a file to your friend's phone... maybe your favorite MP3... what would you call that? If you are the same kind of person as me, you will say "squirting," of course. You squirt a file to someone.

I want to squirt you a picture of my kids. You want to squirt me back a video of your vacation. That's a software experience.

Steve Ballmer said that, in 2006, about a feature of the Zune. Using Wi-Fi Direct, two Zunes could send files to each other. It's a little odd that Ballmer doesn't mention using this to send music, but keep in mind that Microsoft had to tread very carefully to avoid the attention of the RIAA. The point is that it was a Wi-Fi version of IrDA's OBEX file sharing, and Microsoft was widely mocked for calling it "squirting." The funny thing is that I think this is a little overplayed, the Zune didn't actually it "squirting" in the UI and I don't think that term was even used in documentation. But Steve Ballmer used it, as did other Microsoft employees, so it seems to have been the term they used internally.

The other funny thing is that they didn't come up with it.

Squirting was already the accepted term for this feature by the time the Zune came around. It's generally assumed that one-shot file transfers came to be called "squirts" because "squirt" is used in a similar way for brief radio signals (back to at least WWII). I can't promise you that I have found when this happened, but I have a good contender: HP's CoolTown research project, which ran from 1999 to 2000 or so and developed an IrDA protocol they called "e-Squirt." By 2000, papers about IrDA referred to file sharing as squirting. I have no doubt that the Zune team would have picked up the term from there, given Microsoft's extensive involvement in IrDA.

Well, I didn't expect the tangent about squirting, but we had to do it. Back to our main program: whatever happened to IrDA?

It's frustrating, because IrDA had a lot of potential. The work on GigaIR, even though unrealized, showed that IrDA doesn't have to be slow. IBM invested a lot of time and verbiage in IrDA AIR, or Advanced Infrared, which circled back on the whole "point to multi-point is possible but unimplemented" thing. AIR was an overhaul of the whole IrDA stack that made networks of three or more devices possible although, as it turned out, not necessarily practical. AIR was supported by a lot of IBM products but never really used for anything, because by that point devices were getting Bluetooth and Wi-Fi.

That's about it: IrDA just got wiped out by Bluetooth and Wi-Fi. Wi-Fi meant that "thick" mobile devices like PDAs would just connect to a network, wiping out the whole synchronization scenario. For more transient purposes like file sharing, Bluetooth promised more convenience, even if I'm not sure it delivered it. IrDA did require that the two devices were in direct sight of each other, which imposed design constraints on mobile devices and implied that you would hold them pointed at each other the whole time.

By the mid-2000s, things just got worse. Integration of Bluetooth and Wi-Fi modules meant that Bluetooth was "free" in a lot of mobile devices, which is even cheaper than the IrDA components. IrDA mostly dealt with lighting well, but not direct sunlight, and smartphones probably made that scenario pretty frequent.

IrDA published IrSimple, a comprehensive set of improvements to IrDA, in 2005. It was mostly a Japanese effort, because Japan being the way it is, IrDA was more widely used there. Infrared was disappearing from mobile devices when the iPhone killed IrDA entirely. Without mobile devices, IrDA was a solution without a problem. The technology, the standard, and the Infrared Data Association itself all faded into obscurity.

This isn't to say that IrDA is gone. It's probably actually pretty widely used, as far as short-range wireless protocols go. There are several common embedded applications of IrDA and IrDA-derived protocols, ranging from power meters to laundry machine diagnostics. These applications benefit from IrDA's low cost: modern documents on IrDA often call out that it can be implemented with bit-banged GPIO or unused UARTs and very few additional components. They also value that IrDA is easily compatible with waterproofing and that it doesn't provoke the regulatory requirements associated with RF.

There's another upside to IrDA, as well: security. IrDA isn't quite visible light, but infrared behaves similarly. Most materials are opaque. If you can seal a room against light, you can seal a room against IrDA—and that's a lot easier than reliably blocking RF. There are probably some enduring applications of IrDA because it's permitted in some areas where RF communications are not, due to concern of eavesdropping or malicious interference.

Okay, you've made it this far, and I did not expect this article to be this long. Let's have a little dessert. As we've seen with Li-Fi, there's interest in FSO communications as a way to connect portable devices to IP networks. IrDA sure has a lot of the ingredients, it just wasn't built for IP.

Well, don't worry, there was a solution to that: IrLAN. IrLAN implemented IP over IrDA with a surprisingly Wi-Fi-like architecture, including support for multiple clients. That part is particularly interesting: IrLAN supports an "access point" mode with one AP communicating with multiple clients. You could, in theory, use it as a direct alternative to Wi-Fi. It seems like HP even built a device for this, the HP NetBeamer, although it is so obscure that I suspect that it only ever existed as a prototype.

But, remember, point-to-multi-point was unimplemented in IrDA. Well, except for AIR, which saw little adoption. How to square the circle?

It's strange, I'm honestly a little confused by it, but the original IrLAN specification has this odd sentence buried in a definition in the glossary:

IrLMP is multi-point-capable even though IrLAP is not. When IrLAP becomes multipoint-capable, multiple machines will be able to communicate concurrently over an infrared link.

Well, it's true that nothing about IrLMP really prevented point-to-multipoint. But just saying that IrLAP didn't support it is rather understating the problem, AIR had to make changes to the physical layer to get feasible multipoint support (and AIR had a bad reputation for performance as a result). Later in the specification, we read that "it is quite reasonable to expect future implementation of access point devices to support multiple concurrent clients connecting to the LAN."

So, it turns out, the authors of the IrLAN spec defined support for multiple clients on the assumption that it would become possible, through some effort like IBM's AIR... and then that just didn't happen.

Honestly, IrLAN doesn't seem to have gone anywhere. Even at the time, many reportedly thought it was a bad idea. Much better to just use PPP, designed for a serial channel with the exact behavior provided by IrCOMM. It's nice to know that at least the protocol side of an IrDA-based alternate Li-Fi was developed. I hope to one day find an HP NetBeamer. I want to pick up some IrDA devices and experiment with OBEX. We could send some files around at 4 Mbps. With a quiet IR background, later-generation IrDA transceivers apparently worked over impressive ranges. AIR targeted five to ten meters. We could build a LAN. We could squirt. That's a software experience.

Years ago, when I was in college, I had one of those friends who never quite had it together. You know the type; I'm talking lost a debit card and took three months to get a new one because of some sort of "mixup" with the credit union that I think consisted mostly of not calling them for three months. In the mean time, our mutual friend ended up in a quandry: at WalMart, at one in the morning, with a $2 purchase and no cash. Well, this was no problem for that particular space case: he had his checkbook.

If you think about it, it's actually pretty remarkable that grocery stores accept personal checks. It's a very high risk form of payment. Even if the check is genuine, the customer could be writing it against an empty account. On top of that, with modern printers and the declining use of MICR, forging checks is trivial. When you offer a check, the retailer has very little to go on to decide whether or not you're good for the money. Surely, fraud must run out of hand—and yet, just about every major grocer still accepts personal checks.

Retail point-of-sale acceptance of personal checks is the product of an intriguing industry that handles all the challenges of checks at once: a combination of digital payment network, credit reporting firm, insurer, and debt collector known as a check guarantee service. The check guarantee is older than the ATM, and depending on how you squint, check guarantees are quite possibly the first form of real-time, telecommunications-based point of sale payment processing.

---

Harry M. Flagg was born in Frankfurt in 1935, but spent most of his childhood in Milwaukee, Wisconsin. He attended MIT, major unknown, and graduated in 1957. I think he was probably an ROTC student, because some sort of Navy service took him from Massachusetts to Hawaii, where just a few years later he was out of the Navy and working as some sort of "management consultant." Flagg was entrepreneurial to his core, so while I knew few details about it his consulting work is unsurprising given the wide variety of business ventures he was soon involved in. We can be fairly confident, though, that his clients included retailers—retailers who struggled with personal checks. In 1964¹, Flagg quit consulting to focus on checks alone.

His idea was straightforward: keep a list of people known to pass bad checks. When a retailer is given a check, they just check the list, and if the writer's name appears they should turn the check away. As legend has it, Flagg took the idea to a Boy Scout meeting where he happened to describe it to a crowd of Honolulu business leaders, one that presumably included his soon cofounder Bob Baer. They agreed on an informal arrangement: Honolulu businesses would report writers of bad checks to Flagg's consulting office, where his small staff would look up names on request. It was such a success that Flagg's staff were soon overwhelmed. Tracking the writers of bad checks became Flagg's full time business.

He christened his new venture TeleCheck—Tele, perhaps, for Telephone, or Telecommunications. Whether his MIT education or his Navy experience, something had introduced Flagg to the potential of the computer. Having seen his busy office staff, taking calls and digging through files, he imagined TeleCheck as a centralized, real-time computer system. By the time he announced the new company, an IBM system was already on order. General manager George Duncan set about designing and testing the process, and somewhere along the way they picked up the engineering talent to build a database for questionable checks.

As explained in TeleCheck's ads, accepting checks required only a phone call. Once connected to a TeleCheck operator, customers curtly said their TeleCheck account number followed by the driver's license number of the person who had written them a check. By the time TeleCheck matured, they settled on a system of three possible results: a "code 1" indicated a low-risk check that TeleCheck would guarantee. A "code 3" meant that TeleCheck didn't have any specific evidence against the check writer, but the value of the check or other risk patterns meant that TeleCheck was not willing to guarantee it. Worst of all was "code 4," telling the retailer that TeleCheck would absolutely not guarantee the check, because the writer already owed them money.

A 1964 newspaper photo shows TeleCheck operator Dorothy Nicholson sitting at the console of an IBM 1440, Harry Flagg looking on from the side. She's answering the phone with her left hand, right hand poised on the keyboard of the teletypewriter. This is probably a staged shot for the newspaper, I sincerely hope that they found Dorothy a headset (admittedly a surprisingly expensive proposition in the 1960s). It also contradicts claims in other newspaper articles that TeleCheck didn't go into operation until 1965, but I think that there was an extended "trial" phase before the service was generally available. I pay attention to these details because they tell us something about the company's early days. Flagg brought on quite a few business partners, so many that I struggle to keep track of them, and I assume that the computer was much of the reason. They were probably renting it, but that rate would have apparently been at least $1,500 a month, equivalent to about ten times that today. TeleCheck had capital. I assume that many of their early customers, taken from that Honolulu Boy Scout meeting, were investors as well.

Into the IBM 1440, TeleCheck combined several data sources: they formed a partnership with the Honolulu Police, from which they received copies of police reports on bad checks. They invited banks to submit records of bad checks they'd received. This information formed what Baer called a "positive" credit file. Instead of collecting data on all consumers, TeleCheck collected data on only the writers of bad checks. This distinction doesn't seem particularly interesting today, but TeleCheck really leaned into it, perhaps because consumer credit bureaus were both a growing business and a growing source of controversy in the mid-1960s. It probably served TeleCheck's interests to maintain some space between themselves and proto-Equifax organizations like the Hawaii Credit Bureau.

You might wonder about the business model; one of the advantages of checks is that they are relatively cheap to process. TeleCheck charged businesses a fee, at least initially set at 2%, but that wasn't just for the risk database. For the merchants, TeleCheck actually had a much more compelling offer than tracking check frauds. TeleCheck would guarantee each check they approved. If a merchant accepted a check on TeleCheck's advice, and the check bounced, TeleCheck would reimburse the merchant. In exchange, it asked for the bad check to be endorsed over to TeleCheck themselves.

Eating the cost of these bad checks could have been rough on TeleCheck's books, but they had their reasons. First, the reimbursements gave their customers a clear incentive to submit every bad check to TeleCheck. While TeleCheck marketing emphasized police and bank sources, it's clear that the primary source of their data was always their own customers.

You might realize that the guarantee service could create a new kind of fraud: a business might fabricate a bad check, or even knowingly accept one, and then let TeleCheck reimburse it. TeleCheck's insurance scheme was closely coupled to their credit bureau scheme. In other words, TeleCheck was able to control their risk on reimbursing bounced checks by making the decision of whether or not to accept the check at all. For businesses to claim reimbursement on a check, they had to prove that TeleCheck had agreed to guarantee it. They did that with an authorization number, a four-digit code provided over the phone that the cashier wrote on the back of the check before it was deposited.

Second, it was a business of its own: TeleCheck was a debt collector. And not just any debt collector, but one with the leverage of control over check acceptance at hundreds of businesses. TeleCheck presented this as a simple arrangement that does seem quite charming compared to the modern credit reporting industry: you could pass one bad check on TeleCheck's dime, but only one. Your identification remained in TeleCheck's database of unacceptable risks until you contacted them and made good on the original bounce. In other words, rip off TeleCheck and you'll never pay by check in this town again.

When businesses rejected checks, due to a negative TeleCheck response, they were instructed to provide the customer a "courtesy card" with an explanation of TeleCheck's operation, ways to contact them, and a reference number for the database entry that lead to the decline.

---

One of the interesting things about TeleCheck is its place in the history of check guarantee and its rapid growth. TeleCheck was not the first check guarantee service, Flagg personally knew of at least one other in New York City. For that reason, and likely others, TeleCheck had been given legal advice that they could not protect their business model by patents or other means. Flagg told the Honolulu Star-Advisor that "this means that we have to expand just as fast as possible before others get the same idea." And expand they did.

TeleCheck had barely started commercial operations, perhaps not started them at all, when they renamed from TeleCheck to TeleCheck International, signaling ambitions far beyond Hawaii. Existing operations were moved to TeleCheck Hawaii, a subsidiary, which would soon be joined by TeleCheck New York.

Today, checks seem an odd way to pay at retail because of the ubiquity and stronger security guarantees of debit and credit cards. In the 1960s, though, card payments were not widely available—if you weren't carrying cash, you paid by check. Checks were particularly problematic in the case of travelers, and that explains TeleCheck's Hawaiian origins. Checks are much easier to confidently accept when the bank, or even better the writer, are known to the merchant. That usually meant that personal checks had to be drawn on a local bank, at the very minimum, and initially even TeleCheck only guaranteed checks from Hawaiian banks.

But what, then, of tourists? Merchants would sometimes accept out-of-town checks with additional identification measures that ranged from copying down a driver's license to taking a photograph and thumbprint. Most just didn't, expecting visitors to obtain "travelers checks" issued by a well-known national bank and then usually cashed at another of that bank's own branches. Besides the inconvenience to the traveler, tourism economies like Honolulu's must have acutely felt the unwillingness of visitors to spend money when it involved multiple preparatory steps.

TeleCheck knew this going in, so expansion was inevitable. TeleCheck Hawaii and TeleCheck New York were set up as independent operations with their own computers and databases, but they were connected: bad check records from each were automatically transmitted to the other. As TeleCheck expanded, data sharing between regional operations built a distributed nationwide database, one that allowed a merchant in, say, Honolulu to accept a personal check from New York under full guarantee. If you asked Flagg, or Baer, all that was needed for complete nationwide acceptance of personal checks was a TeleCheck computer in every major city. Within a few years, TeleCheck International reorganized as TeleCheck Services, a franchise corporation. They started recruiting franchises in every state and, within a decade, Canada.

TeleCheck grew extremely rapidly, the kind of growth we might call a "unicorn" today. In 1969, TeleCheck estimated that their seven full-time operators took about 70,000 calls a month, 100,000 in the holiday shopping season. They guaranteed $6.75 million in purchases each month and paid out over a million a year in bad check reimbursements.

Check guarantees weren't everything, though. TeleCheck also diversified, expanding into just about every business they could think of until it started to seem comical. The same year that TeleCheck started, they acquired a company called Professional Services Inc. that did something we would now recognize as medical billing. It only took a couple of years for TeleCheck to dominate the Hawaiian medical and dental billing industry. In the words of one journalist, TeleCheck was hooked on computers, and hunted for any opportunity to make money off of the IBM 1440 and the larger machines that soon joined it.

Consider, for example, Match-Mate: the premier computerized matchmaking service of 1960s Hawaii. Lonely islanders filled out a questionnaire, conveniently distributed as a newspaper ad, and mailed it in with a payment of $7.00—by check, of course. The questionnaires were entered into TeleCheck computers and participants received a report with two likely matches and a booklet entitled Dining in the Islands. Considering that the book alone would "retail for $5.00 and has a full purchase price of $75.00" it was quite the deal for love. Well, maybe not, Match-Mate didn't last for long. It's interesting though that, alongside the address of TeleCheck International, the newspaper ads mentioned a CDC 3100.

The IBM 1440 was something of a budget computer, intended as a lower-end alternative to the "flagship" IBM 1401 mainframe for the many small businesses and accounting firms that couldn't afford a 1401. Within a couple of years, TeleCheck appeared in a directory of computer services provider as a consulting, accounting, payroll, etc. data processing firm with a Honeywell 200, a semi-clone of the IBM 1401 that could mostly run the same software, so they had apparently upgraded. Then, in 1966, Match-Mate associates TeleCheck with the CDC 3100. The 3100 was the runt of the CDC 3000 family but still ran about $120,000, over a million dollars today. Once again, all of the machines were likely rented, but still... in its first two years, TeleCheck acquired more computers than most established businesses would over five.

Some of TeleCheck's side ventures were quite logical. They had accounting and payroll businesses, which naturally fit the transaction processing skillset they had built for check guarantees. Consumer credit cards emerged during the 1960s, and TeleCheck was enthusiastic about those too. I don't think the scale of this operation was ever that large, but TeleCheck apparently handled online verification for some retailer credit cards in Hawaii, quite possibly by treating them as a special kind of check (a trick that TeleCheck would use repeatedly over the years to offer new services over existing equipment). Once again leading me to suspect that Flagg was doing some kind of engineering at MIT and in the Navy, TeleCheck's software situation seemed sophisticated for the 1960s. Newspaper articles describe real-time multitasking between batch processing of medical billing folios and online check inquiries. In a couple of years, they threw some sort of telephone order business and construction supply catalog into the mix.

But TeleCheck didn't keep to its computer roots. By the end of the 1960s, TeleCheck owned Honolulu Business College and Cannon's College of Commerce. They owned Minneapolis-based Boatel, manufacturer of houseboats and snowmobiles (Flagg seemed to have moved to Minneapolis at some point along the way). TeleCheck's Marine Science Division, made up mostly of subsidiary Pacific Submersibles, operated a Perry PC5C research submarine for which they were building a custom robotic manipulator. In 1968, TeleCheck Hawaii announced a complete rebranding to Data-Pac, a name that would better reflect their diversified interests. The franchise parent, TeleCheck International, kept the TeleCheck name.

In 1972, TeleCheck International went bankrupt.

---

The story of Harry Flagg is a complicated one, and I do not think that I have all of the information. There are just certain, you know, oddities. In the mid-'60s, Flagg was repeatedly lauded as the founder of TeleCheck. Robert Baer seems to have been around from the beginning, but it's not until later on, in the '70s and '80s, that he is widely referred to as TeleCheck's founder. Flagg is conspicuously absent from these versions of the company's history.

Similarly, the 1972 bankruptcy, triggered by the parent of a company TeleCheck had acquired calling the loans it granted to facilitate the acquisition, left little paper trail. Well, it was a bankruptcy, so there's a voluminous docket of the legal and financial details, but TeleCheck's leadership and their thinking are now opaque. We do know this: after the 1972 bankruptcy, Flagg was no longer in charge of TeleCheck International.

In 2005, a court ruled for the FTC in its case against Trek Alliance and its founders, including Harry Flagg and his son Kale Flagg. Flagg had moved to Arizona and founded Trek Alliance in 1997, a vague company with a confusing set of subsidiaries that included some sort of sales training. Primarily, though, Trek—which is unrelated to the better-known bicycle manufacturer—was a pyramid scheme. At least, that's what the court ruled. According to the FTC's complaint, Trek's "Independent Business Owners" sold water filters, cleaning products, nutritional supplements, and beauty aids. Trek's compensation plan, "one of the most lucrative in the industry," included a series of Bonuses and a 22-level Pay Plan assigned according to dollar volume of an Independent Business Owner's "downline."

This is, of course, the gold standard of business excellence in modern Utah, but Flagg was in Arizona and the 2000s. He and the other parties settled, denying fault but agreeing to shut down Trek. Together with their insurance company, they paid millions in restitution and suffered a permanent injunction against involvement in any multi-level marketing schemes.

Honestly, I'm not inclined to view Flagg as a fraudster, although it would be deliciously ironic considering where he started. I think he was just a little too ambitious and not quite cautious enough, entangling himself in everything that sounded like a good idea until it was just too many things to keep up. Still, how remarkable it is that the creator of the nation's most successful anti-check-fraud scheme would become separated from it, only to later be caught cashing checks from the top of a pyramid scheme. Now that's vision.

---

Despite TeleCheck's over-expansion and leadership troubles, the company was unstoppable. Baer became president of TeleCheck Hawaii while his son, Jeffery Baer, moved to Denver and established the headquarters of TeleCheck Services Inc., the new parent company of the TeleCheck franchise system. Those franchises multiplied: here in the Southwest, TeleCheck Texas was founded in 1982 and rebranded to TeleCheck Southwest two years later, when it bought the franchise rights for New Mexico, Oklahoma, and Arkansas. TeleCheck Texas had processed $325 million in checks in 1983. Elsewhere, there were TeleCheck franchises operating in more than half of the US states, several Canadian provinces, and Hong Kong.

Along with expansion came technical improvements. TeleCheck Hawaii, perhaps because of its origin as the first TeleCheck franchise, was always independently minded and eventually left the franchise system to go it alone as Uni-Check. Before they left, the introduced the first interactive voice response (IVR) check verification system. The fully-automated, touch-tone based IVR system expanded to other TeleCheck franchises, who named the automated voice "Samantha." TeleCheck liked it so much that they bought the developer, a company called Real-Share.

TeleCheck had other ideas, as well. Perhaps inspired by Ma Bell's "dataphone" efforts, TeleCheck launched the first point of sale electronic payment terminal I know of, if you are generous about the definition ². The first-generation TeleCheck Terminal, introduced 1980, was a protrusion of the front of a standard pyramid phone that read magnetic stripes and sent them over the phone line. Instead of typing the digits from a check and driver's license on their phone keypads, merchants could call into TeleCheck and just swipe a card. Of course, this only worked for cards, drivers licenses and credit cards that TeleCheck processed, but it was a hit nonetheless.

Nationwide expansion of the TeleCheck service necessarily entailed nationwide expansion of the TeleCheck network. With each franchise operating independent computers that shared records with the other, TeleCheck was a sophisticated, networked operation by the standards of the time. As it turned out, they had help, from one of the most advanced computer networking companies. In 1980, TeleCheck Services was acquired by Tymshare.

Tymshare is, to me, one of the signature brands of the era of Business Computing. As the name suggests, twee spelling and all ³, Tymshare started out as a pay-by-the-minute time-sharing provider. Founded in the same year as TeleCheck by two ex-General Electric Computer employees, Tymshare grew out of UC Berkeley by selling time on an SDS 940 computer (initially borrowed from UC, later rented themselves) running the Berkeley Timesharing System. The combination of the 940 computer (itself mostly designed by UC Berkeley), the BTS operating system, and the dial-in timesharing model made Tymshare one of the most affordable routes to serious computing. The company was tremendously successful, but time-sharing was a short lived industry. Computers were getting faster, smaller, and cheaper every year, so the set of customers that needed a computer but could not afford their own got smaller and smaller. Tymshare probably saw that coming, because like early TeleCheck, they aggressively diversified into just about anything that a computer could do—including check guarantees.

As Tymshare grew, they purchased more computers, and larger. In the late '60s, they were operating dozens of machines running a largely custom operating system. They had economized on many of their acquisitions by running the acquired software on their time-sharing fleet, which was efficient but challenging for applications like TeleCheck that were designed around a central computer (for each franchise). Tymshare's business wasn't as simple as connecting a user to their nearest computer; they needed to accept dial-in calls from around the nation and then connect them to whichever computer ran the requested application, potentially on the other side of the country.

The solution, remarkably prescient of later wide-area networks, was a cutting-edge architecture of Varian 620 minicomputers that controlled banks of telephone modems and forwarded traffic to a set of SDS 940 computers called "supervisors." The role of the "supervisor" was very much what we would call a "router" today: they packetized data from the Varians and then routed it to other 940s according to a virtual circuit switching scheme. While the system initially connected dial-in users to 940s, it was readily extended to building arbitrary circuits between any of the serial interfaces of the Varian "edge nodes."

A business that wanted to offer a dial-in service to a wide area could save a tremendous amount of money on phone lines and modem banks by instead purchasing a small number of leased lines to a Tymshare data center. Their users could then call any of the Tymshare access phone numbers, where they would receive a command prompt from the answering Varian. When they typed the keyword assigned to the interconnected computer system, Tymshare's network built a circuit from a modem in one data center to a modem in another, connecting the caller to the customer's computer across their own nationwide network.

In 1976, Tymshare separated their network infrastructure into a separate company, Tymnet, which registered as a telecommunications carrier. Tymnet would ultimately outlive Tymshare itself, becoming the "industrial internet" before the contemporary buzzword or, really, the internet that spawned it. Despite some technical challenges originating from Tymnet's proto-internet architecture, everything from credit card transactions to supply chain notifications to consumer dial-up ISP traffic ran over Tymnet for decades after. Tymnet provided modem bank services to AOL, for example, and some of the vintage 1970s Tymshare dial-in numbers are still in service with various contract modem providers.

---

After its Tymshare acquisition, TeleCheck had a nationwide computer network, significant computer capacity, and an appetite for technical sophistication. It was a troubling time for a check guarantee firm, though. A new technology called Electronic Funds Transfer, or EFT, let retailers withdraw money directly from customer's accounts using only their debit cards. TeleCheck and Tymshare had actually found some business processing these transactions, but for the most part it was a separate industry that competed with checks. Baer cautioned that there was no reason to panic, as consumers would continue to use checks for many years to come, but there were clearly other things underway at TeleCheck. They were building their own transaction processing network.

I started thinking about TeleCheck because I was pumping gas and looking around for anything to distract me from how much it costs these days. Attached to the fuel dispenser, in between a half dozen other mandatory regulatory notices, was a sticker with the bright red and white TeleCheck logo. Rather than the "Your Check Is Welcome Here" verbiage used by early TeleCheck decals (back when many retailers did not yet accept personal checks from unknown customers), this one had decidedly less interesting text: "When you provide a check as payment, you authorize us either to use information from your check to make a one-time electronic fund transfer from your account or to process the payment as a check transaction."

In 1984, Tymshare, and TeleCheck with it, were sold to McDonnell-Douglas. McD-D, as I like to call them, was a formidable aerospace and defense contractor that was feeling an acute need to diversify as "peace broke out." They are also known, perhaps mostly due to their 1997 merger with Boeing and its effects on that company, as a bit of a backwater for actual engineering. They didn't hold on to Tymshare for very long, just a few years to give TeleCheck the curious property of a check guarantee service backed by F-15s.

Around the same time they were acquired, TeleCheck introduced a next-generation payment terminal that incorporated an MICR check reader along with support for newly standardized credit cards. This terminal followed the same basic model, of calling in via Tymnet and then sending the card or check contents over the phone line. This method of handling credit cards proved short lived as the industry reorganized and security requirements became far stricter, but it got TeleCheck equipment into a huge number of retailers. In 1989, McD-D, unsatisfied with the finance industry and perhaps ginning up the Gulf War, sold their entire software division. TeleCheck's increasing success as a general payments processor no doubt helped attract the buyer: First Data.

First Data probably qualifies as obscure, as they have few consumer-facing options, but they're a giant in payment processing. First Data provided the original infrastructure for both Visa and MasterCard before becoming part of American Express, who continued to operate the company as a general financial information systems company until they spun it out again. Back in the early '90s, though, TeleCheck found itself as a subsidiary of a company that also processed credit and debit transactions. It was only natural to unify those business lines into one, which TeleCheck called Electronic Check Acceptance.

Picture with me, in your mind's eye, the Verifone TRANZ 330. You have no idea what I'm talking about, of course, but if you're about my age or older you would recognize one. The Tranz 330 was the first widely successful credit card payments terminal, and anchors the Verifone brand name as a manufacturer of devices that verified payments over the phone. In practice, the TRANZ 330's main purpose was to read the data from a credit card, accept a keyed-in dollar amount, and then connect (usually over Tymnet) to a backend computer to authorize the transaction.

It could do much the same for checks: one of the features of the TRANZ 330 was check authorization, designed specifically for TeleCheck. The cashier could press a key to select check authorization, follow prompts to enter the check information and swipe the writer's driver's license, and then get back an authorization code (or a decline) on the terminal's screen.

The TRANZ 330 represented a milestone in two ways. It was the first payment terminal with TeleCheck support that resembled modern payment terminals in function. Earlier TeleCheck terminals were primarily phones with a payment card reader attached to them, the TRANZ 330 was not a phone at all and managed modem calls transparently to the user. Second, it represented a shift from TeleCheck providing a complete end-to-end solution to TeleCheck as one of a number of services supported by a common payment frontend.

After the First Data acquisition, TeleCheck was further integrated into other payment equipment, but it won back the branding. The TeleCheck Accelera and TeleCheck Eclipse, both manufactured by Verifone but bearing the TeleCheck logo, looked very much like every other credit card terminal but with the addition of an MICR check reader and slip printer. The added hardware allowed the terminals to read the check automatically, and also to print the authorization code on the back.

When these devices were introduced, the expectation was that a merchant would use the terminal to "authorize" a check (getting a TeleCheck guarantee on it), and then separately deposit the check with their bank. This wasn't all that different from how credit card transactions were handled at the time, with authorization usually done in real time (if at all) and "capture" (the actual payment) submitted as part of a batch at the end of the day. There was still a lot of paper handling involved, though, and during the 1990s it was clear that shipping slips of paper around the country was not a suitable long-term plan for checking.

Since the 1980s, a system called the Automated Clearing House (ACH) had been brewing among various bank coalitions and, later, the Federal Reserve Bank and an organization called NACHA: the National Automated Clearing House Association. ACH was one of the first standardized forms of computer payment, basically just a specification for a text file that contained the basic information for check-like transactions. Instead of exchanging paper checks, banks uploaded a text file to the ACH and then downloaded another text file later. Those files represented transactions in and out, processed in the banks and the ACH platform as a once-daily batch. ACH caught on for many of the same purposes that checks had fulfilled, like payroll (commonly called "direct deposit") and payments to savvy billers that wanted to avoid card payment fees (commonly called "e-check").

Since an ACH transaction is pretty much just a line in a text file with the same numbers you would find on a check, it is superficially possible to take someone's check, copy down the numbers, and then enter it as an ACH transaction. In actual practice this wasn't allowed—until 2000. That year, NACHA adopted a provisional set of rules for "point of purchase entry," the on-the-fly creation of ACH transactions from a point of sale system. In the real world of retail, where cashiers are not excited to ask for someone's bank account and routing numbers, wait for the consumer to figure them out, and then try to key them in correctly, this pretty much universally turned into "check conversion."

The idea of check conversion is pretty simple: you "pay by check," but when the cashier takes the check from you, they actually put it into a terminal that reads the check information and uses it to create a technically unrelated transaction in an ACH batch. Most retailers use slip printers that add some tracking information (like a transaction number) to the back of the check, and stamp it "void" with a message that it has been "converted" into an ACH transaction. Some retailers would even return the voided check to the consumer as a sort of "receipt" of the ACH transaction, although I don't believe this practice is still common.

To the average person, there is hardly any difference: you write a number on a check, sign it, the money eventually disappears from your account. Since there are differences in the legalities of check and ACH processing, though, businesses are required to disclose that they convert checks to ACH. The main difference, in practice, is that ACH transactions tend to clear more quickly than checks. That does cause occasional problems for consumers who are "kiting," writing checks that they do not yet have the money to cover, since they may be counting on the slower process of exchanging slips of paper. In our modern world, traditional processing of checks has been completely eliminated in favor of image-based processing, which is something like ACH conversion built into the check clearing process itself. That means that the actual difference in processing time between ACH and checks is no longer as significant (although the availability of same-day ACH potentially complicates this, I do not believe that NACHA rules allow check conversions to be opted in for same-day clearing). The whole story of check conversion has become mostly a forgotten detail of the beautiful tapestry of transaction processing.

Naturally, TeleCheck integrated ACH conversion into their product. Many businesses now handle checks via payment terminals that perform a TeleCheck authorization and ACH conversion in one step, all facilitated by TeleCheck for a fee that is a bit lower than an equivalent payment card transaction. The function of writing TeleCheck authorization numbers is integrated into the slip printer, which used to endorse checks but now decorates them with ACH conversion details.

---

In 2019, First Data was acquired by financial technology giant Fiserv. Fiserv continues to operate TeleCheck to this day, but with ACH conversion now a commodity service and retail of personal checks so standard that we are forgetting about it, TeleCheck has started to look less like an interesting technology company and more like every other credit bureau.

Here's one of the reasons I find TeleCheck so interesting: search for them. I mean, just type it into Google or whatever. What do you see?

A very minimal corporate website, curiously at "getassistance.telecheck.com" (the bare "telecheck.com" redirects to the subdomain as well), with zero information except for law enforcement contact info, a form to look up declined checks, and a set of mandatory regulatory notices. "Victims of Human Trafficking" is a top-level navigation item, peeking out from above the hero banner of typing hands.

TeleCheck is now a ghost, a specter of financial technology past. Baer's 1980s predictions about EFT not cutting into TeleCheck's business fared well only if you ignore the closely related phenomenon of credit card networks. Nationwide, check volume, especially at retail, has collapsed to almost nothing. Fiserv continues to operate TeleCheck as, essentially, a legacy cash cow. They don't market the service at all, and maintain only the brand presence that is legally mandated of credit bureaus.

TeleCheck has a twisting, confusing corporate history, and besides Flagg's larger-than-life ambitions, credit reporting and debt collection are the reasons why. Consumer credit bureaus started to get a bad rap in the 1960s and have never recovered, they must be among the most hated brands in America today. Debt collectors have never had many friends. TeleCheck is, in various ways, both of those things, and they are now more important functions than technicalities of handling checks.

TeleCheck has been the target of dozens of lawsuits under the Fair Credit Reporting Act. To be fair, I don't think that they're worse than usual in this regard. One of the major implications of the FCRA is that credit bureaus can be liable for having incorrect information about consumers, but prior to the passage of the FCRA most credit bureaus were, shall we say, fast and loose about details.

Let's consider an example, of TeleCheck's practice of linked identities. If a person is writing bad checks serially, they will probably not write bad checks from the same account every time. So, as part of the "30 million facts" that TeleCheck's 1980s ads claimed their computers contained, TeleCheck saved relationships between identifiers. If you presented a driver's license and a check from one account, and then a month later at another store presented a driver's license and a check from a different account, TeleCheck permanently recorded the association between all three.

Say the first check bounced. If someone else, months later, at a different store, presented a check from the second account, TeleCheck would decline it. Why? Because they followed the links, from the second bank account to your driver's license to the first account. We might recognize this as taint analysis, and TeleCheck would follow connections through multiple steps until a bad check written by one person could result in "Code 4" declines of checks on different accounts by different people. To say that this was controversial with affected consumers understates the problems, and the way that account linking worked (especially between people of whom TeleCheck otherwise had no evidence of a relationship) became a legal morass. Several of TeleCheck's franchises seem to have left the TeleCheck brand in an effort to manage their risk, especially considering state-specific regulations on credit reporting.

As another way to manage regulatory complexity and liability, TeleCheck spun out its debt collection function into an independent company (although also owned by First Data) called TRS. Or TRS Recovery Services. Here's the thing, I am 90% sure that TRS stands for TeleCheck Recovery Services, but their own website says "TRS Recovery Services (TRS)", which would imply TeleCheck Recovery Services Recovery Services. I think the intent of the whole thing was to divorce the TRS brand from TeleCheck for reputational reasons (consider TRS has a totally different logo), which lead to some "TRS doesn't stand for anything" nonsense. TRS has a website that is almost but not quite identical to TeleCheck, with mandatory regulatory notices only, and they are universally hated as the people that hound you for life over a bounced check at Walmart.

Within the story of TeleCheck we see the full arc of payments technology: 1960s idealism at a new world in which everyone's checks are welcome, 2020s cynicism with an ailing conglomerate interested mostly in not losing lawsuits. TeleCheck is completely unexciting, the cartoon opposite of innovation, but it is very much still with it. Did you know that you can pay at amazon.com via direct ACH withdrawal from your checking account? Mass retailers are still surprisingly likely to accept checks as payment, and they are still, for the most part, doing that via TeleCheck. Even small businesses don't have to miss out: Fiserv also owns Clover, and Clover integrates TeleCheck electronic acceptance.

Deep inside Amazon's help system, under Payment Issues, an article explains how to "Correct a Failed Checking Account Authorization." Besides making sure you typed your account number correctly, the advice is: Call TRS Recovery Solutions. Sometime, somewhere, you must have written a bad check. A nationwide network of computers took note. Honolulu businessmen hobnobbed at a Boy Scout council meeting. TeleCheck took phone calls, McDonnell-Douglas forever changed Boeing, Harry Flagg went on to running MLMs. "Many Hawaii organizations currently are buying time on comparable computer facilities on the Mainland," Flagg said, when they bought the CDC 3100, the first in Hawaii. "Our installation will save them time and money." It might find them a date, too. The Nai'a explores the ocean, two small business colleges fold, two companies from Hawaii make competing claims about an obscure part of history. You're at WalMart, the total is $2, and the cashier is saying something about "Code 4." These things are all connected. They are all connected by Tymnet.

I tend to focus on the origin of the computer within the military. Particularly in the early days of digital computing, the military was a key customer, and fundamental concepts of modern computing arose in universities and laboratories serving military contracts. Of course, the war would not last forever, and computing had applications in so many other fields—fields that, nonetheless, started out as beneficiaries of military largesse.

Consider education. The Second World War had a profound impact on higher education in the US. The GI bill made college newly affordable to veterans, who in the 1950s made up a large portion of the population. That was only the tip of the iceberg, though: military planners perceived the allied victory as a result of technical and industrial excellence. Many of the most decisive innovations of the war—radar and radionavigation, scientific management and operations research, nuclear weapons—had originated in academic research laboratories at the nation's most prestigious universities. Many of those universities, MIT, Stanford, University of California, created subsidiaries and spinoffs that act as major defense contractors to this day.

Educational institutions bent themselves, to some degree, to the needs of the military. The relationship was not at all one-sided. Besides direct funding for defense-oriented research, in the runup to the Cold War the military started to shower money on education itself. Research contracts from uniformed services and grant programs from the young DoD supported all kinds of educational programs. For the military, there were two general goals: first, it was assumed that R&D in civilian education would lead to findings that directly improved the military's own educational system. Weapons and tactics of war were increasingly technical, even computer controlled, and the military was acutely aware that training a large number of 18-year-old enlistees to operate complex equipment according to tactical doctrine under pressure was, well, to call it a challenge would be an understatement.

Second, the nation's ranks of academics made up something like a military auxiliary. The Civil Air Patrol built up a base of trained pilots, in case there was ever a need to quickly expand the Air Force. By the same logic, university programs in management, sciences, and education itself produced a corps of well-educated people who would form the staff of the next era of secret military laboratories. Well, that's not exactly how it turned out, with the Cold War's radical turn to privatization, but it was an idea, anyway.

That spirit of military-academic collaboration is how a group of researchers, mostly physicists, at the University of Illinois found themselves with military funding to develop a system called "Programmed Logic for Automatic Teaching Operations," or PLATO. With its origins in the late 1950s, and heyday in the 1970s, PLATO is usually considered the first effort in computerized teaching. It's a fascinating sibling to other large-scale computer systems of the time, like those in air traffic control. There are many similarities: PLATO struggled with connecting terminals and computers over a large area, before "the Internet" was even an idea. It had to display graphics, a very primitive computer capability at the time but one that was thought to be vital for classroom demonstrations. The system supported many simultaneous users, and had to process data in real-time to synchronize their various workspaces.

There were also important differences. Unlike SAGE and the 9020, unlike business accounting and tabulation systems, unlike almost every computer application yet devised, PLATO was designed for user-facilitated content.

Reflecting its origins among academic physicists, PLATO heavily emphasized collaboration. Many of the earlier, 1960s-era PLATO developments focused on simplifying the development of learning modules so that teachers could create interactive PLATO courses with less specialized computer training. By the 1970s, as PLATO terminals were increasingly installed in schools and other institutions in Illinois, the emphasis on collaboration turned towards communication. If learning modules were easy for teachers to develop, the students should also be able to use the system to create their own coursework and study materials. Researchers and other academic users had a similar desire for a computer system where they could keep notes, write reports, and stay in touch with their colleagues.

PLATO did not prove an enduring success—despite a decades-long effort toward commercialization, it was expensive and the actual benefits of computer-aided teaching remained unproven¹. Few PLATO systems ever escaped the University of Illinois and its network of satellite delivery locations. Follow-on projects fizzled out and, despite PLATO's incredible ascent from a 1960 concept to an elaborate 1970s multi-user interactive system, PLATO spent the 1980s in decline. No one thinks about PLATO very much any more, which is unfortunate, because it is one of those remarkable, isolated moments in history, a sort of conceptual singularity, in which a project with limited real success incubates so many concepts that it sets the course of history afterwards.

When you look at our modern computers, smartphones and social media and Farmville and so on, it's hard to find a single thing that isn't somehow derived from PLATO. Through PLATO's 1970s NSF funding and resulting interaction with other NSF efforts, it is my opinion that PLATO is probably a more important precursor to the modern internet than ARPANET and NSFNET. Not so much in a technical way; PLATO was closely tied to a mainframe-terminal architecture that would not have likely lead to our flexible packet-routing internet architecture. Rather, in a vibes way. PLATO was a large-area, networked computer system that emphasized posting things and looking at things other people had posted. It offered math lessons, but it also offered games.

Perhaps most importantly, it had notes.

PLATO's developers at the Computer-Based Education Research Lab of the University of Illinois had long had a simple way of communicating with each other, by editing a series of lessons titled "notes1" through "notes19." While functional, it was imperfect: Google Wave had not yet appeared to tell us that everyone being able to edit everything is good actually, so the complete lack of access controls, or formatting, or really organization of any kind was making the "notes" lessons a headache as the system gained users. In 1972, the PLATO IV terminal, new funding, and new backend computers had PLATO growing fast. There was an obvious need for a better version of the notes lessons.

Through the machinations of academia, the task of building a better "notes" file fell on two high-school students, on their summer break from the University of Illinois-affiliated laboratory high school. Using PLATO's native TUTOR programming language and facilities intended for exams and course discussions, Dave Woolley and Kim Mast wrote a lesson called "=notes=". The =notes= lesson was originally intended for system announcements, trouble reports, and communications between PLATO's operators. Soon, though, it was doing far more.

PLATO's notes were not the first implementation of a computer-based discussion board. Similar capabilities had been implemented by ARPANET users at least a couple of years earlier. It was also not the first email system, and indeed, was not email at all: notes only offered public posts. PLATO didn't gain a private messaging capability at all until a year later. The notes lesson wasn't even the only message board on PLATO, although it was the most sophisticated.

What stood out about notes is that it was popular. Everyone used it. By 1976, the notes lesson had evolved into a generalized application that allowed any PLATO user to create and manage their own notes file. That management included access controls and capabilities we might now call moderation. It was one of the most popular applications on all of PLATO, and that was against the competition of the several notable early video games created there.

Brian Dear, author of "The Friendly Orange Glow," a book on PLATO, argues that the notes lesson's peculiar history as a public messaging system that came before a private one had a critical impact on PLATO's users and culture. Communications on PLATO were "public-first." While a "private notes" feature was added later, users were already in the habit of doing things in the open. This sense of community, of close collaboration with some and passive awareness of others, must be a cultural precursor to the BBSs of the early internet and the social networks of today.

But that's not even what I'm here to talk about. During the late '70s, there was something else going on at the University of Illinois: future Microsoft CTO Ray Ozzie was working on his undergraduate in CS, a large part of which involved PLATO. After his graduation in 1979, he worked for Data General, manufacturer of a popular series of minicomputers, where he reported to Jonathan Sachs. After Data General, he did a stint at Software Arts, the development firm behind VisiCalc. He must have stayed in touch with Jonathan Sachs, though, who in 1982 had left Data General to found his own company: Lotus Development.

Lotus is widely remembered for Lotus 1-2-3, the hit spreadsheet application that displaced VisiCalc as possibly the most important software package in all of personal computing. Lotus's two founders were Sachs and Mitchell Kapor, both of whom had connections to Data General and Software Arts, in an era in which blockbuster software products often came from companies founded by a few entrepreneurial employees of the incumbent that was on its way out. And, indeed, they stuck to the familiar recruiting strategy: Ray Ozzie, of similar employment background, joined Lotus in the early '80s as a developer on their complete office suite, Lotus SmartSuite ².

Lotus 1-2-3 was one of the most successful software products ever, but the frontier of office computing was quickly expanding to word processing and presentations. Far from our modern monoculture of Google Docs and Microsoft Office still kind of hanging on I guess, the productivity software market was extremely competitive in the 1980s and just about every Independent Software Vendor had some kind of productivity or office suite under development. The vast majority of these were commercial failures, and Lotus SmartSuite was no different ³. It is fascinating to consider that SmartSuite included an (apparently mediocre) desktop database/rapid application development tool called FORMS. Desktop databases were once considered table stakes for productivity suites, but are now almost completely forgotten, recast as expensive and standalone SaaS offerings like AirTable. But I digress...

By 1985, Lotus had acquired VisiCorp and consolidated its dominance in PC spreadsheets. Despite its overwhelming success in spreadsheets, Lotus struggled elsewhere: not just SmartSuite and Jazz but standalone word processor Manuscript and modeling-oriented spreadsheet Improv were often technically impressive but were also consistently poor sellers. I think that part of the problem is that the people at Lotus were a little too smart. This is best exemplified by their unsuccessful "personal information management" or PIM product (this is the same general category as Microsoft Outlook and Mozilla Thunderbird, although historically PIMs were less email-centric and more focused on calendars and contacts).

Lotus Agenda was marketed as a PIM, but unlike other contenders of the era, it came without a fixed schema into which the user inserted data. Instead, it behaved more like a very simple desktop database, with the user entering data into spreadsheet-like tables however they wanted and then defining views based on column and row filters. The result was highly generalized, able to fit just about any use case with sufficient time and effort. It also did very little to help: you started with a blank grid, left to your own devices. It reminds me of modern PIM-adjacent offerings like Emacs org mode or Obsidian that attract a fervent following of dedicated users who ascribe some sort of life-changing experience to them, while the other 99% of us launch the software and then wonder what we are supposed to do. Lotus Agenda, for its part, reportedly had a similar cult fame.

I imagine that Lotus's difficulty entering new markets must have encouraged them to get creative. They had quite a few products, many of them the subject of awards or technical papers or patents, no lack of engineering capability. What they seemed short on was vision and marketing. They had not quite figured out what customers wanted, or at least what customers wanted that was different from what their competitors already offered. I suppose they were willing to take a bet, as long as the risk could be adequately controlled.

In 1984, Ray Ozzie left Lotus to found a new company called Iris Associates. Like Software Arts developer-publisher relationship with VisiCorp, Iris Associates was an independent company but was contractually obligated to offer exclusive publishing rights on its products to Lotus Development. In exchange, the first few years of Iris's operation were funded by a substantial investment from Lotus. Ozzie, now in control of his own kingdom, quickly hired three other University of Illinois alums—people he knew from his time working on PLATO.

It's not clear to me if this was the original goal of Iris Associates or if it became the goal as a result of their experience with PLATO. However it happened, Iris Associates spent about the next five years building a version of PLATO's =notes= lesson that could run on a network of Windows PCs. Through their publishing arrangement with Lotus, this would come to be known as Lotus Notes.

Lotus Notes is one of the most famous, or infamous, examples of "groupware." Groupware is a hard concept to put your finger on, in good part because of its history. The whole thing, as a category, is easily traceable to Douglas Engelbart's work on Human Augmentation at SRI—work that, like PLATO, failed to gain market adoption but was nonetheless tremendously influential on the larger art. Human augmentation is the core idea in groupware, also called computer-supported cooperative work (CSCW). These were lofty academic ideas around the ways that computers could augment the social processes behind most productive work; they were ideas that grew and forked like branches from the middle of the 20th century to today. Many of those branches withered and died, others flourished before some sort of rotting disease set in (looking at you, Sharepoint). The result is that "groupware" meant different things at different times, and today is so non-specific that everything from enterprise business process automation platforms to slightly dressed-up webmail are lumped together.

In the days of Iris Associates, groupware meant something like this: software that assisted humans in communicating, collaborating, and tracking and executing processes. In practical terms, this meant core PIM applications like email and calendaring, but also included collaborative editing, workflow automation, document and policy management, and all kinds of other core business needs that involved more than one person. It's hard to apply a definition of groupware to Lotus Notes because Lotus Notes was one of the major products that originally defined the category, so its capabilities and limitations have become part of the background.

Lotus was under development from roughly 1984 (founding of Iris) to 1989 (first commercial release). The time period also put Lotus Notes in another category, although this one even more tenuous: network operating systems. Novell NetWare, for example, hit the scene in 1983. Network operating systems revolved mostly around file and printer sharing, but most gained some form of email. Lotus Notes had substantial overlap with network operating system features, but ran on Windows, making it an important step in the decline of fully integrated network products like NetWare in favor of application software on Windows which, more and more with each Windows release, used underlying operating system capabilities instead of implementing a complete application-specific network stack. Novell would go in a similar direction, with GroupWise, part of a wild industry restructuring during the 1990s.

It's hard to put a finger on what exactly Lotus Notes was from a modern perspective because the software itself was highly generalized. Wikipedia puts it like this: "Notes and Domino is a cross-platform, distributed document-oriented NoSQL database and messaging framework and rapid application development environment that includes pre-built applications like email, calendar, etc." Let's set aside for the moment the matter of Domino, which we have not introduced, and focus on the parts like "distributed document-oriented NoSQL database" and "messaging framework" which are strange ways to describe a groupware package but actually a very logical way to describe a rapid application development product—mentioned here as just one of the things that Lotus Notes is. I suppose that's a bit of synecdoche.

Like Lotus Agenda before it, Lotus Notes was something of a blank canvas that could be configured, customized, and designed to meet virtually any need. Unlike Lotus Agenda, they had learned a lesson about onboarding and Lotus Notes came out of the box with a set of familiar groupware features. What set Notes apart from other implementations of email, for example, is that Notes email was just another set of views and logic built on a common database. It was a custom application like the others, just one already provided as a sample.

Much of the strangeness of Lotus Notes reflects its origins in PLATO. Most of the microcomputer network products of the 1980s were built around peer-to-peer networking, the idea that multiple small computers could communicate with each other directly (NetWare, for example, used a peer-to-peer architecture despite the fact that certain machines were usually clearly logical servers). That was the hot new thing in the '80s, but PLATO was not of the '80s, it was of the '60s. PLATO wasn't even really a client-server architecture: it was a terminal-computer architecture, in which individual PLATO terminals interacted with one of multiple (mostly DEC) mainframe or midsize computers that ran the actual logic.

PLATO wasn't just multi-user, though, it was multi-computer. To form one integrated system out of multiple independent sets of computers and terminals, PLATO replicated all of the user data between the computers. Lotus Notes inherited the same approach: data was stored in a database, and communicated between users by replicating that database between machines.

Everything in Lotus Notes is a note, and a database is a collection of notes that are identified by unique IDs. When a note is updated, that note is replicated to other copies of the database. Over time, the obvious performance and reliability problems with this architecture became apparent and the replication process became a lot more sophisticated, but it always worked on this simple logical model of updating the local database and then replicating it to others.

And that was about it—the database full of notes was all there ever was to Lotus Notes. That and an incredible amount of functionality exposed to the user by the fact that notes could incorporate simple scripts, arbitrary programs, workflow rules, and entire GUI layouts. Lotus Notes did not differentiate between "user data" and "program data" at a low level, and indeed users could edit views, write scripts, and build entire applications on top of Lotus Notes that replicated to other users just like an email.

The kind of freedom that Lotus Notes gave its users was incredibly empowering, but also a bit of a nightmare for usability. I am reminded of the early object-oriented work, like Smalltalk, much of which came from a similar milieu of SRI and Xerox-inspired applied computing research. If everything is a note, then you can use the tools for manipulating notes to create anything. The Lotus Notes data model was extremely flexible, about as close to completely schema-less as practical, but it still offered indexing and full text search. There wasn't much that you couldn't shove into Lotus Notes if you tried, and for businesses that adopted it wholesale, it could expand in scope until it took over everything. Not just email and calendars but enterprise resource planning!

Early versions of Lotus Notes, before the IBM acquisition, allowed for programming using either an expression language very similar to Lotus 1-2-3 spreadsheet formulas (and called, sensibly, Formula) or an imperative language very similar to Visual Basic (called LotusScript). Using a tool called Designer, it was possible to visually edit GUI interfaces like forms and reports. These were backed by a Notes-specific query language, although later versions of Notes also gained an SQL compatibility layer and some options for interoperating directly with relational databases.

Because of the example of PLATO, Notes was architected around a client-server model. The client, generally called Lotus Notes, partially replicated the database to allow the user to view and edit notes. The server, which came to be called Domino, kept a complete copy of the database and executed enough of its logic to handle access controls. Like PLATO's replicated model, though, it was not expected that there be a single, central server. Many Domino servers could share the same database, and replicated changes among themselves. This architecture was very convenient for the era's business deployments, where each office location would have a local area network but network connections between sites were expensive and comparatively slow. Domino servers had the effect of consolidating user activity into a reduced volume of replication traffic, and when inter-office network links were lost the office still functioned normally, albeit as an island.

Despite its origins as a product explicitly for Windows, Notes maintained a degree of cross-platform capability. Domino was available for several platforms and the Lotus Notes client was available for pretty much anything. The simplicity of the Formula and LotusScript languages did a lot to simplify porting, although the system itself was originally written in C++. That all started to change when Lotus rolled in Iris and shortly after, in 1995, merged into IBM.

Lotus Notes is hard to write about because of the naming. The curious decision to call the server (which was, originally, called Lotus Notes Server) Domino already complicates the situation, and that itself appears to be a result of IBM's unique product vision. The IBM renaming churn went on for the whole span that Lotus operated under IBM, meaning that whether you call it "Notes" or "Lotus Notes" or "IBM Notes" or even refer to the entire thing as "Domino," you're probably correct for some point in the history. Modern IBM documents use elegant language like "the Notes and Domino product family." I have opted to just stick to Notes.

IBM was tremendously invested in the development of client-server microcomputer applications in the 1990s, unsurprisingly since they were the closest match to IBM's traditional strength in mainframe systems. The acquisition of Lotus was, no doubt, intended to advance that focus: Lotus Notes must have been one of the most prominent client-server products of the Windows '95 era. Its architecture, something like a "thin client" system that was derived from its mainframe precursor, felt akin to IBM's block terminals and generally fit well with IBM's approach to microcomputer applications (small client, big backend).

IBM was also investing in another project that fit their client-server vision: Java. Java was originally designed for embedded applications (interactive television!), but by its first official release in 1996 it had grown into more of a general purpose business application platform. The Microsoft .NET platform is a good later comparable, starting out where Java ended up. The "standard edition" and "enterprise edition" split of Java, and web service platform concept developing around them, fit client-server applications well. Java was widely seen as an easier, more productive language than C++. Best of all, Java's origins as a portable language to run on lightweight, embedded VMS made it a perfect fit for software that would need to run on a variety of different clients.

Lotus Notes 1 through 3 had been iterative improvements, from 1989 to 1993, adding features and fixing defects. With Lotus Notes 4, spanning 1996 to 1999, IBM Lotus Development started on a wholesale port to Java. Java was added as a native scripting language, alongside Formula and LotusScript, and many newer Lotus Notes features were "Java first." The Java-ization of Lotus Notes could be said to have completed in 2008, when the Lotus Notes client was completely replaced with a full-Java implementation based on Eclipse ⁴.

Alongside the new world of the object-oriented user interface, IBM also adapted Lotus Notes to the web. Version 4 introduced a very simple web interface where users could view static notes through a web browser, and IBM added an SMTP bridge for interoperation with other email systems.

At the dawn of the 2000s, Lotus Notes' technical leadership had eroded. The state of the art in GUI applications was far more advanced, and Microsoft had invested in their own groupware products (Exchange and Sharepoint). As part of the Windows Server offering, the Microsoft products were better integrated into the Windows desktop experience. IBM's investment in web technologies had lagged, so while Sharepoint was never exactly a hot rod, it was easier to interact with than Lotus Notes.

Still, development continued apace. IBM retired the Lotus branding around 2002, decisively folding IBM Notes and IBM Domino into Big Blue. Domino Web Access provided a web interface by 2002, although it was limited and not much different from what we would now consider mere webmail. IBM introduced SameTime, an instant messaging system that integrated with Lotus Notes. QuickPlace, similarly, was a web-based file/document management system that integrated with Lotus Notes as well. Both of these were even initially Lotus-branded, although IBM dropped that practice a few years later (and, for some reason, renamed QuickPlace to Quickr. It was a time).

When we talk about Java and the web, you might remember something, perhaps with a shudder: applets. In 2008, IBM introduced XPages. XPages were basically Java Server Faces components that ran on Domino server. They interacted directly with the Notes database and were, in some ways, just web-based versions of Notes forms. You even edited them using Domino Designer, the IDE for "serious" Notes development. Unfortunately, XPages hewed too close to JSF and remained a separate platform from Notes itself. You could not simply view existing Notes forms via XPages, or even readily port Notes interfaces to XPages format. If you wanted to support both the desktop and web experience, as most businesses would around 2010, you tended to end up writing things twice. Next thing you know, you're only really actively maintaining the web version, and then you might as well be on Sharepoint which was then well established.

To me, the most captivating part of Lotus Notes is its decline. In the mid '90s, Lotus Notes was the dominant groupware platform by a wide margin—Forbes put Lotus Notes market share at 64% in 1995. In 1997, they had fallen to 47%. A 2008 survey put Lotus Notes at 10%. Considering the long cycles and sheer staying power of enterprise software, that was a remarkable fall—all the more so considering that at that peak, IBM spent over $3 billion to get their hands on it.

Admittedly, IBM buying a popular software product at great expense and running it into the ground is an old story. But there's another major player here too: Microsoft was at their best. The vast majority of Lotus Notes' users went directly to Microsoft Exchange, which besides its higher level of integration with Windows (both at a technical and sales level) was also agreed to be more performant and easier to maintain. I would wager that Microsoft was at their peak engineering competence in the early '00s, and they had a considerable late-mover advantage over software that carried all the baggage of predating open-standards TCP/IP networking. NetWare went into a precipitous slide for the same reason.

Standards themselves are another important part of the story. Early email systems were all proprietary, and weren't necessarily networked systems at all—the first email implementations just stored mail on the single machine that all of the users accessed via terminal. Lotus Notes having a completely proprietary implementation of email didn't stand out in that world. When Microsoft first launched Exchange, it had the same limitations. That changed fast, albeit more by accident than intention. ITU X.400 had been expected to provide the standards for email on the public internet, replacing the hodgepodge of proprietary and network-specific open standards. Instead, X.400 failed to do much of anything at all—but not before it became the basis of Microsoft Exchange.

Exchange ended up in sort of a standards purgatory, an implementation of an open standard that died before Exchange could even launch. Exchange's X.400 capabilities became a proprietary standard of unusual origin, and most of industry and academia opted to follow the NSF example and adopt the older and simpler SMTP. Microsoft apparently understood that interoperability would be critical for the future of email, so Exchange was marketed from the very start as a multi-protocol system that could speak SMTP as fluently as anything else. That wasn't ever 100% true, but it was the idea that mattered. Lotus Notes had SMTP support but it was lagging behind, Exchange was the future focused on interoperation.

The web story was almost exactly the same. That's not to say that Exchange had a good version of a web interface, Microsoft has famously struggled with messes like Outlook Web Access. But they were still more agile than IBM, and their complete vertical control of the desktop experience meant that their weird proprietary web technology (ActiveX) fared better than IBM's weird proprietary web technology (a confusing tangle of Notes browser plugins and Java applets). Besides, Lotus's web features were apparently separately licensed at great cost. Microsoft wasn't cheap but, in this case, they were the budget option.

Last of all is the elephant in the room. It is hard, today, to explain exactly what Lotus Notes was. That's not a recent problem. In 1998, Forbes wrote:

Even before IBM arrived, Notes' identity had been blurred; it's no clearer now. First the software was marketed as a system to make it easy for a widely dispersed group of employees to work together editing a document or managing a project. Then Notes was redefined as a tool for building customized collaborative applications. Now Lotus seems to be playing up the E-mail function in Notes and positioning it as a replacement for cc:Mail. The latter was once a leader in its field but is now suffering a slow death.

cc:Mail! That's a whole different article. So let's stick to the point: the fact that Lotus Notes sort of defined a product category became a liability as other vendors muscled in with more targeted, more narrowly focused, and more obviously useful competition. Lotus Notes started to sound like Zombocom: You can do anything! Microsoft was just selling email and calendar. Notes' large scope was a problem for the user experience as well; besides the fact that the Notes client looked and felt dated by the '00s, the generality and depth of its capabilities meant that it was also just plain hard to use.

In 2018, IBM sold what remained of Lotus Development to an Indian software company called HCL. HCL is a classic IT consulting firm, with a portfolio that spans "Industry 4.0" to "SAP consulting." They acquired Lotus as part of a bundle of IBM castoffs: some of you might remember BigFix. Lotus Notes is now HCL Notes, and as far as I can tell HCL intends to just enjoy the revenue as long as legacy customers will pay them to keep Notes running.

That's another part of the allure of Lotus Notes: it might be the most legacy of legacy software. I have never worked at an organization that was using Lotus Notes, but everywhere I've worked that wasn't a startup had a powerful institutional memory of back when, in the Lotus Notes Times. Some people remembered it fondly, most people remembered it hatefully, but they all remembered it.

Will GMail ever inspire such emotion?

In the United States, we are losing our fondness for cash. As in many other countries, cards and other types of electronic payments now dominate everyday commerce. To some, this is a loss. Cash represented a certain freedom from intermediation, a comforting simplicity that you just don't get from Visa. It's funny to consider, then, how cash is in fact quite amenable to automation. Even Benjamin Franklin's face on a piece of paper can feel like a mere proxy for a database transaction. How different is cash itself from "e-cash", when it starts and ends its lifecycle through automation?

Increasing automation of cash reflects the changing nature of banking: decades ago, a consumer might have interacted with banking primarily through a "passbook" savings account, where transactions were so infrequent that the bank recorded them directly in the patron's copy of the passbook. Over the years, increasing travel and nationwide communications led to the ubiquitous use of inter-bank money transfers, mostly in the form of the check. The accounts that checks typically drew on—checking accounts—were made for convenience and ease of access. You might deposit your entire paycheck into an account—it might even be sent there automatically—and then when you needed a little walking around money, you would withdraw cash by the assistance of a teller. By the time I was a banked consumer, even the teller was mostly gone. Today, we get our cash from machines so that it can be deposited into other machines.

Cash handling is fraught with peril. Bills are fairly small and easy to hide, and yet quite valuable. Automation in the banking world first focused on solving this problem, of reliable and secure cash handling within the bank branch. The primary measure against theft by insiders was that the theft would be discovered, as a result of the careful bookkeeping that typifies banks. But, well, that bookkeeping was surprisingly labor-intensive in even the bank of the 1950s.

Histories of the ATM usually focus on just that: the ATM. It's an interesting story, but one that I haven't been particularly inclined to cover due to the lack of a compelling angle. Let's try IBM. IBM is such an important, famous player in business automation that it forms something of a synecdoche for the larger industry. Even so, in the world of bank cash handling, IBM's efforts ultimately failed... a surprising outcome, given their dominance in the machines that actually did the accounting.

In this article, we'll examine the history of ATMs—by IBM. IBM was just one of the players in the ATM industry and, by its maturity, not even one of the more important ones. But the company has a legacy of banking products that put the ATM in a more interesting context, and despite lackluster adoption of later IBM models, their efforts were still influential enough that later ATMs inherited some of IBM's signature design concepts. I mean that more literally than you might think. But first, we have to understand where ATMs came from. We'll start with branch banking.

When you open a bank account, you typically do so at a "branch," one of many physical locations that a national bank maintains. Let us imagine that you are opening an account at your local branch of a major bank sometime around 1930; whether before or after that year's bank run is up to you. Regardless of the turbulent economic times, the branch became responsible for tracking the balance of your account. When you deposit money, a teller writes up a slip. When you come back and withdraw money, a different teller writes up a different slip. At the end of each business day, all of these slips (which basically constitute a journal in accounting terminology) have to be rounded up by the back office and posted to the ledger for your account, which was naturally kept as a card in a big binder.

A perfectly practicable 1930s technology, but you can already see the downsides. Imagine that you appear at a different branch to withdraw money from your account. Fortunately, this was not very common at the time, and you would be more likely to use other means of moving money in most scenarios. Still, the bank tries to accommodate. The branch at which you have appeared can dispense cash, write a slip, and then send it to the correct branch for posting... but they also need to post it to their own ledger that tracks transactions for foreign accounts, since they need to be able to reconcile where their cash went. And that ignores the whole issue of who you are, whether or not you even have an account at another branch, and whether or not you have enough money to cover the withdrawal. Those are problems that, mercifully, could mostly be sorted out with a phone call to your home branch.

Bank branches, being branches, do not exist in isolation. The bank also has a headquarters, which tracks the finances of its various branches—both to know the bank's overall financial posture (critical considering how banks fail), and to provide controls against insider theft. Yes, that means that each of the branch banks had to produce various reports and ledger copies and then send them by courier to the bank headquarters, where an army of clerks in yet another back office did yet another round of arithmetic to produce the bank's overall ledgers.

As the United States entered World War II, an expanding economy, rapid industrial buildup, and a huge increase in national mobility (brought on by things like the railroads and highways) caused all of these tasks to occur on larger and larger scales. Major banks expanded into a tiered system, in which branches reported their transactions to "regional centers" for reconciliation and further reporting up to headquarters. The largest banks turned to unit record equipment or "business machines," arguably the first form of business computing: punched card machines that did not evaluate programs, but sorted and summed.

Simple punched card equipment gave way to more advanced automation, innovations like the "posting machine." These did exactly what they promised: given a stack of punched cards encoding transactions, they produced a ledger with accurately computed sums. Specialized posting machines were made for industries ranging from hospitality (posting room service and dining charges to room folios) to every part of finance, and might be built custom to the business process of a large customer.

If tellers punched transactions into cards, the bank could come much closer to automation by shipping the cards around for processing at each office. But then, if transactions are logged in a machine readable format, and then processed by machines, do we really need to courier them to rooms full of clerks?

Well, yes, because that was the state of technology in the 1930s. But it would not stay that way for long.

In 1950, Bank of America approached SRI about the feasibility of an automated check processing system. Use of checks was rapidly increasing, as were total account holders, and the resulting increase in inter-branch transactions was clearly overextending BoA's workforce—to such an extent that some branches were curtailing their business hours to make more time for daily closing. By 1950, computer technology had advanced to such a state that it was obviously possible to automate this activity, but it still represented one of the most ambitious efforts in business computing to date.

BoA wanted a system that would not only automate the posting of transactions prepared by tellers, but actually automate the handling of the checks themselves. SRI and, later, their chosen manufacturing partner General Electric ran a multi-year R&D campaign on automated check handling that ultimately led to the design of the checks that we use today: preprinted slips with account holder information, and account number, already in place. And, most importantly, certain key fields (like account number and check number) represented in a newly developed machine-readable format called "MICR" for magnetic ink character recognition. This format remains in use today, to the extent that checks remain in use, although as a practical matter MICR has given way to the more familiar OCR (aided greatly by the constrained and standardized MICR character set).

The machine that came out of this initiative was called ERMA, the Electronic Recording Machine, Accounting. I will no doubt one day devote a full article to ERMA, as it holds a key position in the history of business computing while also managing to not have much of a progeny due to General Electric's failure to become a serious contender in the computer industry. ERMA did not lead to a whole line of large-scale "ERM" business systems as GE had hoped, but it did firmly establish the role of the computer in accounting, automate parts of the bookkeeping through almost the entirety of what would become the nation's largest bank, and inspire generations of products from other computer manufacturers.

The first ERMA system went into use in 1959. While IBM was the leader in unit record equipment and very familiar to the banking industry, it took a few years for Big Blue to bring their own version. Still, IBM had their own legacy to build on, including complex electromechanical machines that performed some of the tasks that ERMA was taking over. Since the 1930s, IBM had produced a line of check processing or "proofing" machines. These didn't exactly "automate" check handling, but they did allow a single operator to handle a lot of documents.

The IBM 801, 802, and 803 line of check proofers used what were fundamentally unit record techniques—keypunch, sorting bins, mechanical totalizers—to present checks one at a time in front of the operator, who read information like the amount, account number, and check number off of the paper slip and entered it on a keypad. The machine then whisked the check away, printing the keyed data (and reference numbers for auditing) on the back of the check, stamped an endorsement, added the check's amounts to the branch's daily totals (including subtotals by document type), and deposited the check in an appropriate sorter bin to be couriered to the drawer's bank. While all this happened, the machines also printed the keyed check information and totals onto paper tapes.

By the early 1960s, with ERMA on the scene, IBM started to catch up. Subsequent check processing systems gained support for MICR, eliminating much (sometimes all!) of the operator's keying. Since the check proofing machines could also handle deposit slips, a branch that generated MICR-marked deposit slips could eliminate most of the human touchpoints involved in routine banking. A typical branch bank setup might involve an IBM 1210 document reader/sorter machine connected by serial channel to an IBM 1401 computer. This system behaved much like the older check proofers, reading documents, logging them, and calculating totals. But it was now all under computer control, with the flexibility and complexity that entails.

One of these setups could process almost a thousand checks a minute with a little help from an operator, and adoption of electronic technology at other stages made clerks lives easier. For example, IBM's mid-1960s equipment introduced solid-state memory. The IBM 1260 was used for adding machine-readable MICR data to documents that didn't already have it. Through an innovation that we would now call a trivial buffer, the 1260's operator could key in the numbers from the next document while the printer was still working on the previous.

Along with improvements in branch bank equipment came a new line of "high-speed" systems. In a previous career, I worked at a Federal Reserve bank, where "high-speed" was used as the name of a department in the basement vault. There, huge machines processed currency to pick out bad bills. This use of "high-speed" seems to date to an IBM collaboration with the Federal Reserve to build machines for central clearinghouses, handling checks by the tens of thousands. By the time I found myself in central banking, the use of "high-speed" machinery for checks was a thing of the past—"digital substitute" documents or image-based clearing having completely replaced physical handling of paper checks. Still, the "high-speed" staff labored on in their ballistic glass cages, tending to the green paper slips that the institution still dispenses by the millions.

One of the interesting things about the ATM is when, exactly, it pops up in the history of computers. We are, right now, in the 1960s. The credit card is in its nascent stages, MasterCard's predecessor pops up in 1966 to compete with Bank of America's own partially ERMA-powered charge card offering. With computer systems maintaining account sums, and document processing machines communicating with bookkeeping computers in real-time, it would seem that we are on the very cusp of online transaction authorization, which must be the fundamental key to the ATM. ATMs hand out cash, and one thing we all know about cash is that, once you give yours to someone else you are very unlikely to get it back. ATMs, therefore, must not dispense cash unless they can confirm that the account holder is "good for it." Otherwise the obvious fraud opportunity would easily wipe out the benefits.

So, what do you do? It seems obvious, right? You connect the ATM to the bookkeeping computer so it can check account balances before dispensing cash. Simple enough.

But that's not actually how the ATM evolved, not at all. There are plenty of reasons. Computers were very expensive so banks centralized functions and not all branches had one. Long-distance computer communication links were very expensive as well, and still, in general, an unproven technology. Besides, the computer systems used by banks were fundamentally batch-mode machines, and it was difficult to see how you would shove an ATM's random interruptions into the programming model.

Instead, the first ATMs were token-based. Much like an NYC commuter of the era could convert cash into a subway token, the first ATMs were machines that converted tokens into cash. You had to have a token—and to get one, you appeared at a teller during business hours, who essentially dispensed the token as if it were a routine cash withdrawal.

It seems a little wacky to modern sensibilities, but keep in mind that this was the era of the traveler's check. A lot of consumers didn't want to carry a lot of cash around with them, but they did want to be able to get cash after hours. By seeing a teller to get a few ATM tokens (usually worth $10 or £10 and sometimes available only in that denomination), you had the ability to retrieve cash, but only carried a bank document that was thought (due to features like revocability and the presence of ATMs under bank surveillance) to be relatively secure against theft. Since the tokens were later "cleared" against accounts much like checks, losing them wasn't necessarily a big deal, as something analogous to a "stop payment" was usually possible.

Unlike subway tokens, these were not coin-shaped. The most common scheme was a paper card, often the same dimensions as a modern credit card, but with punched holes that encoded the denomination and account holder information. The punched holes were also viewed as an anti-counterfeiting measure, probably not one that would hold up today, but still a roadblock to fraudsters who would have a hard time locating a keypunch and a valid account number. Manufacturers also explored some other intriguing opportunities, like the very first production cash dispenser, 1967's Barclaycash machine. This proto-ATM used punched paper tokens that were also printed in part with a Carbon-14 ink. Carbon-14 is unstable and emits beta radiation, which the ATM detected with a simple electrostatic sensor. For some reason difficult to divine the radioactive ATM card did not catch on.

For roughly the first decade of the "cash machine," they were offline devices that issued cash based on validating a token. The actual decision making, on the worthiness of a bank customer to withdraw cash, was still deferred to the teller who issued the tokens. Whether or not you would even consider this an ATM is debatable, although historical accounts generally do. They are certainly of a different breed than the modern online ATM, but they also set some of the patterns we still follow. Consider, for example, the ATMs within my lifespan that accepted deposits in an envelope. These ATMs did nothing with the envelopes other than accumulate them into a bin to go to a central processing center later on—the same way that early token-based ATMs introduced deposit boxes.

In this theory of ATM evolution, the missing link that made 1960s­–1970s ATMs so primitive was the lack of computer systems that were amenable to real-time data processing using networked peripherals. The '60s and '70s were a remarkable era in computer history, though, seeing the introduction of IBM's System/360 and System/370 line. These machines were more powerful, more flexible, and more interoperable than any before them. I think it's fair to say that, despite earlier dabbling, it was the 360/370 that truly ushered in the era of business computing. Banks didn't miss out.

One of the innovations of the System/360 was an improved and standardized architecture for the connection of peripherals to the machine. While earlier IBM models had supported all kinds of external devices, there was a lot of custom integration to make that happen. With the System/360, this took the form of "Bisync," which I might grandly call a far ancestor of USB. Bisync allowed a 360 computer to communicate with multiple peripherals connected to a common multi-drop bus, even using different logical communications protocols. While the first Bisync peripherals were "remote job entry" terminals for interacting with the machine via punched cards and teletype, IBM and other manufacturers found more and more applications in the following years.

IBM had already built document processing machines that interacted with their computers. In 1971, IBM joined the credit card fray with the 2730, a "transaction" terminal that we would now recognize as a credit card reader. It used a Bisync connection to a System/360-class machine to authorize a credit transaction in real time. The very next year, IBM took the logical next step: the IBM 2984 Cash Issuing Terminal. Like many other early ATMs, the 2984 had its debut in the UK as Lloyds Bank's "Cashpoint."

The 2984 similarly used Bisync communications with a System/360. While not the very first implementation of the concept, the 2984 was an important step in ATM security and the progenitor of an important line of cryptographic algorithms. To withdraw cash, a user inserted a magnetic card that contained an account number, and then keyed in a PIN. The 2984 sent this information, over the Bisync connection, to the computer, which then responded with a command such as "dispense cash." In some cases, the computer was immediately on the other side of the wall, but it was already apparent that banks would install ATMs in remote locations controlled via leased telephone lines—and those telephone lines were not well-secured. A motivated attacker (and with cash involved, it's easy to be motivated!) could probably "tap" the ATM's network connection and issue it spurious "dispense cash" commands. To prevent this problem, and assuage the concerns of bankers who were nervous about dispensing cash so far from the branch's many controls, IBM decided to encrypt the network connection.

The concept of an encrypted network connection was not at all new; encrypted communications were widely used in the military during the second World War and the concept was well-known in the computer industry. As IBM designed the 2984, in the late '60s, encrypted computer links were nonetheless very rare. There were not yet generally accepted standards, and cryptography as an academic discipline was immature.

IBM, to secure the 2984's network connection, turned to an algorithm recently developed by an IBM researcher named Horst Feistel. Feistel, for silly reasons, had named his family of experimental block ciphers LUCIFER. For the 2984, a modified version of one of the LUCIFER implementations called DSD-1¹. Through a Bureau of Standards design competition and the twists and turns of industry politics, DSD-1 later reemerged (with just slight changes) as the Data Encryption Standard, or DES. We owe the humble ATM honors for its key role in computer cryptography.

The 2984 was a huge step forward. Unlike the token-based machines of the 1960s, it was pretty much the same as the ATMs we use today. To use a 2984, you inserted your ATM card and entered a PIN. You could then choose to check your balance, and then enter how much cash you wanted. The machine checked your balance in real time and, if it was high enough, debited your account immediately before coughing up money.

The 2984 was not as successful as you might expect. The Lloyd's Bank rollout was big, but very few were installed by other banks. Collective memory of the 2984 is vague enough that I cannot give a definitive reason for its limited success, but I think it likely comes down to a common tale about IBM: price and flexibility. The 2984 was essentially a semi-custom peripheral, designed for Lloyd's Bank and the specific System/360 environment already in place there. Adoption for other banks was quite costly. Besides, despite the ATM's lead in the UK, the US industry had quickly caught up. By the time the 2984 would be considered by other banks, there were several different ATMs available in the US from other manufacturers (some of them the same names you see on ATMs today). The 2984 is probably the first "modern" ATM, but since IBM spent 4-5 years developing it, it was not as far ahead of the curve on launch day as you might expect. Just a year or two later, a now-forgotten company called Docutel was dominating the US market, leaving IBM little room to fit in.

Because most other ATMs were offered by companies that didn't control the entire software stack, they were more flexible, designed to work with simpler host support. There is something of an inverse vertical integration penalty here: when introducing a new product, close integration with an existing product family makes it difficult to sell! Still, it's interesting that the 2984 used pretty much the same basic architecture as the many ATMs that followed. It's worth reflecting on the 2984's relationship with its host, a close dependency that generally holds true for modern ATMs as well.

The 2984 connected to its host via a Bisync channel (possibly over various carrier or modem systems to accommodate remote ATMs), a communications facility originally provided for remote job entry, the conceptual ancestor of IBM's later block-oriented terminals. That means that the host computer expected the peripheral to provide some input for a job and then wait to be sent the results. Remote job entry devices, and block terminals later, can be confusing when compared to more familiar Unix-family terminals. In some ways, they were quite sophisticated, with the host computer able to send configuration information like validation rules for input. In other ways, they were very primitive, capable of no real logic other than receiving computer output (which was dumped to cards, TTY, or screen) and then sending computer input (from much the same devices). So, the ATM behaved the same way.

In simple terms, the ATM's small display (called a VDU or Video Display Unit in typical IBM terminology) showed whatever the computer sent as the body of a "display" command. It dispensed whatever cash the computer indicated with a "dispense cash" command. Any user input, such as reading a card or entry of a PIN number, was sent directly to the computer. The host was responsible for all of the actual logic, and the ATM was a dumb terminal, just doing exactly what the computer said. You can think of the Cash Issuing Terminal as, well, just that: a mainframe terminal with a weird physical interface.

Most modern ATMs follow this same model, although the actual protocol has become more sophisticated and involves a great deal more XML. You can be reassured that when the ATM takes a frustratingly long time to advance to the next screen, it is at least waiting to receive the contents of that screen from a host computer that is some distance away or, even worse, in The Cloud.

Incidentally, you might wonder about the software that ran on the host computer. I believe that the IBM 2984 was designed for use with CICS, the Customer Information Control System. CICS will one day get its own article, but it launched in 1966, built specifically for the Michigan Bell to manage customer and billing data. Over the following years, CICS was extensively expanded for use in the utility and later finance industries. I don't think it's inaccurate to call CICS the first "enterprise customer relationship management system," the first voyage in an adventure that took us through Siebel before grounding on the rocks of Salesforce. Today we wouldn't think of a CRM as the system of record for depository finance institutions like banks, but CICS itself was very finance-oriented from the start (telephone companies sometimes felt like accounting firms that ran phones on the side) and took naturally to gathering transactions and posting them against customer accounts. Since CICS was designed as an online system to serve telephone and in-person customer service reps (in fact making CICS a very notable early real-time computing system), it was also a good fit for handling ATM requests throughout the day.

Despite the 2984's lackluster success, IBM moved on. I don't think IBM was particularly surprised by the outcome, the 2984 was always a "request quotation" (e.g. custom) product. IBM probably regarded it as a prototype or pilot with their friendly customer Lloyds Bank. More than actual deployment, the 2984's achievement was paving the way for the IBM 3614 Consumer Transaction Facility.

In 1970, IBM had replaced the System/360 line with the System/370. The 370 is directly based on the 360 and uses the same instruction set, but it came with numerous improvements. Among them was a new approach to peripheral connectivity that developed into the IBM Systems Network Architecture, or SNA, basically IBM's entry into the computer networking wars of the 1970s and 1980s. While SNA would ultimately cede to IP (with, naturally, an interregnum of SNA-over-IP), it gave IBM the foundations for networked systems that are almost modern in their look and feel.

I say almost because SNA was still very much a mainframe-oriented design. An example SNA network might look like this: An S/370 computer running CICS (or one of several other IBM software packages with SNA support) is connected via channel (the high-speed peripheral bus on mainframe computers, analogous to PCI) to an IBM 3705 Communications Controller running the Network Control Program (analogous to a network interface controller). The 3705 had one or more "scanners" installed, which supported simple low-speed serial lines or fast, high-level protocols like SDLC (synchronous data link control) used by SNA. The 3705 fills a role sometimes called a "front-end processor," doing the grunt work of polling (scanning) communications lines and implementing the SDLC protocol so that the "actual computer" was relieved of these menial tasks.

At the other end of one of the SDLC links might be an IBM 3770 Data Communications System, which was superficially a large terminal that, depending on options ordered, could include a teletypewriter, card reader and punch, diskette drives, and a high speed printer. Yes, the 3770 is basically a grown-up remote job entry terminal, and the SNA/SDLC stack was a direct evolution from the Bisync stack used by the 2984. The 3770 had a bit more to offer, though: in order to handle its multiple devices, like the printer and card punch, it acted as a sort of network switch—the host computer identified the 3770's devices as separate endpoints, and the 3770 interleaved their respective traffic. It could also perform that interleaving function for additional peripherals connected to it by serial lines, which depending on customer requirements often included additional card punches and readers for data entry, or line printers for things like warehouse picking slips.

In 1973, IBM gave banks the SNA treatment with the 3600 Finance Communication System ². A beautifully orange brochure tells us:

The IBM 3600 Finance Communication System is a family of products designed to provide the Finance Industry with remote on-line teller station operation.

System/370 computers represented an enormous investment, generally around a million dollars and more often above that point than below. They were also large and required both infrastructure and staff to support them. Banks were already not inclined to install an S/370 in each branch, so it became a common pattern to place a "full-size" computer like an S/370 in a central processing center to support remote peripherals (over leased telephone line) in branches. The 3600 was a turn-key product line for exactly this use.

An S/370 computer with a 3704 or 3705 running the NCP would connect (usually over a leased line) to a 3601 System, which IBM describes as a "programmable communications controller" although they do not seem to have elevated that phrase to a product name. The 3601 is basically a minicomputer of its own, with up to 20KB of user-available memory and diskette drive. A 3601 includes, as standard, a 9600 bps SDLC modem for connection to the host, and a 9600 bps "loop" interface for a local multidrop serial bus. For larger installations, you could expand a 3601 with additional local loop interfaces or 4800 or 9600 bps modems to extend the local loop interface to a remote location via telephone line.

In total, a 3601 could interface up to five peripheral loops with the host computer over a single interleaved SDLC link. But what would you put on those peripheral loops? Well, the 3604 Keyboard Display Unit was the mainstay, with a vacuum fluorescent display and choice of "numeric" (accounting, similar to a desk calculator) or "data entry" (alphabetic) keyboard. A bank would put one of these 3604s in front of each teller, where they could inquire into customer accounts and enter transactions. In the meantime, 3610 printers provided general-purpose document printing capability, including back-office journals (logging all transactions) or filling in pre-printed forms such as receipts and bank checks. Since the 3610 was often used as a journal printer, it was available with a take-up roller that stored the printed output under a locked cover. In fact, basically every part of the 3600 system was available with a key switch or locking cover, a charming reminder of the state of computer security at the time.

The 3612 is a similar printer, but with the addition of a dedicated passbook feature. Remember passbook savings accounts, where the bank writes every transaction in a little booklet that the customer keeps? They were still around, although declining in use, in the 1970s. The 3612 had a slot on the front where an appropriately formatted passbook could be inserted, and like a check validator or slip printer, it printed the latest transaction onto the next empty line. Finally, the 3618 was a "medium-speed" printer, meaning 155 lines per minute. A branch bank would probably have one, in the back office, used for printing daily closing reports and other longer "administrative" output.

A branch bank could carry out all of its routine business through the 3600 system, including cash withdrawals. In fact, since a customer withdrawing cash would end up talking to a teller who simply keyed the transaction into a 3604, it seems like a little more automation could make an ATM part of the system.

Enter the 3614 Consumer Transaction Facility, the first IBM ATM available as a regular catalog item. The 3614 is actually fairly obscure, and doesn't seem to have sold in large numbers. Some sources suggest that it was basically the same as the 2984, but with a general facelift and adaptations to connect to a 3601 Finance Communication Controller instead of directly to a front-end processor. Some features which were optional on the 2984, like a deposit slot, were apparently standard on 3614. I'm not even quite sure when the 3614 was introduced, but based on manual copyright dates they must have been around by 1977.

One of the reasons the 3614 is obscure is that its replacement, the IBM 3624 Consumer Transaction Facility, hit the market in 1978—probably very shortly after the 3614. The 3624 was functionally very similar to the 3614, but with maintainability improvements like convenient portable cartridges for storing cash. It also brought a completely redesigned front panel that is more similar to modern ATMs. I should talk about the front panels—the IBM ATMs won a few design awards over their years, and they were really very handsome machines. The backlit logo panel and function-specific keys of the 3624 look more pleasant to use than most modern ATMs, although they would, of course, render translation difficult.

The 3614/3624 series established a number of conventions that are still in use today. For example, they added an envelope deposit system in which the machine accepted an envelope (with cash or checks) and printed a transaction identifier on the outside of the envelope for lookup at the processing center. This relieved the user of writing up a deposit slip when using the ATM. It was also capable of not only reading but, optionally, writing to the magnetic strips on ATM cards. To the modern reader that sounds strange, but we have to discuss one of the most enduring properties of the 3614/3624: their handling of PIN numbers.

I believe the 2984 did something fairly similar, but the details are now obscure (and seem to get mixed up with its use of LUCIFER/DSD-1/DES for communications). The 3614/3624, though, so firmly established a particular approach to PIN numbers that it is now known as the 3624 algorithm. Here's how it works: the ATM reads the card number (called Primary Account Number or PAN) off of the ATM card, reads a key from memory, and then applies a convoluted cryptographic algorithm to calculate an "intermediate PIN" from it. The "intermediate PIN" is then summed with a "PIN offset" stored on the card itself, modulo 10, to produce the PIN that the user is actually expected to enter. This means that your "true" PIN is a static value calculated from your card number and a key, but as a matter of convenience, you can "set" a PIN of your choice by using an ATM that is equipped to rewrite the PIN offset on your card. This same system, with some tweaks and a lot of terminological drift, is still in use today. You will sometimes hear IBM's intermediate PIN called the "natural PIN," the one you get with an offset of 0, which is a use of language that I find charming.

Another interesting feature of the 3624 was a receipt printer—I'm not sure if it was the first ATM to offer a receipt, but it was definitely an early one. The exact mechanics of the 3624 receipt printer are amusing and the result of some happenstance at IBM. Besides its mainframes and their peripherals, IBM in the 1970s was increasingly invested in "midrange computers" or "midcomputers" that would fill in a space between the mainframe and minicomputer—and, most importantly, make IBM more competitive with the smaller businesses that could not afford IBM's mainframe systems and were starting to turn to competitors like DEC as a result. These would eventually blossom into the extremely successful AS/400 and System i, but not easily, and the first few models all suffered from decidedly soft sales.

For these smaller computers, IBM reasoned that they needed to offer peripherals like card punches and readers that were also smaller. Apparently following that line of thought to a misguided extent, IBM also designed a smaller punch card: the 96-column three-row card, which was nearly square. The only computer ever to support these cards was the very first of the midrange line, the 1969 System/3. One wonders if the System/3's limited success led to excess stock of 96-column card equipment, or perhaps they just wanted to reuse tooling. In any case, the oddball System/3 card had a second life as the "Transaction Statement Printer" on the 3614 and 3624. The ATM could print four lines of text, 34 characters each, onto the middle of the card. The machines didn't actually punch them, and the printed text ended up over the original punch fields. You could, if you wanted, actually order a 3624 with two printers: one that presented the slip to the customer, and another that retained it internally for bank auditing. A curious detail that would so soon be replaced by thermal receipt printers.

Unlike IBM's ATMs before it, and, as we will see, unlike those after it as well, the 3624 was a hit. While IBM never enjoyed the dominance in ATMs that they did in computers, and companies like NCR and Diebold had substantial market share, the 3624 was widely installed in the late 1970s and would probably be recognized by anyone who was withdrawing cash in that era. The machine had technical leadership as well: NCR built their successful ATM line in part by duplicating aspects of the 3624 design, allowing interoperability with IBM backend systems. Ultimately, as so often happens, it may have been IBM's success that became its undoing.

In 1983, IBM completely refreshed their branch banking solution with the 4700 Finance Communication System. While architecturally similar, the 4700 was a big upgrade. For one, the CRT had landed: the 4700 peripherals replaced several-line VFDs with full-size CRTs typical of other computer terminals, and conventional computer keyboards to boot. Most radically, though, the 4700 line introduced distributed communications to IBM's banking offerings. The 4701 Communications Controller was optionally available with a hard disk, and could be programmed in COBOL. Disk-equipped 4701s could operate offline, without a connection to the host, or in a hybrid mode in which they performed some transactions locally and only contacted the host system when necessary. Local records kept by the 4701 could be automatically sent to the host computer on a scheduled basis for reconciliation.

Along with the 4700 series came a new ATM: the IBM 473x Personal Banking Machines. And with that, IBM's glory days in ATMs came crashing to the ground. The 473x series was such a flop that it is hard to even figure out the model numbers, the 4732 is most often referenced but others clearly existed, including the 4730, 4731, 4736, 4737, and 4738. These various models were introduced from 1983 to 1988, making up almost a decade of IBM's efforts and very few sales. The 4732 had a generally upgraded interface, including a CRT, but a similar feature set—unsurprising, given that the 3624 had already introduced most of the features ATMs have today. It also didn't sell. I haven't been able to find any numbers, but the trade press referred to the 4732 with terms like "debacle," so they couldn't have been great.

There were a few faults in the 4732's stars. First, IBM had made the decision to handle the 4700 Finance Communication System as a complete rework of the 3600. The 4700 controllers could support some 3600 peripherals, but 4700 peripherals could not be used with 3600 controllers. Since 3600 systems were widely installed in banks, the compatibility choice created a situation where many of the 4732's prospective buyers would end up having to replace a significant amount of their other equipment, and then likely make software changes, in order to support the new machine. That might not have been so bad on its own had IBM's competitors not provided another way out.

NCR made their fame in ATMs in part by equipping their contemporary models with 3624 software emulation, making them a drop-in modernization option for existing 3600 systems. Other ATM manufacturers had pursued a path of interoperability, with multiprotocol ATMs that supported multiple hosts, and standalone ATM host products that could interoperate with multiple backend accounting systems. For customers, buying an NCR or Diebold product that would work with whatever they already used was a more appealing option than buying the entire IBM suite in one go. It also matched the development cycle of ATMs better: as a consumer-facing device, ATMs became part of the brand image of the bank, and were likely to see replacement more often than back-office devices like teller terminals. NCR offered something like a regular refresh, while IBM was still in a mode of generational releases that would completely replace the bank's computer systems.

The 4732 and its 473x compatriots became the last real IBM ATMs. After a hiatus of roughly a decade, IBM reentered the ATM market by forming a joint venture with Diebold called InterBold. The basic terms were that Diebold would sell its ATMs in the US, and IBM would sell them overseas, where IBM had generally been the more successful of the two brands. The IBM 478x series ATMs, which you might encounter in the UK for example, are the same as the Diebold 1000 series in the US. InterBold was quite successful, becoming the dominant ATM manufacturer in the US, and in 1998 Diebold bought out IBM's share.

IBM had won the ATM market, and then lost it. Along the way, they left us with so much texture: DES's origins in the ATM, the 3624 PIN format, the dumb terminal or thin client model... even InterBold, IBM's protracted exit, gave us quite a legacy: now you know the reason that so many later ATMs ran OS/2. IBM, a once great company, provided Diebold with their once great operating system. Unlike IBM, Diebold made it successful.

The way I see it, few parts of American life are as quintessentially American as buying gas. We love our cars, we love our oil, and an industry about as old as automobiles themselves has developed a highly consistent, fully automated, and fairly user friendly system for filling the former with the latter.

I grew up in Oregon. While these rules have since been relaxed, many know Oregon for its long identity as one of two states where you cannot pump your own gas (the other being New Jersey). Instead, an attendant, employee of the gas station, operates the equipment. Like Portland's lingering indoor gas station, Oregon's favor for "full-service" is a holdover. It makes sense, of course, that all gas stations used to be full-service.

The front part of a gas station, where the pumps are and where you pull up your car, is called the Forecourt. The practicalities of selling gasoline, namely that it is a liquid sold by volume, make the forecourt more complex than you might realize. It's a set of devices that many of us interact with on a regular basis, but we rarely think about the sheer number of moving parts and long-running need for digital communications. Hey, that latter part sounds interesting, doesn't it?

Electric vehicles are catching on in the US. My personal taste in vehicles tends towards "old" and "cheap," but EVs have been on the market for long enough that they now come in that variety. Since my daily driver is an EV, I don't pay my dues at the Circle K nearly as often as I used to. One of the odd little details of EVs is the complexity hidden in the charging system or "EVSE," which requires digital communications with the vehicle for protection reasons. As consumers across the country install EVSE in their garages, we're all getting more familiar with these devices and their price tags. We might forget that, well, handling a fluid takes a lot of equipment as well... we just don't think about it, having shifted the whole problem to a large industry of loosely supervised hazardous chemical handling facilities.

Well, I don't mean to turn this into yet another discussion of the significant environmental hazard posed by leaking underground storage tanks. Instead, we're going to talk about forecourt technology. Let's start, then, with a rough, sketchy history of the forecourt.

The earliest volumetric fuel dispensers used an elevated glass tank where fuel was staged and measured before gravity drained it through the hose into the vehicle tank. Operation of these pumps was very manual, with an attendant filling the calibrated cylinder with the desired amount of gas, emptying it into the vehicle, and then collecting an appropriate sum of money. As an upside, the customer could be quite confident of the amount of fuel they purchased, since they could see it temporarily stored in the cylinder.

As cars proliferated in the 1910s, a company called Gilbarco developed a fuel dispenser that actually measured the quantity of fuel as it was being pumped from storage tank to vehicle... with no intermediary step in a glass cylinder required. The original Gilbarco design involved a metal turbine in a small glass sphere; the passing fuel spun the turbine which drove a mechanical counter. In truth, the design of modern fuel dispensers hasn't changed that much, although the modern volumetric turbines are made more accurate with a positive displacement design similar to a Roots blower.

Even with the new equipment, fuel was sold in much the same way: an attendant operated the pump, read the meter, and collected payment. There was, admittedly, an increased hazard of inattentive or malicious gas stations overcharging. Volumetric dispensers thus lead to dispensers that automatically calculated the price (now generally a legal requirement) and the practice of a regulatory authority like the state or tribal government testing fuel dispensers for calibration. Well, if consumers were expected to trust the gas station, perhaps the gas station ought to trust the consumer... and these same improvements to fuel dispensers made it more practical for the motorist to simply pump their own gas.

At the genesis of self-serve gasoline, most stations operated on a postpayment model. You pulled up, pumped gas, and then went inside to the attendant to pay whatever you owed. Of course, a few unscrupulous people would omit that last step. A simple countermeasure spread in busy cities: the pumps were normally kept powered off. Before dispensing gasoline, you would have to speak with the attendant. Depending on how trustworthy they estimated you to be, they might just turn on power to the pump or they might require you to deposit some cash with them in advance. This came to be known as "prepayment," and is now so universal in th US that the "prepay only" stickers on fuel dispensers seem a bit anachronistic ¹.

It's simple enough to imagine how this scheme worked, electronically. There is separate power wiring to the pumps for each dispenser (and these stations usually only had two dispensers anyway), and that wiring runs to the counter where the attendant can directly switch power. Most gas stations do use submersible pumps in the tank rather than in the actual dispenser, but older designs still had one pump per dispenser and were less likely to use submersible pumps anyway.

Soon, things became more complex. Modern vehicles have big gas tanks, and gas has become fairly expensive. What happens when a person deposits, say, $20 of "earnest cash" to get a pump turned on, and then pumps $25 worth of gas? Hopefully they have the extra $5, but the attendant doesn't know that. Besides, gas stations grew larger and it wasn't always feasible for the attendant to see the dispenser counters out the window. You wouldn't want to encourage people to just lie about the amount of gas they'd dispensed.

Gas stations gained remote control: using digital communications, fuel dispensers reported the value of their accumulators to a controller at the counter. The attendant would use the same controller to enable dispenser, potentially setting a limit at which the dispenser would automatically shut off. If you deposit $20, they enable the pump with a limit of $20. If you pay by card, they will likely authorize the card for a fixed amount (this used to routinely be $40 but has gone up for reasons you can imagine), enable the dispenser with no limit or a high limit, and then capture the actual amount after you finished dispensing ².

And that's how gas stations worked for quite a few decades. Most gas stations that you use today still have this exact same system in operation, but it may have become buried under additional layers of automation. There are two things that have caused combinatorial complexity in modern forecourt control: first, any time you automate something, there is a natural desire to automate more things. With a digital communications system between the counter and the forecourt, you can do more than just enable the dispensers! You might want to monitor the levels in the tanks, update the price on the big sign, and sell car wash vouchers with a discount for a related fuel purchase. All of these capabilities, and many more, have been layered on to forecourt control systems through everything from serial bus accessories to REST API third party integrations.

Speaking of leaking underground storage tanks, you likely even have a regulatory obligation to monitor tank levels and ensure they balance against bulk fuel deliveries and dispenser totals. This detects leakage, but it also detects theft, still a surprisingly common problem for gas stations. Your corporate office, or your bulk fuel provider, may monitor these parameters remotely to schedule deliveries and make sure that theft isn't happening with the cooperation of the station manager. Oh, and prices, those may be set centrally as well.

The second big change is nearly universal "CRIND." This is an awkward industry acronym for everyone's favorite convenience feature, Card Reader IN Dispenser. CRIND fuel dispensers let payment card customers complete the whole authorize, dispense, and capture process right at the dispenser, without coming inside at all. CRIND is so common today that it's almost completely displaced even its immediate ancestor, "fuel island" outdoor payment terminals (OPTs) that provide a central kiosk where customers make payments for multiple dispensers. This used to be a pretty common setup in California where self-service caught on early but, based on my recent travels, has mostly evaporated there.

So you can see that we have a complicated and open-ended set of requirements for communication and automation in the fuel court: enabling and monitoring pumps, collecting card payments, and monitoring and controlling numerous accessories. Most states also require gas stations to have an intercom system so that customers can request help from the attendant inside. Third-party loyalty systems were briefly popular although, mercifully, the more annoying of them have mostly died out... although only because irritating advertising-and-loyalty technology has been better integrated into the dispensers themselves.

Further complicating things, gas station forecourts are the epitome of legacy integration. Fuel dispensers are expensive, concrete slabs are expensive, and gas stations run on thin margins. While there aren't very many manufacturers of fuel dispensers, or multi-product dispensers as they're typically called today, the industry of accessories, control systems, and replacement parts is vast. Most gas stations have accumulated several different generations of control systems and in-dispenser accessories like tree rings. New features like CRIND, chip payment, touchless payment, and "Gas Station TV" have each motivated another round of new communications protocols.

And that's how we get to our modern world, where the brochure for a typical gas station forecourt controller lists 25+ different communications protocols—and assures that you can use "any mix."

Variability between gas stations increases when you consider the differing levels of automation available. It used to be common for gas stations to use standalone pump controllers that didn't integrate with much else—when you prepaid, for example, the cashier would manually enter the pump number and prepayment limit on a separate device from the cash register.

Here in New Mexico, quite a few stations used to use the Triangle MicroSystems MPC family, a wedge-shaped box with an industrial-chic membrane keypad in grey and bright red. Operation of the MPC is pretty simple, basically pressing a pump number and then entering a dollar limit. Of course, the full set of features runs much deeper, including financial reporting and fleet fueling contracts.

This is another important dimension of the gas station control industry: fleet fueling. It used to be that gas stations were divided into two categories, consumer stations that took cash payment and "cardlock" stations that used an electronic payment system. Since cardlock stations originally relied on proprietary, closed payment agreements, they didn't sell to consumers and had different control requirements (often involving an outside payment terminal). As consumers widely adopted card payments, the lines between the two markets blurred. Modern cardlock fueling networks, like CFN and Wex, are largely just another set of payment processors. Most major gas stations participate in most major cardlock networks, just the same as they participate in most major ATM networks for lower-cost processing of debit cards.

Of course, more payment networks call for more integrations. The complexity of the modern payment situation has generally outgrown standalone controllers, and they seem to be fading away. Instead, the typical gas station today has forecourt control completely integrated into their POS system. Forecourt integration is such an important requirement that gas station convenience stores, mostly handling normal grocery-type transactions, nevertheless rely almost exclusively on dedicated gas station POS solutions. In other words, next time you buy a can of Monster and a bag of chips, the cashier most likely rings you up and takes payment through a POS solution offered by the dispenser manufacturer (like Gilbarco Passport Retail) or one of dozens of vendors that caters specifically to gas stations (including compelling names like Petrosoft). Control of fuel dispensers is just too weird of a detail to integrate into other POS platforms... or so it was thought, although things clearly get odd as Gilbarco has to implement basic kitchen video system integration for the modern truck stop.

So how does this all work technically? That's the real topic of fascination, right? Well, it's a mess and hard to describe succinctly. There are so many different options, and particularly legacy retrofit options, that one gas station will be very different from the next.

In the days of "mechanical pumps," simple designs with mechanical counters, control wiring was simple: the dispenser (really a mechanical device called a pulser) was expected to provide "one pulse per penny" on a counting circuit for dollars dispensed, which incremented a synchronized counter on the controller. For control the other way, the controller just closed relays to open "fast" or "slow" valves on the dispenser. The controller might also get a signal when a handle lever is activated, to alert the attendant that someone is trying to use a dispenser, but that was about it.

Later on, particularly as multi-product dispensers with two hoses and four rates (due to diesel and three grades) became common, wiring all the different pulse and valve circuits became frustrating. Besides, pumps with digital counters no longer needed mechanical adjustment when prices changed, allowing for completely centralized price calculation. To simplify wiring while enabling new features, fuel dispenser manufacturers introduced simple current-loop serial buses. These are usually implemented as a single loop that passes through each dispenser, carrying small packets with addressed commands or reports, usually at a pretty low speed. The dispensers designed for use with these systems are much more standalone than the older mechanical dispensers, and perform price accumulation internally, so they only needed to report periodic totals during fueling and at the end of the transaction.

An upside of these more standalone dispensers is that they made CRIND easier to implement: the payment terminal in the dispenser could locally enable the pump, including setting limits, by a direct interface to the pump controller. Still, the CRIND needed some way to actually authorize and report transactions. Solution: another current loop. Most CRIND installations involved a second, similar, but usually higher-speed serial bus that specifically handled payment processing. The CRIND terminals in such a system usually communicated with a back-office payment server using a very simple protocol, sending card information in the clear. That back-office server might be in the back of the convenience store, but it could also be remote.

As gas stations introduced CRIND, plastic card sales became a key part of the business. Card volume is much greater than cash volume at most stations, and it's known that customers will often leave rather than go inside if there is a problem with CRIND payment. So gas stations prioritized reliability of payments. To this day, if you look at the roof of many gas stations, you'll find a small parabolic antenna pointed aimlessly skywards. By the end of the 1990s, many chain gas stations used satellite networks for payment processing, either routinely (cheaper than a leased telephone line!) or as a contingency. Cisco's VSAT terminal modules for edge routers, combined with a boutique industry of Mbps-class data networks on leased transponders, made satellite a fairly inexpensive and easy-to-obtain option for handling small payment processing messages.

This arrangement of one current loop for dispenser control and one current loop for payment terminals lasted for long enough that it became a de facto wiring standard for the gas station forecourt. New construction gas stations provided conduits from the convenience store to the pumps, and those conduits were usually spec'd for an A/C power circuit (controlled, per code, by an emergency stop button) and two low-voltage data circuits. The low-voltage data circuits were particularly important because the electrical code and fire code impose specific rules on electrical systems used in proximity to flammable fluids—what's called a "hazardous environment" in the language of safety codes. Dispenser manufacturers sold specialized field interconnection enclosures that isolated the data circuits to the required safety standard, lowering the complexity of the installation in the dispensers themselves ³.

The next event to challenge forecourt infrastructure was the introduction of EMV chip and tap-to-pay payment cards. Many Americans will remember how fuel dispensers routinely had tap-to-pay terminals physically installed for years, even a decade, before they actually started working. Modernizing dispensers usually meant installing a new CRIND system with EMV support, but upgrades to the underlying network to support them took much longer. The problem was exactly the simplicity of the CRIND current loop design: EMV standards required that all data be encrypted (you couldn't just send card numbers to the backend in the clear as older systems did), and required larger and more numerous messages between the payment network, the terminal, and the card itself. Even if supporting EMV transactions on the serial bus was possible, most manufacturers chose not to, opting for the vastly simpler design of direct IP connectivity to each CRIND terminal.

But how do you put IP over a simple two-wire serial bus? Well, there are a lot of options, and the fuel dispenser industry chose basically all of them. There were proprietary solutions, but more common were IP networking technologies adapted to the forecourt application. Consider DSL: for a good decade, many forecourt interconnection boxes and fuel dispenser controllers supported good old fashioned DSL over the payment loop (not to be confused with DSL as in Diesel, an abbreviation also used in the fuel industry).

Bandwidth requirements increased yet further, though, with the introduction of Maria Menounos. "Forecourt media" advertising systems can deliver full video to each dispenser, a golden opportunity to pitch credit cards and monetize something called "Cheddar." While there was a long era of satellite transponders delivering analog video to chains for in-store marketing (I will one day write about WalMart TV), the "GSTV" phenomenon is newer and completely internet-based. For HD video you need a little better than the 5Mbps performance that industrial DSL systems were delivering. Enter HomePlug.

Despite HomePlug's limited market success, it has been widely adopted in gas station forecourts. The advantage of HomePlug is clear: it dispenses with the control wiring loops entirely, providing IP communications with dispensers over the electrical supply wiring. It usually presents an almost zero-wiring upgrade, just adding HomePlug boards on both ends, so even in stations with good forecourt serial loops, dispenser upgrades often end in a switch to HomePlug.

The most interesting thing about these networks is just how modular it all still is: somewhere in your local gas station, there is a forecourt controller. Depending on the age of the system, that might be a bespoke embedded system with plug-in modules, or it might be a generic N100 Mini PC with a few serial ports and mostly IP connectivity. There is likely a forecourt interconnection box that holds not just the wiring terminals but also adapter boards that convert between various serial protocols, IP carriers, and control signals. The point of sale backend server might interact with the forecourt controller via IP, but older systems used RS-232... and systems in between might use the same logical protocol as they did with RS-232, but encapsulated in TCP. The installation manuals for all of these products include pages of wiring diagrams for each different scenario.

Next time you stop at a gas station and find the CRIND not working, think about all of that: whatever technician comes out to fix it will have their work cut out for them, just to figure out which way that gas station is set up.

The front of the American grocery store contains a strange, liminal space: the transitional area between parking lot and checkstand, along the front exterior and interior of the building, that fills with oddball commodities. Ice is a fixture at nearly every store, filtered water at most, firewood at some. This retail purgatory, both too early and too late in the shopping journey for impulse purchases, is mostly good only for items people know they will need as they check out. One of the standard residents of this space has always struck me as peculiar: dry ice.

Carbon dioxide ice is said to have been invented, or we might better say discovered, in the 1830s. For whatever reason, it took just about a hundred years for the substance to be commercialized. Thomas B. Slate was a son of Oregon, somehow ended up in Boston, and then realized that the solid form of CO2 was both fairly easy to produce and useful as a form of refrigeration. With an eye towards marketing, he coined the name Dry Ice—and founded the DryIce Corporation of America. The year was 1925, and word quickly spread. In a widely syndicated 1930 article, "Use of Carbon Dioxide as Ice Said to be Developing Rapidly," the Alamogordo Daily News and others reported that "the development of... 'concentrated essence of frigidity' for use as a refrigerant in transportation of perishable products, is already taxing the manufacturing facilities of the Nation... So rapidly has the use of this new form of refrigeration come into acceptance that there is not sufficient carbon dioxide gas available."

The rush to dry ice seems strange today, but we must consider the refrigeration technology of the time. Refrigerated transportation first emerged in the US during the middle of the 19th century. Train boxcars, packed thoroughly with ice, carried meat and fruit from midwestern agriculture to major cities. This type of refrigerated transportation greatly expanded the availability of perishables, and the ability to ship fruits and vegetables between growing regions made it possible, for the first time, to get some fresh fruit out of season. Still, it was an expensive proposition: railroads built extensive infrastructure to support the movement of trains loaded down with hundreds of tons of ice. The itself had to be quarried from frozen lakes, some of them purpose-built, a whole secondary seasonal transportation economy.

Mechanical refrigeration, using some kind of phase change process as we are familiar with today, came about a few decades later and found regular use on steamships by 1900. Still, this refrigeration equipment was big and awkward; steam power was a practical requirement. As the Second World War broke out, tens of thousands of refrigerated railcars and nearly 20,000 refrigerated trucks were in service—the vast majority still cooled by ice, not mechanical refrigeration.

You can see, then, the advantages of a "dryer" and lighter form of ice. The sheer weight of the ice significantly reduced the capacity of refrigerated transports. "One pound of carbon dioxide ice at 110 degrees below zero is declared to be equivalent to 16 pounds of water ice," the papers explained, for the purposes of transportation. The use of dry ice could reduce long-haul shipping costs for fruit and vegetables by 50%, the Department of Commerce estimated, and dry ice even opened the door to shipping fresh produce from the West Coast to the East—without having to "re-ice" the train multiple times along the way. Indeed, improvements in refrigeration would remake the American agricultural landscape. Central California was being irrigated so that produce could grow, and refrigeration would bring that produce to market.

1916 saw the American Production Company drilling on the dusty plains of northeastern New Mexico, a few miles south of the town of Bueyeros. On the banks of an anonymous wash, in the shadow of Mesa Quitaras, they hoped to strike oil. Instead, at about 2,000 feet, they struck something else: carbon dioxide. The well blew wide open, and spewed CO2 into the air for about a year, the production estimated at 25,000,000 cubic feet of gas per day under natural pressure. For American Production, this was an unhappy accident. They could identify no market for CO2, and a year later, they brought the well under control, only to plug and abandon it permanently.

Though the "No. 1 Bueyeros" well was a commercial failure at the time, it was not wasted effort. American Production had set the future for northeastern New Mexico. There was oil, if you looked in the right place. American Production found its own productive wells, and soon had neighbors. Whiting Brothers, once operator of charismatic service stations throughout the Southwest and famously along Route 66, had drilled their own wells by 1928. American Production became part of British Petroleum. Breitburn Production of Texas has now consolidated much of the rest of the field, and more than two million cubic feet of natural gas come from northeastern New Mexico each month.

If you looked elsewhere, there was gas—not natural gas, but CO2. Most wells in the region produced CO2 as a byproduct, and the less fortunate attempts yielded nothing but CO2. The clear, non-flammable gas was mostly a nuisance in the 1910s and 1920s. By the 1930s, though, promotion by the DryIce Corporation of America (in no small part through the Bureau of Commerce) had worked. CO2 started to be seen as a valuable commodity.

The production of dry ice is deceptively simple. Given my general knowledge about producing and handling cryogenic gases, I was surprised to read of commercial-scale production with small plants in the 1930s. There is, it turns out, not that much to it. One of the chief advantages of CO2 as an industrial gas is its low critical temperature and pressure. If you take yourself back to high school chemistry, and picture a phase diagram, we can think about liquifying the CO2 gas coming out of a well. The triple point of carbon dioxide, where increasing pressure and temperature will make it a liquid, is at around -60 Celsius and 5 atmospheres. The critical point, beyond which CO2 becomes a supercritical gas-fluid hybrid, is only at 30 degrees Celsius and 72 atmospheres. In terms more familiar to us Americans, that's about 88 degrees F and 1,000 PSI.

In other words, CO2 gas becomes a liquid at temperatures and pressures that were readily achievable, even with the early stages of chemical engineering in the 1930s. With steam-powered chillers and compressors, it wasn't difficult to produce liquid CO2 in bulk. But CO2 makes the next step even more convenient: liquid CO2, released into open air, boils very rapidly. As it bubbles away, the phase change absorbs energy, leaving the remaining liquid CO2 even colder. Some of it freezes into ice, almost like evaporating seawater to extract the salt, evaporating liquid CO2 leaves a snow-like mass of flaky, loose CO2 ice. Scoop that snow up, pack it into forms, and use steam power or weight to compress it, and you have a block of the product we call dry ice.

The Bueyeros Field, as it was initially known, caught the interest of CO2 entrepreneurs in 1931. A company called Timmons Carbonic, or perhaps Southern Dry Ice Company (I suspect these to be two names for the same outfit), produced a well about a mile east, up on the mesa.

Over the next few years, the Estancia Valley Carbon Dioxide Development Company drilled a series of wells to be operated by Witt Ice and Gas. These were located in the Estancia field, further southwest and closer to Albuquerque. Witt built New Mexico's first production dry ice plant, which operated from 1932 to 1942 off of a pipeline from several nearby wells. Low pressure and difficult drilling conditions in the Estancia field limited the plant's output, so by the time it shut down Witt had already built a replacement. This facility, known as the Bueyeros plant, produced 17 tons of dry ice per day starting in 1940. It is located just a couple of miles from the original American Production well, north of Mesa Quitaras.

About 2,000' below the surface at Bueyeros lies the Tubb Sandstone, a loose aggregation of rock stuck below the impermeable Cimarron Anhydrite. Carbon dioxide can form underground through several processes, including the breakdown of organic materials under great heat and pressure (a process that creates petroleum oil as well) and chemical reactions between different minerals, especially when volcanic activity causes rapid mixing with plenty of heat. There are enough mechanisms of formation, either known or postulated, that it's hard to say where exactly the CO2 came from. Whatever its source, the gas flowed upwards underground into the sandstone, where it became trapped under the airtight layer of Anhydrite. It's still there today, at least most of it, and what stands out in particular about northeastern New Mexico's CO2 is its purity. Most wells in the Bueyeros field produce 99% pure CO2, suitable for immediate use.

Near Solano, perhaps 20 miles southwest of Bueyeros by air, the Carbonic Chemical Co built the state's largest dry ice plant. Starting operation in 1942, the plant seems to have initially gone by the name "Dioxice," immortalized as a stop on the nearby Union Pacific branch. Dioxice is an occasional synonym for Dry Ice, perhaps intended to avoid the DryIce Corporation's trademark, although few bothered. The Carbonic Chemical Plant relied on an 18 mile pipeline to bring gas from the Bueyeros field. Uniquely, this new plant used a "high pressure process." By feeding the plant only with wells producing high pressure (hundreds of PSI, as much as 500 PSI of natural pressure at some wells), the pipeline was made more efficient and reliable. Further, the already high pressure of the gas appreciably raised the temperature at which it would liquefy.

The Carbonic Chemical plant's ammonia chillers only had to cool the CO2 to -15 degrees F, liquifying it before spraying it into "snow chambers" that filled with white carbon dioxide ice. A hydraulic press, built directly into the snow chamber, applied a couple of hundred tons of force to create a solid block of dry ice weighing some 180 pounds. After a few saw cuts, the blocks were wrapped in paper and loaded onto insulated train cars for delivery to customers throughout the west—and even some in Chicago.

The main applications of CO2, a 1959 New Mexico Bureau of Mines report explains, were dry ice for shipping. Secondarily, liquid CO2 was shipped in tanks for use in carbonating beverages. Witt Ice and Gas in particular built a good business out of distributing liquid CO2 for beverage and industrial use, and for a time was a joint venture with Chicago-based nationwide gas distributor Cardox. Bueyeros's gas producers found different customers over time, so it is hard to summarize their impact, but we know some salient examples. Most beverage carbonation in mid-century Denver, and perhaps all in Albuquerque, used Bueyeros gas. Dry ice from Bueyeros was used to pack train cars passing through from California, and accompanied them all the way to the major cities of the East Coast.

By the 1950s, much of the product went to a more modern pursuit. Experimental work pursued by the military and the precursors to the Department of Energy often required precise control of low temperatures, and both solid and liquid CO2 were suitable for the purpose. In the late 1950s, Carbonic Chemical listed Los Alamos Scientific Laboratory, Sandia Laboratories, and White Sands Missile Range as their primary customers.

Bueyeros lies in Harding County, New Mexico. Harding County is home to two incorporated cities (Roy and Mosquero), a couple of railroad stops, a few highways, and hardly 650 people. It is the least populous county of New Mexico, but it's almost the size of Delaware. Harding County has never exactly been a metropolis, but it did used to be a more vital place. In the 1930s, as the CO2 industry built out, there were almost 4,500 residents. Since then, the population has declined about 20% from each census to the next.

CO2 production went into a similar decline. After the war, significant improvements in refrigeration technology made mechanical refrigeration inevitable, even for road transportation. Besides, the growing chemical industry had designed many industrial processes that produced CO2 as a byproduct. CO2 for purposes like carbonation and gas blanketing was often available locally at lower prices than shipped-in well CO2, leading to a general decline in the CO2 industry.

Growing understanding of New Mexico geology and a broader reorganizing of the stratigraphic nomenclature lead the Bueyeros Field to become part of the Bravo Dome. Bravo Dome CO2 production in the 1950s and 1960s was likely supported mostly by military and weapons activity, as by the end of the 1960s the situation once again looked much like it did in the 1910s: the Bravo Dome had a tremendous amount of gas to offer, but there were few applications. The rate of extraction was limited by the size of the market. Most of the dry ice plants closed, contributing, no doubt, to the depopulation of Harding County.

The whole idea of drilling for CO2 is now rather amusing. Our modern problems are so much different: we have too much CO2, and we're producing even more without even intending to. It has at times seemed like the industry of the future will be putting CO2 down into the ground, not taking it out. What happened out in Harding County was almost the opening of Pandora's box. A hundred years ago, before there was a dry ice industry in the US, newspaper articles already speculated as to the possibility of global warming by CO2. At the time, it was often presented as a positive outcome: all the CO2 released by burning coal would warm the environment and thus reduce the need for that coal, possibly even a self-balancing problem. It's even more ironic that CO2 was extracted mostly to make things colder, given the longer-term consequences. Given all that, you would be forgiven for assuming that drilling for CO2 was a thing of the past.

The CO2 extraction industry has always been linked to the oil industry, and oil has always been boom and bust. In 1982, there were 16 CO2 wells operating in the Bravo Dome field. At the end of 1985, just three years later, there were 258. Despite the almost total collapse of demand for CO2 refrigeration, demand for liquid CO2 was up by far. It turns out that American Production hadn't screwed up in 1917, at least not if they had known a little more about petroleum engineering.

In 1972, the Scurry Area Canyon Reef Operators Committee of West Texas started an experiment, attempting industrial application of a technique first proposed in the 1950s. Through a network of non-productive oil wells in the Permian Basin, they injected liquid CO2 deep underground. The rapidly evaporating liquid raised the pressure in the overall oil formation, and even lubricated and somewhat fractured the rock, all of which increased the flow rate at nearby oil wells. A decade later, the concept was proven, and CO2 Enhanced Oil Recovery (EOR) swept across the Permian Basin.

Today, it is estimated that about 62% of the global industrial production of CO2 is injected into the ground somewhere in North America to stimulate oil production. The original SACROC system is still running, now up to 414 injection wells. There are thousands more. Every day, over two billion cubic feet of CO2 are forced into the ground, pushing back up 245,000 barrels of additional oil.

British Petroleum's acquisition of American Production proved fortuitous. BP became one of the country's largest producers of CO2, extracted from the ground around Bueyeros and transported by pipeline directly to the Permian Basin for injection. In 2000, BP sold their Bravo Dome operations to Occidental Petroleum ¹. Now going by Oxy, the petroleum giant has adopted a slogan of "Zero In". That's zero as in carbon emissions.

I would not have expected to describe Occidental Petroleum as "woke," but in our contemporary politics they stand out. Oxy mentions "Diversity, Inclusion, and Belonging" on the front page of their website, which was once attractive to investors but now seems more attractive to our nation's increasingly vindictive federal government. Still, Oxy is sticking to a corporate strategy that involves acknowledging climate change as real, which I suppose counts as refreshing. From a 2025 annual report:

Oxy is building an integrated portfolio of low-carbon projects, products, technologies and companies that complement our existing businesses; leveraging our competitive advantages in CO2 EOR, reservoir management, drilling, essential chemicals and major infrastructure projects; and are designed to sustain long term shareholder value as we work to implement our Net-Zero Strategy.

Yes, Oxy has made achieving net-zero carbon a major part of their brand, and yes, this model of reducing carbon emissions relies heavily on CO2 EOR: the extraction of CO2 from the ground.

In a faltering effort to address carbon emissions, the United States has leaned heavily on the promise of Carbon Capture and Storage (CCS) technologies. The idea is to take CO2 out of the environment (potentially by separating it from the air but, more practically, by capturing it in places where it is already concentrated by industrial processes) and to put it somewhere else. Yes, this has shades of the Australian television sketch about the ship whose front fell off, but the key to "sequestration" is time. If we can put enough carbon somewhere that it will say for enough time, we can reduce the "active" greenhouse gas content of our environment. The main way we have found of doing this is injecting it deep underground. How convenient, then, that the oil industry is already looking for CO2 for EOR.

CCS has struggled in many ways, chief among them that the majority of planned CCS projects have never been built. As with most of our modern carbon reduction economy, even the CCS that has been built is, well, a little bit questionable. There is something of a Faustian bargain with fossil fuels. As we speak, about 45 megatons of CO2 are captured from industrial processes each year for CCS. Of that 45 Mt, 9 Mt are injected into dedicated CO2 sequestration projects. The rest, 80%, is purchased by the oil industry for use in EOR.

This form of CCS, in which the captured CO2 is applied to an industrial process that leads to the production of more CO2, has taken to the name CCUS. That's Carbon Capture, Utilization, and Storage. Since the majority of the CO2 injected for EOR never comes back up, it is a form of sequestration. Although the additional oil produced will generally be burned, producing CO2, the process can be said to be inefficient in terms of CO2. In other words, the CO2 produced by burning oil from EOR is less in volume than the CO2 injected to stimulate recovery of that oil.

Mathematically, CCUS, the use of CO2 to produce oil, leads to a net reduction in released CO2. Philosophically, though, it is deeply unsatisfying. This is made all the worse by the fact that CCUS has benefited from significant government support. Outright subsidies for CCS are uncommon, although they do exist. What are quite common are grants and subsidized financing for the capital costs of CCS facilities. Nearly all CCS in the US has been built with some degree of government funding, totaling at least four billion dollars, and regulatory requirements for CCS to offset new fossil fuel plants may create a de facto electrical ratepayer subsidy for CCS. Most of that financial support, intended for our low-carbon future, goes to the oil producers.

The Permian Basin is well-positioned for CCS EOR because it produces mostly natural gas. Natural gas in its raw form, "well gas," almost always includes CO2. Natural gas processing plants separate the combustible gases from noncombustible ones, producing natural gas that has a higher energy content and burns more cleanly—but, in the process, venting large quantities of CO2 into the atmosphere. Oxy is equipping its Permian Basin natural gas plants with a capture system that collects the CO2 and compresses it for use in EOR.

The problem is that CO2 consumption for EOR has, as always, outpaced production. There aren't enough carbon capture systems to supply the Permian Basin fields, so "sequestered" CO2 is mixed with "new" CO2. Bravo Dome CO2 production has slowly declined since the 1990s, due mostly to declining oil prices. Even so, northeastern New Mexico is still full of Oxy wells bringing up CO2 by the millions of cubic feet. 218 miles of pipeline deliver Bueyeros CO2 into West Texas, and 120 miles of pipeline the other way land it in the oil fields of Wyoming. There is very nearly one producing CO2 well per person in Harding County.

Considering the totality of the system, it appears that government grants, financing incentives, and tax credits for CCS are subsidizing not only natural gas production but the extraction of CO2 itself. Whether this is progress on climate change or a complete farce depends a mathematical analysis. CO2 goes in, from several different sources; CO2 goes out, to several different dispositions. Do we remove more from the atmosphere than we end up putting back? There isn't an obvious answer.

The oil industry maintains that CCS is one of the most practical means of reducing carbon emissions, with more CO2 injected than produced and a resulting reduction in the "net CO2 impact" of the product natural gas.

As for more independent researchers, well, a paper finding that CCS EOR "cannot contribute to reductions" isn't the worst news. A 2020 literature review of reports on CCS EOR projects found that they routinely fail to account for significant secondary carbon emissions and that, due to a mix of the construction and operational realities of CCS EOR facilities and the economics of oil consumption, CCS EOR has so far produced a modest net increase in greenhouse gas emissions.

They're still out there today, drilling for carbon dioxide. The reports from the petroleum institute today say that the Permian Basin might need even more shipped in. New Mexico is an oil state; Texas gets the reputation but New Mexico has the numbers. Per-capita oil production here is significantly higher than Texas and second only to North Dakota. New Mexico now produces more oil than Old Mexico, if you will, the country to our south.

Per capita, New Mexico ranks 12th for CO2 emissions, responsible for about 1% of the nation's total. Well, I can do a bit better: for CO2 intentionally extracted from the ground, New Mexico is #3, behind only Colorado and Mississippi for total production. We produce something around 17% of the nation's supply of extracted CO2, and we even use most of it locally. I guess that's something you could put a good spin on.

Previously on Computers Are Bad, we discussed the early history of air traffic control in the United States. The technical demands of air traffic control are well known in computer history circles because of the prominence of SAGE, but what's less well known is that SAGE itself was not an air traffic control system at all. SAGE was an air defense system, designed for the military with a specific task of ground-controlled interception (GCI). There is natural overlap between air defense and air traffic control: for example, both applications require correlating aircraft identities with radar targets. This commonality lead the Federal Aviation Agency (precursor to today's FAA) to launch a joint project with the Air Force to adapt SAGE for civilian ATC.

There are also significant differences. In general, SAGE did not provide any safety functions. It did not monitor altitude reservations for uniqueness, it did not detect loss of separation, and it did not integrate instrument procedure or terminal information. SAGE would need to gain these features to meet FAA requirements, particularly given the mid-century focus on mid-air collisions (a growing problem, with increasing air traffic, that SAGE did nothing to address).

The result was a 1959 initiative called SATIN, for SAGE Air Traffic Integration. Around the same time, the Air Force had been working on a broader enhancement program for SAGE known as the Super Combat Center (SCC). The SCC program was several different ideas grouped together: a newer transistorized computer to host SAGE, improved communications capabilities, and the relocation of Air Defense Direction Centers from conspicuous and vulnerable "SAGE Blockhouses" to hardened underground command centers, specified as an impressive 200 PSI blast overpressure resistance (for comparison, the hardened telecommunication facilities of the Cold War were mostly specified for 6 or 10 PSI).

At the program's apex, construction of the SCCs seemed so inevitable that the Air Force suspended the original SAGE project under the expectation that SCC would immediately obsolete it. For example, my own Albuquerque was one of the last Air Defense Sectors scheduled for installation of a SAGE computer. That installation was canceled; while a hardened underground center had never been in the cards for Albuquerque, the decision was made to otherwise build Albuquerque to the newer SCC design, including the transistorized computer. By the same card, the FAA's interest in a civilian ATC capability, and thus the SATIN project, came to be grouped together with the SCC program as just another component of SAGE's next phase of development.

SAGE had originally been engineered by MIT's Lincoln Laboratory, then the national center of expertise in all things radar. By the late 1950s a large portion of the Lincoln Laboratory staff were working on air defense systems and specifically SAGE. Those projects had become so large that MIT opted to split them off into a new organization, which through some obscure means came to be called the MITRE Corporation. MITRE was to be a general military R&D and consulting contractor, but in its early years it was essentially the SAGE company.

The FAA contracted MITRE to deliver the SATIN project, and MITRE subcontracted software to the Systems Development Corporation, originally part of RAND and among the ancestors of today's L3Harris. For the hardware, MITRE had long used IBM, who designed and built the original AN/FSQ-7 SAGE computer and its putative transistorized replacement, the AN/FSQ-32. MITRE began a series of engineering studies, and then an evaluation program on prototype SATIN technology.

There is a somewhat tenuous claim that you will oft see repeated, that the AN/FSQ-7 is the largest computer ever built. It did occupy the vast majority of the floorspace of the four-story buildings built around it. The power consumption was around 3 MW, and the heat load required an air conditioning system at the very frontier of HVAC engineering (you can imagine that nearly all of that 3 MW had to be blown out of the building on a continuing basis). One of the major goals of the AN/FSQ-32 was reduced size and power consumption, with the lower heat load in particular being a critical requirement for installation deep underground. Of course, the "deep underground" part more than wiped out any savings from the improved technology.

From Air Defense to Air Traffic Control

By the late 1950s, enormous spending for the rapid built-out of defense systems including SAGE and the air defense radar system (then the Permanent System) had fatigued the national budget and Congress. The winds of the Cold War had once again changed. In 1959, MITRE had begun operation of a prototype civilian SAGE capability called CHARM, the CAA High Altitude Remote Monitor (CAA had become the FAA during the course of the CHARM effort). CHARM used MIT's Whirlwind computer to process high-altitude radar data from the Boston ARTCC (Air Route Traffic Control Center), which it displayed to operators while continuously evaluating aircraft movements for possible conflicts. CHARM was designed for interoperability with SAGE, the ultimate goal being the addition of the CHARM software package to existing SAGE computers. None of that would ever happen; by the time the ball dropped for the year 1960 the Super Combat Center program had been almost completely canceled. SATIN, and the whole idea of civilian air traffic control with SAGE, became blast damage.

In 1961, the Beacon Report concluded that there was an immediate need for a centralized, automated air traffic control system. Mid-air collisions had become a significant political issue, subject of congressional hearings and GAO reports. The FAA seemed to be failing to rise to the task of safe civilian ATC, a perilous situation for such a new agency... and after the cancellation of the SCCs, the FAA's entire plan for computerized ATC was gone.

During the late 1950s and 1960s, the FAA adopted computer systems in a piecemeal fashion. Many enroute control centers (ARTCCs), and even some terminal facilities, had some type of computer system installed. These were often custom software running on commodity computers, limited to tasks like recording flight plans and making them available to controllers at other terminals. Correlation of radar targets with flight plans was generally manual, as were safety functions like conflict detection.

These systems were limited in scale—the biggest problem being that some ARTCCs remained completely manual even in the late 1960s. On the upside, they demonstrated much of the technology required, and provided a test bed for implementation. Many of the individual technical components of ATC were under development, particularly within IBM and Raytheon, but there was no coordinated nationwide program. This situation resulted in part from a very intentional decision by the FAA to grant more decision making power to its regional offices, a concept that was successful in some areas but in retrospect disastrous in others. In 1967, the Department of Transportation was formed as a new cabinet-level executive department. The FAA, then the Federal Aviation Agency, was reorganized into DOT and renamed the Federal Aviation Administration. The new Administration had a clear imperative from both the President and Congress: figure out air traffic control.

In the late 1960s, the FAA coined a new term: the National Airspace System ¹, a fully standardized, nationwide system of procedures and systems that would safely coordinate air traffic into the indefinite future. Automation of the NAS began with NAS Enroute Stage A, which would automate the ARTCCs that handled high-altitude aircraft on their way between terminals. The remit was more or less "just like SAGE but with the SATIN features," and when it came to contracting, the FAA decided to cut the middlemen and go directly to the hardware manufacturer: IBM.

The IBM 9020

It was 1967 by the time NAS Enroute Stage A was underway, nearly 20 years since SAGE development had begun. IBM would thus benefit from considerable advancements in computer technology in general. Chief among them was the 1965 introduction of the System/360. S/360 was a milestone in the development of the computer: a family of solid-state, microcoded computers with a common architecture for software and peripheral interconnection. S/360's chief designer, Gene Amdahl, was a genius of computer architecture who developed a particular interest in parallel and multiprocessing systems. Soon after the S/360 project, he left IBM to start the Amdahl Corporation, briefly one of IBM's chief competitors. During his short 1960s tenure at IBM, though, Amdahl contributed IBM's concept of the "multisystem."

A multisystem consisted of multiple independent computers that operated together as a single system. There is quite a bit of conceptual similarity between the multisystem and modern concepts like multiprocessing and distributed computing, but remember that this was the 1960s, and engineers were probing out the possibilities of computer-to-computer communication for the first time. Some of the ideas of S/360 multisystems read as strikingly modern and prescient of techniques used today (like atomic resource locking for peripherals and shared memory), while others are more clearly of their time (the general fact that S/360 multisystems tended to assign their CPUs exclusively to a specific task).

One of the great animating tensions of 1960s computer history is the ever-moving front between batch processing systems and realtime computing systems. IBM had its heritage manufacturing unit record data processing machines, in which a physical stack of punched cards was the unit of work, and input and output ultimately occurred between humans on two sides of a service window. IBM computers were designed around the same model: a "job" was entered into the machine, stored until it reached the end of the queue, processed, and then the output was stored for later retrieval. One could argue that all computers still work this way, it's just process scheduling, but IBM had originally envisioned job queuing times measured in hours rather than milliseconds.

The batch model of computing was fighting a battle on multiple fronts: rising popularity of time-sharing systems meant servicing multiple terminals simultaneously and, ideally, completing simple jobs interactively while the user waited. Remote terminals allowed clerks to enter and retrieve data right where business transactions were taking place, and customers standing at ticket counters expected prompt service. Perhaps most difficult of all, fast-moving airplanes and even faster-moving missiles required sub-second decisions by computers in defense applications.

IBM approached the FAA's NAS Enroute Stage A contract as one that required a real-time system (to meet the short timelines necessary in air traffic control) and a multisystem (to meet the FAA's exceptionally high uptime and performance requirements). They also intended to build the NAS automation on an existing, commodity architecture to the greatest extent possible. The result was the IBM 9020.

The 9020 is a fascinating system, exemplary of so many of the challenges and excitement of the birth of the modern computer. On the one hand, a 9020 is a sophisticated, fault-tolerant, high-performance computer system with impressive diagnostic capabilities and remarkably dynamic resource allocation. On the other hand, a 9020 is just six to seven S/360 computers married to each other with a vibe that is more duct tape and bailing wire than aerospace aluminum and titanium.

The first full-scale 9020 was installed in Jacksonville, Florida, late in 1967. Along with prototype systems at the FAA's experimental center and at Raytheon (due to the 9020's close interaction with Raytheon-built radar systems), the early 9020 computers served as development and test platforms for a complex and completely new software system written mostly in JOVIAL. JOVIAL isn't a particularly well-remembered programming language, based on ALGOL with modifications to better suit real-time computer systems. The Air Force was investing extensively in real-time computing capabilities for air defense and JOVIAL was, for practical purposes, an Air Force language.

It's not completely clear to me why IBM selected JOVIAL for enroute stage A, but we can make an informed guess. There were very few high-level programming languages that were suitable for real-time use at all in the 1960s, and JOVIAL had been created by Systems Development Corporation (the original SAGE software vendor) and widely used for both avionics and air defense. The SCC project, if it had been completed, would likely have involved rewriting large parts of SAGE in JOVIAL. For that reason, JOVIAL had been used for some of the FAA's earlier ATC projects including SATIN. At the end of the day, JOVIAL was probably an irritating (due to its external origin) but obvious choice for IBM.

More interesting than the programming language is the architecture of the 9020. It is, fortunately, well described in various papers and a special issue of IBM Systems Journal. I will simplify IBM's description of the architecture to be more legible to a modern reader who hasn't worked for IBM for a decade.

Picture this: seven IBM S/360 computers, of various models, are connected to a common address and memory bus used for interaction with storage. These computers are referred to as Compute Elements and I/O Control Elements, forming two pools of machines dedicated to two different sets of tasks. Also on that bus are something like 10 Storage Elements, specialized machines that function like memory controllers with additional features for locking, prioritization, and diagnostics. These Storage Elements provide either 131 kB or about 1 MB of memory each; due to various limitations the maximum possible memory capacity of a 9020 is about 3.4 MB, not all of which is usable at any given time due to redundancy.

At least three Compute Elements, and up to four, serve as the general-purpose part of the system where the main application software is executed. Three I/O Control Elements existed mostly as "smart" controllers for peripherals connected to their channels, the IBM parlance for what we might now call an expansion bus.

The 9020 received input from a huge number of sources (radar digitizers, teletypes at airlines and flight service stations, controller workstations, other ARTCCs). Similarly, it sent output to most of these endpoints as well. All of these communications channels, with perhaps the exception of the direct 9020-to-9020 links between ARTCCs, were very slow even by the standards of the time. The I/O Control Elements each used two of their high-speed channels for interconnection with display controllers (discussed later) and tape drives in the ARTCC, while the third high-speed channel connected to a multiplexing system called the Peripheral Adapter Module that connected the computer to dozens of peripherals in the ARTCC and leased telephone lines to radar stations, offices, and other ATC sites.

Any given I/O Control Element had a full-time job of passing data between peripherals and storage elements, with steps to validate and preprocess data. In addition to ATC-specific I/O devices, the Control Elements also used their Peripheral Adapter Modules to communicate with the System Console. The System Console is one of the most unique properties of the 9020, and one of the achievements of which IBM seems most proud.

Multisystem installations of S/360s were not necessarily new, but the 9020 was one of the first attempts to present a cluster of S/360s as a single unified machine. The System Console manifested that goal. It was, on first glance, not that different from the operator's consoles found on each of the individual S/360 machines. It was much more than that, though: it was the operator's console for all seven of them. During normal 9020 operation, a single operator at the system console could supervise all components of the system through alarms and monitors, interact with any element of the system via a teletypewriter terminal, and even manually interact with the shared storage bus for troubleshooting and setup. The significance of the System Console's central control was such that the individual S/360 machines, when operating as part of the Multisystem, disabled their local operator's consoles entirely.

One of the practical purposes of the System Console was to manage partitioning of the system. A typical 9020 had three compute elements and three I/O control elements, an especially large system could have a fourth compute element for added capacity. The system was sized to produce 50% redundancy during peak traffic. In other words, a 9020 could run the full normal ATC workload on just two of the compute elements and two of the I/O control elements. The remaining elements could be left in a "standby" state in which the multisystem would automatically bring them online if one of the in-service elements failed, and this redundancy mechanism was critical to meeting the FAA's reliability requirement. You could also use the out-of-service elements for other workloads, though.

For example, you could remove one of the S/360s from the multisystem and then operate it manually or run "offline" software. An S/360 operating this way is described as "S/360 compatibility mode" in IBM documentation, since it reduces the individual compute element to a normal standalone computer. IBM developed an extensive library of diagnostic tools that could be run on elements in standby mode, many of which were only slight modifications of standard S/360 tools. You could also use the offline machines in more interesting ways, by bringing up a complete ATC software chain running on a smaller number of elements. For training new controllers, for example, one compute element and one I/O control element could be removed from the multisystem and used to form a separate partition of the machine that operated on recorded training data. This partition could have its own assigned peripherals and storage area and largely operate as if it were a complete second 9020.

Multisystem Architecture

You probably have some questions about how IBM achieved these multisystem capabilities, given the immature state of operating systems design at the time. The 9020 used an operating system derived from OS/360 MVT, an advanced form of OS/360 with a multitasking capability that was state-of-the-art in the mid-1960s but nonetheless very limited and with many practical problems. Fortunately, IBM was not exactly building a general-purpose machine, but a dedicated system with one function. This allowed the software to be relatively simple.

The core of the 9020 software system is called the control program, which is similar to what we would call a scheduler today. During routine operation of the 9020, any of the individual computers might begin execution of the control program at any time—typically either because the computer's previous task was complete (along the lines of cooperative multitasking) or because an interrupt had been received (along the lines of preemptive multitasking). To meet performance and timing requirements, especially with the large number of peripherals involved, the 9020 extensively used interrupts which could either be generated and handled within a specific machine or sent across the entire multisystem bus.

The control program's main function is to choose the next task to execute. Since it can be started on any machine at any time, it must be reentrant. The fact that all of the machines have shared memory simplifies the control program's task, since it has direct access to all of the running programs. Shared memory also added the complexity that the control program has to implement locking and conflict detection to ensure that it doesn't start the same task on multiple machines at once, or start multiple tasks that will require interaction with the same peripheral.

You might wonder about how, exactly, the shared memory was implemented. The storage elements were not complete computers, but did implement features to prevent conflicts between simultaneous access by two machines, for example. By necessity, all of the memory management used for the multisystem is quite simple. Access conflicts were resolved by choosing one machine and making the other wait until the next bus cycle. Each machine had a "private" storage area, called the preferential storage area. A register on each element contained an offset added to all memory addresses that ensured the preferential storage areas did not overlap. Beyond that, all memory had to be acquired by calling system subroutines provided by the control program, so that the control program could manage memory regions. Several different types of memory allocations were available for different purposes, ranging from arbitrary blocks for internal use by programs to shared buffer areas that multiple machines could use to queue data for an I/O Control Element to send elsewhere.

At any time during execution of normal programs, an interrupt could be generated indicating a problem with the system (IBM gives the examples of a detection of high temperature or loss of A/C power in one of the compute elements). Whenever the control program began execution, it would potentially detect other error conditions using its more advanced understanding of the state of tasks. For example, the control program might detect that a program has exited abnormally, or that allocation of memory has failed, or an I/O operation has timed out without completing. All of these situations constitute operational errors, and result in the Control Program ceding execution to the Operational Error Analysis Program or OEAP.

The OEAP is where error-handling logic lives, but also a surprising portion of the overall control of the multisystem. The OEAP begins by performing self-diagnosis. Whatever started the OEAP, whether the control program or a hardware interrupt, is expected to leave some minimal data on the nature of the failure in a register. The OEAP reads that register and then follows an automated data-collection procedure that could involve reading other registers on the local machine, requesting registers from other machines, and requesting memory contents from storage elements. Based on the diagnosis, the OEAP has different options: some errors are often transient (like communications problems), so the OEAP might do nothing and simply allow the control program to start the task again.

On the other hand, some errors could indicate a serious problem with a component of the system, like a storage element that is no longer responding to read and write operations in its address range. In those more critical cases, the OEAP will rewrite configuration registers on the various elements of the system and then reset them—and on initialization, the configuration registers will cause them to assume new states in terms of membership in the multisystem. In this way, the OEAP is capable of recovering from "solid" hardware failures by simply reconfiguring the system to no longer use the failed hardware. Most of the time, that involves changing the failed element's configuration from "online" to "offline," and choosing an element in "online standby" and changing its configuration to "online." During the next execution of the control program, it will start tasks on the newly "online" element, and the newly "offline" element may as well have never existed.

The details are, of course, a little more complex. In the case of a failed storage element, for example, there's a problem of memory addresses. The 9020 multisystem doesn't have virtual memory in the modern sense, addresses are more or less absolute (ignoring some logical addressing available for specific types of memory allocations). That means that if a storage element fails, any machines which have been using memory addresses within that element will need to have a set of registers for memory address offsets reconfigured and then execution reset. Basically, by changing offsets, the OEAP can "remap" the memory in use by a compute or I/O control element to a different storage element. Redundancy is also built in to the software design to make these operations less critical. For example, some important parts of memory are stored in duplicate with an offset between the two copies large enough to ensure that they will never fall on the same physical storage element.

So far we have only really talked about the "operational error" part, though, and not the "analysis." In the proud tradition of IBM, the 9020 was designed from the ground up for diagnosis. A considerable part of IBM's discussion of the architecture of the Control Program, for example, is devoted to its "timing analysis" feature. That capability allows the Control Program to commit to tape a record of when each task began execution, on which element, and how long it took. The output is a set of start and duration times, with task metadata, remarkably similar to what we would now call a "span" in distributed tracing. Engineers used these records to analyze the performance of the system and more accurately determine load limits such as the number of in-air flights that could be simultaneously tracked. Of course, details of the time analysis system remind us that computers of this era were very slow: the resolution on task-start timestamps was only 1/2 second, although durations were recorded at the relatively exact 1/60th of a second.

That was just the control program, though, and the system's limited ability to write timing analysis data (which, even on the slow computers, tended to be produced faster than the tape drives could write it and so had to fit within a buffer memory area for practical purposes) meant that it was only enabled as needed. The OEAP provided long-term analysis of the performance of the entire machine. Whenever the OEAP was invoked, even if it determined that a problem was transient or "soft" and took no action, it would write statistical records of the nature of the error and the involved elements. When the OEAP detected an unusually large number of soft errors from the same physical equipment, it would automatically reconfigure the system to remove that equipment from service and then generate an alarm.

Alerts generated by the OEAP were recorded by a printer connected to the System Console, and indicated by lights on the System Console. A few controls on the System Console allowed the operator manual intervention when needed, for example to force a reconfiguration.

One of the interesting aspects of the OEAP is where it runs. The 9020 multisystem is truly a distributed one in that there is no "leader." The control program, as we have discussed, simply starts on whichever machine is looking for work. In practice, it may sometimes run simultaneously on multiple machines, which is acceptable as it implements precautions to prevent stepping on its own toes.

This model is a little more complex for the OEAP, because of the fact that it deals specifically with failures. Consider a specific failure scenario: loss of power. IBM equipped each of the functional components of the 9020 with a battery backup, but they only rate the battery backup for 5.5 seconds of operation. That isn't long enough for a generator to reliably pick up the load, so this isn't a UPS as we would use today. It's more of a dying gasp system: the computer can "know" that it has lost power and continue to operate long enough stabilize the state of the system for faster resumption.

When a compute element or I/O control element loses power, an interrupt is generated within that single machine that starts the OEAP. The OEAP performs a series of actions, which include generating an interrupt across the entire system to trigger reconfiguration (it is possible, even likely given the physical installations, that the power loss is isolated to the single machine) and resetting task states in the control program so that the machine's tasks can be restarted elsewhere. The OEAP also informs the system console and writes out records of what has happened. Ideally, this all completes in 5.5 seconds while battery power remains reliable.

In the real world, there could be problems that lead to slow OEAP execution, or the batteries could fail to make it for long enough, or for that matter the compute element could encounter some kind of fundamentally different problem. The fact that the OEAP is executing on a machine means that something has gone wrong, and so until the OEAP completes analysis, the machine that it is running on should be considered suspect. The 9020 resolves this contradiction through teamwork: beginning of OEAP execution on any machine in the total system generates an interrupt that starts the OEAP on other machines in a "time-down" mode. The "time-down" OEAPs wait for a random time interval and then check the shared memory to see if the original OEAP has marked its execution as completed. If not, the first OEAP to complete its time-down timer will take over OEAP execution and attempt to complete diagnostics from afar. That process can, potentially, repeat multiple times: in some scenario where two of the three compute elements have failed, the remaining third element will eventually give up on waiting for the first two and run the OEAP itself. In theory, someone will eventually diagnose every problem. IBM asserts that system recovery should always complete within 30 seconds.

Let's work a couple of practical examples, to edify our understanding of the Control Program and OEAP. Say that a program running on a Compute Element sets up a write operation for an I/O Control Element, which formats and sends the data to a Peripheral Adapter Module which attempts to send it to an offsite peripheral (say an air traffic control tower teleprinter) but fails. A timer that tracks the I/O operation will eventually fail, triggering the OEAP on the I/O control element running the task. The OEAP reads out the error register on its new home, discovers that it is an I/O problem related to a PAM, and then speaks over the channel to request the value of state registers from the PAM. These registers contain flags for various possible states of peripheral connections, and from these the OEAP can determine that sending a message has failed because there was no response. These types of errors are often transient, due to telephone network trouble or bad luck, so the OEAP increments counters for future reference, looks up the application task that tried to send the message and changes its state to incomplete, clears registers on the PAM and I/O control element, and then hands execution back to the Control Program. The Control Program will most likely attempt to do the exact same thing over again, but in the case of a transient error, it'll probably work this time.

Consider a more severe case, where the Control Program starts a task on a Compute Element that simply never finishes. A timer runs down to detect this condition, and an interrupt at the end of the timer starts the Control Program, which checks the state and discovers the still-unfinished task. Throwing its hands in the air, the Control Program sets some flags in the error register and hands execution to the OEAP. The OEAP starts on the same machine, but also interrupts other machines to start the OEAP in time-down mode in case the machine is too broken to complete error handling. It then reads the error register and examines other registers and storage contents. Determining that some indeterminate problem has occurred with the Compute Element, the OEAP triggers what IBM confusingly calls a "logout" but we might today call a "core dump" (ironically an old term that was more appropriate in this era). The "logout" entails copying the contents of all of the registers and counters to the preferential storage area and then directing, via channel, one of the tape drives to write it all to a tape kept ready for this purpose—the syslog of its day. Once that's complete, the OEAP will reset the Compute Element and hand back to the Control Program to try again... unless counters indicate that this same thing has happened recently. In that case, the OEAP will update the configuration register on the running machine to change its status to offline, and choose a machine in online-standby. It will write to that machine's register, changing its status to online. A final interrupt causes the Control Program to start on both machines, taking them into their new states.

Lengthy FAA procedure manuals described what would happen next. These are unfortunately difficult to obtain, but from IBM documentation we know that basic information on errors was printed for the system operator. The system operator would likely then use the system console to place the suspicious element in "test" mode, which completely isolates it to behave more or less like a normal S/360. At that point, the operator could use one of the tape drives attached to the problem machine to load IBM's diagnostic library and perform offline troubleshooting. The way the tape drives are hooked up to specific machines is important; in fact, since the OEAP is fairly large, it is only stored in one copy on one Storage Element. The 9020 requires that one of the tape drives always have a "system tape" ready with the OEAP itself, and low-level logic in the elements allows the OEAP to be read from the ready-to-go system tape in case the storage element that contains it fails to respond.

A final interesting note about the OEAP is a clever optimization called "problem program mode." During analysis and handling of an error, the OEAP can decide that the critical phase of error handling has ended and the situation is no longer time sensitive. For example, the OEAP might decide that no action is required except for updating statistics, which can comfortably happen with a slight delay. These lower-priority remaining tasks can be added to memory as "normal" application tasks, to be run by the Control Program like any other task after error handling is complete. Think of it as a deferral mechanism, to avoid the OEAP locking up a machine for any longer than necessary.

For the sake of clarity, I'll note again an interesting fact by quoting IBM directly: "OEAP has sole responsibility for maintaining the system configuration." The configuration model of the 9020 system is a little unintuitive to me. Each machine has its own configuration register that tells it what its task is and whether it is online or offline (or one of several states in between like online-standby). The OEAP reconfigures the system by running on any one machine and writing the configuration registers of both the machine it's running on, and all of the other machines via the shared bus. Most reconfigurations happen because the OEAP has detected a problem and is working around it, but if the operator manually reconfigures the system (for example to facilitate testing or training), they also do so by triggering an interrupt that leads the Control Program to start the OEAP. The System Console has buttons for this, along with toggles to set up a sort of "main configuration register" that determines how the OEAP will try to set up the system.

The Air Traffic Control Application

This has become a fairly long article by my norms, and I haven't even really talked about air traffic control that much. Well, here it comes: the application that actually ran on the 9020, which seems to have had no particular name, besides perhaps Central Computing Complex (although this seems to have been adopted mostly to differentiate it from the Display Complex, discussed soon).

First, let's talk about the hardware landscape of the ARTCC and the 9020's role. An ARTCC handles a number of sectors, say around 30. Under the 9020 system, each of these sectors has three controllers associated with it, called the R, D, and A controllers. The R controller is responsible for monitoring and interpreting the radar, the D controller for managing flight plans and flight strips, and the A controller is something of a generalist who assists the other two. The three people sit at something like a long desk, made up of the R, D, and A consoles side by side.

The R console is the most recognizable to modern eyes, as its centerpiece is a 22" CRT plan-view radar display. The plan-view display (PVD) of the 9020 system is significantly more sophisticated than the SAGE PVD on which it is modeled. Most notably, the 9020 PVD is capable of displaying text and icons. No longer does a controller use a light gun to select a target for a teleprinter to identify; the "data blocks" giving basic information on a flight were actually shown on the PVD next to the radar contact. A trackball and a set of buttons even allowed the controller to select targets to query for more information or update flight data. This was quite a feat of technology even in 1970, and in fact one that the 9020 was not capable of. Well, it was actually capable of it, but not initially.

The original NAS stage A architecture separated the air traffic control data function and radar display function into two completely separate systems. The former was contracted to IBM, the latter to Raytheon, due to their significant experience building similar systems for the military. Early IBM 9020 installations sat alongside a Raytheon 730 Display Channel, a very specialized system that was nearly as large as the 9020. The Display Channel's role was to receive radar contact data and flight information in digital form from the 9020, and convert it into drawing instructions sent over a high-speed serial connection to each individual PVD. A single Display Channel was responsible for up to 60 PVDs. Further complicating things, sector workstations were reconfigurable to handle changing workloads. The same sector might be displayed on multiple PVDs, and where sectors met, PVDs often overlapped so the same contact would be visible to controllers for both sectors. The Display Channel had a fairly complex task to get the right radar contacts and data blocks to the right displays, and in the right places.

Later on, the FAA opted to contract IBM to build a slightly more sophisticated version of the Display Channel that supported additional PVDs and provided better uptime. To meet that contract, IBM used another 9020. Some ARTCCs thus had two complete 9020 systems, called the Central Computer Complex (CCC) and the Display Channel Complex (DCC).

The PVD is the most conspicuous part of the controller console, but there's a lot of other equipment there, and the rest of it is directly connected to the 9020 (CCC). At the R controller's position, a set of "hotkeys" allow for quickly entering flight data (like new altitudes) and a computer readout device (CRD), a CRT that displays 25x20 text for general output. For example, when a controller selects a target on the PVD to query for details, that query is sent to the 9020 CCC which shows the result on the R controller's CRD above the PVD.

At the D controller's position, right next door, a large rack of slots for flight strips (small paper strips used to logically organize flight clearances, still in use today in some contexts) surrounds the D controller's CRD. The D controller also has a Computer Entry Device, or CED, a specialized keyboard that allows the D controller to retrieve and update flight plans and clearances based on requests from pilots or changes in the airspace situation. To their right, a modified teleprinter is dedicated to producing the flight strips that they arrange in front of them. Flight strips are automatically printed when an aircraft enters the sector, or when the controller enters changes. The A controller's position to the right of the flight strip printer is largely the same as the D controller's position, with another CRD and CED that operate independently from the D controller's—valuable during peak traffic.

While controller consoles are the most visible peripherals of the system, they're far from the only ones. Each 9020 system had an extensive set of teletypewriter circuits. Some of these were local; for example, the ATC supervisor had a dedicated TTY where they could not only interact with flight data (to assist a sector controller for example) but also interact with the status of the NAS automation itself (for example to query the status of a malfunctioning radar site and then remove it from use for PVDs).

Since the 9020 was also the locus of flight planning, TTYs were provided in air traffic control towers, terminal radar facilities, and even the dispatch offices of airlines. These allowed flight plans to be entered into the 9020 before the aircraft was handed off to enroute control. Flight service stations functioned more or less as the dispatch offices for general aviation, so they were similarly equipped with TTYs for flight plan management. In many areas, military controllers at air defense sectors were also provided with TTYs for convenient access to flight plans. Not least of all, each 9020 had high-speed leased lines to its neighboring 9020s. Flights passing from one ARTCC to the next had their flight strip "digitally passed" by transmission from one 9020 to the next.

A set of high-speed line printers connected to the 9020 printed diagnostic data as well as summary and audit reports on air traffic. Similar audit data, including a detailed record of clearances, was written to tape drives for future reference.

To organize the whole operation, IBM divided the software architecture of the system into the "supervisor state" and the "problem state." These are reasonably analogous to kernel and user space today, and "problem" is meant as in "the problem the computer solves" rather than "a problem has occurred." The Control Program and OEAP run in the supervisor state, everything else runs after the Control Program has set up a machine in the Problem State and started a given program.

IBM organized the application software into five modules, which they called the five Programs. These are Input Processing, Flight Processing, Radar Processing, Output Processing, and Liaison Management. Most of these are fairly self-explanatory, but the list reveals the remarkably asynchronous design of the system. Consider an example, we'll say a general aviation flight taking off from an airport inside of one of the ARTCC's sectors.

The pilot first contacts a Flight Service Station, which uses their TTY to enter a flight plan into the 9020. Next, the pilot interacts with the control tower, which in the process of giving a takeoff clearance uses their TTY to inform the 9020 that the flight plan is active. They may also update the flight plan with the aircraft's planned movements shortly after takeoff, if they have changed due to operating conditions. The Input Processing program handles all of these TTY inputs, parsing them into records stored on a Storage Element. In case any errors occur, like an invalid entry, those are also written to the Storage Element, where the Output Processing program picks them up and sends an appropriate message to the originating TTY. IBM notes that there were, as originally designed, about 100 types of input messages parsed by the input processing program.

As the aircraft takes off, it is detected by a radar site (such as a Permanent System radar or Air Route Surveillance Radar) which digitally encodes the radar contact (a Raytheon system) for transmission to the 9020. The Radar Processing program receives these messages, converts radial radar coordinates to the XY plane used by the system, correlates contacts with similar XY positions from multiple radar sites into a single logical contact, and computes each contact's apparent heading and speed to extrapolate future positions. Complicating things, the 9020 went into service during the development of secondary surveillance radar, also known as the transponder system ². On appropriately equipped aircraft, the transponder provides altitude. The Radar Processing system makes an altitude determination on each aircraft, a slightly more complicated task than you might expect as, at the time, only some radar systems and some transponders provided altitude information. The Radar Processing program thus had to track if it had altitude information at all and, if so, where from. In the mean time, the Radar Processing program tracked the state of the radar sites and reported any apparent trouble (such as loss of data or abnormal data) to the supervisor.

The Flight Processing program periodically evaluates all targets from the Radar Processing program against all filed flight plans, correlating radar targets with filed flight plans, calculating navigational deviations, and predicting future paths. Among other outputs, the Flight Processing program generated up-to-date flight strips for each aircraft and predicted their arrival times at each flight plan fix for controller's planning purposes. The Flight Processing program hosted a set of rules used for safety protections, such as separation distances. This capability was fairly minimal during the 9020's original development, but was enhanced over time.

The Output Processing program had two key roles. First, it handled data that was specifically queued for it because of a reactive need to send data to a given output. For example, if someone made a data entry error or a controller queried for a specific aircraft's flight plan, the Input Processing program placed the resulting data in memory, where the Output Processing program would "find it" to format and send to the correct device. The Output Processing program also continuously prepared common outputs like flight data blocks and radar station status messages that were formatted once to a common memory buffer to be sent to many devices in bulk. For example, a new flight strip for an aircraft would be formatted and stored once, and then sent in sequence to every controller position with a relation to that aircraft.

Legacy

The 9020 is just one corner of the evolution of air traffic control during the 1960s and 1970s, a period that also saw the introduction of secondary radar for civilian flights and the first effort to automate the role of flight service stations. These topics quickly spiral out into others: unlike the ARTCCs of the time, the flight service stations dealt extensively with weather and interacted with both FAA and National Weather Service teletype networks and computer systems. An early effort to automate the flight service function involved the use of a teletext system originally developed for agricultural use as a "flight briefing terminal." That wasn't the agricultural teletext system in Kentucky that I discussed, but a different one, in Kansas. Fascinating things everywhere you look!

This article has already become long, though, and so we'll have to save plenty for later. To round things out, let's consider the fate of the 9020. SAGE is known not only for its pioneering role in the computing art, but because of its remarkably long service life, roughly from 1958 to 1984. The 9020 was almost 20 years younger than SAGE, and indeed outlived it, but not by much. In 1982, IBM announced the IBM 3083, a newer implementation of the Enhanced S/370 architecture that was directly descended from S/360 but with greatly improved I/O capabilities. In 1986, the FAA accepted a new 3083-based system called "HOST." Over the following three years, all of the 9020 CCCs were replaced by HOST systems.

The 9020 was not to be forgotten so easily, though. First, the HOST project was mostly limited to hardware modernization or "rehosting." The HOST 3083 computers ran most of the same application code as the original 9020 system, incorporating many enhancements made over the intervening decades.

Second, there is the case of the Display Channel Complex. Once again, because of the complexity of the PVD subsystem the FAA opted to view it as a separate program. While an effort was started to replace the 9020 DCCs alongside the 9020 CCCs, it encountered considerable delays and was ultimately canceled. The 9020 DCCs remained in service controlling PVDs until the ERAM Stage A project replaced the PVD system entirely in the 1990s.

While IBM's efforts to market the 9020 overseas generally failed, a 9020 CCC system (complete with simplex test machine) was sold to the UK Civil Aviation Authority for use in the London Air Traffic Centre. This 9020 remained in service until 1990, and perhaps because of its singularity and unusually long life, it is better remembered as a historic object. There are photos.

Some years ago, I had a frustrating and largely fruitless encounter with the politics of policing. As a member of an oversight commission, I was particularly interested in the regulation of urban surveillance. The Albuquerque Police Department, for reasons good and bad, has often been an early adopter of surveillance technology. APD deployed automated license plate readers, mounted on patrol cars and portable trailers, in 2013. Initially, the department kept a six-month history of license plate data. For six months, police could retrospectively search the database to reconstruct a vehicle, or person's, movements—at least, those movements that happened near select patrol cars and "your speed is" trailers. Lobbying by the American Civil Liberties Union and public pressure on APD and city council lead to a policy change to retain data for only 14 days, a privacy-preserving measure that the ACLU lauded as one of the best ALPR policies in the nation.

Today, ALPR is far more common in Albuquerque. Lowering costs and a continuing appetite for solving social problems with surveillance technology means that some parts of the city have ALPR installed at every signalized intersection—every person's movements cataloged at a resolution of four blocks. The data is retained for a full year. Some of it is offered, as a service, to law enforcement agencies across the country.

One of the most frustrating parts of the mass surveillance debate is the ability of law enforcement agencies and municipal governments to advance wide-scale monitoring programs, weather the controversy, and then ratchet up retention and sharing after public attention fades. For years, expansive ALPR programs spread through most American cities with little objection. In my part of the country, it seemed that the controversy over ALPR had been completely forgotten until one particularly significant ALPR vendor—Flock Safety—started repeatedly stepping in long-festering controversies with such wild abandon that they are clearly either idiots or entirely unconcerned about public perception.

I try not to be too cynical but I am, unfortunately, more inclined to the latter. Companies like Flock know that they are in treacherous territory, morally and legally. They know that their customers are mostly governments or organizations with elected leaders that are subject to popular opinion. They know that helping Texas law enforcement track down abortion seekers in other states is a "bad look." They know all of these things, but they do not particularly care. They don't have to care: decades of incipient corruption, legal and political maneuvering, and the routine inefficacy of municipal politics has created an environment where public opinion doesn't matter.

I can't definitely tell you where public opinion lies on ALPR, although it seems like the average person might be mildly in support. From at least my experience, in my corner of the world, I will tell you this: it doesn't matter. Police departments and the means by which they purchase and field technology are so isolated from the political process that it is extremely difficult to imagine a scenario where voters could affect change. Year by year, city by city, the police become more dug in. Law enforcement agencies across the country have found that the most straightforward way to address privacy concerns around surveillance technology is to keep the department's purchase and deployment of that technology a secret. Most city governments at least passively support this approach. The vendors of surveillance systems facilitate, support, and even demand secrecy through their contract terms.

More recently, Las Vegas and the Bay Area have offered a model even more opaque to public scrutiny: law enforcement surveillance technology is simply purchased by wealthy private donors, almost invariably from the software industry, and then either the systems or their use are donated to the city. If carefully designed, these programs can be completely exempt from public information rules. They can take the form, for example, of a business association that runs its own private surveillance state, involving the public and ostensibly accountable police only when an arrest is made. We are privatizing mass surveillance.

Flock continues to generate enormous press, mostly on the back of persistent investigation by 404 Media. More recently, security researchers have published significant defects in the design of Flock's technology that can make the original video publicly accessible. Just about every time that someone looks into Flock, the company turns out to be less ethical, the users less concerned about compliance, the design of the system itself less competent than charitable viewers had assumed.

I'm trying not to be a doomer for Christmas, but I am sometimes frustrated with Flock coverage because it can miss the entire history of this issue. I think that a more contextually complete discussion of urban surveillance could be useful. And I am sitting in a coffee shop, in a trendy part of town, looking out the window at a PTZ camera on the side of a traffic light. That camera, I know, is owned and operated by APD's Real Time Crime Center (RTCC). Between the police department itself, city agencies that participate in the RTCC, and businesses that volunteer real-time access to their surveillance, there are thousands of others like it. Courtesy of a transit station, there are at least a dozen within my view.

Whether or not this is a good or bad idea, whether or not it is effective in reducing crime, whether or not it will be leveraged against political opposition; these are tricky questions. I suppose what worries me is that it feels like hardly anyone is asking them any more. It took Flock's remarkable ability to step on rakes and the apparent victory of fascism in national politics for anyone to remember that the construction of ubiquitous surveillance is a project that started many years ago, and that has proceeded largely unhindered ever since.

Domestic Signals Intelligence

Within the tech community, there has historically been much attention to the ability to track people by passively observing Bluetooth traffic. This technique has been widely used, both in commercial and government applications. There are popular "smart city" street lighting systems, for example, that allow every street light to passively collect signatures of the people passing underneath it. To my knowledge, these techniques are not actually very widely used by law enforcement. There are, perhaps, two reasons: one of mechanisms of government, the other of countermeasures.

First, "smart city" data collection systems are usually funded and deployed by municipal works or environmental departments. While police could make arrangements for access to that data, those arrangements would require the kind of inter-agency Memoranda of Understanding that tend to lead to far more public scrutiny than police acquisitions of surveillance products. Besides, since they aren't deployed for law enforcement purposes, they're often not useful sources for the areas that law enforcement find most interesting: higher-crime areas that are usually lower-income and, thus, less likely to have working street lights at all—much less "smart" ones. Besides, smartphones have widely adopted randomization of Bluetooth and WiFi identifiers, and protocol revisions have reduced the number of identifiers transmitted in plaintext. Passive signals intelligence just doesn't work as well as it used to, at least at the capability level of a municipality.

Talking about Bluetooth and WiFi on phones does raise the question of the "phone" part, the cellular interface, which is targeted by the family of devices often known as "Stingrays" after the trademark of a particular manufacturer. Fortunately, improvements in the security design of cellular protocols is making these less effective over time. Unfortunately, the technology continues to advance, sometimes undoing the improvements of newer GSM revisions. Federal and local law enforcement continue to purchase and use these devices, largely in secret, benefiting from a particular model of federal ownership/local use that makes it especially difficult to get a police department to even confirm or deny that they have ever deployed them. "Stingrays" or IMSI surveillance is mostly a shadow world of rumors and carefully worded non-denials. Despite technical measures against them, they are clearly still in use, and thus clearly still useful.

Optics

Out in the rest of the world, visual surveillance is more salient than radio. Early rollouts of police-operated surveillance, dating back to at least the 1970s, generated some controversy over privacy implications. Two things have since happened: first, law enforcement have relied on an increasing number of public-private partnerships and commercial vendors to gain access to surveillance without directly owning it. Second, police video surveillance has largely been normalized, and no longer faces much opposition or even public notice.

One of the interesting changes here is one of visibility: police video surveillance has often edged in and out of public awareness to fit the politics. In periods of pro-privacy, anti-surveillance sentiment, departments rely more on voluntary arrangements for access to cameras installed by others. In periods of pro-police, anti-crime sentiment, departments install cameras with flashing lights and police badges. Both types tend to persist after the next change in the tide.

The public usually knows little about these systems, a result of intentional opacity by police departments and a general lack of interest by the press. That leads to a lot of confusion. Where I live, we do have an extensive network of police-operated cameras on intersection traffic signal arms. And yet, if you ask the average person to identify a police surveillance camera, they will point to the camera for the traffic signal's video-based lane occupancy sensing every single time. That's not a police camera, it's barely even a camera as the video is rarely retained. All of these people, it seems, have worked themselves into a sort paranoia where they think that every camera is an eye of the police. Well, the police are watching them, from about ten feet over. The "speed dome" PTZ cameras get so much less attention, perhaps because they are usually mounted on the pole further from sightline, or perhaps because they are of a type more common for commercial surveillance systems that we have learned to simply ignore.

Video surveillance is an interesting topic to me, philosophically. I am largely unconcerned with the privacy implications of most video surveillance installations (such as the one on my own house) because, historically, the video was recorded locally and generally reviewed only when there was a specific reason. The simple fact that reviewing large amounts of video is so time consuming meant that the pervasive surveillance potential of video surveillance was, at one time not so long ago, quite limited.

Of course, the age of the computer has somewhat changed that situation. There are two phenomenon of the automation of surveillance that I think should be considered separately: first, machine vision has improved to an extent that computers can automatically process surveillance video to extract events and identities. Second, the appified, everything-social-media attitude of consumer products creates new dynamics in access to surveillance data, and those dynamics are spreading upwards into the commercial segment.

Machine Vision

Historically, much of the attention to video pervasive surveillance has centered around facial recognition. Facial recognition has indeed been applied to video surveillance for years, but I think that the average person vastly overestimates how effective and widely used facial recognition is.

The vast majority of currently installed surveillance cameras do not produce video of sufficient quality for facial recognition. That has less to do with the quality of the video itself (although that is poorer than you think for most real systems) and more to do with the way that surveillance cameras are used and installed. Most cameras are positioned high up with wide coverage of a room; this perspective is ideal for reconstructing a series of events but just about the worst case for facial recognition. For most surveillance cameras, human faces are small and at indirect angles. There is little geometry that you can extract from a face that is about ten pixels wide and subjected to aggressive h264 compression, which is how most surveillance video comes out.

Practical facial recognition systems involve cameras installed specifically for that purpose, roughly at eye level where they will get close-up, straight-on images of people who pass by. Next time you visit a bank branch, look by the exit doors for a conspicuously thick height strip. Height strips by exit doors were invented to allow clerks to give police a more accurate description of a robber, but they have since evolved to serve largely as subterfuge. Somewhere around 5' 6", you will notice a small hole, and behind that hole is a camera. Several manufacturers offer these and they seem very popular in financial services.

Kroger is more to the point: they've just been installing dome cameras right at eye height on their exit doors, for at least a decade.

When discussing surveillance, it's important to remember that the vast majority of real-world video surveillance systems are old, inexpensive, and poorly maintained. Even where cameras are installed specifically for a good view of faces, there probably isn't any facial recognition in use, most of the time. Facial recognition products are expensive and don't currently manifest many benefits unless the organization is large enough to have a security department to work with the resulting data, which requires a degree of operational maturity beyond most surveillance users (e.g. gas stations).

All of that said, there are plenty of real-world facial recognition deployments. Rite Aid, for example, prominently rolled out facial recognition to flag known shoplifters at their stores... a rollout that went so poorly that it lead to a lawsuit and an FTC settlement including the end of the facial recognition program and a five-year moratorium on further attempts. This is not to say that facial recognition on video surveillance isn't legal (although in some states it isn't or at least requires a lot of disclosure), but there is definitely a degree of legal and reputational hazard involved.

The ACLU has periodically conducted call-around surveys on use of facial recognition. The most notable trend is that most large chains now refuse to talk about it. I could be wrong, but my experience with corporate communications behavior leads to me to interpret a refusal to comment along these lines: Home Depot, for example, is a very large company that will have various initiatives underway, and either confirming or denying their use of facial recognition would probably be wrong in some cases and expose them to compliance or legal risk in others. I would bet good money that Home Depot has some facial recognition technology deployed at some locations, but knowing how slowly security technology tends to roll out in that kind of company and how complicated the legal and compliance situation can become, it's probably limited. They are probably acutely aware of the controversy surrounding this type of surveillance and, given the example of Rite Aid, the ways a rollout could go wrong. That means that surveillance will spread slowly.

But it will spread. At this point, I think it is inevitable that facial recognition will become widely used in video surveillance. I just think the point at which "facial recognition is everywhere" remains some years away, due to all the normal reasons: technical limitations, slow-moving bureaucracies, and a somewhat complex and unclear regulatory situation. It will inevitably happen, for the same reasons as well: aggressive sales by facial recognition vendors.

There are some sectors where facial recognition is very common, although I don't get the impression that retail is one of these yet. Casinos, for example—some of the larger Las Vegas casinos have reportedly had facial recognition systems in use for decades, and Nevada law is very permissive of them. Casinos are, of course, pretty much ideal users. Large institutions with a lot of financial risk and large, sophisticated security departments. Few other businesses outside of Target can compete with the size and sophistication of casino security departments. There's a whole lot of money flying around, and they can spend some of it on expensive per-camera licensing without much leadership objection.

License Plate Reading

Popular attention to facial recognition has mostly fallen away as industry and media focus has shifted to another application of machine vision that is, it turns out, a whole lot easier: license plates. License plates are designed for readability, and most states use retroreflective paints that give you an absolutely beautiful high-contrast image under coaxial (i.e. mounted alongside the camera) infrared illumination. There's not much that is easier to read by machine vision. Automated license plate readers have been available for quite a long time: US CBP had an experimental ALPR installation at a Texas border crossing in 1994. That system was actually deemed a failure and removed, but technology improved and there were permanent installations at larger border crossings by the end of the 1990s.

For a long time, the dominant vendor of ALPR equipment in the US was Motorola. Motorola's product line remains popular for vehicle-mounted systems, but the high price of the cameras, controllers, and software package had a side benefit of limiting the pervasiveness of ALPR. The equipment was just too expensive to put up all over the place.

In Albuquerque, for example, the ALPR program long consisted of Motorola systems mounted on portable "your speed is" trailers. The portable nature of these setups made the expense more worthwhile, and besides, portability has its own utilities: an APD detective once told me, for example, of how they would leave ALPR trailers in front of the houses of people suspected to lead criminal gangs. While there was value in the intelligence collection, the main motivation was intimidation: while you might call the "your speed is" trailers a concealed system, they're not all that subtle, and one supposes that most vehicle-based criminals (the main kind here) are aware that they function as the eyes of the police.

At some point, in response to growing budgets or lowered costs I'm not sure, APD began installing fixed Motorola ALPR systems on the light arms of major intersections. I know of around a dozen installations of this type in Albuquerque, which is the beginning of a widespread capability to monitor public movements but not exactly the dystopian pervasive surveillance of Minority Report.

ALPR works pretty well, but it is not perfect. Cameras need to be installed with fairly narrow optics aimed at the right spot, and infrared illumination makes reading far more reliable. Speaking of Las Vegas, I used to use a certain casino parking garage with an ALPR-based payment system with some regularity. It printed the license plate, as read by the ALPR, on the parking ticket, which is why I know that it was almost comically inept at reading my very legible California plate. It always got about half the characters wrong. I have gotten much better results in my own home experiments with budget equipment, so I figure that system must have been very poorly installed or maintained, but I'm sure there are plenty of others out there just like it.

That's the tricky thing about video surveillance, from the "blue team" side of the house: people don't tend to pay a lot of attention to it until there's been an incident, at which point they find out that the lens has had mud on it for the last three months (a much bigger problem with ALPR cameras that used to be mounted pretty low to the ground for a better look angle). I bring this up because I think that people tend to vastly overestimate the quality of real-world video surveillance, and I like to take every opportunity to remind people that the main failure case of retail video surveillance used to be failure to replace the continuous-loop tape cassette before it was completely demagnetized by repeated recording. Now, in 2025, the continuous-loop tape cassettes are pretty much gone, but maintenance practices haven't improved. Lots of the cameras you see in public only barely work or don't work at all. So it goes.

Flock

In 2017, though, a VC-backed (and specifically YCombinator) company called Flock Safety introduced a bold new idea to ALPR: a Silicon Valley sales model. Flock's system is built to be low-cost, and the sensors are smaller, simpler, and cheaper than Motorola's. I suppose they might be less effective as a result, but a reduced "catch" rate doesn't really detract from mass-surveillance ALPR installations that much. Flock has also greatly expanded their customer base, emphasizing sales to private organizations as well as law enforcement and government. Speaking of Home Depot, for example, Home Depot seems to have installed their own Flock cameras in all of their parking lots. Lowes Home Improvement has done the same.

I wanted to know more definitely how much Flock systems cost, because I suspected they were making significant inroads just through low pricing. It's a little tricky to say definitively because Flock is a "call for quote" kind of company and I think they offer contracts on different price bases. Scouring contracting documents, meeting notes, etc., it seems like a "typical" cost for a Flock camera is around $4,000 with about $3,000 a year in per-camera software licensing fees.

That might seem expensive but it compares well to the five-figure prices I have heard associated with Motorola systems, especially since the Flock offering is more "white glove" with installation and maintenance packaged. Motorola systems are usually purchased through an integrator who adds their own considerable margin.

Flock also designs their cameras to be amenable to solar power, which radically reduces install costs compared to Motorola systems that usually need a utility worker out to splice power from a streetlight. More even than a price reduction, it makes Flock cameras much more available to organizations like HOAs that control territory in a sense but do not have the full bucket truck or utility work order capabilities of a municipal government.

Another recent innovation in ALPR is less traceable to Flock but certainly seems associated with them: flexible funding sources. Police departments have limited budgets with which to acquire new technology, and technology vendors have to compete with other budget priorities like salaries, vehicles, and black-on-black tactical vinyl jobs for those vehicles. ALPR seems especially attractive for public-private partnership mechanisms, so there are a lot of Flock installations that were funded by business associations, HOAs, neighborhood associations, and other "indirect" sources. Some of these systems are owned and operated by the police with only the funding donated, others are owned and operated by the private group that paid for them. This can result in curious deployment decisions: sometimes the lowest-crime neighborhoods are the most replete with ALPR, as they tend to be wealthier and more politically organized communities with the wherewithall to put up the money.

The most important thing to understand about Flock, though, is that it has built on Amazon's concept of "Ring neighbors" to build a sort of nationwide, ALPR-centric Nextdoor. Flock customers can basically check a box that allows other Flock customers to access data from their sensors, and of course Flock strongly encourages users to turn sharing on. While there are some audit and access controls available on Flock data sharing, they seem like pretty minimal efforts that are often ignored.

Flock sharing has generated a lot of press, especially with some dramatic examples like use by a Texas sheriff to locate an abortion patient and use by ICE/CBP to track suspects. These are both examples that raise one of the most alarming aspects of the Flock situation: many states and municipalities have laws in place that limit or at least monitor collaboration of local police with other police agencies and federal law enforcement. Some people find this surprising, but it's important to understand that the United States is a republic of nominally independent governments. Laws, policies, and priorities can vary greatly from jurisdiction to jurisdiction. There is a specific historical thread, related to the tracing of escaped slaves, that has made resource sharing between different law enforcement agencies a known area of moral and legal treachery for a very long time.

And yet, it turns out, most Flock customers seem to have sharing turned on, possibly entirely without their knowledge. There are now multiple well-established cases of local law enforcement agencies violating state laws by having data sharing with ICE/CBP enabled. It is possible for Flock users to turn off or restrict sharing, but I think a lot of them honestly don't know that. Some state Attorneys General have ordered Flock users to disable sharing, some have restricted or banned Flock products entirely, but in general it's a very messy situation. It appears that a lack of care by law enforcement and other Flock customers, facilitate and no doubt encouraged by Flock's motivation towards "network effect" lock-in, has resulted in widespread and brazen violation of privacy laws that is only now making its way to the courts.

In other words, the tech industry happened.

Acoustics

I will not spend much time here discussing wide-area acoustic surveillance like ShotSpotter, in part because I have written a bit about it before. It's a complex issue: a well-designed gunshot detection system would probably be a good thing, but I find SoundThinking (manufacturer of the ShotSpotter system) to be profoundly untrustworthy.

Futures

The changes we are already seeing will continue: ALPR will become more ubiquitous, facial recognition will advance further into the public sphere, and the tech industry will continue to centralize data and facilitate queries by law enforcement. There's a lot of money to me made out of the whole thing, and funding towards law enforcement or public safety purchases are usually politically safe. Pretty much everything is stacked in the direction of more pervasive surveillance in the United States.

Do you find that upsetting? It seems that some people do, and some people do not. I am probably not as opposed to surveillance of public spaces as the most vocal privacy advocates, but I am also convinced that vendor-enabled mass surveillance technology like Flock is subject to enormous abuse and will inevitably undermine constitutional protections. Unfortunately, vocal organizing against mass surveillance has become pretty limited. The ACLU is doing a lot of good work in this area, but I don't see much public organizing.

The best thing you can do is probably to advocate for transparency. The most alarming part of this whole thing, to me, is the way that police departments have brazenly structured purchases of surveillance technology to get around public record and approval requirements. Companies like Flock and SoundThinking encourage this, and write it into their contracts. The end result is that many police departments have installed cameras and microphones in all kinds of places, and will not disclose when, where, how many, or how they are used. We should not allow that kind of secrecy, but preventing it seems to require legislation. The federal situation seems like a loss, so the best pressure point might be to lobby for municipal or state legislation that will require police departments to disclose their surveillance programs. Even better would be a requirement for review and approval of surveillance purchases, but unfortunately that kind of rule often already exists and police departments still structure their purchase arrangements to avoid invoking it.

I suppose the bottom line is this: keep bringing it up. Mass surveillance in the US often feels like a lost cause, but I suppose it's only lost if we give up. It doesn't take that many people showing up at a city council meeting to make something a priority to the councilors; and perhaps the police can only stonewall for so long. It's worth a shot.

One of the difficult things about describing a grift, or at least what became a grift, is judging the sincerity with which the whole thing started. Scams often crystallize around a kernel of truth: genuinely good intentions that start rolling down the hill to profitability and end up crashing through every solid object along the way. I'm not totally sure about Evelyn Wood; she seems to have had all the best in mind but still turned so quickly to hotel conference room seminars that I have trouble lending her the benefit of the doubt.

Still, she was a teacher, and I am inclined to be sympathetic to teachers. Funny, then, that Wood's journey to fame started with another teacher. His curious reading behavior, whether interpreted as intense attention or half-assed inattention, set into motion one of the mid-century's greatest and, perhaps, most embarrassing executive self-help sensations.

In 1929, Evelyn Wood earned a bachelor's in English at the University of Utah. The following two decades are a bit obscure; she took various high-school jobs around Utah leading ultimately to Salt Lake City's Jordan High School. There, as a counselor to girl students, Wood found that many students struggled because of their reading. Assigned books were arduous, handouts discarded. These students struggled to read so severely that it hampered their performance in every area. She launched a remedial reading program of her own design, during which she made her first discovery: as her students learned to read faster, their comprehension improved. Then their grades—in every subject—followed suit. Reading, she learned, was a foundational skill. A person could learn more, do more, achieve more, if only they could read faster.

Wood became fascinated with reading, probably the reason for her return to the University of Utah for a master's degree in speech. Around 1946, she turned her thesis in to Dr. Lowell Lees. Lees was the chair of the Speech and Theater Department, and had a hand in much of the development of Utah theater from the Great Depression until his death in the 1950s. A period photo of Lees depicts him with a breastplate-microphone intercom headset and a look of concentration, hands on the levers of a mechanical variac dimmer rack. He is backstage of either "Show Boat" or "A Midsummer Night's Dream" at the university's summer theater festival. A theater department chair on lights seems odd, yes, but theater was Lees passion.

Perhaps reading was not. When Wood turned her thesis into Lees, he "read, graded, and returned the thesis within a matter of minutes." Wood was amazed that he seemed to just leaf through the pages, but then still had insightful questions to ask. Perhaps I am too cynical. It feels most likely to me that Lees was already familiar with the contents (he was probably Wood's advisor and would have discussed the research plenty of times before) and just didn't bother to read the final document. To Wood, though, something more remarkable had happened. With a series of tests, she convinced herself that Dr. Lees could read over 6,000 words per minute with full comprehension.

A typical American college graduate can read at about 250 words per minute, at least if the material isn't too challenging. Some people, Wood contends, are "10x readers." They read so quickly, and with such good understanding, that they simply outpace the rest of us at every intellectual pursuit. What's more, Wood could make you one of those people. As she tells it, she spent two years, probably in the 1950s, tracking down fifty some examples of other exceptional readers. She published "Reading Skills" in 1958, a book evidently based on some of this research but more focused on remedial skills for grade students than executive achievement.

The introduction of Reading Skills tells us of ten different students. Anna was pretty, but she couldn't read. Joseph hated school, because he couldn't read. Carl's hair is a mess, and his parents neglectful. He also can't read. All of them became proficient readers through Wood's program. But Wood had more in mind than grade students. A year later, with her business-educated husband, she brought her reading program to adults by launching a chain of training centers under the name Evelyn Wood Reading Dynamics.

Books neither bored nor scared me any longer. I could read almost any book within an hour, and more important, I better understood that which I read.

Reading Dynamics became a sensation. Over the following years, Evelyn Wood institutes opened across the country. The speed reading movement received a considerable boost from President John F. Kennedy—he claimed to read at 1,200 words per minute, a skill he learned in part through a correspondence speed reading course. It wasn't one of Evelyn Wood's, but that detail was mostly lost on the public and the success of the Kennedies became linked to Reading Dynamics. He seems to have bought the same course for his brother Ted, and encouraged his staff to take speed reading courses as well. Reading Dynamics didn't miss the marketing opportunity, and indeed the very first dedicated Institute opened in Washington, D.C. and advertised specifically to politicians. Senators and representatives were among her earliest students and her strongest advocates.

Evelyn Wood Reading Dynamics underwent several changes of ownership through the 1960s, but Wood stayed on as developer of the training materials. Soon there were more than 60 institutes, and newspaper ads directed the interested public to "free mini-lessons" held in the meeting rooms of fine hotels across the country. It became a franchise system, with the Woods personally owning the franchise for Utah and Idaho. The company's fortunes have trended up and down with those of speed reading as a concept, but genuine Evelyn Wood speed reading courses are still available today from business training firm Pryor. There has been a bit of a reckoning: far from the 1,000+ WPM rates promised by early Evelyn Wood marketing material, Pryor now advertises "a potential rate of 400-700 words per minute." These numbers align with the upper end of reading speeds observed among the general population. In effect, Pryor no longer claims that speed reading courses will make you a faster reader than more conventional methods of training reading, like just doing a lot of it.

The science has never really been with speed reading. As early as 1959, when Reading Dynamics hit Washington, researchers and educators called Wood's data and methods into question. As with most self-help materials, Wood's writing was heavy on anecdotes and light on quantitative analysis. Certain elements of her method contradicted psychology's growing understanding of human language and perception. At the core of the problem, though, was her promise of comprehension.

It is obvious that a person can "read" a document very quickly, if we relax our definition of "read." This is just as obvious to the developers of speed reading courses. Many advise students to start by skimming, flipping through the whole book or document and taking in the headings and subjects. You can certainly get through a book under an hour that way, but of course, you haven't exactly read it. Think of it as a lossy process: the less time you spend on a document, the less you comprehend and retain its contents. That seems pretty intuitive, doesn't it?

But Wood disagreed, or at least, the company she founded did. It can be a little difficult to untangle Evelyn Wood's original theory from the many generations of Reading Dynamics and competing speed reading systems that followed. Subsequent owners of Reading Dynamics, which included companies like the publisher of Encyclopedia Britannica, made significant revisions to the material. By the 1970s, Wood's role was more as a celebrity spokesperson than a teacher. In any case, Reading Dynamics came to emphasize a key principle that reading faster actually improves comprehension. The most skilled readers, Reading Dynamics taught, don't even read words. They scan a page vertically, not horizontally, taking in an entire line at a time by peripheral vision. There is no need to sound out, read, recognize, or even really see individual words, as the mind actually processes language in large chunks at a time. Reading occurs mostly subconsciously, so in a way all you have to do is see the text and believe that you have read it, and you will retain the content.

In a 2016 review paper on speed reading, a team of psychologists deliver bad news: it just doesn't hold up. Laboratory studies confirm that the eye only has the acuity to distinguish words in a small area, and that reading requires fixating on just about every word individually. That doesn't even matter, though, because other laboratory experiments strongly suggest that the limiting factor on reading speed is not the eyes at all but the mind. Even when clever computer techniques are developed to present text more quickly, comprehension trails off at about the same speed. In fact, when humans read, we regard an even smaller area of our vision than the limits of the fovea would suggest. When looking at a word, we are basically blind to anything further than about seven characters away.

The problems with speed reading are not merely theoretical, though. The researchers considered studies of actual speed readers, people who had either completed speed reading courses or claimed to naturally read at exceptional speeds. Almost no studies can be found that support the claim of faster reading with retained comprehension. Speed readers perform poorly on comprehension tests on new material. People who "speed read" a document generally show similar comprehension to people who have no speed reading training but skimmed the document in the same period of time. When speed readers have performed better, researchers suspect the result comes more from advanced familiarity with the material (a common problem with speed reading courses that use the same texts repeatedly), broader general education (you retain more from non-fiction material if you already knew the information to begin with), and greater experience and confidence in "interpolating" by speculating as to the content of the text that wasn't actually read.

Ultimately, eye tracking experiments tend to confirm the worst. People who speed read don't do all that much actual reading. Skilled speed readers skip much of the text completely, and tend to make things up when asked about things they never fixated on. Most interesting, there seems to be a certain Dunning-Kruger effect at play. People who have speed-read a book on a subject, for example, tend to rate their knowledge of the subject highly and then perform very poorly on questions about it (often scoring similar to chance on multiple choice tests). Speed reading, it turns out, is a placebo. It makes you feel like you have read something, even though you haven't.

And yet we still have speed reading. Wood's efforts were perhaps sincere, but the commercial imperative of the growing Reading Dynamics institutes steered the whole thing away from evidence-based methods and towards ideas with an increasingly tenuous connection to reality. The on-again, off-again success of Reading Dynamics left a lot of room for imitators, or innovators, depending on your perspective. Evelyn Wood's original strain of speed reading has mostly fallen away, replaced by a new set of courses that build on Wood's ideas—the worst of them.

Take, for example, the work of Paul Scheele. Scheele is one of those business conference motivational speaker types, the kind of person who is introduced with a vast and impressive resume but who doesn't seem to have really done anything. With a PhD in "Leadership and Change," he founded Scheele Learning Systems to market a series of innovations. His work is so interconnected with other self-help and new-age grifts that it can be hard to untangle what comes from where, but one of their key programs clearly builds on the Evelyn Wood method: PhotoReading.

The basic concept of PhotoReading is that the mind is able to subconsciously process far more information than the conscious mind. In a marketing sheet, he writes:

Your conscious mind can handle seven pieces of information at a time, while your subconscious mind can handle a staggering 20,000 pieces of information. That's the difference between regular reading and PhotoReading.

So imagine a future in which you pick up a book, flip through the pages, and in a matter of minutes gain a full command of the material contained therein. The key is that you don't actually have to read anything, you just have to see it and your subconscious mind files every word away for later retrieval.

Well, of course, it's not quite that simple. There's a whole technique to it, a technique that you can learn from a self-guided digital course for only $530. Sure, that might seem a little steep, but consider that it's a package that includes not only the course but "The PhotoReading Activator Paraliminal CD." Paraliminal activation or paraliminal hypnosis is another major product from Scheele Learning Systems, although I think it's licensed at least in part from a different organization (Centerpointe Research Institute) founded by different cranks ("transcendental meditation" enthusiasts Bill Harris and Wes Wait). The idea of paraliminal activation is roughly halfway between subliminal inducement videos ¹ and binaural beats ², in that it's both of them mixed together. Incidentally, a lot of subliminal videos are like that anyway, so I'm not sure that Scheele is offering anything you can't get for free. All of these organizations offer rotating carousels of endorsements from famous and successful customers. The fact that these happy customers are almost invariably self-help authors or business conference motivational speakers goes unremarked upon.

Scheele's ultimate claim is that PhotoReading allows you "to 'mentally photograph' the printed page at 25,000 words per minute." 600-800 WPM is an excellent, exceptional reading rate among the normal population. For today's speed reading industry, though, 25,000 WPM is the bar to meet. "Harry Potter and the Deathly Hallows" counts up to about 198,000 words. An experienced PhotoReader, then, ought to be able to complete it in around eight minutes. Well, celebrity speed reader Ann Jones says it took her 47, so no one is perfect. She knocked out "Go Set a Watchman" in 25 and a half, and that on live television. Yes, for the most successful speed readers, people who claim rates in excess of 10,000 WPM, there's almost always some aspect of performance involved... whether that's television appearances or elected office. 25,000 WPM became cemented because it's the rate at which the Guinness World Records clocked celebrity speed reader Howard Berg. Actually, there's a woman who claims to have a Guinness World Record at 80,000 WPM, but it's hard to substantiate as Guinness stopped publishing the speed reading record at all decades ago. I suppose it became too questionable for even them.

The reason I'm so fascinated by speed reading is its close interconnection to the concept of the executive. One of the earliest newspaper ads for Reading Dynamics reads "For Executives, Businessmen, Students, Housewives." The housewives part doesn't quite fit the theme, but I think that might be better understood with the Utah LDS context of the housewife side hustle. Multi-level marketing schemes were becoming a cornerstone of the Salt Lake City business scene during the 1960s, a role they still fill today, and MLM brands like Avon found their success in part by melding the two worlds of the housewife and the business executive. Feminine products sold with masculine hustle, you might venture; some housewives were applying themselves to business with a zeal that would make a railroad baron blush.

For students, the motivation is more obvious. Much of education comes down to reading, and we all remember the feeling of a paper due in two days on a book that you haven't yet opened. For the student, getting good comprehension of a text in a fraction of the time is an incredible offer. So promising was speed reading for education that, in its early days, it found considerable adoption in the educational establishment. Many universities offered speed reading courses, some even made them core curriculum. A particularly prominent speed reading course at Harvard served as the pattern on which many others were taught. Besides a series of demonstration films developed Harvard, devices called "reading regulators" or "reading accelerators" were popular lab equipment for these courses. They automated Evelyn Wood's idea of running a ruler down the page, sliding a metal shield down the page faster and faster to force the student to read at a higher and higher rate. For a few years, speed reading for universities became an entire industry, but it was short lived. Academic speed reading courses faded away as criticisms of Wood's theories became better known and attempts at validating speed reading continued to fail.

"Speed reading," it turns out, did not work out in education. But perhaps that's a matter of framing. If we consider the broader landscape of "things that promise to save you time reading," speed reading is just one in a long line of ideas. It turns out that students have been trying to skip the reading for just about as long as there has been reading—consider Monarch Notes, a line of book summaries and critical commentary already available a hundred years ago. From Monarch to CliffsNotes to Chegg, students have looked to a whole sector of the publishing industry to do the hard work of actually reading books for them. You could say that the purpose of these digests or study guides is to help a student maintain the appearance that they have read a text even though they have not, by imparting only the parts of the text that are most important... important either because they are key to the plot or theme, or because they are likely to appear on exams or be expected in papers.

While students have an obvious need for these types of summaries (scoring well on assignments with less time invested), the appeal to the business executive might seem a little fuzzier. Well, unless we take the cynical view that the ultimate goal of an executive is to look smart, and I'm not sure that you really have to be so cynical to accept that as truth.

Summarizations are obviously "lossy," in that a digest form of a book cannot possibly contain the full information of the original book. Similarly, the weight of scientific evidence, as well as most credible practical experience, tells us that speed reading is a lossy process. There is, as the psychologists put it, no silver bullet in reading. Comprehension takes time; less time means less comprehension; and while you likely can improve your reading speed it will take years of practice.

And yet book summaries are an even larger industry than speed reading, and one that is both older and better adapted to the modern age. There clearly is a market for fast, low-comprehension reading of large texts. The audience is not purely made up of people seeking to create the appearance of work they have not done, although that's clearly a large part of it. Consider the magazine book review: long a staple of magazines, book reviews serve two purposes. They give you an idea of whether or not a book is worth reading, but they also summarize the book, or at least explain the major themes. That gives you some of the content of the book, the major ideas and a few choice details, in just a page of three-column prose. A third of that might be taken up by a wine club ad, to boot.

The case of the magazine book review reminds us that there is a serious, a respectable application for summaries. The perfect example might be the lawyer or doctor, people who are paid explicitly for their expertise and education but who also make heavy use of digests and summaries and desk references. There is a lot of information in the modern world, even in any given field, and no one can keep track of all of it. You might need to speed read, to use the CliffsNotes, just to keep up with the state of the field and find the things that you do need to read in their full length.

And so we have seen the dual facets of the executive demand for speed reading: the businessman, the leader, the executive is the perfect intersection of the professional need to find what to read and the personal need to look like you have done a lot of reading. Executives are expected to know what's out there, but also to seem like they already know all of it. It's a matter of opinion which of these is more prominent, but I think we can all agree that publications like CTO Magazine are aimed at that dual purpose.

Well, these days, publications like CTO Magazine are mostly aimed at drumming up AI hype. That's the other thing about business publishing: it is itself a business, and as beholden to the trends as any other.

The funny thing about speed reading is that it has never been that credible. Evelyn Wood's theories were inconsistent with the research and, frankly, a bit "out there" even as she developed them into a business in the 1960s. Experiments on speed reading, some of them conducted by the same people selling courses, have always shown iffy to clearly negative results. And yet speed reading has, in its good times, enjoyed a level of credibility and popularity that seems out of step with even its promises and certainly with its outcomes.

US Presidents Kennedy, Carter, and Nixon were all speed readers. Carter and Nixon both arranged Evelyn Wood Reading Dynamics courses for their staff, and it seems that Kennedy probably purchased some sort of course for White House staff as well. This was very much perceived as an endorsement from the top, and speed reading became not just a new innovation in education, not just a trend, but practically a requirement for any serious leader. Marketing, and the celebrity adoption that it intentionally engineered, outpaced the results. Evelyn Wood's newspaper ads and reputation got so far out front of the actual pedagogy that today's speed reading industry, spinning ever farther from reason, continues to coast on the same set of presidents.

That's not to say that there has been nothing new in speed reading. In 1984, psychologist Mary C. Potter described a method called "rapid serial visual presentation" or RSVP. The idea of RSVP is to eliminate the whole eye movement part of reading entirely, instead using a computer to present one word at a time, each centered in the same location. In theory, the words can be presented faster and faster until the user is reading more quickly than the visual system allows. Well, that's a theory at least. It's inconsistent with later research suggesting that reading speed is limited by cognition rather than perception, but most of that wasn't yet known at the time. Even so, Potter doesn't seem to have viewed RSVP as a speed reading technique. She described it as a method for cognitive research, one that could enable new experiments and improve old results by controlling for the many variables involved in scanning a page of text.

The idea of RSVP as a speed reading technique seems to have been popularized by software startup Spritz, who launched an RSVP speed reading application in 2014. Spritz seems to have spun it as "text streaming," although I think that might have been a later branding innovation. The claims of Spritz are relatively modest, only 1,000 WPM in most cases and sometimes as low as 600 WPM. These are speeds achievable (even if only narrowly) without technical assistance for exceptionally fast readers. Even so, it doesn't really work out. Research on the RSVP method of speed reading finds that comprehension decreases with increasing speed. Amusingly, some experiments show that RSVP results in decreased comprehension even when run at the same speed the subject reads naturally. Psychologists tend to attribute that effect to the fact that RSVP prevents going back and rereading a sentence that you didn't fully understand—a behavior that seems to be a natural and even required part of good reading, despite the fact that Evelyn Wood and virtually every speed reading theorist since has outlawed it.

The fact that the RSVP concept is fundamentally at odds with blinking is probably the major cause of a reported increase in fatigue, as well, but none of these shortcomings have prevented the massive popularity of RSVP within the tech industry especially. Spritz, the company, has gone basically nowhere, but the concept has graduated from TED talks to a huge inventory of browser extensions, mobile apps, CLI tools, and various and sundry GitHub projects that all make the same claims about increased reading speed. "Speed Reading Makes a Comeback" was the title of an NBC News spot on Iris Reading, more of a traditional Wood-style speed reading training company that has since wholeheartedly embraced the RSVP concept.

If software is part of the speed reading story, and a particularly core part of it today, we will have to take on the elephant in the room: in a certain sense, a very real sense, summarizing text is now the largest single driver of the US economy.

The appeal of summarization to the business executive has never gone away; the underlying technology has just evolved. Since the 2022 launch of ChatGPT, television spots, bus shelter ads, and the collective buzz of the south end of the San Francisco Peninsula have promised first and foremost that AI will relieve us of the obligation of reading. An LLM can read your email, read the news, read a book, or read the comments. Actually, the LLM has already read a lot of these things. On command, it can summarize them to you.

AI advertising seems to imagine a world that is, well, oddly familiar: one in which students, housewives, and, yes, business executives can save hours of each day by using the LLM to, in effect, read at 25,000 WPM. It also seems that the same basic principles apply: the LLM's output loses some of the content of the original material. It might also gain some content, a benefit of all of the other things that the LLM has also been trained on. Still: there's always a certain rounding out, a sanding down of the details.

What strikes me most about LLM summaries is just how long they are. When I have asked Claude to summarize reading notes, it has routinely produced output that is longer than the original notes. This problem can probably be addressed by prompting, although my efforts at appending everything from "be brief" to "for the love of God keep it to one paragraph" have failed to produce a good result. Maybe I'm holding it wrong, maybe I'm an idiot, I possess no qualifications in this area besides decades as a natural language user and an unfinished degree in technical writing. But experience suggests that my coworkers have the same problem. I see AI generated meeting summaries, AI generated issue descriptions, AI generated sales documents. One of their common properties is that they are astoundingly, uselessly verbose.

Of course, modern AI can do so much more than summarize text. "Generative AI" promises not only to summarize, but also to create something new. Perhaps that's why the LLM is so verbose. I, personally, find that I make up for my lackluster interpersonal skills by writing. Perhaps LLMs make up for their similar limitations, their fundamentally text-based, screen-resident nature, by using the one tool that they have. The LLM cannot think or feel, yet it can write. So it writes: a simple question answered with such energy that it merits four distinct bulleted lists, each with an emoji-laden heading and an introductory paragraph. I suppose I can sympathize. We must imagine Grok happy.

I do not mean to criticize AI too harshly, although I think the level of criticism that this entire industry phenomena deserves is high enough that you have to go big.

But the relationship between speed reading and the LLM—between Sam Altman and Evelyn Wood—is vague but vivid. The software industry's imagined future, in which people use LLMs to generate text that other people use LLMs to summarize, genuinely haunts me. AI has created a profound contradiction: it promises the productivity gains of speed reading, the ease of CliffsNotes, but it doesn't just shorten text. It also lengthens it. My joking reference to Camus, shoddy as it is, becomes more meaningful. ChatGPT pushes the written word up the hill, it watches it roll back down again.

I read "The Myth of Sisyphus" for the same reason everyone else did: high school. IB English HL. Yes, I went to one of those schools. If you are not familiar you can look it up and one of the top results, at least for me, is a clearly LLM-generated article that is four or five times longer than it should be based on the factual content. You can have your web browser's LLM feature summarize it back down for you, if you want. The result comes out a lot less useful than the Wikipedia article but it is, as they say, disruptive nonetheless.

If the purpose of reading is solely to acquire information and accumulate thought units, then surely speed and efficiency are the essential criteria. Regressing is obviously a morbid symptom since it is destructive of time and energy, while horizontal reading not only taxes the optic muscles, but requires that the same tome remain clutched by fingers which could be more profitably employed in reaching for yet another volume.

We're all full of opinions on the era of AI. I am perhaps not as pessimistic as you might think: the machine learning innovations of the last few years clearly do have useful applications. Even summarizing text has its time and place. I suppose that what frustrates me most about it all is the lack of ambition. LLMs train on text, take text as input, and generate text as output. A room of Silicon Valley visionaries, presented with this astounding tool, came up with such world-changing applications as "reading emails" and "writing emails." The whole industry is still struggling to move past this trivial, boring, frequently nonproductive use.

As for lip motions, any toddler knows that the mouth is tardier than the eye, and retardation is one of the most dreaded words in an educator's terminology. Furthermore, the speed cult is quick to point out that slow readers are rarely careful ones, and generally speaking, comprehension appears to increase with reading velocity. Speed reading, it would appear, is all profit and no loss and if it can make good its claims at linking efficiency and comprehension, then it is well that its methodology and objectives are incorporated into any reading program.

There is the potential, the AI's industries advocates say, of AI actually expanding human creativity. Machine learning methods of producing "art," whether text or image or audio or video, will lower the barrier of entry to artistic production. Of course, that depends a lot on how you define "artistic production," but at least it's a rare promise of a better future rather than a worse one. It only takes a brief interaction with the modern internet to realize that we do live in an age blessed with text. We are rich in the written word like never before, so wealthy with words that they crowd out the actual information. Search results are mostly AI-generated, but the search engine doesn't want you to look at them anyway, it's provided its own AI-generated treatise. The headings, the paragraphs, the bulleted lists, they run down the page, drip from our screens, they leave our desks filthy with content.

But prior to debating the plausibility of the claims in an Evelyn Wood brochure, it would seem logical to consider the desirability of the goals—goals which appear to have slipped unchallenged into the realm of pedagogical axioms. Are such facile reading practices worthy of unqualified adulation? A careful look at their implications suggests that such seeming saints can in fact be devils.

It's enough to drive you to madness. Why do we use computers to write text that no one will read? Why do we use computers to read text that no one wrote?

Years ago, in college, during a previous AI winter, I sat in my room reading a shitty science fiction novel. Leo, from across the hall, walked in. "What class is that for?" he asked.

"Not for a class," I responded.

"So you're just reading it?"

Taken by themselves, the cardinal virtues of reading efficiency can collectively demean the entire reading process by treating it as a function rather than as an art.

There is nothing new under the sun. We have done this all before: we have fixated on reading as production, production as profitable, and reading thus, ultimately, unimportant. A detail to be optimized away. An expense. ChatGPT didn't start this. It won't end it. That's what I remind myself: we are living through just another step in the evolution of culture.

But then I still worry. What if this is it? Between short-form video and AI, between social media's pivot to stoking fascism and the publishing industry's pivot to reprinting AO3, what if language arts are done?

None of these people care. That's the one thing I can say confidently, or at least say that I truly believe. These people building the cutting edge of natural language, these industry titans who style themselves as the loyalists of our nation and revolutionaries of the arts, they don't give a damn about writing or reading. Text is an asset, an asset to extract, refine, and dispense. They're just trying to make it through the news and their Twitter feeds and a half dozen pop-science books as fast as possible so that they can be, feel, or at least look like they're well-read. They assume that everyone else feels the same way.

How can any teacher extol the pleasures of reading when classroom practice implicitly asserts that books are mines to be stripped and not pastures in which to dwell and delight?

I've been quoting from Leonard R. Mendelsohn, whose paper "Jetting to Utopia: The Speed Reading Phenomenon" ignores the question of whether or not speed reading works and instead considers whether or not it is a good idea. His context was the classroom of the 1970s: speed reading had caught on in education, and Mendelsohn worried. Well-intentioned teachers were training their students to absolutely optimize the mechanics of reading. In the process, Mendelsohn feared, they had forgotten the point.

Reading can provide fodder for the brain by the ready conversion of wood pulp and printer's ink into social poise, persuasiveness, and a financially rewarding livelihood.

Things have changed a great deal since Mendelsohn's day. The wood pulp is gone, so too the printer ink, and so too the financial rewards. Writing is, I suppose, more of an art than ever before, as my chosen industry devotes its full might to destroying my chosen avocation.

Although reading might be branded with the explicit label "fun," it is not long before the apt student reaches the conclusion that speed, concepts, and information are all one knows and all one needs to know.

Mendelsohn's paper ran in the journal "Language Arts." It's about four pages long, about 2,300 words. It took me around ten minutes to read. An accomplished student of Evelyn Wood could read it in just a couple of minutes. With some chiding to stay brief and cut it out with the bulleted lists, Claude summarized it in a few sentences.

For the journal, though, the paper is not quite long enough. Its last page is only half full. The journal editor made up the difference, they found some filler. It's a poem about clouds.

ChatGPT can do so much, but it can't do the work of a poet. It can't match Christa Kessler, age 10, Powhatan School, Boyce, Virginia. She wrote "Clouds" almost fifty years ago, an editor used it to round out the layout of a journal, JSTOR coughed it up along with my article, and now I am thinking about how clouds really are interludes in the middle of a great blue sea.

That's what it's like to read slow. That's what it means to write.

I have at least a few readers for which the sound of a man's voice saying "government cell phone detected" will elicit a palpable reaction. In Department of Energy facilities across the country, incidences of employees accidentally carrying phones into secure areas are reduced through a sort of automated nagging. A device at the door monitors for the presence of a tag; when the tag is detected it plays an audio clip. Because this is the government, the device in question is highly specialized, fantastically expensive, and says "government cell phone" even though most of the phones in question are personal devices. Look, they already did the recording, they're not changing it now!

One of the things that I love is weird little wireless networks. Long ago I wrote about ANT+, for example, a failed personal area network standard designed mostly around fitness applications. There's tons of these, and they have a lot of similarities---so it's fun to think about the protocols that went down a completely different path. It's even better, of course, if the protocol is obscure outside of an important niche. And a terrible website, too? What more could I ask for.

The DoE's cell-phone nagging boxes, and an array of related but more critical applications, rely on an unusual personal area networking protocol called RuBee.

RuBee is a product of Visible Assets Inc., or VAI, founded in 2004 ¹ by John K. Stevens. Stevens seems a somewhat improbable founder, with a background in biophysics and eye health, but he's a repeat entrepreneur. He's particularly fond of companies called Visible: he founded Visible Assets after his successful tenure as CEO of Visible Genetics. Visible Genetics was an early innovator in DNA sequencing, and still provides a specialty laboratory service that sequences samples of HIV in order to detect vulnerabilities to antiretroviral medications.

Clinical trials in the early 2000s exposed Visible Genetics to one of the more frustrating parts of health care logistics: refrigeration. Samples being shipped to the lab and reagents shipped out to clinics were both temperature sensitive. Providers had to verify that these materials had stayed adequately cold throughout shipping and handling, otherwise laboratory results could be invalid or incorrect. Stevens became interested in technical solutions to these problems; he wanted some way to verify that samples were at acceptable temperatures both in storage and in transit.

Moreover, Stevens imagined that these sensors would be in continuous communication. There's a lot of overlap between this application and personal area networks (PANs), protocols like Bluetooth that provide low-power communications over short ranges. There is also clear overlap with RFID; you can buy RFID temperature sensors. VAI, though, coined the term visibility network to describe RuBee. That's visibility as in asset visibility: somewhat different from Bluetooth or RFID, RuBee as a protocol is explicitly designed for situations where you need to "keep tabs" on a number of different objects. Despite the overlap with other types of wireless communications, the set of requirements on a visibility network have led RuBee down a very different technical path.

Visibility networks have to be highly reliable. When you are trying to keep track of an asset, a failure to communicate with it represents a fundamental failure of the system. For visibility networks, the ability to actually convey a payload is secondary: the main function is just reliably detecting that endpoints exist. Visibility networks have this in common with RFID, and indeed, despite its similarities to technologies like BLE RuBee is positioned mostly as a competitor to technologies like UHF RFID.

There are several differences between RuBee and RFID; for example, RuBee uses active (battery-powered) tags and the tags are generally powered by a complete 4-bit microcontroller. That doesn't necessarily sound like an advantage, though. While RuBee tags advertise a battery life of "5-25 years", the need for a battery seems mostly like a liability. The real feature is what active tags enable: RuBee operates in the low frequency (LF) band, typically at 131 kHz.

At that low frequency, the wavelength is very long, about 2.5 km. With such a long wavelength, RuBee communications all happen at much less than one wavelength in range. RF engineers refer to this as near-field operation, and it has some properties that are intriguingly different from more typical far-field RF communications. In the near-field, the magnetic field created by the antenna is more significant than the electrical field. RuBee devices are intentionally designed to emit very little electrical RF signal. Communications within a RuBee network are achieved through magnetic, not electrical fields. That's the core of RuBee's magic.

The idea of magnetic coupling is not unique to RuBee. Speaking of the near-field, there's an obvious comparison to NFC which works much the same way. The main difference, besides the very different logical protocols, is that NFC operates at 13.56 MHz. At this higher frequency, the wavelength is only around 20 meters. The requirement that near-field devices be much closer than a full wavelength leads naturally to NFC's very short range, typically specified as 4 cm.

At LF frequencies, RuBee can achieve magnetic coupling at ranges up to about 30 meters. That's a range comparable to, and often much better than, RFID inventory tracking technologies. Improved range isn't RuBee's only benefit over RFID. The properties of magnetic fields also make it a more robust protocol. RuBee promises significantly less vulnerability to shielding by metal or water than RFID.

There are two key scenarios where this comes up: the first is equipment stored in metal containers or on metal shelves, or equipment that is itself metallic. In that scenario, it's difficult to find a location for an RFID tag that won't suffer from shielding by the container. The case of water might seem less important, but keep in mind that people are made mostly of water. RFID reading is often unreliable for objects carried on a person, which are likely to be shielded from the reader by the water content of the body.

These problems are not just theoretical. WalMart is a major adopter of RFID inventory technology, and in early rollouts struggled with low successful read rates. Metal, moisture (including damp cardboard boxes), antenna orientation, and multipath/interference effects could cause read failure rates as high as 33% when scanning a pallet of goods. Low read rates are mostly addressed by using RFID "portals" with multiple antennas. Eight antennas used as an array greatly increase read rate, but at a cost of over ten thousand dollars per portal system. Even so, WalMart seems to now target a success rate of only 95% during bulk scanning.

95% might sound pretty good, but there are a lot of visibility applications where a failure rate of even a couple percent is unacceptable. These mostly go by the euphemism "high value goods," which depending on your career trajectory you may have encountered in corporate expense and property policies. High-value goods tend to be items that are both attractive to theft and where theft has particularly severe consequences. Classically, firearms and explosives. Throw in classified material for good measure.

I wonder if Stevens was surprised by RuBee's market trajectory. He came out of the healthcare industry and, it seems, originally developed RuBee for cold chain visibility... but, at least in retrospect, it's quite obvious that its most compelling application is in the armory.

Because RuBee tags are small and largely immune to shielding by metals, you can embed them directly in the frames of firearms, or as an aftermarket modification you can mill out some space under the grip. RuBee tags in weapons will read reliably when they are stored in metal cases or on metal shelving, as is often the case. They will even read reliably when a weapon is carried holstered, close to a person's body.

Since RuBee tags incorporate an active microcontroller, there are even more possibilities. Temperature logging is one thing, but firearm-embedded RuBee tags can incorporate an accelerometer (NIST-traceable, VAI likes to emphasize) and actually count the rounds fired.

---

Sidebar time: there is a long history of political hazard around "smart guns." The term "smart gun" is mostly used more specifically for firearms that identify their user, for example by fingerprint authentication or detection of an RFID fob. The idea has become vague enough, though, that mention of a firearm with any type of RFID technology embedded would probably raise the specter of the smart gun to gun-rights advocates.

Further, devices embedded in firearms that count the number of rounds fired have been proposed for decades, if not a century, as a means of accountability. The holder of a weapon could, in theory, be required to positively account for every round fired. That could eliminate incidents of unreported use of force by police, for example. In practice I think this is less compelling than it sounds, simple counting of rounds leaves too many opportunities to fudge the numbers and conceal real-world use of a weapon as range training, for example.

That said, the NRA has long been vehemently opposed to the incorporation of any sort of technology into weapons that could potentially be used as a means of state control or regulation. The concern isn't completely unfounded; the state of New Jersey did, for a time, have legislation that would have made user-identifying "smart guns" mandatory if they were commercially available. The result of the NRA's strident lobbying is that no such gun has ever become commercially available; "smart guns" have been such a political third rail that any firearms manufacturer that dared to introduce one would probably face a boycott by most gun stores. For better or worse, a result of the NRA's powerful political advocacy in this area is that the concept of embedding security or accountability technology into weapons has never been seriously pursued in the US. Even a tentative step in that direction can produce a huge volume of critical press for everyone involved.

I bring this up because I think it explains some of why VAI seems a bit vague and cagey about the round-counting capabilities of their tags. They position it as purely a maintenance feature, allowing the armorer to keep accurate tabs on the preventative maintenance schedule for each individual weapon (in armory environments, firearm users are often expected to report how many rounds they fired for maintenance tracking reasons). The resistance of RuBee tags to concealment is only positioned as a deterrent to theft, although the idea of RuBee-tagged firearms creates obvious potential for security screening. Probably the most profitable option for VAI would be to promote RuBee-tagged firearms as tool for enforcement of gun control laws, but this is a political impossibility and bringing it up at all could cause significant reputational harm, especially with the government as a key customer. The result is marketing copy that is a bit odd, giving a set of capabilities that imply an application that is never mentioned.

---

VAI found an incredible niche with their arms-tracking application. Institutional users of firearms, like the military, police, and security forces, are relatively price-insensitive and may have strict accounting requirements. By the mid-'00s, VAI was into the long sales cycle of proposing the technology to the military. That wasn't entirely unsuccessful. RuBee shot-counting weapon inventory tags were selected by the Naval Surface Warfare Center in 2010 for installation on SCAR and M4 rifles. That contract had a five-year term, it's unclear to me if it was renewed. Military contracting opened quite a few doors to VAI, though, and created a commercial opportunity that they eagerly pursued.

Perhaps most importantly, weapons applications required an impressive round of safety and compatibility testing. RuBee tags have the fairly unique distinction of military approval for direct attachment to ordnance, something called "zero separation distance" as the tags do not require a minimum separation from high explosives. Central to that certification are findings of intrinsic safety of the tags (that they do not contain enough energy to trigger explosives) and that the magnetic fields involved cannot convey enough energy to heat anything to dangerous temperatures.

That's not the only special certification that RuBee would acquire. The military has a lot of firearms, but military procurement is infamously slow and mercurial. Improved weapon accountability is, almost notoriously, not a priority for the US military which has often had stolen weapons go undetected until their later use in crime. The Navy's interest in RuBee does not seem to have translated to more widespread military applications.

Then you have police departments, probably the largest institutional owners of firearms and a very lucrative market for technology vendors. But here we run into the political hazard: the firearms lobby is very influential on police departments, as are police unions which generally oppose technical accountability measures. Besides, most police departments are fairly cash-poor and are not likely to make a major investment in a firearms inventory system.

That leaves us with institutional security forces. And there is one category of security force that are particularly well-funded, well-equipped, and beholden to highly R&D-driven, almost pedantic standards of performance: the protection forces of atomic energy facilities.

Protection forces at privately-operated atomic energy facilities, such as civilian nuclear power plants, are subject to licensing and scrutiny by the Nuclear Regulatory Commission. Things step up further at the many facilities operated by the National Nuclear Security Administration (NNSA). Protection forces for NNSA facilities are trained at the Department of Energy's National Training Center, at the former Manzano Base here in Albuquerque. Concern over adequate physical protection of NNSA facilities has lead Sandia National Laboratories to become one of the premier centers for R&D in physical security. Teams of scientists and engineers have applied sometimes comical scientific rigor to "guns, gates, and guards," the traditional articulation of physical security in the nuclear world.

That scope includes the evaluation of new technology for the management of protection forces, which is why Oak Ridge National Laboratory launched an evaluation program for the RuBee tagging of firearms in their armory. The white paper on this evaluation is curiously undated, but citations "retrieved 2008" lead me to assume that the evaluation happened right around the middle of the '00s. At the time, VAI seems to have been involved in some ultimately unsuccessful partnership with Oracle, leading to the branding of the RuBee system as Oracle Dot-Tag Server. The term "Dot-Tag" never occurs outside of very limited materials around the Oracle partnership, so I'm not sure if it was Oracle branding for RuBee or just some passing lark. In any case, Oracle's involvement seems to have mainly just been the use of the Oracle database for tracking inventory data---which was naturally replaced by PostgreSQL at Oak Ridge.

The Oak Ridge trial apparently went well enough, and around the same time, the Pantex Plant in Texas launched an evaluation of RuBee for tracking classified tools. Classified tools are a tricky category, as they're often metallic and often stored in metallic cases. During the trial period, Pantex tagged a set of sample classified tools with RuBee tags and then transported them around the property, testing the ability of the RuBee controllers to reliably detect them entering and exiting areas of buildings. Simultaneously, Pantex evaluated the use of RuBee tags to track containers of "chemical products" through the manufacturing lifecycle. Both seem to have produced positive results.

There are quite a few interesting and strange aspects of the RuBee system, a result of its purpose-built Visibility Network nature. A RuBee controller can have multiple antennas that it cycles through. RuBee tags remain in a deep-sleep mode for power savings until they detect a RuBee carrier during their periodic wake cycle. When a carrier is detected, they fully wake and listen for traffic. A RuBee controller can send an interrogate message and any number of tags can respond, with an interesting and novel collision detection algorithm used to ensure reliable reading of a large number of tags.

The actual RuBee protocol is quite simple, and can also be referred to as IEEE 1902.1 since the decision of VAI to put it through the standards process. Packets are small and contain basic addressing info, but they can also contain arbitrary payload in both directions, perfect for data loggers or sensors. RuBee tags are identified by something that VAI oddly refers to as an "IP address," causing some confusion over whether or not VAI uses IP over 1902.1. They don't, I am confident saying after reading a whole lot of documents. RuBee tags, as standard, have three different 4-byte addresses. VAI refers to these as "IP, subnet, and MAC," ² but these names are more like analogies. Really, the "IP address" and "subnet" are both configurable arbitrary addresses, with the former intended for unicast traffic and the latter for broadcast. For example, you would likely give each asset a unique IP address, and use subnet addresses for categories or item types. The subnet address allows a controller to interrogate for every item within that category at once. The MAC address is a fixed, non-configurable address derived from the tag's serial number. They're all written in the formats we associate with IP networks, dotted-quad notation, as a matter of convenience.

And that's about it as far as the protocol specification, besides of course the physical details which are a 131,072 Hz carrier, 1024 Hz data clock, either ASK or BPSK modulation. The specification also describes an interesting mode called "clip," in which a set of multiple controllers interrogate in exact synchronization and all tags then reply in exact synchronization. Somewhat counter-intuitively, because of the ability of RuBee controllers to separate out multiple simultaneous tag transmissions using an anti-collision algorithm based on random phase shifts by each tag, this is ideal. It allows a room, say an armory, full of RuBee controllers to rapidly interrogate the entire contents of the room. I think this feature may have been added after the Oak Ridge trials...

RuBee is quite slow, typically 1,200 baud, so inventorying a large number of assets can take a while (Oak Ridge found that their system could only collect data on 2-7 tags per second per controller). But it's so robust that it an achieve a 100% read rate in some very challenging scenarios. Evaluation by the DoE and the military produced impressive results. You can read, for example, of a military experiment in which a RuBee antenna embedded in a roadway reliably identified rifles secured in steel containers in passing Humvees.

Paradoxically, then, one of the benefits of RuBee in the military/defense context is that it is also difficult to receive. Here is RuBee's most interesting trick: somewhat oversimplified, the strength of an electrical radio signal goes as 1/r, while the strength of a magnetic field goes as 1/r^3. RuBee equipment is optimized, by antenna design, to produce a minimal electrical field. The result is that RuBee tags can very reliably be contacted at short range (say, around ten feet), but are virtually impossible to contact or even detect at ranges over a few hundred feet. To the security-conscious buyer, this is a huge feature. RuBee tags are highly resistant to communications or electronic intelligence collection.

Consider the logical implications of tagging the military's rifles. With conventional RFID, range is limited by the size and sensitivity of the antenna. Particularly when tags are incidentally powered by a nearby reader, an adversary with good equipment can detect RFID tags at very long range. VAI heavily references a 2010 DEFCON presentation, for example, that demonstrated detection of RFID tags at a range of 80 miles. One imagines that opportunistic detection by satellite is feasible for a state intelligence agency. That means that your rifle asset tracking is also revealing the movements of soldiers in the field, or at least providing a way to detect their approach.

Most RuBee tags have their transmit power reduced by configuration, so even the maximum 100' range of the protocol is not achievable. VAI suggests that typical RuBee tags cannot be detected by radio direction finding equipment at ranges beyond 20', and that this range can be made shorter by further reducing transmit power.

Once again, we have caught the attention of the Department of Energy. Because of the short range of RuBee tags, they have generally been approved as not representing a COMSEC or TEMPEST hazard to secure facilities. And that brings us back to the very beginning: why does the DoE use a specialized, technically interesting, and largely unique radio protocol to fulfill such a basic function as nagging people that have their phones? Because RuBee's security properties have allowed it to be approved for use adjacent to and inside of secure facilities. A RuBee tag, it is thought, cannot be turned into a listening device because the intrinsic range limitation of magnetic coupling will make it impossible to communicate with the tag from outside of the building. It's a lot like how infrared microphones still see some use in secure facilities, but so much more interesting!

VAI has built several different product lines around RuBee, with names like Armory 20/20 and Shot Counting Allegro 20/20 and Store 20/20. The founder started his career in eye health, remember. None of them are that interesting, though. They're all pretty basic CRUD applications built around polling multiple RuBee controllers for tags in their presence.

And then there's the "Alert 20/20 DoorGuard:" a metal pedestal with a RuBee controller and audio announcement module, perfect for detecting government cell phones.

One of the strangest things about RuBee is that it's hard to tell if it's still a going concern. VAI's website has a press release section, where nothing has been posted since 2019. The whole website feels like it was last revised even longer ago. When RuBee was newer, back in the '00s, a lot of industry journals covered it with headlines like "the new RFID." I think VAI was optimistic that RuBee could displace all kinds of asset tracking applications, but despite some special certifications in other fields (e.g. approval to use RuBee controllers and tags around pacemakers in surgical suites), I don't think RuBee has found much success outside of military applications.

RuBee's resistance to shielding is impressive, but RFID read rates have improved considerably with new DSP techniques, antenna array designs, and the generally reduced cost of modern RFID equipment. RuBee's unique advantages, its security properties and resistance to even intentional exfiltration, are interesting but not worth much money to buyers other than the military.

So that's the fate of RuBee and VAI: defense contracting. As far as I can tell, RuBee and VAI are about as vital as they have ever been, but RuBee is now installed as just one part of general defense contracts around weapons systems, armory management, and process safety and security. IEEE standardization has opened the door to use of RuBee by federal contractors under license, and indeed, Lockheed Martin is repeatedly named as a licensee, as are firearms manufacturers with military contracts like Sig Sauer.

Besides, RuBee continues to grow closer to the DoE. In 2021, VAI appointed Lisa Gordon-Hagerty to it board of directors. Gordon-Hagerty was undersecretary of Energy and had lead the NNSA until the year before. This year, the New Hampshire Small Business Development Center wrote a glowing profile of VAI. They described it as a 25-employee company with a goal of hitting $30 million in annual revenue in the next two years.

Despite the outdated website, VAI claims over 1,200 RuBee sites in service. I wonder how many of those are Alert 20/20 DoorGuards? Still, I do believe there are military weapons inventory systems currently in use. RuBee probably has a bright future, as a niche technology for a niche industry. If nothing else, they have legacy installations and intellectual property to lean on. A spreadsheet of VAI-owned patents on RuBee, with nearly 200 rows, encourages would-be magnetically coupled visibility network inventors not to go it on their own. I just wish I could get my hands on a controller....

How much do you remember from elementary school? I remember vinyl tile floors, the playground, the teacher sentencing me to standing in the hallway. I had a teacher who was a chess fanatic; he painted a huge chess board in the paved schoolyard and got someone to fabricate big wooden chess pieces. It was enough of an event to get us on the evening news. I remember Run for the Arts, where I tried to talk people into donating money on the theory that I could run, which I could not. I'm about six months into trying to change that and I'm good for a mediocre 5k now, but I don't think that's going to shift the balance on K-12 art funding.

I also remember a domain name: bridger.pps.k12.or.us

I have quipped before that computer science is a field mostly concerned with assigning numbers to things, which is true, but it only takes us so far. Computer scientists also like to organize those numbers into structures, and one of their favorites has always been the tree. The development of wide-area computer networking surfaced a whole set of problems around naming or addressing computer systems that belong to organizations. A wide-area network consists of a set of institutions that manage their own affairs. Each of those institutions may be made up of departments that manage their own affairs. A tree seemed a natural fit. Even the "low level" IP addresses, in the days of "classful" addressing, were a straightforward hierarchy: each dot separated a different level of the tree, a different step in an organizational hierarchy.

The first large computer networks, including those that would become the Internet, initially relied on manually building lists of machines by name. By the time the Domain Name System was developed, this had already become cumbersome. The rapid growth of the internet was hard to keep up with, and besides, why did any one central entity---Jon Postel or whoever---even care about the names of all of the computers at Georgia Tech? Like IP addressing, DNS was designed as a hierarchy with delegated control. A registrant obtains a name in the hierarchy, say gatech.edu, and everything "under" that name is within the control, and responsibility, of the registrant. This arrangement is convenient for both the DNS administrator, which was a single organization even after the days of Postel, and for registrants.

We still use the same approach today... mostly. The meanings of levels of the hierarchy have ossified. Technically speaking, the top of the DNS tree, the DNS root, is a null label referenced by a trailing dot. It's analogous to the '/' at the beginning of POSIX file paths. "gatech.edu" really should be written as "gatech.edu." to make it absolute rather than relative, but since resolution of relative URLs almost always recurses to the top of the tree, the trailing dot is "optional" enough that it is now almost always omitted. The analogy to POSIX file paths raises an interesting point: domain names are backwards. The 'root' is at the end, rather than at the beginning, or in other words, they run from least significant to most significant, rather than most significant to least significant. That's just... one of those things, you know? In the early days one wasn't obviously better than the other, people wrote hierarchies out both ways, and as the dust settled the left-to-right convention mostly prevailed but right-to-left hung around in some protocols. If you've ever dealt with endianness, this is just one of those things about computers that you have to accept: we cannot agree on which way around to write things.

Anyway, the analogy to file paths also illustrates the way that DNS has ossified. The highest "real" or non-root component of a domain name is called the top-level domain or TLD, while the component below it is called a second-level domain. In the US, it was long the case that top-level domains were fixed while second-level domains were available for registration. There have always been exceptions in other countries and our modern proliferation of TLDs has changed this somewhat, but it's still pretty much true. When you look at "gatech.edu" you know that "edu" is just a fixed name in the hierarchy, used to organize domain names by organization type, while "gatech" is a name that belongs to a registrant.

Under the second-level name, things get a little vague. We are all familiar with the third-level name "www," which emerged as a convention for web servers and became a practical requirement. Web servers having the name "www" under an organization's domain was such a norm for so many years that hosting a webpage directly at a second-level name came to be called a "naked domain" and had some caveats and complications.

Other than www, though, there are few to no standards for the use of third-level and below names. Larger organizations are more likely to use third-level names for departments, infrastructure operators often have complex hierarchies of names for their equipment, and enterprises the world 'round name their load-balanced webservers "www2," "www3" and up. If you think about it, this situation seems like kind of a failure of the original concept of DNS... we do use the hierarchy, but for the most part it is not intended for human consumption. Users are only expected to remember two names, one of which is a TLD that comes from a relatively constrained set.

The issue is more interesting when we consider geography. For a very long time, TLDs have been split into two categories: global TLDs, or gTLDs, and country-code TLDs, or ccTLDs. ccTLDs reflect the ISO country codes of each country, and are intended for use by those countries, while gTLDs are arbitrary and reflect the fact that DNS was designed in the US. The ".gov" gTLD, for example, is for use by the US government, while the UK is stuck with ".gov.uk". This does seem unfair but it's now very much cemented into the system: for the large part, US entities use gTLDs, while entities in other countries use names under their respective ccTLDs. The ".us" ccTLD exists just as much as all the others, but is obscure enough that my choice to put my personal website under .us (not an ideological decision but simply a result of where a nice form of my name was available) sometimes gets my email address rejected.

Also, a common typo for ".us" is ".su" and that's geopolitically amusing. .su is of course the ccTLD for the Soviet Union, which no longer exists, but the ccTLD lives on in a limited way because it became Structurally Important and difficult to remove, as names and addresses tend to do.

We can easily imagine a world where this historical injustice had been fixed: as the internet became more global, all of our US institutions could have moved under the .us ccTLD. In fact, why not go further? Geographers have long organized political boundaries into a hierarchy. The US is made up of states, each of which has been assigned a two-letter code by the federal government. We have ".us", why not "nm.us"?

The answer, of course, is that we do.

In the modern DNS, all TLDs have been delegated to an organization who administers them. The .us TLD is rightfully administered by the National Telecommunications and Information Administration, on the same basis by which all ccTLDs are delegated to their respective national governments. Being the US government, NTIA has naturally privatized the function through a contract to telecom-industrial-complex giant Neustar. Being a US company, Neustar restructured and sold its DNS-related business to GoDaddy. Being a US company, GoDaddy rose to prominence on the back of infamously tasteless television commercials, and its subsidiary Registry Services LLC now operates our nation's corner of the DNS.

But that's the present---around here, we avoid discussing the present so as to hold crushing depression at bay. Let's turn our minds to June 1993, and the publication of RFC 1480 "The US Domain." To wit:

Even though the original intention was that any educational institution anywhere in the world could be registered under the EDU domain, in practice, it has turned out with few exceptions, only those in the United States have registered under EDU, similarly with COM (for commercial). In other countries, everything is registered under the 2-letter country code, often with some subdivision. For example, in Korea (KR) the second level names are AC for academic community, CO for commercial, GO for government, and RE for research. However, each country may go its own way about organizing its domain, and many have.

Oh, so let's sort it out!

There are no current plans of putting all of the organizational domains EDU, GOV, COM, etc., under US. These name tokens are not used in the US Domain to avoid confusion.

Oh. Oh well.

Currently, only four year colleges and universities are being registered in the EDU domain. All other schools are being registered in the US Domain.

Huh?

RFC 1480 is a very interesting read. It makes passing references to so many facets of DNS history that could easily be their own articles. It also defines a strict, geography-based hierarchy for the .us domain that is a completely different universe from the one in which we now live. For example, we learned above that, in 1993, only four-year institutions were being placed under .edu. What about the community colleges? Well, RFC 1480 has an answer. Central New Mexico Community College would, of course, fall under cnm.cc.nm.us. Well, actually, in 1993 it was called the Technical-Vocational Institute, so it would have been tvi.tec.nm.us. That's right, the RFC describes both "cc" for community colleges and "tec" for technical institutes.

Even more surprising, it describes placing entities under a "locality" such as a city. The examples of localities given are "berkeley.ca.us" and "portland.wa.us", the latter of which betrays an ironic geographical confusion. It then specifies "ci" for city and "co" for county, meaning that the city government of our notional Portland, Washington would be ci.portland.wa.us. Agencies could go under the city government component (the RFC gives the example "Fire-Dept.CI.Los-Angeles.CA.US") while private businesses could be placed directly under the city (e.g. "IBM.Amonk.NY.US"). The examples here reinforce that the idea itself is different from how we use DNS today: The DNS of RFC 1480 is far more hierarchical and far more focused on full names, without abbreviations.

Of course, the concept is not limited to local government. RFC 1480 describes "fed.us" as a suffix for the federal government (the example "dod.fed.us" illustrates that this has not at all happened), and even "General Independent Entities" and "Distributed National Institutes" for those trickier cases.

We can draw a few lessons from how this proposal compares to our modern day. Back in the 1990s, .gov was limited to the federal government. The thinking was that all government agencies would move into .us, where the hierarchical structure made it easier to delegate management of state and locality subtrees. What actually happened was the opposite: the .us thing never really caught on, and a more straightforward and automated management process made .gov available to state and local governments. The tree has effectively been flattened.

That's not to say that none of these hierarchical names saw use. GoDaddy continues to maintain what they call the "usTLD Locality-Based Structure". At the decision of the relevant level of the hierarchy (e.g. a state), locality-based subdomains of .us can either be delegated to the state or municipality to operate, or operated by GoDaddy itself as the "Delegated Manager." The latter arrangement is far more common, and it's going to stay that way: RFC 1480 names are not dead, but they are on life support. GoDaddy's contract allows them to stop onboarding any additional delegated managers, and they have.

Few of these locality-based names found wide use, and there are even fewer left today. Multnomah County Library once used "multnomah.lib.or.us," which I believe was actually the very first "library" domain name registered. It now silently redirects to "multcolib.org", which we could consider a graceful name only in that the spelling of "Multnomah" is probably not intuitive to those not from the region. As far as I can tell, the University of Oregon and OGI (part of OHSU) were keeping very close tabs on the goings-on of academic DNS, as Oregon entities are conspicuously over-represented in the very early days of RFC 1480 names---behind only California, although Georgia Tech and Trent Heim of former Colorado company XOR both registered enough names to give their states a run for the money.

"co.bergen.nj.us" works, but just gets you a redirect notice page to bergencountynj.gov. It's interesting that this name is actually longer than the RFC 1480 name, but I think most people would agree that bergencountynj.gov is easier to remember. Some of that just comes down to habit, we all know ".gov", but some of it is more fundamental. I don't think that people often understand the hierarchical structure of DNS, at least not intuitively, and that makes "deeply hierarchical" (as GoDaddy calls them) names confusing.

Certainly the RFC 1480 names for school districts produced complaints. They were also by far the most widely adopted. You can pick and choose examples of libraries (.lib.[state].us) and municipal governments that have used RFC 1480 names, but school districts are another world: most school districts that existed at the time have a legacy of using RFC 1480 naming. As one of its many interesting asides, RFC 1480 explains why: the practice of putting school districts under [district].k12.[state].us actually predates RFC 1480. Indeed, the RFC seems to have been written in part to formalize the existing practice. The idea of the k12.[state].us hierarchy originated within IANA in consultation with InterNIC (newly created at the time) and the Federal Networking Council, a now-defunct advisory committee of federal agencies that made a number of important early decisions about internet architecture.

RFC 1480 is actually a revision on the slightly older RFC 1386, which instead of saying that schools were already using the k12 domains, says that "there ought to be a consistent scheme for naming them." It then says that the k12 branch has been "introduced" for that purpose. RFC 1386 is mostly silent on topics other than schools, so I think it was written to document the decision made about schools with other details about the use of locality-based domains left sketchy until the more thorough RFC 1480.

The decision to place "k12" under the state rather than under a municipality or county might seem odd, but the RFC gives a reason. It's not unusual for school districts, even those named after a municipality, to cover a larger area than the municipality itself. Albuquerque Public Schools operates schools in the East Mountains; Portland Public Schools operates schools across multiple counties and beyond city limits. Actually the RFC gives exactly that second one as an example:

For example, the Portland school district in Oregon, is in three or four counties. Each of those counties also has non-Portland districts.

I include that quote mostly because I think it's funny that the authors now know what state Portland is in. When you hear "DNS" you think Jon Postel, at least if you're me, but RFC 1480 was written by Postel along with a less familiar name, Ann Westine Cooper. Cooper was a coworker of Postel at USC, and RFC 1480 very matter-of-factly names the duo of Postel and Cooper as the administrator of the .US TLD. That's interesting considering that almost five years later Postel would become involved in a notable conflict with the federal government over control of DNS---one of the events that precipitated today's eccentric model of public-private DNS governance.

There are other corners of the RFC 1480 scheme that were not contemplated in 1993, and have managed to outlive many of the names that were. Consider, for example, our indigenous nations: these are an exception to the normal political hierarchy of the US. The Navajo Nation, for example, exists in a state that is often described as parallel to a state, but isn't really. Native nations are sovereign, but are also subject to federal law by statute, and subject to state law by various combinations of statute, jurisprudence, and bilateral agreement. I didn't really give any detail there and I probably still got something wrong, such is the complicated legal history and present of Native America. So where would a native sovereign government put their website? They don't fall under the traditional realm of .gov, federal government, nor do they fall under a state-based hierarchy. Well, naturally, the Navajo Nation is found at navajo-nsn.gov.

We can follow the "navajo" part but the "nsn" is odd, unless they spelled "nation" wrong and then abbreviated it, which I've always thought is what it looks like on first glance. No, this domain name is very much an artifact of history. When the problem of sovereign nations came to Postel and Cooper, the solution they adopted was a new affinity group, like "fed" and "k12" and "lib": "nsn", standing for Native Sovereign Nation. Despite being a late comer, nsn.us probably has the most enduring use of any part of the RFC 1480 concept. Dozens of pueblos, tribes, bands, and confederations still use it. squamishtribe.nsn.us, muckleshoot.nsn.us, ctsi.nsn.us, sandiapueblo.nsn.us.

Yet others have moved away... in a curiously "partial" fashion. navajo-nsn.gov as we have seen, but an even more interesting puzzler is tataviam-nsn.us. It's only one character away from a "standardized" NSN affinity group locality domain, but it's so far away. As best I can tell, most of these governments initially adopted "nsn.us" names, which cemented the use of "nsn" in a similar way to "state" or "city" as they appear in many .gov domains to this day. Policies on .gov registration may be a factor as well, the policies around acceptable .gov names seem to have gone through a long period of informality and then changed a number of times. Without having researched it too deeply, I have seen bits and pieces that make me think that at various points NTIA has preferred that .gov domains for non-federal agencies have some kind of qualifier to indicate their "level" in the political hierarchy. In any case, it's a very interesting situation because "native sovereign nation" is not otherwise a common term in US government. It's not like lawyers or lawmakers broadly refer to tribal governments as NSNs, the term is pretty much unique to the domain names.

So what ever happened to locality-based names? RFC 1480 names have fallen out of favor to such an extent as to be considered legacy by many of their users. Most Americans are probably not aware of this name hierarchy at all, despite it ostensibly being the unified approach for this country. In short, it failed to take off, and those sectors that had widely adopted it (such as schools) have since moved away. But why?

As usual, there seem to be a few reasons. The first is user-friendliness. This is, of course, a matter of opinion---but anecdotally, many people seem to find deeply hierarchical domain names confusing. This may be a self-fulfilling prophecy, since the perception that multi-part DNS names are user-hostile means that no one uses them which means that no users are familiar with them. Maybe, in a different world, we could have broken out of that loop. I'm not convinced, though. In RFC 1480, Postel and Cooper argue that a deeper hierarchy is valuable because it allows for more entities to have their "obviously correct" names. That does make sense to me, splitting the tree up into more branches means that there is less name contention within each branch. But, well, I think it might be the kind of logic that is intuitive only those who work in computing. For the general public, I think long multi-part names quickly become difficult to remember and difficult to type. When you consider the dollar amounts that private companies have put into dictionary word domain names, it's no surprise that government agencies tend to prefer one-level names with full words and simple abbreviations.

I also think that the technology outpaced the need that RFC 1480 was intended to address. The RFC makes it very clear that Postel and Cooper were concerned about the growing size of the internet, and expected the sheer number of organizations going online to make maintenance of the DNS impractical. They correctly predicted the explosion of hosts, but not the corresponding expansion of the DNS bureaucracy. Between the two versions of the .us RFC, DNS operations were contracted to Network Solutions. This began a winding path that lead to delegation of DNS zones to various private organizations, most of which fully automated registration and delegation and then federated it via a common provisioning protocol. The size of, say, the .com zone really did expand beyond what DNS's designers had originally anticipated... but it pretty much worked out okay. The mechanics of DNS's maturation probably had a specifically negative effect on adoption of .us, since it was often under a different operator from the "major" domain names and not all "registrars" initially had access.

Besides, the federal government never seems to have been all that on board with the concept. RFC 1480 could be viewed as a casualty of the DNS wars, a largely unexplored path on the branch of DNS futures that involved IANA becoming completely independent of the federal government. That didn't happen. Instead, in 2003 .gov registration was formally opened to municipal, state, and tribal governments. It became federal policy to encourage use of .gov for trust reasons (DNSSEC has only furthered this), and .us began to fall by the wayside.

That's not to say that RFC 1480 names have ever gone away. You can still find many of them in use. state.nm.us doesn't have an A record, but governor.state.nm.us and a bunch of other examples under it do. The internet is littered with these locality-based names, many of them hiding out in smaller agencies and legacy systems. Names are hard to get right, and one of the reasons is that they're very hard to get rid of.

When things are bigger, names have to be longer. There is an argument that with only 8-character names, and in each position allow a-z, 0-9, and -, you get 37**8 = 3,512,479,453,921 or 3.5 trillion possible names. It is a great argument, but how many of us want names like "xs4gp-7q". It is like license plate numbers, sure some people get the name they want on a vanity plate, but a lot more people who want something specific on a vanity plate can't get it because someone else got it first. Structure and longer names also let more people get their "obviously right" name.

You look at Reddit these days and see all these usernames that are two random words and four random numbers, and you see that Postel and Cooper were right. Flat namespaces create a problem, names must either be complex or long, and people don't like it either. What I think they got wrong, at a usability level, is that deep hierarchies still create names that are complex and long. It's a kind of complexity that computer scientists are more comfortable with, but that's little reassurance when you're staring down the barrel of "bridger.pps.k12.or.us".

We think that we're converting time into energy... that's the engineering principle.

In the 1820s, stacks, ovens, and gasometers rose over the docklands of Dublin. The Hibernian Gas Company, one of several gasworks that would eventually occupy the land around the Grand Canal Docks, heated coal to produce town gas. That gas would soon supply thousands of lights on the streets of Dublin, a quiet revolution in municipal development that paved the way for electrification---both conceptually, as it proved the case for public lighting, and literally, as town gas fired the city's first small power plants.

Ireland's supply of coal became scarce during the Second World War; as part of rationing of the town gas supply most street lights were shut off. Town gas would never make a full recovery. By that time, electricity had proven its case for lighting. Although coal became plentiful after the war, imported from England and transported from the docks to the gasworks by horse teams, even into the 1960s---this form of energy had become obsolete. In the 1980s, the gasworks stoked their brick retorts for the last time. Natural gas had arrived. It was cheaper, cleaner, safer.

The Docklands still echo with the legacy of the town gas industry. The former site of the Hibernian gasworks is now the Bord Gáis Energy Theatre, a performing arts center named for the British-owned energy conglomerate that had once run Ireland's entire gas industry as a state monopoly. Metal poles, the Red Sticks, jut into the sky from the Grand Canal Square. They are an homage to the stacks that once loomed over the industrial docks. Today, those docks have turned over to gastropubs, offices, the European headquarters of Google.

Out on the water, a new form of energy once spun to life. In December of 2009, a man named Sean McCarthy booked the Waterways Ireland Visitors Centre for a press event. In the dockside event space, and around the world through a live stream, he invited the public to witness a demonstration of his life's work. The culmination of three years of hype, controversy, and no small amount of ridicule, this was his opportunity to prove what famed physicist Michio Kaku and his own hand-picked jury of scientists had called a sham.

He had invented a perpetual motion machine.

---

Sean McCarthy had his start in the energy industry. Graduating with an engineering degree in the 1980s, he took a software position with a company that quickly became part of European energy equipment giant ABB. By some route now lost to history, he made is way into sales. 1994 found McCarthy in Bahrain, managing sales for the Middle East.

McCarthy is clearly a strong salesman. In Bahrain, he takes credit for growing ABB's business by orders of magnitude. Next, ABB sent him to Houston, where he worked sales to Central and South America. In 1999, the company called him to Azerbaijan---but he chose otherwise, leaving ABB and returning to Dublin.

Contacts from the energy industry, and no doubt his start in software, brought Graham to the world of the internet. 1999 in Dublin was, like so many places, crowded with dotcom boom startups. McCarthy saw opportunity: not necessarily by joining the fray, but by selling expertise and experience to those who already had. With his brother-in-law, a lawyer, he formed Steorn.

The nature of Steorn's business during these early years is a bit hazy. As McCarthy recounts it, to Barry Whyte for the book The Impossible Dream, Steorn gathered some early support from the Dublin internet industry that connected him with both capital and clients. One of their most notable clients, attested by several sources, was dotcom wonder WorldOfFruit.com. The World of Fruit, a subsidiary of fruit importer Fyffes, was a web-based auction platform for wholesale import and export of fruit. I have not been able to clarify how exactly Steorn was involved in World Of Fruit, although it cannot be ignored that the aforementioned brother-in-law and cofounder, Francis Hackett, had been the general counsel of World Of Fruit before becoming involved in Steorn. As for the future of online fruit trading, it went about as well as most dotcom-era market platforms did, stumbling on for a few years before folding with millions of Euro lost.

Later, by McCarthy's telling, Steorn shifted its business more towards R&D. They worked on a project for a client fighting credit card fraud, which led to later work for Irish and British law enforcement organizations, mostly related to video surveillance and machine vision. For Ireland's DART commuter train, for example, Steorn attempted a system that would automatically detect aggressive behavior in platform surveillance video---though it was canceled before completion. These projects floated Steorn through the worst of the dotcom bust and supported an in-house R&D operation that trialed a variety of ideas, including something like an early version of California's modern electronic license plates.

I am not sure who well this account squares with Steorn's public presence. Steorn was incorporated in July of 2000, and when their website appeared in 2001 it advertised "programme management and technical assessment advice for European companies engaging in e-commerce projects." The main emphasis of the website was in the project management experience of McCarthy and the company's CTO, Sean Menzies, formerly of recruiting and staffing firm Zartis. The primary claim was that Steorn could oversee projects to develop and launch e-commerce websites, reducing the risk and expense of jumping into the world wide web.

Steorn has been formed by individuals with significant experience in this area who have recognised that there is an absence in the market of genuinely impartial and objective advice as to the best way of implementing e-commerce ideas.

A few months later, Steorn's business became a little more general: "Steorn is an expert in the field of technology risk management." In August of 2001, Steorn launched its first concrete product offering, one that is curiously absent from McCarthy's later stories of the company. Not only would Steorn supervise technology projects, it had partnered with an insurance company to underwrite surety bonds on the completion of those projects.

Steorn had a proprietary program management methodology which they called GUIDE. Frustratingly, although GUIDE is always styled in all caps, I have not found any indication that it stood for something. What I do know is that GUIDE "offers predictability, visibility, control and results," and that it has some relation to a project planning methodology called DIRECT.

In all truth, while Steorn's 2001 websites are laden with bespoke buzzwords, the project and risk management services they offer seem unremarkable in a good way. Probabalistic project scheduling models, checkpoints and interim deliverables, "an approach which is systematic yet sufficiently flexible." The only way that Steorn's services seem foolish is the extent to which these methods have been adopted by every technology project---or have become so dated or unfashionable as to appear obsolete.

This was 2001, though, and many established companies found themselves plunging into the deep end of the online economy for the first time. No doubt the guidance of an outside expert with experience in online projects could be valuable. But, then, I am a sympathetic audience: they were hiring technical architects. That's my day job! The benefits package was, they said, best in class. Alas, the conditions that may necessitate my own move to Dublin come more than twenty years too late.

Despite McCarthy's description of their multiple projects in security, surveillance, and machine vision, Steorn never seems to have advertised or publicized any work in those areas. In 2003, their website underwent a redesign that added "services for technology inventors" and advertised a subsidiary web hosting provider called HostsMe. Perhaps my view is too colored by the late-2000s context in which I came to understand the internet industry, but pivoting to the low barrier of entry, low chance of success shared hosting market feels like a move of desperation. In 2004, the website changed again, becoming a single page of broken links with a promise that it is "being redesigned."

For this entire period, from late 2001 after the announcement of their risk underwriting service to the company's complete swerve into energy from nothing, I can find no press mentions of Steorn or any of their projects. McCarthy seems to suggest that Steorn had a staff of engineers and technicians, but I struggle to find any names or details. Well, that was a long time ago, and of course the staff are not all listed on LinkedIn. On the other hand, one of McCarthy's investors, shown a laboratory full of white-coated scientists, later came to suspect they were temps hired only to create an appearance.

This is not to accuse McCarthy of lying. Some details definitely check out. For example, their project with Fraudhalt to detect tampering with ATMs via machine vision is the subject of a patent listing Sean McCarthy's name. But I do think that McCarthy is prone to exaggerating his successes and, in the process, muddling the details. When it comes to Steorn's most famous invention, McCarthy and early employee and project manager Michael Daly agree that it was an accidental discovery. Both tell detailed stories of a project for which Steorn purchased and modified small wind turbines. The trouble is that they are not describing the same project. Two different customers, two different contexts, two different objectives; but both lead to the Orbo.

Other attempts at checking into the particulars of Steorn's early business lead in more whimsical directions. To Whyte, McCarthy described a sort of joke project by Steorn to create an "online excuse exchange" called WhatsMyExcuse.com. I can't find any evidence of this project, although there's no particular reason I would. What I can say is that some ideas are evergreen: four months ago, a Reddit user posted that they "built a tiny excuse generator for fun." It's live right now, at whatsmyexcuse.com.

In any case, the nature of Steorn around 2003 remains mysterious to me. McCarthy's claims to have been a regional expert on credit card fraud are hard to square with the complete lack of any mention of this line of business on Steorn's website. On the other hand, Steorn had a confusing but clearly close relationship with Fraudhalt to such an extent that I think Steorn may have gone largely dormant as McCarthy spent his time with the other company.

---

After five years of near silence, a period that McCarthy characterized as modest success, Steorn captured the world's attention with what must be one of the boldest pivots of all time. An August 2006 full-page ad in the Economist began with a George Bernard Shaw quote, "all great truths begin as blasphemies." "Imagine," it reads below, "a world with an infinite supply of pure energy. Never having to recharge your phone. Never having to refuel your car."

Steorn's website underwent a complete redesign, now promoting the free energy device they dubbed the Orbo. A too-credulous press went wild with the story, especially in Ireland where Steorn's sudden prominence fit a narrative of a new Irish technology renaissance. The Irish Independent ran a photo of McCarthy kneeling on the floor beside a contraption of aluminum extrusion and wheels and wiring. "Irish firm claims to have solved energy crisis" is the headline. The photo caption reminds us that it "has been treated with some skepticism in the scientific community because it appears to contravene the laws of physics."

I will not devote much time to a detailed history of the development of the Orbo; that topic is much better addressed by Barry Whyte's book, which benefits greatly from interviews with McCarthy. The CliffsNotes ¹ go something like this: Steorn was working on some surveillance project and wanted to power the cameras with small wind turbines. The exact motivations here are confusing since, whichever version of the story you go with, the cameras were being installed in locations that already had readily available electrical service. It may have been simple greenwashing. In any case, the project lead Steorn to experimenting with modifications to off-the-shelf compact wind turbines.

At some point, presumably one or two years before 2006, it was discovered that a modified turbine seemed to produce more energy in output than it received as input. I have not been able to find enough detail on these experiments to understand what exactly was being done, so I am limited to this vague description. Steorn at first put it out of mind, but McCarthy later returned to this mystery and built a series of prototypes that further demonstrated the effect. A series of physicists and research teams were brought in to evaluate the device. This part of the story is complicated and details conflict, but an opinionated summary is that entrepreneurially-inclined technology development executives tended to find the Orbo promising while academic physicists found the whole thing so laughable that they declined to look at it. Trinity College contained both, and in 2005 McCarthy's discussions with a technology commercialization team there lead to a confrontation with physicist Michael Coey. From that point on, McCarthy viewed the scientific establishment as resistance to be overcome.

McCarthy determined that the only way to bring Orbo to the world was to definitively prove that it worked, and to do so largely without outside experience. For that, he would need equipment and a team. After long discussions with a venture capital firm, McCarthy seems to have balked at the amount of due diligence involved in such formal investment channels. Instead, he brought on a haphazard series of individuals, mostly farmers, who collectively put up about 20 million euro.

---

One of the curious things about the Orbo was Steorn's emphasis on cell phone batteries. I suppose it is understandable, in that ubiquitous mobile phones were new at the time and the hassle of keeping them charged was suddenly a familiar problem to all. It also feels like a remarkable failure of ambition. Having invented a source of unlimited free energy, most of what McCarthy had to say was that you wouldn't have to charge your phone any more. Sure, investment documents around Steorn laid out a timeline to portable generators, vehicles, standby generators for telecom applications... a practical set of concerns, sure, but one that is rather underwhelming from the creator of the first over-unity energy device.

It is also, when you think about it, a curious choice. McCarthy later talked about pitching the device to Apple's battery suppliers. The timeline here is a bit confusing, given that the iPhone launched after the period I think he was discussing, but still, it makes a good example. The first-generation iPhone was about a half inch thick, hefty by today's standards but a very far cry from Steorn's few publicized prototypes, which occupied perhaps two feet square. Miniaturization is a challenge for most technologies, with the result that pocket-portable devices are usually a late-stage application, not an early one. That's especially true for an electromechanical generator, one that involves bearings and electromagnets and a spinning rotor. Still, it was all about phones.

Steorn's trouble getting the attention of battery manufacturers lead them to seek independent validation. Turned away, as McCarthy tells it, by university physics departments and industrial laboratories, Steorn came up with an unconventional plan. In the 2006 Economist advertisement, they announced "The Challenge." Steorn was recruiting scientists and engineers to form a panel of independent jurors who would evaluate the technology. You could apply at steorn.com. Nearly 500 people did.

The steorn.com of the Orbo era is a fascinating artifact. It has a daily poll in a sidebar, just for some interactive fun. You could apply for the scientific jury, or just sign up to express your interest in the technology. Most intriguingly, though, there was a forum.

This is some intense mid-2000s internet. There was a time when it just made sense for every website to have a forum; I guess the collective internet industry hadn't yet learned that user-generated content is, on average, about as good as toxic waste, and that giving the general public the ability to publish their opinions under your brand is rarely a good business decision. Still, Steorn did, and what few threads of the steorn.com discussion forum survive in the Internet Archive are a journey through a bygone era. Users have the obvious arguments: is Orbo real, or a scam? They have the less obvious arguments, over each other's writing styles. There is a thread for testing BBCode. A few overactive users appear seemingly everywhere, always hostile and dismissive. An occasional thread will become suddenly, shockingly misogynistic. This is, of course, the soup we all grew up in.

The forum exemplifies one of the things that I find most strange about Steorn. The company was often obsessively secret, apparently working on the Orbo technology in quiet for years, but it was also incredibly bold, choosing a full-page ad for a publicity stunt as their first public announcement. McCarthy was well aware that the scientific establishment viewed him as a fraud, by this time individual investors seemed to be growing skeptical, and yet they had a discussion forum.

---

The late years of Steorn are best summarized as a period of humiliation. In 2007, the company promised a public demonstration of the Orbo at a science museum in London. I remember following this news in real time: the announcement of the demonstration triggered a new round of press coverage, and then it was canceled before it even began. Steorn claimed that the lighting in the museum had overheated the device and damaged it. The public mostly laughed.

In 2009, Steorn arranged a second public demonstration, the one at the Waterways Visitors Centre in Dublin. This one actually happened, but the device clearly incorporated a battery and Steorn didn't allow any meaningful level of inspection. The demonstration proved nothing but that Steorn could use a battery to make a wheel spin, hardly a world-changing innovation. Not even a way to charge a phone.

Later that same year, The Challenge came to its close. The independent jury of experts, after meetings, demonstrations, experiments, and investigations, returned its verdict. The news was not good. Jurors unanimously approved a statement that Steorn had not shown any evidence of energy production.

When something is as overhyped and underdeveloped as the Orbo, when a company claims the impossible and still attracts millions, you sort of expect a dramatic failure. Surely, you cannot just claim to have created a source of free energy and then wander on through life as usual. Or, perhaps you can. In 2010, steorn.com gained the Developer Community, where you could pay a licensing fee for access to information on the Steorn technology.

Even the mobile phones were not abandoned. In 2015, Steorn mounted another demonstration, of a sort. A small Orbo device, now apparently solid-state, was displayed supposedly charging a phone in a Dublin pub. A lot of big claims are made and tested in pubs, but few with so much financial backing. Steorn had been reduced to a story told over beer and the internet: at the end of 2015, McCarthy announced via Facebook that an Orbo phone and phone charger would soon go on sale. The charger, the Orbo Cube, ran €1,200 and was due to ship within the month.

As far as I can tell, these devices were actually completely unrelated to the original Orbo. The "solid-state Orbo" was completely novel design, to the extent that you would call it a design. Some units did ship, and reverse engineering efforts revealed an eccentric set of components including two alkaline 9v batteries and shrinkwrapped mystery cells that have the appearance of capacitors but are, supposedly, proprietary components relying on some kind of magnetic effect. It's hard to find much solid information about these devices, I think very few were made and they shipped mostly to "friendly" customers. It is said that many of them failed to work at all, the result of an acknowledged quality control problem with the clearly handmade interior. There is a lot of electrical tape involved. Like a lot. And some potting compound, but mostly electrical tape. The two I've found detailed accounts of failed to charge a phone even when new out of the box.

Through this whole period, Steorn was still McCarthy's day job, at least on paper. But he had turned towards a new way to make money: poker. Starting around 2015, McCarthy became a professional poker player, apparently his main form of income to this day. He announced that Steorn would be liquidated in 2017; it was finally dissolved in 2024.

---

History is replete with perpetual motion machines. There are rumors of some sort of machine in the 8th century, although questions about the historicity of these accounts lead to difficult questions around what makes a perpetual motion machine "real." Of the many putative free energy sources of history, many have involved some configuration of magnets. I can understand why: magnets offer just the right combination of the mysterious and the relatable. They seem to work as if by magic, and yet, they are physical objects that we can hold and manipulate. You can see how someone might develop an intuitive understanding of magnetic behavior, extend it to an impossible idea, and remain quite convinced of their discovery.

If perpetual motion machines are a dime a dozen, why does the story of Steorn so fascinate me? I think that it has many parallels to our situation today. The Orbo is not merely the product of a deluded inventor, it's the product of a period of Irish economic history. Whyte's book is focused primarily on this aspect of the story. He articulates how the overstuffed, overhyped Irish economy of the 2000s, a phenomenon known as the Celtic Tiger, created an environment in which just about any claim could raise millions---no matter how outlandish.

And here we are today, with some of the largest components of our economy run by people who fashion themselves as technical experts but have backgrounds mostly in business. They, too, are collecting money on the premise that they may build a world-changing technology---AGI.

Sure, this analogy is imperfect in many ways. But I think it is instructive. Sean McCarthy wrote of Steorn's early days:

It was just a gold rush: if you weren't doing web, you didn't exist. So, a banana import-export business needs to invest millions in e-commerce. They probably did need to invest, but it was driven, in my opinion, by all of these companies driving their share price up by doing something on the web rather than by any real business cases.

The dotcom boom and bust was an embarrassment to the computer industry, and it was also a pivotal moment in its growth. E-commerce, the graveyard of many '90s businesses, is one of the pillars of modern life. The good days of easy money can be an incubator for important ideas. They can also be an incubator for idiocy.

Whenever I read about some historic scam, I always wonder about the people at the top. Was Sean McCarthy a fraud, or was he deluded? Did he play up the Orbo to keep the money coming in, or did he really believe that he just needed one more demonstration?

Whyte's book takes a strong position for the latter, that McCarthy had the best intentions and got in over his head. He still believes in Orbo, to this day. Of course, Whyte is far more sympathetic than I find myself. The book is almost half written by McCarthy himself in the form of its lengthy excerpts from his statements. I am more cynical, I think that McCarthy must have known things were going sideways well before he shipped out broken power banks with nine volt batteries covered in electrical tape. I don't think he started out that way, though.

I think he started out with the best intentions: the leader of a moderately successful tech company, one that wasn't getting much business but had ready access to capital. He went fishing for ideas, for opportunities for growth, and he hooked the biggest of them all. A world-changing idea, one so profound that he himself struggled to understand its implications. An impossible dream of not having to charge his phone. To sustain the company, he had to bring in money. To bring in money, he had to double down. Perpetual motion was always just around the corner.

I believe that McCarthy is a con man, in that he has too much confidence. It's a trait that makes for a good poker player. McCarthy saw something remarkable, perhaps an error in measurement or an error in understanding. He placed a bet, and spent the next decade pushing more money into the pot, trying to make it work out. I would hope that our industry has learned to be more cautious, but then, we're only human. We watched the wheels spin, faster and faster, and we convince ourselves that they can spin forever.

Where we left off, Albuquerque's boosters, together with the Forest Service, had completed construction of the Ellis Ranch Loop and a spur to the Sandia Crest. It was possible, even easy, to drive from Albuquerque east through Tijeras Pass, north to the present-day location of Sandia Park, and through the mountains to Placitas before reaching Bernalillo to return by the highway. The road provided access to the Ellis Ranch summer resort, now operated by the Cooper family and the First Presbyterian Church, and to the crest itself.

The road situation would remain much the same for decades to come, although not due to a lack of investment. One of the road-building trends of the 1920s and 1930s was the general maturation of the United States' formidable highway construction program. The Federal Aid Highway Act of 1921 established the pattern that much of western road building would follow: the federal government would split costs 50:50 to help western states build highways. This funding would bring about many of the US highways that we use today.

A share of the money, called the forest highway fund, was specifically set aside for highways that were in national forests or connected national forests to existing state highways. By 1926, the Federal Lands Transportation Program had taken its first form, a set of partnerships between the Bureau of Public Roads (later the Federal Highway Administration) and federal land management agencies to develop roads for economic and recreational use of federal land. For the Forest Service of the era, a forest without road access was of limited use. The following years saw a systematic survey of the national forests of New Mexico with an eye towards construction.

The Federal Aid Highway Act presaged the later interstate freeway program in several ways. First, the state-federal cost sharing model would become the norm for new highways and drive the politics of road construction to this day. Second, despite the nominal responsibility of the states for highways, the Act established the pattern of the federal government determining a map of "desirable" or "meritorious" road routes that states would be expected to follow. And finally, the Act enshrined the relationship between the military and road building. The first notional map of an integrated US highway system, developed mostly by the Army for the Bureau of Public Roads, was presented to Congress by esteemed General of the Armies John Pershing. This plan, the Pershing Map, is now over 100 years old but still resembles our contemporary freeway system.

The Great Depression did not generally drive construction, but roads would prove an exception. Roosevelt's New Deal, and the dryer language of the Emergency Relief Acts of the early 1930s, provided substantial federal funding for construction and improvement of highways. The impact on New Mexico was considerable. "Surveys, plans, and estimates of Navajo Canyon Section of State Highway No. 2 within the Carson National Forest, south of Canjilon, and survey, plans, and estimates on State Highway No. 12 between Reserve and Apache Creek on the Datil-Reserve Forest Highway route." The lists of highway projects in newspapers become difficult to track. The forest highway program's budget doubled, and doubled again. "The building of new Forest Highways in the National Forests of New Mexico will be rushed as rapidly as possible... to open up the National Forests to greater use and protection from fires."

The papers were optimistic. The Carlsbad Current-Argus ran a list of Forest Highway efforts in 1932; the headline read simply "Forest Service To Solve Unemployment."

---

The depression-era building campaign left the Ellis Ranch loop on the same route, but saw much of the route improved to a far higher standard than before. Graveling, oiling, and even paving stabilized the road surface while CCC crews built out embankments on the road's mountainside segments. By 1940, much of the loop had been incorporated into New Mexico Highway 44---a continuous route from Cedar Crest to Aztec.

Former NM 44 exemplifies changes to the loop's northern segment. Much of NM 44 is now known as NM 550, a busy highway from Bernalillo and alongside the far reaches of Los Lunas that eventually becomes one of the major routes in the state's northwest corner. The connection to Cedar Crest, though, is now difficult to fathom. The former route of NM 44 continued east of the freeway through Placitas and along NM 165, over the mountains to become NM 536 at Balsam Glade and then NM 14 at San Antonito. NM 14 is a major route as well, and NM 536 is at least paved---but as discussed in part I, NM 165 is now a rough unpaved road appealing mostly to OHV users and impassable in winter ¹.

Sections of NM 165 do seem to have been paved, mostly closer to Placitas, but maintenance as a main highway had ended at least by 1988 when the NM 44 designation was retired.

This is the major reason for my fascination with the Ellis Ranch Loop: today, it is hardly a loop---the whole loop is still there, and you can drive it at least in summer, but the segment of the highway in the north part of the Sandias, what is now 165, doesn't feel at all like a continuation of the same loop as NM 536. On the ground, when driving up 536, it's clear that the "main" route at Balsam Glade is to follow the curve west onto Sandia Crest Scenic Highway. And yet, the Scenic Highway was only a spur of the original loop.

This oddity of history is still reflected in the road designations and, as a result, modern maps. NM 536 and NM 165 are state highways. Sandia Crest Scenic Highway, despite the name, is not. As you zoom out, Google Maps hides the Scenic Highway entirely, depicting the former Ellis Ranch Loop as the main route. This is very different from what you will find if you actually drive it.

This halfway state of change, with NM 165 still designated as a highway but effectively abandoned and the more major Scenic Highway not designated, partly reflects the bureaucratic details of the 1930s. NM 165 was considered part of a proper inter-city route; the Scenic Highway deadends at the crest and thus never really "went anywhere." But there is more to the story: it also reflects the ambitions, errors, and shifting priorities of the mid-century.

---

The first formal ski area in the Sandia Mountains opened in 1936, developed by Robert Nordhaus's Albuquerque Ski Club. The ski area, then called La Madera for the canyon that tops out at the ski slopes, became a center of southwestern wintersports. In the following decades, the ski area would displace Ellis Ranch as the main destination in the mountains.

World War II disrupted skiing and recreation more broadly; progress on Sandia Mountain development stalled into the 1940s. The decline wouldn't last: WWII raised the possibility of ground combat in Northern Europe or even a need to defend the inland United States against invasion. Military planners considered Denver the best choice for an emergency relocation of the national capital in response to an Atlantic assault. While never a formal policy, the concept of Denver as "national redoubt" survived into the Cold War and directed military attention to the Rocky Mountains.

From 1939 into the 1940s, the National Ski Patrol lobbied for development of ski-mounted military units to the extent that it became a de facto auxiliary of the US Army. Ultimately only one Army "Mountain Division" would be established, but its members---recruited and trained by the Ski Patrol itself---went on to an instrumental role in securing the surrender of German forces in Austria.

While the Alpine Light Infantry were never a large part of the Army, they brought public attention to skiing and mountain recreation. Military support enabled expansion of the Ski Patrol's training programs, and returning soldiers had stories of skiing in the Alps. Much like aviation, skiing surged after the war, elevated from a fairly obscure extreme sport to one of the country's best known forms of recreation.

In 1946, the La Madera Ski Area installed what was then the longest T-bar lift in the country. It built by Ernst Constam, a Swiss engineer who invented the modern ski lift and became trapped in the United States when the war broke out during his business trip. Constam found himself an American by accident, but developed a fierce allegiance to his new home and soon worked for the Army. He participated in the training of the Mountain Division, contributed his understanding of European alpine combat to military intelligence, and even built the first military ski lift: a T-bar at Camp Hale in Colorado, now called Ski Cooper.

During the 1950s, the ski area underwent ownership changes and some financial difficulty. Nordhaus operated the area profitably, but needed cash to build the infrastructure for continued growth. That cash came in the form of Albuquerque businessman and real estate developer Ben Abruzzo, who is remembered first as a famous balloonist (responsible for much of Albuquerque's prominence in hot air ballooning) but was also an avid skier. Nordhaus and Abruzzo formed the Sandia Peak Ski Company and construction was soon underway.

You could access the base of the ski area from the Ellis Ranch Loop, but it was slow going, especially in winter. The trip from Albuquerque reportedly took hours. Ski area expansion required easier access. After negotiations related to the ski area permit expansion, the Forest Service paved the entirety of Ellis Loop from the base of the mountains to the ski area. This was the first time that a large section of the road in the mountains was fully paved, and it appears to be the Forest Service's decision to pave from the ski area southeast to San Antonito, rather than north to Placitas, that sealed the fate of NM 165. From that point on, the "main" way from Albuquerque to the Mountains was via Tijeras, not via Placitas.

---

The late 1950s were a busy time in the Sandias. The ski area was not the only growing tourism attraction; a post-war resurgence in travel and automobilism renewed the pro-tourism boosterism of the 1920s. Albuquerque business interests were back to lobbying the Forest Service for tourist facilities, and Abruzzo and Nordhaus's efforts at the ski area inspired confidence in a bright future for mountain recreation. All that was needed were better roads.

Sometime during the 1950s, the names "Ellis Ranch Loop" or "Ellis Loop" gave way to "Sandia Loop." Despite the new name, the road kept to its old history: the state and the Forest Service announced their intent to complete a project that, as far as they were concerned, had been in intermittent progress since the 1930s: paving and widening the entire route as a two-lane highway.

The project had never been completed in part because of the difficult conditions, but there were politics at play as well. Federal funding, critical to completing highway projects, was allocated based on the recommendations of committees at each level of government. The development of "secondary highways," a category that included NM 44, fell to county committees for prioritization. Despite the emphasis that Bernalillo County put on the Sandia Loop, Sandoval County was less motivated. Another nail in NM-165's coffin came in 1958 when the Sandoval County Road Committee definitively stated that their budget was quite limited and the Sandia Loop was not a priority. As far as I can tell, serious efforts to pave the existing road between Balsam Glade and Placitas ended at that meeting. A few years later, the state highway department announced that the project had been canceled.

Despite the fate of the loop's northern half, the rest of the Sandia mountain roads enjoyed constant attention. The southern portion of the loop was widened (by four feet, three inches) with new road base laid. The road to the crest itself, despite the involvement of the state highway department in the original construction, was decisively the problem of the Forest Service---at least, this was the point repeatedly made by the state highway department as Albuquerque interests campaigned for its improvement. Still, boosters did not have to wait for long. In 1958 the Forest Service let contracts for improvement of the Sandia Crest road as an "anti-recession project."

While many reports describe the late-1950s Sandia Crest project as widening and resurfacing, some changes to the alignment were made as well. The most severe of its many switchbacks, at the Nine Mile Picnic Area, was reworked for fewer curves. The original alignment is still in use for access to the picnic area.

In the mean time, crews contracted by the Forest Service completed yet another round of improvements to the north section of the loop road. This project spanned from a "preliminary engineering survey" in 1958 to the point where the gravel base was largely complete in 1959, but while paving of the entire route was planned it does not seem to have happened. The Albuquerque business community felt that there was good reason for the paving project: the Albuquerque Tribune reported in 1958 that "upwards of 100,000 cars annually use the Loop Road in making the punishing trip up to La Madera ski run and the Crest." At the same time, though, Albuquerque's once unified push for mountain access started to face political headwinds.

---

The 1960s were a different world, but some things stayed the same: Bob Cooper advertised cabins for lease at Ellis Ranch, and the business community wanted to see an improved, paved loop highway around the Sandias. The original completion of the Ellis Ranch Loop appeared in the Albuquerque Tribune's "30 Years Ago" feature in 1960, alongside coverage of the growing debate over placement of the city's two upcoming interstate freeways.

Despite efforts by the Chamber of Commerce, Highland Business Men's Association, and Downtown Business Men's Association to restart the Sandia Loop project, the state Highway Department remained silent on the subject. This was the peak era of freeway construction, and the department had many construction problems on the I-25 corridor especially. The Sandia Loop remained a popular facility, though, recommended by the Mobil travel guide starting in 1962. That same year, the La Madera ski area, now called Sandia Peak, opened the "Summit House" restaurant at the top. Summit House would later be known as High Finance before closing for replacement by today's Ten 3.

This might be confusing to today's Burqueños, as the main attraction at Sandia Peak would not begin construction until 1964. The "Spectacular Sandia Peak Chairlift Ride" described in 1962 newspaper advertisements is the ski area's Lift #1, a chairlift that is no longer in service and slated for replacement with a mixed chair/gondola system in coming years. And yet, the restaurant and lift were operated by the "Sandia Peak Ski and Aerial Tram Co." This somewhat contradicts the "official" history of the Sandia Peak Tramway, but one would think that Nordhaus and Abruzzo must have already had a bigger vision for access to the mountaintop.

The full story of the Sandia Peak Tramway could occupy its own article, and perhaps it will, but the short version is this: Bob Nordhaus visited Switzerland, where he learned that ski areas were being made more accessible by means of gondola lifts. He thought that the same scheme would work in Albuquerque, and together with Abruzzo, mounted a long campaign for permitting and funding. The plan was ambitious and more than just a bit quixotic, but Nordhaus and Abruzzo were both big personalities with extensive business connections. In 1966, they opened what was then, and for many decades after, the longest aerial tramway in the world. It runs from the northeast fringe of Albuquerque directly up the western slope of the Sandias, delivering passengers to the very top of the ski area. The 3,819 foot climb, over 2.7 miles and just two towers, takes about fifteen minutes. It is far faster, of course, than the drive around and up the east side. And yet, in the fashion of Albuquerque, one of its biggest impacts on our modern city is a road: Tramway Boulevard, or NM 556, roughly one quarter of what was originally planned as a beltway highway around the city.

While the tramway opened the steep western slope to tourism, attention to the east side was reinvigorated. The Albuquerque business community had continued to lobby the Forest Service for new and better roads, and as the tramway went in the road boosters found success. The Forest Service circulated a draft plan that delighted some and terrified others: a brand new loop¸ the Crest Loop, that would replace the road from Balsam Glade to the crest and extend it straight up the very spine of the mountain, allowing views down both sides, until coming down the northern ridge to meet the existing Sandia Loop Road near Placitas. It would be a first-class skyline road, 24' wide pavement over almost exactly the route of today's crest trail.

---

Previous efforts at radical Sandia mountain road construction were held off by weather, funding, and war. For the Crest Loop, there was a new enemy: bighorn sheep.

Rocky Mountain Bighorn Sheep don't appear to have ever been common in New Mexico, but a small population around the Rio Grande Valley was completely wiped out by human impacts in the late 19th and early 20th centuries. Efforts to reintroduce bighorns started in the 1930s, with more translocated sheep in the 1940s and 1960s. These populations struggled severely, and despite the attempts there are no bighorn sheep in the Sandia Mountains today. Still, the population in the mid-'60s was likely the largest it would ever be, a hard-won victory that the state was inclined to protect.

Besides, Rachel Caron's Silent Spring was published in 1962, a landmark in the history of American environmentalism. The environmentalist movement wasn't as powerful as it would become, but it was certainly gaining momentum. Not only the state Game and Fish Department opposed, but community groups like the New Mexico Mountain Club stepped up to oppose development along the ridge.

On Monday, August 8th, 1966, the Forest Service called a public hearing to be held at the East Central Branch of Albuquerque National Bank. To an audience of businessmen and community groups, regional forester William D. Hurst presented the official plan for the Sandias. The existing road from Balsam Glade up to the crest, the Crest Scenic Highway, would be significantly reworked. The old alignment had over a dozen switchbacks, the new alignment only four. The wider, straighter road would offer access to a completely new ski area at about the location of today's Capulin snow play area.

The Crest Loop would not make it quite all the way to the crest, instead turning north a bit to the east (and below) the crest where a parking lot and turnout area formed the main attraction. The highway itself no longer followed the ridge, an admission to the Bighorn Sheep interests, but instead paralleled it about 1/4 mile to the east. About 2.5 miles north of the crest, the highway would switch back and start it a descent down the eastern face, through Las Huertas canyon, to meet the Sandia Loop before Placitas.

It was noted that, in the future, an additional road could be planned up the canyon above Sandia Cave for yet a third road all the way up the eastern side. At least you can't say they weren't ambitious.

The 1966 Crest Loop plan was presented by the Forest Service in a rather matter-of-fact way. Hurst told the press that he did not expect much objection, and apparently the presentation itself was well received. Still, Hurst's quip that the plan presented "the greatest good for the largest numbers of people" hinted that it was not uncontroversial, as did the fact that every newspaper article about the presentation made some reference to the "hubbub."

While the 1/4 mile shift away from the ridge had won the support of the state, it was not enough to please the New Mexico Wildlife and Conservation Association and gained only the most tacit support from the New Mexico Mountain Club, which described their position to the papers as "wait and see." They had time: the Forest Service expected the project to take as much as ten years, even with surveying work starting immediately.

The Crest Loop was only part of a broader plan for Sandia recreation, one that responded to Albuquerque's rapid mid-century population growth. There were more than twice as many people in Albuquerque then as there had been before the war, for example, and the Forest Service considered the nine picnic areas on the existing road to be overused. The new crest loop would enable 200 acres of new picnic grounds

For all of the promise of the new Crest Loop, there was a curious omission: the view from the crest. The road's alignment 1/4 mile off the crest meant that the highest point on the road, the turnout with bathrooms and facilities, was on the other side of the ridge from Albuquerque. It would have views to the north and northeast, but not at all to the west or southwest. In other words, you would no longer be able to drive up to the crest and then look down on the city---getting the same view offered by the old road would require a hike.

This is a fascinating decision and one that, if you will allow me to editorialize, seems to reflect the Forest Service's process-focused approach to planning. The original Crest Loop plan would have placed the road directly on the ridge, probably providing one of the finest roadside views in the nation. Well, in actuality, there was always disagreement over the aesthetic merits of the plan. The forest service had always denied any intent to clearcut trees to improve sightlines, so it was likely that even the original ridgetop alignment would have had trees blocking the view along much of the route. The Forest Service even conceded to environmental groups by promising to keep the road off of the large clear bluff just north of the crest, the spot with the very best potential for automobile sightseeing.

The final proposed alignment, downslope from the ridge, had the view completely blocked on the west... and it was away from the windswept, rocky western face, decidedly in the forest. Even to the east, where the terrain dropped away precipitously, you would be unlikely to see anything other than the forest right in front of you.

Proponents said the Crest Loop would be a natural attraction like no other, with a 360 degree perspective over the Rio Grande Valley and the High Plains. Opponents said it would be a "green tunnel," so densely forested on both sides that it might as well have been down in the valley for all the sightseeing it would afford. They seem to have been right: as ultimately proposed, the "skyline" Crest Loop actually offered less of a view than the existing route.

What it would offer is capacity. The road was designed for traffic, intended to support visitors to the new picnic areas, the expanding Sandia Peak Ski Area, and the new ski area yet to be named. The new ski area penciled out as 1,000 acres, larger than Sandia Peak. A new snow play area was expected to attract thousands of non-skiing visitors. And parking---parking was a problem, the Forest Service reported, with just 750 spaces at the bottom of the ski area. The Forest Service intended to provide 800 spaces at the new snow play area, and the bypassed switchbacks of the old crest road would themselves be repurposed as new parking areas.

---

As far as the Forest Service was concerned, the Crest Loop was a done deal. They had already committed $600,000 a year for the next several years to fund surveys and planning, and expected more federal funding to match state support. And yet, as always, progress was slow. 1967 passed with little mention of the road. In 1968 the Albuquerque Chamber of Commerce must have been getting nervous, as they passed a resolution in support of its completion, for no reason in particular.

In 1969, an interesting new proposal for the Crest Scenic Highway put it back into the papers: a race. Albuquerque advertising firm Eicher and Associates proposed to one-up the Pikes Peak by hosting a Labor Day hill climb. The state Highway Commission approved the plan, but the race stalled when a State Police captain pointed out that state law forbid racing on public roads---and the Attorney General issued an opinion that the Highway Commission had no authority to grant an exception. Nothing ultimately came of the Sandia Peak Climb, or at least, nothing other than a series of headlines and another round of objections to planned improvements.

Central to the opponent's arguments was a disagreement over the nature of forest recreation. The New Mexico Wildlife and Conservation Association advocated for the establishment of wilderness areas covering much of the Sandias, precluding further development. Their proposal actually accommodated the Crest Loop by putting it just off of the edge of the wilderness area covering the western face, but Cibola National Forest Supervisor George Proctor was still hesitant: "Proctor made clear he did not rule out wilderness use for the land, but only wanted time to weigh the proposals against possible other needs---as for future additional tramways or recreation areas." Perhaps one tramway and two highways was not enough; we can't know what the future will hold.

The conservationists said exactly what you would expect:

It would be nice to have complete access for large volumes of people---since so many have seen nothing like the top of the Sandias---said [NMCWA director] McDonald, but when large volumes come, then the beauty of the area often ceases to exist. ²

While the Forest Service had been quiet, it had not been entirely idle. Surveying for the route was nearly complete, and the Forest Service contracted the first phase of the work: clearing trees for the new route, starting in the immediate area of the crest. A bit over three miles of right of way were clearcut, mostly along the "top" of the road a quarter mile from the crest. The new upper alignment of the existing road was cleared as well, wider switchbacks well to the north and south of the current road.

Almost immediately after the clearing, Forest Service funding for the project unexpectedly dried up. Work was discontinued until new funding for the project could be secured, a delay that was not expected to last long but that fell at the worst possible time.

The Crest Loop had been abstract, seen only in the form of Forest Service presentations and hand-drawn newspaper illustrations. In that form, it had attracted considerable support from the business community and only muted objections from conservationists. But then the trees fell. Acres of clearcut forest, so close to the ridge and so visible from the road, turned the highway plan into an eyesore. As they sat, unchanged, month after month, an eyesore turned into a scandal.

---

By the summer of 1970, the Wildlife and Conservation Association, the New Mexico Conservation Coordination, and the National Wildlife Federation sent letters opposing the project. State legislators became involved. There were accusations of treachery, with Forest Service employees allegedly lobbying business leaders to support the project while on the clock, and the district forester holding secret meetings with state officials to coordinate.

Over the following year, opposition stacked up. US Rep. Manuel Lujan, a prominent New Mexico politician, wrote to the director of the Forest Service with his objections. NM Attorney General David Norvell joined with a press release asking the Forest Service to reconsider the plan. These objections were actually rather weak: Lujan, in particular, mostly just thought the proposed road was too expensive, a waste of money considering that there was already a highway connecting the same places. Even environmental groups tempered their criticism, with at least two clarifying that they objected only to the highway along the ridge, and not to improvements to the road between Balsam Glade and the crest.

The Forest Service's motivations can be difficult to understand. The whole Crest Loop plan had reportedly begun with a presidential directive to identify opportunities for new recreational facilities, and it had certainly had the strong support of the business community in the 1960s. At the turn of the 1970s, the situation had noticeably changed. Most press coverage of the project focused on the opposition, and Albuquerque's business clubs had fallen silent, perhaps averse to the controversy. There's not much to say for the Forest Service's lobbying effort beyond inertia: as far as they were concerned, they had committed to the plan in 1966 and were now simply executing on an old decision.

The federal government, once it gets into motion, does not easily change directions. For Forest Service staff, careers, or at least plans to climb the corporate ladder, must have been staked on the success of the Crest Loop. Even as the budget escalated from $3 million to $3.5, then as high as $5 million, the Forest Service remained committed to finishing what it had started. "[Opponents] claim the Forest Service won't heed their cries of distress," the Albuquerque Journal summarized. "The Forest Service says the decision to build the road was made a long time ago."

In May of 1971, the controversy broke out into demonstrations. This was no Vietnam war; anti-Crest Loop protests had a more whimsical quality. On Sunday the 16th, around two hundred demonstrators went up the Crest Scenic Highway and gathered in the cleared path of the future road. The plan, originally, had been to replant the clearing with new trees, a slow way of walking back the road's likewise slow advance. This threat of minor vandalism was met by a threat of minor reprisal when the Forest Service informed the press that unauthorized planting of trees in a National Forest could be charged as a misdemeanor.

The result, a mountaintop clash between Albuquerque's most motivated environmentalists and the force majeure of the Forest Service, must be one of the stranger moments in Albuquerque's environmental history.

One group---lead by Dr. Gerald Bordin---advocated "peaceful protest." Another, lead partly by Jesse Cole, planted three symbolic trees. Finally, two individuals---Florencio Baca and Marvin Price---began a private discussion that turned into a simulated panel discussion on the negative aspects of the proposed road....

Few persons started the planned three-mile walk down the graded Ellis Loop in the 10,000-foot altitude. Several protesters, disillusioned for the lack of trees to plant, began to leave by noon.

Bordin produced a cardboard replica of a pine tree, dissected by a highway, and placed it on the proposed road. "This type of planting is legal," he said; and another protester, watching two others dig a hole for a tree a few yards away, asked Bordin "Where's your tree, man?" ³

---

On the first day of 1970, president Richard Nixon signed the National Environmental Policy Act. Nixon was famously a mixed bag on environmental issues; his creation of the Environmental Protection Agency is one of the largest single steps forward in the history of US conservation, but he also pinched pennies to a degree that smothered many environmental programs. As it would turn out, the fate of the Crest Loop hinged on the interplay of these two sides of Nixon.

To satisfy its obligations under NEPA, the Forest Service completed an environmental impact assessment of the proposed Crest Road in 1973. It was now almost five years since the clearing of part of the right of way, but the only progress made had been on paper. Unsurprisingly, the impact assessment found the road to be feasible. "The highway is completely on the east slope and won't even be able to be seen from Albuquerque," the forest supervisor explained.

The environmental analysis spurred another series of exchanges between the Forest Service and environmental interests. The state Environmental Planning Commission filed comment, suggesting that the Crest Loop would better be replaced by a system of two one-way roads at different altitudes. One thing the EPC did agree on was the necessity that something be done. Like the groups who had assured their support for improving the road to the crest in the years before, the EPC recommended against taking no action. The existing crest road was unsustainable, they found, and needed to be improved in some way.

During the plant-in, one organizer suggested letters to the editor. Perhaps he was heard, as letters in opposition stacked up during 1973. Frank Marquart wrote that the highway would be good for General Motors and no one else. A letter from L. Graham got the simple headline "Crest Road Not Necessary." Among the growing public protest, the gears of government were turning: the Bernalillo County Commission made a resolution asking the Forest Service to stop.

Joe Whiton attended a hearing on the matter in Tijeras, leaving a Journal reporter with a memorable quote:

The Crest Road is a juggernaut already aimed, the switch is on and it is going to be fired no matter what we say. I have a feeling it is going to happen anyway and I don't think it should. ⁴

---

Despite the Forest Service's momentum, two significant roadblocks had arisen in the Crest Loop plan: first, the Forest Service was having a hard time getting to a final environmental impact statement on the project. Several drafts had been circulated, and from what I can tell one was even published as final, although it only covered a portion of the work rather than the entirety. The National Environmental Policy Act has many problems and often fails to achieve its aims, but it does have its virtues: while NEPA does not necessarily require agencies to listen to public feedback, it does require them to solicit it, and the hearings and comment periods that make up federal environmental policy gave the Crest Loop's opponents a platform on which to organize.

Second, there was President Nixon. Nixon had generally supported the Federal Lands Highway Program, under which the Crest Loop now fell, but his efforts to reduce federal spending through impoundment quickly became controversial. A fight with Congress over changes to the federal highway funding model in 1972 lead to a lapse in highway funding, a situation that became even more chaotic as Nixon specifically refused funding for a long list of highway grants. The funding situation probably didn't kill the Crest Loop, but it delayed further progress during the 1970s, contributing to both the sense of scandal over the lack of progress and to the opposition's developing power.

In 1973, the Forest Service produced new draft environmental documents covering four variant plans, which other than the no action alternative all involved some form of new highway. This was, as it turns out, the last gasp of the Crest Loop: in 1975, over a decade after planning first started, the Forest Service released a new environmental impact statement covering the entire Sandia mountain recreational plan. Its alternatives included new picnic grounds, bathrooms, and campgrounds. It included various improvements to the Crest Scenic Highway. But more important is what it did not include: any mention of the Crest Loop.

In the following years, the Forest Service would release further environmental and planning documents on the remaining scope of work, which was only the improvement of the road between Balsam Glade and the Crest. A 1976 impact statement covered three alternatives: the first would use the area already cleared for realignment, the second would keep the old alignment, and the third would finish realignment of the entire route, as included in the original Crest Loop plan. Based on the Albuquerque Tribune's count of public comments, Burqueños overwhelmingly favored the second option---keeping the existing route. While the road has been widened and resurfaced over the years, it remains on its original, winding route.

---

In 1927, roads reached Sandia Crest. Today, just shy of one hundred years later, the drive to the crest is essentially the same as this original route. It's all paved now, it's wider, and the curves has softened. It is still, by our modern standards, a rough road. The biggest change from the long-ago Ellis Ranch Loop to today's forest highways is actually a loss: the reduction of NM 165 to a minor, unpaved road. One might say, then, that progress since 1927 has been backwards. We are less ambitious than we once were.

On the flip side, the ambitions of the mid-century are so difficult to square with our modern sense of environmental preservation. The Forest Service dreamed, at times, of thousands of parking spaces, of cabins, a resort. We now value conservation over improvement, wilderness over access. It almost seems like it has always been that way---but it hasn't. This is a modern idea in land management, an innovation of the post-war era and the American environmental movement.

The old tensions between tourist promotion and wilderness preservation have never gone away. In 2023, Mountain Capital Partners signed a joint venture with the Sandia Peak Ski Company. The Sandia Peak Ski Area has barely opened for the last five years; climate change has shortened the season so severely that the Sandias can go all winter without enough snow to ski. Mountain Capital Partners how hopes, through artificial snowmaking, to bring the ski area back to life. Still, it's clear that snow will never be enough: ski areas are, in general, facing tough times.

Sandia Peak Ski Company has developed an ambitious plan for a "four season" recreational destination. It called for a mountain roller coaster, mountainbike trails, a complete replacement of Lift #1. The mountain coaster has already been abandoned, having attracted more environmental controversy than the Forest Service was prepared to handle.

In aerial images, you can still clearly see the path of the Crest Loop, at least to the first switchback where 1968 clearing work ended. Some of the modern trails follow the unused highway rights of way. They are incongruously described by in some Forest Service documents as firebreaks, a function that they do serve but were never intended for. I get the impression that even some of the Forest Service staff have forgotten the entire Crest Loop story. Well, it's one of many things about the Sandias that have been forgotten.

Ellis Ranch is now hardly remembered, up a canyon from a picnic area. It is once again difficult to access, past the part of the former Ellis Loop that is still paved. The Crest Loop was canceled in its early stages, the notional second highway was never even planned.

But Ellis Ranch Loop Road is still there. Millions of people drive up it every year. They look down on the city, and marvel at the view. I wonder how many realize---that the climb started so long ago, and that it has still never been finished. I think that it's better this way. A summit should always be an achievement, so easy to see, but so hard to reach. We get used to living in the shadows of mountains.

The Rotary Club will take immediate action on the Ellis ranch loop project. The Rotarians reached this decision at their weekly luncheon, held yesterday at the Albarado hotel.

The club's plan is not merely to give Albuquerque a good, short road to the Ellis ranch... They embrace the building of a seventy-mile scenic loop. ¹

Many Western cities are defined, in part, by their mountains. Those moving from town to town often comment on the disorientation, the disruption, caused by a change in the city's relation to the peaks. If you have ever lived in a place with the mountains on the west, and then a place with the mountains on the east, you will know what I mean. We get used to living in the shadows of mountains.

One of the appeals of mountains, perhaps the source of their mysticism, is inaccessibility. Despite their close proximity to Albuquerque, steep slopes and difficult terrain kept the Sandias a world apart from the city, even to this day. Yet we have always been driven to climb, to ascend to the summit.

Humans climb mountains not only as a matter of individual achievement, but also as a matter of infrastructure. Whether the inaccessibility of mountain peaks is a good thing or a bad thing depends on the observer; and even the most challenging mountain ascents are sometimes developed to the scale of an industrial tourism operation.

And somewhere, in between, are the Sandias. Not technically part of the Rocky Mountains but roughly aligned with them, the Sandias lack a clear peak. Instead, the range is an elongated ridge, making up the entire eastern boundary of the city of Albuquerque and extending some distance further north into the Sandia Pueblo. The highest point is at 10,679', relatively modest for the mountain states---but still one of the most prominent in New Mexico, moreso even than the higher-elevation Mt. Taylor and Wheeler Peak.

Today, the Sandias are a major site for recreation. Tourists reach the upper parts of the mountain by one of two means: Sandia Crest Scenic Highway, which climbs the gentler eastern slope of the mountain to the crest itself; or the Sandia Peak Aerial Tramway, which makes a daring ascent from the edge of the city up the western side. Either provides access to an extensive network of trails, as do numerous points in the foothills and along the scenic highway.

Like bagging any peak, these access routes were hard-won. Their present contours---engineering achievements, twists and turns, scars on the landscape---reflect over one hundred years of ambition and conflict. Some of Albuquerque's most prominent figures left their marks on the mountain, as did everyday Burqueños, President Richard M. Nixon, and the slow grind of state and federal bureaucracy. The lay of the land today tells a story about the changing relationship of the city with the mountains, and more broadly of the American public with the wilderness.

It also explains why so many older documents and maps refer to the highway to the crest as a "loop," when it does no such thing. Here's the clickbait headline: Where does the Sandia Loop Highway go? The answer will surprise you.

---

Exploration of the Sandias by Spanish expeditions were motivated in large part by rumors of great mineral wealth. Certainly there are minerals in the mountains, and the Pueblo people had told the Spanish expeditions coming up the valley of gold and other precious metals, both in the mountains and in the plains beyond them. With the full perspective of history, it seems that these reports were less grounded in fact and more in a well-founded desire to get the Spanish to move on elsewhere. Supposedly several mines were established, but they left little mark on the landscape and their locations are now forgotten.

The Sandias, while rich in scenery, were not rich in gold. For centuries, they remained mostly undeveloped, besides Pueblo settlements in areas such as Tijeras canyon which were later abandoned in large part due to the increasing encroachment of settlers. Just a small number of people, mostly hunters, called the mountains their home. A few mining operations made their way into the mountains during the 19th century, perhaps repeating the mistakes of the Spanish several hundred years before.

It was these mining camps that brought the Ellis family to Albuquerque. They had run a ranch in the eastern plains of New Mexico, but were driven into the city by lack of water. Patriarch George Ellis found work for a fruit distributor, delivering produce to the mining camps in the mountains. Perhaps missing their days out in the country, George took his time as he traveled the canyons and ridges between camps. His wanderings were rewarded when he found Las Huertas Canyon, a narrow slot with a creek, protection from the wind, and what George Ellis himself called "extraordinary beauty."

The Ellis family moved into the canyon as early as 1893, but in 1905 George Ellis filed a land patent for what was, as an outcome of the Treaty of Guadalupe Hidalgo, frontier up for the taking. Ellis Ranch, as it was called, was indeed extraordinary: one of the most pleasant places in the mountains. But, then, the Ellis family were rather extraordinary as well. Life on the mountainside at 7,600 feet wasn't easy, but they kept cattle, chickens, and a modest farm.

Charlotte Ellis, George's daughter, had not yet a year of education at UNM when they moved out of town and up the slopes on the east side of the ridge. Still, she built a notable career as an amateur botanist. It is because of her efforts that Ellis ranch is often described in terms of its plant life---Charlotte collected specimens of hundreds of plant species, some of them new discoveries, and the ranch is said to have contained examples of every flowering plant known to exist in the mountains.

Plants were not Charlotte's only interest, though. She was apparently a skier, in a time well before skiing was known as a recreational pastime. A photo taken in 1896 shows her in a snow-covered landscape, facing away from the camera. She's looking down the slope, wearing a full dress and a pair of wooden skis. Her pose suggests a certain resolve, the kind you would need to spend the winter high above the nearest towns. It is often captioned as the first photograph of a person skiing in New Mexico. It may be the first photo of a woman skiing at all ².

It also foreshadows the tremendous impact that skiing would have on the mountains and on Albuquerque. We don't know exactly what she was thinking, but what we do know about her suggests that she viewed the mountains much as we do today: for their beauty, their diversity, and for their thrill.

George Ellis had perhaps filed his land patent just in time. The Forest Service had existed for some years but was reorganized under its current name, and as part of the Department of the Interior, in 1905. The forests of the Sandia Mountains have a complicated legal history owing to the interaction of Spanish and Mexican land grants with the Treaty of Guadalupe Hidalgo at the conclusion of the Mexican-American war. You could say that the matter of the forest's legal ownership was not entirely settled until 2004.

None of that stopped President Roosevelt declaring the Sandias part of the Manzano Forest Reserves in 1906. Shortly after, the Forest Reserves became a National Forest and a ranger station was under construction near Tijeras. Paul Ellis, the son of the family, was soon hired on as one of the national forest's first staff; the Ellis Ranch one of the few private parcels within the federal boundaries.

---

The Forest Service has always served multiple purposes: "Land of Many Uses," as Forest Service signs still proclaim. Principle concerns of the forest service included lumber, cattle and sheep grazing, mining, and hunting. These uses were already widespread at the dawn of the modern Forest Service, so in practice, much of the role of early Forest Rangers was to bring forest users under control. Extensive regulations and rules were backed by a small set of rangers of the "Stetson era," rugged types not unlike the Ellises.

What we don't always picture as a part of that era was recreation. The "crunchy granola" brand of wilderness recreation most widely promoted today is indeed a more modern phenomenon, but forests have been viewed as recreational assets since the dawn of forest management---the form of that recreation has just changed, largely in response to new concepts of environmental stewardship that emphasize preservation over "improvement."

In 1913, George Ellis passed away. After a small ceremony, he was buried in a cemetery established next to his ranch. With George gone, the rest of the Ellis family began to scatter, moving into towns, closer to work. This left the question of the ranch.

By the time of George's death, Ellis Ranch had already become renowned for hospitality. The newspapers of the day recounted some of Albuquerque's finest taking vacations at the Ellis Ranch, a trip that was something of an adventure in the 1900s. It took a few hours, in fine conditions, to make the way up a rough trail to the Ellis homestead. One local businessman, known to newspaper readers as "the bee man," took some pride in having gotten a Ford automobile to within a mile of the site. Even so, the journey was well-agreed to be worthwhile. Las Huertas canyon was beautiful, and the Ellises known to offer fresh produce, spring water, and mountain air---a commodity in increasingly high demand as tuberculosis swept the country.

The reputation of the Ellis ranch for taking visitors explains the Albuquerque Tribune's matter-of-fact statement that the Ellis ranch would likely become a tourist resort, despite the lack of known buyers. At that time, the Forest Service was already in an infrastructure-building mood. La Madera, then one of the larger settlements in the east mountains, had telephone service and the Forest Service was extending the line to the ranch.

Shortly after, the buyers emerged, Raymond Stamm (uncle of the Stamm of Bradbury Stamm Construction) and Jack Sheehan of Albuquerque.

These young men propose to fit up the ranch as a mountain resort, the house being splendidly adapted to a mountain club house, with fine fishing, hunting, and camping facilities all about. ³

One challenge for this ambitious pair was, of course, the road: for the ranch to be the resort they imagined it needed an easier means of access. It was estimated that this effort would only require a few hundred dollars, reworking about three miles of trail between the ranch and a better road to Tijeras.

Even by 1913, an improved road to Ellis Ranch was a recurring theme in the newspapers. George Ellis, before his death, had estimated $500 to improve the road between the ranch and Placitas to passable condition. After his death, the estimate on this longer segment reached the thousands.

Some of the problem was, of course, hubris: the Albuquerque boosters of the day could look at three miles of faint road climbing up and down cabins and call it a few days of work. When these projects went underway, they always took longer and cost more than expected. To be fair, though, they were also fighting a living foe. Thunderstorm downpours and winter snow repeatedly washed out the road in the steeper sections. Reading of road improvement efforts, you get repeated deja vu. They are, in fact, announcing plans to fix the same section of road over and over again---it having always washed out the previous year.

The Stamm-Sheehan resort seems to have been popular, even as the road stayed rough. In September of 1914, the faculty of UNM held what we might now call an "offsite" there. Indeed, during 1914 and 1915 it seemed that practically everybody who was anybody was vacationing at Ellis Ranch. All the while, though, they were still struggling to get there... a situation that the Rotarians resolved to correct.

---

The road to Ellis Ranch has been known by many names and promoted by many people. The Rotary Club was a powerful force in the Albuquerque of 1916, though, and their proposal seems to have set the pattern for those that followed. The Ellis Ranch Loop Road, as they called it, would span 70 miles from Albuquerque to Albuquerque. The route went through the Tijeras pass, turned north to San Antonito ⁴ and passed Ellis Ranch on the way to La Madera and then Placitas. From there, it went nearly to Bernalillo, meeting the Santa Fe-Albuquerque road that would soon become part of Route 66---until 1937, at least.

A big argument for the loop, besides the fact that it makes the Ellis summer resort more easily accessible, is its scenery.

At the time of the Rotary Club proposal, the region's road map was in some ways familiar, and in some ways rather different. Many of the today's routes were already known but their condition was widely variable. The trip from Albuquerque to Tijeras, which would become the ultimate Route 66 alignment, was already in place. It was shorter in 1913, though, not extending much further east, and far from a highway. There was a rough road north to San Antonito, and there had once been a road further north from there to La Madera, but by 1916 it had been closed where it entered a private land grant.

This is probably the biggest difference from our modern road network: the route through Tijeras Canyon was not used for much other than access to Tijeras, and for much of the era covered in this article the preferred way to negotiate the Sandias and Manzanos was to go around them entirely. This is a good part of the reason for Route 66's original north-south alignment through the Rio Grande Valley, one that added significant mileage but avoided the mountain pass that was slow going in good weather and could be outright impassable in winter.

Similarly, from the north, there was an existing road from Bernalillo through Placitas and into the East Mountains. It was, in fact, the main route used by many of the residents there. It was not an easy route, though, long and prone to washouts.

And while these roads came close to forming a loop, they didn't. A section of about four miles was missing, right in the middle. Ellis Ranch was, of course, somewhere in that in-between.

So, the Rotary Club project was mostly the improvement of existing roads, along with the construction of new road in a couple of areas required to complete the loop around the Sandia range. The Rotarians estimated the cost at $5,000, and the Forest Service offered to pitch in.

Now, we must consider that $5,000 was a significant amount of money in 1916, something around $150,000 today. This inflation calculation probably underestimates the ask, because extensive investment in highway construction was not yet the pillar of American culture that it is today. Still, public enthusiasm for the project was high, and by the end of 1916 Bernalillo County had enacted a tax assessment to to support the road. The state legislature joined on, and with the Forest Service's share the project was fully funded. Construction seems to have began in earnest in 1917, on improvements to the road through Tijeras Pass.

Efforts on the loop road coincided, of course, with work to develop Ellis Ranch as a summer resort. Starting in 1917, regular newspaper ads appear for vacation cabin rentals at Ellis Ranch, with instructions to telephone Hugh Cooper. This High Cooper was Hugh Cooper Jr., son of the Hugh Cooper who was pastor of the First Presbyterian Church. Much like George Ellis, Rev. Cooper was a well-known figure around Albuquerque. Given that he officiated the wedding of at least one of Ellis's children, he was presumably friendly with the Ellises as well. The newspapers are not exactly clear on how it came to happen, but Rev. Cooper Sr would soon become much more involved in the ranch: in 1923, he and his sister went in together to buy it.

"We hope to keep this beauty spot of nature, so near our city, from falling into the hands of sports, who will advertise dance pavilions and entertainments of a sordid nature. If our plans mature we hope to make it a place where parents can go with their children and find recreation of an exalted character." ⁵

Rev. Cooper's motivations were, he assured the newspaper, godly rather than pecuniary. The phrase "exalted recreation" was singled out for the headline and is certainly one to remember as you explore the Sandias. Cooper was not, though, immune to the sort of boosterism that so animated the Rotary Club.

"We hope, with the help of the good people of Albuquerque, to make the famous Ellis Ranch a second Estes park."

---

The 1910s and '20s were boom years in Albuquerque, with a confluence of the railroad shops, the sawmill, and tuberculosis bringing new residents at a fast clip. The population more than doubled from 1900 to 1920, and with the newcomers came urban development. Subdivisions went under construction west of New Town, a skyscraper rose, and car ownership proliferated. The same period, for many of the same reasons, saw the genesis of American tourism. Trains brought visitors from out of town, and cars brought visitors and residents alike out into the surroundings. Boosters, from the city council to the chamber of commerce to a half dozen clubs, all had plans to make the best of it.

There was, to be sure, a certain aspect of keeping up with the joneses. Colorado was building a strong reputation as a tourist destination, and Denver as a beautiful and prosperous city. Arizona and Nevada had not yet gained populous cities, and west Texas was no less sparse than it is now. For the businessmen of Albuquerque, Colorado was the competition... and Colorado was building summer resorts.

The concept of the "summer resort" is mostly forgotten today, but in the 1920s there was little air conditioning and cities were gaining a reputation as crowded and polluted. The spread of Tuberculosis had brought particular attention to "clean, fresh air," and nothing was quite as clean and fresh as a summer day in the mountains. Estes Park, a town in the northern Colorado rockies, had multiple fine hotels by 1910 and attracted the tourism to match.

Further, in 1910, Denver's own boosters (groups like the Real Estate Exchange and the Denver Motor Club) began to acquire land for the Denver Mountain Parks. The beautiful Genesee Park was under construction in 1920, with a lodge and campground. Daniel Park had opened as a scenic drive (or "Auto View"). Plans for Mount Blue Sky Scenic Byway, the highest paved road in North America, had been finalized and announced to great fanfare.

To the modern conservationist it seems odd that these plans were so focused on roads, but the automobile was new and exciting and in the West had almost singlehandedly popularized the idea of "wilderness recreation." The mountains were no longer a destination only for the hardiest adventurers, they were also for city dwellers, out for a drive and a picnic. The ideal of natural preservation was a long, winding road with regular scenic pullouts. If the road could access a summer resort, all the better. Denver was doing it, and so Albuquerque would too.

Behind the scenes, the wheels of government had been turning. The Forest Service approved construction through the mountains in 1919, which would "make accessible some of the most beautiful mountain scenery in the state, in addition to bringing within easy reach sites for summer homes and temporary camping places." At this point in its history, the Forest Service routinely leased space for the construction of private homes, and a small number were established in the national forest as the road progressed inwards.

Bernalillo County had raised $12,500, the Forest Service committed $25,000, and the state was lobbied for another $12,500. A Forest Service engineer, speaking to the County Commission, noted that the road would provide access to 30 million board feet of lumber. The Forest Service said that construction could be finished by the end of the summer, and in September, the section from Placitas to Ellis Ranch opened to drivers. The road from Ellis Ranch to San Antonito had been surveyed, but work had not yet started.

---

We should consider again how this historic project relates to the modern roads, because there is a surprise. Today, the primary access from Albuquerque to the eastern slopes of the Sandias is via I-40 to Tijeras, and then NM-14 north through Cedar Crest and to then to San Antonito. There, NM-536 branches to the west, becoming Sandia Crest Scenic Highway.

NM-536 turns into Sandia Crest Scenic Highway, which is actually not a numbered highway at all, at a 90-degree turn by the Balsam Glade picnic area. From the picnic area, a much smaller, rougher dirt road descends to the north: NM-165. Today, NM-165 is a rugged and adventurous route to Placitas, impassable in winter and unfriendly to passenger vehicles.

And yet, in the summer of 1919, it was this route that was made a "first class road" to Ellis Ranch. The former site of Ellis Ranch now has a surprisingly well-paved parking lot off of the dirt NM-165, signed as "Las Huertas Canyon," and a trail continues up the creek through much of what was the summer resort. It took about an hour and a half, the Journal reported, to drive from Albuquerque to Ellis Ranch by this route around the north side. The travel time probably isn't all that much faster today, given the slow going on the dirt road.

In 1920, Bernalillo County ran into some trouble on the project funding. The county's plan to fund much of the project through bonds on the tax assessment was, it turns out, in violation of state law. For accounting reasons, the total sum of the bonds would have to be much lower. The county was effectively out of the project, at least as far as money was concerned.

The Forest Service worked according to its own plans. Construction started on a forest road, what is now NM-536, that would later be home to Tinkertown. In summer of 1922, it reached Sulphur Spring, the first of many recreation sites on its long trip up. Progress was brisk: the newspaper reported that 2.5 miles had been completed, then a few more, then eight. The forest road seems to have gone by different names at different times, and perhaps just depending on who was talking, but as 1922 came to a close the Forest Service named it for the canyon that it followed from San Antonito: Tejano Road.

Tejano Road ended about where the bottom of the ski area is found today, some four miles short of Ellis Ranch. Much of the Ellis Ranch Loop was complete, but it wasn't a loop. Plans had been completed for the final stretch, but it was unclear who would pay for the work---and when. The Journal printed an appeals to finish the project, several times mentioning the importance of a "sky line road."

The sky line road was something of a fad in those days. California and Oregon both had them, and plans were underway in many other parts of the west. A skyline road was what we might now call a scenic byway, but with a particular emphasis on views. Many followed a ridge, right down the center, so that drivers could get a view down both sides. Extensive earth-moving and forestry were sometimes involved, modifying the environment to create these views.

The West had a lead on the idea: it wasn't until 1925 that planning started on the most famous, Skyline Drive of Shenandoah National Park. The 100-mile highway was viewed as a wilderness adventure, but was about as far from our modern conception of wilderness as possible. Extensive clearing, of both trees and residents, were required to manicure its vistas. It was seen as one of the finest examples of American recreation, a model to be followed.

---

In July of 1923, the Ellis Ranch Loop was considered complete except for the four-mile segment between Tejano Road and Ellis Ranch Road and, oddly enough, the reworking of the road from Placitas to Ellis Ranch---the very one that had been completed in 1922. Road construction in the mountains was a fight against both funding and nature. The road seems to have been minimally improved, to keep the fast schedule and tight budget. During winter, snowfall and then snowmelt would wash parts of the road out. Throughout the 1920s and 1930s, the road was constantly being repaired and reworked. This probably explains why NM-165, once the principle route up the Sandias, is now so minimally maintained: the state and the Forest Service gave up.

During 1924, the Forest Service closed two miles of the gap and had the other two underway. In a turn of luck for the Albuquerque boosters, the Forest Service also offered to pick up work on the road to Placitas. The entire Ellis Ranch Loop project had become a Forest Service initiative, and they did what they set out to do.

In September 24th of 1924, an Albuquerque man named Arthur Ford set out through Tijeras Canyon with a friend, Mrs. Karskadon. They left Albuquerque at 10 am, finding that the new section connecting Ellis Ranch was not, officially, open; Ford was little discouraged and simply moved the barriers aside. At 11:45 they reached Ellis Ranch. After lunch, they continued northwards, to Placitas, through Bernalillo, and back into town. Ford's odometer measured the trip at 68 miles. The Ellis Ranch Loop was complete.

Imagine a circle approximately 20 miles in diameter.

Then imagine that this circle encloses some of the most beautiful mountain, valley, and mesa scenery in the world.

The city has always existed in tension with the mountains. The Ellis Family, who most famously opened the mountain to visitors, also oversaw a sort of closure to nature. George Ellis was the last person to see a Grizzly Bear in the Sandias. He came across the bear in 1905; he shot it.

From 1916 to 1924, Albuquerque business leaders admired the mountains for their beauty but lamented their inaccessibility. The Sandias could attract visitors from around the nation, they argued, if only you could get to them. Charlotte Ellis would hoof it on foot and on skis, but, then, she was Charlotte.

Then imagine that this circle is bounded by a highway that is traversible every day in the year.

Imagine, within the 70-mile circumference of this circle, near its eastern rim, a cluster of summer houses, supplied with water, light, and other necessaries. Imagine, at various spots within the circle, fine picnic and camping grounds, where people from the hot city may go for a day's outing.

We have always been driven to climb. As soon as it was possible to drive the full loop, to reach Ellis Ranch on an easy road, to take in the "Million Dollar Playground" of the Sandias, the Kiwanis Club formed a committee of business leaders and Forest Service representatives to consider the future of Sandia development.

It was "not only practicable, but highly necessary," they said, to take the next logical step. Ellis Ranch, at 7,500 feet, was only halfway from the city to the crest. A road all the way to the top---from Ellis Ranch to the crest---would complete their vision of the Sandias. "An effort will be made to have work begun as soon as possible."

You may think you are dreaming. And perhaps you are. But some day you will wake up and find the dream come true. ⁶

The Forest Service cleared six miles of steep, tight switchbacks, from the Balsam Glade area just above Ellis Ranch to the crest itself, over 10,500 feet. The New Mexico Highway Department, bolstered by the completion of Route 66, laid a gravel roadbed. Automobiles had become more common, driving a more popular pastime. It didn't take the adventurous Arthur Ford and Mrs. Karskadon to inaugurate the crest spur. On October 10th, 1927, Highway Department officials at the crest counted 110 cars.

Albuquerque had summited the mountain---but the greatest controversy was still to come.

Few aspects of commercial telecommunications have quite the allure of the T-carrier. Well, to me, at least, but then I have very specific interests.

T-carrier has this odd, enduring influence on discussion of internet connections. I remember that for years, some video game installers (perhaps those using Gamespy?) used to ask what kind of internet service you had, with T1 as the "highest" option. The Steam Hardware Survey included T1 among the options for a long time. This was odd, in a lot of ways. It set T1 as sort of the "gold standard" in the minds of gamers, but residential internet service over T1 would have been very rare. Besides, even by the standards of the 2000s T1 service was actually pretty slow.

Still, T1 involved a healthy life as an important "common denominator" in internet connectivity. As a regulated telephone service, it was expensive, but available pretty much anywhere. It also provided a very high standard for reliability and latency, beyond many of the last-mile media we use today.

Telephone Carriers

We think of telephone calls as being carried over a pair of wires. In the early days of the telephone system, it wasn't all that far off to imagine a phone call as a single long circuit of two wires that extended all the way from your telephone to the phone you had called. This was the most naive and straightforward version of circuit switching: connections were established by creating a circuit.

The era of this extremely literal form of circuit switching did not last as long as you might think. First, we have to remember that two-wire telephone circuits don't actually work that well. Low install cost and convenience means that they are the norm between a telephone exchange and its local callers, but for long-distance carriage over the phone network, you get far better results by splitting the "talk" and "listen" sides into two separate pairs. This is called a four-wire telephone circuit, and while you will rarely see four-wire service at a customer's premises, almost all connectivity between telephone exchanges (and even in the internals of the telephone exchange itself) has been four-wire since the dawn of long-distance service.

Four-wire circuits only exacerbated an obvious scaling problem: in the long distance network, you have connections called a toll lead between two exchanges. In a very simple case, two towns might have a toll lead between them. For simple four-wire telephone lines, that toll lead needs four wires for each channel. If it has four wires, only one phone call can take place between the towns at a time. If it has eight wires, two telephone calls. This got very expensive very fast, considering that even heavily-built four-crossarm open wire routes might only have a capacity for eight simultaneous calls.

For obvious reasons, research in the telephone industry focused heavily on ways to combine more calls onto fewer wires. Some simple electrical techniques could be used, like phantoms that combined two underlying pairs into a single additional "phantom" pair for a 50% increase in capacity. You could extend this technique to create more channels, with a noticeable loss in quality.

By the 1920s, the Bell System relied on a technique that we would later call frequency division multiplexing (FDM). By modulating a phone call onto a higher-frequency carrier, you can put it over the same wires as other phone calls modulated onto different frequencies. The devices that actually did this combined multiple channels onto a single line, so they were known as channel banks. The actual formats they used over the wire, since they originally consisted of modulation onto a carrier, came to be known themselves as carriers. AT&T identified the carriers they developed with simple sequential letters. In the 1940s, the state of the art was up to J- and K-carrier, which allowed 16 channels on a four-wire circuit (over open wire and multipair cable, respectively). A four-crossarm open-wire circuit, with sixteen pairs, could support 256 unidirectional circuits for 128 channels---or simultaneous phone calls.

FDM carriers reached their apex with the coaxial-cable based L-carrier and microwave radio TH and TD carriers ¹, which combined channels into groups, groups into supergroups, and supergroups into mastergroups for total capacities that reached into thousands of channels. Such huge FDM groups required very large bandwidths, though, which could not be achieved over copper pairs.

In the 1950s, rapidly developing digital electronics lead to the development of digital carriers. These carriers relied on a technique called Pulse-Code Modulation or PCM. PCM has sort of an odd name due to the history; it dates so far back into the history of digital communications that it was named before the terminology was well-established. "Pulse-code modulation" really just means "quantizing an analog signal to numbers and sending the numbers," which is now intuitive and obvious, but was an exciting new development of the 1940s.

PCM had a lot of potential for carrying phone calls, because the relaxed needs of voice transmission meant that calls could be digitized to fairly low-rate streams (8kHz by 8 bits) and engineers could see that there was a huge variety of possible techniques for combining and transporting digital signals. The best thing, though, was that digital signals could reliably be transmitted as exact copies rather than analog recreations. That meant that PCM telephone calls could pass through a complex network, with many mux/demux steps, without the reduction in quality that analog FDM carriers caused.

Even better, analog channel banks were large systems with a lot of sensitive analog components. They were expensive to build, required controlled environments, and were still subject to drift that required regular maintenance by technicians. Digital technology involved far fewer sensitive analog components, promising cheaper equipment with less maintenance. Digital was clearly the future.

The question, though, was how to best carry these digital signals over the wiring that made up much of the telephone system: copper pairs. In the late 1950s, Bell Laboratories developed T-carrier as the answer.

T-Carrier

T-carrier is a specification for transporting a stream of bits over copper pairs. The plural here is important: because T-carrier supported multiple channels, it was developed as a trunk technology, used for communication between telephone exchanges rather than between a telephone exchange and a customer. So, like other carriers used for trunks, T-carrier was four-wire, requiring two pairs to carry bidirectional signals.

T-carrier operates at 1.544Mbps, and that's about all you can say about it. The logical protocol used over T-carrier, the actual application of those bits, is determined by a separate protocol called the digital signal or DS. You can roughly think of this as a layer model, with DS running on top of T.

Here, we need to address the elephant in the room: the numbers. The numbers follow a convention used throughout AT&T-developed digital standards that is most clearly illustrated with DS. A DS0, by analogy to DS raised to the zeroeth power, is one single telephone channel expressed as a PCM digital signal. Since a telephone call is conveyed as 8-bit samples at 8kHz, a DS0 is 64kbps of data.

A DS1 is a combination of 24 DS0s, for 1.544Mbps.

A DS2 is a combination of 4 DS1s, for 96 channels or 6.312Mbps.

A DS3 is a combination of 7 DS2s, for 672 channels or 44.736Mbps.

Each level of the DS hierarchy is a TDM combination of several instances of the level below. The numbers are kind of odd, though, right? 24, 4, 7, it has the upsetting feeling of a gallon being four quarts each of which is two pints.

The DS system was developed in close parallel with the carriers actually used to convey the signal, so the short explanation for this odd scheme is that a DS1 is the number of DS0s that fit onto a T1 line, and a DS2 is the number of DS1s that fit onto a T2 line. The numbers are thus parallel: DS1 over T1, DS2 over T2, DS3 over T3. The distinction between T and DS is thus not always that important, and the terms do get used interchangeably.

But still, why 24?

Well, it's said that the number of channels on a T1 was just determined empirically. The 64kbps rate of a DS0 was fixed by the 8b x 8kHz digital format. A test T1 installation was built using a typical in-place copper telephone cable, and the number of channels was increased until it no longer functioned reliably. 24 channels was the magic number, the most achieved without unacceptable errors.

T-Carrier Infrastructure

T1 was designed for operation over a "typical" telephone cable trunk. In the 1950s, this meant a twisted-pair telephone cable installed in 6,600 foot sections with a loading coil at the junction of each section. A loading coil was essentially a big inductor hooked up to a telephone line at regular intervals to compensate for the capacitance of the line---long telephone cables, even four-wire, needed loading coils at regular intervals or the higher frequencies of speech would be lost. Loading coils also had disadvantages, though, in that they imposed a pretty sharp maximum frequency cutoff on the line. High-speed digital signaling needed to operate at those high frequencies, so T1 was designed to fit into existing long cables by replacing the loading coils with repeaters.

That means that T1 required a repeater very 6,600 feet. This repeaters were fairly small devices, often enclosed in pressurized cans to keep water out. 6,600 feet might sound pretty frequent, but because of the loading coil (and splice box) requirements trunk lines usually had underground vaults or equipment cabinets at that interval anyway.

Over time, the 6,600 foot interval became increasingly inconvenient. This was especially true as end-users started to order T1 service, requiring that T1 be supported on local loops that were often longer than 6,600 feet. Rather than installing new repeaters out in the field, it became a widespread practice to deliver T1 over a version of DSL called HDSL. HDSL is older and slower than the newer ADSL and VDSL protocols, and requires four wires, but it was fast enough to carry a DS1 signal and could cover a much longer span than traditional T-carrier. HDSL used the voice frequency band and thus could not coexist with voice calls like ADSL or VDSL, but this had the upside that it "fully controlled" the telephone line and could use measures like continuous monitoring (using a mark signal when there was no traffic) to maintain high reliability.

For the era of internet-over-T1, then, it was far more likely that a given customer actually had an HDSL connection that was converted to T1 at the customer premises by a device called a "smart jack." This pattern of the telco providing a regulated T1 service over a different media of their choice, and converting it at the customer premises, is still common today. T1s ordered later on may have actually been delivered via fiber with a similar on-premises media converter, depending on what was most convenient to the telco.

T1 is typically carried over telephone cable with 8P8C modular connectors, much like Ethernet. It requires two pairs, much like the two-line telephone wiring commonly installed in buildings. However, like most digital carriers, T-carrier is more particular about the wiring than conventional telephone. T1 wiring must be twisted-pair, and it is polarity sensitive.

DS1 Protocol

The DS1 protocol defines the carriage of 24 64kbps channels over a T1 interface. This basically amounts to TDM-muxing the 24 channels by looping over them sending one byte at a time, but there are a surprising number of nuances and details.

Early versions of DS1 only actually carried 7 bits for each sample, which was sufficient for a telephone call when companding was used to recover some of the dynamic range. The eighth bit was used for framing. T-carrier is a completely synchronous system, requiring that all of the equipment on the line have perfectly synchronized "frame clocks" to understand what bits belong to which logical channels. The framing mechanism provided a synchronization signal to achieve this perfect coordination. Later improvements in the framing protocol allowed for the use of all eight bits in some or even all of the samples. This gets to be a complicated and tangled story with many caveats, so I am going to leave it out here or this article would get a lot longer and probably contain a bunch more mistakes.

The various combinations of technologies and conventions used at different points get confusing. If you are curious, look into "robbed bit signaling," an artifact of the transition of where framing and control signals in T1 were placed that was, for some reason, a pet topic of one of my college professors. I think we spent more time on robbed-bit signaling than we did on all of MPLS, which is way cooler. Anyway, the point of this is to understand the protocol overhead involved: T1 operates at 1.544Mbps, but at least one 8-bit "sample" must be used for framing purposes, leaving 1.536Mbps of actual payload. The payload may be further reduced by other framing/signaling overhead, depending on exactly how the channel bank is configured. Most of these issues are specific to carrying telephone calls (and their related signaling); "internet service" T1 lines typically used a maximally-efficient configuration.

The Internet

So far we have pretty much only talked about telephone calls, because that's what T-carrier was developed for. By the 1980s, though, the computer industry was producing a lot of new applications for high-speed digital connections. T1 was widely available, and in many cases a tariffed/regulated service, so it was one of the most widely available high-speed data connections. Especially in the very early days of the internet, it was often the only option.

Into the 1990s, T1 was one of the dominant options for commercial internet access. It was rarely seen in homes, though, as it was quite expensive. Keep in mind that, from the perspective of the phone company, a T1 line was basically 24 phone lines. They charged accordingly.

To obtain internet service, you would order either from the telco itself or from an independent provider that then ordered a connection from your premises to their point of presence on an open access basis. In this case you were effectively paying two bills, one to the telco for the T1 line and the other to the independent provider for internet connectivity... but the total was still often more affordable than the telco's high rates for internet services.

Because of the popularity of T1 for internet access, routers with T1 interfaces were readily available. Well, really, the wide variety of media used for data connections before Ethernet became such a common standard means that many routers of the era took interchangeable modules, and T1 was one of the modules you could get for them.

In implementation, a T1 line was basically a fast serial line from your equipment to your ISP's equipment. What actually ran over that serial line was up to the ISP, and there were multiple options. The most classical would be frame relay, a successor to X.25 mostly associated with ISDN. PPP was also a fairly common option, as with consumer DSL, and more exotic protocols existed for specialized purposes or ISPs that were just a little weird.

When the internet was new, 1.5Mbps T1 was really very fast---the NSFNET backbone was lauded for its speed when it went online as an all-T1 network in 1991. Of course, today, a 1.5Mbps "backbone" internet connection is pretty laughable. Even as the '90s progressed, 1.5Mbps started to feel tight.

One of the things I find odd about the role of T1 in the history of internet access is that the era when a T1 was "blazing fast" was really pretty short. By 2000, when online gaming for example was taking off, both DSL and cable offered significantly better downstream speeds than T1. However, the nature of T1 as a circuit-switched, telephone-engineered TDM protocol made it very reliable and low-latency, properties that most faster internet media performed poorly on (early DSL notoriously so). Multiplayer gaming would likely have been a better experience on T1 than on a DSL connection offering multiples of the theoretical bandwidth.

A faster T1

Of course, there were options to speed up T-carrier. The most obvious is to combine multiple T1 connections, which was indeed a common practice. Later T1 interfaces were often supplied in multiple for that reason. MLPPP is a variant of the PPP protocol intended for combining the bandwidth of multiple links, referred to in the telephone industry as bonding.

But there were also higher levels in the hierarchy. Remember DS2 and DS3? Well, in practice, T2 wasn't really used. It was far more common to bond multiple T1 connections to reach the equivalent speed of a T2. 44.736Mbps T3 did find use, though. The catch is that T3 required specialized cabling (coaxial pairs!) and had a fairly short range, so it was usually not practical between a telephone exchange and a business.

Fortunately, by the time these larger bandwidths were in demand, fiber optic technology had become well-established. The telephone industry primarily used SONET, a fiber media over which Synchronous Digital Hierarchy (SDH) channels were carried. SONET comes in formats identified by OC (Optical Carrier) numbers in a way very similar to T-carrier numbers. An OC-1 is 51.840Mbps, already faster than T3/DS3. So, in practice, DS3 service was pretty much always provided by a media converter from an OC-1 SONET ring. As bandwidth demands further increased, businesses were much more likely to directly order SONET service rather than T-carrier. SONET was available into the multiple Gbps and enjoyed a long life as a popular internet carrier.

Of course, as the internet proliferated, so too did the stack of network media designed specifically for opportunistic packet-switching computer networks. Chief among them was Ethernet. These protocols have now overtaken traditional telephony protocols in most internet applications, so SONET as well is now mostly out of the picture. On the upside, Ethernet networks are generally more cost-effective on a bandwidth basis and allow you to obtain much faster service than you would be able to afford over the telephone network. On the downside, Ethernet networks have no circuit switching or traffic engineering, and so they do not provide the quality of service that the telephone network did. This means more jitter and less predictability of the available bandwidth, an enduring challenge for real-time media applications.

Miscellaneous

One of the confusing things about T1 to modern readers is its interaction with ISDN. ISDN was a later development that introduced a lot more standardization to digital communications over the telephone network, but it incorporated and reused existing digital technology including T-carrier. In the world of ISDN, a "DS0" or single telephone channel is usually part of a basic rate interface (BRI), while the 24-channel T1 bandwidth is called a primary rate interface (PRI). Many T1 connections in the 1990s were actually ISDN PRIs.

The difference between the two is in the fine details: many of the details related to framing and control signals that were, shall we say, "loosey-goosey" with T-carrier are completely standardized under ISDN. An ISDN PRI always consists of 23 bearer channels ("payload" channels) and one control channel, and the framing is always the same. Since there is a dedicated control channel, there's no need to do weird things with the bits in the bearer channels, and so the overhead is standard across all ISDN PRIs.

In practice, the difference between T1 and ISDN PRI was usually just a matter of configuring your router's interface appropriately. Because of the curious details of the regulation and tariff processes, one was sometimes cheaper than the other, and in general the choice of whether to use a "T1" or a "PRI" was often somewhat arbitrary. It's even possible to use some T1 channels in the traditional way and others as ISDN.

While T1 is now mostly forgotten, some parts of its design live on. DSL and T1 have always had a relationship, DSL having originally been developed as basically a "better T-carrier" for ISDN use. In the modern world, DSL pretty much always refers to either ADSL or VDSL, newer protocols designed for consumer service that can coexist with a voice line and provide very high speeds. Many aspects of how DSL works have their roots in T-carrier, including the common use of protocols like ATM (now rare) or PPP (fading out) to encapsulate IP over DSL.

Okay, I realize this article has been sort of dry and factual, but I thought it'd be interesting to share a bit about T-carrier. I think it's something that people my age vaguely remember as important but have never thought about that much. I, personally, am probably just a bit too young to have had much real interaction with T-carrier. When I worked for an MSP for a bit in high school I saw a few customers that still had HDSL-based T1 service, and when I later interned at General Electric we had an OC-3 SONET connection that was in the process of being turned down. Just really catching the trail end... and yet for years later the Steam Hardware Survey was still asking if I had a T1.

Why did T1 get stuck so long in the very specific context of video games? I assume because video game developers frequently had T-carrier connections to their offices and knew that its guaranteed bandwidth provided very good performance for video games. The poor latency of ADSL meant that, despite a theoretical bandwidth several times larger, it was not really a better choice for the specific application of multiplayer games. So the "T1 is god tier" thing hung around for longer than you would have otherwise expected.

An opening note: would you believe that I have been at this for five years, now? If I planned ahead better, I would have done this on the five-year anniversary, but I missed it. Computers Are Bad is now five years and four months old.

When I originally launched CAB, it was my second attempt at keeping up a blog. The first, which I had called 12 Bit Word, went nowhere and I stopped keeping it up. One of the reasons, I figured, is that I had put too much effort into it. CAB was a very low-effort affair, which was perhaps best exemplified by the website itself. It was monospace and 80 characters wide, a decision that I found funny (in a shitposty way) and generated constant complaints. To be fair, if you didn't like the font, it was "user error:" I only ever specified "monospace" and I can't be blamed that certain platforms default to Courier. But there were problems beyond the appearance; the tool that generated the website was extremely rough and made new features frustrating to implement.

Over the years, I have not invested much (or really any) effort in promoting CAB or even making it presentable. I figured my readership, interested in vintage computing, would probably put up with it anyway. That is at least partially true, and I am not going to put any more effort into promotion, but some things have changed. Over time I have broadened my topics quite a bit, and I now regularly write about things that I would have dropped as "off topic" three or four years ago. Similarly, my readership has broadened, and probably to a set of people that find 80 characters of monospace text less charming.

I think I've also changed my mind in some ways about what is "special" about CAB. One of the things that I really value about it, that I don't think comes across to readers well, is the extent to which it is what I call artisanal internet. It's like something you'd get at the farmer's market. What I mean by this is that CAB is a website generated by a static site generator that I wrote, and a newsletter sent by a mailing list system that I wrote, and you access them by connecting directly to a VM that I administer, on a VM cluster that I administer, on hardware that I own, in a rack that I lease in a data center in downtown Albuquerque, New Mexico. This is a very old-fashioned way of doing things, now, and one of the ironies is that it is a very expensive way of doing things. It would be radically cheaper and easier to use wordpress.com, and it would probably go down less often and definitely go down for reasons that are my fault less often. But I figure people listen to me in part because I don't use wordpress.com, because I have weird and often impractical opinions about how to best contribute to internet culture.

I spent a week on a cruise ship just recently, and took advantage of the great deal of time I had to look at the sea to also get some work done. Strategically, I decided, I want to keep the things that are important to me (doing everything myself) and move on from the things that are not so important (the website looking, objectively, bad). So this is all a long-winded announcement that I am launching, with this post, a complete rewrite of the site generator and partial rewrite of the mailing list manager.

This comes with several benefits to you. First, computer.rip is now much more readable and, arguably, better looking. Second, it should be generally less buggy (although to be fair I had eliminated most of the problems with the old generator through sheer brute force over the years). Perhaps most importantly, the emails sent to the mailing list are no longer the unrendered Markdown files.

I originally didn't use markup of any kind, so it was natural to just email out the plaintext files. But then I wanted links, and then I wanted pictures, leading me to implement Markdown in generating the webpages... but I just kept emailing out the plaintext files. I strongly considered switching to HTML emails as a solution and mostly finished the effort, but in the end I didn't like it. HTML email is a massive pain in the ass and, I think, distasteful. Instead, I modified a Markdown renderer to create human-readable plaintext output. Things like links and images will still be a little weird in the plaintext emails, but vastly better than they were before.

I expect some problems to surface when I put this all live. It is quite possible that RSS readers will consider the most recent ten posts to all be new again due to a change in how the article IDs are generated. I tried to avoid that happening but, look, I'm only going to put so much time into testing and I've found RSS readers to be surprisingly inconsistent. If anything else goes weird, please let me know.

---

There has long been a certain connection between the computer industry and the art of animation. The computer, with a frame-oriented raster video output, is intrinsically an animation machine. Animation itself is an exacting, time-consuming process that has always relied on technology to expand the frontier of the possible. Walt Disney, before he was a business magnate, was a technical innovator in animation. He made great advances in cel animation techniques during the 1930s, propelling the Disney Company to fame not only by artistic achievement but also by reducing the cost and time involved in creating feature-length animated films.

Most readers will be familiar with the case of Pixar, a technical division of Lucasfilm that operated primarily as a computer company before its 1986 spinoff under computer executive Steve Jobs---who led the company through a series of creative successes that overshadowed the company's technical work until it was known to most only as a film studio.

Animation is hard. There are several techniques, but most ultimately come down to an animator using experience, judgment, and trial and error to get a series of individually composed frames to combine into fluid motion. Disney worked primarily in cel animation: each element of each frame was hand-drawn, but on independent transparent sheets. Each frame was created by overlaying the sheets like layers in a modern image editor. The use of separate cels made composition and corrections easier, by allowing the animator to move and redraw single elements of the final image, but it still took a great deal of experience to produce a reasonable result.

The biggest challenge was in anticipating how motion would appear. From the era of Disney's first work, problems like registration (consistent positioning of non-moving objects) had been greatly simplified by the use of clear cels and alignment pegs on the animator's desk that held cels in exact registration for tracing. But some things in an animation are supposed to move, I would say that's what makes it animation. There was no simple jig for ensuring that motion would come out smoothly, especially for complex movements like a walking or gesturing character. The animator could flip two cels back and forth, but that was about as good as they could get without dedicating the animation to film.

For much of the mid-century, a typical animation workflow looked like this: a key animator would draw out the key frames in final or near-final quality, establishing the most important moments in the animation, the positions and poses of the characters. The key animator or an assistant would then complete a series of rough pencil sketches for the frames that would need to go in between. These sketches were sent to the photography department for a "pencil test."

In the photography department, a rostrum camera was used: a cinema camera, often 16mm, permanently mounted on an adjustable stand that pointed it down at a flat desk. The rostrom camera looked a bit like a photographic enlarger and worked much the same way, but backwards: the photographer laid out the cels or sketches on the desk, adjusted the position and focus of the camera for the desired framing, and then exposed one frame. This process was repeated, over and over and over, a simple economy that explains the common use of a low 12 FPS frame rate in animation.

Once the pencil test had been photographed, the film went to the lab where it was developed, and then returned to the animation studio where the production team could watch it played on a cinema projector in a viewing room. Ideally, any problems would be identified during this first viewing before the key frames and pencil sketches were sent to the small army of assistant animators. These workers would refine the cels and redraw the pencil sketches in part by tracing, creating the "in between" frames of the final animation. Any needed changes were costly, even when caught at the earliest stage, as it usually took a full day for the photography department to return a new a pencil test (making the pencil test very much analogous to the dailies used in film). What separated the most skilled animators from amateurs, then, was often their ability to visualize the movement of their individual frames by imagination. They wanted to get it right the first time.

---

Graphics posed a challenge to computers for similar reasons. Even a very basic drawing involves a huge number of line segments, which a computer will need to process individually during rendering. Add properties such as color, consider the practicalities of rasterizing, and then make it all move: just the number of simple arithmetic problems involved in computer graphics becomes enormous. It is not a coincidence that we picture all early computer systems as text-only, although it is a bit unfair. Graphical output is older than many realize, originating with vector-mode CRT displays in the 1950s. Still, early computer graphics were very slow. Vector-mode displays were often paired with high-end scientific computers and you could still watch them draw in real time. Early graphics-intensive computer applications like CAD used specialized ASICs for drawing and yet provided nothing like the interactivity we expect from computers today.

The complexity of computer graphics ran head-first against an intense desire for more capable graphical computers, driven most prominently by the CAD industry. Aerospace and other advanced engineering fields were undergoing huge advancements during the second half of the 20th century. World War II had seen adoption of the jet engine, for example, machines which were extremely powerful but involved complex mathematics and a multitude of 3D parts that made them difficult for a human to reason over. The new field of computer-aided design promised a revolutionary leap in engineering capability, but ironically, the computers were quite bad at drawing. In the first decades, CAD output was still being sent to traditional draftsmen for final drawings. The computers were not only slow, but unskilled at the art of drafting: limitations on the number and complexity of the shapes that computers could render limited them to only very basic drawings, without the extensive annotations that would be needed for manufacturing.

During the 1980s, the "workstation" began to replace the mainframe in engineering applications. Today, "workstation" mostly just identifies PCs that are usually extra big and always extra expensive. Historically, workstations were a different class of machines from PCs that often employed fundamentally different architectures. Many workstations were RISC, an architecture selected for better mathematical performance. They frequently ran UNIX or a derivative, and featured the first examples of what we now call a GPU. Some things don't change: they were also very big, and very expensive.

It was the heady days of the space program and the Concorde, then, that brought us modern computer graphics. The intertied requirements for scientific computing, numerical simulation, and computer graphics that emerged from Cold War aerospace and weapons programs forged a strong bond between high-end computing and graphics. One could perhaps say that the nexus between AI and GPUs today is an extension of this era, although I think it's a bit of a stretch given the text-heavy applications. The echoes of the dawn of computer graphics are much quieter today, but still around. They persist, for example, in the heavy emphasis on computer visualization seen throughout scientific computing but especially in defense-related fields. They persist also in the names of the companies born in that era, names like Silicon Graphics and Mentor Graphics.

---

The development of video technology, basically the combination of preexisting television technology with new video tape recorders, lead to a lot of optimizations in film. Video was simply not of good enough quality to displace film for editing and distribution, but it was fast and inexpensive. For example, beginning in the 1960s filmmakers began to adopt a system called "video assist." A video camera was coupled to the film camera, either side-by-side with matched lenses or even sharing the same lens via a beam splitter. By running a video tape recorder during filming, the crew could generate something like an "instant daily" and play the tape back on an on-set TV. For the first time, a director could film a scene and then immediately rewatch it. Video assist was a huge step forward, especially in the television industry where it furthered the marriage of film techniques and television techniques for the production of television dramas.

It certainly seems that there should be a similar optimization for animation. It's not easy, though. Video technology was all designed around sequences of frames in a continuous analog signal, not individual images stored discretely. With the practicalities of video cameras and video recorders, it was surprisingly difficult to capture single frames and then play them back to back.

In the 1970s, animators Bruce Lyon and John Lamb developed the Lyon-Lamb Video Animation System (VAS). The original version of the VAS was a large workstation that replaced a rostrum camera with a video camera, monitor, and a custom video tape recorder. Much like the film rostrum camera, the VAS allowed an operator to capture a single frame at a time by composing it on the desk. Unlike the traditional method, the resulting animation could be played back immediately on the included monitor.

The VAS was a major innovation in cel animation, and netted both an Academy Award and an Emmy for technical achievement. While it's difficult to say for sure, it seems like a large portion of the cel-animated features of the '80s had used the VAS for pencil tests. The system was particularly well-suited to rotoscoping, overlaying animation on live-action images. Through a combination of analog mixing techniques and keying, the VAS could directly overlay an animator's work on the video, radically accelerating the process. To demonstrate the capability, John Lamb created a rotoscoped music video for the Tom Waits song "The One That Got Away." The resulting video, titled "Tom Waits for No One," was probably the first rotoscoped music video as well as the first production created with the video rotoscope process. As these landmarks often do, it languished in obscurity until it was quietly uploaded to YouTube in 2006.

The VAS was not without its limitations. It was large, and it was expensive. Even later generations of the system, greatly miniaturized through the use of computerized controls and more modern tape recorders, came in at over $30,000 for a complete system. And the VAS was designed around the traditional rostrom camera workflow, intended for a dedicated operator working at a desk. For many smaller studios the system was out of reach, and for forms of animation that were not amenable to top-down photography on a desk, the VAS wasn't feasible.

---

There are some forms of animation that are 3D---truly 3D. Disney had produced pseudo-3D scenes by mounting cels under a camera on multiple glass planes, for example, but it was obviously possible to do so in a more complete form by the use of animated sculptures or puppets. Practical challenges seem to have left this kind of animation mostly unexplored until the rise of its greatest producer, Will Vinton.

Vinton grew up in McMinnville, Oregon, but left to study at UC Berkeley. His time in Berkeley left him not only with an architecture degree (although he had studied filmmaking as well), but also a friendship with Bob Gardiner. Gardiner had a prolific and unfortunately short artistic career, in which he embraced many novel media including the hologram. Among his inventions, though, was a novel form of animation using clay: Gardiner was fascinated with sculpting and posing clay figures, and demonstrated the animation potential to Vinton. Vinton, in turn, developed a method of using his student film camera to photograph the clay scenes frame by frame.

Their first full project together, Closed Mondays, took the Academy Award for Best Animated Short Film in 1975. It was notable not only for the moving clay sculptures, but for its camerawork. Vinton had realized that in clay animation, where scenes are composed in real 3D space, the camera can be moved from frame to frame just like the figures. Not long after this project, Vinton and Gardiner split up. Gardiner seems to have been a prolific artist in that way where he could never stick to one thing for very long, and Vinton had a mind towards making a business out of this new animation technology. It was Vinton who christened it Claymation, then a trademark of his new studio.

Vinton returned to his home state and opened Will Vinton Studios in Portland. Vinton Studios released a series of successful animated shorts in the '70s, and picked up work on numerous other projects, contributing for example to the "Wizard of Oz" film sequel "Return to Oz" and the Disney film "Captain EO." By far Vinton Studios most famous contributions to our culture, though, are their advertising projects. Will Vinton Studios brought us the California Raisins, the Noid, and walking, talking M&M's.

Will Vinton Studios struggled with producing claymation at commercial scale. Shooting with film cameras, it took hours to see the result. Claymation scenes were more difficult to rework than cel animation, setting an even larger penalty for reshoots. Most radically, claymation scenes had to be shot on sets, with camera and light rigging. Reshooting sections without continuity errors was as challenging as animating those sections in the first place.

To reduce rework, they used pencil tests: quicker, lower-effort versions of scenes shot to test the lighting, motion, and sound synchronization before photography with a film camera. Their pencil tests were apparently captured on a crude system of customized VCRs, allowing the animator to see the previous frame on a monitor as they composed the next, and then to play back the whole sequence. It was better than working from film, but it was still slow going.

---

The area from Beaverton to Hillsboro, in Oregon near Portland, is sometimes called "the silicon forest" largely on the influence of Intel and Tektronix. As in the better known silicon valley, these two keystone companies were important not only on their own, but also as the progenitors of dozens of new companies. Tektronix, in particular, had a steady stream of employees leaving to start their own businesses. Among these alumni was Mentor Graphics.

Mentor Graphics was an early player in electronic design automation (EDA), sort of like a field of CAD specialized to electronics. Mentor products assisted not just in the physical design of circuit boards and ICs, but also simulation and validation of their functionality. Among the challenges of EDA are its fundamentally graphical nature: the final outputs of EDA are often images, masks for photolithographic manufacturing processes, and engineers want to see both manufacturing drawings and logical diagrams as they work on complex designs.

When Mentor started out in 1981, EDA was in its infancy and relied mostly on custom hardware. Mentor went a different route, building a suite of software products that ran on Motorola 68000-based workstations from Apollo. The all-software architecture had cost and agility advantages, and Mentor outpaced their competition to become the field's leader.

Corporations want for growth, and by the 1990s Mentor had a commanding position in EDA and went looking for other industries to which their graphics-intensive software could be applied. One route they considered was, apparently, animation: computer animation was starting to take off, and there were very few vendors for not just the animation software but the computer platforms capable of rendering the product. In the end, Mentor shied away: companies like Silicon Graphics and Pixar already had a substantial lead, and animation was an industry that Mentor knew little about. As best I can tell, though, it was this brief investigation of a new market that exposed Mentor engineering managers Howard Mozeico and Arthur Babitz to the animation industry.

I don't know much about their career trajectories in the years shortly after, only that they both decided to leave Mentor for their own reasons. Arthur Babitz went into independent consulting, and found a client reminiscent of his work at Mentor, an established animation studio that was expanding into computer graphics: Will Vinton Studios. Babitz's work at Will Vinton Studios seems to have been largely unrelated to claymation, but it exposed him to the process, and he watched the way they used jury-rigged VCRs and consumer video cameras to preview animations.

Just a couple of years later, Mozeico and Babitz talked about their experience with animation at Mentor, a field in which they were both still interested. Babitz explained the process he had seen at Will Vinton Studios, and his ideas for improving it. Both agreed that they wanted to figure out a sort of retirement enterprise, what we might now call a "lifestyle business": they each wanted to found a company that would keep them busy, but not too busy. The pair incorporated Animation Toolworks, headquartered in Mozeico's Sherwood, Oregon home.

---

In 1998 Animation Toolworks hit trade shows with the Video Lunchbox. The engineering was mostly by Babitz, the design and marketing by Mozeico, and the manufacturing done on contract by a third party. The device took its name from its form factor, a black crinkle paint box with a handle on top of its barn-roof-shaped lid. It was something like the Lyon Lamb VAS, if it was portable, digital, and relatively inexpensive.

The Lunchbox was essentially a framegrabber, a compact and simplified version of the computer framegrabbers that were coming into use in the animation industry. You plugged a video camera into the input, and a television monitor into the output. You could see the output of the camera, live, on the monitor while you composed a scene. Then, one press of a button captured a single frame and stored it. With a press of another button, you could swap between the stored frame and the live image, helping to compose the next. You could even enable an automatic "flip-flop" mode that alternated the two rapidly, for hands-free adjustment.

Each successive press of the capture button stored another frame to the Lunchbox's memory, and buttons allowed you to play the entire set of stored frames as a loop, or manually step forward or backward through the frames. And that was basically it: there were a couple of other convenience features like an intervalometer (for time lapse) and the ability to record short sections of real-time video, but complete operation of the device was really very simple. That seems to have been one of its great assets. The Lunchbox was much easier to sell after Mozeico gave a brief demonstration and said that that was all there is to it.

To professionals, the Lunchbox was a more convenient, more reliable, and more portable version of the video tape recorder or computer framegrabber systems they were already using for pencil tests. Early customers of Animation Toolworks included Will Vinton Studios alongside other animation giants like Disney, MTV, and Academy Award-winning animator Mark Osborne. Animation Toolworks press quoted animators from these firms commenting on the simplicity and ease of use, saying that it had greatly sped up the animation test process.

In a review for Animation World Magazine, Kellie-Bea Rainey wrote:

In most cases, computers as framegrabbers offer more complications than solutions. Many frustrations stem from the complexity of learning the computer, the software and it's constant upgrades. But one of the things Gary Schwartz likes most about the LunchBox is that the system requires no techno-geeks. "Computers are too complex and the technology upgrades are so frequent that the learning curve keeps you from mastering the tools. It seems that computers are taking the focus off the art. The Video LunchBox has a minimum learning curve with no upgrade manuals. Everything is in the box, just plug it in."

Indeed, the Lunchbox was so simple that it caught on well beyond the context of professional studios. It is remembered most as an educational tool. Disney used the Lunchbox for teaching cel animation in a summer program, but closer to home, the Lunchbox made its way to animation enthusiast and second-grade teacher Carrie Caramella. At Redmond, Oregon's John Tuck Elementary School, Caramella acted as director of a student production team that brought their short film "The Polka Dot Day" to the Northwest Film Center's Young People's Film and Video Festival. During the early 2000s, after-school and summer animation programs proliferated, many using claymation, and almost all using the Video Lunchbox.

At $3,500, the Video Lunchbox was not exactly cheap. It cost more than some of the more affordable computer-based options, but it was so much easier to use, and so much more durable, that it was very much at home in a classroom. Caramella:

"By using the lunchbox, we receive instant feedback because the camera acts > as an eye. It is also child-friendly, and you can manipulate the film a lot more."

Caramella championed animation at John Tuck, finding its uses in other topics. A math teacher worked with students to make a short animation of a chicken. In a unit on compound words, Caramella led students in animating their two words together: a sun and a flower dance; the word is "sunflower." Butter and milk, base and ball.

In Lake Oswego, an independent summer program called Earthlight Studios took up the system.

With the lunchbox, Corey's black-and-white drawings spring to life, two catlike animé characters circling each other with broad edged swords. It's the opening seconds of what he envisions will be an action-adventure film.

We can imagine how cringeworthy these student animations must be to their creators today, but early-'00s education was fascinated with multimedia and it seems rare that technology served the instructional role so well.

It was in this context that I crossed ways with the Lunchbox. As a kid, I went to a summer animation program at OMSI---a claymation program, which I hazily remember was sponsored by a Will Vinton Studios employee. In an old industrial building beside the museum, we made crude clay figures and then made them crudely walk around. The museum's inventory of Lunchboxes already showed their age, but they worked, in a way that was so straightforward that I think hardly any time was spent teaching operation of the equipment. It was a far cry from an elementary school film project in which, as I recall, nearly an entire day of class time was burned trying to get video off of a DV camcorder and into iMovie.

---

Mozeico and Babitz aimed for modest success, and that was exactly what they found. Animation Toolworks got started on so little capital that it turned a profit the first year, and by the second year the two made a comfortable salary---and that was all the company would ever really do. Mozeico and Babitz continued to improve on the concept. In 2000, they launched the Lunchbox Sync, which added an audio recorder and the ability to cue audio clips at specific frame numbers. In 2006, the Lunchbox DV added digital video.

By the mid-2000s, computer multimedia technology had improved by leaps and bounds. Framegrabbers and real-time video capture devices were affordable, and animation software on commodity PCs overtook the Lunchbox on price and features. Still, the ease of use and portability of the Lunchbox was a huge appeal to educators. By 2005 Animation Toolworks was basically an educational technology company, and in the following years computers overtook them in that market as well.

The era of the Lunchbox is over, in more ways than one. A contentious business maneuver by Phil Knight saw Will Vinton pushed out of Will Vinton Studios. He was replaced by Phil Knight's son, Travis Knight, and the studio rebranded to Laika. The company has struggled under its new management, and Laika has not achieved the renaissance of stop-motion that some thought Coraline might bring about. Educational technology has shifted its focus, as a business, to a sort of lightweight version of corporate productivity platforms that is firmly dominated by Google.

Animation Toolworks was still selling the Lunchbox DV as late as 2014, but by 2016 Mozeico and Babitz had fully retired and offered support on existing units only. Mozeico died in 2017, crushed under a tractor on his own vineyard. There are worse ways to go. Arthur Babitz is a Hood River County Commissioner.

Kellie-Bea Rainey:

I took the two-minute tutorial and taped it to the wall. I cleaned off a work table and set up a stage and a character. Then I put my Sharp Slimcam on a tripod... Once the camera was plugged into the LunchBox, I focused it on my animation set-up. Next, I plugged in my monitor.

All the machines were on and all the lights were green, standing by. It's time to hit the red button on the LunchBox and animate!

Yippee! Look Houston, we have an image! That was quick, easy and most of all, painless. I want to do more, and more, and even more.

The next time you hear from me I'll be having fun, teaching my own animation classes and making my own characters come to life. I think Gary Schwartz says it best, "The LunchBox brings the student back to what animation is all about: art, self-esteem, results and creativity."

I think we're all a little nostalgic for the way technology used to be. I know I am. But there is something to be said for a simple device, from a small company, that does a specific thing well. I'm not sure that I have ever, in my life, used a piece of technology that was as immediately compelling as the Video Lunchbox. There are numerous modern alternatives, replete with USB and Bluetooth and iPad apps. Somehow I am confident that none of them are quite as good.

I have an ongoing fascination with "interactive TV": a series of efforts, starting in the 1990s and continuing today, to drag the humble living room television into the world of the computer. One of the big appeals of interactive TV was adoption, the average household had a TV long before the average household had a computer. So, it seems like interactive TV services should have proliferated before personal computers, at least following the logic that many in the industry did at the time.

This wasn't untrue! In the UK, for example, Ceefax was a widespread success by the 1980s. In general, TV-based teletext systems were pretty common in Europe. In North America, they never had much of an impact---but not for lack of trying. In fact, there were multiple competing efforts at teletext in the US and Canada, and it may very well have been the sheer number of independent efforts that sunk the whole idea. But let's start at the beginning.

The BBC went live with Ceefax in 1974, the culmination of years of prototype development and test broadcasts over the BBC network. Ceefax was quickly joined by other teletext standards in Europe, and the concept enjoyed a high level of adoption. This must have caught the attention of many in the television industry on this side of the ocean, but it was Bonneville International that first bit ¹. Its premier holding, KSL-TV of Salt Lake City, has an influence larger than its name suggests: KSL was carried by an extensive repeater network and reached a large portion of the population throughout the Mountain States. Because of the wide reach of KSL and the even wider reach of the religion that relied on Bonneville for communications, Bonneville was also an early innovator in satellite distribution of television and data. These were ingredients that made for a promising teletext network, one that could quickly reach a large audience and expand to broader television networks through satellite distribution.

KSL applied to the FCC for an experimental license to broadcast teletext in addition to its television signal, and received it in June of 1978. I am finding some confusion in the historical record over whether KSL adopted the BBC's Ceefax protocol or the competing ORACLE, used in the UK by the independent broadcasters. A 1982 paper on KSL's experiment confusingly says they used "the British CEEFAX/Oracle," but then in the next sentence the author gives the first years of service for Ceefax and ORACLE the wrong way around, so I think it's safe to say that they were just generally confused. I think I know the reason why: in the late '70s, the British broadcasters were developing something called World System Teletext (WST), a new common standard based on aspects of both Ceefax and ORACLE. Although WST wasn't quite final in 1978, I believe that what KSL adopted was actually a draft of WST.

That actually hints at an interesting detail which becomes important to these proposals: in Europe, where teletext thrived, there were usually not very many TV channels. The US's highly competitive media landscape lead to a proliferation of different TV networks, and local operations in addition. It was a far cry from the UK, for example, where 1982 saw the introduction of a fourth channel called, well, Channel 4. By contrast, Salt Lake City viewers with cable were picking from over a dozen channels in 1982, and that wasn't an especially crowded media market. This difference in the industry, between a few major nationwide channels and a longer list of often local ones, has widespread ramifications on how UK and US television technology evolved.

One of them is that, in the UK, space in the VBI to transmit data became a hotly contested commodity. By the '80s, obtaining a line of the VBI on any UK network to use for your new datacasting scheme involved a bidding war with your potential competitors, not unlike the way spectrum was allocated in the US. Teletext schemes were made and broken by the outcomes of these auctions. Over here, there was a long list of television channels and on most of them only a single line of the VBI was in use for data (line 21 for closed captions). You might think this would create fertile ground for VBI-based services, but it also posed a challenge: the market was extensively fractured. You could not win a BBC or IBA VBI allocation and then have nationwide coverage, you would have to negotiate such a deal with a long list of TV stations and then likely provide your own infrastructure for injecting the signal.

In short, this seems to be one of the main reasons for the huge difference in teletext adoption between Europe and North America: throughout Europe, broadcasting tended to be quite centralized, which made it difficult to get your foot in the door but very easy to reach a large customer base once you had. In the US, it was easier to get started, but you had to fight for each market area. "Critical mass" was very hard to achieve ².

Back at KSL, $40,000 (~$200,000 today) bought a General Automation computer and Tektronix NTSC signal generator that made up the broadcast system. The computer could manage as many as 800 pages of 20x32 teletext, but KSL launched with 120. Texas Instruments assisted KSL in modifying thirty television sets with a new decoder board and a wired remote control for page selection. This setup, very similar to teletext sets in Europe, nearly doubled the price of the TV set. This likely would have become a problem later on, but for the pilot stage, KSL provided the modified sets gratis to their 30 test households.

One of the selling points of teletext in Europe was its ability to provide real-time data. Things like sports scores and stock quotations could be quickly updated in teletext, and news headlines could make it to teletext before the next TV news broadcast. Of course, collecting all that data and preparing it as teletext pages required either a substantial investment in automation or a staff of typists. At the pilot stage, KSL opted for neither, so much of the information that KSL provided was out-of-date. It was very much a prototype. Over time, KSL invested more in the system. In 1979, for example, KSL partnered with the National Weather Service to bring real-time weather updates to teletext---all automatically via the NWS's computerized system called AFOS.

At that time, KSL was still operating under an experimental license, one that didn't allow them to onboard customers beyond their 30-set test market. The goal was to demonstrate the technology and its compatibility with the broader ecosystem. In 1980, the FCC granted a similar experimental license to CBS affiliated KMOX in St. Louis, who started a similar pilot effort using a French system called Antiope. Over the following few years, the FCC allowed expansion of this test to other CBS affiliates including KNXT in Los Angeles. To emphasize the educational and practical value of teletext (and no doubt attract another funding source), CBS partnered with Los Angeles PBS affiliate KCET who carried their own Teletext programming with a characteristic slant towards enrichment. Meanwhile, in Chicago, station WFLD introduced a teletext service called Keyfax, built on Ceefax technology as a joint venture with Honeywell and telecom company Centel. Despite the lack of consumer availability, teletext was becoming a crowded field---and for the sake of narrative simplicity I am leaving out a whole set of other North American ventures right now.

In 1983, there were at least a half dozen stations broadcasting teletext based on British or French technology, and yet, there were zero teletext decoders on the US market. Besides their use of an experimental license, the teletext pilot projects were constrained by the need for largely custom prototype decoders integrated into customer's television sets. Broadcast executives promised the price could come down to $25, but the modifications actually available continued to cost in the hundreds. The director of public affairs at KSL, asked about this odd conundrum of a nearly five-year-old service that you could not buy, pointed out that electronics manufacturers were hesitant to mass produce an inexpensive teletext decoder as long as it was unclear which of several standards would prevail. The reason that no one used teletext, then, was in part the sheer number of different teletext efforts underway. And, of course, things were looking pretty evenly split: CBS had fully endorsed the French-derived system, and was a major nationwide TV network. But most non-network stations with teletext projects had gone the British route. In terms of broadcast channels, it was looking about 50/50.

Further complicating things, teletext proper was not the only contender. There was also videotex. The terminology has become somewhat confused, but I will stick to the nomenclature used in the 1980s: teletext services used a continuous one-way broadcast of every page and decoders simply displayed the requested page when it came around in the loop. Videotex systems were two-way, with the customer using a phone line to request a specific page which was then sent on-demand. Videotex systems tended to operate over telephone lines rather than television cable, but were frequently integrated into television sets. Videotex is not as well remembered as teletext because it was a massive commercial failure, with the very notable exception of the French Minitel.

But in the '80s they didn't know that yet, and the UK had its own videotex venture called Prestel. Prestel had the backing of the Post Office, because they ran the telephones and thus stood to make a lot of money off of it. For the exact same reason, US telephone company GTE bought the rights to the system in the US.

Videotex is significantly closer to "the internet" in its concept than teletext, and GTE was entering a competitive market. In 1981, Radio Shack had introduced a videotex terminal for several years already, a machine originally developed as the "AgVision" for use with an experimental Kentucky agricultural videotex service and then offered nationwide. This creates an amusing irony: teletext services existed but it was very difficult to obtain a decoder to use them. Radio Shack was selling a videotex client nationwide, but what service would you use it with? In practice, the "TRS-80 Videotex" as the AgVision came to be known was used mostly as a client for CompuServe and Dow Jones. Neither of these were actually videotex services, using neither the videotex UX model nor the videotex-specific features of the machine. The TRS-80 Videotex was reduced to just a slightly weird terminal with a telephone modem, and never sold well until Radio Shack beefed it up into a complete microcomputer and relaunched it as the TRS-80 Color Computer.

Radio Shack also sold a backend videotex system, and apparently some newspapers bought it in an effort to launch a "digital edition." The only one to achieve long-term success seems to have been StarText, a service of the Fort Worth Star-Telegram. It was popular enough to be remembered by many from the Fort Worth area, but there was little national impact. It was clearly not enough to float sales of the TRS-80 Videotex and the whole thing has been forgotten. Well, with such a promising market, GTE brought its US Prestel service to market in 1982. As the TRS-80 dropped its Videotex ambitions, Zenith launched a US television set with a built-in Prestel client.

Prestel wasn't the only videotex operation, and GTE wasn't the only company marketing videotex in the US. If the British Post Office and GTE thought they could make money off of something, you know AT&T was somewhere around. They were, and in classic AT&T fashion. During the 1970s, the Canadian Communications Research Center developed a vector-based drawing system. Ontario manufacturer Norpak developed a consumer terminal that could request full-color pages from this system using a videotex-like protocol. Based on the model of Ceefax, the CRC designed a system called Telidon that worked over television (in a more teletext-like fashion) or phone lines (like videotex), with the capability of publishing far more detailed graphics than the simple box drawings of teletext.

Telidon had several cool aspects, like the use of a pretty complicated vector-drawing terminal and a flexible protocol designed for interoperability between different communications media. That's the kind of thing AT&T loved, so they joined the effort. With CRC, AT&T developed NABTS, the North American Broadcast Teletext Specification---based on Telidon and intended for one-way broadcast over TV networks.

NABTS was complex and expensive compared to Ceefax/ORACLE/WST based systems. A review of KSL's pilot notes how the $40,000 budget for their origination system compared to the cost quoted by AT&T for an NABTS headend: as much as $2 million. While KSL's estimates of $25 for a teletext decoder had not been achieved, the prototypes were still running cheaper than NABTS clients that ran into the hundreds. Still, the graphical capabilities of NABTS were immediately impressive compared to text-only services. Besides, the extensibility of NABTS onto telephone systems, where pages could be delivered on-demand, made it capable of far larger databases.

When KSL first introduced teletext, they spoke of a scheme where a customer could call a phone number and, via DTMF menus, request an "extended" page beyond the 120 normally transmitted. They could then request that page on their teletext decoder and, at the end of the normal 120 page loop, it would be sent just for them. I'm not sure if that was ever implemented or just a concept. In any case, videotex systems could function this way natively, with pages requested and retrieved entirely by telephone modem, or using hybrid approaches.

NABTS won the support of NBC, who launched a pilot NABTS service (confusingly called NBC Teletext) in 1981 and went into full service in 1983. CBS wasn't going to be left behind, and trialed and launched NABTS (as CBS ExtraVision) at the same time. That was an ignominious end for CBS's actual teletext pilot, which quietly shut down without ever having gone into full service. ExtraVision and NBC Teletext are probably the first US interactive TV services that consumers could actually buy and use.

Teletext was not dead, though. In 1982, Cincinnati station WKRC ran test broadcasts for a WST-based teletext service called Electra. WKRC's parent company, Taft, partnered with Zenith to develop a real US-market consumer WST decoder for use with the Electra service. In 1983, the same year that ExtraVision and CBS Teletext went live, Zenith teletext decoders appeared on the shelves of Cincinnati stores. They were plug-in modules for recent Zenith televisions, meaning that customers would likely also need to buy a whole new TV to use the service... but it was the only option, and seems to have remained that way for the life of US teletext.

I believe that Taft's Electra was the first teletext service to achieve a regular broadcast license. Through the mid 1980s, Electra would expand to more television stations, reaching similar penetration to the videotex services. In 1982, KeyFax (remember KeyFax? it was the one on WFLD in Chicago) had made the pivot from teletext to videotex as well, adopting the Prestel-derived technology from GTE. In 1984, KeyFax gave up on their broadcast television component and became a telephone modem service only. Electra jumped on the now-free VBI lines of WFLD and launched in Chicago. WTBS in Atlanta carried Electra, and then in the biggest expansion of teletext, Electra appeared on SPN---a satellite network that would later become CNBC.

While major networks, and major companies like GTE and AT&T, pushed for the videotex NABTS, teletext continued to have its supporters among independent stations. Los Angeles's KTTV started its own teletext service in 1984, which combined locally-developed pages with national news syndicated from Electra. This seemed like the start of a promising model for teletext across independent stations, but it wasn't often repeated.

Oh, and KSL? at some point, uncertain to me but before 1984, they switched to NABTS.

Let's stop for a moment and recap the situation. Between about 1978 and 1984, over a dozen major US television stations launched interactive TV offerings using four major protocols that fell into two general categories. One of those categories was one-way over television while the other was two-way over telephone or one-way over television with some operators offering both. Several TV stations switched between types. The largest telcos and TV networks favored one option, but it was significantly more expensive than the other, leading smaller operators to choose differently. The hardware situation was surprisingly straightforward in that, within teletext and videotex, consumers only had one option and it was very expensive.

Oh, and that's just the technical aspects. The business arrangements could get even stranger. Teletext services were generally free, but videotex services often charged a service fee. This was universally true for videotex services offered over telephone and often, but not always, true for videotex services over cable. Were the videotex services over cable even videotex? doesn't that contradict the definition I gave earlier? is that why NBC called their videotex service teletext? And isn't videotex over telephone barely differentiated from computer-based services like CompuServe and The Source that were gaining traction at the same time?

I think this all explains the failure of interactive TV in the 1980s. As you've seen, it's not that no one tried. It's that everyone tried, and they were all tripping over each other the entire time. Even in Canada, where the government had sponsored development of the Telidon system ground-up to be a nationwide standard, the influence of US teletext services created similar confusion. For consumers, there were so many options that they didn't know what to buy, and besides, the price of the hardware was difficult to justify with the few stations that offered teletext. The fact that teletext had been hyped as the "next big thing" by newspapers since 1978, and only reached the market in 1983 as a shambled mess, surely did little for consumer confidence.

You might wonder: where was the FCC during this whole thing? In the US, we do not have a state broadcaster, but we do have state regulation of broadcast media that is really quite strict as to content and form. During the late '70s, under those first experimental licenses, the general perception seemed to be that the FCC was waiting for broadcasters to evaluate the different options before selecting a nationwide standard. Given that the FCC had previously dictated standards for television receivers, it didn't seem like that far of a stretch to think that a national-standard teletext decoder might become mandatory equipment on new televisions.

Well, it was political. The long, odd experimental period from 1978 to 1983 was basically a result of FCC indecision. The commission wasn't prepared to approve anything as a national standard, but the lack of approval meant that broadcasters weren't really allowed to use anything outside of limited experimental programs. One assumes that they were being aggressively lobbied by every side of the debate, which no doubt factored into the FCC's 1981 decision that teletext content would be unregulated, and 1982 statements from commissioners suggesting that the FCC would not, in fact, adopt any technical standards for teletext.

There is another factor wrapped up in this whole story, another tumultuous effort to deliver text over television: closed captioning. PBS introduced closed captioning in 1980, transmitting text over line 21 of the VBI for decoding by a set-top box. There are meaningful technical similarities between closed captioning and teletext, to the extent that the two became competitors. Some broadcasters that added NABTS dropped closed captioning because of incompatibility between the equipment in use. This doesn't seem to have been a real technical constraint, and was perhaps more likely cover for a cost-savings decision, but it generated considerable controversy that lead to the National Association for the Deaf organizing for closed captioning and against teletext.

The topic of closed captioning continued to haunt interactive TV. TV networks tended to view teletext or videotex as the obvious replacements for line 21 closed captioning, due to their more sophisticated technical features. Of course, the problems that limited interactive TV adoption in general, high cost and fragmentation, made it unappealing to the deaf. Closed captioning had only just barely become well-standardized in the mid-1980s and its users were not keen to give it up for another decade of frustration. While some deaf groups did support NABTS, the industry still set up a conflict between closed captioning and interactive TV that must have contributed to the FCC's cold feet.

In April of 1983, at the dawn of US broadcast teletext, the FCC voted 6-1 to allow television networks and equipment manufacturers to support any teletext or videotex protocol of their choice. At the same time, they declined to require cable networks to carry teletext content from broadcast television stations, making it more difficult for any TV network to achieve widespread adoption ³. The FCC adopted what was often termed a "market-based" solution to the question of interactive TV.

The market would not provide that solution. It had already failed.

In November of 1983, Time ended their teletext service. That's right, Time used to have a TV network and it used to have teletext; it was actually one of the first on the market. It was also the first to fall, but they had company. CBS and NBC had significantly scaled back their NABTS programs, which were failing to make any money because of the lack of hardware that could decode the service.

On the WST side of the industry, Taft reported poor adoption of Electra and Zenith reported that they had sold very few decoders, so few that they were considering ending the product line. Taft was having a hard time anyway, going through a rough reorganization in 1986 that seems to have eliminated most of the budget for Electra. Electra actually seems to have still been operational in 1992, an impressive lifespan, but it says something about the level of adoption that we have to speculate as to the time of death. Interactive TV services had so little adoption that they ended unnoticed, and by 1990, almost none remained.

Conflict with closed captioning still haunted teletext. There had been some efforts towards integrating teletext decoders into TV sets, by Zenith for example, but in 1990 line 21 closed caption decoding became mandatory. The added cost of a closed captioning decoder, and the similarity to teletext, seems to have been enough for the manufacturing industry to decide that teletext had lost the fight. Few, possibly no teletext decoders were commercially available after that date.

In Canada, Telidon met a similar fate. Most Telidon services were gone by 1986, and it seems likely that none were ever profitable. On the other hand, the government-sponsored, open-standards nature of Telidon mean that it and descendants like NABTS saw a number of enduring niche uses. Environment Canada distributed weather data via a dedicated Telidon network, and Transport Canada installed Telidon terminals in airports to distribute real-time advisories. Overall, the Telidon project is widely considered a failure, but it has had enduring impact. The original vector drawing language, the idea that had started the whole thing, came to be known as NAPLPS, the North American Presentation Layer Protocol Syntax. NAPLPS had some conceptual similarities to HTML, as Telidon's concept of interlinking did to the World Wide Web. That similarity wasn't just theoretical: Prodigy, the second largest information service after CompuServe and first to introduce a GUI, ran on NAPLPS. Prodigy is now viewed as an important precursor to the internet, but seen in a different light, it was just another videotex---but one that actually found success.

I know that there are entire branches of North American teletext and videotex and interactive TV services that I did not address in this article, and I've become confused enough in the timeline and details that I'm sure at least one thing above is outright wrong. But that kind of makes the point, doesn't it? The thing about teletext here is that we tried, we really tried, but we badly fumbled it. Even if the internet hadn't happened, I'm skeptical that interactive television efforts would have gotten anywhere without a complete fresh start. And the internet did happen, so abruptly that it nearly killed the whole concept while television carriers were still tossing it around.

Nearly killed... but not quite. Even at the beginning of the internet age, televisions were still more widespread than computers. In fact, from a TV point of view, wasn't the internet a tremendous opportunity? Internet technology and more compact computers could enable more sophisticated interactive television services at lower prices. At least, that's what a lot of people thought. I've written before about Cablesoft and it is just one small part of an entire 1990s renaissance of interactive TV. There's a few major 1980s-era services that I didn't get to here either. Stick around and you'll hear more.

You know what's sort of funny? Remember the AgVision, the first form of the TRS-80? It was built as a client for AGTEXT, a joint project of Kentucky Educational Television (who carried it on the VBI of their television network) and the Kentucky College of Agriculture. At some point, AGTEXT switched over to the line 21 closed captioning protocol and operated until 1998. It was almost the first teletext service, and it may very well have been the last.

One of the most significant single advancements in telecommunications technology was the development of microwave radio. Essentially an evolution of radar, the middle of the Second World War saw the first practical microwave telephone system. By the time Japan surrendered, AT&T had largely abandoned their plan to build an extensive nationwide network of coaxial telephone cables. Microwave relay offered greater capacity at a lower cost. When Japan and the US signed their peace treaty in 1951, it was broadcast from coast to coast over what AT&T called the "skyway": the first transcontinental telephone lead made up entirely of radio waves. The fact that live television coverage could be sent over the microwave system demonstrated its core advantage. The bandwidth of microwave links, their capacity, was truly enormous. Within the decade, a single microwave antenna could handle over 1,000 simultaneous calls.

Microwave's great capacity, its chief advantage, comes from the high frequencies and large bandwidths involved. The design of microwave-frequency radio electronics was an engineering challenge that was aggressively attacked during the war because microwave frequency's short wavelengths made them especially suitable for radar. The cavity magnetron, one of the first practical microwave transmitters, was an invention of such import that it was the UK's key contribution to a technical partnership that lead to the UK's access to US nuclear weapons research. Unlike the "peaceful atom," though, the "peaceful microwave" spread fast after the war. By the end of the 1950s, most long-distance telephone calls were carried over microwave. While coaxial long-distance carriers such as L-carrier saw continued use in especially congested areas, the supremacy of microwave for telephone communications would not fall until adoption of fiber optics in the 1980s.

The high frequency, and short wavelength, of microwave radio is a limitation as well as an advantage. Historically, "microwave" was often used to refer to radio bands above VHF, including UHF. As RF technology improved, microwave shifted higher, and microwave telephone links operated mostly between 1 and 9 GHz. These frequencies are well beyond the limits of beyond-line-of-sight propagation mechanisms, and penetrate and reflect only poorly. Microwave signals could be received over 40 or 50 miles in ideal conditions, but the two antennas needed to be within direct line of sight. Further complicating planning, microwave signals are especially vulnerable to interference due to obstacles within the "fresnel zone," the region around the direct line of sight through which most of the received RF energy passes.

Today, these problems have become relatively easy to overcome. Microwave relays, stations that receive signals and rebroadcast them further along a route, are located in positions of geographical advantage. We tend to think of mountain peaks and rocky ridges, but 1950s microwave equipment was large and required significant power and cooling, not to mention frequent attendance by a technician for inspection and adjustment. This was a tube-based technology, with analog and electromechanical control. Microwave stations ran over a thousand square feet, often of thick hardened concrete in the post-war climate and for more consistent temperature regulation, critical to keeping analog equipment on calibration. Where commercial power wasn't available they consumed a constant supply of diesel fuel. It simply wasn't practical to put microwave stations in remote locations.

In the flatter regions of the country, locating microwave stations on hills gave them appreciably better range with few downsides. This strategy often stopped at the Rocky Mountains.

In much of the American West, telephone construction had always been exceptionally difficult. Open-wire telephone leads had been installed through incredible terrain by the dedication and sacrifice of crews of men and horses. Wire strung over telephone poles proved able to handle steep inclines and rocky badlands, so long as the poles could be set---although inclement weather on the route could make calls difficult to understand. When the first transcontinental coaxial lead was installed, the route was carefully planned to follow flat valley floors whenever possible. This was an important requirement since it was installed mostly by mechanized equipment, heavy machines, which were incapable of navigating the obstacles that the old pole and wire crews had on foot.

The first installations of microwave adopted largely the same strategy. Despite the commanding views offered by mountains on both sides of the Rio Grande Valley, AT&T's microwave stations are often found on low mesas or even at the center of the valley floor. Later installations, and those in the especially mountainous states where level ground was scarce, became more ambitious. At Mt. Rose, in Nevada, an aerial tramway carried technicians up the slope to the roof of the microwave station---the only access during winter when snowpack reached high up the building's walls. Expansion in the 1960s involved increasing use of helicopters as the main access to stations, although roads still had to be graded for construction and electrical service.

These special arrangements for mountain locations were expensive, within the reach of the Long Lines department's monopoly-backed budget but difficult for anyone else, even Bell Operating Companies, to sustain. And the West---where these difficult conditions were encountered the most---also contained some of the least profitable telephone territory, areas where there was no interconnected phone service at all until government subsidy under the Rural Electrification Act. Independent telephone companies and telephone cooperatives, many of them scrappy operations that had expanded out from the manager's personal home, could scarcely afford a mountaintop fortress and a helilift operation to sustain it.

For the telephone industry's many small players, and even the more rural Bell Operating Companies, another property of microwave became critical: with a little engineering, you can bounce it off of a mirror.

James Kreitzberg was, at least as the obituary reads, something of a wunderkind. Raised in Missoula, Montana, he earned his pilots license at 15 and joined the Army Air Corps as soon as he was allowed. The Second World War came to a close shortly after, and so, he went on to the University of Washington where he studied aeronautical engineering and then went back home to Montana, taking up work as an engineer at one of the states' largest electrical utilities. His brother, George, had taken a similar path: a stint in the Marine Corps and an aeronautical engineering degree from Oklahoma. While James worked at Montana Power in Butte, George moved to Salem, Oregon, where he started an aviation company that supplemented their cropdusting revenue by modifying Army-surplus aircraft for other uses.

Montana Power operated hydroelectric dams, coal mines, and power plants, a portfolio of facilities across a sparse and mountainous state that must have made communications a difficult problem. During the 1950s, James was involved in an effort to build a new private telephone system connecting the utility's facilities. It required negotiating some type of obstacle, perhaps a mountain pass. James proposed an idea: a reflector.

Because the wavelength of microwaves are so short, say 10cm, it's practical to build a flat metallic panel that spans multiple wavelengths. Such a panel will function like a reflector or mirror, redirecting microwave energy at an angle proportional to the angle on which it arrived. Much like you can redirect a laser using mirrors, you can also redirect a microwave signal. Some early commenters referred to this technique as a "radio mirror," but by the 1950s the use of "active" microwave repeaters with receivers and transmitters had become well established, so by comparison reflectors came to be known as "passive repeaters."

James believed a passive repeater to be a practical solution, but Montana Power lacked the expertise to build one. For a passive repeater to work efficiently, its surface must be very flat and regular, even under varying temperature. Wind loading had to be accounted for, and the face sufficiently rigid to not flex under the wind. Of course, with his education in aeronautics, James knew that similar problems were encountered in aircraft: the need for lightweight metal structures with surfaces that kept an engineered shape. Wasn't he fortunate, then, that his brother owned a shop that repaired and modified aircraft.

I know very little about the original Montana Power installation, which is unfortunate, as it may very well be the first passive microwave repeater ever put into service. What I do know is that in the fall of 1955, James called his brother George and asked if his company, Kreitzberg Aviation, could fabricate a passive repeater for Montana Power. George, he later recounted, said that "I can build anything you can draw." The repeater was made in a hangar on the side of Salem's McNary Field, erected by the flightline as a test, and then shipped in parts to Montana for reassembly in the field. It worked. It worked so well, in fact, that as word of Montana Power's new telephone system spread, other utilities wrote to inquire about obtaining passive repeaters for their own telephone systems.

In 1956, James Kreitzberg moved to Salem and the two brothers formed the Microflect Company. From the sidelines of McNary Field, Microflect built aluminum "billboards" that can still be found on mountain passes and forested slopes throughout the western United States, and in many other parts of the world where mountainous terrain, adverse weather, and limited utilities made the construction of active repeaters impractical.

Passive repeaters can be used in two basic configurations, defined by the angle at which the signal is reflected. In the first case, the reflection angle is around 90 degrees (the closer to this ideal angle, of course, the more efficiently the repeater performs). This situation is often encountered when there is an obstacle that the microwave path needs to "maneuver" around. For example, a ridge or even a large structure like a building in between two sites. In the second case, the microwave signal must travel in something closer to a straight line---over a mountain pass between two towns, for example. When the reflection angle is greater than 135 degrees, the use of a single passive repeater becomes inefficient or impossible, so Microflect recommends the use of two. Arranged like a dogleg or periscope, the two repeaters reflect the signal to the side and then onward in the intended direction.

Microflect published an excellent engineering manual with many examples of passive repeater installations along with the signal calculations. You might think that passive repeaters would be so inefficient as to be impractical, especially when more than one was required, but this is surprisingly untrue. Flat aluminum panels are almost completely efficient reflectors of microwave, and somewhat counterintuitively, passive repeaters can even provide gain.

In an active repeater, it's easy to see how gain is achieved: power is added. A receiver picks up a signal, and then a powered transmitter retransmits it, stronger than it was before. But passive repeaters require no power at all, one of their key advantages. How do they pull off this feat? The design manual explains with an ITU definition of gain that only an engineer could love, but in an article for "Electronics World," Microflect field engineer Ray Thrower provided a more intuitive explanation.

A passive repeater, he writes, functions essentially identically to a parabolic antenna, or a telescope:

Quite probably the difficulty many people have in understanding how the passive repeater, a flat surface, can have gain relates back to the common misconception about parabolic antennas. It is commonly believed that it is the focusing characteristics of the parabolic antenna that gives it its gain. Therefore, goes the faulty conclusion, how can the passive repeater have gain? The truth is, it isn't focusing that gives a parabola its gain; it is its larger projected aperture. The focusing is a convenient means of transition from a large aperture (the dish) to a small aperture (the feed device). And since it is projected aperture that provides gain, rather than focusing, the passive repeater with its larger aperture will provide high gain that can be calculated and measured reliably. A check of the method of determining antenna gain in any antenna engineering handbook will show that focusing does not enter into the basic gain calculation.

We can also think of it this way: the beam of energy emitted by a microwave antenna expands in an arc as it travels, dissipating the "density" of the energy such that a dish antenna of the same size will receive a weaker and weaker signal as it moves further away (this is the major component of path loss, the "dilution" of the energy over space). A passive repeater employs a reflecting surface which is quite large, larger than practical antennas, and so it "collects" a large cross section of that energy for reemission.

Projected aperture is the effective "window" of energy seen by the antenna at the active terminal as it views the passive repeater. The passive repeater also sees the antenna as a "window" of energy. If the two are far enough away from one another, they will appear to each other as essentially point sources.

In practice, a passive repeater functions a bit like an active repeater that collects a signal with a large antenna and then reemits it with a smaller directional antenna. To be quite honest, I still find it a bit challenging to intuit this effect, but the mathematics bear it out as well. Interestingly, the effect only occurs when the passive repeater is far enough from either terminal so as to be usefully approximated as a point source. Microflect refers to this as the far field condition. When the passive repeater is very close to one of the active sites, within the near field, it is more effective to consider the passive repeater as part of the transmitting antenna itself, and disregard it for path loss calculations. This dichotomy between far field and near field behavior is actually quite common in antenna engineering (where an "antenna" is often multiple radiating and nonradiating elements within the near field of each other), but it's yet another of the things that gives antenna design the feeling of a dark art.

One of the most striking things about passive repeaters is their size. As a passive repeater becomes larger, it reflects a larger cross section of the RF energy and thus provides more gain. Much like with dish or horn antennas, the size of a passive repeater can be traded off with transmitter power (and the size of other antennas involved) to design an economical solution. Microflect offered as standard sizes ranging from 8'x10' (gain at around 6.175GHz: 90.95 dB) to 40'x60' (120.48dB, after a "rough estimate" reduction of 1dB due to interference effects possible from such a short wavelength reflecting off of such a large panel as to invoke multipath effects).

By comparison, a typical active microwave repeater site might provide a gain of around 140dB---and we must bear in mind that dB is a logarithmic unit, so the difference between 121 and 140 is bigger than it sounds. Still, there's a reason that logarithms are used when discussing radio paths... in practice, it is orders of magnitude that make the difference in reliable reception. The reduction in gain from an active repeater to a passive repeater can be made up for with higher-gain terminal antennas and more powerful transmitters. Given that the terminal sites are often at far more convenient locations than the passive repeater, that tradeoff can be well worth it.

Keep in mind that, as Microflect emphasizes, passive repeaters require no power and very little ("virtually no") maintenance. Microflect passive repeaters were manufactured in sections that bolted together in the field, and the support structures provided for fine adjustment of the panel alignment after mounting. These features made it possible to install passive repeaters by helicopter onto simple site-built foundations, and many are found on mountainsides that are difficult to reach even on foot. Even in less difficult locations, these advantages made passive repeaters less expensive to install and operate than active repeaters. Even when the repeater side was readily accessible, passives were often selected simply for cost savings.

Let's consider some examples of passive repeater installations. Microflect was born of the power industry, and electrical generators and utilities remained one of their best customers. Even today, you can find passive repeaters at many hydroelectric dams. There is a practical need to communicate by telephone between a dispatch center (often at the utility's city headquarters) and the operators in the dam's powerhouse, but the powerhouse is at the base of the dam, often in a canyon where microwave signals are completely blocked. A passive repeater set on the canyon rim, at an angle downwards, solves the problem by redirecting the signal from horizontal to vertical. Such an installation can be seen, for example, at the Hoover Dam. In some sense, these passive repeaters "relocate" the radio equipment from the canyon rim (where the desirable signal path is located) to a more convenient location with the other powerhouse equipment. Because of the short distance from the powerhouse to the repeater, these passives were usually small.

This idea can be extended to relocating en-route repeaters to a more serviceable site. In Glacier National Park, Mountain States Telephone and Telegraph installed a telephone system to serve various small towns and National Park Service sites. Glacier is incredibly mountainous, with only narrow valleys and passes. The only points with long sight ranges tend to be very inaccessible. Mt. Furlong provided ideal line of sight to East Glacier and Essex along highway 2, but it would have been extremely challenging to install and maintain a microwave site on the steep peak. Instead, two passive repeaters were installed near the mountaintop, redirecting the signals from those two destinations to an active repeater installed downslope near the highway and railroad.

This example raises another advantage of passive repeaters: their reduced environmental impact, something that Microflect emphasized as the environmental movement of the 1970s made agencies like the Forest Service (which controlled many of the most appealing mountaintop radio sites) less willing to grant permits that would lead to extensive environmental disruption. Construction by helicopter and the lack of a need for power meant that passive repeaters could be installed without extensive clearing of trees for roads and power line rights of way. They eliminated the persistent problem of leakage from standby generator fuel tanks. Despite their large size, passive repeaters could be camouflaged. Many in national forests were painted green to make them less conspicuous. And while they did have a large surface area, Microflect argued that since they could be installed on slopes rather than requiring a large leveled area, passive repeaters would often fall below the ridge or treeline behind them. This made them less visually conspicuous than a traditional active repeater site that would require a tower. Indeed, passive repeaters are only rarely found on towers, with most elevated off the ground only far enough for the bottom edge to be free of undergrowth and snow.

Other passive repeater installations were less a result of exceptionally difficult terrain and more a simple cost optimization. In rural Nevada, Nevada Bell and a dozen independents and coops faced the challenge of connecting small towns with ridges between them. The need for an active repeater at the top of each ridge, even for short routes, made these rural lines excessively expensive. Instead, such towns were linked with dual passive repeaters on the ridge in a "straight through" configuration, allowing microwave antennas at the towns' existing telephone exchange buildings to reach each other. This was the case with the installation I photographed above Pioche. I have been frustratingly unable to confirm the original use of these repeaters, but from context they were likely installed by the Lincoln County Telephone System to link their "hub" microwave site at Mt. Wilson (with direct sight to several towns) to their site near Caliente.

The Microflect manual describes, as an example, a very similar installation connecting Elko to Carlin. Two 20'x32' passive repeaters on a ridge between the two (unfortunately since demolished) provided a direct connection between the two telephone exchanges.

As an example of a typical use, it might be interesting to look at the manual's calculations for this route. From Elko to the repeaters is 13.73 miles, the repeaters are close enough to each other as to be in near field (and so considered as a single antenna system), and from the repeaters to Carlin is 6.71 miles. The first repeater reflects the signal at a 68 degree angle, then the second reflects it back at a 45 degree angle, for a net change in direction of 23 degrees---a mostly straight route. The transmitter produces 33.0 dBm, both antennas provide a 34.5 dB gain, and the passive repeater assembly provides 88 dB gain (this calculated basically by consulting a table in the manual). That means there is 190 dB of gain in the total system. The 6.71 and 13.73 mile paths add up to 244 dB of free space path loss, and Microflect throws in a few more dB of loss to account for connectors and cables and the less than ideal performance of the double passive repeater. The net result is a received signal of -58 dBm, which is plenty acceptable for a 72-channel voice carrier system. This is all done at a significantly lower price than the construction of a full radio site on the ridge [1].

The combination of relocating radio equipment to a more convenient location and simply saving money leads to one of the iconic applications of passive repeaters, the "periscope" or "flyswatter" antenna. Microwave antennas of the 1960s were still quite large and heavy, and most were pressurized. You needed a sturdy tower to support one, and then a way to get up the tower for regular maintenance. This lead to most AT&T microwave sites using short, squat square towers, often with surprisingly convenient staircases to access the antenna decks. In areas where a very tall tower was needed, it might just not be practical to build one strong enough. You could often dodge the problem by putting the site up a hill, but that wasn't always possible, and besides, good hilltop sites that weren't already taken became harder to find.

When Western Union built out their microwave network, they widely adopted the flyswatter antenna as an optimization. Here's how it works: the actual microwave antenna is installed directly on the roof of the equipment building facing up. Only short waveguides are needed, weight isn't an issue, and technicians can conveniently service the antenna without even fall protection. Then, at the top of a tall guyed lattice tower similar to an AM mast, a passive repeater is installed at a 45 degree angle to the ground, redirecting the signal from the rooftop antenna to the horizontal. The passive repeater is much lighter than the antenna, allowing for a thinner tower, and will rarely if ever need service. Western Union often employed two side-by-side lattice towers with a "crossbar" between them at the top for convenient mounting of reflectors each direction, and similar towers were used in some other installations such as the FAA's radar data links. Some of these towers are still in use, although generally with modern lightweight drum antennas replacing the reflectors.

Passive microwave repeaters experienced their peak popularity during the 1960s and 1970s, as the technology became mature and communications infrastructure proliferated. Microflect manufactured thousands of units from their new, larger warehouse, across the street from their old hangar on McNary Field. Microflect's customer list grew to just about every entity in the Bell System, from Long Lines to Western Electric to nearly all of the BOCs. The list includes GTE, dozens of smaller independent telephone companies, most of the nation's major railroads, electrical utilities from the original Montana Power to the Tennessee Valley Authority. Microflect repeaters were used by ITT Arctic Services and RCA Alascom in the far north, and overseas by oil companies and telecoms on islands and in mountainous northern Europe.

In Hawaii, a single passive repeater dodged a mountain to connect Lanai City telephones to the Hawaii Telephone Company network at Tantalus on Oahu---nearly 70 miles in one jump. In Nevada, six passive repeaters joined two active sites to connect six substations to the Sierra Pacific Power Company's control center in Reno. Jamaica's first high-capacity telephone network involved 11 passive repeaters, one as large as 40'x60'.

The Rocky Mountains are still dotted with passive repeaters, structures that are sometimes hard to spot but seem to loom over the forest once noticed. In Seligman, AZ, a sun-faded passive repeater looks over the cemetery. BC Telephone installed passive repeaters to phase out active sites that were inaccessible for maintenance during the winter. Passive repeaters were, it turns out, quite common---and yet they are little known today.

First, it cannot be ignored that passive repeaters are most common in areas where communications infrastructure was built post-1960 through difficult terrain. In North America, this means mostly the West [2], far away from the Eastern cities where we think of telephone history being concentrated. Second, the days of passive repeaters were relatively short. After widespread adoption in the '60s, fiber optics began to cut into microwave networks during the '80s and rendered microwave long-distance links largely obsolete by the late '90s. Considerable improvements in cable-laying equipment, not to mention the lighter and more durable cables, made fiber optics easier to install in difficult terrain than coaxial had ever been.

Besides, during the 1990s, more widespread electrical infrastructure, miniaturization of radio equipment, and practical photovoltaic solar systems all combined to make active repeaters easier to install. Today, active repeater systems installed by helicopter with independent power supplies are not that unusual, supporting cellular service in the Mojave Desert, for example. Most passive repeaters have been obsoleted by changes in communications networks and technologies. Satellite communications offer an even more cost effective option for the most difficult installations, and there really aren't that many places left that a small active microwave site can't be installed.

Moreover, little has been done to preserve the history of passive repeaters. In the wake of the 2015 Wired article on the Long Lines network, considerable enthusiasm has been directed towards former AT&T microwave stations, having been mostly preserved by their haphazard transfer to companies like American Tower. Passive repeaters, lacking even the minimal commercial potential of old AT&T sites, were mostly abandoned in place. Often being found in national forests and other resource management areas, many have been demolished for restoration. In 2019, a historic resources report was written on the Bonneville Power Administration's extensive microwave network. It was prepared to address the responsibility that federal agencies have for historical preservation under the National Historic Preservation Act and National Environmental Policy Act, policies intended to ensure that at least the government takes measures to preserve history before demolishing artifacts. The report reads: "Due to their limited features, passive repeaters are not considered historic resources, and are not evaluated as part of this study."

In 1995, Valmont Industries acquired Microflect. Valmont is known mostly for their agricultural products, including center-pivot irrigation systems, but they had expanded their agricultural windmill business into a general infrastructure division that manufactured radio masts and communication towers. For a time, Valmont continued to manufacture passive repeaters as Valmont Microflect, but business seems to have dried up.

Today, Valmont Structures manufactures modular telecom towers from their facility across the street from McNary Field in Salem, Oregon. A Salem local, descended from early Microflect employees, once shared a set of photos on Facebook: a beat-up hangar with a sign reading "Aircraft Repair Center," and in front of it, stacks of aluminum panel sections. Microflect workers erecting a passive repeater in front of a Douglas A-26. Rows of reflector sections beside a Shell aviation fuel station. George Kreitzberg died in 2004, James in 2017. As of 2025, Valmont no longer manufactures passive repeaters.

Postscript

If you are interested in the history of passive repeaters, there are a few useful tips I can give you.

• Nearly all passive repeaters in North America were built by Microflect, so they have a very consistent design. Locals sometimes confuse passive repeaters with old billboards or even drive-in theater screens, the clearest way to differentiate them is that passive repeaters have a face made up of aluminum modules with deep sidewalls for rigidity and flatness. Take a look at the Microflect manual for many photos.
• Because passive repeaters are passive, they do not require a radio license proper. However, for site-based microwave licenses, the FCC does require that passive repeaters be included in paths (i.e. a license will be for an active site but with a passive repeater as the location at the other end of the path). These "other location" entries often have names ending in "PR" and their type set to "Passive Repeater."
• I don't have any straight answer on whether or not any passive repeaters are still in use. It has likely become very rare but there are probably still examples. Two sources suggest that Rachel, NV still relies on a passive repeater for telephone and DSL, but I'm pretty sure this hasn't been true for some years (I can't find any license covering it). I have so far found one active site-based microwave license covering a passive repeater, but it serves a mine that has been closed since the 1980s and I suspect the license has only been renewed due to a second, different path that does not involve a passive. A reader let me know that Industry Canada has some 80 passive repeaters licensed, but I do not know how many (if any) are in active use.
• For the sake of simplicity I have used "passive repeater" here to refer to microwave reflectors only, but the same term is also used for arrangements of two antennas connected back-to-back. These are much more common in VHF/UHF than in the microwave, although microwave passive repeaters of two parabolic antennas have been used in limited cases.
• Microflect dominated the US and European market for passive repeaters, but the technology was also used in the Soviet Union, seemingly around the same time. I do not know where it was developed first, or whether it was a case of independent invention. The Soviet examples I have seen use a noticeably different support structure from Microflect, and seem to have been engineered for helicopter hoisting in complete form rather than in parts. Passive repeaters proved very useful in the arctic and so I would assume that the Soviet Union installed quite a few.
• Most passive repeaters were installed by "classic communications organizations," meaning telephone companies, power utilities, and railroads---industries that used long-distance communications systems since the turn of the century. I have heard of one passive repeater installed by a television studio for an STL link, and there might be others, but I don't think it was common.

[1] If you find these dB gain/loss calculations confusing, you are not alone. It is deceptively simple in a way that was hard for me to learn, and perhaps I will devote an article to it one day.

[2] Although not exclusively, with installations in places like Vermont and Newfoundland where similar constraints applied.

Alcatraz first operated as a prison in 1859, when the military fort first held convicted soldiers. The prison technology of the time was simple, consisting of little more than a basement room with a trap-door entrance. Only small numbers of prisoners were held in this period, but it established Alcatraz as a center of incarceration. Later, the Civil War triggered construction of a "political prison," a term with fewer negative connotations at the time, for confederate sympathizers.

This prison was more purpose-built (although actually a modification of an existing shop), but it was small and not designed for an especially high security level. It presaged, though, a much larger construction project to come.

Alcatraz had several properties that made it an attractive prison. First, it had seen heavy military construction as a Civil War defensive facility, but just decades later improvements in artillery made its fortifications obsolete. That left Alcatraz surplus property, a complete military installation available for new use. Second, Alcatraz was formidable. The small island was made up of steep rock walls, and it was miles from shore in a bay known for its strong currents. Escape, even for prisoners who had seized control of the island, would be exceptionally difficult.

These advantages were also limitations. Alcatraz was isolated and difficult to support, requiring a substantial roster of military personnel to ferry supplies back and forth. There were no connections to the mainland, requiring on-site power and water plants. Corrosive sea spray, sent over the island by the Bay's strong winds, lay perpetual siege on the island. Buildings needed constant maintenance, rust covered everything. Alcatraz was not just a famous prison, it was a particularly complicated one.

In 1909, Alcatraz lost its previous defensive role and pivoted entirely to military prison. The Citadel, a hardened barracks building dating to the original fortifications, was partially demolished. On top of it, a new cellblock was built. This was a purpose-built prison, designed to house several hundred inmates under high security conditions.

Unfortunately, few records seem to survive from the construction and operation of the cellblock as a disciplinary barracks. At some point, a manual telephone exchange was installed to provide service between buildings on the island. I only really know that because it was recorded as being removed later on. Communications to and from Alcatraz were a challenge. Radio and even light signals were used to convey messages between the island and other military installations on the bay. There was a constant struggle to maintain cables.

Early efforts to lay cables in the bay were less about communications and more about triggering. Starting in 1883, the Army Corps of Engineers began the installation of "torpedoes" in the San Francisco bay. These were different from what we think of as torpedoes today, they were essentially remotely-operated mines. Each device floated in the water by its own buoyancy, anchored to the bottom by a cable that then ran to shore. An electrical signal sent down the cable detonated the torpedo. The system was intended primarily to protect the bay from submarines, a new threat that often required technically complex defenses.

Submarines are, of course, difficult to spot. To make the torpedoes effective, the Army had to devise a targeting system. Observation posts on each side of the Golden Gate made sightings of possible submarines and reported them to a control post, where they were plotted on the map. With a threat confirmed, the control post would begin to detonate nearby torpedoes. A second set of observation posts, and a second line of torpedoes, were located further into the bay to address any submarines that made it through the first barrage.

By 1891, there were three such control points in total: Fort Mason, Angel Island, and Yerba Buena. The rather florid San Francisco Examiner of the day described the control point at Fort Mason, a "chamber of death and destruction" in a tunnel twenty feet underground. The Army "death-dealers" that manned the plotting table in that bunker had access to a board that "greatly resemble[d] the switch board in the great operating rooms of the telephone companies." By cords and buttons, they could select chains of mines and send the signal to fire.

NPS historians found that a torpedo control point had been planned at Alcatraz, and one of the fortifications modified to accommodate it, but never seems to have been used. The 1891 article gives a hint of the reason, noting that the line from Alcatraz to Fort Mason was "favorable for a line of torpedoes" but that currents were so strong that it was difficult to keep them anchored. Perhaps this problem was discovered after construction was already underway.

Somewhere around 1887-1888, the Army Signal Corps had joined the cable-laying fray. A telegraph cable was constructed from the Presidio to Alcatraz, and provided good service except for the many times that it was drug up by anchors and severed. This was a tremendous problem: in 1898, Gen. A. W. Greely of the Signal Corps called San Francisco the "worst bay in the country" for cable laying and said that no cable across the Golden Gate had lasted more than three years. The General attributed the problem mainly to the heavy shipping traffic, but I suspect that the notorious currents must have been a factor in just how many anchors were dragged through cables [1].

In 1889, a brand new Army telegraph cable was announced, one that would run from Alcatraz to Angel Island, and then from Angel Island to Marin County. An existing commercial cable crossed the Golden Gate, providing a connection all the way to the Presidio.

The many failures of Alcatraz cables makes it difficult to keep track. For example, a cable from Fort Mason to Alcatraz Island was apparently laid in 1891---but a few years later, it was lamented that Alcatraz's only cable connection to Fort Mason was indirect, via the 1889 Angel Island cable. Presumably the 1891 cable was damaged at some point and not replaced, but that event doesn't seem to have made the papers (or at least my search results!).

In 1900, a Signal Corps officer on Angel Island made a routine check of the cable to Alcatraz, finding it in good working order---but noticing that a "four masted schooner... in direct line with the cable" seemed to be in trouble just off the island and was being assisted by a tug. That evening, the officer returned to the cable landing box to find the ship gone... along with the cable. A French ship, "Lamoriciere," had drifted from anchor overnight. A Signal Corps sergeant, apparently having spoken with harbor officials, reported that the ship would have run completely aground had the anchor not caught the Alcatraz cable and pulled it taught. Of course the efforts of the tug to free Lamoriciere seems to have freed a little more than intended, and the cable was broken away from its landing. "Its end has been carried into the bay and probably quite a distance from land," the Signal Corps reported.

This ongoing struggle, of laying new cables to Alcatraz and then seeing them dragged away a few years later, has dogged the island basically to the modern day---when we have finally just given up. Today, as during many points in its history, Alcatraz must generate its own power and communicate with the mainland via radio.

When the Bureau of Prisons took control of Alcatraz in 1933, they installed entirely new radio systems. A marine AM radio was used to reach the Coast Guard, their main point of contact in any emergency. Another radio was used to contact "Alcatraz Landing" from which BOP ferries sailed, and over the years several radios were installed to permit direct communications with military installations and police departments around the Bay Area.

At some point, equipment was made available to connect telephone calls to the island. I'm not sure if this was manual patching by BOP or Coast Guard radio operators, or if a contract was made with PT&T to provide telephone service by radio. Such an arrangement seems to have been in place by 1937, when an unexplained distress call from the island made the warden impossible to contact (by the press or Bureau of Prisons) because "all lines [were] tied up."

Unfortunately I have not been able to find much on the radiotelephone arrangements. The BOP, no doubt concerned about security, did not follow the Army's habit of announcing new construction projects to the press. Fortunately, the BOP-era history of Alcatraz is much better covered by modern NPS documentation than the Army era (presumably because the more recent closure of the BOP prison meant that much of the original documentation was archived). Unfortunately, the NPS reports are mostly concerned with the history of the structures on the island and do not pay much attention to outside communications or the infrastructure that supported it.

Internal arrangements on the island almost completely changed when the BOP took over. The Army had left Alcatraz in a degree of disrepair (discussions about closing it having started by at least 1913), and besides, the BOP intended to provide a much higher level of security than the Army had. Extensive renovations were made of the main cellblock and many supporting buildings from 1933 to about 1939.

The 1930s had seen a great deal of innovation in technical security. Technologies like electrostatic and microwave motion sensors were available in early forms. On Alcatraz, though, the island was small and buildings tightly spaced. The prison staff, and in some cases their families, would be housed on the island just a stones throw from the cellblock. That meant there would be quite a few people moving around exterior to the prison, ruling out motion sensors as a means of escape detection. Exterior security would instead be provided by guard and dog patrols.

There was still some cutting-edge technical security when Alcatraz opened, including early metal detectors. At first, the BOP contracted the Teletouch Corporation of New York City. Teletouch, a manufactured burglar alarms and other electronic devices, was owned by or at least affiliated with famed electromagnetics inventor and Soviet spy Leon Theremin. Besides the instrument we remember him for today, Theremin had invented a number of devices for security applications, and the metal detectors were probably of his design. In practice, the Teletouch machines proved unsatisfactory. They were later replaced with machines made by Forewarn. I believe the metal detector on display today is one of the Forewarn products, although the NPS documents are a little unclear on this.

Sensitive common areas like the mess hall, kitchen, and sallyport wre fitted with electrically-activated teargas canisters. Originally, the mess hall teargas was controlled by a set of toggle switches in a corner gun gallery, while the sallyport teargas was controlled from the armory. While the teargas system was never used, it was probably the most radical of Alcatraz's technical security measures. As more electronic systems were installed, the armory, with its hardened vault entrance and gun issue window, served as a de facto control center for Alcatraz's initial security systems.

The Army's small manual telephone switchboard was considered unsuitable for the prison's use. The telephone system provided communication between the guards, making it a critical part of the overall security measures, and the BOP specified that all equipment and cabling needed to be better secured from any access by prisoners. Modifications to the cellblock building's entrance created a new room, just to the side of the sallyport, that housed a 100-line automatic exchange. Automatic Electric telephones that appear throughout historic photos of the prison would suggest that this exchange had been built by AE.

Besides providing dial service between prison offices and the many other structures on the island, the exchange was equipped with a conference circuit that included annunciator panels in each of the prison's main offices. Assuming this was the type provided by Automatic Electric, it provided an emergency communications system in which the guard telephones could ring all of the office and guard phones simultaneously, even interrupting calls already in progress. Annunciator panels in the armory and offices showed which phone had started the emergency conference, and which phones had picked up. From the armory, a siren on the building roof could be sounded to alert the entire island to any attempted escape.

Some locations, including the armory and the warden's office, were also fitted with fire annunciators. I am less clear on this system. Fire circuits similar to the previously described conference circuit (and sometimes called "crash alarms" after their use on airfields) were an optional feature on telephone exchanges of the time. Crash alarms were usually activated by dedicated "hotline" phones, and mentions of "emergency phones" in various prison locations support that this system worked the same way. Indeed, 1950s and 60s photos show a red phone alongside other telephones in several prison locations. The fire annunciator panels probably would have indicated which of the emergency phones had been lifted to initiate the alarm.

One of the most fascinating parts of Alcatraz, to a person like me, is the prison doors. Prison doors have a long history, one that is interrelated with but largely distinct from other forms of physical security. Take a look, for example, at the keys used in prisons. Prisons of the era, and even many today, rely on lever locks manufactured by specialty companies like Folger Adams and Sargent and Greenleaf. These locks are prized for their durability, and that extends to the keys, huge brass plates that could hold up to daily wear well beyond most locks.

At Alcatraz, the first warden adopted a "sterile area" model in which areas accessible to prisoners should be kept as clear as possible of dangerous items like guns and keys. Guards on the cellblock carried no keys, and cell doors lacked traditional locks. Instead, the cell doors were operated by a central mechanical system designed by Stewart Iron Works.

To let prisoners out of cells in the morning, a guard in the elevated gun gallery passed keys to a cellblock guard in a bucket or on a string. The guard unlocked the cabinet of a cell row's control system, revealing a set of large levers. The design is quite ingenious: by purely mechanical means, the guard could select individual cells or the entire row to be unlocked, and then by throwing the largest lever the guard could pull the cell doors open---after returning the necessary key to the gun gallery above. This 1934 system represents a major innovation in centralized access control, designed specifically for Alcatraz.

Stewart Iron Works is still in business, although not building prison doors. Some years ago, the company assisted NPS's work to restore the locking system to its original function. The present-day CEO provided replicas of the original Stewart logo plate for the restored locking cabinets. Interviewing him about the restoration work, the San Francisco Chronicle wrote that "Alcatraz, he believes, is part of the American experience."

The Stewart mechanical system seems to have remained in use on the B and C blocks until the prison closed, but the D block was either originally fitted, or later upgraded, with electrically locked cell doors. These were controlled from a set of switches in the gun gallery.

In 1960, the BOP launched another wave of renovations on Alcatraz, mostly to modernize its access and security arrangements to modern standards. The telephone exchange was moved away from the sallyport to an upper floor of the administration building, freeing up its original space for a new control center. This is the modern sallyport control area that visitors look into through the ballistic windows; the old service windows and viewports into the armory anteroom that had been the de facto control center are now removed.

This control center is more typical of what you will see in modern prisons. Through large windows, guards observed the sallyport and visitor areas and controlled the electrically operated main gates. An electrical interlock prevented opening the full path from the cellblock to the outside, creating a mantrap in the visitor area through which the guards in the control room could identify everyone entering and leaving.

Photos from the 1960 control room, and other parts of the prison around the same time, clearly show consoles for a Western Electric 507B PBX. The 507B is really a manual exchange, although it used keys rather than the more traditional plugboard for a more modern look. It dates back to about 1929---so I assume the 507B had been installed well before the 1960 renovation, and its appearance then is just a bias of more and better photos available from the prison's later days.

Fortunately, the NPS Historic Furnishings Report for the cellblock building includes a complete copy of a 1960s memo describing the layout and requirements for the control center. We're fortunate to get such a detailed listing of the equipment:

• Four phones (these are Automatic Electric instruments, based on the photo). One is a fire reporting phone (presumably on the exchange's "crash alarm" circuit), one is the watch call reporting phone (detailed in a moment), a regular outgoing call telephone, and an "executive right of way" phone that I assume will disconnect other calls from the outgoing trunks.
• The 507B PBX switchboard
• An intercom for communication with each of the guard towers
• Controls for five electrically operated doors
• Intercoms to each of the electrically operated doors (many of these are right outside of the control center, but the glass is very thick and you would not otherwise be able to converse)
• An "annunciator panel for the interior telephone system" which presumably combines the conference circuit, fire circuit, and watch call annunciators.
• An intercom to the visitor registration area
• A "paging intercom for group control purposes." I don't really know what that is, possibly it is for the public address speakers installed in many parts of the cellblock.
• Monitor speaker for the inmate radio system. This presumably allowed the control center to check the operation of the two-channel wired radio system installed in the cells.
• The "watch call answering device," discussed later.
• An indicator panel that shows any open doors in the D cell block (which is the higher security unit and the only one equipped with electrically locking cell doors).
• Two-way radio remote console
• Tear gas controls

Many of these are things we are already familiar with, but the watch call telephone system deserves some more discussion. It was clearly present back in the 1930s, but it wasn't clear to me what it actually did. Fortunately this memo gives some details on the operation.

Guards calling in to report their watch call extension 3331. This connects to the watch call answering device in the control center, which when enabled, automatically answers the call during the first ring. The answering device then allows a guard anywhere in the control center to converse with the caller via a loudspeaker and microphone. So, the watch call system is essentially just a speaker phone. This approach is probably a holdover from the 1930s system (older documents mention a watch call phone as well), and that would have been the early days for speakerphones, making it a somewhat specialized device. Clearly it made these routine watch calls a lot more convenient for the control center, especially since the guard there didn't even have to do anything to answer.

It might be useful to mention why this kind of system was used: I have never found any mention of two-way radios used on Alcatraz, and that's not surprising. Portable two-way radios were a nascent technology even in the 1960s---the handheld radio had basically been invented for the Second World War, and it took years for them to come down in size and price. If Alcatraz ever did issue radios to guards, it probably would have been in the last decade of operation. Instead, telephones were provided at enough places in the facility that guards could report their watch tour and any important events by finding a phone and calling the control center.

Guards were probably required to report their location at various points as they patrolled, so the control center would receive quite a few calls that were just a guard saying where they were---to be written down in a log by a control room guard, who no doubt appreciated not having to walk to a phone to hear these reports. This provided both the functions of a "guard tour" system, ensuring that guards were actually performing their rounds, and improved the safety of guards by making it likely that the control center would notice fairly promptly that they had stopped reporting in.

Alcatraz closed as a BOP prison in 1963, and after a surprising number of twists and turns ranging from plans to develop a shopping center to occupation by the Indians of All Tribes, Alcatraz opened to tourists. Most technology past this point might not be considered "historic," having been installed by NPS for operational purposes. I can't help but mention, though, that there were more attempts at a cable. For the NPS, operating the power plant at Alcatraz was a significant expense that they would much rather save.

The idea of a buried power cable isn't new. I have seen references, although no solid documentation, that the BOP laid a power cable in 1934. They built a new power plant in 1939 and operated it for the rest of the life of the prison, so either that cable failed and was never replaced, or it never existed at all...

I should take a moment here to mention that LLM-generated "AI slop" has become a pervasive and unavoidable problem around any "hot SEO topic" like tourism. Unfortunately the history of tourist sites like Alcatraz has become more and more difficult to learn as websites with well-researched history are displaced in search results by SEO spam---articles that often contains confident but unsourced and often incorrect information. This has always been a problem but it has increased by orders of magnitude over the last couple of years, and it seems that the LLM-generated articles are more likely to contain details that are outright made up than the older human-generated kind. It's really depressing. That's basically all I have to say about it.

It seems that a power cable was installed to Alcatraz sometime in the 1960s but failed by about 1971. I'm a little skeptical of that because that was the era in which it was surplus GSA property, making such a large investment an odd choice, so maybe the 1980s article with that detail is wrong or confusing power with one of the several telephone cables that seem to have been laid (and failed) during BOP operations). In any case, in late 1980 or early 1981, Paul F. Pugh and Associates of Oakland designed a novel type of underwater power cable for the NPS. It was expected to provide power to Alcatraz at much reduced cost compared to more traditional underwater power cable technologies. It never even made it to day 1: after the cable was laid, but before commissioning, some failure caused a large span of it to float to the surface. The cable was evidently not repairable, and it was pulled back to shore.

'I don't know where we go from here,' William J. Whalen, superintendent of the Golden Gate National Recreation Area, said after the broken cable was hauled in.

We do know now: where the NPS went from there was decades of operating two diesel generators on the island, until a 2017 DoE-sponsored project that installed solar panels on the cellblock building roof. The panels were intentionally installed such that they are not visible anywhere from the ground, preserving the historic integrity of the site. In aerial photos, though, they give Alcatraz a curiously modern look. The DoE calls the project, which incorporates battery storage and backup diesel generators, as "one of the largest microgrids in the United States." That is an interesting framing, one that emphasizes the modern valance of "microgrid," since Alcatraz had been a self-sufficient electrical system since the island's first electric lights. But what's old is, apparently, new again.

---

I originally wrote much of this as part of a larger travelogue on my most recent trip to Alcatraz, which was coincidentally the same day as a visit by Pam Bondi and Doug Burgum to "survey" the prison for potential reopening. That piece became long and unwieldy, so I am breaking it up into more focused articles---this one on the technical history, a travelogue about the experience of visiting the island in this political context and its history as a symbol of justice and retribution, and probably a third piece on the way that the NPS interprets the site today. I am pitching the travelogue itself to other publications so it may not have a clear fate for a while, but if it doesn't appear here I'll let you know where. In any case there probably will be a loose part two to look forward to.

[1] Greely had a rather illustrious Army career. His term as chief of the Signal Corps was something of a retirement after he led several arctic expeditions, the topic of his numerous popular books and articles. He received the Medal of Honor shortly before his death in 1935.

A long time ago I wrote about secret government telephone numbers, and before that, secret military telephone buttons. I suppose this is becoming a series. To be clear, the "secret" here is a joke, but more charitably I could say that it refers to obscurity rather than any real effort to keep them secret. Actually, today's examples really make this point: they're specifically intended to be well known, but are still pretty obscure in practice.

If you've been around for a while, you know how much I love telephone numbers. Here in North America, we have a system called the North American Numbering Plan (NANP) that has rigidly standardized telephone dialing practices since the middle of the 20th century. The US, Canada, and a number of Central American countries benefit from a very orderly system of area codes (more formally numbering plan areas or NPAs) followed by a subscriber number written in the format NXX-XXXX (this is a largely NANP-centric notation for describing phone number patterns, N represents the digits 2-9 and X any digit). All of these NANP numbers reside under the country code 1, allowing at least theoretically seamless international dialing within the NANP community. It's really a pretty elegant system.

NANP is the way it is for many reasons, but it mostly reflects technical requirements of the telephone exchanges of the 1940s. This is more thoroughly explained in the link above, but one of the goals of NANP is to ensure that step-by-step (SxS) exchanges can process phone numbers digit by digit as they are dialed. In other words, it needs to be possible to navigate the decision tree of telephone routing using only the digits dialed so far.

Readers with a computer science education might have some tidy way to describe this in terms of Chompsky or something, but I do not have a computer science education; I have an Information Technology education. That means I prefer flow charts to automata, and we can visualize a basic SxS exchange as a big tree. When you pick up your phone, you start at the root of the tree, and each digit dialed chooses the edge to follow. Eventually you get to a leaf that is hopefully someone's telephone, but at no point in the process does any node benefit from the context of digits you dial before, after, or how many total digits you dial. This creates all kinds of practical constraints, and is the reason, for example, that we tend to write ten-digit phone numbers with a "1" before them.

That requirement was in some ways long-lived (The last SxS exchange on the public telephone network was retired in 1999), and in other ways not so long lived... "common control" telephone exchanges, which did store the entire number in electromechanical memory before making a routing decision, were already in use by the time the NANP scheme was adopted. They just weren't universal, and a common nationwide numbering scheme had to be designed to accommodate the lowest common denominator.

This discussion so far is all applicable to the land-line telephone. There is a whole telephone network that is, these days, almost completely separate but interconnected: cellular phones. Early cellular phones (where "early" extends into CDMA and early GSM deployments) were much more closely attached to the "POTS" (Plain Old Telephone System). AT&T and Verizon both operated traditional telephone exchanges, for example 5ESS, that routed calls to and from their customers. These telephone exchanges have become increasingly irrelevant to mobile telephony, and you won't find a T-Mobile ESS or DMS anywhere. All US cellular carriers have adopted the GSM technology stack, and GSM has its own definition of the switching element that can be, and often is, fulfilled by an AWS EC2 instance running RHEL 8. Calls between cell phones today, even between different carriers, are often connected completely over IP and never touch a traditional telephone exchange.

The point is that not only is telephone number parsing less constrained on today's telephone network, in the case of cellular phones, it is outright required to be more flexible. GSM also defines the properties of phone numbers, and it is a very loose definition. Keep in mind that GSM is deeply European, and was built from the start to accommodate the wide variety of dialing practices found in Europe. This manifests in ways big and small; one of the notable small ways is that the European emergency number 112 works just as well as 911 on US cell phones because GSM dictates special handling for emergency numbers and dictates that 112 is one of those numbers. In fact, the definition of an "emergency call" on modern GSM networks is requesting a SIP URI of "urn:service:sos". This reveals that dialed number handling on cellular networks is fundamentally different.

When you dial a number on your cellular phone, the phone collects the entire number and then applies a series of rules to determine what to do, often leading to a GSM call setup process where the entire number, along with various flags, is sent to the network. This is all software-defined. In the immortal words of our present predicament, "everything's computer."

The bottom line is that, within certain regulatory boundaries and requirements set by GSM, cellular carriers can do pretty much whatever they want with phone numbers. Obviously numbers need to be NANP-compliant to be carried by the POTS, but many modern cellular calls aren't carried by the POTS, they are completed entirely within cellular carrier systems through their own interconnection agreements. This freedom allows all kinds of things like "HD voice" (cellular calls connected without the narrow filtering and companding used by the traditional network), and a lot of flexibility in dialing.

Most people already know about some weird cellular phone numbers. For example, you can dial *#06# to display your phone's various serial numbers. This is an example of a GSM MMI (man-machine interface) code, phone numbers that are handled entirely within your device but nonetheless defined as dialable numbers by GSM for compatibility with even the most basic flip phones. GSM also defined numbers called USSD for unstructured supplementary service data, which set up connections to the network that can be used in any arbitrary way the network pleases. Older prepaid phone services used to implement balance check and top-up operations using USSD numbers, and they're also often used in ways similar to Vertical Service Codes (VSCs) on the landline network to control carrier features. USSDs also enabled the first forms of mobile data, which involved a "special telephone call" to a USSD in order to download a cut-down form of ESPN in a weird mobile-specific markup language.

Now, put yourself in the shoes of an enterprising cellular network. The flexibility of processing phone numbers as you please opens up all kinds of possibilities. Innovative services! Customer convenience! Sell them for money! Oh my god, sell them for money!

It seems like this started with customer service. It is an old practice, dating to the Bell operating companies, to have special short phone numbers to reach the telephone company itself. The details varied by company (often based on technical constraints in their switching system), but a common early setup was that dialing 114 got you the repair service operator to report a problem with your phone line. These numbers were usually listed in the front of the phone book, and for the phone company the fact that they were "special" or nonstandard was sort of a feature, since they could ensure that they were always routed within the same switch. The selection of "911" as the US emergency number seems rooted in this practice, as later on several major telcos used the "N11" numbers for their service lines. This became immortalized in the form of 611, which will get you customer service for most phone carriers.

So cellular companies did the same, allocating themselves "special" numbers for various service lines. Verizon offers #PMT to make a payment. Naturally, there's also room for upsell services: #ROAD for roadside assistance on Verizon.

The odd thing about these phone numbers is that there's really no standard involved, they're just the arbitrary practices of specific cellular companies. The term "mobile dial code" (MDC) is usually used to refer to them, although that term seems to have arisen organically rather than by intent. Remember, these aren't a real thing! The carriers just make them up, all on their own.

The only real constraint on MDCs is that they need to not collide with any POTS number, which is most easily achieved by prefixing them with some combination of * and #, and usually not "*#" because it's referenced by the GSM standard for MMI.

MDCs are available for purchase, but the terms don't seem to be public and you have to negotiate separately with each carrier. That's because there is no centralization. This is where MDCs stand in clear contrast to the better known SMS Short Code, or SMSSC. Those are the five or six-digit numbers widely used in advertising campaigns.

SMSSCs are centrally managed by the SMS Short Code Registry, which is a function of industry association CTIA but contracted to iConectiv. iConectiv is sort of like the SAIC of the communications industry, a huge company that dates back to the Bell System (where it became Bellcore after divestiture) and that no one has heard of but nonetheless is a critically important part of the telephone system.

Providers that want to have an SMSSC (typically on behalf of one of their customers) pay a fee, and usually recoup it from the end user. That fee is not cheap, typical end-user rates for an SMSSC run over $10k a year. But at least it's straightforward, and your SMS A2P or marketing company can make it happen for you.

MDCs have no such centralization, no standardized registration process. You negotiate with each carrier individually. That means it's pretty difficult to put together "complete coverage" on an MDC by getting the same one assigned by every major carrier. And this is one of those areas where "good enough" is seldom good enough; people get pissed off when something you advertise doesn't work. Putting a phone number that only works for some people on a billboard can quickly turn into an expensive embarrassment, so companies will be wary of using an MDC in marketing if they don't feel really confident that it works for the vast majority of cellphone users.

Because of this fragmentation, adoption of MDCs for marketing purposes has been very low. The only going concern I know of is #250, operated by a company called Mobile Direct Response. The premise of #250 is very simple: users call #250 and are greeted by a simple IVR. They say a keyword, and they're either forwarded to the phone number of the business that paid for the keyword or they receive a text message response with more information. #250 is specifically oriented towards radio advertising, where asking people to remember a ten-digit phone number is, well, asking a lot. It's also made the jump to podcast advertising. #250 is priced in a very radio-centric way, by the keyword and the size of the market area in which the advertisement that gives the keyword is played.

#250 was founded by Dave Robinett, who used to work on marketing at Sprint, presumably where he became aware that these MDCs were a possibility. He has negotiated for #250 to work across a substantial list of cellular carriers in the US and Canada, providing almost complete coverage. That wasn't easy, Robinett said in an interview that it took five years to get AT&T, T-Mobile, Verizon, and Sprint on board.

#250 does not appear to be especially widely used. For one, the website is a little junky, with some broken links and other indications that it is not backed by a large communications department. Dave Robinett may be the entire company. They've been operating since at least 2017, and I've only ever heard it in an ad once---a podcast ad that ended with "Call #250 and say I need a dentist." One thing you quickly notice when you look into telephone marketing is that dentists are apparently about 80% of the market. He does mention success with shows like "Rush, Hannity, and Levin," so it's safe to say that my radio habits are a little different from Robinett's.

That's not to say that #250 is a failure. In the same interview Robinett says that the company pays his mortgage and, well, that ain't too bad. But it's also nothing like the widespread adoption of SMSSCs. One wonders if the limitation of MDCs to one company that is so focused on radio marketing limits their potential. It might really open things up if some company created a registration service, and prenegotiated terms with carriers so that companies could pick up their own MDCs to use as they please.

Well, yeah, someone's trying. Around 2006, a recently-founded mobile marketing company called Zoove announced StarStar dialing. I'm a little unclear on Zoove's history. It seems that they were originally founded as Teleractive in Rhode Island as an SMS short code keyword response service, and after an infusion of VC cash moved to Palo Alto and started looking for something bigger. In 2016, they were acquired by a call center technology company called Mindful. Or maybe Zoove sold the StarStar business to Mindful? Stick a pin in that.

I don't love the name StarStar, which has shades of Spacestar Ordering. But it refers to their chosen MDC prefix, two stars. Well, that point is a little odd, according to their marketing material you can also get numbers with a # prefix or * prefix, but all of the examples use **. I would say that, in general, StarStar has it a little less together than #250. Their website is kind of broken, it only loads intermittently and some of the images are missing. At one point it uses the term "CADC" to describe these numbers but I can't find that expanded anywhere. Plus the "About" page refers repeatedly to Virtual Hold Technologies, which renamed to VHT in 2018 and Mindful 2022. It really feels like the vestigial website of a dead company.

I know about StarStar because, for a time, trucks from moving franchise All My Sons prominently bore the number **MOVE on the side. Indeed, this is still one of the headline examples on the StarStar website, but it doesn't work. I just get a loud click and then the call ends. And it's not that StarStar doesn't work with my mobile carrier, because StarStar's own number **MOBILE does connect to their IVR. That IVR promises that a representative will speak with me shortly, plays about five seconds of hold music, and then dumps me on a voicemail system. Despite StarStar numbers apparently basically working, I'm finding that most of the examples they give on their website won't even connect. Perhaps results will vary depending on the mobile network.

Well, perhaps not that much is lost. StarStar was founded by Steve Doumar, a serial telephone marketing entrepreneur with a colorful past founding various inbound call center companies. Perhaps his most famous venture is R360, a "lead acquisition" service memorialized by headlines like "Drug treatment referral service took advantage of addictions to make a quick buck" from the Federal Trade Commission. He's one of those guys whose bio involves founding a new company every two years, which he has to spin as entrepreneurial dynamism rather than some combination of fleeing dissatisfied investors and fleeing angered regulators.

Today he runs whisp.io, a "customer activation platform" that appears to be a glorified SMS advertising service featuring something ominously called "simplified opt-in." Whisp has a YouTube channel which features the 48-second gem "Fun Fact We Absolutely Love About Steve Doumar". Description:

Our very own CEO, Steve Doumar is a kind and generous person who has given back to the community in many ways; this man is absolutely a man with a heart of gold.

Do you want to know the fun fact? Yes you do! Here it is: "He is an incredible philanthropist. He loves helping other people. Every time I'm with him he comes up with new ways and new ideas to help other people. Which I think is amazing. And he doesn't brag about it, he doesn't talk about it a lot." Except he's got his CMO making a YouTube video about it?

From Steve Doumar's blog:

American entrepreneur Ray Kroc expressed the importance of persisting in a busy world where everyone wants a bite of success.

This man is no exception.

An entrepreneur. A family man. A visionary.

These are the many names of a man that has made it possible for opt-ins to be safe, secure, and accurate; Steve Doumar.

I love this stuff, you just can't make it up. I'm pretty sure what's going on here is just an SEO effort to outrank the FTC releases and other articles about the R360 case when you search for his name. It's only partially working, "FTC Hits R360 and its Owner With $3.8 Million Civil ..." still comes in at Google result #4 for "Steve Doumar," at least for me. But hey, #4 is better than #1.

Well, to be fair to StarStar, I don't think Steve Doumar has been involved for some years, but also to be fair, some of their current situation clearly dates to past behavior that is maybe less than savory.

Zoove originally styled itself as "The National StarStar Registry," clearly trying to draw parallels to CTIA/iConectiv's SMSSC registry. Their largest customer was evidently a company called Sumotext, which leased a number of StarStar numbers to offer an SMS and telephone marketing service. In 2016, Sumotext sued StarStar, Zoove, VHT (now Mindful), and a healthy list of other entities all involved in StarStar including the intriguingly named StarSteve LLC. I'm not alone in finding the corporate history a little baffling; in a footnote on one ruling the court expressed confusion about all the different names and opted to call them all Zoove.

In any case, Sumotext alleged that Zoove, StarSteve, and VHT all merged as part of a scheme to illegally monopolize the StarStar market by undercutting the companies that had been leasing the numbers and effectively giving VHT (Mindful) an exclusive ability to offer marketing services with StarStar numbers. The case didn't end up going anywhere for Sumotext, the jury found that Sumotext hadn't established a relevant market which is a key part of a Sherman act case. An appeal was made all the way to the Supreme Court, but they didn't take it up. What the case did do was publicize some pretty sketchy sounding details, like the seemingly uncontested accusation that VHT got Sumotext's customer list from the registry database and used it to convert them all into StarSteve customers.

And yes, the Steve in StarSteve is Steve Doumar. As best I can tell, the story here is that Steve Doumar founded Zoove (or bought Teleractive and renamed it or something?) to establish the National StarStar Registry, then founded a marketing company called StarSteve that resold StarStar numbers, then merged StarSteve and the National StarStar Registry together and cut off all of the other resellers. Apparently not a Sherman act violation but it sure is a bad look, and I wonder how much it contributed to the lack of adoption of the whole StarStar idea---especially given that Sumotext seems to have been responsible for most of that adoption, including the All My Sons deal for **MOVE. I wonder if All My Sons had to take **MOVE off of their trucks because of the whole StarSteve maneuver? That seems to be what happened.

Look, ten-digit phone numbers are had to remember, that much is true. But as is, the "MDC" industry doesn't seem stable enough for advertising applications where the number needs to continue to work into the future. I think the #250 service is probably here to stay, but confined to the niche of audio advertising. StarStar raised at least $30 million in capital in the 2010s, but seems to have shot itself in the foot. StarStar owner VHT/Mindful, now acquired by Medallia, doesn't even mention StarStar as a product offering.

Hey, remember how Steve Doumar is such a great philanthropist? There are a lot of vestiges around of StarStar Inc., a nonprofit that made StarStar numbers available to charitable organizations. Their website, starstar.org, is now a Wix error page. You can find old articles about StarStar Me, also written **me, which sounds lewd but was a $3/mo offering that allowed customers to get a vanity short code (such as ** followed by their name)---the original form of StarStar, dating back to 2012 and the beginning of Zoove.

In a press release announcing the StarStar Me, Zoove CEO Joe Gillespie said:

With two-thirds of smartphone users having downloaded social networking apps to their phones, there’s a rapidly growing trend in today's on-the-go lifestyle to extend our personal communications and identity into the digital realm via our mobile phones.

And somehow this leads to paying $3 to get StarStarred? I love it! It's so meaningless! And years later it would be StarStar Mobile formerly Zoove by VHT now known as Mindful a Medallia company. Truly an inspiring story of industry, and just one little corner of the vast tapestry of phone numbers.

Some time ago, via a certain orange website, I came across a report about a mission to recover nuclear material from a former Soviet test site. I don't know what you're doing here, go read that instead. But it brought up a topic that I have only known very little about: Hydronuclear testing.

One of the key reasons for the nonproliferation concern at Semipalatinsk was the presence of a large quantity of weapons grade material. This created a substantial risk that someone would recover the material and either use it directly or sell it---either way giving a significant leg up on the construction of a nuclear weapon. That's a bit odd, though, isn't it? Material refined for use in weapons in scarce and valuable, and besides that rather dangerous. It's uncommon to just leave it lying around, especially not hundreds of kilograms of it.

This material was abandoned in place because the nature of the testing performed required that a lot of weapons-grade material be present, and made it very difficult to remove. As the Semipalatinsk document mentions in brief, similar tests were conducted in the US and led to a similar abandonment of special nuclear material at Los Alamos's TA-49. Today, I would like to give the background on hydronuclear testing---the what and why. Then we'll look specifically at LANL's TA-49 and the impact of the testing performed there.

First we have to discuss the boosted fission weapon. Especially in the 21st century, we tend to talk about "nuclear weapons" as one big category. The distinction between an "A-bomb" and an "H-bomb," for example, or between a conventional nuclear weapon and a thermonuclear weapon, is mostly forgotten. That's no big surprise: thermonuclear weapons have been around since the 1950s, so it's no longer a great innovation or escalation in weapons design.

The thermonuclear weapon was not the only post-WWII design innovation. At around the same time, Los Alamos developed a related concept: the boosted weapon. Boosted weapons were essentially an improvement in the efficiency of nuclear weapons. When the core of a weapon goes supercritical, the fission produces a powerful pulse of neutrons. Those neutrons cause more fission, the chain reaction that makes up the basic principle of the atomic bomb. The problem is that the whole process isn't fast enough: the energy produced blows the core apart before it's been sufficiently "saturated" with neutrons to completely fission. That leads to a lot of the fuel in the core being scattered, rather than actually contributing to the explosive energy.

In boosted weapons, a material that will fusion is added to the mix, typically tritium and deuterium gas. The immense heat of the beginning of the supercritical stage causes the gas to undergo fusion, and it emits far more neutrons than the fissioning fuel does alone. The additional neutrons cause more fission to occur, improving the efficiency of the weapon. Even better, despite the theoretical complexity of driving a gas into fusion¸ the mechanics of this mechanism are actually simpler than the techniques used to improve yield in non-boosted weapons (pushers and tampers).

The result is that boosted weapons produce a more powerful yield in comparison to the amount of fuel, and the non-nuclear components can be made simpler and more compact as well. This was a pretty big advance in weapons design and boosting is now a ubiquitous technique.

It came with some downsides, though. The big one is that whole property of making supercriticality easier to achieve. Early implosion weapons were remarkably difficult to detonate, requiring an extremely precisely timed detonation of the high explosive shell. While an inconvenience from an engineering perspective, the inherent difficulty of achieving a nuclear yield also provided a safety factor. If the high explosives detonated for some unintended reason, like being struck by canon fire as a bomber was intercepted, or impacting the ground following an accidental release, it wouldn't "work right." Uneven detonation of the shell would scatter the core, rather than driving it into supercriticality.

This property was referred to as "one point safety:" a detonation at one point on the high explosive assembly should not produce a nuclear yield. While it has its limitations, it became one of the key safety principles of weapon design.

The design of boosted weapons complicated this story. Just a small fission yield, from a small fragment of the core, could potentially start the fusion process and trigger the rest of the core to detonate as well. In other words, weapon designers became concerned that boosted weapons would not have one point safety. As it turns out, two-stage thermonuclear weapons, which were being fielded around the same time, posed a similar set of problems.

The safety problems around more advanced weapon designs came to a head in the late '50s. Incidentally, so did something else: shifts in Soviet politics had given Khrushchev extensive power over Soviet military planning, and he was no fan of nuclear weapons. After some on-again, off-again dialog between the time's nuclear powers, the US and UK agreed to a voluntary moratorium on nuclear testing which began in late 1958.

For weapons designers this was, of course, a problem. They had planned to address the safety of advanced weapon designs through a testing campaign, and that was now off the table for the indefinite future. An alternative had to be developed, and quickly.

In 1959, the Hydronuclear Safety Program was initiated. By reducing the amount of material in otherwise real weapon cores, physicists realized they could run a complete test of the high explosive system and observe its effects on the core without producing a meaningful nuclear yield. These tests were dubbed "hydronuclear," because of the desire to observe the behavior of the core as it flowed like water under the immense explosive force. While the test devices were in some ways real nuclear weapons, the nuclear yield would be vastly smaller than the high explosive yield, practically nill.

Weapons designers seemed to agree that these experiments complied with the spirit of the moratorium, being far from actual nuclear tests, but there was enough concern that Los Alamos went to the AEC and President Eisenhower for approval. They evidently agreed, and work started immediately to identify a suitable site for hydronuclear testing.

While hydronuclear tests do not create a nuclear yield, they do involve a lot of high explosives and radioactive material. The plan was to conduct the tests underground, where the materials cast off by the explosion would be trapped. This would solve the immediate problem of scattering nuclear material, but it would obviously be impractical to recover the dangerous material once it was mixed with unstable soil deep below the surface. The material would stay, and it had to stay put!

The US Army Corps of Engineers, a center of expertise in hydrology because of their reclamation work, arrived in October 1959 to begin an extensive set of studies on the Frijoles Mesa site. This was an unused area near a good road but far on the east edge of the laboratory, well separated from the town of Los Alamos and pretty much anything else. More importantly, it was a classic example of northern New Mexican geology: high up on a mesa built of tuff and volcanic sediments, well-drained and extremely dry soil in an area that received little rain.

One of the main migration paths for underground contaminants is their interaction with water, and specifically the tendency of many materials to dissolve into groundwater and flow with it towards aquifers. The Corps of Engineers drilled test wells, about 1,500' deep, and a series of 400' core samples. They found that on the Frijoles Mesa, ground water was over 1,000' below the surface, and that everything above was far from saturation. That means no mobility of the water, which is trapped in the soil. It's just about the ideal situation for putting something underground and having it stay.

Incidentally, this study would lead to the development of a series of new water wells for Los Alamos's domestic water supply. It also gave the green light for hydronuclear testing, and Frijoles Mesa was dubbed Technical Area 49 and subdivided into a set of test areas. Over the following three years, these test areas would see about 35 hydronuclear detonations carried out in the bottom of shafts that were about 200' deep and 3-6' wide.

It seems that for most tests, the hole was excavated and lined with a ladder installed to reach the bottom. Technicians worked at the bottom of the hole to prepare the test device, which was connected by extensive cabling to instrumentation trailers on the surface. When the "shot" was ready, the hole was backfilled with sand and sealed at the top with a heavy plate. The material on top of the device held everything down, preventing migration of nuclear material to the surface. The high explosives did, of course, destroy the test device and the cabling, but not before the instrumentation trailers had recorded a vast amount of data.

If you read these kinds of articles, you must know that the 1958 moratorium did not last. Soviet politics shifted again, France began nuclear testing, negotiations over a more formal test ban faltered. US intelligence suspected that the Soviet Union had operated their nuclear weapons program at full tilt during the test ban, and the military suspected clandestine tests, although there was no evidence they had violated the treaty. Of course, that they continued their research efforts is guaranteed, we did as well. Physicist Edward Teller, ever the nuclear weapons hawk, opposed the moratorium and pushed to resume testing.

In 1961, the Soviet Union resumed testing, culminating in the test of the record-holding "Tsar Bomba," a 50 megaton device. The US resumed testing as well. The arms race was back on.

US hydronuclear testing largely ended with the resumption of full-scale testing. The same safety studies could be completed on real weapons, and those tests would serve other purposes in weapons development as well. Although post-moratorium testing included atmospheric detonations, the focus had shifted towards underground tests and the 1963 Partial Test Ban Treaty restricted the US and USSR to underground tests only.

One wonders about the relationship between hydronuclear testing at TA-49 and the full-scale underground tests extensively performed at the NTS. Underground testing began in 1951 with Buster-Jangle Uncle, a test to determine how big of a crater could be produced by a ground-penetrating weapon. Uncle wasn't really an underground test in the modern sense, the device was emplaced only 17 feet deep and still produced a huge cloud of fallout. It started a trend, though: a similar 1955 test was set 67 feet deep, producing a spectacular crater, before the 1957 Plumbbob Pascal-A was detonated at 486 feet and produced radically less fallout.

1957's Plumbbob Rainier was the first fully-contained underground test, set at the end of a tunnel excavated far into a hillside. This test emitted no fallout at all, proving the possibility of containment. Thus both the idea of emplacing a test device in a deep hole, and the fact that testing underground could contain all of the fallout, were known when the moratorium began in 1959.

What's very interesting about the hydronuclear tests is the fact that technicians actually worked "downhole," at the bottom of the excavation. Later underground tests were prepared by assembling the test device at the surface, as part of a rocket-like "rack," and then lowering it to the bottom just before detonation. These techniques hadn't yet been developed in the '50s, thus the use of a horizontal tunnel for the first fully-contained test.

Many of the racks used for underground testing were designed and built by LANL, but others (called "canisters" in an example of the tendency of the labs to not totally agree on things) were built by Lawrence Livermore. I'm not actually sure which of the two labs started building them first, a question for future research. It does seem likely that the hydronuclear testing at LANL advanced the state of the art in remote instrumentation and underground test design, facilitating the adoption of fully-contained underground tests in the following years.

During the three years of hydronuclear testing, shafts were excavated in four testing areas. It's estimated that the test program at TA-49 left about 40kg of plutonium and 93kg of enriched uranium underground, along with 92kg of depleted uranium and 13kg of beryllium (both toxic contaminants). Because of the lack of a nuclear yield, these tests did not create the caverns associated with underground testing. Material from the weapons likely spread within just a 10-20' area, as holes were drilled on a 25' grid and contamination from previous neighboring tests was encountered only once.

The tests also produced quite a bit of ancillary waste: things like laboratory equipment, handling gear, cables and tubing, that are not directly radioactive but were contaminated with radioactive or toxic materials. In the fashion typical of the time, this waste was buried on site, often as part of the backfilling of the test shafts.

During the excavation of one of the test shafts, 2-M in December 1960, contamination was detected at the surface. It seems that the geology allowed plutonium from a previous test to spread through cracks into the area where 2-M was being drilled. The surface soil contaminated by drill cuttings was buried back in hole 2-M, but this incident made area 2 the most heavily contaminated part of TA-49. When hydronuclear testing ended in 1961, area 2 was covered by a 6' of gravel and 4-6" of asphalt to better contain any contaminated soil.

Several support buildings on the surface were also contaminated, most notably a building used as a radiochemistry laboratory to support the tests. An underground calibration facility that allowed for exposure of test equipment to a contained source in an underground chamber was also built at TA-49 and similarly contaminated by use with radioisotopes.

The Corps of Engineers continued to monitor the hydrology of the site from 1961 to 1970, and test wells and soil samples showed no indication that any contamination was spreading. In 1971, LANL established a new environmental surveillance department that assumed responsibility for legacy sites like TA-49. That department continued to sample wells, soil, and added air sampling. Monitoring of stream sediment downhill from the site was added in the '70s, as many of the contaminants involved can bind to silt and travel with surface water. This monitoring has not found any spread either.

That's not to say that everything is perfect. In 1975, a section of the asphalt pad over Area 2 collapsed, leaving a three foot deep depression. Rainwater pooled in the depression and then flowed through the gravel into hole 2-M itself, collecting in the bottom of the lining of the former experimental shaft. In 1976, the asphalt cover was replaced, but concerns remained about the water that had already entered 2-M. It could potentially travel out of the hole, continue downwards, and carry contamination into the aquifer around 800' below. Worse, a nearby core sample hole had picked up some water too, suggesting that the water was flowing out of 2-M through cracks and into nearby features. Since the core hole had a slotted liner, it would be easier for water to leave it and soak into the ground below.

In 1980, the water that had accumulated in 2-M was removed by lifting about 24 gallons to the surface. While the water was plutonium contaminated, it fell within acceptable levels for controlled laboratory areas. Further inspections through 1986 did not find additional water in the hole, suggesting that the asphalt pad was continuing to function correctly. Several other investigations were conducted, including the drilling of some additional sample wells and examination of other shafts in the area, to determine if there were other routes for water to enter the Area 2 shafts. Fortunately no evidence of ongoing water ingress was found.

In 1986, TA-49 was designated a hazardous waste site under the Resource Conservation and Recovery Act. Shortly after, the site was evaluated under CERCLA to prioritize remediation. Scoring using the Hazard Ranking System determined a fairly low risk for the site, due to the lack of spread of the contamination and evidence suggesting that it was well contained by the geology.

Still, TA-49 remains an environmental remediation site and now falls under a license granted by the New Mexico Environment Department. This license requires ongoing monitoring and remediation of any problems with the containment. For example, in 1991 the asphalt cover of Area 2 was found to have cracked and allowed more water to enter the sample wells. The covering was repaired once again, and investigations made every few years from 1991 to 2015 to check for further contamination. Ongoing monitoring continues today. So far, Area 2 has not been found to pose an unacceptable risk to human health or a risk to the environment.

NMED permitting also covers the former radiological laboratory and calibration facility, and infrastructure related to them like a leach field from drains. Sampling found some surface contamination, so the affected soil was removed and disposed of at a hazardous waste landfill where it will be better contained.

TA-49 was reused for other purposes after hydronuclear testing. These activities included high explosive experiments contained in metal "bottles," carried out in a metal-lined pit under a small structure called the "bottle house." Part of the bottle house site was later reused to build a huge hydraulic ram used to test steel cables at their failure strength. I am not sure of the exact purpose of this "Cable Test Facility," but given the timeline of its use during the peak of underground testing and the design I suspect LANL used it as a quality control measure for the cable assemblies used in lowering underground test racks into their shafts. No radioactive materials were involved in either of these activities, but high explosives and hydraulic oil can both be toxic, so both were investigated and received some surface soil cleanup.

Finally, the NMED permit covers the actual test shafts. These have received numerous investigations over the sixty years since the original tests, and significant contamination is present as expected. However, that contamination does not seem to be spreading, and modeling suggests that it will stay that way.

In 2022, the NMED issued Certificates of Completion releasing most of the TA-49 remediation sites without further environmental controls. The test shafts themselves, known to NMED by the punchy name of Solid Waste Management Unit 49-001(e), received a certificate of completion that requires ongoing controls to ensure that the land is used only for industrial purposes. Environmental monitoring of the TA-49 site continues under LANL's environmental management program and federal regulation, but TA-49 is no longer an active remediation project. The plutonium and uranium is just down there, and it'll have to stay.

In a previous life, I worked for a location-based entertainment company, part of a huge team of people developing a location for Las Vegas, Nevada. It was COVID, a rough time for location-based anything, and things were delayed more than usual. Coworkers paid a lot of attention to another upcoming Las Vegas attraction, one with a vastly larger budget but still struggling to make schedule: the MSG (Madison Square Garden) Sphere.

I will set aside jokes about it being a square sphere, but they were perhaps one of the reasons that it underwent a pre-launch rebranding to merely the Sphere. If you are not familiar, the Sphere is a theater and venue in Las Vegas. While it's know mostly for the video display on the outside, that's just marketing for the inside: a digital dome theater, with seating at a roughly 45 degree stadium layout facing a near hemisphere of video displays.

It is a "near" hemisphere because the lower section is truncated to allow a flat floor, which serves as a stage for events but is also a practical architectural decision to avoid completely unsalable front rows. It might seem a little bit deceptive that an attraction called the Sphere does not quite pull off even a hemisphere of "payload," but the same compromise has been reached by most dome theaters. While the use of digital display technology is flashy, especially on the exterior, the Sphere is not quite the innovation that it presents itself as. It is just a continuation of a long tradition of dome theaters. Only time will tell, but the financial difficulties of the Sphere suggest that it follows the tradition faithfully: towards commercial failure.

You could make an argument that the dome theater is hundreds of years old, but I will omit it. Things really started developing, at least in our modern tradition of domes, with the 1923 introduction of the Zeiss planetarium projector. Zeiss projectors and their siblings used a complex optical and mechanical design to project accurate representations of the night sky. Many auxiliary projectors, incorporated into the chassis and giving these projectors famously eccentric shapes, rendered planets and other celestial bodies. Rather than digital light modulators, the images from these projectors were formed by purely optical means: perforated metal plates, glass plates with etched metalized layers, and fiber optics. The large, precisely manufactured image elements and specialized optics created breathtaking images.

While these projectors had considerable entertainment value, especially in the mid-century when they represented some of the most sophisticated projection technology yet developed, their greatest potential was obviously in education. Planetarium projectors were fantastically expensive (being hand-built in Germany with incredible component counts) [1], they were widely installed in science museums around the world. Most of us probably remember a dogbone-shaped Zeiss, or one of their later competitors like Spitz or Minolta, from our youths. Unfortunately, these marvels of artistic engineering were mostly retired as digital projection of near comparable quality became similarly priced in the 2000s.

But we aren't talking about projectors, we're talking about theaters. Planetarium projectors were highly specialized to rendering the night sky, and everything about them was intrinsically spherical. For both a reasonable viewing experience, and for the projector to produce a geometrically correct image, the screen had to be a spherical section. Thus the planetarium itself: in its most traditional form, rings of heavily reclined seats below a hemispherical dome. The dome was rarely a full hemisphere, but was usually truncated at the horizon. This was mostly a practical decision but integrated well into the planetarium experience, given that sky viewing is usually poor near the horizon anyway. Many planetaria painted a city skyline or forest silhouette around the lower edge to make the transition from screen to wall more natural. Later, theatrical lighting often replaced the silhouette, reproducing twilight or the haze of city lights.

Unsurprisingly, the application-specific design of these theaters also limits their potential. Despite many attempts, the collective science museum industry has struggled to find entertainment programming for planetaria much beyond Pink Floyd laser shows [2]. There just aren't that many things that you look up at. Over time, planetarium shows moved in more narrative directions. Film projection promised new flexibility---many planetaria with optical star projectors were also equipped with film projectors, which gave show producers exciting new options. Documentary video of space launches and animations of physical principles became natural parts of most science museum programs, but were a bit awkward on the traditional dome. You might project four copies of the image just above the horizon in the four cardinal directions, for example. It was very much a compromise.

With time, the theater adapted to the projection once again: the domes began to tilt. By shifting the dome in one direction, and orienting the seating towards that direction, you could create a sort of compromise point between the traditional dome and traditional movie theater. The lower central area of the screen was a reasonable place to show conventional film, while the full size of the dome allowed the starfield to almost fill the audience's vision. The experience of the tilted dome is compared to "floating in space," as opposed to looking up at the sky.

In true Cold War fashion, it was a pair of weapons engineers (one nuclear weapons, the other missiles) who designed the first tilted planetarium. In 1973, the planetarium of what is now called the Fleet Science Center in San Diego, California opened to the public. Its dome was tilted 25 degrees to the horizon, with the seating installed on a similar plane and facing in one direction. It featured a novel type of planetarium projector developed by Spitz and called the Space Transit Simulator. The STS was not the first, but still an early mechanical projector to be controlled by a computer---a computer that also had simultaneous control of other projectors and lighting in the theater, what we now call a show control system.

Even better, the STS's innovative optical design allowed it to warp or bend the starfield to simulate its appearance from locations other than earth. This was the "transit" feature: with a joystick connected to the control computer, the planetarium presenter could "fly" the theater through space in real time. The STS was installed in a well in the center of the seating area, and its compact chassis kept it low in the seating area, preserving the spherical geometry (with the projector at the center of the sphere) without blocking the view of audience members sitting behind it and facing forward.

And yet my main reason for discussing the Fleet planetarium is not the the planetarium projector at all. It is a second projector, an "auxiliary" one, installed in a second well behind the STS. The designers of the planetarium intended to show film as part of their presentations, but they were not content with a small image at the center viewpoint. The planetarium commissioned a few of the industry's leading film projection experts to design a film projection system that could fill the entire dome, just as the planetarium projector did.

They knew that such a large dome would require an exceptionally sharp image. Planetarium projectors, with their large lithographed slides, offered excellent spatial resolution. They made stars appear as point sources, the same as in the night sky. 35mm film, spread across such a large screen, would be obviously blurred in comparison. They would need a very large film format.

Fortuitously, almost simultaneously the Multiscreen Corporation was developing a "sideways" 70mm format. This 15-perf format used 70mm film but fed it through the projector sideways, making each frame much larger than typical 70mm film. In its debut, at a temporary installation in the 1970 Expo Osaka, it was dubbed IMAX. IMAX made an obvious basis for a high-resolution projection system, and so the then-named IMAX Corporation was added to the planetarium project. The Fleet's film projector ultimately consisted of an IMAX film transport with a custom-built compact, liquid-cooled lamphouse and spherical fisheye lens system.

The large size of the projector, the complex IMAX framing system and cooling equipment, made it difficult to conceal in the theater's projector well. Threading film into IMAX projectors is quite complex, with several checks the projectionist must make during a pre-show inspection. The projectionist needed room to handle the large film, and to route it to and from the enormous reels. The projector's position in the middle of the seating area left no room for any of this. We can speculate that it was, perhaps, one of the designer's missile experience that lead to the solution: the projector was serviced in a large projection room beneath the theater's seating. Once it was prepared for each show, it rose on near-vertical rails until just the top emerged in the theater. Rollers guided the film as it ran from a platter, up the shaft to the projector, and back down to another platter. Cables and hoses hung below the projector, following it up and down like the traveling cable of an elevator.

To advertise this system, probably the greatest advance in film projection since the IMAX format itself, the planetarium coined the term Omnimax.

Omnimax was not an easy or economical format. Ideally, footage had to be taken in the same format, using a 70mm camera with a spherical lens system. These cameras were exceptionally large and heavy, and the huge film format limited cinematographers to short takes. The practical problems with Omnimax filming were big enough that the first Omnimax films faked it, projecting to the larger spherical format from much smaller conventional negatives. This was the case for "Voyage to the Outer Planets" and "Garden Isle," the premier films at the Fleet planetarium. The history of both is somewhat obscure, the latter especially.

"Voyage to the Outer Planets" was executive-produced by Preston Fleet, a founder of the Fleet center (which was ultimately named for his father, a WWII aviator). We have Fleet's sense of showmanship to thank for the invention of Omnimax: He was an accomplished business executive, particularly in the photography industry, and an aviation enthusiast who had his hands in more than one museum. Most tellingly, though, he had an eccentric hobby. He was a theater organist. I can't help but think that his passion for the theater organ, an instrument almost defined by the combination of many gizmos under electromechanical control, inspired "Voyage." The film, often called a "multimedia experience," used multiple projectors throughout the planetarium to depict a far-future journey of exploration. The Omnimax film depicted travel through space, with slide projectors filling in artist's renderings of the many wonders of space.

The ten-minute Omnimax film was produced by Graphic Films Corporation, a brand that would become closely associated with Omnimax in the following decades. Graphic was founded in the midst of the Second World War by Lester Novros, a former Disney animator who found a niche creating training films for the military. Novros's fascination with motion and expertise in presenting complicated 3D scenes drew him to aerospace, and after the war he found much of his business in the newly formed Air Force and NASA. He was also an enthusiast of niche film formats, and Omnimax was not his first dome.

For the 1964 New York World's Fair, Novros and Graphic Films had produced "To the Moon and Beyond," a speculative science film with thematic similarities to "Voyage" and more than just a little mechanical similarity. It was presented in Cinerama 360, a semi-spherical, dome-theater 70mm format presented in a special theater called the Moon Dome. "To the Moon and Beyond" was influential in many ways, leading to Graphic Films' involvement in "2001: A Space Odyssey" and its enduring expertise in domes.

The Fleet planetarium would not remain the only Omnimax for long. In 1975, the city of Spokane, Washington struggled to find a new application for the pavilion built for Expo '74 [3]. A top contender: an Omnimax theater, in some ways a replacement for the temporary IMAX theater that had been constructed for the actual Expo. Alas, this project was not to be, but others came along: in 1978, the Detroit Science Center opened the second Omnimax theater ("the machine itself looks like and is the size of a front loader," the Detroit Free Press wrote). The Science Museum of Minnesota, in St. Paul, followed shortly after.

Omnimax hit prime time the next year, with the 1979 announcement of an Omnimax theater at Caesars Palace in Las Vegas, Nevada. Unlike the previous installations, this 380-seat theater was purely commercial. It opened with the 1976 IMAX film "To Fly!," which had been optically modified to fit the Omnimax format. This choice of first film is illuminating. "To Fly!" is a 27 minute documentary on the history of aviation in the United States, originally produced for the IMAX theater at the National Air and Space Museum [4]. It doesn't exactly seem like casino fare.

The IMAX format, the flat-screen one, was born of world's fairs. It premiered at an Expo, reappeared a couple of years later at another one, and for the first years of the format most of the IMAX theaters built were associated with either a major festival or an educational institution. This noncommercial history is a bit hard to square with the modern IMAX brand, closely associated with major theater chains and the Marvel Cinematic Universe.

Well, IMAX took off, and in many ways it sold out. Over the decades since the 1970 Expo, IMAX has met widespread success with commercial films and theater owners. Simultaneously, the definition or criteria for IMAX theaters have relaxed, with smaller screens made permissible until, ultimately, the transition to digital projection eliminated the 70mm film and more or less reduce IMAX to just another ticket surcharge brand. It competes directly with Cinemark xD, for example. To the theater enthusiast, this is a pretty sad turn of events, a Westinghouse-esque zombification of a brand that once heralded the field's most impressive technical achievements.

The same never happened to Omnimax. The Caesar's Omnimax theater was an odd exception; the vast majority of Omnimax theaters were built by science museums and the vast majority of Omnimax films were science documentaries. Quite a few of those films had been specifically commissioned by science museums, often on the occasion of their Omnimax theater opening. The Omnimax community was fairly tight, and so the same names recur.

The Graphic Films Corporation, which had been around since the beginning, remained so closely tied to the IMAX brand that they practically shared identities. Most Omnimax theaters, and some IMAX theaters, used to open with a vanity card often known as "the wormhole." It might be hard to describe beyond "if you know you know," it certainly made an impression on everyone I know that grew up near a theater that used it. There are some videos, although unfortunately none of them are very good.

I have spent more hours of my life than I am proud to admit trying to untangle the history of this clip. Over time, it has appeared in many theaters with many different logos at the end, and several variations of the audio track. This is in part informed speculation, but here is what I believe to be true: the "wormhole" was originally created by Graphic Films for the Fleet planetarium specifically, and ran before "Voyage to the Outer Planets" and its double-feature companion "Garden Isle," both of which Graphic Films had worked on. This original version ended with the name Graphic Films, accompanied by an odd sketchy drawing that was also used as an early logo of the IMAX Corporation. Later, the same animation was re-edited to end with an IMAX logo.

This version ran in both Omnimax and conventional IMAX theaters, probably as a result of the extensive "cross-pollination" of films between the two formats. Many Omnimax films through the life of the format had actually been filmed for IMAX, with conventional lenses, and then optically modified to fit the Omnimax dome after the fact. You could usually tell: the reprojection process created an unusual warp in the image, and more tellingly, these pseudo-Omnimax films almost always centered the action at the middle of the IMAX frame, which was too high to be quite comfortable in an Omnimax theater (where the "frame center" was well above the "front center" point of the theater). Graphic Films had been involved in a lot of these as well, perhaps explaining the animation reuse, but it's just as likely that they had sold it outright to the IMAX corporation which used it as they pleased.

For some reason, this version also received new audio that is mostly the same but slightly different. I don't have a definitive explanation, but I think there may have been an audio format change between the very early Omnimax theaters and later IMAX/Omnimax systems, which might have required remastering.

Later, as Omnimax domes proliferated at science museums, the IMAX Corporation (which very actively promoted Omnimax to education) gave many of these theaters custom versions of the vanity card that ended with the science museum's own logo. I have personally seen two of these, so I feel pretty confident that they exist and weren't all that rare (basically 2 out of 2 Omnimax theaters I've visited used one), but I cannot find any preserved copies.

Another recurring name in the world of IMAX and Omnimax is MacGillivray Freeman Films. MacGillivray and Freeman were a pair of teenage friends from Laguna Beach who dropped out of school in the '60s to make skateboard and surf films. This is, of course, a rather cliché start for documentary filmmakers but we must allow that it was the '60s and they were pretty much the ones creating the cliché. Their early films are hard to find in anything better than VHS rip quality, but worth watching: Wikipedia notes their significance in pioneering "action cameras," mounting 16mm cinema cameras to skateboards and surfboards, but I would say that their cinematography was innovative in more ways than just one. The 1970 "Catch the Joy," about sandrails, has some incredible shots that I struggle to explain. There's at least one where they definitely cut the shot just a couple of frames before a drifting sandrail flung their camera all the way down the dune.

For some reason, I would speculate due to their reputation for exciting cinematography, the National Air and Space Museum chose MacGillivray and Freeman for "To Fly!". While not the first science museum IMAX documentary by any means (that was, presumably, "Voyage to the Outer Planets" given the different subject matter of the various Expo films), "To Fly!" might be called the first modern one. It set the pattern that decades of science museum films followed: a film initially written by science educators, punched up by producers, and filmed with the very best technology of the time. Fearing that the film's history content would be dry, they pivoted more towards entertainment, adding jokes and action sequences. "To Fly!" was a hit, running in just about every science museum with an IMAX theater, including Omnimax.

Sadly, Jim Freeman died in a helicopter crash shortly after production. Nonetheless, MacGillivray Freeman Films went on. Over the following decades, few IMAX science documentaries were made that didn't involve them somehow. Besides the films they produced, the company consulted on action sequences in most of the format's popular features.

I had hoped to present here a thorough history of the films that were actually produced in the Omnimax format. Unfortunately, this has proven very difficult: the fact that most of them were distributed only to science museums means that they are very spottily remembered, and besides, so many of the films that ran in Omnimax theaters were converted from IMAX presentations that it's hard to tell the two apart. I'm disappointed that this part of cinema history isn't better recorded, and I'll continue to put time into the effort. Science museum documentaries don't get a lot of attention, but many of the have involved formidable technical efforts.

Consider, for example, the cameras: befitting the large film, IMAX cameras themselves are very large. When filming "To Fly!", MacGillivray and Freeman complained that the technically very basic 80 pound cameras required a lot of maintenance, were complex to operate, and wouldn't fit into the "action cam" mounting positions they were used to. The cameras were so expensive, and so rare, that they had to be far more conservative than their usual approach out of fear of damaging a camera they would not be able to replace. It turns out that they had it easy. Later IMAX science documentaries would be filmed in space ("The Dream is Alive" among others) and deep underwater ("Deep Sea 3D" among others). These IMAX cameras, modified for simpler operation and housed for such difficult environments, weighed over 1,000 pounds. Astronauts had to be trained to operate the cameras; mission specialists on Hubble service missions had wrangling a 70-pound handheld IMAX camera around the cabin and developing its film in a darkroom bag among their duties. There was a lot of film to handle: as a rule of thumb, one mile of IMAX film is good for eight and a half minutes.

I grew up in Portland, Oregon, and so we will make things a bit more approachable by focusing on one example: The Omnimax theater of the Oregon Museum of Science and Industry, which opened as part of the museum's new waterfront location in 1992. This 330-seat boasted a 10,000 sq ft dome and 15 kW of sound. The premier feature was "Ring of Fire," a volcano documentary originally commissioned by the Fleet, the Fort Worth Museum of Science and Industry, and the Science Museum of Minnesota. By the 1990s, the later era of Omnimax, the dome format was all but abandoned as a commercial concept. There were, an announcement article notes, around 90 total IMAX theaters (including Omnimax) and 80 Omnimax films (including those converted from IMAX) in '92. Considering the heavy bias towards science museums among these theaters, it was very common for the films to be funded by consortia of those museums.

Considering the high cost of filming in IMAX, a lot of the documentaries had a sort of "mashup" feel. They would combine footage taken in different times and places, often originally for other projects, into a new narrative. "Ring of Fire" was no exception, consisting of a series of sections that were sometimes more loosely connected to the theme. The 1982 Loma Prieta earthquake was a focus, and the eruption of Mt. St. Helens, and lava flows in Hawaii. Perhaps one of the reasons it's hard to catalog IMAX films is this mashup quality, many of the titles carried at science museums were something along the lines of "another ocean one." I don't mean this as a criticism, many of the IMAX documentaries were excellent, but they were necessarily composed from painstakingly gathered fragments and had to cover wide topics.

Given that I have an announcement feature piece in front of me, let's also use the example of OMSI to discuss the technical aspects. OMSI's projector cost about $2 million and weighted about two tons. To avoid dust damaging the expensive prints, the "projection room" under the seating was a positive-pressure cleanroom. This was especially important since the paucity of Omnimax content meant that many films ran regularly for years. The 15 kW water-cooled lamp required replacement at 800 to 1,000 hours, but unfortunately, the price is not noted.

By the 1990s, Omnimax had become a rare enough system that the projection technology was a major part of the appeal. OMSI's installation, like most later Omnimax theaters, had the audience queue below the seating, separated from the projection room by a glass wall. The high cost of these theaters meant that they operated on high turnovers, so patrons would wait in line to enter immediately after the previous showing had exited. While they waited, they could watch the projectionist prepare the next show while a museum docent explained the equipment.

I have written before about multi-channel audio formats, and Omnimax gives us some more to consider. The conventional audio format for much of Omnimax's life was six-channel: left rear, left screen, center screen, right screen, right rear, and top. Each channel had an independent bass cabinet (in one theater, a "caravan-sized" enclosure with eight JBL 2245H 46cm woofers), and a crossover network fed the lowest end of all six channels to a "sub-bass" array at screen bottom. The original Fleet installation also had sub-bass speakers located beneath the audience seating, although that doesn't seem to have become common.

IMAX titles of the '70s and '80s delivered audio on eight-track magnetic tape, with the additional tracks used for synchronization to the film. By the '90s, IMAX had switched to distributing digital audio on three CDs (one for each two channels). OMSI's theater was equipped for both, and the announcement amusingly notes the availability of cassette decks. A semi-custom audio processor made for IMAX, the Sonics TAC-86, managed synchronization with film playback and applied equalization curves individually calibrated to the theater.

IMAX domes used perforated aluminum screens (also the norm in later planetaria), so the speakers were placed behind the screen in the scaffold-like superstructure that supported it. When I was young, OMSI used to start presentations with a demo program that explained the large size of IMAX film before illuminating work lights behind the screen to make the speakers visible. Much of this was the work of the surprisingly sophisticated show control system employed by Omnimax theaters, a descendent of the PDP-15 originally installed in the Fleet.

Despite Omnimax's almost complete consignment to science museums, there were some efforts at bringing commercial films. Titles like Disney's "Fantasia" and "Star Wars: Episode III" were distributed to Omnimax theaters via optical reprojection, sometimes even from 35mm originals. Unfortunately, the quality of these adaptations was rarely satisfactory, and the short runtimes (and marketing and exclusivity deals) typical of major commercial releases did not always work well with science museum schedules. Still, the cost of converting an existing film to dome format is pretty low, so the practice continues today. "Star Wars: The Force Awakens," for example, ran on at least one science museum dome. This trickle of blockbusters was not enough to make commercial Omnimax theaters viable.

Caesars Palace closed, and then demolished, their Omnimax theater in 2000. The turn of the 21st century was very much the beginning of the end for the dome theater. IMAX was moving away from their film system and towards digital projection, but digital projection systems suitable for large domes were still a nascent technology and extremely expensive. The end of aggressive support from IMAX meant that filming costs became impractical for documentaries, so while some significant IMAX science museum films were made in the 2000s, the volume definitely began to lull and the overall industry moved away from IMAX in general and Omnimax especially.

It's surprising how unforeseen this was, at least to some. A ten-screen commercial theater in Duluth opened an Omnimax theater in 1996! Perhaps due to the sunk cost, it ran until 2010, not a bad closing date for an Omnimax theater. Science museums, with their relatively tight budgets and less competitive nature, did tend to hold over existing Omnimax installations well past their prime. Unfortunately, many didn't: OMSI, for example, closed its Omnimax theater in 2013 for replacement with a conventional digital theater that has a large screen but is not IMAX branded.

Fortunately, some operators hung onto their increasingly costly Omnimax domes long enough for modernization to become practical. The IMAX Corporation abandoned the Omnimax name as more of the theaters closed, but continued to support "IMAX Dome" with the introduction of a digital laser projector with spherical optics. There are only ten examples of this system. Others, including Omnimax's flagship at the Fleet Science Center, have been replaced by custom dome projection systems built by competitors like Sony.

Few Omnimax projectors remain. The Fleet, to their credit, installed the modern laser projectors in front of the projector well so that the original film projector could remain in place. It's still functional and used for reprisals of Omnimax-era documentaries. IMAX projectors in general are a dying breed, a number of them have been preserved but their complex, specialized design and the end of vendor support means that it may become infeasible to keep them operating.

We are, of course, well into the digital era. While far from inexpensive, digital projection systems are now able to match the quality of Omnimax projection. The newest dome theaters, like the Sphere, dispense with projection entirely. Instead, they use LED display panels capable of far brighter and more vivid images than projection, and with none of the complexity of water-cooled arc lamps.

Still, something has been lost. There was once a parallel theater industry, a world with none of the glamor of Hollywood but for whom James Cameron hauled a camera to the depths of the ocean and Leonardo DiCaprio narrated repairs to the Hubble. In a good few dozen science museums, two-ton behemoths rose from beneath the seats, the zenith of film projection technology. After decades of documentaries, I think people forgot how remarkable these theaters were.

Science museums stopped promoting them as aggressively, and much of the showmanship faded away. Sometime in the 2000s, OMSI stopped running the pre-show demonstration, instead starting the film directly. They stopped explaining the projectionist's work in preparing the show, and as they shifted their schedule towards direct repetition of one feature, there was less for the projectionist to do anyway. It became just another museum theater, so it's no wonder that they replaced it with just another museum theater: a generic big-screen setup with the exceptionally dull name of "Empirical Theater."

From time to time, there have been whispers of a resurgence of 70mm film. Oppenheimer, for example, was distributed to a small number of theaters in this giant of film formats: 53 reels, 11 miles, 600 pounds of film. Even conventional IMAX is too costly for the modern theater industry, though. Omnimax has fallen completely by the wayside, with the few remaining dome operators doomed to recycling the same films with a sprinkling of newer reformatted features. It is hard to imagine a collective of science museums sending another film camera to space.

Omnimax poses a preservation challenge in more ways than one. Besides the lack of documentation on Omnimax theaters and films, there are precious few photographs of Omnimax theaters and even fewer videos of their presentations. Of course, the historian suffers where Madison Square Garden hopes to succeed: the dome theater is perhaps the ultimate in location-based entertainment. Photos and videos, represented on a flat screen, cannot reproduce the experience of the Omnimax theater. The 180 horizontal degrees of screen, the sound that was always a little too loud, in no small part to mask the sound of the projector that made its own racket in the middle of the seating. You had to be there.

IMAGES: Omnimax projection room at OMSI, Flickr user truk. Omnimax dome with work lights on at MSI Chicago, Wikimedia Commons user GualdimG. Omnimax projector at St. Louis Science Center, Flickr user pasa47.

[1] I don't have extensive information on pricing, but I know that in the 1960s an "economy" Spitz came in over $30,000 (~10x that much today).

[2] Pink Floyd's landmark album Dark Side of The Moon debuted in a release event held at the London Planetarium. This connection between Pink Floyd and planetaria, apparently much disliked by the band itself, has persisted to the present day. Several generations of Pink Floyd laser shows have been licensed by science museums around the world, and must represent by far the largest success of fixed-installation laser projection.

[3] Are you starting to detect a theme with these Expos? the World's Fairs, including in their various forms as Expos, were long one of the main markets for niche film formats. Any given weird projection format you run into, there's a decent chance that it was originally developed for some short film for an Expo. Keep in mind that it's the nature of niche projection formats that they cannot easily be shown in conventional theaters, so they end up coupled to these crowd events where a custom venue can be built.

[4] The Smithsonian Institution started looking for an exciting new theater in 1970. As an example of the various niche film formats at the time, the Smithsonian considered a dome (presumably Omnimax), Cinerama (a three-projector ultrawide system), and Circle-Vision 360 (known mostly for the few surviving Expo films at Disney World's EPCOT) before settling on IMAX. The Smithsonian theater, first planned for the Smithsonian Museum of Natural History before being integrated into the new National Air and Space Museum, was tremendously influential on the broader world of science museum films. That is perhaps an understatement, it is sometimes credited with popularizing IMAX in general, and the newspaper coverage the new theater received throughout North America lends credence to the idea. It is interesting, then, to imagine how different our world would be if they had chosen Circle-Vision. "Captain America: Brave New World" in Cinemark 360.

Sometimes I think I should pivot my career to home automation critic, because I have many opinions on the state of the home automation industry---and they're pretty much all critical. Virtually every time I bring up home automation, someone says something about the superiority of the light switch. Controlling lights is one of the most obvious applications of home automation, and there is a roughly century long history of developments in light control---yet, paradoxically, it is an area where consumer home automation continues to struggle.

An analysis of how and why billion-dollar tech companies fail to master the simple toggling of lights in response to human input will have to wait for a future article, because I will have a hard time writing one without descending into incoherent sobbing about the principles of scene control and the interests of capital. Instead, I want to just dip a toe into the troubled waters of "smart lighting" by looking at one of its earliest precedents: low-voltage lighting control.

A source I generally trust, the venerable "old internet" website Inspectapedia, says that low-voltage lighting control systems date back to about 1946. The earliest conclusive evidence I can find of these systems is a newspaper ad from 1948, but let's be honest, it's a holiday and I'm only making a half effort on the research. In any case, the post-war timing is not a coincidence. The late 1940s were a period of both rapid (sub)urban expansion and high copper prices, and the original impetus for relay systems seems to have been the confluence of these two.

But let's step back and explain what a relay or low-voltage lighting control system is. First, I am not referring to "low voltage lighting" meaning lights that run on 12 or 24 volts DC or AC, as was common in landscape lighting and is increasingly common today for integrated LED lighting. Low-voltage lighting control systems are used for conventional 120VAC lights. In the most traditional construction, e.g. in the 1940s, lights would be served by a "hot" wire that passed through a wall box containing a switch. In many cases the neutral (likely shared with other fixtures) went directly from the light back to the panel, bypassing the switch... running both the hot and neutral through the switch box did not become conventional until fairly recently, to the chagrin of anyone installing switches that require a neutral for their own power, like timers or "smart" switches.

The problem with this is that it lengthens the wiring runs. If you have a ceiling fixture with two different switches in a three-way arrangement, say in a hallway in a larger house, you could be adding nearly 100' in additional wire to get the hot to the switches and the runner between them. The cost of that wiring, in the mid-century, was quite substantial. Considering how difficult it is to find an employee to unlock the Romex cage at Lowes these days, I'm not sure that's changed that much.

There are different ways of dealing with this. In the UK, the "ring main" served in part to reduce the gauge (and thus cost) of outlet wiring, but we never picked up that particular eccentricity in the US (for good reason). In commercial buildings, it's not unusual for lighting to run on 240v for similar reasons, but 240v is discouraged in US residential wiring. Besides, the mid-century was an age of optimism and ambition in electrical technology, the days of Total Electric Living. Perhaps the technology of the relay, refined by so many innovations of WWII, could offer a solution.

Switch wiring also had to run through wall cavities, an irritating requirement in single-floor houses where much of the lighting wiring could be contained to the attic. The wiring of four-way and other multi-switch arrangements could become complex and require a lot more wall runs, discouraging builders providing switches in the most convenient places. What if relays also made multiple switches significantly easier to install and relocate?

You probably get the idea. In a typical low-voltage lighting control system, a transformer provides a low voltage like 24VAC, much the same as used by doorbells. The light switches simply toggle the 24VAC control power to the coils of relays. Some (generally older) systems powered the relay continuously, but most used latching relays. In this case, all light switches are momentary, with an "on" side and an "off" side. This could be a paddle that you push up or down (much like a conventional light switch), a bar that you push the left or right sides of, or a pair of two push buttons.

In most installations, all of the relays were installed together in a single enclosure, usually in the attic where the high-voltage wiring to the actual lights would be fairly short. The 24VAC cabling to the switches was much smaller gauge, and depending on the jurisdiction might not require any sort of license to install.

Many systems had enclosures with separate high voltage and low voltage components, or mounted the relays on the outside of an enclosure such that the high voltage wiring was inside and low voltage outside. Both arrangements helped to meet code requirements for isolating high and low voltage systems and provided a margin of safety in the low voltage wiring. That provided additional cost savings as well; low voltage wiring was usually installed without any kind of conduit or sheathed cable.

By 1950, relay lighting controls were making common appearances in real estate listings. A feature piece on the "Melody House," a builder's model home, in the Tacoma News Tribune reads thus:

Newest features in the house are the low voltage touch plate and relay system lighting controls, with wide plates instead of snap buttons---operated like the stops of a pipe organ, with the merest flick of a finger.

The comparison to a pipe organ is interesting, first in its assumption that many readers were familiar with typical organ stops. Pipe organs were, increasingly, one of the technological marvels of the era: while the concept of the pipe organ is very old, this same era saw electrical control systems (replete with relays!) significantly reduce the cost and complexity of organ consoles. What's more, the tonewheel electric organ had become well-developed and started to find its way into homes.

The comparison is also interesting because of its deficiencies. The Touch-Plate system described used wide bars, which you pressed the left or right side of---you could call them momentary SPDT rocker switches if you wanted. There were organs with similar rocker stops but I do not think they were common in 1950. My experience is that such rocker switch stops usually indicate a fully digital control system, where they make momentary action unobtrusive and avoid state synchronization problems. I am far from an expert on organs, though, which is why I haven't yet written about them. If you have a guess at which type of pipe organ console our journalist was familiar with, do let me know.

Touch-Plate seems to have been one of the first manufacturers of these systems, although I can't say for sure that they invented them. Interestingly, Touch-Plate is still around today, but their badly broken WordPress site ("Welcome to the new touch-plate.com" despite it actually being touchplate.com) suggests they may not do much business. After a few pageloads their WordPress plugin WAF blocked me for "exceed[ing] the maximum number of page not found errors per minute for humans." This might be related to my frustration that none of the product images load. It seems that the Touch-Plate company has mostly pivoted to reselling imported LED lighting (touchplateled.com), so I suppose the controls business is withering on the vine.

The 1950s saw a proliferation of relay lighting control brands, with GE introducing a particularly popular system with several generations of fixtures. Kyle Switch Plates, who sell replacement switch plates (what else?), list options for Remcon, Sierra, Bryant, Pyramid, Douglas, and Enercon systems in addition to the two brands we have met so far. As someone who pays a little too much attention to light switches, I have personally seen four of these brands, three of them still in use and one apparently abandoned in place.

Now, you might be thinking that simply economizing wiring by relocating the switches does not constitute "home automation," but there are other features to consider. For one, low-voltage light control systems made it feasible to install a lot more switches. Houses originally built with them often go a little wild with the n-way switching, every room providing lightswitches at every door. But there is also the possibility of relay logic. From the same article:

The necessary switches are found in every room, but in the master bedroom there is a master control panel above the bed, from where the house and yard may be flooded with instant light in case of night emergency.

Such "master control panels" were a big attraction for relay lighting, and the finest homes of the 1950s and 1960s often displayed either a grid of buttons near the head of the master bed, or even better, a GE "Master Selector" with a curious system of rotary switches. On later systems, timers often served as auxiliary switches, so you could schedule exterior lights. With a creative installer, "scenes" were even possible by wiring switches to arbitrary sets of relays (this required DC or half-wave rectified control power and diodes to isolate the switches from each other).

Many of these relay control systems are still in use today. While they are quite outdated in a certain sense, the design is robust and the simple components mean that it's usually not difficult to find replacement parts when something does fail. The most popular system is the one offered by GE, using their RR series relays (RR3, RR4, etc., to the modern RR9). That said, GE suggests a modernization path to their LightSweep system, which is really a 0-10v analog dimming controller that has the add-on ability to operate relays.

The failure modes are mostly what you would expect: low voltage wiring can chafe and short, or the switches can become stuck. This tends to cause the lights to stick on or off, and the continuous current through the relay coil often burns it out. The fix requires finding the stuck switch or short and correcting it, and then replacing the relay.

One upside of these systems that persists today is density: the low voltage switches are small, so with most systems you can fit 3 per gang. Another is that they still make N-way switching easier. There is arguably a safety benefit, considering the reduction in mains-voltage wire runs.

Yet we rarely see such a thing installed in homes newer than around the '80s. I don't know that I can give a definitive explanation of the decline of relay lighting control, but reduced prices for copper wiring were probably a main factor. The relays added a failure point, which might lead to a perception of unreliability, and the declining familiarity of electricians means that installing a relay system could be expensive and frustrating today.

What really interests me about relay systems is that they weren't really replaced... the idea just went away. It's not like modern homes are providing a master control panel in the bedroom using some alternative technology. I mean, some do, those with prices in the eight digits, but you'll hardly ever see it.

That gets us to the tension between residential lighting and architectural lighting control systems. In higher-end commercial buildings, and in environments like conference rooms and lecture halls, there's a well established industry building digital lighting control systems. Today, DALI is a common standard for the actual lighting control, but if you look at a range of existing buildings you will find everything from completely proprietary digital distributed dimming to 0-10v analog dimming to central dimmer racks (similar to traditional theatrical lighting).

Relay lighting systems were, in a way, a nascent version of residential architectural lighting control. And the architectural lighting control industry continues to evolve. If there is a modern equivalent to relay lighting, it's something like Lutron QSX. That's a proprietary digital lighting (and shade) control system, marketed for both residential and commercial use. QSX offers a wide range of attractive wall controls, tight integration to Lutron's HomeSense home automation platform, and a price tag that'll make your eyes water. Lutron has produced many generations of these systems, and you could make an argument that they trace their heritage back to the relay systems of the 1940s. But they're just priced way beyond the middle-class home.

And, well, I suppose that requires an argument based on economics. Prices have gone up. Despite tract construction being a much older idea than people often realize, it seems clear that today's new construction homes have been "value engineered" to significantly lower feature and quality levels than those of the mid-century---but they're a lot bigger. There is a sort of maxim that today's home buyers don't care about anything but square footage, and if you've seen what Pulte or D. R. Horton are putting up... well, I never knew that 3,000 sq ft could come so cheap, and look it too.

Modern new-construction homes just don't come with the gizmos that older ones did, especially in the '60s and '70s. Looking at the sales brochure for a new development in my own Albuquerque ("Estates at La Cuentista"), besides 21st century suburbanization (Gated Community! "East Access to Paseo del Norte" as if that's a good thing!) most of the advertised features are "big." I'm serious! If you look at the "More Innovation Built In" section, the "innovations" are a home office (more square footage), storage (more square footage), indoor and outdoor gathering spaces (to be fair, only the indoor ones are square footage), "dedicated learning areas" for kids (more square footage), and a "basement or bigger garage" for a home gym (more square footage). The only thing in the entire innovation section that I would call a "technical" feature is water filtration. You can scroll down for more details, and you get to things like "space for a movie room" and a finished basement described eight different ways.

Things were different during the peak of relay lighting in the '60s. A house might only be 1,600 sq ft, but the builder would deck it out with an intercom (including multi-room audio of a primitive sort), burglar alarm, and yes, relay lighting. All of these technologies were a lot newer and people were more excited about them; I bring up Total Electric Living a lot because of an aesthetic obsession but it was a large-scale advertising and partnership campaign by the electrical industry (particularly Westinghouse) that gave builders additional cross-promotion if they included all of these bells and whistles.

Remember, that was when people were watching those old videos about the "kitchen of the future." What would a 2025 "Kitchen of the Future" promotional film emphasize? An island bigger than my living room and a nook for every meal, I assume. Features like intercoms and even burglar alarms have become far less common in new construction, and even if they were present I don't think most buyers would use them.

But that might seem a little odd, right, given the push towards home automation? Well, built-in home automation options have existed for longer than any of today's consumer solutions, but "built in" is a liability for a technology product. There are practical reasons, in that built-in equipment is harder to replace, but there's also a lamer commercial reason. Consumer technology companies want to sell their products like consumer technology, so they've recontextualized lighting control as "IoT" and "smart" and "AI" rather than something an electrician would hook up.

While I was looking into relay lighting control systems, I ran into an interesting example. The Lutron Lu Master Lumi 5. What a name! Lutron loves naming things like this. The Lumi 5 is a 1980s era product with essentially the same features as a relay system, but architected in a much stranger way. It is, essentially, five three way switches in a box with remote controls. That means that each of the actual light switches in the house (which could also be dimmers) need mains-voltage wiring, including runner, back to the Lumi 5 "interface."

Pressing a button on one of the Lutron wall panels toggles the state of the relay in the "interface" cabinet, toggling the light. But, since it's all wired as a three-way switch, toggling the physical switch at the light does the same thing. As is typical when combining n-way switches and dimming, the Lumi 5 has no control over dimmers. You can only dim a light up or down at the actual local control, the Lumi 5 can just toggle the dimmer on and off using the 3-way runner. The architecture also means that you have two fundamentally different types of wall panels in your house: local switches or dimmers wired to each light, and the Lu Master panels with their five buttons for the five circuits, along with "all on" and "all off."

The Lumi 5 "interface" uses simple relay logic to implement a few more features. Five mains-voltage-level inputs can be wired to time clocks, so that you can schedule any combination(s) of the circuits to turn on and off. The manual recommends models including one with an astronomical clock for sunrise/sunset. An additional input causes all five circuits to turn on; it's suggested for connection to an auxiliary relay on a burglar alarm to turn all of the lights on should the alarm be triggered.

The whole thing is strange and fascinating. It is basically a relay lighting control system, like so many before it, but using a distinctly different wiring convention. I think the main reason for the odd wiring was to accommodate dimmers, an increasingly popular option in the 1980s that relay systems could never really contend with. It doesn't have the cost advantages of relay systems at all, it will definitely be more expensive! But it adds some features over the fancy Lutron switches and dimmers you were going to install anyway.

The Lu Master is the transitional stage between relay lighting systems and later architectural lighting controls, and it straddled too the end of relay light control in homes. It gives an idea of where relay light control in homes would have evolved, had the whole technology not been doomed to the niche zone of conference centers and universities.

If you think about it, the Lu Master fills the most fundamental roles of home automation in lighting: control over multiple lights in a convenient place, scheduling and triggers, and an emergency function. It only lacks scenes, which I think we can excuse considering that the simple technology it uses does not allow it to adjust dimmers. And all of that with no Node-RED in sight!

Maybe that conveys what most frustrates me about the "home automation" industry: it is constantly reinventing the wheel, an oligopoly of tech companies trying to drag people's homes into their "ecosystem." They do so by leveraging the buzzword of the moment, IoT to voice assistants to, I guess now AI?, to solve a basic set of problems that were pretty well solved at least as early as 1948.

That's not to deny that modern home automation platforms have features that old ones don't. They are capable of incredibly sophisticated things! But realistically, most of their users want only very basic functionality: control in convenient places, basic automation, scenes. It wouldn't sting so much if all these whiz-bang general purpose computers were good at those tasks, but they aren't. For the very most basic tasks, things like turning on and off a group of lights, major tech ecosystems like HomeKit provide a user experience that is significantly worse than the model home of 1950.

You could install a Lutron system, and it would solve those fundamental tasks much better... for a much higher price. But it's not like Lutron uses all that money to be an absolute technical powerhouse, a center of innovation at the cutting edge. No, even the latest Lutron products are really very simple, technically. The technical leaders here, Google, Apple, are the companies that can't figure out how to make a damn light switch.

The problem with modern home automation platforms is that they are too ambitious. They are trying to apply enormously complex systems to very simple tasks, and thus contaminating the simplest of electrical systems with all the convenience and ease of a Smart TV.

Sometimes that's what it feels like this whole industry is doing: adding complexity while the core decays. From automatic programming to AI coding agents, video terminals to Electron, the scope of the possible expands while the fundamentals become more and more irritating.

But back to the real point, I hope you learned about some cool light switches. Check out the Kyle Switch Plates reference and you'll start seeing these buildings and homes, at least if you live in an area that built up during the era that they were common (1950s to the 1970s).

Air traffic control has been in the news lately, on account of my country's declining ability to do it. Well, that's a long-term trend, resulting from decades of under-investment, severe capture by our increasingly incompetent defense-industrial complex, no small degree of management incompetence in the FAA, and long-lasting effects of Reagan crushing the PATCO strike. But that's just my opinion, you know, maybe airplanes got too woke. In any case, it's an interesting time to consider how weird parts of air traffic control are. The technical, administrative, and social aspects of ATC all seem two notches more complicated than you would expect. ATC is heavily influenced by its peculiar and often accidental development, a product of necessity that perpetually trails behind the need, and a beneficiary of hand-me-down military practices and technology.

Aviation Radio

In the early days of aviation, there was little need for ATC---there just weren't many planes, and technology didn't allow ground-based controllers to do much of value. There was some use of flags and signal lights to clear aircraft to land, but for the most part ATC had to wait for the development of aviation radio. The impetus for that work came mostly from the First World War.

Here we have to note that the history of aviation is very closely intertwined with the history of warfare. Aviation technology has always rapidly advanced during major conflicts, and as we will see, ATC is no exception.

By 1913, the US Army Signal Corps was experimenting with the use of radio to communicate with aircraft. This was pretty early in radio technology, and the aircraft radios were huge and awkward to operate, but it was also early in aviation and "huge and awkward to operate" could be similarly applied to the aircraft of the day. Even so, radio had obvious potential in aviation. The first military application for aircraft was reconnaissance. Pilots could fly past the front to find artillery positions and otherwise provide useful information, and then return with maps. Well, even better than returning with a map was providing the information in real-time, and by the end of the war medium-frequency AM radios were well developed for aircraft.

Radios in aircraft led naturally to another wartime innovation: ground control. Military personnel on the ground used radio to coordinate the schedules and routes of reconnaissance planes, and later to inform on the positions of fighters and other enemy assets. Without any real way to know where the planes were, this was all pretty primitive, but it set the basic pattern that people on the ground could keep track of aircraft and provide useful information.

Post-war, civil aviation rapidly advanced. The early 1920s saw numerous commercial airlines adopting radio, mostly for business purposes like schedule coordination. Once you were in contact with someone on the ground, though, it was only logical to ask about weather and conditions. Many of our modern practices like weather briefings, flight plans, and route clearances originated as more or less formal practices within individual airlines.

Air Mail

The government was not left out of the action. The Post Office operated what may have been the largest commercial aviation operation in the world during the early 1920s, in the form of Air Mail. The Post Office itself did not have any aircraft; all of the flying was contracted out---initially to the Army Air Service, and later to a long list of regional airlines. Air Mail was considered a high priority by the Post Office and proved very popular with the public. When the transcontinental route began proper operation in 1920, it became possible to get a letter from New York City to San Francisco in just 33 hours by transferring it between airplanes in a nearly non-stop relay race.

The Post Office's largesse in contracting the service to private operators provided not only the funding but the very motivation for much of our modern aviation industry. Air travel was not very popular at the time, being loud and uncomfortable, but the mail didn't complain. The many contract mail carriers of the 1920s grew and consolidated into what are now some of the United States' largest companies. For around a decade, the Post Office almost singlehandedly bankrolled civil aviation, and passengers were a side hustle [1].

Air mail ambition was not only of economic benefit. Air mail routes were often longer and more challenging than commercial passenger routes. Transcontinental service required regular flights through sparsely populated parts of the interior, challenging the navigation technology of the time and making rescue of downed pilots a major concern. Notably, air mail operators did far more nighttime flying than any other commercial aviation in the 1920s. The post office became the government's de facto technical leader in civil aviation. Besides the network of beacons and markers built to guide air mail between cities, the post office built 17 Air Mail Radio Stations along the transcontinental route.

The Air Mail Radio Stations were the company radio system for the entire air mail enterprise, and the closest thing to a nationwide, public air traffic control service to then exist. They did not, however, provide what we would now call control. Their role was mainly to provide pilots with information (including, critically, weather reports) and to keep loose tabs on air mail flights so that a disappearance would be noticed in time to send search and rescue.

In 1926, the Watres Act created the Aeronautic Branch of the Department of Commerce. The Aeronautic Branch assumed a number of responsibilities, but one of them was the maintenance of the Air Mail routes. Similarly, the Air Mail Radio Stations became Aeronautics Branch facilities, and took on the new name of Flight Service Stations. No longer just for the contract mail carriers, the Flight Service Stations made up a nationwide network of government-provided services to aviators. They were the first edifices in what we now call the National Airspace System (NAS): a complex combination of physical facilities, technologies, and operating practices that enable safe aviation.

In 1935, the first en-route air traffic control center opened, a facility in Newark owned by a group of airlines. The Aeronautic Branch, since renamed the Bureau of Air Commerce, supported the airlines in developing this new concept of en-route control that used radio communications and paperwork to track which aircraft were in which airways. The rising number of commercial aircraft made in-air collisions a bigger problem, so the Newark control center was quickly followed by more facilities built on the same pattern. In 1936, the Bureau of Air Commerce took ownership of these centers, and ATC became a government function alongside the advisory and safety services provided by the flight service stations.

En route center controllers worked off of position reports from pilots via radio, but needed a way to visualize and track aircraft's positions and their intended flight paths. Several techniques helped: first, airlines shared their flight planning paperwork with the control centers, establishing "flight plans" that corresponded to each aircraft in the sky. Controllers adopted a work aid called a "flight strip," a small piece of paper with the key information about an aircraft's identity and flight plan that could easily be handed between stations. By arranging the flight strips on display boards full of slots, controllers could visualize the ordering of aircraft in terms of altitude and airway.

Second, each center was equipped with a large plotting table map where controllers pushed markers around to correspond to the position reports from aircraft. A small flag on each marker gave the flight number, so it could easily be correlated to a flight strip on one of the boards mounted around the plotting table. This basic concept of air traffic control, of a flight strip and a position marker, is still in use today.

Radar

The Second World War changed aviation more than any other event of history. Among the many advancements were two British inventions of particular significance: first, the jet engine, which would make modern passenger airliners practical. Second, the radar, and more specifically the magnetron. This was a development of such significance that the British government treated it as a secret akin to nuclear weapons; indeed, the UK effectively traded radar technology to the US in exchange for participation in US nuclear weapons research.

Radar created radical new possibilities for air defense, and complimented previous air defense development in Britain. During WWI, the organization tasked with defending London from aerial attack had developed a method called "ground-controlled interception" or GCI. Under GCI, ground-based observers identify possible targets and then direct attack aircraft towards them via radio. The advent of radar made GCI tremendously more powerful, allowing a relatively small number of radar-assisted air defense centers to monitor for inbound attack and then direct defenders with real-time vectors.

In the first implementation, radar stations reported contacts via telephone to "filter centers" that correlated tracks from separate radars to create a unified view of the airspace---drawn in grease pencil on a preprinted map. Filter center staff took radar and visual reports and updated the map by moving the marks. This consolidated information was then provided to air defense bases, once again by telephone.

Later technical developments in the UK made the process more automated. The invention of the "plan position indicator" or PPI, the type of radar scope we are all familiar with today, made the radar far easier to operate and interpret. Radar sets that automatically swept over 360 degrees allowed each radar station to see all activity in its area, rather than just aircraft passing through a defensive line. These new capabilities eliminated the need for much of the manual work: radar stations could see attacking aircraft and defending aircraft on one PPI, and communicated directly with defenders by radio.

It became routine for a radar operator to give a pilot navigation vectors by radio, based on real-time observation of the pilot's position and heading. A controller took strategic command of the airspace, effectively steering the aircraft from a top-down view. The ease and efficiency of this workflow was a significant factor in the end of the Battle of Britain, and its remarkable efficacy was noticed in the US as well.

At the same time, changes were afoot in the US. WWII was tremendously disruptive to civil aviation; while aviation technology rapidly advanced due to wartime needs those same pressing demands lead to a slowdown in nonmilitary activity. A heavy volume of military logistics flights and flight training, as well as growing concerns about defending the US from an invasion, meant that ATC was still a priority. A reorganization of the Bureau of Air Commerce replaced it with the Civil Aeronautics Authority, or CAA. The CAA's role greatly expanded as it assumed responsibility for airport control towers and commissioned new en route centers.

As WWII came to a close, CAA en route control centers began to adopt GCI techniques. By 1955, the name Air Route Traffic Control Center (ARTCC) had been adopted for en route centers and the first air surveillance radars were installed. In a radar-equipped ARTCC, the map where controllers pushed markers around was replaced with a large tabletop PPI built to a Navy design. The controllers still pushed markers around to track the identities of aircraft, but they moved them based on their corresponding radar "blips" instead of radio position reports.

Air Defense

After WWII, post-war prosperity and wartime technology like the jet engine lead to huge growth in commercial aviation. During the 1950s, radar was adopted by more and more ATC facilities (both "terminal" at airports and "en route" at ARTCCs), but there were few major changes in ATC procedure. With more and more planes in the air, tracking flight plans and their corresponding positions became labor intensive and error-prone. A particular problem was the increasing range and speed of aircraft, and corresponding longer passenger flights, that meant that many aircraft passed from the territory of one ARTCC into another. This required that controllers "hand off" the aircraft, informing the "next" ARTCC of the flight plan and position at which the aircraft would enter their airspace.

In 1956, 128 people died in a mid-air collision of two commercial airliners over the Grand Canyon. In 1958, 49 people died when a military fighter struck a commercial airliner over Nevada. These were not the only such incidents in the mid-1950s, and public trust in aviation started to decline. Something had to be done. First, in 1958 the CAA gave way to the Federal Aviation Administration. This was more than just a name change: the FAA's authority was greatly increased compared to the CAA, most notably by granting it authority over military aviation.

This is a difficult topic to explain succinctly, so I will only give broad strokes. Prior to 1958, military aviation was completely distinct from civil aviation, with no coordination and often no communication at all between the two. This was, of course, a factor in the 1958 collision. Further, the 1956 collision, while it did not involve the military, did result in part from communications issues between separate distinct CAA facilities and the airline's own control facilities. After 1958, ATC was completely unified into one organization, the FAA, which assumed the work of the military controllers of the time and some of the role of the airlines. The military continues to have its own air controllers to this day, and military aircraft continue to include privileges such as (practical but not legal) exemption from transponder requirements, but military flights over the US are still beholden to the same ATC as civil flights. Some exceptions apply, void where prohibited, etc.

The FAA's suddenly increased scope only made the practical challenges of ATC more difficult, and commercial aviation numbers continued to rise. As soon as the FAA was formed, it was understood that there needed to be major investments in improving the National Airspace System. While the first couple of years were dominated by the transition, the FAA's second director (Najeeb Halaby) prepared two lengthy reports examining the situation and recommending improvements. One of these, the Beacon report (also called Project Beacon), specifically addressed ATC. The Beacon report's recommendations included massive expansion of radar-based control (called "positive control" because of the controller's access to real-time feedback on aircraft movements) and new control procedures for airways and airports. Even better, for our purposes, it recommended the adoption of general-purpose computers and software to automate ATC functions.

Meanwhile, the Cold War was heating up. US air defense, a minor concern in the few short years after WWII, became a higher priority than ever before. The Soviet Union had long-range aircraft capable of reaching the United States, and nuclear weapons meant that only a few such aircraft had to make it to cause massive destruction. Considering the vast size of the United States (and, considering the new unified air defense command between the United States and Canada, all of North America) made this a formidable challenge.

During the 1950s, the newly minted Air Force worked closely with MIT's Lincoln Laboratory (an important center of radar research) and IBM to design a computerized, integrated, networked system for GCI. When the Air Force committed to purchasing the system, it was christened the Semi-Automated Ground Environment, or SAGE. SAGE is a critical juncture in the history of the computer and computer communications, the first system to demonstrate many parts of modern computer technology and, moreover, perhaps the first large-scale computer system of any kind.

SAGE is an expansive topic that I will not take on here; I'm sure it will be the focus of a future article but it's a pretty well-known and well-covered topic. I have not so far felt like I had much new to contribute, despite it being the first item on my "list of topics" for the last five years. But one of the things I want to tell you about SAGE, that is perhaps not so well known, is that SAGE was not used for ATC. SAGE was a purely military system. It was commissioned by the Air Force, and its numerous operating facilities (called "direction centers") were located on Air Force bases along with the interceptor forces they would direct.

However, there was obvious overlap between the functionality of SAGE and the needs of ATC. SAGE direction centers continuously received tracks from remote data sites using modems over leased telephone lines, and automatically correlated multiple radar tracks to a single aircraft. Once an operator entered information about an aircraft, SAGE stored that information for retrieval by other radar operators. When an aircraft with associated data passed from the territory of one direction center to another, the aircraft's position and related information were automatically transmitted to the next direction center by modem.

One of the key demands of air defense is the identification of aircraft---any unknown track might be routine commercial activity, or it could be an inbound attack. The air defense command received flight plan data on commercial flights (and more broadly all flights entering North America) from the FAA and entered them into SAGE, allowing radar operators to retrieve "flight strip" data on any aircraft on their scope.

Recognizing this interconnection with ATC, as soon as SAGE direction centers were being installed the Air Force started work on an upgrade called SAGE Air Traffic Integration, or SATIN. SATIN would extend SAGE to serve the ATC use-case as well, providing SAGE consoles directly in ARTCCs and enhancing SAGE to perform non-military safety functions like conflict warning and forward projection of flight plans for scheduling. Flight strips would be replaced by teletype output, and in general made less necessary by the computer's ability to filter the radar scope.

Experimental trial installations were made, and the FAA participated readily in the research efforts. Enhancement of SAGE to meet ATC requirements seemed likely to meet the Beacon report's recommendations and radically improve ARTCC operations, sooner and cheaper than development of an FAA-specific system.

As it happened, well, it didn't happen. SATIN became interconnected with another planned SAGE upgrade to the Super Combat Centers (SCC), deep underground combat command centers with greatly enhanced SAGE computer equipment. SATIN and SCC planners were so confident that the last three Air Defense Sectors scheduled for SAGE installation, including my own Albuquerque, were delayed under the assumption that the improved SATIN/SCC equipment should be installed instead of the soon-obsolete original system. SCC cost estimates ballooned, and the program's ambitions were reduced month by month until it was canceled entirely in 1960. Albuquerque never got a SAGE installation, and the Albuquerque air defense sector was eliminated by reorganization later in 1960 anyway.

Flight Service Stations

Remember those Flight Service Stations, the ones that were originally built by the Post Office? One of the oddities of ATC is that they never went away. FSS were transferred to the CAB, to the CAA, and then to the FAA. During the 1930s and 1940s many more were built, expanding coverage across much of the country.

Throughout the development of ATC, the FSS remained responsible for non-control functions like weather briefing and flight plan management. Because aircraft operating under instrument flight rules must closely comply with ATC, the involvement of FSS in IFR flights is very limited, and FSS mostly serve VFR traffic.

As ATC became common, the FSS gained a new and somewhat odd role: playing go-between for ATC. FSS were more numerous and often located in sparser areas between cities (while ATC facilities tended to be in cities), so especially in the mid-century, pilots were more likely to be able to reach an FSS than ATC. It was, for a time, routine for FSS to relay instructions between pilots and controllers. This is still done today, although improved communications have made the need much less common.

As weather dissemination improved (another topic for a future post), FSS gained access to extensive weather conditions and forecasting information from the Weather Service. This connectivity is bidirectional; during the midcentury FSS not only received weather forecasts by teletype but transmitted pilot reports of weather conditions back to the Weather Service. Today these communications have, of course, been computerized, although the legacy teletype format doggedly persists.

There has always been an odd schism between the FSS and ATC: they are operated by different departments, out of different facilities, with different functions and operating practices. In 2005, the FAA cut costs by privatizing the FSS function entirely. Flight service is now operated by Leidos, one of the largest government contractors. All FSS operations have been centralized to one facility that communicates via remote radio sites.

While flight service is still available, increasing automation has made the stations far less important, and the general perception is that flight service is in its last years. Last I looked, Leidos was not hiring for flight service and the expectation was that they would never hire again, retiring the service along with its staff.

Flight service does maintain one of my favorite internet phenomenon, the phone number domain name: 1800wxbrief.com. One of the odd manifestations of the FSS/ATC schism and the FAA's very partial privatization is that Leidos maintains an online aviation weather portal that is separate from, and competes with, the Weather Service's aviationweather.gov. Since Flight Service traditionally has the responsibility for weather briefings, it is honestly unclear to what extent Leidos vs. the National Weather Service should be investing in aviation weather information services. For its part, the FAA seems to consider aviationweather.gov the official source, while it pays for 1800wxbrief.com. There's also weathercams.faa.gov, which duplicates a very large portion (maybe all?) of the weather information on Leidos's portal and some of the NWS's. It's just one of those things. Or three of those things, rather. Speaking of duplication due to poor planning...

The National Airspace System

Left in the lurch by the Air Force, the FAA launched its own program for ATC automation. While the Air Force was deploying SAGE, the FAA had mostly been waiting, and various ARTCCs had adopted a hodgepodge of methods ranging from one-off computer systems to completely paper-based tracking. By 1960 radar was ubiquitous, but different radar systems were used at different facilities, and correlation between radar contacts and flight plans was completely manual. The FAA needed something better, and with growing congressional support for ATC modernization, they had the money to fund what they called National Airspace System En Route Stage A.

Further bolstering historical confusion between SAGE and ATC, the FAA decided on a practical, if ironic, solution: buy their own SAGE.

In an upcoming article, we'll learn about the FAA's first fully integrated computerized air traffic control system. While the failed detour through SATIN delayed the development of this system, the nearly decade-long delay between the design of SAGE and the FAA's contract allowed significant technical improvements. This "New SAGE," while directly based on SAGE at a functional level, used later off-the-shelf computer equipment including the IBM System/360, giving it far more resemblance to our modern world of computing than SAGE with its enormous, bespoke AN/FSQ-7.

And we're still dealing with the consequences today!

[1] It also laid the groundwork for the consolidation of the industry, with a 1930 decision that took air mail contracts away from most of the smaller companies and awarded them instead to the precursors of United, TWA, and American Airlines.

You know sometimes a technology just sort of... comes and goes? Without leaving much of an impression? And then gets lodged in your brain for the next decade? Let's talk about one of those: the iBeacon.

I think the reason that iBeacons loom so large in my memory is that the technology was announced at WWDC in 2013. Picture yourself in 2013: Steve Jobs had only died a couple of years ago, Apple was still widely viewed as a visionary leader in consumer technology, and WWDC was still happening. Back then, pretty much anything announced at an Apple event was a Big Deal that got Big Coverage. Even, it turns out, if it was a minor development for a niche application. That's the iBeacon, a specific solution to a specific problem. It's not really that interesting, but the valance of it's Apple origin makes it seem cool?

iBeacon Technology

Let's start out with what iBeacon is, as it's so simple as to be underwhelming. Way back in the '00s, a group of vendors developed a sort of "Diet Bluetooth": a wireless protocol that was directly based on Bluetooth but simplified and optimized for low-power, low-data-rate devices. This went through an unfortunate series of names, including the delightful Wibree, but eventually settled on Bluetooth Low Energy (BLE). BLE is not just lower-power, but also easier to implement, so it shows up in all kinds of smart devices today. Back in 2011, it was quite new, and Apple was one of the first vendors to adopt it.

BLE is far less connection-oriented than regular Bluetooth; you may have noticed that BLE devices are often used entirely without conventional "pairing." A lot of typical BLE profiles involve just broadcasting some data into the void for any device that cares (and is in short range) to receive, which is pretty similar to ANT+ and unsurprisingly appears in ANT+-like applications of fitness monitors and other sensors. Of course, despite the simpler association model, BLE applications need some way to find devices, so BLE provides an advertising mechanism in which devices transmit their identifying info at regular intervals.

And that's all iBeacon really is: a standard for very simple BLE devices that do nothing but transmit advertisements with a unique ID as the payload. Add a type field on the advertising packet to specify that the device is trying to be an iBeacon and you're done. You interact with an iBeacon by receiving its advertisements, so you know that you are near it. Any BLE device with advertisements enabled could be used this way, but iBeacons are built only for this purpose.

The applications for iBeacon are pretty much defined by its implementation in iOS; there's not much of a standard even if only for the reason that there's not much to put in a standard. It's all obvious. iOS provides two principle APIs for working with iBeacons: the region monitoring API allows an app to determine if it is near an iBeacon, including registering the region so that the app will be started when the iBeacon enters range. This is useful for apps that want to do something in response to the user being in a specific location.

The ranging API allows an app to get a list of all of the nearby iBeacons and a rough range from the device to the iBeacon. iBeacons can actually operate at substantial ranges---up to hundreds of meters for more powerful beacons with external power, so ranging mode can potentially be used as sort of a lightweight local positioning system to estimate the location of the user within a larger space.

iBeacon IDs are in the format of a UUID, followed by a "major" number and a "minor" number. There are different ways that these get used, especially if you are buying cheap iBeacons and not reconfiguring them, but the general idea is roughly that the UUID identifies the operator, the major a deployment, and the minor a beacon within the deployment. In practice this might be less common than just every beacon having its own UUID due to how they're sourced. It would be interesting to survey iBeacon applications to see which they do.

Promoted Applications

So where do you actually use these? Retail! Apple seems to have designed the iBeacon pretty much exclusively for "proximity marketing" applications in the retail environment. It goes something like this: when you're in a store and open that store's app, the app will know what beacons you are nearby and display relevant content. For example, in a grocery store, the grocer's app might offer e-coupons for cosmetics when you are in the cosmetics section.

That's, uhh, kind of the whole thing? The imagined universe of applications around the launch of iBeacon was pretty underwhelming to me, even at the time, and it still seems that way. That's presumably why iBeacon had so little success in consumer-facing applications. You might wonder, who actually used iBeacons?

Well, Apple did, obviously. During 2013 and into 2014 iBeacons were installed in all US Apple stores, and prompted the Apple Store app to send notifications about upgrade offers and other in-store deals. Unsurprisingly, this Apple Store implementation was considered the flagship deployment. It generated a fair amount of press, including speculation as to whether or not it would prove the concept for other buyers.

Around the same time, Apple penned a deal with Major League Baseball that would see iBeacons installed in MLB stadiums. For the 2014 season, MLB Advanced Marketing, a joint venture of team owners, had installed iBeacon technology in 20 stadiums.

Baseball fans will be able to utilize iBeacon technology within MLB.com At The Ballpark when the award-winning app's 2014 update is released for Opening Day. Complete details on new features being developed by MLBAM for At The Ballpark, including iBeacon capabilities, will be available in March.

What's the point? the iBeacons "enable the At The Ballpark app to play specific videos or offer coupons."

This exact story repeats for other retail companies that have picked the technology up at various points, including giants like Target and WalMart. The iBeacons are simply a way to target advertising based on location, with better indoor precision and lower power consumption than GPS. Aiding these applications along, Apple integrated iBeacon support into the iOS location framework and further blurred the lines between iBeacon and other positioning services by introducing location-based-advertising features that operated on geofencing alone.

Some creative thinkers did develop more complex applications for the iBeacon. One of the early adopters was a company called Exact Editions, which prepared the Apple Newsstand version of a number of major magazines back when "readable on iPad" was thought to be the future of print media. Exact Editions explored a "read for free" feature where partner magazines would be freely accessible to users at partnering locations like coffee shops and book stores. This does not seem to have been a success, but using the proximity of an iBeacon to unlock some paywalled media is at least a little creative, if probably ill-advised considering security considerations we'll discuss later.

The world of applications raises interesting questions about the other half of the mobile ecosystem: how did this all work on Android? iOS has built-in support for iBeacons. An operating system service scans for iBeacons and dispatches notifications to apps as appropriate. On Android, there has never been this type of OS-level support, but Android apps have access to relatively rich low-level Bluetooth functionality and can easily scan for iBeacons themselves. Several popular libraries exist for this purpose, and it's not unusual for them to be used to give ported cross-platform apps more or less equivalent functionality. These apps do need to run in the background if they're to notify the user proactively, but especially back in 2013 Android was far more generous about background work than iOS.

iBeacons found expanded success through ShopKick, a retail loyalty platform that installed iBeacons in locations of some major retailers like American Eagle. These powered location-based advertising and offers in the ShopKick app as well as retailer-specific apps, which is kind of the start of a larger, more seamless network, but it doesn't seem to have caught on. Honestly, consumers just don't seem to want location-based advertising that much. Maybe because, when you're standing in an American Eagle, getting ads for products carried in the American Eagle is inane and irritating. iBeacons sort of foresaw cooler screens in this regard.

To be completely honest, I'm skeptical that anyone ever really believed in the location-based advertising thing. I mean, I don't know, the advertising industry is pretty good at self-deception, but I don't think there were ever any real signs of hyper-local smartphone-based advertising taking off. I think the play was always data collection, and advertising and special offers just provided a convenient cover story.

Real Applications

iBeacons are one of those technologies that feels like a flop from a consumer perspective but has, in actuality, enjoyed surprisingly widespread deployments. The reason, of course, is data mining.

To Apple's credit, they took a set of precautions in the design of the iBeacon iOS features that probably felt sufficient in 2013. Despite the fact that a lot of journalists described iBeacons as being used to "notify a user to install an app," that was never actually a capability (a very similar-seeming iOS feature attached to Siri actually used conventional geofencing rather than iBeacons). iBeacons only did anything if the user already had an app installed that either scanned for iBeacons when in the foreground or registered for region notifications.

In theory, this limited iBeacons to companies with which consumers already had some kind of relationship. What Apple may not have foreseen, or perhaps simply accepted, is the incredible willingness of your typical consumer brand to sell that relationship to anyone who would pay.

iBeacons became, in practice, just another major advancement in pervasive consumer surveillance. The New York Times reported in 2019 that popular applications were including SDKs that reported iBeacon contacts to third-party consumer data brokers. This data became one of several streams that was used to sell consumer location history to advertisers.

It's a little difficult to assign blame and credit, here. Apple, to their credit, kept iBeacon features in iOS relatively locked down. This suggests that they weren't trying to facilitate massive location surveillance. That said, Apple always marketed iBeacon to developers based on exactly this kind of consumer tracking and micro-targeting, they just intended for it to be done under the auspices of a single brand. That industry would obviously form data exchanges and recruit random apps into reporting everything in your proximity isn't surprising, but maybe Apple failed to foresee it.

They certainly weren't the worst offender. Apple's promotion of iBeacon opened the floodgates for everyone else to do the same thing. During 2014 and 2015, Facebook started offering bluetooth beacons to businesses that were ostensibly supposed to facilitate in-app special offers (though I'm not sure that those ever really materialized) but were pretty transparently just a location data collection play.

Google jumped into the fray in their Signature Google style, with an offering that was confusing, semi-secret, incoherently marketed, and short lived. Google's Project Beacon, or Google My Business, also shipped free Bluetooth beacons out to businesses to give Android location services a boost. Google My Business seems to have been the source of a fair amount of confusion even at the time, and we can virtually guarantee that (as reporters speculated at the time) Google was intentionally vague and evasive about the system to avoid negative attention from privacy advocates.

In the case of Facebook, well, they don't have the level of opsec that Google does so things are a little better documented:

Leaked documents show that Facebook worried that users would 'freak out' and spread 'negative memes' about the program. The company recently removed the Facebook Bluetooth beacons section from their website.

The real deployment of iBeacons and closely related third-party iBeacon-like products [1] occurred at massive scale but largely in secret. It became yet another dark project of the advertising-industrial complex, perhaps the most successful yet of a long-running series of retail consumer surveillance systems.

Payments

One interesting thing about iBeacon is how it was compared to NFC. The two really aren't that similar, especially considering the vast difference in usable ranges, but NFC was the first radio technology to be adopted for "location marketing" applications. "Tap your phone to see our menu," kinds of things. Back in 2013, Apple had rather notably not implemented NFC in its products, despite its increasing adoption on Android.

But, there is much more to this story than learning about new iPads and getting a surprise notification that you are eligible for a subsidized iPhone upgrade. What we're seeing is Apple pioneering the way mobile devices can be utilized to make shopping a better experience for consumers. What we're seeing is Apple putting its money where its mouth is when it decided not to support NFC. (MacObserver)

Some commentators viewed iBeacon as Apple's response to NFC, and I think there's more to that than you might think. In early marketing, Apple kept positioning iBeacon for payments. That's a little weird, right, because iBeacons are a purely one-way broadcast system.

Still, part of Apple's flagship iBeacon implementation was a payment system:

Here's how he describes the purchase he made there, using his iPhone and the EasyPay system: "We started by using the iPhone to scan the product barcode and then we had to enter our Apple ID, pretty much the way we would for any online Apple purchase [using the credit card data on file with one's Apple account]. The one key difference was that this transaction ended with a digital receipt, one that we could show to a clerk if anyone stopped us on the way out."

Apple Wallet only kinda-sorta existed at the time, although Apple was clearly already midway into a project to expand into consumer payments. It says a lot about this point in time in phone-based payments that several reporters talk about iBeacon payments as a feature of iTunes, since Apple was mostly implementing general-purpose billing by bolting it onto iTunes accounts.

It seems like what happened is that Apple committed to developing a pay-by-phone solution, but decided against NFC. To be competitive with other entrants in the pay-by-phone market, they had to come up with some kind of technical solution to interact with retail POS, and iBeacon was their choice. From a modern perspective this seems outright insane; like, Bluetooth broadcasts are obviously not the right way to initiate a payment flow, and besides, there's a whole industry-standard stack dedicated to that purpose... built on NFC.

But remember, this was 2013! EMV was not yet in meaningful use in the US; several major banks and payment networks had just committed to rolling it out in 2012 and every American can tell you that the process was long and torturous. Because of the stringent security standards around EMV, Android devices did not implement EMV until ARM secure enclaves became widely available. EMVCo, the industry body behind EMV, did not have a certification program for smartphones until 2016.

Android phones offered several "tap-to-pay" solutions, from Google's frequently rebranded Google Wallet^w^wAndroid Pay^w^wGoogle Wallet to Verizon's embarrassingly rebranded ISIS^wSoftcard and Samsung Pay. All of these initially relied on proprietary NFC protocols with bespoke payment terminal implementations. This was sketchy enough, and few enough phones actually had NFC, that the most successful US pay-by-phone implementations like Walmart's and Starbucks' used barcodes for communication. It would take almost a decade before things really settled down and smartphones all just implemented EMV.

So, in that context, Apple's decision isn't so odd. They must have figured that iBeacon could solve the same "initial handshake" problem as Walmart's QR codes, but more conveniently and using radio hardware that they already included in their phones. iBeacon-based payment flows used the iBeacon only to inform the phone of what payment devices were nearby, everything else happened via interaction with a cloud service or whatever mechanism the payment vendor chose to implement. Apple used their proprietary payments system through what would become your Apple Account, PayPal slapped together an iBeacon-based fast path to PayPal transfers, etc.

I don't think that Apple's iBeacon-based payments solution ever really shipped. It did get some use, most notably by Apple, but these all seem to have been early-stage implementations, and the complete end-to-end SDK that a lot of developers expected never landed.

You might remember that this was a very chaotic time in phone-based payments, solutions were coming and going. When Apple Pay was properly announced a year after iBeacons, there was little mention of Bluetooth. By the time in-store Apple Pay became common, Apple had given up and adopted NFC.

Limitations

One of the great weaknesses of iBeacon was the security design, or lack thereof. iBeacon advertisements were sent in plaintext with no authentication of any type. This did, of course, radically simplify implementation, but it also made iBeacon untrustworthy for any important purpose. It is quite trivial, with a device like an Android phone, to "clone" any iBeacon and transmit its identifiers wherever you want. This problem might have killed off the whole location-based-paywall-unlocking concept had market forces not already done so. It also opens the door to a lot of nuisance attacks on iBeacon-based location marketing, which may have limited the depth of iBeacon features in major apps.

iBeacon was also positioned as a sort of local positioning system, but it really wasn't. iBeacon offers no actual time-of-flight measurements, only RSSI-based estimation of range. Even with correct on-site calibration (which can be aided by adjusting a fixed RSSI-range bias value included in some iBeacon advertisements) this type of estimation is very inaccurate, and in my little experiments with a Bluetooth beacon location library I can see swings from 30m to 70m estimated range based only on how I hold my phone. iBeacon positioning has never been accurate enough to do more than assert whether or not a phone is "near" the beacon, and "near" can take on different values depending on the beacon's transmit power.

Developers have long looked towards Bluetooth as a potential local positioning solution, and it's never quite delivered. The industry is now turning towards Ultra-Wideband or UWB technology, which combines a high-rate, high-bandwidth radio signal with a time-of-flight radio ranging protocol to provide very accurate distance measurements. Apple is, once again, a technical leader in this field and UWB radios have been integrated into the iPhone 11 and later.

Senescence

iBeacon arrived to some fanfare, quietly proliferated in the shadows of the advertising industry, and then faded away. The Wikipedia article on iBeacons hasn't really been updated since support on Windows Phone was relevant. Apple doesn't much talk about iBeacons any more, and their compatriots Facebook and Google both sunset their beacon programs years ago.

Part of the problem is, well, the pervasive surveillance thing. The idea of Bluetooth beacons cooperating with your phone to track your every move proved unpopular with the public, and so progressively tighter privacy restrictions in mobile operating systems and app stores have clamped down on every grocery store app selling location data to whatever broker bids the most. I mean, they still do, but it's gotten harder to use Bluetooth as an aid. Even Android, the platform of "do whatever you want in the background, battery be damned," strongly discourages Bluetooth scanning by non-foreground apps.

Still, the basic technology remains in widespread use. BLE beacons have absolutely proliferated, there are plenty of apps you can use to list nearby beacons and there almost certainly are nearby beacons. One of my cars has, like, four separate BLE beacons going on all the time, related to a phone-based keyless entry system that I don't think the automaker even supports any more. Bluetooth beacons, as a basic primitive, are so useful that they get thrown into all kinds of applications. My earbuds are a BLE beacon, which the (terrible, miserable, no-good) Bose app uses to detect their proximity when they're paired to another device. A lot of smart home devices like light bulbs are beacons. The irony, perhaps, of iBeacon-based location tracking is that it's a victim of its own success. There is so much "background" BLE beacon activity that you scarcely need to add purpose-built beacons to track users, and only privacy measures in mobile operating systems and the beacons themselves (some of which rotate IDs) save us.

Apple is no exception to the widespread use of Bluetooth beacons: iBeacon lives on in virtually every apple device. If you do try out a Bluetooth beacon scanning app, you'll discover pretty much every Apple product in a 30 meter radius. From MacBooks Pro to Airpods, almost all Apple products transmit iBeacon advertisements to their surroundings. These are used for the initial handshake process of peer-to-peer features like Airdrop, and Find My/AirTag technology seems to be derived from the iBeacon protocol (in the sense that anything can be derived from such a straightforward design). Of course, pretty much all of these applications now randomize identifiers to prevent passive use of device advertisements for long-term tracking.

Here's some good news: iBeacons are readily available in a variety of form factors, and they are very cheap. Lots of libraries exist for working with them. If you've ever wanted some sort of location-based behavior for something like home automation, iBeacons might offer a good solution. They're neat, in an old technology way. Retrotech from the different world of 2013.

It's retro in more ways than one. It's funny, and a bit quaint, to read the contemporary privacy concerns around iBeacon. If only they had known how bad things would get! Bluetooth beacons were the least of our concerns.

[1] Things can be a little confusing here because the iBeacon is such a straightforward concept, and Apple's implementation is so simple. We could define "iBeacon" as including only officially endorsed products from Apple affiliates, or as including any device that behaves the same as official products (e.g. by using the iBeacon BLE advertisement type codes), or as any device that is performing substantially the same function (but using a different advertising format). I usually mean the latter of these three as there isn't really much difference between an iBeacon and ten million other BLE beacons that are doing the same thing with a slightly different identifier format. Facebook and Google's efforts fall into this camp.

When we last talked about Troposcatter, it was Pole Vault. Pole Vault was the first troposcatter communications network, on the east coast of Canada. It would not be alone for long. By the time the first Pole Vault stations were complete, work was already underway on a similar network for Alaska: the White Alice Communication System, WACS.

Alaska has long posed a challenge for communications. In the 1860s, Western Union wanted to extend their telegraph network from the United States into Europe. Although the technology would be demonstrated shortly after, undersea telegraph cables were still notional and it seemed that a route that minimized the ocean crossing would be preferable---of course, that route maximized the length on land, stretching through present-day Alaska and Siberia on each side of the Bering Strait. This task proved more formidable than Western Union had imagined, and the first transatlantic telegraph cable (on a much further south crossing) was completed before the arctic segments of the overland route. The "Western Union Telegraph Expedition" abandoned its work, leaving a telegraph line well into British Columbia that would serve as one of the principle communications assets in the region for decades after.

This ill-fated telegraph line failed to link San Francisco to Moscow, but its aftermath included a much larger impact on Russian interests in North America: the purchase of Alaska in 1867. Shortly after, the US military began its expansion into the new frontier. The Army Signal Corps, mostly to fulfill its function in observing the weather, built and staffed small installations that stretched further and further west. Later, in the 1890s, a gold rush brought a sudden influx of American settlers to Alaska's rugged terrain. The sudden economic importance of Klondike, and the rather colorful personalities of the prospectors looking to exploit it, created a much larger need for military presence. Fortuitously, many of the forts present had been built by the Signal Corps, which had already started on lines of communication. Construction was difficult, though, and without Alaskan communications as major priority there was only minimal coverage.

Things changed in 1900, when Congress appropriated a substantial budget to the Washington-Alaska Military Cable and Telegraph System. The Signal Corps set on Alaska like, well, like an army, and extensive telegraph and later telephone lines were built to link the various military outposts. Later renamed the Alaska Communications System, these cables brought the first telecommunication to much of Alaska. The arrival of the telegraph was quite revolutionary for remote towns, who could now receive news in real-time that had previously been delayed by as much as a year [1]. Telegraphy was important to civilians as well, something that Congress had anticipated: The original act authorizing the Alaska Communications System dictated that it would carry commercial traffic as well. The military had an unusual role in Alaska, and one aspect of it was telecommunications provider.

In 1925, an outbreak of diphtheria began to claim the lives of children in Nome, a town in far western Alaska on the Seward Peninsula. The daring winter delivery of antidiphtheria serum by dog sled is widely remembered due to its tangential connection to the Iditarod, but there were two sides of the "serum run." The message from Nome's sole doctor requesting the urgent shipment was transmitted from Nome to the Public Health Service in DC over the Alaska Communications System. It gives us some perspective on the importance of the telegraph in Alaska that the 600 mile route to Nome took five days and many feats of heroism---but at the same time could be crossed instantaneously by telegrams.

The Alaska Communications System included some use of radio from the beginning. A pair of HF radio stations specifically handled traffic for Nome, covering a 100-mile stretch too difficult for even the intrepid Signal Corps. While not a totally new technology to the military, radio was quite new to the telegraph business, and the ACS to Nome was probably the first commercial radiotelegraph system on the continent. By the 1930s, the condition of the Alaskan telegraph cables had decayed while demand for telephony had increased. Much of ACS was upgraded and modernized to medium-frequency radiotelephone links. In towns small and large, even in Anchorage itself, the sole telephone connection to the contiguous United States was an ACS telephone installed in the general store.

Alaskan communications became an even greater focus of the military with the onset of the Second World War. A few weeks after Pearl Harbor, the Japanese attacked Fort Mears in the Aleutian Islands. Fort Mears had no telecommunications connections, so despite the proximity of other airbases support was slow to come. The lack of a telegraph or telephone line contributed to 43 deaths and focused attention on the ACS. By 1944, the Army Signal Corps had a workforce of 2,000 dedicated to Alaska.

WWII brought more than one kind of attention to Alaska. Several Japanese assaults on the Aleutian islands represented the largest threats to American soil outside of Pearl Harbor, showing both Alaska's vulnerability and the strategic importance given to it by its relative proximity to Eurasia. WWII ended but, in 1949, the USSR demonstrated an atomic weapon. A combination of Soviet expansionism and the new specter of nuclear war turned military planners towards air defense. Like the Canadian Maritimes in the East, Alaska covered a huge swath of the airspace through which Soviet bombers might approach the US. Alaska was, once again, a battleground.

The early Cold War military buildup of Alaska was particularly heavy on air defense. During the late '40s and early '50s, more than a dozen new radar and control sites were built. The doctrine of ground-controlled interception requires real-time communication between radar centers, stressing the limited number of voice channels available on the ACS. As early as 1948, the Signal Corps had begun experiments to choose an upgrade path. Canadian early-warning radar networks, including the Distant Early Warning Line, were on the drawing board and would require many communications channels in particularly remote parts of Alaska.

Initially, point-to-point microwave was used in relatively favorable terrain (where the construction of relay stations about every 50 miles was practical). For the more difficult segments, the Signal Corps found that VHF radio could provide useful communications at ranges over 100 miles. VHF radiotelephones were installed at air defense radar stations, but there was a big problem: the airspace surveillance radar of the 1950s also operated in the VHF band, and caused so much interference with the radiotelephones that they were difficult to use. The radar stations were probably the most important users of the network, so VHF would have to be abandoned.

In 1954, a military study group was formed to evaluate options for the ACS. That group, in turn, requested a proposal from AT&T. Bell Laboratories had been involved in the design and evaluation of Pole Vault, the first sites of which had been completed two years before, so they naturally positioned troposcatter as the best option.

It is worth mentioning the unusual relationship AT&T had with Alaska, or rather, the lack of one. While the Bell System enjoyed a monopoly on telephony in most of the United States [2], they had never expanded into Alaska. Alaska was only a territory, after all, and a very sparsely populated one at that. The paucity of long-distance leads to or from Alaska (only one connected to Anchorage, for example) limited the potential for integration of Alaska into the broader Bell System anyway. Long-distance telecommunications in Alaska were a military project, and AT&T was involved only as a vendor.

Because of the high cost of troposcatter stations, proven during Pole Vault construction, a hybrid was proposed: microwave stations could be spaced every 50 miles along the road network, while troposcatter would cover the long stretches without roads.

In 1955, the Signal Corps awarded Western Electric a contract for the White Alice Communications System. The Corps of Engineers surveyed the locations of 31 sites, verifying each by constructing a temporary antenna tower. The Corps of Engineers led construction of the first 11 sites, and the final 20 were built on contract by Western Electric itself. All sites used radio equipment furnished by Western Electric and were built to Western Electric designs.

Construction was far from straightforward. Difficult conditions delayed completion of the original network until 1959, two years later than intended. A much larger issue, though, was the budget. The original WACS was expected to cost $38 million. By the time the first 31 sites were complete, the bill totaled $113 million---equivalent to over a billion dollars today. Western Electric had underestimated not only the complexity of the sites but the difficulty of their construction. A WECo report read:

On numerous occasions, the men were forced to surrender before the onslaught of cold, wind and snow and were immobilized for days, even weeks . This ordeal of waiting was of times made doubly galling by the knowledge that supplies and parts needed for the job were only a few miles distant but inaccessible because the white wall of winter had become impenetrable

WACS initial capability included 31 stations, of which 22 were troposcatter and the remainder only microwave (using Western Electric's TD-2). A few stations were equipped with both troposcatter and microwave, serving as relays between the two carriers.

In 1958, construction started on the Ballistic Missile Early Warning System or BMEWS. BMEWS was an over-the-horizon radar system intended to provide early warning of a Soviet attack. BMEWS would provide as little as 15 minutes warning, requiring that alerts reach NORAD in Colorado as quickly as possible. One BMEWS set was installed in Greenland, where the Pole Vault system was expanded to provide communications. Similarly, the BMEWS set at Clear Missile Early Warning Station in central Alaska relied on White Alice. Planners were concerned about the ability of the Soviet Union to suppress an alert by destroying infrastructure, so two redundant chains of microwave sites were added to White Alice. One stretched from Clear to Ketchikan where it connected to an undersea cable to Seattle. The other went east, towards Canada, where it met existing telephone cables on the Alaska Highway.

A further expansion of White Alice started the next year, in 1959. Troposcatter sites were extended through the Aleutian islands in "Project Stretchout" to serve new DEW Line stations. During the 1960s, existing WACS sites were expanded and new antennas were installed at Air Force installations. These were generally microwave links connecting the airbases to existing troposcatter stations.

In total, WACS reached 71 sites. Four large sites served as key switching points with multiple radio links and telephone exchanges. Pedro Dome, for example, had a 15,000 square foot communications building with dormitories, a power plant, and extensive equipment rooms. Support facilities included a vehicle maintenance building, storage warehouse, and extensive fuel tanks. A few WACS sites even had tramways for access between the "lower camp" (where equipment and personnel were housed) and the "upper camp" (where the antennas were located)... although they apparently did not fare well in the Alaskan conditions.

While Western Electric had initially planned for six people and 25 KW of power at each station, the final requirements were typically 20 people and 120-180 KW of generator capacity. Some sites stored over half a million gallons of fuel---conditions often meant that resupply was only possible during the summer.

Besides troposcatter and microwave radios, the equipment included tandem telephone exchanges. These are described in a couple of documents as "ATSS-4A," ATSS standing for Alaska Telephone Switching System. Based on the naming and some circumstantial evidence, I believe these were Western Electric 4A crossbar exchanges. They were later incorporated into AUTOVON, but also handled commercial long-distance traffic between Alaskan towns.

With troposcatter comes large antennas, and depending on connection lengths, WACS troposcatter antennas ranged from 30' dishes to 120' "billboard" antennas similar to those seen at Pole Vault sites. The larger antennas handled up to 50kW of transmit power. Some 60' and 120' antennas included their own fuel tanks and steam plants that heated the antennas through winter to minimize snow accumulation.

Nearly all of the equipment used by WACS was manufactured by Western Electric, with a lot of reuse of standard telephone equipment. For example, muxing on the troposcatter links used standard K-carrier (originally for telephone cables) and L-carrier (originally for coaxial cables). Troposcatter links operated at about 900 MHz with a wide bandwidth, and carrier two L-carrier supergroups (60 channels) and one K-carrier (12 channels) for a nominal capacity of 132 channels, although most stations did not have fully-populated L-carrier groups so actual capacity varied based on need. This was standard telephone carrier equipment in widespread use on the long-distance network, but some output modifications were made to suit the radio application.

The exception to the Western Electric rule was the radio sets themselves. They were manufactured by Radio Engineering Laboratories, the same company that built the radios for Pole Vault. REL pulled out all of the tricks they had developed for Pole Vault, and the longer WACS links used two antennas at different positions for space diversity. Each antenna had two feed horns, of orthoganal polarization, matching similar dual-transmitters for further diversity. REL equipment selected the best signal of the four available receiver options.

WACS represented an enormous improvement in Alaskan communications. The entire system was multi-channel with redundancy in many key parts of the network. Outside of the larger cities, WACS often brought the first usable long-distance telephone service. Even in Anchorage, WACS provided the only multi-channel connection. Despite these achievements, WACS was set for much the same fate as other troposcatter systems: obsolescence after the invention of communications satellites.

The experimental satellites Telstar 1 and 2 launched in the early 1960s, and the military began a shift towards satellite communications shortly after. Besides, the formidable cost of WACS had become a political issue. Maintenance of the system overran estimates by just as much as construction, and placing this cost on taxpayers was controversial since much of the traffic carried by the system consisted of regular commercial telephone calls. Besides, a general reticence to allocate money to WACS had lead to a general decay of the system. WACS capacity was insufficient for the rapidly increasing long-distance telephone traffic of the '60s, and due to decreased maintenance funding reliability was beginning to decline.

The retirement of a Cold War communications system is not unusual, but the particular fate of WACS is. It entered a long second life.

After acting as the sole long-distance provider for 60 years, the military began its retreat. In 1969, Congress passed the Alaska Communications Disposal Act. It called for complete divestment of the Alaska Communications System and WACS, to a private owner determined by a bidding process. Several large independent communications companies bid, but the winner was RCA. Committing to a $28.5 million purchase price followed by $30 million in upgrades, RCA reorganized the Alaska Communications System as RCA Alascom.

Transfer of the many ACS assets from the military to RCA took 13 years, involving both outright transfer of property and complex lease agreements on sites colocated with military installations. RCA's interest in Alaskan communications was closely connected to the coming satellite revolution: RCA had just built the Bartlett Earth Station, the first satellite ground station in Alaska. While Bartlett was originally an ACS asset owned by the Signal Corps, it became just the first of multiple ground stations that RCA would build for Alascom. Several of the new ground stations were colocated with WACS sites, establishing satellite as an alternative to the troposcatter links.

Alascom appears to have been the first domestic satellite voice network in commercial use, initially relying on a Canadian communications satellite [3]. In 1974, SATCOM 1 and 2 launched. These were not the first commercial communications satellites, but they represented a significant increase in capacity over previous commercial designs and are sometimes thought of as the true beginning of the satellite communications era. Both were built and owned by RCA, and Alascom took advantage of the new transponders.

At the same time, Alascom launched a modernization effort. 22 of the former WACS stations were converted to satellite ground stations, a project that took much of the '70s as Alascom struggled with the same conditions that had made WACS so challenging to begin with. Modernization also included the installation of DMS-10 telephone switches and conversion of some connections to digital.

A series of regulatory and business changes in the 1970s lead RCA to step away from the domestic communications industry. In 1979, Alascom sold to Pacific Power and Light, now for $200 million and $90 million in debt. PP&L continued on much the same trajectory, expanding the Alascom system to over 200 ground stations and launching the satellite Aurora I---the first of a small series of satellites that gave Alaska the distinction of being the only state with its own satellite communications network. For much of the '70s to the '00s, large parts of Alaska relied on satellite relay for calls between towns.

In a slight twist of irony considering its long lack of interest in the state, AT&T purchased parts of Alascom from PP&L in 1995, forming AT&T Alascom which has faded away as an independent brand. Other parts of the former ACS network, generally non-toll (or non-long-distance) operations, were split off into then PP&L subsidiary CenturyTel. While CenturyTel has since merged into CenturyLink, the Alaskan assets were first sold to Alaska Communications. Alaska Communications considers itself the successor of the ACS heritage, giving them a claim to over 100 years of communications history.

As electronics technology has continued to improve, penetration of microwave relays into inland Alaska has increased. Fewer towns rely on satellite today than in the 1970s, and the half-second latency to geosynchronous orbit is probably not missed. Alaska communications have also become more competitive, with long-distance connectivity available from General Communications (GCI) as well as AT&T and Alaska Communications.

Still, the legacy of Alaska's complex and expensive long-distance infrastructure still echoes in our telephone bills. State and federal regulators have allowed for extra fees on telephone service in Alaska and calls into Alaska, both intended to offset the high cost of infrastructure. Alaska is generally the most expensive long-distance calling destination in the United States, even when considering the territories.

But what of White Alice?

The history of the Alaska Communications System's transition to private ownership is complex and not especially well documented. While RCA's winning bid following the Alaska Communications Disposal Act set the big picture, the actual details of the transition were established by many individual negotiations spanning over a decade. Depending on the station, WACS troposcatter sites generally conveyed to RCA in 1973 or 1974. Some, colocated with active military installations, were leased rather than included in the sale. RCA generally decommissioned each WACS site once a satellite ground station was ready to replace it, either on-site or nearby.

For some WACS sites, this meant the troposcatter equipment was shut down in 1973. Others remained in use later. The Boswell Bay troposcatter station seems to have been the last turned down, in 1985. The 1980s were decidedly the end of WACS. Alascom's sale to PP&L cemented the plan to shut down all troposcatter operations, and the 1980 Comprehensive Environmental Response, Compensation, and Liability Act lead to the establishment of the Formerly Used Defense Sites (FUDS) program within DoD. Under FUDS, the Corps of Engineers surveyed the disused WACS sites and found nearly all had significant contamination by asbestos (used in seemingly every building material in the '50s and '60s) and leaked fuel oil.

As a result, most White Alice sites were demolished between 1986 and 1999. The cost of demolition and remediation in such remote locations was sometimes greater than the original construction. No WACS sites remain intact today.

Postscript:

A 1988 Corps of Engineers historical inventory of WACS, prepared due to the demolition of many of the stations, mentions that meteor burst communications might replace troposcatter. Meteor burst is a fascinating communications mode, similar in many ways to troposcatter but with the twist that the reflecting surface is not the troposphere but the ionized trail of meteors entering the atmosphere. Meteor burst connections only work when there is a meteor actively vaporizing in the upper atmosphere, but atmospheric entry of small meteors is common enough that meteor burst communications are practical for low-rate packetized communications. For example, meteor burst has been used for large weather and agricultural telemetry systems.

The Alaska Meteor Burst Communications System was implemented in 1977 by several federal agencies, and was used primarily for automated environmental telemetry. Unlike most meteor burst systems, though, it seems to have been used for real-time communications by the BLM and FAA. I can't find much information, but they seem to have built portable teleprinter terminals for this use.

Even more interesting, the Air Force's Alaskan Air Command built its own meteor burst network around the same time. This network was entirely for real-time use, and demonstrated the successful transmission of radar track data from radar stations across the state to Elmendorf Air Force base. Even better, the Air Force experimented with the use of meteor burst for intercept control by fitting aircraft with a small speech synthesizer that translated coded messages into short phrases. The Air Force experimented with several meteor burst systems during the Cold War, anticipating that it might be a survivable communications system in wartime. More details on these will have to fill a future article.

[1] Crews of the Western Union Telegraph Expedition reportedly continued work for a full year after the completion of the transatlantic telephone cable, because news of it hadn't reached them yet.

[2] Eliding here some complexities like GTE and their relationship to the Bell System.

[3] Perhaps owing to the large size of the country and many geographical challenges to cable laying, Canada has often led North America in satellite communications technology.

Note: I have edited this post to add more information, a couple of hours after originally publishing it. I forgot about a source I had open in a tab. Sorry.

We've talked before about carphones, and certainly one of the only ways to make phones even more interesting is to put them in modes of transportation. Installing telephones in cars made a lot of sense when radiotelephones were big and required a lot of power; and they faded away as cellphones became small enough to have a carphone even outside of your car.

There is one mode of transportation where the personal cellphone is pretty useless, though: air travel. Most readers are probably well aware that the use of cellular networks while aboard an airliner is prohibited by FCC regulations. There are a lot of urban legends and popular misconceptions about this rule, and fully explaining it would probably require its own article. The short version is that it has to do with the way cellular devices are certified and cellular networks are planned. The technical problems are not impossible to overcome, but honestly, there hasn't been a lot of pressure to make changes. One line of argument that used to make an appearance in cellphones-on-airplanes discourse is the idea that airlines or the telecom industry supported the cellphone ban because it created a captive market for in-flight telephone services.

Wait, in-flight telephone services?

That theory has never had much to back it up, but with the benefit of hindsight we can soundly rule it out: not only has the rule persisted well past the decline and disappearance of in-flight telephones, in-flight telephones were never commercially successful to begin with.

Let's start with John Goeken. A 1984 Washington Post article tells us that "Goeken is what is called, predictably enough, an 'idea man.'" Being the "idea person" must not have had quite the same connotations back then, it was a good time for Goeken. In the 1960s, conversations with customers at his two-way radio shop near Chicago gave him an idea for a repeater network to allow truckers to reach their company offices via CB radio. This was the first falling domino in a series that lead to the founding of MCI and the end of AT&T's long-distance monopoly. Goeken seems to have been the type who grew bored with success, and he left MCI to take on a series of new ventures. These included an emergency medicine messaging service, electrically illuminated high-viz clothing, and a system called the Mercury Network that built much of the inertia behind the surprisingly advanced computerization of florists [1].

"Goeken's ideas have a way of turning into dollars, millions of them," the Washington Post continued. That was certainly true of MCI, but every ideas guy had their misses. One of the impressive things about Goeken was his ability to execute with speed and determination, though, so even his failures left their mark. This was especially true of one of his ideas that, in the abstract, seemed so solid: what if there were payphones on commercial flights?

Goeken's experience with MCI and two-way radios proved valuable, and starting in the mid-1970s he developed prototype air-ground radiotelephones. In its first iteration, "Airfone" consisted of a base unit installed on an aircraft bulkhead that accepted a credit card and released a cordless phone. When the phone was returned to the base station, the credit card was returned to the customer. This equipment was simple enough, but it would require an extensive ground network to connect callers to the telephone system. The infrastructure part of the scheme fell into place when long-distance communications giant Western Union signed on with Goeken Communications to launch a 50/50 joint venture under the name Airfone, Inc.

Airfone was not the first to attempt air-ground telephony---AT&T had pursued the same concept in the 1970s, but abandoned it after resistance from the FCC (unconvinced the need was great enough to justify frequency allocations) and the airline industry (which had formed a pact, blessed by the government, that prohibited the installation of telephones on aircraft until such time as a mature technology was available to all airlines). Goeken's hard-headed attitude, exemplified in the six-year legal battle he fought against AT&T to create MCI, must have helped to defeat this resistance.

Goeken brought technical advances, as well. By 1980, there actually was an air-ground radiotelephone service in general use. The "General Aviation Air-Ground Radiotelephone Service" allocated 12 channels (of duplex pairs) for radiotelephony from general aviation aircraft to the ground, and a company called Wulfsberg had found great success selling equipment for this service under the FliteFone name. Wulfsberg FliteFones were common equipment on business aircraft, where they let executives shout "buy" and "sell" from the air. Goeken referred to this service as evidence of the concept's appeal, but it was inherently limited by the 12 allocated channels.

General Aviation Air-Ground Radiotelephone Service, which I will call AGRAS (this is confusing in a way I will discuss shortly), operated at about 450MHz. This UHF band is decidedly line-of-sight, but airplanes are very high up and thus can see a very long ways. The reception radius of an AGRAS transmission, used by the FCC for planning purposes, was 220 miles. This required assigning specific channels to specific cities, and there the limits became quite severe. Albuquerque had exactly one AGRAS channel available. New York City got three. Miami, a busy aviation area but no doubt benefiting from its relative geographical isolation, scored a record-setting four AGRAS channels. That meant AGRAS could only handle four simultaneous calls within a large region... if you were lucky enough for that to be the Miami region; otherwise capacity was even more limited.

Back in the 1970s, AT&T had figured that in-flight telephones would be very popular. In a somewhat hand-wavy economic analysis, they figured that about a million people flew in the air on a given day, and about a third of them would want to make telephone calls. That's over 300,000 calls a day, clearly more than the limited AGRAS channels could handle... leading to the FCC's objection that a great deal of spectrum would have to be allocated to make in-flight telephony work.

Goeken had a better idea: single-sideband. SSB is a radio modulation technique that allows a radio transmission to fit within a very narrow bandwidth (basically by suppressing half of the signal envelope), at the cost of a somewhat more fiddly tuning process for reception. SSB was mostly used down in the HF bands, where the low frequencies meant that bandwidth was acutely limited. Up in the UHF world, bandwidth seemed so plentiful that there was little need for careful modulation techniques... until Goeken found himself asking the FCC for 10 blocks of 29 channels each, a lavish request that wouldn't really fit anywhere in the popular UHF spectrum. The use of UHF SSB, pioneered by Airfone, allowed far more efficient use of the allocation.

In 1983, the FCC held hearings on Airfone's request for an experimental license to operate their SSB air-ground radiotelephone system in two allocations (separate air-ground and ground-air ranges) around 850MHz and 895MHz. The total spectrum allocated was about 1.5MHz in each of the two directions. The FCC assented and issued the experimental license in 1984, and Airfone was in business.

Airfone initially planned 52 ground stations for the system, although I'm not sure how many were ultimately built---certainly 37 were in progress in 1984, at a cost of about $50 million. By 1987, the network had reportedly grown to 68. Airfone launched on six national airlines (a true sign of how much airline consolidation has happened in recent decades---there were six national airlines?), typically with four cordless payphones on a 727 or similar aircraft. The airlines received a commission on the calling rates, and Airfone installed the equipment at their own expense. Still, it was expected to be profitable... Airfone projected that 20-30% of passengers would have calls to make.

I wish I could share more detail on these ground stations, in part because I assume there was at least some reuse of existing Western Union facilities (WU operated a microwave network at the time and had even dabbled in cellular service in the 1980s). I can't find much info, though. The antennas for the 800MHz band would have been quite small, but the 1980s multiplexing and control equipment probably took a fare share of floorspace.

Airfone was off to a strong start, at least in terms of installation base and press coverage. I can't say now how many users it actually had, but things looked good enough that in 1986 Western Union sold their share of the company to GTE. Within a couple of years, Goeken sold his share to GTE as well, reportedly as a result of disagreements with GTE's business strategy.

Airfone's SSB innovation was actually quite significant. At the same time, in the 1980s, a competitor called Skytel was trying to get a similar idea off the ground with the existing AGRAS allocation. It doesn't seem to have gone anywhere, I don't think the FCC ever approved it. Despite an obvious concept, Airfone pretty much launched as a monopoly, operating under an experimental license that named them alone. Unsurprisingly there was some upset over this apparent show of favoritism by the FCC, including from AT&T, which vigorously opposed the experimental license.

As it happened, the situation would be resolved by going the other way: in 1990, the FCC established the "commercial aviation air-ground service" which normalized the 800 MHz spectrum and made licenses available to other operators. That was six years after Airfone started their build-out, though, giving them a head start that severely limited competition.

Still, AT&T was back. AT&T introduced a competing service called AirOne. AirOne was never as widely installed as Airfone but did score some customers including Southwest Airlines, which only briefly installed AirOne handsets on their fleet. "Only briefly" describes most aspects of AirOne, but we'll get to that in a moment.

The suddenly competitive market probably gave GTE Airfone reason to innovate, and besides, a lot had changed in communications technology since Airfone was designed. One of Airfone's biggest limitations was its lack of true roaming: an Airfone call could only last as long as the aircraft was within range of the same ground station. Airfone called this "30 minutes," but you can imagine that people sometimes started their call near the end of this window, and the problem was reportedly much worse. Dropped calls were common, adding insult to the injury that Airfone was decidedly expensive. GTE moved towards digital technology and automation.

1991 saw the launch of Airfone GenStar, which used QAM digital modulation to achieve better call quality and tighter utilization within the existing bandwidth. Further, a new computerized network allowed calls to be handed off from one ground station to another. Capitalizing on the new capacity and reliability, the aircraft equipment was upgraded as well. The payphone like cordless stations were gone, replaced by handsets installed in seatbacks. First class cabins often got a dedicated handset for every seat, economy might have one handset on each side of a row. The new handsets offered RJ11 jacks, allowing the use of laptop modems while in-flight. Truly, it was the future.

During the 1990s, satellites were added to the Airfone network as well, improving coverage generally and making telephone calls possible on overseas flights. Of course, the rise of satellite communications also sowed the seeds of Airfone's demise. A company called Aircell, which started out using the cellular network to connect calls to aircraft, rebranded to Gogo and pivoted to satellite-based telephone services. By the late '90s, they were taking market share from Airfone, a trend that would only continue.

Besides, for all of its fanfare, Airfone was not exactly a smash hit. Rates were very high, $5 a minute in the late '90s, giving Airfone a reputation as a ripoff that must have cut a great deal into that "20-30% of fliers" they hoped to serve. With the rise of cellphones, many preferred to wait until the aircraft was on the ground to use their own cellphone at a much lower rate. GTE does not seem to have released much in the way of numbers for Airfone, but it probably wasn't making them rich.

Goeken, returning to the industry, inadvertently proved this point. He aggressively lobbied the FCC to issue competitive licenses, and ultimately succeeded. His second company in the space, In-Flight Phone Inc., became one of the new competitors to his old company. In-Flight Phone did not last for long. Neither did AT&T AirOne. A 2005 FCC ruling paints a grim picture:

Current 800 MHz Air-Ground Radiotelephone Service rules contemplate six competing licensees providing voice and low-speed data services. Six entities were originally licensed under these rules, which required all systems to conform to detailed technical specifications to enable shared use of the air-ground channels. Only three of the six licensees built systems and provided service, and two of those failed for business reasons.

In 2002, AT&T pulled out, and Airfone was the only in-flight phone left. By then, GTE had become Verizon, and GTE Airfone was Verizon Airfone. Far from a third of passengers, the CEO of Airfone admitted in an interview that a typical flight only saw 2-3 phone calls. Considering the minimum five-figure capital investment in each aircraft, it's hard to imagine that Airfone was profitable---even at $5 minute.

Airfone more or less faded into obscurity, but not without a detour into the press via the events of 9/11. Flight 93, which crashed in Pennsylvania, was equipped with Airfone and passengers made numerous calls. Many of the events on board this aircraft were reconstructed with the assistance of Airfone records, and Claircom (the name of the operator of AT&T AirOne, which never seems to have been well marketed) also produced records related to other aircraft involved in the attacks. Most notably, flight 93 passenger Todd Beamer had a series of lengthy calls with Airfone operator Lisa Jefferson, through which he relayed many of the events taking place on the plane in real time. During these calls, Beamer seems to have coordinated the effort by passengers to retake control of the plane. The significance of Airfone and Claircom records to 9/11 investigations is such that 9/11 conspiracy theories may be one of the most enduring legacies of Claircom especially.

In an odd acknowledgment of their aggressive pricing, Airfone decided not to bill for any calls made on 9/11, and temporarily introduced steep discounts (to $0.99 a minute) in the weeks after. This rather meager show of generosity did little to reverse the company's fortunes, though, and it was already well into a backslide.

In 2006, the FCC auctioned the majority of Airfone's spectrum to new users. The poor utilization of Airfone was a factor in the decision, as well as Airfone's relative lack of innovation compared to newer cellular and satellite systems. In fact, a large portion of the bandwidth was purchased by Gogo, who years later would use to to deliver in-flight WiFi. Another portion went to a subsidiary of JetBlue that provided in-flight television. Verizon announced the end of Airfone in 2006, pending an acquisition by JetBlue, and while the acquisition did complete JetBlue does not seem to have continued Airfone's passenger airline service. A few years later, Gogo bought out JetBlue's communications branch, making them the new monopoly in 800MHz air ground radiotelephony. Gogo only offered telephone service for general aviation aircraft; passenger aircraft telephones had gone the way of the carphone.

It's interesting to contrast the fate of Airfone to to its sibling, AGRAS. Depending on who you ask, AGRAS refers to the radio service or to the Air Ground Radiotelephone Automated Service operated by Mid-America Computer Corporation. What an incredible set of names. This was a situation a bit like ARINC, the semi-private company that for some time held a monopoly on aviation radio services. MACC had a practical monopoly on general aviation telephone service throughout the US, by operating the billing system for calls. MACC still exists today as a vendor of telecom billing software and this always seems to have been their focus---while I'm not sure, I don't believe that MACC ever operated ground stations, instead distributing rate payments to private companies that operated a handful of ground stations each. Unfortunately the history of this service is quite obscure and I'm not sure how MACC came to operate the system, but I couldn't resist the urge to mention the Mid-America Computer Corporation.

AGRAS probably didn't make anyone rich, but it seems to have been generally successful. Wulfsberg FliteFones operating on the AGRAS network gave way to Gogo's business aviation phone service, itself a direct descendent of Airfone technology.

The former AGRAS allocation at 450MHz somehow came under the control of a company called AURA Network Systems, which for some years has used a temporary FCC waiver of AGRAS rules to operate data services. This year, the FCC began rulemaking to formally reallocate the 450MHz air ground allocation to data services for Advanced Air Mobility, a catch-all term for UAS and air taxi services that everyone expects to radically change the airspace system in coming years. New uses of the band will include command and control for long-range UAS, clearance and collision avoidance for air taxis, and ground and air-based "see and avoid" communications for UAS. This pattern, of issuing a temporary authority to one company and later performing rulemaking to allow other companies to enter, is not unusual for the FCC but does make an interesting recurring theme in aviation radio. It's typical for no real competition to occur, the incumbent provider having been given such a big advantage.

When reading about these legacy services, it's always interesting to look at the licenses. ULS has only nine licenses on record for the original 800 MHz air ground service, all expired and originally issued to Airfone (under both GTE and Verizon names), Claircom (operating company for AT&T AirOne), and Skyway Aircraft---this one an oddity, a Florida-based company that seems to have planned to introduce in-flight WiFi but not gotten all the way there.

Later rulemaking to open up the 800MHz allocation to more users created a technically separate radio service with two active licenses, both held by AC BidCo. This is an intriguing mystery until you discover that AC BidCo is obviously a front company for Gogo, something they make no effort to hide---the legalities of FCC bidding processes are such that it's very common to use shell companies to hold FCC licenses, and we could speculate that AC BidCo is the Aircraft Communications Bidding Company, created by Gogo for the purpose of the 2006-2008 auctions. These two licenses are active for the former Airfone band, and Gogo reportedly continues to use some of the original Airfone ground stations.

Gogo's air-ground network, which operates at 800MHz as well as in a 3GHz band allocated specifically to Gogo, was originally based on CDMA cellular technology. The ground stations were essentially cellular stations pointed upwards. It's not clear to me if this CDMA-derived system is still in use, but Gogo relies much more heavily on their Ku-band satellite network today.

The 450MHz licenses are fascinating. AURA is the only company to hold current licenses, but the 246 reveal the scale of the AGRAS business. Airground of Idaho, Inc., until 1999 held a license for an AGRAS ground station on Brundage Mountain McCall, Idaho. The Arlington Telephone Company, until a 2004 cancellation, held a license for an AGRAS ground station atop their small telephone exchange in Arlington, Nebraska. AGRAS ground stations seem to have been a cottage industry, with multiple licenses to small rural telephone companies and even sole proprietorships. Some of the ground stations appear to have been the roofs of strip mall two-way radio installers. In another life, maybe I would be putting a 450MHz antenna on my roof to make a few dollars.

Still, there were incumbents: numerous licenses belonged to SkyTel, which after the decline of AGRAS seems to have refocused on paging and, then, gone the same direction as most paging companies: an eternal twilight as American Messaging ("The Dependable Choice"), promoting innovation in the form of longer-range restaurant coaster pagers. In another life, I'd probably be doing that too.

[1] This is probably a topic for a future article, but the Mercury Network was a computerized system that Goeken built for a company called Florist's Telegraph Delivery (FTD). It was an evolution of FTD's telegraph system that allowed a florist in one city to place an order to be delivered by by a florist in another city, thus enabling the long-distance gifting of flowers. There were multiple such networks and they had an enduring influence on the florist industry and broader business telecommunications.

I have a rough list of topics for future articles, a scratchpad of two-word ideas that I sometimes struggle to interpret. Some items have been on that list for years now. Sometimes, ideas languish because I'm not really interested in them enough to devote the time. Others have the opposite problem: chapters of communications history with which I'm so fascinated that I can't decide where to start and end. They seem almost too big to take on. One of these stories starts in another vast frontier: northeastern Canada.

It was a time, rather unlike our own, of relative unity between Canada and the United States. Both countries had spent the later part of World War II planning around the possibility of an Axis attack on North America, and a ragtag set of radar stations had been built to detect inbound bombers. The US had built a series of stations along the border, and the Canadians had built a few north of Ontario and Quebec to extend coverage north of those population centers. Then the war ended and, as with so many WWII projects, construction stopped. Just a few years later, the USSR demonstrated a nuclear weapon and the Cold War was on. As with so many WWII projects, freshly anxious planners declared the post-war over and blew the dust off of North American air defense plans. In 1950, US and Canadian defense leaders developed a new plan to consolidate and improve the scattershot radar early warning plan.

This agreement would become the Pinetree Line, the first of three trans-Canadian radar fences jointly constructed and operated by the two nations. For the duration of the Cold War, and even to the present day, these radar installations formed the backbone of North American early warning and the locus of extensive military cooperation. The joint defense agreement between the US and Canada, solidified by the Manhattan Project's dependence on Canadian nuclear industry, grew into the 1968 establishment of the North American Air Defense Command (NORAD) as a binational joint military organization.

This joint effort had to rise to many challenges. Radar had earned its place as a revolutionary military technology during the Second World War, but despite the many radar systems that had been fielded, engineer's theoretical understanding of radar and RF propagation were pretty weak. I have written here before about over-the-horizon radar, the pursuit of which significantly improved our scientific understanding of radio propagation in the atmosphere... often by experiment, rather than model. A similar progression in RF physics would also benefit radar early warning in another way: communications.

One of the bigger problems with the Pinetree Line plan was the remote location of the stations. You might find that surprising; the later Mid-Canada and DEW lines were much further north and more remote. The Pinetree Line already involved stations in the far reaches of the maritime provinces, though, and to provide suitable warning to Quebec and the Great Lakes region stations were built well north of the population centers. Construction and operations would rely on aviation, but an important part of an early warning system is the ability to deliver the warning. Besides, ground-controlled interception had become the main doctrine in air defense, and it required not just an alert but real-time updates from radar stations for the most effective response. Each site on the Pinetree Line would require a reliable real-time communications capability, and as the sites were built in the 1950s, some were a very long distance from telephone lines.

Canada had only gained a transcontinental telephone line in 1932, seventeen years behind the United States (which by then had three different transcontinental routes and a fourth in progress), a delay owing mostly to the formidable obstacle of the Canadian Rockies. The leaders in Canadian long-distance communications were Bell Canada and the two railways (Canadian Pacific and Canadian National), and in many cases contracts had been let to these companies to extend telephone service to radar stations. The service was very expensive, though, and the construction of telephone cables in the maritimes was effectively ruled out due to the huge distances involved and uncertainty around the technical feasibility of underwater cables to Newfoundland due to the difficult conditions and extreme tides in the Gulf of St. Lawrence.

The RCAF had faced a similar problem when constructing its piecemeal radar stations in Ontario and Quebec in the 1940s, and had addressed them by applying the nascent technology of point-to-point microwave relays. This system, called ADCOM, was built and owned by RCAF to stretch 1,400 miles between a series of radar stations and other military installations. It worked, but the construction project had run far over budget (and major upgrades performed soon after blew the budget even further), and the Canadian telecom industry had vocally opposed it on the principle that purpose-built military communications systems took government investment away from public telephone infrastructure that could also serve non-military needs.

These pros and cons of ADCOM must have weighed on Pinetree Line planners when they chose to build a system directly based on ADCOM, but to contract its construction and operation to Bell Canada [1]. This was, it turned out, the sort of compromise that made no one happy: the Canadian military's communications research establishment was reluctant to cede its technology to Bell Canada, while Bell Canada objected to deploying the military's system rather than one of the commercial technologies then in use across the Bell System.

The distinct lack of enthusiasm on the part of both parties involved was a bad omen for the future of this Pinetree Line communications system, but as it would happen, the whole plan was overcome by events. One of the great struggles of large communications projects in that era, and even today, is the rapid rate of technological progress. One of ADCOM's faults was that the immense progress Bell Labs and Western Electric made in microwave equipment during the late '40s meant that it was obsolete as soon as it went into service. This mistake would not be repeated, as ADCOM's maritimes successor was obsoleted before it even broke ground. A promising new radio technology offered a much lower cost solution to these long, remote spans.

At the onset of the Second World War, the accepted theory of radio propagation held that HF signals could pass the horizon via ground wave propagation, curving to follow the surface of the Earth, while VHF and UHF signals could not. This meant that the higher-frequency bands, where wideband signals were feasible, were limited to line-of-sight or at least near-line-of-sight links... not more than 50 miles with ideal terrain, often less. We can forgive the misconception, because this still holds true today, as a rule of thumb. The catch is in the exceptions, the nuances, that during the war were already becoming a headache to RF engineers.

First, military radar operators observed mysterious contacts well beyond the theoretical line-of-sight range of their VHF radar sets. These might have been dismissed as faults in the equipment (or the operator), but reports stacked up as more long-range radar systems were fielded. After the war, relaxed restrictions and a booming economy allowed radio to proliferate. UHF television stations, separated by hundreds of miles, unexpectedly interfered with each other. AT&T, well into deployment of a transcontinental microwave network, had to adjust its frequency planning after it was found that microwave stations sometimes received interfering signals from other stations in the chain... stations well over the horizon.

This was the accidental discovery of tropospheric scattering.

The Earth's atmosphere is divided into five layers. We live in the troposhere, the lowest and thinnest of the layers, above which lies the stratosphere. Roughly speaking, the difference between these layers is that the troposphere becomes colder with height (due to increasing distance from the warm surface), while the stratosphere becomes warmer with height (due to decreasing shielding from the sun) [2]. In between is a local minimum of temperature, called the tropopause.

The density gradients around the tropopause create a mirror effect, like the reflections you see when looking at an air-water boundary. The extensive turbulence and, well, weather present in the troposhere also refract signals on their way up and down, making the true course of radio signals reflecting off of the tropopause difficult to predict or analyze. Because of this turbulence, the effect has come to be known as scattering: radio signals sent upwards, towards the troposphere, will be scattered back downwards across a wide area. This effect is noticeable only at high frequencies, so it remained unknown until the widespread use of UHF and microwave, and was still only partially understood in the early 1950s.

The locii of radar technology at the time were Bell Laboratories and the MIT Lincoln Laboratory, and they both studied this effect for possible applications. Presaging one of the repeated problems of early warning radar systems, by the time Pinetree Line construction began in 1951 the Lincoln Laboratory was already writing proposals for systems that would obsolete it. In fact, construction would begin on both of the Pinetree Line's northern replacements before the Pinetree Line itself was completed. Between rapid technological development and military planners in a sort of panic mode, the early 1950s were a very chaotic time. Underscoring the ever-changing nature of early warning was the timeline of Pinetree Line communications: as the Pinetree Line microwave network was in planning, the Lincoln Laboratory was experimenting with troposcatter communications. By the time the first stations in Newfoundland completed construction, Bell Laboratories had developed an experimental troposcatter communications system.

This new means of long-range communications would not be ready in time for the first Pinetree Line stations, so parts of the original ADCOM-based microwave network would have to be built. Still, troposcatter promised to complete the rest of the network at significantly reduced cost. The US Air Force, wary of ADCOM's high costs and more detached from Canadian military politics, aggressively lobbied for the adoption of troposcatter communications for the longest and most challenging Pinetree Line links.

Bell Laboratories, long a close collaborator with the Air Force, was well aware of troposcatter's potential for early warning radar. Bell Canada and Bell Laboratories agreed to evaluate the system under field conditions, and in 1952 experimental sites were installed in Newfoundland. These tests found reliable performance over 150 miles, far longer than achievable by microwave and---rather conveniently---about the distance between Pinetree Line radar stations. These results suggested that the Pinetree Line could go without an expensive communications network in the traditional sense, instead using troposcatter to link the radar stations directly to each other.

Consider a comparison laid out by the Air Force: one of the most complex communications requirements for the Pinetree Line was a string of stations running not east-west like the "main" line, but north-south from St. John's, Newfoundland to Frobisher Bay, Nunavut. These stations were critical for detection of Soviet bombers approaching over the pole from the northeast, otherwise a difficult gap in radar coverage until the introduction of radar sites in Greenland. But the stations covered a span of over 1,000 miles, most of it in formidably rugged and remote arctic coastal terrain. The proposed microwave system would require 50 relay stations, almost all of which would be completely new construction. Each relay's construction would have to be preceded by the construction of a harbor or airfield for access, and then establishment of a power plant, to say nothing of the ongoing logistics of transporting fuel and personnel for maintenance. The proposed troposcatter system, on the other hand, required only ten relays. All ten would be colocated with radar stations, and could share infrastructure and logistical considerations.

Despite the clear advantages of troposcatter and its selection by the USAF, the Canadian establishment remained skeptical. One cannot entirely blame them, considering that troposcatter communications had only just been demonstrated in the last year. Still, the USAF was footing most of the bill for the overall system (and paying entirely for the communications aspect, depending on how you break down the accounting) and had considerable sway. In 1954, well into construction of the radar stations (several had already been commissioned), the Bell Canada contract for communications was amended to add troposcatter relay in addition to the original microwave scheme. Despite the weaselly contracting, the writing was on the wall and progress on microwave relay stations almost stopped. By the latter part of 1954, the microwave network was abandoned entirely. Bell Canada moved at incredible speed to complete the world's first troposcatter long-distance route, code named Pole Vault.

One of the major downsides of troposcatter communications is its inefficiency. Only a very small portion of the RF energy reaching the tropopause is reflected, and of that, only a small portion is reflected in the right direction. Path loss from transmitter to receiver for long links is over -200 dB, compared to say -130 dB for a microwave link. That difference looks smaller than it is; dB is a logarithmic comparison and the decrease from -130 dB to -200 dB is a factor of ten million.

The solution is to go big. Pole Vault's antennas were manufactured as a rush order by D. S. Kennedy Co. of Massachusetts. 36 were required, generally four per site for transmit and receive in each direction. Each antenna was a 60' aluminum parabolic dish held up on edge by truss legs. Because of the extreme weather at the coastal radar sites, the antennas were specified to operate in a 120 knot wind---or a 100 knot wind with an inch of ice buildup. These were operating requirements, so the antenna had not only to survive these winds, but to keep flexing and movements small enough to not adversely impact performance. The design of the antennas was not trivial; even after analysis by both Kennedy Co. and Bell Canada, after installation some of the rear struts supporting the antennas buckled. All high-wind locations received redesigned struts.

To drive the antennas, Radio Engineering Laboratories of Long Island furnished radio sets with 10 kW of transmit power. Both D. S. Kennedy and Radio Engineering Laboratories were established companies, especially for military systems, but were still small compared to Bell System juggernauts like Western Electric and Northern Electric. They had built the equipment for the experimental sites, though, and the timeline for construction of Pole Vault was so short that planners did not feel there was time to contract larger manufacturers. This turn of events made Kennedy Co. and REL the leading experts in troposcatter equipment, which became their key business in the following decade.

The target of the contract, signed in January of 1954, was to have Pole Vault operational by the end of that same year. Winter conditions, and indeed spring and fall conditions, are not conducive to construction on the arctic coast. All of the equipment for Pole Vault had to be manufactured in the first half of the year, and as weather improved and ice cleared in the mid-summer, everything was shipped north and installation work began. Both militaries had turned down involvement in the complex and time-consuming logistics of the project, so Bell Canada chartered ships and aircraft and managed an incredibly complex schedule. To deliver equipment to sites as early as possible, icebreaker CCGS D'Iberville was chartered. C-119 and DC3 aircraft served alongside numerous small boats and airplanes.

All told, it took about seven months to manufacture and deliver equipment to the Pole Vault sites, and six months to complete construction. Construction workers, representing four or five different contractors at each site and reaching about 120 workers to a site during peak activity, had to live in construction camps that could still be located miles from the station. Grounded ships, fires, frostbite, and of course poor morale lead to complications and delays. At one site, Saglek, project engineers recorded a full 24-hour day with winds continuously above 75 miles per hour, and then weeks later, a gust of 135 mph was observed. Repairs had to be made to the antennas and buildings before they were even completed.

In a remarkable feat of engineering and construction, the Pole Vault system was completed and commissioned more or less on schedule: amended into the contract in January of 1954, commissioning tests of the six initial stations were successfully completed February of 1955. Four additional stations were built to complete the chain, and Pole Vault was declared fully operational December of 1956 at a cost of $24.6 million (about $290 million today).

Pole Vault operated at various frequencies between 650 and 800 MHz, the wide range allowing for minimal frequency reuse---interference was fairly severe, since each station's signal scattered and could be received by stations further down the line in ideal (or as the case may be, less than ideal) conditions. Frequency division multiplexing equipment, produced by Northern Electric (Nortel) based on microwave carrier systems, offered up to 36 analog voice circuits. The carrier systems were modular, and some links initially supported only 12 circuits, while later operational requirements lead to an upgrade to 70 circuits.

Over the following decades, the North Atlantic remained a critical challenge for North American air defense. It also became the primary communications barrier between the US and Canada and European NATO allies. Because Pole Vault provided connections across such a difficult span, several later military communications systems relied on Pole Vault as a backhaul connection.

An inventory of the Saglek site, typical of the system, gives an idea of the scope of each of the nine primary stations. This is taken from "Special Contract," a history by former Bell Canada engineer A. G. Lester:

(1) Four parabolic antennas, 60 feet in diameter, each mounted on seven mass concrete footings. (2) An equipment building 62 by 32 feet to house electronic equipment, plus a small (10 by 10 feet) diversity building. (3) A diesel building 54 by 36 feet, to house three 125 KVA (kilovolt amperes) diesel driven generators. (4) Two 2500 gallon fuel storage tanks. (5) Raceways to carry waveguide and cables. (6) Enclosed corridors interconnecting buildings, total length in this case 520 feet.

Since the Pole Vault stations were colocated with radar facilities, barracks and other support facilities for the crews were already provided for. Of course, you can imagine that the overall construction effort at each site was much larger, including the radar systems as well as cantonment for personnel.

Pole Vault would become a key communications system in the maritime provinces, remaining in service until 1975. Its reliable performance in such a challenging environment was a powerful proof of concept for troposcatter, a communications technique first imagined only a handful of years earlier. Even as Pole Vault reached its full operating capability in late 1956, other troposcatter systems were under construction. Much the same, and not unrelated, other radar early warning systems were under construction as well.

The Pinetree Line, for all of its historical interest and its many firsts, ended as a footnote in the history of North American air defense. More sophisticated radar fences were already under design by the time Pinetree Line construction started, leaving some Pinetree stations to operate for just four years. It is a testament to Pole Vault that it outlived much of the radar system it was designed to support, becoming an integral part of not one, or even two, but at least three later radar early warning programs. Moreover, Pole Vault became a template for troposcatter systems elsewhere in Canada, in Europe, and in the United States. But we'll have to talk about those later.

[1] Alexander Graham Bell was Scottish-Canadian-American, and lived for some time in rural Ontario and later Montreal. As a result, Bell Canada is barely younger than its counterpart in the United States and the early history of the two is more one of parallel development than the establishment of a foreign subsidiary. Bell's personal habit of traveling back and forth between Montreal and Boston makes the early interplay of the two companies a bit confusing. In 1955, the TAT-1 telephone cable would conquer the Atlantic ocean to link the US to Scotland via Canada, incidentally making a charming gesture to Bell's personal journey.

[2] If you have studied weather a bit, you might recognize these as positive and negative lapse rates. The positive lapse rate in the troposphere is a major driver in the various phenomenon we call "weather," and the tropopause forms a natural boundary that keeps most weather within the troposphere. Commercial airliners fly in the lower part of the stratosphere, putting them above most (but not all) weather.

I have been slowly working on a book. Don't get too excited, it is on a very niche topic and I will probably eventually barely finish it and then post it here. But in the mean time, I will recount some stories which are related, but don't quite fit in. Today, we'll learn a bit about the self-illumination industry.

At the turn of the 20th century, it was discovered that the newfangled element radium could be combined with a phosphor to create a paint that glowed. This was pretty much as cool as it sounds, and commercial radioluminescent paints like Undark went through periods of mass popularity. The most significant application, though, was in the military: radioluminescent paints were applied first to aircraft instruments and later to watches and gunsights. The low light output of radioluminescent paints had a tactical advantage (being very difficult to see from a distance), while the self-powering nature of radioisotopes made them very reliable.

The First World War was thus the "killer app" for radioluminescence. Military demand for self-illuminating devices fed a "radium rush" that built mines, processing plants, and manufacturing operations across the country. It also fed, in a sense much too literal, the tragedy of the "Radium Girls." Several self-luminous dial manufacturers knowingly subjected their women painters to shockingly irresponsible conditions, leading inevitably to radium poisoning that disfigured, debilitated, and ultimately killed. Today, this is a fairly well-known story, a cautionary tale about the nuclear excess and labor exploitation of the 1920s. That the situation persisted into the 1940s is often omitted, perhaps too inconvenient to the narrative that a series of lawsuits, and what was essentially the invention of occupational medicine, headed off the problem in the late 1920s.

What did happen after the Radium Girls? What was the fate of the luminous radium industry?

A significant lull in military demand after WWI was hard on the radium business, to say nothing of a series of costly settlements to radium painters despite aggressive efforts to avoid liability. At the same time, significant radium reserves were discovered overseas, triggering a price collapse that closed most of the mines. The two largest manufacturers of radium dials, Radium Dial Company (part of Standard Chemical who owned most radium mines) and US Radium Corporation (USRC), both went through lean times. Fortunately, for them, the advent of the Second World War reignited demand for radioluminescence.

The story of Radium Dial and USRC doesn't end in the 1920s---of course it doesn't, luminous paints having had a major 1970s second wind. Both companies survived, in various forms, into the current century. In this article, I will focus on the post-WWII story of radioactive self-illumination and the legacy that we live with today.

During its 1920s financial difficulties, the USRC closed the Orange, New Jersey plant famously associated with Radium Girls and opened a new facility in Brooklyn. In 1948, perhaps looking to manage expenses during yet another post-war slump, USRC relocated again to Bloomsburg, Pennsylvania. The Bloomsburg facility, originally a toy factory, operated through a series of generational shifts in self-illuminating technology.

The use of radium, with some occasional polonium, for radioluminescence declined in the 1950s and ended entirely in the 1970s. The alpha radiation emitted by those elements is very effective in exciting phosphors but so energetic that it damages them. A longer overall lifespan, and somewhat better safety properties, could be obtained by the use of a beta emitter like strontium or tritium. While strontium was widely used in military applications, civilian products shifted towards tritium, which offered an attractive balance of price and half life.

USRC handled almost a dozen radioisotopes in Bloomsburg, much of them due to diversified operations during the 1950s that included calibration sources, ionizers, and luminous products built to various specific military requirements. The construction of a metal plating plant enabled further diversification, including foil sources used in research, but eventually became an opportunity for vertical integration. By 1968, USRC had consolidated to only tritium products, with an emphasis on clocks and watches.

Radioluminescent clocks were a huge hit, in part because of their practicality, but fashion was definitely a factor. Millions of radioluminescent clocks were sold during the '60s and '70s, many of them by Westclox. Westclox started out as a typical clock company (the United Clock Company in 1885), but joined the atomic age through a long-lived partnership with the Radium Dial Company. The two companies were so close that they became physically so: Radium Dial's occupational health tragedy played out in Ottawa, Illinois, a town Radium Dial had chosen as its headquarters due to its proximity to Westclox in nearby Peru [1].

Westclox sold clocks with radioluminescent dials from the 1920s to probably the 1970s, but one of the interesting things about this corner of atomic history is just how poorly documented it is. Westclox may have switched from radium to tritium at some point, and definitely abandoned radioisotopes entirely at some point. Clock and watch collectors, a rather avid bunch, struggle to tell when. Many consumer radioisotopes are like this: it's surprisingly hard to know if they even are radioactive.

Now, the Radium Dial Company itself folded entirely to a series of radium poisoning lawsuits in the 1930s. Simply being found guilty of one of the most malevolent labor abuses of the era would not stop free enterprise, though, and Radium Dial's president founded a legally distinct company called Luminous Processes just down the street. Luminous Processes is particularly notable for having continued the production of radium-based clock faces until 1978, making them the last manufacturer of commercial radioluminescent radium products. This also presents compelling circumstantial evidence that Westclox continued to use radium paint until sometime around 1978, which lines up with the general impressions of luminous dial collectors.

While the late '70s were the end of Radium Dial, USRC was just beginning its corporate transformation. From 1980 to 1982, a confusing series of spinoffs and mergers lead to USR Industries, parent company of Metreal, parent company of Safety Light Corporation, which manufactured products to be marketed and distributed by Isolite. All of these companies were ultimately part of USR Industries, the former USRC, but the org chart sure did get more complex. The Nuclear Regulatory Commission expressed some irritation in their observation, decades later, that they weren't told about any of this restructuring until they noticed it on their own.

Safety Light, as expressed by the name, focused on a new application for tritium radioluminescence: safety signage, mostly self-powered illuminated exit signs and evacuation signage for aircraft. Safety Light continued to manufacture tritium exit signs until 2007, when they shut down following some tough interactions with the NRC and the EPA. They had been, in the fashion typical of early nuclear industry, disposing of their waste by putting it in a hole in the ground. They had persisted in doing this much longer than was socially acceptable, and ultimately seem to have been bankrupted by their environmental obligations... obligations which then had to be assumed by the Superfund program.

The specific form of illumination used in these exit signs, and by far the most common type of radioluminescence today, is the Gaseous Tritium Light Source or GTLS. GTLS are small glass tubes or vials, usually made with borosilicate glass, containing tritium gas and an internal coating of phosphor. GTLS are simple, robust, and due to the very small amount of tritium required, fairly inexpensive. They can be made large enough to illuminate a letter in an exit sign, or small enough to be embedded into a watch hand. Major applications include watch faces, gun sights, and the keychains of "EDC" enthusiasts.

Plenty of GTLS manufacturers have come and gone over the years. In the UK, defense contractor Saunders-Roe got into the GTLS business during WWII. Their GTLS product line moved to Brandhurst Inc., which had a major American subsidiary. It is an interesting observation that the US always seems to have been the biggest market for GTLS, but their manufacture has increasingly shifted overseas. Brandhurst is no longer even British, having gone the way of so much of the nuclear world by becoming Canadian. A merger with Canadian company SRB created SRB Technologies in Pembroke, Ontario, which continues to manufacture GTLS today.

Other Canadian GTLS manufacturers have not fared as well. Shield Source Inc., of Peterborough, Ontario, began filling GTLS vials in 1987. I can't find a whole lot of information on Shield Source's early days, but they seem to have mostly made tubes for exit signs, and perhaps some other self-powered signage. In 2012, the Canadian Nuclear Safety Commission (CNSC) detected a discrepancy in Shield Source's tritium emissions monitoring. I am not sure of the exact details, because CNSC seems to make less information public in general than the US NRC [2].

Here's what appears to have happened: tritium is a gas, which makes it tricky to safely handle. Fortunately, the activity of tritium is relatively low and its half life is relatively short. This means that it's acceptable to manage everyday leakage (for example when connecting and disconnecting things) in a tritium workspace by ventilating it to a stack, releasing it to the atmosphere for dilution and decay. The license of a tritium facility will specify a limit for how much radioactivity can be released this way, and monitoring systems (usually several layers of monitoring systems) have to be used to ensure that the permit limit is not exceeded. In the case of Shield Source, some kind of configuration error with the tritium ventilation monitoring system combined with a failure to adequately test and audit it. The CNSC discovered that during 2010 and 2011, the facility had undercounted their tritium emissions, and in fact exceeded the limits of their license.

Air samplers located around the facility, some of which were also validated by an independent laboratory, did not detect tritium in excess of the environmental limits. This suggests that the excess releases probably did not have an adverse impact on human health or the environment. Still, exceeding license terms and then failing to report and correct the problem for two years is a very serious failure by a licensee. In 2012, when the problem was discovered, CNSC ordered Shield Source's license modified to prohibit actual tritium handling. This can seem like an odd maneuver but something similar can happen in the US. Just having radioisotope-contaminated equipment, storing test sources, and managing radioactive waste requires a license. By modifying Shield Source's license to prohibit tritium vial filling, the CNSC effectively shut the plant down while allowing Shield Source to continue their radiological protection and waste management functions. This is the same reason that long-defunct radiological facilities often still hold licenses from NRC in the US: they retain the licenses to allow them to store and process waste and contaminated materials during decommissioning.

In the case of Shield Source, while the violation was serious, CNSC does not seem to have anticipated a permanent shutdown. The terms agreed in 2012 were that Shield Source could regain a license to manufacture GTLS if it produced for CNSC a satisfactory report on the root cause of the failure and actions taken to prevent a recurrence. Shield Source did produce such a report, and CNSC seems to have mostly accepted it with some comments requesting further work (the actual report does not appear to be public). Still, in early 2013, Shield Source informed CNSC that it did not intend to resume manufacturing. The license was converted to a one-year license to facilitate decommissioning.

Tritium filling and ventilation equipment, which had been contaminated by long-term exposure to tritium, was "packaged" and disposed. This typically consists of breaking things down into parts small enough to fit into 55-gallon drums, "overpacking" those drums into 65-gallon drums for extra protection, and then coordinating with transportation authorities to ship the materials in a suitable way to a facility licensed to dispose of them. This is mostly done by burying them in the ground in an area where the geology makes groundwater interaction exceedingly unlikely, like a certain landfill on the Texas-New Mexico border near Eunice. Keep in mind that tritium's short half life means this is not a long-term geological repository situation; the waste needs to be safely contained for only, say, fifty years to get down to levels not much different from background.

I don't know where the Shield Source waste went, CNSC only says it went to a licensed facility. Once the contaminated equipment was removed, drywall and ceiling and floor finishes were removed in the tritium handling area and everything left was thoroughly cleaned. A survey confirmed that remaining tritium contamination was below CNSC-determined limits (for example, in-air concentrations that would lead to a dose of less than 0.01 mSv/year for 9-5 occupational exposure). At that point, the Shield Source building was released to the landlord they had leased it from, presumably to be occupied by some other company. Fortunately tritium cleanup isn't all that complex.

You might wonder why Shield Source abruptly closed down. I assume there was some back-and-forth with CNSC before they decided to throw in the towel, but it is kind of odd that they folded entirely during the response to an incident that CNSC seems to have fully expected them to survive. I suspect that a full year of lost revenue was just too much for Shield Source: by 2012, when all of this was playing out, the radioluminescence market had seriously declined.

There are a lot of reasons. For one, the regulatory approach to tritium has become more and more strict over time. Radium is entirely prohibited in consumer goods, and the limit on tritium activity is very low. Even self-illuminating exit signs now require NRC oversight in the US, discussed shortly. Besides, public sentiment has increasingly turned against the Friendly Atom is consumer contexts, and you can imagine that people are especially sensitive to the use of tritium in classic institutional contexts for self-powered exit signs: schools and healthcare facilities.

At the same time, alternatives have emerged. Non-radioactive luminescent materials, the kinds of things we tend to call "glow in the dark," have greatly improved since WWII. Strontium aluminate is a typical choice today---the inclusion of strontium might suggest otherwise, but strontium aluminate uses the stable natural isotope of strontium, Sr-88, and is not radioactive. Strontium aluminate has mostly displaced radioluminescence in safety applications, and for example the FAA has long allowed it for safety signage and path illumination on aircraft. Keep in mind that these luminescent materials are not self-powered. They must be "charged" by exposure to light. Minor adaptations are required, for example a requirement that the cabin lights in airliners be turned on for a certain period of time before takeoff, but in practice these limitations are considered preferable to the complexity and risks involved in the use of radioisotopes.

You are probably already thinking that improving electronics have also made radioluminescence less relevant. Compact, cool-running, energy-efficient LEDs and a wide variety of packages and form factors mean that a lot of traditional applications of radioluminescence are now simply electric. Here's just a small example: in the early days of LCD digital watches, it was not unusual for higher-end models to use a radioluminescent source as a backlight. Today that's just nonsensical, a digital watch needs a power source anyway and in even the cheapest Casios a single LED offers a reasonable alternative. Radioluminescent digital watches were very short lived.

Now that we've learned about a few historic radioluminescent manufacturers, you might have a couple of questions. Where were the radioisotopes actually sourced? And why does Ontario come up twice? These are related. From the 1910s to the 1950s, radioluminescent products were mostly using radium sourced from Standard Chemical, who extracted it from mines in the Southwest. The domestic radium mining industry collapsed by 1955 due to a combination of factors: declining demand after WWII, cheaper radium imported from Brazil, and a broadly changing attitude towards radium that lead the NRC to note in the '90s that we might never again find the need to extract radium: radium has a very long half life that makes it considerably more difficult to manage than strontium or tritium. Today, you could say that the price of radium has gone negative, in that you are far more likely to pay an environmental management company to take it away (at rather high prices) than to buy more.

But what about tritium? Tritium is not really naturally occurring; there technically is some natural tritium but it's at extremely low concentrations and very hard to get at. But, as it happens, irradiating water produces a bit of tritium, and nuclear reactors incidentally irradiate a lot of water. With suitable modifications, the tritium produced as a byproduct of civilian reactors can be concentrated and sold. Ontario Hydro has long had facilities to perform this extraction, and recently built a new plant at the Darlington Nuclear Station that processes heavy water shipped from CANDU reactors throughout Ontario. The primary purpose of this plant is to reduce environmental exposure from the release of "tritiated" heavy water; it produces more tritium than can reasonably be sold, so much of it is stored for decay. The result is that tritium is fairly abundant and cheap in Ontario.

Besides SRB Technologies which packages tritium from Ontario Hydro into GTLS, another major manufacturer of GTLS is the Swiss company mb-microtec. mb-microtec is the parent of watch brand Traser and GTLS brand Trigalight, and seem to be one of the largest sources of consumer GTLS overall. Many of the tritium keychains you can buy, for example, use tritium vials manufactured by mb-microtec. NRC documents suggest that mb-microtec contracts a lot of their finished product manufacturing to a company in Hong Kong and that some of the finished products you see using their GTLS (like watches and fobs) are in fact white-labeled from that plant, but unfortunately don't make the original source of the tritium clear. mb-microtec has the distinction of operating the only recycling plant for tritium gas, and press releases surrounding the new recycling operation say they purchase the rest of their tritium supply. I assume from the civilian nuclear power industry in Switzerland, which has several major reactors operating.

A number of other manufacturers produce GTLS primarily for military applications, with some safety signage side business. And then there is, of course, the nuclear weapons program, which consumes the largest volume of tritium in the US. The US's tritium production facility for much of the Cold War actually shut down in 1988, one of the factors in most GTLS manufacturers being overseas. In the interim period, the sole domestic tritium supply was recycling of tritium in dismantled weapons and other surplus equipment. Since tritium has such a short half-life, this situation cannot persist indefinitely, and tritium production was resumed in 2004 at the Tennessee Valley Authority's Watts Bar nuclear generating station. Tritium extracted from that plant is currently used solely by the Department of Energy, primarily for the weapons program.

Finally, let's discuss the modern state of radioluminescence. GTLS, based on tritium, are the only type of radioluminescence available to consumers. All importation and distribution of GTLS requires an NRC license, although companies that only distribute products that have been manufactured and tested by another licensee fall under a license exemption category that still requires NRC reporting but greatly simplifies the process. Consumers that purchase these items have no obligations to the NRC. Major categories of devices under these rules include smoke detectors, detection instruments and small calibration sources, and self-luminous products using tritium, krypton, or promethium. You might wonder, "how big of a device can a I buy under these rules?" The answer to that question is a bit complicated, so let me explain my understanding of the rules using a specific example.

Let's say you buy a GTLS keychain from massdrop or wherever people get EDC baubles these days [3]. The business you ordered it from almost certainly did not make it, and is acting as an NRC exempt distributor of a product. In NRC terms, your purchase of the product is not the "initial sale or distribution," that already happened when the company you got it from ordered it from their supplier. Their supplier, or possibly someone further up in the chain, does need to hold a license: an NRC specific license is required to manufacture, process, produce, or initially transfer or sell tritium products. This is the reason that overseas companies like SRB and mb-microtec hold NRC licenses; this is the only way for consumers to legally receive their products.

It is important to note the word "specific" in "NRC specific license." These licenses are very specific; the NRC approves each individual product including the design of the containment and and labeling. When a license is issued, the individual products are added to a registry maintained by the NRC. When evaluating license applications, the NRC considers a set of safety objectives rather than specific criteria. For example, and if you want to read along we're in 10 CFR 32.23:

In normal use and disposal of a single exempt unit, it is unlikely that the external radiation dose in any one year, or the dose commitment resulting from the intake of radioactive material in any one year, to a suitable sample of the group of individuals expected to be most highly exposed to radiation or radioactive material from the product will exceed the dose to the appropriate organ as specified in Column I of the table in § 32.24 of this part.

So the rules are a bit soft, in that a licensee can argue back and forth with the NRC over means of calculating dose risk and so on. It is, ultimately, the NRC's discretion as to whether or not a device complies. It's surprisingly hard to track down original licensing paperwork for these products because of how frequently they are rebranded, and resellers never seem to provide detailed specifications. I suspect this is intentional, as I've found some cases of NRC applications that request trade secret confidentiality on details. Still, from the license paperwork I've found with hard numbers, it seems like manufacturers keep the total activity of GTLS products (e.g. a single GTLS sold alone, or the total of the GTLS in a watch) under 25 millicurie.

There do exist larger devices, of which exit signs are the largest category. Self-powered exit signs are also manufactured under NRC specific licenses, but their activity and resulting risk is too high to qualify for exemption at the distribution and use stage. Instead, all users of self-powered safety signs do so under a general license issued by the NRC (a general license meaning that it is implicitly issued to all such users). The general license is found in 10 CFR 31. Owners of tritium exit signs are required to designate a person to track and maintain the signs, inform the NRC of that person's contact information and any changes in that person, to inform the NRC of any lost, stolen, or damaged signs. General licensees are not allowed to sell or otherwise transfer tritium signs, unless they are remaining in the same location (e.g. when a building is sold), in which case they must notify the NRC and disclose NRC requirements to the transferee.

When tritium exit signs reach the end of their lifespan, they must be disposed of by transfer to an NRC license holder who can recycle them. The general licensee has to notify the NRC of that transfer. Overall, the intent of the general license regulations is to ensure that they are properly disposed of: reporting transfers and events to the NRC, along with serial numbers, allows the NRC to audit for signs that have "disappeared." Missing tritium exit signs are a common source of NRC event reports. It should also be said that, partly for these reasons, tritium exit signs are pretty expensive. Roughly $300 for a new one, and $150 to dispose of an old one.

Other radioluminescent devices you will find are mostly antiques. Radium dials are reasonably common, anything with a luminescent dial made before, say, 1960 is probably radium, and specifically Westclox products to 1978 likely use radium. The half-life of radium-226 is 1,600 years, so these radium dials have the distinction of often still working, although the paints have usually held up more poorly than the isotopes they contain. These items should be handled with caution, since the failure of the paint creates the possibility of inhaling or ingesting radium. They also emit radon as a decay product, which becomes hazardous in confined spaces, so radium dials should be stored in a well-ventilated environment.

Strontium-90 has a half-life of 29 years, and tritium 12 years, so vintage radioluminescent products using either have usually decayed to the extent that they no longer shine brightly or even at all. The phosphors used for these products will usually still fluoresce brightly under UV light and might even photoluminesce for a time after light exposure, but they will no longer stay lit in a dark environment. Fortunately, the decay that makes them not work also makes them much safer to handle. Tritium decays to helium-3 which is quite safe, strontium-90 to yttrium-90 which quickly decays to zirconium-90. Zirconium-90 is stable and only about as toxic as any other heavy metal. You can see why these radioisotopes are now much preferred over radium.

And that's the modern story of radioluminescence. Sometime soon, probably tomorrow, I will be sending out my supporter's newsletter, EYES ONLY, with some more detail on environmental remediation at historic processing facilities for radioluminescent products. You can learn a bit more about how US Radium was putting their waste in a hole in the ground, and also into a river, and sort of wherever else. You know Radium Dial Company was up to similar abuses.

[1] The assertion that Ottawa is conveniently close to Peru is one of those oddities of naming places after bigger, more famous places.

[2] CNSC's whole final report on Shield Source is only 25 pages. A similar decommissioning process in the US would produce thousands of pages of public record typically culminating in EPA Five Year Reviews which would be, themselves, perhaps a hundred pages depending on the amount of post-closure monitoring. I'm not familiar with the actual law but it seems like most of the difference is that CNSC does not normally publish technical documentation or original data (although one document does suggest that original data is available on request). It's an interesting difference... the 25-page report, really only 20 pages after front matter, is a lot more approachable for the public than a 400 page set of close-out reports. Much of the standard documentation in the US comes from NEPA requirements, and NEPA is infamous in some circles for requiring exhaustive reports that don't necessarily do anything useful. But from my perspective it is weird for the formal, published documentation on closure of a radiological site to not include hydrology discussion, demographics, maps, and fifty pages of data tables as appendices. Ideally a bunch of one-sentence acceptance emails stapled to the end for good measure. When it comes to describing the actual problem, CNSC only gives you a couple of paragraphs of background.

[3] Really channeling Guy Debord with my contempt for keychains here. during the writing of this article, I bought myself a tritium EDC bauble, so we're all in the mud together.

September 6th, 1996. Eddie Murray, of the Baltimore Orioles, is at bat. He has had 20 home runs in the season; 499 in his career. Anticipation for the 500th had been building for the last week. It would make Murray only the third player to reach 500 home runs and 3000 hits. His career RBI would land in the top ten hitters in the history of the sport; his 500th home run was a statistical inevitability. Less foreseeable was the ball's glancing path through one of the most famous stories of the telephone business.

Statistics only tell you what might happen. Michael Lasky had made a career, a very lucrative one, of telling people what would happen. Lasky would have that ball. As usual, he made it happen by advertising.

Clearing right field, the ball landed in the hands of Dan Jones, a salesman from Towson, Maryland. Despite his vocation, he didn't immediately view his spectacular catch in financial terms. He told a newspaper reporter that he looked forward to meeting Murray, getting some signatures, some memorabilia. Instead, he got offers. At least three parties inquired about purchasing the ball, but the biggest offer came far from discreetly: an ad in the Baltimore Sun offering half a million dollars to whoever had it. Well, the offer was actually for a $25,000 annuity for 20 years, with a notional cash value of half a million but a time-adjusted value of $300,000 or less. I couldn't tell for sure, but given events that would follow, it seems unlikely that Jones ever received more than a few of the payments anyway. Still, the half a million made headlines, and NPV or not the sale price still set the record for a public sale of sports memorabilia.

Lasky handled his new purchase with his signature sense of showmanship. He held a vote, a telephone vote: two 1-900 numbers, charging $0.95 a call, allowed the public to weigh in on whether he should donate the ball to the Babe Ruth Birthplace museum or display it in the swanky waterfront hotel he part-owned. The proceeds went to charity, and after the museum won the poll, the ball did too. The whole thing was a bit of a publicity stunt, Lasky thrived on unsubtle displays and he could part with the money. His 1-900 numbers were bringing in over $100 million a year.

---

Lasky's biography is obscure. Born 1942 in Brooklyn, he moved to Baltimore in the 1960s for some reason connected to a conspicuous family business: a blood bank. Perhaps the blood bank was a grift, it's hard to say now, but Lasky certainly had a unique eye for business. He was fond of horse racing, or really, of trackside betting. His father, a postal worker, had a proprietary theory of mathematics that he applied to predicting the outcome of the race. This art, or science, or sham, is called handicapping, and it became Lasky's first real success. Under the pseudonym Mike Warren, he published the Baltimore Bulletin, a handicapping newsletter advertising sure bets at all the region's racetracks.

Well, there were some little details of this business, some conflicts of interest, a little infringement on the trademark of the Preakness. The details are neither clear nor important, but he had some trouble with the racing commissions in at least three states. He probably wouldn't have tangled with them at all if he weren't stubbornly trying to hold down a license to breed racehorses while also running a betting cartel, but Lasky was always driven more by passion than reason.

Besides, he had other things going. Predicting the future in print was sort of an industry vertical, and he diversified. His mail-order astrology operation did well, before the Postal Service shut it down. He ran some sort of sports pager service, probably tied to betting, and I don't know what came of that. Perhaps on the back of a new year's resolution, he ran a health club, although it collapsed in 1985 with a bankruptcy case that revealed some, well, questionable practices. Strange that a health club just weeks away from bankruptcy would sell so many multi-year memberships, paid up front. And where did that money go, anyway?

No matter, Lasky was onto the next thing. During the 1980s, changes had occurred that would grow Lasky's future-predicting portfolio into a staple of American media. First, in 1984, a Reagan-era FCC voted to end most regulation of television advertising. Gone was the limit of 16 minutes per hour of paid programming. An advertiser could now book entire half-hour schedule slots. Second, during the early '80s AT&T standardized and promoted a new model in telephone billing. The premium-rate number, often called a "1-900 number" after the NPA assigned for their use, incurred a callee-determined per-minute toll that the telco collected and paid on to the callee. It's a bit like a nascent version of "Web 3.0": telephone microtransactions, an innovative new way to pay for information services.

It seems like a fair assumption that handicapping brought Lasky to the 1-900 racket, and he certainly did offer betting tip lines. But he had learned a thing or two from the astrology business, even if it ran afoul of Big Postal. Handicapping involved a surprising amount of work, and its marketing centered around the supposedly unique insight of the handicapper. Fixed recordings of advice could only keep people on a telephone line for so long, anyway. Astrology, though, involved even fewer facts, and even more opportunity to ramble. Best of all, there was an established industry of small-time psychics working out of their homes. With the magic of the telephone, every one of them could offer intuitive readings to all of America, for just $3.99 a minute.

In 1990, Lasky's new "direct response marketing" company Inphomation, Inc. contracted five-time Grammy winner Dionne Warwick, celebrity psychic Linda Georgian, and a studio audience to produce a 30 minute talk-show "infomercial" promoting the Psychic Friends Network. Over the next few years, Inphomation conjoined with an ad booking agency and a video production company under the ownership of Mike Lasky's son, Marc Lasky. Inphomation spent as much as a million a week in television bookings, promoting a knitting machine and a fishing lure and sports tips, but most of all psychics. The original half-hour Psychic Friends Network spot is often regarded as the most successful infomercial in history. It remade Warwick's reputation, turning her from a singer to a psychic promoter. Calls to PFN's 1-900 number, charged at various rates that could reach over $200 an hour, brought in $140 million in revenue in its peak years of the mid 1990s.

Lasky described PFN as an innovative new business model, but it's one we now easily recognize as "gig work." Telephone psychics, recruited mostly by referral from the existing network, worked from home, answering calls on their own telephones. Some read Tarot, some gazed into crystals, others did nothing at all, but the important thing was that they kept callers on the line. After the phone company's cut and Inphomation's cut, they were paid a share of the per-minute rate that automatically appeared on caller's monthly phone bills.

---

A lot of people, and even some articles written in the last decade, link the Psychic Friends Network to "Miss Cleo." There's sort of a "Berenstain Bears" effect happening here; as widely as we might remember Miss Cleo's PFN appearances there are no such thing. Miss Cleo was actually the head psychic and spokeswoman of the Psychic Reader's Network, which would be called a competitor to the Psychic Friends Network except that they didn't operate at the same time. In the early '00s, the Psychic Reader's Network collapsed in scandal. The limitations of its business model, a straightforward con, eventually caught up. It was sued out of business by a dozen states, then the FTC, then the FCC just for good measure.

The era of the 1-900 number was actually rather short. By the late '80s, it had already become clear that the main application of premium rate calling was not stock quotations or paid tech support or referral services. It was scams. An extremely common genre of premium rate number, almost the lifeblood of the industry, were joke lines that offered telephonic entertainment in the voice of cartoon characters. Advertisements for these numbers, run during morning cartoons, advised children to call right away. Their parents wouldn't find out until the end of the month, when the phone bill came and those jokes turned out to have run $50 in 1983's currency. Telephone companies were at first called complicit in the grift, but eventually bowed to pressure and, in 1987, made it possible for consumers to block 1-900 calling on their phone service. Of course, few telephone customers took advantage, and the children's joke line racket went on into the early '90s when a series of FTC lawsuits finally scared most of them off the telephone network.

Adult entertainment was another touchstone of the industry, although adult lines did not last as long on 1-900 numbers as we often remember. Ripping off adults via their children is one thing; smut is a vice. AT&T and MCI, the dominant long distance carriers and thus the companies that handled most 1-900 call volume, largely cut off phone sex lines by 1991. Congress passed a law requiring telephone carriers to block them by default anyway, but of course left other 1-900 services as is. Phone sex lines were far from gone, of course, but they had to find more nuanced ways to make their revenue: international rates and complicit telephone carriers, dial-around long distance revenue, and whatever else they could think of that regulators hadn't caught up to yet.

When Miss Cleo and her Psychic Reader's Network launched in 1997, psychics were still an "above board" use of the 1-900 number. The Psychic Readers lived to see the end of that era. In the late '90s, regulations changed to make unpaid 1-900 bills more difficult to collect. By 2001, some telephone carriers had dropped psychic lines from their networks as a business decision. The bill disputes simply weren't worth the hassle. In 2002, AT&T ended 1-900 billing entirely. Other carriers maintained premium-rate billing for a decade later, but AT&T had most of the customer volume anyway.

The Psychic Friends Network, blessed by better vision, struck at the right time. 1990 to 1997 were the golden age of 1-900 and the golden age of Inphomation. Inphomation's three-story office building in Baltimore had a conference room with a hand-painted ceiling fresco of cherubs and clouds. In the marble-lined lobby, a wall of 25 televisions played Inphomation infomercials on repeat. At its peak, the Psychic Friends Network routed calls to 2,000 independent psychic contractors. Dionne Warwick and Linda Georgian were famous television personalities; Warwick wasn't entirely happy about her association with the brand but she made royalties whenever the infomercial aired. Some customers spent tens of thousands of dollars on psychic advice.

In 1993, a direct response marketing firm called Regal Communications made a deal to buy Inphomation. The deal went through, but just the next year Regal spun their entire 1-900 division off, and Inphomation exercised an option to become an independent company once again. A decade later, many of Regal's executives would face SEC charges over the details of Regal's 1-900 business, foreshadowing a common tendency of Psychic Friends Network owners.

The psychic business, it turns out, was not so unlike the handicapping business. Both were unsavory. Both made most of their money off of addicts. In the press, Lasky talked about casual fans that called for two minutes here and there. What's $5 for a little fun? You might even get some good advice. Lawsuits, regulatory action, and newspaper articles told a different story. The "30 free minutes" promotion used to attract new customers only covered the first two minutes of each call, the rest were billed at an aggressive rate. The most important customers stayed on the line for hours. Callers had to sit through a few minutes of recordings, charged at the full rate, before being connected to a psychic who drew out the conversation by speaking slowly and asking inane questions. Some psychics seem to have approached their job rather sincerely, but others apparently read scripts.

And just like the horse track, the whole thing moved a lot of money. Lasky continued to tussle with racing commissions over his thoroughbred horses. He bought a Mercedes, a yacht, a luxury condo, a luxury hotel whose presidential suite he used as an apartment, a half-million-dollar baseball. Well, a $300,000 baseball, at least.

Eventually, the odds turned against Lasky. Miss Cleo's Psychic Reader's Network was just one of the many PFN lookalikes that popped up in the late '90s. There was a vacuum to fill, because in 1997, Inphomation was descending into bankruptcy.

---

Opinions differ on Lasky's management and leadership. He was a visionary at least once, but later decisions were more variable. Bringing infomercial production in-house through his son's Pikesville Pictures might have improved creative control, but production budgets ballooned and projects ran late. PFN was still running mainly off of the Dionne Warwick shows, which were feeling dated, especially after a memorable 1993 Saturday Night Live parody featuring Christopher Walken. Lasky's idea for a radio show, the Psychic Friends Radio Network, had a promising trial run but then faltered on launch. Hardly a half dozen radio stations picked it up, and it lost Inphomation tens of millions of dollars. While they were years ahead of the telephone industry cracking down on psychics, PFN still struggled with a timeless trouble of the telephone network: billing.

AT&T had a long-established practice of withholding a portion of 1-900 revenue for chargebacks. Some customers see the extra charges on their phone bills and call in with complaints; the telephone company, not really being the beneficiary of the revenue anyway, was not willing to go to much trouble to keep it and often agreed to a refund. Holding, say, 10% of a callee's 1-900 billings in reserve allowed AT&T to offer these refunds without taking a loss. The psychic industry, it turned out, was especially prone to end-of-month customer dissatisfaction. Chargebacks were so frequent that AT&T raised Inphomation's withholding to 20%, 30%, and even 40% of revenue.

At least, that's how AT&T told it. Lasky always seemed skeptical, alleging that the telephone companies were simply refusing to hand over money Inphomation was owed, making themselves a free loan. Inphomation brokered a deal to move their business elsewhere, signing an exclusive contract with MCI. MCI underdelivered: they withheld just as much revenue, in violation of the contract according to Lasky, and besides the MCI numbers suffered from poor quality and dropped calls. At least, that's how Inphomation told it. Maybe the dropped calls were on Inphomation's end, and maybe they had a convenient positive effect on revenue as callers paid for a few minute of recordings before being connected to no one at all. By the time the Psychic Friends Network fell apart, there was a lot of blame passed around. Lasky would eventually prevail in a lawsuit against MCI for unpaid revenue, but not until too late.

By some combination of a lack of innovation in their product, largely unchanged since 1991, and increasing expenses for both advertising and its founder's lifestyle, Inphomation ended 1997 over $20 million in the red. In 1998 they filed for Chapter 11, and Lasky sought to reorganize the company as debtor-in-possession.

The bankruptcy case brought out some stories of Lasky's personal behavior. While some employees stood by him as a talented salesman and apt finder of opportunities, others had filed assault charges. Those charges were later dropped, but by many accounts, he had quite a temper. Lasky's habit of not just carrying but brandishing a handgun around the office certainly raised eyebrows. Besides, his expensive lifestyle persisted much too far into Inphomation's decline. The bankruptcy judge's doubts about Lasky reached a head when it was revealed that he had tried to hide the company's assets. Much of the infrastructure and intellectual property of the Psychic Friends Network, and no small amount of cash, had been transferred to the newly formed Friends of Friends LLC in the weeks before bankruptcy.

The judge also noticed some irregularities. The company controller had been sworn in as treasurer, signed the bankruptcy petition, and then resigned as treasurer in the space of a few days. When asked why the company chose this odd maneuver over simply having Lasky, corporate president, sign the papers, Lasky had trouble recalling the whole thing. He also had trouble recalling loans Inphomation had taken, meetings he had scheduled, and actions he had taken. When asked about Inphomation's board of directors, Lasky didn't know who they were, or when they had last met.

The judge used harsh language. "I've seen nothing but evidence of concealment, dishonesty, and less than full disclosure... I have no hope this debtor can reorganize with the present management." Lasky was removed, and a receiver appointed to manage Inphomation through a reorganization that quietly turned into a liquidation. And that was almost the end of the Psychic Friends Network.

The bankruptcy is sometimes attributed to Lasky's failure to adapt to the times, but PFN wasn't entirely without innovation. The Psychic Friends Network first went online, at psychicfriendsnetwork.com, in 1997. This website, launched in the company's final days, offered not only the PFN's 1-900 number but a landing page for a telephone-based version of "Colorgenics." Colorgenics was a personality test based on the "Lüscher color test," an assessment designed by a Swiss psychotherapist based on nothing in particular. There are dozens of colorgenics tests online today, many of which make various attempts to extract money from the user, but none with quite the verve of a color quiz via 1-900 number.

Inphomation just didn't quite make it in the internet age, or at least not directly. Most people know 1998 as the end of the Psychic Friends Network. The Dionne Warwick infomercials were gone, and that was most of PFN anyway. Without Linda Georgian, could PFN live on? Yes, it turns out, but not in its living form. The 1998 bankruptcy marked PFN's transition from a scam to the specter of a scam, and then to a whole different kind of scam. It was the beginning of the PFN's zombie years.

In 1999, Inphomation's assets were liquidated at auction for $1.85 million, a far cry from the company's mid-'90s valuations in the hundreds of millions. The buyer: Marc Lasky, Michael Lasky's son. PFN assets became part of PFN Holdings Inc., with Michael Lasky and Marc Lasky as officers. PFN was back.

It does seem that the Laskys made a brief second crack at a 1-900 business, but by 1999 the tide was clearly against expensive psychic hotlines. Telephone companies had started their crackdown, and attorney general lawsuits were brewing. Besides, after the buyout PFN Holdings didn't have much capital, and doesn't seem to have done much in the way of marketing. It's obscure what happened in these years, but I think the Laskys licensed out the PFN name.

psychicfriendsnetwork.com, from 2002 to around 2009, directed visitors to Keen. Keen was the Inphomation of the internet age, what Inphomation probably would have been if they had run their finances a little better in '97. Backed by $60 million in venture funding from names like Microsoft and eBay, Keen was a classic dotcom startup. They launched in '99 with the ambitious and original idea of operating a web directory and reference library. Like most of the seemingly endless number of reference website startups, they had to pivot to something else. Unlike most of the others, Keen and their investors had a relaxed set of moral strictures about the company's new direction. In the early 2000s, keen.com was squarely in the ethical swamp that had been so well explored by the 1-900 business. Their web directory specialized in phone sex and psychic advice---all offered by 1-800 numbers with convenient credit card payment, a new twist on the premium phone line model that bypassed the vagaries and regulations of telephone billing.

Keen is, incidentally, still around today. They'll broker a call or chat with empath/medium Citrine Angel, offering both angel readings and clairsentience, just $1 for the first 5 minutes and $2.99 a minute thereafter. That's actually a pretty good deal compared to the Psychic Friends Network's old rates. Keen's parent company, Ingenio, runs a half dozen psychic advice websites and a habit tracking app. But it says something about the viability of online psychics that Keen still seems to do most of their business via phone. Maybe the internet is not as much of a blessing for psychics as it seems, or maybe they just haven't found quite the right business model.

The Laskys enjoyed a windfall during PFN's 2000s dormancy. In 2004, the Inphomation bankruptcy estate settled its lawsuit against bankrupt MCI for withholding payments. The Laskys made $4 million. It's hard to say where that money went, maybe to backing Marc's Pikesville Pictures production company. Pikesville picked up odd jobs producing television commercials, promotional documentaries, and an extremely strange educational film intended to prepare children to testify in court. I only know about this because parts of it appear in the video "Marc Lasky Demo Reel," uploaded to YouTube by "Mike Warren," the old horse race handicapping pseudonym of Michael Lasky. It has 167 views, and a single comment, "my dad made this." That was Gabriela Lasky, Marc's daughter. It's funny how much of modern life plays out on YouTube, where Marc's own account uploaded the full run of PFN infomercials.

---

Some of that $4 million in MCI money might have gone into the Psychic Friends Networks' reboot. In 2009, Marc Lasky produced a new series of television commercials for PFN. "The legendary Psychic Friends Network is back, bigger and bolder than ever." An extremely catchy jingle goes "all new, all improved, all knowing: call the Psychic Friends Network." On PFN 2.0, you can access your favorite psychic whenever you wish, on your laptop, computer, on your mobile, or on your tablet.

These were decidedly modernized, directing viewers to text a keyword to an SMS shortcode or visit psychicfriendsnetwork.com, where they could set up real-time video consultations with PFN's network of advisors. Some referred to "newpfn.com" instead, perhaps because it was easier to type, or perhaps there was some dispute around the Keen deal. There were still echoes of the original 1990s formula. The younger Lasky seemed to be hunting for a new celebrity lead like Warwick, but having trouble finding one. Actress Vivica A. Fox appeared in one spot, but then sent a cease and desist and went to the press alleging that her likeness was used without her permission. Well, they got her to record the lines somehow, but maybe they never paid. Maybe she found out about PFN's troubled reputation after the shoot. In any case, Lasky went hunting again and landed on Puerto Rican astrologer and television personality Walter Mercado.

Mercado, coming off something like Liberace if he was a Spanish-language TV host, sells the Psychic Friends Network to a Latin beat and does a hell of a job of it. He was a recognizable face in the Latin-American media due to his astrology show, syndicated for many years by Univision, and he appears in a sparkling outfit that lets him deliver the line "the legend is back" with far more credibility than anyone else in the new PFN spots. He was no Dionne Warwick, though, and the 2009 PFN revival sorely lacked the production quality or charm of the '90s infomercial. It seems to have had little impact; this iteration of PFN is so obscure that many histories of the company are completely unaware of it.

Elsewhere, in Nevada, an enigmatic figure named Ya Tao Chang had incorporated Web Wizards Inc. I can tell you almost nothing about this; Chang is impossible to research and Web Wizards left no footprints. All I know is that, somehow, Web Wizards made it to a listing on the OTC securities market. In 2012, PFN Holdings needed money and, to be frank, I think that Chang needed a real business. Or, at least, something that looked like one. In a reverse-merger, PFN Holdings joined Web Wizards and renamed to Psychic Friends Network Inc., PFNI on the OTC bulletin board.

The deal was financed by Right Power Services, a British Virgin Islands company (or was it a Singapore company? accounts disagree), also linked to Chang. Supposedly, there were millions in capital. Supposedly, exciting things were to come for PFN.

---

Penny stocks are stocks that trade at low prices, under $5 or even more classically under $1. Because these prices are too low to quality for listing on exchanges, they trade on less formal, and less heavily regulated, over-the-counter markets. Related to penny stocks are microcap stocks, stocks of companies with very small market capitalizations. These companies, being small and obscure, typically see miniscule trading volumes as well.

The low price, low volume, and thus high volatility of penny stocks makes them notoriously prone to manipulation. Fraud is rampant on OTC markets, and if you look up a few microcap names it's not hard to fall into a sort of alternate corporate universe. There exists what I call the "pseudocorporate world," an economy that relates to "real" business the same way that pseudoscience relates to science. Pseudocorporations have much of the ceremony of their legitimate brethren, but none of the substance. They have boards, executives, officers, they issue press releases, they publish annual reports. What they conspicuously lack is a product, or a business. Like NFTs or memecoins, they are purely tokens for speculation, and that speculation is mostly pumping and dumping.

Penny stock pseudocompanies intentionally resemble real ones; indeed, their operation, to the extent that they have one, is to manufacture the appearance of operating. They announce new products, that will never materialize, they announce new partnerships, that will never amount to anything, they announce mergers, that never close. They also rearrange their executive leadership with impressive frequency, due in no small part to the tendency of those leaders to end up in trouble with the SEC. All of this means that it's very difficult to untangle their history, and often hard to tell if they were once real companies that were hollowed out and exploited by con men, or whether they were a sham all along.

Web Wizards does not appear to have had any purpose prior to its merger with PFN, and as part of the merger deal the Laskys became the executive leadership of the new company. They seem to have legitimately approached the transaction as a way to raise capital for PFN, because immediately after the merger they announced PFN's ambitious future. This new PFN would be an all-online operation using live webcasts and 1:1 video calling. The PFN website became a landing page for their new membership service, and the Laskys were primed to produce a new series of TV spots. Little more would ever be heard of this.

In 2014, PFN Inc renamed itself to "Peer to Peer Network Inc.," announcing their intent to capitalize on PFN's early gig work model by expanding the company into other "peer to peer" industries. The first and only venture Peer to Peer Network (PTOP on OTC Pink) announced was an acquisition of 321Lend, a silicon valley software startup that intended to match accredited investors with individuals needing loans. Neither company seems to have followed up on the announcement, and a year later 321Lend announced its acquisition by Loans4Less, so it doesn't seem that the deal went through.

I might be reading too much between the lines, but I think there was a conflict between the Laskys, who had a fairly sincere intent to operate the PFN as a business, and the revolving odd lot of investors and executives that seem to grow like mold on publicly-traded microcap companies.

Back in 2010, a stockbroker named Joshua Sodaitis started work on a transit payment and routing app called "Freemobicard." In 2023, he was profiled in Business Leaders Review, one of dozens of magazines, podcasts, YouTube channels, and Medium blogs that exist to provide microcap executives with uncritical interviews that create the resemblance of notability. The Review says Sodaitis "envisioned a future where seamless, affordable, and sustainable transportation would be accessible to all." Freemobicard, the article tells us, has "not only transformed the way people travel but has also contributed to easing traffic congestion and reducing carbon emissions."

It never really says what Freemobicard actually is, but that doesn't matter, because by the time it gets involved in our story Sodaitis had completely forgotten about the transportation thing anyway.

---

In 2015, disagreements between the psychic promoters and the stock promoters had come to a head. Attributing the move to differences in business vision, the Laskys bought the Psychic Friends Network assets out of Peer to Peer Network for $20,000 and resigned their seats on PTOP's board. At about the same time, PTOP announced a "licensing agreement" with a software company called Code2Action. The licensing agreement somehow involved Code2Action's CEO, Christopher Esposito, becoming CEO of PTOP itself. At this point Code2Action apparently rolled up operations, making the "licensing agreement" more of a merger, but the contract as filed with the SEC does indeed read as a license agreement. This is just one of the many odd and confusing details of PTOP's post-2015 corporate governance.

I couldn't really tell you who Christopher Esposito is or where he came from, but he seems to have had something to do with Joshua Sodaitis, because he would eventually bring Sodaitis along as a board member. More conspicuously, Code2Action's product was called Mobicard---or Freemobicard, depending on which press release you read. This Mobicard was a very different one, though. Prior to the merger it was some sort of SMS marketing product (a "text this keyword to this shortcode" type of autoresponse/referral service), but as PTOP renamed itself to Mobicard Inc. (or at least announced the intent to, I don't think the renaming ever actually happened) the vision shifted to the lucrative world of digital business cards. Their mobile app, Mobicard 1.0, allowed business professionals to pay a monthly fee to hand out a link to a basic profile webpage with contact information and social media links. Kind of like Linktree, but with LinkedIn vibes, higher prices, and less polish.

One of the things you'll notice about Mobicard is that, for a software company, they were pretty short on software engineers. Every version of the products (and they constantly announce new ones, with press releases touting Mobicard 1.5, 1.7, and 2.0) seems to have been contracted to a different low-end software house. There are demo videos of various iterations of Mobicard, and they are extremely underwhelming. I don't think it really mattered, PTOP didn't expect Mobicard to make money. Making money is not the point of a microcap pseudocompany.

That same year, Code2Action signed another license agreement, much like the PTOP deal, but with a company called Cannabiz. Or maybe J M Farms Patient Group, the timeline is fuzzy. This was either a marketing company for medical marijuana growers or a medical marijuana grower proper, probably varying before and after they were denied a license by the state of Massachusetts on account of the criminal record of one of the founders. The whole cannabis aside only really matters because, first, it matches the classic microcap scam pattern of constantly pivoting to whatever is new and hot (which was, for a time, newly legalized cannabis), and second, because a court would later find that Cannabiz was a vehicle for securities fraud.

Esposito had a few years of freedom first, though, to work on his new Peer to Peer Network venture. He made the best of it: PTOP issued a steady stream of press releases related to contracts for Mobicard development, the appointment of various new executives, and events as minor as having purchased a new domain name. Despite the steady stream of mentions in the venerable pages of PRNewswire, PTOP doesn't seem to have actually done anything. In 2015, 2016, 2017, and 2018, PTOP failed to complete financial audits and SEC reports. To be fair, in 2016 Esposito was fined nearly $100,000 by the SEC as part of a larger case against Cannabiz and its executives. He must have had a hard time getting to the business chores of PTOP, especially since he had been barred from stock promotion.

In 2018, with PTOP on the verge of delisting due to the string of late audits, Joshua Sodaitis was promoted to CEO and Chairman of "Peer to Peer Network, Inc., (Stock Ticker Symbol PTOP) a.k.a. Mobicard," "the 1st and ONLY publicly traded digital business card company." PTOP's main objective became maintaining its public listing, and for a couple of years most discussion of the actual product stopped.

In 2020, PTOP made the "50 Most Admired Companies" in something called "The Silicon Valley Review," which I assume is prestigious and conveniently offers a 10% discount if you nominate your company for one of their many respected awards right now. "This has been a monumental year for the company," Sodaitis said, announcing that they had been granted two (provisional) patents and appointed a new advisory board (including one member "who is self-identified as a progressive millennial" and another who was a retired doctor). The bio of Sodaitis mentions the Massachusetts medical marijuana venture, using the name of the company that was denied a license and shuttered by the SEC, not the reorganized replacement. Sodaitis is not great with details.

It's hard to explain Mobicard because of this atmosphere of confusion. There was the complete change in product concept, which is itself confusing, since Sodaitis seems to have given the interview where he discussed Mobicard as a transportation app well after he had started describing it as a digital business card. Likewise, Mobicard has a remarkable number of distinct websites. freemobicard.com, mobicard.com, ptopnetwork.com, and mobicards.ca all seem oddly unaware of each other, and as the business plan continues to morph, are starting to disagree on what mobicard even is. The software contractor or staff developing the product keep changing, as does the version of mobicard they are about to launch. And on top of it all are the press releases.

Oh, the press releases. There's nary a Silicon Valley grift unmentioned in PTOP's voluminous newswire output. Crypto, the Metaverse, and AI all make appearances as part of the digital business card vision. As for the tone, the headlines speak for themselves.

"MOBICARD Set for Explosive Growth in 2024"

"MobiCard's Digital Business Card Revolutionizes Networking & Social Media"

"MOBICARD Revolutionizes Business Cards"

"Peer To Peer Network, aka Mobicard™ Announces Effective Form C Filing with the SEC and Launch of Reg CF Crowdfunding Campaign"

"Joshua Sodaitis, Mobicard, Inc. Chairman and CEO: 'We’re Highly Committed to Keeping Our 'One Source Networking Solution' Relevant to the Ever-Changing Dynamics of Personal and Professional Networking'"

"PTOP ANNOUNCES THE RESUBMISSION OF THE IMPROVED MOBICARD MOBILE APPS TO THE APPLE STORE AND GOOGLE PLAY"

"Mobicard™ Experienced 832% User Growth in Two Weeks"

"Peer To Peer Network Makes Payment to Attorney To File A Provisional Patent for Innovative Technology"

Yes, this company issues a press release when they pay an invoice. To be fair, considering the history of bankruptcy, maybe that's more of an achievement than it sounds.

In one "interview" with a "business magazine," Sodaitis talks about why Mobicard has taken so long to reach maturity. It's the Apple app store review, he explains, a story to which numerous iOS devs will no doubt relate. Besides, based on their press releases, they have had to switch contractors and completely redevelop the product multiple times. I didn't know that the digital business card was such a technical challenge. Sodaitis has been working on it for perhaps as long as fifteen years and still hasn't quite gotten to MVP.

---

You know where this goes, don't you? After decades of shady characters, trouble with regulators, cosplaying at business, and outright scams, there's only one way the story could possibly end.

All the way back in 2017, PTOP announced that they were "Up 993.75% After Launch Of Their Mobicoin Cryptocurrency." PTOP, the release continues, "saw a truly Bitcoin-esque move today, completely outdoing the strength of every other stock trading on the OTC market." PTOPs incredible market move was, of course, from $0.0005 to $0.0094. With 22 billion shares of common stock outstanding, that gave PTOP a valuation of over $200 million my the timeless logic of the crypto investor.

Of course, PTOP wasn't giving up on their OTC listing, and with declining Bitcoin prices their interest in the cryptocurrency seems to have declined as well. That was, until the political and crypto market winds shifted yet again. Late last year, PTOP was newly describing Mobicoin as a utility token. In November, they received a provisional patent on "A Cryptocurrency-Based Platform for Connecting Companies and Social Media Users for Targeted Marketing Campaigns." This is the latest version of Mobicard. As far as I can tell, it's now a platform where people are paid in cryptocurrency for tweeting advertising on behalf of a brand.

PTOP had to beef up their crypto expertise for this exciting new frontier. Last year, they hired "Renowned Crypto Specialist DeFi Mark," proprietor of a cryptocurrency casino and proud owner of 32,000 Twitter followers. "With Peer To Peer Network, we're poised to unleash the power of blockchain, likely triggering a significant shift in the general understanding of web3," he said.

"I have spoken to our Senior Architect Jay Wallace who is a genius at what he does and he knows that we plan to Launch Mobicard 1.7 with the MOBICOIN fully implemented shortly after the New President is sworn into office. I think this is a great time to reintroduce the world to MOBICOIN™ regardless of how I, or anyone feels about politics we can't deny the Crypto markets exceptional increase in anticipation to major regulatory transformations. I made it very clear to our Tech Team leader that this is a must to launch Mobicard™ 1.7.

Well, they've outdone themselves. Just two weeks ago, they announced Mobicard 2.0. "With enhanced features like real-time analytics, seamless MOBICOIN™ integration, and enterprise-level onboarding for up to 999 million employees, this platform is positioned to set new standards in the digital business card industry."

And how does that cryptocurrency integration work?

"Look the Mobicard™ Reward system is simple. We had something like it previously implemented back in 2017. If a MOBICARD™ user shares his MOBICARD™ 50 times in one week then he will be rewarded with 50 MOBICOIN's. If a MOBICARD user attends a conference and shares his digital business card MOBICARD™ with 100 people he will be granted 100 MOBICOIN™'s."

Yeah, it's best not to ask.

---

I decided to try out this innovative new digital business card experience, although I regret to say that the version in the Play Store is only 1.5. I'm sure they're just waiting on app store review. The dashboard looks pretty good, although I had some difficulty actually using it. I have not so far been able to successfully create a digital business card, and most of the tabs just lead to errors, but I have gained access to four or five real estate brokers and CPAs via the "featured cards." One of the featured cards is for Christopher Esposito, listed as "Crypto Dev" at NRGai.

Somewhere around 2019, Esposito brought Code2Action back to life again. He promoted a stock offering, talking up the company's bright future and many promising contracts. You might remember that this is exactly the kind of thing that the SEC got him for in 2016, and the SEC dutifully got him again. He was sentenced to five of probation after a court found that he had lied about a plan to merge Code2Action with another company and taken steps to conceal the mass sale of his own stock in the company.

NRGai, or NRG4ai, they're inconsistent, is a token that claims to facilitate the use of idle GPUs for AI training. According to one analytics website, it has four holders and trades at $0.00.

The Laskys have moved on as well. Michael Lasky is now well into retirement, but Marc Lasky is President & Director of Fernhill Corporation, "a publicly traded Web3 Enterprise Software Infrastructure company focused on providing cloud based APIs and solutions for digital asset trading, NFT marketplaces, data aggregation and DeFi/Lending". Fernhill has four subsidiaries, ranging from a cryptocurrency market platform to mining software. None appear to have real products. Fernhill is trading on OTC Pink at $0.00045.

Joshua Sodaitis is still working on Mobicard. Mobicard 2.0 is set for a June 1 launch date, and promises to "redefine digital networking and position [PTOP] as the premier solution in the digital business card industry." "With these exciting developments, we anticipate a positive impact on the price of PTOP stock." PTOP is trading on OTC Pink at $0.00015.

Michael Lasky was reportedly fond of saying that "you can get more money from people over the telephone than using a gun." As it happens, he wielded a gun anyway, but he had a big personality like that. One wonders what he would say about the internet. At some point, in his golden years, he relaunched his handicapping business Mike Warren Sports. The website sold $97/month subscriptions for tips on the 2015 NFL and NCAA football seasons, and the customer testimonials are glowing. One of them is from CNN's Larry King, although it doesn't read much like a testimonial, more like an admission that he met Lasky once.

There might still be some hope. A microcap investor, operating amusingly as "FOMO Inc.," has been agitating to force a corporate meeting for PTOP. PTOP apparently hasn't held one in years, is once again behind on audits, and isn't replying to shareholder inquiries. Investors allege poor management by Sodaitis. The demand letter, in a list of CC'd shareholders the author claims to represent by proxy, includes familiar names: Mike and Marc Lasky. They never fully divested themselves of their kind-of-sort-of former company.

A 1998 article in the Baltimore Sun discussed Lasky's history as a handicapper. It quotes a former Inphomation employee, whose preacher father once wore a "Mike Warren Sports" sweater at the mall.

"A woman came up to him and said 'Oh, I believe in him, Mike Warren.' My father says, 'well, ma'am, everybody has to believe in something."

Lasky built his company on predicting the future, but of course, he was only ever playing the odds. Eventually, both turned on him. His company fell to a series of bad bets, and his scam fell to technological progress. Everyone has to believe in something, though, and when one con man stumbles there are always more ready to step in.

Recently, I covered some of the history of Ethernet's tenuous relationship with installed telephone cabling. That article focused on the earlier and more business-oriented products, but many of you probably know that there have been a number of efforts to install IP networking over installed telephone wiring in a residential and SOHO environment. There is a broader category of "computer networking over things you already have in your house," and some products remain pretty popular today, although seemingly less so in the US than in Europe.

The grandparent of these products is probably PhoneNet, a fairly popular product introduced by Farallon in the mid-'80s. At the time, local area networking for microcomputers was far from settled. Just about every vendor had their own proprietary solution, although many of them had shared heritage and resulting similarities. Apple Computer was struggling with the situation just like everyone; in 1983 they introduced an XNS-based network stack for the Lisa called AppleNet and then almost immediately gave up on it [1]. Steve Jobs made the call to adopt IBM's token ring instead, which would have seemed like a pretty safe bet at the time because of IBM's general prominence in the computing industry. Besides, Apple was enjoying a period of warming relations with IBM, part of the 1980s-1990s pattern of Apple and Microsoft alternately courting IBM as their gateway into business computing.

The vision of token ring as the Apple network standard died the way a lot of token ring visions did, to the late delivery and high cost of IBM's design. While Apple was waiting around for token ring to materialize, they sort of stumbled into their own LAN suite, AppleTalk [2]. AppleTalk was basically an expansion of the unusually sophisticated peripheral interconnect used by the Macintosh to longer cable runs. Apple put a lot of software work into it, creating a pretty impressive zero-configuration experience that did a lot to popularize the idea of LANs outside of organizations large enough to have dedicated network administrators. The hardware was a little more, well, weird. In true Apple fashion, AppleTalk launched with a requirement for weird proprietary cables. To be fair, one of the reasons for the system's enduring popularity was its low cost compared to Ethernet or token ring. They weren't price gouging on the cables the way they might seem to today. Still, they were a decided inconvenience, especially when trying to connect machines across more than one room.

One of the great things about AppleTalk, in this context, is that it was very slow. As a result, even though the physical layer was basically RS-422, the electrical requirements for the cabling were pretty relaxed. Apple had already taken advantage of this for cost reduction, using a shared signal ground on the long cables rather than the dedicated differential pairs typical for RS-422. A hobbyist realized that you could push this further, and designed a passive dongle that used telephone wiring as a replacement for Apple's more expensive dongle and cables. He filed a patent and sold it to Farallon, who introduced the product as PhoneNet.

PhoneNet was a big hit. It was cheaper than Apple's solution for the same performance, and even better, because AppleTalk was already a bus topology it could be used directly over the existing parallel-wired telephone cabling in a typical house or small office. For a lot of people with heritage in the Apple tradition of computing, it'll be the first LAN they ever used. Larger offices even used it because of the popularity of Macs in certain industries and the simplicity of patching their existing telephone cables for AppleTalk use; in my teenage years I worked in an office suite in downtown Portland that hadn't seen a remodel for a while and still had telephone jacks labeled "PhoneNet" at the desks.

PhoneNet had one important limitation compared to the network-over-telephone products that would follow: it could not coexist with telephony. Well, it could, in a sense, and was advertised as such. But PhoneNet signaled within the voice band, so it required dedicated telephone pairs. In a lot of installations, it could use the second telephone line that was often wired but not actually used. Still, it was a bust for a lot of residential installs where only one phone line was fully wired and already in use for phone calls.

As we saw in the case of Ethernet, local area networking standards evolved very quickly in the '80s and '90s. IP over Ethernet became by far the dominant standard, so the attention of the industry shifted towards new physical media for Ethernet frames. While 10BASE-T Ethernet operated over category 3 telephone wiring, that was of little benefit in the residential market. Commercial buildings typically had "home run" telephone wiring, in which each office's telephone pair ran directly to a wiring closet. In residential wiring of the era, this method was almost unheard of, and most houses had their telephone jacks wired in parallel along a small number of linear segments (often just one).

This created a cabling situation much like coaxial Ethernet, in which each telephone jack was a "drop" along a linear bus. The problem is that coaxial Ethernet relied on several different installation measures to make this linear bus design practical, and home telephone wiring had none of these advantages. Inconsistently spaced drops, side legs, and a lack of termination meant that reflections were a formidable problem. PhoneNet addressed reflections mainly by operating at a very low speed (allowing reflections to "clear out" between symbols), but such low bitrate did not befit the 1990s.

A promising solution to the reflection problem came from a company called Tut Systems. Tut's history is unfortunately obscure, but they seem to have been involved in what we would now call "last-mile access technologies" since the 1980s. Tut would later be acquired by Motorola, but not before developing a number of telephone-wiring based IP networks under names like HomeWire and LongWire. A particular focus of Tut was multi-family housing, which will become important later.

I'm not even sure when Tut introduced their residential networking product, but it seems like they filed a relevant patent in 1995, so let's say around then. Tut's solution relied on pulse position modulation (PPM), a technique in which data is encoded by the length of the spacing between pulses. The principal advantage of PPM is that it allows a fairly large number of bits to be transmitted per pulse (by using, say, 16 potential pulse positions to encode 4 bits). This allowed reflections to dissipate between pulses, even at relatively high bitrates.

Following a bit of inter-corporate negotiation, the Tut solution became an industry standard under the HomePNA consortium: HomePNA 1.0. HomePNA 1.0 could transmit 1Mbps over residential telephone wiring with up to 25 devices. A few years later, HomePNA 1.0 was supplanted by HomePNA 2.0, which replaced PPM with QAM (a more common technique for high data rates over low bandwidth channels today) and in doing so improved to 10Mbps for potentially thousands of devices.

I sort of questioned writing an article about all of these weird home networking media, because the end-user experience for most of them is pretty much the same. That makes it kind of boring to look at them one by one, as you'll see later. Fortunately, HomePNA has a property that makes it interesting: despite a lot of the marketing talking more about single-family homes, Tut seems to have envisioned HomePNA mainly as a last-mile solution for multi-family housing. That makes HomePNA a bit different than later offerings, landing in a bit of a gray area between the LAN and the access network.

The idea is this: home run wiring is unusual in residential buildings, but in apartment and condo buildings, it is typical for the telephone lines of each unit to terminate in a wiring closet. This yields a sort of hybrid star topology where you have one line to each unit, and multiple jacks in each unit. HomePNA took advantage of this wiring model by offering a product category that is at once bland and rather unusual for this type of media: a hub.

HomePNA hubs are readily available, even today in used form, with 16 or 24 HomePNA interfaces. The idea of a hub can be a little confusing for a shared-bus media like HomePNA, but each interface on these hubs is a completely independent HomePNA network. In an apartment building, you could connect one interface to the telephone line of each apartment, and thus offer high-speed (for the time) internet to each of your tenants using existing infrastructure. A 100Mbps Ethernet port on the hub then connected to whatever upstream access you had available.

The use of the term "hub" is kind of weird, and I do believe that at least in the case of HomePNA 2.0, they were actually switching devices. This leads to some weird labeling like "hub/switch," perhaps a result of the underlying oddity of a multi-port device on a shared-media network that nonetheless performs no routing.

There's another important trait of HomePNA 2.0 that we should discuss, at least an important one to the historical development of home networking. HomePNA 1.0 was designed not to cause problematic interference with telephone calls but still effectively signaled within the voice band. HomePNA 2.0's QAM modulation addressed this problem completely: it signaled between 4MHz and 10MHz, which put it comfortably above not only the voice band but the roughly up-to-1MHz band used by early ADSL. HomePNA could coexist with pretty much anything else that would have been used on a telephone line at the time.

Over time, control of HomePNA shifted away from Tut Systems and towards a competitor called Epigram, who had developed the QAM modulation for HomePNA 2.0. Later part of Broadcom, Epigram also developed a 100Mbps HomePNA 3.0 in 2005. The wind was mostly gone from HomePNA's sails by that point, though, more due to the rise of WiFi than anything else. There was a HomePNA 3.1, which added support for operation over cable TV wiring, but shortly after, in 2009, the HomePNA consortium endorsed the HomeGrid Forum as a successor. A few years later, HomePNA merged into HomeGrid Forum and faded away entirely.

The HomeGrid Forum is the organization behind G.hn, which is to some extent a successor of HomePNA, although it incorporates other precedents as well. G.hn is actually fairly widely used for the near-zero name recognition it enjoys, and I can't help but suspect that that's a result of the rather unergonomic names that ITU standards tend to take on. "G.hn" kind-of-sort-of stands for Gigabit Home Networking, which is at least more memorable than the formal designation G.9960, but still isn't at all distinctive.

G.hn is a pretty interesting standard. It's quite sophisticated, using a complex and modern modulation scheme (OFDM) along with forward error correction. It is capable of up to 2Gbps in its recent versions, and is kind of hard to succinctly discuss because it supports four distinct physical media: telephone, coaxial (TV) cable, powerline, and fiber.

G.hn's flexibility is probably another reason for its low brand recognition, because it looks very different in different applications. Distinct profiles of G.hn involve different band plans and signaling details for each physical media, and it's designed to coexist with other protocols like ADSL when needed.

Unlike HomePNA, multi-family housing is not a major consideration in the design of G.hn and combining multiple networks with a "hub/switch" is unusual. There's a reason: G.hn wasn't designed by access network companies like Tut; it was mostly designed in the television set-top box (STB) industry.

When G.hn hit the market in 2009, cable and satellite TV was rapidly modernizing. The TiVo had established DVRs as nearly the norm, and then pushed consumers further towards the convenience of multi-room DVR systems. Providing multi-room satellite TV is actually surprisingly complex, because STV STBs (say that five times fast) actually reconfigure the LNA in the antenna as part of tuning. STB manufacturers, dominated by EchoStar (at one time part of Hughes and closely linked to the Dish Network), had solved this problem by making multiple STBs in a home communicate with each other. Typically, there is a "main" STB that actually interacts with the antenna and decodes TV channels. Other STBs in the same house use the coaxial cabling to communicate with the main STB, requesting video signals for specific channels.

Multi-room DVR was basically an extension of this same concept. One STB is the actual DVR, and other STBs remote-control it, scheduling recordings and then having the main STB play them back, transmitting the video feed over the in-home coaxial cabling. You can see that this is becoming a lot like HomePNA, repurposing CATV-style or STV-style coaxial cabling as a general-purpose network in which peer devices can communicate with each other.

As STB services have become more sophisticated, "over the top" media services and "triple play" combo packages have become an important and lucrative part of the home communications market. Structurally, these services can feel a little clumsy, with an STB at the television and a cable modem with telephone adapters somewhere else. STBs increasingly rely on internet-based services, so you then connect the STB to your WiFi so it can communicate via the same cabling but a different modem. It's awkward.

G.hn was developed to unify these communications devices, and that's mostly how it's used. Providers like AT&T U-verse build G.hn into their cable television devices so that they can all share a DOCSIS internet connection. There are two basic ways of employing G.hn: first, you can use it to unify devices. The DOCSIS modem for internet service is integrated into the STB, and then G.hn media adapters can provide Ethernet connections wherever there is an existing cable drop. Second, G.hn can also be applied to multi-family housing, by installing a central modem system in the wiring closet and connecting each unit via G.hn. Providers that have adopted G.hn often use both configurations depending on the customer, so you see a lot of STBs these days with G.hn interfaces and extremely flexible configurations that allow them to either act as the upstream internet connection for the G.hn network, or to use a G.hn network that provides internet access from somewhere else. The same STB can thus be installed in either a single-family home or a multi-family unit.

We should take a brief aside here to mention MoCA, the Multimedia over Coax Alliance. MoCA is a somewhat older protocol with a lot of similarities to G.hn. It's used in similar ways, and to some extent the difference between the two just comes down to corporate alliances: AT&T is into G.hn, but Cox, both US satellite TV providers, and Verizon have adopted MoCA, making it overall the more common of the two. I just think it's less interesting. Verizon FiOS prominently uses MoCA to provide IP-based television service to STBs, via an optical network terminal that provides MoCA to the existing CATV wiring.

We've looked at home networking over telephone wiring, and home networking over coaxial cable. What about the electrical wiring? G.hn has a powerline profile, although it doesn't seem to be that widely used. Home powerline networking is much more often associated with HomePlug.

Well, as it happens, HomePlug is sort of dead, the industry organization behind it having wrapped up operations in 2016. That might not be such a big practical problem, though, as HomePlug is closely aligned with related IEEE standards for data over powerline and it's widely used in embedded applications.

As a consumer product, HomePlug will be found in the form of HomePlug AV2. AV2 offers Gigabit-plus data rates over good quality home electrical wiring, and compared to G.hn and MoCA it enjoys the benefit that standalone, consumer adapters are very easy to buy.

HomePlug selects the most complex modulation the wiring can support (typically QAM with a large constellation size) and uses multiple OFDM carriers in the HF band, which it transmits onto the neutral conductor of an outlet. The neutral wiring in the average house is also joined at one location in the service panel, so it provides a convenient shared bus. On the downside, the installation quality of home electrical wiring is variable and the neutral conductor can be noisy, so some people experience very poor performance from HomePlug. Others find it to be great. It really depends on the situation.

That brings us to the modern age: G.hn, MoCA, and HomePlug are all more or less competing standards for data networking using existing household wiring. As a consumer, you're most likely to use G.hn or MoCA if you have an ISP that provides equipment using one of the two. Standalone consumer installations, for people who just want to get Ethernet from one place to another without running cable, usually use HomePlug.

It doesn't really have to be that way, G.hn powerline adapters have come down in price to where they compete pretty directly with HomePlug. Coaxial-cable and telephone-cable based solutions actually don't seem to be that popular with consumers any more, so powerline is the dominant choice. I can take a guess at the reason: electrical wiring can be of questionable quality, but in a lot of houses I see the coaxial and telephone wiring is much worse. Some people have outright removed the telephone wiring from houses, and the coaxial plant has often been through enough rounds of cable and satellite TV installers that it's a bit of a project to sort out which parts are connected. A large number of cheap passive distribution taps, common in cable TV where the signal level from the provider is very high, can be problematic for coaxial G.hn or MoCA. It's usually not hard to fix those problems, but unless an installer from the ISP sorts it out it usually doesn't happen. For the consumer, powerline is what's most likely to work.

And, well, I'm not sure that any consumers care any more. WiFi has gotten so fast that it often beats the data rates achievable by these solutions, and it's often more reliable to boot. HomePlug in particular has a frustrating habit of working perfectly except for when something happens, conditions degrade, the adapters switch modulations, and the connection drops entirely for a few seconds. That's particularly maddening behavior for gamers, who are probably the most likely to care about the potential advantages of these wired solutions over WiFi.

I expect G.hn, MoCA, and HomePlug to stick around. All three have been written into various embedded standards and adopted by ISPs as part of their access network in multi-family or at least as an installation convenience in single-family contexts. But I don't think anyone really cares about them any more, and they'll start to feel as antiquated as HomePNA.

And here's a quick postscript to show how these protocols might adapt to the modern era: remember how I said G.hn can operate over fiber? Cheap fiber, too, the kind of plastic cables used by S/PDIF. The HomePlug Forum is investigating the potential of G.hn over in-home passive optical networks, on the theory that these passive optical networks can be cheaper (due to small conductor size and EMI tolerance) and more flexible (due to the passive bus topology) than copper Ethernet. I wouldn't bet money on it, given the constant improvement of WiFi, but it's possible that G.hn will come back around for "fiber in the home" internet service.

[1] XNS was a LAN suite designed by Xerox in the 1970s. Unusually for the time, it was an openly published standard, so a considerable number of the proprietary LANs of the 1980s were at least partially based on XNS.

[2] The software sophistication of AppleTalk is all the more impressive when you consider that it was basically a rush job. Apple was set to launch LaserWriter, and as I mentioned recently on Mastodon, it was outrageously expensive. LaserWriter was built around the same print engine as the first LaserJet and still cost twice as much, due in good part to its flexible but very demanding PostScript engine. Apple realized it would never sell unless multiple Macintoshes could share it---it cost nearly as much as three Mac 128ks!---so they absolutely needed to have a LAN solution ready. LaserWriter would not wait for IBM to get their token ring shit together. This is a very common story of 1980s computer networks; it's hard to appreciate now how much printer sharing was one of the main motivations for networking computers at all. There's this old historical theory that hasn't held up very well but is appealing in its simplicity, that civilization arises primarily in response to the scarcity of water and thus the need to construct irrigation works. You could say that microcomputer networking arises primarily in response to the scarcity of printers.

I've seen them at least twice on /r/whatisthisthing, a good couple dozen times on the road, and these days, even in press photos: GMC trucks with custom square boxes on the back, painted dark blue, with US Government "E" plates. These courier escorts, "unmarked" but about as subtle as a Crown Vic with a bull bar, are perhaps the most conspicuous part of an obscure office of a secretive agency. One that seems chronically underfunded but carries out a remarkable task: shipping nuclear weapons.

The first nuclear weapon ever constructed, the Trinity Device, was transported over the road from Los Alamos to the north end of the White Sands Missile Range, near San Antonio, New Mexico. It was shipped disassembled, with the non-nuclear components strapped down in a box truck and the nuclear pit nestled in the back seat of a sedan. Army soldiers, of the Manhattan Engineering District, accompanied it for security. This was a singular operation, and the logistics were necessarily improvised.

The end of the Second World War brought a brief reprieve in the nuclear weapons program, but only a brief one. By the 1950s, an arms race was underway. The civilian components of the Manhattan Project, reorganized as the Atomic Energy Commission, put manufacturing of nuclear arms into full swing. Most nuclear weapons of the late '40s, gravity bombs built for the Strategic Air Command, were assembled at former Manhattan Project laboratories. They were then "put away" at one of the three original nuclear weapons stockpiles: Manzano Base, Albuquerque; Killeen Base, Fort Hood; and and Clarksville Base, Fort Campbell [1].

By the mid-1950s, the Pantex Plant near Amarillo had been activated as a full-scale nuclear weapons manufacturing center. Weapons were stockpiled not only at the AEC's tunnel sites but at the "Q Areas" of about 20 Strategic Air Command bases throughout the country and overseas. Shipping and handling nuclear weapons was no longer a one-off operation, it was a national enterprise.

To understand the considerations around nuclear transportation, it's important to know who controls nuclear weapons. In the early days of the nuclear program, all weapons were exclusively under civilian control. Even when stored on military installations (as nearly all were), the keys and combinations to the vaults were held by employees of the AEC, not military personnel. Civilian control was a key component of the Atomic Energy Act, an artifact of a political climate that disfavored the idea of fully empowering the military with such destructive weapons. Over the decades since, larger and larger parts of the nuclear arsenal have been transferred into military control. The majority of "ready to use" nuclear weapons today are "allocated" to the military, and the military is responsible for storing and transporting them.

Even today, though, civilian control is very much in force for weapons in any state other than ready for use. Newly manufactured weapons (in eras in which there were such a thing), weapons on their way to and from refurbishment or modification, and weapons removed from the military allocation for eventual disassembly are all under the control of the Department of Energy's National Nuclear Security Administration [2]. So too are components of weapons, test assemblies, and the full spectrum of Special Nuclear Material (a category defined by the Atomic Energy Act). Just as in the 1940s, civilian employees of the DoE are responsible for securing and transporting a large inventory of weapons and sensitive assets.

As the Atomic Energy Commission matured, and nuclear weapons became less of an experiment and more of a product, transportation arrangements matured as well. It's hard to find much historical detail on AEC shipping before the 1960s, but we can pick up a few details from modern DoE publications showing how the process has improved. Weapons were transported in box trucks as part of a small convoy, accompanied by "technical couriers, special agents, and armed military police." Technical courier was an AEC job title, one that persisted for decades to describe the AEC staff who kept custody of weapons under transport. Despite the use of military security (references can be found to both Army MPs and Marines accompanying shipments), technical couriers were also armed. A late 1950s photo published by DoE depicts a civilian courier on the side of a road wielding a long suit jacket and an M3 submachine gun.

During that period, shipments to overseas test sites were often made by military aircraft and Navy vessels. AEC couriers still kept custody of the device, and much of the route (for example, from Los Alamos to the Navy supply center at Oakland) was by AEC highway convoy. There have always been two key considerations in nuclear transportation: first, that an enemy force (first the Communists and later the Terrorists) might attempt to interdict such a shipment, and second, that nuclear weapons and materials are hazardous and any accident could create a disaster. More "broken arrow" incidents involve air transportation than anything else, and it seems that despite the potentially greater vulnerability to ambush, the ground has always been preferred for safety.

A 1981 manual for military escort operations, applicable not only to nuclear but also chemical weapons, lays out some of the complexity of the task. "Suits Uncomfortable," "Radiation Lasts and Lasts," quick notes in the margin advise. The manual describes the broad responsibilities of escort teams, ranging from compliance with DOT hazmat regulations to making emergency repairs to contain leakage. It warns of the complexity of such operations near civilians: there may be thousands of civilians nearby, and they might panic.

Escort personnel must be trained to be prepared for problems with the public. If they are not, their problems may be multiplied---perhaps to a point where satisfactory solutions become almost impossible.

During the 1960s, heightened Cold War tensions and increasing concern of terrorism (likely owing to the increasingly prominent anti-war and anti-nuclear movements, sometimes as good as terrorists in the eyes of the military they opposed) lead to a complete rethinking of nuclear shipping. Details are scant, but the AEC seems to have increased the number of armed civilian guards and fully ended the use of any non-government couriers for special nuclear material. I can't say for sure, but this seems to be when the use of military escorts was largely abandoned in favor of a larger, better prepared AEC force. Increasing protests against nuclear weapons, which sometimes blocked the route of AEC convoys, may have made posse comitatus and political optics a problem with the use of the military on US roads.

In 1975, the Atomic Energy Commission gave way to the Energy Research and Development Administration, predecessor to the modern Department of Energy. The ERDA reorganized huge parts of the nuclear weapons complex to align with a more conventional executive branch agency, and in doing so created the Office of Transportation Safeguards (OTS). OTS had two principal operations: the nuclear train, and nuclear trucks.

Trains have been used to transport military ordnance for about as long as they have existed, and in the mid-20th century most major military installations had direct railroad access to their ammunition bunkers. When manufacturing operations began at the Pantex Plant, a train known as the "White Train" for its original color became the primary method of delivery of new weapons. The train was made up of distinctive armored cars surrounded by empty buffer cars (for collision safety) and modified box cars housing the armed escorts. Although the "white train" was repainted to make it less obvious, railfans demonstrate that it is hard to keep an unusual train secret, and anti-nuclear activists were often aware of its movements. While the train was considered a very safe and secure option for nuclear transportation (considering the very heavy armored cars and relative safety of established rail routes), it had its downsides.

In 1985, a group of demonstrators assembled at Bangor Submarine Base. Among their goals was to bring attention to the Trident II SLBM by blocking the arrival of warheads on the White Train. 19 demonstrators were arrested and charged with conspiracy for their interference with the shipment. The jury found all 19 not guilty.

The DoE is a little cagey, in their own histories, about why they stopped using the train. We can't say for sure that this demonstration was the reason, but it must have been a factor. At Bangor, despite the easy rail access, all subsequent shipments were made by truck. Trucks were far more flexible and less obvious, able to operate on unpredictable schedules and vary their routes to evade protests. In the two following years, use of the White Train trailed off and then ended entirely. From 1987, all land transportation of nuclear weapons would be by semi-trailer.

This incident seems to have been formative for the OTS, which in classic defense fashion would be renamed the Office of Secure Transportation, or OST. A briefing on the OST, likely made for military and law enforcement partners, describes their tactical doctrine: "Remain Unpredictable." Sub-bullets of this concept include "Chess Match" and "Ruthless Adherence to Deductive Thought Process," the meaning of which we could ponder for hours, but if not a military briefing this is at least a paramilitary powerpoint. Such curious phrases accompanied by baffling concept diagrams (as we find them here) are part of a fine American tradition.

Beginning somewhere around 1985, the backbone of the OST's security program became obscurity. An early '00s document from an anti-nuclear weapons group notes that there were only two known photographs of OST vehicles. At varying times in their recent history, OST's policy seems to have been to either not notify law enforcement of their presence at all, or to advise state police only that there was a "special operation" that they were not to interfere with. Box trucks marked "Atomic Energy Commission," or trains bearing the reporting symbol "AEC," are long gone. OST convoys are now unmarked and, at least by intention, stealthy.

It must be because of this history that the OST is so little-known today. It's not exactly a secret, and there have been occasional waves of newspaper coverage for its entire existence. While the OST remains low-profile relative to, say, the national laboratories, over the last decade the DoE has rather opened up. There are multiple photos, and even a short video, published by the DoE depicting OST vehicles and personnel. The OST has had a hard time attracting and retaining staff, which is perhaps the biggest motivator of this new publicity: almost all of the information the DoE puts out to the public about OST is for recruiting.

It is, of course, a long-running comedy that the federal government's efforts at low-profile vehicles so universally amount to large domestic trucks in dark colors with push bumpers, spotlights, and GSA license plates. OST convoys are not hard to recognize, and are conspicuous enough that with some patience you can find numerous examples of people with no idea what they are finding them odd enough to take photos. The OST, even as an acknowledged office of the NNSA with open job listings, still feels a bit like a conspiracy.

During the early 1970s, the AEC charged engineers at Sandia with the design of a new, specialized vehicle for highway transportation of nuclear weapons. The result, with a name only the government could love, was the Safe Secure Transporter (SST, which is also often expanded as Safe Secure Trailer). Assembly and maintenance of the SSTs was contracted to Allied Signal, now part of Honeywell. During the 1990s, the SST was replaced by the Safeguards Transporter (SGT), also designed by Sandia. By M&A, the Allied Signal contract had passed to Honeywell Federal Manufacturing & Technology (FM&T), also the operating contractor of the Kansas City Plant where many non-nuclear components of nuclear weapons are made. Honeywell FM&T continues to service the SGTs today, and is building their Sandia-designed third-generation replacement, the Mobile Guardian [3].

Although DoE is no longer stingy about photographs of the SGT, details of its design remain closely held. The SGT consists of a silver semi-trailer, which looks mostly similar to any other van trailer but is a bit shorter than the typical 53' (probably because of its weight). Perhaps the most distinctive feature of the trailers is an underslung equipment enclosure which appears to contain an air conditioner; an unusual way to mount the equipment that I have never seen on another semi-trailer.

Various DoE-released documents have given some interior details, although they're a bit confusing on close reading, probably because the trailers have been replaced and refurbished multiple times and things have changed. They are heavily armored, the doors apparently 12" thick. They are equipped with a surprising number of spray nozzles, providing fire suppression, some sort of active denial system (perhaps tear gas), and an expanding foam that can be released to secure the contents in an accident. There is some sort of advanced lock system that prevents the trailer being opened except at the destination, perhaps using age-old bank vault techniques like time delay or maybe drawing from Sandia's work on permissive action links and cryptographic authentication.

The trailers are pulled by a Peterbilt tractor that looks normal until you pay attention. They are painted various colors, perhaps a lesson learned from the conspicuity of the White Train. They're visibly up-armored, with the windshield replaced by two flat ballistic glass panels, much like you'd see on a cash transport. The sleeper has been modified to fit additional equipment and expand seating capacity to four crew members.

Maybe more obvious, they're probably the only semitrailers and tractors that you'll see with GSA "E" prefix license plates (for Department of Energy).

SGTs are accompanied on the road by a number of escort vehicles, although I couldn't say exactly how many. From published photographs, we can see that these fall into two types: the dark blue, almost black GMC box trucks with not-so-subtle emergency lights and vans with fiberglass bodies that you might mistake for a Winnebago were they not conspicuously undecorated. I've also seen at least one photo of a larger Topkick box truck associated with the OST, as well as dark-painted conventional cargo vans with rooftop AC.

If you will forgive the shilling for my Online Brand, I posted a collection of photos on Mastodon. These were all released by NNSA and were presumably taken by OST or Honeywell staff, you can see that many of them are probably from the same photoshoot. Depending on what part of the country you are in, you may very well be able to pick these vehicles out on the freeway. Hint: they don't go faster than 60, and only operate during the day in good weather.

These escort vehicles probably mostly carry additional guards, but one can assume that they also have communications equipment and emergency supplies. Besides security, one of the roles of the OST personnel is prompt emergency response, taking the first steps to contain any kind of radiological release before larger response forces can arrive. Documents indicate that OST has partnerships with both DoE facilities (such as national labs) and the Air Force to provide a rapid response capability and offer secure stopping points for OST convoys.

The OST has problems to contend with besides security and anti-nuclear activism: its own management. The OST is sort of infamously not in great shape.

Some of the vehicles were originally fabricated in Albuquerque in a motley assortment of leased buildings put together temporarily for the task, others were fabricated at the Kansas City Plant. It's hard to tell which is which, but when refurbishment of the trailers was initiated in the 2000s, it was decided to centralize all vehicle work near the OST's headquarters (also a leased office building) in Albuquerque. At the time, the OST's warehouses and workshops were in poor and declining condition, and deemed too small for the task. OST's communications center (discussed in more detail later) was in former WWII Sandia Base barracks along with NNSA's other Albuquerque offices, and they were in markedly bad shape.

To ready Honeywell FM&T for a large refurbishment project and equip OST with more reliable, futureproof facilities, it was proposed to build the Albuquerque Transportation Technology Center (ATTC) near the Sunport. In 2009, the ATTC was canceled. To this day, Honeywell FM&T works out of various industrial park suites it has leased, mostly the same ones as the 1980s. Facilities plans released by the DoE in response to a lawsuit by an activist organization end in FY2014 but tell a sad story of escalating deferred maintenance, buildings in unknown condition because of the lack of resources to inspect them, and an aging vehicle fleet that was becoming less reliable and more expensive to maintain.

The OST has 42 trucks and about 700 guards, now styled as Federal Agents. They are mostly recruited from military special forces, receive extensive training, and hold limited law enforcement powers and a statutory authorization to use deadly force in the defense of their convoys. Under a little-known and (fortunately) little-used provision of the Atomic Energy Act, they can declare National Security Areas, sort of a limited form of martial law. Despite these expansive powers, a 2015 audit report from the DoE found that OST federal agents were unsustainably overworked (with some averaging nearly 20 hours of overtime per week), were involved in an unacceptable number of drug and alcohol-related incidents for members of the Human Reliability Program, and that a series of oversights and poor management had lead to OST leadership taking five months to find out that an OST Federal Agent had threatened to kill two of his coworkers. Recruiting and retention of OST staff is poor, and this all comes in the context of an increasing number of nuclear shipments due to the ongoing weapons modernization program.

The OST keeps a low profile perhaps, in part, because it is troubled. Few audit reports, GSA evaluations, or even planning documents have been released to the public since 2015. While this leaves the possibility that the situation has markedly improved, refusal to talk about it doesn't tend to indicate good news.

OST is a large organization for its low profile. It operates out of three command centers: Western Command, at Kirtland AFB, Central Command, in Texas at Pantex, and Eastern Command, at Savannah River. The OST headquarters is leased space in an Albuquerque office building near the Sunport, and the communications and control center is nearby in the new NNSA building on Eubank. Agent training takes place primarily on a tenant basis at a National Guard base in Arkansas. OST additionally operates four or five (it was five but I believe one has been decommissioned) communications facilities. I have not been successful in locating those exactly besides that they are in New Mexico, Idaho, Missouri, South Carolina, and Maryland. Descriptions of these facilities are consistent with HF radio sites.

That brings us to the topic of communications, which you know I could go on about at length. I have been interested in OST for a long time, and a while back I wrote about the TacNet Tracker, an interesting experiment in early mobile computing and mesh networking that Sandia developed as a tactical communications system for OST. OST used to use a proprietary, Sandia-developed digital HF radio system for communications between convoys and the control center. That was replaced by ALE, for commonality with military systems, sometime in the 1990s.

More recent documents show that OST continues to use HF radio via the five relay stations, but also uses satellite messaging (which is described as Qualcomm, suggesting the off-the-shelf commercial system that is broadly popular in the trucking industry). Things have no doubt continued to advance since that dated briefing, as more recent documents mention real-time video links and extensive digital communications.

These communications systems keep all OST convoys in constant contact with the communications center in Albuquerque, where dispatchers monitor their status and movements. Communications center personnel provide weather and threat intelligence updates to convoys en route, and in the event of some sort of incident, will request assistance from the DoE, military, and local law enforcement. Some of the detailed communications arrangements emphasize the cautious nature of the OST. When requesting law enforcement assistance, communications center dispatchers provide law enforcement with codewords to authenticate themselves. An OST training video advises those law enforcement responders that, should they not have the codeword or the OST guards refuse the codeword they provide, they are to "take cover."

Paralleling a challenge that exists in the cash handling industry, the fact that law enforcement are routinely armed makes them an especially large threat to secure operations. OST may be required to use force to keep armed people away from a convoy, even when those people appear to be law enforcement. The way that this is communicated to law enforcement---that they must approach OST convoys carefully and get authorization from a convoy commander before approaching the truck---is necessarily a bit awkward. The permits and travel authorizations for the convoy are, law enforcement are warned, classified. They will not be able to check the paperwork.

The OST has assets beyond trucks, although the trucks are the backbone of the system. Three 737s, registered in the NNSA name, make up their most important air assets. Released documents don't rule out the possibility of these aircraft being used to transport nuclear weapons, but suggest that they're primarily for logistical support and personnel transport. Other smaller aircraft are in the OST inventory as well, all operating from a hanger at the Albuquerque Sunport. They fly fairly often, perhaps providing air support to OST convoys, but the NNSA indicates that they also use the OST aircraft for other related NNSA functions like transportation of the Radiological Assistance Program teams.

It should be said that despite the OST's long-running funding and administrative problems, it has maintained an excellent safety record. Some sources state that there has only been one road accident involving an OST convoy, a 1996 accident in which the truck slid off the road during an ice storm in Nebraska. I have actually seen OST documents refer to another incident in Oregon in the early '80s, in which an escort vehicle was forced off the road by a drunk driver and went into the ditch. I think it goes mostly unmentioned since only an escort vehicle was involved and there was no press attention at the time. Otherwise, despite troubling indications of its future sustainability, OST seems to have kept an excellent track record.

Finally, if you have fifteen minutes to kill, this video is probably the most extensive source of information on OST operations to have been made public. Even though I'm pretty sure a couple of the historical details it gives are wrong, but what's new. Special credit if you notice the lady that's still wearing her site-specific Q badge in the video. Badges off! Badges!

Also, if you're former military and can hold down a Q, a CDL, EMT-B, and firearms qualifications, they're hiring. I hear the overtime is good. But maybe the threats of violence not so much.

[1] The early Cold War was a very dynamic time in nuclear history, and plans changed quickly as the AEC and Armed Forces Special Weapons Project developed their first real nuclear strategy. Many of these historic details are thus complicated and I am somewhat simplifying. There were other stockpile sites planned that underwent some construction, and it is not totally clear if they were used before strategies changed once again. Similarly, manufacturing operations moved around quite a bit during this era and are hard to summarize.

[2] The NNSA, not to be confused with the agency with only one N, is a semi-autonomous division of the Department of Energy with programmatic responsibility for nuclear weapons and nuclear security. Its Administrator, currently former Sandia director Jill Hruby, is an Under Secretary of Energy and answers to the Secretary of Energy (and then to the President). I am personally very fond of Jill Hruby because of memorable comments she made after Trump's first election. They were not exactly complimentary to the new administration and I have a hard time thinking her outspokenness was not a factor in her removal as director of the laboratory. I assume her tenure as NNSA Administrator is about to come to an end.

[3] Here's a brief anecdote about how researching these topics can drive you a little mad. Unclassified documents about OST and their vehicles make frequent reference to the "Craddock buildings," where they are maintained and overhauled in Albuquerque. For years, this lead me to assume that Craddock was the name of a defense contractor that originally held the contract and Honeywell had acquired. There is, to boot, an office building near OST headquarters in Albuquerque that has a distinctive logo and the name "Craddock" in relief, although it's been painted over to match the rest of the building. Only yesterday did I look into this specifically and discover that Craddock is a Colorado-based commercial real estate firm that developed the industrial park near the airport, where MITS manufactured the Altair 8800 and Allied Signal manufactured the SSTs (if I am not mistaken Honeywell FM&T now uses the old MITS suite). OST just calls them the Craddock buildings because Craddock is the landlord! Craddock went bankrupt in the '80s, sold off part of its Albuquerque holdings, and mostly withdrew to Colorado, probably why they're not a well-known name here today.

So we all know about twisted-pair ethernet, huh? I get a little frustrated with a lot of histories of the topic, like the recent neil breen^w^wserial port video, because they often fail to address some obvious questions about the origin of twisted-pair network cabling. Well, I will fail to answer these as well, because the reality is that these answers have proven very difficult to track down.

For example, I have discussed before that TIA-568A and B are specified for compatibility with two different multipair wiring conventions, telephone and SYSTIMAX. And yet both standards actually originate within AT&T, so why did AT&T disagree internally on the correspondence of pair numbers to pair colors? Well, it's quite likely that some of these things just don't have satisfactory answers. Maybe the SYSTIMAX people just didn't realize there was an existing convention until they were committed. Maybe they had some specific reason to assign pairs 3 and 4 differently that didn't survive to the modern era. Who knows? At this point, the answer may be no one.

There are other oddities to which I can provide a more satisfactory answer. For example, why is it so widely said that twisted-pair ethernet was selected for compatibility with existing telephone cabling, when its most common form (10/100) is in fact not compatible with existing telephone cabling?

But before we get there, let's address one other question that the Serial Port video has left with a lot of people. Most office buildings, it is mentioned, had 25-pair wiring installed to each office. Wow, that's a lot of pairs! A telephone line, of course, uses a single pair. UTP ethernet would be designed to use two. Why 25?

The answer lies in the key telephone system. The 1A2 key telephone system, and its predecessors and successors, was an extremely common telephone system in the offices of the 1980s. Much of the existing communications wiring of the day's commercial buildings had been installed specifically for a 1A2-like system. I have previously explained that key telephone systems, for simplicity of implementation, inverted the architecture we expect from the PBX by connecting many lines to each phone, instead of many phones to each line. This is the first reason: a typical six-button key telephone, with access to five lines plus hold, needed five pairs to deliver those five lines. An eighteen button call director would have, when fully equipped, 17 lines requiring 17 pairs. Already, you will see that we can get to some pretty substantial pair counts.

On top of that, though, 1A2 telephones provided features like hold, busy line indication (a line key lighting up to indicate its status), and selective ringing. Later business telephone systems would use a digital connection to control these aspects of the phone, but the 1A2 is completely analog. It uses more pairs. There is an A-lead pair, which controls hold release. There is a lamp pair for each line button, to control the light. There is a pair to control the phone's ringer, and in some installations, another pair to control a buzzer (used to differentiate outside calls from calls on an intercom line). So, a fairly simple desk phone could require eight or more pairs.

To supply these pair counts, the industry adopted a standard for business telephone wiring: 25-pair cables terminated in Amphenol connectors. A call director could still require two cables, and two Amphenol connectors, and you can imagine how bulky this connection was. 25-pair cable was fairly expensive. These issues all motivated the development of digitally-controlled systems like the Merlin, but as businesses looked to install computer networks, 25-pair cabling remained very common.

But, there is a key difference between the unshielded twisted-pair cables used for telephones and the unshielded twisted-pair we think of today: the twist rate. We mostly interact with this property through the proxy of "cable categories," which seem to have originated with cable distributors (perhaps Anixter) but were later standardized by TIA-568.

• Category 1: up to 1MHz (not included in TIA-568)
• Category 2: up to 4MHz (not included in TIA-568)
• Category 3: up to 16MHz
• Category 4: up to 20MHz (not included in TIA-568)
• Category 5: up to 100MHz
• Category 6: up to 250MHz
• Category 7: up to 600MHz (not included in TIA-568)
• Category 8: up to 2GHz

Some of these categories are not, in fact, unshielded twisted-pair (UTP), as shielding is required to achieve the specified bandwidth. The important thing about these cable categories is that they sort of abstract away the physical details of the cable's construction, by basing the definition around a maximum usable bandwidth. At that maximum bandwidth, the cable must meet defined limits for attenuation and crosstalk.

Among the factors that determine the bandwidth capability of a cable is the twist rate, the frequency with which the two wires in a pair switch positions. The idea of twisted pair is very old, dating to the turn of the 20th century and open wire telephone leads that used "transposition brackets" to switch the order of the wires on the telephone pole. More frequent twisting provides protection against crosstalk at higher frequencies, due to the shorter spans of unbalanced wire. As carrier systems used higher frequencies on open wire telephone leads, transposition brackets became more frequent. Telephone cable is much the same, with the frequency of twists referred to as the pitch. The pitch is not actually specified by category standards; cables use whatever pitch is sufficient to meet the performance requirements. In practice, it's also typical to use slightly different pitches for different pairs in a cable, to avoid different pairs "interlocking" with each other and inviting other forms of EM coupling.

Inside telephone wiring in residential buildings is often completely unrated and may be more or less equivalent to category 1, which is a somewhat informal standard sufficient only for analog voice applications. Of course, commercial buildings were also using their twisted-pair cabling only for analog voice, but the higher number of pairs in a cable and the nature of key systems made crosstalk a more noticeable problem. As a result, category 3 was the most common cable type in 1A2-type installations of the 1980s. This is why category 3 was the first to make it into the standard, and it's why category 3 was the standard physical medium for 10BASE-T.

In common parlance, wiring originally installed for voice applications was referred to as "voice grade." This paralleled terminology used within AT&T for services like leased lines. In inside wiring applications, "voice grade" was mostly synonymous with category 3. StarLAN, the main predecessor to 10BASE-T, required a bandwidth of 12MHz... beyond the reliable capabilities of category 1 and 2, but perfectly suited for category 3.

This brings us to the second part of the twisted-pair story that is frequently elided in histories: the transition from category 3 cabling to category 5 cabling, as is required by 100BASE-TX "10/100" ethernet.

On the one hand, the explanation is simple: To achieve 100Mbps, 100BASE-TX requires a 100MHz cable, which means it requires category 5.

On the other hand, remember the whole entire thing about twisted-pair being intended to reuse existing telephone cable? Yes, the move from 10BASE-T to 100BASE-TX, and from category 3 to category 5, abandoned this advantage. The path by which this happened was not an simple one. The desire to reuse existing telephone cabling was still very much alive, and several divergent versions of twisted-pair ethernet were created for this purpose.

Ethernet comes with these kind of odd old conventions for describing physical carriers. The first part is the speed, the second part is the bandwidth/position (mostly obsolete, with BASE for baseband being the only surviving example), and the next part, often after a hyphen, identifies the medium. This medium code was poorly standardized and can be a little confusing. Most probably know that 10BASE5 and 10BASE2 identify 10Mbps Ethernet over two different types of coaxial cable. Perhaps fewer know that StarLAN, over twisted pair, was initially described as 1BASE5 (it was, originally, 1Mbps). The reason for the initial "5" code for twisted pair seems to be lost to history; by the time Ethernet over twisted pair was accepted as part of the IEEE 802.3 standard, the medium designator had changed to "-T" for Twisted Pair: 10BASE-T.

And yet, 100Mbps "Fast Ethernet," while often referred to as 100BASE-T, is more properly 100BASE-TX. Why? To differentiate it from the competing standard 100BASE-T4, which was 100Mbps Ethernet over Category 3 twisted pair cable. There were substantial efforts to deploy Fast Ethernet without requiring the installation of new cable in existing buildings, and 100BASE-TX competed directly with both 100BASE-T4 and the oddly designated 100BaseVG. In 1995, all three of these media were set up for a three-way faceoff [1].

For our first contender, let's consider 100BASE-T4, which I'll call "T4" for short. The T4 media designator means Twisted pair, 4 pairs. Recall that, for various reasons, 10BASE-T only used two pairs (one each direction). Doubling the number of required pairs might seem like a bit of a demand, but 10BASE-T was already routinely used with four-pair cable and 8P8C connectors, and years later Gigabit 1000BASE-T would do the same. Using these four pairs, T4 could operate over category 3 cable at up to 100 meters.

T4 used the pairs in an unusual way, directly extending the 10BASE-T pattern while compromising to achieve the high data rate over lower bandwidth cable. T4 had one pair each direction, and two pairs that dynamically changed directions as required. Yes, this means that 100BASE-T4 was only half duplex. T4 was mostly a Broadcom project, who offered chipsets for the standard and brought 3Com on board as the principal (but not only) vendor of network hubs.

The other category 3 contender, actually a slightly older one, was Hewlett-Packard's 100BaseVG. The "VG" media designator stood for "voice grade," indicating suitability for category 3 cables. Like T4, VG required four pairs. VG also uses those pairs in an unusual way, but a more interesting one: VG switches between a full-duplex, symmetric "control mode" and a half-duplex "transmission mode" in which all four pairs are used in one direction. Coordinating these transitions required a more complex physical layer protocol, and besides, HP took the opportunity to take on the problem of collisions. In 10BASE-T networks, the use of hubs meant that multiple hosts were in a collision domain, much like with coaxial Ethernet. As network demands increased, collisions became more frequent and the need to retransmit after collisions could appreciably reduce the effective capacity of the network.

VG solved both problems at once by introducing, to Ethernet, one of the other great ideas of the local area networking industry: token-passing. The 100BaseVG physical layer incorporated a token-passing scheme in which the hub assigned tokens to nodes, both setting the network operation mode and preventing collisions. The standard even included a simple quality of service scheme to the tokens, called demand priority, in which nodes could indicate a priority level when requesting to transmit. The token-passing system made the effective throughput of heavily loaded VG networks appreciably higher than other Fast Ethernet networks. Demand priority promised to make VG more suitable for real-time media applications in which Ethernet had traditionally struggled due to its nondeterministic capacity allocation.

Given that you have probably never heard of either of these standards, you are probably suspecting that they did not achieve widespread success. Indeed, the era of competition was quite short, and very few products were ever offered in either T4 or VG. Considering the enormous advantage of using existing Category 3 cabling, that's kind of a surprise, and it undermines the whole story that twisted pair ethernet succeeded because it eliminated the need to install new cabling. Of course, it doesn't make it wrong, exactly. Things had changed: 10BASE-T was standardized in 1990, and the three 100Mbps media were adopted in 1994-1995. Years had passed, and purpose-built computer network cabling had become more common. Besides, despite their advantages, T4 and VG were not without downsides.

To start, both were half-duplex. I don't think this was actually that big of a limitation at the time; half-duplex 100Mbps was still a huge improvement in real performance over even full-duplex 10Mbps, and the vast majority of 10BASE-T networks were hub-based and only half-duplex as well. A period document from a network equipment vendor notes this limitation of T4 but then describes full-duplex as "unneeded for workstations." That might seem like an odd claim today, but I think it was a pretty fair one in the mid-'90s.

A bigger problem was that both T4 and VG were meaningfully more complicated than TX. T4 used a big and expensive DSP chip to recover the complex symbols from the lower-grade cable. VG's token passing scheme required a more elaborate physical layer protocol implementation. Both standards were correspondingly more expensive, both for adapters and network appliances. The cost benefit of using existing cabling was thus a little fuzzier: buyers would have to trade off the cost of new cabling vs. the savings of using less complex, less expensive TX equipment.

For similar reasons, TX is also often said to have been more reliable than T4 or VG, although it's hard to tell if that's a bona fide advantage of TX or just a result of TX's much more widespread adoption. TX transceivers benefited from generations of improvement that T4 and VG transceivers never would.

Let's think a bit about that tradeoff between new cable and more expensive equipment. T4 and VG both operated on category 3, but they required four pairs. In buildings that had adopted 10BASE-T on existing telephone wiring, they would most likely have only punched down two pairs (out of a larger cable) to their network jacks and equipment. That meant that an upgrade from 10BASE-T to 100BASE-T4, for example, still involved considerable effort by a telecom or network technician. There would often be enough spare pairs to add two more to each network device, but not always. In practice, upgrading an office building would still require the occasional new cable pull. T4 and VG's poor reputation for reliability, or moreover poor reputation for tolerating less-than-perfect installations, meant that even existing connections might need time-consuming troubleshooting to bring them up to full category 3 spec (while TX, by spec, requires the full 100MHz of category 5, it is fairly tolerant of underperforming cabling).

There's another consideration as well: the full-duplex nature of TX makes it a lot more appealing in the equipment room and data center environment, and for trunk connections (between hubs or switches). These network connections see much higher utilization, and often more symmetric utilization as well, so a full-duplex option really looks 50% faster than a half-duplex one. Historically, plenty of network architectures have included the use of different media for "end-user" vs trunk connections. Virtually all consumer and SMB internet service providers do so today. It has never really caught on in the LAN world, though, where a smaller staff of network technicians are expected to maintain both sides.

Put yourself in the shoes of an IT manager at a midsized business. One option is T4 or VG, with more expensive equipment and some refitting of the cable plant, and probably with TX used in some cases anyway. Another option is TX, with less expensive equipment and more refitting of the cable plant. You can see that the decision is less than obvious, and you could easily be swayed in the all-TX direction, especially considering the benefit of more standardization and fewer architectural and software differences from 10BASE-T.

That seems to be what happened. T4 and VG found little adoption, and as inertia built, the cost and vendor diversity advantage of TX only got bigger. Besides, a widespread industry shift from shared-media networks (with hubs) to switched networks (with, well, switches) followed pretty closely behind 100BASE-TX. A lot of users went straight from 10BASE-T to switched 100BASE-TX, which almost totally eliminated the benefits of VG's token-passing scheme and made the cost advantage of TX even bigger.

And that's the story, right? No, hold on, we need to talk about one other effort to upon 10BASE-T. Not because it's important, or influential, or anything, but because it's very weird. We need to talk about IsoEthernet and IsoNetworks.

As I noted, Ethernet is poorly suited to real-time media applications. That was true in 1990, and it's still true today, but network connections have gotten so fast that the level of performance overhead available mitigates the problem. Still, there's a fundamental limitation: real-time media, like video and audio, requires a consistent amount of delivered bandwidth for the duration of playback. The Ethernet/IP network stack, for a couple of different reasons, provides only opportunistic or nondeterministic bandwidth to any given application. As a result, achieving smooth playback requires some combination of overprovisioning of the network and buffering of the media. This buffering introduces latency, which is particularly intolerable in real-time applications. You might think this problem has gone away entirely with today's very fast networks, but you can still see Twitch streamers struggling with just how bad the internet is at real-time media.

An alternative approach comes from the telephone industry, which has always had real-time media as its primary concern. The family of digital network technologies developed in the telephone industry, SONET, ISDN, what have you, provide provisioned bandwidth via virtual circuit switching. If you are going to make a telephone call at 64Kbps, the network assigns an end-to-end, deterministic 64Kbps connection. Because this bandwidth allocation is so consistent and reliable, very little or no buffering is required, allowing for much lower latency.

There are ways to address this problem, but they're far from perfect. The IP-based voice networks used by modern cellular carriers make extensive use of quality of service protocols but still fail to deliver the latency of the traditional TDM telephone network. Even with QoS, VoIP struggles to reach the reliability of ISDN. For practical reasons, consumers are rarely able to take any advantage of QoS for ubiquitous over-the-top media applications like streaming video.

What if things were different? What if, instead of networks, we had IsoNetworks? IsoEthernet proposed a new type of hybrid network that was capable of both nondeterministic packet switching and deterministic (or, in telephone industry parlance, isochronous) virtual circuit switching. They took 10BASE-T and ISDN and ziptied them together, and then they put Iso in front of the name of everything.

Here's how it works: IsoEthernet takes two pairs of category 3 cabling and runs 16.144 Mbps TDM frames over them at full duplex. This modest 60% increase in overall speed allows for a 10Mbps channel (called a P-channel by IsoEthernet) to be used to carry Ethernet frames, and the remaining 6.144Mbps to be used for 96 64-Kbps B-channels according to the traditional ISDN T2 scheme.

An IsoEthernet host (sadly not called an IsoHost, at least not in any documents I've seen) can use both channels simultaneously to communicate with an IsoHub. An IsoHub functions as a standard Ethernet hub for the P-channel, but directs the B-channels to a TDM switching system like a PABX. The mention of a PABX, of course, illustrates the most likely application: telephone calls over the computer.

I know that doesn't sound like that much of a win: most people just had a computer on their desk, and a phone on their desk, and despite decades of effort by the Unified Communications industry, few have felt a particular need to marry the two devices. But the 1990s saw the birth of telepresence: video conferencing. We're doing Zoom, now!

Videoconferencing over IP over 10Mbps Ethernet with multiple hosts in a collision domain was a very, very ugly thing. Media streaming very quickly caused almost worst-case collision behavior, dropping the real capacity of the medium well below 10Mbps and making even low resolution video infeasible. Telephone protocols were far more suited to videoconferencing, and so naturally, most early videoconferencing equipment operated over ISDN. I had a Tandberg videoconferencing system, for example, which dated to the mid '00s. It still provided four jacks on the back suitable for 4x T1 connections or 4 ISDN PRIs (basically just a software difference), providing a total of around 6Mbps of provisioned bandwidth for silky smooth real-time video.

These were widely used in academia and large corporations. If you ever worked somewhere with a Tandberg or Cisco (Cisco bought Tandberg) curved-monitor-wall system, it was most likely running over ISDN using H.320 video and T.120 application sharing ("application sharing" referred to things like virtual whiteboards). Early computer-based videoconferencing systems like Microsoft NetMeeting were designed to use existing computer networks. They used the same protocols, but over IP, with a resulting loss in reliability and increase in latency [2].

With IsoEthernet, there was no need for this compromise. You could use IP for your non-realtime computer applications, but your softphone and videoconferencing client could use ISDN. What a beautiful vision! As you can imagine, it went nowhere. Despite IEEE acceptance as 802.9 and promotion efforts by developer National Semiconductor, IsoEthernet never got even as far as 100BASE-T4 or 100BaseVG. I can't tell you for sure that it ever had a single customer outside of evaluation environments.

[1] A similar 100Mbps-over-category 3 standard, called 100BASE-T2, also belongs to this series. I am omitting it from this article because it was standardized in 1998 after industry consolidation on 100BASE-TX, so it wasn't really part of the original competition.

[2] The more prominent WebEx has a stranger history which will probably fill a whole article here one day---but it did also use H.320.

There were thousands of reports: strange aircraft, floating through the sky. A retrospective sum of press accounts finds that some 100,000 people were reported to have witnessed aerial intruders. Despite the scant details associated with most reports, an eager press repeated the claims with fervor. The claims became more fantastical. Prominent people claimed secret knowledge of the origins of the crafts. This was 1896. The airship had just barely been invented, and already the public was seeing them everywhere they looked.

John Keel was a writer and prominent UFOlogist, although he's probably remembered most of all for his cryptozoological book, The Mothman Prophecies. Like most UFOlogists of his era, Keel was sort of a mixed bag to those readers who are at least attempting to keep a rational perspective. In some ways he was more critical than average, turning against the extraterrestrial hypothesis as impractical and always calling for a shift away from "investigating" based on lone contactee accounts. On the other hand, he was as prone as anyone to fancy and it now seems that his books took some liberties with the information he'd been given. Still, his popular newspaper articles during the 1960s shaped much of our modern parlance around UFOs. Among the terms he seems to have introduced, or at the least popularized, is the "flap."

A flap is a concentrated set of UFO reports in a specific place and time. The 1896-1897 airship flap, which started in California and eventually spread across the nation to New York City, might be called the first. Of course, there is a straightforward argument that the airship flap was the first only in that it was the first flap during which aviation was on the public mind; by this token other paranormal episodes like dancing plagues and witch trials could be considered flaps. Still, "flap" is usually reserved for those times during which the general public is seeing things in the sky: something up there.

Flaps are a well-known phenomenon in UFOlogical circles (although not always by that name) since 1947. Widespread reports of flying saucers that year kicked off our modern UFO culture. Almost every decade had some sort of major flap until the 1990s, the decade during which UFOlogy could be said to have died. This is a more complex topic than I can explain here as preamble, and my opinion is somewhat controversial, but UFOlogy enjoyed a golden age during the '60s and '70s and by the time I came onto the scene had largely collapsed. The end of the Cold War, improving digital media, and sidelining (and often outright suppression) of serious investigations into UFOs were all factors. There was also a certain qualitative change in the UFO community: the most prominent names in UFOs were increasingly untrustworthy, forced by desperation or, more cynically, encouraged by money to become less and less careful about the ideas they endorsed.

It cannot be ignored that there are complexities, UFOlogical mysteries, to some of this decline as well. The single hardest blow to UFOlogy came in 1989, when Bill Moore stood before a MUFON conference to admit that the UFO materials he had distributed throughout the community, including the Majestic 12 papers, were fakes. This confession triggered a dramatic unraveling of the established canon of paranormal knowledge. By the early '90s, it seemed that nearly all of the major UFO news of the decade before had originated with a small number of people, often in collusion, who ranged from extremely unreliable (Bob Lazar) to admitted fabricators (Richard Doty). The fact that some of these people had connections to military intelligence, and that there remains some reason to believe they were intentionally spreading disinformation on behalf of those agencies, leaves plenty of intrigue but does nothing to resolve the basic fact that the UFOlogy of the '80s and '90s turned out to be mostly bullshit---not even of the vague cultural kind, but with specific, known authors [1].

It was this climate that lead us to the 21st century, which for nearly two decades was surprisingly devoid of UFO discourse. Around 2017, though, a motley crew including such personalities as a famous rock musician, a powerful US senator, and an eccentric hotel-aerospace billionaire thrust UFOs back into the popular media. I have written before with my analysis that the late-2010s UFO "revelations" (and, moreover, the lack thereof) were most likely the result of Bigelow taking advantage of the DoD's lax contract supervision and Sen. Harry Reid's personal interest in order to fund his hobby projects. Still, the whole unfortunate affair seems to have had the upside of renewing public and political attention to the topic.

The DoD was forced to at least try to get its act together, creating a new organization (the AARO) with tighter reins and more credibility. NASA formed its own review. The government seems to now be involved in its most serious efforts to understand the UFO phenomenon since the 1960s, which we can dream will be a departure from the conflicted, shambolic, and dismissive way that it addressed strange objects in the sky for fifty years. Or, like every other such effort to date, it will collapse into a hasty effort to close the whole topic and avoid admitting the failure of the intelligence community to make any real progress on a matter of obvious military and public interest. Only time will tell.

Anyway, that is all setting the stage for what has been going on for the last month in New Jersey: people are seeing drones.

The New Jersey Drones have all of the makings of a classic UFO flap. Unmanned aircraft are a topic of widespread public attention, tied up in everything from global conflict (Ukrainian combat drones) to intelligence intrigue (Chinese spy balloons) to domestic politics (DJI import bans). The real prevalence of drones flying around is increasing as they continue to come down in price and the FAA adopts a more permissive regulatory scheme. The airship flap happened a few years after airships started to make headlines (manned flight was barely achievable at the time, but there had been promising experiments and they inspired a great deal of speculation). Similarly, the drone flap happens a few years after foreign unmanned aircraft gained widespread media attention.

And this is the simplest explanation of what is happening in New Jersey: when people look up at the sky, they see things there.

The universe is full of more than humans can comprehend, but we make our peace with that by engaging with it through the sky. There is so much that we do not know about stars, galaxies, and the myriad of objects that surround us constantly, but we do know that they exist and we can see them. Even that can quickly become hazy when you really look up, though. Perceptual psychology offers a variety of explanations. For example: when the visual field is lacking in reliable, easily distinguishable features, our eyes can lose their ability to maintain a fixed target. The stars themselves begin to wander, moving erratically, as if under the control of some unknown intelligence. They are, in a sense, but that unknown intelligence is our own visual system performing poorly in a challenging environment. When a camera aimlessly hunts for focus we understand that it is a technical problem with the observation, but when our own eyes have similar trouble we have a hard time being so objective.

And then there are those phenomenon that are less common but still well known: meteorites, for example, which incidentally reached their peak frequency of the year, in the northern hemisphere, during the New Jersey flap. There are satellites, some of which can reflect light from the sun beyond the horizon in odd flashing patterns, and which are becoming far more numerous as Starlink continues its bulk launches. In the good seeing conditions of the rural Southwest you can hardly look at the sky and not find a satellite, or two, or three, or four, lazily wandering between the stars. Failing to find a moving light is more unusual than looking up and having one catch your eye.

But, most of all, there are airplanes. The FAA reports that their Air Traffic Organization provides services to about 45,000 flights per day, an underestimate of the total number of aircraft operations. There are some 800,000 certificated pilots in the US. During the peak aviation hours of the mid-day to evening, there are about 5,000 IFR flights over the US at any given moment---and that's excluding the many VFR operations. The nation's busiest airports, several of which are located in the New Jersey region, handle more than one arriving or departing flight per minute.

The sky is increasingly a busy place.

When the drone flap reached its apex a few weeks ago, news stations and websites posted video montages of the "drone sightings" sent in by their viewers or, well, found on Twitter by their staff. The vast majority of the objects in these videos were recognizably commercial aircraft. Green light, left wingtip. Red light, right wingtip. White light, tail. Flashing light, location varies, usually somewhere in the middle. During approach and departure, airliners are likely to have landing lights (forward-facing) and inspection lights (pointed back at the engines and wings) turned on. If you live near an airport, you probably see this familiar constellation every day, but you aren't calling it in to the news.

And this is where the UFO phenomenon is unavoidably psychosocial.

For as long as UFOs have been observed, skeptics (and psychosocial theorists) have noted that those observations tend to follow a fashion. In the late nineteenth century, the only thing anyone had made to fly were airships, and so everyone saw airships. By the mid-20th century, the flying saucer had been introduced. The exact origin of the flying saucer is actually surprisingly complicated (having precedents going back decades in fiction), but the 1947 UFO flap solidified it as the "classic" form of UFO. For most of the golden age of UFOlogy, flying saucers were a norm punctuated only by the occasional cigar.

During the 1970s, the development of computer modeling for the radar return of flat surfaces (ironically by a Soviet physicist who seemed largely unaware of the military applications and so published his work openly) enabled the development of "stealth" aircraft. Practical matters involving the limitations of the modeling methods (the fewer vertices the better) and the low-RF-reflectivity materials known at the time meant that these aircraft were black and triangular. During the 1980s and 1990s, a wave of "black triangle" UFO sightings spanned the country, almost displacing the flying saucer as the archetypal UFO. Some of these were probably genuine sightings of the secret F-117, but far more were confirmation bias. The popular media and especially UFO newsletters promulgated this new kind of craft. People were told to look for black triangles, so they looked, and they saw black triangles.

This phenomenon is often termed "mass hysteria," but I try to avoid that language. "Hysteria" can invoke memories of "female hysteria" and a long history of dismissive and unhelpful treatment of disempowered individuals. To the extent that mass hysteria has a formal definition, it tends to refer to symptoms of such severity as to be considered illness. A flap has a different character: I am not sure that it is fair to say that someone is "hysterical" when they look in an unfamiliar place and see what they have been told everyone else is seeing.

While rather less punchy, I think that "mass confirmation bias" is a better term. "Mass susceptibility to suggestion," perhaps. "Mass priming effects." "Mass misunderstanding." "Mass surprise at the facets of our world that are always there but you seldom care to notice."

There are a surprising number of balloons in the air. Researchers launch them, weather agencies launch them, hobbyists launch them. They can drift around for days, or longer if carefully engineered. They are also just about as innocuous as an aerial object can be, rarely doing anything more nefarious than transmitting their location and some environmental measurements. And yet, when a rare sophisticated spy balloon drifts across the country, everyone starts noticing balloons for the first time. The Air Force shoots a few down. Then, cooler heads prevail, and we all end up feeling a bit silly.

There are some lessons we can learn from the Chinese spy balloon incident. First, there are strange things up there: spy balloons have a long history, having been used by the United States to observe the Soviet Union in the 1950s. That balloon program, short-lived for diplomatic reasons, laid the groundwork for a surprising number of following military and scientific developments on the part of both countries (and, in true American fashion, General Mills). From this perspective it is no surprise that the Chinese have a spy balloon program, they are treading down a proven path and once again finding that the political problems are more difficult than the technical ones (In the 1950s, the United States took the position that countries did not have a claim to control of their upper airspace, an argument that the Chinese would have a hard time making today).

Second, there are a lot of routine things up there. In the great menagerie of aerial balloons, spy balloons are probably the rarest type. Any wispy, high-altitude drifter you might see is vastly more likely to be a scientific or hobby project. Far from unusual, they are in this field the definition of "usual." Normal denizens of the sky, like airliners and satellites and stars.

Third, it is difficult to tell the difference. Even the military struggles to tell one from the other, since balloons operate at high altitudes, are small in size, and even smaller in radar cross section due to the radio transparency of the envelope. The general public has little hope. So, they interpret things as they have been primed.

Normally, people do not see balloons, because they do not look. On the occasion they happen to notice one, they dismiss it as, well, probably a weather balloon. Then, a Chinese spy balloon makes the news. Suddenly people look: they notice more balloons, and when they do, their first thought is of Chinese intelligence. They interpret things as they have just been told to.

---

I do most of my writing from Flying Star, and you can help pay for my posole and cake. That is a sentence that will probably only make sense to people in the immediate area. Anyway, the point is, if you enjoy my writing consider supporting me on ko-fi. I send out an occasional special newsletter, EYES ONLY, to my supporters.

I have another appeal as well: I am considering starting a separate newsletter, probably once monthly, in which I round up the UFO/UAP news with an eye towards cutting it down to just the meaningful new information. If you're vaguely aware that there keep being congressional hearings and occasionally new reports, this would bring you up to date on the important parts. Is that something you'd be interested in? Let me know, there's contact info in the footer.

---

If I seem to be belaboring the point, appreciate that I am trying to thread a needle. It is ridiculous, unreasonable, and frankly embarrassing for the media to disseminate "evidence" of a "drone incursion" that are plainly just blurry videos of Southwest flights on final. I am fast to fault the media. At the same time, I am much slower to blame the people who take these videos. They are, in a sense, just doing what they were told. They started looking for the drones, and now they are seeing drones.

The media has never been a friend to serious inquiry into UFOs. For much of the 20th century, "yellow journalism," intentional sensationalism, was the main vector by which UFO reports spread. These newspaper accounts held up to no scrutiny, and the journalists that filed them were often fully aware of that fact. The papers would print pretty much anything. There was a certain wink-and-nod aspect to most UFO reporting, which both spread UFOs as a popular phenomenon and hopelessly undermined the credibility of any actual sightings.

Today, yellow journalism is mostly a thing of the past, but it has been replaced by a new practice with similar outcomes. I think it has a lot to do with the fundamental collapse of journalism as an industry: the average city newsroom seems to consist of about three half-time reporters whose main source is their Twitter feed and primary interest is keeping their jobs by producing content fast enough to stay "fresh." They hardly have time to find out what happened at the City Council meeting, much less to critically evaluate twenty different UFO tips. The papers will print just about anything.

To the workaday New Jersey reporter, the drone flap must be a bit of a godsend. News is falling right into their laps. Video---the most important form of online content, the best engagement driver, the promised beachhead of the media conglomerate into the TikTokified culture of youths, is just showing up in their inboxes. This person says they saw a drone! Just like everyone's talking about! They have video! Of course you publish it. You'd be stupid not to.

It is, of course, an airplane. Maybe the reporter knows that, I think they often do. The text somewhere around the video player, for anyone that reads it, usually has an appropriate number of weasel words cushioned in vague language. They're not saying that this person caught a drone on video, they're just saying that this person says they caught a drone on video. Please watch the video. Share it with your friends, on one of the platforms where that's still worth something.

Okay, I'll knock it off, I'm trying not to just be a doomer about the decline of the media to such an extent that no one knows what's going on anywhere except for Twitter and ivy league universities for some reason. I have to skim the City Council meeting videos myself because there are sometimes literally zero journalists who are paid to sit through them. I once gave an impassioned speech about some homelessness project at a city meeting, and when some guy walked up to me after the meeting and introduced himself as a reporter from the Journal, I actually said "the Journal has reporters?" to his face. I thought they just syndicated from the five remaining AP writers and select Facebook pages. And I guess whatever Doug Peterson is on about, but seriously, now that 've gotten onto local issues I really need to stop before I get into Larry Barker Investigates memes.

So let's talk about the drones. Drones are in the news, in a military context, in a regulatory context, in popular media. Tensions with China continue to heighten, and it's clear that China doesn't have too many compunctions about US airspace sovereignty. I mean, I think I actually believe them that the balloon incursion into US airspace was unintentional (better to stay off the coasts, right? that's where a lot of the good military exercises are anyway, and we can imagine that the balloon's maneuvering capabilities are probably quite limited and flight planning depends a lot on wind forecasting which is not exact). But if they were really that broken up about it, they probably would have apologized via diplomatic channels before it became a major event. Clearly they were hoping it would go unnoticed.

First some items hit the news about mysterious drones. I'd love to identify a Patient Zero, but I don't think it's quite that simple, there was a confluence of a few things. Another congressional UAP hearing, reporting of drone incursions over Ramstein air base and Picatinny arsenal, and then a few random public reports of odd lights in the sky, as have always happened from time to time. But these separate incidents come together in the minds of the American public. A few people who are already inclined towards seeing strange things in the sky start looking for drones, and they see drones, or at least things that they are willing to conform to that expectation, even if only tentatively. They post on the internet. A cycle starts; it feeds on itself; more people looking, more sightings, more people looking, more sightings.

Somewhere along the way, US politics being what they are, Rep. Jeff Van Drew of New Jersey reports that he heard from "high sources" that the "drones" were coming from an "Iranian" "mothership" off the coast in the "Atlantic."

"These are from high sources. I don't say this lightly."

He added that the drones should be "shot down". [2]

Where the hell did that come from?! The thing is, it doesn't matter. Congresspeople going off on wild tangents repeating completely unconfirmed information that probably came via email from someone claiming to work for the CIA or whatever is just par for the course. I suppose it's always been true that if you want to find the truth you have to ignore the politicians, but it sure feels extra true right now. I don't think they're even exactly lying, they're just repeating whatever they hear that might serve an aim. It's almost an involuntary reflex. The entire series of congressional UAP hearings have been like this, basically devoid of any meaningful new information, but completely full of bizarre claims from unnamed sources that will never be seriously evaluated because no one thinks there's really anything to seriously evaluate.

The New Jersey Drone Flap is definitely that, a flap. Virtually everything you have heard is probably meaningless, just routine misperceptions that are both induced and amplified by the media. Politicians making solemn statements about needing to get on top of this, demanding a serious investigation, the DoD not doing enough, how we should shoot them down, are just doing what politicians do: they are Taking It Seriously, whatever It is. In a few weeks they will be Taking Something Else Seriously and American political discourse will move on without ever following up on any of it.

There's something curious about this flap, though, that I think does actually make it fundamentally different from the UFO flaps of yesteryear. It's the degree of strangeness involved. UFO enthusiasts sometimes use the phrase "high strangeness" to describe the more outlandish, the more inexplicable parts of UFO encounters. What people are claiming to see in New Jersey, though, is not high strangeness. It is not even strangeness. It's just... a little odd, at most.

The most authoritative government response to the New Jersey drones comes in the form of the "DHS, FBI, FAA & DoD Joint Statement on Ongoing Response to Reported Drone Sightings". Such a boring title gives you a degree of confidence that this is a Genuine Government Position, straight out of some ponderous subcommittee of the faceless bureaucracy. In other words, it's the real shit, too worked over by public information staff to likely contain baseless speculation or meaningless repetition of political discourse. If it's untruthful, it's at least intentionally untruthful, in some big organizational sense. It reads in part:

Having closely examined the technical data and tips from concerned citizens, we assess that the sightings to date include a combination of lawful commercial drones, hobbyist drones, and law enforcement drones, as well as manned fixed-wing aircraft, helicopters, and stars mistakenly reported as drones.

Here the government is saying: those aircraft you're seeing in the sky? Well, they're aircraft. You know, airplanes and stuff. Some of them are even drones! You know people just have drones, right? You can buy them at Costco. I don't think they have Costco in Iran, so I don't know where the Mothership gets them, but here in the god-bless-the-USA the DJI Mavic 3 Pro is $3,000 on Amazon and you can fly it all around New Jersey, at least for the moment. Probably just for the moment. If you're thinking about it I'd recommend that you buy now.

The real Fortean strangeness of the drone flap is that it is not Fortean. It's not paranormal, it's not mysterious. People are just looking at the sky and claiming to see something that is manifestly, objectively, actually a real thing that exists in the sky.

And yet they are still wrong about it most of the time.

I think that's why the government's messaging has been so weird and scattered. It's not like the Air Force is going to reassure us that there are no drones in the sky, because there are. I know people are getting really tired of the "does not pose a threat" language but what else are they supposed to say? It's like if there was a New Jersey Bird Flap. The National Audobon Society continues to examine the data, but to date the reported sightings of birds are assessed to be lawfully operating birds, or airplanes or helicopters or stars mistaken for birds. There is no indication that they pose a danger to national security.

And after all of this, what is left? Well, as always, the mystery is left.

For every ten thousand sounding balloons, there is a Chinese Spy Balloon (these numbers are made up for the purpose of rhetoric, please do not check my math). For every ten thousand "drone sightings," there is a real drone, operating somewhere it shouldn't, for unknown reasons.

The Joint Statement again:

Additionally, there have been a limited number of visual sightings of drones over military facilities in New Jersey and elsewhere, including within restricted air space. Such sightings near or over DoD installations are not new.

The military, and airports, and other security-sensitive installations have experienced occasional drone incursions for years. It rarely gets press. Most of the time it's some clueless hobbyist who crosses a line they shouldn't' have; this problem got bad enough that the FAA ended up deciding technical controls were required to make these mistakes more difficult.

There may be more more afoot: weeks ago some Chinese citizen, Yinpiao Zhou, was arrested for flying a consumer drone over Vandenberg Space Force Base to take photos. He reportedly said to federal investigators that the whole thing was "probably not a good idea," and it seems most likely he was just a SpaceX fan who wanted to get closeups of their facility at Vandenberg and severely didn't think things out. But there are reasons to be suspicious, a couple of months ago five Chinese nationals who had been attending a US college were arrested for sneaking around a military exercise taking photos of sensitive equipment. Their whole sequence of activities, including lying about their travel and coordinating to destroy evidence, can succinctly be described as "very suspicious." They seem to have been fully aware that they were doing something illegal, which encourages one to speculate about their motivations even if the charges of espionage have not yet been adjudicated in court.

There is good evidence that Chinese intelligence coordinates with more or less random people that travel between the US and China to opportunistically collect information on military capabilities, so the idea that there are people operating consumer drones around military bases in service of Chinese interests is not a particularly far-fetched one. It just kind of makes sense. If you were a Chinese intelligence agent, wouldn't you give it a try? It's so low risk and low cost it could practically be some handler's side project.

Foreign adversaries do provide reasons to keep a close eye on drones, especially as they interact with sensitive sites and military operations. The DoD has an admitted inability to do so effectively, leading to a significant investment in methods of detecting and countering small drones. There is a drone problem. It's just not new, it's not specific to New Jersey, and it's not some big dramatic event, but a slow evolution of military and intelligence practice akin to the development of aviation itself.

The FAA has issued a number of temporary flight restrictions in the area, and the media has made a pretty big deal of that. But most of the flight restrictions aren't even that restrictive (they allow private operations if the FAA is notified and provided with a statement of work), and the FAA tends to reflexively issue flight restrictions when anyone gets nervous. It's probably a wise decision: all this talk of drones has, ironically, almost certainly brought the drones out. People probably are more likely to operate in an unsafe fashion near sensitive infrastructure sites. They're using their drones to look for all these drones they're hearing about! And they barely even know what drones are!

[1] One of the reasons I don't write about UFOs that often, besides the fact that it gets me more weird threatening emails than any other topic, is that it's very hard to explain a lot of the events of UFO history without providing extensive background. The beliefs of individual people in the UFO community vary widely with respect to the credibility of well-known individuals. When someone admits a hoax, there is virtually always someone else who will claim the admission of the hoax to itself be a hoax (if not CIA disinfo). Some people, like Doty, have gone through this cycle so many times that it's hard to tell which of his lies he's lying about. The point is that you can't really say anything about UFOs without someone disagreeing with you to the point of anger, and so if I'm going to say anything at all I have to sort of push through and just write what I think. I encourage vigorous debate, and historically it has often been the lack of such debate that has created the biggest problems. But, you know, please be polite. If I am a CIA shill they're not paying me much for it.

[2] Inconsistent quotation-and-punctuation style is in the original due to the BBC's internally consistent but odd looking style manual rules for putting the punctuation inside or outside of the quote. They are, incidentally, pretty close to what I usually do. See, it's not just me struggling with where to put the period.

Histories of radio broadcasting often make a particular focus on the most powerful stations. For historic reasons, WBCT of Grand Rapids, Michigan broadcasts FM at 320¸000 watts. Many AM stations are licensed to operate at 50,000 watts, but this modern license limit represented a downgrade for some. WLW, of Cincinnati, once made 500,000. Less is made of the fun you can have under 10 watts: what we now call the Traveler's Information Station (TIS).

The TIS was not formally established as a radio service until 1977, but has much earlier precedents. The American Association of Information Radio Operators, an advocacy group for TIS, has collected some of the history of early experimental low-power radio stations. Superintendent James R. McConaghie of Vicksburg National Military Park must have been something of a tinkerer, as he built a low-power AM transmitter for his car in the mid-1950s and used it to lead auto tours. He suggested that a tape recorder might be added to provide a pre-recorded narration, and so anticipated not only the TIS but a common system of narration for group tours to this day.

During the New York World's Fair in 1964, a "leaky cable" AM system was installed on the George Washington Bridge to provide driving directions to visitors. This is the first example I can find of a low-power AM station used for traffic guidance. I can't find much information about this system except that it was the work of William Halstead, a pioneering radio engineer. Halstead is best known for developing FM stereo, but as we will see, he was a major force in TIS as well.

The National Park Service continued to innovate in radio. Low-power stations offered a promising solution to the challenge of interpreting a park to increasing numbers of visitors, especially in the era of the automobile, when rangers no longer lead tour groups from place to place. In 1968, Yellowstone acquired six custom-built low power AM transmitters that were installed at fixed locations around the park. Connected to an 8-track player with a continuous loop cartridge, they broadcast park announcements and interpretive information to visitors approaching popular attractions.

As an experiment, Yellowstone installed a five-mile "auto nature trail," a road with regularly spaced AM transmitters built for the experiment by Montana State University. The notion of an "auto nature trail" confounds our modern sensibilities, but such were the 1960s, when experiencing the world from the interior of your car was an American pastime. In a 1972 article on the effort, park service employees once again pointed out applications beyond park interpretation:

Not only is this new aspect of radio communications opening interpretation of natural areas to motorists, but the idea of being able to communicate with hundreds of motorists without having them stop their cars is a patrolman's blessing.

Along these lines, the NPS article mentions that the California Department of Transportation had deployed a low-power radio station to advise travelers of a detour on I-5 following the San Fernando earthquake. I have, unfortunately, not been able to find much information about this station---but the NPS article does tell us it used equipment from Info Systems.

Info Systems, Inc. appears to have been the first vendor of purpose-built transmitters for low-power informational stations. I haven't been able to find much information about them, and I'm a little unclear on the nature of the company--- they were apparently reselling transmitters built by vendors including ITT. I'm not sure if they were built to Info Systems designs, or if Info Systems was merely a reseller of equipment originally intended for some other application. Of course, I'm not sure what that application would have been, because at the time no such radio service existed. These transmitters operated either at milliwatt power levels under Part 15 rules, or at 10w under experimental licenses. This perhaps explains why the National Park Service figures so prominently into the history of low-power radio: as a federal agency, they presumably obtained their authorization to use radio equipment from the NTIA, not the FCC. The NTIA was likely more willing (or at least faster) to issue these experimental licenses. Info Systems transmitters were extensively installed by NPS, likely over a dozen just at Yellowstone.

In 1970, the general manager of Los Angeles International Airport became frustrated with the traffic jams at the arrival and departure lanes. He hoped to find a way to communicate with approaching drivers to better direct them---a project for which he hired William Halstead. Halstead partnered with radio consultant Richard Burden to design and install the system, and we are fortunate that Burden wrote a history of the project.

In 1972, a leaky cable antenna was buried along the median of Century Boulevard as it approached the airport. A second antenna was buried along the main airport loop, and two different NAB cartridge message repeaters (tape loop players) drove two separate transmitters. Drivers would thus begin to hear a different message as they crossed the overpass at Sepulveda Boulevard. Here, the short range of the low-power transmitters and inefficient antennas became an advantage, enabling a fairly small transition area between the two signals that would otherwise interfere.

Each of the message repeaters had three different cartridges they rotated through: a list of airlines using each terminal, parking information, and traffic information. Some of these recordings, like the traffic information, had different prerecorded variations that could be used depending on the weather and traffic conditions.

An interesting detail of the LAX radio system is that it was coupled to a new signage strategy. During development of the recordings, Burden realized that it was very difficult to direct drivers to terminals, since the terminal numbers were indicated by high-up signs that weren't noticeable from road level. Brand new signs were installed that were color coded (to identify terminals or parking areas) and bore large terminal numbers and a list of airlines served. The signs from this project were apparently in use at LAX at least until 2012. There is, of course, a lesson here, in that any new interpretive or information system will be most effective when it's installed as part of a larger, holistic strategy.

LAX's new traffic radio station operated at 830 kHz under an experimental license. Unfortunately, early experience with the system showed that drivers had a hard time tuning to 830 kHz using the slider-type tuners of the era, creating a dangerous wave of distraction as they passed the signs advertising the new radio station. Burden wanted to move the station to an extreme end of the AM band, where drivers could just push the slider until it stopped. Unfortunately, 540 kHz, the bottom of the established AM band, was licensed to a Mexican clear-channel station and could not be allocated so near to the border. Instead, Burden convinced the FCC to allow an experimental license for 530 kHz: the vast majority of cars, they found, would receive 530 kHz just fine when tuned to the bottom of their range. The frequency was formally allocated for aviation NDBs, but not in use at LAX or almost any other airport. Thus we have the origin of 530 kHz as one of the two standard frequencies for TIS [1].

By 1973, the FCC had started the rulemaking process to create a 10w TIS radio service. The National Park Service, apparently wanting to take a conservative approach to equipment purchasing, chose to stop buying new low-power AM transmitters until transmitters certified under the new FCC rules were available. In practice, this would take four years, during which time the lost sales to NPS were so great that Info Systems went out of business.

During this period, a company called Audio-Sine continued to manufacture and promote Part 15 AM transmitters---but for a different application. The "talking billboard," they proposed, would improve outdoor advertising by allowing travelers to tune their radio for more information on a product they saw along the roadside. The talking billboard concept never really caught on---a prototype, in Minneapolis, advertised for the idea of the talking billboard itself. "Look for talking billboards throughout this area in the near future." At least one other was installed, but in Duluth, advertising for Dean Nyquist's primary race for Minnesota Attorney General. "The Audio Sign... gives a very positive pitch for the City of Duluth..." the campaign manager said. "I would advise the city or chamber of commerce to use one or more all the time." I wonder if he was invested in Audio-Sine. A newspaper article a few days later comments that the talking billboard apparently did not work, something the same campaign manager attributed to a railroad trestle blocking the signal.

This is an obvious limitation of Part 15 AM transmitters: the power limit is very low. Audio-Sine only really claimed a range of "4-8 blocks," and today I think you would struggle to meet even that. The more powerful 10W stations, operated under experimental licenses, could reach as much as eight miles in good conditions.

Despite their limitations, the Audio-Sine milliwatt transmitters did find some use as early equivalents of TIS. This overlap does make it amusing that when the California Department of Transportation introduced their first changeable message signs around the same time, they called them "talking billboards" in the press.

There exists to this day a "microbroadcasting" hobby, of individuals who operate low-power FM and AM transmitters under Part 15 rules. To these hobbyists, who are always looking to transmit the best signal they can within the rules, the specific technical details of these early transmitters are of great interest. They remain, to this day, just about the state of the art in intentional broadcast radio transmission within Part 15 rules. In fact, the availability of these commercially-manufactured low-power AM transmitters seems to have lead to a short-lived boom of "whip and mast" Part 15 AM stations that attracted the attention of the FCC---not in a good way. Various details of our contemporary Part 15, such as the 3-meter antenna, feed line, and ground lead limitation of 47 CFR 15.219, seem to have been written to limit the range of the early 1970s Info Systems and Audio-Sine transmitters, along with a few other less prominent manufacturers of the day.

There are historical questions here that are very difficult to answer, which is frustrating. The exact interpretation of the limits on Part 15 intentional radiators are of great interest to hobbyists in the pirate-radio-adjacent space of legal unlicensed broadcasting, but the rules can be surprisingly confusing. You can imagine this leads to a lot of squinting at the CFRs, the history, and what exactly the FCC intended the rules to be when they were originally written. The fact that the FCC actually enforces according to a booklet of standards that it won't release but may be based on 1970s installation practices only makes the matter more intriguing.

In 1977, the FCC promulgated Part 90 rules formally establishing the Traveler's Information Station/Highway Advisory Radio service. TIS were allocated 530 kHz and 1610kHz, the two extremes of the American AM broadcast band at the time. Incidentally, the AM broadcast band would later be extended up to 1700kHz, but TIS on 1610 has not been moved. 530 and 1610 remain de facto exclusively allocated to TIS today. TIS rules remain largely unchanged today, although there have been some revisions to clarify that the established practice of "ribbons" (sequences of TIS transmitters) was permissible and to allow 5 kHz of audio bandwidth rather than the former 3 kHz.

Part 90-certified TIS transmitters are now commercially available from several manufacturers, and widely installed. Power is limited primarily in terms of field strength, although there is an RF output power limit as well. Leaky cable systems are permitted up to 50 watts into a 3 km long antenna to produce a field of 2 mV/m at 60 m from the antenna; conventional antenna stations are limited to 10 watts power into a vertically polarized antenna up to 15 m high and a field strength of 2 mV/m at 1.5 km. Most TIS installations are "whip and mast" types similar to those at the genesis of the category, using a monopole antenna mounted at the top of a signpost-type mast with the transmitter in a weathertight enclosure mounted to the side of the mast. You learn to recognize them. Typical coverage for a TIS station is 3 km (indeed, that is the limit on the planned coverage area).

Searching for TIS licenses is a little odd because of the formalities of the licensing. All TIS licenses must be issued to "government entities or park districts," in part because TIS is technically part of the public safety pool. The AM frequencies allocated to TIS stations are sort of "transferred" to the public safety pool (on a primary basis for 530 kHz and secondary basis for 1600-1700 kHz). In other words, TIS licenses are best found in ULS by searching the PW (public safety pool, conventional) service for frequencies between 0.530-1.700 MHz. There are 1,218 such licenses active.

I'm not going to provide a breakdown on all thousand-plus licenses, but I did take a quick look for any "interesting" entries, and some boring ones as examples of a typical application.

Consider the very first result, KMH441, licensed to the State of Illinois for 1610 kHz. It appears to have a surprisingly large tophat antenna. It probably serves weather advisories for the nearby freeway. Rather dull, but most TIS are just like this, except with less impressive antennas. KNIP553 is licensed to the Foothill-De Anza Community College District Police in Los Altos Hills, CA, at 1610 kHz as well. It's probably on the roof of one of the campus buildings. Like most TIS, there are essentially no mentions of this station on the internet, except in listings of TIS based on licenses.

KNNN871 1610 kHz is licensed to the city of Vail, Colorado, and this one got a local news article when it was installed. There are two transmitters. WNKG901, Greater New Orleans Expressway Commission, is on 1700 kHz and has four licensed transmitters at various toll plazas. The transmitters are standard whips on masts, but this one is in an unusual place.

WNRO290, State of New Mexico, operates at 530 kHz at the St. Francis/I-25 interchange in Santa Fe. The transmitter is totally typical and shoved into a median space.

WPEZ840 is assigned to the Lower Colorado River Authority and covers 1610 or 1670 kHz at six locations, each a power plant (some of them hydroelectric, but the Lower Colorado River Authority apparently operates some coal plants). Like many emergency-oriented TIS, these stations normally rebroadcast NOAA All-Hazards Weather Radio.

While TIS are limited to government agencies, there are definitely some cases of private organizations finding a government sponsor to obtain a TIS license. For example, Meteor Crater in Arizona has signs at the freeway advising that there is attraction information on 1610 kHz. This is WQDF361, which is actually licensed to the nearby City of Winslow. Like many TIS stations, the license contact is Information Station Specialists, a company that specializes in TIS including both equipment and licensing.

Because TIS are ubiquitous low-power AM stations, some DX (long-distance receiving) enthusiasts will try to pick up very distant TIS. Historically, some TIS operators would issue QSL cards. Considering that there are quite a few TIS in service that are government-registered but seem to be physically maintained by radio clubs or amateur radio operators, there are probably still a fair number out there that will return a QSL card if you try.

Having discussed TIS, we finally need to consider the fact that there are a lot of things that look and feel like TIS but are not. Most notably, when the Low Power FM (LPFM) class was established in 2000, one of the authorized functions of LPFM stations is something that is very much like, but not quite, TIS. A notable advantage of LPFM stations for this purpose (besides the higher popularity of FM radio despite its poorer range) is that the license class explicitly allows large-area networks composed of many low-power transmitters---something that is kind-of-sort-of possible with TIS using very long "ribbon" sequences, but not encouraged. These rules mean that TIS-type LPFM networks can feasibly cover multiple towns.

A major example is in Colorado, where the state operates eleven LPFM stations such as KASP-LP, 107.9 FM Aspen. Anyone familiar with the extreme difficulty of actually getting LPFM licenses will be rather jealous of the State of Colorado for bagging eleven, but then government agencies do get preference. The Colorado stations rebroadcast NOAA All-Hazards Weather Radio with 100 W of power, mostly just allowing people to listen to them without having a tuner capable of covering the 160MHz weather band (an unfortunately common problem).

It's hard to know what the future holds for TIS. The broad decline in AM radio suggests that TIS may fade away as well, although it appears that AM receivers will be mandated in vehicles sold in the US. Some states, such as Virginia, have significantly reduced the number of TIS in operation. Still, some TIS systems are popular enough with drivers that plans to eliminate them lead to public objections. Most TIS operators are increasingly focusing on emergency communications rather than traffic advisories, since TIS offers a very reliable option for communications that is completely under local control---very local control, considering the short range.

[1] Wikipedia suggests that an NDB on 529 kHz at Manchester, TN can be heard in many parts of the US. There's a weird lack of basic information on this NDB, such as its location or the name of the airport it is located at. It seems to have been installed at a private airport by an amateur radio operator, probably as more of a hobby project than anything. I cannot find it on contemporary charts or even find an airport that fits the description, and I don't see references to it newer than 2009, so I think at least the NDB and possibly the entire airport are gone to history.

At the very core of telephone history, there is the telephone operator. For a lot of people, the vague understanding that an operator used to be involved is the main thing they know about historic telephony. Of course, telephone historians, as a group, tend to be much more inclined towards machinery than people. This shows: websites with information on, say, TD-2, seldom tell you much about the operators as people.

Fortunately, telephone operators have merited more than just a bit of discussion in the social sciences. It was a major field of employment, many ideas in management were tested out in the telephone companies, and moreover, telephone operators were women.

It wasn't always that way. The first central exchange telephone system, where you would meaningfully "place a call" to a specific person, arose in 1877. It was the invention not of Bell, but of Edwin Holmes, a burglar alarm entrepreneur. His experience with wiring burglar alarms back to central stations for monitoring may have made the idea of a similarly-wired telephone exchange more obvious to Holmes than to those in other industries. Holmes initially staffed his telephone exchange the same way he had staffed his alarm and telegraph businesses: with boys. The disposition of these young and somewhat unruly staff members became problematic when they spoke direct with customers, though, so Holmes tried something else. He hired Emma Nutt, the first female telephone operator.

Women telephone operators were a hit. Holmes quickly hired more and customers responded positively. The matter probably had little to do with their gender, but rather with the cultural norms expected of young men and young women at the time, but it takes a certain perspective to differentiate the two (for example, it cannot be ignored that the switch from boys to women as telephone operators also involved the other change implied by the terms: most women telephone operators were hired as young adults, not at 12 or 13 as telegraph boys often were). The way the decision was reported at the time, and sometimes still today, is simply that women were better for the job: calmer, more friendly, more professional, more obedient.

With her extreme youth, her gentle voice, musical as the woodsy voices of a summer day, her always friendly way of answering calls, she is a sensible little thing, tranquilly serene through all the round of jollies, kicks and nerve-racking experiences which are the result of a day's labor. She likes her place, she knows her work, and she is prepared with quick-witted, instinctive readiness for every emergency which comes her way. [1]

Alexander Graham Bell was very much aware of the goings-on at Holmes' company, which AT&T would purchase in 1905. So the Holmes Telephone Despatch Co., the first telephone exchange, became the pattern for many to follow. During the last decades of the 19th century, the concept of exchange telephone service rapidly spread, as did the role of the operator. Virtually all of these new telephone workers were women, building a gender divide in the telephone industry that would persist for as long as the operator.

Operators stood out not just for being women, but also for their constant direct interaction with customers. To a telephone user, the operator was part of the machine. The operator's diminished humanity was not unintentional. The early telephone industry was obsessed with the appearance of order and reliability. The role of fallible humans in such a core part of the system would undermine that appearance, and with the social mores of the time, the use of women would do so even more. The telephone companies were quick to emphasize to customers that operators were precisely trained, tightly managed, and a model of efficiency. The virtues of telephone operators, as described by the Bell companies, reflect the semi-mechanical nature of their professional identities: a good operator was fast, precise, efficient, reliable.

Within the Bell System, new operators attended training schools that taught both the technical skills of telephony (operation of the exchange, basic understanding of telephone technology, et) and the behavior and manner expected from operators. Operators were not expected to bring their personalities to the workplace: they followed a detailed standard practice, and any deviation from it would be seen as inefficiency. In many companies, they were identified by number.

There was, of course, a tension underlying the role of the operator: operators were women, chosen for their supposed subservience and then trained to follow an exact procedure. At the same time, operators were women, in the workforce in a time when female employment remained unusual. The job of telephone operator was one of few respectable professions available to women in the late 19th and early 20th centuries, alongside nursing. It seemed to attract the more ambitious and independent-minded women, and modern studies have noted that telephone operators were far more likely to be college-educated heads of households than women in nearly any other field.

A full examination of telephone operators, their role in the normalization of working women, the suffrage movement, and etc., would require a much better education in the liberal arts than I have. Still, I plan to write a few articles which will lend some humanity to the telephone industry's first, and most important, switching system: first, I will tell you of a few particularly famous telephone operators. Second, I plan to write on the technical details of the work of operators, which will hopefully bring you to appreciate the unusual and often very demanding career---and the women that took it up.

We will begin, then, with one of my favorite telephone operators: Susie Parks. Parks grew up in Kirkland, Washington, at the very turn of the 20th century. After a surprising amount of family relocation for the era, she found herself in Columbus, New Mexico. At age 17, she met a soldier assigned to a nearby camp, and they married. He purchased the town newspaper, and the two of them worked together operating the press. Columbus was a small town, then as well as now, and Parks wore multiple hats: she was also a telephone operator for the Columbus Telephone Company.

The Columbus Telephone Company seems to have started as an association around 1914, when the first telephone line was extended from Deming to a series of telephones located along the route and in Columbus itself. An exchange must have been procured by 1915, when Henry Burton moved to Columbus to serve as the young telephone company's full-time manager. Burton purchased land for the construction of a new telephone office and brought on his sister as the first operator.

Rural telephone companies were a world apart from the big-city exchanges of the era. Many were staffed only during the day; emergency service was often provided at night by dint of the manager living in a room of the telephone office. Operators at these small exchanges had wide-ranging duties, not just connecting calls but giving out the time, sending help to those experiencing emergencies, and troubleshooting problematic lines.

By 1916, Susie Parks sat at a 75-line common battery manual exchange. Unlike the long line multiple boards used at larger exchanges, this one was compact, a single cabinet. When a nearby lumberyard burned in January and the fire damaged the telephone office, Parks stepped in with a handy solution: the telephone exchange was temporarily moved to the newspaper office, where she lived.

The temporary relocation of the exchange would prove fortuitous. Unknown to Parks and everyone else, in February or March, Mexican revolutionary Pancho Villa sent spies into Columbus. His army was in a weakened state, traveling between temporary camps in northern Mexico and attempting to gather the supplies to resume their campaign against the Federal forces of President Carranza.

The exact reason for Villa's attack on Columbus remains disputed; perhaps they hoped to capture US Army weaponry from the nearby fort or perhaps they intended to destroy an ammunition depot to deter US advance into Mexican territory. We also don't know if Villa directed his spies to locate the communications facilities in Columbus, but it's said that they failed to identify the telephone exchange because of its temporary relocation. The spies were evidently not that good at their jobs anyway, as they significantly under counted the number of US infantry stationed at Columbus.

On March 9th, 1916, Villa's army mounted what might be considered the most recent land invasion of the United States [2]. Almost 500 of Villa's men moved into downtown Columbus in the early morning, setting fire to buildings and looting homes. Susie Parks awoke to screams, gunfire, and the glow of a burning town. A day before, her husband had left town for the homestead the two worked. Parks was alone with their baby, and bullets flew through the modest building.

At the nearby infantry camp, two machine gun units came together to mount a hasty defense. While much more formidable than Villa had believed, they were nonetheless outnumbered and caught off guard, some of them barefoot as they advanced towards town with a few M1909s.

Susie Parks was barefoot, too, as gunfire shattered a window of the newspaper office. Keeping her head down, she maneuvered in the dark, knowing that a light would no doubt attract the attention of the raiders. Parks found her way to the exchange and, cord in hand, tried their few long distance leads. El Paso was no good: Villa's forces had cut the line. The line to the north, though, to Deming, had escaped damage. The Deming operator must have had her own fright as Parks described the violence around her. In short order, the message was passed to Captain A. W. Brock of the National Guard.

Somewhere along the way, a bullet or at least a fragment hit her in the throat. Unsure if she would survive, she hid her baby under the bed. According to most accounts, she stayed with the switchboard, keeping a low profile until the battle ended. According to her son, in an obituary, she took up a rifle of her own and made way for the Army camp. I suspect there are elements of the truth in both: she probably did get a gun, but I think she was more intending to defend the baby than the soldiers, who were apparently able to take care of themselves.

The Battle of Columbus ended as quickly as it began, and the exact order of events is told in different ways. Villa may have already given the order to retreat, seeing his substantial losses against the increasingly organized machine gunners from the Columbus camp. In a version more complimentary to our hero, it was the arrival of Brock's company, spotted coming into town, that lead to the withdrawal. In any case, the sunrise appearance of the National Guard in Columbus decisively ended the invasion.

It began a series of campaigns against Villa, culminating in the assignment of General John Pershing to oversee a six-month "Punitive Expedition." They didn't find Villa, but they did prove out the use of air support and truck transport for a wide-ranging expedition through northern Mexico. The experience gathered in the expedition would be invaluable in the First World War soon to follow.

Susie Parks is remembered as a hero. Charlotte Prince, a former first lady of the New Mexico Territory, and the Daughters of the American Revolution presented her with a gold watch and silverware set at a celebration in Columbus's small theater. General Pershing, on his arrival to begin the Punitive Campaign, paid her a visit to commend her for keeping her post through a raging battle [3].

The original Columbus telephone exchange, and other memorabilia of the Columbus Telephone Company and Susie Parks, are on display in the top floor of the Telephone Pioneer Museum of New Mexico.

Parks had set a high standard for her fellow telephone operators, not just in New Mexico but beyond.

The next year, the United States would enter the First World War. Major General Fred Funston, an accomplished military leader and veteran of the Spanish-American War, was favored to lead the US Army into Europe. By bad luck, he died of a heart attack just a couple of months before the declaration of war. Funston was replaced by the General he had sent into Mexico. John Pershing traveled to France as commander of the American Expeditionary Forces.

Upon Pershing's arrival, he found Europe in disarray. Communications in France were tremendously more difficult than the high standard the Army maintained at home. There were both technical and organizational challenges: telephone lines and exchanges had been damaged by fighting, and the Army Signal Corps lacked the personnel to improve service.

The idea to dispatch American telephone operators to Europe likely originated in the Army Signal Corps and AT&T, with whom they already maintained a close relationship. But I like to think that Pershing remembered the bravery of Susie Parks when he signed on to the plan, cabling the US to send send "a force of Woman telephone operators."

At the time, women had been admitted to the military only as nurses, and those nurses were kept far from the front. There was substantial doubt about the fortitude of these women, especially as they would be called on to staff exchanges near combat. The Secretary of War allowed the plan to go forward only on the condition that men would be hired preferentially and women would be carefully selected and closely supervised.

Operators were selected in by the Army Signal Corps in cooperation with AT&T. It was initially thought that they would be found among the staff of the many Bell operating companies, but the practicalities of the AEF (which was headquartered and primarily fought in France in collaboration with French units) required that operators speak both French and English fluently. There were few French-speaking telephone operators, so AT&T expanded their search, hiring women with no telephone experience as long as they were fluent in French and passed AT&T's standardized testing process for telephone proficiency. These recruits were sent to the Bell System's operator training schools, and all selectees attended the Army Signal Corps' training center at what is now Fort Meade.

The first unit of the Signal Corps Female Telephone Operators Unit [4] consisted of 33 operators under the leadership of Chief Operator Grace Banker, who had learned French at Barnard College before finding work at AT&T as an instructor in an operator training school. Their 1918 journey to France was a long and difficult one, as transport ships were in short supply early in the war and subject to German attack. The ferry crossing of the English Channel, not a long voyage by any means, turned into a 48-hour ordeal as the ship was stuck in dense fog in a vulnerable position. Despite the cold and damp conditions, the operators waited two days on deck in preparation to take to the life boats if necessary. Two men on the ship died; at one point French forces mistook its faint outline for an attacker and surrounded it. As Banker would tell the story later, her operators were in good spirits.

Their cheerful disposition in the face of the harsh journey served as good preparation for the conditions the operators faced in the field. There was hardly a barracks or telephone exchange for the women that wasn't plagued by leaks, rats, fleas, or disarray as the AEF scrambled to find facilities for their use. The simple mechanics of the telephone system required that exchanges be located fairly close to concentrations of command staff and, thus, fairly close to the fighting. The operators were constantly in motion, moving from camp to camp, and ever closer to the front.

Banker's first unit of 33 women quickly proved themselves invaluable, providing faster and more reliable telephone service as they leveraged their French to handle all allied traffic and developed directories and route guides to keep up with the rapid work of the Signal Corps' men in building out new telephone lines. The Female Telephone Operators had proven themselves, and Pershing called for more. A few months later, hundreds more were in France or on their way. Despite the War Department's concern about the willingness of women to work in wartime conditions, telephone operators turned out to be as ready to fight as anyone: when AT&T solicited applications from among the Bell companies, they were swamped by thousands of postcard forms.

While some sectors the military were clear that the women operators were brought to Europe for their technical proficiency, there remained a clear resistance to recognition of their work as part of the military art. "Even telephone operators were persistently told that their presence and their girlish American voices would benefit the war effort by comforting home-sick soldiers and lifting their morale" [5]. The operators were, at times, regarded in the same stead as the women "morale volunteers" fielded by organizations like the YWCA.

The military was so quick to categorize them as such that, shortly after their arrival, the YWCA was made responsible for their care. Operators were accompanied by YWCA chaperones, furnished to protect their moral virtues from the soldiers they worked alongside. Despite their long shifts at the exchanges, the YWCA expected them to attend military dances and keep up appearances at social functions. Many of the women associated with the AEF, telephone operators and nurses alike, took to cutting their hair short---no doubt a practical decision given the poor housing and inconsistent access to washrooms, but one that generated complaints from the Army and the YWCA.

In September of 1918, the AEF and French troops---a quarter million men in all---took on their first great offensive. The logistics of supporting and organizing such a large fighting force proved formidable, and the Signal Corps relied on the telephone to coordinate a coherent assault. The thunder of artillery was heard over the chatter of telephone calls. For the duration of the offensive, a system of field phones and hastily laid long-distance connections, known as the "fighting lines," fell under the control of Grace Banker and five operators she hand-picked to move up to the front with her. They donned helmets and coats, toted gas masks, and took up their positions at temporary exchanges, some of them in trenches. Infantry orders, emergency calls for supply, and even artillery fire control passed through their plugboards as the allies took Saint-Mihiel.

As a reward, they moved forward once again, taking up a new "telephone office" at the allied advance headquarters in Bar-le-Duc. There, they camped in old French army buildings and weathered German bombing as they provided 24/7 telephone service for the Meuse-Argonne offensive. Military service was demanding, but still subject to the "scientific management" trend of the time and the particular doctrine of the Bell System. Their long shifts were carefully supervised, subject to performance evaluations and numerical scoring. There was a certain subtext that the women operators had to perform better than the Signal Corps' men who they had replaced.

Fighting ended in November of 1918, although many of the operators were assigned to various post-war duties in Europe (including Grace Banker's assignment to the French residence of President Wilson) during 1919. The first 33 operators had spent 20 months in France before they returned to the United States, where Banker would complain of the low stakes of civilian work.

After the war, the Female Telephone Operators received numerous commendations. Major General H. L. Rogers of the Signal Corps spoke of their efficiency and the quality of the telephone service under their watch. The Chief Signal Officer reported that "a large part of the success of the communications of this Army is due to... a competent staff of women operators." Pershing personally signed letters of commendation to a number of the operators, referring to their "exceptionally meritorious and conspicuous services." Operators who had worked near the front received ribbons and clasps for their involvement in the offensives. Grace Banker, for her own part, was awarded the Army's Distinguished Service Medal. Of 16,000 officers of the Signal Corps in the First World War, only 18 received such an honor.

Considering the decorations these women wore on their Signal Corps jackets as they returned to the United States, it is no wonder that modern accounts often style them as the "first women soldiers." The female nurses of the Red Cross, while far more numerous, were never as close to the front or as involved in combat operations as the operators. The operators were unique in the extent to which they considered themselves---and they were often seen by others---to be members of the Army.

After the war, they would learn at the same time as many of their commanding officers they were not. Earlier, the Army had quietly determined them to be contracted civilian employees. None other than General Pershing himself had ordered them to be inducted into to the army in his original letter to headquarters, and recruiting materials explicitly used the terms "enlistment" and "regular Army," even introducing the term "women soldiers." But even before the first 33 shipped out for France, Army legal counsel had determined that military code prohibited the involvement of women. None of the women were told; instead, they were issued uniforms.

450 members of the Female Telephone Operators Unit worked 12-hour shifts, handling 150,000 telephone calls per day, often not only making connections but serving as interpreters between French and American officers. The Signal Corps' male telephone operators, more experienced in the Army, were of such noticeably poorer performance that they were restricted to night shifts---and even then, only in safe territory well behind the front. Two operators, Corah Bartlett and Inez Crittenden, died in the service of the United States and were buried in France with military honors. Years later, it was noted that because of their critical role in military logistics, the operators were among the first Americans to reach the combat theater and among the last to leave.

They were discharged as civilians---or rather, they were not discharged at all. Because of the Army's legal determination, the women received no Army papers and were deemed ineligible for veteran's benefits or even to receive the Victory Medal which the Signal Corps had promised them.

Despite its recognition of their exceptional service, the military was slow to admit women's role outside of wartime exigency, or even in it. The United States as a whole was even slower to recognize the work of the telephone operators. Despite the introduction of 24 bills to congress, starting in 1927, it was not until 1977 that the operators were declared regular members of the Army and granted military benefits. By the time the act was put into effect in 1979, only 33 operators lived to receive their discharge papers and the Victory Medal.

AEF telephone operator Olive Shaw, who tirelessly lobbied for military recognition of her fellow women, was the first burial at the new Massachusetts National Cemetery in 1980. Her wartime uniform, fitted as always with the brass devices of the Signal Corps and the letters "U.S.," was presented to congress as evidence of their rightful role as veterans in 1977 and cited again, in 2024, when all of the members of the Army Signal Corps Female Telephone Operators Unit were awarded the Congressional Gold Medal. It is now on display at the National World War I Museum in Kansas City.

The Female Telephone Operators Unit laid the groundwork for the induction of women during World War II---the Women's Army Auxiliary Corps and the United States Navy's Women's Reserve, or WAVES, which is remembered today for its exceptional contributions in the fields of cryptography and computer science. It is fitting, of course, that the achievements of the WAVES would be exemplified by another Grace, Rear Admiral Grace Hopper.

"Women's work," far from being frivolous, was now defined as essential to the war effort, and the U.S. military found itself in the uncomfortable position of being dependent on female labor to meet the structural needs of the war economy. Ironically, then, it was the logic of sex segregation in the civilian economy that compelled the U.S. government to grant women entry into the armed services, the ultimate masculine preserve. [5]

[1] "A Study of the Telephone Girl," Telephony magazine (1905).

[2] A 1918 conflict at Nogales, AZ, involving similar combatants, might also lay claim to that description. I will argue in favor of the Battle of Columbus, which was an unprovoked invasion, as compared to the Battle of Ambos Nogales which was more of a border security conflict in reaction to years of rising tensions.

[3] Parks was an interesting figure for the rest of her life. She and her husband continued to move around, buying the Clackamas News in Oregon. Her husband's condition declined, a result of surgical complications and a morphine addiction, and they split up. During the Second World War, Parks found herself back in wartime service, as a sheet metal worker at the Seattle-Tacoma Shipbuilding Company. In 1981, the Deming Headlight, closest newspaper to Columbus, reprinted her obituary from the Seattle Post-Intelligencer. It recounts a half dozen careers, two husbands, and 36 grandchildren.

[4] The members of the Female Telephone Operators Unit are frequently referred to as the "hello girls," but this is a more generic term for telephone operators that would also come to refer to other groups, be used as the title of works about telephone operators, etc. I prefer to stick to something a little more precise.

[5] Susan Zeiger, "In Uncle Sam's Service: Women Workers with the American Expeditionary Force, 1917-1919" (2019).

As an American, I often feel an intense jealousy of Ceefax, one of several commercially successful teletext services in the UK and Europe. "Teletext" is sometimes a confusing term because of its apparent relation to telecom-industry technologies like the teletypewriter and telegram, but it refers specifically to a "broadcast text" technology usually operated over TV networks. Teletext could be considered the first form of "interactive television," a reimagining of traditional television as a more WWW-like service that allows viewers to navigate through information and retrieve content on-demand.

Despite many, many attempts, interactive television was never particularly successful in the US. Nor, I believe, did it fare well in Europe after the retirement of teletext. It was an artifact of a specific time and place; once PC ownership and internet access expanded they handily filled the niche of interactive text. That feels a little surprising, televisions being a big screen that so many consumers already had in their home, but offerings like MSN TV sucked to use compared to PCs. The technology for interacting with PC software from a couch honestly still isn't quite there [1], and it was even worse in the '90s.

Despite its general failure to launch, Interactive TV was, for a time, a big field with big ideas. For example, you've heard of MPEG-4, but what about MHEG-5? That's the Multimedia Hypermedia Expert Group's effort towards an object-oriented, television-native hypermedia environment, and it's exactly as terrible and fascinating as that description would lead you to believe. But I'm not going to talk about that today. Here's what's on my mind: what if I told you that MSN TV was Microsoft's second attempt at interactive television?

In 1994, Microsoft formed a partnership with two cable television carriers to launch Cablesoft. It was such a big hit that Microsoft spent most of its brief life trying not to talk about it.

You might remember the days when a television with a standard QAM tuner could often pick up on-demand content being watched by other people. And that's the basic interactive television model at work: as a cable customer with on-demand features, your STB presents a set of menus to select a program. When you start it, a video server at the head end selects an unused digital television channel and plays back the content on it. Your STB is sent a command to tune to the correct channel, and all of your playback control actions (play/pause etc) are sent to the video server. Today the video downlink is always encrypted, but in the heyday of CATV on-demand encryption was inconsistent and some providers didn't use it at all, leaving these downlink channels viewable by anyone with a tuner not configured to hide them.

On-demand isn't particularly exciting, and perhaps only barely counts as "interactive television," but it is the most substantial thing to come out of US efforts. The late '80s and early '90s saw plenty of interactive television ideas, which tended to envision the TV as the main way consumers would get information. Because of the limitations of the technology, this exciting world was mostly text. Text on the TV. Nothing would really catch on, even to the trial phase, until interactive TV started to become synonymous with multimedia.

The 1990s were an exciting time in computer multimedia. Larger storage devices (namely CD-ROMs), faster processors, and better display controllers all meant that computers were becoming more and more practical for audio and video. In the video game industry, the period famously lead to a surge of "full motion video" (FMV) games that used live-action cutscenes or elaborate pre-rendered 3D scenes. Of course, most enthusiasts of video game history also wince at the thought of figuring out which exact version of Apple QuickTime will work right with any given title.

Despite its surging popularity, computer multimedia was also in its infancy. Audio and video encoding were dominated by proprietary systems like QuickTime and RealMedia. SOver time, these products and their underlying codecs would largely converge into the relatively consistent and consumer-friendly ecosystem of media formats we use today (i.e. everything is MPEG and consumers don't care about the rest, mostly, as long as they don't want h.265 in a web browser on Linux or something wild like that).

Some of that convergence happened because of vendors actively contributing to standardization and promoting licensing pools, but some of it also happened because one of the biggest players in the PC software industry saw Apple's success with QuickTime and didn't want to fall behind. Microsoft developed a major focus on multimedia, leading to their own family of codecs, containers, and protocols, some of which remain in common use today. What's more, Microsoft had a long-running fascination with the television distribution industry, which it tended to view as the future of media delivery due to its very high capacity compared to telephone lines. Microsoft itself, and its executives as individuals, had a variety of interests in cable TV starting in the '90s. Perhaps most prominently, Paul Allen was controlling owner of Charter for a decade, and Microsoft invested a billion dollars in Comcast in 1997 to support their effort to pivot towards data.

So, the application of Microsoft technology to cable television was inevitable. Microsoft brought Tele-Communications Inc (TCI, now part of Comcast) and Time Warner (somehow not part of Comcast yet) on board as CATV partners and set about building Microsoft Media for CATV. Or, perhaps, TCI and Time Warner formed a joint initiative to develop an interactive TV platform and selected Microsoft as a partner. The history is a little fuzzy, but somehow, these three companies ended up in high-level talks about a new, standard platform for technology-enabled TV. Cablesoft, as they called it, would include an electronic program guide, TV shopping, and, most importantly, on-demand streaming media.

I'm trying not to say this over and over again, but Microsoft and Bill Gates and Paul Allen were all kind of obsessed with streaming media and on-demand delivery in the 1990s. It's hard to keep track of all the failed ventures they either launched or invested in, there were several each year. If you read into the history of the TV distribution industry Paul Allen especially just keeps popping up in weird places. It's fascinating to me because our modern experience shows that they were very much right, in that on-demand streaming delivery via computers would become the dominant way media is distributed. But they were also pathologically ahead of their time; Paul Allen was basically trying to do Netflix in 1993 and all of these efforts just sucked. The infrastructure was simply not there and the many companies trying to build it tripped over each other as often as they made progress.

To be fair, Microsoft was not the only faction making repeated stabs at streaming media, and by 1994 investors were already starting to tire of it. A 1994 News Tribune (Tacoma) article on Cablesoft's announcement captures the attitude with this spectacular quote from industry newsletter editor Denise Caruso: "Anybody who bothers to get excited about another interactive TV trial at this point deserves everything they get in terms of disappointment." In 1994! All These Goddamned Streaming Services is a complaint pretty much as old as computer multimedia. I wonder what Denise Caruso would have to say about Tubi.

As you know, things didn't get a whole lot better. In 1996, the Boston Globe's "FastTrack" technology column began: "Interactive television is the Loch Ness Monster of the information age---much talked about but rarely seen." And it's not hard to find these quotes, those are like the first two search results in the archive I use. The consumer internet was barely a thing and industry commentators were already rolling their eyes at each new streaming service.

The difference, of course, is this: back in the '90s, these streaming multimedia efforts were collapsing fast, generally before they signed up actual consumers. Now they collapse very slowly, after producing about a hundred original TV series that none of us will ever hear of. Say what you will of Cablesoft, at least they didn't make Tall Girl and Tall Girl 2.

So what did they make? Look, Cablesoft didn't get very far, and there's not a lot of historical information about them. You have to be careful not to confuse Cablesoft with CableSoft, a completely separate company that was working on the exact same thing at pretty much the same time (CableSoft had spun off of television technology giant General Instrument and thus had a considerable advantage, but it didn't work out for them either) [2].

By early 1994, Microsoft was already involved in other interactive TV ventures, leading to a somewhat critical interview of future Microsoft CTO and cookbook author Nathan Myhrvold by the Seattle Post-Intelligencer. "In the long run," he opined, "it's very likely there will be some form of a smart TV... it's not very input intensive, you don't have a keyboard for your TV." 100% correct!

But then the interviewer, Jim Erickson, asks something along the line of "what's with these three different interactive TV things that Microsoft is doing at once?" Myhrvold answers that "there is more uniformity and more synergy than may meet the eye with the series of things that we have done so far," which sounds like a comedy sketch of a Google exec explaining the difference between Duo and Allo. Erickson digs a little deeper, asking what's going on with Cablesoft, prompting Myhrvold to say "it's a funny thing to give status on something you never announced and never admitted to." And that is a very interesting response indeed.

The Wikipedia article is an absolute stub, giving us just one tantalizing factoid that has my practically foaming at the mouth: "...a custom version of the Windows NT operating system known as NTAS, which was essentially a series of fine-tuning efforts to drive ATM switches." We'll get back to that. But the Wikipedia article also says that Cablesoft was announced in 1994, which isn't wrong, but is a little misleading. As far as I can tell, Microsoft "announced" Cablesoft in March 1994 only under duress. Rumors of Cablesoft started to swirl about nine months earlier, in 1993, and the media did not look on it very positively. The most widely published article quoted then-chairman of Apple John Sculley accusing Microsoft of an anticompetitive move to corner the interactive TV market.

There is, of course, nothing more quintessentially Microsoft than an anticompetitive move to capture a market that would never actually emerge.

The first widespread mention of Cablesoft running under the headline "Big Software-Cable Deal Criticized" does a bit to explain Microsoft's odd cageyness about Cablesoft, repeatedly denying that any final deal had been signed and even downplaying the likelihood of the product launching. TCI and Time Warner refused to talk about it. Charmingly, a Phil Rowe of Battle Creek, Michigan wrote in to the editor of the Battle Creek Enquirer that Microsoft, TCI, and AT&T (I think Rowe was just confused? AT&T had its own interactive television efforts going on) would soon monopolize the interactive TV market, and that to hold them off, Battle Creek should swiftly franchise wireless cable.

It seems that Cablesoft died under the same cloud that it emerged. No one is really that clear on what happened. A trial program apparently launched where TCI and Microsoft employees in the Puget Sound area could try it out. It must not have lasted very long, by 1995 an article about Microsoft's antitrust woes listed Cablesoft as one of the ventures that Microsoft had abandoned due to the scrutiny. "Everyone backed off," an anonymous Microsoft employee told another reporter. "They were all afraid that this thing would be regulated out of existence."

Cablesoft didn't make much of a contribution to the business, but was it technically significant? And what about that customized version of Windows NT? Denise Caruso comes up again: an archived version of her personal website is the Wikipedia article's main source. She wrote:

Code-named Tiger and now called the Microsoft Media Server, the innovative design is based on a version of the Windows NT operating system, called NTAS, that uses standard PCs and cutting-edge ATM (asynchronous transfer mode) networking products to deliver video, audio, animation and information services into the home.

Streaming media was difficult in 1999; it was very difficult in 1993, when Microsoft's efforts began. Hard disks were slow, and head contention meant that it was very hard to serve multiple video streams from one disk. Networks were slow and, worse, had high levels of latency and jitter compared to what we are used to today. Feasibly providing real-time unicast streams to a large set of users would require some sort of large, very high performance storage system---or, in a strategic move that has repeatedly revolutionized the server side, a lot of consumer hardware and a clever system of coordination.

Microsoft technical report 96-09 describes the Tiger Video Fileserver. It was released after the Cablesoft project had faltered and never mentions it by name, but it clearly describes the head-end equipment for an on-demand video streaming system. Its authors, including Myhrvold, include a half dozen people with long careers in distributed systems and high performance storage.

A Tiger fileserver consists of multiple consumer PCs running Windows NT. Each of these nodes, called "cubs," has multiple hard disks. Files are separated into blocks (64kB-1MB) which are distributed across disks in the cluster; there are no constraints on the nature of the files except that they must be at the same playback bitrate. This constraint exists because the entire Tiger system operates on a synchronized timeslot schedule, consisting of block service times which are equal to the time required to read one block from disk, plus some margin for transient events and error recovery.

When a viewer requests a video, a controller node allocates the viewer to slots in a schedule of block service times and cubs. This is done such that each successive block of a given video will be handled by a different disk, and such that no one disk will be needed by more than one viewer in a given block service time. In other words, video playback consists of a series of cubs each delivering a single block of the file in order, and each disk retrieves only one block at a time. Because the block service time (and thus rate at which the schedule is executed) is appreciably shorter than the time viewers spent playing back back that same block, the cubs are able to support multiple viewers and still deliver blocks on time.

Because disks were not fast enough to reliably perform two block reads within a block service time, and aggregation of multiple viewers into one logical stream remained an elusive challenge, Tiger used a simple hack to avoid noisy neighbor problems: the controller ensured that there was only one viewer of a given file at a given time index. In practice, if two user were to hit "play" on the same movie at the exact same time, the controller would slightly delay the beginning of streaming to one of the viewers in order to introduce a time offset. Combined with the striping of each file across multiple nodes, this naturally distributed load to allow a large number of simultaneous viewers of the same media without having to create additional replicas of the media.

The controller determines and distributes the schedule in advance, and each cub is permitted (and expected) to retrieve blocks early as its I/O allows. But cubs are required to send that block in the correct schedule slot, so that storage buffering occurs only at the cub level and the outgoing network stream is in perfect realtime. When the file is distributed across cubs and disks, extra copies of each block are stored for redundancy, in case of a disk or cub failure. Extra slack in the block service time allows a failed block retrieval to be moved to a different cub. Secondary blocks are allocated by organizing the cubs into a ring. Each primary block has secondary copies stored on one or more cubs "to the right," and each cub is responsible for monitoring the liveness of its neighbor "to the left" and assuming its schedule if required.

The physical layout of files on each disk is optimized for fault load; the "primary" copy of each block is stored on the (physical) outer part of the disk surface while the inner part of the disk surface is used for secondary (backup) copies. Because the outer surface of the disk moves physically faster, it can be read more quickly. By placing primary blocks on the outer half, the disk's normal "primary" workload runs at nearer the disk's maximum read speed, leaving spare time for retrieval of secondary blocks when block sizes and bitrates are optimized for the disk's average read speed. When primary blocks are lost due to disk failure, they are automatically restored to a new disk as soon as one is available.

Besides real-time streaming of files, Tiger also supported ad-hoc read and write operations. These were performed on an opportunistic basis, sent to cubs as "extra" jobs to execute when they were ahead on carrying out scheduled reads. When viewers fast-forwarded or rewound playback, these opportunistic jobs were used to jump-start playback at other points in the file, with the caveat of reduced reliability.

During the course of normal video playback, individual blocks will come from an arbitrary sequence of different cubs. There are several approaches to the network design, and Tiger supports UDP over both Ethernet and ATM, but ATM is preferred. ATM is Asynchronous Transfer Mode, a network protocol that originated in the telephone industry as part of the ISDN stack. Unlike Ethernet, ATM was designed for real-time data streams, and uses prescheduled time-division muxing to provide guaranteed-bandwidth virtual circuits over a switched fabric. This made ATM inherently more suited to streaming media than Ethernet, a difference that Ethernet only made up for with quality of service protocols and, mostly, just getting to be so fast that streaming media mostly worked out despite having only intermittent, opportunistic access to the network media.

Microsoft further enhanced ATM for the Tiger application by introducing the ATM "funnel," a multipoint-to-point networking mode that allows many cubs to send packets into a single virtual circuit. ATM subdivides packets into multiple frames, meaning that if two cubs were to send packets too close together, they may become interleaved (both violating the design of IP-over-ATM and complicating the work of the viewer). To resolve this problem, Tiger uses a token-passing scheme where each cub transmits its block and then passes a token to the next cub in the schedule for that viewer. The implementation of this token-passing ATM variant is one of two customizations to the NT kernel involved in Tiger.

The other will be familiar to readers in the modern high-performance networking industry: Tiger implemented a basic form of kernel-bypass networking in which the network interface read the file block directly from the read buffer via DMA. Tiger thus required some special kernel-mode code to implement this DMA-to-UDP mode and ensure that video data passed over the bus only twice, once from the disk controller to memory, and once from memory to the network controller.

These kernel features, which I believe were less modifications than device drivers, seem to be the "customizations" that Caruso referred to. To my frustration, the connection to the term "NTAS" seems to be mistaken. I cannot find any instance of Microsoft using it in relation to Cablesoft or Tiger. Most likely it arose from confusion with the branding of NT 3.1's server edition as NT Advanced Server; NT 3.1 must have been the basis for the Tiger system since it was released in 1993 and had considerable emphasis on performance as a network file server to compete with Netware.

The technical report describes a model Tiger deployment: Five Gateway Pentium 133MHz machines served as cubs, each with 48MB of RAM, three 2GB Seagate drives, and an OC-3 ATM adapter. Larger than usual 2KB sectors were used on the hard disks for better throughput when handling large files (this is an interesting detail since support for non-default sector sizes was apparently rather cantankerous in the PCs and storage interfaces of the time). A Gateway 486/66 machine served as the controller for the cluster. Ten 486/66s, each attached to the ATM network by 100Mbps fiber, served as test clients.

The controller, on the other hand, used 10Mbps ethernet to communicate with the five cubs, while the cubs communicated among themselves using the ATM network. The paper explains this by noting that the 486-based controller was very slow compared to the Pentium-based cubs, perhaps no benefit was seen in the added cost of extending the ATM network to the controller.

This system, with a total of 15 data drives, stored a little over 6 hours of media at 6Mbps. 0.75MB blocks were used, for a block playback time of roughly one second. Based on calculations, the OC-3 interfaces were the bottleneck of the system, allowing the five cubs to provide a total of 68 simultaneous streams. The 486-based viewer machines actually weren't fast enough to decode that many streams, so some requested videos and simply discarded the packets, while others actually checked the received blocks for correctness. Based on this sampling method, a lost or late block rate of only 0.02% was observed. Performance data collected from the cubs indicated that, with faster network controllers, they would have kept up with additional viewers.

Despite Cablesoft's failure to ever reach the market, Tiger appears to have included novel work in several areas: scheduling real-time demand across a cluster, distributed storage with striping and replication for fault-tolerance and performance. At the bottom line, Tiger demonstrated the use of a cluster of commodity PCs to perform a task that, at the time, was seen as requiring specialized and costly high-performance machines from SGI or Oracle.

In the end, it wasn't enough. Caruso again:

Unlike the PC business, where it has ultimate leverage over all the links of a relatively short chain, Microsoft has no native influence in the large and existing infrastructures it wants to penetrate (cable networks, telcos and content providers)---except that it is Microsoft.

Cablesoft ended so quietly that we will probably never know for sure, but I think that Caruso's larger argument in her article about Cablesoft is right: licensing software for STBs and cable headends would never bring in Bill Gates money. Microsoft's real play was for revenue share: wrestling matches and hotel pornography had proven that on-demand content could move a lot of money, and as the operator of the technical platform, Microsoft stood to impose royalties. 5% of pay-per-view revenue is the kind of thing that motivates Microsoft. When Microsoft went to major cable operators to try to standardize them on its platform, the cable oligopoly saw the software monopoly coming in for a slice of the pie. They probably thought they could do it just fine on their own.

And they turned out to be right. Interactive TV would die off as a buzzword in the US market, but not that much later, in the '00s, General Instrument successor Motorola would partner with the manufacturers of some of those specialized costly machines and quietly introduce video on demand to the American cable consumer as a standard offering of digital TV packages.

I think they met more success because they didn't try as hard: '00s on-demand infrastructure involved some formidable machines, like the Broadbus (later Motorola) B-1 that served thousands of simultaneous video streams by storing the entire active video library in RAM. But no one was talking about hypertext or smart TVs or the TV as the center of digital family life. The aspects of interactive TV that were familiar to the television industry, electronic program guides and more convenient pay-per-view, did just fine once they became technically feasible.

What of Tiger?

As part of NT4 in 1996, Microsoft introduced NetShow. NetShow would later be known as Microsoft Media Server, and then Windows Media Server, before fading into obscurity. It was a real-time media server that used a proprietary protocol to deliver Advanced Streaming Format media, competing directly with streaming pioneer RealNetworks. The details are fuzzy enough that I have a hard time saying this for sure, and Tiger as a distributed system definitely didn't make it past Cablesoft, but it does seem very likely that NetShow is a descendent of Tiger. Over time, the Windows Media family moved over to industry standard protocol RTSP, and then the streaming server was merged into IIS. Perhaps nothing of Tiger survived into the 21st century. But, you know how Microsoft is. Maybe there's a bit of Tiger inside of Windows to this day.

[1] As a dedicated computer-on-TV person, I use a ThinkPad Trackpoint Keyboard II as the "remote control." It's the best I've found so far but still large and clunky compared to a traditional remote. More recently, my husband also added an HDMI-CEC adapter that allows the TV remote to control the computer via a daemon that generates keyboard events. This is pretty slick for applications like Plex and Steam Big Picture that were designed with television use in mind, but in the web browser the experience leaves much to be desired. We're basically facing all the same struggles as similar dweebs did thirty years ago.

[2] And neither should be confused with Cablesoft Ltd., an English company that developed software for the telephone and television industry during the same years, or CableSoft, a company that made a stock quotation and analytics product a couple of years later.

I have quipped before about "underground datacenters," and how they never succeed. During the late decades of the Cold War and even into the '00s, the military and (to a lesser extent) the telecommunications industry parted ways with a great number of underground facilities. Missile silos, command bunkers, and hardened telephone exchanges were all sold to the highest bidder or---often in the case of missile silos---offered at a fixed price to the surrounding land owner. Many of them ended up sealed, the new owner only being interested in the surface property. But others...

There are numerous examples of ex-defense facilities with more ambitious owners. There ought to be some commercial interest in a hardened, underground facility, right? After all, the investment to build them was substantial. Perhaps a data center?

There are several ways this goes wrong. First, there are not actually that many data center clients who will pay extra to put their equipment underground. That's not really how modern disaster recovery plans work. Second, and probably more damning, these ventures often fail to anticipate the enormous cost of renovating an underground facility. Every type of construction is more expensive when you do it underground, and hardened facilities have thick, reinforced concrete walls that are difficult to penetrate. Modernizing a former hardened telecom site or, even worse, missile site for data center use will likely cost more than constructing a brand new one. Indeed, the military knows this: that's why they just sold them, often at rock-bottom prices.

Even if these "secure datacenters" almost never succeed (and rarely even make it to a first paying client), they've provided a lot of stories over the years. CyberBunker, one of the less usual European examples (a former NATO facility), managed to become entangled in cybercrime and the largest DDoS attack ever observed at the time, all while claiming to be an independent nation. They were also manufacturing MDMA, and probably lying about most of the equipment being in a hardened facility to begin with.

So that's obviously a rather extreme example, sort of a case study in the stranger corners of former military real estate and internet crime. But just here in New Mexico I know of at least two efforts to adopt Atlas silos as secure datacenters or document storage facilities, neither of which got off the ground (or under the ground, as it were). It seems like a good idea until, you know, you actually think about it. You might recall that I wrote about a secure data center claiming to be located in a hardened facility with CIA and/or SDI ties. That building doesn't even appear to have been hardened at all, and they still went bankrupt.

What if I told you that they were all barking up the wrong tree? If you really want to make a business out of secure underground storage, you need something bigger and with better access. You need a mine.

It will also be very important to make this play in the early Cold War, when there was a much clearer market for hardened facilities, as evidenced by the military spending that period building them rather than selling them off. The 1980s and on were just too late.

There are actually several rather successful businesses built on the premise of secure, hardened storage, and they are distinctively pre-computer. The best known of them is diversified "information management" firm Iron Mountain. You know, with the shredding trucks. And an iron mountain, or rather, an iron mine by that name.

Like most of the large underground facilities that are still commercially successful today, the story of Iron Mountain involves mushrooms. Efficient cultivation of mushrooms requires fairly tightly controlled conditions that are not dissimilar to those you already find underground: cool temperatures, high humidity, and low light. Culinary mushrooms are widely produced in large caves and former mines, which often provide more convenient access, having been designed for bulk transportation of ores.

This might be a little surprising, because we tend to think of underground mines as being small, cramped spaces. That's true of many mines, often for precious metals, for example. But there are also mines that extract ores that are present over large areas and have relatively low value. This requires an efficient method of removing a very large quantity of rock. Modern mines employ some very clever techniques, like "block caving," where a very large rock mass is intentionally collapsed into a prepared chamber that it can be scooped out of like the bottom of a hopper. One of the most common methods, though, and one that has been in use for a very long time, is "room and pillar" mining.

The idea is pretty simple: you excavate a huge room, leaving pillars to hold up the material above. Depending on the economics, geology, etc., you might then "retreat," digging out the pillars behind you as you work your way out of the room. This causes the room to collapse, ideally away from the working area, but not always. Retreat mining is dangerous and doesn't always produce ore of that much value, so a lot of mines didn't do it. They just left the rooms, and their pillars: huge underground chambers, tens and hundreds of thousands of square feet, empty space. Many were dug back into a hill or mountain, providing direct drive-in access via adits. Most, almost all, successful underground storage facilities are retired room and pillar mines.

In the first-half of the 20th century, mushroom cultivation was the most common application of underground space. That's what led "Mushroom King" Herman Knaust to purchase the disused Mt. Thomas iron mine near Livingston, NY in 1936. Knaust's company, Knaust's Cavern Mushrooms, was the largest producer of culinary mushrooms in the world. Ten crops of mushrooms were produced each year in the Livingston mine, and as Knaust continued to grow his operations, the mushroom mine became one of the area's principal employers. Knaust dubbed it Iron Mountain.

By 1950, things had changed. Knaust had at least two motivations for his pivot: first, the US mushroom industry was rapidly weakening, upset by lower-cost mushrooms imported from Europe and Asia. Second, WWII had come to a close, and the Cold War was beginning.

In 1952, Knaust told reporters of his experience working with refugees from Europe resettled in New York. Most of them had lost everything to bombing, and they told Knaust how they had attempted to hide their most valuable possessions, and their paperwork, in safer places. The Germans, Knaust read, had come up with the best hiding place of all: disused mines. During the course of the war, the Nazi administration stored valuables ranging from gold bullion to original works of art in former mines throughout their occupied territories. Some of them were large-scale logistical operations, with rail access and organized archival systems.

Now, in the age of nuclear weapons, Knaust thought that this kind of protection would be in more demand than ever. In 1951, he renovated the old mine and installed new ventilation equipment. Most importantly, he bought a 28-ton vault door secondhand from a failed bank. A generator and a security force of armed former police officers rounded out Knaust's new venture: Iron Mountain Atomic Storage.

The bank vault door was mostly for show, and Knaust's description of the mine as "A-bomb proof" and "radiation proof" somewhat stretch the science. But Knaust was a born marketer; his version of nuclear alarmism drew the attention of corporate America like the Civil Defense Administration's pamphlets gripped the public. The entrance to his mine was a sturdy stone block building with iron bars over the windows and "Atomic Storage Vaults" inscribed at the top. He was sure to tell reporters of his estate, home to the world's only mushroom-shaped swimming pool. Over the years of newspaper coverage, the bank vault door at the front of the mine got heavier and heavier.

In the event of a nuclear attack, Knaust reasoned, banks could lose their records of deposits. Insurers could lose track of their policies. Companies of all kinds could lose inventory sheets. The resulting economic chaos would be as destructive as the bomb that started it. If we were to shelter lives, we also needed to shelter information. By the time Iron Mountain Atomic Storage was open for business he had already signed up the first customers, who shipped copies of their records for storage in individual vaults constructed within chambers of the mine. The East River Savings Bank, of New York, proudly described how each of their branches microfilmed new deposits daily for transport to the Mountain.

Iron Mountain sold its services to individuals as well. For $2.50 a year, a consumer or small business could pack records into a tin can for storage in the mine. The cans could be stored or retrieved with local agents stationed in major cities, who sealed them in the presence of the customer and shipped them to the mine by courier.

By 1954, Iron Mountain boasted over 400 customers, mostly large corporations and institutions. It was a surprising hit with newspapers: about 150 rented space to store their archives. Recordak, a subsidiary of Kodak that provided microfilming services, set up a branch office at the mine with representatives who could convert records to microfilm or turn them back into full-sized versions on demand. The consumer part of the business was reorganized in partnership with the Railway Express Agency, one of the descendants and ancestors of our modern American Express and Wells Fargo. Individuals and small businesses could deposit records with any Railway Express agent and request records sent back to their nearest train station.

They quickly faced competition. Perhaps the most interesting was Western States Atomic Storage Vaults, who purchased a disused South Pacific Coast Railroad tunnel in the Santa Cruz mountains. The railroad right of way, near Zayante, California, received a similar conversion to caged storage units. At least a half dozen underground storage companies would be organized between 1950 and 1960.

The atomic storage industry was not always an easy one. Iron Mountain had a slow start, signing up few customers after their initial set of banks and newspapers. The Cuban Missile Crisis gave a considerable boost to sales, though, and revenue almost doubled in 1963 alone. Iron Mountain's inventory expanded from paper and microfilm records to original works of art, and they purchased a second mine, a former limestone mine nearby at Kingston, NY, to expand. They added office and dormitory facilities at both sites, to protect both their extensive staff of clerks and representatives of their customers in the event of nuclear war. "What good are the records if everyone in the firm is blown up," Iron Mountain's executive vice president offered.

Inside of Iron Mountain, behind the vault door, steel doors with combination locks protected individual vault suites ranging from closets to hundreds of square feet. Racks held boxes and cans of individual records deposited by smaller customers. The whole facility was maintained at 70 degrees Fahrenheit and 50% humidity, a task greatly eased by the surrounding earth.

They were dedicated to the privacy of their customers but also had a hard time passing up an opportunity for promotion, telling reporters of some of the publishers and television companies that stored their archives at Iron Mountain, and hinting at a "major New York City art museum" that leased space for its collection. Individual customers included doctors, stamp collectors, and "a whole lot of people who aren't talking as long as the outside world lasts."

The mid-1960s were the apex of the Cold War in the popular consciousness, and Iron Mountain's luck would not last through the broader decline in planning for all-out Soviet attack. Now called Iron Mountain Security Storage, quotes for newspaper articles shifted their focus towards civil unrest. In a world of campus riots, president James Price said, universities were moving their academic records underground.

Iron Mountain's good (for business) and bad (for society) mood must have been infectious, because competitors flourished even in 1970. Bekins, the moving and storage company, purchased 200 acres in the Diablo mountains of California. They intended to open the first underground storage facility specifically built for that purpose, and plans included a hotel and heliport for convenient customer access. The July 11th, 1970 edition of The Black Panther's Black Community News Service contains perhaps them most blunt assessment of the Bekins plan, one that would prove prescient.

The possibility of World War III is not as much an immediate threat to the life and well being of America's greedy capitalists as is the strong probability of the more severe "political consequences" that might be meted out by the masses, the people, for selfish crimes committed against them.

A year later, the plan had expanded to shelter for 1,000 "executives and office workers" for up to 30 days, an airport that could serve business jets, and a computer and communications center. Bekins said that it would help corporations survive a nuclear war, but was even more important in the event of rioting or terrorism. Bekins specifically called out unrest at UC Berkeley, and damage it had caused to academic records, as evidence of the need.

Blaine L. Paris, number one stooge manager of Bekins Company.., acknowledged that the hideaway, hideout survival shelter's main draw is the widespread fear, on the business executive level, of bombings, the random tossing of molotov cocktails, possibilities of kidnapping...

Some large companies, Bekins said, were planning to set up alternate corporate headquarters in the facility as soon as it opened. It would be something like the Mount Weather of the corporate world. Two years passed, and Bekins reimagined the facility as a regular-use business park rather than a contingency site, but still underground. Joseph Raymond, Bekins' director of Archival Services, quipped that employees might be more productive underground where they'd be "free from the distractions of the surface."

Bekins had bad news coming. The era of atomic storage had come to an end. Corporate fear of popular revolution proved insufficient to fund the ten million dollar project. The Bekins facility would never break ground. Iron Mountain, quietly and under cover of their typical boosterism, had run out of money.

In 1971, a group of investors formed Schooner Capital and bought them out. Their strategy: to focus on business records management and compliance, and largely drop the "underground" and "atomic" part. Beginning in 1978, Iron Mountain built dozens of new storage facilities that were normal, above-ground warehouses. At the same time, they shifted the focus of their sales from security to "information management." New legal requirements and tax regulations meant that records retention had become a complex and costly part of many businesses; Iron Mountain offered to outsource the entire matter. Their clerks collected records from businesses, filed them away, and destroyed them when retention was no longer required.

Iron Mountain remains the largest company in the business today. Most US cities have an expansive Iron Mountain warehouse somewhere on their outskirts, and their mobile shredding trucks are a regular sight in business districts. Still, a certain portion of the Cold War attitude remains. Unshredded records are said to be transported in unmarked vehicles, to avoid attracting attention. Iron Mountain facilities are not exactly hidden, but their locations are not well publicized, and they continue to use armed guards. Distinctive red "Restricted Area" signs surround each one.

And they still have plenty underground.

When you look into the history of Iron Mountain, you will see frequent reference to the Corbis Collection. The story of Corbis would easily make its own article, but the short version is that Corbis was founded by Bill Gates as a sort of poorly-thought-out electronic picture frame company. Over the span of decades, they amassed one of the world's largest private collections of historic photographs and media, and then collapsed into an influencer marketing firm. It is often noted that the Corbis collection, of over 15 million photographs spanning 150 years, is stored at Iron Mountain. This isn't quite correct, but it's wrong in an interesting enough way to make it worth unpacking.

In the 1950s, the Northeast Pennsylvania Industrial Development Commission (NPIDC) formed a task force to investigate opportunities for the reuse of the state's growing number of abandoned coal mines. Coal is mined almost entirely by the room and pillar method, and while there are practical challenges in reusing coal mines in particular, the amount of space involved was considerable. The NPIDC's first proposal was right in line with the cold war: they proposed that the Civil Defense Administration use the mines to store their stockpiles of equipment and supplies.

The Civil Defense Administration wasn't interested, they were worried that firedamp (flammable coal gases) would make the mines dangerous and high humidity would cause stored equipment to rust. Still, the idea rattled around the state of Pennsylvania for years, and sometime around 1953 one such mine near Boyers, PA was purchased by the newly formed National Storage Company. National Storage became one of Iron Mountain's key competitors.

Iron Mountain has become as large as it is by following a fine American economic tradition: monopolization. It outlasted its erstwhile atomic storage competitors by buying them. Western States Atomic Storage Vaults and their railroad tunnel, National Storage and their coal mine, and at least two other similar ventures became part of Iron Mountain in the 1990s.

It is the former National Storage facility in Boyers that holds the Corbis collection. It has a notable neighbor: the largest tenant at Boyers is the United States Office of Personnel Management, which famously holds both clearance investigation files and federal employee retirement records down in the old mine. In 2014, the Washington Post called the Boyers mine a "sinkhole of bureaucracy", describing the 600 OPM employees who worked underground manually processing retirement applications. These employees, toiling away in a literal paperwork mine, were the practical result of a decades-long failed digitization program.

Underground storage is still a surprisingly large business. Some readers may be familiar with "SubTropolis," an extensive limestone mine near Kansas City, which offers 55 million square feet of underground space. SubTropolis has never particularly marketed itself as a hardened or secure facility. Instead, it offers very cost-effective storage space with good natural climate control. Tenants include refrigerated logistics companies and the National Archives. There are a number of facilities like it, particularly in parts of the eastern United States where the geography has been amenable to room and pillar mining.

That's the irony of Iron Mountain: their original plan was a little too interesting. Iron Mountain continues to operate multiple underground facilities, both their own and those they have acquired. Some of them, including Boyers, even have datacenters. The clients are mostly media companies, with original materials they cannot easily duplicate, and legacy government and financial records that would be too costly to digitize. Sony Music stores their studio masters with Iron Mountain, a big enough operation that some of Iron Mountain's underground sites have small recording studios to allow for restoration without removing the valuable originals from safekeeping. Miles of film are stored alongside miles of pension accounts. No one talks about nuclear war. The bigger fear is fire, which is more difficult to contain and fight in these old mines than in purpose-built archival warehouses.

There are only so many masters to store, and the physical volume of corporate records is quickly declining. Atomic vaults hit a limit to their growth. The total inventory of underground corporate storage facilities in the United States today is much the same as it was in the 1960s, with more closing than opening. Offsite records storage is shrinking overall, and Iron Mountain is effectively in the process of a pivot towards (above-ground) datacenters and services.

Still, when you read about Mark Zuckerberg's 5,000 square foot bunker in Hawaii, or Peter Thiel's planned underground project in New Zealand, one can't help but wonder if the predictions of Bekins, and the Black Panthers, were just ahead of their time.

---

I hope you enjoy this kind of material on Cold War defense and culture. It's one of my greatest interests besides, you know, anything underground. For those of you who support me on Ko-Fi, in the next day or two my supporter newsletter EYES ONLY will be a short followup to this piece. It will discuss underground storage facilities of a slightly different kind: the records vaults constructed by the Church of Scientology and the Latter-Day Saints, and the extent to which these facilities also reflect Cold War concerns.

I am also working on something about waste-to-energy facilities that will probably be an EYES ONLY article, as a companion to an upcoming CAB article on the history of an experimental Department of Energy biomass power plant in Albany, Oregon. But first, I will write something about computers. I have to every once in a while.

PAYPHONES at High Volume

Existing sites! Earn BIG $$. Money Back Guarantee!

Dropshipping AliExpress watches, AI-generated SEO spam websites... marginally legal and ethical passive income schemes, that serve to generate that income mostly for their promoters, can feel like a modern phenomenon. The promise of big money for little work is one of the fundamental human weaknesses, though, and it has been exploited by "business coaches" and "investment promoters" for about as long as the concept of invesstment has existed. We used to refer mostly to the "get rich quick" scheme, but fashions change with the time, and at the moment "passive income" is the watchword of business YouTubers and Instagram advertising.

And what income is more passive than vending machine coin revenue? Automated vending has had a bit of a renaissance, with social media influencers buying old machines and turning them into a business. The split of their revenue between vending machine income and social media sponsorship is questionable, but it's definitely brought some younger eyes to an industry that is as rife with passive income scams as your average spam folder. Perhaps it's the enforcement efforts of the SEC, or perhaps today's youth just need a little more time to advance their art, but I haven't so far seen a vending machine hustle quite as financialized as the post-divestiture payphone industry.

For much of the history of the telephone system, payphones were owned and operated by telephone carriers. As with the broader telephone monopoly, there were technical reasons for this integration. Payphones, more specifically called coin operated telephones, were "dumb" devices that relied on the telephone exchange for control. In the case of a manual exchange, you would pick up a payphone and ask the operator for your party---and they would advise you of the price and tell you to insert coins. The coin acceptor in the payphone used a simple electrical signaling scheme to notify the operator of which and how many coins you had inserted, and it was up to the operator to check that it was correct and connect the call. If coins needed to be returned after the call, the operator would signal the phone to do so.

With the introduction of electromechanical and then digital exchanges, coin control became automated, but payphones continued to use specialized signaling schemes to communicate with the coin control system. They had to be connected to special loops, usually called "coin lines," with the equipment to receive and send these signals. The payphone itself was a direct extension of the telephone system, under remote control of the exchange, much like later devices like line concentrators. It was only natural that they would be operated by the same company that operated the control system they relied on.

Well, a lot of things have changed about the payphone industry. The 1968 Carterfone decision revolutionized the telephone industry by allowing the customer to connect their own device. Coin operated telephones in the traditional sense were unaffected, but Carterfone opened the door to a whole new kind of payphone.

In 1970, burglar alarm manufacturer Robotguard blazed the trail into a new telephone business. They imported a Japanese payphone that was a little different from the American models of the time: it implemented coin payment internally. Robotguard connected the payphone through one of their burglar alarm autodialers, a device that was already fully compliant with telephone industry regulations, and then hooked it up to a Southwestern Bell telephone line in a department store in in St. Louis. By inserting a dime, the phone was enabled and you could make a local call (the autodialer was used, in part, to limit dialing to 7 digits to ensure that only local calls were made).

Robotguard had done their homework, consulting the same law firm that represented Carterfone in the 1968 case. They believed the scheme to be legal, since the modified Japanese payphone behaved, to the telephone company, just like any other customer-owned phone. The New York Times quotes Southwestern Bell, whose attitude is perhaps best described as resignation:

Spokesmen for the Southwestern Bell Telephone Company, the operating company in that area, acknowledge that the equipment is in the store, that it is working as described and that it appears completely legal. There is nothing they can do about it at this time, they say.

There was, indeed, nothing that they could do about it. Robotguard had introduced the Customer-Owned Coin-Operated Telephone, or COCOT, to the United States. Payphones were now a competitive business.

Despite a certain air of inevitability, COCOTs had a slow start. First, there would indeed be an effort by telephone companies to legally restrict COCOTs. This was never entirely successful, but did result in a set of state regulations (and to a lesser extent, federal regulations related to long-distance calls) that made the payphone business harder to get into. More importantly, though, the technical capabilities of COCOTs were limited. The Robotguard design could charge only a fixed fee per call, which made it a practical necessity to limit the payphone to local calls. Telephone company payphones, which allowed long-distance calls at a higher rate, had an advantage. Long-distance calls were also typically billed by minute, which made it important for a payphone to impose a time limit before charging more. These capabilities were difficult to implement in a reasonably compact, robust device in the 1970s.

A number of articles will tell you that COCOTs became far more common as a result of payphone deregulation stemming from the 1984 breakup of AT&T. I would love to hear evidence to the contrary, but from my research I believe this is a misconception, or at least not the entire story. In fact, payphones were deregulated by the Telecommunications Act of 1996, but that was done in large part because COCOTs were already common and telephone companies were unhappy that conventional payphones were subject to rate regulation while COCOTs were not [1].

Divestiture did definitely open the floodgates of COCOTs, although I think that the advances in electronics around that time were also a significant factor in their proliferation. In any case, several manufacturers introduced COCOTs in 1984 and 1985.

These later-generation COCOTs were significantly more sophisticated than the mechanical system used by Robotguard. To the user, they were pretty much indistinguishable from carrier-operated payphones, charging varying rates based on call duration and local or long distance. This local simulation of the telephone exchange's charging decisions required that each COCOT have, in internal memory, a prefix and rate table to determine charges. Early examples used ROM chips shipped by their manufacturer, but over time the industry shifted to remote programming via modems. These sophisticated, electronically-controlled coin operated phones that did not rely on an exchange-provided coin line came to be known as "smart payphones" and even, occasionally, as "smartphones."

Smart payphones greatly simplified payphone operations and were even adopted by the established telephone companies, where they could save money compared to the more complex exchange-controlled system. But they also made COCOTs completely practical, as good to the consumer as any other payphone. As COCOTs became remotely programmable, the payphone business started to feel like a way to generate---dare I say it---passive income. All you had to do was collect the coins. Well, that and keep the phone in working order, which would become a struggle for the thinly staffed and overleveraged Payphone Service Providers (PSPs) that would come to dominate the industry.

One of the new entrants into the payphone business was a company that specialized in exactly the kind of remote management these new smart payphones required: Jaroth Inc., which would do business as Pacific Telemanagement Solutions or PTS. Today, PTS is the largest PSP in the United States, but that isn't saying a whole lot. They enjoyed great success in the 1990s, though, and were so well-positioned as a PSP in the '00s that they often purchased the existing payphone fleet from former Bell Operating Companies that decided to abandon the payphone business.

The 1990s were a good time for payphones, and they were also a good time for investment scams. Loose enforcement of regulations around investment offerings, the Dot Com Boom, and a generally strong economy created a lot of opportunities for "telecom entrepreneurs" that were more interested in moving money than information.

---

The problem of 1990s telecommunications companies funded in unscrupulous ways is not at all unique to payphones, although it did reach a sort of apex there. I will take this opportunity to go on a tangent, one of those things that I have always wanted to write an article about but have never quite had enough material for: MMDS, the Multichannel Multipoint Distribution Service.

MMDS was, essentially, cable television upconverted to a microwave band and then put through directional antennas. It was often marketed as "Wireless Cable," sort of an odd term, but it was intended as a direct competitor to conventional cable television. I think it's fair to call it an ancestor of what we now call WISPs, using small roof-mounted parabolic antennas as an alternative to costly CATV outside plant. Some MMDS installations literally were early WISPs: MMDS could carry a modified version of DOCSIS.

Wireless cable got a pretty bad rap, though. If you pay attention to WISPs, you will no doubt have noticed that while the low capital investment required can enable beneficial competition, it also enables a lot of companies that you might call "fly by night." Some start out with good intentions and just aren't up to the task, while some come from "entrepreneurs" with a history of fraud, but either way they end up collecting money and then disappearing with it.

MMDS had a huge problem with shady operators, and more often of the "history of fraud" type. Supposed MMDS startups would take out television and newspaper ads nationwide offering an incredible opportunity to invest in this exciting new industry. The scam took different forms in the details, but the most common model was to sell "shares" of a new MMDS company in the four-to-five-digit range. Investors were told that the company was using the capital to build out their network and would shortly have hundreds of customers.

In practice, most of these "MMDS startups" were in cities with powerful incumbent cable companies and, even worse, preexisting MMDS operators using the limited spectrum available for such a wideband service. They never had any chance of getting a license, and didn't have anyone with the expertise to actually build an MMDS system even if they got one. They just pocketed the money and were next seen on a beach in Mexico or in prison, depending on the whims of fortune.

These wireless cable schemes became so common, and so notorious, that if you asked a lot of people what wireless cable was the two answers you'd get are probably "no idea" and "an old scam."

---

It only takes a brief look at newspaper archives to find that the payphone industry was a little sketchy. There are constant, nationwide, near-identical classified ads with text like "buy and retire now" and "$150k yearly potential" and "CALL NOW!". Sometimes more than one appear back to back, and they're still nearly identical. None of these ads give a company name or really anything but a phone number, and the phone numbers repeat so infrequently that I suspect the advertisers were intentionally rotating them. This was pretty much the Craigslist "work from home" post of the era.

To understand payphone economics better, let's talk a little about how the payphone business operated. Telephone companies had long run payphones on the same payment model, by finding a location for the payphone (or being contacted by the proprietor of a location) and then offering the location a portion of revenue. In the case of incumbent telcos, this was often a fixed rate per call. So someone owned the location and the payphone operator paid them in the form of a royalty.

COCOTs enabled a somewhat more complex model. A COCOT might be located in a business, connected to a telephone company line, and remotely programmed by a service provider, all of which were different companies from the person that actually collected the money. The revenue had to get split between all of these parties somehow, but COCOTS weren't regulated and that was all a matter of negotiation.

Much like the vending machine industry today, one of the most difficult parts of making money with a payphone was actually finding a good location---one that wasn't already taken by another operator. As more and more PSPs spread across the country, this became more and more of a challenge. So you can imagine the appeal of getting into the payphone hustle without having to do all that location scouting and negotiation. Thus all the ads for payphone routes for sale... ostensibly a turnkey business, ready to go.

Ah, but people with turnkey, profitable businesses don't tend to sell them. Something is up.

Not all of these were outright scams, or at least I assume some of them weren't. There probably were some PSPs that financed expansion by selling or leasing rights to some of their devices. But there were also a lot of... well, let's talk about the second largest PSP of the late '90s.

Somewhere around 1994, Charles Edwards of Atlanta, Georgia had an idea. His history is obscure, but he seems to have been an experienced salesman, perhaps in the insurance industry. He put his talent for sales to work raising capital for ETS Payphones, Inc., which would place and operate payphones on the behalf of investors.

The deal was something like this: ETS identified locations for payphones and negotiated an agreement to place them. Then, they sold the payphone itself, along with rights to the location, to an investor for five to seven thousand dollars a pop. ETS would then operate and maintain the payphone while paying a fixed monthly lease to the investor who had purchased it---something like $83 a month.

It was a great deal for the investors---they didn't need any expertise or really to do any work, since ETS arranged the location, installed the phones, and even collected the coins. In fact, most investors purchased phones in cities far from where they lived, such was the convenience of the ETS model. There was virtually no risk for investors, either. ETS promised a monthly payment up front, and the contract said that they would refund the investor if the payphone didn't work out.

The ETS network was far larger than just Edwards could manage. Most of the investment deals were sold by independent representatives, the majority of them insurance agents, who could pick it up as a side business to earn some commission. Edwards sold nearly 50,000 payphones on this basis, many of them in deals of over $100,000. Small-time investors convinced of the value by their insurance agents, many of them retirees, put over $300 million into ETS from 1996 to 2000.

There was, as you might have guessed, a catch. One wonders if the payphones were even real. I think that at least many of them were; ETS ran job listings for payphone technicians in multiple cities and occasionally responded to press inquiries and complaints about malfunctioning payphones bearing their logo. Besides, the telecom industry recognized ETS as a huge PSP in terms of both installed base and call volume.

What definitely wasn't real was the revenue. ETS was a ponzi scheme. In 2000, the SEC went for Charles Edwards, showing that ETS had never been profitable. Edwards sponsored a NASCAR team and directed millions of dollars in salary and consulting fees to himself, but in the first half of 2000 ETS lost $33 million. The monthly lease payments to investors were being made from the capital put in by newer investors, and even that was drying up.

SEC v. ETS went on for six years, in good part due to an appeal to the Supreme Court based on ETS' theory that a contract that paid a fixed, rather than variable, monthly rate could not be considered a security. In 2006, Charles Edwards was convicted of 83 counts of wire fraud and sentenced to thirteen years in prison.

Edwards was far from the only coin-op fraudster. ETS was not unusual except in that it managed to be the largest. When a class-action firm and several state attorneys general went after ETS, their press releases almost always mentioned a few other similar payphone schemes facing similar legal challenges. Remember all of those classified ads? I suspect some of them were ETS, but ETS also had a more sophisticated sales operation than two-line classifieds. Most of them were probably from competitors.

The payphone industry crashed alongside ETS; ETS almost certainly would have collapsed (albeit likely more slowly) even if it had been above board. Increasing cellphone ownership from the '90s to '00s made payphones largely obsolete, and more and more established telcos and PSPs decided to drop them. One of the reasons for PTS's ascent was its willingness to buy out operators who wanted out: in 2008, PTS bought most of AT&T's fleet. In 2011, they bought most of Verizon's fleet. Almost every incumbent telephone company got out of the payphone business and most of them sold to PTS.

Given all that, you might think that payphone scams were only a thing of the '90s. And they mostly were, but you can imagine that there was an opportunity for anyone who could adapt the ETS model to the internet age.

Pantheon Holdings did just that. It's even more difficult to untangle the early days of Pantheon than it is ETS. Pantheon operated through a variety of shell companies and brands, but "the Internet Machine Company" was perhaps the most to the point. Around 2005, Pantheon built "internet kiosks" where customers could check their email, print documents, and even make phone calls for a nominal cash or credit card payment. Sometimes called "global business centers," these kiosks were presented as an exciting business opportunity to mostly elderly investors who were given the opportunity to buy one for just $18,000.

Once again, the kiosks were real, but the revenue was not. Pantheon placed the machines in low-traffic locations and did nothing to market them. By 2009, more than a dozen people had been convicted of fraud in relation to the Internet Machines.

Pantheon kiosks still turn up on the junk market.

[1] I spent quite a bit of time researching the history of payphone regulation to try to understand exactly what did change in 1984, how many COCOTs operated and on what legal basis from 1970-1984, etc. I did not have much success. What I can tell is that COCOTs were very rare prior to 1984 (so rare that the FCC apparently didn't know of any, according to a 1984 memo, despite the 1970 example), and by the late '80s were very common. The FCC seems to have taken the view, in 1984, that COCOTs had always been legal, and just weren't being made or used on any significant scale. That's somewhat inconsistent, though, with the fact that suddenly after 1984 divestiture a bunch of companies started making COCOTs for the first time. My best guess right now is that from 1970-1984 COCOTs were probably legal but were something of a gray area because of the lack of any regulations specifically applying to them. Some combination of divestiture broadly "shaking up" the phone industry, electronics making COCOTs much more feasible, and who knows what else lead multiple companies to get into the COCOT business in the mid-'80s. That lead the FCC to issue a series of regulatory opinions on COCOTs that consistently upheld them as legal, culminating in the 1996 act dropping payphone regulation entirely.

Prescript: I originally started writing this with the intent to send it out to my supporter's newsletter, EYES ONLY, but it got to be long and took basically all day so I feel like it deserves wider circulation. You will have to tolerate that it begins in the more conversational tone I use for the supporters newsletter. I am going to write a bit about some related local works of art and send that to EYES ONLY instead.

Over on pixelfed I posted these two photos.

It's Saturday morning, I have coffee and the cat is here and the work thing I was planning to do has mercifully turned out to not need to be done today, so I have time to kill. Let's talk a little bit about Art before I take on a real project for the day.

So let's talk about the photos first, and then we'll sort of widen our view to the big picture. Neither photo is that good, tbh, I really enjoy architectural photography but I rarely have more than my phone. It's actually rather difficult to find any good angles to photograph this particular corner from anyway, which is one of the common criticisms of it as a work of art. But I have to actually say what it is! It's a corner of the upper floor of The Shops at Crystals, an upscale shopping center attached to the Aria on the Las Vegas strip. It opened in '09, and the building was designed by prominent architect Daniel Libeskind. It is, in my opinion, not really that interesting of a building. It has a combination of dead mall vibes and Las Vegas energy that is not all that compelling, and it doesn't have the surrealism of The Forum Shops at Caesars, the Las Vegas destination I recommend if you want to see a real shopping mall situation. But, it is connected to an APM (the Aria Express), and I will take the slimmest of excuses to ride an APM.

There aren't a lot of good things I can say about Las Vegas architecture, but one compliment I can extend is that the casino developers have mostly retained a tradition of commissioning fine art for their buildings. Nothing can really unseat Dale Chihuly's "Fiori di Como," the 2,000 square foot, 40,000 pound glass sculpture that occupies the ceiling of the Bellagio's lobby. But the Shops at Crystals took a pleasingly modernist direction by commissioning James Turrell. Turrell is probably the most prominent member of the "Light and Space movement," which can be simply described as an installation art view of architecture with a particular emphasis on architectural lighting. Turrell's portfolio includes an array of works he calls "Skyspaces," often vaguely gazebo-like structures with apertures in their ceilings intended to frame the sky as if it were a canvas. Most examples are in private ownership, the only one I have seen is "Dividing the Light" at Pomona College (Turrel's alma mater) in Claremont, California. I do recommend a visit.

There's something important to understand about the Skyspaces: the idea is more or less that the art is the sky, and the space is only there to structure your perception of it. Turrell has described them as naked-eye observatories. The naked-eye observatory has a long tradition, perhaps unsurprisingly, since it was the only kind of observatory until the development of the telescope. Stonehenge is a rather famous one (I am referring, of course, to the Stonehenge of Odessa, Texas. What else?). A lot of Turrell's work is like this, creating a space to structure your view of the world beyond it, and he is particularly interested in the sky as a subject. Stick a pin in that.

So back to the shopping mall. Turrell did a scattering of different projects for the Crystals (I am hereforward dropping "The Shops At" for sanity), including the monorail station. Now, I know you are thinking, what could possibly appeal to me more than a James Turrell monorail station? Why have I not up and moved to Las Vegas to devote my life to the preservation and interpretation of this remarkable artifact?

Well, because it kind of sucks, is why. Almost all of the work Turrell did at the Crystals feels badly hampered by the design of the larger building and the practical necessities of an upscale shopping mall. It's hard to produce that remarkable of a perceptual experience in a wide, crowded hallway. The monorail station doesn't even always have Turrell's lighting turned on, is my experience. What stands out more is the space I photographed, basically an awkward mezzanine floor that exists mainly to be a hallway to the monorail station. You can't help but feel that it was handed over to Turrell because they realized they'd made a mistake and there wasn't really anything else they could do with the square footage. If you're in Las Vegas I would go to the Crystals and take a look, because it is something, but it's a very long ways from a masterpiece. Pretty much every view of it is intermediated by escalators or food court signage, it's jarringly out of context, and exists in a broadly uncomfortable part of the building that is too far from the ground floor to appeal to shops. Instead, it hosts a traveling exhibition on Princess Diana. And if that's not dead mall energy, I don't know what is.

So why are we talking about this, besides that I get to gush a little about Turrell? Well, I really wanted to talk a little bit about land art, and I think these smaller Turrell works are a good inroads. Land art can be succinctly described as art that makes use of, or consists of, the landscape. One of the prototypical works of land art is the Spiral Jetty, built by Robert Smithson on the shores of the Great Salt Lake. It is what it says on the tin, a rock jetty that reaches out into the lake before turning in on itself in a tightening spiral. It was installed in 1970 and wore away with time, so for many years it was completely submerged below the water. But the water is now receding, and it is revealed once again, as a faint rock spiral in a dry plain. There is some active debate over whether or not it would be appropriate to make repairs. It is often an accepted part of land art that it will change over time due to natural processes. At the same time, the Spiral Jetty's exact fate (to be submerged and then later beached, as it were) was not foreseen and is largely a result of human disruption of the ecosystem. So you can make arguments either way.

In any case, it is an interesting aspect of the work (which merits a discussion that makes up more than half of the Wikipedia page) that it exists, today, mostly in the form of photographs. For much of its history it was entirely invisible, and more recently it has reemerged, but in a severely eroded state. If you are killing time in Utah on the wrong side of the lake, say because you have been at the nearby Golden Spike National Historic Park, you should go see it. Or perhaps not see it, as it no longer resembles Smithson's original creation. This is, ultimately, the fate of all land art, as it is the fate of the land. Think about that while you look at it.

One of the most famous works of landscape art is Walter De Maria's "The Lightning Field," in Catron County, New Mexico. The Lightning Field has something in common with the Spiral Jetty and several other prominent works of land art: the Dia Art Foundation. Dia was founded in New York City in the '70s with Schlumberger money, and had a rather explicit mission of funding art works that are particularly expensive. This focus on large projects and the timing of their heyday lead them naturally to land art. I would wager, just off the top of my head, that roughly half of the prominent works of land art in the United States are owned by Dia.

Land art really is big, and that can be its undoing. Turrell's Skyspace at Pomona ran over $2 million, and it's not even really land art, I'm just making a connection there in service of what I originally set out to talk about but still haven't gotten to. I'm not sure if the total original cost of The Lightning Field has been publicized, but it relied on grant funding from multiple art foundations and the state, and a refurbishment about a decade ago ran nearly half a million.

But, well, let's be real, it's not that unusual for original works of fine art to run into the millions, and that's especially true of sculpture which often requires fairly sophisticated fabrication and installation techniques. Land art at its best tends to rely on sophisticated construction techniques, as well. The Lightning Field, for example, required five months of extensive surveying by land and air. Part of this was to produce what we would now call a digital elevation map, in order to create the field's flat top despite the varying terrain.

I am sort of purposely not describing The Lightning Field here. It's not that it defies explanation, it's actually very easy to describe. In an article that De Maria wrote himself for "Art Forum" to describe the project, he says that "the sum of the facts does not constitute the work or determine its esthetics." This is sort of a pretentious thing for a sculptor to say before rattling off a bunch of large numbers, but he has a point: The Lightning Field is literally a bunch of poles stuck in the ground, which is easy to tell you, but gives you very little idea of what it is actually like.

This is a common and important aspect of land art. When I worked for Meow Wolf, we talked a lot about "immersive art," which is pretty much the term that has come to describe "whatever Meow Wolf is." If we allow ourselves some rose-colored glasses, most land art projects were a form of immersive art, an earlier form that seems to predate the forthright commercialization of the projects I got to work on, like Meow Wolf's Omega Mart. To be fair, the commercialization is part of the work, but it sure is on display. Immersive art means that "you have to be there," and "you have to be there" is a great opportunity for the real estate developer.

Ah, but this is indeed a rosy view of the past. There is another very interesting trait of prominent works of land art. Since I have done such a paltry job of describing The Lightning Field, go ahead and look up some photos. They're worth a thousand words, and so you'll get... several thousand words of information. You will quickly notice that there are very few photos of The Lightning Field, and some of them turn out to be crops or recolors of the others. The Dia Art Foundation, apparently according to Walter De Maria's wishes, prohibits photography, or even the presence of cameras.

I have never been to The Lightning Field. This is also a surprise, I'm in Catron County reasonably often. But it costs either $250 or $150 to visit depending on the season, and the larger issue I have encountered is that the visiting nights sell out immediately.

There is no public access to The Lightning Field. You have to make arrangements with Dia to stay the night at a cabin on the property. Lots of people do this and end up writing travelogues about how the experience changed them, and God knows that I probably will too, some day. But after reading enough of these travelogues you start to go a little mad. You are reading someone else's accounting of an experience---something that is fundamentally an experience, not a sight or a sound, not even space or light---that you have not had and probably never will. It's like when you meet someone and the main and only thing they have to talk about is their nomadic travels of Europe, but instead of the routine lifestyle of people with Silicon Valley salaries and few attachments, it is supposedly a great work of art. Like photographs of the spiral jetty, personal essays of The Lightning Fields describe something that hardly exists.

The magazine articles replace the art.

The charmingly-named journal "Art World Follies" featured an essay about The Lightning Fields by art historian John Beardsley. It is titled "Art and Authoritarianism." It's an exercise in self-control to not quote nearly the entire thing here, but it is available on JSTOR if you have just three pages worth of time to kill. Let me take just this, from the introduction:

...The directive posture assumed toward the viewer by De Maria and Dia suggests that both artist and patron lack confidence in either the quality of the work or the discernment of the viewer.

I can respect that Dia has certain practical concerns that encourage them to limit access to the site, such as preservation of both the artwork and the delicate desert ecosystem that it incorporates. But Beardsley points out that, like most landscape art, The Lightning Field is in a remote location. Distance and unimproved roads provide a natural limiting effect on visitation to these sites; I've been to the Spiral Jetty several times and seldom seen more than one other visitor around---fewer than are permitted to stay at The Lightning Field each night. And that's another Dia-owned site, although not one originally commissioned by them. They clearly do not apply such restrictive measures to everything.

It might be tempting to attribute it to finances, especially after seeing Beardsley complain about the required $30 donation, which has gone up by 500% at the least. I do think that Dia has some financial struggles, but no doubt they could raise more revenue by accepting more visitors to the site.

So, while it is tempting to blame Dia for their restrictive attitude, it's clear that Walter De Maria shares in the fault. Many of the restrictions are in place at his request; the whole notion that you can only see the artwork through a 24-hour stay in a remote cabin to which you were transported by Dia staff was apparently part of his vision.

Artist's vision or not, it is an affront to the public.

There are no doubt some regional politics at play. I cannot help but view Dia with skepticism. Dia is a high-society NYC institution with galleries in New York and, incongruously, several of its most prominent holdings in remote parts of Utah and New Mexico. That land art requires land is self-evident, but it also tends to require a degree of isolation. The most prominent and ambitious land art projects, even when conceptualized in the more populous East, tend to be actualized in the West. Here, land is our most important asset, and in places like Catron County, it often seems to be our only asset.

So you can see the appeal to land artists. But you can also see the indignation when those land artists claim the very land as their art, and keep it for themselves.

To say that The Lightning Field is a betrayal of Western values is probably a little over the top. Besides, it takes only a casual reference to the Bundys to show that those values are not universal. But it is fair to say that the kind of person who travels the west and is interested in land art is the type of person who is interested in the land itself, and holds it dearly. There is something unbearable about Walter De Maria, an artist from and in New York City, making his most famous work out of a slice of our desert and then narrowly dictating the terms on which we can see it.

The earth did not set forth one hundred thousand acres of lava across El Malpais and then establish an elaborate booking policy, except that you must put in the effort to cross such difficult terrain. The beautiful and unique Quebradas are remote but open to anyone willing to make the trip. De Maria seems to view his work as part of this tradition: "the land is not the setting for the work but a part of the work." And yet he apparently thinks of himself as being far above it.

Much of the beauty of the land is in the discovery. De Maria writes that "the sky-ground relationship is central to the work." Anyone who loves the desert can tell you that the sky-ground relationship is different everywhere you look, that it does not photograph well but must be experienced, that the most important and striking examples of it are found by chance, or by the dedication of long hours spent looking. Yet The Lightning Field, ironically, offers no such experience. It is intensely curated, guarded by Dia's many restrictions, relentlessly interpreted for its scant audience by the demands its now-dead creator makes of them. "Isolation is the essence of Land Art," he said.

Smithson built the Spiral Jetty and then walked away. It is technically owned by Dia but you would be hard-pressed to tell it apart from the public land that surrounds it. It is, nonetheless, truly isolated. At the lightning field, they make a show of leaving visitors on their own, but in a way that provides the exact opposite message: you are being allowed to glimpse something special, but only on its creator's terms, a creator who has made very certain that his presence will not be forgotten.

I shouldn't be too hard on the Dia foundation. Not only the Spiral Jetty, but also Nancy Holt's "Sun Tunnels" are owned by Dia and open to the public in the way typical of things found in the desert. Like the land itself, they leave it to the visitor to have an experience of their own.

Unfortunately, this demand for isolation that turns to isolationism has become too typical of land art.

We started, you might remember, with James Turrell. His work at the Crystals is open to the public in the most commercial sense possible, perhaps to its detriment. Even there, he has made a concession to isolationism. By far the best part of his multi-installation work there, titled "Akhob," was upstairs in the Luis Vuitton. Access required not only a reservation but passing muster with the high-end store's doorman. Unfortunately, it doesn't seem to have survived COVID: Luis Vuitton stopped advertising Akhob in 2021 and, today, its fate is unknown besides that there is no access. Like the submerged Spiral Jetty, we can experience it only by photos, unless perhaps some shift in the economy causes the waters to recede.

Turrell, as you might suspect, can be viewed as a land artist. The Skyspaces have a natural connection to the land art movement, and as Turrell became more ambitious, his projects became larger, taking on the scale of landforms. Since 1979, Turrell has been working on "Roden Crater." It is a cindercone near Flagstaff that may one day become the greatest of the Skyspaces, excavated and reconstructed as a naked-eye observatory.

It is a huge vision that has faced a great deal of struggle. Despite many announced opening dates, it remained "under construction," closed to the public for 45 years. More recently, a $10 million donation from Kanye West and a partnership program with Arizona State University have brought in renewed funding, but a "tentative opening date" of 2024 looks set to pass just like the last five. In the mean time, it is mainly ASU students (a few of which have been entitled to visit the site by ASU's partnership agreement) and celebrities that have been found worthy. Besides Kanye West, with Kim Kardashian as a +1, Drake recently used Roden Crater as an Instagram backdrop. Clearly, the mandatory donation to experience the isolation of Roden Crater is a great deal more than $250. Perhaps I shouldn't be so cynical, but the main benefactor of Roden Crater is the Dia Art Foundation, and plans call for a set of cabins by which visitors will experience it.

Massive land art projects have a tendency towards vaporware, but that's not to say that they never escape. Michael Heizer's "City" is in a similar vein to Roden Crater, also initiated in the 1970s, also blowing through its proposed opening dates, also admitting no visitors due to its incompleteness. But City made it out: in 2022, it was finally declared open. It is managed by the Triple Aught Foundation, which has apparently learned a thing or two from Dia.

Triple Aught Foundation, the 501(c)(3) that oversees and operates Michael Heizer's City, has complete discretion as to the acceptance of any visitor request. City is located on private property and only invited guests are permitted on the property. All other visitors will be denied access to the property. Invited guests must advise Triple Aught Foundation of any medical conditions. The sculpture City is a registered work, protected by federal copyright law. Triple Aught Foundation has a strict copyright enforcement policy regarding unauthorized photographing or filming of the work. No unauthorized reproductions, public display or distribution of copies of the work, in whole or in part are permitted. Anyone violating this policy will be immediately asked to leave.

Only six visitors are allowed per day, three days a week, for a maximum of three hours, weather permitting, for a fee of $150, from May to November. Reservations are available on a strictly first-come-first-served basis.

You, I can say with a fair degree of confidence, will only ever experience City in the form of the few photographs the Triple Aught Foundation has seen fit to release. The Lightning Field, Roden Crater, they are all submerged, not by the waters of the Great Salt Lake but by the inability of their creators to let land art be like the land itself: unrestricted, unconfined, unassuming.

In the essay collection "LAND/ART New Mexico," curator Lucy Lippard writes that "I've come to the reluctant conclusion that Land Art is for city people."

I live between inhabited and mostly uninhabited areas---which makes this essay a kind of NIMBY rant: not in my backyard, not on my back forty. Given the fact that I have spent my life writing about art (sometimes Land Art), and ranting about the importance of public art, this sounds like a kind of betrayal. But it's hard to imagine what kind of art would work here, at the edge of a tiny village in north central New Mexico, looking out across a highway to private ranchlands and distant mountains. When I was a citydweller, I might have welcomed the sight of some visual extravagance, or oddity, or subtle highlights to my daily surroundings. But the fact remains that even semi-rural New Mexico is hard to improve upon.

Perhaps that's the problem, perhaps land art as a movement is fundamentally at odds with appreciation of the land itself. We might view Turrell, Smithson, De Maria, Heizer as entitled for thinking that the land needed their help. It is already art, and it always has been.

But we are humans, and we have always been inclined to interpret the land in the context of its impact on us, and our impact on it. So often that impact is happenstance, and more often for the worse than for the better. There must be some room to manipulate the land entirely by intent, in service of aesthetics and meaning rather than commercial exploitation. Indeed, the Land Art movement viewed itself in part as an anti-commercial backlash to the museums and galleries that held, and confined, so many forms of fine art. And yet, some of the greatest works of the movement are displayed in conditions more restrictive, more removed from the nature of the land, than the upper floor of a Las Vegas Luis Vuitton.

"Isolation is the essence of Land Art." I am inclined to agree. But isolation is not made, it is found. De Maria went to Catron County to seek it out, but somehow left thinking that he had created it. This is not New York City, you do not find space to think and experience behind a rope stanchion and a guest list. The land is already there, and land artists should trust their audience to experience it.

According to a traditional system of classification, "high frequency" or HF refers to the radio spectrum between 3 and 30 MHz. The label now seems anachronistic, as HF is among the lowest ranges of radio frequencies that see regular use. This setting of the goalposts in the early days of radio technology means that modern communications standards like 5G are pushing major applications into the EHF or "extremely high frequency" band. The frontiers of basic radio technology now lie in the terahertz range, where the demarcation between radio waves and light is blurred and the known techniques for both only partially apply. HF, by contrast, is ancient technology. HF emissions can be generated by simple, brute-force means. Ironically, this makes HF a bit difficult: the incredible miniaturization and energy efficiency of modern electronics makes HF radio hard to receive and transmit in a reasonable footprint, one of several reasons that HF radio sees little consumer use.

Let's briefly consider the propagation characteristics of the HF band, which are its most remarkable property. HF frequencies have long enough wavelengths that they can reflect and refract in the earth's atmosphere. Somewhat like the skin effect observed with AC electricity or surface tension in liquids, HF emissions have a tendency to bend their path to follow dielectric boundaries. All of these effects are mercurial and difficult to predict; the reliability of sky or ground wave HF propagation can depend on the time of day, the weather, the number of sunspots. All of this makes HF radio a bit of a pain in the ass, but it can be worth it to achieve a feat that higher radio bands cannot: propagation beyond the line of sight.

As a rule of thumb, radio emissions in the VHF band and above behave much like light. Many materials are more transparent to RF than they are to light, but still, most modern radio communications will not propagate beyond the horizon, over a hill, or even past a sturdy building. An HF transmission, by contrast, can be received around the globe in good conditions.

HF radio thus appeals mostly to users that desire long-range communications with minimal infrastructure, and that have the sophistication (of operating practice or technology) to handle the vagaries of HF. The usual suspects are militaries, who fall more on the side of technical sophistication by using computer-driven link establishment systems, and amateurs, who enjoy the complexity of the operating practice. Other major HF radio applications include international broadcasting (often of either national or religious propaganda), intelligence and law enforcement agencies, and communication with ships and aircraft at sea. Nearly all of these applications are giving way to satellite communications, but the relative simplicity and low cost of maintaining HF equipment, and its independence from vulnerable satellites, give it enduring appeal to government users.

Let's consider a few interesting examples of these government applications, although they are not the focus of this article. Military radio is a curious combination of secretive and well-documented. The US military possesses multiple major radio systems, but is theoretically consolidating onto the Joint Tactical Radio System (JTRS). JTRS is an infamous military acquisition debacle and has run vastly overbudget and behind schedule while failing to deliver on many of its promises, but is now in daily use and consists of a diverse lineup of software-defined radios that can operate a variety of modes across many bands. In other words, it is a conceptually fairly simple system that hides enormous, combinatorial complexity in its software. This is modern military tradition.

Non-military systems are more recognizable to those without a background in 1990s military technology programs. One of the largest civilian government HF radio systems is COTHEN, the Customs Over-The-Horizon Enforcement Network. COTHEN was constructed, as the name suggests, by Customs and Border Patrol. It is now widely used by other federal law enforcement agencies with remote field operations, as well as by the military when cooperating with law enforcement agencies. The principle day-to-day business on COTHEN is drug interdiction by CBP and the Coast Guard. COTHEN employs second-generation Automatic Link Establishment (ALE), a popular system developed by the military to allow HF radios to "discover" working frequencies between two locations. Besides various mobile radios, COTHEN has fixed radio sites throughout the country, including one in the high plains east of Albuquerque.

There are many smaller HF radio systems operated by executive agencies, mostly for continuity of operations. Many are integrated into SHARES, a joint system sponsored by the Department of Homeland Security. For a more specific example, the Department of Energy has installed HF radio equipment at many of its facilities including national laboratories, infrastructure sites, and the historic AEC campus at Germantown. The DoE operates its own ALE network, but most (and very probably all) of its sites are also seconded to SHARES. DoE also makes use of HF radio for communications with Office of Secure Transportation vehicles.

We understand that there are government applications of HF radio, that's probably no surprise to anyone. But what about commercial applications? The complexity of HF operations, the size of the antennas, and ready availability of other communications options (like the internet) limit the appeal of HF to business users. Still, there must be some? Obtaining a license to use HF radio is a reasonably simple procedure, the FCC has allocated a number of HF ranges to the IG (industrial/business pool) service. As is usual in the IG service, frequency allocations are not exclusive and there may be other users. Interestingly, FCC regulations place a significant break point at 25MHz: 25MHz to 30MHz is technically HF, but FCC rules don't really differentiate between these frequencies and the more common business pool uses in VHF. Below 25MHz, though, special rules apply.

47 CFR 90.266(b):

Only in the following circumstances will authority be extended to stations to operate on the frequencies below 25 MHz:

(1) To provide communications circuits to support operations which are highly important to the national interest and where other means of telecommunication are unavailable;

(2) To provide standby and/or backup communications circuits to regular domestic communications circuits which have been disrupted by disasters and/or emergencies.

As is often the case in federal regulation, there are some additional terms that require a little closer reading. 47 CFR 90.35, which governs the Industrial/Business Pool radio service, has a table of possible frequency allocations. For the range below 25MHz, the table specifies the following restriction, in part:

(c)(i) Only entities engaged in the following activities are eligible to use this spectrum, and then only in accordance with s 90.266:

(A) Prospecting for petroleum, natural gas or petroleum products;

(B) Distribution of electric power or the distribution by pipeline of fuels or water;

(C) Exploration, its support services, and the repair of pipelines; or

(D) The repair of telecommunications circuits.

Available bandwidth at these low frequencies is already rather constrained by the allocations, but the situation is worse than it appears. HF propagation behavior mean that radio operators seldom have their choice of anywhere in the HF spectrum; usually there are only limited "windows" in which propagation is good. It would not take very many users to create congestion in this valuable long-range spectrum, so licenses are limited to users without other options.

Unsurprisingly, special restrictions below 25MHz mean that frequencies just above 25MHz are quite popular. For example, from 25MHz-25.5MHz you can find a veritable who's-who of the petroleum industry, who use HF radio for communications with off-shore oil platforms. You also find some other peculiar licenses in this range. For example, Ritron is a manufacturer of radio equipment and holds a license for the use of 25-50MHz for the purpose of demonstrating and testing that equipment. This 25-50MHz range is more or less "low band VHF," which was formerly in reasonably common business use, but is now becoming rare. Online discussion leads me to believe that it may not be possible (or is at least very difficult) to obtain IG licenses for the low band today, but some people that had them continue to renew them.

When I say "people" here, I mean it. One of the more common types of licensee for this range is... people, filling out their application with a title of "self" or "person." Although it is public record I am hesitant to give names or addresses of these people, but it explains a lot about them that you can almost always (invariably, from my spot-checking) find an amateur radio license for the same person. I have known some amateur radio operators that obtained their General Radiotelephone Operator license, a broad license for maintaining certain types of commercial radio equipment, more or less for the hell of it. I suppose obtaining a low-band IG license is similar, and adds some more bands to your potential operations.

Let's limit our consideration further, then, to the rarefied frequencies below 25MHz. Besides special justification, applicants for these frequencies are limited to certain emission types (generally narrow 2.8kHz emissions), must use equipment capable of tuning across the entire range, must submit their written communications plan, and are prohibited from testing or exercises that exceed seven hours per week. There are relatively few such licenses, the ULS returns just 61.

I totaled up the licenses by user type. For companies that provide radio services, I categorized them with the industry they normally serve, except when that industry was emergency communications itself. This sometimes required a bit of a subjective call, as we'll see when looking at some specific licenses. But here are license counts by type:

• Telecom providers: 23
• Electrical utilities: 15
• Emergency communications/disaster relief contractors: 11
• Petroleum: 7
• Railroads: 2
• Ranches: 1
• Weird: 1

Let's discuss these a bit by category. Telecom providers are a fairly obvious user group, as are electrical utilities. Both types of organizations operate infrastructure over large areas and will be expected to begin recovery quickly after a natural disaster or another event that disrupts conventional communications infrastructure. These licensees include AT&T, with the single largest license count (9), Pacific Bell dba AT&T California at the second largest (4), down to smaller entities like the Grand River Dam Authority. A notable selection is National Grid USA Service Company, which despite its generic and almost sketchy name holds a fixed location license for the Nine Mile Point Atomic Power Station among other power plants.

Verizon New York Inc. holds a license for a number of fixed locations including an inconspicuous brick building (reminiscent in its design of early telephone infrastructure but likely today a remote exchange) in Philmont, NY and a mountaintop site near Schenectady. The license seems to cover multiple telephone exchanges, only some of which have apparent HF antennas... some of the locations listed may be historic, nearly all commercial HF licenses are old, with renewal histories stretching back to the beginning of online records in 2001. Besides, virtually all of these licenses include either a nationwide mobile or nationwide temporary fixed location, making the listed locations less important than they might otherwise be.

Most licenses are also nonspecific as to frequency. It is the nature of HF radio that any given frequency will not reliably provide good propagation. The frequency lists on commercial HF licenses routinely stretch on for multiple pages of 20 ranges each, giving operators ample choice. Besides, the CFR requires these licensees to be frequency-agile across the band, in part because the FCC may require them to stop using a given frequency at any time.

The next largest category of licensees are companies that provide communications services to relatively nonspecific customers. Some, such as L3Harris, are principally in the defense industry but likely also provide services to infrastructure providers and regional governments. A few are tiny, like Hazard Zone Technology LLC, which does business from a residential address. Judging by forum discussions of the FCC's approach to HF licensing, it is possible that some of these smaller licensees are largely fictitious entities created to entice the FCC to issue a sub-25MHz license despite the restriction to emergency communications. While not conclusive, the fact that the same individual holds both a general radiotelephone license and an amateur extra license with a vanity call sign is certainly suggestive of a certain personality. There are several such examples, not an inconsiderable portion of the 61 total.

Others are familiar names in an unfamiliar context. Cisco Systems holds a license that lists their San Jose and Research Triangle Park campuses, besides a nationwide mobile location. This license is likely intended to cover the use of HF equipment as a backup point-to-point link for customers of fully managed industrial communications services, but it's hard to say exactly. They have apparently demonstrated a mobile communications trailer for emergency response coordination at conferences.

Just those three categories get us into a long tail. A railroad, CSX, holds two licenses. A ranch holds one; it's unclear how the ranch would qualify under the requirements but they may have held the license long enough to be grandfathered. The justification listed on the license is simply farm operations, not a 90.266 justification statement as found on most of these. The sub-25MHz restrictions have applied since 1983, making it difficult but certainly not impossible to hold a grandfathered license. One license is held by a missionary transportation group, they may have also held the license since before 1983 and list a location in Guam. They likely use HF to communicate with facilities on outlying islands.

There is exactly one license that I have described as "Weird." It was issued to the Bran Ferren Corporation in 2000. Bran Ferren is a bit of a character, a former executive at Walt Disney Imagineering who apparently has a side business of building off-road vehicles. This helps explain the justification statement, "MANUFACTURER AND DEVELOPER OF ALL TERRAIN COMMUNICATIONS VEHICLE." I would like to know exactly which vehicle this relates to, but the Bran Ferren Corporation keeps a low profile as compared to Ferren's main business venture, Applied Minds. Bran Ferren Corporation holds an active USDOT motor carrier number, but has had zero vehicles inspected in the last two years. That may not mean anything, they are listed as a private (not for hire) carrier and inspections may not be required. It's all a bit of a head-scratcher.

These 61 licenses for sub-25MHz commercial radio represent only a tiny fraction of the activity in the HF band. Besides amateur radio, a long list of government users are authorized to use HF radio by the NTIA, rather than FCC. Indeed, the NTIA master file includes at least hundreds of entries under 25MHz, more detail will have to wait on my finally finishing the parser for the more recent format they have used for FOIA disclosures (universal rule: if you spend many hours writing tools to parse a PDF export of a relational database back into a relational database, they will change the format of the PDF).

High-frequency traders have created renewed interest in HF radio, because of its low latency for global communications and the increasing ease of implementing automatic link negotiation with SDRs. Sniper in Mahwah has some well-known writing on this topic. To date, the FCC has authorized this activity only on an experimental basis. In 2023, a group called the Shortwave Modernization Coalition (SMC) and consisting of a group of HFT firms submitted a petition for rulemaking to allow regular use of the 2-25MHz range. The petition opened docket RM-11953, which remains open. Most recently, the FCC conducted a series of meetings between the SMC and federal spectrum users to discuss the impacts of such a new radio service.

Various documents filed by SMC do contain interesting details. SMC members are operating at least fourteen experimental HF sites, with one operating since 2016 and most since 2020. The median transmit power (EIRP) is 21.5kW, and the experimental licenses authorized 5.06-30MHz while SMC members conducted most activity between 6.675 and 21MHz. The ARRL and various amateur radio operators have filed comments opposing the change, a competing HFT radio operator has discussed a counter-proposal that would impose performance obligations on commercial HF radio operators ("use it or lose it" rules). The FCC has not yet produced a Notice of Proposed Rulemaking, the next step in the process. They are not obligated to, and it is possible the proposal will not reach that stage. For its part, the federal government itself has weighed in (in the form of comment from the NTIA), and has requested that the FCC either extensively study how such commercial HF use would mitigate interference with federal users, or exclude the use of any frequencies allocated to federal use. This includes certain bands like the Radio Astronomy service allocation at 13MHz that exist as a matter of federal policy due to foreign agreements. For my own part, I am skeptical that the FCC will act on the SMC petition unless the scope of the SMC's proposed use is reduced.

Finally, it is worth noting that these commercial HF licenses do not represent the full extent of private industry use of HF radio for continuity of operations. Some critical infrastructure operators have been sponsored by federal agencies as operators of SHARES stations. However, SHARES documents suggest that there are relatively few of these stations (around 100), suggesting that they almost completely overlap with commercial HF licenses. State and municipal governments also operate HF radio stations, which are generally licensed under public safety radio services. A curious exception is the City of Lafayette, Louisiana, which holds an IG commercial license for HF frequencies. I suspect that license is actually for the use of the publicly-owned Lafayette Utilities System (it lists electrical distribution as justification), and was issued in the name of the City of Lafayette for bureaucratic reasons.

We currently find ourselves in something of a series, working our way from private lines to large private line systems like the four-wire private-line national warning system. Let's continue to build on the concept of the private line into large corporate systems.

In principle, a large organization in want of a private telephone system could build one out of a set of private lines and switches, such as under a Centrex CU (Customer Unit) arrangement. And this did happen: one common type of private line was the tie line, a private line used to link two switches together (which could be PABXs or Centrex) so that users could call from one to the other without using a conventional dial telephone line. This could save money, if usage of the tie line was heavy enough and especially if the two switches were far enough part that a standard call would be long-distance.

Consider a corporation with two large offices, each with a PABX. If they are in different local calling areas, calls between them placed by dial line would be long distance. If employees at the two offices call each other often, the long distance bills would add up to more than the fixed monthly cost of a tie line from one office to the other. There are a few different ways to solve this problem, such as getting WATS (wide area telephone service) at one or both offices, but it illustrates the general idea that getting a fixed private line can sometimes be a cost-saving measure compared to placing a lot of calls over standard dial service.

But what about a bigger organization, with many offices? You can imagine that getting a huge number of tie lines between different offices, planning where those tie lines should be located and how many were needed on each link, could become a feat of traffic engineering on par with the telephone company's own work. It might be easier to just pay the telephone company to work it out, and indeed, that's what large organizations often did.

So let's say the telephone company meets this request by designing a scheme of tie lines and Centrex exchanges. It's not so far off the mark to say that this describes AUTOVON. AUTOVON was a complete system of tandem exchanges and at least semi-private telephone lines provisioned by AT&T [1] for use by the military. The problem with this arrangement is that it is very expensive: the customer is paying the telephone companies to purchase, install, and maintain a huge amount of hardware, just for the customer's private use.

Now, comparing AUTOVON on a price basis is both difficult and unfair. Difficult because AUTOVON was paid for in a somewhat complex way, by the military paying a central, cumulative rate for the entire system and then performing cost recovery from individual user agencies and installations using a non-trivial cost assignment calculation. It is also often said that the Bell System themselves did not recover the full cost of AUTOVON from the military and that it was, to some extent, subsidized by other telephone services.

And I say unfair, because AUTOVON was more than just a private telephone network. It was a hardened private telephone network, with four-wire service and a precedence capability that required the development of novel equipment. It wasn't really expected to save money compared to the public telephone system, because it was acquired in order to provide capabilities that the public telephone system did not.

Still, we can safely say that AUTOVON was expensive. A 1979 study by the Defense Communications Agency, responsible for AUTOVON cost recovery at that time, comes out to an impressive total of $255,492,000 in AUTOVON operating costs for FY 1978. By way of example, the report puts the monthly service cost of a two-way capability from CONUS to Europe with priority precedence at $1,182. Obviously this example case is one of the most expensive, but I still shudder to imagine a monthly phone bill of over eleven hundred dollars in 1979 money. The military was willing to swing the stiff cost of AUTOVON because, first, it was the military and they were willing to swing the stiff cost of many things, and second because AUTOVON's military capabilities would be very expensive to build by any means. It was the Cold War, after all, and it could be said that outspending the Soviet Union was a military objective.

The situation was rather different when it came to non-military communications. The civilian federal government ran up some enormous telephone bills between its many offices, and initially considered purchasing an AUTOVON-like system to serve as a private network between federal offices. The concept simply wasn't cost effective, it likely would have increased the cost of federal telephone calls overall. The Federal Telecommunications System or FTS would eventually come to be, but not in the form of a private switched system. It is, after all, intuitive that cost savings would not come from installing a great deal of dedicated hardware. Rather, the Bell System would have to find a way to serve these large institutional customers with less investment. And that was the Common Control Switching Arrangement, or CCSA.

It is very tempting to draw an analogy between the CCSA and virtualization in contemporary computing, but it is probably more accurate to draw an analogy between the CCSA and LPARs in IBM hardware, or even more aptly, to virtualization's early precedent in the limited subdivision capability of Babbage's difference engine. Let's stop indulging the temptation and explain it more directly: a CCSA is created by configuring existing telephone switches to treat a subset of their lines as part of a separate network.

The technical details by which this was achieved varied significantly by the switch. CCSAs were introduced in the early 1960s and could be configured on the #5 crossbar exchange, where "configuration" consisted of strapping or jumpering certain components of the switch to operate independently of the others. CCSAs continued just about to the modern era, where configuration became a matter of selecting the appropriate lines in the business office system that generates configurations for computer-controlled exchanges.

I think that it's most interesting to examine the CCSA by way of example---by looking at a specific, real CCSA. BSP 310-200-007 I2 (1966) conveniently provides a directory of the code numbers that were used to identify CCSAs within the telephone system. Number 02 is FTS, the Federal Telecommunications System. I didn't bring it up without reason, the concept of the CCSA was developed in large part in order to bring the cost of FTS under control. We can ponder what happened to number 01, but I'm guessing that AT&T used that code for testing and thus reserved it, or maybe even to identify the public telephone system.

One could think of the normal, public telephone system as just another CCSA, although as I understand it this was not the nature of the actual implementation. Another appealing analogy for the CCSA is the VLAN, we could think of these CCSA network numbers as VLAN tags. In this analogy, the public telephone system is the PVLAN, sometimes called 0 or sometimes called 1 at the whim of vendors. If you are familiar with VLANs, that somewhat illuminates why I say that the public telephone network is not just another CCSA: it is the "untagged" network into which equipment not capable of CCSAs and lines not attached to a CCSA are presumed to exist. Anyway, that's all besides the point, what other CCSAs existed?

04, General Electric. 05, New York Central (railroad), 06 Lockheed, 07 State of California, 08 AUTOVON (used to facilitate expansions of AUTOVON over non-AUTOVON telephone infrastructure, as a more cost effective way to provide AUTOVON lines at smaller installations), 09 American Airlines, 10 Boeing, 11 Westinghouse, 12 Western Electric, 13 IBM, 14 North American Aviation. That's the complete list as of 1966, and while short, it is a who's-who of the industrial giants of the post-war United States. Plus the State of California. Most state governments used large Centrex-and-WATS arrangements, but some combination of the large size of California and GTE's different approach to the network steered them in the CCSA direction.

Of these CCSAs, I will focus on General Electric. There are two reasons: first, GE had an early and large CCSA---the largest CCSA outside of the federal government, at the time. Second, I was an intern at a failing GE business in a large, half-abandoned corporate campus [2] during the summer that the last vestige of the GE Switched Services Network, as AT&T called it, was retired. Among my scattershot duties was working on the decommissioning of the campus's Nortel PABX in favor of Cisco UCM. GE SSN would go with it, replaced by IP trunking between UCM sites.

In 1963, the GE SSN spanned fifteen central offices ranging from New York to Los Angeles, all #5 crossbars. It was intended to provide voice as well as data at 1200bps. Unlike some (mostly federal) CCSAs, it was designed to provide standard two-wire dial service only, without station-to-station four-wire connections or call precedence. In other words, it was a standard telephone network, but intended to make calls between GE offices more reliable and less costly than calls over the long-distance telephone network.

One of the complications of the GE SSN, and of CCSAs in general, is the diversity of telephone equipment in use across the different corporate offices. The GE of 1963 had Centrex service, step-by-step PABXs, crossbar PABXs, key systems, and manually operated PBXs. All of these were integrated into a 7-digit dialing scheme for GE SSN, with the NNX prefix (different from NXX used in the public telephone network by prohibiting a 0 or 1 in the second digit position) identifying a location on the network such as a PBX, and the four-digit subscriber number generally being the telephone's local extension, padded with arbitrary digits as needed to be four digits long. Of course, the details were less tidy, with smaller locations sharing prefixes and some locations acting almost like toll stations with single telephones on the GE SSN and selecting it by key.

In general, though, at PABX-served locations, extension users had a choice as to how to place their call. The configuration wasn't the same at every office, but the recommended practice was to use a 9 prefix (or "exit code") to dial on the public telephone network, and an 8 prefix to dial on the GE SSN. Most PABXs have some version of this capability: specific trunks can be selected for outgoing calls based on the dialing prefix.

At locations with manual exchanges and locations without compatible PABXs, GE SSN calls had to be placed with the assistance of the local PBX operator. Still other locations used a small PABX connected via tie line to a larger PABX at a larger office, in this case the dialing prefix "18" was recommended to first dial a trunk from the satellite PABX to the main PABX, and second from the main PABX to the telephone exchange providing GE SSN service.

Indeed, let's reflect a bit on the wiring scheme involved.

CCSAs were served by Offices, like the fifteen I mentioned for GE. FTS and AUTOVON had more offices (AUTOVON's CCSA office list tellingly includes the proper AUTOVON exchanges), but most CCSAs had fewer, sometimes only a handful. Between these offices, trunk capacity could be shared with normal telephone traffic, giving CCSAs a significant cost advantage. Individual phones (or PABXs or etc) on CCSAs needed to be connected to an actual CCSA office, though, in order to have access to the CCSA at all. This made practical CCSAs sort of a hybrid situation.

A given GE office might have lines running directly to the serving office, for example in New York City where there were two offices on the GE SSN. Offices that weren't near a 5XB included in the scheme, though, would need to somehow be connected up to one. The BSPs are not replete with details, but presumably this was done using the fairly conventional foreign exchange service.

I think I have mentioned this before but I will provide a very short summary. In the VoIP industry, it is extremely common to identify the "ends" of a telephone subscriber loop using the terms FXO and FXS, for Foreign Exchange Office and Foreign Exchange Station. Confusingly, the terms refer to where a given connection "goes" rather than where it "is," with the result that a Foreign Exchange Office connection is what you would plug into a phone, and a Foreign Exchange Station connection is what you would plug into a device like an ATA---something that provides talk battery, ringing, etc., the traditional role of the telephone office.

So the "Office" and "Station" part of those terms makes sense, besides the fact that they are arguably the opposite way around from what you would first think. But what about the Foreign Exchange? Well, these terms predate VoIP by decades, and were originally used to identify the ends of a foreign exchange service.

Foreign exchange was a specific type of private line that allowed a phone to be connected to a different central office from the one that physically served it. There are different reasons that this was useful, but a common one had to do with long-distance rates and suburban areas: it was historically common in large metro areas that suburbs could call into the city at local rates, but suburbs could not call into other suburbs at local rate... that would be a long distance call. You can see that this provides a bit of an economic advantage to city phones. So if you are, say, a plumber with your shop located in a suburb, you might pine after a big-city telephone line that would allow more of your prospective customers to call you for free. Foreign exchange could solve that problem.

When you ordered foreign exchange service, the telco connected your phone line, at the distribution frame of your local central office, directly to a private line. The private line went to a central office in the Big City, where it was connected at the distribution frame to a local line served by the switch. In practice there were complications and details to how this was set up, but this description gives you the idea: your telephone was now connected to the switch in a different central office from the one your local loop was actually connected to.

Foreign exchange service was expensive, because it took up private line capacity, so various combinations of WATS, InWATS, zenith numbers, toll-free numbers, etc. have pretty much replaced it. Some of the terminology got stuck in our modern telephone parlance, though. FXO and FXS were designations used by the telco to keep track of which ends of the private line needed to connect to what equipment. Why was I talking about this, though? Oh, right, because large CCSAs in practice also relied on what was basically foreign exchange service in order to connect outlying locations to CCSA offices.

It's a good thing, here, that most of these GE offices had PABXs. This limited the number of outside lines that needed to actually go to a CCSA office. What I am calling foreign exchange lines could also be viewed as tie lines, not really serving phones but providing trunks from PABXs to CCSA offices.

The nature of the CCSA is pretty much in its name: it is a switched private line service, but it makes use of common control equipment to minimize costs. I have tried to make terminology a little simpler here, but I have kept saying "GE SSN." Switched Services Network, or SSN, seems to have been a term used by the Bell System to refer to any private switched network. A CCSA was one of the ways of implementing an SSN, and seems to have been the most common throughout telephone history. There were not many truly private switched systems. AUTOVON could be considered an example, although it had requirements above and beyond typical telephone service. That leaves the FAA as a purer example, as the FAA used a significant number of private line services for both switched and unswitched communications between air traffic control sites and equipment.

Incidentally, a precursor to AUTOVON was called SCAN, the Switched Circuit Automatic Network. SCAN was a US Army four-wire system, because four-wire service was required for the cryptographic equipment of the era to function. AUTOVON seems to have inherited its four-wire nature directly from SCAN. Skimming through a telephone tariff almost always turns up some interesting details, one of them being that a few state telephone tariffs describe Switched Service Networks as being private line service based on either CCSA or SCAN. Given that these same 2024 telephone tariffs define SCAN as a federal government service for secure communications, this definition (and the presence of an entry for SCAN at all) seems to be purely a holdover from some fifty-year-old tariff documents. It does go to show that non-common control switched services networks were uncommon enough that telcos viewed AUTOVON as the odd exception to the CCSA rule.

I am trying not to get too tied up in the history of AUTOVON, because it is easily its whole own article. I do think it is fair to say that the CCSA emerged largely as a response to the high price of AUTOVON, as building FTS based on the pattern of AUTOVON was deemed completely unrealistic on a cost basis. FTS launched in 1963, not far behind AUTOVON at all, but consisted of CCSA service for long-distance calling with PABXs in government offices furnished under lease agreements. Once the heavy lifting had been done for FTS, it was natural to extend CCSA as an offering to large private companies.

FTS also holds some of the seeds of the Bell System's undoing. High costs and lackluster service plagued FTS in its early years. During the 1970s the General Service Administration, which was responsible for FTS, decided to introduce a competitive bidding process for long-distance capacity. That lead to companies like Western Union and MCI joining the network, and the introduction of least-cost routing to select the carrier. These ideas would also spread from government to private industry, helping to set up the industry-wide tensions that culminated in the 1982 breakup of AT&T.

A common term in the telephone industry is "Universal Service." In the modern world, universal service is understood to be the goal of providing telephone service to all customers. The Universal Service Fund, for example, levies a fee on telephone lines to subsidize the provision of telephone service to those who would otherwise be unable to afford it. This is a recent invention. 1960s documentation on CCSAs makes repeated reference to Universal Service, a very different form of the concept championed by AT&T more or less until its breakup: that everyone would be served by one, unified telephone system, the Bell System. During the early days of the telephone, before AT&T's monopoly was cemented, Universal Service was a rallying cry against competing telephone companies, whose independent networks interfered with the ability of any telephone to call any other. In the mid-century, it became a term somewhat like public switched telephone network (PSTN). CCSAs were capable of Universal Service, where desired, on a somewhat limited basis, in that CCSA exchanges could be configured to allow calling outside of the CCSA, into the public system.

It is amusing, then, that AT&T was so willing to abandon the ideal of universal service when their customers offered to pay for it. But that's business, and in the mid-century, AT&T was one of the biggest businesses in the world. CCSAs show us some of the ups and downs of the age of the telephone monopoly: CCSAs were an innovative concept that was rapidly developed and delivered, first to the government and then to private customers, over a span of just a few years. They were also frightfully expensive, and offered a new level of lock-in that kept customers from using competitive carriers.

It's hard to find any contemporary information about GE's private telephone system. It has almost entirely vanished from history, except in the form documented by the early BSPs. I don't even quite remember what it was called when I was at GE, I think it might have had "star" in the name. There was a printed directory to help you figure out the correct office code and means of transforming an extension in order to dial over the system. I don't think the copy I saw was recent, I'm not sure if a recent copy even existed. The system was barely used at all. It was replaced by a common modern arrangement, least cost routing with IP trunks.

When calling another GE office, the Cisco Call Manager installation would connect the call over the data network to the other office's Call Manager. Very practical, very easy to use, kind of boring.

[1] Even prior to divestiture, the practical construction and operation of these telephone systems was split between AT&T Long Lines and the telephone operating companies. Many, but not all, of these were subsidiaries of AT&T. In the case of AUTOVON, for example, we must consider that non-AT&T subsidiary GTE built and operated part of the system. I'm trying not to get bogged down in this complexity, but I'm also trying not to keep writing "AT&T" when referring to work done by multiple companies, some of them independent. Please do me the kindness of understanding that when I use terms like "the telephone company" or even "the Bell system" I am trying to encompass all of the parties, AT&T, Bell Operating Companies, independents, etc. that were involved in this work. The term I use may not be exactly correct.

[2] It was GE Intelligent Platforms, and the beautiful but poorly maintained corporate estate in Charlottesville, itself the remains of a failed joint venture, might have been a more suitable exterior for Severance than Bell Labs Holmdel.

Previously on Deep Space Nine, we discussed the extensive and variable products that AT&T and telephone operating companies sold as private lines. One of the interesting properties of private line systems is that they can be ordered as four-wire. Internally, the telephone network handles calls as four-wire with separate talk and listen pairs (or at least, it did before digitization). For cost reasons, though, service to individual customers is virtually always two-wire, with talk and listen combined onto a single pair via hybrid transformers. Four-wire private lines are just about the only exception.

Why? Well, one of the major advantages of four-wire service to the telephone instrument is that it avoids the echo and sidetone that normally occur within the hybrid transformers. On a call between two telephones, this effect is acceptable and even desirable. In conference systems, though, with many phones attached, echo accumulates until the line is almost unusable. Prior to the introduction of DSP technology to "clean up" the audio, multiparty conferences were a lot more limited than we take for granted today... except for the four-wire private line systems specifically built for large conference calls. The most notable of these is the National Warning System, or NAWAS, operated by AT&T for the Federal Emergency Management Agency (FEMA).

FEMA has an interesting history. It is most directly a product of the Department of Housing and Urban Development, where was originally established in 1973 with the responsibility for coordinating reconstruction after natural disasters. Over time, a series of federal reorganizations expanded FEMA and added additional roles. Most notably, in 1979 President Jimmy Carter instituted a major reorganization of federal emergency agencies that dissolved the Office of Civil Defense, made FEMA an independent agency, and placed all civil defense responsibilities within FEMA. As a result, part of FEMA is a direct descendent of the civil defense efforts at the peak of the Cold War. FEMA operates the government relocation bunker at Mt. Weather, for example, and by the same coin is responsible for the dissemination of attack warnings to the contiguous United States.

The origins of NAWAS can be traced back to the Civil Defense Warning System (CDWS), often known as the "Bell and Lights System," which was introduced in the 1960s and is itself an extension of some earlier precedents. The various iterations and renaming of NAWAS make it a little bit difficult to trace its history exactly. Wikipedia says that NAWAS was formed in 1978, a reasonable claim given that FEMA organized around that same time. But NAWAS cannot have been new in '78: AT&T published a BSP covering the "OCD NAWAS," prior to FEMA's existence, a full decade earlier in 1968. Indeed, NAWAS and the CDWS or "Bell and Lights" must have operated in parallel, as both had BSPs issued that same year.

That's not actually that surprising: one of the reasons that the nation's emergency communications networks kept being replaced is because the requirements kept changing. CDWS was designed primarily as an automated or machine-to-machine system, capable of activating air raid sirens and sounding local alarms automatically. During the height of the Cold War this was a good fit for the intent. The emergency scenario was nuclear attack, and a warning would need to be disseminated as rapidly as possible for optimum lifesaving effect. The Office of Civil Defense once targeted a 30-second timeline from declaration of an alert to the American public becoming aware.

Even as CDWS was put into service, though, OCD was aware that a more extensive communications capability would be required to distribute information on attack outcomes, evacuation and recovery efforts, and to enable continuity of government even in a possible scenario of devolution of control to local emergency response authorities. That need would be served by NAWAS: not an alarm system, but a voice communications system, ready for two-way use on state, regional, and national scales.

Over time, FEMA has shifted its emergency alert programs away from nuclear conflict and towards the "All-Hazards" model, in which the scope of the systems is interpreted broadly and the primary use tends to be natural disaster and weather alerts. The "All-Hazards" concept came about mostly because of a sense that FEMA and the National Weather Service were pointlessly duplicating capabilities; so because of All-Hazards thinking, FEMA NAWAS distributes critical weather alerts and NWS's weather radio network distributes FEMA alerts. NAWAS has thus changed its identity from a wartime system to a more general emergency management system.

Let's see how NAWAS actually works. NAWAS is, at its core, a network of interconnected four-wire conference circuits. A national line connects the Warning Centers to the Regional Warning Centers, eight regional loops connect the Regional Warning Centers to states and other federal warning points, and 48 state circuits connect warning points within each state. Each of these circuits or loops is essentially a party-line or conference line. If you pick up one of the phones, you'll hear if someone is talking on any of the others.

Nationwide messages generally originate in the National Warning Center or its alternate. Historically, the National Warning Center was at Cheyenne Mountain, Colorado (with NORAD where alerts would most likely originate), and the Alternate National Warning Center was at Denton, Texas. There was a second alternate facility in 1968 at Olney, Maryland. At some point, FEMA seems to have "promoted" the sites to remove Cheyenne Mountain, making Denton the primary and Olney the alternate.

Denton and Olney were both the locations of FEMA regional headquarters, which have gone by various names over time like Special Facilities or Federal Support Centers. They were originally built by Civil Defense or FEMA (depending on the year) to coordinate the recovery from nuclear attack, and as such they were hardened. Perhaps the most famous is the Region 8 "FEMA Bunker" at the Denver Federal Center. Other FEMA regional headquarters tended to be either in more remote areas or were built by repurposing existing military facilities (such as that at Olney, a reused Nike missile site), and as such kept a lower profile.

It's hard to know what exactly is going on today. In the 2016 NAWAS Operating Manual, the most recent version that seems to have been made public, the Alternate Warning Center is directly named as Olney, MD but the location of the primary is left unsaid. The Olney facility has been transferred to the Naval Surface Warfare Center and is no longer in use by FEMA. A document suggests that the Alternate Warning Center may have moved to Thomasville Federal Center, near Atlanta. Denton, TX remains a major FEMA site and may still be the primary.

The primary control circuit connected the (originally) three warning centers with the eight regional headquarters, but considering that some of the regional headquarters were themselves warning centers, it had fewer points that it might sound like. That circuit served as one of several ways (including the HF radio system FNARS) that these major FEMA sites could communicate with each other in a major disaster, but it was less important for alert dissemination.

The more important parts are the regional circuits, which are configured as loop to provide redundancy against a line break. The regional circuits connect a FEMA Regional Headquarters, at least one Warning Center for redundancy, federal alerting points (an AT&T document lists Coast Guard stations as an example), and a primary and alternate warning point within each state. FEMA would use these circuits as the main way to distribute a nationwide alert, reading it directly to the state Warning Points, which are generally the offices of the state's emergency management agency. For example, in New Mexico, the Primary Warning Point is the Department of Public Safety office in Santa Fe; the Alternate Warning Point is the state Emergency Operations Complex at the National Guard complex by Santa Fe.

There are likely some additional levels of redundancy in most cases. For example, state emergency planning documents imply that the Department of Energy has a NAWAS site at Kirtland Air Force Base that is connected to at least the regional and state networks, and possibly the national control circuit as well. It thus serves as an additional contingency for distribution of alerts on the state network, were the Primary and Alternate Warning Points to be lost.

Each state circuit is left in part to the discretion of the state, with criteria stated by FEMA. Generally, it includes the state warning points along with county emergency operations centers and major infrastructure facilities like hospitals and power plants.

Along with the specialized purpose of NAWAS come specialized equipment. NAWAS relies on a system that AT&T calls SS1, presumably Signaling System 1, but is more similar to a selective calling scheme than to a more general signaling system. SS1 appears fairly similar to the control pulses used by CDWS, but repurposed to ring phones on NAWAS to get a user's attention. On the national control circuit, for example, the National Warning Centers have SS1 transmitters that can be used to signal SS1 receivers at the Regional Headquarters to ring. Keep in mind that this is conference system, so there is no real sense of "placing a call" or "hanging up." The telephones are always connected. The provision of selective calling is just to get another user's attention so that they pick up the phone. In practice, they likely won't even pick up the phone, as most NAWAS sets are equipped with an always-on speaker to monitor activity on the circuit.

The SS1 selective ringing system also allows the National Warning Centers to selectively call state warning points on the regional circuits. State warning points, at least those operated by the state itself, are able to selectively call other sites on the state network. And that's actually nearly the limit of the SS1 capabilities. All other selective calling is done by "voice paging," basically yelling into the phone in the hopes that the party you want is listening to their speaker.

There is one other interesting capability of SS1, which requires understanding the structure of the network, beyond just the circuits. At each state primary warning point, the regional NAWAS circuit is actually bridged to the state warning circuit, so that any traffic on the regional circuit will also be heard on the state circuit. Essentially, in its "normal" state, the whole state circuit is just a leg of the regional circuit, and all sites on it hear regional traffic. This ensures that a warning read by one of the National Warning Points will be heard as quickly as possible through the over 2,200 phones in the total NAWAS system. This connection is only present on one pair, though, so it's one way: the state circuit hears traffic on the regional circuit, but the regional circuit does not normally hear traffic on the state circuit.

As originally designed, a foot switch in each state Primary Warning Point disconnects the two networks when depressed, allowing the Primary Warning Point to "speak" on the state circuit only. The foot switch basically selects which of the two networks the Primary Warning Point phone will transmit onto, but also disconnects the bridge so that the Primary Warning Point can speak on the state network even if the regional network is busy. You can imagine, though, that this would pose a problem when a critical alert needs to be distributed. To guard against a stuck footswitch or just a particularly chatty state emergency manager, the National Warning Centers can send an SS1 code that will throw a relay to bypass the footswitch and reconnect the regional and state networks. This code would be sent just before any critical nationwide warning.

The use of four-wire conference systems usually requires a slightly different type of telephone set, anyway. One of the goals of four-wire systems is often to function as a "squawk box" or "hotline" (one of the many definitions of the word), an always-on system that can be heard on a speaker at every connected location. As a result, every NAWAS warning point has a loudspeaker. There is also a telephone handset, used to speak into the system. Because there are many phones on each NAWAS circuit and you know how large Zoom calls tend to get, each NAWAS handset has a push-to-talk button. You don't see these very often today, but Western Electric offered telephone handsets with a PTT button mounted on the inside of the handle as a standard product. Apparently depending on the preference of the installation site, the loudspeaker is automatically disabled either when the handset is picked up or only when the PTT button is depressed. This prevents feedback or echo. It might seem a little odd to hear a conversation on the speaker but speak into a handset, but this was very common in mid-century telephony, as the echo and feedback problems of a true "speakerphone" proved hard to solve.

At State Warning Points, an indicator light attached to the telephone set helps the operator understand what circuit they are connected to. Green means that the footswitch has been depressed to disconnect the state network from the regional network. White means that another location, presumably the State Alternate Warning Point, has depressed their footswitch to disconnect the two systems (the bridge was duplicated at the two sites, but one site pressing their footswitch would disconnect the bridge at the other site as well). A red light indicated that the National Warning Center had bypassed the footswitches to distribute an alert.

BSP 310-530-901 LL Issue A (1968) provides a general description and operating information on NAWAS as originally built. Much of the BSP relates to the bureaucracy of operating a special service on a nationwide telephone system: it lists, for example, the specific plant control offices responsible for the testing and maintenance of each circuit. The Cheyenne Mountain Central Office must have been an interesting place, with primary responsibility to handle trouble reports and network management on the national and regional circuits. Each regional circuit was assigned to a CO within the region as well, typically one in a major city near the corresponding FEMA regional headquarters.

These plant control offices are informed that they must report all outages and work done on the circuits to OCD, and that they must take special precautions to ensure uptime on the NAWAS circuits. 8.05 states, in characteristically dry AT&T tone, that "the circuit may be required by the customer at a moment's notice due to the nature of the business for which it is used." I have heard anecdotes that switchmen working in central offices were sometimes used to hearing routine chatter on the NAWAS, and I see that the BSP suggests (but does not require) that control offices responsible for the system have a dedicated monitoring speaker. Proving that customer service is always the hard part, the BSP requires that these control offices make regular visits to customer sites and emphasize the importance of keeping the loudspeaker volume turned up, using the PTT button on the handset, and promptly reporting any trouble.

The BSP also lays out a process for routine maintenance on this critical, high-uptime system, which AT&T refers to as "circuit line-up." Such line-ups are performed only on Saturday, and must be authorized by the National Warning Center and announced to all Warning Points. Once started, the line-up time allows the test rooms at each control office to perform routine quality measurements on the circuits and detect any problems requiring maintenance. The test room must keep the circuit audible on a speaker for the duration, so that they can immediately stop their testing should the National Warning Center transmit that they need the circuit returned to carry emergency traffic. Following each line-up, the National Warning Center would place test calls to each State Warning Point to verify correct functioning, and the test room was expected to stay on the line to monitor this process. One can imagine that the whole thing felt like a hassle to the test room personnel who had to stay perpetually on edge.

The modern Operations Manual for NAWAS is available via FOIA, or at least a version from 2016. It mostly describes the details of operating the station equipment, which have not changed all that much since the 1968 BSP. The original Western Electric sets have been replaced by NAWAS terminals manufactured by Comlabs, a small company focused on emergency communications. They appear to be modified AT&T or Lucent phones with a new dialpad (including the elusive DMTF ABCD "digits"), an alert light, speaker volume knob, and presumably the electronics swapped out for four-wire operation. Phones used in local warning points have no keypad at all, since they are not intended to issue alerts. The pulse-based SS1 system has been replaced by DTMF; selective calling is done using four-digit numbers while commands like linking and unlinking regional and state networks use the DTMF "A" digit.

Besides some modernization of the terminals, the sites on the system have expanded along with its scope. NWS forecast offices, NOAA facilities like the Hurricane Center, and FEMA's "MERS" mobile response teams have been added to the regional circuits. The NWS runs its own round of tests to ensure it can distribute weather alerts to State Warning Points.

You might be somewhat familiar with NAWAS. I have mentioned it before, and in general, it's one of the best known of the government's various emergency communications system. No small part of this is due to a compelling bit of drama available in the documents. The manual has scripts.

This is the FEMA (Alternate) Operations Center. A nuclear weapon detonated in (city, county, state) at _____ Zulu. Radioactive fallout is possible! Persons in (city, county, state) should be advised to remain under cover and await further instructions from state or local authorities. Residents are advised to take protective actions in accordance with local community shelter plans and to be alert for further instructions from state or local authorities. Residents in all other areas are advised that protective action is not required at this time.

This is actually somewhat lengthy considering that the audience for the message is mostly state emergency management agencies. Alerts of this type on NAWAS would be proceeded by an audible alert tone to attract attention, and followed by a roll-call of all state warning points to ensure that the message was received. Local authorities would do... something. In the event of an attack, they will theoretically activate local sirens with a wavering tone for 3-5 minutes. FEMA procedures dictate that this signal exclusively indicates a confirmed impending or in-progress attack. In practice, it is either the same as or not readily distinguishable from the signal used for tornadoes in many tornado-prone areas of the country---generally the only areas with sirens at all. FEMA procedures have often coped poorly with the use of warning systems for purposes other than those that were apparent during the Cold War.

The manual provides scripts for other scenarios, ranging from a detection by NORAD (probably actually by intelligence community assets) of possible fires to to reentering space debris, errant weapon launches (nuclear or conventional), and various natural disasters. One of the newer additions to the procedure is use by the Pacific Tsunami Warning Center to disperse warnings of tsunamis affecting the west coast. This is one of the newest disaster scenarios to prompt serious investment in mass notification, and large parts of the west coast are now equipped with sirens in case of tsunami.

NAWAS has a close connection to IPAWS, the Integrated Public Alert and Warning System. Most NAWAS alerts would simultaneously be issued as IPAWS alerts, distributed across radio and television stations and via the newer Wireless Emergency Alert system. Much like IPAWS, only federal authorities can issue national NAWAS alerts, but regional authorities like state governors are permitted and even encouraged to use state NAWAS circuits to disseminate local alerts. Indeed, most state governors residences have NAWAS terminals installed for ready access.

NAWAS has provided reliable service for many decades, but now shows its age. Private-line telephone systems like NAWAS are fundamentally challenging to harden against attack and disaster due to their fixed routing. Besides, the statewide conference line capability of NAWAS now seems rather limited compared to the popularity of text messaging in emergency management. In 2022, FEMA awarded AT&T a $167 million contract to modernize multiple FEMA communications systems, including NAWAS. The plans for NAWAS are vague:

AT&T will transition the NAWAS legacy technologies to newer services available via EIS through a well-planned, phased, cost-effective, and non-disruptive approach to the new solution with government oversight.

Sounds great. Can't wait for the new solution.

So there we have it, a large, nationwide communications network made up of four-wire private lines. There have actually been a number of these used over time, including notably the Strategic Air Command's C2 network which involved both radio links and private lines terminating at telco-furnished turrets at SAC bases. Some smaller-scale four-wire systems remain in use today for local emergency management purposes.

Later, possibly as soon as next week, we will take a look at a much more complex type of private line service: Common Carrier Switching Arrangements. But I might get sidetracked on the road to CCSAs once again, and post first about the federal private line services that lead to the invention of the CCSA: AUTOVON and the Federal Telephone System (FTS).

I have been meaning, for some time, to write about common carrier switching arrangements (CCSAs). These could be considered an early form of products like "virtual private ethernet:" a private telephone network that was served by the same switching machines that handled the public telephone system. A CCSA is, in effect, a "virtual telephone network." AT&T operated a number of these for both government agencies and large private organizations, and they might be viewed in a way as precursors to the large CENTREX-and-WATS arrangements that became a common fixture of state governments and school districts.

The problem is that I fear I am putting the cart before the horse. CCSAs, and even the fully private telephone systems they were intended to replace, are basically the extreme extension of the private line. Besides, private lines are an important part of the history of computing, as well: they were the pattern for the digital "leased line" services that formed the bulk of computer network connections through the early days of the internet.

Let's ease into it by starting with an important source in telephone history: the Bell System Practices.

Large organizations tend to function like religions. This is more or less overt, depending on the organization and the fashions of the time. For example, in the period we will mostly discuss in this article, a number of large companies published song books. IBM is the best known for this type of corporate spirit, ranging from their de facto hymnal to the "Think" signs customarily installed in IBM worker's offices. The television series "Severance" frequently referred to this aspect of post-war corporate culture, depicting a corporation that unified its employees through songs, quotations, and an internal museum. This type of pseudoreligious corporate culture is, of course, taken to an extreme in "Severance," but nonetheless resembles real practices that still echo through corporate America today.

Severance depicts its ominous corporation with exterior shots of the Bell Laboratories offices at Holmdel, a prestige design by Eero Saarinen that dates to an era in which the architecture of corporate offices was often an idealistic representation of their intended culture [1]. This is just a little bit ironic, as Severance is clearly patterned more after the computer industry than AT&T, which was comparatively subtle in its corporate religion. Still, there were aspects of the sublime: Saul Bass's pitch film for AT&T's 1960s rebrand devotes much of its length to a hagiography of the Telephone Men. Universal Service, AT&T's stated goal, was both as ambitious and ever-changing as "redemption," personified as the enormous, golden "Spirit of Communication" depicted in sculpture and mural at former AT&T offices. And AT&T had signs! Of course they had signs; the best known being the "Bell System Safety Creed" that hung in most work areas.

This is all to preface my use of the word "doctrine." The kind of shared culture that companies attempted to establish through song books is also an important part of practical operations. We associate doctrine mostly with religion, but more broadly, a doctrine is a set of codified, shared beliefs and practices that define the work of an organization. The military has doctrine in this sense, and AT&T has doctrine as well. Much of that doctrine was compiled into a lengthy, almost sacred text: the Bell System Practices, or BSPs.

BSPs are an invaluable source of information, and fortunately organizations like TCI have put a lot of effort into finding and preserving the remaining copies. Unlike publications like the Bell System Technical Journal (BSTJ), BSPs were for internal use only, and so the collection available today is scattered and incomplete. That's not the only challenge when using BSPs for research. Fitting the analogy to holy texts, the organization of the BSPs is arcane and has undergone a number of changes. For the latter half of the 20th century, BSPs were identified by nine digit numbers, separated into three groups of three digits. The first group identified the Division, the second the Section, and the third the Document.

BSP 000-000-000 provides an index to the Divisions. Each division begins with a section index, numbered NNN-000-000. Sections are typically grouped by their first digit, producing topics referred to by NNN-N. NNN-0 is devoted to indexes, NNN-1 is usually a "general information" topic. There are loose conventions for document numbering. For the weightier topics, -100 is often the overview or general procedures. There are as many exceptions as examples of these rules. Further complicating document lookup, BSPs were published roughly from 1930 to 1990, numbering conventions were changed and sections reorganized, and documents were replaced by updates identified by an issue number. The aspect of time can make it a frustrating and time-consuming process to match up a set of BSPs that not only cover a topic but represent a single point in time, rather than being confused by a decade of changes from one to the next.

We can hope that, one day, a monastic order of telephone enthusiasts will take up the compilation and, while they're at it, illumination of the BSPs. They will produce a Xanadu-esque compendium of AT&T doctrine navigable along dimensions of both time and space; a Grand Enfilade of communications technology. For my own part, I am perhaps alarmingly close but not quite ready to take a vow of silence, move yet further into the desert, and devote my life to the task. For the time being we'll have to settle for search engines, the indexing efforts of a few dedicated people, and no small amount of sweat.

Enough of that, let's take a look at divisions 310 through 312, which cover private lines for voice and data, as well as special switched services. A discussion of private lines starts rather naturally with 310-300-100, Two Point Private Line Systems: Two Point Private Line Telephone Circuits, Voice Only---Description.

Of course, we will not start there, because of the joy of BSPs: Naturally, I have searched several archives and not been able to find a copy of 310-300-100. We can fill in some of this gap from 310-300-300 and 310-300-500, though, Test Objectives and Test Procedures. 310-300-300 I1 (1975) 1.04:

The circuits discussed in this section are served over nonswitched facilities with no access to the message network. A two-point private line involves a channel between two terminal locations. The channel may or may not be entirely a metallic path.

A private line is, at its essence, a pair of wires that runs from one location to another. The locations are customer-specified, and the private line is not connected to anything else, only to the two service points. Some version of this service has existed for pretty much the entire history of the telephone, although it has often, as you would imagine, been very expensive.

The private lines covered in this section are for voice use only. This is important, because voice use implies a specific set of requirements. The topic of private lines, leased lines, and related services is actually a very difficult one to discuss succinctly. First, because there is a 100-year history of these services, and the technology, capabilities, and use-cases have all changed over time. Second, because there is a very diverse set of uses for private lines, each with a different set of requirements.

Voice use, for example, implies a customary telephone voice passband of about 300-3400Hz, and more precisely limited by the 8kHz sampling rate of digital telephony. This might strike you as surprisingly narrow, but it's enough for reliable speech intelligibility. It really is quite narrow though, part of the reason that "HD Voice" codecs now offered by VoIP technology sound startlingly different from traditional calls.

The fact that the circuit may not be entirely metallic is important as well. It might be a little unclear what a "nonmetallic" path would even mean, but remember that we are looking at a 1975 revision. By that time digital telephony was in full swing; a customer's "pair from one place to another" may very well be digitized, carried by the TDM switching network, and converted back to analog (analogized?) near the other end. The possibility of carrying private line service over the digital switching system radically reduced the cost of private lines, and enabled the proliferation of "leased lines" for use with computers.

But, it did have limitations. A private line of the 1930s would have been able to carry polarity reversal signaling, a common scheme at the time for alarm monitoring and remote control. A nonmetallic path was not electrically continuous, so it could not be used for any type of electrical signaling. It would only work for the signaling scheme it was provisioned for, namely, voice.

The BSPs describe a set of tests intended to ensure suitability for that purpose. There are standards for loss, frequency response, and noise. We also learn that private lines for voice use can be configured for ringing, so that one end can signal the other to pick up the phone. Elsewhere in the BSPs the supported types of ringing are enumerated, although they no doubt changed over time: ringing by a ring key (today we'd call that a ring button), by a hand-crank generator, by central office equipment, or no ringing at all.

I would like to cover this topic without getting too stuck in terminology, but it's hard to avoid because there is a lot of terminological confusion. For example, what is the difference between a "private line" and a "leased line" anyway? Well, it's mostly a matter of who's talking. AT&T seems to have always used the term "Private Line" except when discussing data services in the context of switched data networks, in which case they use "Leased Line" to refer to a fixed-capacity data connectivity arrangement. We'll get to that in a bit.

Another problem is how to describe the configuration of private lines. A private line provisioned for voice use, as described in 310-300, is not connected to any switching equipment but would be connected to battery power (AT&T called this "common battery," meaning that the phones did not require their own batteries to function). Private lines with common battery were sometimes referred to as "wet," while private lines without common battery were "dry." The latter were mostly used for non-telephone signaling applications.

Private lines might also be connected to other types of equipment to provide useful features. For example, you might wonder about how ringing applied by the central office would work. A popular type of private line for voice, especially later on, was the "Private Line Automatic Ringdown" (PLAR), also just called a "Ringdown" circuit. These private lines are configured so that picking up the phone at either end automatically applies ringing voltage to the other end.

---

Shameless cross-promotion: I uploaded a YouTube video yesterday where, among other things, I briefly mention that "hotline" is a heavily overloaded term that means different things in different contexts. One of the things that "hotline" often refers to is a PLAR. Mostly, though, I try out a very cheap PABX I bought on the internet and find out that, while not exactly great, it is certainly pretty good for the price.

---

Another option on private lines was whether they were two-wire or four-wire. A typical telephone uses only two wires, one pair. If you think about it, that's a little bit magical. The trick is a device called a hybrid transformer, installed in each telephone, that basically "subtracts" the signal it is transmitting from the signal on the line in order to isolate the "receive" signal. At a modern telephone exchange, another hybrid transformer splits them out once again for handling by the exchange equipment. This is a two-wire circuit, and has the advantage of lower-cost wiring. On private lines, you can also get four-wire service, with dedicated pairs for transmit and receive (also called, for obvious reasons, talk and listen). The major advantage of four-wire circuits is that they make "conference calling" much easier. When you put a lot of telephones onto one two-wire circuit, it becomes very difficult to manage the cumulative echo produced by the imperfection of the hybrid transformers. A full four-wire system avoids this problem.

What were these private lines for? Well, you can probably imagine a few applications for voice private lines, but keep in mind that for much of their history they were very expensive... prior to the use of TDM digital networks, you were basically paying AT&T for the installation and upkeep of however many miles of telephone wire were required to get between the two locations... plus carrier equipment, line conditioning, etc.

One of the cool things about the BSPs, and one of the reasons they were kept internal, is that it's not at all unusual for them to go into detail on specific customers. We can get a feel for the use-cases for these service offerings from the specific customers that used them heavily enough to merit BSPs. For two-point private line telephone service, that customer is the FAA. The FAA has long had a tight relationship with AT&T, rivaled only by the military for reliance on dedicated telephone infrastructure. Air-traffic controllers used voice private lines both to talk to each other across facilities, and to remotely operate the radios they used to communicate with aircraft.

Voice private lines were also popular in other situations where people across multiple locations needed to communicate in real-time. For example, television and radio networks often had private lines between studios and network control centers to aid the staff in coordinating the start and stop of programs from different locations. The telephone system was an obvious choice, because for the early history of radio and television most networks used "broadcast-grade" private lines (with larger bandwidths than voice lines, especially for television!) to distribute their programming to the member stations. The radio and television networks were a huge business for AT&T prior to the use of satellite transponders for the same purpose.

I know of some more exotic applications as well. Some critical infrastructure or large industrial facilities, and in particular nuclear power plants with their extensive emergency preparation requirements, had four-wire private line systems that linked their control rooms to contingency sites, disaster response agencies, and even the homes of their senior staff. These were configured as conference lines for use in coordinating an emergency response. Indeed, emergency management was another major application of private lines, and I will eventually write about FEMA's nationwide four-wire private line system, still used for dissemination of emergency warnings to state and regional emergency management agencies.

These conference lines are, as you have likely suspected, not really two-station private line systems. They are multistation voice systems, described in 310-405-100. TCI has a much older issue of this BSP, from 1957, and it describes the electrical resistor networks used to achieve the "conference line" configuration in which the talk pairs are mixed and sent on the listen pairs of all stations. 310-405-100 I1 (1957) 1.03:

One of the most important requirements of a multistation private line circuit is that the volume delivered to each receiver be substantially the same no matter which station is talking. The above holds true even when two or more circuits are switched together.

Indeed, and we are still struggling with this today!

1.04 is interesting:

Various signaling systems and combinations of signaling systems can be used to "call" the various stations on a private line circuit. Some types of signaling systems are: loudspeaker signaling, manual code ringing, ringdown signaling, 600-1500-cycle (2-tone) selective signaling, and code selective signaling.

Explaining all of those would be an article of its own, but it is an interesting note that multistation systems challenge the typical sense of "ringing" on a telephone line, and conventional ringing doesn't seem to have been very common on four-wire systems. A common replacement is what is described as "loudspeaker signaling" here, where the stations have an always-on speaker and you get someone's attention by... yelling at them, basically.

AT&T has never limited themselves to voice. There have long been various types of control or signaling equipment available, and 310-435 describes the SC2 Selective Control System available for use with multistation private lines. This telegraph-like system involves a control station sending series of coded pulses, which are detected by satellite stations that open and close relays in response. There are still a lot of products like this today, but the SC2 as a private line offering is a good time to make a point about the historic phone system: it used to be that all telephone equipment was the property of the telephone company. That included the SC2 control and satellite stations, and they had a terminal strip on the outside of the cabinet that exposed the relay contacts and served as the demarcation point between telco and customer property.

That has an interesting implication for these private line systems: the operating company needed a way to test and diagnose them. As a result, larger or higher-criticality private line systems usually included some kind of terminal equipment at the customer location that provided test functionality. On four-wire systems, a loopback test was the norm: some sort of signal sent on the line would cause a relay to shunt the talk pair to the listen pair at the customer premises, allowing the test board at the exchange to send signals all the way "around" the private line. The SC2 had a pretty complicated piece of terminal equipment, since it had to decode the signal pulses, and as a result there is an extensive test procedure with parts that can be performed remotely and parts that must be done on-site. The BSP cautions that arrangements must be made with the customer to disconnect their equipment before a telephone technician starts opening and closing relays to test.

A final common application of voice private lines was for "tie lines," or any of the other names they went by. These were lines that ran between two switching systems, providing something like a long-distance trunk between them. Imagine a corporation with two offices, each with a private branch exchange. The corporation could contract for a private line between the two PBXs that served as a tie line, and the operators at each location could then use it to directly connect calls between the two offices. Besides avoiding the operators having to dial those calls, tie lines could also save money over long-distance calling, if utilization was high enough. A number of services offered by AT&T and operating companies basically amounted to different ways of using tie lines, so Division 311 "Switched Special Services Systems" covers tie line configurations including WATS and operating company-managed PBXs.

Of course, if you have read this far, you are probably wondering about data. In the contemporary computer industry we associate this kind of private service entirely with the leased lines of the '70s and '80s. Division 312 covers Private Line Data Systems and Services, beginning with Electronic Telegraph Loops and teletypewriters. In the 1950s, "data" pretty much meant teletypewriter, and there is a dedicated customer section on Western Union who used AT&T circuits to extend their own network. The first half-dozen topics are devoted to various types of telegraph systems, including DC telegraphy (over continuous metallic circuits) and carrier telegraphy (combining multiple telegraph channels onto one private line using frequency division muxing).

312-8 is where we get to the good stuff, Data Sets. Data set was the term used by AT&T for what we now know as a modem, for example, the Bell 103 modem is more properly the 103 Data Set. The term "data set" predates acoustic modems and is somewhat more general, though, having been used by AT&T to refer to simple relay closure systems as well. Still, by the 1978 312-000-000 index there are a range of different data sets available, covering different speeds and applications. The systems covered in this section are intended for use with a complete private line data system, that is, the lines themselves were part of the system. They were typically private lines that terminated at dedicated equipment in the telephone exchange, where a data bus was used to carry signals between different lines.

It is in Division 313 that we find what we'll recognize today: Voice and Voiceband Data Circuits. These are data systems that operate over voice-type lines, using acoustic modem methods to encode data within the frequency response of those lines. This division is actually surprisingly short, for a reason. 313-100-100 I2 (1982) 1.01:

Telephone company testing of circuits that terminate in customer premises outlined in these sections includes only that portion of the circuit up to the network interface. The circuit testing procedure does not include customer premises terminal equipment.

In practice, a lot of voiceband equipment would be provided by AT&T, but they are establishing a clear separation of concerns between a voice-type private line and the modem used with it. It's important to understand that "network interface" as used here has a specific meaning within the telephone industry, it is the demarcation point between the telephone network and the customer's equipment.

Higher-speed telephone modems like the Bell 201, capable of 2400bps, could be used on four-wire circuits for full-duplex operation. Indeed, four-wire circuits were quite common for data use as well because they enabled full duplex operation. AT&T was also not the only option for voiceband modems. Division 314, Digital and Analogue Data Transmission Systems, is dedicated to a variety of modems and I49 (1983) includes familiar names like IBM and Data General.

This practice of using voiceband equipment to put data over private lines, without requiring extensive special equipment at the central office, started a shift in data communications practices that greatly blurred the lines between different types of service. Some of the confusion of terms we encounter today comes from these gray lines. The Bell 103 is a data set and can be used on private lines, but it can also be used on a conventional dial line. Similarly, the lines between voice and data were blurred. SAGE incorporated a nationwide computer network that operated over voiceband modems, often considered the first precursor to the modern internet. The digitization of the telephone network would further complicate definitions.

---

I have a long list of telephone-related topics to cover and I will probably never get to all of them, although I sure will do my best. It takes quite a bit of my free time to write these articles, and I'm also starting work on a more ambitious project around telephone history in more of a reference format. I'd appreciate your support in pursuing these projects---consider supporting me on ko-fi, which will also get you my subscribers-only newsletter EYES ONLY.

---

During the 1960s, AT&T introduced TDM digital trunks to the telephone network. Using digital technology, a large number of telephone calls could be digitized into samples and those samples multiplexed onto a single high-speed data connection between two telephone exchanges. This method of multiplexing was more reliable and less prone to noise than analog FDM methods, and it could be adapted to a wide variety of carriers. Over the following decades the telephone network underwent a wholesale conversion to digital, and it is now typical that the only "analog" parts of an analog telephone call are the last mile connections between the exchange and the customer premises. Modern telephone exchanges digitize calls at the line cards and it remains digital until reaching a line card on the other end. We were doing data over voiceband, and then we were doing voiceband over data.

The implied result, data over voiceband over data, was in fact very common and the apex of dial-up internet standards (v.90/v.92) assume that the underlying telephone connection is digital. The oddity of stacking multiple layers of digitization was far more apparent on private lines, though, where there was no requirement to retain compatibility with a standard telephone loop.

And thus the leased line was introduced. The closest thing I know of to a technical difference between a "private line" and a "leased line" is that a private line is assumed to be private over the entire span, while a "leased line," in practice, refers only to the last-mile. The actual telephone network is digital and using TDM or even packet switching methods, there is no need for a dedicated physical connection between two central offices to carry data. The leased line just allows a customer a way to insert data into the telephone network.

While v.92 assumes a bidirectional digital connection, it is limited to 56kbps because of properties of standard phone lines including the use of companding. Leased lines didn't have to put up with this limitation, they could omit the functions of a telephone line card and instead deliver a digital signal all the way to the customer premises. While the telephone network itself went through several major iterations of digital media (including the charmingly named "plesiochronous" network), the last-mile digital connection to customers has been well-standardized for a surprisingly long time: the T-carrier.

Let's talk quickly about the Digital Hierarchy, the scheme around which the TDM telephone network was designed. TDM involves packing samples into time slots in a round-robin fashion, so the digital hierarchy is similarly organized around cycles. As a hierarchy, those cycles get larger and larger. A telephone call consists of 8-bit samples at 8kHz, which multiples to 64kbps of data. That 64kbps channel, in the digital hierarchy, is referred to as DS0 or Digital Signal 0. By the somewhat arbitrary but pragmatic design of the digital hierarchy, 24 DS0s are multiplexed to form a DS1. These DS designations refer to the actual payload, not to the carrier technology used to transmit it. DS1, in the United States, was most commonly carried by Transmission System 1, commonly called T1 or T-carrier. 24 64kbps channels adds up to 1.544mbps, and that's exactly what a T1 delivered.

This connection was digital the entire way through, with no need for conversion to analog and the resulting quantization noise and bandwidth limitations. T1 was originally designed for trunk connections between telephone equipment, but it became quite natural to extend T1 connections to customer sites as a form of high-speed (for the time) data. T1 was the typical format of the leased line even into the '00s, and most people my age probably remember having dial-up service and coveting the remarkable speed of a T1 connection.

This is not to say that leased lines were limited to T1. There were higher-speed systems used within the telephone network that carried higher steps on the digital hierarchy, DS2, DS3, and so on, and these could be extended to customer premises as well. At the high end, larger businesses would be placed directly on a SONET fiber-optic ring via add-drop multiplexers, an arrangement capable of multi-gigabit speeds.

It's a little odd, actually, that the "leased line" we think of has very little to do with actual telephones. It's a digital network connection much like we use today, except that it functions on top of the provisioned-bandwidth, synchronous, TDM network originally built in order to carry telephone calls. There are probably still organizations today running ethernet over SDH for their internet connection, and it won't feel much different from anything else we use.

[1] The use of architecture as a symbol of corporate power, once a fundamental part of the computer and telecommunications industry, is largely lost today. There are more than a few reasons, but one of them is the development history of Silicon Valley. It is remarkable how underwhelming the offices of today's most powerful corporations are, consisting of scattered low-rise office parks with no identity besides the earth tones and angles of the 1970s. Of course, when modern tech companies do build prestige headquarters, they tend to be unspeakably ugly. I'm speaking mostly of Meta, I will give Apple's effort a mediocre grade.

I am making steady progress towards moving the Computers Are Bad enterprise cloud to its new home, here in New Mexico. One of the steps in this process is, of course, purchasing a new server... the current Big Iron is getting rather old (probably about a decade!) and here in town I'll have the rack space for more machines anyway.

In our modern, cloud-centric industry, it is rare that I find myself comparing the specifications of a Dell PowerEdge against an HP ProLiant. Because the non-hyperscale server market has increasingly consolidated around Intel specifications and reference designs, it is even rarer that there is much of a difference between the major options.

This brings back to mind one of those ancient questions that comes up among computer novices and becomes a writing prompt for technology bloggers. What is a server? Is it just, like, a big computer? Or is it actually special?

There's a lot of industrial history wrapped up in that question, and the answer is often very context-specific. But there are some generalizations we can make about the history of the server: client-server computing originated mostly as an evolution of time-sharing computing using multiple terminals connected to a single computer. There was no expectation that terminals had a similar architecture to computers (and indeed they were usually vastly simpler machines), and that attitude carried over to client-server systems. The PC revolution instilled a WinTel monoculture in much of client-side computing by the mid-'90s, but it remained common into the '00s for servers to run entirely different operating systems and architectures.

The SPARC and Solaris combination was very common for servers, as were IBM's minicomputer architectures and their numerous operating systems. Indeed, one of the key commercial contributions of Java was the way it allowed enterprise applications to be written for a Solaris/SPARC backend while enabling code reuse for clients that ran on either stalwarts like Unix/RISC or "modern" business computing environments like Windows/x86. This model was sometimes referred to as client-server computing with "thick clients." It preserved the differentiation between "server" and "client" as classes of machines, and the universal adherance of serious business software to this model lead to an association between server platforms and "enterprise computing."

Over time, things have changed, as they always do. Architectures that had been relegated to servers became increasingly niche and struggled to compete with the PC architecture on cost and performance. The general architecture of server software shifted away from vertical scaling and high-uptime systems to horizontal scaling with relaxed reliability requirements, taking away much of the advantage of enterprise-class computers. For the most part, today, a server is just a big computer. There are some distinguishing features: servers are far more likely to be SMP or NUMA, with multiple processor sockets. While the days of SAS and hardware RAID are increasingly behind us, servers continue to have more complex storage controllers and topologies than clients. And servers, almost by definition, offer some sort of out of band management.

Out-of-band management, sometimes also called lights-out management, identifies a capability that is almost unheard of in clients. A separate, smaller management computer allows for remote access to a server even when it is, say, powered off. The terms out-of-band and in-band in this context emerge from their customery uses in networking and telecom, meaning that out of band management is performed without the use of the standard (we might say "data plane") network connection to a machine. But in practice they have drifted in meaning, and it is probably better to think of out-of-band management as meaning that the operating system and general-purpose components are not required. This might be made clearer by comparison: a very standard example of in-band management would be SSH, a service provided by the software on a computer that allows you to interact with it. Out-of-band management, by contrast, is provided by a dedicated hardware and software stack and does not require the operating system or, traditionally, even the CPU to cooperate.

You can imagine that this is a useful capability. Today, out-of-band management is probably best exemplified by the remote console that most servers offer. It's basically an embedded IP KVM, allowing you to interact with the machine as if you were at a locally connected monitor and keyboard. A lot of OOB management products also offer "virtual media," where you can upload an ISO file to the management interface and then have it appear to the computer proper as if it were a physical device. This is extremely useful for installing operating systems.

OOB management is an interesting little corner of computer history. It's not a new idea at all; in fact, similar capabilities can be found through pretty much the entire history of business computing. If anything, it's gotten simpler and more boring over time. A few evenings ago I was watching a clabretro video about an IBM p5 he's gotten working. As is the case in most of his videos about servers, he has to give a brief explanation of the multiple layers of lower-level management systems present in the p5 and their various textmode and web interfaces.

If we constrain our discussion of "servers" to relatively modern machines, starting say in the late '80s or early '90s, there are some common features:

• Some sort of local operator interface (this term itself being a very old one), like an LCD matrix display or grid of LED indicators, providing low-level information on hardware health.
• A serial console with access to the early bootloader and a persistent low-level management system.
• A higher-level management system, with a variable position in the stack depending on architecture, for remote management of the machine workload.

A lot of this stuff still hangs around today. Most servers can tell you on the front panel if a redundant component like a fan or power supply has failed, although the number of components that are redundant and can be replaced online has dwindled with time from "everything up to and including CPUs" on '90s prestige architectures to sometimes little more than fans. Serial management is still pretty common, mostly as a holdover of being a popular way to do OS installation and maintenance on headless machines [1].

But for the most part, OOB management has consolidated in the exact same way as processor architecture: onto Intel IPMI.

IPMI is confusing to some people for a couple of reasons. First, IPMI is a specification, not an implementation. Most major vendors have their own implementation of IPMI, often with features above and beyond the core IPMI spec, and they call them weird acronyms like HP iLO and Dell DRAC. These vendor-specific implementations often predate IPMI, too, so it's never quite right to say they are "just IPMI." They're independent systems with IPMI characteristics. On the other hand, more upstart manufacturers are more likely to just call it IPMI, in which case it may just be the standard offering from their firmware vendor.

Further confusing matters is a fair amount of terminological overlap. The IPMI software runs on a processor conventionally called the baseboard management controller or BMC, and the terms IPMI and BMC are sometimes used interchangeably. Lights-out management or LOM is mostly an obsolete term but sticks around because HP(E) is a fan of it and continues to call their IPMI implementation Integrated Lights-Out. The BMC should not be confused with the System Management Controller or SMC, which is one of a few terms used for a component present in client computers to handle tasks like fan speed control. These have an interrelated history and, indeed, the BMC handles those functions in most servers.

IPMI also specifies two interfaces: an out-of-band interface available over the network or a serial connection, and an in-band interface available to the operating system via a driver (and, in practice, I believe communication between the CPU and the baseboard management controller via the low-pin-count or LPC bus, which is a weird little holdover of ISA present in most modern computers). The result is that you can interact with the IPMI from a tool running in the operating system, like ipmitool on Linux. That makes it a little confusing what exactly is going on, if you don't understand that the IPMI is a completely independent system that has a local interface to the running operating system for convenience.

What does the IPMI actually do? Well, like most things, it's mostly become a webapp. Web interfaces are just too convenient to turn down, so while a lot of IPMI products do have dedicated client software, they're porting all the features into an embedded web application. The quality of these web interfaces varies widely but is mostly not very good. That raises a question, of course, of how you get to the IPMI web interface.

Most servers on the market have a dedicated ethernet interface for the IPMI, often labelled "IPMI" or "management" or something like that. Most people would agree that the best way to use IPMI is to put the management network interface onto a dedicated physical network, for reasons of both security and reliability (IPMI should remain accessible even in case of performance or reliability problems with your main network). A dedicated physical network costs time, space, and money, though, so there are compromises. For one, your "management network" is very likely to be a VLAN on your normal network equipment. That's sort of like what AT&T calls a common-carrier switching arrangement, meaning that it behaves like an independent, private network but shares all of the actual equipment with everything else, the isolation being implemented in software. That was a weird comparison to make and I probably just need to write a whole article on CCSAs like I've been meaning to.

Even that approach requires extra cabling, though, so IPMI offers "sideband" networking. With sideband management, the BMC communicates directly with the same NIC that the operating system uses. The implementation is a little bit weird: the NIC will pretend to be two different interfaces, mixing IPMI traffic into the same packet stream as host traffic but using a different MAC address. This way, it appears to other network equipment as if there are two different network interfaces in use, as usual. I will leave judgment as to how good of an idea this is to you, but there are obvious security considerations around reducing the segregation between IPMI and application traffic.

And yes, it should be said, a lot of IPMI implementations have proven to be security nightmares. They should never be accessible to any untrusted person.

Details of network features vary between IPMI implementations, but there is a standard interface on UDP 623 that can be used for discovery and basic commands. There's often SSH and a web interface, and VNC is pretty common for remote console.

There are some neat basic functions you can perform with the IPMI, either over the network or locally using an in-band IPMI client. A useful one, if you are forgetful and keep poor records like I do, is listing the hardware modules making up the machine at an FRU or vendor part number level. You can also interact with basic hardware functions like sensors, power state, fans, etc. IPMI offers a standard watchdog timer, which can be combined with software running on the operating system to ensure that the server will be reset if the application gets into an unhealthy state. You should set a long enough timeout to allow the system to boot and for you to connect and disable the watchdog timer, ask me how I know.

One of the reasons I thought to write about IPMI is its strange relationship to the world of everyday client computers. IPMI is very common in enterprise servers but very rare elsewhere, much to the consternation of people like me that don't have the space or noise tolerance for a 1U pizzabox in their homes. If you are trying to stick to compact or low-power computers, you'll pretty much have to go without.

But then, there's kind of a weird exception. What about Intel ME and AMD ST? These are essentially OOB management controllers that are present in virtually all Intel and AMD processors. This is kind of an odd story. Intel ME, the Management Engine, is an enabling component of Intel Active Management Technology (Intel AMT). AMT was pretty much an attempt at popularizing OOB management for client machines, and offers most of the same capabilities as IPMI. It has been considerably less successful. Most of that is probably due to pricing, Intel has limited almost all AMT features to use with their very costly enterprise management platforms. Perhaps there is some industry in which these sell well, but I am apparently not in it. There are open-source AMT clients, but the next problem you will run into is finding a machine where AMT is actually usable.

The fact that Intel AMT has sideband management capability, and that therefore the Intel ME component on which AMT runs has sideband management capability, was the topic of quite some consternation in the security community. Here is a mitigating factor: sideband management is only possible if the processor, motherboard chipset, and NIC are all AMT-capable. Options for all three devices are limited to Intel products with the vPro badge. The unpopularity of Intel NICs in consumer devices alone means that sideband access is rarely possible. vPro is also limited to relatively high-end processors and chipsets. The bad news is that you will have a hard time using AMT in your homelab, although some people certainly do. The upside is that the widely-reported "fact" that Intel ME is accessible via sideband networking on consumer devices is typically untrue, and for reasons beyond Intel software licensing.

That leaves an odd question around Intel ME itself, though, which is certainly OOB management-like but doesn't really have any OOB management features without AMT. So why do nearly all processors have it? Well, this is somewhat speculative, but the impression I get is that Intel ME exists mostly as a convenient way to host and manage trusted execution components that are used for things like Secure Boot and DRM. These features all run on the same processor as ME and share some common technology stack. The "management" portion of Intel ME is thus largely vestigial, and it's part of the secure computing infrastructure.

This is not to make excuses for Intel ME, which is entirely unauditable by third parties and has harbored significant security vulnerabilities in the past. But, remember, we all use one processor architecture from one of two vendors, so Intel doesn't have a whole lot of motivation to do better. Lest you respond that ARM is the way, remember that modern ARM SOCs used in consumer devices have pretty much identical capabilities.

It is what it is.

[1] The definition of "headless" is sticky and we have to not get stuck on it too much. People tend to say "headless" to mean no monitor and keyboard attached, but keep in mind that slide-out rack consoles and IP KVMs have been common for a long time and so in non-hyperscale environments truly headless machines are rarer than you would think. Part of this is because using a serial console is a monumental pain in the ass, so your typical computer operator will do a lot to avoid dealing with it. Before LCD displays, this meant a CRT and keyboard on an Anthro cart with wheels, but now that we are an enlightened society, you can cram a whole monitor and keyboard into 1U and get a KVM switching fabric that can cover the whole rack. Or swap cables. Mostly swap cables.

Billboards

Route 66 is often viewed through the lens of its billboards. The Jack Rabbit Trading Post, a small store a few miles out of Joseph City, would hardly be remembered were it not for its billboards spanning four states. The tradition of far-advance billboards is still observed today. Albuquerque's roadside stop operator Bowlin puts billboards six hours and two freeway exchanges out from its combined gas station-Dairy Queens. One can ponder the mystery of "The Thing" (near Benson, Arizona) throughout nearly the entire state of New Mexico.

So, if you have driven anywhere over a several-hundred-mile span of Interstate 40 (the modern-day successor to Route 66 in this area), you are probably aware of the Meteor Crater [1]. At Meteor, Arizona, between Flagstaff and Winslow, the Meteor Crater earns its nonspecific name: it was the first crater definitely shown to have been the result of a meteor impact. It is a spectacular sight, almost 4,000 feet wide and 600 feet deep. It is also known by another name: Barringer Crater, for the family that has owned it for over one hundred years.

Today, Meteor Crater is one of the few traditional Route 66 roadside attractions to have held on to much of its vitality. A steady flow of visitors pay admission to see the crater and its attached visitors center, somewhat ostentatiously styled as the Barringer Space Museum. The museum focuses on the crater's two major connections to space: first, that it was formed by a meteor that came from there. Second, and most importantly, that the crater has been used as a training site for astronauts since the Apollo era.

I assume it has been calculated that these are the two topics that draw visitors, because the museum devotes almost no space at all to what I consider the most fascinating of the crater's many stories: that the crater was a mine. Forget space; the Meteor Crater is the greatest artifact of a fascinating, but brief, chapter of American mining history.

Meteorites

But first, as dismissive as I may be, we must talk a bit about space. During the 19th century, detailed observations of meteor showers established that the bright trails of light seen in the night sky---previously assumed to be some atmospheric phenomenon---must in fact be the collision of objects from space with the atmosphere. The idea that rocks or something were flying out of space and into the upper reaches of our world naturally suggests that some of them might make it all the way down. Indeed, as early as 1803 such a meteorite [2] had been found on the ground in France following a meteor shower, although the idea that it had fallen from above was not universally accepted until much later.

Chemists analyzed a number of meteorites recovered during that era and found that almost all of them contained significant amounts of iron, and their geological oddities (such as the inclusion of spherical globules of metal) supported the idea that they had formed in space. By the turn of the 20th century, it was generally understood that meteors were chunks of mostly iron that came from somewhere out there and collided with our planet, and that some of them made it all the way to the ground.

What was not well understood by that time was the fate of those meteorites. Since the 17th century it was known that there were craters on the moon, and the idea that they had been formed by impacts is nearly that old as well. Even three hundred years later, though, it was far from a settled matter. Volcanic activity was also a promising explanation, and one that many felt to be more comfortably within the bounds of reason. In the year 1900, many astronomers would have been quite dismissive of the moon-meteorite theory, as the opinion of the day favored a moon with an active volcanic core.

The idea that impacts had formed vast craters on Earth must have been even more far-fetched. Besides, the known meteorites were quite small, more prone to knocking holes in roofs than in the very desert. It was in this context that the Meteor Crater was first examined.

Okay, we're done with that boring space stuff. Let's talk about mines!

In 1891, a small mining firm based out of Albuquerque received a sample from a prospector in Arizona. The prospector had found an ore vein in a remote part of that territory that he believed to be quite valuable. Indeed, an assayer put the included sample at a remarkable 77% iron, with a bit of lead, silver, and gold to boot. Observing its purity and unusual structure, the assayer figured that it had been melted in a furnace.

The mining firm, amazed by this new ore and skeptical of the assayer's attribution to a furnace, circulated parts of their sample among a number of business leaders---men who might put up the money for a large-scale mining operation. One of them, James Williamson of Civil War fame and by then an executive the Atlantic and Pacific Railroad, sent his sample to mineralogist Albert E. Foote of Philadelphia for his thoughts on its commercial value and exploitability. Foote, recognizing the sample's unusual structure, knew immediately that the prospectors claim of a vein two miles long and up to forty yards wide could not be entirely true [3]. It was a meteorite.

In truth, the "ore" was one of a number of small fragments (where "small" still often surpassed 100 pounds) that could be found over a large area near Cañon Diablo and a feature known to locals as "Crater Mountain." Foote described the "so-called 'crater'", "the sides of which are so steep that animals that have descended into it have been unable to escape and have left their bleached bones at the bottom." Foote carefully searched the rim, but found no evidence of volcanic activity, leaving the crater's origin a mystery. Foote's blindness to the crater's connection with the meteorite fragments is almost hard to believe, but he seems to have taken it as a coincidence.

The remarkable quantity of oxidized black fragmental material that was found at those points where the greatest number of small fragments of meteoric iron were found, would seem to indicate that an extraordinarily large mass of probably 500 or 600 pounds had become oxidized while passing through the air and was so weakened in its internal structure that it had burst into pieces not long before reaching the earth.

Indeed, that same year, the chief geologist of the USGS visited the crater and reached the same conclusion. The crater's proximity to the meteorites seemed to be chance. The crater itself must be the result of a vast steam explosion, which was, after all, not an unknown phenomenon in that part of the Arizona territory.

The late 19th century saw the formation of the Division of Forestry (of the Department of Agriculture) and a related reorganization of the General Land Office, precursors to today's Forest Service and Bureau of Land Management. Large areas of public land would be withdrawn from the GLO's management (which consisted mostly of sale to private owners) and reserved as National Forests. In eastern Arizona, this controversial task fell on GLO agent S. J. Holsinger. He traveled the region extensively in studying forest issues and establishing the boundaries of what would become the Coconino National Forest. Although he never saw it for himself, he heard stories of an enormous crater, near what he called Coon Mountain or Coon Butte. The stories held that fragments of iron, from a meteor, had been found within and around it.

The record is unclear on where or how, but one day in October of 1902, Holsinger found himself in a casual conversation with a mine engineer by the name of Daniel Barringer. Perhaps they sat by each other in a saloon or a train station. Perhaps Barringer told Holsinger that he worked in iron mining, and Holsinger said something along the lines of "I'll tell you about some iron." In any case, Holsinger described the meteorites, and the crater---as they had been described to him. Some of the locals, Holsinger said, had a theory: that the crater had actually been formed by the meteorites. That a single meteorite larger than ever seen before had crashed into the earth, burying itself far below, and leaving behind the feature known as Coon Mountain, or Crater Mountain, or Sunset Knoll as the USGS had labeled it on maps.

Barringer was hooked. He wrote back to Holsinger for more information, he researched the area, and he involved his friend Benjamin Tilghman, the inventor of sandblasting and a general scientific type in the pattern of the era, in the pursuit. They became convinced that the USGS had been wrong, and the locals right: that it was not just a crater, but a Meteor Crater. So, they bought it.

Iron

Although Barringer was well-qualified as a geologist and was clearly fascinated by the crater, his interest in it was not purely scientific. In a 1905 paper making his argument for its meteoritic origin, Barringer reports that the iron fragments from the meteorite had already been commercially exploited. A nearby merchant had hired laborers to search the area around the crater for the "iron ore," and they had found some pieces as heavy as 1,000 pounds. Of the smaller pieces that they collected, the merchant estimated that between his efforts and those of another businessman, perhaps 15 tons of the iron-rich material had been shipped away for smelting.

Barringer and his business partners had found thousands of fragments, ranging from over 200 pounds to less than an ounce. These "Cañon Diablo siderites" could be over 90% iron, the rest being mostly nickel.

Among the evidence he cites for the meteoritic theory, Barringer observes that every single one of these fragments had been found immediately on the surface, and only a handfull had ever been found inside of the crater. They were not themselves buried by impact; they seemed to be found right where they lay after they were ejected from the crater with great force.

Barringer's tone in the 1905 paper is, well, critical, particularly where it comes to the USGS geologist who had declared the crater a product of a steam explosion, Gilbert. Barringer suggests that Gilbert's rejection of the meteoritic theory could only be the result of a profound failure to notice geological inconsistencies that ought to have been obvious. The presence of a great amount of fine silica in every direction from the crater (sometimes called "rock flour," this very fine sand is the result of impact forces instantaneously shattering a large area of bedrock), the distribution of iron fragments neatly centered on the crater, and the upturned and sideways geological layers found in the rim all indicated that the rim had been "heaved out" of the crater itself by a great force.

To definitively prove his theory, though, Barringer relied on trenches in search of buried fragments. If Barringer was correct, there should be fragments of the meteorite mixed randomly within the other ejecta of the crater. It took some effort, but eventually Barringer reported clear cases, including a large iron fragment found underneath a slab of sandstone that must have come from 400 feet below the surface.

Still, Barringer had bigger plans: he intended to find the original meteorite itself--the core from all of the fragments had broken. With a horse-driven drill, his employees sunk numerous shafts from 200 to over 1,000 feet deep. Many seemed to strike iron material, but Barringer believed them to be mere fragments, not the great cluster of broken iron that must be present somewhere beneath the crater.

His belief in this huge iron core explains the mining patent he took out on the behalf of his company, Standard Iron. He planned to find it and sell it. By his estimate, based on a crude analysis he developed, it would have a market value in the range of one billion dollars.

---

If you have made it this far, you are either fairly committed to my writing or very bored. Either way, would you consider supporting my work with a monthly contribution? Contributors receive EYES ONLY, a special newsletter on special topics (but mostly computer trivia).

https://ko-fi.com/jbcrawford

---

Shafts

Barringer was enthusiastic about his meteor theory in a charmingly turn-of-the-century way. He had invented, he said, the field of "meteoritics," or the study of meteors. A central concern of meteoritics was the ballistics of a meteorite: what happened when a meteorite struck the ground.

Today, thanks in large part to World War II munitions research and the nuclear weapons program, we have considerable theoretical and experimental information on how solid objects penetrate soil and rock---a topic that Sandia Laboratory, operating with similar zeal for the new frontiers of science, dubbed "terradynamics." This work began in the 1930s; Barringer was too early to benefit. He had to develop a similar theory on his own.

Understanding the geology of the crater involved a huge effort, particularly in such a remote location. Barringer had to build out a considerable operation, and he had to do so by remote. He was already a known figure in Arizona, having launched a particularly successful mine, and he worried that anyone that got word of his interest in the crater would attempt to jump his claim. Instead of traveling to Arizona, be brought in relatives and friends as business partners and even hired Holsinger away from the Division of Forestry to act as land manager. Barringer and Holsinger chartered a railroad to the site, established mill sites on the Little Colorado River and Oak Creek, and constructed a water reservoir. A camp was established at the base of the rim, and an expanding workforce started on holes, trenches, and a shaft straight down in the center of the crater.

Barringer first reached the crater in 1904, where he learned that the central shaft had been abandoned at 200 feet of depth due to the fine silica forming a quicksand that quickly filled the excavation. He opted for a different approach, buying a 4" drill and sinking five smaller shafts in the crater floor. Most of these shafts ended when they struck meteorite fragments too hard for the drill. One, managing to dodge any large pieces of iron, found no new meteoritic material after 550 feet, and reached intact bedrock at 1000 feet. These observations lead Barringer to conclude that the crater had originally been deeper but had been partially filled back in by the material that was thrown into the air.

Barringer thought that a steam hoist might allow them to excavate the quicksand more quickly than it filled, allowing progress on the larger shaft in the crater's center. Barringer called off the drilling effort and shifted focus once again to the shaft, spending 1905 excavating a larger clean shaft to the level of the quicksand. In 1906, the race against the soil was attempted, but failed. The next year, drilling resumed at an accelerated pace, with sixteen new boreholes completed in 1907.

Each bore found meteor fragments down to nearly 600 feet, but there was no evidence of the meteorite itself. Barringer began to develop a new theory: the meteorite was not directly below the crater.

He conducted a series of simple ballistic experiments: he shot a rifle into the ground. The impact of the bullet into the desert soil, he reasoned, would behave similarly to the impact of a meteorite into the same. His easy target shooting lead to an interesting finding. Regardless of the angle at which he shot the ground, the resulting hole appeared round. The meteor, he realized, likely didn't come straight down; it struck at an angle.

Analyzing the rim, he came to believe that the meteorite would be found somewhere beneath the south rim. More fragments were found to the north, and the south rim had been lifted higher than in other directions. He had a new target, but there was a problem... he was out of money. Barringer had started the project with considerable wealth and several partners, but he had spent $100,000 searching and Tilghman, his friend and investor in the project, decided to back out.

Progress at Meteor Crater was slow for the next decade as Barringer marketed the project to prospective investors. It was not until 1918 that he signed up a new partner, the United States Smelting and Refining Company. USSRC signed up to put $75,000 into exploratory drilling, but $60,000 was spent installing supporting infrastructure, including a ten mile water pipe, before drilling began from near the top of the south rim.

Progress was slow, and stopped entirely at 282 feet when the drill became jammed and the $75,000 exhausted. Barringer seems to have pulled off some feat of salesmanship by convincing USSRC to continue to support the project, and in 1921 a horizontal tunnel was dug from the interior of the crater to the end of the drill, revealing the two iron balls that had stopped the work. While the tunnel plan was innovative, it was not particularly effective, as the bore turned out to be in poor shape and difficult to continue.

Drilling practically started over again, reaching 600 feet at the end of 1921. Barringer's theory appeared to pan out: at 1,100 feet, well below the crater floor, meteor fragments started to appear. The drill stuck again at 1,300 feet, but not before finding dense a span dense with iron fragments. Drilling to that depth had cost $200,000, and while promising, none of the material recovered had been of any significant value. USSRC pulled out.

Barringer was back to fundraising, courting various mining companies including one, United Verde Extension, who rejected the project based on their conclusion that the crater was the result not of a meteor, but of a steam explosion. You see, even in 1924, Barringer's theory of the crater's origin came off as crackpot. The steam explosion theory was still the accepted one, and the crater had attracted surprisingly little attention from professional scientists. Barringer's abrasive response to the USGS survey was no doubt a factor; a USGS geologist by the name of N. H. Darton who had worked for Gilbert during the original survey took the cause to heart.

Throughout Barringer's mineral exploration of the crater, Darton published papers arguing for a steam explosion and dismissing the meteorite theory. He refused to change the name of the site on USGS maps away from Coon Butte until 1916, when he begrudgingly accepted "Crater Mound." Barringer's son, Brandon Barringer, wrote that a USGS geologist once told him that he thought the meteor theory to be correct but "it would cost me my job if I was heard saying so."

Drilling resumed again that year, the effort of a new stockholder corporation that leased the crater from Standard iron and worked on the advice of Barringer. This corporation, the Meteor Crater Exploration and Mining Company, was backed in part by Boston business magnate Quincy Adams Shaw. Shaw would bring about Barringer's vindication, and his downfall.

One of Daniel Barringer's sons, Daniel Barringer Jr., followed his father into the study of meteoritics. He would make a discovery similar to his fathers: a letter to a mining journal described the discovery, in 1921, of a large iron meteorite near a hole in the area of Odessa, Texas.

Barringer Jr. arranged a deal with G. M. Colvocoresses, a smelter executive and one of the investors in the Exploration Company, to inspect a set of mines in Texas on his behalf. This provided a convenient excuse to travel through Odessa, where he examined the area and found signs much like those near Cañon Diablo. In 1926, he discovered the world's second known meteor impact crater, the Odessa Crater.

Incidentally, something else had been discovered near Odessa: oil. The Odessa Crater turned out to be owned by an oil interest, which was not interested in selling.

Fragments

According to a pattern, the Exploration company completed a new shaft near the USSRC effort, which struck water at 600 feet and about $200,000 in expense. Shaw, beginning to question the project, engaged the services of an expert: astronomer Forest Ray Moulton. Moulton was a distinguished scientist, a professor at the University of Chicago, and even better, an expert in ballistics, having been in charge of ballistics research at the Aberdeen Proving Ground during the First World War.

Moulton seemed to accept the meteorite theory from the start. After all, despite the objections of the USGS, General Electric cofounder and MIT president Elihu Thomson had visited the crater and reached the same conclusion. Still, it gives some of the flavor of the debate that Barringer sometimes referred to his scientific supporters as "converts."

Moulton's report, published in 1929, was a mixed result for Barringer. He concluded that the crater was indeed the result of an impact by a meteorite, perhaps 50,000 to 3,000,000 tons. He also concluded that the meteorite would never be found. The impact energy was more than enough to vaporize it entirely, leaving only the fragments scattered across the plain.

Three months later, Daniel Barringer died.

His dreams of one billion dollars of iron and nickel buried beneath the crater died with him. While his family carried on the effort, the Great Depression ensured that only two further shafts would be drilled, after which mining exploration ended.

By 1930, the passage of time, the discovery of the Odessa Crater, and Moulton's report had solidified the meteorite theory to such a degree that the Meteor Crater became generally accepted as just that. Meteor mining, though, was over before it had started.

Billboards

If not valuable, meteor craters remain unique and fascinating geological features. The Odessa Crater was leased by Ector County as a tourist site, and efforts by various parties including the Texas Memorial Museum lead to the discovery of a six-ton meteorite at the bottom of a second, smaller crater nearby. Efforts by the Works Progress Administration and later the Barringer Family to locate the meteorite that formed the larger main crater failed, leaving a covered shaft still visible today.

At Meteor Crater itself, the mission has changed in a classic Route 66 fashion: from mining to tourism. In 1953, the Standard Iron Company renamed itself to the Barringer Crater Company. The Company continues to operate Meteor Crater as a tourist attraction and scientific resource, hosting NASA training efforts and geological experiments. Ongoing research at Meteor Crater produced many conclusions about meteorites and impact events, including extensive research by Eugene Shoemaker.

It was Shoemaker who directed NASA to the site for training purposes: he had been considered as an astronaut himself, but excluded for medical reasons. Instead, he took to Meteor Crater as a moon of his own. It is due in large part to Shoemaker's comparisons between Meteor Crater and the lunar surface that we now know the the craters of the Moon to be a result of meteorites as well.

Today, Meteor Crater appears as a roadside stop, not that different from Jack Rabbit or "The Thing." Billboards precede it by hours, ignored by their audience of long-haul truckers. It is dusty, and minimally staffed, and has the feeling of a forgotten place; a spacesuited mannequin and a 4D Experience only add to the impression of a tourist trap with a brighter past. The museum, with so much focus on the only tangentially related Apollo program, forms a stark contrast with brass plaques devoted to a hagiography of the Barringers. They bring you out to see the crater, but then they say very little about what is in the crater. You have to read between the lines to find the reason why: Barringer never found what he was looking for.

Still, there are things that you cannot see anywhere else. The crater itself, bigger than the Sedan Crater excavated by a nuclear weapon, can only hint at the unthinkable energy released some 50,000 years ago. A huge fragment of the meteorite, weighing 3/4 ton and named for Holsinger, is prominently displayed in the museum. Apartments built for staff remind us of just how remote "between Flagstaff and Winslow" once was.

Barringer's obsession with meteorites was sincere. Based on his writings, it predated the discovery of the Meteor Crater, and he devoted as much of his life to better understanding the crater as he did to mining it. His family and the Barringer Crater Company continue to make grants in meteoritics research, and each year they give the Barringer Medal. Its first honoree was Shoemaker, its most recent Canadian geology professor John Spray, who studies deformation and friction at extreme speeds and pressures.

Like so much of Route 66, it feels dated, and more than a little tacky. And like Route 66 as well, it is a fascinating chapter in the story of the American West. An eastern businessman, as eccentric as he was passionate, left for Arizona in pursuit of a wild idea. He was only half right.

[1] Curiously, the most recent billboards for Meteor Crater mainly feature an anthropomorphic rabbit, apparently a character designed for a "4D experience" at the museum. The connection between the rabbit and the crater is left unexplained; an odd mascot that you might be tempted to ascribe to furries except that if the furries were in charge it would have looked better. I cautiously speculate that it may have been intended as a reference to the Jack Rabbit.

[2] A meteor is seen in the sky, a meteoroid is the actual object that burned up to be seen as a meteor, and a meteorite hits the ground. It is said that you can remember this by recalling that meteorites are "right" in that they successfully made it all the way. As with so many memory devices, I think you could also argue this one the other way, so it's not really that helpful.

[3] Foote's paper to the AAAS about the discovery makes it clear that this kind of wild exaggeration was not unusual when coming from prospectors. "There were some remarkable mineralogical and geological features which together with the character of the iron itself, would allow of a good deal of self-deception in a man who wanted to sell a mine."

Did you hear that Elon Musk dug a tunnel under the Las Vegas Convention Center?

I think it is pretty universally known by now that the "Las Vegas Loop" is impractical, poorly thought out, and generally an embarrassment to society and industry. I will spare an accounting of the history and future of the system, but I will give a bit of context for the unfamiliar reader. The Las Vegas Loop is a (supposed) mass-transit system built and operated by The Boring Company for the Las Vegas Convention and Visitors Authority at the Las Vegas Convention Center. Besides four (ish) stations in the Convention Center, it has been expanded to serve Resorts World as well. It will, according to plan, be expanded to as many as 93 stops throughout the Las Vegas metropolitan area, despite the mayor of Las Vegas calling it "impractical" and "unsafe and inaccessible." This odd contradiction comes about because The Boring Company is footing a very large portion of the construction cost, while much of the rest is coming from casinos and resorts, making it extremely inexpensive for regional government agencies.

In practice, the Loop consists of a set of mostly double-bore tunnels of small diameter, which are traversed by Tesla Model 3 and Tesla Model X vehicles manually driven by humans at up to 40 mph. They have more recently switched to Model Y, but the operations manual I have predates that change, so let's stick with the older models for consistency. Each vehicle seats up to four. The system is nominally a PRT, or personal rapid transit, as the drivers take you to the specific station you request. The tunnel to Resorts World is single bore, and can admit vehicles in only one direction. A simple signaling scheme serves to prevent vehicles meeting head-on in single tunnels. While Loop and Boring Company marketing focuses heavily on the single underground station, all other stations are above ground. In the current state, I think it is actually somewhat generous to call the Loop an underground system, as most maneuvers and operations occur at surface level. It is perhaps best thought of as a taxi system that makes use of underground connectors to bypass traffic. Future expansion plans involve significantly more tunnel length and more underground stations, which will probably cause the system overall to feel more like a below-surface transit system and less like an odd fleet of hotel courtesy cars.

I am not going to provide a general review of the system, because many others have, and you can probably already guess what I think of it. Instead, I want to focus on some aspects that have not been as heavily discussed in other reporting: detailed operational practices, and safety and communications technology.

We are fortunate that, as part of its fire safety permitting, the Loop has been required to file its operations manual with Clark County. Unfortunately, the newest revision I can find online is 2021's Revision 7, which predates the Resorts World station and may be out of date in other ways as well. Still, it appears to be substantially correct, and much of what I will discuss is based on Revision 7 of the manual alongside several trips I have taken in the system.

Interestingly, the operations manual refers to the system only as the "Campus-Wide People Mover" or CWPM. This term seems to date to the original solicitations by LVCVA, but is not used in marketing.

Rules and Discipline

Like most detailed policies, the operations manual is an interesting read for the pedantic. Some parts are odd in a classically Elon Musk way, like the manual's use of "What's Elon like?" as the first example of a question that passengers might ask a driver. Other parts are weird in a more conventional way, like a paragraph that says first that the tunnels are connected to the operations control center (OCC) by single-mode fiber, second by two redundant fibers, and third by two single-mode fibers taking separate paths. I am pretty sure the correct interpretation of this paragraph is that there are two fiber routes and they just said that three times to pad for length, but it's hard to be totally sure. Why it's so important to clarify that it's single-mode is anyone's guess, perhaps because during the review stage a regulator asked. You find this kind of thing a lot in these sorts of policies, which are usually edited extremely carefully for regulatory compliance and not at all for plain reading.

One interesting aspect of the Loop, that has been rather heavily reported on, is that the whole thing feels remarkably cheap. Perhaps that's not surprising, as The Boring Company's central claim is to be able to construct underground transit on a tight budget. They have indeed delivered on this promise in the construction of the Loop, but it's hard not to feel like they did so more by pinching pennies and eating development costs than by innovation. Nothing speaks to this more than the photo of the OCC the manual provides, which depicts two cheap office chairs at a long desk in a room with worn linoleum floors and distinctly portable-building vibes. It is, by far, the most underwhelming mass-transit control room I have ever seen. I strongly suspect that the OCC is a reuse of the old on-site construction office. To be fair, they will surely have to build something more sophisticated for future expansion, as the current space only accommodates two operators. We will assume it is temporary.

Early in the manual, hiring and training requirements are discussed. They are fairly standard for transportation drivers, with hiring requirements mostly amounting to a clean driving record and a clean drug test. Drivers must undergo 10 hours of in-vehicle training, including a four hour half-shift endurance exercise with mock passengers. There is classroom (or more likely computer-delivered) training as well, but the manual doesn't enumerate it. Drivers are required to wear a provided uniform with plain black shoes and no jewelry or accessories, and are prohibited from initiating conversations with passengers.

Which leads us somewhat naturally to the next section of the manual, on rules and discipline. There is usually a wide gap between rules written into policy and rules followed in practice, with "real" rules being determined by enforcement behavior---what practically can be enforced, and what supervisors choose to enforce. For example, the manual prohibits the drivers listening to the radio in the cars, something that zero percent of the drivers I have had complied with. There is no discipline outlined in the manual for this infraction, so, is it even really a rule?

The speed limits for the system are 40 in straight tunnels, 30 in turns, 15 on ramps, and 10 in stations. Most of this was unsurprising except for the "15 on ramps" part, as my drivers have consistently taken full advantage of the electric vehicle's torque, hitting 30 before the end of the descent ramp. This would appear to be a violation of policy. But, it's interesting to note, discipline (a "demerit") is only listed for a speed excursion of at least 5 seconds. Because of the short length of the ramps, it is likely not actually possible to incur a demerit for violating the 15 mph ramp speed limit. I wonder if the authors of this policy realized that.

Drivers are strictly prohibited from using any assistive driving features. This is sort of a moot point in practice, as maintenance staff are required to disable the assistive driving features of the vehicles before they are put into service in the Loop. This isn't at all surprising considering the highly regulated nature of transit operations, but it is pretty funny considering that The Boring Company originally promised automation, and that closely related Tesla has made self-driving a key part of their marketing.

Emergency Procedures

So, with the rules stuff out of the way, let's talk about emergency procedures. One of the problems with underground transit is that tunnels can be very dangerous: they are enclosed spaces where exits may be far away, and in a fire they can quickly fill with toxic and opaque smoke. Many historic incidents have illustrated the inherent danger of tunnels, and so modern tunnel designs incorporate extensive safety measures which typically include smoke extraction systems, evacuation guidance, emergency exits or refuges, and increasingly, fire suppression systems.

The Loop has been widely criticized for incorporating very few of these features. It does have a basic smoke evacuation system, but there is no evacuation guidance in the tunnels (no signage to indicate the nearest exit in low visibility conditions), and no evacuation points or refuges except for, oddly enough, marked refuges at the end of some tunnels that seem to be largely an ADA compliance measure because the ramps are too steep to be considered ADA-compliant egress (they are remarkably steep!).

To be fair, I think some of these criticisms are somewhat overblown. It does appear to be possible to open the car doors just about fully within the tunnels, although I think they are likely to strike the walls. And the thing is, the tunnels are very short! like really short! some tunnels seem to be shorter than the typical interval between refuge points in modern highway tunnels, and those that are longer probably aren't longer by much. The expansion system may incorporate more extensive safety measures due to longer tunnel runs.

Evacuation procedure basically consist of driving the car out of the tunnel, via the next station. If evacuation must be made in the opposite direction, the manual says the driver must await instructions from the OCC, as they are not generally permitted to drive in reverse. This is probably an accommodation for the poor rear visibility of the vehicle; drivers are normally prohibited by policy from driving in reverse. The OCC would likely have to coordinate vehicles reversing out by track warrant (tunnel warrant?) to avoid collisions. This is a common pain point with evacuation of train tunnels, for example, where there may not be a cab on the rear and even if there is there may not be enough time for the operator to switch ends.

In the event it becomes necessary to abandon the vehicle, the driver is to have passengers get out, and then lead them to the closest exit. The driver will presumably have to know the nearest exit by heart, since there isn't clear evacuation guidance in the tunnel. The manual addresses difficulty opening the vehicle doors, a common concern with Teslas that have electrically operated door releases. My understanding is that both the Model 3 and the Model X do have a mechanical release for all passenger doors, although it's pretty hidden on the rear doors. Oddly enough, the manual doesn't seem to know that, as the Loop operations manual strongly implies that there is no manual release for the rear doors of a Model 3. I would think that authorities would have immediately noticed that implication, so it makes me wonder if the Model 3 rear door release (a wire loop hidden under a panel) was simply ruled out as infeasible to use in an emergency scenario. Of course, that's odd, because the operations manual... just doesn't tell you what to do with rear passengers in a Model 3. You are basically SOL, as far as the operations manual is concerned. Only passengers of a Model X are allowed to escape in a scenario where the vehicle loses power. To my eyes, that is by far the biggest unresolved problem with the emergency operations plans. Perhaps a later revision of the manual addresses it, because it seems like more of a documentation error than a real problem.

The manual does not address evacuation procedures. There may be a document with that information that did not make it to the internet. At at least two points in the tunnels I spotted golf carts (the type with four rows of seats) stashed in corners, and I suspect they would be used if there was a need to retrieve passengers from a disabled car. My husband pointed out that due to their larger seating capacity and faster boarding/deboarding, the Loop would likely achieve a higher capacity if they just shifted operations entirely to the Club Cars.

Customer Service

There is a section of the manual on customer service and interactions with customers I actually don't think it's that unusual for this kind of policy, so I don't want to mock it too much, but I will tell you this: drivers are told to keep conversations as short as possible and give as little information about themselves as possible. They are not to tell passengers how long they have worked for The Boring Company, how old they are, their last names, or information about TBC employee counts or pay rates, even if asked. There is a surprisingly long (in comparison to other items) script for answering questions about the flamethrower. Drivers are told to tell passengers that the Loop operates at "about 35 average and 50 maximum," an interesting answer since 40 is the maximum speed in any segment and exceeding it for more than 5 seconds would lead to a demerit (exceeding 50 for more than 5 seconds would lead to suspension).

The correct answer to "What's Elon like" is "He's awesome [inspiring / motivating / etc.]", and drivers are not sure how often he is around. There is a whole section about how to answer questions about Elon Musk, including how to respond to questions about his tweets. I enjoy that Elon Musk is a person such that every employee of one of his companies needs some basic press training on how to deal with his social media habits. Drivers are, if we take this script more literally than its authors probably thought out, to say that they do not have personal experience of Elon Musk smoking weed.

The operations manual spends more page length on answering questions about Elon Musk and The Boring Company than it does on fires.

Communications Technology

So, what communications technology does the Loop employ? Communications in tunnels is an interesting problem, especially in a life-safety critical environment. Unfortunately, The Boring Company has opted for a pretty boring approach that also seems... questionably safe?

From the manual and various public press, we can infer that the tunnel has some type of LTE. "Leaky" cables are an interesting RF technique often used for tunnels, but the Loop tunnels are short enough that directional antennas at the end might be sufficient. Still, it seems very plausible that there is leaky feeder embedded in the overhead light trough.

Loop operators are equipped with an iPad and a bluetooth headset. The iPad runs a very basic looking app with a "call" button the operator can use to reach the OCC. It's probably using a very straightforward third-party library to either make a VoIP (e.g. WebRTC) call or to use the iOS dialer to do the same (does iPad OS have a dialer? I don't know how these newfangled Apples work).

An interesting note - you might notice that Tesla vehicles characteristically include a big touchscreen in the middle. The Las Vegas Loop vehicles do run modified software, but based on the 2021 operations manual and the scant more recent information I can find, it has been modified only to allow the OCC to remotely access vehicle status information and cameras. There don't appear to be any extra driver-facing features, leaving a need for the iPad.

But what about a failure? Well, there is an auxiliary system, one that is very similar to that used in other types of underground transportation systems. At regular intervals in each tunnel, as well as at stations and other critical points, there is a "blue light station." The blue light station consists of a blue fire pull device that presumably reports an emergency to the OCC (but probably does not automatically trigger an evacuation, as that can be very bad), as well as a phone. The phone is configured as a "hotline" in the modern-traditional sense, meaning it automatically dials the OCC when taken off hook. This appears to be the only emergency communications system, although it seems unlikely to me that they don't have a public safety repeater system in the tunnels (e.g. for 900MHz P25), as fire authorities often require them.

Mass notification can serve as an important secondary communications system, either when there are problems with primary systems or in emergencies that require action as quickly as possible. Here, the Loop takes an interesting approach. The operations manual includes a screenshot of the OCC operator interface, and the amount of screen space devoted to controlling the tunnel's RGB gamer lights seems odd (given that they use white lighting except for during special events, it seems like this is more of a marketing concern than an operations task). That makes a lot more sense when you discover that the mass notification plan is all based on the lighting: in an evacuation scenario, the lights will flash red and white in the intended direction of travel, and remain solid red in the direction of danger. I honestly think this is a clever use of the lighting equipment and I like how it indicates the direction of evacuation, but I do worry a little about whether or not the color of the lighting is set over a life-safety grade network, or just via the LTE or something.

Accessibility

A quick side-note about accessibility. Transit enthusiasts probably know that ADA requirements for public transportation are quite strict, and there's not much of a way around offering wheelchair service. The Boring Company seems to mostly address this question by saying that passengers are expected to transfer from wheelchair into sedan, which... sucks and isn't going to pass ADA review. So they have a secret: a GEM cart. GEM is a brand of Polaris, the parent company of RZR and Indian Motorcycle and assorted other small vehicle brands, aimed at institutional customers. GEM carts are golf-cart or neighborhood electric vehicle (NEV) class electric vehicles with 72-volt systems, and they make a wheelchair accessible version. Apparently the Las Vegas Loop now uses one to ferry around wheelchair customers.

The punch line here is... remember what my husband said about the golf cart we saw? GEM makes carts that seat five in addition to the driver, with a higher seating position and open sides or optionally large doors for faster board/deboard. Even with the 25mph stock speed limiter for NEV/LSV regulatory compliance (and believe me, with some adjustments to the motor controller they can go faster), I suspect that switching the Loop entirely to GEMs would increase its total capacity. And the GEMs honestly suck, in the world of light electric vehicles. They just kind of pulled off a regulatory capture move and got the NEV rules written to pretty much require something that sucks as much as they do for street legality.

Subjective Experience

So as I said, this is not a review, just trying to focus on some things of interest to transit, communications, and policy dweebs. Which I assume pretty much describes my core readers. But I do want to point out a couple of oddities that add to the "wow, this is cheap" sensation:

The ride is surprisingly rough, even in a Model Y with highway-grade suspension. I am concerned that they may not be able to do much better when paving in the confined tunnels, given that I don't think standard paving equipment would fit in the loading gauge. The ride experience was not "oooo electric car luxury," it was more on par with the Orlando Airport APM100s with sketchy steering gear.

For the segment that requires tickets (to Resorts World), the ticketing system is based on a QR code. The customer-side implementation is fine enough, but the ticket checking is laughable. It's an iPad where you have to show a QR code to the front-facing camera, meaning you have to present the QR code with your phone facing away from you, looking at the image on the iPad for alignment. It is very awkward and there is no reason for it besides cheapness. Plus there's not really any way for the attendant to see if the ticket is valid without standing awkwardly close to you to look at the same iPad screen you are, and indeed, I accidentally scored a free ride by merit of the attendant's inability to see the actual result of the ticket check.

The stations are not especially well thought out. People walking in and out of the stations have to cross the path of the Loop vehicles in some places. The attendants are supposed to direct people and, for trips to Resorts World, collect fare, but the design of some stations lacks a chokepoint at which to do so. The attendants have to kind of chase people down after they've already walked straight to a vehicle.

The tunnel to Resorts World is one-way. Its portal is connected to the West LVCC station by a tunnel, but the station and the Resorts World portal are actually in the same parking lot. They seem to have adopted a practice of cars one way going through the tunnel, and cars the other way just driving... across the parking lot. This is very funny to experience and contributes a lot to the feeling that the Loop is only marginally an underground system. I doubt the original designers intended for this outcome, it seems like the money spent on the connecting tunnel was completely wasted, but I'm assuming that eliminating one segment of single-track tunnel helped with throughput. Their approach to managing traffic at the Resorts World portal also involves a sort of approach-pattern-esque architecture where every car has to drive in a circle around the portal before entering, which is funny.

This stuff matters in my mind because it gets to the question of what the Loop... is for? The capacity of the Loop is very low. The expansion plan calls for a lot of tunnels, doubled up for capacity in places, but targets only 90k passengers per day. That would put it at around 8x the current daily ridership of the monorail, but with a vastly larger network of stations. Presumably they will expand fare collection, and I would have to think that tickets will actually become fairly expensive. so it's probably not intended to be a high-capacity, low-cost option.

So what else could it be? Well, some press and discussion around the Loop figures it as more of a luxury option: something that casinos can comp for high rollers, that will spare people dealing with the general disaster of getting around the strip. But it also doesn't feel like that. The outdoor stations, need to quickly board and deboard a sedan, and general chaos level of the stations (i.e. attendant chasing you down for ticket) make it feel more "courtesy car" than "black car."

I don't know, they could totally dress it up a bit and make it feel fancier. Some paint here and there, train the attendants better, do more to direct traffic. They could! But right now I think the best way to describe the Las Vegas Loop is... "cheap and amateurish." Surprisingly fitting with the Las Vegas vibe, in a way.

So, let's say that a security vendor, we'll call them ClownStrike, accidentally takes down most of their Windows install base with a poorly tested content update. Rough day at the office, huh? There are lots of things you could say about this, lots of reasons it happens this way, lots of people to blame and not to blame, etc., etc., but nearly every time a major security incident like this hits the news, you see a lot of people repeating an old refrain:

these systems shouldn't be connected to the internet.

Every time, I get a little twitch.

The idea that computer systems just "shouldn't be connected to the internet," for security or reliability purposes, is a really common one. It's got a lot of appeal to it! But there's not really that many environments where it's done. In this unusually applied and present-era article, I want to talk a little about the real considerations around "just not connecting it to the internet," and why I wish people wouldn't bring it up if they aren't ready for some serious considerations.

We Live in a Society

In the abstract, computers can perform valuable work by doing, well, computation. In practice, the computation is rarely that important. In industry, there is a lot more "information technology" than there is "computation." Information technology inherently needs to ingest and produce information, and while that was once facilitated by a department of Operators loading tapes, we have found the whole Operator thing to be costly and slow compared to real-time communications.

In other words, the modern business computer is almost primarily a communications device.

There are not that many practical line-of-business computer systems that produce value without interconnection with other line-of-business computer systems. These interconnections often cross organizational and geographical boundaries.

I am thinking, for example, of the case of airline reservation and scheduling systems disabled by the CrowdStrike, er, sorry, whatever I called them incident. These are fundamentally communications systems, and have their origins as replacements for the telephone and telegraph. It is not possible to simply not internetwork them, because networking is inherent to their function.

Networking is important to maintenance and operations

But let's consider systems that don't actually require real-time communications to perform their business purpose. Network connectivity still tends to be really valuable for these.

For one, consider maintenance: how does a system obtain software updates if you have no internet connection? How is that system monitored?

And even if you think you can avoid those requirements by declaring a system "complete" and without the need for any updates or real-time monitoring or intervention, business requirements have the frustrating habit of changing over time, and network connectivity reduces the cost of handling those changes tremendously.

What does it mean for a system to not be connected to the internet?

First, we need to consider the fact that there are as many forms of "not connected to the internet" as there are ways of being connected to the internet. For this reason alone, proposing that a system shouldn't be internet-connected is usually too nonspecific to really discuss. Let's consider a menu of possibilities:

List 1:

1. A single device with no network connection at all.
2. A system of devices that is "air-gapped" in the strictest sense, with no connection to any network other than its private local-area one, where data never crosses the security boundary.
3. That same system, but someone carries DVD-Rs across the security boundary to introduce new data to the private network.
4. That same system, but a cross-domain solution or "data diode" allows movement of data from a wider (or lower-security) network into the private (or higher-security) network.
5. That same system, where the cross-domain solution does not have a costly and difficult to obtain NSA certification.

List 2:

1. A system of devices which interconnect over a private wide-area network using fully independent physical infrastructure with physical precautions against tampering.
2. That same system, but the independent physical infrastructure is run through commodity shared ducts.
3. That same system, but the infrastructure is leased dark fiber.
4. That same system, but the infrastructure is wavelengths on lit fiber.
5. That same system, but the infrastructure is "virtual private ethernet" implemented by the provider using, let's say, MPLS.
6. That same system, but the infrastructure is "virtual private ethernet" implemented by the provider using a tunneling solution with encryption and authentication.

List 3:

1. A system of devices which interconnect over a common-carrier network (such as, we might even dare say, the internet), where private network traffic is tunneled through encryption and authentication performed by hardware devices.
2. That same system, but the hardware devices do not have a costly and difficult to obtain NSA certification.
3. That same system, but the tunneling is performed by a software solution that is well-designed such that it configures the operating system network stack, at a low level, to prevent any traffic bypassing the tunnel, and this has been validated by someone much smarter than me.
4. That same system, but not so well designed and validated by someone like me.
5. That same system, but the "software solution" is like Wireguard and an iptables script that has been "thoroughly tested" by someone on Reddit.

List 4:

1. A system of devices which interconnect on a private network that has interconnection to the internet that is strictly limited by policy-based routing or other reliable methods, such that only very narrowly defined traffic flows are possible.
2. That same system, but the permissible network flows are documented in some old Jira tickets and some of them were, you know, just thrown in to make it work.
3. That same system, but it's basically protected by a firewall that's pretty liberal about outbound flows (maybe with IPS or something), and pretty restrictive about inbound flows.

List 5:

1. An AWS private VPC without any routing elsewhere.
2. An AWS private VPC with PrivateLinks and other AWS networking baubles that allow it communicate with other private VPCs.
3. That same system, but some of the interconnected VPCs can route traffic to/from the internet.
4. An AWS private VPC with NAT GW and IGW but the security groups are set up pretty tight in both directions.

These are all things that I have seen described as non-internet-connected. Take a moment to work through each list and mark the point at which you think that is no longer a reasonable claim. It's okay, I'll wait.

I'm not going to provide threat modeling for all of these scenarios because it would go on for pages, but you can probably see that pretty much every option is at least slightly different in terms of attack surface and risk.

This might seem like an annoying or pedantic argument, but this is actually the biggest reason I get irritated when people say that something should never be connected to the internet. What do they mean by that? When someone says that an airline reservation system shouldn't be internet-connected, they clearly don't actually mean the strictest form of that contention (no network connection at all) unless their name is Adama and they liked when airline reservation centers had big turntables of paper cards they spun around to check off your seat. They must mean one of the midpoints presented above, which are pretty much all coherent positions, but all positions with different practical considerations.

This ambiguity makes it hard to actually, seriously consider the merits of dropping internet connectivity.

Non-internet connected systems are so very, very annoying

In my day job, I work with a wide variety of clients with a wide variety of cultures, IT architectures, and so on. Some of them are in highly regulated industries or defense or whatever, and so they actually conduct software operations in networks with either no internet connectivity or tightly restricted internet connectivity.

When I discover this to be the case, I mentally multiply all of the schedule/cost estimates by a factor of, I would say, 3 to 10, depending on where they fall on the above lists (usually 3x to 5x for list 5 and 10x to a bajillion times forever for list 1, just rule of thumb).

Here's the thing: virtually the entire software landscape has been designed with the assumption of internet connectivity. Your operating system wants to obtain its updates from online servers. If you are paying for expensive licenses for your operating system, the vendor probably offers additional expensive licenses for infrastructure to perform updates within your private network. If you are getting your operating system for free-as-in-beer, there's a good bet you can figure it out yourself, but if you're using anything too new and cutting-edge it might be a massive hassle.

But that just, you know, scratches the surface. You probably develop and deploy software using a half dozen different package managers with varying degrees of accommodation for operating against private, internal repositories. Some of them make this easy, some of them don't, but the worst part is that you will have to figure it out about fifty times because of the combinatorial complexity of multiple package managers, multiple ways of invoking them, and multiple environments in which they are invoked.

If you are operating a private network, your internal services probably don't have TLS certificates signed by a popular CA that is in root programs. You will spend many valuable hours of your life trying to remember the default password for the JRE's special private trust store and discovering all of the other things that have special private trust stores, even though your operating system provides a perfectly reasonable trust store that is relatively easy to manage, because of Reasons. You will discover that in some tech stacks this is consistent but in others it depends on what libraries you use.

A bunch of the software you use will want to perform cloud licensing and get irritated when it cannot phone home for entitlements. You will have to go back and forth with your vendors to figure out a workaround somewhere between "add these ninety seven /16s to your firewall exceptions" and "wait six months while we figure out the internal process to issue you a bespoke licensing scheme."

All of your stuff that requires updates or content updates will have some different process you have to follow to obtain those updates and then provide them internally. Here's a not at all made up example, but a real one I have personally lived through: you will find that a particular (and particularly hated) enterprise software vendor provides content updates for offline use only through a customer support portal that is held over from three acquisitions ago, and that it is only possible to get an account in that customer support portal by getting an entitlement manually added in a different customer support portal held over from two acquisitions ago. It will take over three months of support tickets and escalations through your named account executive to get accounts opened in successively older customer support portals until you can finally get into the right one, which incidentally has an invalid TLS cert you are reassured is not something to worry about. Once you download your offline content update, you will find that the documented process to apply it no longer works, and it will take a long email chain with one of the engineers to get the right instructions. You paid a five-figure sum for a 1-year license to this software and it has now nearly elapsed while you figured out how to use it. You will of course get an extension on that license pro bono, because this is enterprise software sales and what is a quarter worth of my salary between friends, but they won't manage to issue the extension license until after your original one has already expired, causing a painful interruption in CI pipelines and a violent revolution by the developers.

I am sorry, you are not my therapist, I will try to stop remembering that dark time in my career. Don't worry, the software in question seems to have fallen out of favor and cannot hurt you.

So, like, that's an over-the-top example (but seriously, a real one!), but you get the point. It's not really that any individual part of operating in an offline environment is hard---I mean some of them are, but most of them aren't. It's a death by a thousand cuts. Every single thing you ever do is harder when you do not have internet connectivity, and you will pay for it in money and time.

The largest problem by far is that almost everyone who develops software assumes that their product will not need to operate in an offline environment, and if they find out that it does they will fix that with duct tape and shell scripts because it only matters for a small portion of their customers. You, the person with the offline environment, will become the proud owner of their technical debt.

None of this really needs to be that way, it's just how it is! There are not really that many offline environments, and they tend to be found in big institutions that have adapted to the fact that they make everything cost more and take longer, and are surprisingly tolerant of vendors who perform a three stooges routine every time you say "air-gap," because that's what pretty much every vendor does. Except for like Red Hat, I genuinely think Red Hat is pretty good about this, but you betcha what you save in time you are paying in cash.

Not many people do this

That's kind of the point, right? The problem with non-internet-connected environments is that they are rare. The stronger versions, things from List 1 and List 2, are mostly only seen in defense and intelligence, although I have also seen some banks with pretty impressive practices. You will note that defense and intelligence, and even banks, are also famously industries where everything costs way too much and takes way too long. These correlations are probably not coincidences.

Even the weaker forms tend to be limited to highly-regulated industries (finance and healthcare are the big ones), although you see the occasional random software company that just takes security really seriously and keeps things locked down. Occasionally.

Okay, let's stop just complaining

Here's the thing: I genuinely do not think that "fewer systems should be connected to the internet" is a bad idea. I really wish that things were different, and that every part of the software industry was more prepared and more comfortable operating in environments with no or limited internet connectivity. But that is not the world that we currently live in! So let's get optimistic, what should we be doing right now?

1. Apply restrictive network policy on as much of your stuff as possible. Cloud providers generally make this easier than it has ever been before, it's not all that easy but it's also not all that hard to operate a practical non-internet-routed environment in AWS. If you stay within the lanes of all the AWS managed services, it's mostly pain-free. You will pay for this, but, you know, AWS always gets their check anyway.

2. Build software with offline environments in mind. Any time that you need to phone home to get something, provide a way to disable it (if practical) or a way to override the endpoint that will be used. If the latter, keep in mind that you will also need to come up with a way for a customer to feasibly host their own endpoint. If you keep to simple static files, that's really easy, just nginx and a directory or whatever. If it's an API or something, well, you're probably going to have to ship your internal implementation. Brace yourself for the maintenance overhead.

3. Try to think about the little assumptions that go into connecting to other services that become more complex in an offline environment. Please, for the love of God, do not assume you can reach LetsEncrypt. But that's not the only TLS problem, offline environments virtually always imply internal certificate authorities. Use the system trust store. Please. I am begging you.

4. Avoid fetching any kind of requirements or dependencies at deploy time. One of the advantages Docker supposedly brought us was making all of the requirements of a given package self-contained, but then I still run into Docker containers that can't start if they can't reach the npm repos or something. And now I have yet another place to fix configuration and trust store and etc., in your stupid Docker container. It has made things more difficult instead of less.

Have I mentioned that Docker, paradoxically, actually makes offline environments more difficult to manage? Yeah, because virtually every third-party Docker container has at least a TLS trust store you'll have to modify. Docker is, itself, a profound example of how the modern software industry simply assumes that everything is running On The Internet.

Anyway

I wrote this out in a bit of a huff because I have seen "why were they connected to the internet at all?" like four times in response to the CrowdStrike incident. I know, I am committing the cardinal sin of taking things that people on the internet say seriously, but I feel obligated to point out: internet connectivity is pretty much completely orthogonal to what happened. CrowdStrike content updates are the kind of thing that, in a perfect world, you would promptly make available in your offline environment. In practice, an internal CrowdStrike update mirror would probably lag days, weeks, months, or years behind, because that's what usually ends up happening in "hard" offline environments, but that's a case of two wrongs making a right.

Which they do, more often than you would think, in the world of information technology.

Don't worry, I'll be back next time with something more carefully written and less relevant to the world we live in. I just got in a mood, you know? I just spent like half the day copying Docker images into an offline environment and then fixing them all. I have to find something to occupy the time while a certain endpoint security agent pegs the CPU and makes every "docker save" take ten minutes.

A programming note: I am looking at making some changes to how I host things like Computers Are Bad and wikimap that are going to involve a lot more recurring expense. For that and other reasons, I want to see if y'all would be willing to throw some money my way. If everyone reading this gave $3 a month, we could probably buy Jimbo Wales a nice lunch or something.

I do not intend to paywall anything that I post here. Instead, I'm going to take some of the things that tend to be very long Mastodon threads (how this article originally started!) and send them out to supporters, probably about once a month. They'll be things that I wouldn't post here, usually because they're too short or don't quite have a hook to make a while article interesting.

So consider supporting me on ko-fi---or don't, it's your decision, and I respect that. I'll probably work a plug into each article but I promise not to be annoying about it. I'm also going to be making some new YouTube videos and I'll probably make those available to supporters first, so that's something else to look forward to.

Speaking of annoying, this is kind of a long and dry one. I started looking into something called HICS after visiting a historic site, posted about it a bit on Mastodon, realized that there just wasn't a lot of good historic information about it in general. And I felt like if I was going to talk about communications in Minuteman missile fields, I also had to cover how they would get their war orders.

Look on the bright side: it's got pictures!

Minuteman Missiles

Since the early days of the Cold War, the United States has maintained a nuclear triad: independent capabilities to deliver nuclear weapons from land, sea, and air. The sea and air components are straightforward, consisting of submarines carrying submarine-launched ballistic missiles (SLBMs) and long-range strategic bombers. The land leg is more often forgotten: the intercontinental ballistic missiles, or ICBMs.

The US ICBM arsenal currently consists of 400 Minuteman III missiles emplaced throughout the midwest: the Air Force's 90th Missile Wing, 150 missiles, Wyoming Nebraska, and Colorado; the 91st Missile Wing, 150 missiles, in North Dakota; and the 341st Missile Wing, 100 missiles, in Montana. Historically, there were as many as 1,000 active Minuteman missiles, to say nothing of retired missile programs like Titan and Atlas. At least three Minuteman missile facilities are now historic sites open to the public, a somewhat incongruous experience considering their broad similarity to the facilities still in active use. Many others are abandoned, typically in various states of permanent destruction to satisfy treaty obligations. Fifty Minuteman IIIs of the 341st are currently held in an inactive "reserve" state, out of service but ready for future emplacement, a notable situation given the strict limits Cold War treaties place on stockpiled ICBMs.

Minuteman employed a significantly different launch configuration from earlier ICBM programs. Large facilities were difficult to protect from a first strike; distance was the only effective protection from increasingly accurate Soviet weapons. Scattered single facilities were difficult and costly to staff. Minuteman selected a compromise point: clusters of ten independent Launch Facilities (LFs), spaced miles apart and called a "flight," are remotely monitored and operated from a single Missile Alert Facility (MAF). Groups of four to five missile flights constitute a squadron, and about four squadrons compose a wing, which is supported by an Air Force Base.

In each Missile Alert Facility, a Missile Combat Crew Commander (MCCC) and Deputy Missile Combat Crew Commander (DMCCC) lock themselves into an underground capsule called a Launch Control Center (LCC) for each 24 hour watch. Originally designed by Boeing, the LCC resembles the interior of an aircraft more than a building, fitting its crew of Air Force officers. The LCC is an isolated, self-contained system with all of the equipment needed to monitor, configure, and launch the missiles. A surface building above contains a security control center and quarters for the security force, responsible not only for the MAF but also for the ten LFs under its supervision. Still, the surface building is both powerless over and largely unneeded by the LCC beneath it. In the event of nuclear war, it was assumed, surface structures in these sparsely populated but strategically critical parts of the country would be wiped cleanly away from the earth. Only the hardened infrastructure would remain: the LCCs, the LFs, and communications infrastructure.

Minuteman missile combat crews had more duties than just to wait. They performed remote tests on the missiles, launch, and control equipment; they monitored alarm systems that reported malfunctions and remotely supervised the work of maintenance crews; and they monitored the security systems that protected the unmanned LFs, authorizing access and dispatching security forces on the surface to any unknown intrusion. Still, their primary responsibility, the one for which we all know them, is the disposition of Emergency War Orders (EWO).

In fine Air Force tradition, actions that may very well presage the end of the world as we know it are presented in the form of a checklist. EWOs are authenticated against codes and secrets. Two keys, just like in the movies, are inserted. The missiles are enabled by remote command. Targeting information is transmitted to the missiles. The keys are turned, the launch code sent, and the rest is automated, under the control of the missiles themselves. The missile crew are miles away, so the Real Thing, the Shit Hitting the Fan, must feel pretty anticlimactic. The only apocalyptic horsemen they'll see are a series of indicator lights on the MCCC's console: LCH CMD. LCH IN PROC. MISSILE AWAY.

The realities of ICBM operation are fascinating and incline one towards drama. Somehow the work of submarine and bomber crews seems more ordinary; they are at least "out there," in or near enemy territory. Missile crews are sealed in a very small room buried below a small building in a corner cut out of a farm field near, but not too near, to a highway for logistical convenience. They are entirely dependent on electronic communications, not only to receive their orders, but even to use their weapons. They have the original email job: since 1962, they have served primarily to send and receive messages, mostly by text.

With the rather purple introduction complete, I am going to talk about this communications technology. But first, just a little more preface.

Most of the reduction of the Minuteman force has been a direct response to treaties, which imposed progressively lower caps on the nuclear stockpile. Some reductions were more of a historical accident. In the 1980s, political considerations lead to a decision to "temporarily" deploy the Peacekeeper missile, with 10 MIRV (multiple independent reentry vehicle) warheads, to a set of fifty silos of the 90th Missile Wing's 400th Missile Squadron, in Wyoming. The Peacekeeper, fielded late in the Cold War, was a profoundly controversial program. The fifty Peacekeepers retrofitted into Minuteman silos would be the only ever installed, and their temporary homes became permanent. Most of the missile's warheads were removed for compliance with Start II treaty, which Russia never ratified and the United States withdrew from. Still, for cost-savings reasons, the odd-duck Peacekeeper program was terminated. The last Peacekeeper missiles were retired in 2005.

I got it into my head to write a detailed description of the missile field communications system because of my visit to QUEBEC-01, one of the five MAFs associated with these Peacekeeper missiles, and now a Wyoming State Historical Site. For that reason, I will most closely describe the communications system as installed in the 90th Missile Wing, the remainder of which is still active today. Minuteman missile fields were built over a period of years by different contractors and have since been through multiple modernization programs. Each change has introduced inconsistencies. While I will point out some of the more interesting variations between Minuteman installations, this is best taken as a description of the "average" Minuteman squadron, one that is typical of the others but does not exactly exist.

Although I am not exactly aiming for academic rigor, this information is based mostly on documents available through the Defense Technical Information Center, which include both original documentation from the Minuteman program and more recent documents related to modernization programs, proposed changes, and the retirement of many Minuteman facilities. I have supplemented those documents with recollections by former Air Force personnel when available, and as always, I welcome any corrections or additional information. One of the pleasures of writing about military history is the tendency of veterans to reach out to me with corrections and stories; I apologize that I am not always good about getting back to people, particularly phone calls.

The Minuteman III, a fairly direct evolution of the original Minuteman design, is still in active service. Many detailed materials about the Minuteman program are probably currently classified, most of the others are formerly classified and thus have not consistently made their way to archives. Certain basic questions remain frustratingly unanswered. That's just how it goes.

I'm also going to try really hard not to be too annoying with the acronyms, but it's not easy.

Personnel

While there were originally three-person crews, Minuteman LCCs have had two crew members for many decades, the MCCC and DMCCC. The MCCC is superior to the DMCCC, but the nature of missile operations and such a small crew mean that their roles are somewhat more complex than commander and deputy. Missile crews operate according to the "two-person concept," a general prohibition on any person working alone. This rule is intended to improve safety, reduce mistakes, and most importantly, mitigate the risk of an unauthorized launch. There are additional safeguards against unauthorized launch in the Minuteman system which will be discussed later. Both the MCCC and the DMCCC are required to initiate a launch.

The MCCC sits at a console that is focused around monitoring and control of the launch facilities. They have ready access to procedures and documentation. The DMCCC sits at a separate console that is focused on communications. Their chair slides on rails, allowing them to access the equipment racks and teleprinters to the sides of the DMCCC position. The DMCCC is primarily responsible for communications, so we are most interested in the equipment under their control.

Missile Alert Facilities were designed with a goal of self-containment for survivability. Most communications equipment is within the LCC itself, readily available to the DMCCC so that they can at least diagnose problems, if not make a repair. To this end, DMCCCs receive significant training on technical details of the communications and computer systems.

Should a problem occur outside of the LCC, the MCC would request assistance from the Air Force Base. Several Air Force ratings had expertise in communications equipment, ranging from communications technicians that would investigate problems within the LF to cable splicers that would repair damage in the outside plant.

The Minuteman program is somewhat unusual in the extensive construction of long-distance communications equipment by the Air Force. AT&T's role in the missile fields was surprisingly limited; most communications followed routes fully under the control of the Air Force.

External Communications

We can generally divide Minuteman communications systems into two categories: external and internal. External communications systems are primarily used by the Missile Combat Crew (MCC) to receive orders, including Emergency War Orders authorizing the use of nuclear weapons. Internal communications systems are used within the missile field, primarily to allow the MCC to communicate with the launch facilities under their control. Some details blur the lines: for example, there are communications systems which allow the MCC to contact their Air Force Base, where support facilities and maintenance crews are found. I will consider these internal systems, but you could argue for the opposite.

The external communications systems available to Minuteman crews have varied over time. Perhaps the most exotic was the Survivable Low Frequency Communications System (SLFCS), based on the LF equipment used by the Navy for communications with submarines. Missile facilities are not underwater, but nuclear detonations cause significant disruption to the atmosphere that greatly interferes with radio propagation in the HF range. LF communications are expected to be less affected in a nuclear combat environment. SLFCS specifically operated between 14kHz and 60kHz. Some, but not all, Minuteman MAFs were equipped with a magnetic loop antenna, about 6' in diameter, buried shallowly underground.

All MAFs were equipped with HF antennas, although they were decommissioned in the 1980s. The HF antennas are described as hardened, but it is not feasible to truly harden an HF antenna. They must be fairly large, and HF does not penetrate the ground well, making it impractical to bury them. Instead, hardened HF antennas are perhaps better described as "hidden" HF antennas. The typical design is a monopole that stores in a long, narrow silo underground, awaiting post-attack deployment. MAFs had two separate hardened HF antennas, one for transmit, and one for receive.

The receive antenna was the most critical, as it would be needed to receive war orders over the High Frequency Global Communications System (HFGCS). HFGCS is one of the primary ways that an Emergency War Order would be distributed to Air Force units including both missiles and bombers. The hardened HF receive antenna assembly actually included six 160' monopoles: one was extended for normal use, but in the event of nuclear attack, the five others were stored telescoped in a silo about 30' deep and could be deployed by a small explosive charge. There were, in the parlance apparently used by the Air Force, five "reloads."

The HF transmit antenna, being less important in an attack scenario, had only one replacement. A "soft" HF transmit antenna of conventional design was in normal use but backed up by a single 120' hardened antenna stored in a separate silo. A 50' radius buried ground plane surrounded the hardened antenna.

Near the MAF surface building, a small, white metal cone protrudes from the ground. The cone consists of a huge cast steel blast deflector with a depression in the center, which is covered by a fiberglass cone. The cone houses a compact UHF antenna. This antenna, a 1970s upgrade, can receive war orders via several satellite systems or directly from an aircraft such as the E-6. The E-6 airborne command post can serve in various roles, including as a "Looking Glass" airborne command post (taking control of nuclear forces in the event of a loss of ground-based command posts) or a "TACAMO" Take Charge and Move Out relay, transmitting a war order from elsewhere to forces in the field.

UHF communications are essentially line of sight, especially with the use of a partially in-ground hardened antenna. In a scenario with extensive loss of communications infrastructure, particularly with ASAT warfare disabling military communications satellites, an E-6 or another similarly equipped aircraft would fly over Minuteman missile fields and deliver an emergency war order directly to each LCC.

Complimenting this capability, in-service Minuteman launch facilities have themselves been equipped with a similar UHF antenna. Looking Glass and TACAMO aircraft actually have Air Force missileers on board who serve as an Airborne Launch Control Center (ALCC). In the event of a loss of most of the LCCs in a missile field, an ALCC can issue launch instructions directly to each LF without any need for the regular missile crews or internal communications infrastructure.

During the early '90s, a super high frequency (SHF) small satellite terminal was installed at each active MAF. It is housed in a small, white radome at the top of a pole near the surface building. SHF is widely used by modern military satellite systems, such as the Air Force's Wideband Global SATCOM.

Each of these antennas is connected by buried conduits to radio equipment in the LCC's radio racks. At the DMCCC's position, the telephone console allows the DMCCC to talk or listen on each radio by selecting it with a pushbutton. Over time, the radios were also attached to more modern digital systems. Depending on the year, teleprinters or computer displays would receive text messages via the UHF and SHF radio systems.

The external radio systems were actually all backup or secondary. The primary means of nuclear C2 within the Strategic Air Command and, later, Global Strike Command, has long been a digital computer network. While Minuteman installations began with just a teleprinter to receive orders via leased telephone line, the late '60s saw the introduction of the Strategic Automated Command and Control System (SACCS), which was itself replaced by the Strategic Air Command Digital Network (SACDIN). A small (for the era) computer in a rack to the right of the DMCCC's station allowed two-way messaging with SAC headquarters over leased telephone lines. Reportedly, these and other Bell System telephone lines to LCCs were carried by buried telephone cable to small hardened exchange buildings serving each missile field. I have not yet researched this topic closely.

In Peacekeeper LCCs as well as Minuteman LCCs from the '70s to the '90s, the specific computer used for this purpose was called the Command Data Buffer (CDB). The CDB was connected to both SACCS/SACDIN and the internal communications network, in order to accurately relay targeting information to the missiles. This will be discussed later in the context of rapid retargeting. In the 1990s, the REACT system was installed for a similar purpose.

Quebec-01 LCC was equipped with a teleprinter with a selector between SACDIN and AFSAT (UHF satellite) receivers. I'm not sure why the teleprinter was retained after the CDB upgrade, very possibly just for redundancy.

Somewhere between external and internal, each LCC had access to two dial telephone lines. These connected directly to the PSTN. Some circumstantial evidence leads me to think these dial lines were shared with the surface building, which probably explains the need for two. The dial lines were mostly used to contact support crews at the Air Force Base for routine maintenance issues.

HICS Digital Communications

I was most interested in the internal communications systems, which have garnered less historical documentation than the external links. In most cases, there is only really one: the Hardened Intersite Cable System, HICS.

HICS consists of multipair pressurized telephone cables trenched between Minuteman facilities. HICS carries digital traffic for C2, and many Air Force documents use the term "HICS" exclusively to mean the digital channel, but the same cables carried multiple voice pairs. Let's consider the digital capability first, though.

HICS, as originally installed, operated at 1.3Kbps. Details on the actual encoding are hard to come by, but given the time period I assume it was generally similar to the AFSK schemes used by other early telephone data links. The punch line, of course, is that the 1.3Kbps stuck---based on some Air Force journal articles on options for upgrades, it seems that contemporary Minuteman III fields still communicate over HICS at 1.3Kbps. Remember that when we get to retargeting.

The topology of the digital HICS network is rather interesting. It was designed for redundancy and reliability, but prior to most of our modern understanding of computer networking. There's a mix of a few different ideas.

One of the things I'm not completely confident of is the size of the collision domain within HICS, or how much of the cable network was a common bus. From reading between the lines of some different reports and considering the overall design, I'm fairly confident that the entire digital HICS network was a single shared bus within each flight, and I think it is likely that it was a shared bus within each squadron. This bus would be tens of miles long with multiple branches, a challenging electrical situation that perhaps explains why the Air Force has repeatedly found it to be infeasible to make HICS faster.

Thanks to minutemanmissile.com for this image of the Warren AFB/90th Missile Wing HICS map, which is far more legible than the photo I had taken.

Each of the LCCs, denoted in the map by open rectangles, is connected to four "loops" of HICS cable. The legs of adjacent loops to the LCC are shared, though, so it's perhaps easier to describe this way: each LCC is surrounded by a ring of HICS cable, to which it is connected by four legs spread roughly 90 degrees apart. This design gives a fair amount of redundancy, a break anywhere in the ring or even breaks of more than one of the LCC's legs would still leave a working path.

This latter scenario was probably one of the designer's greatest concerns, as the LCCs would be obvious targets for inbound nuclear attacks. The cable layout provides four-times redundancy on the cables to the LCC, but no redundancy at all on the cables to individual LFs. That tells you a lot about their threat modeling. Facilities were spaced far enough apart that a precision strike on an LF would probably disable only that single LF; a precision strike on an LCC, though, could potentially disable ten LFs at once. As the accuracy and power of nuclear warheads improved, it seemed more likely that a first strike would succeed in disabling at least some LCCs. A lot of the complexity of HICS is intended to account for that possibility.

Each of the LFs is connected to the ring via a single leg. In some cases, multiple LFs are along the same leg. These are often the same long runs that connect the rings of two different LCCs together. In general, each LCC ring is connected to two of its neighbors, although sometimes it will instead have two redundant connections to a single neighbor. Based on the map and situation on the ground, these inter-flight connections don't seem to have required any active equipment, only a splice case. That supports the theory that an entire squadron was a shared bus, although it's possible that a separate pair to the "foreign" LCC ring would home-run to the LCC to allow separately sending messages to either. The network doesn't seem to have that kind of selective routing capability, though, so I find it unlikely.

Digital messages do seem to have been packetized, and were distributed through the network on a "flood fill" basis. That is, every active node on the digital network repeated every message it received. You might wonder about flow control and the avoidance of cycles; only a very primitive method was used. Each node, after transmitting a message, would "lock out" the cable it was transmitted on for a long enough period for the node on the other end to finish repeating the message.

This explanation is a little more difficult to understand, though, when applied to the actual layout of the HICS system. What exactly constitutes a node? You will note that the map distinctly shows intersquadron connections, with both thicker lines and open circles where they connect to an LCC ring. My theory is that these intersquadron connections where the only places where active repeating of messages was required. Whether or not repeating messages between squadrons was selective is unclear. Did a message from an LCC to one of its nearby LFs get repeated across squadrons to the opposite end of the field? If repeating had originally been completely non-selective, I suspect that was changed as part of the work done to facilitate retargeting.

We can infer certain things about the HICS network from the equipment in Quebec-01. For example, HICS must have had a fair number of active repeaters. along a path.

Inside of the LCC, we find something like a tiny long-distance telephone test desk. A rack includes pressurization alarms for five cables (we know of four legs to the LCC ring, is the fifth perhaps a cable to the local telephone exchange?), and a fault isolation panel. When a cable seemed to have been lost, the DMCCC could use this panel to locate the problem along the cable. This probably relied on a loopback test feature of the repeaters, but I'm not sure exactly the operating principle. Further down in the rack is what appears to be a cable power supply.

Repeaters were definitely installed inside the LCCs, but the number of selections on this test panel makes me suspect that there were also in-line repeaters on the cable, perhaps taking power from that power supply. This is entirely speculative, but A repeaters may have been located along the LCC ring and B repeaters on legs, making the two-knob selector arrangement useful to test a specific repeater on a specific leg.

In the equipment side of the LCC, where the generator and chiller are located, we also find the terminations of the HICS cables. Note the mostly empty rack that would have held repeater equipment, and the air dryer and flow gauges for cable pressurization.

Finally, I should talk a bit about the exception to all of this: the 321st Missile Wing, in North Dakota, was built later than the 90th and 91st and by a different contractor. Sylvania, not Boeing, won the bid to build the LCCs and LFs. Much of the equipment is the same, but Sylvania did inject a few of their own ideas, and one of them was radio redundancy for HICS. The 321st apparently had a simplified HICS topology; I'm not sure of the details but I would guess that they may not have provided the four redundant cables to each LCC.

To make up for it, each LCC and LF in the 321st is equipped with a large, buried antenna, a grid-like arrangement of crossing dipoles that took up an area similar to the sewage lagoons outside of the fence. These antennas made up a medium-frequency, ground-wave communications network that could be used as an alternative to HICS. The 321sts redundant radio system, apparently called "Deuce" at the time, could be viewed as a precursor to the later nationwide GWEN radio C2 network. It seems to have carried the same digital messages as HICS, and the DMCCC had selectors to choose whether messages would be sent by cable or radio.

HICS Voice Communications

Now, let's take a close look at the DMCCC's communications console, which tells us a lot about the voice capabilities.

They get a lot of buttons! There appear to be two separate busses, I'm not completely sure of the significance of that layout. I know from an airman's anecdote that the DMCCCs could conference together LF phones and the dial telephone lines, and sometimes did so that maintenance crews stuck at an LF overnight could make apologies to their families. This makes me think that it is not a matter of "one selection per bus," but rather probably indicates that lines can only be joined within a bus. The logic behind that design is not clear to me.

Anyway, let's see if we know enough to explain all of these buttons. Some are easy: for the speaker and handset, there are selectors each of the radios. The "LF Lines" correspond to each of the ten LFs, numbered 2-11 since the LCC is numbered as site 1. We see the two dial lines, regular telephone lines provided at each site, and they are even labeled with their phone numbers. The five-digit notation dates this hand-written addition to the 2L-5N era, which probably persisted unusually late in rural Wyoming.

The rest of the buttons correspond to specific pairs that would emerge in different places in the HICS network. The "SCC" button likely allows communications with the security command center in the surface building, just up the elevator from the LCC. The "LCC" button I am less sure of; perhaps it was a party line of other LCCs in the squadron? The "LCC Ring" selections must correspond to the four HICS cable rings extending from the LCC, but I'm not sure which devices would be found on those pairs. They may be "order wires," available in the splice boxes and as jacks at sites and normally used only by maintenance crews working on cables outside of the LFs.

The EWO buttons are interesting. EWO is, of course, Emergency War Order, but in the context of Minuteman was also the term used for party lines connecting the Air Force Base to the LCCs. These could be used, of course, as a redundant way to deliver EWOs, as well as for general communications across the missile field. There are two for redundancy: one was routed via AT&T infrastructure, following a cable from the LCC to a telephone exchange. The other was routed via HICS. I am not sure why only one merits a "RNG" button, that could apply ringing voltage to get the attention of other stations.

I am assuming, by the magic of speculation, that the "OPR" button on the left of each bus probably selected which bus the headset was connected to. There is also a dial, for use with the dial lines.

These voice connections within the field were of critical importance because of Minuteman's strict security posture. The unattended LFs were equipped with intrusion alarms for physical security, initially a bistatic radar system more similar to that used at Titan, and later a DSP-based monostatic radar system called the IMPSS. Any personnel or, reportedly, large rabbits approaching an LF would cause an alarm, and security forces were dispatched to investigate unless the intruder used a HICS voice circuit to authenticate themselves to the LCC. The process of "penetrating" a secure LF could take a missile crew thirty minutes or more, and involved multiple calls to the LCC as different alarms were triggered.

HICS Outside Plant

Old aerial images and historical documents from the Air Force give us some insight into the construction of HICS. HICS cables were installed in open trenches and then covered, rather than placed directly with a vibratory plow as would become common later. Splices were done in large holes with scotch-lok connectors and cast iron splice casings. Over time, many of the splice casings had to be replaced due to premature corrosion, and different materials were tried before settling on brass. A cathodic protection system was installed as a permanent solution to the problem.

I have done my best to trace some of the HICS cables along their routes. The holes used for splicing are sometimes visible as scars, but it does not appear that any manholes were installed; instead splices were made near RoW markers and will have to be excavated for repairs. The lack of manholes suggests that there may not be active equipment along the cable routes, I'm not really sure.

The RoW markers used by the Air Force are substantially similar to the style used by AT&T at the time. They are round wooden posts, about 6' tall, with metal bands around the top. Unlike AT&T, the Air Force used white bands, and it seems that there are always two. Shorter markers are used in some places, I suspect where there are splice cases not near road crossings. Where the cables cross roads, the Air Force usually installed gates with sturdy metal posts in the roadside fences. Sometimes these gates are the easiest evidence to find in aerial photos.

One of my biggest questions is about the inter-squadron relays. The map depicts them as nodes, but they aren't located at LFs or other facilities. I wondered if there might be active equipment, but I found one of the locations where in inter-squadron cable takes off from an LCC ring and there is no indication of even a manhole. In case you might be interested, here is a KML with the cable tracks I have worked out so far.

Cross-Flight Communications

HICS served primarily to allow an LCC to communicate within the ten missiles in its flight. However, the entire squadron and then the entire missile wing were interconnected, and Minuteman took advantage of this capability for several purposes.

First, a specific LCC in each squadron was designated as the Squadron Command Post (SCP). The SCP was capable of sending launch orders to any missile within the squadron, and of countermanding launch orders issued by any of the squadron's LCCs. This provided a measure of protection against a destroyed or compromised LCC.

Over time, the security of Minuteman missiles was further enhanced by the addition of a "vote to launch" system. Minuteman missiles can only be launched if launch orders are sent by at least two LCCs, requiring a total of four individuals.

In some Minuteman fields (and all current fields), a Wing command post serves in a role similar to the SCP but across the entire wing. It provides one central point where the entire wing's missile inventory can be monitored and, if necessary, controlled.

Alarms

Besides missile C2 and voice communications, one of the main functions of the HICS was the reporting of alarms within unattended LFs to the LCC. Some alarms, particularly related to the missile itself, would be sent by the missile guidance computer over the HICS digital network. These alarms would be printed by the teleprinter below the DMCCC's desk, along with confirmations of commands received and other routine traffic.

There was also a second, dedicated alarm system called the Voice Reporting Status Assembly or VRSA. VRSA seems to have relied on its own pairs in the HICS cables, and resembled the simple alarm reporting telephones coming into use by AT&T. The DMCCC could select an LF and press a button to send a tone, which triggered a device at the LF to "read back" any status alarms via voice recordings. At the time this almost certainly involved some interesting magnetic tape equipment, but I haven't found much information on the LF end of the VRSA. Toggle switches on the VRSA console allowed the DMCCC to reset the alarm device in the LF, clearing any recorded alarms that weren't active.

The VRSA was an upgrade over the original Minuteman installations, which used a very similar panel to send a safe/arm tone to the LFs as part of the launch process. Since part of standard testing practice was for DMCCCs to flip the toggle switch for an LF to remove the "safe" tone and arm the site, it was a fairly obvious evolution to have the site report any faults in response to a tone. Reportedly, the VRSA panels were the original safe/arm panels with modifications.

Retargeting

When Minuteman was originally installed, each missile's targeting data was loaded from tape using equipment in the LF. To retarget a missile, a maintenance crew had to travel to the site, access it, run the new target tape through equipment in the LF that sent the data to the guidance computer, and complete a recalibration of the inertial reference platform in the missile. This was something like a 12-hour process overall, and retargeting a squadron would take weeks.

Fixed targets were practical when the "enemy" was self-evidently the Soviet Union and any attack would be all-out. Over the 70-year lifespan of the Minuteman program, though, the geopolitical and military environment has changed. There are now other nuclear adversaries, and their military assets are increasingly mobile. The biggest challenge to Minuteman's targeting was the Soviet Union's development of road-mobile ICBMs like the RT-21. To eliminate the USSR's nuclear capability, we would have to fire on these mobile systems wherever they were located. Aerial and satellite surveillance could be surprisingly effective in keeping track of these large, slow-moving TELs, but the Minuteman missiles could not be retargeted to keep up with that intelligence.

In response, a series of enhancements were made (often as part of the Minuteman II program) to introduce "rapid retargeting." Rapid retargeting allowed the missiles to be retargeted from within the LCC. During the 1970s, a computer system called the Command Data Buffer (CDB) was installed in each LCC. The CDB could receive targeting parameters from SAC and then transmit them to the LFs. It was theoretically possible to retarget missiles shortly before launch. In practice, the "shortly" wasn't very achievable

The HICS network was capable of 1.3Kbps, and because of the "flood" design of the network, that was essentially 1.3Kbps of total capacity across a single collision domain. In other words, only 1.3Kbps of total traffic could be handled, with far less available from point to point when the network is under heavy use. Further, enhancements to the Minuteman system added cryptographic authentication of messages over HICS and, later, encryption of the messages themselves. The added overhead of the cryptographic system further reduced network capacity.

Retargeting a squadron of Minuteman missiles via the CDB took over 20 hours. Retargeting a single missile could take 30 minutes.

CDB represented a major step forward in Minuteman C2, particularly with its real-time messaging capability. Retargeting was still a severely limited capability, though.

During the 1990s, active Minuteman sites were upgraded to the Rapid Execution and Combat Targeting System, or REACT. More than just an upgrade for retargeting, REACT brought a completely new control system that significantly changed the layout of LCCs. Instead of sitting at opposite ends of the tube, REACT put the MCCC and DMCCC directly alongside each other and centralized almost all control functionality onto computer displays.

It also further refined retargeting: retargeting an entire squadron now takes only ten hours. More radically, though, a single missile can be retargeted in only a couple of minutes, making it feasible to retarget a missile just before firing in a limited attack scenario.

The future of HICS

While both over budget and behind schedule, the Sentinel program is expected to replace the Minuteman missiles. Sentinel will likely be an in-place upgrade, installing new missiles and control systems in the existing Minuteman silos. It has been clear for decades now that HICS isn't capable of meeting modern expectations, so Sentinel will include a complete replacement.

Various options including DSL over HICS cables and radio were considered, but the current plan is to trench new fiber-optic cables across the launch fields. They're less interesting, but fiber optic cables have both capacity and reliability advantages over telephone cables, and could easily remain in service for the life of the Sentinel program.

Cathode Ray Dude, in one of his many excellent "Quick Start" videos, made an interesting observation that has stuck in my brain: sophisticated computer users, nerds if you will, have a tendency to completely ignore things that they know are worthless. He was referring to the second power button present on a remarkably large portion of laptops sold in the Windows Vista era. Indeed, I owned at least two of these laptops and never gave it any real consideration.

I think the phenomenon is broader. As consumers in general, we've gotten very good at completely disregarding things that don't offer us anything worthwhile, even when they want to be noticed. "Banner blindness" is a particularly acute form of this adaptation to capitalism. Our almost subconscious filtering of our perception to things that seem worth the intellectual effort allows a lot of ubiquitous features of products to fly under the radar. Buttons that we just never press, because sometime a decade ago we got the impression they were useless.

I haven't written for a bit because I've been doing a lot of traveling. Somewhere in the two thousand miles or so we covered, my husband gestured vaguely at the headliner of our car. "what is that button for?" He was referring to a button that I have a learned inability to perceive: the friendly blue "information" button, right next to the less friendly red "SOS" button. Most cars on the US market today have these buttons, and in Europe they're mandatory (well, at least the red one, but I suspect the value-add potential of the blue on is not one that most automakers would turn down). And there's a whole story behind them.

It all started in 1996 at General Motors. Wikipedia tells us that it actually started with a collaboration of General Motors (GM), Electronic Data Systems (EDS), and Hughes Electronics. That isn't incorrect, but misses the interesting point that EDS and Hughes were both subsidiaries of GM at the time. GM was a massive company, full of what you might call vim and vigor, and it happened to own both a major IT services firm (EDS) and a major communications technology company (Hughes). It was sort of inevitable that GM would try integrating these into some sort of sophisticated car-technology-communications platform. They went full-steam ahead on this ambitious project, and what they delivered is OnStar.

But first, a brief tangent into corporate history. I won't say much about GM because I am not an automotive history person at all, but I will say a bit about EDS and Hughes. Hughes was obviously the product of a notable and often enigmatic figure of history, Howard Hughes. GM owned Hughes Electronics because Howard Hughes had cleverly placed his business ventures under the ownership of a massive and brazen tax shelter called the Howard Hughes Medical Institute. Howard Hughes died without leaving a will or any successor as trustee of HHMI, putting HHMI into an awkward legal and organizational struggle as it abruptly pivoted from "Howard Hughes' personal tax scheme" to "independent foundation that incidentally owned a major defense contractor." HHMI ultimately made the decision to turn Hughes' business empire into an endowment, and sold Hughes Aircraft. General Motors was the high bidder. A set of confusing details of the Hughes amalgamation, like the fact that Hughes Aircraft didn't own all of the Hughes aircraft, lead to the whole thing becoming the Hughes Electronics subsidiary of GM.

After GM essentially stripped it for parts, Hughes Electronics lives on today under the name DirecTV. The satellite internet company that actually markets under the name Hughes is, oddly enough, one of the parts that GM stripped off. Hughes Communications became part of EchoStar, operator of Hughes Electronics competitor DISH Network, which then spun Hughes Communications out, and then bought it back again. You can't make this stuff up. The point is that the strange legacy of Howard Hughes, the HHMI, and GM's ownership of Hughes Aircraft mean that the name "Hughes" is now sort of randomly splashed across the satellite communications industry. It's sort of like how Martin Marietta still paves freeways in Colorado.

Electronic Data Systems is not quite as interesting, but it was run by two-time minor presidential candidate Ross Perot, so that's something. GM dumped EDS almost immediately after launching OnStar. EDS eventually became part of Hewlett Packard, which, by that time, had become a sort of retirement home for enterprise technology companies. It more or less survives today as part of various large companies that you've never heard of but have nonetheless secured 9-digit contracts to do ominous things for the Department of Defense.

What a crowd, huh? It's a good thing that nothing strange and terrible happened to General Motors in approximately 2008.

So anyway, OnStar. OnStar was, basically, a straightforward evolution of the carphone backed by a concierge-like telephone service center. In that light, it's an unsurprising development: the carphone was just on its way out in the mid-'90s, falling victim to increasingly portable handheld phones. Hughes, by its division Hughes Network Systems, was an established carphone manufacturer but seems to have had few or no offerings in the mobile phone space [1]. To Hughes long-timers, OnStar was probably an obvious way to preserve the popularity of carphones: build them into the car at the factory, with factory-quality finish.

GM had their own goals. Ironically, it is in large part due to GM's efforts that built-in telephony is so common (and yet so ignored) in cars today. The situation was much different in 1996: OnStar was a new offering, only available from GM. It had the promise of competitive differentiation from other automakers, but for that to work, GM would have to differentiate it from the carphones widely available on the aftermarket. This tension, the conflict between "we built a carphone in at the factory" and "carphones are going out of style," probably explains why OnStar marketing focused on safety and security.

"General Motors has come up with the ultimate safety system" lead a '96 newspaper article. Marketing materials prominently positioned roadside assistance, automatic emergency calls on airbag deployment, remote door unlock, and locating stolen cars. These were features that your average carphone couldn't offer, because they required closer integration with the vehicle itself. OnStar was more than a carphone, it was a telematics system.

"Telematics" is one of those broad, cross-discipline concepts that we don't really talk about any more because it's become so ubiquitous as to be uninteresting. Like Cybernetics, but without a tantalizing but lost historical promise in Chile. Telematics has often been more or less synonymous with "putting phones into cars," but is more broadly concerned with communications technology as it applies to moving vehicles. There is a particular emphasis on the vehicle part, and telematics has always been interested in vehicle-specific concerns like positioning, navigation, and the collection of real-time data.

Telematics was already a developed field by the '90s, although the high cost and large size of communications equipment made it less universal than it is today. OnStar would lead one of the biggest changes in the modern automotive industry: the extension of telematics from commercial and industrial equipment to consumer automobiles. In doing so, it would introduce select GM drivers to an impressive set of benefits, almost a form of ambient computing. It would also start a cascade of falling dominoes that lead, rather directly, to a remarkable lack of privacy in modern vehicles and getting an email that something mysterious is wrong with your car two to four hours after the tire pressure light comes on. The computer gives and takes.

And what of EDS? EDS provided the other half of OnStar's differentiation from a mere carphone. OnStar was not only integrated into your vehicle, it was backed by a team of Service Advisors with training and tools to use that integration. The OnStar equipment included a GPS receiver, still a fairly cutting-edge technology at the time, and continuously provided your location to the OnStar service center in Michigan. Advisors had access to maps and travel directories and the ability to dispatch tow trucks and emergency responders. They could even send a limited set of remote commands to OnStar vehicles. The infrastructure to support this modern telematic call center was built by EDS, and the staff of human advisors provided a friendly face and a level of flexibility that was difficult to achieve by automation alone.

Besides emergencies and roadside assistance, the advisors could solve one of the most formidable problems in automotive technology: navigation. When GM's advertising and press coverage strayed from emergency assistance, they focused on concierge-like services focused around navigation. OnStar could direct you to gas or food. They could not only reserve a hotel room, but get you to the hotel. If you have seen the wacky turn-by-turn navigation technology that proliferated in the late 20th century, you might wonder how exactly that worked. Did an advisor stay awkwardly on the line? No, of course not, that would be both awkward and costly. They read out driving directions, which the OnStar equipment recorded for playback.

I really wish I could find a complete description of the user experience, because I suspect it was bad. The basic idea of recording spoken guidance and playing it back for reference is a common feature in aviation radios, but that's mostly for dealing with characteristically terse and fast-talking ground controllers, and usually consists of a short playback buffer that always starts from the beginning. Given the technology available, I suspect the OnStar approach was similar, but just with a... longer playback buffer. Thinking about listening through the directions over and over again to find one turn gives me anxiety, but it was 1996.

Technology advanced like it always does, and by the mid '00s at least some GM vehicles had the ability to display turn-by-turn instructions, provided by OnStar, as the driver needed them. Fortunately there are videos from this era, so I know that the UX was... better than expected, but strange. It's odd to see an LCD-matrix radio display, with no promise of navigation features, start displaying large turn arrows and distances after an OnStar call. One of the interesting things about OnStar is that the "human in the loop" nature of OnStar features makes it sort of a transitional stage between cassette tapes and Apple CarPlay. OnStar allowed human operators and remote computer systems to do the hard parts, allowing cars to behave in a way that seemed very ahead of their time.

One of the interesting things about OnStar, given the constant mention of satellites in its marketing, was the lack of actual satellite communications. Hughes, a satellite technology company, was involved. Articles about OnStar coyly refer to satellite technology, or say it's "powered by satellites." Of course, OnStar cost $22.50 a month in 1996, and $22.50 a month didn't entitle you to so much as look at a satellite phone in 1996. The satellite technology was limited to the GPS receiver; all voice communications were cellular. AMPS, specifically. The first several generations of OnStar, into the early '00s, relied on AMPS.

Telematics, telemetry, and the applications we now call "IoT" often struggle with the realities of communications networks. AMPS, often just referred to as "analog," was the first cellular communications standard to reach widespread popularity. For over a decade, everything cellular used AMPS. Then CDMA and GSM and even, may we all shed a tear, iDEN took over. These were digital standards with improved capacity and capabilities. It was inevitable that they would replace AMPS, and with the short lifespan of a consumer cellular phone, devices without support for digital networks naturally faded away... except for a bunch of them. OnStar and burglar alarms are two famous AMPS-retirement scandals. The deactivation of AMPS networks in 2008 left cars and alarm communicators across the country unable to communicate, and prompted a series of replacement programs, lawsuits, trade-in deals, lawsuits, and more lawsuits that are influential on how cellular networks are retired today (meaning: as rarely as possible).

The obsolescence of OnStar equipment in older vehicles by AMPS retirement left a black mark on OnStar's history that still hangs over it today. It was, I think, a vanguard of the larger impacts of fast-changing technology being integrated into cars. While vehicles have indeed become more reliable over time, there is an ever-present anxiety that new cars are more like consumer electronics, built for a three-year replacement cycle. The forced retirement of half a million OnStar buttons is probably one of the most visible examples of automotive equipment failing due to industry change rather than age.

In 2022, 2G cellular service was retired in the United States. With it went another generation of OnStar-equipped vehicles. For a combination of reasons, though, both a more conservative approach to 2G retirement in the cellular industry and likely GM's planning further ahead, only two model years were impacted.

Incidentally, Ford also had an offering very much like OnStar, called RESCU and introduced in 1996 as well. It was pretty universally agreed by automotive journalists at the time that RESCU was more primitive than OnStar and amounted mostly to a knee-jerk "we also have one of those" response to GM's launch. RESCU is perhaps worth mentioning, though, for its contribution to the lineage of Ford's SYNC platform, at least in the form of gratuitous all-caps.

In 2002, GM offered OnStar for licensing to other automotive manufacturers. Subarus, among others, began to sprout blue buttons in the overhead. But what had happened to competitive differentiation? Well, automotive technology tends to go through two phases: First, it differentiates. Second, it's mandated. The originating manufacturer can make quite a bit of money off of both.

In 1995, a year prior to the launch of OnStar, the National Highway Transportation Safety Administration (NHTSA) was already investigating the possibility of an Automated Collision Notification (ACN) system. ACN would automatically call 911 in the event of a dangerous crash, improving driver safety. As far as I can tell, GM is not the origin of the ACN concept. NHTSA's work on ACN started with the National Automated Highway System (NAHS), an ambitious technology development program launched in 1991 that imagined a very different self-driving car from the ones that we see today. The NAHS involved mesh networking between automated vehicles to form "platoons," close-following cars (for fuel efficiency) that synchronized their control actions. The mesh network would extend to road-side signaling systems, and would lead eventually to the end of traffic signals as cars automatically negotiated intersection time slots.

The NAHS never came to be and probably never will, but the NHTSA's retro-futuristic graphics of '90s sedans linked by blue waves echo through my childhood like they do through the pages of Popular Mechanics and the academic literature on self-driving. Or, they did, until a new generation of Silicon Valley companies coopted self-driving for their own purposes. This is not an entirely fair take on the history, I am certainly applying rose-colored (or is it cerulean blue?) glasses to the NAHS, but I think it is hard to argue that there has not been a loss of ambition in our vision of the self-driving future. For one, we stopped drawing blue waves on everything.

Anyway, GM may not have created ACN. If anyone, I think that honor might fall on Johns Hopkins University. But they sure did get involved: by 1996, the year OnStar launched, Delco Electronics was building ACN prototypes for NHTSA. Delco Electronics, a division of GM (Delco's history is closely intertwined with that of Hughes in this period, parts of Delco were and would be parts of Hughes and vice-versa). Over the following years, GM really jumped in: OnStar was ACN, and ACN should be mandatory.

Here's the thing: it's never really worked. The move of introducing a technology and then pushing for it to become mandatory is a fairly well-known one in the automotive industry, and to its credit, has lead to numerous safety advancements in consumer cars and no doubt a meaningful reduction on fatalities (to its discredit, it is often cited as one of the reasons for steeply rising prices on new cars).

Universal OnStar has come tantalizingly close. Europe mandated "eCall," functionally identical to ACN, in 2018. I'm not sure how directly GM was involved, but there are GM patents in the licensing pool required to implement eCall, so it's at least more than "not at all." But despite its increasing presence, ACN isn't required in the US. Automakers aren't even consistent on whether it's standard or a paid add-on.

GM is still hacking away at this one... as recently as last year, GM was taking federal grants to study ACN and propose standards. In collaboration with CDC, GM developed a system called AACN that uses accelerometer data to predict the severity of injury to occupants and difficulty of rescue. It's installed in newer OnStar vehicles, and Ford has even licensed it for Ford SYNC, but the data rarely goes anywhere at all... 911 PSAPs that receive the calls from ACN systems aren't equipped to receive the extra metadata; extensions to E911 to facilitate AACN data exchange are another thing that GM is actively involved in.

GM really seems to have put a 30-year effort into mandating OnStar in the US, but they just can't get it over the finish line. In the mean time, OnStar has stopped mattering.

GM's program to license OnStar to other automakers was short-lived. I'm not sure exactly why, but GM also gave up on their "OnStar For My Vehicle" aftermarket product. Even as OnStar continued to gain features, its ambition waned. I think that the problem was simple: by the mid-2000s, putting a phone into a car was becoming pretty easy. Besides, the "connected car" offered too many advantages for any automaker to turn down. Can you imagine the benefits of storing location history for the entire fleet of vehicles you've sold? You can sell that to the insurance industry! You know GM did, of course they did, and of course it's the subject of an ongoing class-action lawsuit.

OnStar just stopped being special. I was actually a little surprised to notice that the blue button in the overhead of my modern Subaru isn't an OnStar button; Subaru stopped licensing OnStar in '06. It's just another manifestation of Subaru StarLink, a confusing menagerie of vaguely-telematic features that are mostly built on contract by Samsung. Once the car has an LTE modem for remote start and maintenance telemetry and selling your driving habits to LexisNexis, throwing in a button that makes a phone call is hardly an engineering achievement.

You know, sometimes it feels like smartphones can only incidentally make phone calls. With the move to VoLTE, it's not even really a deeply-embedded functionality any more. "Phone" is just an application on the thing that, for reasons of habit, we call a "phone."

The legacy of OnStar is much the same: of course your car can make phone calls, GM shoved a carphone in the trunk in 1996 and it's still in there somewhere. It's just one of a million things modern vehicle telematics do, and frankly, it's one of the least interesting ones. Ironically, GM is taking the carphone back out: in 2022, GM discontinued the OnStar telephone service. It's no longer possible to have a phone number assigned to your car and use OnStar for routine calls. Everyone uses an app on their phone for that.

[1] I am excluding here their satellite phones, although they were surprisingly advanced for the mid-'90s and probably would have competed well with cellular phones if the service wasn't so costly.

The majority of US states have something called a "Department of Motor Vehicles," or DMV. Actually, the universality of the term "DMV" seems to be overstated. A more general term is "motor vehicle administrator," used for example by the American Association of Motor Vehicle Administrators to address the inconsistent terminology.

Not happy with merely noting that I live in a state with an "MVD" rather than a "DMV," I did the kind of serious investigative journalism that you have come to expect from me. Of These Fifty United States plus six territories, I count 28 DMVs, 5 MVDs, 5 BMVs, 2 OMVs, 2 "Driver Services," and the remainder are hard to describe succinctly. In fact, there's a surprising amount of ambiguity across the board. A number of states don't seem to formally have an agency or division called the DMV, but nonetheless use the term "DMV" to describe something like the Office of Driver Licensing of the Department of Transportation.

Indeed, the very topic of where the motor vehicle administrator is found is interesting. Many exist within the Department of Transportation or Department of Revenue (which goes by different names depending on the state, such as DTR or DFA). Some states place driver's licensing within the Department of State. One of the more unusual cases is Oklahoma, which recently formed a new state agency for motor vehicle administration but with the goal of expanding to other state customer service functions... leaving it with the generic name of Service Oklahoma.

The most exceptional case, as you'll find with other state government functions as well, is Hawaii. Hawaii has deferred motor vehicle administration to counties, with the Honolulu CSD or DCS (they are inconsistent!) the largest, alongside others like the Hawaii County VRL.

So, the point is that DMV is sort of a colloquialism, one that is widely understood since the most populous states (CA and TX for example) have proper DMVs. Florida, third most populous state, actually has a DHSMV or FLHSMV depending on where you look... but their online services portal is branded MyDMV, even though there is no state agency or division called the DMV. See how this can be confusing?

Anyway, if you are sitting around on a Saturday morning searching for the name of every state plus "DMV" like I am, you will notice something else: a lot of... suspicious results. guamtax.com is, it turns out, actually the website of the Guam Department of Revenue and Taxation. dmvflorida.org is not to be confused with the memorable flhsmv.gov, and especially not with mydmvportal.flhsmv.gov. You have to put "portal" in the domain name so people know it's a portal, it's like how "apdonline.com" has "online" in it so you know that it's a website on the internet.

dmvflorida.org calls itself the "American Safety Council's Guide to the Florida Department of Motor Vehicles." Now, we have established that the "Florida Department of Motor Vehicles" does not exist, but the State of Florida itself seems a little confused on that point, so I'll let it slide. But that brings us to the American Safety Council, or ASC.

ASC is... It's sort of styled to sound like the National Safety Council (NSC) or National Sanitation Foundation (NSF), independent nonprofits that publish standards and guidance. ASC is a different deal. ASC is a for-profit vendor of training courses. Based on the row of badges on their homepage, ASC wants you to know not only that they are "Shopper Approved," "Certifiably Excellent (The Stats To Prove It)," they have won a "5-Star Excellence Award" (from whom not specified), and that the Orlando Business Journal included their own John Comly on its 2019 list of "CEOs of the Year."

This is the most impressive credential they have on offer besides accreditation by IACET, an industry association behind the "continuing education units" used by many certifications, and which is currently hosting a webinar series on "how AI is reshaping learning landscapes from curriculum design to compliance." This does indeed mean that, in the future, your corporate sexual harassment training will be generated by Vyond Formerly GoAnimate based on a script right out of ChatGPT. The quality of the content will, surprisingly, not be adversely affected. "As you can see, this is very important to Your Company. Click Here [broken link] to read your organization's policy."

In reality, ASC is a popular vendor of driver safety courses that businesses need their employees to take in order to get an insurance discount. Somewhere in a drawer I have a "New Mexico Vehicle Operator's Permit," a flimsy paper credential issued to state employees in recognition of their having completed an ASC course that consisted mostly of memorizing that "LOS POT" stands for "line of sight, path of travel." Years later, I am fuzzy on what that phrase actually means, but expanding the acronym was on the test.

We can all reflect on the fact that the state's vehicle insurance program is not satisfied with merely possessing the driver's license that the state itself issues, but instead requires you to pass a shorter and easier exam on actually driving safely. Or knowing about the line of sight and the path of travel, or something. I once took a Motorcycle Safety Foundation course that included a truly incomprehensible diagram of the priorities for scanning points of conflict at an intersection, a work of such information density that any motorcyclist attempting to apply it by rote would be entirely through the intersection and to the next one before completing the checklist. We were, nonetheless, taught as if we were expected to learn it that way. Driver's education is the ultimate test of "Adult Learning Theory," a loose set of principles influential on the design of Adobe Captivate compliance courses, and the limitations of its ability to actually teach anyone anything.

This is all a tangent, so let's get back to the core. ASC sells safety courses and... operates dmvflorida.org?

Here's the thing: running DMV websites is a profitable business. Very few people look for the DMV website because they just wanted to read up on driver's license endorsements. Almost everyone who searches for "<state name> DMV" is on the way to spending money: they need to renew their license, or their registration, or get a driving test, or ideally, a driver's ed course or traffic school.

The latter are ideal because a huge number of states have privatized them, at least to some degree. Driver's ed and traffic school are both commonly offered by competitive for-profit ventures that will split revenue in exchange for acquiring a customer. I would say that dmvflorida.org is a referral scam, but it's actually not! it's even better: it's owned by ASC, one of the companies that competes to offer traffic school courses! It's just a big, vaguely government-looking funnel into ASC's core consumer product.

In some states, the situation is even better. DMV services are partially privatized or "agents" can submit paperwork on the behalf of the consumer. Either of these models allow a website that tops Google results to submit your driver's license renewal on your behalf... and tack on a "convenience fee" for doing so. Indeed, Florida allows private third-parties to administer the written exam for a driver's license, and you know dmvflorida.org offers such an online exam for just $24.95.

You can, of course, renew your driver's license online directly with the state, at least in the vast majority of cases. so how does a website that does the same thing, with the same rates, plus their own fee, compete? SEO. Their best bet is to outrank the actual state website, grabbing consumers and funneling them towards profitable offerings before they find the actual DMV website.

There's a whole world of DMV websites that operate in a fascinating nexus of SEO spam, referral farm, and nearly-fraudulent imitation of official state websites. This has been going on since, well, I have a reliable source that claims since 1999: dmv.org.

dmv.org is an incredible artifact of the internet. It contains an enormous amount of written content, much of it of surprisingly high quality, in an effort to maintain strong search engine rankings. It used to work: for many years, dmv.org routinely outranked state agency websites for queries that were anywhere close to "dmv" or "renew driver's license" or "traffic school." And it was all in the pursuit of referral and advertising revenue. Take it from them:

Advertise with DMV.ORG

Partner with one of the most valuable resource for DMV & driving - driven by 85% organic reach that captures 80% of U.S drivers, DMV.ORG helps organize the driver experience across the spectrum of DMV and automotive- related information. Want to reach this highly valued audience?

dmv.org claims to date back to 1999, and I have no reason to doubt them, but the earliest archived copies I can find are from 2000 and badly broken. By late 2001 the website has been redesigned, and reads "Welcome to the Department of Motor Vehicles Website Listings." If you follow the call to action and look up your state, it changes to "The Department of Motor Vehicles Portal on the Web!"

They should have gone for dmvportal.org for added credibility.

In 2002, dmv.org takes a new form: before doing pretty much anything, it asks you for your contact information, including an AOL, MSN, or Yahoo screen name. They promise not to sell your address to third parties but this appears to be a way to build their own marketing lists. They now prominently advertise vehicle history reports, giving you a referral link to CarFax.

Over the following months, more banner ads and referral links appear: vehicle history reports, now by edriver.com, $14.99 or $19.99. Driving record status, by drivingrecord.org, $19.99. Traffic School Online, available in 8 states, dmv-traffic-school.com and no price specified. The footer: "DMV.ORG IS PRIVATELY OPERATED AND MAINTAINED FOR THE BENEFIT OF ITS USERS."

In mid-2003, there's a rebranding. The header now reads "DMV Online Services." There are even more referral links. Just a month later, another redesign, a brief transitional stage, before in September 2003, dmv.org achieves the form familiar to most of us today: A large license-plate themed "DMV.ORG" logotype, referral links everywhere, map of the US where you can click on your state. "Rated #1 Site, CBS Early Show."

This year coincides, of course, with rapid adoption of the internet. Suddenly consumers really are online, and they really are searching for "DMV." And dmv.org is building a good reputation for itself. A widely syndicated 2002 newspaper article about post-marriage bureaucracy (often appearing in a Bridal Guide supplement) sends readers to dmv.org for information on updating their name. The Guardian, of London, points travelers at dmv.org for information on obtaining a handicap placard while visiting the US.

You also start to see the first signs of trouble. Over the following years, an increasing number of articles both in print and online refer to dmv.org as if it is the website of the Department of Motor Vehicles. We cannot totally blame them for the confusion. First, the internet was relatively new, and reporters had perhaps not learned to be suspicious of it. Second, states themselves sometimes fanned the flames. In a 2005 article, the director of driver services for the Mississippi Department of Transportation tells the reporter that you can now renew your driver's license online... at dmv.org.

dmv.org was operated by a company called eDriver. It's hard to find much about them, because they have faded into obscurity and search results are now dominated by the lawsuit that you probably suspected is coming. The "About Us" page of the dmv.org of this period is a great bit of copywriting, complete with dramatic stories, but almost goes out of its way not to name the people involved. "One of our principals likes to say..."

eDriver must not have been very large, their San Diego office address was a rented mail box. Whether or not it started out that way is hard to say, but by 2008 eDriver was a subsidiary of Biz Groups Inc., along with Online Guru Inc and Find My Specialist Inc. These corporate names all have intense "SEO spam" energy, and they seem to have almost jointly operated dmv.org through a constellation of closely interlinked websites. In 2008, eDriver owned dmv.org but didn't even run it: they contracted Online Guru to manage the website.

Biz Groups Inc was owned by brothers Raj and Ravi Lahoti. Along with third brother David, the Lahotis were '00s domain name magnates. They often landed on the receiving end of UDRP complaints, ICANN's process for resolving disputes over the rightful ownership of domain names. Well, they were in the business: David Lahoti owns UDRP-tracking website udrpsearch.com to this day.

Their whole deal was, essentially, speculating on domain names. Some of them weren't big hits. An article on a dispute between the MIT Scratch project and the Lahotis (as owners of scratch.org) reads "Ravi updated the site at Scratch.org recently to includes news articles and videos with the word scratch in them. It also has a notice that the domain was registered in 1998 and includes the dictionary definition of scratch."

Others were more successful. In 2011, Raj Lahoti was interviewed by a Korean startup accelerator called beSuccess:

My older brother Ravi was the main inspiration behind starting OnlineGURU. Ravi owned many amazing domain names and although he didn't build a website on every one of his domains, he DID build a small website at www.DMV.org and this website started doing well. Well enough that he saw an opportunity to do something bigger with it and turn it into a bigger business.

And he is clear on how the strategy evolved to focus on SEO farming:

Search Engine Marketing and Search Engine Optimization has definitely been most effective in my overall marketing strategy. The beautiful thing about search engines is that you can target users who are looking for EXACTLY what you offer at the EXACT moment they are looking for it. Google Adwords has so many tools, such as the Google Keyword Tool where you can learn what people are searching for and how many people are searching the same thing. This has allowed me to learn about WHAT the world wants and gives me ideas on how I can provide solutions to help people with what they are looking for.

Also, San Diego Business Journal named Raj Lahoti "among the finalists of the publication's Most Admired CEO award" in 2011. So if he ever meets John Comly, they'll have something to talk about.

The thing is, the relationship between dmv.org and actual state motor vehicle administrators became blurrier over time... perhaps not coincidentally, just as dmv.org ascended to a top-ranking result across a huge range of Google queries. It really was a business built entirely on search engine ranking, and they seemed to achieve that ranking in part through a huge amount of content (that is distinctly a cut above the nearly incoherent SEO farms you see today), but also in part through feeding consumer confusion between them and state agencies. I personally remember ending up on dmv.org when looking for the actual DMV's website, and that was probably when I was trying to get a driver's license to begin with. It was getting a bit of a scammy reputation, actual DMVs were sometimes trying to steer people away from it, and in 2007 they were sued.

A group of website operators in basically the same industry, TrafficSchool.com Inc and Driver's Ed Direct, LLC, filed a false advertising suit against the Online Guru family of companies. They claimed not that dmv.org was fraudulent, but that it unfairly benefited from pretending to be an official website.

Their claim must have seemed credible. At the beginning of 2008, before the lawsuit made it very far, dmv.org's tagline changed from "No need to stand IN LINE. Your DMV guide is now ON LINE!" to "Your unofficial guide to the DMV." This became the largest indication that dmv.org was not an official website, supplementing the small, grey text that had been present in the footer for years.

The judge was not satisfied.

See, the outcome of the lawsuit was sort of funny. The court agreed that dmv.org was engaging in false advertising under the Lanham Act, but then found that the plaintiffs were doing basically the same thing, leaving them with "unclean hands." Incidentally, they would appeal and the appeals court would disagree on some details of the "unclean hands" finding, but the gist of the lower court's ruling held: the plaintiffs would not receive damages, since they had been pursuing the same strategy, but the court did issue an injunction requiring dmv.org to add a splash screen clearly stating that it was not an official website.

The lawsuit documents are actually a great read. The plaintiffs provided the court with a huge list of examples of confusion, including highlights like a Washington State Trooper emailing dmv.org requesting a DUI suspect's Oregon driving records. dmv.org admitted to the court that they received emails like this on "a daily basis," many of them being people attempting to comply with mandatory DUI reporting laws by reporting their recent DUI arrest... to Online Guru.

The court noted the changes made to dmv.org in early 2008, including the "Unofficial" heading and changing headings from, for example, "California DMV" to "California DMV Info." But those weren't sufficient: going forward, users would have to click "acknowledge" on a page warning them.

It is amusing, of course, that the SEO industry of the time interpreted the injunction mainly in the SEO context. This was, after all, a website that lived and died by Google rankings, part of a huge industry of similar websites. Eric Goldman's Technology and Marketing Law Blog wrote that "My hypothesis is that such an acknowledgment page wrecks DMV.org’s search engine indexing by preventing the robots from seeing the page content."

The takeaway:

This suggests a possible lesson to learn from this case. The defendants had a great domain name (DMV.org) that they managed to build nicely, but they may have too aggressive about stoking consumer expectations about their affiliation with the government.

It's wild that "get a good domain name and pack it with referral links" used to be a substantial portion of the internet economy. Good thing nothing that vapid survives today! Speaking of today, what happened to dmv.org?

Well, the court order softened over time, and the acknowledgment page ultimately went away. It was replaced by a large, top-of-page banner, almost comically reminiscent of those appearing on cigarettes. "DMV.ORG IS A PRIVATELY OWNED WEBSITE THAT IS NOT OWNED OR OPERATED BY ANY STATE GOVERNMENT AGENCY." Below that, the license plate dmv.org logotype, same as ever.

Besides, they reformed. At sustainablebrands.com we read:

Over our 10-year history, DMV.org’s mission has shifted entirely from profit to purpose. We not only want to bring value to our users by making their DMV experience easier, we ultimately want to reduce transportation-related deaths, encourage eco-friendly driving habits, and influence other businesses to reduce their carbon footprints and become stewards of change themselves.

This press-release-turned-article says that they painted "the company’s human values on our wall, to remind ourselves every day what we’re here for and why" and that, curiously, dmv.org "potentially aim[s] to" "eliminate Styrofoam from local eateries." The whole thing is such gross greenwashing, bordering on incoherent, that I might accuse it of being AI-generated were it not a decade old.

dmv.org lived by Google and, it seems, it will die by Google. Several SEO blogs report that, sometime in 2019, Google seems to have applied aggressive manual adjustments to a list of government-agency-like domain names that include irs.com (its whole own story) and dmv.org. Their search traffic almost instantaneously dropped by 80%.

dmv.org is still going today, but I'm not sure that it's relevant any more. I tried a scattering of Google queries like "new mexico driver's license" and "traffic school," the kind of thing where dmv.org used to win the top five results, and they weren't even on the first page. Online Guru still operates dmv.org, and "dmv.org is NOT your state agency" might as well be the new tagline. Phrases like that one constantly appear in headings and sidebars.

They advertise auto insurance, and will sell you online practice tests for $10. Curiously, when I look up how to renew my driver's license in New Mexico, dmv.org sends me to the actual NM MVD website. That's sort of a funny twist, because New Mexico does indeed allow renewal through private service providers that are permitted to charge a service fee. I don't think dmv.org makes enough money to manage compliance with all these state programs, though, so it's actually returned to its roots, in a way: just a directory of links to state websites.

Also, there's a form you can fill out to become a contributor! Computers Are Bad has been fun, but I'm joining the big leagues. Now I write for dmv.org.

In a previous episode, I discussed audio transports and mention that they have become a much less important part of the modern home theater landscape. One reason is the broad decline of the component system: most consumers aren't buying a television, home theater receiver, several playback devices, and speakers. Instead, they use a television and perhaps (hopefully!) a soundbar system, which often supports wireless satellites if there are satellites at all. The second reason for the decline of audio transports is easy to see when we examine these soundbar systems: most connect to the television by HDMI.

This is surprising if you consider that soundbars are neither sources nor sinks for video. But it's not so surprising if you consider the long-term arc of HDMI [1], towards being a general-purpose consumer AV interconnect. HDMI has become the USB of home theater, and I mean that as a compliment and an insult. So, like USB, HDMI comes in a confusing array of versions with various mandatory, optional, and extension features. The capabilities of HDMI vary by the devices on the ends, and in an increasing number of cases, even by the specific port you use on the device.

HDMI actually comes to this degree of complexity more honestly than USB. USB started out as a fairly pure and simple serial link, and then more use-cases were piled on, culminating in the marriage of two completely different interconnects (USB and Thunderbolt) in one physical connector. HDMI has always been a Frankenstein creation. At its very core, HDMI is "DVI plus some other things with a smaller connector."

DVI, or really its precursors that established the actual video format, was intended to be a fairly straightforward step from the analog VGA. As a result, the logical design of DVI (and thus HDMI) video signals are pretty much the same as the signals that have been used to drive CRT monitors for almost as long as they've existed. There are four TMDS data lines on an HDMI cable, each a differential pair with its own dedicated shield. The four allow for three color signals (which can be used for more than one color space) and a clock. Two data pins plus a shield, times four, means 12 pins. That's most, but not all, of the 19 pins on an HDMI connector.

A couple of other pins are used for an I2C connection, to allow a video source to query the display for its specifications. A couple more are used for the audio return channel or ethernet (you can't do both at the same time) feature of HDMI. There's a 5V and a general signal ground. And then there's the CEC pin.

The fact that CEC merits its own special pin suggests that it is an old part of the standard, and indeed it is. CEC was planned from the very beginning, although it didn't get a full specification as part of the HDMI standard until HDMI 1.2a. Indeed, CEC is older than HDMI, dating to at least 1998, when it was standardized as part of SCART. But let's take a step back and consider the application.

One of the factors in the decline of component stereo systems is the remote control. In the era of vinyl, when you had to get off the couch to start a record anyway, remote controls weren't such an important part of the stereo market. The television changed everything about the way consumers interact with AV equipment: now we all stay on the couch.

I think we all know the problem, because we all lived through it: the proliferation of remotes. When your TV, your VCR, and your home theater receiver all have remote controls, you end up carrying around a bundle of cheap plastic. You will inevitably drop them, and the battery cover will pop off, and the batteries will go under the couch. This was one of the principal struggles faced by the American home for decades.

There are, of course, promised solutions on the market. Many VCR remotes had the ability to control a TV, and often the reverse as well. If you bought your TV and VCR from the same manufacturer this worked. If you didn't, it might not, or at least setup will be more complex. This is because the protocols used by IR remotes are surprisingly unstandardized. Surprisingly unstandardized in that curious way where there are few enough IR transceiver ICs that a lot of devices actually are compatible (consider the ubiquitous Philips protocol), but no one documents it and detailed button codes often vary in small and annoying ways.

So we got the universal remote. These remotes, often thrown in with home theater receivers as a perk, have some combination of a database of remote protocols pre-reverse-engineered by the manufacturer and a "learn" mode in which they can record unknown protocols for naive playback. Results were... variable. I heard that some of the expensive universal remotes like Logitech Harmony (dead) and Control4 (still around) were actually pretty good, but they required some emphasis on the word "expensive." Universal remotes were sort of a mixed bag, but they were fiddly enough to keep working that consumer adoption doesn't seem to have been high.

So, another approach came to us from the French. In the Europe of the 1970s, there was not yet a widely accepted norm for connecting a video source to a TV (besides RF modulation). France addressed the matter by legislation, mandating SCART in 1980. Over the following years, SCART became a norm in much of Europe. SCART is a bit of an oddity to Americans, as it never achieved a footprint on this continent. That's perhaps a bit disappointing, because SCART was ahead of its time.

For example, much like HDMI, SCART carried bidirectional audio. It supported multiple video formats over one cable. Most notably, though, SCART was designed for daisy chaining. Some simple aspects of the SCART design provided a basic form of video routing, where the TV could bidirectionally exchange video signals with one of several devices in a chain. The idea of daisy-chainable video interconnects continuously reappears but seldom finds much success, so I'd call this one of the more notable aspects of SCART.

That's not why we're here, though. Another interesting aspect of SCART was its communications channel between devices. The core SCART specification included a basic system of voltage signaling to indicate which device was active, but in 1998 CENELEC EN 50157-1 was standardized as a flexible serial link between devices over the SCART cable. Most often called AV.link, this channel could be used for video format negotiation, but also promised a solution to multiplying remotes: the AV.link channel can transmit remote control commands between devices. For example, your TV remote can have play/pause buttons, and when you push them the TV can send AV.link play/pause commands to whichever video source is active.

AV.link is a very simple design. A one wire (plus ground) serial bus operates at a slow (but durable) 333bps with collision detection. Devices are identified by four-bit addresses chosen at random (but checked for collision). Messages have a simple format: a one-byte header with the sending and receiving addresses, a one-byte opcode, and then whatever bytes are expected as parameters to the opcode.

AV.link is one of those standards that never quite got its branding together. Unlike, say, USB, where a consistent trademark identity is used, AV.link goes by different names from different vendors. Wikipedia offers the names nexTViewLink (horrible), SmartLink (mediocre), Q-Link (lazy), and EasyLink (mediocre again). One wonders if consumers were confused by these different vendor brands for the same thing, it's not a situation that happens very often with consumer interconnects.

When HDMI was developed, the provision of a pin for AV.link was pretty much copied over from SCART. Originally, the functionality wasn't even really specified, and just assumed to be similar to SCART. Later HDMI versions included a much more complete description of CEC as a supplement. Hardware support for CEC is mandated for devices like TVs as part of the HDMI certification process, but curiously, software support isn't really included. As a result, it is very common, but not universal, that TVs fully support CEC. Other AV devices like home theater receivers almost universally have CEC support. Computers almost universally do not, as cost and licensing considerations mean that GPUs do not provide a CEC transceiver.

Inconsistent implementations are not the only way that CEC is a little sketchy. Remember how different vendors referred to SCART AV.link by different names? CEC has the same problem. I won't bother with the whole list, but the names you're more likely to have seen include Samsung Anynet+, LG SimpLink, and... well, Philips EasyLink is still with us. In practice, a lot of people seem to ignore these names, and CEC is a lot more common than Anynet+ when discussing Samsung TVs. That doesn't stop Samsung from pushing their own branding in their menus and port labeling, though.

Because CEC inherits the AV.link features designed for SCART, it has a surprisingly rich featureset. For example, if you have an HDMI switch with real CEC support (these don't seem to be that common!) and a TV with software support, the TV can discover the topology of connected devices and remote control the switch to use the switch inputs as an extension of its own input selection menu.

Most CEC features are more prosaic, though. Considering the list of high-level features in the specification, "One Touch Play" means that a device can indicate that it has video to show (causing a TV to turn on and select that input) while "System Standby" means that a device being turned off can tell the other devices on the bus to turn off as well. "One Touch Record," "Deck Control," "Tuner Control," "Device Menu Control," and "System Audio Control" are all variations on devices forwarding simple remote commands (play, pause, up, down, etc) to other devices that might care about them more. For example, when you use a TV with a stereo receiver or soundbar, it should forward volume up/down commands from the remote to the audio device via System Audio Control.

Considering the decline of component systems, there are basically two common scenarios where CEC is used today. These are really the same scenario in a lot of ways, but they vary in the details.

• The connection of a television to a home theater receiver. In these kinds of configurations, the home theater receiver is often used for at least some video switching. That means that the receiver is sending video to the TV, receiving audio from the TV via audio return channel (ARC), and both are exchanging commands. For example, the receiver can use CEC to turn on the TV when a video input is selected. Conversely, the TV can turn on the receiver when it is turned on.
• The connection of a television to a soundbar. In this case, ARC is used to send audio from the TV to the soundbar. There really is no video involved in this scenario, so in a sense the HDMI cable is mostly unused. CEC is used by the TV to control the soundbar. Because soundbars don't often have a remote that the user cares to keep around, this control tends to be mostly unidirectional, used by the TV to turn the soundbar on and off.

It is interesting, isn't it, that an interconnect with four very-high-speed serial video channels is often put into use in a scenario where those channels are useless. Instead, the much lower-rate ARC and CEC channels are the important ones. Well, think about USB as a power connector... these things happen.

CEC could be used in much more complex scenarios. For example, if you had a DVR connected to your TV via CEC, you could browse the electronic program guide (EPG) on your TV and choose a program to record. This would cause the TV to use CEC Timer Programming to send the program details from the TV EPG to the DVR to schedule the recording. How widely was this ever used? I don't know, I suspect not very, because these days DVRs are almost invariably provided by the cable or satellite company, who expect you to use the DVR's EPG rather than your TVs anyway.

This is actually one of the scenarios where you see ARC used for reasons other than synchronizing control of an audio output: set-top boxes (STBs). Media companies that distribute STBs, mostly cable and satellite operators, tend to be in a bit of a war to own your television watching experience. They face stiff competition from "Smart TVs." I have a suspicion that the complete proliferation of smart TVs is largely an artifact of the television manufacturers trying to win advertising surface area away from the STB manufacturers, who have traditionally held most of it via the EPG.

As some evidence of this fight, consider the case of Xfinity Xumo (formerly Flex), the compact STB that Xfinity offers to its internet customers for free. Since it's advertised to people who don't necessarily have any TV service from Xfinity, it's not really a conventional STB. It's more of a slightly-weird-but-free Roku or Amazon Fire Stick. It doesn't really offer anything that your TV doesn't already, but unlike your TV, it's controlled by Comcast. This gives them the opportunity to upsell you on IPTV services, but Comcast never seems to have pursued this route that far. Mostly it gives them the opportunity to advertise to you, and to grab some partner revenue from various streaming apps.

Anyway, that was a bit of a digression. The point is that Comcast and Dish Network and all of their compatriots don't want you using your TV, they want you using your STB. So they give you a big chunky remote ("With Voice Control!") and the STB attempts to use CEC to control the TV so you never have to touch its small, svelte remote ("With Voice Control!") and split their sponsored content revenue with LG.

That's an interesting detail of this whole landscape, isn't it? CEC was developed as a solution to a technical problem: people had multiple devices, and hauling around multiple remotes was frustrating. Over the decades since, it has evolved into a strategy to address a business problem: everyone that sells you AV equipment prefers that you passionately navigate their on-screen menus while completely forgetting about those of your other components.

That's pretty much what's happening with the audio devices as well. TV manufacturers want to capture as much of your entertainment attention and budget as possible, so ideally they sell you a TV and their matching soundbar system (which can be fairly inexpensive since it is closely coupled to the TV and needs very little of its own control logic). CEC here is an under-the-hood implementation detail, something that happens behind the scenes to make your soundbar do the few things it does.

Say you're a higher-end customer, though, with a home theater receiver. The AV receiver industry has been surprisingly unambitious about capturing Platform Revenue, probably because soundbars have pretty much eliminated everything but higher-end, "audiophile"-focused brands. These companies either lack the technical resources to develop a good Entertainment Platform or don't think their customers will respond well to yet another remote with a Pluto TV button. I would like to say it's mostly the latter, but given my experience with the on-screen design and mobile apps of several leading AV receiver manufacturers, I suspect it's mostly the former.

So CEC functions perhaps the most as it was originally intended: you can mainly interact with your TV, and CEC carries control messages to the receiver as needed so that you don't need to find its remote to select the right input. Conceptually you can even use the TV to control non-video functions. For example, my particular combination of a Samsung TV and Yamaha receiver implements CEC completely enough that I can turn on the receiver, select the turntable preamp input, and control the volume via the TV if I want to. Then I still have to get up to actually put a record onto the turntable, and now the TV is just on the whole time, so this isn't that appealing in practice. I am still rummaging fro the receiver's own remote, that or using its terrible Android app.

In the STB scenario, something like Xfinity X1 or Dish Hopper, we have an inversion of control: the only remote you'll need, they hope, is the STB's remote. It will remote control the TV via CEC as needed. This inevitably sets up a power struggle where your Smart TV gets lonely and wants attention. I am mostly kidding about this emotional interpretation of the situation, but obviously the TV manufacturer does have an incentive to distract your attention from the STB, which probably has something to do with the tendency of Smart TVs to pop up a lot of on-screen chrome whenever you turn them on.

The coolest thing about CEC, in my mind, is that unlike HDMI it is multi-drop. That is, when you connect a bunch of HDMI sources to a multi-input TV or receiver or another switching device, they can all be connected to the same unified CEC bus. That means that HDMI devices can communicate with each other via CEC even when there's no active video or audio connection.

CEC even has a fairly complex addressing mechanism to take advantage. CEC physical addresses are assigned based on bus topology, and a mapping protocol is used to advertise a correspondence between new physical addresses and logical addresses. Logical addresses, the same 4-bit addresses from AV.link, are assigned based on capabilities. Typically logical address 0 will be the TV, 1 and 2 will be recorders, 3 will be a tuner, 4 a playback device, 5 a home stereo receiver. You can have a fairly large component setup where everything is controllable by sending CEC messages to standard logical addresses.

And other aspects of CEC are designed to accommodate these kinds of more complex networks. For example, when the user selects a device to watch on their TV, the TV can send a "Set Stream Path" message (opcode 0x86). The parameter on this message is the physical CEC address of the desired device, and any CEC switches in the path are expected to see the message and select the appropriate input to form a path from the selected device to the TV. It's a little bit of centrally-controlled circuit switching right in your entertainment center. Neat!

You can even do broadcast messaging across the entire CEC topology. TVs often use this to discover what devices they're connected to, saving the user some menu setup. That's about the only time you'd notice it, though: like CEC's other more advanced capabilities, routing and multi-device messaging are rarely used outside of its very simplest application.

I want to pass some sort of tidy moral judgment, but CEC is a hard case. It's kind of a mixed bag. The more basic functionality tends to work well and adds convenience. The more complex functionality tends to either not work or be buried deeply enough in configuration menus that no one uses it. It inevitably leads to some weird, inelegant behavior. My husband will put a cab ride video on the TV and then Spotify Cast to the receiver. But then what if you want to listen to the video audio? Easiest way to get the receiver switched back to the TV audio is, of course, to turn the TV off and on again. When you turn it on, it uses CEC "One Touch Play" to signal the receiver to select it again. The particular convergence of technologies here leads to a strange tic, sort of a superstitious behavior, that works fine but feels bad.

If you're a weirdo like me, you use your TV heavily as a monitor for a computer. You might find the gap here rather conspicuous: when I wiggle the mouse to wake the computer up, the TV doesn't turn on. HDMI keeps gaining features, video games are a big driver of high-end PC and television sales, there is an inevitable convergence happening between "monitor" and "TV," and between "video source" and "computer." But the computer video industry is, well, a little slow to catch on.

You might remember that it took an awkwardly long time for PC GPUs to have consistent support for HDMI audio, and then it was still weird and sketchy for a good few years. Well, we haven't even quite made it to that point on the CEC front. I don't think any conventional PCs have CEC transceivers. The solution, if you are mad enough to want one, is a USB CEC adapter. They're basically passthrough devices for HDMI, they just tap the CEC pin and hook it up to a UART. Not many companies make them but they're cheap enough. Software support is... minimal, but it'll let Kodi turn your TV on.

It's fun to think about, though. You know how CEC is multi-drop? You could hook up multiple computers to an HDMI switch and they could talk to each other with CEC. You could use some vendor-specific opcodes to convey IP. You could log onto the internet over HDMI, at 333bps. You could put OpenSC over IP over HDMI CEC and turn your lights on via your stereo receiver. What a dream! I was going to say you could do DMX-512 over CEC but actually at CEC's slow speed the register-broadcast model of DMX would become a pretty significant problem.

You could also log onto the internet over HDMI at 100Mbps, but that's using different pins, your GPU definitely doesn't support it, and I don't even know of a way to do HDMI Ethernet from a PC. CEC may be a bit of an awkward cousin but at least it's more popular than HDMI Ethernet.

[1] pun not intended

I feel like I used to spend an inordinate amount of time dealing with suspect hard drives. I mean, like, back in high school. These days I almost never do, or on the occasion that I have storage trouble, it's a drive that has completely stopped responding at all and there's little to do besides replacing it. One time I had two NVMe drives in two different machines do this to me the same week. Bad luck or quantum phenomenon, who knows.

What accounts for the paucity of "HDD recovery" in my adult years? Well, for one, no doubt HDD technology has improved over time and modern drives are simply more reliable. The well-aged HDDs I have running without trouble in multiple machines right now support this theory. But probably a bigger factor is my buying habits: back in high school I was probably getting most of the HDDs I used second-hand from the Free Geek thrift store. They were coming pre-populated with problems for my convenience.

Besides, the whole storage industry has changed. What's probably more surprising about my situation is how many "spinning rust" HDDs I still own. Conventional magnetic storage only really makes sense in volume. These days I would call an 8TB HDD a small one. The drives that get physical abuse, say in laptops, are all solid state. And solid state drives... while there is no doubt performance degradation over their lifetimes, failure modes tend to be all-or-nothing.

I was thinking about all of this as I ruminated on one of the "holy grail" tools of the late '00s: SpinRite, by Gibson Research Corporation.

The notion that HDDs aren't losing data like they used to is supported by the dearth of data recovery tools on the modern shareware market. Well, maybe that's more symptomatic of the complete hollowing out of the independent software industry by the interests of capitalism, but let's try to dwell on the positive. Some SEO-spam blog post titled "Best data recovery software of 2024" still offers some classic software names like "UnDeleteMyFiles Pro," but some items on the list are just backup tools, and options like Piriform Recuva and the open-source PhotoRec still rank prominently... as they did when I was in high school and my ongoing affection for Linkin Park was less embarrassing [1].

Back in The Day, freeware, shareware, and commercial (payware?) data recovery software proliferated. It was advertised in the back of magazines, the sidebar banner ads of websites, and even appeared in the electronics department of Fred Meyer's. You also saw a lot of advertisements for services that could perform more intensive methods, like swapping an HDD's controller for one from another unit of the same model. These are all still around today, just a whole lot less prominent. Have you ever seen an Instagram ad for UnDeleteMyFiles Pro?

First, we should talk a bit about the idea of data recovery in general. There are essentially two distinct fields that we might call "data recovery": consumers or business users trying to recover their Important Files (say, accounting spreadsheets) from damaged or failed devices, and forensic analysts trying to recover Important Files (say, the other accounting spreadsheets) that have been deleted.

There is naturally some overlap between these two ventures. Consumers sometimes accidentally delete their Important Files and want them back. Suspects sometimes intentionally damage storage devices to complicate forensics. But the two different fields use rather different techniques.

Let's start by examining forensics, both to set up contrast to consumer data recovery and because I know a lot more about it. One of the quintessential techniques of file system forensics is "file carving." A file carving tool examines an arbitrary sequence of bytes (say, from a disk image) and looks for the telltale signs of known file formats. For example, most common file formats have a fixed prefix of some kind. ZIP files start with 0x504B0304, the beginning of which is the ASCII "PK" for Phil Katz who designed the format. Some formats also have a fixed trailer, but many more have structure that can be used to infer the location of the end of the file. For example, in ZIP files the main header structure, the "central directory," is actually a trailer found at the end of the file.

If you can find the beginning and end of a file, and it's stored sequentially, you've now got the whole file. When the file is fragmented in the byte stream (commonly the case with disk images), the problem is a little tougher, but still you can find a lot of value. A surprising number of files are stored sequentially because they are small, some filetypes have internal structure that can be used to infer related blocks and their order, and even finding a single block of a file can be useful if it happens to contain a spreadsheet row starting "facilitating payments to foreign officials" or, I don't know, "Fiat@".

You end up doing this kind of thing a lot because of a detail of file systems that all of my readers probably know. It's often articulated as something like "when you delete a file, it's not deleted, just marked as having been deleted." That's not exactly wrong but it's also an oversimplification in a way that makes it more difficult to understand why that is the case. There's a whole level of indirection due to block allocation, updating the bitmap on every file delete is a relatively time-consuming process that offers little value, actually overwriting blocks would be even more time consuming with even less value, etc. Read Brian Carrier for the whole story.

Actually, screw Brian Carrier, I've written before about the adjacent topic of secure erasure of computer media.

My point is this: these forensic methods are performed on a fully functional storage device (or more likely an image of one), where "recovery" is necessary and possible because of the design of the file system. The storage device, as hardware, is not all that involved. Well, that's really an oversimplification, and points to an important consideration in modern data recovery: storage devices have gotten tremendously more complex, and that's especially true of SSDs.

Even HDDs tend to have their own thoughts and feelings. They can have a great deal of internal logic dedicated to maintaining the disk surface, optimizing performance, working around physical defects on the surface, caching, encryption, etc. Pretty much all of this is proprietary to the manufacturer, undocumented, and largely a mystery to the person performing recovery. Thinking of the device as a "sequence of bytes" throws out a lot of what's really going on, but it's a necessary compromise.

SSDs have gone even further. Flash storage is less durable than magnetic storage but also more flexible. It requires new optimizations to maximize life and facilitates optimizations for access time and speed. Some models of SSDs vary from each other only by their software configuration (this has long been suspected of some HDDs as well, but I have no particular insight into Western Digital color coding). Even worse for the forensic analyst, the TRIM command creates a whole new level of active management by the storage device: SSDs know which blocks are in use, allowing them to constantly remap blocks on the fly. It is impossible, without hardware reverse engineering techniques, to produce a true image of an SSD. You are always working with a "view" of the SSD mediated by its firmware.

So let's compare and contrast forensic analysis to consumer data recovery. The problem for most consumers is sort of the opposite: they didn't delete the file. If they could get the sequence of bytes off the storage device, they could just access the file through the file system. The problem is that the storage device is refusing to produce bytes at all, or it's producing the wrong ones.

Techniques like file carving are not entirely irrelevant to consumer data recovery because it's common for storage devices to fail only partially. There are different ways of referring to the physical geometry of HDDs, and besides, modern storage devices (HDDs and SSDs alike) abstract away their true geometry. Different file systems also use different terminology for their own internal system of mapping portions of the drive to logical objects. So while you'll find people say things like "bad cluster" and "bad sector," I'm just going to talk about blocks. The block is the smallest elementary unit by which your file system interacts with the device. The size of a block is typically 512B for smaller devices and 4k for larger devices.

A common failure mode for storage devices (although, it seems, not so much today) is the loss of a specific block: the platter is damaged, or some of the flash silicon fails, and a specific spot just won't read any more. The storage device can, and likely will, paper over this problem by moving the block to a different area in the storage medium. But, in the process, the contents of the block are probably lost. The new location just contains... whatever was there before [2]. Sometimes the bad block is in the middle of a file, and that sucks for that file. Sometimes the bad block is in the middle of a file system structure like the allocation table, and that sucks for all of the files.

More complicated file systems tend to incorporate precautionary measures against this kind of thing, so the blast radius is mostly limited to single files. For example, NTFS keeps a second copy of the allocation table as a backup. Journaling can also provide a second source of allocation data when the table is damaged.

Simpler file systems, like the venerable FAT, don't have any of these tricks. They are, after all, old and simple. But old age and simplicity gives FAT a "lowest common denominator" status that sees it widely used on removable devices. PhotoRec, while oriented towards the consumer data recovery application, is actually a file carving tool. It's no coincidence that it's called PhotoRec. Removable flash devices like SD cards have simple controllers and host simple file systems. They are, as a result, some of the most vulnerable devices to block failures that render intact files undiscoverable.

What about the cases where file isn't intact, though? Where the block that has become damaged is part of the file that we want? What about cases where a damaged head leaves an HDD unable to read an entire surface?

Well, the news isn't that great. Despite this being one of the most common types of consumer storage failure for a decade or more, and despite the enormous inventory of software that promises to help, your options are limited. A lot of the techniques that software packages used in these situations lack supporting research or are outright suspect. Let's start on solid ground, though, with the most obvious and probably safest option.

One of the problems you quickly encounter when working with a damaged storage device is the file system and operating system. File systems don't like damaged storage devices, and operating systems don't like file systems that refuse to give up a file they say exists. So you try to copy files off of the bad device and onto a good one using your daily-driver file browser, and it hits a block that won't read and gets stuck. Maybe it hangs almost indefinitely, maybe you get an obscure error and the copy operation stops. Your software is working against you.

One of the best options for data recovery from suspect devices is an open-source tool called ddrescue. ddrescue is very simple and substantially similar to dd. It has one critical trick up its sleeve: when reading a block fails, ddrescue retries a limited number of times and then moves on. With that little adaptation, you can recover all of the working blocks from a device and so likely recover all of the files but a few.

Besides, just retrying a few times has value. Especially on magnetic devices, the result of reading the surface can be influenced by small perturbances. An unreadable sector might be readable every once in a while. This doesn't seem to happen as much with SSDs due to the dynamics of flash storage and preemptive correction of weak or ambiguous values, but I'm sure it still happens every once in a while.

At the end of the day, though, this method still means accepting the loss of some data. Losing some data is better than losing all of it, but it might not be good enough. Isn't there anything we can do?

HDDs used to be different. For one, they used to be bigger. But there's more to it than that. Older hard drives used stepper motors to position the head stack, and so head positioning was absolute but subject to some mechanical error. Although this was rarely the case on the consumer market, early hard drives were sometimes sold entirely uninitialized, without the timing marks the controller used to determine sector positions. You had to use a special tool to get the drive to write them [3]. It was common for older drives to come with a report (often printed on the label) of known bad sectors to be kept in mind when formatting.

We now live in a different era. Head stacks are positioned by a magnetic coil based on servo feedback from the read head; mechanical error is virtually absent and positioning is no longer absolute but relative to the cylinder being read. Extensive low-level formatting is required but is handled completely internally by the controller. Controllers passively detect bad blocks and reallocate around them. Honestly, there's just not a lot you can do. There are too many levels of abstraction between even the ATA interface and the actual storage to do anything meaningful at the level of the magnetic surface. And all of this was pretty much true in the late '00s, even before SSDs took over.

So what about SpinRite?

SpinRite dates back to 1987 and is apparently still under development by its creator Steve Gibson. Gibson is an interesting figure, one of the "Tech Personalities" that contemporary media no longer creates (insert comment about decay in the interest of capitalism here). Think Robert Cringely or Leo Laporte, with whom Gibson happens to cohost a podcast. In my mind, Gibson is perhaps most notable for his work as an early security researcher, which had its misses but also had its hits. Through the whole thing he's run Gibson Research Corporation. GRC offers a variety of one-off web services, like a password generator (generated, erm, server-side) and something that displays the TLS fingerprint of a website you enter. There's a user-triggered port scanner called ShieldsUp, which might be interesting were it not for the fact that its port list seems limited to the Windows RPC mapper and some items of that type... things that were major concerns in the early '00s but rarely a practical problem today.

It's full of some gems. Consider the password generator...

What makes these perfect and safe? Every one is completely random (maximum entropy) without any pattern, and the cryptographically-strong pseudo random number generator we use guarantees that no similar strings will ever be produced again. Also, because this page will only allow itself to be displayed over a snoop-proof and proxy-proof high-security SSL connection, and it is marked as having expired back in 1999, this page which was custom generated just now for you will not be cached or visible to anyone else. ... The "Techie Details" section at the end describes exactly how these super-strong maximum-entropy passwords are generated (to satisfy the uber-geek inside you).

You know I'm reading the Techie Details. They describe a straightforward approach using AES in CBC mode, fed by a counter and its own output. It's unremarkable except that just about any modern security professional would have paroxysms at the fact that he seems to have implemented it himself. Sure, there are better methods (like AES CTR), but this is the kind of thing where you shouldn't even really be using methods. "I read it from /dev/urandom" is a far more reassuring explanation than a block diagram of cryptographic primitives. /dev/urandom is a well-audited implementation, whatever is behind your block diagram is not. Besides, it's server side!

My point is not so much to criticize Gibson's technical expertise, although I certainly think you could, but to say that he doesn't seem to have updated his website in some time. A lot of little details like references to WEP and the fact that the PDFs are Corel Ventura output support this theory. By association, I suspect that GRC's flagship product, SpinRite, doesn't get a lot of active maintenance either.

Even back around 2007 when I first encountered SpinRite it was already a little questionable, and I remember a rough internet consensus of "it likely doesn't do anything but it probably doesn't hurt to try." A little research finds that "is SpinRite snake oil?" threads date back to the Usenet era. It doesn't help that Steve Gibson's writing is pervaded by a certain sort of... hucksterism. A sort of ceaseless self-promotion that internet users associate mostly with travel influencers selling courses about how to make money as a travel influencer.

But what does SpinRite even claim? After a charming disclaimer that GRC is opposed to software patents but nonetheless involved in "extensive ongoing patent acquisition" related to SpinRite, a document titled "SpinRite: What's Under the Hood" gives some details. It's undated but has metadata pointing at 1998. That's rather vintage I see several reasons to think that there have been few or no functional changes in SpinRite since that time.

SpinRite is a bootable tool based on FreeDOS. It originated as an interleaving tool, which I won't really explain because it's quite irrelevant to modern storage devices and really just a historic detail of SpinRite. It also "introduc[ed] the concept of non-destructive low-level reformatting," which I won't really explain because I don't know what it means, other than it seems to fall into the broad category of no one really knowing what "low level formatting" means. It's a particularly amusing example, because most modern software vendors use "low level formatting" to refer explicitly to a destructive process.

SpinRite "completely bypasses the system's motherboard BIOS software when used on any standard hard disk system." I assume this means that SpinRite directly issues ATA commands, which probably has some advantages, although the specific ones the document calls out seem specious.

In reference to SpinRite's data recovery features, we read that "The DynaStat system's statistical analysis capability frequently determines a sector's correct data even when the data could never be read correctly from the mass storage medium." This is what I remember as the key claim of SpinRite marketing over a decade ago: that SpinRite would attempt rereading a block a very large number of times and then determine on a bit-by-bit basis what the most likely value is. It seems reasonable on the surface, but it wouldn't make much sense with a drive with internal error correction. That's universal today but I'm not sure how long that's been true, presumably in the late '90s this was a better idea.

That's probably the high point of this document's credibility. Everything from there gets more suspect. It claims that SpinRite has a proprietary system that models the internal line coding used by "every existing proprietary" hard drive, an unlikely claim in 1998 and an impossible one today without a massive reverse engineering effort. Consider also "its second recovery strategy of deliberately wiggling the drive's heads." It seems to achieve this by issuing reads to cylinders on either side of the cylinder in question, but it's questionable if that would even work in principle on a modern drive. You must then consider the use of servo positioning on modern drives, which means that the head will likely oscillate around the target cylinder before settling on it anyway.

This gives the flavor of the central problem with SpinRite: it claims to perform sophisticated analysis at a very low level of the drive's operation, but it claims to do that with hard drives that intentionally abstract away all of their low level details.

A lot of the document reads, to modern eyes, like pure flimflam, written by someone who knew enough about HDDs to sound technical but not enough to really understand the implications of what they were saying. The thing is, though, this document is from '98 and the software was already a decade old at the time! The document does note that SpinRite 3.0 was a complete rewrite, but I suspect it was the last complete rewrite and probably carried a lot of its functionality over from the first two versions.

I think that SpinRite probably does implement the functionality that it claims and that those features might have been of some value in the late '80s and much of the '90s. Then technology moved on and SpinRite became irrelevant. Probably the only thing that SpinRite does of any value on a modern drive is just rewriting the entire addressable area, which gives the controller an opportunity to detect bad blocks and remap them. That should also happen in the course of normal operation, though, and even tools dedicated to that purpose (like the open-source badblocks) are becoming rather questionable in comparison to the preemptive capabilities of modern HDDs. This type of bad-block-detecting rewrite pass is probably only useful in pathological cases on older devices, but it's also the only real claim of the vast majority of modern "hard drive repair" software.

It seems a little mean-spirited to go after GRC for their old software, but they continue to promote it at a cost of $89. The FAQ tells us that "SpinRite is every bit as necessary today as it ever was — maybe even more so since people store so much valuable personal 'media' data on today's massive drives." I resent the implication of the scare-quoted "media," Mr. Gibson, but what I do with my hard drives in my own home is none of your business.

The FAQ tells us "SpinRite is often credited with performing "true miracles" of data recovery," but is oddly silent on the topic of SSDs. Some dedicated Wikipedia editor rounded up a number of occasions on which Gibson said that SpinRite was of limited or no use with SSDs, and yet the GRC website currently includes the heading "Amazingly effective for SSDs!" There is no technical explanation offered for how SpinRite's exceptionally platter-centric features affect an SSD, nor mention of any new functionality targeting flash storage. Instead, there's just anecdotal claims that SpinRite made SSDs faster and a suggestion that the reader google a well-known behavior of flash storage for which SSD controllers have considerable mitigations.

It is an odd detail of the GRC website that most of the new information about the product is provided in the form of video. Specifically, videos excerpted from recent episodes of Gibson and Laporte's podcast "Security Now." Security Now is weekly, so I don't think that SpinRite promotional material makes up a large portion of it, but it does seem conspicuous that Gibson uses the podcast as a platform for 15 minute stories about how SpinRite worked miracles. These segments, and their mentions of how SpinRite is a very powerful tool that one shouldn't run on SSDs too often, absolutely reek of the promotional techniques behind Orgone accumulators, Hulda Clark's "Zapper," and color therapy. It is, it seems, quack medicine for the hard drive.

I don't think SpinRite started as a scam, but I sure think it ended as one.

A lot of this was already apparent back in the late '00s, and I can't honestly say that bootleg copies of SpinRite every improved anything for me. So why did I love it so much? The animations!

SpinRite's TUI was truly a work of art. Just watch it go!.

[1] I recently bought the 20th anniversary vinyl box set of Meteora, which emphasizes that (1) 20 years have passed and (2) I am still a loser.

[2] This kind of visible failure seems uncommon with SSDs, likely because SSD controllers tend to read out the flash in a critical, suspicious way and take preemptive action when the physical state is less than perfectly clear cut. In a common type of engineering irony, the fact that flash storage is less reliable than magnetic media requires aggressive management of the problem that makes the overall system more reliable. Or at least that's what I tell myself when another SSD has gone completely unresponsive.

[3] Honestly this doesn't seem to have been typical with any hard drives by the microcomputer era, which makes perfect sense if you consider that these hard drives were sold with bad sector lists and therefore must have been factory tested. The whole "low level formatting" thing has been 70% a scam and 30% confusion with the very different technical tradition of magnetic diskettes, since probably 1990 at least.

Some things have been made nearly impossible to search for. Say, for example, the long-running partnership between Epson and Catalina: a query that will return pages upon pages of people trying to use Epson printers with an old version of MacOS.

When you think of a point of sale printer, you probably think of something like the venerable Epson TM-T88. A direct thermal printer that heats small sections of specially coated paper, causing it to turn black. Thermal paper of this type is made in various widths, but the 80mm or 3 1/8" used by the TM-T88 is the most common. The thermally-reactive coating on the paper incorporates some, umm, questionable chemicals, but moreover, the durability of direct thermal prints is poor. The image tends to fade over not that long of a timespan. Besides, the need for special paper is an irritation.

So, there are other technologies available. Thermal transfer, in which a ribbon of ink (I suspect actually a thermoplastic) is pressed against the paper and heated to cause the ink to stick, is often used for more durability-sensitive applications like warehouse labeling. The greater flexibility of paper (or plastic) stock sees thermal transfer used in specialty applications as well, like conference attendee badges. Thermal transfer printers tend to be more expensive and more complex than direct thermal, though, and are rarely used at the POS.

Impact printers are actually fairly common in a POS-adjacent application. These printers punch metal pins against an inked ribbon, pushing it against the paper to leave a mark. Impact printers were actually the norm for receipt printing prior to the development of inexpensive thermal printers. They remain popular in restaurant kitchens: the plain paper they use is less readily damaged by oils, and won't turn entirely black if exposed to too much heat, as might happen when a ticket is clipped above a grill. Impact receipt printers today are often referred to as kitchen ticket printers as a result.

Impact receipt printers, and many impact printers in general, have a neat trick: you can manufacture an ink ribbon in two colors, say, black on one half and red on the other. By either using two sets of impact pins or shifting the position of the impact head, either black or red can be printed. Dual-color printers with black and red ribbons became ubiquitous for kitchen tickets, although the red doesn't tend to reproduce well from an old, dry ribbon.

The ability of impact printers to use plain paper had another advantage: slip printing. A slip printer is a device intended to print characters on a small piece of paper inserted into it. Historically they were often used by bank tellers to print account and reference numbers onto deposit slips, for later auditing. In other applications they functioned as more sophisticated "received" stamps, adding not just the time and date but customer account or transaction numbers to received paperwork. The legal profession has a tradition of "Bates numbering," which traces its history to a rather different printing device, but Bates numbers could be applied by slip printers as well. In this case, of course, we would need to refer to them as Generic Sequential Page Numbers, Compare to Bates (TM).

A variant of the slip printer, really a receipt printer (often thermal) and slip printer (often impact) married into one box, is known as a check validator. Very common in grocery stores until recently, these printers both produced receipts and printed an audit number and endorsement on the back of the check a customer might offer in payment. It's difficult to imagine paying for groceries with a check, but it used to be a common practice. For many years, the practicalities of accepting checks were a major driver of POS technology. When a cashier rung you up, there were two options: they pushed the cash button, and the POS "bumped" the cash drawer open, or they pushed the check button, and the POS sent an endorsement to the check validator. The close coupling of these two features means that cash drawer bumping is traditionally the task of the receipt printer, and cash bump outputs are common to this day.

But where, exactly, is this tour of POS printing technology taking us? Well, you might notice the absence of the humble inkjet. It might seem surprising: inkjet mechanisms can actually be quite compact, and they tend to be a natural evolution of impact printing. Well, there are indeed inkjet printers in the receipt printer class, but there are some practical considerations. Moving a smaller print head across the paper in bands requires a more complex mechanism, and it's slow compared to printing in one pass. Inkjet heads large enough to span the whole width of the receipt tape are fairly expensive.

And after all that, inkjet seems high maintenance compared to the almost bulletproof reliability of direct thermal printers. Consider the state of the average gas pump "CRIND" (Card Reader In Dispenser) receipt, and then consider that the small thermal mechanism is still managing to produce that output after many years in the harsh conditions of the outdoors. Inkjets tend to quickly malfunction without some sort of automated mechanical cleaning, and that's under office conditions.

So, to put it succinctly, inkjet receipt printers just aren't popular.

You could make similar comments about office printers, where inkjet suffers in many ways when compared to laser or LED printers. But they have been a tremendous success at the lower end of the market. There are a few reasons for this outcome, but one of the bigger ones is color: for a laser or LED printer to produce color used to be rather complicated. In the '00s, many inexpensive color laser printers were "four-pass" printers: the page had to be looped through the print engine four times, one for each color! It saved a lot of parts but made printing more than four times slower. Inkjets were far from this problem. It's a fairly simple matter to make an inkjet print head that serves multiple colors in one assembly!

The same ideas are applicable to receipt printers. If you, for some reason, want a full-color receipt, inkjet is the way to go. But no one wanted a full-color receipt. Even dual-color impact printers disappeared into the kitchen.

And then a company called Catalina came along. Catalina keeps a somewhat low profile among consumers, certainly lower than the MacOS release. Search results suggest lower even than the island off of Los Angeles, for which the company, and the MacOS release, are named. There's no Wikipedia article about Catalina, and their own About Us is brief and made up mostly of nonsense like this:

Transforming data into insights, and insights into action through a seamless consumer experience that drives results.

Catalina is one of those companies that you never think about, but that is constantly thinking about you. Today we would call it ad-tech.

Catalina is tough to research. Obviously they did not intentionally choose a name that would become a MacOS release; they were using the Catalina name many years earlier. But it does seem like they have participated in a bit of obfuscation. Today, they continue to advertise a charming phone number: 1-800-8-COUPON. This "translates," of course, to 1-800-826-8766. During the 1990s they ran numerous classified ads using this phone number, but the numeric version instead of the easier to remember "vanity" representation. The ads were for advertising associate positions, but curiously did not mention the name of the company at all.

Actually, some of these ads give a slightly different phone number, 1-800-826-8768. It is quite conceivable that both phone numbers were issued to the company, given the different toll-free number industry of the '90s. But the fact that OCR frequently confuses these two numbers leads one to suspect that some of the 8768 ads may have been a copy mistake.

Even better, a few of the ads for the 8768 number, and one ad with the 8766 number, do give the name of a company, but an unfamiliar one: Aquarius Enterprises.

Aquarius Enterprises was a "register tape advertising" or "receipt back advertising" venture. In other words, they sold advertising on the backs of receipts. Curiously, while Catalina mentions their 40-year history, Aquarius Enterprises calls themselves "the most successful register tape advertising" for "over 25 years"... in 1993. Are they the same company? Well, they used the same phone number. Catalina is headquartered in St. Petersburg, Florida today, but seems to have moved, as early articles describe then as Anaheim-based... rather closer to the El Segundo address often used by Aquarius Enterprises.

Perhaps it is a coincidence of similar phone numbers and similar industries, but I strongly suspect that Catalina was a spin-out of Aquarius Enterprises. I tried finding shared employees, but there is remarkably little information about Aquarius Enterprises outside of their classified ads for sales associates. But then, once again, it's not an easy name to search for.

Whatever its origins, Catalina launched in 1985 with "Coupon $olutions." Besides the cringeworthy name, this venture was remarkably similar to what consumers will know them for today: Coupon $olutions consisted of software that recorded a consumer's purchases at the POS, and then printed on-demand targeted coupons.

Early articles about Catalina describe the system as relatively simple. Coupons would be printed for "complimentary items." For example, the purchase of baby food would result in an coupon for diapers. The coupons themselves were also simple: printed in monochrome on tape with a distinctive printed edge.

Coupon $olutions debuted at two Boys markets in Los Angeles. It grew fast. By 1990, Catalina's coupon printers were installed in 3,300 grocery stores nationwide. Newspaper coverage started to mention privacy concerns in the 1990s, waving them away with Catalina's assurances that there was no privacy concern because they tracked only purchases and not the shopper's identity. Of course, in the late '80s Catalina had trialed a shopper loyalty card program that would rather change that situation, but it seems to have been unsuccessful.

As time passed, Catalina expanded further into retail technology. They opened their own clearinghouse service for coupons, and marketed their on-demand coupon system to stores as an analytics product, since it provided real-time reporting on purchases (in this era even large retailers would often not have granular, fast reporting from their POS system).

The 1990s treated Catalina well, but they seem to have flown a little too close to technology, and the dot com bust hit them as well. In the early '00s, they weathered layoffs, an accounting probe, and a stock dive. Still, 2005 brought a big step forward: color.

Yes, we're finally back to the point. Catalina Marketing partnered with Epson to introduce a special variant of the TM-C610 color receipt printer, called the TM-C600. Called the CMC-6 by Catalina, the printer uses a full-width inkjet head to produce 360 DPI full color on 57.5mm paper.

Lately, though, you may have noticed these printers yielding unsatisfactory results. When I've gotten Checkout Coupons at all, they've been barely legible or, increasingly, completely blank. Curious.

Catalina went bankrupt in 2018, and underwent a reorganization. The company emerged, but apparently not by that much, as it went bankrupt once again in 2023. Catalina offers a fully managed service, meaning that they ship stores new ink cartridges when remote monitoring of the printers indicates that it will be needed. I have a suspicion that Catalina's second bankruptcy has introduced some disruptions. And yet, in an article they claim:

Catalina is assuring clients and shoppers that it’s still business as usual, and ongoing promotions won’t be affected. “There will be no interruption in Catalina’s ability to serve its customers or any impact on how it works with them,” Catalina says.

I'm not sure that this is working out, even a year into the bankruptcy process. Safeway/Albertsons has apparently decided to remove the Catalina printers entirely. Smith's (Kroger) doesn't seem to maintain them at all. Walgreens is apparently more committed to the cause, as they are with the cooler screens, but even there checkout coupons have become inconsistent.

Besides, I don't think even Catalina views the printers as very important any more. They're relegated to a small corner of Catalina's website, with the vast majority of their marketing material dedicated to analytics, targeting, and digital marketing. Catalina seems to be a major player in the in-app digital coupons now emphasized by a lot of grocers, although I've personally found the system to be laughably unusable. But it's not surprising that you get a laughably unusable app from an industry that churns out this kind of copy:

84.51° currently delivers personalized promotional offers to Kroger’s digitally engaged shoppers via its website, mobile app, and more broadly via its Loyal Customer Mailer. Catalina Reach Extender is a complementary solution to the way current offers are delivered and will expand the impact of promotional offers by aligning those offers to the way customers shop – in-store, online or both.

As far as I can tell, this press release is just describing making digital coupons (managed by a company that is, improbably, called 84.51°) also print out on the Catalina printers. The ones that barely work any more. Well, that was January of '23, they didn't know about the second bankruptcy yet.

Catalina may date to 1985, but it's sort of a case study in the advertising industry. It's a huge, publicly traded company, with a market cap that's reached at least $1.7 billion, and two bankruptcies. They write such obtuse copy that it's hard to understand what exactly they do these days, which is probably mainly a way to distract from the fact that their main business is now collecting and selling consumer data. And I would say that no one likes them... subreddits of retail employees are full of comments expressing relief when the Catalina printers would break, since unplugging them would result in multiple phone calls a day from Catalina investigating the "problem."

BUT: there are couponers.

That's right, there's a whole internet subculture that is obsessed with these checkout coupons. They catalog the coupons on offer, and document the process for requesting a replacement coupon from Catalina when the one you expected failed to print. So very strange to me, a reminder of the many people out there and their many strange hobbies.

Why would you ever waste your time on these coupons? I have real things to do, like collecting thermal printers.

For those of you who are members of the Matrix project, I wanted to let you know that I am running for the Governing Board, and a bit about why. For those of you who are not, I hope you will forgive the intrusion. Maybe you'll find my opinions on the topic interesting anyway.

I am coming off of a period of intense involvement in an ill-fated government commission, and I wanted to find another way to meaningfully contribute to the governance of something I care about. Auspiciously, the newly constituted Matrix foundation is forming a governing board. I am up for one of the individual member seats.

Why do I care?

Instant messaging is a fascinating case study in the history of technology. It is nearly as old as networked computing, and you could make a decent argument that it is older, running only into dithering around the definitions. We've always wanted to communicate, and text has always been an obvious option. It is probably because of the obviousness of instant messaging that it has repeatedly been coopted by commercial interests.

You don't have to be very old to have lived through several iterations of this process. I'm not quite the right person to remember ICQ fondly; for me it was AIM. But what I remember most fondly is more obscure: XFire. It had an in-game overlay and killcounter integration, both critical features for my computer habits that consisted heavily of Jedi Knight: Jedi Academy. That isn't actually important, I'm just reminiscing, but I think most people have a story like this.

If you have read much of my back catalog you know that I am not always optimistic about federated systems. They face a lot of challenges, which range from the technical complexity of changing federated protocol specifications to a whole category of opposing forces that can be vaguely chalked up as capitalism. And yet, textual communications bring us what is probably federation's greatest and most enduring success: email. Email is also a cautionary tale in a lot of ways, but it gives us a cause for optimism.

The history of federated messaging is rather more varied. XMPP was, in its heyday, nearly on track to mass adoption. High quality clients emerged, XMPP was adopted by grassroots projects and then, as at least an implementation detail, by Facebook and Google. We all know what happened. I think most people today are too quick to blame XMPP's downfall on inconsistent implementation of protocol extensions (XEPs) rather than complete cooption by two of the era's largest internet companies, but to be clear, inconsistent implementation was indeed a problem.

Matrix and Me

I have used Matrix as my main, day-to-day messaging solution since 2016. I have also operated a homeserver with open registration for that entire span. In some ways this has been a rather passive venture, but as the user count of that homeserver has grown I've struggled more with performance and moderation issues. A few months ago things tipped over and I had to spend a weekend doing some serious work on both fronts. This lead me to pay a lot more attention to the Matrix project and the state of the art.

I wish that I had been more involved in the Matrix project to date, but I try very hard to avoid software engineering, and Matrix governance and community efforts, the area that matters to me most, have often been hard for me to follow. This situation has improved significantly recently, and I think that the Matrix foundation deserves enormous credit for the work they have done to pick up the level of community engagement.

Of course I come to the topic with some opinions. Who would expect anything less?

Polish over Features

The Matrix project, especially as personified by Element, has added a huge number of new features. It's hard to call this a bad thing, and some of them have been notable successes. For example, E2E is a challenging feature to deliver, but has indeed become table stakes for a messaging product that attracts a privacy-minded userbase.

Still, there is one criticism of Matrix that has remained constant over its entire lifespan, and its one that needs to be attended to: the level of consistency, usability, and polish.

Polish is tricky in a federated system. It's more the domain of clients than the protocol, but the protocol directly affects the situation by determining how easy it is to develop and maintain high-quality clients. For many years it was clear that the change rate in the Matrix protocol made it difficult to develop a good client. Element often felt like the only complete client, and even it was pretty rocky. Fortunately there has been a lot of progress; Element has greatly improved and the stable of third-party clients like my own choice, Nheko, has a lot to offer.

Still, there's a lot of progress to be made. Matrix competes directly with commercial products that come from vendors with a heavy focus on usability and user experience. It only takes one instance of the dreaded "Unable to decrypt" for casual users to bounce. Element continues to be a de facto "primary" implementation that can make the road more difficult for others.

I think that protocol changes should be evaluated conservatively, with an eye towards providing a level of stability that enables multiple top-tier clients. The Matrix Foundation should actively seek ways to support the enhancement and maintenance of clients beyond Element, supporting the healthy ecosystem of independent implementations that are required for an open protocol to be sustainable.

Moderation

Moderation is one of the great struggles of the internet, if not the greatest. Some advocates of federated systems opine that they make moderation easier or more tractable. I disagree; while federation enables more flexibility in how users experience moderation it makes many of the underlying problems more difficult. Moderation decisions across the system are made in an ad-hoc, distributed way. The rich network of homeservers presents many opportunities for bad actors, including every poorly maintained (or unmaintained) node.

Matrix imposes a moderation challenge at two levels: within communities and within homeservers. Relatively good tools exist at the community level, but still, too many basic functions require introducing the Mjolnir moderation bot. At the level of the homeserver, moderation tools are frustratingly limited. The administration API is minimal in severely limiting ways and there do not appear to be any complete implementations of a client for it.

I applaud the various efforts that have popped up, things like the community moderation initiative's blocklist effort and the "awesome technologies" Synapse administration tool. But we need more, and we need more in two ways.

First, we need technical progress. The in-protocol moderation capabilities of Matrix should be improved over time with a north-star vision of eliminating Mjolnir, an approach to community moderation that was carried over from IRC but probably should have stayed there. The Synapse admin API should be improved and better tooling around it developed.

Second, we need progress in governance. I would like to see an open initiative to develop best practices for moderation of communities and homeservers. This can include the development of shared blocklists through a documented, auditable process (although not necessarily an open one, for reasons of user privacy). I would like to see a sincere effort to advance the state of the art in distributed moderation, bringing together diverse users to learn their concern and developing tools to make consistent and active moderation the default.

The number of independently operated homeservers in Matrix can be a strength, but in this area it can be a weakness. ActivityPub, with its heavier orientation towards public discussion, has served as a laboratory for abuse and moderation issues. Matrix could learn a lot from the efforts going on in the Mastodon community, for example, towards practical means of moderating across instances.

For homeserver operators, moderation is an immense practical concern due to risks from load and CSAM. The volume of CSAM traffic on Matrix, while not a problem beyond solving, seems badly under-discussed and particularly calls for some sort of distributed moderation program to relieve public homeserver operators of ongoing whac-a-mole. Sometimes a graph is only as strong as its weakest node---this is the kind of hard problem we have to take on to build a sustainable future for federated systems, and we should take it on enthusiastically.

I would like to see the Matrix project boldly take on moderation at multiple levels. First, improving the moderation tools and capabilities of the Matrix protocol should always be part of the discussion. Second, I would like to see the Matrix Foundation support the development of improved moderation and abuse tools, preferably including them as part of Synapse or providing a very easy setup process so that good abuse management can be the norm rather than the exception. Third, I would like the Matrix foundation to facilitate community discussion around best practices, tools, and techniques for moderation.

Not everyone will agree on the way to perform moderation, or even the goals of moderation. That's the nature of the internet, and more broadly of communications. We can't let it stop us from trying. This can be one of the hardest areas to build consensus, but that will always be the case, and so we need to include the inherent social complexity of moderation as part of the technical requirements. Once again: we need to be bold and take on the hard problems, and this might be the hardest.

Chat, First and Mostly

One of the concerning trends I have seen in a lot of adjacent nonprofit tech projects lately is dilution of mission. We could also call this "distractions." Unfortunately, Matrix has not been immune. The most obvious example is Third Room, the Matrix metaverse project. I want to temper my criticism by saying that the level of effort devoted to Third Room has evidently been low, but I think that the optical problem created by Third Room (the appearance that Matrix has been capered, one might even say Zuck'd, into a distracting focus on the latest trend) is certainly real. For a community venture, appearances are important, and this means applying discipline in how side projects are presented, especially in this era of so many projects presaging their downfall with some buzzword-reaction initiative.

I might go just a bit further. I don't think that the VoIP features of Matrix (voice and video communications) are a bad idea per se, but I think that that's a complex problem space and the current landscape of instant messaging products suggests that it's not a particularly important one. In other words, people seem happy to do their voice/video chat in a different product than their text chat. You could say that this presents an opportunity for Matrix: to double down on providing a best-in-class textual messaging experience, without having to expend significant resources on real-time media.

I wouldn't want to see existing features removed, but I think that features other than core instant messaging should be deprioritized, at least in the short term.

Onboarding

Sometimes caring a lot about onboarding can be kind of gross. It has the scent of focusing on conversions. But it's a really important issue for IM, when the onbarding experience of a lot of the other options is "you already have it." The Matrix Foundation is well-positioned to demonstrate leadership in the onbaording experience, across the protocol, clients, and public communications. Let's make Matrix easy to get into.

A Consistent Direction

I don't want to dwell too long on how many times a certain prominent Matrix client has been renamed, launched new App Store listings, etc. It's old news and fortunately things seem to have settled down. Still, I think a lot of reputational damage happened that has not fully been forgotten. This history serves as a reminder that significant user-facing changes need to be made carefully. New social applications in general, and especially federated ones, have a bad reputation for churn. The most successful are often the most boring. Let's think carefully about things, and look before we leap.

What Do You Think?

I have a lot of opinions and of course all of them are correct, but usually only in my eccentric construction of reality. Your experience may vary. Please feel free to reach out with your thought on the Matrix project, an offer that stands whether I'm elected or not, because I love to talk about it.

And that concludes my stump speech. I'll be back again soon with a normal post about some useless trivia. I think it might be about a specific kind of printer that you've probably seen but not thought much about, other than slight irritation. I'm also spending some time right now playing video games^w^w^w working on a more ambitious writing project that is out of my normal lane but you might still enjoy. It's about dogs. It's also very sad and I'm not entirely sure what to think about it. You'll see what I mean if I ever finish.

I haven't written anything for a bit. I'm not apologizing, because y'all don't pay me enough to apologize, but I do feel a little bad. Part of it is just that I've been busy, with work and travel and events. Part of it is that I've embarked on a couple of writing projects only to have them just Not Work Out. It happens sometimes: I'll notice something interesting, spend an evening or two digging into it, but find that I just can't make a story out of it. There isn't enough information; it's not really that interesting; the original source turned out to just be wrong. Well, this one is a bit of all three. Join me, if you will, on a journey to nowhere in particular.

One of the things I am interested in is embedded real-time operating systems. Another thing I am interested in is Unified Communications. Yet another is failed Microsoft research projects. So if you've ever heard of Microsoft At Work, you probably won't be surprised that it has repeatedly caught my eye. Most likely, you haven't heard of it. Few have; even the normal sources of information on these kinds of things appear to be inaccurate or at least confused about the details.

Microsoft went to work in the summer of 1993, or at least that's when they announced Microsoft At Work. This kind of terrible product naming was rampant in the mid-'90s, perhaps more from Microsoft than usual. MAW, as I and a few others call it, was marketed with a healthy dose of software sales obfuscation. What was it, exactly? an Architecture, Microsoft said. It would enable all kinds of new applications. With MAW, one would be able to seamlessly access the wealth of information on their personal computers. Some reporters called it an Environment. Try this for a lede: "Microsoft Corp. unveils integrated computer program."

The announcement included a demo that got a lot more to the point: a fax machine that ran Windows.

Even this was strangely obfuscated: enough newspaper reports described it as a "fax like product" that I think this verbiage was sincerely used in the announcement. Today, we would refer to MAW as an effort towards "smart" office machines, but in 1993 we hadn't quite learned that vocabulary yet. Microsoft must have been worried that it would be dismissed as "just a fax machine." It couldn't be that, it had to be something more. It had to be a "fax like product," built with "Windows architecture."

I am being a bit dismissive for effect. MAW was more ambitious than just installing Windows on a grape. The effort included a unified communications protocol for the control of office machines, including printers, for which a whole Microsoft stack was envisioned. This built on top of the Windows Printing System, a difficult-to-search-for project that apparently predated MAW by a short time, enough so that Windows Printing System products were actually on the market when MAW was announced---MAW products were, we will learn, very much not.

Windows Printing System modules were sold for at least the HP LaserJet II and III. If you did not experience them, these printers placed their actual rasterization logic onto a modular card that could be swapped out, usually to switch between PCL or PostScript "personalities." The PostScript module was offered mostly for MacOS compatibility, Apple having selected PostScript as a common printer control language. The Windows Printing System module took this operating system specialization a step further, using Windows' simple GDI graphics protocol to draw output to the printer.

I am actually a little unclear on whether or not the Windows Printing System lead directly to the cheap "WinPrinters" that are also associated with the idea of GDI-based printing. "WinPrinters," so-called by analogy to WinModems, are entirely dependent on the host computer to perform rasterization. While extremely irritating from the perspective of software support, this was an important cost-savings measure in consumer printers. Executing a capable printer control language was rather demanding; the Apple LaserWriter famously had a faster processor than the Macintosh computers it was a peripheral to. Printers with independent rasterization, particularly the more complex PostScript, came at a substantial price premium to those that required the host to perform rasterization.

While some details of reporting on the Windows Printing System make me worry that it was in fact rasterizing on device (like the curiously specific limit of "up to 79" TrueType fonts), I'm fairly sure it was indeed a precursor to the later inexpensive designs. Rather than a cost-savings measure, though, Microsoft seems to have marketed it as a premium feature. Because of the Windows Printing System's higher level of integration with the operating system, it brought numerous new features, many of which we take for granted today. TrueType font support at all, for example, a cutting-edge feature in '93. Duplex control from the print dialog rather than the printer's own display, and for that matter, the ability to see printer status messages (like "PC LOAD LETTER") on the computer you just printed from.

And at the end of the day, offloading rasterization from the printer had an advantage: the Windows Printing System was faster than PCL or PostScript.

Even if it did become the dominant printing method years later, the Windows Printing System of the MAW era doesn't seem to have fared very well. Because it took the position of an add-on cartridge (like a font cartridge), it would have been an added-cost option for printer buyers---an added cost of $132.99, according to a period advertisement. The dearth of available documentation or even post-launch advertising for the Windows Printing System cartridge suggests disappointing sales numbers.

The fortunes of Windows Printing Technology would turn a year later, though, as Lexmark introduced their WinWriter series: "With the Microsoft Windows Printing System Built In!" Speaking of the Lexmark WinWriter series, this whole printing thing is kind of a tangent. What about MAW? The Windows Printing System, it seems, was not really a part of MAW. It was just generally related and available when MAW was announced, so it was rolled into the press conference. It is a bit ironic that the Lexmark WinWriter, truly the Printer for Windows, was not a MAW device despite shipping well after MAW was announced.

So, back to the course: MAW was not just Windows on a fax machine, not just the Windows Printing System, but an integrated system of Windows on a fax machine, the Windows Printing System, a generalized network protocol, and apparently a page description language. This was all, as you can see, rather document-focused. MAW would allow Windows users to easily, seamlessly interact with these common office machines, sending and receiving documents like it was 1999.

And later, it would do more: Microsoft was clear from the beginning that MAW had a higher vision, one that is remarkably similar to the later concept of Unified Communications. Microsoft envisioned Windows on a phone, bringing desk phones into the same architecture, or environment, or whatever. Remember the phone part, it comes back.

In practice, MAW would do nothing. It was a complete and total failure. It took two years for the first MAW office machine to reach the market, a Ricoh fax machine. Fortunately, a television commercial has been preserved, giving us a small window into the Windows on a Fax Machine experience. "Microsoft's At Work Still Loafing on the Job," is how the Washington Post put it in 1995.

They call it "the first real step toward the paperless digital office," a nod towards the promise of Microsoft's document-messaging vision, before noting that virtually no products had shipped, everything was behind schedule, and Microsoft had reorganized the At Work team out of existence. Microsoft At Work was seldom spoken of again. Few products ever launched, those that did sold poorly (the Windows licensing fee imposed on them being one of several factors contributing to noncompetitive price tags), and by the time Windows gained proper USB support few would remember it had ever happened.

In other words, a classic Microsoft story.

But I'm not here to chronicle Microsoft's foibles, there are other writers for that. I'm here to chronicle their weird operating system projects. And that's what got me reading into MAW: the promise of not just one, but two weird operating system projects.

Regard that promise with suspicion.

Wikipedia tells us that MAW included "Microsoft At Work Operating System, a small RTOS to be embedded in devices." That's very interesting. I love a small RTOS to be embedded in devices! Tell us more.

Researching this MAW embedded operating system turns out to be a challenge. You see, it is not the better known of the operating systems produced by the MAW initiative. That would be WinPad, curiously not mentioned at all in the MAW Wikipedia article, but instead in the Windows CE article, as a precursor to CE. Windows CE gets a lot more affection than MAW, and so we know quite a bit more about WinPad. It was an early attempt at an operating system for a touchscreen mobile device, one that, in classic Microsoft fashion, competed internally with another project to build an operating system for a touchscreen mobile device (called Pegasus) and died out along with the rest of MAW.

It was based on 16-bit Windows 3.1, using a stripped-down UI layer that resembled Windows 95. Probably not coincidentally, there seems to have been an effort to port WinPad onto Windows 95, and fortunately developer releases of WinPad have been preserved. With some effort, you can get them running on top of appropriate Windows versions in an emulator.

WinPad was envisioned as a core part of MAW, the key enabler of that paperless office. With MAW and WinPad, you could synchronize documents, emails, and faxes, everything you could ever want in 1995, onto your handheld device and then carry it with you. WinPad also didn't work. Evidently the performance was lousy and it required entirely unrealistic battery capacities. Not a surprising outcome when one ports a mid-'90s desktop operating system to a tablet. How charming! But not exactly my target. What about this RTOS?

If you dig into these things for too long, you start to question your life, or at least reality. References to this MAW embedded operating system are so sparse that I quickly started to wonder if it existed at all, or if it was simply confused with WinPad. This MAW OS would run directly on the office machines. Is it possible that it was, in fact, WinPad that ran on a fax machine? Or at least that whatever ran on the fax machine was a direct precursor to WinPad, an earlier new UI layer on top of 16-bit Windows?

The nagging thing that kept me on the hunt for this MAW embedded OS was, oddly enough, the Sega Saturn. A series of newspaper archives, many gathered by Mega Drive Shock, tell an interesting story. Microsoft, it seemed, had been contracted to provide the operating system for the Sega Saturn. Well, this seems to have been a misconception, although clearly a period one. As the news cycle carried on, the scope of this Microsoft-Sega partnership (at first denied by Microsoft!) was reduced to Microsoft providing some sort of firmware related to the Saturn's CD drive.

There is, though, a tantalizing detail. The Electronics Times reported that "Microsoft looks set to port its Microsoft At Work operating system to Hitachi's new SH series of microprocessors." The article explicitly linked the porting to the Saturn effort, but also mentioned that the MAW operating system was being ported to Motorola 68000.

Do you know what never ran on the Hitachi SH or Super-H architecture? 16-bit Windows.

Do you know what did? Windows CE.

Is it possible? Do you think? Is Windows CE a derivative of Windows for Fax Machines?

I'm pretty sure the answer is no. A reader pointed me at John Murray's 1998 book Inside Windows CE, which provides a brief and presumably authoritative history of the platform. It specifically discusses Windows CE as a follow-on project to the failed WinPad, which it describes as 16-bit Windows 3.1, and goes on to say it "was designed for office equipment such as copiers and fax machines."

It is, of course, possible that the book is incorrect. But given the dearth of references to this MAW embedded RTOS, I think this is the more likely scenario:

MAW devices like the Ricoh IFS77 ran 16-bit Windows 3.1 with a new GUI intended to appear more modern while reducing resource requirements. Some reporters at the time noted that Microsoft was cagey about the supported architectures, I suspect they were waiting on ports to be completed. The fax machine was probably x86, though, as there's little evidence MAW actually ran on anything else.

This operating system was extended for the WinPad project, and efforts were made to port it to architectures more common in the embedded devices of the time like SH and 68000. Microsoft may have reached some level of completion on that project and sold it to Sega for the Saturn's complicated storage controller, but it's also possible that the connection between the Saturn and MAW is mistaken and the software Microsoft delivered to Sega was a simple, from-scratch effort. The strange arc of media reporting on the Microsoft-Sega relationship offers the tantalizing possibility that Microsoft was intended to deliver a complete OS for the Saturn but had to pare it back as a result of problems with porting WinPad, but it seems more likely it just results from an overeager electronics industry press and the Sega NDA that a Microsoft spokesperson admitted to being subject to.

MAW failed to win the market, and WinPad failed to win a BillG review. The project was canceled. From the ashes of WinPad and the similarly failed Pegasus, some of the same people started work on a brand new project, Pulsar, which would become Windows CE.

MAW didn't survive the '90s.

Well, some things are like that. I still got 240 lines out of it.

Update: Alert reader abrasive (James Wah) writes in that they had previously dumped the CD-ROM firmware from the Saturn and performed some reverse engineering. Several things suggest that it was not developed by Microsoft, including a Hitachi copyright notice. It seems likely, then, that the supposed Microsoft-Sega partnership never produced anything or was never real in the first place.

Sometimes, when I am feeling down, I read about failed satellite TV (STV) services. Don't we all? As a result, I've periodically come across a company called AlphaStar Television Network. PrimeStar may have had a rough life, but AlphaStar barely had one at all: it launched in 1996 and went bankrupt in 1997. All told, AlphaStar's STV service only operated for 13 months and 6 days.

AlphaStar is sort of an interesting story on its own. Much like the merchant marine, satellites are closely tied to the identity of their home state. Many satellites are government owned and operated, and several prominent satellite communications networks were chartered by governments or intergovernmental organizations. Consider the example of Inmarsat, a pioneer of private satellite communications born of a UN agency, or Telesat, originally a Crown corporation of Canada. As space technology became more proven, private investors started to fund their own satellite projects, but they continued to operate with the imprimatur of their licensing state.

AlphaStar was sort of an oddity in that sense: a subsidiary of a Canadian company set up to offer an STV service in the United States. Understanding this situation seems to require some background in the Canadian STV industry. 1995 saw the announcement of Expressvu, a satellite television service by telecom company BCE and satellite receiver manufacturer Tee-Comm. Canadian satellite operator Cancom would provide the space segment, and Tee-Comm the ground segment.

Expressvu looked to be headed directly for monopoly: despite attempts by a coalition of Montreal company Power and Hughes/DirecTV to launch a competing service, only Expressvu could meet a regulatory requirement that Canadian broadcast services be served by Canadian satellites. Power's efforts to change the rules involved considerable political controversy as politicians up to the prime minister became involved in the back-and-forth between the two hopeful STV operators.

Foreshadowing Alphastar, both potential Canadian STV operators struggled. Neither Expressvu nor PowerDirecTV would ever begin operations as originally planned. While regulatory uncertainty contributed to schedule delays, and the complexity of still relatively new satellite TV technology drove up costs, one of the biggest problems was a lack of satellite capacity. Most Canadian communications satellites were launched and operated by Telesat, and in the mid '90s Telesat's fleet fit onto a small list. Expressvu had been slated to use a set of transponders on Telesat's Anik E1, but in successive events Anik E1 lost a solar panel and then several of its transponders.

The lack of Canadian satellite capacity created a regulatory conundrum for Canadian STV: Industry Canada was requiring that operators show they had access to satellite capacity in order to obtain an STV license. No capacity was available on Canadian satellites, though. For STV to become available at all in Canada, some compromise needed to be found.

PowerDirecTV and a new satellite venture by Shaw Communications applied for an exception, allowing them to use US satellites until transponders were available on Canadian satellites. Industry Canada was reticent to approve the arrangement, considering the uncertainty over what satellites could be used and when.

As Expressvu failed to get off the ground, several of the partners in the project backed out, and Tee-Comm decided to set off on their own. Considering the licensing situation in Canada, they devised a clever plan: they would launch an STV service in the United States. Such a service, delivering US-made content to US customers, could clearly be served by US-owned satellites according to Canadian policy. But it would also secure long-term satellite carriage agreements and fund the construction of infrastructure. When Tee-Comm later returned to apply for an STV license in the Canadian market, they would have fully operational infrastructure and an existing customer base. They could make a far stronger argument that they would be a reliable, affordable service that could transition to Canadian satellites when capacity allowed.

So Tee-Comm started AlphaStar.

AlphaStar carried over several signs of their Canadian origin, including the basic broadcast technology. They would broadcast DVB-S, the norm overseas but new to the United States where DirecTV and the Dish Network used their own protocols. With DVB-S and more powerful Ku-band transponders on AT&T's Telstar 402R satellite, AlphaStar customers needed a 30" dish---smaller than the C-band TVRO dishes associated with earlier STV, but still larger than the 24" and smaller dishes used with DirecTV's DSS.

Of course, satellite feeds have to come from somewhere. AlphaStar purchased an existing earth station in the town of Oxford, Connecticut and adapted it for television use, adding TVRO antennas to receive programming alongside the large steerable dishes used to transmit to the satellite. An on-site network control center ensured the quality and reliability of their television service; corporate headquarters were located nearby in Stamford.

They never signed up many customers. There may have been a high point of around 40,000, but that wasn't enough to cover the cost of operations. Tee-Comm had barely received authorization to launch the Canadian version of the service (AlphaStar Canada) when they went belly-up in both countries. AlphaStar in the US managed over a year, but AlphaStar Canada only made it a few months. In the mean time, the old Expressvu project, minus Tee-Comm, had finally lurched to life. Expressvu went live in 1997, and the AlphaStar story was forgotten.

During the bankruptcy proceedings in the US and Canada, the courts solicited bids to take over AlphaStar's assets. These included, according to a document prepared by AlphaStar, their Oxford earth station which had been built for the Strategic Defense Initiative and hardened to withstand nuclear attack.

See, this is where I really got interested. An SDI satellite earth station in Oxford? What part of SDI was it built for? I started hunting for the location of this earth station. Not far from Oxford I found an obvious candidate, an isolated facility with a half dozen large, steerable antennas. But no, it was built by Inmarsat and is operated today by Comsat (also originally government-chartered).

Finally, digging through FCC rulings, I found an address: 66 Hawley Road. There was nothing to see there, though, just a tilt-up warehouse for a bearing company that showed no signs of satellite communications heritage. It's funny, Google Maps itself intermittently shows images from before or after the bearing company moved in, but I never noticed that. It took Department of Agriculture aerials from the '90s for me to realize the address was correct; the earth station was demolished just a few years ago.

There are few photos of the building. The best I've seen, from a marketing presentation from one of AlphaStar's successors, is only a partial view. The building doesn't look to be nuclear-hardened, though. It has a glass-walled lobby, and no sign of blast deflectors on its ventilation openings. It seemed like it had been renovated, though. Perhaps they tore out its original hardened features?

---

Historic aerial imagery tells a story. The facility was first built sometime in the 1980s, and in the early '90s featured two large, likely steerable antennas. They were in the open, not enclosed by radomes, an observation that points away from a military application. It is a fairly simple matter to estimate the altitude and azimuth of a satellite antenna from aerial photographs, so antennas used for military and intelligence purposes are almost always kept under inflatable cover.

In the mid-'90s, around when AlphaStar moved in, small antennas proliferated on the site, peaking at probably a dozen. By the turn of the millenium the antennas receded, dwindling in number as the largest were demolished.

AlphaStar's remains were purchased out of bankruptcy by Egyptian telecom entrepreneur Mahmoud Wahba, who operated them as Champion Telecom Platform. Champion was a general-purpose satellite communications company, but took advantage of the network control center and television equipment at the Oxford facility to focus on television distribution. Making the record a bit confusing, Champion advertised many of its services under the AlphaStar name. They seem to have been reasonably successful, but never attracted much press.

Still, there were interesting aspects to the business. They offered a service where Champion used their small network of earth stations to receive international channels, streaming them over IP to cable television operators who could beef up their lineup without the cost of added headend receivers. At one point, it seems, they even provided infrastructure for a nascent direct-to-consumer IPTV service. They offered the Oxford network control center as an amenity to their earth station customers, and had relationships with a few national television networks, likely as a backup site.

Champion had a better run than AlphaStar but still faded away. Their "remote cable headend" service was innovative in the worst way; in the 2000s the model was widely adopted by the increasingly monopolized cable industry. "Virtual headends" became the norm, with each cable network operating central receivers and network control in-house. IPTV was quite simply a commercial failure, but perhaps we can give them the credit of saying that they were ahead of their time. Earth stations became more available and affordable, and the fees Champion could extract from television networks must have gotten thinner.

Champion Telecom shut down sometime in the '00s. Through their holding company, JJT&M Inc., Champion and Wahba held onto the building and leased it to a tenant, SteelVault Data Centers. For several years, SteelVault operated the building as a colocation center. In their marketing materials, they said "The data center building was originally built for [the] CIA in the early 1980's" [1].

Oh? Now the CIA is involved.

At one point, I felt the trail had gone cold on the history of the Oxford earth station. It clearly predated AlphaStar, and it seemed likely that it was built sometime in the early '80s as several sources claimed. But by whom, and for what? Newspaper archives turned up very little. Ironically, any search with the word "satellite" in the 1980s turns up an unlimited number of articles on the Strategic Defense Initiative, but none have any relation to Oxford.

I put down the case for a month or more. I must have looked into property records, but to be honest, I think I was thrown off the case by Connecticut's curious convention of putting tax assessors and clerks in city government rather than the county. Oxford is in New Haven County, but the New Haven assessor works for the city by that name. Of course they have nothing on parcels in Oxford.

It pays to return with fresh eyes, and today I found what should have been obvious: the Oxford assessor has record of the parcel. The Oxford clerk, in a feat rare in my part of the country, has digitized their books. I didn't even have to brave a phone call, just a frustrating web application. It was a simple trail to follow from the current deed to the survey that first described the parcel---in 1982.

---

In the era of SteelVault, 66 Hawley takes a strange turn. Like most "secure data centers," the sector of the market that often make claim to having renovated a government bunker, SteelVault did not flourish. In 2013, SteelVault was bankrupt and left the building. Of course, that doesn't stop numerous data center directories from repeating their CIA claims today.

JJT&M, too, was bankrupt, and the building at least seemed to be tied up in the matter. There was a lien, then a foreclosure, then a tax auction; unpaid property taxes of over one million dollars.

Then, there was a twist: the Oxford tax collector went to prison. She had been pocketing property tax payments. JJT&M sued the Town of Oxford, alleging the unpaid taxes had, in fact, been paid to begin with. They also sued the town marshal, who conducted the auction, alleging that he failed to tell the bidders that JJT&M might still hold title.

None of these attempts were successful: there were various technical problems with JJT&M's claims, but the larger finding was that JJT&M had been given ample notice of the unpaid taxes, the foreclosure, and the tax auction, but had failed to object until after the whole thing was done. Wahba had a number of business ventures in the television industry and elsewhere, and he must have been an absentee owner. A good reminder for us all to check the mail every once in a while.

The auction purchaser transferred the building to a holding LLC, probably as an investment, and then a few years later sold it to the Roller Bearing Company of America. They tore it down and built a new warehouse, and that's the end of the story.

But what about the beginning?

Several of the deeds on the property, which is variously listed with an address on Hawley or on the adjacent Willenbrock Road, include the same metes-and-bounds description. It ends: "Being the premises shown and described on a certain map entitled 'Survey & Topographical Map Prepared for G.T.E. Satellite Corp, Oxford.'"

In 1981, the Southern Pacific Railroad, owner of Sprint, launched a satellite communications business under the name Southern Pacific Communications Corporation (SPCC). In 1983, GTE acquired both Sprint and SPCC, rebranding SPCC as GTE Satellite and then shortly after as GTE Spacenet. In 1994, GTE sold Spacenet to GE, where it became GE Capital Spacenet Services, who sold the Oxford earth station to AlphaStar in 1995.

Before AlphaStar, it was a commercial earth station for satellite data network Spacenet, who had built the property to begin with. So what about the SDI? The CIA? AlphaStar had, I think, stretched the truth.

Spacenet was a major satellite data operator in the '90s. They had many commercial customers, but also government customers, and so it is not inconceivable that they held defense contracts. GTE Government Systems had definitely been involved in the SDI, contributing to computer systems and radar technology. But GTE was a huge company with many divisions, and the jump from its Government Services arm to Spacenet being built for the SDI is not one that I can find any backing for. Besides, it doesn't make much sense: SDI was, itself, a satellite program. Why would they use a commercial teleport built for civilian communications satellites?

And what of the CIA? As soon as those three letters are invoked, any claim takes on the odor of urban legend. The CIA has been accused of a great many things, and certainly has done some of them, but I can find nothing to substantiate any connection to Oxford.

It seems more likely that the Oxford earth station fits into the history of satellite communications in the obvious way. GTE Satellite was rapidly growing. From its beginning as SPCC, it had ordered the construction of two satellites that would launch in 1984. In 1982, they were making preparations, purchasing property in Oxford CT and completing a survey and zoning approvals. Over the following year the Oxford Earth Station was constructed, and when Spacenet 1 reached orbit in May 1984 it was ready for service. Oxford was just one of a half dozen earth stations built from 1982-1984 by GTE.

But there's a little more: the Oxford earth station has always had an affinity for television. Paul Allen's Skypix, a spectacularly failed satellite pay-per-view movie service, used GTE's Oxford earth station to uplink its 80 channels of video feeds in the early '90s. Perhaps this was the origin of the site's television equipment, or perhaps there had been a TV venture with GTE even earlier.

What we know for sure is that the Oxford earth station didn't make the cut when GE acquired Spacenet. They sold the earth station shortly after the acquisition. A few years later, in the words of a bankrupt company looking to sell its assets, GTE became the SDI. In the eyes of a failing data center, it became the CIA. And now those claims are rattling around in Wikipedia.

[1] The original just says "built for CIA," which has charming echoes of Arrested Development's "going to Army."

So let's say you're working on a household project and need around a dozen telephone cables---the ordinary kind that you would use between your telephone and the wall. It is, of course, more cost effective to buy bulk cable, or simply a long cable, and cut it to length and attach jacks yourself. This is even mercifully easy for telephone cable, as the wires come out of the flat cable jacket in the same order they go into the modular connector. No fiddly straightening and rearranging, you can just cut off the jacket and shove it into the jack.

But, wait, what's up with that whole thing anyway? and are telephone cables really as simple as stripping the jacket and shoving them in?

There's a lot of weirdness about modular cables. I use modular cable to refer to a cable assembly that is terminated in modular connectors, a standard type of multipin connector developed by the Bell System in the 1960s and now widely used for telephones, Ethernet, and occasionally other applications. These types of connectors are often referred to as RJ connectors, although that's a bit problematic for the pedantic. The modular connector itself is more properly designated in terms of its positions and contacts. Telephone connections predominantly use a 6P4C modular connector: the connector has six positions, but only four are populated with actual contacts. Ethernet uses an 8P8C modular connector, a bit larger with eight positions, all of which are used. The handset of a telephone typically connects to the base with a 4P4C connector: smaller than the 6P4C, but still with four contacts.

Why? And what do the RJ designations actually have to do with it?

Well, historically, telephones would be hardwired to the wall by the telephone installer. This proved inconvenient, and so the connection between the telephone and wall started to be connectorized. Telephones of the early 20th century were unlike the ones we use today, though, and were not fully self contained. A "desk set," the part of the telephone that sat on your desk, would be connected to an electrical box, usually mounted on the wall. The box was often called the ringer box, because it contained the ringer, but in many cases it also contained the hybrid transformer that achieved the telephone's key feat of magic: the combination of bidirectional signals onto one wire pair.

The hybrid transformer performed the conversion between a two-wire (one pair) signal and a four-wire (two-pair) signal with 'talk' and 'listen' on separate circuits. Since the hybrid was in the box on the wall, the telephone needed to be connected to the box by four wires. Thus the first standard telephone connector, a chunky block with protruding pins, had four contacts. These connectors were in use even after the end of separate ringer boxes, making two of the four wires vestigial. They were still in use into the 1960s, and so you might still find them in older houses.

As you will gather from the fact that the hybrid may have been in the phone or in a box on the wall, and thus the telephone connection to the wall may require four or two wires, the interface between telephone and wall was poorly standardized. This wasn't much of a problem in practice: at the time, you did not own a telephone, you rented it. When you rented a phone, an installer would be sent to your house, and if any wiring was already present they would check it and adjust the connections as required. Depending on the specific type of service you had, the type of phone you had, and when it was all installed, there were a number of ways things might actually be connected.

By the 1950s, as the Model 500 telephone became the norm, a separate hybrid became very unusual: the Model 500 had a hybrid built into its base and only needed the two wires, which could be connected directly to the exchange without an intermediary box. So what of the other two wires? Just about anyone will tell you that the other two wires are present to allow for a second telephone line. This isn't wrong in the modern context, but it is ahistorical to the origin of the wiring convention. The four wires originated with the use of an external hybrid, and when they became vestigial, other uses were sometimes found for them.

For example, the "Princess" phone, a rather slick phone introduced as more of a consumer-friendly product in 1959, had a cool new feature: a lighted dial. The Princess phone was advertised specifically for home use, and particularly as a bedside telephone, so the lighted dial was a convenient feature if you wanted to make a telephone call at night. I realize that might sound a bit strange to the modern reader, but a lot of people used to put a phone extension on their nightstand. If you wanted to place a call after you had turned out the lights, wouldn't it be nice to not have to get up and turn them back on just to see the dial? Anyway, the whole concept of the Princess phone was this kind of dialing-in-bed luxury, and the glowing dial was a nice touch.

There's a problem, though: how to power the dial light? It could potentially be powered by the loop current, but the loop current is very small, likely to be split across multiple extensions, and the exchange would not appreciate the increased load of a lot of tiny dial lights. Instead, Princess phones were installed with a transformer that produced 6VAC from wall power for the dial light. That power was delivered to the phone using the two unused wires in its wall connection. This sounds rather slick in the era of DECT phones that require a separate power cable to the wall, and was one of the upsides of the complete integration of the telephone system. One of the downsides was, of course, that you were paying a monthly rental rate for all of this convenience.

In the late 1960s, the nature of telephone ownership radically changed. A series of judicial and regulatory decisions, culminating in the Carterfone decision, unleashed the telephone itself from the phone company. In the 1970s, consumers gained the ability to purchase their own phone and connect it to the telephone network without a rental fee. Increasingly, they chose to do so. Suddenly, the loose standardization of the telephone-to-wall interface became a very real problem, and one that impeded the ability of consumers to choose their own telephone.

The solution was the Registered Jack, originally a set of standardized wiring configurations developed within the Bell System and later a matter of federal regulation. Wiring installed by telephone companies was required to provide a standard Registered Jack so that consumers could easily connect their own device. It is important to understand that the Registered Jack standards are really about wiring, not connectors. They describe the way that connectors should be wired to meet specific standard applications.

The most straightforward is number 11, RJ11, which specifies a 6P2C connector with a single telephone pair. But what of the 6P4C connector we use today? Well, that's RJ14, a 6P4C with two telephone lines. The problem is that neither consumers nor the telephone cable industry have much of any appetite for understanding these distinctions, and so today the RJ standards have become misunderstood to such a degree that they are only poor synonyms for the modular jack configuration.

Cables with 6P4C connectors are routinely advertised as RJ11 or RJ14, sometimes RJ11/RJ14. Most of the time RJ11 is manifestly incorrect as they do, in fact, contain four wires and thus provide 6P4C connectors. Actual 6P2C telephone cables are uncommon, as they don't really cost any less than 6P4C (manufacturing cost by far dominating the small-gauge copper) and consumers tend to expect any telephone cable to work with a two-line phone. RJ14 here is even incorrect, as there really is no such thing as an RJ14 cable. It's in the name, Registered Jack: RJ14 describes the jack you plug the cable into, the electrical interface presented on the wall. Any 6P4C cable could be used with any RJ that specifies a 6P4C connector. Incidentally, this is only academic, as RJ14 is the only 6P4C jack. This is, of course, much of why the terminology has become confused: Most of the time it doesn't matter! If the connector fits, it will work.

This whole thing becomes famously complex with Ethernet. It is common, but entirely incorrect, to refer to the 8P8C connector used for Ethernet as RJ45. This terminology is purely the result of confusion, a real RJ45 connector is actually keyed differently (and thus incompatible with) the 8P8C non-keyed connector used for Ethernet. They just look similar, if you don't look too close. A true RJ45 connector provides one telephone line and a resistor with a value that would tell a modem what transmit power it should use. In practice this jack was rarely used and it is entirely obsolete today.

In fact, Ethernet is wired according to a standard called TIA 568, which famously has two different variants, A and B. A and B are electrically identical and differ only in the mapping of color pairs to pins. The origin of this standard, and its two variants, is arcane and basically a result of awkwardly shoehorning Ethernet into telephone wiring while trying not to interfere with the telephone lines, or the RJ45 resistor if present. The connectors are wired strangely in order to provide crossover of transmit and receive while using the pins not used by the RJ45 standard: ironically, Ethernet is very intentionally incompatible with RJ45. It's sort of the inverse, plus a twist to swap RX and TX.

So you have to know why? Well, on any modular wiring, the center pins (4 and 5 for an 8P connector) are almost guaranteed to carry a telephone line. That's what modular wiring was for! Additionally, the RJ45 standard that closely resembles Ethernet uses pins 7 and 8 for the resistor. For these reasons, Ethernet originally avoided those pins, using only pins 1, 2, 3, and 6. Pins 3 and 6 would likely already be a pair, as they are the conventional position for either a second telephone line or a key system control circuit. That maintains, of course, the symmetry that is standard for telephone wiring. But that leaves pins 1 and 2 to be used for the other pair. And this is where we get the weird, inconsistent wiring pattern: 1 and 2, and 3 and 6, respectively were used for pairs by 10/100. When Gigabit ethernet came around and used four pairs, 4 and 5 were obvious since they were already going to be a telephone pair, and 7 and 8 were left. Ethernet connectors grew like tree rings: the middle is symmetric according to telephone convention, the outside is weird, according to Ethernet convention.

And as for why there are two different color conventions... well, the "A" variant was identical to the telephone industry convention for the two center pairs, which was very convenient for any installation that reused or coexisted with telephone wiring. The "B" pattern was actually included only for backwards compatibility with a pre-Ethernet, pre-TIA 568 structured wiring system called SYSTIMAX. SYSTIMAX was widely installed for a variety of applications in early business networking, carrying everything from analog voice to token ring, but particularly emphasized serial terminal connections. Since both telephone wiring and SYSTIMAX wiring were widely installed, using different color conventions for mapping pairs to 8P8C connectors, TIA-568 decided to encompass both.

It is ironic, of course, that SYSTIMAX was originally an AT&T product, and so AT&T created the whole confusion themselves. Today, it is the legalistic view that TIA-568A is "correct" as the standard says it is preferred. TIA-568B, despite being included in the standard for backwards compatibility, is nonetheless extremely common. People will tell you various rules of thumb, like "government uses A and business uses B," or "horizontal wiring uses A and patch cables use B," but really, you just have to check.

But that's not what I meant to talk about here, and I don't think I even explained it very well. Ethernet is weird, that's the point. It's the odd one out, because it was shoehorned into a wiring convention originally designed for another purpose, and in many cases it had to coexist with that other purpose. It's some real legacy stuff. And also Ethernet was originally used with coaxial cables, yes I know, that's why it only needed one pair to begin with, but then we wanted full duplex.

So that's the great thing about phone cables: they're actually using the cable and modular connector the way they were intended to be used, so they fit right into each other. So quick and easy, and there's nothing to think about.

Except...

With Ethernet, there used to be this confusion about whether or not RX and TX were swapped by the cable. Today, because of something originally called auto-MDIX and replaced by the media-independent interface part of GbE, we rarely have to worry about this. But with older 10/100 equipment, there was a wiring convention for one end, and a wiring convention for the other, but if you tried to connect two things that were wired to be the same end, you had to swap RX and TX in the cable. This was called a crossover cable, and is directly analogous to the confusingly named "null modem" serial cable.

Telephone cables are... well, if you go shopping for RJ11 or RJ14 telephone cables, you might run into something odd. Some sellers, typically the more knowledgeable ones, may identify their cables as "straight" or "reverse." Even more confusingly, you will often read that "straight" is for data applications (like fax machines!) while "reverse" is for voice applications. If you consider that the majority of fax machines provide a telephone handset and are, in fact, capable of voice, this is particularly confusing.

See, the thing is, a reverse cable has the two ends swapped relative to each other. It's not like Ethernet, the RX and TX pairs aren't swapped, because there are no such pairs. Remember, the two pairs of a 6P4C telephone cable are used as two separate circuits. Instead, the polarity is swapped within each pair.

Telephone cables are wired in such a way that this is easy: in a 6P4C connector, the "first" pair is the middle two pins (3 and 4), while the "second" pair is the next two pins out (2 and 5). That makes them symmetric, so you can swap the polarity of all of the pairs by simply putting one of the modular jacks on the other way around. With Ethernet, not coincidentally, the "inner" two pairs still work this way. It's the outer ones that buck convention.

When the jacks are connected such that the pins are consistent---that is, pin 1 on one connector is connected to pin 1 on the other, we could call that a straight cable. If the ends are mirrored, that is, pin 1 on one end is connected to pin 6 on the other, we could call it a reverse cable.

With a telephone, we already talked about the hybrid situation: the two directions are not separated on the telephone line. We don't need to swap out RX and TX. So... why? why are there straight and reverse cables? Why do they have different applications?

Telephone lines have a distinct polarity, because of the DC battery voltage. For historic reasons, the two "sides" of a telephone pair are referred to as "tip" and "ring," referring to where they would land on the 1/4" connector that we no longer call a "phone" connector and instead associate mostly with electric guitars and expensive headphones. The ring is the negative side of the battery power, and the tip is the positive side. As standard, these are identified as -48v and 0v, because the exchange equipment is grounded on the positive side. Both sides should be regarded as floating at the subscriber end, though, so the voltages and positive or negative aren't that important. It's just tip and ring.

There is a correct way to connect a phone, but older phones with entirely analog wiring wouldn't notice the difference. When touch-tone phones introduced active digital electronics, polarity suddenly mattered, but you can imagine how this went over with consumers: some people had telephone jacks wired the wrong way around, and had for years, without any problems. When they upgraded to a touch-tone phone and it didn't work, the phone was clearly at fault, not the wiring. So, quite a few touch-tone phones were made with circuitry to "fix" a reverse-wired telephone connection. Besides, just to keep things complex, there were some types of pre-touch-tone phones that required tip and ring be correctly preserved for biasing the magnetic ringer.

But wait... why, then, would so many sources assert that reverse-wired cables are appropriate for voice use? Well, there is a major problem of internet advice here. Look carefully at the websites that are the top results for the question of straight vs. reverse telephone cables, and you will find that they don't actually agree on what those terms mean. There are, in fact, two ways to look at it: you could say that a straight cable is a cable with the same correspondence of color to pin, or you could say that a straight cable has the two modular connectors installed the same way up.

If you think about it, you will realize that these conflict: if you attach both modular connectors with the latch on the same side of the cable, they will have mirrored pinouts and thus opposite polarity. To have a 1:1 pin correspondence that preserves polarity, you must attach the connectors such that one has the latch up and the other has the latch down. Now, this only makes sense if you lay your cable out perfectly flat, and for a round cable (like the twisted pair cables used for ethernet) you still wouldn't be able to tell. But telephone cables are flat, and what's more, the manufacturing process leaves a distinct ridge on one side that makes it obvious which way the connector is oriented. Latch on the ridge side, or latch on the smooth side?

There's another way to look at it: put two 6P4C connectors face-to-face, like you are trying to plug the two into each other. You will notice that, if the wiring is pin-to-pin, they don't match each other. Pin 2 on one connector is a different color from the adjacent pin 5 on the other connector. This isn't all that surprising, because we're basically doing the same thing: we're focusing on the physical orientation of the connectors instead of the electrical connection.

Whether "straight" refers to the wiring or the connector orientation varies from author to author. I will confidently assert that the correct definition of "straight" is a cable where a given pin on one end corresponds to the same pin on the other, but there are certainly some that will disagree with me!

Here's the thing: as far as I can tell, the entire issue of straight vs. reverse telephone cables comes from this exact confusion. Oddly enough, non-pin-consistent wiring (e.g. with pin 2 on one connector going to pin 4 on the other) seems to have been the historical convention. Many manufactured telephone cables are made this way, even today. I am not sure, but I will speculate it might be an artifact of the manufacturing technique, or at least the desire of those manufacturing telephone cables to have an easy, consistent way to put the connector on. Non pin-consistent cables are often articulated as placing the connector latch on the ridge side of the cable at both ends. Which makes sense, in a way!

The thing is, these cables, standard though they apparently are, will reverse the polarity of the telephone line. If you connect two with a mating connector, the second one might reverse it back to the way it was before... but it might not! mating connectors are made in both straight and reverse variants, although in this case straight seems much more common.

And I believe this is the whole origin of the "data" vs "voice" advice: telephones, the voice application, rarely care about line polarity. Data applications, because of the diversity of the equipment in use, are more likely to care about polarity. Indeed, for true digital applications like T-carrier, the cable must be straight. The whole thing is perhaps more succinctly described as "straight vs. don't care" rather than "straight vs. reverse," because as far as I can tell, there is no true application for what I am calling a reverse cable (one that does not preserve pin consistency). They're just common because of the applications in which polarity need not be maintained.

But I would love to hear if anyone knows otherwise! Truthfully I am very frustrated by this whole thing. The inconsistency of naming conventions, confusion over applications and the history, and argumentative forum threads about this have all deeply unsettled my belief in the consistency of telecommunications wiring.

Also, if you're making telephone cables, just make them straight (pin-consistent). It seems to be the safer way. I've never had it not work!

In the 1450s, German inventor Johannes Gutenburg designed the movable-type printing press, the first practical method of mass-duplicating text. After various other projects, he applied his press to the production of the Bible, yielding over one hundred copies of a text that previously had to be laboriously hand-copied.

His Bible was a tremendous cultural success, triggering revolutions not only in printed matter but also in religion. It was not a financial success: Gutenburg had apparently misspent the funds loaned to him for the project. Gutenburg lost a lawsuit and, as a result of the judgment, lost his workshop. He had made printing vastly cheaper, but it remained costly in volume. Sustaining the revolution of the printing press evidently required careful accounting.

For as long as there have been documents, there has been a need to copy. The printing press revolutionized printed matter, but setting up plates was a labor-intensive process, and a large number of copies needed to be produced at once for the process to be feasible. Into the early 20th century, it was not unusual for smaller-quantity business documents to be hand-copied. It wasn't necessarily for lack of duplicating technology; if anything, there were a surprising number of competing methods of duplication. But all of them had considerable downsides, not least among them the cost of treated paper stock and photographic chemicals.

The mimeograph was the star of the era. Mimeograph printing involved preparing a wax master, which would eventually be done by typewriter but was still a frustrating process when you only possessed a printed original. Photographic methods could be used to reproduce anything you could look at, but required expensive equipment and a relatively high skill level. The millennial office's proliferation of paper would not fully develop until the invention of xerography.

Xerography is not a common term today, first because of the general retreat of the Xerox corporation from the market, and second because it specifically identifies an analog process not used by modern photocopiers. In the 1960s, Xerox brought about a revolution in paperwork, though, mass-producing a reprographic machine that was faster, easier, and considerably less expensive to operate than contemporaries like the Photostat. The photocopier was now simple and inexpensive enough that they ventured beyond the print shop, taking root in the hallways and supply rooms of offices around the nation.

They were cheap, but they were costly in volume. Cost per page for the photocopiers of the '60s and '70s could reach $0.05, approaching $0.40 in today's currency. The price of photocopies continued to come down, but the ease of photocopiers encouraged quantity. Office workers ran amok, running off 30, 60, even 100 pages of documents to pass around. The operation of photocopiers became a significant item in the budget of American corporations.

The continued proliferation of the photocopier called for careful accounting.

---

Wilhelm Haller was born in Swabia, in Germany. Details of his life, in the English language and seemingly in German as well, are sparse. His Wikipedia biography has the tone of a hagiography; a banner tells us that its neutrality is disputed.

What I can say for sure is that, in the 1960s, Haller found the start of his career as a sales apprentice for Hengstler. Hengstler, by then nearly a hundred years old, had made watches and other fine machinery before settling into the world of industrial clockwork. Among their products were a refined line of mechanical counters, of the same type we use today: hour meters, pulse counters, and volume meters, all driving a set of small wheels printed with the digits 0 through 9. As each wheel rolled from 9 to 0, a peg pushed a lever to advance the next wheel by one digit. They had numerous applications in commercial equipment and Haller must have become quite familiar with them before he moved to New York City, representing Hengstler products to the American market.

Perhaps he worked in an office where photocopier expenses were a complaint. I wish there was more of a story behind his first great invention, but it is quite overshadowed by his later, more abstract work. No source I can find cares to go deeper than to say that, along with Hengstler employee Paul Buser, he founded an American subsidiary of Hengstler called the Hecon Corporation. I can speculate somewhat confidently that Hecon was short for "Hengstler Counter," as Hecon dealt entirely in counters. More specifically, Hecon introduced a new application of the mechanical counter invented by Haller himself: the photocopier key counter.

Xerox photocopiers already included wiring that distributed a "pulse per page" signal, used to advance a counter used for scheduled maintenance. The Hecon key counter was a simple elaboration on this idea: a socket and wiring harness, furnished by Hecon, was installed on the photocopier. An "enable" circuit for the photocopier passed through the socket, and had to be jumpered for the photocopier to function. The socket also provided a pulse per page wire.

Photocopier users, typically each department, were issued a Hecon mechanical counter that fit into the socket. To make photocopies, you had to insert your key counter into the socket to enable the photocopier. The key counter was not resettable, so the accounting department could periodically collect key counters and read the number displayed on them like a utility meter. Thus the name key counter: it was a key to enable the photocopier, and a counter to measure the keyholder's usage.

Key counters were a massive success and proliferated on office photocopiers during the '70s. Xerox, and then their competitors, bought into the system by providing a convenient mounting point and wiring harness connector for the key counter socket. You could find photocopiers that required a Hecon key counter well into the 1990s. Threads on office machine technician forums about adapting the wiring to modern machines suggest that there were some users into the 2010s.

---

Hecon would not allow the technology to stagnate. The mechanical key counter was reliable but had to be collected or turned in for the counter to be read. The Hecon KCC, introduced by the mid-1990s, replaced key counters with a microcontroller. Users entered an individual PIN or department number on a keypad mounted to the copier and connected to the key counter socket. The KCC enabled the copier and counted the page pulses, totalizing them into a department account that could be read out later from the keypad or from a computer by serial connection.

Hecon was not only invested in technological change, though. At some point, Hecon became a major component of Hengstler, with more Hengstler management moving to its New Jersey headquarters. "Must have good command of German and English," a 1969 newspaper listing for a secretarial job stated, before advising applicants to call a Mr. Hengstler himself.

By 1976, the "Liberal Benefits" in their job listing had been supplemented by a new feature: "Hecon Corp, the company that pioneered & operates on flexible working hours."

During the late '60s, Wilhelm Haller seems to have returned to Germany and shifted his interests beyond photocopiers to the operations of corporations themselves. Working with German management consultant Christel Kammerer, he designed a system for mechanical recording of employee's working hours.

This was not the invention of the time clock. The history of the time clock is obscure but they were already in use during the 19th century. Haller's system implemented a more specific model of working hours promoted by Kammerer: flexitime (more common in Germany) or flextime (more common in the US).

Flextime is a simple enough concept and gained considerable popularity in the US during the 1970s and 1980s, making it almost too obvious to "invent" today. A flextime schedule defines "core hours," such as 11a-3p, during which employees are required to be present in the office. Outside of core hours, employees are free to come and go so long as their working hours total eight each day. Haller's time clock invention was, like the key counter, a totalizing counter: one that recorded not when employees arrived and left, but how many hours they were present each day.

It's unclear if Haller still worked for Hengstler, but he must have had some influence there. Hecon was among the first, perhaps the first, companies to introduce flextime in the United States.

---

Photocopier accounting continued apace. Dallas Semiconductor and Sun Microsystems popularized the iButton during the late 1990s, a compact and robust device that could store data and perform cryptographic operations. Hecon followed in the footprints of the broader stored value industry, introducing the Hecon Quick Key system that used iButtons for user authentication at the photocopier. Copies could even be "prepaid" onto an iButton, ideal for photocopiers with a regular cast of outside users, like those in courthouses and county clerk's offices.

The Quick Key had a distinctive, angular copier controller apparently called the Base 10. It had the aesthetic vibes of a '90s contemporary art museum, all white and geometric, although surviving examples have yellowed to to the pallor of dated office equipment.

As the Xerographic process was under development, British Bible scholar Hugh Schonfield spent the 1950s developing his Commonwealth of World Citizens. Part micronation, part NGO, the Commonwealth had a mission of organizing its members throughout many nations into a world community that would uphold the ideals of equality and peace while carrying out humanitarian programs.

Adopting Esperanto as its language, it renamed itself to the Mondcivitan Republic, publishing a provisional constitution and electing a parliament. The Mondcivitan Republic issued passports; some of its members tried to abandon citizenship of their own countries. It was one of several organizations promoting "world citizenship" in the mid-century.

In 1972, Schonfield published a book, Politics of God, describing the organization's ideals. Those politics were apparently challenging. While the Mondcivitan Republic operated various humanitarian and charitable programs through the '60s and '70s, it failed to adopt a permanent constitution and by the 1980s had effectively dissolved. Sometime around then, Wilhelm Haller joined the movement and established a new manifestation of the Mondcivitan Republic in Germany. Haller applied to cancel his German citizenship, he would be a citizen of the world.

As a management consultant and social organizer, he founded a series of progressive German organizations. Haller's projects reached their apex in 2004, with the formation of the "International Leadership and Business Society," a direct extension of the Mondcivitan project. That same year, Haller passed away, a victim of thyroid cancer.

---

A German progressive organization, Lebenshaus Schwäbische Alb eV, published a touching obituary of Haller. Hengstler and Hecon are mentioned only as "a Swabian factory," his work on flextime earns a short paragraph.

In translation:

He was able to celebrate his 69th birthday sitting in a wheelchair with a large group of his family and the circle of friends from the Reconciliation Association and the Life Center. With a weak and barely audible voice, he took part in our discussion about new financing options for the local independent Waldorf school from the purchasing power of the affected parents' homes.

Haller is, to me, a rather curious type of person. He was first an inventor of accounting systems, second a management consultant, and then a social activist motivated by both his Christian religion and belief in precision management. His work with Hengstler/Hecon gave way to support and adoption programs for disadvantaged children, supportive employment programs, and international initiatives born of unique mid-century optimism.

Flextime, he argued, freed workers to live their lives on their own schedules, while his timekeeping systems maintained an eight-hour workday with German precision. The Hecon key counter, a footnote of his career, perhaps did the same on a smaller scale: duplication was freed from the print shop but protected by complete cost recovery. Later in his career, he would set out to unify the world.

But then, it's hard to know what to make of Haller. Almost everything written about him seems to be the work of a true believer in his religious-managerial vision. I came for a small detail of photocopier history, and left with this strange leader of West German industrial thought, a management consultant who promised to "humanize" the workplace through time recording.

For him, a new building in the great "city on a hill" required only two things: careful commercial accounting with the knowledge of our own limited possibilities, and a deep trust in God, who knows how to continue when our own strength has come to an end.

Across the United States, streets are taking on a strange hue at night. Purple.

Purple streetlights have been reported in Tampa, Vancouver, Wichita, Boston. They're certainly in evidence here in Albuquerque, where Coal through downtown has turned almost entirely to mood lighting. Explanations vary. When I first saw the phenomenon, I thought of fixtures that combined RGB elements and thought perhaps one of the color channels had failed.

Others on the internet offer more involved explanations. "A black light surveillance network," one conspiracist calls them, as he shows his mushroom-themed blacklight poster fluorescing on the side of a highway. I remain unclear on what exactly a shadowy cabal would gain from installing blacklights across North America, but I am nonetheless charmed by his fluorescent fingerpainting demonstration. The topic of "blacklight" is a somewhat complex one with LEDs.

Historically, "blacklight" had referred to long-wave UV lamps, also called UV-A. These lamps emitted light around 400nm, beyond violet light, thus the term ultraviolet. This light is close to, but not quite in, the visible spectrum, which is ideal for observing the effect of fluorescence. Fluorescence is a fascinating but also mundane physical phenomenon in which many materials will absorb light, becoming excited, and then re-emit it as they relax. The process is not completely efficient, so the re-emited light is longer in wavelength than the absorbed light.

Because of this loss of energy, a fluorescent material excited by a blacklight will emit light down in the visible spectrum. The effect seems a bit like magic: the fluorescence is far brighter, to the human eye, than the ultraviolet light that incited it. The trouble is that the common use of UV light to show fluorescence leads to a bit of a misconception that ultraviolet light is required. Not at all, fluorescent materials will emit just about any light at a slightly lower wavelength. The emitted light is relatively weak, though, and under broad spectrum lighting is unlikely to stand out against the ambient lighting. Fluorescence always occurs, it's just much more visible under a light source that humans can't see.

When we consider LEDs, though, there is an economic aspect to consider. The construction of LEDs that emit UV light turns out to be quite difficult. There are now options on the market, but only relatively recently, and they run a considerable price premium compared to visible wavelength LEDs. The vast majority of "LED blacklights" are not actually blacklights; they don't actually emit UV. They're just blue. Human eyes aren't so sensitive to blue, especially the narrow emission of blue LEDs, and so these blue "blacklights" work well enough for showing fluorescence, although not as well as a "real" blacklight (still typically gas discharge).

This was mostly a minor detail of theatrical lighting until COVID, when some combination of unknowing buyers and unscrupulous sellers lead to a wave of people using blue LEDs in an attempt to sanitize things. That doesn't work, long-wave UV already barely has enough energy to have much of a sanitizing effect and blue LEDs have none at all. For sanitizing purposes you need short wave UV, or UV-C, which has so much energy that it is almost ionizing radiation. The trouble, of course, is that this energy damages most biological things, including us. UV-C lights can quickly cause mild (but very unpleasant) eye damage called flashburn or "welder's eye," and more serious exposure can cause permanent damage to your eyes and skin. Funny, then, that all the people waving blue LEDs over their groceries on Instagram reels were at least saving themselves from an unpleasant learning experience.

You can probably see how this all ties back to streetlights. The purple streetlights are not "blacklights," but the clear fluorescence of our friend's psychedelic art tells us that they are emitting energy mostly at the short end of the visible spectrum, allowing the longer wave light emitted by the poster to appear inexplicably bright to our eyes. We are apparently looking at some sort of blue LED.

Those familiar with modern LED lighting probably easily see what's happening. LEDs are largely monochromatic lighting sources, they emit a single wavelength that results in very poor color rendering, which is both aesthetically unpleasing and produces poor perception for drivers. While some fixtures do indeed combine LEDs of multiple colors to produce white output, there's another technique that is less expensive, more energy efficient, and produces better quality light. Today's inexpensive, good quality LED lights have been enabled by phosphor coatings.

Here's the idea: LEDs of a single color illuminate a phosphorous material. Phosphorescence is actually a closely related phenomenon to fluorescence, but involves kicking an electron up to a different spin state. Fewer materials exhibit this effect than fluorescence, but chemists have devised synthetic phosphors that can sort of "rearrange" light energy within the spectrum.

Blue LEDs are the most energy efficient, so a typical white LED light uses blue LEDs coated in a phosphor that absorbs a portion of the blue light and re-emits it at longer wavelengths. The resulting spectrum, the combination of some of the blue light passing through and red and green light emitted by the phosphor, is a high-CRI white light ideal for street lighting.

Incidentally, one of the properties of phosphorescence that differentiates it from fluorescence is that phosphors take a while to "relax" back to their lower energy state. A phosphor will continue to glow after the energy that excited it is gone. This effect has long been employed for "glow in the dark" materials that continue to glow softly for an extended period of time after the room goes dark. During the Cold War, the Civil Defense Administration recommended outlining stair treads and doors with such phosphorescent tape so that you could more safely navigate your home during a blackout. The same idea is still employed aboard aircraft and ships, and I suppose you could still do it to your house, it would be fun.

Phosphor-conversion white LEDs use phosphors that minimize this effect but they still exhibit it. Turn off a white LED light in a dark room and you will probably notice that it continues to glow dimly for a short time. You are observing the phosphor slowly relaxing.

So what of the purple streetlights? The phosphor has failed, at least partially, and the lights are emitting the natural spectrum of their LEDs rather than the "adjusted" spectrum produced by the phosphor. The exact reason for this failure doesn't seem to have been publicized, but judging by the apparently rapid onset most people think the phosphor is delaminating and falling off of the LEDs rather than slowly burning away or undergoing some sort of corrosion. They may have simply not used a very good glue.

So we have a technical explanation: white LED streetlights are not white LEDs but blue LEDs with phosphor conversion. If the phosphor somehow fails or comes off, their spectrum shifts towards deep blue. Some combination of remaining phosphor on the lights and environmental conditions (we are not used to seeing large areas under monochromatic blue light) causes this to come off as an eery purple.

There is also, though, a system question. How is it that so many streetlights across so many cities are demonstrating the same failure at around the same time?

The answer to that question is monopolization.

Virtually all LED street lighting installed in North America is manufactured by Acuity Brands. Based in Atlanta, Acuity is a hundred-year-old industrial conglomerate that originally focused on linens and janitorial supplies. In 1969, though, Acuity acquired Lithonia: one of the United States' largest manufacturers of area lighting. Acuity gained a lighting division, and it was on the war path. Through a huge number of acquisitions, everything from age-old area lighting giants like Holophane to VC-funded networked lighting companies have become part of Acuity.

In the mean time, GE's area lighting division petered out along with the rest of GE (they recently sold their entire lighting division to a consumer home automation company). Directories of street lighting manufacturers now list Acuity followed by a list of brands Acuity owns. Their dominant competitor for traditional street lighting are probably Cree and Cooper (part of Eaton), but both are well behind Acuity in municipal sales.

Starting around 2017, Acuity started to manufacture defective lights. The exact nature of the defect is unclear, but it seems to cause abrupt failure of the phosphor after around five years. And here we are, over five years later, with purple streets.

The situation is not quite as bad as it sounds. Acuity offered a long warranty on their street lighting, and the affected lights are still covered. Acuity is sending contractors to replace defective lights at their expense, but they have to coordinate with street lighting operators to identify defective lights and schedule the work. It's a long process. Many cities have over a thousand lights to replace, but finding them is a problem on its own.

Most cities have invested in some sort of smart streetlighting solution. The most common approach is a module that plugs into the standard photocell receptacle on the light and both controls the light and reports energy use over a municipal LTE network. These modules can automatically identify many failure modes based on changes on power consumption. The problem is that the phosphor failure is completely nonelectrical, so the faulty lights can't be located by energy monitoring.

So, while I can't truly rule out the possibility of a blacklight surveillance network, I'd suggest you report purple lights to your city or electrical utility. They're likely already working with Acuity on a replacement campaign, but they may not know the exact scale of the problem yet.

---

While I'm at it, let's talk about another common failure mode of outdoor LED lighting: flashing. LED lights use a constant current power supply (often called a driver in this context) that regulates the voltage applied to the LEDs to achieve their rated current. Unfortunately, several failure modes can cause the driver to continuously cycle. Consider the common case of an LED module that has failed in such a way that it shorts at high temperature. The driver will turn on until the faulty module gets warm enough and the driver turns off again on current protection. The process repeats indefinitely. Some drivers have a "soft start" feature and some failure modes cause current to rise beyond limits over time, so it's not unusual for these faulty lights to fade in before shutting off.

It's actually a very similar situation to the cycling that gas discharge street lighting used to show, but as is the way of electronics, it happens faster. Aged sodium bulbs would often cause the ballast to hit its current limit over the span of perhaps five minutes, cycling the light on and off. Now it often happens twice in a second.

I once saw a parking lot where nearly every light had failed this way. I would guess that lightning had struck, creating a transient that damaged all of them at once. It felt like a silent rave, only a little color could have made it better. Unfortunately they were RAB, not Acuity, and the phosphor was holding on.

Last week, someone leaked a spreadsheet of SoundThinking sensors to Wired. You are probably asking "What is SoundThinking," because the company rebranded last year. They used to be called ShotSpotter, and their outdoor acoustic gunfire detection system still goes by the ShotSpotter name.

ShotSpotter has attracted a lot of press and plenty of criticism for the gunfire detection service they provide to many law enforcement agencies in the US. The system involves installing acoustic sensors throughout a city, which use some sort of signature matching to detect gunfire and then use time of flight to determine the likely source.

One of the principle topics of criticism is the immense secrecy with which they operate: ShotSpotter protects information on the location of its sensors as if it were state secret, and does not disclose them even to the law enforcement agencies that are its customers. This secrecy attracts accusations that ShotSpotter's claims of efficacy cannot be independently validated, and that ShotSpotter is attempting to suppress research into the civil rights impacts of its product.

I have encountered this topic before: the Albuquerque Police Department is a ShotSpotter customer, and during my involvement in police oversight was evasive in response to any questions about the system and resisted efforts to subject its surveillance technology purchases to more outside scrutiny. Many assumed that ShotSpotter coverage was concentrated in disadvantaged parts of the city, an unsurprising outcome but one that could contribute to systemic overpolicing. APD would not comment.

I have always assumed that it would not really be that difficult to find the ShotSpotter sensors, at least if you have my inclination to examine telephone poles. While the Wired article focuses heavily on sensors installed on buildings, it seems likely that in environments like Albuquerque with city-operated lighting and a single electrical utility, they would be installed on street lights. That's where you find most of the technology the city fields.

The thing is, I didn't really know what the sensors looked like. I've seen pictures, but I know they were quite old, and I assumed the design had gotten more compact over time. Indeed it has.

An interesting thing about the Wired article is that it contains a map, but the MapBox embed produced with Flourish Studio had a surprisingly high maximum zoom level. That made it more or less impossible to interpret the locations of the sensors exactly. I'm concerned that this was an intentional decision by Wired to partially obfuscate the data, because it is not an effective one. It was a simple matter to find the JSON payload the map viewer was using for the PoI overlay and then convert it to KML.

I worried that the underlying data would be obscured; it was not. The coordinates are exact. So, I took the opportunity to enjoy a nice day and went on an expedition.

The sensors are pretty much what I imagined, innocuous beige boxes clamped to street light arms. There are a number of these boxes to be found in modern cities. Some are smart meter nodes, some are base stations for municipal data networks, others collect environmental data. Some are the police, listening in on your activities.

This is not as hypothetical of a concern as it might sound. Conversations recorded by ShotSpotter sensors have twice been introduced as evidence in criminal trials. In one case the court allowed it, in another the court did not. The possibility clearly exists, and depending on interpretation of state law, it may be permissible for ShotSpotter to record conversations on the street for future use as evidence.

This ought to give us pause, as should the fact that ShotSpotter has been compellingly demonstrated to manipulate their "interpretation" of evidence to fit a prosecutor's narrative---even when ShotSpotter's original analysis contradicted it.

But pervasive surveillance of urban areas and troubling use of that evidence is nothing new. Albuquerque already has an expansive police-operated video surveillance network connected to the Real-Time Crime Center. APD has long used portable automated license plate readers (ALPR) under cover of "your speed is" trailers, and more recently has installed permanent ALPR at major intersections in the city.

All of this occurs with virtually no public oversight or even public awareness.

What most surprised me is the density of ShotSpotter sensors. In my head, I assumed they were fairly sparse. A Chicago report on the system says there are 20 to 25 per square mile. Density in Albuquerque is lower, probably reflecting the wide streets and relative lack of high rises. Still, there are a lot of them. 721 in Albuquerque, a city of about 190 square miles. At present, only parts of the city are covered.

And those coverage decisions are interesting. The valley (what of it is in city limits) is well covered, as is the west side outside of Coors/Old Coors. The International District, of course, is dense with sensors, as is inner NE bounded by roughly by the freeways to Louisiana and Montgomery.

Conspicuously empty is the rest of the northeast, from UNM's north campus area to the foothills. Indian School Road makes almost its entire east side length without any sensors.

The reader can probably infer how this coverage pattern relates to race and class in Albuquerque. It's not perfect, but the distance from your house to a ShotSpotter sensor correlates fairly well with your household income. The wealthier you are, the less surveilled you are.

The "pocket of poverty" south of Downtown where I live, the historically Spanish Barelas and historically Black South Broadway, are predictably well covered. All of the photos here were taken within a mile, and I did not come even close to visiting all of the sensors. Within a one mile radius of the center of Barelas, there are 31 sensors.

Some are conspicuous. Washington Middle School, where 13-year-old Bennie Hargrove was shot by another student, has a sensor mounted at its front entrance. Another sensor is in the cul de sac behind the Coors and I-40 Walmart, where a body was found in a burned-out car. Perhaps the deep gulch of the freeway poses a coverage challenge, there are two more less than a thousand feet away.

In the Downtown Core, buildings were preferred to light poles. The PNM building, the Anasazi condos, and the Banque building are all feeding data into the city's failing scheme of federal prosecutions for downtown gun crime.

The closest sensor to the wealthy Heights is at Embudo Canyon, and coverage stops north of Central in the affluent Nob Hill residential area. Old Town is almost completely uncovered, as is the isolationist Four Hills.

Highland High School has a sensor on its swimming pool building. The data says there are two at the intersection of Gibson and Chavez, probably an error, it also says there are two sensors on "Null Island." Don't worry about coverage in the south campus area, though. There are 16 in the area bounded by I-25 to Yale and Gibson to Coal.

KOB quotes APD PIO Gallegos saying "We don't know, technically, where all the sensors are." Well, I suppose they do now, the leak has been widely reported on. APD received about 14,000 ShotSpotter reports last year. The accuracy of these reports, in terms of their correctly identifying gunfire, is contested. SoundThinking claims impressive statistics, but has actively resisted independent evaluation. A Chicago report found that only 11.3% of ShotSpotter reports could be confirmed as gunfire. APD, for its part, reports a few hundred suspects or victims identified as a result of ShotSpotter reports.

APD has used a local firearms training business, Calibers, to fire blanks around the city to verify detection. They say the system performed well.

But, if asked, they provide a form letter written by ShotSpotter. Their contract prohibits the disclosure of any actual data.

It's one of those anachronisms that is deeply embedded in modern technology. From cloud operator servers to embedded controllers in appliances, there must be uncountable devices that think they are connected to a TTY.

I will omit the many interesting details of the Linux terminal infrastructure here, as it could easily fill its own article. But most Linux users are at least peripherally aware that the kernel tends to identify both serial devices and terminals as TTYs, assigning them filesystem names in the form of /dev/tty*. Probably a lot of those people remember that this stands for teletype or perhaps teletypewriter, although in practice the term teleprinter is more common.

Indeed, from about the 1950s (the genesis of electronic computers) to the 1970s (the rise of video display terminals/VDTs), teleprinters were the most common form of interactive human-machine interface. The "interactive" distinction here is important; early computers were built primarily around noninteractive input and output, often using punched paper tape. Interactive operation was a more advanced form of computing, one that took almost until the widespread use of VDTs to mature. Look into the computers of the 1960s especially, the early days of interactive operation, and you will be amazed at how bizarre and unfriendly the command interface is. It wasn't really intended for people to use; it was for the Computer Operator (who had attended a lengthy training course on the topic) to troubleshoot problems in the noninteractive workload.

But interactive computing is yet another topic I will one day take on. Right now, I want to talk about the heritage of these input/output mechanisms. Why is it that punched paper tape and the teleprinter were the most obvious way to interact with the first electronic computers? As you might suspect, the arrangement was one of convenience. Paper tape punches and readers were already being manufactured, as were teleprinters. They were both used for communications.

Most people who hear about the telegraph think of Morse code keys and rhythmic beeping. Indeed, Samuel Morse is an important figure in the history of telegraphy. The form of "morse code" that we tend to imagine, though, a continuous wave "beep," is mostly an artifact of radio. For telegraphs, no carrier wave or radio modulation was required. You can transmit a message simply by interrupting the current on a wire.

This idea is rather simple to conceive and even to implement, so it's no surprise that telegraphy has a long history. By the end of the 18th century inventors in Europe and Great Britain were devising simple electrical telegraphs. These early telegraphs had limited ranges and even more limited speeds, though, a result mostly of the lack of a good way to indicate to the operator whether or not a current was present. It is an intriguing aspect of technical history that the first decades of experimentation with electricity were done with only the clumsiest means of measuring or even detecting it.

In 1820, three physicists or inventors (these were vague titles at the time) almost simultaneously worked out that electrical current induced a magnetic field. They invented various ways of demonstrating the effect, usually by deflecting a magnetic needle. This innovation quickly lead to the "electromagnetic telegraph," in which a telegrapher operates a key to switch current, which causes a needle or flag to deflect at the other end of the circuit. This was tremendously simpler than previous means of indicating current and was applied almost immediately to build the first practical telegraphs. During the 1830s, the invention of the relay allowed telegraph signals to be repeated or amplified as the potential weakened (the origin of the term "relay"). Edward Davy, one of the inventors of the relay, also invented the telegraph recorder.

From 1830 to 1850, so many people invented so many telegraph systems that it is difficult to succinctly describe how an early practical telegraph worked. There were certain themes: for non-recording systems, a needle was often deflected one way or the other by the presence or absence of current, or perhaps by polarity reversal. Sometimes the receiver would strike a bell or sound a buzzer with each change. In recording systems, a telegraph printer or telegraph recorder embossed a hole or left a small mark on a paper tape that advanced through the device. In the first case, the receiving operator would watch the needle, interpreting messages as they came. In the second case, the operator could examine the paper tape at their leisure, interpreting the message based on the distances between the dots.

Recording systems tended to be used for less time-sensitive operations like passing telegrams between cities, while non-recording telegraphs were used for more real-time applications like railroad dispatch and signaling. Regardless, it is important to understand that the teleprinter is about as old as the telegraph. Many early telegraphs recorded received signals onto paper.

The interpretation of telegraph signals was as varied as the equipment that carried them. Samuel Morse popularized the telegraph in the United States based in part on his alphabetic code, but it was not the first. Gauss famously devised a binary encoding for alphabetic characters a few years earlier, which resembles modern character encodings more than Morse's scheme. In many telegraph applications, though, there was no alphabetic code at all. Railroad signal telegraphs, for example, often used application-specific schemes that encoded types of trains and routes instead of letters.

Morse's telegraph system was very successful in the United States, and in 1861 a Morse telegraph line connected the coasts. It surprises some that a transcontinental telegraph line was completed some fifty years before the transcontinental telephone line. Telegraphy is older, though, because it is simpler. There is no analog signaling involved; simple on/off or polarity signals can be amplified using simple mechanical relays. The tendency to view text as more complex than voice (SMS came after the first cellphones, for one) has more to do with the last 50 years than the 50 years before.

The Morse telegraph system was practical enough to spawn a large industry, but suffered a key limitation: the level of experience required to key and copy Morse quickly and reliably is fairly high. Telegraphers were skilled and, thus, fairly well paid and sometimes in short supply [1]. To drive down the cost of telegraphy, there would need to be more automation.

Many of the earliest telegraph designs had employed parallel signaling. A common scheme was to provide one wire for each letter, and a common return. These were impractical to build over any meaningful distance, and Morse's one-wire design (along with one-wire designs by others) won out for obvious reasons. The idea of parallel signaling stayed around, though, and was reintroduced during the 1840s with a simple form of multiplexing: one "logical channel" for each letter could be combined onto one wire using time division muxing, for example by using a transmitter and receiver with synchronized spinning wheels. Letters would be presented by positions on the wheel, and a pulse sent at the appropriate point in the revolution to cause the teleprinter to produce that letter. With this alphabetic teleprinter, an experienced operator was no longer required to receive messages. They appeared as text on a strip of paper, ready for an unskilled clerk to read or paste onto a message card.

This system proved expensive but still practical to operate, and a network of such alphabetic teleprinters was built in the United States during the mid 19th century. A set of smaller telegraph companies operating one such system, called the Hughes system after its inventor, joined together to become the Western Union Telegraph Company. In a precedent that would be followed even more closely by the telephone system, practical commercial telegraphy was intertwined with a monopoly.

The Hughes system was functional but costly. The basic idea of multiplexing across 30 channels was difficult to achieve with mechanical technology. Émile Baudot was employed by the French telegraph service to find a way to better utilize telegraph lines. He first developed a proper form of multiplexing, using synchronized switches to combine five Hughes system messages onto one wire and separate them again at the other end. Likely inspired by his close inspection of the Hughes system and its limitations, Baudot went on to develop a more efficient scheme for the transmission of alphabetic messages: the Baudot code.

Baudot's system was similar to the Hughes system in that it relied on a transmitter and receiver kept in synchronization to interpret pulses as belonging to the correct logical channel. He simplified the design, though, by allowing for only five logical channels. Instead of each pulse representing a letter, the combination of all five channels would be used to form one symbol. The Baudot code was a five-bit binary alphabetic encoding, and most computer alphabetic encodings to the present day are at least partially derived from it.

One of the downsides of Baudot's design is that it was not quite as easy to operate as telegraphy companies would hope. Baudot equipment could keep up 30 words per minute with a skilled operator who could work the five-key piano-style keyboard in good synchronization with the mechanical armature that read it out. This took a great deal of practice, though, and pressing keys out of synchronization with the transmitter could easily cause incorrect letters to be sent.

In 1901, during the early days of the telephone, Donald Murray developed an important enhancement to the Baudot system. He was likely informed by an older practice that had been developed for Morse telegraphs, of having an operator punch a Morse message into paper tape to be transmitted by a simple tape reader later. He did the same for Baudot code: he designed a device with an easy to use typewriter-like keyboard that punched Baudot code onto a strip of paper tape with five rows, one for each bit. The tape punch had no need to be synchronized with the other end, and the operator could type at whatever pace they were comfortable.

The invention of Murray's tape punch brought about the low-cost telegram networks that we are familiar with from the early 20th century. A clerk would take down a message and then punch it onto paper tape. Later, the paper tape would be inserted into a reader that transmitted the Baudot message in perfect synchronization with the receiver, a teleprinter that typed it onto tape as text once again. The process of encoding and decoding messages for the telegraph was now fully automated.

The total operation of the system, though, was not. For one, the output was paper tape, that had to be cut and pasted to compose a paragraph of text. For another, the transmitting and receiving equipment operated continuously, requiring operators to coordinate on the scheduling of sending messages (or they would tie up the line and waste a lot of paper tape).

In a wonderful time capsule of early 20th century industrialism, the next major evolution would come about with considerable help from the Morton Salt Company. Joy Morton, its founder, agreed to fund Frank Pearne's efforts to develop an even more practical printing telegraph. This device would use a typewriter mechanism to produce the output as normal text on a page, saving considerable effort by clerks. Even better, it would use a system of control codes to indicate the beginning and end of messages, allowing a teleprinter to operate largely unattended. This was more complex than it sounded, as it required finding a way for the two ends to establish clock synchronization before the message.

There were, it turned out, others working on the same concept. After a series of patent disputes, mergers, and negotiations, the Morkrum-Kleinschmidt Company would market this new technology. A fully automated teleprinter, lurching into life when the other end had a message to send, producing pages of text like a typewriter with an invisible typist.

In 1928, Morkrum-Kleinschmidt adopted a rather more memorable name: the Teletype Corporation. During the development of the Teletype system, the telephone network had grown into a nationwide enterprise and one of the United States' largest industrial ventures (at many points in time, the country's single largest employer). AT&T had already entered the telegraph business by leasing its lines for telegraph use, and work had already begun on telegraphs that could operate over switched telephone lines, transmitting text as if it were a phone call. The telephone was born of the telegraph but came to consume it. In 1930, the Teletype Corporation was purchased by AT&T and became part of Western Electric.

That same year, Western Electric introduced the Teletype Model 15. Receiving Baudot at 45 baud [2] with an optional tape punch and tape reader, the Model 15 became a workhorse of American communications. By some accounts, the Model 15 was instrumental in the prosecution of World War II. The War Department made extensive use of AT&T-furnished teletype networks and Model 15 teleprinters as the core of the military logistics enterprise. The Model 15 was still being manufactured as late as 1963, a production record rivaled by few other electrical devices.

It is difficult to summarize the history of the networks that teleprinters enabled. The concept of switching connections between teleprinters, as was done on the phone network, was an obvious one. The dominant switched teleprinter network was Telex, not really an organization but actually a set of standards promulgated by the ITU. The most prominent US implementation of Telex was an AT&T service called TWX, short for Teletypewriter Exchange Service. TWX used Teletype teleprinters on phone lines (in a special class of service), and was a very popular service for business use from the '40s to the '70s.

Incidentally, TWX was assigned the special purpose area codes 510, 610, 710, 810, and 910, which contained only teleprinters. These area codes would eventually be assigned to other uses, but for a long time ranked among the "unusual" NPAs.

Western Union continued to develop their telegraph network during the era of TWX, acting in many ways as a sibling or shadow of AT&T. Like AT&T, Western Union developed multiplexing schemes to make better use of their long-distance telegraph lines. Like AT&T, Western Union developed automatic switching systems to decrease operator expenses. Like AT&T, Western Union built out a microwave network to increase the capacity of their long-haul network. Telegraphy is one of the areas where AT&T struggled despite their vast network, and Western Union kept ahead of them, purchasing the TWX service from AT&T. Western Union would continue to operate the switched teleprinter network, under the Telex name, into the '80s when it largely died out in favor of the newly developed fax machine.

During the era of TWX, encoding schemes changed several times as AT&T and Western Union developed better and faster equipment (Western Union continued to make use of Western Electric-built Teletype machines among other equipment). ASCII came to replace Baudot, and so a number of ASCII teleprinters existed. There were also hybrids. For some time Western Union operated teleprinters on an ASCII variant that provided only upper case letters and some punctuation, with the benefit of requiring fewer bits. The encoding and decoding of this reduced ASCII set was implemented by the Bell 101 telephone modem, designed in 1958 to allow SAGE computers to communicate with one another and then widely included in TWX and Telex teleprinters. The Bell 101's descendants would bring about remote access to time-sharing computer systems and, ultimately, one of the major forms of long-distance computer networking.

You can see, then, that the history of teleprinters and the history of computers are naturally interleaved. From an early stage, computers operated primarily on streams of characters. This basic concept is still the core of many modern computer systems and, not coincidentally, also describes the operation of teleprinters.

When electronic computers were under development in the 1950s and 1960s, teleprinters were near the apex of their popularity as a medium for business communications. Most people working on computers probably had experience with teleprinters; most organizations working on computers already had a number of teleprinters installed. It was quite natural that teleprinter technology would be repurposed as a means of input and output for computers.

Some of the very earliest computers, for example those of Konrad Zuse, employed punched tape as an input medium. These were almost invariably repurposed or modified telegraphic punched tape systems, often in five-bit Baudot. Particularly in retrospect, as more materials have become available to historians, it is clear that much of the groundwork for digital computing was laid by WWII cryptological efforts.

Newly devised cryptographic machines like the Lorenz ciphers were essentially teleprinters with added digital logic. The machines built to attack these codes, like Colossus, are now generally recognized as the first programmable computers. The line between teleprinter and computer was not always clear. As more encoding and control logic was added, teleprinters came to resemble simple computers.

The Manchester Mark I, a pioneer of stored-program computing built in 1949, used a 5-bit code adopted from Baudot by none other than Alan Turing. The major advantage of this 5-bit encoding was, of course, that programs could be read and written using Baudot tape and standard telegraph equipment. The addition of a teleprinter allowed operators to "interactively" enter instructions into the computer and read the output, although the concept of a shell (or any other designed user interface) had not yet been developed. EDSAC, a contemporary of the Mark I and precursor to a powerful tea logistics system that would set off the development of business computing, also used a teleprinter for input and output.

Many early commercial computers limited input and output to paper tape, often 5-bit for Baudot or 8-bit for ASCII with parity, as in the early days of computing preparation of a program was an exacting process that would not typically be done "on the fly" at a keyboard. It was, of course, convenient that teleprinters with tape punches could be used to prepare programs for entry into the computer.

Business computing is most obviously associated with IBM, a company that had large divisions building both computers and typewriters. The marriage of the two was inevitable considering the existing precedent. Beginning around 1960 it was standard for IBM computers to furnish a teleprinter as the operator interface, but IBM had a distinct heritage from the telecommunications industry and, for several reasons, was intent on maintaining that distinction. IBM's teleprinter-like devices were variously called Data Communications Systems, Printer-Keyboards, Consoles, and eventually Terminals. They generally operated over proprietary serial channels.

Other computer manufacturers didn't have typewriter divisions, and typewriters and teleprinters were actually rather complex mechanical devices and not all that easy to build. As a result, they tended to buy teleprinters from established manufacturers, often IBM or Western Electric. Consider the case of a rather famous non-IBM computer, the DEC PDP-1 of 1960. It came with a CRT graphics display as standard, and many sources will act as if this was the primary operator interface, but it is important to understand that early CRT graphics displays had a hard time with text. Text is rather complex to render when you are writing point-by-point to a CRT vector display from a rather slow machine. You would be surprised how many vertices a sentence has in it.

So despite the ready availability of CRTs in the 1960s (they were, of course, well established in the television industry), few computers used them for primary text input/output. Instead, the PDP-1 was furnished with a modified IBM typewriter as its console. This scheme of paying a third-party company (Soroban Engineering) to modify IBM typewriters for teleprinter control was apparently not very practical, and later DEC PDP models tended to use Western Electric Teletypes as user terminals. These had the considerable advantage that they were already designed to operate over long telephone circuits, making it easy to install multiple terminals throughout a building for time sharing use.

Indeed, time sharing was a natural fit for teleprinter terminals. With a teleprinter and a computer with a suitable modem, you could "call in" to a time sharing computer over the telephone from a remote office. Most of the first practical "computer networks" (term used broadly) were not actually networks of computers, but a single computer with many remote terminals. This architecture evolved into the BBS and early Internet-like services such as CompuServe. The idea was surprisingly easy to implement once time sharing operating systems were developed; the necessary hardware was already available from Western Electric.

While I cannot swear to the accuracy of this attribution, many sources suggest that the term "tty" as a generic reference to a user terminal or serial I/O channel originated with DEC. It seems reasonable; DEC's software was very influential on the broader computer industry, particularly outside of IBM. UNIX originally targeted a PDP-11 with teleprinters. While I can't prove it, it seems quite believable that the tty terminology was adopted directly from RT-11 or another operating system that Bell Labs staff might have used on the PDP-11.

Computers were born of the teleprinter and would inevitably come to consume them. After all, what is a computer but a complex teleprinter? Today, displaying text and accepting it from a keyboard is among the most basic functions of computers, and computers continue to perform this task using an architecture that would be familiar to engineers in the 1970s. They would likely be more surprised by what hasn't changed than what has: many of us still spend a lot of time in graphical software pretending to be a video display terminal built for compatibility with teleprinters.

And we're still using that 7-bit ASCII code a lot, aren't we. At least Baudot died out and we get to enjoy lower case letters.

[1] Actor, singer, etc. Gene Autry had worked as a telegrapher before he began his career in entertainment. This resulted in no small number of stories of a celebrity stand-in at the telegraph office. Yes, this is about to be a local history anecdote. It is fairly reliably reported that Gene Autry once volunteered to stand in for the telegrapher and station manager at the small Santa Fe Railroad station in Socorro, New Mexico, as the telegrapher had been temporarily overwhelmed by the simultaneous arrival of a packed train and a series of telegrams. There are enough of these stories about Gene that I think he really did keep his Morse sharp well into his acting career.

[2] Baud is a somewhat confusing unit derived from Baudot. Baud refers to the number of symbols per second on the underlying communication medium. For simple binary systems (and thus many computer communications systems we encounter daily), baud rate is equivalent to bit rate (bps). For systems that employ multi-level signaling, the bit rate will be higher than the baud rate, as multiple bits are represented per symbol on the wire. Methods like QAM are useful because they result in bit rates that are many multiples of the baud rate, reducing the bandwidth on the wire.

In the past (in fact two years ago, proof I have been doing this for a while now!) I wrote about the "inconvenient truth" that structural aspects of the Internet make truly decentralized systems infeasible, due to the lack of a means to perform broadcast discovery. As a result, most distributed systems rely on a set of central, semi-static nodes to perform initial introductions.

For example, Bitcoin relies on a small list of volunteer-operated domain names that resolve to known-good full nodes. Tor similarly uses a small set of central "directory servers" that provide initial node lists. Both systems have these lists hardcoded into their clients; coincidentally, both have nine trusted, central hostnames.

This sort of problem exists in basically all distributed systems that operate in environments where it is not possible to shout into the void and hope for a response. The internet, for good historic reasons, does not permit this kind of behavior. Here we should differentiate between distributed and decentralized, two terms I do not tend to select very carefully. Not all distributed systems are decentralized, indeed, many are not. One of the easiest and most practical ways to organize a distributed system is according to a hierarchy. This is a useful technique, so there are many examples, but a prominent and old one happens to also be part of the drivetrain mechanics of the internet: DNS, the domain name system.

My reader base is expanding and so I will provide a very brief bit of background. Many know that DNS is responsible for translating human-readable names like "computer.rip" into the actual numerical addresses used by the internet protocol. Perhaps a bit fewer know that DNS, as a system, is fundamentally organized around the hierarchy of these names. To examine the process of resolving a DNS name, it is sometimes more intuitive to reverse the name, and instead of "computer.rip", discuss "rip.computer" [1].

This name is hierarchical, it indicates that the record "computer" is within the zone "rip". "computer" is itself a zone and can contain yet more records, we tend to call these subdomains. But the term "subdomain" can be confusing as everything is a subdomain of something, even "rip" itself, which in a certain sense is a subdomain of the DNS root "." (which is why, of course, a stricter writing of the domain name computer.rip would be computer.rip., but as a culture we have rejected the trailing root dot).

Many of us probably know that each level of the DNS hierarchy has authoritative nameservers, operated typically by whoever controls the name (or their third-party DNS vendor). "rip" has authoritative DNS servers provided by a company called Rightside Group, a subsidiary of the operator of websites like eHow that went headfirst into the great DNS land grab and snapped up "rip" as a bit of land speculation, alongside such attractive properties as "lawyer" and "navy" and "republican" and "democrat", all of which I would like to own the "computer" subdomain of, but alas such dictionary words are usually already taken.

"computer.rip", of course, has authoritative nameservers operated by myself or my delegate. Unlike some people I know, I do not have any nostalgia for BIND, and so I pay a modest fee to a commercial DNS operator to do it for me. Some would be surprised that I pay for this; DNS is actually rather inexpensive to operate and authoritative name servers are almost universally available as a free perk from domain registrars and others. I just like to pay for this on the general feeling that companies that charge for a given service are probably more committed to its quality, and it really costs very little and changing it would take work.

To the observant reader, this might leave an interesting question. If even the top-level domains are subdomains of a secret, seldom-seen root domain ".", who operates the authoritative name servers for that zone?

And here we return to the matter of even distributed systems requiring central nodes. Bitcoin uses nine harcoded domain names for initial discovery of decentralized peers. DNS uses thirteen harcoded root servers to establish the top level of the hierarchy.

These root servers are commonly referred to as a.root-servers.net through m.root-servers.net, and indeed those are their domain names, but remember that when we need to use those root servers we have no entrypoint into the DNS hierarchy and so are not capable of resolving names. The root servers are much more meaningfully identified by their IP addresses, which are "semi-harcoded" into recursive resolves in the form of what's often called a root hints file. You can download a copy, it's a simple file in BIND zone format that BIND basically uses to bootstrap its cache.

And yes, there are other DNS implementations too, a surprising number of them, even in wide use. But when talking about DNS history we can mostly stick to BIND. BIND used to stand for Berkeley Internet Name Domain, and it is an apt rule of thumb in computer history that anything with a reference to UC Berkeley in the name is probably structurally important to the modern technology industry.

One of the things I wanted to get at, when I originally talked about central nodes in distributed systems, is the impact it has on trust and reliability. The TOR project is aware that the nine directory servers are an appealing target for attack or compromise, and technical measures have been taken to mitigate the possibility of malicious behavior. The Bitcoin project seems to mostly ignore that the DNS seeds exist, but of course the design of the Bitcoin system limits their compromise to certain types of attacks. In the case of DNS, much like most decentralized systems, there is a layer of long-lived caching for top-level domains that mitigates the impact of unavailability of the root servers, but still, in every one of these systems, there is the possibility of compromise or unavailability if the central nodes are attacked.

And so there is always a layer of policy. A trusted operator can never guarantee the trustworthiness of a central node (the node could be compromised, or the trusted operator could turn out to be the FBI), but it sure does help. Tor's directory servers are operated by the Tor project. Bitcoin's DNS seeds are operated by individuals with a long history of involvement in the project. DNS's root nodes are operated by a hodgepodge of companies and institutions that were important to the early internet.

Verisign operates two, of course. A California university operates one, of course, but amusingly not Berkeley. Three are operated by various arms of US defense. Some internet industry associations, an NCC, another university, ICANN runs one of them themselves. It's pretty random, though, and just reflects a set of organizations prominently involved in the early internet.

Some people, even some journalists I've come across, hear that there are 13 name servers and picture 13 4U boxes with a lot of blinking lights in heavily fortified data centers. Admittedly this description was more or less accurate in the early days, and a couple of the smaller root server operators did have single machines until surprisingly recently. But today, all thirteen root server IP addresses are anycast groups.

Anycast is not a concept you run into every day, because it's not really useful on local networks where multicast can be used. But it's very important to the modern internet. The idea is this: an IP address (really a subnetwork) is advertised by multiple BGP nodes. Other BGP nodes can select the advertisement they like the best, typically based on lowest hop count. As a user, you connect to a single IP address, but based on the BGP-informed routing tables of internet service providers your traffic could be directed to any number of sites. You can think of it as a form of load balancing at the IP layer, but it also has the performance benefit of users mostly connecting to nearby nodes, so it's widely used by CDNs for multiple reasons.

For DNS, though, where we often have a bootstrapping problem to solve, anycast is extremely useful as a way to handle "special" IP addresses that are used directly. For authoritative DNS servers like 192.5.5.241 [2001:500:2f::f] [2] (root server F) or recursive resolvers like 8.8.8.8 [2001:4860:4860::8888] (Google public DNS), anycast is the secret that allows a "single" address to correspond to a distributed system of nodes.

So there are thirteen DNS root servers in the sense that there are thirteen independently administered clusters of root servers (with the partial exception of A and J, both operated by Verisign, due to their acquisition of former A operator Network Solutions). Each of the thirteen root servers is, in practice, a fairly large number of anycast sites, sometimes over 100. The root server operators don't share much information about their internal implementation, but one can assume that in most cases the anycast sites consist of multiple servers as well, fronted by some sort of redundant network appliance. There may only be thirteen of them, but each of the thirteen is quite robust. For example, the root servers typically place their anycast sites in major internet exchanges distributed across both geography and provider networks. This makes it unlikely that any small number of failures would seriously affect the number of available sites. Even if a root server were to experience a major failure due to some sort of administration problem, there are twelve more.

Why thirteen, you might ask? No good reason. The number of root servers basically grew until the answer to an NS request for "." hit the 512 byte limit on UDP DNS responses. Optimizations over time allowed this number to grow (actually using single letters to identify the servers was one of these optimizations, allowing the basic compression used in DNS responses to collapse the matching root-servers.net part). Of course IPv6 blew DNS response sizes completely out of the water, leading to the development of the EDNS extension that allows for much larger responses.

13 is no longer the practical limit, but with how large some of the 13 are, no one sees a pressing need to add more. Besides, can you imagine the political considerations in our modern internet environment? The proposed operator would probably be Cloudflare or Google or Amazon or something and their motives would never be trusted. Incidentally, many of the anycast sites for root server F (operated by ISC) are Cloudflare data centers used under agreement.

We are, of course, currently trusting the motives of Verisign. You should never do this! But it's been that way for a long time, we're already committed. At least it isn't Network Solutions any more. I kind of miss when SRI was running DNS and military remote viewing.

But still, there's something a little uncomfortable about the situation. Billions of internet hosts depend on thirteen "servers" to have any functional access to the internet.

What if someone attacked them? Could they take the internet down? Wouldn't this cause a global crisis of a type seldom before seen? Should I be stockpiling DNS records alongside my canned water and iodine pills?

Wikipedia contains a great piece of comedic encyclopedia writing. In its article on the history of attacks on DNS root servers, it mentions the time, in 2012, that some-pastebin-user-claiming-to-be-Anonymous (one of the great internet security threats of that era) threatened to "shut the Internet down". "It may only last one hour, maybe more, maybe even a few days," the statement continues. "No matter what, it will be global. It will be known."

That's the end of the section. Some Wikipedia editor, no doubt familiar with the activities of Anonymous in 2012, apparently considered it self-evident that the attack never happened.

Anonymous may not have put in the effort, but others have. There have been several apparent DDoS attacks on the root DNS servers. One, in 2007, was significant enough that four of the root servers suffered---but there were nine more, and no serious impact was felt by internet users. This attack, like most meaningful DDoS, originated with a botnet. It had its footprint primarily in Korea, but C2 in the United States. The motivation for the attack, and who launched it, remains unknown.

There is a surprisingly large industry of "booters," commercial services that, for a fee, will DDoS a target of your choice. These tend to be operated by criminal groups with access to large botnets; the botnets are sometimes bought and sold and get their tasking from a network of resellers. It's a competitive industry. In the past, booters and botnet operators have sometimes been observed announcing a somewhat random target and taking it offline as, essentially, a sales demonstration. Since these demonstrations are a known behavior, any time a botnet targets something important for no discernible reason, analysts have a tendency to attribute it to a "show of force." I have little doubt that this is sometimes true, but as with the tendency to attribute monumental architecture to deity worship, it might be an overgeneralization of the motivations of botnet operators. Sometimes I wonder if they made a mistake, or maybe they were just a little drunk and a lot bored, who is to say?

The problem with this kind of attribution is evident in the case of the other significant attack on the DNS root servers, in 2015. Once again, some root servers were impacted badly enough that they became unreliable, but other root servers held on and there was little or even no impact to the public. This attack, though, had some interesting properties.

In the 2007 incident, the abnormal traffic to the root servers consisted of large, mostly-random DNS requests. This is basically the expected behavior of a DNS attack; using randomly generated hostnames in requests ensures that the responses won't be cached, making the DNS server exert more effort. Several major botnet clients have this "random subdomain request" functionality built in, normally used for attacks on specific authoritative DNS servers as a way to take the operator's website offline. Chinese security firm Qihoo 360, based on a large botnet honeypot they operate, reports that this type of DNS attack was very popular at the time.

The 2015 attack was different, though! Wikipedia, like many other websites, describes the attack as "valid queries for a single undisclosed domain name and then a different domain the next day." In fact, the domain names were disclosed, by at least 2016. The attack happened on two days. On the first day, all requests were for 336901.com. The second day, all requests were for 916yy.com.

Contemporaneous reporting is remarkably confused on the topic of these domain names, perhaps because they were not widely known, perhaps because few reporters bothered to check up on them thoroughly. Many sources make it sound like they were random domain names perhaps operated by the attacker, one goes so far as to say that they were registered with fake identities.

Well, my Mandarin isn't great, and I think the language barrier is a big part of the confusion. No doubt another part is a Western lack of familiarity with Chinese internet culture. To an American in the security industry, 336901.com would probably look at first like the result of a DGA or domain generation algorithm. A randomly-generated domain used specifically to be evasive. In China, though, numeric names like this are quite popular. Qihoo 360 is, after all, domestically branded as just 360---360.cn.

As far as I can tell, both domains were pretty normal Chinese websites related to mobile games. It's difficult or maybe impossible to tell now, but it seems reasonable to speculate that they were operated by the same company. I would assume they were something of a gray market operation, as there's a huge intersection between "mobile games," "gambling," and "target of DDoS attacks." For a long time, perhaps still today in the right corners of the industry, it was pretty routine for gray-market gambling websites to pay booters to DDoS each other.

In a 2016 presentation, security researchers from Verisign (Weinberg and Wessels) reported on their analysis of the attack based on traffic observed at Verisign root servers. They conclude that the traffic likely originated from multiple botnets or at least botnet clients with different configurations, since the attack traffic can be categorized into several apparently different types [3]. Based on command and control traffic from a source they don't disclose (perhaps from a Verisign honeynet?), they link the attack to the common "BillGates" [4] botnet. Most interestingly, they conclude that it was probably not intended as an attack on the DNS root: the choice of fixed domain names just doesn't make sense, and the traffic wasn't targeted at all root servers.

Instead, they suspect it was just what it looks like: an attack on the two websites the packets queried for, that for some reason was directed at the root servers instead of the authoritative servers for that second-level domain. This isn't a good strategy; the root servers are a far harder target than your average web hosting company's authoritative servers. But perhaps it was a mistake? An experiment to see if the root server operators might mitigate the DDoS by dropping requests for those two domains, incidentally taking the websites offline?

Remember that Qihoo 360 operates a large honeynet and was kind enough to publish a presentation on their analysis of root server attacks. Matching Verisign's conclusions, they link the attack to the BillGates botnet, and also note that they often observe multiple separate botnet C2 servers send tasks targeting the same domain names. This probably reflects the commercialized nature of modern botnets, with booters "subcontracting" operations to multiple botnet operators. It also handily explains Verisign's observation that the 2015 attack traffic seems to have come from more than one implementation a DNS DDoS.

360 reports that, on the first day, five different C2 servers tasked bots with attacking 336901.com. On the second day, three C2 servers tasked for 916yy.com. But they also have a much bigger revelation: throughout the time period of the attacks, they observed multiple tasks to attack 916yy.com using several different methods.

360 concludes that the 2015 DNS attack was most likely the result of a commodity DDoS operation that decided to experiment, directing traffic at the DNS roots instead of the authoritative server for the target to see what would happen. I doubt they thought they'd take down the root servers, but it seems totally reasonable that they might have wondered if the root server operators would filter DDoS traffic based on the domain name appearing in the requests.

Intriguingly, they note that some of the traffic originated with a DNS attack tool that had significant similarities to BillGates but didn't produce quite the same packets. Likely we will never know, but a likely explanation is that some group modified the BillGates DNS attack module or implemented a new one based on the method used by BillGates.

Tracking botnets gets very confusing very fast, there are just so many different variants of any major botnet client! BillGates originated, for example, as a Linux botnet. It was distributed to servers, not only through SSH but through vulnerabilities in MySQL and ElasticSearch. It was unusual, for a time, in being a major botnet that skipped over the most common desktop operating system. But ports of BillGates to Windows were later observed, distributed through an Internet Explorer vulnerability---classic Windows. Why someone chose to port a Linux botnet to Windows instead of using one of the several popular Windows botnets (Conficker, for example) is a mystery. Perhaps they had spent a lot of time building out BillGates C2 infrastructure and, like any good IT operation, wanted to simplify their cloud footprint.

High in the wizard's tower of the internet, thirteen elders are responsible for starting every recursive resolver on its own path to truth. There's a whole Neal Stephenson for Wired article there. But in practice it's a large and robust system. The extent of anycast routing used for the root DNS servers, to say nothing of CDNs, is one of those things that challenges are typical stacked view of the internet. Geographic load balancing is something we think of at high layers of the system, it's surprising to encounter it as a core part of a very low level process.

That's why we need to keep our thinking flexible: computers are towers of abstraction, and complexity can be added at nearly any level, as needed or convenient. Seldom is this more apparent than it is in any process called "bootstrapping." Some seemingly simpler parts of the internet, like DNS, rely on a great deal of complexity within other parts of the system, like BGP.

Now I'm just complaining about pedagogical use of the OSI model again.

[1] The fact that the DNS hierarchy is written from right-to-left while it's routinely used in URIs that are otherwise read left-to-right is one of those quirks of computer history. Basically an endianness inconsistency. Like American date order, to strictly interpret a URI you have to stop and reverse your analysis part way through. There's no particular reason that DNS is like that, there was just less consistency over most significant first/least significant first hierarchical ordering at the time and contemporaneous network protocols (consider the OSI stack) actually had a tendency towards least significant first.

[2] The IPv4 addresses of the root servers are ages old and mostly just a matter of chance, but the IPv6 addresses were assigned more recently and allowed an opportunity for something more meaningful. Reflecting the long tradition of identifying the root servers by their letter, many root server operators use IPv6 addresses where the host part can be written as the single letter of the server (i.e. root server C at [2001:500:2::c]). Others chose a host part of "53," a gesture at the port number used for DNS (i.e. root server J, [2001:7fe::53]). Others seem more random, Verisign uses 2:30 for both of their root servers (i.e. root server A, [2001:503:ba3e::2:30]), so maybe that means something to them, or maybe it was just convenient. Amusingly, the only operator that went for what I would call an address pun is the Defense Information Systems Agency, which put root server G at [2001:500:12::d0d].

[3] It really dates this story that there was some controversy around the source IPs of the attack, originating with none other than deceased security industry personality John McAfee. He angrily insisted that it was not plausible that the source IPs were spoofed. Of course botnets conducting DDoS attacks via DNS virtually always spoof the source IP, as there are few protections in place (at the time almost none at all) to prevent it. But John McAfee has always had a way of ginning up controversy where none was needed.

[4] Botnets are often bought, modified, and sold. They tend to go by various names from different security researchers and different variants. I'm calling this one "BillGates" because that's the funniest of the several names used for it.

Last time, we left off at the fact that modern films are distributed with their audio in multiple formats. Most of the time, there is a stereo version of the audio, and a multi-channel version of the audio that is perhaps 5.1 or 7.1 and compressed using one of several codecs that were designed within the film industry for this purpose.

But that was all about film, in physical form. In the modern world, films go out to theaters in the form of Digital Cinema Packages, a somewhat elaborate format that basically comes down to an encrypted motion JPEG 2000 stream with PCM audio. There are a lot of details there that I don't know very well and I don't want to get hung up on anyway, because I want to talk about the consumer experience.

As a consumer, there are a lot of ways you get movies. If you are a weirdo, you might buy a Blu-Ray disc. Optical discs are a nice case, because they tend to conform to a specification that allows relatively few options (so that players are reasonable to implement). Blu-Ray are allowed to encode their audio as linear PCM [1], Dolby Digital, Dolby TrueHD, DTS, DTS-HD, or DRA.

DRA is a common standard in the Chinese market but not in the US (that's where I live), so I'll ignore it. That still leaves three basic families of codecs, each of which have some variations. One of the interesting things about the Blu-Ray specification is that PCM audio can incorporate up to eight channels. The Blu-Ray spec allows up to 27,648 Kbps of audio, so it's actually quite feasible to do uncompressed, 24-bit, 96 kHz, 7.1 audio on a Blu-Ray disc. This is an unusual capability in a consumer standard and makes the terribly named Blu-Ray High Fidelity Pure Audio standard for Blu-Ray audio discs make more sense. Stick a pin in that, though, because you're going to have a tough time actually playing uncompressed 7.1.

On the other hand, you might use a streaming service. There's about a million of those and half of them have inane names ending in Plus, so I'm going to simplify by pretending that we're back in 2012 and Netflix is all that really matters. We can infer from Netflix help articles that Netflix delivers audio as AAC or Dolby Digital.

Or, consider the case of video files that you obtained by legal means. I looked at a few of the movies on my NAS to take a rough sampling. Most older films, and some newer ones, have stereo AAC audio. Some have what VLC describes as A52 aka AC3. A/52 is an ATSC standard that is equivalent to AC3, and AC-3 (hyphen inconsistent) is sort of the older name of Dolby Digital or the name of the underlying transport stream format, depending on how you squint at it. Less common, in my hodgepodge sample, is DTS, but I can find a few.

VLC typically describes the DTS and Dolby Digital as 3F2M/LFE, which is a somewhat eccentric (and I think specific to VLC) notation for 5.1 surround. An interesting detail is that VLC differentiates 3F2M/LFE and 3F2R/LFE, both 5.1, but with the two "surround" channels assigned to either side or rear positions. While 5.1 configurations with the surround channels to the side seem to be more standard, you could potentially put the two surround channels to the rear. Some formats have channel mapping metadata that can differentiate the two.

Because there is no rest for the weary, there is some inconsistency between "5.1 side" and "5.1 rear" in different standards and formats. At the end of the day, most applications don't really differentiate. I tend to consider surround channels on the side to be "correct," in that movie theaters are configured that way and thus it's ostensibly the design target for films. One of few true specifications I could find for general use, rather than design standards specific to theaters like THX, is ITU-R BS 775. It states that the surround channels of a 5.1 configuration should be mostly to the side, but slightly behind the listener.

That digression aside, it's unsurprising that a video file could contain a multi-channel stream. Most video containers today can support basically arbitrary numbers of streams, and you could put uncompressed multichannel audio into such a container if you wanted. And yet, multi-channel audio in films almost always comes in the form of a Dolby Digital or DTS stream. Why is that? Well, in part, because of tradition: they used to be the formats used by theaters, although digital cinema has somewhat changed that situation and the consumer versions have usually been a little different in the details. But the point stands, films are usually mastered in Dolby or DTS, so the "home video" release goes out with Dolby or DTS.

Another reason, though, is the problem of interconnections.

Let's talk a bit about interconnections. In a previous era of consumer audio, the age of "hi-fi," component systems dominated residential living rooms. In a component system, you had various audio sources that connected to a device that came to be known as a "receiver" since it typically had an FM/AM radio receiver integrated. It is perhaps more accurate to refer to it as an amplifier since that's the main role it serves in most modern systems, but there's also an increasing tendency to think of their input selection and DSP features as part of a preamp. The device itself is sometimes referred to as a preamp, in audiophile circles, when component amplifiers are used to drive the actual speakers. You can see that in these conventional component systems you need to move audio signals between devices. This kind of set up, though, is not common in households with fewer than four bathrooms and one swimming pool.

Most consumers today seem to have a television and, hopefully, some sort of audio device like a soundbar. Sometimes there are no audio interconnections at all! Often the only audio interconnection is from the TV to the soundbar via HDMI. Sometimes it's wireless! So audio interconnects as a topic can feel a touch antiquated today, but these interconnects still matter a lot in practice. First, they are often either the same as something used in industry or similar to something used in industry. Second, despite the increasing prevalence of 5.1 and 7.1 soundbar systems with wireless satellites, the kind of people with a large Blu-Ray collection are still likely to have a component home theater system. Third, legacy audio interconnects don't die that quickly, because a lot of people have an older video game console or something that they want to work with their new TV and soundbar, so manufacturers tend to throw in one or two audio interconnects even if they don't expect most consumers to use them.

So let's think about how to transport multi-channel audio. An ancient tradition in consumer audio says that stereo audio will be sent between components on two sets of two-conductor cables terminated by RCA connectors. The RCA connector dates back to to the Radio Corporation of America and, apparently, at least 1937. It remains in widespread service today. There are a surprising number of variations in this interconnect, in practice.

For one, the audio cables may be coaxial or just zipped up in a common jacket. Coaxial audio cables are a bit more expensive and a lot less flexible but admit less noise. There is a lot of confusion in this area because a particular digital transport we'll talk about later specified coaxial cables terminated in RCA connectors, but then is frequently used with non-coaxial cables terminated in RCA connectors, and for reasonable lengths usually still works fine. This has lead to a lot of consumer confusion and people thinking that any cable with RCA connectors is coaxial, when in fact, most of them are not. Virtually all of them are not. Unless you specifically paid more money to get a coaxial one, it's not, and even then sometimes it's not, because Amazon is a hotbed of scams.

Second, though these connections are routinely described as "line level" as if that means something, there is remarkably little standardization of the actual signaling. There are various conventions like 1.7v peak-to-peak and 2v peak-to-peak and about 1v peak-to-peak, and few consumer manufacturers bother to tell you which convention they have followed. There are also a surprising number of ways of expressing signaling levels, involving different measurement bases (peak vs RMS) and units (dBv vs dBu), making it a little difficult to interpret specifications when they are provided. This whole mess is just one of the reasons you find yourself having to make volume adjustments for different sources, or having to tune input levels on receivers with that option [2].

But that's all sort of a tangent, the point here is multi-channel audio. You could, conceptually, move 5.1 over six RCA cables, or 7.1 over eight RCA cables. Home theater receivers used to give you this option, but much like analog HDTV connections, it has largely disappeared.

There is one other analog option: remember Pro Logic, from the film soundtracks? that matrixed five channels into the analog stereo? Some analog formats like VHS and LaserDisc often had a Pro Logic soundtrack that could be "decoded" (really dematrixed) by a receiver with that capability, which used to be common. In this case you can transport multi-channel audio over your normal two RCA cables. The matrixing technique was always sort of cheating, though, and produces inferior results to actual multichannel interconnects. It's no longer common either.

Much like video, audio interconnects today have gone digital. Consumer digital audio really took flight with the elegantly named Sony/Philips Digital Interface, or S/PDIF. S/PDIF specifies a digital format that is extremely similar to, but not quite the same as, a professional digital interconnect called AES3. AES3 is typically carried on a three-conductor (balanced) cable with XLR connectors, though, which are too big an expensive for consumer equipment. In one of the weirder decisions in the history of consumer electronics, one that I can only imagine came out of an intractable political fight, S/PDIF specified two completely different physical transports: one electrical, and one optical.

The electrical format should be transmitted over a coaxial cable with RCA connectors. In practice it is often used over non-coaxial cables with RCA connectors, which will usually work fine if the length is short and nothing nearby is too electrically noisy. S/PDIF over non-coaxial cables is "fine" in the same way that HDMI cables longer than you are tall are "fine." If it doesn't work reliably, try a more expensive cable and you'll probably be good.

The optical format is used with cheap plastic optical cables terminated in a square connector called Toslink, originally for Toshiba Link, after the manufacturer that gave us the optical variant. Toslink is one of those great disappointments in consumer products. Despite the theoretical advantages of an optical interconnect, the extremely cheap cables used with Toslink mean it's mostly just worse than the electrical transport, especially when it comes to range [3].

But the oddity of S/PDIF's sibling formats isn't the interesting thing here. Let's talk about the actual S/PDIF bitstream, the very-AES3-like format the audio actually needs to get through.

S/PDIF was basically designed for CDs, and so it comfortably carries CD audio: two channels of 16 bit samples at 44.1kHz. In fact, it can comfortably go further, carrying 20 (or with the right equipment even 24) bit samples at the 48 kHz sampling rate more common of digital audio other than CDs. That's for two channels, though. Make the leap to six channels for 5.1 and you are well beyond the capabilities of an S/PDIF transceiver.

You see where this is going? compression.

See, the problems that Dolby Digital and DTS solved, of fitting multichannel audio onto the limited space of a 35mm film print, also very much exist in the world of S/PDIF. CDs brought us uncompressed digital audio remarkably early on, but also set sort of a constraint on the bitrate of digital audio streams that ensured the opposite in the world of multi-channel theatrical sound. It sort of makes sense, anyway. DTS soundtracks came on CDs!

Of course even S/PDIF is looking rather long in the tooth today. I don't think I use it at all any more, which is not something I expected to be saying this soon. Today, though, all of my audio sources and sinks are either analog or have HDMI. HDMI is the de facto norm for consumer digital audio today.

HDMI is a complex thing when it comes to audio or, really, just about anything. Details like eARC and the specific HDMI version have all kinds of impacts on what kind of audio can be carried, and the same is true for video as well. I am going to spare a lengthy diversion into the many variants of HDMI, which seem almost as numerous as those of USB, and talk about HDMI 2.1.

Unsurprisingly, considering the numerous extra conductors and newer line coding, HDMI offers a lot more bandwidth for audio than S/PDIF. In fact, you can transport 8 channels of uncompressed 24-bit PCM at 192kHz. That's about 37 Mbps, which is not that fast for a data transport but sure is pretty fast for an audio cable. Considering the bandwidth requirements for 4K video at 120Hz, though, it's only a minor ask. With HDMI, compression of audio is no longer necessary.

But we still usually do it.

Why? Well, basically everything can handle Dolby Digital or DTS, and so films are mostly mastered to Dolby Digital or DTS, and so we mostly use Dolby Digital or DTS. That's just the way of things.

One of the interesting implications of this whole thing is that audio stacks have to deal with multiple formats and figure out which format is in use. That's not really new, with Dolby Pro Logic you either had to turn it on/off with a switch or the receiver had to try to infer whether or not Pro Logic had been used to matrix a multichannel soundtrack to stereo. For S/PDIF, IEC 61937 standardizes a format that can be used to encapsulate a compressed audio stream with sufficient metadata to determine the type of compression. HDMI adopts the same standard to identify compressed audio streams (and, in general, HDMI audio is pretty much in the same bitstream format as good old S/PDIF, but you can have a lot more of it).

In practice, there are a lot of headaches around this format switching. For one, home theater receivers have to switch between decoding modes. They mostly do this transparently and without any fuss, but I've owned a couple that had occasional issues with losing track of which format was in use, leading to dropouts. Maybe related to signal dropouts but my current receiver has the same problem with internal sources, so it seems more like a software bug of some sort.

It's a lot more complicated when you get out of dedicated home theater devices, though. Consider the audio stack of a general-purpose operating system. First, PCs rarely have S/PDIF outputs, so we are virtually always talking about HDMI. For a surprisingly long time, common video cards had no support for audio over HDMI. This is fortunately a problem of the past, but unfortunately ubiquitous audio over HDMI means that your graphics drivers are now involved in the transport of audio, and graphics drivers are notoriously bad at reliably producing video, much less dealing with audio as a side business. I shudder to think of the hours of my life I have lost dealing with defects of AMD's DTS support.

Things are weird on the host software side, though. The operating system does not normally handle sound in formats even resembling Dolby Digital or DTS. So, when you play a video file with audio encoded in one of those formats, a "passthrough" feature is typically used to deliver the compressed stream directly to the audio (often actually video) device, without normal operating system intervention. We are reaching the point where this mostly just works but you will still notice some symptoms of the underlying complexity.

On Linux, it's possible to get this working, but in part because of licensing issues I don't think any distros will do it right out of the box. My knowledge may be out of date as I haven't tried for some time, but I am still seeing Kodi forum threads about bash scripts to bypass PulseAudio, so things seem mostly unchanged.

There are other frustrations, as well. For one, the whole architecture of multichannel audio interconnection is based around sinks detecting the mode used by the source. That means that your home theater receiver should figure out what your video player is doing, but your video player has no idea what your home theater receiver is doing. This manifests in maddening ways. Consider, for example, the number of blog posts I ran across (while searching for something else!) about how to make Netflix less quiet by disabling surround sound.

If Netflix has 5.1 audio they deliver it; they don't know what your speaker setup is. But what if you don't have 5.1 speakers? In principal you could downmix the 5.1 back to stereo, and a lot of home theater receivers have DSP modes that do this (and in general downmix 5.1 or 7.1 to whatever speaker channels are active, good for people with less common setups like my own 3.1). But you'd have to turn that on, which means having a receiver or soundbar or whatever that is capable, understanding the issue, and knowing how to enable that mode. That is way more than your average Netflix watcher wants to think about any of this. In practice, setting the Netflix player to only ever provide stereo audio is an easier fix.

The use of compressed multichannel formats that are decoded in the receiver rather than the computer playing back introduces other problems as well, like source equalization. If you have a computer connected to a home theater receiver (which is a ridiculous thing to do and yet here I am), you have two completely parallel audio stacks: "normal" audio that passes through the OS sound server and goes to the receiver as PCM, and "surround sound" that bypasses the OS sound server and goes to the receiver as Dolby Digital or DTS. It is very easy to have differences in levels, adjustments, latency, etc. between these two paths. The level problem here is just one of the several factors in the perennial "Plex is too quiet" forum threads [4].

Finally, let's talk about what may be, to some readers, the elephant in the room. I keep talking about Dolby Digital and DTS, but both are 5.1 formats, and 5.1 is going out of fashion in the movie world. Sure, there's Dolby Digital Plus which is 7.1, but it's so similar to the non-plus variant that there isn't much use in addressing them separately. Insert the "Plus" after Dolby Digital in the proceeding paragraphs if it makes you feel better.

But there are two significantly different formats appearing on more and more film releases, especially in the relatively space-unconstrained Blu-Ray versions: lossless surround sound and object-based surround sound.

First, lossless is basically what it sounds like. Dolby TrueHD and DTS-HD are both formats that present 7.1 surround with only lossless compression, at the cost of a higher bitrate than older media and interconnects support. HDMI can easily handle these, and if you have a fairly new setup of a Blu-Ray player and recent home theater receiver connected by HDMI you should be able to enjoy a lossless digital soundtrack on films that were released with one. That's sort of the end of that topic, it's nothing that revolutionary.

But what about object-based surround sound? I'm using that somewhat lengthy term to try to avoid singling out one commercial product, but, well, there's basically one commercial product: Dolby Atmos. Atmos is heralded as a revolution in surround sound in a way that makes it sort of hard to know what it actually is. Here's the basic idea: instead of mastering a soundtrack by mixing audio sources into channels, you master a soundtrack by specifying the physical location (in cartesian coordinates) of each sound source.

When the audio is played back, an Atmos decoder then mixes the audio into channels on the fly, using whatever channels are available. Atmos allows the same soundtrack to be used by theaters with a variety of different speaker configurations, and as a result, makes it practical for theaters to expand into much higher channel counts.

Theaters aren't nearly as important a part of the film industry as they used to be, though, and unsurprisingly Atmos is heavily advertised for consumer equipment as well. How exactly does that work?

Atmos is conveyed on consumer equipment as 7.1 Dolby Digital Plus or Dolby TrueHD with extra metadata.

If you know anything about HDR video, also known as SDR video with extra metadata, you will find this unsurprising. But some might be confused. The thing is, the vast majority of consumers don't have Atmos equipment, and with lossless compression soundtracks are starting to get very large so including two complete copies isn't very appealing. The consumer encoding of Atmos was selected to have direct backward compatibility to 7.1 systems, allowing normal playback on pre-Atmos equipment.

For Atmos-capable equipment, an extra PCM-like subchannel (at a reduced bitrate compared to the audio channels) is used to describe the 3D position of specific sound sources. Consumer Atmos decoders cannot support as many objects as the theatrical version, so part of the process of mastering an Atmos film for home release is clustering nearby objects into groups that are then treated as a single object by the consumer Atmos decoder. One way to think about this is that Atmos is downmixed to 7.1, and in the process a metadata stream is created that can be used to upmix back to Atmos mostly correctly. If it sounds kind of like matrix encoding it kind of is, in effect, which is perhaps part of why Dolby's marketing materials are so insistent that it is not matrix encoding. To be fair it is a completely different implementation, but has a similar effect of reducing the channel separation compared to the original source.

Also I don't think Atmos has really taken off in home setups? I might just be out of date here, I think half the soundbars on the market today claim Atmos support and amazing feats with their five channels two of which are pointed up. I'm just pretty skeptical of the whole "we have made fewer, smaller speakers behave as if they were more, bigger speakers" school of audio products. Sorry Dr. Bose, there's just no replacement for displacement.

[1] The term Linear PCM or LPCM is used to clarify that no companding has been performed. This is useful because PCM originated for the telephone network, which uses companding as standard. LPCM clarifies that neither μ-law companding nor A-law companding has been performed. I will mostly just use PCM because I'm talking about movies and stuff, where companding digital audio is rare.

[2] There is also the matter of magnetic sources like turntables and microphones that produce much lower output levels than a typical "line level." Ideally you need a preamplifier with adjustable gain for these, although in the case of turntables there are generally accepted gain levels for the two common types of cartridges. A lot of preamplifiers either let you choose from those two or give you no control at all. Traditionally a receiver would have a built-in preamplifier to bring up the level of the signal on the turntable inputs, but a lot of newer receivers have left this out to save money, which leads to hipsters with vinyl collections having to really crank the volume.

[3] I don't feel like I should have to say this, but in the world of audio, I probably do: if it works, it doesn't matter! The problem with optical is that it develops reliability problems over shorter lengths than the electrical format. If you aren't getting missing samples (dropouts) in the audio, though, it's working fine and changing around cables isn't going to get you anything. In practice the length limitations on optical don't tend to matter very much anyway, since the average distance between two pieces of a component home theater system is, what, ten inches?

[4] Among the myriad other factors here is the more difficult problem that movies mix most of the dialog into the center channel while most viewers don't have a center channel. That means you need to remix the center channel into left and right to recover dialog. So-called professionals mastering Blu-Ray releases don't always get this right, and you're in even more trouble if you're having to do it yourself.

Stereophonic or two-channel audio is so ubiquitous today that we tend to refer to all kinds of pieces of consumer audio reproduction equipment as "a stereo." As you might imagine, this is a relatively modern phenomenon. While stereo audio in concept dates to the late 19th century, it wasn't common in consumer settings until the 1960s and 1970s. Those were very busy decades in the music industry, and radio stations, records, and film soundtracks all came to be distributed primarily in stereo.

Given the success of stereo, though, one wonders why larger numbers of channels have met more limited success. There are, as usual, a number of factors. For one, two-channel audio was thought to be "enough" by some, considering that humans have two ears. Now it doesn't quite work this way in practice, and we are more sensitive to the direction from which sound comes than our binaural system would suggest. Still, there are probably diminishing returns, with stereo producing the most notable improvement in listening experience over mono.

There are also, though, technical limitations at play. The dominant form of recorded music during the transition to stereo was the vinyl record. There is a fairly straightforward way to record stereo on a record, by using a cartridge with coils on two opposing axes. This is the limit, though: you cannot add additional channels as you have run out of dimensions in the needle's free movement.

This was probably the main cause of the failure of quadraphonic sound, the first music industry attempt at pushing more channels. Introduced almost immediately after stereo in the 1970s, quadraphonic or four-channel sound seemed like the next logical step. It couldn't really be encoded on records, so a matrix encoding system was used in which the front-rear difference was encoded as phase shift in the left and right channels. In practice this system worked poorly, and especially early quadraphonic systems could sound noticeably worse than the stereo version. Wendy Carlos, an advocate of quadraphonic sound but harsh critic of musical electronics, complained bitterly about the inferiority of so-called quadraphonic records when compared to true four-channel recordings, for example on tape.

Of course, four-channel tape players were vastly more expensive than record players in the 1970s, as they ironically remain today. Quadraphonic sound was in a bind: it was either too expensive or too poor of quality to appeal to consumers. Quadraphonic radio using the same matrix encoding, while investigated by some broadcasters, had its own set of problems and never saw permanent deployment. Alan Parsons famously produced Pink Floyd's "Dark Side of the Moon" in quadraphonic sound; the effort was a failure in several ways but most memorably because, by the time of the album's release in 1973, the quadraphonic experiment was essentially over.

Three-or-more-channel-sound would have its comeback just a few years later, though, by the efforts of a different industry. Understanding this requires backtracking a bit, though, to consider the history of cinema prints.

Many are probably at least peripherally aware of Cinerama, an eccentric-seeming film format that used three separate cameras, and three separate projectors, to produce an exceptionally widescreen image. Cinerama's excess was not limited to the picture: it involved not only the three 35mm film reels for the three screen panels, but also a fourth 35mm film that was entirely coated with a magnetic substrate and was used to store seven channels of audio. Five channels were placed behind the screen, effectively becoming center, left, right, left side, and right side. The final two tracks were played back behind the audience, as the surround left and surround right.

Cinerama debuted in 1952, decades before 35mm films would typically carry even stereo audio. Like quadraphonic sound later, Cinerama was not particularly successful. By the time stereo records were common, Cinerama had been replaced by wider film formats and anamorphic formats in which the image was horizontally compressed by the lens of the camera, and expanded by the lens of the projector. Late Cinerama films like 2001: A Space Odyssey were actually filmed Super Panavision 70 and projected onto Cinerama screens from a single projector with a specialized lens.

There's a reason people talk so much about Cinerama, though. While it was not a commercial success, it was influential on the film industry to come. Widescreen formats, mostly anamorphic, would become increasingly common in the following decades. It would take years longer, but so would seven-channel theatrical sound.

"Surround sound," as these multi-channel formats came to be known in the late '50s, would come and go in theatrical presentations throughout the mid-century even as the vast majority of films were presented monaurally, with only a single channel. Most of these relied on either a second 35mm reel for audio only, or the greater area for magnetic audio tracks allowed by 70mm film. Both of these options were substantially more expensive for the presenting theater than mono, limiting surround sound mostly to high-end theaters and premiers. For surround sound to become common, it had to become cheap.

1971's A Clockwork Orange (I will try not to fawn over Stanley Kubrick too much but you are learning something about my film preferences here) employed a modest bit of audio technology, something that was becoming well established in the music industry but was new to film. The magnetic recordings used during the production process employed Dolby Type A noise reduction, similar to what became popular on compact cassette tapes, for a slight improvement in audio quality. The film was still mostly screened in magnetic mono, but it was the beginning of a profitable relationship between Dolby Labs and the film industry. Over the following years a number of films were released with Dolby Type A noise reduction on the actual distribution print, and some theaters purchased decoders to use with these prints. Dolby had bigger ambitions, though.

Around the same time, Kodak had been experimenting with the addition of stereo audio to 35mm release prints, using two optical tracks. They applied Dolby noise reduction to these experimental prints, and brought Dolby in to consult. This presented the perfect opportunity to implement an idea Dolby had been considering. Remember the matrix encoded quadraphonic recording that had been a failure for records? Dolby licensed a later-generation matrix decoder design from Sansui, and applied it to Kodak's stereo film soundtracks, allowing separation into four channels. While the music industry had placed the four channels at the four corners of the soundstage, the film industry had different tastes, driven mostly by the need to place dialog squarely in the center of the field. Dolby's variant of quadraphonic audio was used to present left, right, center, and a "surround" or side channel. This audio format went through several iterations, including much improved matrix decoding, and along the way picked up a name that is still familiar today: Dolby Stereo.

That Dolby Stereo is, in fact, a quadraphonic format reflects a general atmosphere of terminological confusion in the surround sound industry. Keep this in mind.

One of Dolby Stereo's most important properties was its backwards compatibility. The two optical tracks could be played back on a two-channel (or actually stereo) system and still sound alright. They could even be placed on the print alongside the older magnetic mono audio, providing compatibility with mono theaters. This compatibility with fewer channels became one of the most important traits in surround sound systems, and somewhat incidentally served to bring them to the consumer. Since the Dolby Stereo soundtrack played fine on a two-channel system, home releases of films on formats like VHS and Laserdisc often included the original Dolby Stereo audio from the print. A small industry formed around these home releases, licensing the Dolby technology to sell consumer decoders that could recover surround sound from home video.

For cost reasons these decoders were inferior to Dolby's own in several ways, and to avoid the hazard of damage to the Dolby Stereo brand, Dolby introduced a new marketing name for consumer Dolby Stereo decoders: Dolby Surround.

By the 1980s, Dolby Stereo, or Dolby Surround, had become the most common audio format on theatrical presentations and their home video releases. Even some television programs and direct-to-video material was recorded in Dolby Surround. Consumer stereo receivers, in the variant that came to be known as the home theater receiver, often incorporated Dolby Surround decoders. Improvements in consumer electronics brought the cost of proper Dolby Stereo decoders down, and so the home systems came to resemble the theatrical systems as well. Seeking a new brand to unify the whole mess of Dolby Stereo and Dolby Surround (which, confusingly, were often 4 and 3 channel, respectively), Dolby seems to have turned to the "Advanced Logic" and "Full Logic" terms once used by manufacturers of quadraphonic decoders. Dolby's theatrical sound solution came to be known as Dolby Pro Logic. A Dolby Pro Logic decoder processed two audio channels to produce a four-channel output. According to a modern naming convention, Dolby Pro Logic is a 4.0 system: four full-bandwidth channels.

This entire thing, so far, has been a preamble to the topic I actually meant to discuss. It's an interesting preamble, though! I just want to apologize that I didn't mean to write a history of multi-channel audio distribution and so this one isn't especially complete. I left out a number of interesting attempts at multi-channel formats, of which the film industry produced a surprising number, and instead focused on the ones that were influential and/or used for Kubrick films [1].

Dolby Pro Logic, despite its impressive name, was still an analog format, based on an early '70s technique. Later developments would see an increase in the number of channels, and the transition to digital audio formats.

Recall that 70mm film provided six magnetic audio channels, which were often used in an approximation of the seven-channel Cinerama format. Dolby experimented with the six-channel format, though, confusingly also under the scope of the Dolby Stereo product. During the '70s, Dolby observed that the ability of humans to differentiate the source of a sound is significantly reduced as the sound becomes lower in frequency. This had obvious potential for surround sound systems, enabling something analogous to chroma subsampling in video. The lower-frequency component of surround sound does not need to be directional, and for a sense of directionality the high frequencies are most important.

Besides, bassheads were coming to the film industry. The long-used Academy response curve fell out of fashion during the '70s, in part due to Dolby's work, in part due to generally improved loudspeaker technology, and in part due to the increasing popularity of bass-heavy action films. Several 70mm releases used one or more of the audio channels as dedicated bass channels.

For the 1979 film Apocalypse Now in its 70mm print, Dolby premiered a 5.1 format in which three full-bandwidth channels were used for center, left, and right, two channels with high-pass filtering were used for surround left and surround right, and one channel with low-pass filtering was used for bass. Apocalypse Now was not, in fact, the first film to use this channel configuration, but Dolby promoted it far more than the studios had.

Interestingly, while I know less about live production history, the famous cabaret Moulin Rouge apparently used a 5.1 configuration during the 1980s. Moulin Rouge was prominent enough to give the 5.1 format a boost in popularity, perhaps particularly important because of the film industry's indecision on audio formats.

The seven-channel concept of the original Cinerama must have hung around in the film industry, as there was continuing interest in a seven-channel surround configuration. At the same time, the music industry widely adopted eight-channel tape recorders for studio use, making eight-channel audio equipment readily available. The extension to 7.1 surround, adding left and right side channels to the 5.1 configuration, was perhaps obvious. Indeed, what I find strangest about 7.1 is just how late it was introduced to film. Would you believe that the first film released (not merely remastered or mixed for Blu-Ray) in 7.1 was 2010's Toy Story 3?

7.1 home theater systems were already fairly common by then, a notable example of a modern trend afflicting the film industry: the large installed base and cost avoidance of the theater industry means that consumer home theater equipment now evolves more quickly than theatrical systems. Indeed, while 7.1 became the gold standard in home theater audio during the 2000s, 5.1 remains the dominant format in theatrical sound systems today.

Systems with more than eight channels are now in use, but haven't caught on in the consumer setting. We'll talk about those later. For most purposes, eight-channel 7.1 surround sound is the most complex you will encounter in home media. The audio may take a rather circuitous route to its 7.1 representation, but, well, we'll get to that.

Let's shift focus, though, and talk a bit about the actual encodings. Audio systems up to 7.1 can be implemented using analog recording, but numerous analog channels impose practical constraints. For one, they are physically large, making it infeasible to put even analog 5.1 onto 35mm prints. Prestige multi-channel audio formats like that of IMAX often avoided this problem by putting the audio onto an entirely separate film reel (much like Cinerama back at the beginning), synchronized with the image using a pulse track and special equipment. This worked well but drove up costs considerably. Dolby Stereo demonstrated that it was possible to matrix four channels into two channels (with limitations), but considering the practical bandwidth of the magnetic or optical audio tracks on film you couldn't push this technique much further.

Remember that the theatrical audio situation changed radically during the 1970s, going from almost universal mono audio to four channels as routine and six channels for premiers and 70mm. During the same decade, the music reproduction industry, especially in Japan, was exploring another major advancement: digital audio encoding.

In 1980, the Compact Disc launched. Numerous factors contributed to the rapid success of CDs over vinyl and, to a lesser but still great extent, the compact cassette. One of them was the quality of the audio reproduction. CDs were a night and day change: records could produce an excellent result but almost always suffered from dirt and damage. Cassette tapes were better than most of us remember but still had limited bandwidth and a high noise floor, requiring Dolby noise reduction for good results. The CD, though, provided lossless digital audio.

Audio is encoded on an audio CD in PCM format. PCM, or pulse code modulation, is a somewhat confusing term that originated in the telephone industry. If we were to reinvent it today, we would probably just call it digital modulation. To encode a CD, audio is sampled (at 44.1 kHz for historic reasons) and quantized to 16 bits. A CD carries two channels, stereo, which was by then the universal format for music. Put together, those add up to 1.4Mbps. This was a very challenging data rate in 1980, and indeed, practical CD players relied on the fact that the data did not need to be read perfectly (error correcting codes were used) and did not need to be stored (going directly to a digital to analog converter). These were conveniently common traits of audio reproduction systems, and the CD demonstrated that digital audio was far more practical than the computing technology of the time would suggest.

The future of theatrical sound would be digital. Indeed, many films would be distributed with their soundtracks on CD.

There remained a problem, though: a CD could encode two channels. Even four channels wouldn't fit within the data rate CD equipment was capable of, much less six or eight. The film industry would need to formats that could encode six or eight channels of audio into either the bandwidth of a two-channel signal or into precious unused space on 35mm film prints.

Many ingenious solutions were developed. A typical 35mm film print today contains three distinct representations of the audio: a two-channel optical signal outside of the sprocket holes (which could encode Dolby Stereo), a continuous 2D barcode between the frame and sprocket holes which carries the SDDS (Sony Dynamic Digital Sound) digital signal, and individual 2D barcodes between the sprocket holes which encode the Dolby digital signal. Finally, a small pulse pattern at the very edge of the film provides a time code used for synchronization with audio played back from a CD, the DTS system.

But then, a typical 35mm film print today wouldn't exist, as 35mm film distribution has all but disappeared. Almost all modern film is played back entirely digitally from some sort of flexible stream container. You would think, then, that the struggles of encoding multi-channel audio are over. Many media container formats can, after all, contain an arbitrary number of audio channels.

Nothing is ever so simple. Much like a dedicated audio reel adds cost, multiple audio channels inflate file sizes, media cost, and in the era of playback from optical media, could stress the practical read rate. Besides, constraints of the past have a way of sticking around. Every multichannel audio format to find widespread success in the film industry has done so by maintaining backwards compatibility with simple mono and stereo equipment. That continues to be true today: modern multi-channel digital audio formats are still mostly built as extensions of an existing stereo encoding, not as truly new arbitrary-channel formats.

At the same time, the theatrical sound industry has begun a transition away from channel-centric audio formats and towards a more flexible system that is much further removed from the actual playback equipment.

Another trend has emerged since 1980 as well, which you probably already suspected from the multiple formats included in 35mm prints. Dolby's supremacy in multi-channel audio was never as complete as I made it sound, although they did become (and for some time remained) the most popular surround sound solution. They have always had competition, and that's still true today. Just as 35mm prints came with the audio in multiple formats, current digitally distributed films often do as well.

In Part 2, I'll get to the topic I meant to write about today before I got distracted by history: the landscape of audio formats included in digitally distributed films and common video files today, and some of the ways they interact remarkably poorly with computers. We're going to talk about:

• Dolby Digital/AC-3/AC-4
• DTS
• Dolby Atmos
• MPEG Surround/MPEG-H 3D
• HDMI (ugh)
• And more!

Postscript: Film dweebs will of course wonder where George Lucas is in this story. His work on the Star Wars trilogy lead to the creation of THX, a company that will long be remembered for its distinctive audio identity. The odd thing is that THX was never exactly a technology company, although it was closely involved in sound technology developments of the time. THX was essentially a certification agency: THX theaters installed equipment by others (Altec Lansing, for much of the 20th century), and used any of the popular multi-channel audio formats.

To be a THX-certified theater, certain performance requirements had to be met, regardless of the equipment and format in use. THX certification requirements included architectural design standards for theaters, performance specifications for audio equipment, and a specific crossover configuration designed by Lucasfilm.

In 2002, Lucasfilm spun out THX and it essentially became a rental brand, shuffled into the ownership of gamer headphone manufacturer Razer today. THX certification still pops up in some consumer home theater equipment but is no longer part of the theatrical audio industry.

Read part 2 >

[1] Incidentally, Kubrick did not adapt to Dolby Stereo. Despite his early experience with Dolby noise reduction, all of his films would be released in mono except for 2001 (six-channel audio only in the Cinerama release) and Eyes Wide Shut (edited in Dolby Stereo after Kubrick's death).

Previously on Deep Space Nine, I wrote that "the mid-2000s were an unsettled time in mobile computing." Today, I want to share a little example. Over the last few weeks, for various personal reasons, I have been doing a lot of reading about embedded operating systems and ISAs for embedded computing. Things like the NXP TriMedia (Harvard architecture!) and pSOS+ (ran on TriMedia!). As tends to happen, I kept coming across references to a device that stuck in my memory: the TacNet Tracker. It prominently features on Wikipedia's list of applications for the popular VxWorks real-time operating system.

It's also an interesting case study in the mid-2000s field of mobile computing, especially within academia (or at least the Department of Energy). You see, "mobile computing" used to be treated as a field of study, a subdiscipline within computer science. Mobile devices imposed practical constraints, and they invited more sophisticated models of communication and synchronization than were used with fixed equipment. I took a class on mobile computing in my undergraduate, although it was already feeling dated at the time.

Today, with the ubiquity of smartphones, "mobile computing" is sort of the normal kind. Perhaps future computer science students will be treated to a slightly rusty elective in "immobile computing." The kinds of strange techniques you use when you aren't constrained by battery capacity. Busy loop to blink the cursor!

Sometime around 2004, Sandia National Laboratory's 6452 started work on the TacNet Tracker. The goal: to develop a portable computer device that could be used to exchange real-time information between individuals in a field environment. A presentation states that an original goal of the project was to use COTS (commercial, off-the-shelf) hardware, but it was found to be infeasible. Considering the state of the mobile computing market in 2004, this isn't surprising. It's not necessarily that there weren't mobile devices available; if anything, the opposite. There were companies popping up with various tablets fairly regularly, and then dropping them two years later. You can find any number of Windows XP tablets; but the government needed something that could be supported long-term. That perhaps explains the "Life-cycle limitations" bullet point the presentation wields against COTS options.

The only products with long-term traction were select phones and PDAs like the iPaq and Axim. Even this market collapsed almost immediately with the release of the iPhone, although Sandia engineers wouldn't have known that would come. Still, the capabilities and expandability of these devices were probably too limited for the Tracker's features. There's a reason all those Windows XP tablets existed. They weighed ten pounds, but they were beefy enough to run the data entry applications that were the major application of commercial mobile computing at the time.

The TacNet Tracker, though, was designed to fit in a pocket and to incorporate geospatial features. Armed with a Tracker, you could see the real-time location of other Tracker users on a map. You could even annotate the map, marking points and lines, and share these annotations with others. This is all very mundane today! At the time, though, it was an obvious and yet fairly complex application for a mobile device.

The first question, of course, is of architecture. The Tracker was built around the XScale PXA270 SoC. XScale, remember, was Intel's marketing name for their ARMv5 chips manufactured during the first half of the '00s. ARM was far less common back then, but was already emerging as a leader in power-efficient devices. The PXA270 was an early processor to feature speed-stepping, decreasing its clock speed when under low load to conserve power.

The PXA270 was attached to 64MB of SDRAM and 32MB of flash. It supported more storage on CompactFlash, had an integrated video adapter, and a set of UARTs that, in the Tracker, would support a serial interface, a GPS receiver, and Bluetooth.

A rechargeable Li-Poly pack allowed the Tracker to operate for "about 4 hours," but the presentation promises 8-12 hours in the future. Battery life was a huge challenge in this era. It probably took about as long to charge as it did to discharge, too. There hadn't been much development in high-rate embedded battery chargers yet.

The next challenge was communication. 802.11 WiFi was achieving popularity by this time, but suffered from a difficult and power-intensive association process even more than it does today. Besides, in mobile applications like those the Tracker was intended for, conventional WiFi's requirement for network infrastructure was impractical. Instead, Sandia turned to Motorola. The Tracker used a PCMCIA WMC6300 Pocket PC MEA modem. MEA stands for "Mesh Enabled Architecture," which seems to have been the period term for something Motorola later rebranded as MOTOMESH.

Marketed primarily for municipal network and public safety applications, MOTOMESH is a vaguely 802.11-adjacent proprietary radio protocol that provides broadband mesh routing. One of the most compelling features of MEA and MOTOMESH is its flexibility: MOTOMESH modems will connect to fixed infrastructure nodes under central management, but they can also connect directly to each other, forming ad-hoc networks between adjacent devices. 802.11 itself was conceptually capable of the same, but in practice, the higher-level software to support this kind of use never really emerged. Motorola offered a complete software suite for MOTOMESH, though, and for no less than Windows CE.

Yes, it really enforces the period vibes that the user manual for the WMC6300 modem starts by guiding you through using Microsoft ActiveSync to transfer the software to an HP iPaq. One did not simply put files onto a mobile device at the time; you had to sync them. Microsoft tried to stamp out an ecosystem of proprietary mobile device sync protocols with ActiveSync. Ultimately none of them would really see much use, PDAs were always fairly niche.

Sandia validated performance of the Tracker's MEA modem using an Elektrobit Propsim C2. I saw one of these at auction once (possibly the same one!), and sort of wish I'd bid on it. It's a chunky desktop device with a set of RF ports and the ability to simulate a wide variety of different radio paths between those ports, introducing phenomena like noise, fading, and multipath that will be observed in the real world. The results are impressive: in a simulated hilly environment, Trackers could exchange a 1MB test image in just 13.6 seconds. Remember that next time you are frustrated by LTE; we really take what we have today for granted.

But what of the software? Well, the Tracker ran VxWorks. Actually, that's how I ran into it: it seems that Wind River (developer of VxWorks) published a whitepaper about the Tracker, which made it onto a list of featured applications, which was the source a Wikipedia editor used to flesh out the article. Unfortunately I can't find the original whitepaper, only dead links to it. I'm sure it would have been a fun read.

VxWorks is a real-time operating system mostly used in embedded applications. It supports a variety of architectures, provides a sophisticated process scheduler with options for hard real-time and opportunistic workloads, offers network, peripheral bus, and file system support, and even a POSIX-compliant userspace. It remains very popular for real-time control applications today, although I don't think you'd find many UI-intensive devices like the Tracker running it. A GUI framework is actually a fairly new feature.

The main application for the Tracker was a map, with real-time location and annotation features. It seems that a virtual whiteboard and instant messaging application were also developed. A charmingly cyberpunk Bluetooth wrist-mounted display was pondered, although I don't think it was actually made.

But what was it actually for?

Well, federal R&D laboratories have a tendency to start a project for one application and then try to shop it around to others, so the materials Sandia published present a somewhat mixed message. A conference presentation suggests it could be used to monitor the health of soldiers in-theater (an extremely frequent justification for grants in mobile computing research!), for situational awareness among security or rescue forces, or for remote control of weapons systems.

I think a hint comes, though, from the only concrete US government application I can find documented: in 2008, Sandia delivered the TacNet Tracker system to the DoE Office of Secure Transportation (OST). OST is responsible for the over-road transportation of nuclear weapons and nuclear materials in the United States. Put simply, they operate a fleet of armored trucks and accompanying security escorts. There is a fairly long history, back to at least the '70s, of Sandia developing advanced radio communications systems for use by OST convoys. Many of these radio systems seemed ahead of their time or at least state of the art, but they often failed to gain much traction outside of DoE. Perhaps this relates to DoE culture, perhaps to the extent to which private contractors have captured military purchasing.

Consider, for example, that Sandia developed a fairly sophisticated digital HF system for communication between OST convoys and control centers. It seemed rather more advanced than the military's ALE solution, but a decade or so later OST dropped it and went to using ALE like everyone else (likely for interoperability with the large HF ALE networks operated by the FBI and CBP for domestic security use, although at some point the DoE itself also procured its own ALE network). A whole little branch of digital HF technology that just sort of fizzled out in the nuclear weapons complex. There's a lot of things like that, it's what you get when you put an enormous R&D capability into a particularly insular and secretive part of the executive branch.

Sandia clearly hoped to find other applications for the system. A 2008 Sandia physical security manual for nuclear installations recommends that security forces consider the TacNet Tracker as a situational awareness solution. It was pitched for several military applications. It's a little hard to tell because the name "TacNet" is a little too obvious, but it doesn't seem that the Sandia device ever gained traction in the military.

As it does with many technical developments that don't go very far, Sandia licensed the technology out. A company called Homeland Integrated Security Systems (HISS) bought it, a very typical name for a company that sells licensed government technology. HISS partnered with a UK-based company called Arcom to manufacture the TacNet Tracker as a commercial product, and marketed it to everyone from the military to search and rescue teams.

HISS must have found that the most popular application of the Tracker was asset tracking. It makes sense, the Tracker device itself lacked a display, under the assumption that it would be in a dock or used with an accessory body-worn display. By the late 2000s, HISS had rebranded the TacNet Tracker as the CyberTracker, and re-engineered it around a Motorola iDEN board. I doubt they actually did much engineering on this product, it seems to have been pretty much an off-the-shelf Motorola iDEN radio that HISS just integrated into their tracking platform. It was advertised as a deterrent to automotive theft and a way to track hijacked school buses in real time---the Chowchilla kidnapping was mentioned.

And that's the curve of millennial mobile computing: a cutting-edge R&D project around special-purpose national security requirements, pitched as a general purpose tactical device, licensed to a private partner, turned into yet another commodity anti-theft tracker. Like if LoJack had started out for nuclear weapons. Just a little story about telecommunications history.

Sandia applied for a patent on the Tracker in 2009, so it's probably still in force (ask a patent attorney). HISS went through a couple of restructurings but, as far as I can tell, no longer exists. The same goes for Arcom, a company by the same name that makes cable TV diagnostic equipment seems to be unrelated. Like the OLPC again, all that is left of the Tracker is a surprising number of used units for sale. I'm not sure who ever used the commercial version, but they sure turn up on eBay. I bought one, of course. It'll make a good paperweight.

USB, the Universal Serial Bus, was first released in 1996. It did not achieve widespread adoption until some years later; for most of the '90s RS-232-ish serial and its awkward sibling the parallel port were the norm for external peripheral. It's sort of surprising that USB didn't take off faster, considering the significant advantages it had over conventional serial. Most significantly, USB was self-configuring: when you plugged a device into a host, a negotiation was performed to detect a configuration supported by both ends. No more decoding labels like 9600 8N1 and then trying both flow control modes!

There are some significant architectural differences between USB and conventional serial that come out of autoconfiguration. Serial ports had no real sense of which end was which. Terms like DTE and DCE were sometimes used, but they were a holdover from the far more prescriptive genuine RS-232 standard (which PCs and most peripherals did not follow) and often inconsistently applied by manufacturers. All that really mattered to a serial connection is that one device's TX pin went to the other device's RX pin, and vice versa. The real differentiation between DCE and DTE was the placement of these pins: in principle, a computer would have them one way around, and a peripheral the other way around. This meant that a straight-through cable would result in a crossed-over configuration, as expected.

In practice, plenty of peripherals used the same DE-9 wiring convention as PCs, and sometimes you wanted to connect two PCs to each other. Some peripherals used 8p8c modular jacks, some peripherals used real RS-232 connectors, and some peripherals used monstrosities that could only have emerged from the nightmares of their creators. The TX pin often ended up connected to the TX pin and vice versa. This did not work. The solution, as we so often see in networking, was a special cable that crossed over the TX and RX wires within the cable (or adapter). For historical reasons this was referred to as a null modem cable.

One of the other things that was not well standardized with serial connections was the gender of the connectors. Even when both ends features the PC-standard DE-9, there was some inconsistency over the gender of the connectors on the devices and on the cable. Most people who interact with serial with any regularity probably have a small assortment of "gender changers" and null-modem shims in their junk drawer. Sometimes you can figure out the correct configuration from device manuals (the best manuals provide a full pinout), but often you end up guessing, stringing together adapters until the genders fit and then trying with and without a null modem adapter.

You will notice that we rarely go through this exercise today. For that we can thank USB's very prescriptive standards for connectors on devices and cables. The USB standard specifies three basic connectors, A, B, and C. There are variants of some connectors, mostly for size (mini-B, micro-B, even a less commonly used mini-A and micro-A). For the moment, we will ignore C, which came along later and massively complicated the situation. Until 2014, there was only A and B. Hosts had A, and devices had B.

Yes, USB fundamentally employs a host-device architecture. When you connect two things with USB, one is the host, and the other is the device. This differentiation is important, not just for the cable, but for the protocol itself. USB prior to 3, for example, does not feature interrupts. The host must poll the device for new data. The host also has responsibility for enumeration of devices to facilitate autoconfiguration, and for flow control throughout a tree of USB devices.

This architecture makes perfect sense for USB's original 1990s use-case of connecting peripherals (like mice) to hosts (like PCs). In fact, it worked so well that once USB1.1 addressed some key limitations it became completely ubiquitous. Microsoft used the term "legacy-free PC" to identify a new generation of PCs at the very end of the '90s and early '00s. While there were multiple criteria for the designation, the most visible to users was the elimination of multiple traditional ports (like the game port! remember those!) in favor of USB.

Times change, and so do interconnects. The consumer electronics industry made leaps and bounds during the '00s and "peripheral" devices became increasingly sophisticated. The introduction of portables running sophisticated operating systems pushed the host-device model to a breaking point. It is, of course, tempting to talk about this revolution in the context of the iPhone. I never had an iPhone though, so the history of the iDevice doesn't have quite the romance to me that it has to so many in this space [1]. Instead, let's talk about Nokia. If there is a Windows XP to Apple's iPhone, it's probably Nokia. They tried so hard, and got so far, but [...].

The Nokia 770 Internet Tablet was not by any means the first tablet computer, but it was definitely a notable early example. Introduced in 2005, it premiered the Linux-based Maemo operating system beloved by Nokia fans until iOS and Android killed it off in the 2010s. The N770 was one of the first devices to fall into a new niche: with a 4" touchscreen and OMAP/ARM SoC, it wasn't exactly a "computer" in the consumer sense. It was more like a peripheral, something that you would connect to your computer in order to load it up with your favorite MP3s. But it also ran a complete general-purpose operating system. The software was perfectly capable of using peripherals itself, and MP3s were big when you were storing them on MMC. Shouldn't you be able to connect your N770 to a USB storage device and nominate even more MP3s as favorites?

Obviously Linux had mainline USB mass storage support in 2005, and by extension Maemo did. The problem was USB itself. The most common use case for USB on the N770 was as a peripheral, and so it featured a type-B device connector. It was not permitted to act as a host. In fact, every PDA/tablet/smartphone type device with sophisticated enough software to support USB peripherals would encounter the exact same problem. Fortunately, it was addressed by a supplement to the USB 2.0 specification released in 2001.

The N770 did not follow the supplement. That makes it fun to talk about, both because it is weird and because it is an illustrative example of the problems that need to be solved.

The N770 featured an unusual USB transceiver on its SoC, seemingly unique to Nokia and called "Tahvo." The Tahvo controller exposed an interface (via sysfs in the Linux driver) that allowed the system to toggle it between device mode (its normal configuration) and host mode. This worked well enough with Maemo's user interface, but host mode had a major limitation. The N770 wouldn't provide power on the USB port; it didn't have the necessary electrical components. Instead, a special adapter cable was needed to provide 5v power from an alternate source.

So there are several challenges for a USB device to operate as host or device:

• The USB controller needs a way to determine if it should behave in host or device mode. Ideally, the user shouldn't have to think about this.
• The USB controller needs to be able to supply power when in host mode, and in most practical situations also needs to accept power (e.g. for charging) when in device mode.

Note that "special cable" involved in host mode for the N770. You might think this was the ugliest part of the situation. You're not wrong, but it's also not really the hack. For many years to follow, the proper solution to this problem would also involve a special cable.

As I mentioned, since 2001 there has been a supplement USB specification called USB On-The-Go, commonly referred to as USB OTG, perhaps because On-The-Go is an insufferably early '00s name. It reminds me of, okay, here goes a full-on anecdote.

Anecdote

I attended an alternative middle school in Portland that is today known as the Sunnyside Environmental School. I could tell any number of stories about the bizarre goings-on at this school that you would scarcely believe, but it also had its merits. One of them, which I think actually came from the broader school district, was a program in which eighth graders were encouraged to "job shadow" someone in a profession they were interested in pursuing. By good fortune, a friend's father was an electrical engineer employed at Intel's Jones Farm campus, and agreed to be my host. I had actually been to Jones Farm a number of times on account of various extracurricular programs (in that era, essentially every STEM program in the Pacific Northwest operated on the largess of either Intel or Boeing, if not both). This was different, though: this guy had a row of engraved brass patent awards lining his cubicle wall and showed me through labs where technicians tinkered with prototype hardware. Foreshadowing a concerning later trend in my career, though, the part that stuck with me most was the meetings. I attended meetings, including one where this engineering team was reporting to leadership on the status of a few of their projects.

I am no doubt primed to make this comparison by the mediocre movie I watched last night, but I have to describe the experience as Wonka-esque. These EEs demonstrated a series of magical hardware prototypes to some partners from another company. Each was more impressive than the last. It felt like I was seeing the future in the making.

My host demonstrated his pet project, a bar that contained an array of microphones and used DSP methods to compare the audio from each and directionalize the source of sounds. This could be used for a sophisticated form of noise canceling in which sound coming from an off-axis direction could be subtracted, leaving only the voice of the speaker. If this sounds sort of unremarkable, that is perhaps a reflection of its success, as the same basic concept is now implemented in just about every laptop on the market. Back then, when the N770 was a new release, it was challenging to make work and my host explained that the software behind it usually crashed before he finished the demo, and sometimes it turned the output into a high pitched whine and he hadn't quite figured out why yet. I suppose that meeting was lucky.

But that's an aside. A long presentation, and then debate skeptical execs, revolved around a new generation of ultramobile devices that Intel envisioned. One, which I got to handle a prototype of, would eventually become the Intel Core Medical Tablet. It featured chunky, colorful design that is clearly of the same vintage as the OLPC. It was durable enough to stand on, which a lab technician demonstrated with delight (my host, I suspect tired of this feat, picked up some sort of lab interface and dryly remarked that he could probably stand on it too). The Core Medical Tablet shared another trait with the OLPC: the kind of failure that leaves no impact on the world but a big footprint at recyclers. Years later, as an intern at Free Geek, I would come across at least a dozen.

Another facet of this program, though, was the Mobile Metro. The Metro was a new category of subnotebook, not just small but thin. A period article compares its 18mm profile to the somewhat thinner Motorola Razr, another product with an outsize representation in the Free Geek Thrift Store. Intel staff were confident that it would appeal to a new mobile workforce, road warriors working from cars and coffee shops. The Mobile Metro featured SideShow, a small e-ink display (in fact, I believe, a full Windows Mobile system) on the outside of a case that could show notifications and media controls.

The Mobile Metro was developed around the same time as the Classmate PC, but seems to have been even less successful. It was still in the conceptual stages when I heard of it. It was announced, to great fanfare, in 2007. I don't think it ever went into production. It had WiMax. It had inductive charging. It only had one USB port. It was, in retrospect, prescient in many ways both good and bad.

The point of this anecdote, besides digging up middle school memories while attempting to keep others well suppressed, is that the mid-2000s were an unsettled time in mobile computing. The technology was starting to enable practical compact devices, but manufacturers weren't really sure how people would use them. Some innovations were hits (thin form factors). Some were absolute misses (SideShow). Some we got stuck with (not enough USB ports).

End of anecdote

As far as I can tell, USB OTG wasn't common on devices until it started to appear on Android smartphones in the early 2010s. Android gained OTG support in 3.1 (codenamed Honeycomb, 2011), and it quickly appeared in higher-end devices. Now OTG support seems nearly universal for Android devices; I'm sure there are lower-end products where it doesn't work but I haven't yet encountered one. Android OTG support is even admirably complete. If you have an Android phone, amuse yourself sometime by plugging a hub into it, and then a keyboard and mouse. Android support for desktop input peripherals is actually very good and operating mobile apps with an MX Pro mouse is an entertaining and somewhat surreal experience. On the second smartphone I owned, I hazily think a Samsung in 2012-2013, I used to take notes with a USB keyboard.

iOS doesn't seem to have sprouted user-exposed OTG support until the iPhone 12, although it seems like earlier versions probably had hardware support that wasn't exposed by the OS. I could be wrong about this; I can't find a straightforward answer in Apple documentation. The Apple Community Forums seem to be... I'll just say "below average." iPads seem to have gotten OTG support a lot earlier than the iPhone despite using the same connector, making the situation rather confusing. This comports with my general understanding of iOS, though, from working with bluetooth devices: Apple is very conservative about hardware peripheral support in iOS, and so it's typical for iOS to be well behind Android in this regard for purely software reasons. Ask me about how this has impacted the Point of Sale market. It's not positive.

But how does OTG work? Remember, USB specifies that hosts must have an A connector, and devices a B connector. Most smartphones, besides Apple products and before USB-C, sported a micro-B connector as expected. How OTG?

The OTG specification decouples, to some extent, the roles of A/B connector, power supply, and host/device role. A device with USB OTG support should feature a type AB socket that accommodates either an A or a B plug. Type AB is only defined for the mini and micro sizes, typically used on portable devices. The A or B connectors are differentiated not only by the shape of their shells (preventing a type-A plug being inserted into a B-only socket), but also electrically. The observant among you may have noticed that mini and micro B sockets and plugs feature five pins, while USB2.0 only uses four. This is the purpose of the fifth pin: differentiation of type A and B plugs.

In a mini or micro type B plug, the fifth pin is floating (disconnected). In a mini or micro type A plug, it is connected to the ground pin. When you insert a plug into a type AB socket, the controller checks for connectivity between the fifth pin (called the ID pin) and the ground. If connectivity is present, the controller knows that it must act as an OTG A-device---it is on the "A" end of the connection. If there is no continuity, the more common case, the controller will act as an OTG B-device, a plain old USB device [2].

The OTG A-device is always responsible for supplying 5v power (see exception in [2]). By default, the A-device also acts as the host. This provides a basically complete solution for the most common OTG use-case: connecting a peripheral like a flash drive to your phone. The connector you plug into your phone identifies itself as an A connector via the ID pin, and your phone thus knows that it must supply power and act as host. The flash drive doesn't need to know anything about this, it has a B connection and acts as a device as usual. This simple case only became confusing when you consider a few flash drives sold specifically for use with phones that had a micro-A connector right on them. These were weird and I don't like them.

In the more common situation, though, you would use a dongle: a special cable. A typical OTG cable, which were actually included in the package with enough Android phones of the era that I have a couple in a drawer without having ever purchased one, provides a micro-A connector on one end and a full-size A socket on the other. With this adapter, you can plug any USB device into your phone with a standard USB cable.

Here's an odd case, though. What if you plug two OTG devices into each other? USB has always had this sort of odd edge-case. Some of you may remember "USB link cables," which don't really have a technical name but tend to get called Laplink cables after a popular vendor. Best Buy and Circuit City used to be lousy with these things, mostly marketed to people who had bought a new computer and wanted to transfer their files. A special USB cable had two A connectors, which might create the appearance that it connected two hosts, but in fact the cable (usually a chunky bit in the middle) acted as two devices to connect to two different hosts. The details of how these actually worked varied from product to product, but the short version is "it was proprietary." Most of them didn't work unless you found the software that came with them, but there are some pseudo-standard controllers supported out of the box by Windows or Linux. I would strongly suggest that you protect your mental state by not trying to use one.

OTG set out to address this problem more completely. First, it's important to understand that this in no way poses an exception to the rule that a USB connection has an A end and a B end. A USB cable you use to connect two phones together might, at first glance, appear to be B-B. But, if you inspect closer, you will find that one end is mini or micro A, and the other is mini or micro B. You may have to look close, the micro connectors in particular have a similar shell!

If you are anything like me, you are most likely to have encountered such a cable in the box with a TI-84+. These calculators had a type AB connector and came with a micro A->B cable to link two units. You might think, by extension, that the TI-84+ used USB OTG. The answer is kind of! The USB implementation on the TI-84+ and TI-84+SE was very weird, and the OS didn't support anything other than TIConnect. Eventually the TI-84+CE introduced a much more standard USB controller, although I think support for any OTG peripheral still has to be hacked on to the OS. TI has always been at the forefront of calculator networking, and it has always been very weird and rarely used.

This solves part of the problem: it is clear, when you connect two phones, which should supply power and which should handle enumeration. The A-device is, by default, in charge. There are problems where this interacts with common USB devices types, though. One of the most common uses of USB with phones is mass storage (and its evil twin MTP). USB mass storage has a very strong sense of host and device at a logical level; the host can browse the devices files. When connecting two smartphones, though, you might want to browse from either end. Another common problem case here is that of the printer, or at least it would be if printer USB host support was ever usable. If you plug a printer into a phone, you might want to browse the phone as mass storage on the printer. Or you might want to use conventional USB printing to print a document from the phone's interface. In fact you almost certainly want to do the latter, because even with Android's extremely half-assed print spooler it's probably a lot more usable than the file browser your printer vendor managed to offer on its 2" resistive touchscreen.

OTG adds Host Negotiation Protocol, or HNP, to help in this situation. HNP allows the devices on a USB OTG connection to swap roles. While the A-device will always be the host when first connected, HNP can reverse the logical roles on demand.

This all sounds great, so where does it fall apart? Well, the usual places. Android devices often went a little off the script with their OTG implementations. First, the specification did not require devices to be capable of powering the bus, and phones couldn't. Fortunately that seems to have been a pretty short lived problem, only common in the first couple of generations of OTG devices. This wasn't the only limitation of OTG implementations; I don't have a good sense of scale but I've seen multiple reports that many OTG devices in the wild didn't actually support HNP, they just determined a role when connected based on the ID pin and could not change after that point.

Finally, and more insidiously, the whole thing about OTG devices having an AB connector didn't go over as well as intended. We actually must admire TI for their rare dedication to standards compliance. A lot of Android phones with OTG support had a micro-B connector only, and as a result a lot of OTG adapters use a micro-B connector.

There's a reason this was common; since A and B plugs are electrically differentiable regardless of the shape of the shell, the shell shape arguably doesn't matter. You could be a heavy OTG user with such a noncompliant phone and adapter and never notice. The problem only emerges when you get a (rare) standards-compliant OTG adapter or, probably more common, OTG A-B cable. Despite being electrically compatible, the connector won't fit into your phone. Of course this behavior feeds itself; as soon as devices with an improper B port were common, manufacturers of cables were greatly discouraged from using the correct A connector.

The downside, conceptually, is that you could plug an OTG A connector (with a B-shaped shell) into a device with no OTG support. In theory this could cause problems, in practice the problems don't seem to have been common since both devices would think they were B devices and (if standards compliant) not provide power. Essentially these improper OTG adapters create a B-B cable. It's a similar problem to an A-A cable but, in practice, less severe. Like an extension cord with two female ends. Home Depot might even help you make one of those.

While trying to figure out which iPhones had OTG support, I ran across an Apple Community thread where someone helpfully replied "I haven't heard of OTG in over a decade." Well, it's not a very helpful reply, but it's not exactly wrong either. No doubt the dearth of information on iOS OTG is in part because no one ever really cared. Much like the HDMI-over-USB support that a generation of Android phones included, OTG was an obscure feature. I'm not sure I have ever, even once, seen a human being other than myself make use of OTG.

Besides, it was completely buried by USB-C.

The thing is that OTG is not gone at all, in fact, it's probably more popular than ever before. There seems to be some confusion about how OTG has evolved with USB specifications. I came across more than one article saying that USB 3.1 Dual Role replaced OTG. This assertion is... confusing. It's not incorrect, but there's a good chance of it leading you int he wrong direction.

Much of the confusion comes from the fact that Dual-Role doesn't mean anything that specific. The term Dual-Role and various resulting acronyms like DRD and DRP have been applied to multiple concepts over the life of USB. Some vendors say "static dual role" to refer to devices that can be configured as either host or device (like the N770). Some vendors use dual role to identify chipsets that detect role based on the ID pin but are not actually capable of OTG protocols like HNP. Some articles use dual role to identify chipsets with OTG support. Subjectively, I think the intent of the changes in USB 3.1 were mostly to formally adopt the "dual role" term that was already the norm in informal use---and hopefully standardize the meaning.

For USB-C connectors, it's more complicated. USB-C cables are symmetric, they do not identify a host or device end in any way. Instead, the USB-C ports use resistance values to indicate their type. When either end indicates that it is only capable of the device role, the situation is simple, behaving basically the same way that OTG did: the host detects that the other end is a device and behaves as the host.

When both ends support the host role, things work differently: the Dual Role feature of USB-C comes into play. The actual implementation is reasonably simple; a dual-role USB-C controller will attempt to set up a connection both ways and go with whichever succeeds. There are some minor complications on top of this, for example, the controller can be configured with a "preference" for host or device role. This means that when you plug your phone into your computer via USB-C, the computer will assume the host role, because although it's capable of either the phone is configured with a preference for the device role. That matches consumer expectations. When both devices are capable of dual roles and neither specifies a preference, the outcome is random. This scenario is interesting but not all that common in practice.

The detection of host or device role by USB-C is based on the CC pins, basically a more flexible version of OTG's ID pin. There's another important difference between the behavior of USB-C and A/B: USB-C interfaces provide no power until they detect, via the CC pins, that the other device expects it. This is an important ingredient to mitigate the problem with A-A cables, that both devices will attempt to power the same bus.

The USB-C approach of using CC pins and having dual role controllers attempt one or the other at their preference is, for the most part, a much more elegant approach. There are a couple of oddities. First, in practice cables from C to A or B connectors are extremely common. These cables must provide the appropriate values on the CC pins to allow the USB-C controller to correctly determine its role, both for data and power delivery.

Second, what about role reversal? For type A and B connectors, this is achieved via HNP, but HNP is not supported on USB-C. Application notes from several USB controller vendors explain that, oddly enough, the only way to perform role reversal with USB-C is to implement USB Power Delivery (PD) and use the PD negotiation protocol to change the source of power. In other words, while OTG allows reversing host and device roles independently of the bus power source, USB-C does not. The end supplying power is always the host end. This apparent limitation probably isn't that big of a deal, considering that the role reversal feature of OTG was reportedly seldom implemented.

That's a bit of a look into what happens when you plug two USB hosts into each other. Are you confused? Yeah, I'm a little confused too. The details vary, and a lot more based on the capabilities of the individual devices rather than the USB version in use. This has been the malaise of USB for a solid decade now, at least: the specification has become so expansive, with so many non-mandatory features, that it's a crapshoot what capabilities any given USB port actually has. The fact that USB-C supports a bevy of alternate modes like Thunderbolt and HDMI only adds further confusion.

I sort of miss when the problem was just inappropriate micro-B connectors. Nonetheless, USB-C dual role support seems ubiquitous in modern smartphones, and that's the only place any of this ever really mattered. Most embedded devices still seem to prefer to just provide two USB ports: a host port and a device port. And no one ever uses the USB host support on their printer. It's absurd, no one ever would. Have you seen what HP thinks is a decent file browser? Good lord.

[1] My first smartphone was the HTC Thunderbolt. No one, not even me, will speak of that thing with nostalgia. It was pretty cool owning one of the first LTE devices on the market, though. There was no contention at all in areas with LTE service and I was getting 75+Mbps mobile tethering in 2011. Then everyone else had LTE too and the good times ended.

[2] There are actually several additional states defined by fixed resistances that tell the controller that it is the A-device but power will be supplied by the bus. These states were intended for Y-cables that allowed you to charge your phone from an external charger while using OTG. In this case neither device supplies power, the external charger does. The details of how this works are quite straightforward but will be confusing to keep adding as an exception, so I'm going to pretend the whole feature doesn't exist.

Programming note/shameless plug: I am finally on Mastodon.

The history of the telephone industry is a bit of an odd one. For the greatest part of the 20th century, telephony in the United States was largely a monopoly of AT&T and its many affiliates. This wasn't always the case, though. AT&T held patents on their telephone implementation, but Bell's invention was not the only way to construct a practical telephone. During the late 19th century, telephone companies proliferated, most using variations on the design they felt would fall outside of Ma Bell's patent portfolio. AT&T was aggressive in challenging these operations but not always successful. During this period, it was not at all unusual for a city to have multiple competing telephone companies that were not interconnected.

Shortly after the turn of the 20th century, AT&T moved more decisively towards monopoly. Theodore Newton Vail, president of AT&T during this period, adopted the term "Universal Service" to describe the targeted monopoly state: there would be one universal telephone system. One operated under the policies and, by implication, the ownership of AT&T. AT&T's path to monopoly involved many political and business maneuvers, the details of which have filled more than a few dissertations in history and economics. By the 1920s the deal was done, there would be virtually no (and in a legal sense literally no) long-distance telephone infrastructure in the United States outside of The Bell System.

But what of the era's many telephone entrepreneurs? For several American telephone companies struggling to stand up to AT&T, the best opportunities were overseas. A number of countries, especially elsewhere in the Americas, had telephone systems built by AT&T's domestic competitors. Perhaps the most neatly named was ITT, the International Telephone and Telegraph company. ITT was formed from the combination of Puerto Rican and Cuban telephone companies, and through a series of acquisitions expanded into Europe.

Telefónica, for example, is a descendent of an early ITT acquisition. Other European acquisitions led to wartime complications, like the C. Lorenz company, which under ITT ownership functioned as a defense contractor to the Nazis during WWII. Domestically, ITT also expanded into a number of businesses outside of the monopolized telephone industry, including telegraphy and international cables.

ITT had been bolstered as well by an effect of AT&T's first round of antitrust cases during the 1910s and 1920s. As part of one of several settlements, AT&T agreed to divest several overseas operations to focus instead on the domestic market. They found a perfect buyer: ITT, a company which already seemed like a sibling of AT&T and through acquisitions came to function as one.

ITT grew rapidly during the mid-century, and in the pattern of many industrial conglomerates of the time ITT diversified. Brands like Sheraton Hotels and Avis Rent-a-Car joined the ITT portfolio (incidentally, Avis would be spun off, conglomerated with others, and then purchased by previous CAB subject Beatrice Foods). ITT was a multi-billion-dollar American giant.

Elsewhere in the early technology industry, salesman Howard W. Sams worked for the P. R. Mallory Company in Indianapolis during the 1930s and 1940s. Mallory made batteries and electronic components, especially for the expanding radio industry, and as Sams sold radio components to Mallory customers he saw a common problem and a sales opportunity: radio technicians often needed replacement components, but had a hard time identifying them and finding a manufacturer. Under the auspices of the Mallory company Sams produced and published several books on radio repair and electronic components, but Mallory didn't see the potential that Sams did in these technical manuals.

Sams, driven by the same electronics industry fervor as so many telephone entrepreneurs, struck out on his own. Incorporated in 1946, the Howard W. Sams Company found quick success with its Photofact series. Sort of the radio equivalent of Haynes and Chilton in the auto industry, Photofact provided schematics, parts lists, and repair instructions for popular radio receivers. They were often found on the shelves of both technicians and hobbyists, and propelled the Sams Company to million-dollar revenues by the early 1950s.

Sams would expand along with the electronics industry, publishing manuals on all types of consumer electronics and, by the 1960s, books on the use of computers. Sams, as a technical press, eventually made its way into the ownership of Pearson. Through Pearson's InformIT, the Sams Teach Yourself series remains in bookstores today. I am not quite sure, but I think one of the first technical books I ever picked up was an earlier edition of Sams HTML in 24 Hours.

The 1960s were an ambitious era, and Sams was not content with just books. Sams had taught thousands electronics technicians through their books. Many radio technicians had demonstrated their qualifications and kept up to date by maintaining a membership in the Howard Sams Radio Institute, a sort of correspondence program. It was a natural extension to teach electronics skills in person. In 1963, Sams opened the Sams Technical Institute in Indianapolis. Shortly after, they purchased the Acme Institute of Technology (Dayton, Ohio) and the charmingly named Teletronic Technical Institute (Evansville, Indiana), rebranding both as Sams campuses.

In 1965, the Sams Technical Institute had 2,300 students across five locations. Sams added the Bramwell Business College to its training division, signaling a move into the broader world of higher education. It was a fast growing business; it must have looked like a great opportunity to a telephone company looking for more ways to diversify. In 1968, ITT purchased the entire training division from Sams, renaming it ITT Educational Services [1].

ITT approached education with the same zeal it had overseas telephone service. ITT Educational Services spent the late '60s and early '70s on a shopping spree, adding campus after campus to the ITT system. Two newly constructed campuses expanded ITT's business programs, and during the '70s ITT introduced formal curriculum standardization programs and a bureaucratic structure to support its many locations. Along with expansion came a punchier name: the ITT Technical Institute.

"Tri-State Businessmen Look to ITT Business Institute, Inc. for Graduates," reads one corner of a 1970 full-page newspaper ad. "ITT adds motorcycle repair course to program," 1973. "THE ELECTRONICS AGE IS HERE. If your eyes are on the future, ITT Technical institute can prepare you for a HIGH PAYING, EXCITING career in... ELECTRONICS," 1971. ITT Tech has always known the value of advertising, and ran everything from full-page "advertorials" to succinct classified ads throughout their growing region.

During this period, ITT Tech clearly operated as a vocational school rather than a higher education institution. Many of its programs ran as short as two months, and they were consistently advertised as direct preparation for a career. These sorts of job-oriented programs were very attractive to veterans returning from Vietnam, and ITT widely advertised to veterans on the basis of its approval (clearly by 1972 based on newspaper advertisements, although some sources say 1974) for payment under the GI Bill. Around the same time ITT Tech was approved for the fairly new federal student loan program. Many of ITT's students attended on government money, with or without the expectation of repayment.

ITT Tech flourished. By the mid-'70s the locations were difficult to count, and ITT had over 1,000 students in several states. ITT Tech was the "coding boot camp" of its day, advertising computer programming courses that were sure to lead to employment in just about six months. Like the coding boot camps of our day, these claims were suspect.

In 1975, ITT Tech was the subject of investigations in at least two states. In Indiana, three students complained to the Evansville municipal government after ITT recruiters promised them financial aid and federally subsidized employment during their program. ITT and federal work study, they were told, would take care of all their living expenses. Instead, they ended up living in a YWCA off of food stamps. The Indiana board overseeing private schools allowed ITT to keep its accreditation only after ITT promised to rework its entire recruiting policy---and pointed out that the recruiters involved had left the company. ITT refunded the tuition of a dozen students who joined the complaint, which no doubt helped their case with the state.

Meanwhile, in Massachusetts, the Boston Globe ran a ten-part investigative series on the growing for-profit vocational education industry. ITT Tech, they alleged, promised recruits to its medical assistant program guaranteed post-graduation employment. The Globe claimed that almost no students of the program successfully found jobs, and the Massachusetts Attorney General agreed. In fact, the AG found, the program's placement rate didn't quite reach 5%. For a settlement, ITT Tech agreed to change its recruiting practices and refund nearly half a million dollars in tuition and fees.

ITT continued to expand at a brisk pace, adding more than a dozen locations in the early '80s and beginning to offer associates degrees. Newspapers from Florida to California ran ads exhorting readers to "Make the right connections! Call ITT Technical Institute." As the 1990s dawned, ITT Tech enjoyed the same energy as the computer industry, and aspired to the same scale. In 1992, ITT Tech announced their "Vision 2000" master plan, calling for bachelor's programs, 80 locations, and 45,000 students for beginning of the new millennium. ITT Tech was the largest provider of vocational training the country.

In 1993, ITT Tech was one of few schools accepted into the first year of the Direct Student Loan program. The availability of these new loans gave enrollment another boost, as ITT Tech reached 54 locations and 20,000 students. In 1994, ITT Tech started to gain independence from its former parent: an IPO sold 17% ownership to the open market, with ITT retaining the remaining 83%. The next year, ITT itself went through a reorganization and split, with its majority share of ITT Tech landing in the new ITT Corporation.

As was the case with so many diversified conglomerates of the '90s (see Beatrice Foods again), ITT's reorganization was a bad portent. ITT Hartford, the spun-out financial services division, survives today as The Hartford. ITT Industries, the spun-out defense contracting division, survives today as well, confusingly renamed to ITT Corporation. But the third part of the 1995 breakup, the ITT Corporation itself, merged with Starwood Hotels and Resorts. The real estate and hospitality side-business of a telephone and telegraph company saw the end of its parent.

Starwood had little interest in vocational education, and over the remainder of the '90s sold off its entire share of ITT Tech. Divestment was a good idea: the end of the '90s hit hard for ITT Tech. Besides the general decline of the tech industry as the dot com bubble burst, ITT Tech's suspect recruiting practices were back. This time, they had attracted federal attention.

In 1999, two ITT Tech employees filed a federal whistleblower suit alleging that ITT Tech trained recruiters to use high-pressure sales tactics and outright deception to obtain students eligible for federal aid. Recruiters were paid a commission for each student they brought in, and ITT Tech obtained 70% of its revenue from federal aid programs. A federal investigation moved slowly, apparently protracted by the Department of Education's nervous approach following the criticism it received for shutting down similar operation Computer Learning Centers. In 2004, federal agents raided ITT Tech campuses across ten states, collecting records on recruitment and federal funding.

During the early 2000s ITT Tech students defaulted on $400 million in federal student loans. The result, a large portion of ITT Tech revenue coming from defaulted federal loans, attracted ongoing attention. ITT Tech was deft in its legal defense, though, and through a series of legal victories and, more often, settlements, ITT Tech stayed in business.

ITT Tech aggressively advertised throughout its history. In the late '90s and early '00s, ITT Tech's constant television spots filled a corner of my brain. "How Much You Know Measures How Far You Can Go," a TV spot proclaims, before ITT's distinctive block letter logo faded on screen in metallic silver. By the year 2000, International Telephone and Telegraph, or rather its scattered remains, no longer had any relationship with ITT Tech. Starwood agreed to license the name and logo to the independent public ITT Technical Institutes corporation, though, and with the decline of ITT's original business the ITT name and logo became associated far more with the for-profit college than the electronics manufacturer.

For-profit universities attracted a lot of press in the '00s---the wrong kind of press. ITT Tech was far from unique in suspicious advertising and recruiting, high tuition rates, and frequent defaults on the federal loans that covered that tuition. For-profit education, it seemed, was more of a scam on the taxpayer dollar than way to secure a promising new career. Publicly traded colleges like DeVry and the University of Phoenix had repeated scandals over their use, or abuse, of federal aid, and a 2004 criminal investigation into ITT Tech for fraud on federal student aid made its future murky.

ITT Tech was a survivor. The criminal case fell apart, the whistleblower lawsuit lead to nothing, and ITT Tech continued to grow. In 2009, ITT Tech acquired the formerly nonprofit Daniel Webster University, part of a wave of for-profit conversions of small colleges. ITT Tech explained the purchase as a way to expand their aeronautics offerings, but observers suspected other motives, ones that had more to do with the perceived legitimacy of what was once a nonprofit, regionally accredited institution. Today, regional accreditors re-investigate institutions that are purchased. There was a series of suspect expansions of small colleges to encompass large for-profit organizations during the '00s that lead to the tightening of these rules.

ITT Tech, numerically, achieved an incredible high. In 2014, ITT Tech reported a total cost of attendance of up to $85,000. I didn't spend that much on my BS and MS combined. Of course, I attended college in impoverished New Mexico, but we can make a comparison locally. ITT Tech operated here as well, and curiously, New Mexico tuition is specially listed in an ITT Tech cost estimate report because it is higher. At its location in Albuquerque's Journal Center office development, ITT Tech charged more than $51,000 in tuition alone for an Associate's in Criminal Justice. The same program at Central New Mexico Community College would have cost under $4,000 over the two years [2].

That isn't the most remarkable, though. A Bachelor's in Criminal Justice would run over $100,000---more than the cost of a JD at UNM School of Law, for an out-of-state student, today.

In 2014, more than 80% of ITT Tech's revenue came from federal student aid. Their loan default rate was the highest of even for-profit programs. With their extreme tuition costs and notoriously poor job placement rates, ITT Tech increasingly had the appearance of an outright fraud.

Death came swiftly for ITT Tech. In 2016, they were a giant with more than 130 campuses and 40,000 students. The Consumer Financial Protection Bureau sued. State Attorneys General followed, with New Mexico's Hector Balderas one of the first two. The killing blow, though, came from the Department of Education, which revoked ITT Tech's eligibility for federal student aid. Weeks later, ITT Tech stopped accepting applications. The next month, they filed for bankruptcy, chapter 7, liquidation.

Over the following years, the ITT Tech scandal would continue to echo. After a series of lawsuits, the Department of Education agreed to forgive the federal debt of ITT Tech attendees, although a decision by Betsy DeVos to end the ITT Tech forgiveness program produced a new round of lawsuits over the matter in 2018. Private lenders faced similar lawsuits, and made similar settlements. Between federal and private lenders, I estimate almost $4.5 billion in loans to pay ITT Tech tuition were written off.

The Department of Education decision to end federal aid to ITT Tech was based, in part, on ITT Tech's fraying relationship with its accreditor. The Accrediting Council for Independent Colleges and Schools (ACICS), a favorite of for-profit colleges, had its own problems. That same summer in 2016, the Department of Education ended federal recognition of ACICS. ACICS accreditation reviews had been cursory, and it routinely continued to accredit colleges despite their failure to meet even ACIC's lax standards. ITT Tech was not the only large ACIC-accredited institution to collapse in scandal.

Two years later, Betsy DeVos reinstated ACICS to federal recognition. Only 85 institutions still relied on ACICS, such august names as the Professional Golfers Career College and certain campuses of the Art Institutes that were suspect even by the norms of the Art Institutes (the Art Institutes folded just a few months ago following a similar federal loan fraud scandal). ACICS lost federal recognition again in 2022. Only time will tell what the next presidential administration holds for the for-profit college industry.

ITT endured a long fall from grace. A leading electronics manufacturer in 1929, a diversified conglomerate in 1960, scandals through the 1970s. You might say that ITT is distinctly American in all the best and worst ways. They grew to billions in revenue through an aggressive program of acquisitions. They were implicated in the CIA coup in Chile. They made telephones and radios and radars and all the things that formed the backbone of the mid-century American electronics industry.

The modern ITT Corporation, descended from spinoff company ITT Industries, continues on as an industrial automation company. They have abandoned the former ITT logo, distancing themselves from their origin. The former defense division became Exelis, later part of Harris, now part of L3, doomed to slowly sink into the monopolized, lethargic American defense industry. German tool and appliance company Kärcher apparently holds a license to the former ITT logo, although I struggle to find any use of it.

To most Americans, ITT is ITT Tech, a so-called college that was actually a scam, an infamous scandal, a sink of billions of dollars in federal money. Dozens of telephone companies around the world, tracing their history back to ITT, are probably better off distancing themselves from what was once a promising international telephone operator, a meaningful technical competitor to Western Electric. The conglomeration of the second half of the 20th century put companies together and then tore them apart; they seldom made it out in as good of condition as they went in. ITT went through the same cycle as so many other large American corporations. They went into hotels, car rentals, then into colleges. They left thousands of students in the lurch on the way out. When ITT Tech went bankrupt, everyone else had already started the semester. They weren't accepting applicants. They wouldn't accept transfer credit from ITT anyway; ITT's accreditation was suspect.

"What you don't know can hurt you," a 1990s ITT Tech advertisement declares. In Reddit threads, ITT Tech alums debate if they're better off telling prospective employers they never went to college at all.

[1] Sources actually vary on when ITT purchased Sams Training Institute, with some 1970s newspaper articles putting it as early as 1966, but 1968 is the year that ITT's involvement in Sams was advertised in the papers. Further confusing things, the former Sams locations continued to operate under the Sams Technical Institute name until around 1970, with verbiage like "part of ITT Educational Services" inconsistently appearing. ITT may have been weighing the value of its brand recognition against Sams but apparently made a solid decision during 1970, after which ads virtually always use the ITT name and logo above any other.

[2] Today, undergraduate education across all of New Mexico's public universities and community colleges is free for state residents. Unfortunately 2014 was not such an enlightened time. I must take every opportunity to brag about this remarkable and unusual achievement in our state politics.

The term "VHF omnidirectional range" can at first be confusing, because it includes "range"---a measurement that the technology does not provide. The answer to this conundrum is, as is so often the case, history. The "range" refers not to the radio equipment but to the space around it, the area in which the signal can be received. VOR is an inherently spatial technology; the signal is useless except as it relates to the physical world around it.

This use of the word "range" is about as old as instrument flying, dating back to the first radionavigation devices in the 1930s. We still use it today, in the somewhat abstract sense of an acronym that is rarely expanded: VOR.

This is Truth or Consequences VOR. Or, perhaps more accurately, the transmitter that defines the center of the Truth or Consequences VOR, which extends perhaps two hundred miles around this point. The range can be observed only by instruments, but it's there, a phase shift that varies like terrain.

The basic concept of VOR is reasonably simple: a signal is transmitted with two components, a 30Hz tone in amplitude modulation and a 30Hz in frequency modulation. The two tones are out of phase, by an amount that is determined by your position in the range, and more specifically by the radial from the VOR transmitter to your position. This apparent feat of magic, a radio signal that is different in different locations, is often described as "space modulation."

The first VOR transmitters achieved this effect the obvious way, by rapidly spinning a directional antenna in time with the electronically generated phase shift. Spinning anything quickly becomes a maintenance headache, and so VOR was quickly transitioned to solid-state techniques. Modern VOR transmitters are electronically rotated, by one of two techniques. They rotate in the same sense as images on a screen, a set of discrete changes in a solid state system that produce the effect of rotation.

The Truth or Consequences VOR operates on 112.7 MHz, near the middle of the band assigned for this use. Patterned after the nearby Truth or Consequences Airport, KTCS, it identifies itself by transmitting "TCS" in Morse code. Modern charts give this identifier in dots and dashes, an affordance to the poor level of Morse literacy among contemporary pilots.

In the airspace, it defines the intersection of several airways. They all go generally north-south, unsurprising considering that the restricted airspace of White Sands Missile Range prevents nearly all flight to the east. Flights following the Rio Grande, most north-south traffic in this area, will pass directly overhead on their way to VOR transmitters at Socorro or Deming or El Paso, where complicated airspace leads to two such sites very nearby.

This is the function that VORs serve: for the most part, you fly to or from them. Because the radial from the VOR to you remains constant, they provide a reliable and easy to use indication that you are still on the right track. A warning sign, verbose by tradition, articulates the significance:

This facility is used in FAA air traffic control. Loss of human life may result from service interruption. Any person who interferes with air traffic control or damages or trespasses on this property will be prosecuted under federal law.

The sign is backed up by a rustic wooden fence. Like most VOR transmitters, this one was built in the late 1950s or 1960s. The structure has seen only minimal changes since then, although the radio equipment has been improved and simplified.

The central, omnidirectional antenna of a VOR transmitter makes for a distinctive silhouette. You have likely noticed one before. I must admit that I have somewhat simplified; most of the volume of the central antenna housing is actually occupied by the TACAN antenna. Most VOR sites in the US are really VORTAC sites, combining the civilian VOR and military TACAN systems into one facility. TACAN has several minor advantages over VOR for military use, but one big advantage: it provides not only a radial but a distance. The same system used by TACAN for distance information, based on an unusual radio modulation technique called "squitter," can be used by civilian aircraft as well in the form of DME. VORTAC sites thus provide VOR, DME, and TACAN service.

True VOR sites, rare in the US but plentiful across the rest of the world, have smaller central antennas. If you are not used to observing the ring of radial antennas, you might not recognize them as the same system.

The radial antennas are placed in a circle some distance away, to open space between them. This reduces, but does not eliminate, the effect of each antenna's radiated power being absorbed by its neighbors. They are often on the roof of the equipment building, and may be surrounded by a metallic ground plane that extends still further. Most US VORTAC sites, originally built before modern RF technology, rely on careful positioning on suitable terrain rather than a ground plane.

Intriguingly, the radial antennas are not directional designs. In a modern VOR site, the radial antennas transmit an in-phase signal. The phase shift used for space modulation is created by rapidly changing the omnidirectional antenna in use. The space modulation is created not by rotating the antenna, but by moving the antenna through a circular path and allowing the Doppler effect to vary the apparent phase of the received signal.

The lower part of the central antenna, the more cone shaped part, is mostly empty. It encloses the structure that supports the cylindrical radome that houses the actual antenna elements. In newer installations it is often an exposed frame, but the original midcentury sites all provide a conical enclosure. I suspect the circular metallic sheathing simplified calculation of the effective radiation pattern at the time.

An access door can be used to reach the interior to service the antennas; the rope holding this one closed is not standard equipment but is perhaps also not very unusual. These are old facilities. When this cone was installed, adjacent Interstate 25 wasn't an interstate yet.

Aviation engineers leave little to chance, and almost never leave a system without a spare. Ground-based infrastructure is no exception. Each VOR transmitter is continuously tested by a monitoring system. A pair of antennas mounted on a post near the fence line feed redundant monitoring systems that ensure the static antennas receive the correct radial. If failure or a bad fix are detected, it switches the transmit antennas over to a second, redundant set of radio equipment. The problem is reported to the FAA, and Tech Ops staff are dispatched to investigate the problem.

Occasionally, the telephone lines VOR stations use to report problems are, themselves, unreliable. When Tech Ops is unable to remotely monitor a VOR station, they issue a NOTAM that it should not be relied upon.

The rear of the building better shows its age. The wall is scarred where old electrical service equipment has been removed; the weather-tight light fixture is a piece of incandescent history. It has probably been broken for longer than I have been alive.

A 1000 gallon propane tank to one side will supply the generator in the enclosure in case of a failure. Records of the Petroleum Storage Bureau of the New Mexico Environment Department show that an underground fuel tank was present at this site but has been removed. Propane is often selected for newer standby generator installations where an underground tank, no longer up to environmental safety standards, had to be removed.

It is indeed in its twilight years. The FAA has shut down about half of the VOR transmitters. TCS was spared this round, with all but one of the VOR transmitters in sparsely covered New Mexico. It is part of the "minimum operational network." It remains to be seen how long VOR's skeleton crew will carry on. A number of countries have now announced the end of VOR service. Another casualty to satellite PNT, joining LORAN wherever dead radio systems go.

The vastness and sparse population of southern New Mexico pose many challenges. One the FAA has long had to contend with is communications. Very near the Truth or Consequences VOR transmitter is an FAA microwave relay site. This tower is part of a chain that relays radar data from southern New Mexico to the air route traffic control center in Albuquerque.

When it was first built, the design of microwave communications equipment was much less advanced than it is today. Practical antennas were bulky and often pressurized for water tightness. Waveguides were expensive and cables were inefficient. To ease maintenance, shorten feedlines, and reduce tower loading, the actual antennas were installed on shelves near the bottom of the tower, pointing straight upwards. At the top of the tower, two passive reflectors acted like mirrors to redirect the signal into the distance. This "periscope" design was widely used by Western Union in the early days of microwave data networking.

Today, this system is partially retired, replaced by commercial fiber networks. This tower survives, maintained under contract by L3Harris. As the compound name suggests, half of this company used to Harris, a pioneer in microwave technology. The other half used to be L3, which split off from Lockheed Martin, which bought it when it was called Loral. Loral was a broad defense contractor, but had its history and focus in radar, another application of microwave RF engineering.

Two old radio sites, the remains of ambitious nationwide systems that helped create today's ubiquitous aviation. A town named after an old radio show. Some of the great achievements of radio history are out there in Sierra County.

I'm heading to Las Vegas for re:invent soon, perhaps the most boring type of industry extravaganza there could be. In that spirit, I thought I would write something quick and oddly professional: I'm going to complain about Docker.

Packaging software is one of those fundamental problems in system administration. It's so important, so influential on the way a system is used, that package managers are often the main identity of operating systems. Consider Windows: the operating system's most alarming defect in the eyes of many "Linux people" is its lack of package management, despite Microsoft's numerous attempts to introduce the concept. Well, perhaps more likely, because of the number of those attempts. And still, in the Linux world, distributions are differentiated primarily by their approach to managing software repositories. I don't just mean the difference between dpkg and rpm, but rather more fundamental decisions, like opinionated vs. upstream configuration and stable repositories vs. a rolling release. RHEL and Arch share the vast majority of their implementation and yet have very different vibes.

Linux distributions have, for the most part, consolidated on a certain philosophy of how software ought to be packaged, if not how often. One of the basic concepts shared by most Linux systems is centralization of dependencies. Libraries should be declared as dependencies, and the packages depended on should be installed in a common location for use of the linker. This can create a challenge: different pieces of software might depend on different versions of a library, which may not be compatible. This is the central challenge of maintaining a Linux distribution, in the classical sense: providing repositories of software versions that will all work correctly together. One of the advantages of stable distributions like RHEL is that they are very reliable in doing this; one of the disadvantages is that they achieve that goal by packaging new versions very infrequently.

Because of the need to provide mutually compatible versions of a huge range of software, and to ensure compliance with all kinds of other norms established by distributions (which may range from philosophical policies like free software to rules on the layout of configuration files), putting new software into Linux distributions can be... painful. For software maintainers, it means dealing with a bunch of distributions using a bunch of old versions with various specific build and configuration quirks. For distribution and package maintainers, it means bending all kinds of upstream software into compliance with distribution policy and figuring out version and dependency problems. It's all a lot of work, and while there are some norms, in practice it's sort of a wild scramble to do the work to make all this happen. Software developers that want their software to be widely used have to put up with distros. Distros that want software have to put up with software developers. Everyone gets mad.

Naturally there have been various attempts to ease these problems. Naturally they are indeed various and the community has not really consolidated on any one approach. In the desktop environment, Flatpak, Snap, and AppImage are all distressingly common ways of distributing software. The images or applications for these systems package the software complete with its dependencies, providing a complete self-contained environment that should work correctly on any distribution. The fact that I have multiple times had to unpack flatpaks and modify them to fix dependencies reveals that this concept doesn't always work entirely as advertised, but to be fair that kind of situation usually crops up when the software has to interact with elements of the system that the runtime can't properly isolate them from. The video stack is a classic example, where errant OpenGL libraries in packages might have to be removed or replaced for them to function with your particular graphics driver.

Still, these systems work reasonably well, well enough that they continue to proliferate. They are greatly aided by the nature of the desktop applications for which they're used (Snapcraft's system ambitions notwithstanding). Desktop applications tend to interact mostly with the user and receive their configuration via their own interface. Limiting the interaction surface mostly to a GUI window is actually tremendously helpful in making sandboxing feasible, although it continues to show rough edges when interacting with the file system.

I will note that I'm barely mentioning sandboxing here because I'm just not discussing it at the moment. Sandboxing is useful for security and even stability purposes, but I'm looking at these tools primarily as a way of packaging software for distribution. Sandboxed software can be distributed by more conventional means as well, and a few crusty old packages show that it's not as modern of a concept as it's often made out to be.

Anyway, what I really wanted to complain a bit about is the realm of software intended to be run on servers. Here, there is a clear champion: Docker, and to a lesser degree the ecosystem of compatible tools like Podman. The release of Docker lead to a surprisingly rapid change in what are widely considered best practices for server operations. While Docker images a means of distributing software first seemed to appeal mostly to large scalable environments with container orchestration, it sort of merged together with ideas from Vagrant and others to become a common means of distributing software for developer and single-node use as well.

Today, Docker is the most widespread way that server-side software is distributed for Linux. I hate it.

This is not a criticism of containers in general. Containerization is a wonderful thing with many advantages, even if the advantages over lightweight VMs are perhaps not as great as commonly claimed. I'm not sure that Docker has saved me more hours than it's cost, but to be fair I work as a DevOps consultant and, as a general rule, people don't get me involved unless the current situation isn't working properly. Docker images that run correctly with minimal effort don't make for many billable hours.

What really irritates me these days is not really the use of Docker images in DevOps environments that are, to some extent, centrally planned and managed. The problem is the use of Docker as a lowest common denominator, or perhaps more accurately lowest common effort, approach to distributing software to end users. When I see open-source, server-side software offered to me as a Docker image or--even worse---Docker Compose stack, my gut reaction is irritation. These sorts of things usually take longer to get working than equivalent software distributed as a conventional Linux package or to be built from source.

But wait, how does that happen? Isn't Docker supposed to make everything completely self-contained? Let's consider the common problems, something that I will call my Taxonomy of Docker Gone Bad.

Configuration

One of the biggest problems with Docker-as-distribution is the lack of consistent conventions for configuration. The vast majority of server-side Linux software accepts its configuration through an ages-old technique of reading a text file. This certainly isn't perfect! But, it is pretty consistent in its general contours. Docker images, on the other hand...

If you subscribe to the principles of the 12-factor-app, the best way for a Docker image to take configuration is probably via environment variables. This has the upside that it's quite straightforward to provide them on the command line when starting the container. It has the downside that environment variables aren't great for conveying structured data, and you usually interact with them via shell scripts that have clumsy handling of long or complicated values. A lot of Docker images used in DevOps environments take their configuration from environment variables, but they tend to make it a lot more feasible by avoiding complex configuration (by assuming TLS will be terminated by "someone else" for example) or getting a lot of their configuration from a database or service on the network.

For most end-user software though, configuration is too complex or verbose to be comfortable in environment variables. So, often, they fall back to configuration files. You have to get the configuration file into the container's file system somehow, and Docker provides numerous ways of doing so. Documentation on different packages will vary on which way it recommends. There are frequently caveats around ownership and permissions.

Making things worse, a lot of Docker images try to make configuration less painful by providing some sort of entry-point shell script that generates the full configuration from some simpler document provided to the container. Of course this level of abstraction, often poorly documented or entirely undocumented in practice, serves mostly to make troubleshooting a lot more difficult. How many times have we all experienced the joy of software failing to start, referencing some configuration key that isn't in what we provided, leading us to have to find have the Docker image build materials and read the entrypoint script to figure out how it generates that value?

The situation with configuration entrypoint scripts becomes particularly acute when those scripts are opinionated, and opinionated is often a nice way of saying "unsuitable for any configuration other than the developer's." Probably at least a dozen times I have had to build my own version of a Docker image to replace or augment an entrypoint script that doesn't expose parameters that the underlying software accepts.

In the worst case, some Docker images provide no documentation at all, and you have to shell into them and poke around to figure out where the actual configuration file used by the running software is even located. Docker images must always provide at least some basic README information on how the packaged software is configured.

Filesystems

One of the advantages of Docker is sandboxing or isolation, which of course means that Docker runs into the same problem that all sandboxes do. Sandbox isolation concepts do not interact well with Linux file systems. You don't even have to get into UID behavior to have problems here, just a Docker Compose stack that uses named volumes can be enough to drive you to drink. Everyday operations tasks like backups, to say nothing of troubleshooting, can get a lot more frustrating when you have to use a dummy container to interact with files in a named volume. The porcelain around named volumes has improved over time, but seemingly simple operations can still be weirdly inconsistent between Docker versions and, worse, other implementations like Podman.

But then, of course, there's the UID thing. One of the great sins of Docker is having normalized running software as root. Yes, Docker provides a degree of isolation, but from a perspective of defense in depth running anything with user exposure as root continues to be a poor practice. Of course this is one thing that often leads me to have to rebuild containers provided by software projects, and a number of common Docker practices don't make it easy. It all gets much more complicated if you use hostmounts because of UID mapping, and slightly complex environments with Docker can turn into NFS-style puzzles around UID allocation. Mitigating this mess is one of the advantages to named volumes, of course, with the pain points they bring.

Non-portable Containers

The irony of Docker for distribution, though, and especially Docker Compose, is that there are a lot of common practices that negatively impact portability---ostensibly the main benefit of this approach. Doing anything non-default with networks in Docker Compose will often create stacks that don't work correctly on machines with complex network setups. Too many Docker Compose stacks like to assume that default, well-known ports are available for listeners. They enable features of the underlying software without giving you a way to disable them, and assume common values that might not work in your environment.

One of the most common frustrations, for me personally, is TLS. As I have already alluded to, I preach a general principle that Docker containers should not terminate TLS. Accepting TLS connections means having access to the private key material. Even if 90-day ephemeral TLS certificates and a general atmosphere of laziness have deteriorated our discipline in this regard, private key material should be closely guarded. It should be stored in only one place and accessible to only one principal. You don't even have to get into these types of lofty security concerns, though. TLS is also sort of complicated to configure.

A lot of people who self-host software will have some type of SNI or virtual hosting situation. There may be wildcard certificates for multiple subdomains involved. All of this is best handled at a single point or a small number of dedicated points. It is absolutely maddening to encounter Docker images built with the assumption that they will individually handle TLS. Even with TLS completely aside, I would probably never expose a Docker container with some application directly to the internet. There are too many advantages to having a reverse proxy in front of it. And yet there are Docker Compose stacks out there for end-user software that want to use ACME to issue their own certificate! Now you have to dig through documentation to figure out how to disable that behavior.

The Single-Purpose Computer

All of these complaints are most common with what I would call hobby-tier software. Two examples that pop into my mind are HomeAssistant and Nextcloud. I don't call these hobby-tier to impugn the software, but rather to describe the average user.

Unfortunately, the kind of hobbyist that deploys software has had their mind addled by the cheap high of the Raspberry Pi. I'm being hyperbolic here, but this really is a problem. It's absurd the number of "self-hosted" software packages that assume they will run on dedicated hardware. Having "pi" in the name of a software product is a big red flag in my mind, it immediately makes me think "they will not have documented how to run this on a shared device." Call me old-fashioned, but I like my computers to perform more than one task, especially the ones that are running up my power bill 24/7.

HomeAssistant is probably the biggest offender here, because I run it in Docker on a machine with several other applications. It actively resists this, popping up an "unsupported software detected" maintenance notification after every update. Can you imagine if Postfix whined in its logs if it detected that it had neighbors?

Recently I decided to give NextCloud a try. This was long enough ago that the details elude me, but I think I burned around two hours trying to get the all-in-one Docker image to work in my environment. Finally I decided to give up and install it manually, to discover it was a plain old PHP application of the type I was regularly setting up in 2007. Is this a problem with kids these days? Do they not know how to fill in the config.php?

Hiding Sins

Of course, you will say, none of these problems would be widespread of people just made good Docker images. And yes, that is completely true! Perhaps one of the problems with Docker is that it's too easy to use. Creating an RPM or Debian package involves a certain barrier to entry, and it takes a whole lot of activation energy for even me to want to get rpmbuild going (advice: just use copr and rpkg). At the core of my complaints is the fact that distributing an application only as a Docker image is often evidence of a relatively immature project, or at least one without anyone who specializes in distribution. You have to expect a certain amount of friction in getting these sorts of things to work in a nonstandard environment.

It is a palpable irony, though, that Docker was once heralded as the ultimate solution to "works for me" and yet seems to just lead to the same situation existing at a higher level of configuration.

Last Thoughts

This is of course mostly my opinion and I'm sure you'll disagree on something, like my strong conviction that Docker Compose was one of the bigger mistakes of our era. Fifteen years ago I might have written a nearly identical article about all the problems I run into with RPMs created by small projects, but what surprises me about Docker is that it seems like projects can get to a large size, with substantial corporate backing, and still distribute in the form of a decidedly amateurish Docker Compose stack. Some of it is probably the lack of distribution engineering personnel on a lot of these projects, since Docker is "simple." Some of it is just the changing landscape of this class of software, with cheap single-board computers making Docker stacks just a little less specialized than a VM appliance image more palatable than they used to be. But some if it is also that I'm getting older and thus more cantankerous.

I have always been fascinated by the PABX - the private automatic branch exchange, often shortened to "PBX" in today's world where the "automatic" is implied. (Relatively) modern small and medium business PABXs of the type I like to collect are largely solid-state devices that mount on the wall. Picture a cabinet that's maybe two feet wide, a foot and half tall, and five inches deep. That's a pretty accurate depiction of my Comdial hybrid key/PABX system, recovered from the offices of a bankrupt publisher of Christian home schooling materials.

These types of PABX, now often associated with Panasonic on the small end, are affordable and don't require much maintenance or space. They have their limitations, though, particularly in terms of extension count. Besides, the fact that these compact PABX are available at all is the result of decades of development in electronics.

Not that long ago, PABX were far more complex. Early PBX systems were manual, and hotels were a common example of a business that would have a telephone operator on staff. The first PABX were based on the same basic technology as their contemporary phone switches, using step-by-step switches or even crossbar mechanisms. They no longer required an operator to connect every call, but were still mostly designed with the assumption that an attendant would handle some situations. Moreover, these early PABX were large, expensive, and required regular maintenance. They were often leased from the telephone company, and the rates weren't cheap.

PABX had another key limitation as well: they were specific to a location. Each extension had to be home-run wired to the PABX, easy in a single building but costly at the level of a campus and, especially, with buildings spread around a city. For organizations with distributed buildings like school districts, connecting extensions back to a central PABX could be significantly more expensive than connecting them to the public telephone exchange.

This problem must have been especially common in a city the size of New York, so it's no surprise that New York Telephone was the first to commercialize an alternative approach: Centrex.

Every technology writer must struggle with the temptation to call every managed service in history a precursor to "the Cloud." I am going to do my very best to resist that nagging desire, but it's difficult not to note the similarity between Centrex service and modern cloud PABX solutions. Indeed, Centrex relied on capabilities of telephone exchange equipment that are recognizably similar to mainframe computer concepts like LPARs and virtualization today. But we'll get there in a bit. First, we need to talk about what Centrex is.

I've had it in my mind to write something about Centrex for years, but I've always had a hard time knowing where to start. The facts about Centrex are often rather dry, and the details varied over years of development, making it hard to sum up the capabilities in short. So I hope that you will forgive this somewhat dry post. It covers something that I think is a very important part of telephone history, particularly from the perspective of the computer industry today. It also lists off a lot of boring details. I will try to illustrate with interesting examples everywhere I can. I am indebted, for many things but here especially, to many members of the Central Office mailing list. They filled in a lot of details that solidified my understanding of Centrex and its variants.

The basic promise of Centrex was this: instead of installing your own PABX, let the telephone company configure their own equipment to provide the features you want to your business phones. A Centrex line is a bit like a normal telephone line, but with all the added capabilities of a business phone system: intercom calling, transfers, attendants, routing and long distance policies, and so on. All of these features were provided by central telephone exchanges, but your lines were partitioned to be interconnected within your business.

Centrex was a huge success. By 1990, a huge range of large institutions had either started their telephone journey with Centrex or transitioned away from a conventional PABX and onto Centrex. It's very likely that you have interacted with a Centrex system before and perhaps not realized. And now, Centrex's days are numbered. Let's look at the details.

Centrex is often explained as a reuse of the existing central office equipment to serve PABX requirements. This isn't entirely incorrect, but it can be misleading. It was not all that unusual for Centrex to rely on equipment installed at the customer site, but operated by the telco. For this reason, it's better to think of Centrex as a managed service than as a "cloud" service, or a Service-as-a-Service, or whatever modern term you might be tempted to apply.

Centrex existed in two major variants: Centrex-CO and Centrex-CU. The CO case, for Central Office, entailed this well-known design of each business telephone line connecting to an existing telco central office, where a switch was configured to provide Centrex features on that line group. CU, for Customer Unit, looks more like a very large PABX. These systems were usually limited to very large customers, who would provide space for the telco to build a new central office on the customer's site. The exchange was located with the customer, but operated by the telco.

These two different categories of service lead to two different categories of customers, with different needs and usage patterns. Centrex-CO appealed to smaller organizations with fewer extensions, but also to larger organizations with extensions spread across a large area. In that case, wiring every extension back to the CO using telco infrastructure was less expensive than installing new wiring to a CU exchange. A prototypical example might be a municipal school district.

Centrex-CU appealed to customers with a large number of extensions grouped in a large building or a campus. In this case it was much less costly to wire extensions to the new CU site than to connect them all over the longer distance to an existing CO. A prototypical Centrex-CU customer might be a university.

Exactly how these systems worked varied greatly from exchange to exchange, but the basic concept is a form of partitioning. Telephone exchanges with support for Centrex service could be configured such that certain lines were grouped together and enabled for Centrex features. The individual lines needed to have access to Centrex-specific capabilities like service codes, but also needed to be properly associated with each other so that internal calling would indeed be internal to the customer. This concept of partitioning telephone switches had several different applications, and Western Electric and other manufacturers continued to enhance it until it reached a very high level of sophistication in digital switches.

Let's look at an example of a Centrex-CO. The State of New Mexico began a contract with Mountain States Telephone and Telegraph [1] for Centrex service in 1964. The new Centrex service replaced 11 manual switchboards distributed around Santa Fe, and included Wide-Area Telephone Service (WATS), a discount arrangement for long-distance calls placed from state offices to exchanges throughout New Mexico. On November 9th, 1964, technicians sent to Santa Fe by Western Electric completed the cutover at the state capitol complex. Incidentally, the capitol phones of the day were being installed in what is now the Bataan Memorial Building: construction of the Roundhouse, today New Mexico's distinctive state capitol, had just begun that same year.

The Centrex service was estimated to save $12,000 per month in the rental and operation of multiple state exchanges, and the combination of WATS and conference calling service was expected to produce further savings by reducing the need for state employees to travel for meetings. The new system was evidently a success, and lead to a series of minor improvements including a scheme later in 1964 to ensure that the designated official phone number of each state agency would be answered during the state lunch break (noon to 1:15). In 1965, Burns Reinier resigned her job as Chief Operator of the state Centrex to launch a campaign for Secretary of State. Many state employees would probably recognize her voice, but that apparently did not translate to recognition on the ballot, as she lost the Democratic party nomination to the Governor's former secretary.

The late 1960s saw a flurry of newspaper advertisements giving new phone numbers for state and municipal agencies, Albuquerque Public Schools, and universities, as they all consolidated onto the state-run Centrex system. Here we must consider the geographical nature of Centrex: Centrex service operates within a single telephone exchange. To span the gap between the capitol in Santa Fe, state offices and UNM in Albuquerque, NMSU in Las Cruces, and even the State Hospital in Las Vegas (NM), a system of tie lines were installed between Centrex facilities in each city. These tie lines were essentially dedicated long distance trunks leased by the state to connect calls between Centrex exchanges at lower cost than even WATS long-distance service.

This system was not entirely CO-based: in Albuquerque, a Centrex exchange was installed in state-leased space at what was then known as the National Building, 505 Marquette. In the late '60s, 505 Marquette also hosted Telepak, an early private network service from AT&T. It is perhaps a result of this legacy that 505 Marquette houses one of New Mexico's most important network facilities, a large carrier hotel now operated by H5 Data Centers. The installation of the Centrex exchange at 505 Marquette saved a lot of expense on new local loops, since a series of 1960s political and bureaucratic events lead to a concentration of state offices in the new building.

Having made this leap to customer unit systems, let's jump almost 30 years forward to an example of a Centrex-CU installation... one with a number of interesting details. In late 1989, Sandia National Laboratories ended its dependence on the Air Force for telephony services by contracting with AT&T for the installation of a 5ESS telephone exchange. The 5ESS, a digital switch and a rather new one at the time, brought with it not just advanced calling features but something even more compelling to an R&D institution at the time: data networking.

The Sandia installation went nearly all-in on ISDN, the integrated digital telephony and data standard that largely failed to achieve adoption for telephone applications. Besides the digital telephone sets, though, Sandia made full use of the data capabilities of the exchange. Computers connected to the data ports on the ISDN user terminals (the conventional term for the telephone instrument itself in an ISDN network) could make "data calls" over the telephone system to access IBM mainframes and other corporate computing resources... all at a blistering 64 kbps, the speed of an ISDN basic rate interface bearer channel. The ISDN network could even transport video calls, by combining multiple BRIs for 384 kbps aggregate capacity.

The 5ESS was installed on a building on Air Force property near Tech Area 1, and the 5ESS's robust support for remote switch modules was fully leveraged to place an RSM in each Tech Area. The new system required renumbering, always a hassle, but allowed for better matching of Sandia's phone numbers on the public network to phone numbers on the Federal Telecommunications System or FTS... a CCSA operated for the Federal Government. But we'll talk about that later. The 5ESS was also equipped with ISDN PRI tie lines to a sibling 5ESS at Sandia California in Livermore, providing inexpensive calling and ISDN features between the two sites.

This is a good time to discuss digital Centrex. Traditional telephony, even today in residential settings, uses analog telephones. Business systems, though, made a transition from analog to digital during the '80s and '90s. Digital telephone sets used with business systems provided far easier access to features of the key system, PABX, or Centrex, and with fewer wires. A digital telephone set on one or two telephone pairs could offer multiple voice lines, caller ID, central directory service, busy status indication for other phones, soft keys for pickup groups and other features, even text messaging in some later systems (like my Comdial!). Analog systems often required as many as a half dozen pairs just for a simple configuration like two lines and busy lamp fields; analog "attendant" sets with access to many lines could require a 25-pair Amphenol connector... sometimes even more than one.

Many of these digital systems used proprietary protocols between the switch and telephones. A notable example would be the TCM protocol used by the Nortel Meridian, an extremely popular PABX that can still be found in service in many businesses. Digital telephone sets made the leap to Centrex as well: first by Nortel themselves, who offered a "Meridian Digital Centrex" capability on their DMS-100 exchange switch that supported telephone sets similar to (but not the same as!) ordinary Meridian digital systems. AT&T followed several years later by offering 5ESS-based digital Centrex over ISDN: the same basic capability that could be used for computer applications as well, but with the advantage of full compatibility with AT&T's broader ISDN initiative.

The ISDN user terminals manufactured by Western Electric and, later, Lucent, are distinctive and a good indication that that digital Centrex is in use. They are also lovely examples of the digital telephones of the era, with LCD matrix displays, a bevy of programmable buttons, and pleasing Bellcore distinctive ringing. It is frustrating that the evolution of telephone technology has seemingly made ringtones far worse. We will have to forgive the oddities of the ISDN electrical standard that required an "NT1" network termination device screwed to the bottom of your desk or, more often, underfoot on the floor.

Thinking about these digital phones, let's consider the user experience of Centrex. Centrex was very flexible; there were a large number of options available based on customer preference, and the details varied between the Centrex host switches used in the United States: Western Electric's line from the 5XB to the 5ESS, Nortel's DMS-100 and DMS-10, and occasionally the Siemens EWSD. This all makes it hard to describe Centrex usage succinctly, but I will focus on some particular common features of Centrex.

Like PABXs, most Centrex systems required that a dialing prefix (conventionally nine) be used for an outside line. This was not universal, "assumed nine" could often be enabled at customer request, but it created a number of complications in the dialplan and was best avoided. Centrex systems, because they mostly belonged to larger customers, were more likely than PABXs to offer tie lines or other private routing arrangements, which were often used by dialing calls with a prefix of 8. Like conventional telephone systems, you could dial 0 for the operator, but on traditional large Centrex systems the operator would be an attendant within the Centrex customer organization.

Centrex systems enabled internal calling by extension, much like PABXs. Because of the large size of some Centrex-CU installations in particular you are probably much more likely to encounter five-digit extensions with Centrex than with a PABX. These types of extensions were usually designed by taking several exchange prefixes in a sequence, and using the last digit of the exchange code as the first digit of the extension. For that reason the extensions are often written in a format like 1-2345. A somewhat charming example of this arrangement was the 5ESS-based Centrex-CU at Los Alamos National Laboratories, which spans exchange prefixes 662-667 in the 505 NPA. Since that includes the less desirable exchange prefix 666, it was skipped. Of course, that didn't stop Telnyx from starting to use it more recently. Because of the history of Los Alamos's development, telephones in the town use these same prefixes, generally the lower ones.

With digital telephones, Centrex features are comparatively easy to access, since they can be assigned to buttons on the telephones. With analog systems there are no such convenient buttons, so Centrex features had to be awkwardly bolted on much like advanced features on non-Centrex lines. Many features are activated using vertical service codes starting with *, although in some systems (especially older systems for pulse compatibility) they might be mapped to codes that look more like extensions. Operations that involve interrupting an active call, like transfer or hold, involve flashing the hookswitch... a somewhat antiquated operation now more often achieved with a "flash" button on the telephone, when it's done at all.

Still, some analog Centrex systems used electrical tricks on the pair (similar to many PABX) to provide a message waiting light and even an extra button for common operations.

While Centrex initially appealed mainly to larger customers, improvements in host switch technology and telephone company practices made it an accessible option for small organizations as well. Verizon's "CustoPAK" was an affordable offering that provided Centrex features on up to 30 extensions. These small-scale services were also made more accessible by computerization. Configuration changes to the first crossbar Centrex service required exchange technicians climbing ladders to resolder jumpers. With the genesis of digital switches, telco employees in translation centers read customer requirements and built switch configuration plans. By the '90s, carriers offered modem services that allowed customers to reconfigure their Centrex themselves, and later web-based self-service systems emerged.

So what became of Centrex? Like most aspects of the conventional copper phone network, it is on the way out. Major telephone carriers have mostly removed Centrex service from their tariffs, meaning they are no longer required to offer it. Even in areas where it is present on the tariff it is reportedly hard to obtain. A report from the state of Washington notes that, as a result particularly of CenturyLink removing copper service from its tariffs entirely, CenturyLink has informed the state that it may discontinue Centrex service at any time, subject to six months notice. Six months may seem like a long time but it is a very short period for a state government to replace a statewide telephone system... so we can anticipate some hurried acquisitions in the next couple of years.

Centrex had always interacted with tariffs in curious ways, anyway. Centrex was the impetus behind multiple lawsuits against AT&T on grounds varying from anti-competitive behavior to violations of the finer points of tariff regulation. For the most part AT&T prevailed, but some of these did lead to changes in the way Centrex service was charged. Taxation was a particularly difficult matter. There were excise taxes imposed on telephone service in most cases, but AT&T held that "internal" calls within Centrex customers should not be subject to these taxes due to their similarity to untaxed PABX and key systems. The finer points of this debate varied from state to state, and it made it to the Supreme Court at least once.

Centrex could also have a complex relationship with the financial policies of many institutional customers. Centrex was often paired with services like WATS or tie lines to make long-distance calling more affordable, but this also encouraged employees to make their personal long-distance calls in the office. The struggle of long-distance charge accounting lead not only to lengthy employee "acceptable use" policies that often survive to this day, but also schemes of accounting and authorization codes to track long distance users. Long-distance phone charges by state employees were a perennial minor scandal in New Mexico politics, leading to some sort of audit or investigation every few years. Long-distance calling was often disabled except for extensions that required it, but you will find stories of public courtesy phones accidentally left with long-distance enabled becoming suddenly popular parts of university buildings.

Today, Centrex is generally being replaced with VoIP solutions. Some of these are fully managed, cloud-based services, analogous to Centrex-CO before them. IP phones bring a rich featureset that leave eccentric dialplans and feature codes mostly forgotten, and federal regulations around the accessibility of 911 have broadly discouraged prefix schemes for outside calls. On the flip side, these types of phone systems make it very difficult to configure dialplan schemes on endpoints, leading office workers to learn a new type of phone oddity: dialing pound after a number to skip the end-of-dialing timeout. This worked on some Centrex systems as well; some things never change.

[1] Later called US West, later called Qwest, now part of CenturyLink, which is now part of Lumen.

Nuclear weapons are complex in many ways. The basic problem of achieving criticality is difficult on its own, but deploying nuclear weapons as operational military assets involves yet more challenges. Nuclear weapons must be safe and reliable, even with the rough handling and potential of tampering and theft that are intrinsic to their military use.

Early weapon designs somewhat sidestepped the problem by being stored in inoperational condition. During the early phase of the Cold War, most weapons were "open pit" designs. Under normal conditions, the pit was stored separately from the weapon in a criticality-safe canister called a birdcage. The original three nuclear weapons stockpile sites (Manzano Base, Albuquerque NM; Killeen Base, Fort Hood TX; Clarksville Base, Fort Campbell KY) included special vaults to store the pit and assembly buildings where the pits would be installed into weapons. The pit vaults were designed not only for explosive safety but also to resist intrusion; the ability to unlock the vaults was reserved to a strictly limited number of Atomic Energy Commission personnel.

This method posed a substantial problem for nuclear deterrence, though. The process of installing the pits in the weapons was time consuming, required specially trained personnel, and wasn't particularly safe. Particularly after the dawn of ICBMs, a Soviet nuclear attack would require a rapid response, likely faster than weapons could be assembled. The problem was particularly evident when nuclear weapons were stockpiled at Strategic Air Command (SAC) bases for faster loading onto bombers. Each SAC base required a large stockpile area complete with hardened pit vaults and assembly buildings. Far more personnel had to be trained to complete the assembly process, and faster. Opportunities for mistakes that made weapons unusable, killed assembly staff, or contaminated the environment abounded.

As nuclear weapons proliferated, storing them disassembled became distinctly unsafe. It required personnel to perform sensitive operations with high explosives and radioactive materials, all under stressful conditions. It required that nuclear weapons be practical to assemble and disassemble in the field, which prevented strong anti-tampering measures.

The W-25 nuclear warhead, an approximately 220 pound, 1.7 kT weapon introduced in 1957, was the first to employ a fully sealed design. A relatively small warhead built for the Genie air-to-air missile, several thousand units would be stored fully assembled at Air Force sites. The first version of the W-25 was, by the AEC's own admission, unsafe to transport and store. It could detonate by accident, or it could be stolen.

The transition to sealed weapons changed the basic model of nuclear weapons security. Open weapons relied primarily on the pit vault, a hardened building with a bank-vault door, as the authentication mechanism. Few people had access to this vault, and two-man policies were in place and enforced by mechanical locks. Weapons stored assembled, though, lacked this degree of protection. The advent of sealed weapons presented a new possibility, though: the security measures could be installed inside of the weapon itself.

Safety elements of nuclear weapons protect against both unintentional and intentional attacks on the weapon. For example, from early on in the development of sealed implosion-type weapons "one-point safety" became common (it is now universal). One-point safe weapons have their high explosive implosion charge designed so that a detonation at any one point in the shell will never result in a nuclear yield. Instead, the imbalanced forces in the implosion assembly will tear it apart. This improper detonation produces a "fizzle yield" that will kill bystanders and scatter nuclear material, but produces orders of magnitude less explosive force and radiation dispersal than a complete nuclear detonation.

The basic concept of one-point safety is a useful example to explain the technical concepts that followed later. One-point safety is in some ways an accidental consequence of the complexity of implosion weapons: achieving a full yield requires an extremely precisely timed detonation of the entire HE shell. Weapons relied on complex (at the time) electronic firing mechanisms to achieve the required synchronization. Any failure of the firing system to produce a simultaneous detonation results in a partial yield because of the failure to achieve even implosion. One-point safety is essentially just a product of analysis (today computer modeling) to ensure that detonation of a single module of the HE shell will never result in a nuclear yield.

This one-point scenario could occur because of outside forces. For example, one-point safety is often described in terms of enemy fire. Imagine that, in combat conditions, anti-air weapons or even rifle fire strike a nuclear weapon. The shock forces will reach one side of the HE shell first. If they are sufficient to detonate it (not an easy task as very insensitive explosives are used), the one-point detonation will destroy the weapon with a fizzle yield.

We can also examine one-point safety in terms of the electrical function of the weapon. A malfunction or tampering with a weapon might cause one of the detonators to fire. The resulting one-point detonation will destroy the weapon. Achieving a nuclear yield requires that the shell be detonated in synchronization, which naturally functions as a measure of the correct operation of the firing system. Correctly firing a nuclear weapon is complex and difficult, requiring that multiple components are armed and correctly functioning. This itself serves as a safety mechanism since correct operation, difficult to achieve by intention, is unlikely to happen by accident.

Like most nuclear weapons, the W-25 received a series of modifications or "mods." The second, mod 1 (they start at 0), introduced a new safety mechanism: an environmental sensing device. The environmental sensing device allowed the weapon to fire only if certain conditions were satisfied, conditions that were indicative of the scenario the weapon was intended to fire in. The details of the ESD varied by weapon and probably even by application within a set of weapons, but the ESD generally required things like a moving a certain distance at a certain speed (determined by inertial measurements) or a certain change in altitude in order to arm the weapon. These measurements ensured that the weapon had actually been fired on a missile or dropped as a bomb before it could arm.

The environmental sensing device provides one of two basic channels of information that weapons require to arm: indication that the weapon is operating under normal conditions, like flying towards a target or falling onto one. This significantly reduces the risk of unintentional detonation.

There is a second possibility to consider, though, that of intentional detonation by an unauthorized user. A weapon could be stolen, or tampered with in place as an act of terrorism. To address this possibility, a second basic channel of input was developed: intent. For a weapon to detonate, it must be proven that an authorized user has the intent to detonate the weapon.

The implementation of these concepts has varied over time and by weapon type, but from unclassified materials a general understanding of the architecture of these safety systems can be developed. I decided to write about this topic not only because it is interesting (it certainly is), but also because many of the concepts used in the safety design of nuclear weapons are also applicable to other systems. Similar concepts are used, for example, in life-safety systems and robotics, fields where unintentional operation or tampering can cause significant harm to life and property. Some of the principles are unsurprisingly analogous to cryptographic methods used in computer security, as well.

The basic principle of weapons safety is called the strong link, weak link principle, and it is paired to the related idea of an exclusion zone. To understand this, it's helpful to remember the W-25's sealed design. For open weapons, a vault was used to store the pit. In a sealed weapon, the vault is, in a sense, built into the weapon. It's called the exclusion zone, and it can be thought of as a tamper-protected, electrically isolated chamber that contains the vital components of the weapon, including the electronic firing system.

In order to fire the weapon, the exclusion zone must be accessed, in that an electrical signal needs to be delivered to the firing system. Like the bank vaults used for pits, there is only one way into the exclusion zone, and it is tightly locked. An electrical signal must penetrate the energy barrier that surrounds the exclusion zone, and the only way to do so is by passing through a series of strong links.

The chain of events required to fire a nuclear weapon can be thought of like a physical chain used to support a load. Strong links are specifically reinforced so that they should never fail. We can also look at the design through the framework of information security, as an authentication and authorization system. Strong links are strict credential checks that will deny access under all conditions except the one in which the weapon is intended to fire: when the weapon is in suitable environmental conditions, has received an authorized intent signal, and the fuzing system calls for detonation.

One of the most important functions of the strong link is to confirm that correct environmental and intent authorization has occurred. The environmental sensing device, installed in the body of the weapon, sends its authorizing signal when its conditions are satisfied. There is some complexity here, though. One of the key concerns in weapons safety was the possibility of stray electrical signals, perhaps from static or lightning or contact with an aircraft electrical system, causing firing. The strong link needs to ensure that the authorization signal received really is from the environmental sensing device, and not a result of some electrical transient.

This verification is performed by requiring a unique signal. The unique signal is a digital message consisting of multiple bits, even when only a single bit of information (that environmental conditions are correct) needs to be conveyed. The extra bits serve only to make the message complex and unique. This way, any transient or unintentional electrical signal is extremely unlikely to match the correct pattern. We can think of this type of unique signal as an error detection mechanism, padding the message with extra bits just to verify the correctness of the important one.

Intent is a little trickier, though. It involves human input. The intent signal comes from the permissive action link, or PAL. Here, too, the concept of a unique signal is used to enable the weapon, but this time the unique signal isn't only a matter of error detection. The correct unique signal is a secret, and must be provided by a person who knows it.

Permissive action links are fascinating devices from a security perspective. The strong link is like a combination lock, and the permissive action link is the key or, more commonly, a device through which they key is entered. There have been many generations of PALs, and we are fortunate that a number of older, out of use PALs are on public display at the National Museum of Nuclear Science and History here in Albuquerque.

Here we should talk a bit about the implementation of strong links and PALs. While newer designs are likely more electronic, older designs were quite literally combination locks: electromechanical devices where a stepper motor or solenoid had to advance a clockwork mechanism in the correct pattern. It was a lot like operating a safe lock by remote. The design of PALs reflected this. Several earlier PALs are briefcases that, when opened, reveal a series of dials. An operator has to connect the PAL to the weapon, turn all the dials to the correct combination, and then press a button to send to the unique signal to the weapon.

Later PALs became very similar to the key loading devices used for military cryptography. The unique signal is programmed into volatile memory in the PAL. To arm a weapon, the PAL is connected, an operator authenticates themselves to the PAL, and then the PAL sends the stored unique signal. Like a key loader, the PAL itself incorporates measures against tampering or theft. A zeroize function is activated by tamper sensors or manually and clears the stored unique key. Too many failures by an operator to authenticate themselves also results in the stored unique signal being cleared.

Much like key loaders, PALs developed into more sophisticated devices over time with the ability to store and manage multiple unique signals, rekey weapons with new unique signals, and to authenticate the operator by more complex means. A late PAL-adjacent device on public display is the UC1583, a Compaq laptop docked to an electronic interface. This was actually a "PAL controller," meaning that it was built primarily for rekeying weapons and managing sets of keys. By this later era of nuclear weapons design, the PAL itself was typically integrated into communications systems on the delivery vehicle and provided a key to the weapon based on authorization messages received directly from military command authorities.

The next component to understand is the weak link. A strong link is intended to never fail open. A weak link is intended to easily fail closed. A very basic type of weak link would be a thermal fuse that burns out in response to high temperatures, disconnecting the firing system if the weapon is exposed to fire. In practice there can be many weak links and they serve as a protection against both accidental firing of a damaged weapon and intentional tampering. The exclusion zone design incorporates weak links such that any attempt to open the exclusion zone by force will result in weak links failing.

A special case of a weak link, or at least something that functions like a weak link, is the command disable feature on most weapons. Command disable is essentially a self-destruct capability. Details vary but, on the B61 for example, the command disable is triggered by pulling a handle that sticks out of the control panel on the side of the weapon. The command disable triggers multiple weak links, disabling various components of the weapon in hard-to-repair ways. An unauthorized user, without the expertise and resources of the weapons assembly technicians at Pantex, would find it very difficult to restore a weapon to working condition after the command disable was activated. Some weapons apparently had an explosive command disable that destroyed the firing system, but from publicly available material it seems that a more common design involved the command disable interrupting the power supply to volatile storage for unique codes and configuration information.

There are various ways to sum up these design features. First, let's revisit the overall architecture. Critical components of nuclear weapons, including both the pit itself and the electronic firing system, are contained within the exclusion zone. The exclusion zone is protected by an energy barrier that isolates it from mechanical and electrical influence. For the weapon to fire, firing signals must pass through strong links and weak links. Strong links are designed to never open without a correct unique signal, and to fail open only in extreme conditions that would have already triggered weak links. Weak links are designed to easily fail closed in abnormal situations like accidents or tampering. Both strong links and weak links can receive human input, strong links to provide intent authorization, and weak links to manually disable the weapon in a situation where custody may be lost.

The physical design of nuclear weapons is intricate and incorporates many anti-tamper and mechanical protection features, and high explosives and toxic and radioactive materials lead to hazardous working conditions. This makes the disassembly of modern nuclear weapons infamously difficult; a major challenge in the reduction of the nuclear stockpile is the backlog of weapons waiting for qualified technicians to take them apart. Command disable provides a convenience feature for this purpose, since it allows weapons to be written off the books before they can be carefully dismantled at one of very few facilities (often just one) capable of doing so. As an upside, these same properties make it difficult for an unauthorized user to circumvent the safety mechanisms in a nuclear weapon, or repair one in which weak links have failed.

Accidental arming and detonation of a nuclear weapon should not occur because the weapon will only arm on receipt of complex unique signals, including an intent signal that is secret and available only to a limited number of users (today, often only to the national command authority). Detonation of a weapon under extreme conditions like fire or mechanical shock is prevented by the denial of the strong links, the failure of the weak links, and the inherent difficulty of correctly firing a nuclear weapon. Compromise of a nuclear weapon, or detonation by an unauthorized user, is prevented by the authentication checks performed by the strong links and the tamper resistance provided by the weak links. Cryptographic features of modern PALs enhance custodial control of weapons by enabling rotation and separation of credentials.

Modern PALs particularly protect custodial control by requiring keys unknown to the personnel handling the weapons before they can be armed. These keys must be received from the national command authority as part of the order to attack, making communications infrastructure a critical part of the nuclear deterrent. It is for this reason that the United States has so many redundant, independent mechanisms of delivering attack orders, ranging from secure data networks to radio equipment on Air Force One capable of direct communication with nuclear assets.

None of this is to say that the safety and security of nuclear weapons is perfect. In fact, historical incidents suggest that nuclear weapons are sometimes surprisingly poorly protected, considering the technical measures in place. The widely reported story that the enable code for the Minuteman warhead's PAL was 00000000 is unlikely to be true as it was originally reported, but that's not to say that there are no questions about the efficacy of PAL key management. US weapons staged in other NATO countries, for example, have raised perennial concerns about effective custody of nuclear weapons and the information required to use them.

General military security incidents endanger weapons as well. Widely reported disclosures of nuclear weapon security procedures by online flash card services and even Strava do not directly compromise these on-weapon security measures but nonetheless weaken the overall, multi-layered custodial security of these weapons, making other layers more critical and more vulnerable.

Ultimately, concerns still exist about the design of the weapons themselves. Most of the US nuclear fleet is very old. Many weapons are still in service that do not incorporate the latest security precautions, and efforts to upgrade these weapons are slow and endangered by many programmatic problems. Only in 1987 was the entire arsenal equipped with PALs, and in 2004 all weapons were equipped with cryptographic rekeying capability.

PALs, or something like them, are becoming the international norm. The Soviet Union developed similar security systems for their weapons, and allies of the United States often use US-designed PALs or similar under technology sharing agreements. Pakistan, though, remains a notable exception. There are still weapons in service in various parts of the world without this type of protection. Efforts to improve that situation are politically complex and run into many of the same challenges as counterproliferation in general.

Nuclear weapons are perhaps safer than you think, but that's certainly not to say that they are safe.

[1] This "popular fact" comes from an account by a single former missileer. Based on statements by other missile officers and from the Air Force itself, the reality seems to be complex. The 00000000 code may have been used before the locking mechanism was officially placed in service, during a transitional stage when technical safeguards had just been installed but missile crews were still operating on procedures developed before their introduction. Once the locking mechanism was placed in service and missile crews were permitted to deviate from the former strict two-man policy, "real" randomized secret codes were used.