Eri has a mix of a "true confession" and a "wait, really?" today.

The programming language Vala bills itself as a C# like language that compiles into something pretty close to C performance, designed specifically for writing code against Gnome and its associated libraries.

One of the C#-isms in brings in is async/await type semantics. You can yield someAsyncFunction(), which returns control to the caller, allowing it to proceed until the yielded function returns an actual value.

Because it has asynchronous functions, many library functions for handling I/O are already async. So you can make_directory_async, which yields control so you can keep executing while waiting for the filesystem to make your directory.

There are also synchronous versions of those methods. And then there's create_directory_with_parents, which will create a chain of directories for you. That's the synchronous version, and Vala's core library has decided not to provide an asynchronous version of it, which is my "wait, really?" I suspect it's really about the race conditions involved and the risks of things going wrong while doing it asynchronously; all solvable problems, but tricky ones to solve.

But it's the problem Eri had, and this is their solution:

/// Note: does not throw if target already exists
async void create_directory_with_parents_async(File file, Cancellable? cancellable = null) throws Error {
	var to_create = new File[0];
	var? current_target = file;
	while(current_target != null) {
		try {
			yield current_target.make_directory_async(Priority.DEFAULT, cancellable);
		} catch(IOError.NOT_FOUND e) {
			to_create += current_target;
			current_target = current_target.get_parent();
			continue;
		} catch(IOError.EXISTS e) {
			break;
		}
		break;
	}

	for (int i = to_create.length - 1; i >= 0; --i) {
		try {
			yield to_create[i].make_directory_async(Priority.DEFAULT, cancellable);
		} catch(IOError.EXISTS e) {
			// Created by another process
		}
	}
}

If I'm reading this correctly, we start by trying to create the full path to our leaf node. If there's a not found error, we go up one level and try and create that one. We keep trying that until we either run out of parent nodes to try against, or we hit a directory that already exists, or we successfully create a directory. All along the way, we keep appending the current_target to our to_create array.

Once we've gotten that baseline, we then iterate across our to_create array, backwards, creating the shortest non-existent paths first.

This works, but it's ugly as sin. Mostly, it's ugly because we're using exceptions for flow control instead of doing things like checking for file existence, though I suppose those checks may also break our goal of doing all our I/O operations in an async context. I don't know enough about Vala to know the better way of doing this.

Eri writes:

The function works as intended, but trying to trace control flow through the first loop is not pleasant. Ironically, the C mechanism Vala wraps is slightly advanced error codes, which would be nicer to work with in this case

Eri also provides a slightly re-worked version of the main loop, that is at least a bit easier to follow, but still an ugly approach:

	while(current_target != null) {
		try {
			yield current_target.make_directory_async(Priority.DEFAULT, cancellable);
			break;
		} catch(IOError.EXISTS e) {
			break;
		} catch(IOError.NOT_FOUND e) {
			to_create += current_target;
			current_target = current_target.get_parent();
		}
	}

Still, since this is an attempt to patch over a missing core library method and solve a tricky problem about how to handle race conditions, I think absolution is reasonable. It's ugly, it's weird, but it does the job. Go hide it in a box, and never touch its implementation again- except to make it go away.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

From our anonymous submitter:

Having reached the end of the road at a company increasingly swallowed up companies further east which you'd never believe were still afloat, I found myself headhunted for certain specialty software skills. I was reaching the final few years of my expected working span, so I jumped at the chance. The money was (to me at that time) spectacularly good, so I jumped into it.

It started when my first day was spent by me being sent home for the weeks it was still going to take to onboard me. Not bad, engaged to wait, as it were, and the first 6 months was thus and so.

The man who had interviewed me, call him Fred, was intelligent and urbane, and was a joy to meet. He and I clearly hit it off, and lo and behold I was in. It was he who gave me my first assignment, which was mathematical analysis of their core milk-cow program because they needed to find out what it did, and how it did it, so they could perhaps implement it in a more contemporary language.

So I did that, and was just about to publish my findings with him, when Fred inconveniently dropped dead suddenly. In the what-are-we-going-to-do-now-our-key-man-is-no-more confusion, we contractors were forgotten.

For the next 18 months or so (may have been more, may have been less) I was more or less ignored. I spent the time writing a development environment to work on any part of the program conveniently, all the while sitting next to a man who was constantly, forcefully and repetitiously speaking ill of the managers in his line structure. The ridiculously garrulous boss who inherited me thought little of me, and handed me the little work that came my way with active hostility. One or two good guys, but mostly a cabal of elderly men trying to preserve their little money-spinner as long as they could, and a johnny-come-lately trying to increase (and even introduce) automatic processes was less than welcome. During that time I spent quite some time on TDWTF, submitting a gem or two here and there.

No surprise when they finally kicked my arse away. No love lost there. Now working my last couple of years to retirement as a postie.

No punchline here. I want you to know that dropping dead from work is all too real and can happen to anyone.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

It's a holiday here in the US, a celebration of labor, so we're reaching back through the archives for a story about an attempt to be labor saving that was not successful. Original. --Remy

I’ve had to write a few domain-specific-languages in the past. As per Remy’s Law of Requirements Gathering, it’s been mostly because the users needed an Excel-like formula language. The danger of DSLs, of course, is that they’re often YAGNI in the extreme, or at least a sign that you don’t really understand your problem.

XML, coupled with schemas, is a tool for building data-focused DSLs. If you have some complex structure, you can convert each of its features into an XML attribute. For example, if you had a grammar that looked something like this:

The Source specification obeys the following syntax

source = ( Feature1+Feature2+... ":" ) ? steps

Feature1 = "local" | "global"

Feature2 ="real" | "virtual" | "ComponentType.all"

Feature3 ="self" | "ancestors" | "descendants" | "Hierarchy.all"

Feature4 = "first" | "last" | "DayAllocation.all"

If features are specified, the order of features as given above has strictly to be followed.

steps = oneOrMoreNameSteps | zeroOrMoreNameSteps | componentSteps

oneOrMoreNameSteps = nameStep ( "." nameStep ) *

zeroOrMoreNameSteps = ( nameStep "." ) *

nameStep = "#" name

name is a string of characters from "A"-"Z", "a"-"z", "0"-"9", "-" and "_". No umlauts allowed, one character is minimum.

componentSteps is a list of valid values, see below.

Valid 'componentSteps' are:

- GlobalValue
- Product
- Product.Brand
- Product.Accommodation
- Product.Accommodation.SellingAccom
- Product.Accommodation.SellingAccom.Board
- Product.Accommodation.SellingAccom.Unit
- Product.Accommodation.SellingAccom.Unit.SellingUnit
- Product.OnewayFlight
- Product.OnewayFlight.BookingClass
- Product.ReturnFlight
- Product.ReturnFlight.BookingClass
- Product.ReturnFlight.Inbound
- Product.ReturnFlight.Outbound
- Product.Addon
- Product.Addon.Service
- Product.Addon.ServiceFeature

In addition to that all subsequent steps from the paths above are permitted, that is 'Board', 
'Accommodation.SellingAccom' or 'SellingAccom.Unit.SellingUnit'.
'Accommodation.Unit' in the contrary is not permitted, as here some intermediate steps are missing.

You could turn that grammar into an XML document by converting syntax elements to attributes and elements. You could do that, but Stella’s predecessor did not do that. That of course, would have been work, and they may have had to put some thought on how to relate their homebrew grammar to XSD rules, so instead they created an XML schema rule for SourceAttributeType that verifies that the data in the field is valid according to the grammar… using regular expressions. 1,310 characters of regular expressions.

<xs:simpleType>
    <xs:restriction base="xs:string">
            <xs:pattern value="(((Scope.)?(global|local|current)\+?)?((((ComponentType.)?
(real|virtual))|ComponentType.all)\+?)?((((Hierarchy.)?(self|ancestors|descendants))|Hierarchy.all)\+?)?
((((DayAllocation.)?(first|last))|DayAllocation.all)\+?)?:)?(#[A-Za-z0-9\-_]+(\.(#[A-Za-z0-9\-_]+))*|(#[A-Za-z0-
9\-_]+\.)*
(ThisComponent|GlobalValue|Product|Product\.Brand|Product\.Accommodation|Product\.Accommodation\.SellingAccom|Prod
uct\.Accommodation\.SellingAccom\.Board|Product\.Accommodation\.SellingAccom\.Unit|Product\.Accommodation\.Selling
Accom\.Unit\.SellingUnit|Product\.OnewayFlight|Product\.OnewayFlight\.BookingClass|Product\.ReturnFlight|Product\.
ReturnFlight\.BookingClass|Product\.ReturnFlight\.Inbound|Product\.ReturnFlight\.Outbound|Product\.Addon|Product\.
Addon\.Service|Product\.Addon\.ServiceFeature|Brand|Accommodation|Accommodation\.SellingAccom|Accommodation\.Selli
ngAccom\.Board|Accommodation\.SellingAccom\.Unit|Accommodation\.SellingAccom\.Unit\.SellingUnit|OnewayFlight|Onewa
yFlight\.BookingClass|ReturnFlight|ReturnFlight\.BookingClass|ReturnFlight\.Inbound|ReturnFlight\.Outbound|Addon|A
ddon\.Service|Addon\.ServiceFeature|SellingAccom|SellingAccom\.Board|SellingAccom\.Unit|SellingAccom\.Unit\.Sellin
gUnit|BookingClass|Inbound|Outbound|Service|ServiceFeature|Board|Unit|Unit\.SellingUnit|SellingUnit))"/>
    </xs:restriction>
</xs:simpleType>
</xs:union>

There’s a bug in that regex that Stella needed to fix. As she put it: “Every time you evaluate it a few little kitties die because you shouldn’t use kitties to polish your car. I’m so, so sorry, little kitties…”

The full, unexcerpted code is below, so… at least it has documentation. In two languages!

<xs:simpleType name="SourceAttributeType">
                <xs:annotation>
                        <xs:documentation xml:lang="de">
                Die Source Angabe folgt folgender Syntax

                        source = ( Eigenschaft1+Eigenschaft2+... ":" ) ? steps

                        Eigenschaft1 = "local" | "global"

                        Eigenschaft2 ="real" | "virtual" | "ComponentType.all"

                        Eigenschaft3 ="self" | "ancestors" | "descendants" | "Hierarchy.all"

                        Eigenschaft4 = "first" | "last" | "DayAllocation.all"

                        Falls Eigenschaften angegeben werden muss zwingend die oben angegebene Reihenfolge der Eigenschaften eingehalten werden.

                        steps = oneOrMoreNameSteps | zeroOrMoreNameSteps | componentSteps

                        oneOrMoreNameSteps = nameStep ( "." nameStep ) *

                        zeroOrMoreNameSteps = ( nameStep "." ) *

                        nameStep = "#" name

                        name ist eine Folge von Zeichen aus der Menge "A"-"Z", "a"-"z", "0"-"9", "-" und "_". Keine Umlaute. Mindestens ein Zeichen

                        componentSteps ist eine Liste gültiger Werte, siehe im folgenden

                Gültige 'componentSteps' sind zunächst:

                        - GlobalValue
                        - Product
                        - Product.Brand
                        - Product.Accommodation
                        - Product.Accommodation.SellingAccom
                        - Product.Accommodation.SellingAccom.Board
                        - Product.Accommodation.SellingAccom.Unit
                        - Product.Accommodation.SellingAccom.Unit.SellingUnit
                        - Product.OnewayFlight
                        - Product.OnewayFlight.BookingClass
                        - Product.ReturnFlight
                        - Product.ReturnFlight.BookingClass
                        - Product.ReturnFlight.Inbound
                        - Product.ReturnFlight.Outbound
                        - Product.Addon
                        - Product.Addon.Service
                        - Product.Addon.ServiceFeature

                Desweiteren sind alle Unterschrittfolgen aus obigen Pfaden erlaubt, also 'Board', 'Accommodation.SellingAccom' oder 'SellingAccom.Unit.SellingUnit'.
                'Accommodation.Unit' hingegen ist nicht erlaubt, da in diesem Fall einige Zwischenschritte fehlen.

                                </xs:documentation>
                        <xs:documentation xml:lang="en">
                                The Source specification obeys the following syntax

                                source = ( Feature1+Feature2+... ":" ) ? steps

                                Feature1 = "local" | "global"

                                Feature2 ="real" | "virtual" | "ComponentType.all"

                                Feature3 ="self" | "ancestors" | "descendants" | "Hierarchy.all"

                                Feature4 = "first" | "last" | "DayAllocation.all"

                                If features are specified, the order of features as given above has strictly to be followed.

                                steps = oneOrMoreNameSteps | zeroOrMoreNameSteps | componentSteps

                                oneOrMoreNameSteps = nameStep ( "." nameStep ) *

                                zeroOrMoreNameSteps = ( nameStep "." ) *

                                nameStep = "#" name

                                name is a string of characters from "A"-"Z", "a"-"z", "0"-"9", "-" and "_". No umlauts allowed, one character is minimum.

                                componentSteps is a list of valid values, see below.

                                Valid 'componentSteps' are:

                                - GlobalValue
                                - Product
                                - Product.Brand
                                - Product.Accommodation
                                - Product.Accommodation.SellingAccom
                                - Product.Accommodation.SellingAccom.Board
                                - Product.Accommodation.SellingAccom.Unit
                                - Product.Accommodation.SellingAccom.Unit.SellingUnit
                                - Product.OnewayFlight
                                - Product.OnewayFlight.BookingClass
                                - Product.ReturnFlight
                                - Product.ReturnFlight.BookingClass
                                - Product.ReturnFlight.Inbound
                                - Product.ReturnFlight.Outbound
                                - Product.Addon
                                - Product.Addon.Service
                                - Product.Addon.ServiceFeature

                                In addition to that all subsequent steps from the paths above are permitted, that is 'Board', 'Accommodation.SellingAccom' or 'SellingAccom.Unit.SellingUnit'.
                                'Accommodation.Unit' in the contrary is not permitted, as here some intermediate steps are missing.

                        </xs:documentation>
                </xs:annotation>
                <xs:union>
                        <xs:simpleType>
                                <xs:restriction base="xs:string">
                                        <xs:pattern value="(((Scope.)?(global|local|current)\+?)?((((ComponentType.)?(real|virtual))|ComponentType.all)\+?)?((((Hierarchy.)?(self|ancestors|descendants))|Hierarchy.all)\+?)?((((DayAllocation.)?(first|last))|DayAllocation.all)\+?)?:)?(#[A-Za-z0-9\-_]+(\.(#[A-Za-z0-9\-_]+))*|(#[A-Za-z0-9\-_]+\.)*(ThisComponent|GlobalValue|Product|Product\.Brand|Product\.Accommodation|Product\.Accommodation\.SellingAccom|Product\.Accommodation\.SellingAccom\.Board|Product\.Accommodation\.SellingAccom\.Unit|Product\.Accommodation\.SellingAccom\.Unit\.SellingUnit|Product\.OnewayFlight|Product\.OnewayFlight\.BookingClass|Product\.ReturnFlight|Product\.ReturnFlight\.BookingClass|Product\.ReturnFlight\.Inbound|Product\.ReturnFlight\.Outbound|Product\.Addon|Product\.Addon\.Service|Product\.Addon\.ServiceFeature|Brand|Accommodation|Accommodation\.SellingAccom|Accommodation\.SellingAccom\.Board|Accommodation\.SellingAccom\.Unit|Accommodation\.SellingAccom\.Unit\.SellingUnit|OnewayFlight|OnewayFlight\.BookingClass|ReturnFlight|ReturnFlight\.BookingClass|ReturnFlight\.Inbound|ReturnFlight\.Outbound|Addon|Addon\.Service|Addon\.ServiceFeature|SellingAccom|SellingAccom\.Board|SellingAccom\.Unit|SellingAccom\.Unit\.SellingUnit|BookingClass|Inbound|Outbound|Service|ServiceFeature|Board|Unit|Unit\.SellingUnit|SellingUnit))"/>
                                </xs:restriction>
                        </xs:simpleType>
                </xs:union>
</xs:simpleType>

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Apple held its annual September major product rollout today announcing, among other things, updated Apple Watch models with “Audio Intelligence” features. One of those features is “Live Rewind”:

A double press of the Digital Crown shows the previous 15 seconds of a conversation as a text snippet, so the user can catch something they may have missed or are less familiar with. Users can ask Siri about the content of the text or save it to the Siri app to revisit later.

This feature is also coming to iPhone 16 and newer models. Apple swears up and down it is doing all it can to make this private and secure, and I have no reason to believe otherwise. The audio is apparently not actually saved and is, instead, merely transcribed by a dedicated coprocessor. That text is automatically deleted unless the user takes an action to save it, too.

But good luck explaining that to anyone around you after you read back an exact transcript of what they just said. Apple says they should be sufficiently notified Live Rewind is active because “an audible chime plays from the speaker on your Apple Watch, even if your Apple Watch is on silent or you have headphones connected”, plus the watch plays an animation, but it is not clear to me when this happens. It is not shown in Apple’s video. The impression I get is that Live Rewind is always running in the background and these notifications are only played after you double-click the Digital Crown to look at the text.

This feels like a glimpse of how Apple will market its inevitable wear-anywhere glasses product. I do not think it does enough to assuage privacy concerns. I think normal people will continue to react negatively when you tell them you have been passively recording them because, outside Silicon Valley, that is considered gross and invasive.

⌥ Permalink

Michael Crider, PCWorld:

[…] So somebody at LG decided that, in addition to the basic monitor driver and management app delivered to its users via Windows Update, it would slip in an additional program, “LG Monitor App Installer.” This extra bit of software includes McAfee, or an ad for it — as if there was any difference.

C. da Costa, Gadget Review:

This is bigger than one bad app. LG and Samsung smart monitors run the same operating systems as their smart TVs — webOS and Tizen — both already built for automatic content recognition and ad targeting. Acer’s monitor privacy policies permit sharing device data with advertisers. The trajectory mirrors the rise of free-to-play gaming: low barrier to entry, monetization creep once you’re locked in.

These findings were based on a Gamers Nexus investigation in July. But it is not an isolated incident.

This week, Gamers Nexus followed up with another blockbuster showing how LG’s smart televisions find other devices on the same network and, in some cases, record their activity; and, because they can recognize what is being played onscreen, LG tracks everything you watch, too. It is presumptively allowed do this because it presents a bunch of user agreements that anyone can bypass without reading, enables tracking by default, and buries all the opt-out stuff. Oh, and there is more.

Scharon Harding, Ars Technica:

The video also demonstrated an LG TV’s ability to record sound around it, even when the TV isn’t online.

“We were even able to capture microphone audio while the TV was unplugged from the network. Burke said the audio was reportedly stored locally via plaintext, which could potentially allow the data to be sent to LG [if] the TV connects online again.”

LG told Harding it is only listening for the wake word, but Gamers Nexus was able to capture audio in a variety of different ways because there are so many microphones and, in many cases, so many vulnerabilities in LG’s software. If I had one criticism, it is that Gamers Nexus connects the voice recording and transcription to LG’s ads service — an apparently real-life version of the conspiracy theory. I do not think there is not enough evidence to support that, even as speculation.

The transformation of seemingly every major company into, ultimately, vendors for advertising is an outcome basically everyone hates. The televisions tested by Gamers Nexus are thousands of dollars in Canada. That exchange of money used to be where the relationship ended, assuming the product did not need servicing. Now, though, selling ads on the screen itself prints so much money that it is nearly impossible to buy something different. You cannot vote with your wallet when advertisers are paying more.

⌥ Permalink

Heather Burns:

I cannot tell you how much it disgusts me that so much of the archival footage from September 11 has now been age-gated and restricted as “mature” or “sensitive or adult” content, as in this example from this Reddit film archive.

It’s history. It’s archival footage of history. It’s a thing that happened.

If you were in school in 2001, there were probably televisions playing live news coverage in your classroom or in the hallways. There were in mine.

I am with Burns, and with the greatest of respect — she was there that day. But that particular subreddit is an unfortunate example as, while I do not think it should be age-gated, it should be moderated far better. So many of the users there are not treating it as an archive or a way of learning, but as a puzzle where they can play detective.

⌥ Permalink

Aaron Vegh and Ben Rice McCarthy have been steadily updating Indigo since they launched it in May. In the latest version, you can now swipe between different filtered timelines, the scroll position is synced between devices, and there is a new lower price point if you use it exclusively for a single network.

As for me, the one big thing I wanted at launch was the ability to easily switch between multiple accounts, and this was added some time ago. It works exactly as you might expect: you can group sets of your own accounts — my personal Bluesky and personal Mastodon are one set, while my Pixel Envy accounts are in another — and just toggle the two sets.

This is one of my very favourite iOS apps. Just a great piece of software made by some great people.

⌥ Permalink

Brian Krebs:

A new identity theft service launched on the dark web this week is selling digital scans of more than 153 million drivers licenses from people in the United States and Canada. Based on interviews with individuals whose licenses are available for purchase on this service, it appears to be siphoning images collected by a widely-used identity verification company based in Louisiana. KrebsOnSecurity also has learned that the New Orleans field office of the Federal Bureau of Investigation (FBI) today launched an official inquiry into the source of the images.

Mike Masnick, of Techdirt, reports the company in question is IDScan:

You cannot do age or identity verification safely. It always creates some sort of record and that set of records will always become a target. That’s what happened here. And it’s what will happen with any such systems.

Dan Gillmor on Bluesky:

If you support “age verification” online, you are supporting a system that GUARANTEES privacy meltdowns — endangering all of us — because the giant databases of scanned IDs are perpetually hacked by criminals who sell private data.

I am opposed to identity verification, but this argument is not particularly effective for me because there are incredibly low-risk solutions. In Canada, for example, we have an interbank service called Interac that offers an identity verification service. I am not naïve, but I would entirely trust this bank-based system to verify me on a regular basis. In fact, I already do — like many Canadians, I use my banking information to log into government websites. Maybe this relatively safe proxy for a centralized identification system is unique to Canada.

I imagine this argument lands fairly well for lots of people elsewhere, however. The rapid introduction of age verification laws has produced a market for these businesses, but handing your identifying information to some random third-party should terrify you, as it is exactly the behaviour any security expert warns against doing. You have no idea who is able to access that scan of your driver’s license or what they can do with it. And, as it turns out, neither do some of these companies, either.

⌥ Permalink

I skipped August's Recently because this summer has been relentless, mostly in the positive sense. July and August were filled with bike rides, 5K races, friends in town, life changes and logistics.

For a friend's birthday, we rode the rickety Coney Island Cyclone, which was faster and more intense than I remembered it. But it's been 18 years since the last injury on the coaster, and the rider was partly at fault: good enough for me.

I rode my first proper randonneuring ride: 200km (124mi) to all of the beaches around Brooklyn. It was pretty challenging, as you'd expect, but I didn't 'find my limit' for endurance, but I did find my limit in terms of how many gels & ride-nutrition packets I could consume before sugar becomes repulsive. I do still heartily endorse maple syrup packets for ride nutrition: compared to the space-age fuels that all taste like fruit or coffee, it's an identifiable substance that tastes like you'd expect.

And, not pictured, but I ran a bunch of 5Ks. Haven't hit my goal this year of going sub-20, but got close in the last one - 20:15. I'm not giving up yet, because the weather is getting better and Brooklyn has a lot of 5K options.

Listening

Duffy x Uhlmann has been in my rotation for a year now and the new album has all the things I like from the combination. This and Shrunken Elvis go along well together and both have been hits at dinner parties.

This album from Frances Quinlan, the frontperson of Hop Along, went under the radar, but it's so good. I've also been playing Hop Along's send album Get Disowned. I really like everything that this band has put out. It's a product of Philadelphia but I hear some of the elements that drew me to DC post-punk in the turn-on-a-dime songwriting.

Watching

This video, 'The surprising truth about what motivates us', is kind of an artifact of the 'Obamaverse', as one of my friends would put it. It's by Daniel Pink, a writer-speaker-self-help kind of guy, and it's accompanied by one of those realtime hand-illustrations. There are lots of reasons to tune it out as YouTube filler.

Still, I thought it was pretty cool for two reasons: first, it connected directly with Andrew Kelley's video, which is the next one. And it connects with my feelings about incentives and how most simple incentives are counterproductive. And second, the drawing is real. I'm so used to the artificial version of this visual style that it took a minutes to realize that the it wasn't AI or some 'drawing hand' style of video essay, it was a real person making quite nice illustrations: specifically Andrew Park.

Found myself nodding along to every bit of this interview with Andrew Kelley. His values and motivation are so nice to encounter in this current phase of tech. He's kind of a role model, which is something that I used to find trite in my early career but I'm coming around to. As Adam Neely pointed out in his video, a lot of the people heavily using Suno (an AI music tool) could not cite any role models, and defaulted to a sort of enclosed, self-centered stance. I see this with engineering too: it's hard to discern what values or talents the people at 'the forefront of the industry' have that I want. Finding personal qualities as something to imitate instead of desires and possessions (cue the segue into Girard's Memetic theory) seems like a better way to live.

Reading

What strengthens a relationship? Almost always, it is personal investment. Perhaps an AI agent might be better at predicting what my father will want for his birthday than I am, but it definitionally cannot give him my time and consideration. Love is not just a feeling; it is a way of paying attention.

Elizabeth Lopatto in The Verge picks apart Mark Zuckerberg's weird, sad view of humanity. It's worth a read even just for the incredible opening anecdote.

Home of the titular 100 foot waves. Which existed when I was a kid of course, but the larger world didn’t know about them yet. They were our secret. My grandfather and I would walk to the lighthouse and watch the waves hitting against the cliffs and he’d tell me that these were the biggest waves in the world. I didn’t believe him, of course. I thought it was standard grandpa hyperbole. But I also didn’t want to believe that I was seeing the biggest of anything. Not yet. I wanted the really great things to be in the future. Something to aim for. Which means I missed out on some great things that were right in front of me.

Mike Monteiro's newsletter is always beautiful and melancholy, and this is a really nice edition of it.

I read and highlighted a few other articles but they were all saying things about AI, mostly about how it makes the world worse and makes people feel bad. I won't add those to the pile because I don't want that kind of content to dominate what I mention, even if it does dominate what I read.

Elsewhere

For the Val Town blog, I wrote about our experience with the bug bounty program. It's been a very interesting experience. Val Town has a pretty difficult security surface: in contrast to Mapbox, which was mostly a read-only API, it's a very read-write system with many access controls. At the center of the product is a sandbox that we want to keep secure. And because we're living in the AI age, a single bit of functionality can be exposed via REST API, React Router loaders, tRPC, and MCP. Building a model layer and a shared authorization layer - modeled after Oso - has helped a bit.

But the reports still come in. It's pretty wild how AI factors into it: I'd say that 100% of the vulnerability reports used AI to write the report itself. From the best reporters, the vulnerabilities come with screen recordings and evidence, and it's clearly not all-robot. But - I suspect that other people manning help desks & bug bounty programs know this experience - it's jarring to go three replies into an email conversation with someone and they suddenly stop using an LLM to write their emails and the voice completely changes, from hyper-literate to abbreviated and misspelled.

I also wrote about our new authentication scheme that makes vals require login and makes it possible to manage permissions to val applications, not just their code. It was pretty fun to implement, and the sixth or seventh time working with HMAC schemes I'm starting to get a natural grasp on them.

Logic for Programmers has now been out for a month! To celebrate, I'm releasing the entire second chapter, "A Crash Course in Logic", for free on my blog. Read it here!

What happens when people who are working to promote, preserve, and develop different languages come together?

They share experiences, learn from one another, discuss the challenges they face, and look for practical ways to strengthen their communities.

That is what the Language Diversity Conference 2026 is about.

From 2 to 4 October 2026, Accra, Ghana, will host the inaugural Language Diversity Conference under the theme

“Strengthening Language Communities in the Open Knowledge Movement.”

The conference will bring together language activists, Wikimedia contributors, researchers, educators, community organizers, and others working to ensure that languages that are often underrepresented online have a place in the digital world.

Language is not only about communication. It carries our history, culture, traditions, and knowledge. Yet many Indigenous and minoritised languages still have limited representation on the internet. This makes it important for language communities to take an active role in documenting their languages and making knowledge available in them.

The conference will provide an opportunity to discuss some of these issues, including language documentation, digital accessibility, Wikimedia projects, artificial intelligence, speech-to-text technologies, language rights, partnerships, and community-led approaches to language development.

A community spotlight: Dagbani Wikimedians

The Dagbani Wikimedians User Group (DWUG) is one of the communities contributing to this conversation.

Over the years, Dagbani Wikimedians have worked to increase the presence of Dagbani and other Ghanaian languages in the digital space through Wikipedia, Wikidata, Wikimedia Commons, training programs, edit-a-thons, Wiki Hubs, and other community activities.

The journey has also involved supporting the growth of other language communities. These experiences have shown the importance of local people taking ownership of the development and documentation of their languages.

At the Language Diversity Conference, members of the Dagbani Wikimedians community will share some of these experiences.

Among the sessions is “Language Rights Online: Preserving Dagbani Heritage,” which will look at the development of the Dagbani Wikimedia community, its work in the Wikipedia Incubator, community mobilization, Wiki Hubs, and the use of tools such as Mozilla Common Voice and Spell4Wiki. The Dagbanli Dictionary: Building a Sovereign Digital Language Infrastructure for an African Language session will also highlight efforts to build an independent digital foundation for Dagbanli by making the language’s words, meanings, and linguistic knowledge more accessible in the digital space. 

There will also be a session on Sopala, an offline learning initiative designed to support Dagbani education in communities where internet access can be a challenge.

For DWUG, this conference is, therefore, an opportunity to share what we have learned, listen to other language communities, and explore ways of working together.

Come and meet the people working to give more languages a stronger presence in the digital world.

An adaptation of Polis for the Wikimedia ecosystem, currently a live prototype. This post continues after ‘We need to innovate with Wikimedia decision-making‘. This prototype is part of the Next25 initiative attempt to arrive at more constructive attitude towards decision making.

We have a lot of decisions to make in the next year in our movement. As the number of visitors to Wikipedia is dropping, the internet around us is changing rapidly. This means that as a movement, we will have some hard decisions to make where we go. As WMF Executive Director Bernadette Meehan summarized: we need to evolve, but in ways that strengthen our mission. 

As I laid out in my last post, we need better processes to make decisions than the Request for Comments. This is true for deciding on our outdated policies in our many communities (how do we welcome new contributors, how do we handle AI, what is the bureaucracy that we can handle as a less-active community?) and on directions for our movement (what should be our priorities? Are there opportunities for new projects? What technology should we develop, and what should it look like?).

Today, I would like to introduce one prototype that could play a role in a more constructive approach to making decisions. 

Often, our current mechanisms focus on a single, large-form proposal, bringing out our critical thinking. There are many ways to improve our decision making, and I would like to see this as a part of a larger effort to experiment with that. Don’t assume that a single process can serve is in every decision, but rather pick the tool that fits the task. 

In my last post, I outlined some recommendations for how we could improve our processes. None of this is hypothetical. The civic-tech world has spent the last decade building and testing exactly these ideas — most famously in Taiwan, where the vTaiwan process used a tool called Polis to find unexpected common ground on regulating Uber.

How it works

This brings me to the prototype we have been working on: Proto, an adaptation of Polis for the Wikimedia ecosystem, running on Toolforge with your regular Wikimedia login. Polis (for example: pol.is) is a well-known innovation that has been used in many citizen engagements, including vTaiwan. We have built on top of it to make it fit a wiki’s needs better: to be more encouraging about editing the statements, and to add argument mapping.

The process consists of three phases: 

• Respond to atomic statements and find the statements that the community already agrees on, and the ones that divide.
• Collect relevant arguments for the most important statements. 
• Collect an informed opinion mapping by showing the most important statements with their selected arguments.

Separate the phases

A system like Proto can be used to collect valuable input on a complex topic, before solidifying a draft. The phase separation gives ideation, refinement, and preference measurement each their own home. With the outcomes of this process, you could organize a more informed voting process on-wiki.

Make participation accessible (atomic statements)

The core mechanic is deliberately simple. A conversation (for example, a new policy dealing with blocking temporary accounts) starts with a question and a set of short, atomic statements — one claim each (“A temporary account should be warned at least once before being blocked for vandalism”). 

Participants respond agree, disagree, or pass on each statement, and can submit statements of their own: to improve the phrasing of existing statements, or to fill a gap. That’s the whole interaction: no threads, no replies, no walls of text to catch up on. Joining a conversation on day twelve is exactly as easy as joining on day one; if you return later, you simply weigh in on whatever new statements have appeared since. Rapid responding make it easier to participate. 

Notably, the same design serves both ends of the scale: a small group gets an honest reading of the people it could never get to write talk page comments, while a large wiki gets a way to digest thousands of voices that no closer could. And the process is composable — an organizer can stop after the opinion mapping and already walk away with something valuable.

(A practical challenge will be how to connect the process to the wiki – notifications and talk page messages are likely the best bet.)

Decouple opinions from identity

Behind the scenes, the responses build an opinion map. Statistical clustering reveals the groups of participants who tend to respond alike — and, more importantly, the statements that are supported across those groups. Instead of amplifying the sharpest disagreement the way a threaded discussion does, the system is designed to surface hidden consensus. 

Your response is private while the conversation runs; opinions become visible as clusters, not as named individuals to follow or oppose. A faction that shows up to swing the outcome doesn’t silently shift a headcount — it appears on the map as exactly what it is. Private responses remove herding and anchoring — and let people disagree with the regulars without it becoming personal. 

Collect the arguments

Where Proto goes beyond standard Polis is in what happens after the opinion mapping. The organizer can curate a small set of the most informative statements — the points of strong consensus and the genuine fault lines — and open an argument layer: participants write and rank short pro and con arguments for each featured statement. No threading here either; just the community’s best reasoning on both sides, sorted by usefulness.

And as an optional final phase, an informed opinion poll: a fresh round on just the featured statements, with the strongest arguments displayed alongside. Comparing the before and after even lets us see whether exposure to arguments actually changed minds — something our current processes are incapable of measuring. Rather than just focusing on whether we like the proposal, consider also what others might think of as relevant arguments. 

One thing Proto deliberately is not: a vote. While everything may feel like voting, it is much more about mapping where the agreement and disagreement are, and collecting arguments in an equitable manner. From there, we have a solid foundation to draft policies from, and to figure out how to compromise between the views on the table. It may also expose where more conversation is needed, and where we already agree and only a vocal minority pushes back. The report from the tool is meant to inform the final step of the policy process on-wiki — because at that point, a Request for Comments or an opinion poll may actually make sense.

An invitation

This is one way of doing consent-building differently, but the underlying principles stay the same. I would invite you to try it, and to think about it from a constructive perspective: how would you improve it? Let us know, on metawiki.

The prototype is live at proto.wiki , and what it needs now is reality: some groups willing to run a real consultation on a real question. If your wiki, WikiProject, or group has a policy discussion that has been stuck for years — and whose hasn’t? — I would love to talk.

Twenty-five years ago we built an encyclopedia. I refuse to believe that the way we made decisions in 2004 is the best we can do in 2026. The editors who join us next year deserve rules they can actually read, trust — and change.

From 23 to 27 August 2026, Wikimedia Tunisia User Group, in partnership with the Association of Culture and Science in Metlaoui, organized Wikipedia Camp Tunisia 2026 at the Camping and Holiday Center in Bkalta, Tunisia.

The camp was the realization of a dream that had been growing within the Tunisian Wikimedia community for several years. Its main inspiration came from the successful WikiCamp Armenia, a pioneering project that brought Wikimedia contributors together through learning, collaboration, and community-building activities. The Armenian experience demonstrated how a camp could create a strong environment for both newcomers and experienced Wikimedians to learn from one another and contribute together.

Inspired by this model, Wikimedia Tunisia began developing the idea of creating a similar experience in Tunisia.

From workshops to the camp

The camp was developed in cooperation with the Scouts of Tunisia. Before the camp, a series of workshops was organized for young participants from different Scout groups. These workshops introduced participants to Wikipedia and other Wikimedia projects and encouraged them to discover how they could contribute to free knowledge.

Following these activities, participants who showed the strongest interest and engagement were selected to take part in Wikipedia Camp Tunisia 2026.

This approach allowed the camp to bring together young people who had already gained their first experience with Wikimedia projects while creating an environment where they could continue learning and improving their skills.

Five days dedicated to free knowledge

Throughout the five-day camp, participants took part in a wide range of activities focused on Wikimedia projects.

Editing sessions were organized on Wikipedia, Wikidata, and Wikimedia Commons, allowing participants to put their newly acquired knowledge into practice. They worked on improving articles, adding and enriching structured data, uploading and documenting media, and discovering the links between the different Wikimedia projects.

One of the most important outcomes was the creation of new articles on Arabic Wikipedia about Tunisia, with a particular focus on sports-related topics. Participants also worked on Wikidata, creating new items and improving information about books and authors using data from the National Library of Tunisia database.

These activities showed participants how different Wikimedia projects can work together: information researched for Wikipedia can be structured through Wikidata, while institutional sources such as library databases can help improve the quality and coverage of Wikimedia’s structured knowledge.

Learning from Wikimedia Libya

The participation of two representatives from Wikimedia Libya was one of the highlights of the camp. Their presence created an opportunity for exchange between Wikimedia communities in the region and added valuable experience to the program.

They organized a presentation about Wikidata, Wikidata items, and the role of Wikidata in relation to Wikipedia. The session helped participants understand that Wikidata is not simply a database, but an important part of the Wikimedia ecosystem that supports and connects information across different Wikimedia projects.

A second presentation focused on user rights on Arabic Wikipedia. The Libyan Wikimedians explained the different rights available to contributors, their importance, and how participants could develop their editing experience and become more involved in the community.

The session also encouraged participants to take the next step in their Wikimedia journey by requesting appropriate user rights on Arabic Wikipedia. By the end of the camp, several participants had already submitted requests for user rights, demonstrating the practical impact of the training and the motivation created during the event.

Building the future of the Wikimedia movement

Wikipedia Camp Tunisia 2026 was more than a five-day event. It was an experiment in community development and a step toward creating a new generation of Tunisian Wikimedians.

The journey that began with inspiration from Armenia and continued through workshops with the Scouts of Tunisia has now resulted in a Tunisian WikiCamp of its own. The experience demonstrated the power of bringing young contributors together, giving them the tools to contribute, and connecting them with experienced members of the Wikimedia movement.

For Wikimedia Tunisia, the camp represents not an endpoint, but the beginning of a longer journey. The participants now have the opportunity to continue contributing to Wikipedia, Wikidata, Wikimedia Commons, and other Wikimedia projects—and to share what they have learned with others.

From Armenia to Tunisia, the idea of WikiCamp has travelled across communities, adapted to a new context, and created new opportunities for free knowledge and collaboration in Tunisia.

Meetups (Kopdar) are always highly anticipated moments for Wikimedia volunteers. Aside from being a space to connect, meetups often spark extraordinary collaborations. Not too long ago, I had the opportunity to participate in a very unique meetup event titled WikiRempah in Bandung, West Java, Indonesia on Sunday, August 9, 2026.

True to its name, this event highlighted the richness of spices and culture, but with an interesting cross-cultural twist: we documented the culinary and cultural heritage of Eastern Indonesia, yet channeled it into Sundanese-language Wikimedia projects and other global platforms. This activity was not just about editing; it was a contribution marathon involving four Wikimedia projects simultaneously.

Crafting Recipes and Visualizing Flavors

The first session, which was highly memorable for me, was culinary documentation. Eastern Indonesian cuisine, famous for its rich herbs and spices, was the main star. We went to an authentic Eastern Indonesian restaurant, photographed, and documented these signature dishes, then uploaded the best visuals to the Wikimedia Commons repository so they could be freely used by anyone. One example I uploaded was a photo of a Ayam Woku Khas Sulawesi.

Pijri Paijar, Ayam Woku khas Sulawesi, Wikimedia Commons, CC BY-SA 4.0

We then moved on to the Sundanese Wikibooks project, which is currently still in the incubator. There, we worked together to write down these authentic Eastern recipes in Sundanese. Translating the spice measurements, cooking methods, and ingredient names into our mother tongue offered a unique sensation, as if we were serving Eastern Indonesian flavors right in the kitchens of the Sundanese people. On that occasion, I tried writing down the recipe for grilled skipjack tuna (ikan cakalang bakar).

Bridging Cultural Heritage to the Sundanese Wikipedia

After being satisfied with crafting recipes, the meetup’s focus shifted from the dining table to historical sites. Eastern Indonesia has an abundance of cultural heritage sites whose histories are vital to the civilization of the archipelago. Unfortunately, information regarding these sites is still very minimal, especially in regional languages.

This is where we contributed through the Sundanese Wikipedia. We wrote and expanded new articles discussing cultural heritage in the eastern part of Indonesia. Writing about geographically distant topics in Sundanese is a tangible manifestation of Wikimedia’s mission: freeing knowledge for everyone, in the language closest to them. I wrote several titles, including Bénténg Middelburg (Middelburg Fort), Bénténg Amsterdam (Amsterdam Fort), and Bénténg Barneveld (Barneveld Fort).

Bringing the Dictionary to Life with the Sounds of Spices

The highlight of this WikiRempah Meetup, and in my opinion the most interactive activity, was the voice recording session. Since the overarching theme was “Spices,” we compiled a long list of vocabulary related to spices and typical Indonesian cooking ingredients.

Using the Lingua Libre software, we took turns recording the pronunciation of this spice vocabulary one by one. The atmosphere became very lively and fun as we had to ensure our Sundanese articulation sounded clear and precise. After the recording process was complete, the audio files were not just left there. We immediately embedded them into the existing entries in the Sundanese Wiktionary.

Now, the digital dictionary does not only present text but also “speaks.” Anyone looking for entries about spices in the Sundanese Wiktionary can instantly hear how the word is pronounced orally by a native speaker. Some of the recorded words included lauk (fish), sagu (sago), and sampeu (cassava).

A Comprehensive Contribution Experience

Oceanmuse, Kopdar WikiRempah Bandung, Wikimedia Commons, CC-BY 4.0

Participating in the WikiRempah Meetup gave me a new perspective on how the Wikimedia ecosystem works. From a culinary dish and a historical story, we could unpack them into photos on Wikimedia Commons, recipes on Wikibooks, encyclopedic articles on Wikipedia, to audio pronunciation recordings on Wiktionary and Lingua Libre.

This activity proves that contributing to the Wikimedia movement is never boring. There is always room for creativity, cross-cultural collaboration, and new ways to preserve the richness of the archipelago in the digital realm. Endless thanks to the organizing committee for this incredibly valuable and memorable opportunity. My hope is that a multi-contribution meetup model like WikiRempah can inspire other communities to keep exploring!

• Wikidata MCP Server
• Wikidata Embedding Project
• Wikidata Vector Database

2026. Aug. 9th (Saturday)

Under the storm of Typhoon Dolphin, COSCUP 2026 commenced, bringing partners from around the world together to celebrate the value of open technology. The Conference for Open Source Coders, Users & Promoters (COSCUP) is one of the biggest open source events in Taiwan. This year, one topic dominated the community: Artificial Intelligence, or Agent in the concurrent stage.

This year, our joint track of Wikidata and OpenStreetMap not only showcased how our respective platforms contribute to the open community, but also explored how open data and open technology can support the rapidly developing world of artificial intelligence.

One highlight was our presentation with our partner from Wikimedia Deutschland, focusing on the Wikidata MCP Server and the Wikidata Embedding Project.

Artificial Intelligence has come to the stage like a storm and quickly become one of the most discussed technologies across every industry. No matter which side of the fence you are on, one thing is the same: we want to know more about it and how it may affect our lives.

However, much of the discussion throughout the internet is filled with empty buzzwords and misinformation from companies with various interests. The problem is not necessarily malicious intent but the fact that people, including the sources they are quoting, simply do not have enough access to the information they need to understand the technology they are discussing.

Just like previous technology ‘booms’, such as cryptocurrency and NFTs, Artificial Intelligence, and now the so-called Agents, is a market phenomenon of attention crowding around something new and potentially profitable, and thus money floods in. However, unlike its predecessors, AI is ‘real’ in a sense that it has the potential to reshape, and possibly damage, many systems of trust that we humans have built through centuries of mutual understanding and respect for ourselves and others.

AI Agents are not inherently disruptive or destructive; it is how we use them that is causing all the trouble. If we want the technology to become a tool for a better future, we first need to understand how the tool works.

The Problem with LLMs

A Large Language Model, or LLM, at its simplest, is an algorithm that predicts words based on the material it has been provided. Its strength comes from recognizing patterns in a given language and mimicking the ‘natural language’ according to the probability of words appearing together. But this also reveals its fundamental limitation.

All it captures is the distribution of words appearing in languages, not necessarily the why and how behind them. It can produce an answer that looks and sounds correct without having any understanding of whether the information corresponds to reality or is actually true. Hallucinations, gibberish, and blatantly false answers are ultimately extensions of this limitation.

The man in the box never understands Chinese; it just appears to.

This does not mean we need the LLM to become a genie in a lamp that knows everything. The bottleneck is never the generation of more knowledge. It is discovering, retrieving, and acting upon the right information. To achieve this, there is no reason to reinvent the wheel of synthesizing the generation of natural knowledge; instead, what we need is to have the preexisting systems align properly.

And this is where Wikidata comes into play.

A Beacon for AI Agents

Wikidata is a Knowledge Graph: a machine-readable database where information is connected through controlled and structured relationships. It is designed to work across languages, interfaces, databases, services, and users — human and machine.

More importantly, its information is human-pruned and anchored in reality. It can provide explicit and exhaustive relationships, traceable sources, and continuously updated information. This makes Wikidata particularly valuable to LLMs. Rather than leaving an AI Agent to decide whether a string of text is factual or fictional, it can reference Wikidata for additional information.

LLMs excel at semantics, but that strength is inherently imprecise. A Knowledge Graph provides the controlled structure underneath it. Instead of asking an Agent to know everything, we can let it discover relevant information through the web of knowledge and crawl through information more efficiently than we humans could.

This is where the Wikidata Embedding Project aims to establish its influence.

The project prepares a vector database from a Wikidata dump and an MCP Server that allows AI Agents to access knowledge residing in it through a standardized and LLM-friendly interface. It combines the fuzzy discovery capabilities of AI with the structured and human-pruned knowledge of Wikidata.

The advantage is not only accuracy. LLMs are trained primarily on mainstream data, with limited information from less represented communities. Wikidata can provide access to knowledge beyond an Agent’s original training, including information from smaller communities that would otherwise be extremely difficult for an AI to discover and access.

By equipping an Agent with Wikidata, it is like a sailor on a misty sea finally seeing the light of a beacon. The Agent can still navigate on its own, but it now has a solid reference point so it won’t get lost as long as it can see the light.

The Current and the Future

Wikidata is still one of the youngest WikiProjects. Though volunteers around the globe are working diligently, there are still gaps in the Knowledge Graph. If we want Wikidata to become the backbone to power the next generation of Artificial Intelligence, we still have a long journey ahead of us.

The emergence of AI Agents brings as many opportunities as it does challenges. For Wikidata Taiwan, our work is becoming increasingly relevant to making sure Taiwan’s audience and data won’t get left behind in the ever-changing landscape of Artificial Intelligence.

The challenge is not simply to make AI more powerful. It is to make sure that the knowledge it discovers is reliable, traceable, and based on reality, and most importantly, fair.

I’m Berkay, also User:Piyanist on Wikimedia projects. I have been contributing to Wikimedia projects since 2020. I’m member of the Wikimedians of Turkic Languages User Group, CEE Youth Group, and WikiPortraits.

My Wikimedia activities mainly focus on edits and corrections to Wikipedia pages related to my area of interest, adding cited informations, photography, Commons, youth-related projects, and topics related to music and culture.

In this report, I will share my experiences, observations, and highlights from Wikimania 2026. I will also describe the events I enjoyed the most during the conference.

Wikimania 2026 in Paris was my first Wikimania. I had attended other Wikimedia events before, but this was my first time experiencing Wikimania itself. During the conference, I joined sessions related to my interests, volunteered as a photographer at some activities, met new people and spent time again with friends I already knew from the CEE community. Some of the sessions also gave me new ideas about how I want to contribute in the future, especially through photography, youth activities and music.

The time between sessions was also an important part of the conference for me. During coffee breaks, meals and while moving around the venue, I met Wikimedians from many different countries and had conversations that I probably would not have had through scheduled sessions alone.

And of course there was also no shortage of small things to bring home. I collected stickers, pens, postcards, badges, notepads and other materials from different Wikimedia communities and projects, along with considerably less permanent souvenirs in the form of snacks and food. I kept some of these for myself, but I also plan to use many of the printed materials, stickers, and other items as materials for events we organize in the future. They can help me create a more visual stand where people can see different Wikimedia communities, projects and events instead of only hearing about them.

During Wikimania, I moved between many different sessions and activities. Sometimes I stayed until the end of a session, while at other times I left to see something else. Below are the sessions that I enjoyed the most or that gave me ideas about things I may want to work on in the future.

AI, Wikidata and the opening ceremony

One of the first sessions I attended was “If AI becomes the first web interface, how can the Wikimedia movement adapt? Communities’ levers.”

The discussion focused on how different Wikipedia communities are dealing with generative AI. Matthias Schindler shared an example about a tool made to check ISBN numbers in Wikipedia references. The tool helped identify articles containing fake references generated by AI. Some of the text looked normal, but the books used as sources did not actually exist.

The speakers also discussed how language communities can experience AI differently. One point I found interesting was that language models can flatten different perspectives instead of understanding the context of each language and community. The discussion was not simply about accepting or rejecting AI. It was also about understanding where these tools fail and keeping human review in the process.

After that, I attended Philippe Saadé’s “Wikidata MCP: Grounded SPARQL Query Generation.”

He demonstrated how a language model can make a basic mistake while still producing something that looks technically convincing. In one example, a model trying to find female physicists who had won the Nobel Prize in Physics initially selected the QID of a dog breed instead of the Nobel Prize.

The workflow presented in the session separated discovery, structure inspection, SPARQL generation and validation. This made it easier to see where a mistake happened instead of only looking at the final answer.

Later, I attended the opening ceremony, which also celebrated Wikipedia’s 25th anniversary.

UNESCO’s Mariya Gabriel spoke about knowledge as a public good and about the responsibility of AI systems that depend on open knowledge. Wikimedia Foundation CEO Bernadette Meehan talked about the people behind Wikimedia. She reminded us that readers normally only see the finished article. They do not see all the discussions, photos, translations, corrections and other work that happens behind it.

Jimmy Wales talked about assuming good faith and making newcomers feel welcome. He also spoke about freedom, equity and the importance of keeping Wikipedia reliable. He also discussed misinformation bubbles and unequal access to information. One point that stayed with me was his description of Wikipedia’s role in the age of AI: its value is not simply providing the fastest answer, but providing information that can be checked and verified.

He also described Wikipedia as something built gradually through edits, lines of code and discussions rather than around one person.

The ceremony continued with the Wikimedian of the Year awards. I photographed all of the award recipients and later uploaded my photographs to Wikimedia Commons. You can access these photos from the following Commons category

A morning inside the Sainte-Geneviève Library

The next morning, I joined an organised visit to the Bibliothèque Sainte-Geneviève.

We met at the Wikimania venue early in the morning, received public transport tickets and travelled to the library together. Before entering, I walked around the Panthéon area with other photographers and took street and architectural photographs.

Inside the library, we visited the main reading room as well as storage and archive areas normally closed to the public. In the reading room, Pauline Rivière showed us the old book lift: a big wheel and rope system that once carried books up from the storage in a small box. She told us that the architect did not want staff walking around the reading room, because he found it not aesthetic, so the librarians asked the minister for extra stairs to make their work easier. In the storage areas, the books are shelved in little “waves”, so that small books do not get lost between two big ones. The old card catalogues are also still in use, because some records are not in the online catalogue. One massive old door even hides its lock; only a small trick of the architect opens it.

In the lower levels, we saw large collections of books, archive materials and older reproduction formats. Pauline showed us the microfilm archive: before digitisation, books and newspapers were copied on small reels, and readers can still use them today. She also explained how physical collections are digitised and made available through the Genovefa digital library. The library holds around two million documents in three buildings. When a reader requests a book, the request is printed in the stacks and the book is sent up with a small lift, in about twenty minutes. The rolling shelves stay only on the lowest level, because anywhere else their weight would harm the foundations of the building.

We also saw historical documents, rare books and materials connected to preservation and digitisation. For me, the most impressive piece was a planetary clock made in southern Germany at the end of the 16th century – one of the very few surviving in the world, and still working, showing the movements of the planets. The photos I took of the library and the archive area during these events are in this Commons category.

Pauline Rivière explained how physical collections are digitised and made available through the Genovefa digital library. We also saw historical documents, rare books and materials connected to preservation and digitisation.

Photo: Microfilm boxes of the French news magazine L’Express (1996–2014) in the microfilm storage of the Bibliothèque Sainte-Geneviève, Paris, during a Wikimania 2026 guided visit. Piyanist | CC BY-SA 4.0

Photography and WikiPortraits

“Behind the Lens: WikiPortraits on a Global Scale”, presented by Kevin Payravi, Jennifer 8. Lee and other WikiPortraits contributors.

The problem behind WikiPortraits is simple: many Wikipedia biographies still have missing or poor photographs, but taking a good photograph of a notable person often requires being at the right event and having media access. WikiPortraits helps photographers attend and cover these kinds of events, so that useful photographs can later be uploaded under free licences and used across Wikimedia projects.

WikiPortraits now has more than 200 photographers from over 40 countries, and together they have uploaded more than 95,000 photographs. The group covered over 180 events in 2025, and another 169 events had already been covered in 2026 by the time Wikimania took place. These photographs are also widely used, receiving around 400 to 500 million views every month.

One part of the model was especially relevant to me. WikiPortraits encourages photographers to begin with events in their own area. As they build experience and a portfolio of Wikimedia work, the group can help with accreditation for larger international events. For some major coverage, there can also be support with travel or accommodation.

This connected closely with my own photography work. In Türkiye, I already photograph musicians, artists, concerts and other events. Until now, these photographs were usually produced for artists, organisers or their own media use, and I had not really connected this part of my photography with Wikimedia Commons.

The session made that connection much clearer for me. When licensing and other conditions are suitable, I hope to make some of my future artist and concert photography useful on Commons as well.

For me, this is a practical way to connect photography work I already do with my Wikimedia activities instead of treating them as completely separate things.

And after that session, I joined WikiPortraits. For me, this was one of the sessions I enjoyed most at Wikimania 2026.

Indigenous knowledge and responsible sharing

“Respecting Indigenous Knowledge in Open Spaces.”

The panel discussed Indigenous Cultural and Intellectual Property and Indigenous Data Sovereignty. The speakers explained why open knowledge is not only about copyright or whether something can technically be uploaded.

Cultural context, consent and the communities connected to that knowledge also matter.

The Wikimedia Australia team presented a guide that helps contributors think about culturally sensitive material and whether a source is culturally appropriate. One of the practical ideas was to ask not only whether material can be uploaded, but whether uploading it is the responsible choice.

Thérèse Ottawa also spoke about the Atikamekw-language Wikipedia project in Canada and its connection with education and language revitalisation. The project included work with younger people and schools, helping the language remain active in a digital environment.

This session connected with what I had seen at the library. Digitising cultural material can help more people access it, but we also need to think about how that material is shared.

CEE Youth and ideas for future activities

Youth participation was another part of my Wikimania programme.

“Bring in the Youth! Discussing Ways to Support Youth on a Regional Level.” Participants shared examples from different regions and talked about mentoring, universities, informal activities and ways to support young contributors who may not have an active local community around them.

One example I liked was the use of simple social activities such as board game meetings. These meetings helped young Wikimedians get to know each other, and some later developed projects together. The discussion also looked at the role of regional groups in connecting young people who may otherwise feel isolated in their local Wikimedia community.

Later, I joined the CEE Youth session and workshop.

The CEE Youth social media working group currently has eight volunteers from different countries. The team meets once a month, talks through Telegram and works with a shared content calendar. The speakers were also quite open about volunteer burnout. Sometimes volunteers simply need to take a break from their responsibilities for some time.

During the workshop, we talked about who we actually want to reach through social media and what could make the content more interesting. We also discussed how the work could be divided between more people instead of relying too much on one person.

Meeting WikiOrchestra

At the end of the day, I went to the WikiOrchestra rehearsal and met the conductor, Lukas Mezger

I told him that I am playing the piano and composing musics. I also mentioned that I had composed the opening music for the Wikimedia CEE Meeting 2024 in Istanbul.

There is also a separate diff post on this topic here:

We talked with Lukas about music at Wikimedia events and possible future collaborations. I had discovered the information about the closing choir late, but after our conversation Lukas invited me to join.

Manuscripts and the closing ceremony

On the final day, I attended “From Wikilontar to Wikisami: How Regional Partnerships Power Community-Led Growth in Bali”, presented by Carma Citrawati and Sakti Hendra Pramudya.

The session explained how work around Balinese manuscripts developed through local community participation and wider partnerships. The discussion was not only about scanning documents. It also focused on trust, local leadership and giving communities a role in decisions about their own cultural heritage.

For me, this connected naturally with both the Sainte-Geneviève Library visit and the Indigenous knowledge panel. All three showed different sides of digitisation and cultural heritage.

After the session, I studied the score for “Oh, Wikipédia!” and joined the WikiOrchestra and choir rehearsal. I spent hours working on the music, pronunciation, stage positions and performance.

Near the end of the closing ceremony, the WikiOrchestra and choir went on stage and we performed the song together.

Performing with the choir at the main amphitheater stage, became one of the most memorable parts of my first Wikimania.

Lukas and I also talked about future music projects. I also plan to continue composing original music for Wikimedia events when there is an opportunity. For me, music is not only something separate from Wikimedia activities.

After the ceremony, we joined the Wikimania 2026 group photo.

On the final day of Wikimania, there was another very literal example of how many small contributions can create something much larger. Wikimania participants worked together on a giant version of the Wikipedia Puzzle Globe outside the venue. When it was finished, the puzzle covered around 100 square metres and had 944 pieces. It was one of the largest jigsaw puzzles ever assembled in France. It later became the backdrop for the final group photo.

I also kept one of the puzzle pieces as a souvenir. Together with the badges, postcards, stickers and other things I collected during the week, it will probably become one of the more unusual pieces of my future student club stand.

Looking back at my first Wikimania

My first Wikimania included discussions about AI, photography, cultural heritage, youth activities and music.

But the experience was not limited to the sessions. Meeting Wikimedians from different countries during breaks, seeing the projects people brought with them and simply spending time with other contributors were also important parts of the week.

I returned from Paris with several ideas for what I want to do next, like connect more of my photography with Wikimedia Commons and WikiPortraits, become more active in CEE Youth and youth-related activities, and continue exploring ways to bring music into Wikimedia events.

I also want to continue contributing to the topics I already edit while trying these different forms of participation.

One useful part of Wikimania for me was seeing that some of the things I already do outside Wikimedia can also become Wikimedia contributions.

The week ended with the group photo, the giant Puzzle Globe (Of course, I took one puzzle piece with me.) and the memories and small objects I brought home from Paris. It was a memorable end to my first Wikimania.

Introduction

On 2 September 2026, the new Wikimedia CEE Hub Steering Committee began its term, following several months of nominations, eligibility checks, community engagement and voting. Eight members were elected: four by individual community members and four by CEE affiliates, and two more were later co-opted.

But this election was also a first test of putting the CEE Hub’s governance model into practice. We had the rules; we still needed to build the process around them. That meant establishing an independent Electoral Commission, translating eligibility rules into practice, working with two different electorates, and building Fala, our own voting tool on Wikimedia Toolforge.

From rules to an election

The election had two constituencies: individual CEE community members and CEE affiliates, each electing four members. That made eligibility one of the first practical challenges. Community voters had to meet both Wikimedia activity requirements and demonstrate a connection to the CEE community, while affiliates had to designate representatives to vote on their behalf.

To oversee the process, an independent three-member Electoral Commission was established: Antanana as Chair, Bencemac and Gdarin. The Commission verified candidates and voter eligibility, interpreted the rules when unclear cases appeared, supervised the election, and ultimately certified the results. Hub staff supported the process and its technical implementation, but decisions about eligibility and the application of election rules remained with the Commission.

Thirteen candidates entered the nomination process, and after the eligibility review, twelve were confirmed for the final ballots: six Community candidates and six Affiliate candidates.

Building Fala

One of the biggest practical questions was how to run the vote itself. The election had two different electorates, different eligibility rules, preferential voting, and the possibility that one person could be eligible to vote in both categories.

Instead of adapting the election to the limits of an existing tool, we built Fala on Wikimedia Toolforge. Voters logged in with their Wikimedia accounts, eligibility could be checked through the system, and Community and Affiliate voting were handled separately. Voters could also return and update their ballot before the deadline.

Privacy was part of the design as well. Fala recorded who had voted and when a ballot was submitted or updated, but it did not keep the voter’s candidate selections or rankings linked to their identity.

Building Fala also strengthened the Hub’s ability to run its own governance processes independently. We were not limited to whatever an existing election platform happened to support; we could build the voting process around the rules and needs defined by the CEE Hub itself.

Voting and results

Voting was open from 6 to 20 July 2026. On each ballot, voters could rank as many or as few candidates as they wanted. The candidate ranked first received six points, the second five, and so on, while unranked candidates received no points.

Fifty-two community members voted in the Community election, while 15 CEE affiliates participated through their designated representatives. The Electoral Commission then certified the results, including the  rules used to decide the winner when two or more candidates received the same number of votes in the Affiliate election.

The elected members were Shahen Araboghlian, Magda Barascu, Wojciech Pędzich, Mārtiņš Bruņenieks, Gorana Gomirac, Toni Sant, Klára Joklová and Vera Pelhan. They later co-opted Iglika Ivanova and Armen Mirzoyan, completing the ten-member Steering Committee.

The first meeting of the new Steering Committee took place on 2 September 2026, which marked the official start of its term. That is also when they selected the Chair: Toni Sant and Co-Chair: Vera Pelhan.

What we learned

The clearest lesson was how difficult it can be to turn governance rules into something a voting system can actually verify. Some criteria, such as account age, edit count and block history, could be checked directly from Wikimedia data. Others could not. For example, eligibility through membership in approved CEE mailing lists could not be verified by existing voting tools such as SecurePoll. That problem led directly to Fala: voters log in with their Wikimedia account, verify their email with a one-time code, and the address is checked against a whitelist built from the approved mailing lists. In other words, the election rules themselves shaped the architecture of the tool.

A different lesson came from affiliate participation. Some eligible affiliates never nominated a representative and therefore never voted. That showed us that simply giving affiliates a formal role in governance is not enough; they also need to see the election as part of their own responsibility and engage with the process early enough. After the election, the Electoral Commission followed up directly with non-participating affiliates to understand why they had dropped out. For future elections, earlier contact, clearer expectations and more active follow-up with affiliates will probably matter as much as the voting system itself.

What comes next

The election also made us think more broadly about how the CEE Hub puts its governance into practice. If the Hub is going to make more of its own decisions through formal community processes, it needs infrastructure that can support them. In some cases, the available tools were designed for different purposes; in others, there was simply no dedicated tool for the kind of internal decision-making the Hub needed.

For that reason, we decided to build Iskara, a voting tool for internal CEE Hub decision-making. The aim is to give the Hub greater independence in implementing its governance and a practical way to run its own votes when they are needed. Fala was our first experience of building such infrastructure around a real governance process; Iskara takes that experience further and applies it to the Hub’s internal decision-making more generally.

To learn more about the Steering Committee and internal governance of Wikimedia CEE Hub, check out the dedicated Meta page.

Imagine a world in which every woman on the planet finds a representative and diverse sum of all human knowledge about women+* available on Wikipedia and on the internet. This is what the Celebrate Women campaign has been trying to accomplish since 2022. 

Every March, during Women’s History Month and in celebration of International Women’s Day (March 8th), the Wikimedia community gathers both online and in person to edit and improve the content available on Wikipedia and the other Wikimedia projects about women+. The editing happens as part of edit-a-thons, workshops, training sessions, and many other types of gatherings, which are spread around the world, with events on every continent. 

For 2026, we had one of the biggest campaigns so far. Considering all the gatherings globally, we had 112 events listed on the official page, which were divided into these sections: 

• Global: 21 events
• Africa: 20 events
• Asia: 15 events
• Europe: 38 events
• Latin America: 12 events
• Middle East: 1 event
• North America: 3 events
• Oceania: 2 events

These numbers clearly demonstrate the global impact of this campaign, as well as serve as a testimony to Wikimedia’s reach and potential for collaboration, even when the world seems to be divided. In 2026, our communities were able to not only come together, but also edit and collaborate towards that one single goal.

In the last few years, this purpose was also achieved. By checking the pages for the previous campaigns, we had 59 events in 2025, 100 in 2024, and 121 gatherings in 2022. However, when we also consider the Programs & Events Outreach Dashboard campaigns (see more about this topic below), we have another 144 events in 2022; 137 events in 2024; 171 events in 2025; and 155 events in 2026. 

For 2026, we were able to assemble some statistics. In terms of engagement, considering the P&E Outreach Dashboard and gatherings using Event Registration (a new CampaignEvents feature), we were able to identify that this year’s campaign assembled 207 organizers and 2,658 event participants, which can be divided into:

• Beginner (less than 10 lifetime edits when participation begins): 1,388 users
• Junior (>= 10, <100 edits): 435 users
• Experienced (>= 100, <300 edits):  189 users
• Very experienced (>= 300 edits): 647 users

When it comes to content, or knowledge being added or improved through events, we had 238,222 unreverted content edits and 2,499,773,577 bytes added. After a few months from the end of the campaign (March 31, 2026), we have a total of 32,388 undeleted content pages added.

Celebrate Women is also responsible for increasing the number of events held in the Wikimedia community. Considering Event Registration usage only, we can see in the next image a major increase from 2025 to the 2026 campaign, and a peak in February and March compared to last year.

In March 2025, we had 152 events created using Event Registration, while in March 2026, we had 251 events. This is a 65% increase. However, for Celebrate Women, it’s also important to consider February, when a good portion of the events are created for organizers in preparation for the early March events, especially around International Women’s Day. In February 2025, 130 events were created, while in February 2026, 240 events were created. This is an 84.6% increase.

Wikimedia Foundation’s support in 2026

Celebrate Women is a community campaign and the result of volunteers and community members from all over the world. The Wikimedia Foundation (WMF) has been helping this campaign by providing support for communities and organizers.

In 2026, we started the support by renovating the Celebrate Women portal. We changed the campaign’s header, reorganized it, and even added a press toolkit tab to help organizers develop a media plan or get media coverage from local journalists and media outlets. This new page includes sections about how to explain Wikipedia and the gender gap, key Wikipedia and gender gap facts and figures, inspiring gender stories, and a list of campaigns and groups dedicated to decreasing the gender gap on the Wikimedia projects. 

We also added resources and guides for how to use the CampaignEvents features, Content Translation Suggestion feature, and Central Notice Banner Editor. Additionally, we collaborated with the WMF Comms team on social media posts, which were shared throughout March on the Foundation’s channels.

We also contacted major gender organizers and community members (via email and talk page) and discussed the idea of adding the Event Registration header to help us understand the campaign’s participation. With Event Registration, the campaign reached 617 registered participants. According to the Connection team, who developed this feature, Celebrate Women 2026 is currently the 3rd biggest event in terms of participation so far, only behind Wikipedia 25 Virtual Celebration, with 3,187 participants, and the Africa Wiki Challenge 2025 with 790 participants.

During the pre-campaign community consultations, we received a request to not only have an Event Registration way to track the campaign, but also an Outreach Dashboard campaign dedicated to 2026. We created the campaign here.

Later, we chose to understand the impact of previous Celebrate Women years using the Outreach Dashboard too. We created campaigns dedicated to 2022, 2023, 2024, and 2025 and decided to search and include all gender events organized during those years using the dashboard. Now, we can better understand the impact of the gender content gap efforts for the past 5 years. 

Finally, by using Event Registration, we were able to communicate with the 617 participants who registered. Once a week, we sent a reminder with the events that were taking place that week, divided into regions, and including day, time, place, and how to register. This established a recurring way to connect with community members and remind them about the campaign as a month-long commitment.

Wikimedia Foundation’s events

The Wikimedia Foundation also organized three events to support the community in organizing their Celebrate Women gatherings. We planned the campaign’s kick-off event and 2 pre-campaign trainings dedicated to Wikimedia organizers.

• On February 25, we had the Content Translation training. This session aimed to empower organizers with the skills to use Content Translation, with a focus on the article suggestion feature. 

• On February 26, we had the Event Registration and Collaborative Contributions session. The training aimed to equip organizers to effectively accomplish event registration, metrics tracking, outreach, and sustained community engagement. 

• On March 5, we held the campaign’s Welcome Session, which covered the 2026 campaign, some of its events, and some of Wikimedia’s Gender Gap milestones presented by Chinmayee Mishra. Participants also learned a bit about Event Registration, Content Translation, and two tools that help organizers with small but powerful edits: Gender Bias Detector and Wikipedia Microtask Generator. 

On the page for the three events, all the resources are available: recordings, slides in different languages (English, Spanish, Portuguese, Arabic, French), and other useful information.

WikiWomen+ Summit 2026

In August 2026, the WikiWomen+ Summit 2026 took place during Wikimania’s pre-conference day. At this gathering, we were able to present about the Celebrate Women and the impact of this campaign in the context of Wikipedia’s content gender gap. 

Please find the recording for this presentation on YouTube. The slides are also available on Wikimedia Commons.

To understand more about the campaign or to stay tuned, check out the Celebrate Women portal on Meta-Wiki.

* The usage of the “+” sign after the word “women” means we are including anyone who identifies as a woman in some way, that being a cis woman, transgender, non-binary, genderqueer, or any gender identity not listed here.

The #SheSaid campaign 2026 officially launched on 1 September and will run until 31 December 2026. Now in its seventh edition, #SheSaid continues to grow as a global campaign, bringing together participants from around the world to add and improve women’s voices on Wikimedia projects.

As part of Wiki Loves Women, an initiative of Wiki in Africa, #SheSaid works to address the gender gap on Wikiquote by making women’s words, experiences and perspectives more visible and accessible.

This year is particularly special: Wiki Loves Women is celebrating 10 years of making women visible in the world’s largest open knowledge ecosystem. #SheSaid is part of this decade-long journey, building on the contributions of thousands of participants who have helped bring women’s voices into the Wikimedia movement and beyond.

More than numbers, SheSaid is a story of collaboration

Six editions. More than 42k contributions. 22 languages from English to Assamese. More than 25 communities from Indonesia to Italy, Ghana and beyond. . 

#SheSaid is more than just the numbers. It is a story of people coming together across languages, countries and cultures, united by one purpose: to make women’s voices more visible.

That spirit of collaboration has taken #SheSaid beyond Wikimedia. FrancAutrices, a card game created through a collaboration between Les sans pagEs, Wiki in Africa and French Wikiquote editors, brings the words of Francophone women writers to new audiences. The game features quotes from Wikiquote, alongside portraits of the authors by French illustrator Claire Gaudriot and a design by South African designer Theresa Shaw.

FrancAutrices is one example of how a contribution to an open platform can become the starting point for something new, connecting communities, cultures and audiences around women’s voices.

A community favourite

The Wikimedia community has celebrated the story of # SheSaid. At Wikimania 2026, the campaign was selected as one of “Wikimedia’s Coolest Projects”.

For this award, in-person and online attendees were invited to vote for the projects that best represented the creativity, diversity and collaborative spirit of the Wikimedia movement. The contest was revived in memory of its founder, Deror Lin, as part of the celebration of 25 years of Wikimedia.

Being chosen among the three coolest Wikimedia projects is a meaningful recognition of what the #SheSaid community has built together.

A campaign for everyone

As this year’s campaign begins, we’d like to extend a heartfelt thank you to everyone who has contributed to #SheSaid over the years; every quote, edit, translation, event and conversation has helped shape this global campaign and made a significant impact on Wikiquote. SheSaid belongs to everyone who takes part in it, and through this collective effort, women’s voices continue to find their place in open knowledge.

The story continues, and everyone can be part of it. Whether you contribute a quote, improve Wikiquote, organise an activity, translate content, or encourage someone else to participate – join us!

Be part of #SheSaid 2026 and help write the next chapter of this global story.
#SheSaid … Give Her A Voice!

Don’t forget to follow us on social media:

• Website: www.wikiloveswomen.org
• Twitter/X: @Wikiloveswomen
• Instagram: @WikiLovesWomen
• Facebook Page: @WikiLovesWomen
• Telegram Group: WikiGenderGap

A reflection from Wikimedia Kenya User Group on why skills development, shared leadership and opportunities for growth are essential to building resilient communities.

There was a point during Wikimedia Kenya User Group’s recent team-building day when somewhere between the conversations, laughter and uncomfortable moments of reflection, the room changed. The titles and responsibilities that usually defined us seemed to fall away, and for a while, we were simply what we had spent years trying to build: a community.

We spent the day competing in well-crafted games, laughing at our desperate attempts to win and the missteps that cost us points, while discovering just how differently each of us approaches a challenge. But beneath the fun, the activities revealed something more important; they showed us how much we rely on trust, communication and listening to work towards a common goal, and how leadership is not always about taking charge, but also about knowing when to step back and make space for someone else.

The people in that room were not only campaign organisers delivering projects and programmes. They were also members of the Core Team supporting Wikimedia Kenya UserGroup’s mission and governance. That made the day about more than team-building but also an opportunity to reflect on how we work together, the kind of leaders we want to become, and ultimately, the kind of community we are building.

“I really enjoyed the team-building activity because it gave us a chance to step away from our usual roles and understand each other differently. It reminded me that building a strong community is also about building trust within the team, understanding the different strengths each one brings to the table, reflecting on and learning from our mistakes, creating space for different people to belong and learning how to support each other better.”  Terry Boke, Community Organiser & Core Team Member

Building Communities Means Investing in People

It is easy to measure a growing community through the number of new contributors, articles created, events delivered, partnerships established or campaigns completed. But behind every one of those numbers are people: people who need the confidence to take initiative, the skills to do it well, the opportunity to take responsibility, and a community that recognises their potential and gives them room to grow.

For Wikimedia Kenya, leadership development is therefore not a separate activity sitting alongside our programmes. It is part of how we are building a resilient and sustainable community.

We don’t just want volunteers whose contribution is limited to editorial work. We want to build a community of people with the skills, confidence and agency to take on different roles, lead initiatives, solve problems and shape the direction of the movement. Whether they are organising campaigns, building partnerships, mentoring others, contributing to governance or creating knowledge, we want people to see themselves not just as participants, but as active contributors to the growth and future of the community across Kenya.

That means mentorship, nurturing talent and creating opportunities for people to practise leadership before they are formally given a leadership title. It means developing the skills to communicate strategically, make decisions, work across different perspectives, take responsibility for outcomes and, just as importantly, make space for others to contribute and lead.

This thinking is reflected in our 2026/27 Strategic Plan, which recognises leadership development as an important part of building a resilient and sustainable community.

From Team Building to Communicating for Impact

Drawing on more than a decade of experience in strategic communications, community building and leadership development, I wanted to inspire the team to think about how we could strengthen the way we work and grow as a community. I therefore led a practical session on “Communicating for Impact: From Strategy to Community Engagement,” built around a simple question:

How do we translate Wikimedia Kenya User Group’s strategy into action, communicate more effectively, engage our community meaningfully, and ensure that we can see, measure and document the impact of our work?

For people taking on leadership and coordination roles, communication is not simply about writing better outreach emails, designing social media posts or making event announcements. It is about understanding people and audiences, listening well, building trust, navigating differences, working collectively and creating the conditions for others to contribute and lead. It is also about connecting the decisions we make and the work we do every day to Wikimedia Kenya’s strategic priorities and the wider movement’s goals.

Developing these skills is an essential part of developing people who can take on greater responsibility, support others and help shape the future of the community.

We explored a simple progression:

Strategy → Communication → Engagement → Action → Impact

The aim was to help us see that communication cannot sit separately from strategy. It should help people understand the purpose behind our work, see where they fit and feel empowered to contribute.

What stood out to me was not simply what people learned, but how quickly the boundaries between training, team building and strategy began to disappear. The conversations moved naturally from how we communicate to how we work together, take responsibility and create opportunities for others to contribute. The session became a practical exercise in leadership development, building the skills needed not only to deliver our work, but to shape what comes next.

That shift was reflected in the feedback from one of our Core Team members:

“This was really great for understanding the bigger picture of our work and learning how to move more strategically. I feel empowered to think differently about how I plan and engage with the community and external stakeholders in future projects and campaigns that I lead.” Faith Mwanyolo, Core Team Member & Lead Organizer, Wiki Loves Africa

One practical lesson from the communications training was the need to make strategic alignment part of our planning process, rather than something we consider after a project is already underway. Going forward, we want every project and campaign to begin by asking how it advances the Wikimedia Kenya Strategic Plan and contributes to what we are working towards as a community.

To support this, we are developing a standard project planning template for organisers and a review checklist for the Core Team. The template will help teams identify the strategic objective their project contributes to, the activities they will undertake, the outputs they expect to deliver, the outcomes they aim to achieve and the metrics they will use to track progress.

The checklist will provide a consistent and transparent approach to reviewing internal proposals, helping us assess not only whether an activity is well designed, but also how clearly it contributes to our strategic priorities and how responsibility for results is shared.

The intended outcome is a more deliberate approach to how we plan, implement and evaluate our work. Over time, this should help us understand which activities are contributing to our goals, where we are making significant progress, where gaps remain and where we need to adapt.

Four Lessons That Have Shaped How We Build Our Community

For emerging communities, especially those building themselves with limited resources, investing in leadership can easily become an afterthought-something to address once the community is larger, programmes are more established or funding is more secure.

Our experience over the past five years has taught us something different: leadership development is not something that should follow community growth; it is one of the things that makes sustainable growth possible. As more people develop the skills and confidence to take responsibility, lead others and shape the direction of the community, growth becomes something the community can build and sustain collectively.

Below are four lessons that have shaped how we build our community.

1. Make leadership development part of the strategy

Don’t wait for people to emerge as leaders by accident. Create deliberate opportunities for mentorship and for people to take responsibility, experiment, make decisions and learn from experience.

Leadership grows when people are trusted with meaningful roles and supported as they take them on. This also means recognising that people develop at different stages and may need different opportunities to grow—from taking on their first organising role to leading a major campaign, mentoring others or contributing to governance.

2. Don’t separate team building from organisational development

Games, reflection and informal conversations can reveal things that formal meetings don’t. A team that understands how its members communicate, solve problems and respond under pressure is better equipped to collaborate when the real work begins.

Team building therefore has a role beyond creating a good atmosphere. It can help people understand one another’s strengths, identify areas where they need to work differently and build the trust needed to share responsibility.

3. Connect activities to purpose

People taking responsibility for campaigns and community activities should understand not only what they are doing, but why it matters and how it contributes to the community’s wider goals and the Wikimedia movement’s strategy.

When people understand the connection between an edit-a-thon, partnership, training or outreach activity and the bigger organisational goal, they are more likely to take ownership of the work and see how their contribution fits into the community’s growth.

As one of our Core Team members reflected:

“When we understand why we are doing something and how it contributes to our wider goals, the work takes on a different meaning. We are no longer just delivering activities; we are making deliberate contributions towards the kind of community and movement we want to build.” Lebu Ayiga, Community Organiser & Core Team Member

4. Create room for leadership to move

Volunteer burnout can quietly undermine growing communities when too much responsibility rests on a small number of committed leaders. Those who help build a community can easily become the people expected to organise every campaign, solve every problem and keep everything moving. Over time, this can limit both their capacity and the community’s growth.

We have learned that investing in people also means creating room for leadership to move. This means creating pathways for members to take on new responsibilities, develop the skills and confidence to lead, and step back when others are ready to step forward.

For us, this is not only about preventing burnout. It is about building a wider base of people who can share responsibility, mentor others and take on greater leadership roles. When leadership is shared, the community becomes less dependent on individuals and better able to sustain its growth.

Invest in Leaders at Every Stage

Wikimedia communities are built around knowledge, but they are sustained by people. If we want our communities to grow and remain sustainable, we have to invest in people at every stage of their journey.

That means creating pathways for new leaders to emerge, while also giving existing leaders opportunities to deepen their skills, take on new challenges and progress into new roles. Leadership development should create space for people to build confidence, share responsibility, mentor others and continue finding meaningful ways to contribute.

Leadership should not be viewed only as a pathway towards replacing one person with another. People who have already taken on leadership responsibilities also need opportunities to learn, evolve and take on new challenges. In a volunteer movement, creating those opportunities can be just as important to sustaining engagement as bringing new people into leadership.

Over time, this creates resilient communities where people at different stages can continue to grow, contribute, lead and support others.

Winnie Kabintie is the Executive Director, Wikimedia Kenya User Group

One of the downsides of having written here for six years now is that my messy, ever-growing text file of topics for future articles contains a bunch of ideas that have been there for about six years. Part of the problem is that I just have to scroll so far up to even see them now, part of it is that some of them are on topics where I do not feel equipped to write a good formal treatment. So let's consider this an easy, breezy episode of Computers Are Bad as I knock out one of those items.

One of these O. G. topics, line three, is "avalanche technology." I will admit that I am a little fuzzy on what this originally meant, but I have a few good guesses. First, though, let's just take a step back and talk a little bit about the practicalities of the avalanche. For a lot of you, avalanches are probably a pretty abstract issue.

For me, as well—I snowboard but I'm not that good at it, and the kind of mid-tier ski area where you will find me tumbling down the hill (Ski Santa Fe, towards Totemoff's) manages their slopes to avoid the potential for avalanche as much as possible. This requires some expertise in avalanche risks and it's never quite perfect, but it's pretty good. Snow that is regularly groomed by machines like PistenBullys will become coherently packed such that avalanches can't really propagate. The bigger risk on groomed ski slopes generally comes from up above, and here in New Mexico our mountains are of such a height that we're usually starting down from the very top anyway.

Still, avalanches pose a real hazard. The Department of Homeland Security, when it can be distracted from terrorizing well-meaning migrants, tells us that avalanches claim about 28 lives each year in the United States. Globally, the number is much larger, and avalanche fatalities seem to be generally more common in Europe. I'm not sure if that reflects European weather, geography, or recreation habits more, but it's the way things pan out and means that Northern Europe tends to be the center of avalanche technology (as it is in alpine technology in general).

We tend to associate avalanche hazard most of all with recreation, and that's definitely fair: the highest risk of avalanche probably comes about in cross-country skiing, where it's fairly common to traverse the lower part of slopes that have otherwise received very little disturbance. Most avalanches occur during storms, as the snow falls and piles up, which is of course not when recreationists are most likely to be present. You can't rely on this, though. One of the ways that an avalanche can initiate is when lots of snow is piled up on a slope on a nice sunny day—good recreation conditions! Various things like water following the ground or stratigraphic layers of different types of snow from different storms can create a condition of uneven melting. Snow that is under the surface might liquefy faster than snow that is closer to the surface. This "detaches" the upper level of snow from the ground, and when it happens over a wide area, that upper level of snow can let loose all at once. This type of avalanche is relatively rare, but they happen without warning in otherwise good weather, so it's much more likely that people will be caught. Further complicating the situation, the proximity of people can be a factor in initiating these avalanches.

Here in the Mountain States, though, avalanche risk often manifests in another context: transportation. As Americans, cars and highways are among our most esteemed achievements, and decades of engineering accomplishment allows us to drive through, over, and even under much of the Rocky Mountains even during the winter. Landscape modifications, chemical measures, and the brute force of snowplows keep roadways clear—but not the slopes above them. Avalanches cascading down onto highways are a major concern in the Mountain States, especially Colorado which has the most impressive set of mountains and mountainous highways.

Forecasting

Some of the things, then, that we might call "avalanche technology" are the techniques used to prevent, mitigate, and warn of potential avalanche. Much of the work is institutional. On the recreational front, this falls mostly with the US Forest Service's National Avalanche Center and a loose confederation of regional organizations such as our own Taos Avalanche Center, here in New Mexico. One of the main functions of these organizations is education: posting warning signs, spreading awareness, offering training courses and materials, and so on. They also tend to employ avalanche specialists, who use a combination of meteorology and specialized snow observations to determine when avalanches are most likely. This can lead to posting warnings or closing areas entirely.

Avalanche specialists in the recreational field are, in this area, joined by a substantial cadre of avalanche specialists employed by state Departments of Transportation. They apply the same techniques with an eye towards avalanches that could affect mountain pass highways. Those highways may be closed in extreme circumstances, but in transportation even more than in recreation it is common to use, shall we say, "active measures" to prevent avalanches. We will discuss this later, but first, let's briefly talk about how avalanche hazards are forecast.

I am not going to go into much detail here, because I frankly do not know much about it, but I do want to touch on a couple of things that are solidly in my wheelhouse. One of these is SNOTEL. Forecasting avalanche risk requires a solid understanding of the general amount of snowpack in the mountains, which can be very tricky to acquire since when snow is heavy getting up there to take measurements becomes a very time-consuming and potentially hazardous venture.

Avalanche forecasting is just one of many functions served by the SNOTEL or Snow Telemetry system, a 1960s-era network of hundreds of sensor sites throughout mountainous parts of the United States. SNOTEL was originally designed to help with forecasting spring meltwater flows, which is why it falls under the aegis of the Department of Agriculture, but as with any good sensor network there are now a lot of different stakeholders.

The tricky thing about SNOTEL, and remember here that it dates back 60 years, is the snow. SNOTEL sites are way up there, inaccessible during winter, so they are designed to be installed and serviced during summer and to operate autonomously until the next summer. SNOTEL instruments are solar powered, but more interesting are the communications arrangements. The initiation of the SNOTEL project predates a robust set of earth observation satellites, to say nothing of cellular or other digital radio networks. But the whole purpose of SNOTEL is to provide snowpack data when the snow is still there, so real-time reporting is critical.

SNOTEL was thus one of the first applications of meteor scatter, a fascinating form of radio communication that will probably get its whole own article one day. Here is the summary: SNOTEL data collection sites, in convenient locations like towns, transmit a VHF carrier or "pilot" signal up towards the sky. Occasionally, but more often than you would think, a meteor enters the Earth's atmosphere and burns up, leaving a trail of ionized gas behind it. This ionized gas is, if you remember previous discussion of troposcatter, reflective to RF energy. Some of the "pilot" signal is thus scattered back down towards earth. When a SNOTEL site receives the pilot signal, it knows that a meteor has just entered the atmosphere and this scattering is happening, so it quickly transmits a data packet towards the sky. Ideally, this data packet is reflected by the same meteor trail such that the data collection site can receive it. If this all sounds rather stochastic and chancey, it is, but enough meteors enter the atmosphere and data packets can be sent quickly enough (and tried multiple times) that it works pretty reliably for low-bandwidth telemetry data.

Unfortunately, as interesting as meteor scatter is, it's mostly been obsoleted by general technological advances in radio communications. SNOTEL was probably the largest meteor scatter network in the world¹, but is no longer: all SNOTEL sites now use other means of reporting data. The most common is probably the generic data collection capability of GOES, NOAA's main fleet of "weather satellites." GOES satellites actually serve many purposes related to earth observation, and one of them is telemetry collection from instruments on the ground. If you've ever come across a USGS stream gauging station or another mysterious hut in the middle of the woods with a cylindrical or conical antenna pointed at the sky, it's transmitting data upwards for reception by a GOES satellite. The GOES earth segment collects all of the little data packets the satellites receive into files which are distributed to the various organizations that report data this way. If you are so inclined, you can also receive this data yourself as the satellites rebroadcast it in batches, but most users don't bother since NOAA will do that part for you.

Even that might be fading away: newer SNOTEL sites apparently use the cellular network for data reporting, unless it's not possible to get a signal, but that's becoming less common over time even in the Colorado mountains. The major benefit here is that the cellular network is low latency and very high bandwidth, which makes it feasible for example to view cameras installed at the SNOTEL sites.

With the data part discussed, let's take a quick second look at the data collected. Temperature, humidity, pressure, wind, liquid precipitation are all measured as usual, but SNOTEL adds a couple of specialty features. Soil moisture and evapotranspiration (basically evaporation rate of water from soil) sensors provide more data on how saturated the ground is, an ultrasonic or optical instrument pointed down from atop a post measures how deep the snowpack is, and a funny looking contraption called a snow pillow takes perhaps the most important measurement. The snow pillow is basically a big rubber balloon that gets inflated on the ground, and then covered with snow. By measuring the air pressure from the snow pillow, the SNOTEL station effectively weighs the snowpack, which combined with its height gives the density or water content of the snow. This varies widely depending on the specific weather when the snow falls, and remember that stratigraphic differences in density are one of the factors that's important in the likelihood of a surprise avalanche, so collecting this water content information as snow falls allows forecasters to remotely estimate whether or not the snow conditions are hazardous.

That is, of course, supplemented by good old fashioned techniques like sending out avalanche specialists with shovels and meter sticks to take a look at the snow themselves. DOTs often have technicians doing a regular tour of mountain pass sites, where they curate plastic tubs and boxes of snow from various storms so that they can get a hands-on impression of what the different layers under the snowpack are like. They also take samples to weigh, perform some tests to measure mechanical properties of the snow, there's basically a whole bag of tricks, but ultimately there is also a subjective element of how "good" or "bad" things look. This is born of experience and is one of many reasons to value our state employees.

Mitigation

Within the recreational context, one common approach to mitigation is simply to avoid the area. Especially for backcountry skiing areas, avalanche centers might post warnings or work with the operator to close the area entirely when forecasting shows that conditions are favorable for an avalanche. Warnings might be more useful than you think, since besides reducing the number of people in the area they should also prime people to recognize particularly risky slopes or signs of snow that is becoming unstable. Avalanches can occur without any surface signs of instability, but in practice there often are some signs, so of course keeping an eye out is better than nothing. Warnings also make people more likely to carry safety equipment, discussed later, which improves the chance of self-rescue if things really do go wrong.

But there are also, shall we say, kinetic options. Many of the methods of modern alpinism evolved during the Second World War, and so did a good portion of the mitigation equipment. Once snow becomes unstable enough that an avalanche could occur, it is basically waiting for something to set it off. Popular media concepts that loud noises will trigger avalanches are, well, fictional, but the point stands that people moving around or especially operating vehicles can trigger avalanches, as can basically random happenstance. You can mitigate the risk of an avalanche by going to unstable areas and intentionally triggering them, or at least trying to, basically the same way that a bomb squad might handle an unstable explosive by exploding it. Better that it go off when you expect it than when you don't.

A classic method, still in some use today, is repurposed artillery. State DOTs and ski area operators own a surprisingly large inventory of M101 Howitzers, a 105mm artillery gun that dates more or less to a 1910s German design but was widely fielded by the US during the Second World War. The avalanche application is simple: when you find a slope with unstable conditions that could pose a hazard, you shoot at it. The concussion from the impact, and often from an explosive round, puts a lot of pressure on the snow that will likely break it loose if it's possible to do so. You just have to make sure you set up outside of the resulting avalanche.

The main problem with this method, which everyone agrees is a whole lot of fun, is that the M101 has not been manufactured since 1953 and, in general, the supply of cheap Army-surplus artillery has dried up. The Howitzers are aging out, and other methods are replacing them. Some of these are similarly dynamic: dropping small improvised bombs from helicopters, for example, which is a very common method in Colorado today. Even just hovering a helicopter over a slope can sometimes trigger an avalanche, although a bit of dynamite makes this a lot more reliable. If you have noticed the surfeit of high-explosives bunkers and unexploded ordnance warnings in some parts of the Colorado mountains, well, now you know why. It can feel a little bit like a very slow-running war.

Even these methods are being replaced by more modern and cost-effective ones, though. A French alpine engineering company called MND manufactures a system called Gazex, which is a bit like an airport bird canon on steroids. Gazex units are pre-installed in avalanche risk areas (by helicopter, for example), and then, under remote control, they mix oxygen and a combustible gas and set it off in a big tube pointed at the ground. The force created is similar to an explosion, but the whole thing is easier to handle (since it uses gas cylinders rather than high explosives) and remote control makes everything cheaper for the operators. MND recently introduced a new version of the system called O'bellx (I assume this reads better in French), which is even more portable, albeit comically egg-shaped. This seems to be catching on with ski areas, and MND also emphasizes that the portable design and low-impact installation makes it suitable for areas like national parks with acute natural preservation concerns. Basically, you can remove them at the end of every winter, so that you aren't leaving equipment scattered around your beautiful mountains.

There are also a number of other mitigation approaches that are less fun, so I will devote less attention to them, but you can find them done around here. Structures similar to snow fences, built high up on a slope, provide an "anchor" for the snow that prevents it coming loose enough for an avalanche. There are many variations on this idea, for example using steel cables supporting nets. Another approach is to permit the avalanches to start but prevent them from reaching areas where they cause harm, either by building wall or dam-like structures (I do not believe this is common in the US as I am not aware of any examples) or by putting corridors like highways inside of "snow sheds" that are sturdy enough to allow the avalanche to simply flow over. This latter method is much more common in Europe than here, but you can find some examples on US railroads, which are often in more difficult terrain than highways.

Warning

One of the more interesting developments in avalanche technology are warning systems. Avalanches wouldn't pose much hazard if people knew that they were coming; what makes melt-triggered avalanches so dangerous is that they can be very unexpected. One of the methods that avalanche safety courses teach is to recognize signs of snow masses that have already started to slide, shift, or collapse on themselves. This would indicate that conditions are favorable for an avalanche and that disturbing or going below the snow is quite dangerous. The usual surface signs are cracks, caused by the snow starting to slide apart, but these cracks can be tricky to see even when they are present.

Another indication of avalanche potential is subsidence, or the snow surface starting to drop in level, or any motion of the snow surface downslope. These effects can start minutes or hours before the real avalanche occurs, but they are subtle and often slow, so humans are unlikely to notice them. Fortunately, machines are well adapted to this task. Several vendors offer radar systems that monitor snow slopes for any shifting, and can sound a siren or another alert if things start to move. Like earthquake warning, the actual lead time provided by these systems is not very well known and could be quite short, but something is better than nothing.

Other technologies, like LIDAR and acoustics (e.g. geophones), can be applied to a similar task. I have even heard of older systems that used thin wires or brittle metal bars embedded in the snow, as part of an electrical circuit that would break if the snow started to move ². Such systems can be quite sophisticated, and MND's sales materials suggest things like closing gates on highways when an avalanche is detected. The current state of the art in radar detection uses pulsed Doppler operation, which can be very sensitive to movement in all kinds of directions. All that said, these don't seem to be popular in the US. I couldn't readily find any installations in this country.

As a final note on this topic, national alerting systems like NAWAS and the Emergency Alert System have defined message types for avalanche warnings. In practice, these are very rare, as it doesn't usually make sense from a time or area perspective to put out an avalanche warning through mass media channels. That said, they do happen, usually on the (fortunately rare) occasion that forecasters predict a high risk of avalanche in a town or other populated area. In that case, the procedure is to evacuate the town, and that's exactly the kind of situation that NAWAS/EAS/NWS All Hazards/etc. are intended for.

Rescue

And finally, we reach the topic that I believe led me to put "avalanche technology" on the list to begin with: technical aids to rescue. Avalanches are, in a certain sense, not that dangerous. They can move at formidable speeds (60+ miles per hour often reported), but snow is relatively soft and low-density. Injury from being crushed or striking against something is certainly a possibility, but the bigger hazard of avalanche is usually when you become buried.

If you are buried in snow, and there are no rescuers or they cannot find you, it can be extremely difficult to work your way out and you are likely to succumb to some combination of asphyiation, exhaustion, and hypothermia in the process. That might sound silly but keep in mind that you have just been buried by an avalanche, probably have a variety of minor injuries, have become disoriented, etc., and you try to dig yourself out of feet of snow in that situation. You really need someone on the surface to come help you out. That's why standard avalanche safety equipment includes a probe and a shovel—a probe for finding people (just a metal rod you use to poke under the snow, basically), and a shovel for getting to them.

The problem is that probing a large area of snow is a very slow process, and even if you were right next to someone before an avalanche, you probably don't know where they ended up afterwards. This is an area where radio technology can be extremely helpful.

The concept of avalanche beacons or avalanche transponders dates back to at least 1968. Dr. John Lawton was an engineer at an aerospace research lab now known as the Calspan Corporation, where he worked on a variety of interesting aviation problems ranging from weapons targeting to meteorological research. Lawton was quite an aviation enthusiast, making the news in 2013 for celebrating his 90th birthday by flying his Cessna Skyhawk over the US-Canada border 90 times. That's a bit of a peculiar way to entertain yourself, but anyone who is still flying themselves around at 90 has clearly found a way to stay vital. His lifelong investment in aviation is emphasized by the fact that he started his winding cross-border journey at the airport he owned.

What is not so well documented is his interest in alpinism, but he must have had some affection for the mountains. He did some kind of research work in Europe related to triggering avalanches by dropping explosives from light aircraft, and whether that put the topic on his mind or something else, he also came up with the first commercial avalanche beacon. Commercialized by his company Lawtronics, the "Skadi" transmitted a tone at 2.275 kHz. Groups of people traveling in the mountains could each carry a Skadi unit, and if anyone was lost in an avalanche, any of the other units could be switched to a receive mode and the person located by simple amplitude search. That is, walking around the snow until you find the spot where the tone comes in the loudest, at which you point you start probing and digging.

This same method is still used today, although the radio system has been standardized at 457 kHz. Modern avalanche rescue systems like the "Barryvox" from Mammut or the Black Diamond Recon X are direct descendants of Lawton's Skadi device, using 457 kHz and digital direction finding techniques (like the use of multiple perpendicular directional antennas to compute a likely heading to target) to speed things up and make radiolocation more user friendly. Because the 457 kHz system is an international standard, they ought to work about the same regardless of manufacturer. That's important in real-world scenarios where the people doing the rescuing might be from a whole different group than the one affected by the avalanche.

Modern avalanche beacons are probably one of the closest real-world products to the way that "radio beacons" tend to work in movies and videogames. The reality is of course messier, especially if there are multiple victims, but in principle a digital avalanche beacon in receive mode can show an arrow on screen that you follow until you are standing on top of the buried person. You know if you're getting closer when they beep more. Neat.

Technological development continues apace, and some newer avalanche beacons also support a second protocol called W-Link. W-Link is actually surprisingly poorly documented, presumably because it appears to be proprietary to Mammut (the only non-Mammut beacons that support it seem to have been joint venture projects with Mammut). W-Link is a digital protocol that operates in the ISM band, introducing the downside that ISM bands are not internationally standardized. This means that US and European-market beacons will always be compatible for conventional 457 kHz operation, but two beacons with W-Link support may not be able to use that protocol if they were sold for different regulatory markets.

The upside of W-Link is that the digital protocol lets the beacon send significantly more information. This includes a unique serial number for the beacon, which allows the receiver to discriminate between pulses being received from two or more beacons at the same time—a radical improvement in the ease of locating multiple people, as conventional direction finding techniques really struggle when you cannot discriminate between multiple targets. You can end up with a "swinging needle," while the W-Link devices are capable of locking to a single target at a time. As this implies, you can also determine how many people are buried, and ignore beacons that turn out to be unimportant (because they are discarded on the surface, or carried by a rescuer, etc).

Perhaps the most interesting feature of W-Link is that some W-Link beacons will use an accelerometer to detect motion, and report whether or not any motion is present. In theory, this allows rescuers to prioritize their efforts by focusing on beacons that report motion, meaning that they are carried by people who are still alive. These enhanced capabilities of W-Link seem to have led to enough controversy that the Wikipedia article devotes much of its length to ethics discussions, and manufacturers have put out position papers on their decisions to implement or not implement certain W-Link features. The concern is that W-Link may give rescuers more information on which to prioritize rescue in a discriminatory way, perhaps by searching for victims with (more expensive) W-Link beacons before less expensive conventional beacons, or by using knowledge of W-Link serial numbers to prioritize the search for specific people.

In practice, I think these ethical concerns are more theoretical than actual, because penetration of W-Link technology doesn't seem all that high and most actual beacons only implement the very most basic features. The ability to display the serial numbers of individual beacons, for example, is pretty much limited to a USB dongle/software package that Mammut sells to organizations like ski patrols that loan out avalanche beacons and need to keep track of their inventory, check battery levels, etc.

Avalanche beacons are not the only radio-based approach to avalanche rescue. Around a decade after Lawton developed Skadi, a Swiss engineer designed a system called RECCO. If you have read my articles on loss prevention, I can succinctly explain RECCO by saying that it is just RF electronic article surveillance applied to the outdoors. If you have not, let's get into a little bit more detail.

If you make an antenna that is a small loop, and then put a nonlinear device like a p-n diode in it, you end up with an antenna that behaves asymmetrically with regards to the phase of the radio wave passing over it. Intuitively, we can say that energy freely flows one way around the antenna, but not the other. The energy going that way becomes trapped, and gets re-emitted at at a harmonic frequency to the one that excited the device. This general principle is known as non-linear junction detection, and it has all kinds of interesting applications in fields like technical security, but its most common application is as one of the common types of retail anti-theft tagging.

RECCO is basically a retail loss prevention system aimed at the ground. RECCO tags, called "reflectors" in RECCO parlance, are a simple foil antenna and diode, which makes them cheap and durable. They are so cheap, and compact, that some brands of alpine gear (especially in Europe) build RECCO tags right into products like snow pants and ski boots. Dainese even offers a bicycle helmet with one. The RECCO detector emits a powerful and very directional RF field in an ISM band, and then monitors for the characteristic harmonics emitted by an excited tag.

This technology is not RFID, although it is conceptually similar, and has some of the same downsides such as a tendency for tags to be undetectable if they fall in the wrong point in the phase of the exciting field. For that reason, RECCO recommends that you have two reflectors at different locations on your person. The other main downside of RECCO is that it shifts most of the complexity into the detector, so they are relatively large and expensive. That's not so bad when you consider that detectors only need to be carried by ski patrol and SAR organizations. Rescuers basically aim the detector around until it picks up a signal, and then walk that way until they are aiming the detector straight down. That's where they dig.

RECCO has even designed a detector that can be underslung by a helicopter, although its 100 meter range will require some sporty flying. The RECCO system is generally much more popular in Europe than in the US, but penetration is increasing here, and RECCO publishes a list of SAR organizations with detectors that now includes a number of helicopters in the US and many major ski areas.

I view RECCO as having a bit of a belt-and-suspenders relationship with traditional avalanche beacons. Avalanche beacons are more common, and all beacons also function as receivers, so "buddy rescue" by a bystander is far more likely with beacon technology. On the other hand, RECCO is usable over a longer range (although not radically longer) and the detectors are more directional and potentially faster to use. Most avalanche centers seem to recommend that you carry both, although the traditional avalanche beacon is more important.

Finally, let's consider a couple of other pieces of avalanche safety equipment, which will lead us to consider efficacy.

The core avalanche PPE consists of a probe and a shovel, and both are available in collapsible form for portability. Besides locating technologies, though, the major innovation in avalanche safety is the avalanche airbag. These are backpacks that you wear with a parachute-like ripcord. When you pull it, the bag inflates into a big airbag behind you. There are two ways this is helpful: first, the airbag will tend to float on moving snow, making it less likely that you are buried. Second, if you do become buried, you can deflate the airbag to leave you with a void in the snow. This gives you more air to breath, and room to move, making it more likely that you can dig yourself out.

Avalanche airbags remind me a bit of the motorcycle airbags, as far as being very cutting-edge but also, you know, kind of questionable when it comes to efficacy. Manufacturers of airbags often advertise a 50% reduction in fatalities, but we have to take those numbers with a grain of salt.

One of the problems with safety technology for extreme risks is that the actual risk event just doesn't happen very often. In less abstract terms, people are not caught in avalanches all that often, so the total number of people who have been caught in an avalanche while wearing an avalanche airbag just isn't that big. This limits the confidence we can have in efficacy findings.

Well, I tracked down the paper that the 50% number comes from, and it's not actually that bad. They identified 424 people caught in severe avalanches, roughly half of which had avalanche airbags, and found that the airbags reduced mortality by 50% for people who successfully deployed them, and 41% overall given that there were various situations where they were not triggered or failed. The sample size is not huge, but it's bigger than I expected, and even at the bottom of the 95% confidence interval you still see a meaningful mortality reduction of around 10%.

So, avalanche airbags seem to work, but they do cost around $1,500, so make of that what you will. Another interesting number from that paper is actually a bit of an aside: of the sample of people considered, 99% had avalanche beacons! A basic avalanche beacon can be had for about $250, which is pretty similar to cheap satellite messenger or very basic PLB ³.

On the other hand, the passive nature of RECCO tags translates to a price advantage: a standalone reflector that you can clip to your pack only costs $30, which frankly still feels like a ripoff considering the simplicity of the design. I wonder if a RECCO detector would pick up enough RF EAS tags sewn into your clothing. Or shoplifted. One of the amusing things, though, is that despite being the cheap option (or perhaps beacuse of it), RECCO is relatively poorly backed by data. The number of actual rescues using RECCO remains small, although the development of the helicopter-based detector will probably lead to more successful uses.

Or, you know, just stay out of the mountains. They may be calling, but you don't have to go.

Astute readers noticed last week that this editor (that is to say, me) had his own error'd failure to remember what day it was. Thank you for pointing it out promptly, and then proceeding to send in a bunch of examples of other sites calendar failures. Misery loves company!

Traveler's travails, from C_Chell "Trying to complete the form on https://www.ihg.com to tell when I plan to arrive at the hotel, I can't complete the form because of this little time problem."

"You Have -1 Month(s) To Order!" announces dragoncoder047. "Ah, GradImages... the company that told all graduates that they'd get a free 5x7 but tried to charge me for it, then refused to honor my "unsubscribe" request and is *still* emailing me to this day... Can't do date math? Par for the course."

"Stansted Temporal UI design" shared by Michael R. "While waiting for a friend to arrive at Stansted I see this. I better fire up the DeLorean to pick her up at 00:06 tomorrow."

While he was hunting through the website, Michael R. also found that "The Stansted airport website seems to suffer from Directional Confusion."

Nothing wrong with the calendar here, but Slaoput simply opposes mandatory existence. "I was filling out a form that said the Birthdate is optional, but when I hit submit I found out it was required. (I guess technically you have to be born to fill out the form.)"
NOT TO BE!

A common option for retrofitting heating and cooling into older homes is a mini-split, frequently tied to a heat pump. They're (relatively) cheap to install, energy efficient, and can be added without substantial modifications to the home. They also, annoyingly, are mostly controlled via IR remotes, making them challenging to wire up to home automation or even a household thermostat.

People have made solutions, and today's code comes from one of those solutions. Which, I want to stress, this code comes from an open source project for home automation, so it's not the code that's wrong, here. At first I thought it was, and had a moment of, "I'm not going to pick on some hobby project," but then I realised the hobby project points at a deeper issue.

// temperature helper these are direct mappings based on the remote
float toFahrenheit(float fromCelsius) {
    // Lookup table for specific mappings
    const std::map<float, int> lookupTable = {
        {16.0, 61}, {16.5, 62}, {17.0, 63}, {17.5, 64}, {18.0, 65},
        {18.5, 66}, {19.0, 67}, {20.0, 68}, {21.0, 69}, {21.5, 70},
        {22.0, 71}, {22.5, 72}, {23.0, 73}, {23.5, 74}, {24.0, 75},
        {24.5, 76}, {25.0, 77}, {25.5, 78}, {26.0, 79}, {26.5, 80},
        {27.0, 81}, {27.5, 82}, {28.0, 83}, {28.5, 84}, {29.0, 85},
        {29.5, 86}, {30.0, 87}, {30.5, 88}
    };

    // Check if the input is in the lookup table
    auto it = lookupTable.find(fromCelsius);
    if (it != lookupTable.end()) {
        return it->second;
    }

    // Default conversion and rounding to nearest integer
    return roundf(fromCelsius * 1.8 + 32.0);
}

Okay, I am going to pick on their code a little bit; using float as a key in a map is asking for trouble, because rounding errors are going to surprise you. But honestly, failing to find the key you're looking for is better than the opposite, since that actually does the correct thing. Because if you look carefully at the table, you'll see that it's wrong.

18C, for example, should be 64F. Well, 64.4F, but we're rounding to an integer. The choice here is to roughly map every 0.5C increase to a 1F increase, which is not the conversion factor. They try and correct- note how the table mostly steps by 0.5C, but skips 19.5C.

The opposite direction is similarly bad:

// temperature helper these are direct mappings based on the remote
float toCelsius(float fromFahrenheit) {
    // Lookup table for specific mappings
    const std::map<int, float> lookupTable = {
        {61, 16.0}, {62, 16.5}, {63, 17.0}, {64, 17.5}, {65, 18.0},
        {66, 18.5}, {67, 19.0}, {68, 20.0}, {69, 21.0}, {70, 21.5},
        {71, 22.0}, {72, 22.5}, {73, 23.0}, {74, 23.5}, {75, 24.0},
        {76, 24.5}, {77, 25.0}, {78, 25.5}, {79, 26.0}, {80, 26.5},
        {81, 27.0}, {82, 27.5}, {83, 28.0}, {84, 28.5}, {85, 29.0},
        {86, 29.5}, {87, 30.0}, {88, 30.5}
    };

    // Check if the input is in the lookup table
    auto it = lookupTable.find(static_cast<int>(fromFahrenheit));
    if (it != lookupTable.end()) {
        return it->second;
    }

    // Default conversion and rounding to nearest 0.5
    return roundf((fromFahrenheit - 32.0) / 1.8 * 2) / 2.0;
}

Here, we can be off by as much as a 1C, which is certainly a noticeable feeling.

At first glance, I thought this was just a misguided attempt at optimizing the lookup. For common values, do a lookup instead of calculating because it's faster. Seems like the kind of mistake a hobby project might make, and definitely not a WTF. But it's the comment which corrects me: these are direct mappings based on the remote.

These remotes usually have a display. So when you see on the remote that you're trying to set the temperature to a comfortable 72F, the remote is actually sending 22.5C to the unit. That's the actual temperature being sent.

Now, why on Earth does the remote behave this way? Well, I haven't cracked one open to read off the part numbers, but I'm going to go out on a limb and guess that the microcontoller in the remote doesn't handle floating point operations all that well. So it almost certainly does use a lookup table to decide what signal to send, and the lookup table is populated by "good enough" approximations of temperature conversions. There aren't a lot of places that use Fahrenheit, so being "close enough" is a reasonable solution. If you want accurate temperatures, use SI units, not "freedom units".

In the end, I'd say that neither the hobby project, nor the remote control are the WTF here; locales that insist on using weird ass units are.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Rachel joined a new team which was proudly "metrics driven". When she first met with her boss, Zane, he explained his thinking.

"We need to be data-driven to make good decisions, right? We're a manufacturing company. We make widgets. At the end of the day, we need to make the most widgets for the lowest cost of goods sold. So we track that, and that feeds into every decision."

The team oversaw an automated production line, which meant the software was a mix of robotics, embedded firmware, high-level web based monitoring tools, and thickets of dreaded PLC code. And because you can't build an entire factory for test purposes, they only way they could test real-world scales with real-world data was to roll changes out to production. They could simulate, they could run tests on subsets of the system, but a change in the production line software couldn't truly be validated until it rolled out into the real world.

Rachel's first task on the new team involved making some changes to their metrics dashboard. It was viewed as a good way to get her feet wet with the new team. As it turned out, the metrics dashboard was a Google Sheet, with a complex series of formulas that involved multi-level INDEX functions- essentially querying the spreadsheets like they were a database. Why not use an actual database? Oh, they did — six actually — but the company obeyed Remy's Law of Requirements Gathering: "no matter what the requirements the users ask for, what they really wanted was Excel". The database data was pulled into the spreadsheet for reporting.

Now, a complicated sheet pulling in data from not one, but six different databases, they must have a pretty complex model to explain how changes to their software would impact productivity. And since they needed to model the software to make predictions about how it'd behave in production, that model must be extremely useful.

Of course it wasn't. The only metrics they tracked were output metrics, variations on "widgets produced per unit time". There were some performance metrics, so you could maybe potentially identify "oh, our overall throughput dropped because unit 5 became a bottleneck and started taking 1.5 extra seconds per widget", but nothing that actually helped you understand how the complex system made decisions. Or even why unit 5 was taking longer.

For example, there was an automated quality control scanner. It examined widgets as they came off the line, and rejected defective ones based on a computer vision algorithm. Did that subsystem record why it rejected a widget? No, it did not. The CV model was able to tag widgets with a defect category based on what it saw, but that information didn't get recorded anywhere. In fact, it didn't even record how many widgets got rejected. The only way to know was to have an operator on the assembly line count widgets in the bin manually. Since that ate up a bunch of an operator's time, it never happened unless the developers begged for it. And since the operator still couldn't answer the question "why was this widget rejected", it wasn't all that useful anyway.

Every change to the software was scored against the overall output metrics. This meant that when Rachel was ready to push out her first software change, something that would record how many widgets were rejected and why, whether or not it could be deployed was dependent on seeing the change improve, or at least not regress, the widgets-over-time scores. But the widgets-over-time were a noisy metric; it varied based on which operators were working any given shift, or based on supply chain constraints. Or sometimes, based on when one of the machines was last calibrated- theoretically something that happened on a set schedule, but really was up to the operators. This meant the first three times Rachel rolled her code out for a test run, the metrics regressed. Nothing she changed should have impacted the metrics, but the metrics regressed due to environmental issues.

This meant making a simple change could take weeks, because you could only do final validation on the real system, which means you had to mark off a block of time for a test run, you could only run a handful of tests a day, and if metrics regressed you had to account for that before you could release the software for actual production use.

Over the first few months, Rachel added instrumentation to the code. Anything along the way to generating an output widget, she recorded. The hope was that once they had enough data, they could build a useful model of the system. Unfortunately, Zane had other ideas.

"So, you haven't improved our metrics," Zane said. "Which, I remind you, we're a metrics driven organization. Every change needs to improve our metrics."

"Sure, but I'm gathering more data so we have a better idea of what makes our metrics tick. We don't know why our system does some of the things it does, because we don't record any logging about the decisions it makes."

"Right, but we already gather the key metrics."

"But you don't gather the data that tells you why those metrics are what they are!"

"Sure," Zane said. "But those aren't our key metrics."

That, unfortunately for Rachel, was where things landed. Understanding their complex system was a low priority. Pushing top-level metrics without understanding what fed into them, that was the priority. That didn't mean Rachel was powerless: any time she made a change that she thought might help the top level metrics, she also made sure to add instrumentation that explained how that change behaved. It was the compromise that kept Zane happy: she released features that impacted the top-level metrics, but she also made the system more observable.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Today's representative comment ran out of room.

int maxLen = getColumnSize(session, "audit", "text_value1") - 16; // Leave some room for

No, it isn't continued on the next line and just got trimmed out, except perhaps by a careless merge. This is the entire comment.

Clearly, written by David Chase, the creator of "The Sopranos".

There are so many things we might be leaving room for. We could leave some room for dessert. Leave some room for activities. Leave some room for the holy spirit. Leave some room for improvisation.

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

All I can say in response to our anonymous submitter's story is, ALMOST?!

With the World Cup being hosted in North America this year, I remembered this story that happened back in 2014. At the time I was working in Brazil, for a company that builds software systems for public services. And, with the World Cup being hosted there, in came the opportunity for local agencies to invest in modernization, with pretty much a blank check to get new services, so long as it was deployed before the end of the World Cup. And so the sales people did what they did best, and went around trying to upsell whoever would be willing to buy — no matter our actual capacity for developing the things.

So it was that I was pulled into this new fancy digital system for the police force of a state capital. However, we had only about 4 engineers available, and what they sold was a project estimated for a team of 20, to be delivered in 3 months, with no room for delay. And it wasn't just our core C&D product, but this massive thing with customized public-facing websites, live tracking of the position of different police cars delivered to a tablet in each car, automated reporting, etc.

First thing: We received a pile of 24 resumes, and were told to choose 16 of those. Maybe 3 were acceptable, but we had to waste 1 month hiring and onboarding 13 other people who were worse than useless. Classic man-month problem. We eventually had to tell management that nothing would be delivered this way, so they did the very best next thing: fly us to this other city, so we could work embedded there, in full crunch mode for the delivery. We pretty much worked 12+ hours a day, 7 days a week, for those next 2 weeks.

Another situation: they wanted this system where people could take a photo of an incident in progress, and submit via this app + website, to be verified by an operator in real-time. We nicknamed it the "dick-pic encyclopedia." Even worse, we only had the budget to run a single server, so this thing receiving public traffic would live in the same system that was tracking police car locations. Luckily they were convinced it was a bad idea so it was only ever online for a short period of time.

Next, was the police car tracking. This was done by a tablet installed in each car, which would be sending and receiving location information. But, 1 week before our deadline, we were hitting a serious bug: everything was working when we ran the tests ourselves, but the cops would report very weird bugs when testing it in the field. So we asked to do some field debugging, and I went on a ride-along. Things were working pretty much fine everywhere, so I asked to be taken to where he remembered seeing the tablets fail—to which the policeman just decides to drive off straight into one of the favelas around the city. I guess I can cross out "doing debugging in a police car passenger seat in a notoriously dangerous neighborhood" off my bucket list. Root cause: turns out cellphone connections would be pretty spotty in those areas, which we weren't handling properly.

Either way, we delivered something on time that was severely below spec, and very much over-budget. Company tried to squirm out of paying overtime (was told that we would gain "prestige" by doing those extra hours), but I put my foot down and left that job shortly after. Last I heard they actually got sued for this and a bunch of similar projects, and almost went under.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Peter G. shared with us yet another ordering bungled example of. "Should really say "please engage in an Easter egg hunt to find your language"."

"Google can't count" claimed Peter S.. It adds up. "Yet another proof that 0=1, this time from Google."

"Thanks, Microsoft" groused Ivan "Ever since Microsoft ate university e-mail services worldwide and became responsible for major free software mailing lists, quality of service has been steadily dropping. In order to report delivery problems to Outlook, you need a Microsoft account. You're prevented from creating it at first because of "suspicious activity". Once you're in, the contact address is pre-filled for you with an invalid email. Once you fix that in the web developer toolbar, fuck you anyway! I think the form isn't actually expected to work; the fact that the request was submitted is an error. The only thing missing from the experience is the "beware of the leopard" sign."

"Mango Math" needs a bit of money math for the rest of the world to understand. Michael R. muttered "I will buy it by the slice then." The joke here is on the tip of my tongue. Explainer: the new pence is one hundredth of the decimal pound. No shillings no more, decreps! At that ratio, 3p per slice of cheesecake would indeed be far less dear than four pounds for the whole thing, barring translucent slices. Alas, the reality is simply the boring fact that the price is 3p per gram. Not as funny but I'm chuckling imagining Michael's transparent serving of diet cheesecake. I'll leave it up to you to decide if a gram really counts as an "item".

Clint clucked "Got this email from Bigbadtoystore. Lots of links available for preorder!" I think the talented website builders behind the New Mexico DOT have been busy.

Some time ago, Charles shared with us some awful PHP, aka the most common sort. Today's code sample is maybe a little too big to sum up, but I'll let Charles take a crack at it.

It's so bad that even analyzing and laughing at it feels impossible. But it’s so bad, I couldn’t not share it.

I’m the only one handling all the IT-related tasks at my company, and I don’t have anyone here to vent or laugh about this kind of thing with. So, I figured, why not share it here? I’m hoping it’ll provide at least a little bit of catharsis or some dark humor.

To make sure the confidentiality of the codebase was respected, I took the liberty of generalizing it. You might notice some inconsistencies, but that’s just me trying to keep things neutral while protecting the original structure and functionality. Apologies if it looks a bit patchy – the goal was to avoid revealing any specific details or sensitive code.

The whole block is north of 400 lines, and it's doing a lot. Or well, maybe it's not, as you'll see.

Let's star with the outermost layer.

$resm_data = $data_source->fetchData("group=" . $item_id);
foreach ($resm_data as $key => $value) {
// rest of the code here
}

We fetch data from a data source, presumably a database, passing our condition as a string, which reeks of probable SQL injection, but I don't know what library they're using. I also note they're using the key/value style of array iteration, but never actually check the key.

    $option_id = $value->option_id;
    $resm_details = $detail_source->fetch($option_id);
    if ($resm_details) {
        $label = $resm_details->{"label$lang"};
        $description = $resm_details->{"description$lang"};
        $category = $resm_details->category;

Nice little bit of "meta" programming to get their localization working, it'll fetch labelen or labelde as needed. Definitely not a horrible, dangerous way to solve that problem.

We use that again to get our currency figured out. That lets us do number formatting. So much number formatting code.

        if ($category == 0) {
            $cost = $resm_details->{"cost" . $currency};
            $child_cost = $resm_details->{"child_cost" . $currency};
            $cost_info =
                $mot[101] . " : +" . number_format($cost, 2, ",", " ") . $currency_symbol . " - " .
                $mot[102] . " : +" . number_format($child_cost, 2, ",", " ") . $currency_symbol;
        } elseif ($category == 1) {
            $cost = $resm_details->{"cost" . $currency};
            $child_cost = 0;
            $cost_info = $mot[101] . " : +" . number_format($cost, 2, ",", " ") . $currency_symbol;
        } elseif ($category == 2) {
            $cost = 0;
            $child_cost = $resm_details->{"child_cost" . $currency};
            $cost_info = $mot[102] . " : +" . number_format($child_cost, 2, ",", " ") . $currency_symbol;
        } elseif ($category == 4) {
            $cost = $resm_details->{"cost" . $currency};
            $child_cost = 0;
            $cost_info = $mot[500] . " : +" . number_format($cost, 2, ",", " ") . $currency_symbol;
        } elseif (
            $resm_details->{"cost" . $currency} == 0 and
            $resm_details->{"child_cost" . $currency} == 0
        ) {
            $cost = 0;
            $child_cost = 0;
            $cost_info = "";
        } else {
            $cost = $resm_details->{"cost" . $currency};
            $child_cost = 0;
            $cost_info = $mot[200] . " : +" . number_format($cost, 2, ",", " ") . $currency_symbol;
        }

What is the $mot array? Why am I jumping to seemingly random locations in that array? Clearly it contains some headers for our output.

Anyway, there's plenty of HTML string munging happening too, don't you worry.

        if ($location == 1) {
            $quantity_block =
                '<div class="quantity-container">
                    <input type="text" value="1" id="quantity-' . $option_id . '" class="qty-control" name="quantity" min="1" max="1">
                    <div class="increase button">+</div>
                    <div class="decrease button">-</div>
                </div>';
            $quantity = 1;
        } else {
            $quantity_block = "";
            $quantity = "all";
        }

And then there's this little treat for parsing the time stored in our database: $time_data = json_decode($resm_details->time_data); That tells me they're storing date times as strings, so that's fun.

There are also a couple more bon mots as they build a drop down list:

            $departure_select = '<option value="-1">' . $mot[300] . '</option>';
            $arrival_select = '<option value="-1">' . $mot[301] . '</option>';

And then there's this monstrosity:

        // Add the main option to the template
        $TEMPLATE->SET_BLOCK_VARIABLES("Block_OPTIONS", [
            "ID" => $option_id,
            "QUANTITY" => $quantity,
            "LABEL_CLASS" => $label_class,
            "ACTION_CLASS" => $action_class,
            "LABEL" => $label,
            "DESCRIPTION" => $description,
            "QUANTITY_BLOCK" => $quantity_block,
            "COST_INFO" => $cost_info,
            "HOST_COST_DISPLAY" => $host_cost_display,
            "COST" => $cost,
            "CHILD_COST" => $child_cost,
            "LOCATION" => $location,
            "CATEGORY" => $category,
            "HOST_CLASS" => $host_class,
            "EXTRA" => $extra,
            "HOST_EXTRA" => $host_extra,
            "TIME_CHECKED" => ($has_times == 1) ? 'selected="' . $option_id . '" checked="true"' : '',
            "TIME_BLOCK" => $time_block
        ]);

All the nonsense that we concatenate together above gets shoved into some sort of template. And after that, that's where the good stuff starts. Because guess what? We have to do the same thing for child items.

        $resm_children_data = $data_source->fetchData("parent=" . $option_id);
        foreach ($resm_children_data as $child_key => $child_value) {
            $child_option_id = $child_value->option_id;
            $resm_child_details = $detail_source->fetch($child_option_id);

That's right, it's the same block of code, not quite copy/pasted, since they needed to put the word child in everything.

                // Add the child option to the template
                $TEMPLATE->SET_BLOCK_VARIABLES("Block_OPTIONS.Block_OPTIONS_CHILD", [
                    "ID" => $child_option_id,
                    "QUANTITY" => $child_quantity,
                    "LABEL_CLASS" => $child_label_class,
                    "ACTION_CLASS" => $child_action_class,
                    "LABEL" => $child_label,
                    "DESCRIPTION" => $child_description,
                    "QUANTITY_BLOCK" => $child_quantity_block,
                    "COST_INFO" => $child_cost_info,
                    "HOST_COST_DISPLAY" => $child_host_cost_display,
                    "COST" => $child_cost,
                    "CHILD_COST" => $child_extra_cost,
                    "LOCATION" => $child_location,
                    "CATEGORY" => $child_category,
                    "HOST_CLASS" => $child_host_class,
                    "EXTRA" => $child_extra,
                    "HOST_EXTRA" => $child_host_extra,
                    "TIME_CHECKED" => ($child_has_times == 1) ? 'selected="' . $child_option_id . '" checked="true"' : '',
                    "TIME_BLOCK" => $child_time_block
                ]);

And now, if you liked the child record, guess what? Those children have got siblings. What can I say, it's a big family.

                $resm_sibling_data = $data_source->fetchData("parent=" . $parent_option_id);
                foreach ($resm_sibling_data as $sibling_key => $sibling_value) {
                    $sibling_option_id = $sibling_value->option_id;
                    $resm_sibling_details = $detail_source->fetch($sibling_option_id);

And that means, yes, we also use that template thing again:

                        // Add the sibling option to the template
                        $TEMPLATE->SET_BLOCK_VARIABLES("Block_OPTIONS.Block_OPTIONS_CHILD", [
                            "ID" => $sibling_option_id,
                            "QUANTITY" => $sibling_quantity,
                            "LABEL_CLASS" => $sibling_label_class,
                            "ACTION_CLASS" => $sibling_action_class,
                            "LABEL" => $sibling_label,
                            "DESCRIPTION" => $sibling_description,
                            "QUANTITY_BLOCK" => $sibling_quantity_block,
                            "COST_INFO" => $sibling_cost_info,
                            "HOST_COST_DISPLAY" => $sibling_host_cost_display,
                            "COST" => $sibling_cost,
                            "SIBLING_COST" => $sibling_extra_cost,
                            "LOCATION" => $sibling_location,
                            "CATEGORY" => $sibling_category,
                            "HOST_CLASS" => $sibling_host_class,
                            "EXTRA" => $sibling_extra,
                            "HOST_EXTRA" => $sibling_host_extra,
                            "TIME_CHECKED" => ($sibling_has_times == 1) ? 'selected="' . $sibling_option_id . '" checked="true"' : '',
                            "TIME_BLOCK" => $sibling_time_block
                        ]);

Someday, I hope the person who wrote this learn about methods and function calls. Maybe they could write their own one day.

In any case, here's the whole thing:

$resm_data = $data_source->fetchData("group=" . $item_id);
foreach ($resm_data as $key => $value) {
    $option_id = $value->option_id;
    $resm_details = $detail_source->fetch($option_id);
    if ($resm_details) {
        $label = $resm_details->{"label$lang"};
        $description = $resm_details->{"description$lang"};
        $category = $resm_details->category;

        // Process cost based on category
        if ($category == 0) {
            $cost = $resm_details->{"cost" . $currency};
            $child_cost = $resm_details->{"child_cost" . $currency};
            $cost_info =
                $mot[101] . " : +" . number_format($cost, 2, ",", " ") . $currency_symbol . " - " .
                $mot[102] . " : +" . number_format($child_cost, 2, ",", " ") . $currency_symbol;
        } elseif ($category == 1) {
            $cost = $resm_details->{"cost" . $currency};
            $child_cost = 0;
            $cost_info = $mot[101] . " : +" . number_format($cost, 2, ",", " ") . $currency_symbol;
        } elseif ($category == 2) {
            $cost = 0;
            $child_cost = $resm_details->{"child_cost" . $currency};
            $cost_info = $mot[102] . " : +" . number_format($child_cost, 2, ",", " ") . $currency_symbol;
        } elseif ($category == 4) {
            $cost = $resm_details->{"cost" . $currency};
            $child_cost = 0;
            $cost_info = $mot[500] . " : +" . number_format($cost, 2, ",", " ") . $currency_symbol;
        } elseif (
            $resm_details->{"cost" . $currency} == 0 and
            $resm_details->{"child_cost" . $currency} == 0
        ) {
            $cost = 0;
            $child_cost = 0;
            $cost_info = "";
        } else {
            $cost = $resm_details->{"cost" . $currency};
            $child_cost = 0;
            $cost_info = $mot[200] . " : +" . number_format($cost, 2, ",", " ") . $currency_symbol;
        }

        $location = $resm_details->location;
        $label_class = $location == 1 ? "has-quantity" : "no-quantity";
        $action_class = $location == 1 ? "active-with-quantity" : "active-no-quantity";
        if ($location == 1) {
            $quantity_block =
                '<div class="quantity-container">
                    <input type="text" value="1" id="quantity-' . $option_id . '" class="qty-control" name="quantity" min="1" max="1">
                    <div class="increase button">+</div>
                    <div class="decrease button">-</div>
                </div>';
            $quantity = 1;
        } else {
            $quantity_block = "";
            $quantity = "all";
        }

        $has_times = $resm_details->has_times;

        if ($has_times == 1) {
            $time_counter++;
            $time_data = json_decode($resm_details->time_data);

            $departure_select = '<option value="-1">' . $mot[300] . '</option>';
            $arrival_select = '<option value="-1">' . $mot[301] . '</option>';

            $departures = $time_data->departures;
            $arrivals = $time_data->arrivals;

            foreach ($departures as $dep_key => $departure) {
                $departure_select .= '<option value="' . $dep_key . '">' . $departure . '</option>';
            }

            foreach ($arrivals as $arr_key => $arrival) {
                $arrival_select .= '<option value="' . $arr_key . '">' . $arrival . '</option>';
            }

            $departure_block = '<div class="col-half time-select-' . $option_id . '" style="padding-right: 0;">
                <select class="form-control time-departure" style="text-align: center;">' . $departure_select . '</select>
                </div>';

            $arrival_block = '<div class="col-half time-select-' . $option_id . '" style="padding-left: 0;">
                <select class="form-control time-arrival" style="text-align: center;">' . $arrival_select . '</select>
                </div>';

            $time_block = '<div class="row time-container">
                ' . $departure_block . $arrival_block . '
            </div><small class="error-message time-error">' . $mot[302] . '</small>';
        } else {
            $time_block = '';
        }

        $host_cost_display = "";
        $extra = $resm_details->extra;
        $host_extra = $resm_details->host_extra;
        $host_class = "";

        if ($extra == 1) {
            $extra_cost_1 = $resm_details->{"cost" . $currency . "_1"};
            $extra_cost_2 = $resm_details->{"cost" . $currency . "_2"};

            $default_extra = $host_price == 0 ? $extra_cost_2 : $extra_cost_1;
            $cost_info = $mot[101] . ' : +<span class="extra-cost">' . number_format($default_extra, 2, ",", " ") .
                "</span>" . $currency_symbol . " - " . $mot[102] . ' : +<span class="child-cost">0.00</span>' . $currency_symbol;

            $host_class = " extra-option";
        }
        if ($host_extra == 1) {
            $host_cost_display =
                '<span class="host-cost">' . $mot[500] . ' : +<span class="host-price">0.00</span>' . $currency_symbol .
                "</span><br>";
        }

        // Add the main option to the template
        $TEMPLATE->SET_BLOCK_VARIABLES("Block_OPTIONS", [
            "ID" => $option_id,
            "QUANTITY" => $quantity,
            "LABEL_CLASS" => $label_class,
            "ACTION_CLASS" => $action_class,
            "LABEL" => $label,
            "DESCRIPTION" => $description,
            "QUANTITY_BLOCK" => $quantity_block,
            "COST_INFO" => $cost_info,
            "HOST_COST_DISPLAY" => $host_cost_display,
            "COST" => $cost,
            "CHILD_COST" => $child_cost,
            "LOCATION" => $location,
            "CATEGORY" => $category,
            "HOST_CLASS" => $host_class,
            "EXTRA" => $extra,
            "HOST_EXTRA" => $host_extra,
            "TIME_CHECKED" => ($has_times == 1) ? 'selected="' . $option_id . '" checked="true"' : '',
            "TIME_BLOCK" => $time_block
        ]);

        $resm_children_data = $data_source->fetchData("parent=" . $option_id);
        foreach ($resm_children_data as $child_key => $child_value) {
            $child_option_id = $child_value->option_id;
            $resm_child_details = $detail_source->fetch($child_option_id);
            if ($resm_child_details) {
                $child_label = $resm_child_details->{"label$lang"};
                $child_description = $resm_child_details->{"description$lang"};
                $child_category = $resm_child_details->category;

                // Process cost for child category
                if ($child_category == 0) {
                    $child_cost = $resm_child_details->{"cost" . $currency};
                    $child_extra_cost = $resm_child_details->{"child_cost" . $currency};
                    $child_cost_info =
                        $mot[101] . " : +" . number_format($child_cost, 2, ",", " ") . $currency_symbol . " - " .
                        $mot[102] . " : +" . number_format($child_extra_cost, 2, ",", " ") . $currency_symbol;
                } elseif ($child_category == 1) {
                    $child_cost = $resm_child_details->{"cost" . $currency};
                    $child_extra_cost = 0;
                    $child_cost_info = $mot[101] . " : +" . number_format($child_cost, 2, ",", " ") . $currency_symbol;
                } elseif ($child_category == 2) {
                    $child_cost = 0;
                    $child_extra_cost = $resm_child_details->{"child_cost" . $currency};
                    $child_cost_info = $mot[102] . " : +" . number_format($child_extra_cost, 2, ",", " ") . $currency_symbol;
                } elseif ($child_category == 4) {
                    $child_cost = $resm_child_details->{"cost" . $currency};
                    $child_extra_cost = 0;
                    $child_cost_info = $mot[500] . " : +" . number_format($child_cost, 2, ",", " ") . $currency_symbol;
                } elseif (
                    $resm_child_details->{"cost" . $currency} == 0 and
                    $resm_child_details->{"child_cost" . $currency} == 0
                ) {
                    $child_cost = 0;
                    $child_extra_cost = 0;
                    $child_cost_info = "";
                } else {
                    $child_cost = $resm_child_details->{"cost" . $currency};
                    $child_extra_cost = 0;
                    $child_cost_info = $mot[200] . " : +" . number_format($child_cost, 2, ",", " ") . $currency_symbol;
                }

                $child_location = $resm_child_details->location;
                $child_label_class = $child_location == 1 ? "has-quantity" : "no-quantity";
                $child_action_class = $child_location == 1 ? "active-with-quantity" : "active-no-quantity";
                if ($child_location == 1) {
                    $child_quantity_block =
                        '<div class="quantity-container">
                    <input type="text" value="1" id="child-quantity-' . $child_option_id . '" class="qty-control" name="quantity" min="1" max="1">
                    <div class="increase button">+</div>
                    <div class="decrease button">-</div>
                </div>';
                    $child_quantity = 1;
                } else {
                    $child_quantity_block = "";
                    $child_quantity = "all";
                }

                $child_has_times = $resm_child_details->has_times;

                if ($child_has_times == 1) {
                    $child_time_counter++;
                    $child_time_data = json_decode($resm_child_details->time_data);

                    $child_departure_select = '<option value="-1">' . $mot[300] . '</option>';
                    $child_arrival_select = '<option value="-1">' . $mot[301] . '</option>';

                    $child_departures = $child_time_data->departures;
                    $child_arrivals = $child_time_data->arrivals;

                    foreach ($child_departures as $child_dep_key => $child_departure) {
                        $child_departure_select .= '<option value="' . $child_dep_key . '">' . $child_departure . '</option>';
                    }

                    foreach ($child_arrivals as $child_arr_key => $child_arrival) {
                        $child_arrival_select .= '<option value="' . $child_arr_key . '">' . $child_arrival . '</option>';
                    }

                    $child_departure_block = '<div class="col-half time-select-' . $child_option_id . '" style="padding-right: 0;">
                <select class="form-control time-departure" style="text-align: center;">' . $child_departure_select . '</select>
                </div>';

                    $child_arrival_block = '<div class="col-half time-select-' . $child_option_id . '" style="padding-left: 0;">
                <select class="form-control time-arrival" style="text-align: center;">' . $child_arrival_select . '</select>
                </div>';

                    $child_time_block = '<div class="row time-container">
                ' . $child_departure_block . $child_arrival_block . '
            </div><small class="error-message time-error">' . $mot[302] . '</small>';
                } else {
                    $child_time_block = '';
                }

                $child_host_cost_display = "";
                $child_extra = $resm_child_details->extra;
                $child_host_extra = $resm_child_details->host_extra;
                $child_host_class = "";

                if ($child_extra == 1) {
                    $child_extra_cost_1 = $resm_child_details->{"cost" . $currency . "_1"};
                    $child_extra_cost_2 = $resm_child_details->{"cost" . $currency . "_2"};

                    $default_child_extra = $host_price == 0 ? $child_extra_cost_2 : $child_extra_cost_1;
                    $child_cost_info = $mot[101] . ' : +<span class="extra-cost">' . number_format($default_child_extra, 2, ",", " ") .
                        "</span>" . $currency_symbol . " - " . $mot[102] . ' : +<span class="child-cost">0.00</span>' . $currency_symbol;

                    $child_host_class = " extra-option";
                }
                if ($child_host_extra == 1) {
                    $child_host_cost_display =
                        '<span class="host-cost">' . $mot[500] . ' : +<span class="host-price">0.00</span>' . $currency_symbol .
                        "</span><br>";
                }

                // Add the child option to the template
                $TEMPLATE->SET_BLOCK_VARIABLES("Block_OPTIONS.Block_OPTIONS_CHILD", [
                    "ID" => $child_option_id,
                    "QUANTITY" => $child_quantity,
                    "LABEL_CLASS" => $child_label_class,
                    "ACTION_CLASS" => $child_action_class,
                    "LABEL" => $child_label,
                    "DESCRIPTION" => $child_description,
                    "QUANTITY_BLOCK" => $child_quantity_block,
                    "COST_INFO" => $child_cost_info,
                    "HOST_COST_DISPLAY" => $child_host_cost_display,
                    "COST" => $child_cost,
                    "CHILD_COST" => $child_extra_cost,
                    "LOCATION" => $child_location,
                    "CATEGORY" => $child_category,
                    "HOST_CLASS" => $child_host_class,
                    "EXTRA" => $child_extra,
                    "HOST_EXTRA" => $child_host_extra,
                    "TIME_CHECKED" => ($child_has_times == 1) ? 'selected="' . $child_option_id . '" checked="true"' : '',
                    "TIME_BLOCK" => $child_time_block
                ]);

                $resm_sibling_data = $data_source->fetchData("parent=" . $parent_option_id);
                foreach ($resm_sibling_data as $sibling_key => $sibling_value) {
                    $sibling_option_id = $sibling_value->option_id;
                    $resm_sibling_details = $detail_source->fetch($sibling_option_id);
                    if ($resm_sibling_details) {
                        $sibling_label = $resm_sibling_details->{"label$lang"};
                        $sibling_description = $resm_sibling_details->{"description$lang"};
                        $sibling_category = $resm_sibling_details->category;

                        // Process cost for sibling category
                        if ($sibling_category == 0) {
                            $sibling_cost = $resm_sibling_details->{"cost" . $currency};
                            $sibling_extra_cost = $resm_sibling_details->{"sibling_cost" . $currency};
                            $sibling_cost_info =
                                $mot[101] . " : +" . number_format($sibling_cost, 2, ",", " ") . $currency_symbol . " - " .
                                $mot[102] . " : +" . number_format($sibling_extra_cost, 2, ",", " ") . $currency_symbol;
                        } elseif ($sibling_category == 1) {
                            $sibling_cost = $resm_sibling_details->{"cost" . $currency};
                            $sibling_extra_cost = 0;
                            $sibling_cost_info = $mot[101] . " : +" . number_format($sibling_cost, 2, ",", " ") . $currency_symbol;
                        } elseif ($sibling_category == 2) {
                            $sibling_cost = 0;
                            $sibling_extra_cost = $resm_sibling_details->{"sibling_cost" . $currency};
                            $sibling_cost_info = $mot[102] . " : +" . number_format($sibling_extra_cost, 2, ",", " ") . $currency_symbol;
                        } elseif ($sibling_category == 4) {
                            $sibling_cost = $resm_sibling_details->{"cost" . $currency};
                            $sibling_extra_cost = 0;
                            $sibling_cost_info = $mot[500] . " : +" . number_format($sibling_cost, 2, ",", " ") . $currency_symbol;
                        } elseif (
                            $resm_sibling_details->{"cost" . $currency} == 0 and
                            $resm_sibling_details->{"sibling_cost" . $currency} == 0
                        ) {
                            $sibling_cost = 0;
                            $sibling_extra_cost = 0;
                            $sibling_cost_info = "";
                        } else {
                            $sibling_cost = $resm_sibling_details->{"cost" . $currency};
                            $sibling_extra_cost = 0;
                            $sibling_cost_info = $mot[200] . " : +" . number_format($sibling_cost, 2, ",", " ") . $currency_symbol;
                        }

                        $sibling_location = $resm_sibling_details->location;
                        $sibling_label_class = $sibling_location == 1 ? "has-quantity" : "no-quantity";
                        $sibling_action_class = $sibling_location == 1 ? "active-with-quantity" : "active-no-quantity";
                        if ($sibling_location == 1) {
                            $sibling_quantity_block =
                                '<div class="quantity-container">
                    <input type="text" value="1" id="sibling-quantity-' . $sibling_option_id . '" class="qty-control" name="quantity" min="1" max="1">
                    <div class="increase button">+</div>
                    <div class="decrease button">-</div>
                </div>';
                            $sibling_quantity = 1;
                        } else {
                            $sibling_quantity_block = "";
                            $sibling_quantity = "all";
                        }

                        $sibling_has_times = $resm_sibling_details->has_times;

                        if ($sibling_has_times == 1) {
                            $sibling_time_counter++;
                            $sibling_time_data = json_decode($resm_sibling_details->time_data);

                            $sibling_departure_select = '<option value="-1">' . $mot[300] . '</option>';
                            $sibling_arrival_select = '<option value="-1">' . $mot[301] . '</option>';

                            $sibling_departures = $sibling_time_data->departures;
                            $sibling_arrivals = $sibling_time_data->arrivals;

                            foreach ($sibling_departures as $sibling_dep_key => $sibling_departure) {
                                $sibling_departure_select .= '<option value="' . $sibling_dep_key . '">' . $sibling_departure . '</option>';
                            }

                            foreach ($sibling_arrivals as $sibling_arr_key => $sibling_arrival) {
                                $sibling_arrival_select .= '<option value="' . $sibling_arr_key . '">' . $sibling_arrival . '</option>';
                            }

                            $sibling_departure_block = '<div class="col-half time-select-' . $sibling_option_id . '" style="padding-right: 0;">
                <select class="form-control time-departure" style="text-align: center;">' . $sibling_departure_select . '</select>
                </div>';

                            $sibling_arrival_block = '<div class="col-half time-select-' . $sibling_option_id . '" style="padding-left: 0;">
                <select class="form-control time-arrival" style="text-align: center;">' . $sibling_arrival_select . '</select>
                </div>';

                            $sibling_time_block = '<div class="row time-container">
                ' . $sibling_departure_block . $sibling_arrival_block . '
            </div><small class="error-message time-error">' . $mot[302] . '</small>';
                        } else {
                            $sibling_time_block = '';
                        }

                        $sibling_host_cost_display = "";
                        $sibling_extra = $resm_sibling_details->extra;
                        $sibling_host_extra = $resm_sibling_details->host_extra;
                        $sibling_host_class = "";

                        if ($sibling_extra == 1) {
                            $sibling_extra_cost_1 = $resm_sibling_details->{"cost" . $currency . "_1"};
                            $sibling_extra_cost_2 = $resm_sibling_details->{"cost" . $currency . "_2"};

                            $default_sibling_extra = $host_price == 0 ? $sibling_extra_cost_2 : $sibling_extra_cost_1;
                            $sibling_cost_info = $mot[101] . ' : +<span class="extra-cost">' . number_format($default_sibling_extra, 2, ",", " ") .
                                "</span>" . $currency_symbol . " - " . $mot[102] . ' : +<span class="sibling-cost">0.00</span>' . $currency_symbol;

                            $sibling_host_class = " extra-option";
                        }
                        if ($sibling_host_extra == 1) {
                            $sibling_host_cost_display =
                                '<span class="host-cost">' . $mot[500] . ' : +<span class="host-price">0.00</span>' . $currency_symbol .
                                "</span><br>";
                        }

                        // Add the sibling option to the template
                        $TEMPLATE->SET_BLOCK_VARIABLES("Block_OPTIONS.Block_OPTIONS_CHILD", [
                            "ID" => $sibling_option_id,
                            "QUANTITY" => $sibling_quantity,
                            "LABEL_CLASS" => $sibling_label_class,
                            "ACTION_CLASS" => $sibling_action_class,
                            "LABEL" => $sibling_label,
                            "DESCRIPTION" => $sibling_description,
                            "QUANTITY_BLOCK" => $sibling_quantity_block,
                            "COST_INFO" => $sibling_cost_info,
                            "HOST_COST_DISPLAY" => $sibling_host_cost_display,
                            "COST" => $sibling_cost,
                            "SIBLING_COST" => $sibling_extra_cost,
                            "LOCATION" => $sibling_location,
                            "CATEGORY" => $sibling_category,
                            "HOST_CLASS" => $sibling_host_class,
                            "EXTRA" => $sibling_extra,
                            "HOST_EXTRA" => $sibling_host_extra,
                            "TIME_CHECKED" => ($sibling_has_times == 1) ? 'selected="' . $sibling_option_id . '" checked="true"' : '',
                            "TIME_BLOCK" => $sibling_time_block
                        ]);
                    }
                }
            }
        }
    }
}

[Advertisement] ProGet’s got you covered with security and access controls on your NuGet feeds. Learn more.

Kevin sends us an exception handler from C++. Let's see if we can spot what's going wrong:

catch (Exception::Deadlock)
{
   retry;
}

When we catch a deadlock happening, we retry. That's not a keyword in C++, and looking at how it's used, it has to be some kind of macro, and I suspect that the macro is hiding a goto underneath it.

The real problem, though, is that we suspect we're in a deadlock situation. That means this thread is waiting on a resource held by another thread which is waiting for a resource held by this thread. Neither train may continue until the other has passed. So this retry only works if it releases the resource held by this thread (letting the deadlocking thread proceed). But does it?

Not according ot Kevin. The code already had a pile of deadlocks in it, so they brought in a highly paid consultant to try and fix them by reordering access and tracing where mutexes were causing issues. This retry just jumps back up to the top of the block, without releasing any resources. It "seems the consultant wanted to add some deadlocks of their own," Kevin says.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!

Dan Luu, after finding one wrong prediction after another made by Ed Zitron, comparing him to mediocre futurists, and exploring his writing style:

That last sentence really sums up Zitron’s position. “There are so many guys to be mad at the moment”. In this talk, he throws in this jab at Andreesen and blames Andreesen for Meta, Google, and Microsoft pursuing growth. In reality, if Marc Andreesen had never existed, Meta, Google, and Microsoft would almost certainly still be trying to grow so we of course cannot actually blame Andreesen for these companies trying to grow. There’s just this thing that he says is bad, and in his usual style, he pulls some person and says they’re the evil villain that’s to blame for this, and then moves on to the next non sequitur.

Instead of more carefully scrutinizing Zitron’s record, outlets like Vanity Fair are publishing soft interviews with him where he gets to make predictions like “large language models, when you remove all of the insane financialization, it’s probably a $30 billion-a-year industry”. Oh, sure, they ask a single question about wrong predictions since 2024, but he brushes it off by saying he has learned lots in those two years — Luu documents incorrect predictions all the way up until November 2025, after which “most further predictions that I saw were either non-falsifiable or resolve in the future” — and ends the interview by saying “[w]hat comes after the A.I. bubble is actually a little scarier”. Ominous.

The beauty of Zitron’s voluminous output, for him, is that there is a vast difference between what he actually writes and what people remember. Financial experts and more reputable journalists have raised plenty of concerns about how much money is being spent on developing this infrastructure, and how highly these companies are valued. But the words Zitron writes are far more incendiary and conspiratorial than many seem to remember. It is frustrating to see his many media experiences filling the role of the A.I. skeptic when there are far more qualified, sober, and accurate options. We have enough boosters; this is an industry that is co-signed by the world’s most powerful economies. We deserve better A.I. criticism in popular media.

⌥ Permalink

William Bishop of Pew Research:

Between scrolling, notifications, and messages, smartphones can be hard to put down. Just over half of U.S. adults say they spend too much time on their smartphone, according to a Pew Research Center survey from May and June 2026. About a third of adults say their smartphone use is about right, while just 3% say they spend too little time using these devices.

Notably, 70% of smartphone owners in the U.S. aged 18–29 say they believe they use it too much. I am skeptical of public polling — maybe we are societally more approving of shaming our own device use — but this also tracks with what I hear casually from friends and family. Anecdotally, many people I know have expressed that they want to be on their phones less often.

But the editors of Andreessen Horowitz’s It’s Time to Build newsletter — hosted on Substack, a platform they are investors in — believe this behaviour actually indicates people are very happy. Ruby Thelot, professor of design and media studies at New York University and “astute tech observer”, says the heavy use of social media platforms, like Meta’s Facebook and Instagram — Andreessen Horowitz invested in both, and Marc Andreessen is on Meta’s board; none of this is disclosed — is simply evidence we love them:

People en masse are getting on social media, by choice. From 2016 until now, the number of social media users has grown from 2.5B to close to 6B people. Enshittification isn’t real. It’s three TikToks in a trenchcoat, and, maybe, a book deal. It’s good for discourse but does not describe actual reality and user patterns.

Returning to Instagram. The average daily usage showcases an increase in time spent on Instagram every day as well. Users are coming back for more, in a highly competitive attention arena, year after year.

There is a lot of assumption in these two paragraphs, and I think the “close to 6B people” is worth examining to start. Thelot attributes this statistic in the chart above to Backlinko. It cites no source, but a web search indicates to me this originates with Manochi’s DataReportal, which disclaims “‘user identities’ may not represent unique human individuals” because, as the company explains on the sixth slide of its report, it may count multiple social media accounts or business accounts as individual identities. These are not necessarily people, and Manochi says it is improper to compare figures year over year as Thelot does. (And, to be fair, which Manochi also does on slide 321.)

A lack of rigour is not unique to this data point.

Thelot next shows a chart indicating daily Instagram use rose from 25 minutes in 2017 to nearly 34 in 2026. This is, to Thelot, simply evidence that people like using it and want more. But, to return to the Pew poll above, it seems that people do not feel good about spending more time on their smartphones. Though Pew did not ask (PDF) about social media specifically, roughly half of respondents aged 18–29 said it negatively affects their productivity. And, according to slide 337 of that Manochi report, 30.7–44.0% of people say they use social media to “fill up spare time”, trending higher for younger generations. It is plausible that more younger people are spending increasing amounts of time on social media apps and they do not feel good about it. In other words, the time spent numbers are not a good proxy for enjoyment or value.

Emanuel Maiberg, 404 Media:

People need to be on Linkedin to find jobs. Municipalities and news organizations share important updates on social media first. You might be pulled onto Facebook or WhatsApp against your will because your local school or community of parents congregate there. That doesn’t mean they like it. It is possible to hate something with your entire being and still participate in it.

Regardless of whether you call it “addiction” or some kind of compulsive behaviour, it is plausible many people dislike their own actions but struggle to change them. It is also possible these products are designed to take advantage of that to extract more time out of each user.

⌥ Permalink

Kalley Huang, New York Times (gift link)

John Ternus became Apple’s chief executive on Tuesday, succeeding Tim Cook, the company’s leader for the last 15 years. The long-anticipated handoff, Mr. Cook has said, will be “perfectly smooth.”

[…]

This summer, Apple hired Nate Gatten from American Airlines to lead government affairs, replacing Kate Adams, who will retire this year. Laura Legros, a hardware engineering vice president and deputy of Mr. Ternus’s before retiring from Apple in 2022, has rejoined the company, three people familiar with her hiring said, speaking on the condition of anonymity. Ms. Legros, who reports to Mr. Ternus, could act as his adviser and emissary to various parts of the company, the three people said.

Juli Clover, MacRumors:

Apple’s Phil Schiller is no longer going to run the App Store or oversee product events, reports Bloomberg. Schiller isn’t leaving Apple, but he is narrowing his responsibilities and working on unspecified projects.

Employees at Apple told Bloomberg that 66-year-old Schiller appears to be taking another step toward retirement.

Cook and Ternus each sent pretty anodyne company-wide memos about the transition. Cook’s tenure was the longest of any CEO in Apple’s history and he was the one who turned it from a successful company into a global behemoth.

The thing I have liked about Apple — one of the things that made me a longtime customer and someone who writes about the company — is that it has historically been a very simple kind of business: it designs products and sells them to people, mostly. Every one of its peers is a more complicated business. They often balance the needs of massive institutional and government customers, advertisers, or two-sided marketplaces.

That change began in the latter years of Jobs’ tenure and accelerated under Cook. Most software, including operating systems, was accounted for as part of device purchases, and was turned into a software-as-a-service model. Its subscription-based business became a revenue growth centre, which was important for Wall Street because it was a way to turn the company’s successful but inconsistent device sales into predictable money printers. And there are now ads and upsells throughout the operating systems, which are shown to all users regardless of how much other money they have already given Apple.

Ternus has inherited that Apple. Regardless of how much he gives off the vibe of a cool Californian — by way of Philadelphia — who just cares about the best stuff, he is also selling ad space and making sure more people upgrade to Apple One.

⌥ Permalink

Joel Dryden, in a CBC News article with the headline “On the Road With Pro-Independence Albertans, One Small Town at a Time”:

The restaurant owner, who gained prominence during the COVID-19 pandemic, has hit the road this summer to talk independence, travelling with a group of people who also support the idea of Alberta becoming its own nation.

[…]

On the back of the vehicle, a Bible verse printed in script: Jeremiah 29:7 — “Seek the peace and prosperity of the city to which I have carried you into exile. Pray to the Lord for it, because if it prospers, you too will prosper.”

Mike Skerrett, in a 2018 McSweeney’s article with the headline “I Traveled to a Diner In Trump Country to Write Another Article On Whether the President’s Supporters Still Want to, Quote, ‘Smash My Libtard Face In'”:

I came to this diner, The No Safe Space Café, to get a taste of the Real America. This America exists outside the liberal echo chambers, somewhere with real diversity of thought: The opinions of straight, white, Christian men.

Chris Scott — the “restaurant owner” driving the separatist campaign bus with Dryden aboard — has previously called the CBC “liars, thieves and Federalist bootlickers” who publish “drivel”. Replace the U.S.-centrism of the McSweeney’s piece with a Canadian vibe, and is it really all that different?

⌥ Permalink

Here is a list of headlines; perhaps you will spot a theme:

• “Six new Apple products will launch early this year, here’s what’s coming”

• “Apple has five all-new products launching this year, here’s what’s coming”

• “Apple will launch 20+ new products this year, here’s what’s coming”

• “Apple has rare updates coming soon for these three products”

• “Apple launching 5+ new products over the next several weeks”

• “Seven new Macs will launch this year, here’s everything coming”

• “Apple still has over a dozen products in the pipeline for 2026, here’s what’s coming”

• “Apple has six new products that could launch soon, per rumors

• “Apple has 10+ new products rumored to launch first with John Ternus as CEO”

• “Here’s everything Apple has coming this May, and what not to expect”

• “Apple will launch 15+ new products this fall, here’s what’s coming”

• “Apple has five new products coming that could launch at WWDC”

• “Next year to be Apple’s ‘biggest product year’ ever, here’s what’s coming”

• “Apple has five products launching this fall that we’ve waited a long time for”

• “Apple will launch five new products next month, here’s what’s coming”

• “Apple will launch 10+ new products soon, with September event coming”

• “Apple is about to launch five new products that I’m very excited for”

• “Apple will launch 15+ new products this fall, here’s what’s coming”

• “Apple has announced 11 new products in 2026 so far, with more coming soon”

All of these since January — and I excluded most rewrites of articles by Mark Gurman, Ming-Chi Kuo, and other well-known Apple rumour writers. Also, my list does not include articles about products rumoured to be announced at specific events. They also have another thing in common: nearly all are by Ryan Christoffel, who seems to have taken up the 9to5Mac beat for making listicles of Apple products that could be updated in the future. Cool.

Careful readers might think I listed one of these articles twice — the “15+ new products this fall” one. But this is because Christoffel originally published it in May, and then changed the date on it to August after making a few changes.

I do not know who this is supposed to serve. Maybe this is a play for search engine traffic or A.I. chatbot citations. Maybe it works pretty well, too, and maybe I should be less cynical about these kinds of churned-out listicles in a time when A.I. search features are capturing traffic that used to go to third-party websites. Or maybe this is all just filler when there is nothing newsworthy but you need to publish a dozen or so articles daily.

Update: D. Griffin Jones, formerly at Cult of Mac, says on Bluesky:

It’s a secret third thing: Google Discover, Google News, Flipboard, Apple News, and other algorithmic aggregators. Articles like that tend to reach a much broader audience.

“One listicle a day” was our goal at Cult of Mac before I was laid off. It’s a tough time out there for independent blogs. The Google AI overview is devastating.

I obviously have no authority to give business advice. I would only point out that it is disappointing to hear about chasing referral strategies. One would think the pivot to video era would be treated as a cautionary tale and not something to repeat — if one, I suppose, is not responsible for staff and paycheques.

⌥ Permalink

Alex Skopic, at Current Affairs, wrote about the functional use and abuse of stimulants by people who are overworked. I stumbled upon it by way of the latest issue of Web Curios, and I think the general thrust of the article is something I find noteworthy enough to link to it. Alas, it is sloppy in the details.

The second paragraph is a good place to start. Skopic dumps a litany of numbers about declining affordability and the difficulties of modern life; among them:

[…] And according to the Bureau of Labor Statistics, 21.6 percent of Americans now work longer than 40 hours a week, with a beleaguered five percent working as many as 60 hours. […]

The way this is phrased — “now work longer than 40 hours a week”, emphasis mine — may give the impression this figure is a new high, either historically or at least in recent memory. But it is not. This figure has been declining for at least twenty years. In 2005, the BLS reported 27.9% of people employed in nonagricultural jobs in the U.S. worked 41 or more hours; by 2015, that was down to 24.8%, compared to 21.6% in 2025. The same drop is also present for workers committing 60 hours per week or more: 7.2% in 2005, 6.3% in 2015, and 5.4% in 2025.

I am not selectively quoting here. I picked ten and twenty years ago for comparative convenience, but you can check the intervening years and find a clear trend. The misuse of this data point is a poor choice when there is perhaps an interesting narrative in reconciling the declining percentage of overworked U.S. adults with the proliferation of highly caffeinated beverages, mysterious energy shots, and powders and supplements of questionable substance.

This is the kind of sentence that made me begin searching up all kinds of stuff in this article. It raises doubts, making me do the kind of work all of us should probably do any time we read something, but do not because we decide we have better things to do. Well, I decided this is what I was going to do today. I am going to skip past the health claims because they are not in my wheelhouse — double-checking easily verified numbers is something I am more comfortable with — and I will jump to this part:

Monster Energy, for instance, seems to think it owns the word “monster.” In 2009, the company threatened to sue a small brewery in Vermont for selling a beer called the “Vermonster,” claiming it infringed their trademark rights. That was at least a beverage, but in 2020 they did the same thing with video game company Ubisoft, forcing them to change a game’s title from “Gods and Monsters” to “Immortals Fenyx Rising.” In 2023 they once again targeted the gaming industry, this time over an independent game called “Dark Deception: Monsters and Mortals”; according to the developers, the drink company wanted them to “agree to never use a green & white logo on a black background for any game we ever make. So they own the colors green & white too apparently.” In all of these cases, Monster’s legal claim is shaky at best — but with a revenue of $7.9 billion in 2025, they have enough money to bankrupt anyone who tries to take them to court, so it doesn’t matter.

Monster Energy is an aggressive litigant when it comes to protecting its trademarks, but it does not always succeed. Glowstick Entertainment, makers of “Dark Deception: Monsters and Mortals”, fought and won its trademark case. Also, I think it would have been useful for Skopic to cite cases where Monster Energy lost — or likely would have lost — but the defendant went out of business anyhow. For example, in a 2022 case against a store with “Monster” in its name, the energy drink company would have been unlikely to prevail (PDF). Even so, a search of state records showed the company was ultimately dissolved. No matter whether this was directly connected to the litigation, it is a stronger argument for having “enough money to bankrupt anyone who tries to take them to court” than this evidence-free article.

Also, it is kind of weird that Coca-Cola — a company which owns 21% of Monster Energy after the latter acquired Coke’s energy drink portfolio, has a line of teas and coffees, and whose namesake beverage used to contain cocaine and still contains caffeine — goes entirely unmentioned in this piece. Perhaps it is too obvious, or feels like too much of a stretch. But it is strange to dedicate a paragraph to Monster’s trademark battles, which has nothing to do with the thesis of this piece, and leave the Coca-Cola stone unturned.

I wish an editor took another pass on this article as I think there are points that could be clarified, facts that desperately need checking, and interesting narratives that could be teased out. Skopic notes, for instance, the supposed incompatibility of sleep with an economy based on relentless production and consumption. It is true that we cannot make or buy things in your sleep, nor can we provide revenue-generating services, but sleep itself is an enormous business. We are encouraged to spend thousands of dollars on beds and mattresses because less expensive options are insufficiently rest-promoting, and then we should spend more money to wake up and keep ourselves alert throughout the day.

I find all of this baffling, even as — perhaps especially as — a daily coffee drinker. I get that people have different relationships with caffeine and alcohol, so do not take this as anything greater than my feelings-based vibe, but I have never really understood these as functional products. Coffee, when made well using the kinds of beans that I like, is a delicious beverage regardless of its effects. Wine is sort of similar to me. They both have effects, of course, but I think of those as a kind of warning sign: if I have consumed so much coffee or wine that it noticeably changes how I feel, I have probably had too much. This makes them, unlike energy drinks, a delicious treat for me. Your mileage may vary.

⌥ Permalink

An un-bylined Canadian Press report:

U.S. President Donald Trump signed an executive order on Thursday to change the name of Lake Ontario to “Lake America” as tensions with Canada worsen.

Elect stupid people and get stupid policies. This report contains a good explanation of the name’s history, and also has a deliciously petty map illustrating where the border is drawn.

Like the moronic Gulf of Mexico relabelling, expect digital mapping companies like Apple and Google to fall in line. MapQuest, a holdout on that change, also says it will keep the correct name. I still think maps should reflect government names regardless of whether they were assigned by a thumb elected by the world’s least responsible country, but I sympathize with MapQuest’s stance.

Update: Just like that, Google has updated its map to reflect “Lake America” when using a U.S. map, “Lake Ontario” in Canada, and “Lake Ontario (Lake America)” elsewhere. Google’s concession — and, soon, Apple’s — is a good reminder to all users that the U.S. may be the most powerful country in the world, but it is no longer a leader.

Update: Apple has also updated its map.

⌥ Permalink

Apple, after previously saying it would move Hide My Email to the private.icloud.com subdomain:

After further consideration and reviewing community feedback, iCloud+ Hide My Email addresses will remain on icloud.com.

The whole point of an email address like this is that it can be used anywhere someone does not want to provide an address that can be tied to them or remarketed to, or if they want to be able to cut off communications. This change, if implemented, would have made these addresses basically useless. I am glad to see this was reversed.

⌥ Permalink

In September, the European Commission began pondering how to correct its 2009 privacy law that resulted in cookie permission banners littering the web, with the resulting proposal announced in November.

Jennifer Rankin, the Guardian:

EU officials said users would remain in control of their data on the internet, but new rules on cookies — the internet files that are stored on a user’s device so a website can remember them — would make life simpler by ensuring one-click consent. “I think we can all agree we have spent too much of our time accepting or rejecting cookies,” [Henna] Virkkunen said.

This was not the first time the Commission had attempted to correct for the permissions pollution that resulted from the e-Privacy Directive. In 2020, its efforts were focused on ineffective consent options like, as reported at the Verge, “a cookie consent policy with no obvious way to opt out of tracking”. I still see many websites, like the Verge itself, providing no meaningful consent for third-party tracking.

This time, though, the Commission said it was trying to make cookie consents less prevalent by allowing, for example, simple statistical cookies without any consent, and it was going to give users an option to decline tracking universally. When I looked into the changes in November, it seemed like this signal could be ignored by publishers and media companies who would be free to ask for consent anyway. As of May, it seemed this proposal was moving forward by requiring consent management platforms to respond to browser signals. By summer, however, things had changed.

Ernestas Naprys, Cybernews (“Article 88b” refers to the universal browser signal proposal):

Google suggested ditching Article 88b.

“Article 88b should be deleted. Retaining this provision risks anchoring the Omnibus to a proven-failed architecture, Google’s position reads.

“It will drastically impair the ability of most websites to monetize content and drive client acquisition. The resulting low consent rates and severely restricted data access.”

Meta suggested removing the entire Article 5(3) of the ePrivacy Directive. This rule is why we have cookie banners in the first place, as it requires website operators to obtain clear user consent before storing or accessing their information.

It was not just U.S.-based companies that argued against better user privacy controls. According to noyb, French, German, and Polish representatives were in alignment with Google’s position, which ultimately led to its scrapping. All three countries are home to companies that would be affected by this regulation. None, however, are as big or as powerful as Google or Meta.

Privacy-defending organizations are understandably not impressed. They have launched Kill the Cookie Banner to drum up support for legal recognition of a browser signal. In the U.S., five state governments say the Global Privacy Control must be respected. At a browser level, it is only implemented in Brave, DuckDuckGo, and Firefox, but it seems that Apple is working to add it to Safari, and it seems it is being actively worked on for Chromium, too. The European Commission should throw its weight behind this control, too.

⌥ Permalink

Felix Kent, Defector (gift link):

[…] I believed then and still believe that operating a car is too difficult for human beings. This is obviously not true at the level of making a car move. Making a car move is easy. A rat can make a car move. But the level of skill involved in driving really well is not dissimilar from that required to be a concert pianist, if the pianist-centered movies I’ve watched are to be believed. Now imagine that in most parts of the country leading a normal life required more or less constant piano playing. Now imagine that playing the piano badly killed people.

Just about everything we do to make driving a car easier instead seems to make people complacent. The lanes on roads are wider than they need to be to account for lateral imperfection, and then there are wide berms so that someone may drive completely off the road in relative safety. Cars that are enormous, heavy, hulking things do not allow drivers to feel their weight because of myriad assistance features that make it possible to steer with a single finger.

But driving is a skill. Like any skill, it is something people will have varying degrees of success learning. Some people simply cannot drive, whether on a temporary or permanent basis. We can bet the farm on autonomous vehicles which, though seemingly capable in warm and dry urban settings, are an expensive and individualized answer that exacerbates all the other problems of car dependency. It is also a risky bet as, like any technology, current performance is not indicative of future gains. Public transit, on the other hand, is a proven technology that, among other things, strengthens the social contract while giving everyone the ability to move around a city.

⌥ Permalink

“zedeus”:

On 24 August 2026 cease and desist letters have been sent by X Corp. demanding a permanent takedown of Nitter instances and the project’s repository.

Nitter mirrors X; it powers websites like XCancel which remains online and functional as of writing. (Update: A few hours after I published this, XCancel says it was also told by X to shut down its service.)

This is not surprising but it is very stupid. Companies like X and Meta are very happy to scrape the web at unprecedented scale and without permission, but have zero tolerance for the same to be done to them by, in effect, individual users. That is not because they are protective of the creative works users have contributed to these platforms. It is simply because users are not trusted to use the platforms. X does not permit third-party readers, and neither does Instagram, because their value does not come from us using these services as we wish, but from how they dictate we must. Bafflingly, however, some argue they should face little responsibility for those choices.

⌥ Permalink

Jeffrey Kopp, of CNBC, wrote a short profile of Judge Yvonne Gonzalez Rogers, whose name you may have heard even if you do not live in her district of Northern California. That is because her district covers such areas as San Francisco and Silicon Valley, so her rulings on the many cases heard there have a disproportionate impact on U.S. technology policy.

These policies should catch the attention of people in the U.S., of course; it is also a factor to be mindful of in the rest of the world. The boundaries and limitations surrounding the behaviour of some of the most impactful companies in the world are decided by a relative handful of people, among whom are the justices of the Northern District of California. This article is not a deep examination of such consequences — but I would love to read that story, if it exists.

⌥ Permalink

Now that it has been a couple of years since Apple began requiring administrators to sign in to the News Publisher dashboard or risk their accounts losing their role, there are a few things we have learned:

1. These notifications arrive on an approximately five-month schedule, perhaps a couple of weeks longer.

2. News Publisher has little additional functionality that requires or would encourage signing in on a more regular basis if you do not use Apple’s special format.

   In my case, I cannot even see the articles I have published, nor are there analytics. I checked on a device that is not my own and I can see that recent articles are broadcasting to Apple News just fine from my website, but there is effectively nothing in News Publisher reflecting that.

3. Apple News is still frighteningly limited from a publisher’s perspective. It launched eleven years ago and there is still no way to set a custom URL in News Publisher or, in fact, anywhere. That is, this website’s Apple News link is https://apple.news/TAjcS0c5sRV2HYftzmJ6UMQ instead of, say, https://apple.news/pixel-envy. It is not a function of size, either — here are the URL slugs of a few publishers you definitely recognize: ThfiauYLtQlOfqyiHkempkw, TVKIb0N6iSsOyGSnR4zF1Kw, and TUKgA_OjuTe20v1ndDwQjxQ. Catchy.

   This limitation also extends to the URLs of individual stories. It is the kind of basic functionality that exists on every other public-facing platform I can think of that a publisher might use, but Apple does not support because its “Services” business prints money anyway.

I still do not know what happens if I fail to sign into News Publisher five months from now, other than that my role becomes reduced to an editor. There is nobody else authorized with News Publisher, so I guess I simply lose administrator access forever if, every five months, I complete the necessary busywork of signing in.

⌥ Permalink

I finally got around completing a long pending project idea: A diffusion image generation model implemented on a microcontroller. See blog article and repo.

Context: Placemark is a tool that lets you view, create, and edit geospatial (map) data. I ran it as a company for a while and open sourced it when I shut down that company.

Yesterday Placemark got a pretty big PR from Birk Skyum, replacing its dependency on Mapbox GL JS with MapLibre, along with some other improvements. Birk is one of the cofounders of MapLibre^[1] and a generally prolific open-source contributor.

It's merged and live, and you can see it in action at placemark.io. This meant swapping out both the Mapbox frontend code as well as switching to a different tile service - we're now using OpenFreeMap for tiles, OpenMapTiles-based styles, openrouteservice for the routing drawing mode, and MapTiler for the satellite layer. Forks of Placemark no longer require a Mapbox API token to work, but they do require a MapTiler token to support Satellite tiles, and an openrouteservice token to do route-drawing.

The big upside here is that this makes Placemark cleanly open source: all of its dependencies have proper open source licenses like MIT, ISC, Apache-2.0, BSD-3-Clause, or similar^[2]. Mapbox GL's license is not open source and requires the developer to have a Mapbox account and follow their Terms of Service. So: you can now fork Placemark and build cleanly-licensed open source applications on its codebase. I recommend it!

You'll notice some visual changes: MapTiler's satellite sources and processing are different than Mapbox's, and the OpenFreeMap styles are a bit different. I hope that it's not a disruptive change for anyone relying on the tool.

Along with this, Birk removed the deck.gl dependency that had been bothering me for a few years, and swapped out the routing functionality for a more permissive provider. It was a really great contribution, and resolved an issue more than two years old. Great stuff, and much appreciated!

Previously

• Earlier this month, Placemark code got readopted into geojson.io, which was also very cool
• Rewind to the 2024 Placemark open source release post

Spacelab was a reusable laboratory that could be carried in the Space Shuttle's cargo bay, providing lab space for astronauts and experiments.1 Because Spacelab was a European project, it used a French-built minicomputer, the Mitra 125 MS,2 rather than the Shuttle's main computers, IBM-built AP-101 systems. For storage, the Spacelab computer contained 128 kilobytes of RAM. Rather than silicon memory, the computer used magnetic core memory, with each bit stored in a tiny ferrite ring. In this article, I take a close look at this computer's core memory system.

The illustration below shows how Spacelab fit inside the Shuttle's cargo bay. The pressurized laboratory is the cylindrical module in the front of the cargo bay, connected to the Shuttle by a tunnel. Experiments were mounted on pallets behind the laboratory. The laboratory held three identical Mitra computers.3 One computer managed Spacelab itself, while the second computer managed the experiments. The third computer provided a backup in case of failures.

The photo below shows the core memory stack, removed from the computer. The core memory stack takes up roughly a third of the computer. The entire side panel of the computer detaches, and the core memory unit slides out. Since the computer is cooled by conduction, firmly attaching the core memory stack to the side panel kept it cool. The core memory stack consists of seven boards: a driver board, four core plane boards, a second driver board, and an interface board. Each board has two 160-pin connectors that plug into a large daughter board on each side, providing extensive connectivity between the boards. The daughter board on the right has another 160-pin connector that links the memory stack to the rest of the computer. (These connectors are the long blue connectors in the photo.)

How core memory works

One of the hardest problems for early computers was storage. Computers of the late 1940s and early 1950s stored data through techniques such as sound waves in mercury, spots on a CRT screen, or spinning magnetic drums, but these all had limitations. What computers needed was dense, inexpensive storage that was fast, reliable, and could be accessed randomly.

During World War II, Germany developed special magnetic alloys that could "flip" from one magnetic state to another. After the war, American researchers realized that these materials could be used for storing binary data: "It was completely obvious that you could make a memory with this material," in the words of Jan Rajchman. Different aspects of core memory were patented by various inventors (including independent inventor Frederick Viehe, An Wang at Harvard, Jan Rajchman at RCA, and Jay Forrester at MIT), leading to expensive patent battles. (IBM ended up paying $400,000 to Wang—who used the money to build the computer company Wang Laboratories—and $13,000,000 to MIT.) I view Jay Forester as the most important inventor, developing the design of practical core memory, researching magnetic materials, and building the first core memory in 1953 for the groundbreaking Whirlwind computer.

Core memory is based around a tiny toroidal magnetic core, one per bit.4 A core can be magnetized clockwise or counterclockwise to store a bit. The core can be magnetized by threading a wire through the core: running a current through the wire produces a magnetic field that magnetizes the core, while running a current in the opposite direction produces the opposite magnetization.

A key problem with core memory was how to wire the cores without an absurd number of wires: if each core had a separate wire, just 16 KB of storage would require over 100,000 wires. The solution was called "coincident current addressing". The cores are arranged in a grid, with horizontal and vertical wires, as shown below. By running a current through one horizontal wire and one vertical wire, the single core at the intersection was selected. But wouldn't that magnetize all the cores along the horizontal and vertical wires? The key was that the cores were constructed from special magnetic materials with a property called hysteresis: a small current leaves the core completely unchanged, while a larger current flips the core's magnetic state. The currents through the horizontal and vertical wires were carefully selected so each wire had half the current necessary to flip the core; where the wires intersected, the two currents provided sufficient magnetic field to flip the core.

The next step was reading the core. A sense wire was threaded through all the cores in the two-dimensional plane. To read a core, the X and Y select wires were driven to flip the desired core to the 0 state. If the core was already in the 0 state, nothing happened. But if the core was originally in the 1 state, the magnetic field changed as the core changed state. This induced a small current in the sense line, indicating that the core held a 1. Note that reading the value of a bit destroys that value. Thus, a core needs to be rewritten after reading, to restore the original data.

To access a word of memory at a time, core planes were combined into a three-dimensional stack (below). Since each plane held one bit of the word, a 16-bit word would have a stack of 16 planes. All the planes shared the signals to drive the X and Y lines, so a one-word column through the stack was accessed in parallel. Each plane had a separate sense line to read out the bit.

But how do you write different values to the different bits? The trick was to put an "inhibit" line through all the cores in a plane, running the inhibit line in the opposite direction to the X lines. Putting a current through the inhibit line would cancel out the current through the X line, preventing the core in that plane from being modified. To summarize, a read-write cycle consisted of first energizing a pair of X and Y lines to select a word and write a 0 to the column of cores in that word. The sense lines provided a readout of the bit values. Next, the X and Y lines were energized in the opposite direction to write a 1 to the cores. At the same time, the inhibit lines were energized for each plane with a 0 bit. Thus, the cores either flipped back to 1 or stayed at 0, as required. Many core memories, such as the one below, used a shared wire for sense and inhibit, so there were three wires through each core.

The final ingredient to make core memory practical was the diode matrix. The X and Y lines require driver circuits that can produce fast, bidirectional high-current (e.g. 600 mA) pulses. A core memory plane can have hundreds of these lines. Providing a separate driver for each wire would be very expensive, especially in the vacuum tube era. The solution was to put separate drivers at each end of the wire, with each driver supporting multiple wires. For a trivial example, suppose you have 9 vertical lines. Put three drivers (A, B, and C) on the top, each connected to three wires, and three drivers on the bottom (1, 2, and 3), each connected to three wires. By energizing a driver at the top and a driver at the bottom (e.g. B and 1), the corresponding wire will be energized. Now, N drivers on each side control N² wires, supporting N⁴ cores in total.

Unfortunately, it's not quite that easy. Current can take "sneak paths" through the cores, such as the path in pink above. The solution is to add diodes to ensure that current can't take the wrong path. Since a wire needs to be driven with currents in both directions (to flip cores both ways), two diodes are required on each wire, as shown below, one in each direction. Each matrix input (A, B, etc.) is replaced with two inputs, one to drive each direction. (The horizontal wires also require diodes, not shown.)

Since each wire requires two diodes, core memories used many diodes. Fortunately, diodes were small and inexpensive, so a large quantity of diodes was manageable. The photo below shows the diode stack for the computer used in the Saturn V rocket, the Launch Vehicle Digital Computer.

Originally, core memories were tediously constructed by hand. For the Whirlwind computer, it took a full 40 hours to wire a 64×64 core plane. Companies such as IBM soon developed automated techniques to manufacture core memory, and the price dropped by a factor of two every two years, similar to Moore's Law.5 Core memories became fast, inexpensive, and reliable, and were the most popular form of main-memory storage until semiconductor memory took over in the 1970s.

The Spacelab computer's core memory

The Spacelab computer's memory was manufactured in 1980, a late date for core memory, so it is advanced and high density. The photo below shows one of the four core plane boards from the computer. Each board holds 16K of 18-bit words (32 KB), so the computer has 128 KB of RAM in total. The computer is a 16-bit computer, but each word also has a parity bit and a "storage protect" bit, bringing the total to 18 bits. (The storage protect bit provided write protection on a word-by-word basis, preventing programs from being accidentally overwritten. Because core memory is nonvolatile, a program could be loaded into memory once and would be immediately available every time the computer was powered on.)

The core memory board is arranged with 1024 vertical (Y) wires and 288 horizontal (X) wires, supporting 294,912 lithium ferrite cores. These very thin wires are soldered to tiny pads on the printed-circuit board. The board supports 18 bits, which is visible as 18 alternating stripes of green and copper because alternating sense lines have different colors. The board has 36 sense lines: the left and right halves of the board have independent sense lines to reduce noise, so the board has 36 sense lines for 18 bits. The sense wires pass through four holes in the board (green arrows) and are soldered on the back of the board.

The photo below shows a close-up of the cores. Each core is approximately 32 mils (0.8mm) in diameter, the same as the IBM System/360 cores shown earlier. However, the cores are stacked much closer, with only a small gap between cores. The X and Y select lines are copper-colored, while the sense lines are green. (The wires are all enameled to prevent short circuits.) The sense wires loop around at the left, forming a single circuit through each bit section. Half the Y lines form loops at the bottom; the other half form loops at the top. Thus, each Y line passes through the plane twice in a U-shaped path, which will turn out to be important.

The other side of each circuit board holds the sense amplifiers and the diode matrix for the core plane. The diode chips are the square black packages, each containing 16 diodes for 8 core lines.6 In the red-outlined regions, one end of each vertical U-loop is connected to a diode chip; the lines of diagonal holes are the vias that pass each signal through the board. The other end of each vertical U-loop is connected to one of the blue board connectors on the side; these vias are in the blue-outlined regions. The horizontal lines use the diode chips and vias in the green regions. One end of each line is connected to a diode chip, while the other end is connected to a board connector through traces on the other side. Note that some vertical lines connect to the diode chips at the top of the board, while others connect at the bottom. Similarly, some horizontal lines connect at the left while others connect at the right.

The central region (yellow) holds 18 sense amplifier chips, the black DIP integrated circuits, each containing two amplifiers.7 The white packages are resistor packages, holding multiple resistors to bias and terminate the sense amplifier lines. The wires from the sense amplifiers are connected as twisted pairs that are soldered to the board right next to the corresponding sense amplifier chips. Using twisted pairs for the whole distance prevents the wires from picking up electrical noise, which could overwhelm the tiny signals in the sense wires. The sense wires pass from one side of the board to the other through four holes in the board (yellow arrows), and then are glued down as they traverse a significant distance on the board. (It must have been difficult to manufacture the board without breaking the tiny, fragile wires.)

Detecting signals on the sense lines is tricky because the pulses are very small, a few millivolts. Because the sense lines run next to the X drive lines, they can easily pick up noise from the high-current pulses on the X lines. To minimize this noise, the sense lines cross each other between two plane sections, forming a "bow tie", as shown below. The result is that an X line runs next to the positive sense line for half the length and the negative sense line for the other half. Thus, the induced noise cancels out.

The core memory in the Spacelab computer used a different architecture from a typical core memory, improving performance by eliminating the inhibit line. This architecture was called a 2½D memory.8 If you're familiar with core memory, the lack of inhibit lines may seem puzzling: how do you write 1 to some bits and 0 to other bits? The trick is to have separate X driver circuitry for each bit.9 When writing data, the X lines are only energized for bits that receive a 1; the other lines are left unenergized, so the bits remain at 0. The disadvantage is that instead of one set of X driver circuits, you now need one set for each bit, a factor of 18 more for an 18-bit word. However, with the development of core drivers on integrated circuits, the cost of the additional driver circuitry became less significant.

The memory system used an technique called phase reversal to cut the number of vertical drivers in half. Recall that pairs of vertical wires are joined by a U-connection. By driving the wire in a particular direction, the left side or the right side of the pair can be selected. For example, the drawing below shows how the two wires select the left core, but not the right core. In the left core, both currents go through the core in the same direction, inducing a magnetic field in the toroid.10 But in the right core, the two currents cancel out, so there is no magnetic field created. But if the current in the vertical loop is reversed, the right core will be selected, rather than the left core. The point is that instead of using two drivers for the vertical wires, one driver is used, reversing the current to select the left or right core.

The diagram below shows the complex wiring for X drive wires. Each band of 16 wires corresponds to one bit in the 18-bit word, and has a separate sense wire. The top band of 16 X lines is connected to four contacts on the board connector; each contact is connected to four X lines through the curving PCB traces. The bottom band of 16 X lines is wired to diode modules on the other side of the board, connected through the round vias. (Each wire has the opposite connections—diode module or board connector—on the other end.)11 One group of four X wires is energized through the connector, while four wires are energized through the diode matrix, selecting one of the 16 X wires in the group.

Other boards in the memory stack

The memory stack has seven boards in total, arranged as a driver board, the four core planes, a second driver board, and an interface board. I haven't examined these boards in detail, but I'll give some preliminary information. The photo below shows one of the two driver boards. It provides the high-current pulses for the X and Y select lines. The board is crammed with specialized core memory driver chips12, along with a few logic chips to control the drivers. It has separate drivers for the two ends of the select lines, allowing the matrix selection described earlier.

Since there are two driver boards and four core memory boards, at first I thought that each driver board controlled two core memory boards. The configuration turns out to be more complicated, with one more layer of matrix selections to cut the number of drivers in half. To simplify slightly, consider the X lines on a core board to have left ends and right ends, both of which must be energized to activate a line. For the left ends, the first driver board powers core boards 1 and 2, while the second driver board powers core boards 3 and 4. The right ends are shuffled: the first driver board powers core boards 1 and 3, while the second driver board powers core boards 2 and 4. Now, if the first driver board powers the left and right ends, core board 1 is the only one with both ends active. If the first driver board powers the left ends while the second board powers the right ends, core board 2 is activated. Similarly, core board 3 or 4 can be activated. The point is that since each set of drivers is connected to two core boards, two sets of drivers are required instead of four.

The final board is the interface to the rest of the computer. It has many transistor arrays in DIP packages, along with many resistors. It seems that the board uses discrete transistors to drive the bus, rather than using interface chips, which is unexpected. The board has some wire-wrapped jumpers in the lower center region, presumably for configuration.

Conclusions

Core memory had a long life, surviving even as computers migrated from vacuum tubes to transistors and then integrated circuits, but eventually semiconductor memory made it obsolete.13 Core memories lasted even longer in aerospace applications since it had two key advantages over semiconductor memory: it retained data even without power, and it was resistant to radiation. The Spacelab computer, manufactured in 1980, was near the end of core memory's reign, so it is more advanced than a typical core memory system, with higher density, extensive use of integrated circuits, and the 2½D architecture. But eventually the high density, low cost, and low power consumption of semiconductor memory won out. In 1991, the Space Shuttle flew with upgraded main computers, the IBM AP-101S that used semiconductor memory instead of magnetic core. Spacelab's Mitra computers were also replaced, using the AP-101SL, which was based on the AP-101S but modified to support the instruction set and peripherals of the original Spacelab computer.14 Although core memory is now firmly in the past, it still lives on in the expression "core dump".

I plan to investigate the Spacelab computer some more. For updates, follow me on Bluesky (@righto.com), Mastodon (@kenshirriff@oldbytes.space), or RSS. Credits: Thanks to Steve Jurvetson for providing the Spacelab computer. Thanks to CuriousMarc for photography and help disassembling the computer. AI statement: Despite the presence of the em dash, no AI was used in the writing of this article (details).

Notes and references

Project Zomboid recently released their long-awaited Build 42. I set out to see if I can discover any vulnerabilities in the multiplayer component of the game. I ended up discovering three distinct vulnerabilities which allow a malicious server to execute code on connecting clients in Project Zomboid 42.20.3.

On 15 August 2006, Wikiversity opened a new space within the Wikimedia movement: a place where people could not only access educational resources, but also create, share, and participate in learning together.

Wikiversity had grown out of work within Wikibooks and became an independent Wikimedia project in August 2006. Its creation expanded the educational possibilities of the movement beyond textbooks, with a focus on both learning resources and learning activities in English and other languages.

Twenty years later, that idea remains at the heart of Wikiversity.

Its motto, “Set learning free,” captures an approach to education built around participation, collaboration, open resources, and the ability for learners and educators to shape the learning environment themselves.

But what does learning in the open actually look like?

Find learning resources for teaching and learning

One of the simplest ways to use Wikiversity is to find and adapt learning resources. The project hosts lesson plans, course notes, reading lists, problem sets, activities, simulations, teaching aids, and other materials that educators and learners can use and develop.

For example, Student Success: Information Literacy introduces learners to library resources, database searching, digital media evaluation, evaluating online sources, academic honesty, and citation. It combines explanations with multimedia resources and activities that can support the development of research and information literacy skills.

Resources like these can be used as they are, adapted to meet a particular learning need, or developed further by other educators and learners.

That ability to build on what already exists is an important part of learning in an open environment.

Build a learning experience

Wikiversity is not limited to individual learning materials. Educators can also use it to bring together the different elements of a course or learning environment.

A Wikiversity course can include a syllabus, learning objectives, readings, activities, assignments, discussions, and other supporting materials. The project’s School and University Projects page documents examples of educators and institutions using Wikiversity in formal teaching and student assignments.

SPIR608 Political Simulation and Gaming, an open educational resource supported by the University of Westminster, shows what this can look like in practice. The module combines lectures, game playing, workshops, tutorials, presentations, and group discussions. Students analyse political simulations, participate in role playing exercises, research historical and contemporary conflicts, and eventually create and play test their own political simulation prototypes.

This shows how Wikiversity can become more than a collection of pages. It can become a shared learning environment, structured around a subject, a course, or a particular group of learners.

Learn by doing

Perhaps one of Wikiversity’s most distinctive features is its emphasis on learning by doing.

Learners can edit pages, contribute to course-related projects, write papers, conduct research, discuss what they are studying, and collaborate with others. Instead of simply consuming educational content, participants can become involved in creating and improving it.

Over the years, this approach has supported a wide range of school and university projects, including Technical Writing, Design for the Environment at the University of Toronto, Instructional Design at Indiana University, and Media Literacy at Temple University.

Wikiversity has also provided space for learners and educators to explore emerging subjects, including Artificial Intelligence.

This approach brings learning and knowledge creation together. Learners do not simply study existing knowledge; they can question it, contribute to it, document what they learn, and share their work with others.

That is one of the ideas that makes Wikiversity part of the wider Wikimedia ecosystem: learning can itself become a form of knowledge creation.

Explore research and scholarly learning

Learning on Wikiversity can also extend into research.

The project supports research activities and provides spaces where learners and educators can explore questions, develop ideas, and share their work. This reflects an aspect of Wikiversity’s original vision: creating not only educational resources, but also opportunities for people to learn through research and participation.

One example is Introduction to Scientific Journalism, an open course that connects scientific journalism with contributions to Wikimedia projects, giving participants opportunities to develop their skills while contributing to open knowledge.

Another is An introduction to the use and needs of Research Groups with WikiEducation tools on Wikiversity. The research examines how Wikiversity could support Scientific Research Groups through shared resources, documentation, collaboration, and the dissemination of research. It draws on questionnaires, interviews, and literature review to explore the needs and experiences of research groups using or considering Wikiversity.

Wikiversity also hosts WikiJournals, where scholarly articles can undergo external peer review before being published in an indexed, citable, open access format.

These examples show how learning can move beyond the classroom: a question can become a research project, a course activity can become an open contribution, and work developed by learners can become part of a wider knowledge ecosystem.

Learning that remains open

A learning resource does not have to remain fixed. A course does not have to belong to a single classroom. A learner does not have to remain only a consumer of information.

On Wikiversity, educational materials can be created and adapted collaboratively. Courses can evolve. Learners can participate in shaping the resources they use. Educators can share what they develop with others. And learning activities can contribute to a growing body of open knowledge.

For educators looking to experiment with open and collaborative approaches to teaching, Wikiversity offers many examples through its School and University Projects. WikiConecta, a free course for university educators, is one example of how learning resources can remain openly available beyond the original course and participants, allowing others to explore and build on them.

Learners can also explore resources such as the Open Educational Resources lesson, German Home Immersion School, and UrSchool to see different approaches to self-directed and community-based learning.

For subject-based and university learning, examples include Environmental Ethics, Programming with Wikidata, and Hindi Literature.

The EduWiki Hub also maintains an OER Documentation page that brings together resources and examples from across the Wikimedia education community.

Twenty years of learning in the open

Twenty years after its launch, Wikiversity with over 166,000 learning modules on the platform, continues to embody a simple but powerful idea: learning does not have to happen behind closed doors.

It can be created, shared, adapted, questioned, and developed together.

That is what makes Wikiversity’s model relevant today; not simply as a repository of educational materials, but as a space where learning and knowledge creation can happen together.

Set learning free.

Happy 20th anniversary, Wikiversity!

This Diff post is brought to you by the EduWiki Hub team, as part of our ongoing effort to highlight practical ways Wikimedia projects can support teaching and learning. As Wikiversity celebrates 20 years of open learning, we hope this overview helps educators and learners discover how the project can support courses, learning activities, research, and collaborative knowledge creation.

Across Africa, teaching people how to edit Wikipedia is only the beginning. Africa Wikipedian Alliance (AWA)’s learning programme is exploring what contributors need to stay, grow and eventually lead.

Opening a Wikipedia account is easy. Learning to use it well is not.

A new contributor must learn how to judge the reliability of a source, navigate unfamiliar editing tools and make changes without disrupting the work of others. This often brings up harder questions. How should an editor respond to artificial intelligence that produces polished prose and invented citations with equal confidence? How can a patroller protect Wikipedia from vandalism without driving away a beginner who has made an honest mistake? What does it take for an African language to become not merely visible online, but usable across the systems on which digital knowledge increasingly depends?

These questions are at the heart of the work of the African Wikipedian Alliance (AWA): building communities capable of creating, protecting and sustaining their own knowledge.

Between February and May 2026, the African Wikipedian Alliance brought together 256 participants from its Anglophone and Francophone communities for eight learning sessions. The subjects ranged from AI and source editing to gender inclusion, storytelling, Wikidata and African-language infrastructure.

For many people, Wikipedia is the first stop when they want quick information about a subject, event or public figure. Far fewer know about AWA, a community of contributors, organisers, and knowledge advocates working to strengthen participation, leadership, and knowledge equity across Africa’s Wikimedia ecosystem.

When producing text becomes easier, judgement matters more

The arrival of generative AI has unsettled some of the basic assumptions of online knowledge production. It can translate a passage, improve its grammar or suggest the structure of an article in seconds. But, as we have often seen, it can also fabricate a source, distort a fact or produce a confident answer that is not supported by evidence.

For Wikipedia, a platform built on verifiability, this presents both an opportunity and a problem..

During AWA’s February anglophone session, “AI + Wikipedia: The Good, the Bad, and the Best Practices,” (Google Drive) participants examined how AI tools can assist with translation, summarisation, grammar and the early structuring of articles. They also considered the risks: fabricated facts, invented citations and misleading content that may initially appear credible.

Rather than presenting AI as either “good” or “bad,” the discussion encouraged contributors to think critically and ethically about how these tools are used. AI can assist editors, but it cannot replace human judgment, verification, and responsibility. At a time when misinformation and synthetic content continue proliferating online, the lesson was that AI is only useful when people know how to use it responsibly.

The francophone community tackled a related subject of patrolling. Patrollers review recent changes, detect vandalism and help maintain the quality of Wikipedia’s articles. At their best, they are caretakers of collaborative knowledge: people who protect quality while helping new contributors understand where they went wrong.

A damaging edit is not always an act of vandalism. It may come from a newcomer who does not yet understand Wikipedia’s rules or technical conventions. Treating those situations in the same way can protect a page at the expense of someone who was trying to improve it.

In both sessions, the technical tools were secondary. The real subject was responsibility: when not to trust a machine, when to undo a contribution and when a new editor simply needs guidance. Editorial quality and community health are not competing priorities. Over time, each depends on the other.

A language needs more than articles

If the February sessions were concerned with protecting knowledge, the language sessions asked who has the infrastructure to produce it in the first place.

Much of the internet treats language as content: words that can be translated and placed on a page. But for a language community seeking a sustained presence on Wikimedia, translation is only one layer of the work.

Editors need an interface they can use. They need agreed terminology, reliable sources and a community large and active enough to sustain a project. The language must also be represented in digital systems that can recognise its words, grammatical forms and meanings.

Expanding African languages and knowledge representation

AWA’s March sessions introduced participants to the Wikimedia Incubator and TranslateWiki. The Incubator allows communities to begin developing Wikimedia projects in languages that do not yet have independent sites. TranslateWiki enables contributors to localise the interfaces through which people navigate and edit those projects.

The work can appear administrative: translating menus, testing pages and building a consistent body of content. Yet these are the foundations on which an independent language project may eventually stand.

AWA returned to the question in April through a session on lexicographical data in Wikidata. Participants were introduced to lexemes, forms and senses-the structures used to describe a word, the shapes it takes and the meanings it carries in a machine-readable form.

Here is why the distinction matters: A word recorded only in an article can be read by the person who finds it. A word represented as structured data can potentially be connected to dictionaries, translation systems and other language technologies. It becomes easier to search, reuse and link across platforms.

Language work in this sense, is infrastructure work. It determines which communities can do more than appear on the internet: which can participate in building the systems through which knowledge will be organised and retrieved.

For African languages that remain poorly represented online, the stakes extend beyond Wikimedia. As digital tools become more deeply embedded in education, communication and public life, languages absent from their underlying systems risk becoming less usable in the spaces those tools create.

Being admitted is not the same as belonging

Building the right infrastructure also means building strong communities and creating the social conditions that allow people to participate consistently.

During AWA’s International Women’s Day discussions in March, contributors, fellows, organisers and movement leaders from the English and French speaking communities considered what sustainable participation looks like for African women in digital knowledge communities.

Training was part of the answer, but only part. A person may learn how to contribute and still lack the time, financial support, mentorship or recognition needed to remain involved. She may be welcomed as a participant without being given a meaningful opportunity to lead.

The discussions identified access, visibility, funding and mentorship as interconnected needs. They also raised the importance of care work: the often-unrecognised labour that supports families, organisations and communities, but can limit the time women have available for volunteer-led digital projects.

Inclusion cannot be measured simply by whether the door has been opened. A new editor is not integrated into a community simply because an account has been created. They need the resources, relationships and room to grow.

Communities must be able to tell their own stories

Some forms of community work can be easily counted: articles edited, references added or data entries created. Other forms such as mentorship or organising local networks are however invisible. Processes that are developed by one group can often disappear because the experience was not documented.

AWA’s April Francophone session focused on Diff, the Wikimedia movement’s platform for community news and learning. Participants considered how to structure an article, record measurable results and use photographs and testimony to communicate what a project accomplished.

When community work goes undocumented, the lessons can disappear, organisers cannot learn from the experience and future participants cannot see the history they are joining.

Storytelling is therefore part of a community’s infrastructure, too. It preserves memory, circulates knowledge and allows people to describe their work in their own terms.

That capacity is especially important in a movement concerned with representation. Communities seeking greater control over how African knowledge appears online also need greater control over how their own contribution to that work is understood.

From using tools to shaping a movement

By May, the programme had turned towards the mechanics of independent contribution.

To an experienced editor, headings, references, templates and wiki markup are just ordinary elements of a Wikipedia page. But to a newcomer looking at dense source code for the first time, they can make the platform feel closed and forbidding.

In the Anglophone source-editing session, participants were shown how breaking a page into its component parts can make an apparently complicated whole become a sequence of manageable decisions.

The francophone Wikidata session approached independence from another direction. Participants learned how structured data connects information across Wikipedia, Wikimedia Commons, Wikivoyage and other projects. A well-structured item can help knowledge travel across languages and platforms rather than remaining confined to a single page.

These are practical skills that every contributor should have, but their knock on effects are much greater. Contributors who understand the tools can work with greater independence, and, with experience, they can help newcomers, lead training sessions, organise projects and strengthen the systems on which others rely.

One session is rarely enough, and even eight sessions cannot resolve the structural barriers facing African Wikimedia communities. Training is just a starting point, one that can open a path towards transformation.

Monthly meetups can help people move from observing the Wikimedia movement to contributing to it; and, in time, to shaping where it goes next.

Join the AWA community here to connect with Wikimedians across Africa and take part in future learning sessions.

The Igala Wikimedia Campus Outreach was successfully implemented across three higher institutions in Kogi State:

• Kogi state Polytechnic Lokoja
• Federal Polytechnic Idah
• Prince Abubakar Audu University, Anyigba

The outreach served as a capacity-building initiative designed to recruit, train, and mentor new Wikimedia contributors while promoting the documentation of Igala language, culture, history, and notable personalities on Wikimedia platforms. Through practical editing sessions, mentorship, and collaborative learning, participants gained the skills required to contribute quality knowledge to Wikipedia and related Wikimedia projects.

Project Objectives

The primary objective of the outreach was to increase the number of active Igala Wikimedia editors and improve the availability of Igala-related knowledge on Wikipedia and other Wikimedia projects. Specifically, the project sought to:

• Train new editors in Wikipedia editing and Wikimedia policies.
• Increase the number of contributors to the Igala Wikimedia community.
• Document and preserve Igala language, culture, history, and notable personalities.
• Improve the quality and accessibility of Igala-related content online.
• Build a sustainable campus-based Wikimedia community through mentorship and continuous engagement.

Project Activities

• Attah Igala – Traditional rulership and the history of the Igala Kingdom.
• Igala Language – Grammar, dialects, and language usage.
• Idah, Kogi State – Historical significance, culture, and landmarks of the ancestral home of the Igala people.

Participants also improved existing Wikipedia articles and enhanced Wikidata items relating to Igala language, places, and distinguished personalities. These contributions help bridge the existing knowledge gap affecting indigenous Nigerian languages on Wikimedia platforms.

Mentorship and Community Growth

Mentorship remained a central component of the outreach. Experienced Wikimedia volunteers provided continuous guidance throughout the training sessions, ensuring that participants understood editing standards and best practices.A significant outcome of the mentorship process was the growth in participants’ confidence and editing skills. Many attendees joined with little or no prior Wikimedia experience but successfully progressed to independently creating and improving articles by the end of the outreach. This demonstrates the importance of sustained mentorship in retaining new editors and strengthening the local Wikimedia community.

Experience

The outreach was both impactful and inspiring. Participants showed enthusiasm throughout the training sessions and demonstrated a genuine interest in preserving and promoting Igala knowledge online. The collaborative atmosphere encouraged peer learning, teamwork, and active participation.It was particularly rewarding to observe participants transition from beginners to confident contributors capable of independently editing Wikipedia. The project also strengthened relationships among the participating institutions and laid the foundation for future Wikimedia activities within the campuses.

Benefits and Impact

The outreach produced several positive outcomes, including:

• Increased awareness of Wikimedia projects among students and academic communities.
• Development of new editing skills among participants.
• Improved representation of Igala language, culture, and history on Wikipedia.
• Strengthened collaboration among campus communities and the Igala Wikimedia user community.
• Contribution towards reducing the content gap affecting indigenous Nigerian languages.
• Establishment of a growing network of volunteers committed to documenting Igala knowledge.

These contributions will continue to benefit researchers, students, and the wider public by making reliable information about the Igala people more visible and accessible globally.

Challenges Encountered

Despite the success of the outreach, several challenges were experienced during implementation:

• Inconsistent internet connectivity affected editing sessions in some locations.
• Limited availability of laptops among participants reduced hands-on participation.
• Power supply interruptions occasionally delayed training activities.
• Some participants required additional time to fully understand Wikipedia editing guidelines and policies.
• Time constraints limited the depth of practical editing during some sessions.

These challenges were addressed through mentoring, flexible training schedules, collaborative editing, and continuous follow-up support after the events.

Outcomes and Contributions

The outreach successfully expanded the Igala Wikimedia community while increasing the quantity and quality of Igala-related content across Wikimedia projects.

Project Metrics Indicator

• Campuses Reached 3
• New Editors Trained 102
• Articles Created 514
• Articles Improved 516
• Total Edit Contributions 2.64k
• Commons upload 266

See also: https://meta.wikimedia.org/wiki/Event:Building_Igala_Wikipedia_Community_Through_Campus_Engagement

Next Steps

The Igala Wikimedia Outreach continues to play an important role in promoting language equity and representation within the Wikimedia movement. Future activities will focus on:

• Sustaining mentorship for newly recruited editors.
• Organizing advanced editing workshops and edit-a-thons.
• Expanding the outreach to additional tertiary institutions and communities.
• Encouraging regular contributions to Wikipedia and Wikidata.
• Building long-term partnerships with educational institutions to strengthen the Igala Wikimedia community.

Conclusion

The Igala Wikimedia Campus Outreach successfully achieved its objective of empowering new contributors to document and preserve Igala knowledge on Wikimedia platforms. Through training, mentorship, and collaborative editing, participants contributed meaningful content that enriches Wikipedia and improves global access to information about the Igala people, their language, history, traditions, and achievements.The project represents a significant step toward addressing the underrepresentation of indigenous Nigerian languages online while building a sustainable community of editors committed to preserving Igala heritage for future generations. Continued mentorship, institutional collaboration, and community engagement will further strengthen the impact and sustainability of this initiative..

As we get closer to the next Wiki Loves Monuments competition in Iran, we wanted to take a moment to look back at the 2025 edition.

A year has passed since photographers across Iran once again took part in Wiki Loves Monuments Iran. The 2025 edition was organized by the Iranian Wikimedians User Group, with the help of volunteers and members of the Wikimedia community. The competition invited people to photograph Iran’s registered cultural and historical monuments and share their images freely on Wikimedia Commons.

We now have some distance from the competition and its results. This gives us a good chance to look at what happened, what we learned, and what we want to take with us into the next edition.

Wiki Loves Monuments Iran 2025

The Iranian competition ran from 1 to 31 October 2025.

During that month, participants contributed 2,513 eligible photographs. In comparison, the 2024 competition received 2,722 eligible photographs.

This was a small decrease, but the number of uploads tells only part of the story.

Iran has more than 33,000 registered monuments, spread across the country. Some are famous and already have many photographs. Others are much less known, and many still need better documentation on Wikimedia projects.

The goal of Wiki Loves Monuments is not simply to collect as many photographs as possible. We want to create a useful and freely available visual record of cultural heritage. A good photograph can illustrate a Wikipedia article, help people learn about a monument, or preserve a view of a place that may change in the future.

This was also an important lesson from our 2024 competition. Each edition helps us understand a little better how we can improve the contest and how we can encourage people to document more of Iran’s cultural heritage.

Selecting Iran’s finalists

After the competition ended, the national jury reviewed the photographs and selected the ten images that would represent Iran in the international round.

The first-place photograph showed Dayr-e Gachin Caravanserai, photographed by Hossein Pourakbarian.

Second place went to Mohammad Ataei for a photograph of the historic district of Gorgan, and third place went to Hadi Dehghanpour for a photograph of Arak Bazaar.

The ten selected photographs covered different parts of Iran’s heritage. They included caravanserais, historic urban areas, mosques, a bazaar, a castle, a monastery and other monuments.

These ten photographs then moved on to the international stage of Wiki Loves Monuments.

An exceptional international result

The 2025 international competition brought together photographs from 56 national contests. In total, almost 228,000 images were contributed by around 4,000 photographers around the world. Each national competition could send up to ten photographs to the international round.

The final result was especially memorable for Iran.

Four photographs from Iran were selected among the international winners, including both first and second place.

The Iranian winners were:

• 1st place: Dayr-e Gachin Caravanserai, by Hossein pourakbarian
• 2nd place: Mehmandust Tower, by Mehdi Akbari
• 5th place: Arak Bazaar, by Hadi Dehghanpour
• 22nd place: Seyyed Mosque in Isfahan, by Siavash Banaei

The full international results are available on Wikimedia Commons.

Seeing an Iranian photograph take first place was already something to celebrate. Having photographs from Iran in both first and second place made the result even more special.

The first-place photograph of Dayr-e Gachin Caravanserai had also won first place in the Iranian competition. It was chosen as the best image first by the national jury and later by the international jury.

But another result was just as interesting for us.

The photograph of Mehmandust Tower, which finished second internationally, had been ranked sixth in the Iranian competition.

This is a good example of why each country sends ten photographs to the international round instead of only its national winner. Different juries can see different strengths in a photograph. An image that places sixth in one jury may become one of the strongest photographs in the international competition.

The same happened, to a smaller degree, with other Iranian finalists. The photograph of Arak Bazaar was third nationally and fifth internationally. The photograph of Seyyed Mosque was fourth nationally and later finished 22nd in the international results.

For us, this showed the strength of the national top ten as a group, not only the strength of the first-place photograph.

More than the winning photographs

The international results were a great achievement for the photographers and for Wiki Loves Monuments Iran. But the winning images are only a small part of what the competition produced.

The 2,513 eligible photographs from the Iranian competition are now part of Wikimedia Commons.

They are freely licensed. They can be used on Wikipedia and other Wikimedia projects. They can also be reused outside Wikimedia by anyone who follows the terms of their licenses.

Many of these photographs will never win an award, but that does not make them less useful.

A clear photograph of a little-known monument can be more valuable to Wikipedia than another beautiful photograph of a landmark that already has hundreds of images. A photograph can also document a building, detail, landscape or condition that may not look the same ten or twenty years from now.

This is one of the reasons Wiki Loves Monuments continues to matter.

What we learned from 2025

Looking back, one lesson is clear: the success of Wiki Loves Monuments cannot be measured only by the number of uploads.

Iran received fewer eligible photographs in 2025 than in 2024, with 2,513 compared with 2,722. At the same time, the photographs selected by the national jury had an exceptional result internationally.

This does not mean that the number of contributions is unimportant. We still want more people to take part, and we want to document more monuments. But we also want to help participants create photographs that are useful for Wikimedia projects.

The international results also reminded us of the importance of having a strong and varied national selection. The ten photographs sent to the international round should represent different photographers, places, styles and ways of looking at cultural heritage.

Another important goal is to encourage more photography of less-documented monuments.

With more than 33,000 registered monuments in Iran, there is still a great deal of work to do. Famous places naturally attract photographers, but Wiki Loves Monuments can have an even greater long-term impact when participants visit places that have few photographs or no useful images on Wikimedia Commons.

We also want to keep making the connection between photography and open knowledge clearer.

Uploading a photograph is an important first step. Correctly identifying the monument, adding useful information, using the right monument ID, and eventually adding the photograph to Wikipedia articles can make that contribution much more useful.

Looking toward the next edition

As the next Wiki Loves Monuments Iran approaches, we want to build on what we learned from both 2024 and 2025.

We want to reach new photographers while welcoming returning participants. We want to encourage people to explore monuments that receive less attention. We want more high-quality photographs, but also more photographs that fill real gaps in Wikimedia Commons and Wikipedia.

Most of all, we want to continue building a free visual record of Iran’s cultural heritage.

The international success of the 2025 competition gave us something special to celebrate. Four Iranian photographs were among the international winners. Two of them took the first and second places in the world.

But the lasting result of the competition is much bigger than four photographs.

It is the thousands of freely licensed images that are now available to everyone.

The Iranian Wikimedians User Group would like to thank every photographer who participated, the members of the national jury, the volunteers who helped organize the competition, and the international Wiki Loves Monuments community.

As we prepare for the next edition, we hope even more people will join us in photographing, documenting and sharing Iran’s cultural heritage.

There are more than 33,000 monuments to document.

There is still a lot to photograph.

Latest tech news from the Wikimedia technical community. Please tell other users about these changes. Not all changes will affect you. Translations are available.

Updates for editors

• The Special:CreateAccount page has been simplified as part of ongoing work to modernize the account creation experience. The panel showing project statistics no longer appears next to the form on desktop and mobile web. Multiple account creation experiments show that a simpler form helps newcomers complete registration. [1]
• In order to improve page performance, images now load when they are viewed. This means images lower down an article will not load if a reader never scrolls to that part of the page, which may affect some image-related metrics. [2]
•  View all 42 community-submitted tasks that were resolved last week. For example, an issue where image thumbnails in Abstract Wikipedia could fail to display after the corresponding file was moved on Wikimedia Commons, has now been fixed. Thumbnails will now update correctly when files are moved. [3]

Updates for technical contributors

• User Info card is a feature that helps patrollers see information about user accounts. So far, it has been available only in places such as page history, logs and recent changes. Now, it’s possible to place it in the page content as well, using the {{#uic:}} parser function. It can be particularly useful in templates like {{Userlinks}} (or their specialized variants), as it will make it easier to see the context about a user on various noticeboard pages. The card will be displayed only to users who have it enabled in their preferences. [4]
• Due to user security and privacy risks, we have disabled access to Special:MyPage URLs when specifically using action=raw. If you are impacted by this, consider whether you can use an alternative approach. Special:MyPage URLs can still be accessed and used without action=raw. Specified user page URLs (e.g. User:Myusername) can still be used with action=raw. [5]
• Due to an update, the thumbnailing software has been improved. This includes upgrading librsvg to 2.60 and ImageMagick to 7, as well as resolving a number of long-standing thumbnailing bugs like rendering errors. [6]
•  Detailed code updates later this week: MediaWiki

Tech news prepared by Tech News writers and posted by bot • Contribute • Translate • Get help • Give feedback • Subscribe or unsubscribe.

The difference in the number of women editors and men editors on Wikipedia affects the representativeness of articles and the quality of content, to the detriment of the project’s aspiration to create a universal encyclopedia that compiles all human knowledge. Only a fraction of those who edit are women, and there are entire countries in Latin America whose local reality, municipalities, communities, and history barely appear on Wikipedia.

The Escuela de lideresas of the UG Muj(lh)eres latinoamericanas en Wikimedia was founded on a simple premise: to create a safe space exclusively for women from different Latin American countries where they could learn, without feeling embarrassed or intimidated, the technical and editorial tools of Wikipedia and its sister projects. The school’s mission is to develop editing and management skills, foster women’s leadership, and provide support to women’s communities to collaborate with Wikimedia projects, so that they can sustain the Wikimedia community in Latin America in the long term and, at the same time, reduce the content gaps that disproportionately affect some countries in the region.

El Salvador is an example of a double absence: gender-based and territorial gap. In this new training program, one of the participants, Kat, from El Salvador, identified a specific gap: following the legislative reform that restructured the country’s territorial organization in 2023, several of the new municipalities had no article of their own at all, or had only scattered mentions on other pages. From this observation came the idea of creating three new articles about the municipalities of Santa Ana Oeste, Santa Ana Norte, and Santa Ana Este. Kat researched, looked for official sources, read the new Special Law for Municipal Restructuring, verified demographic and geographic data, and applied Wikipedia’s policies on reliable sources and encyclopedic style that she had learned during the school’s training sessions. She also learned how to categorize and correct articles.

Other articles created by Kat include Las 14 familias de El Salvador, a term coined by the press to describe the Salvadoran oligarchy, and Las Dignas, a feminist political organization founded in El Salvador in 1990. She is now drafting a rather ambitious article on the history of feminism in El Salvador. Each of the articles that Kat created during the training represents Salvadoran territory that was previously, in terms of free knowledge, practically invisible. When a locality does not have an article on Wikipedia, it does not appear in search results, does not feed voice assistants or artificial intelligence systems that are now trained on this open data, and remains outside the shared knowledge ecosystem that millions of people consult every day.

In addition, Kat created an article about Brenda Cerén and another about Claudia Ortiz on Wikiquote, and has been learning to edit Wikidata. She also uploaded photos of her country to Wikimedia Commons. Kat tells us about her experience in her own words:

Bubulina26, CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0, via Wikimedia Commons

P: Before participating in the Women Leaders School training sessions, did you have any previous experience in writing, research, or history?

A: -I have been a Wikipedia reader for almost my entire life, for academic reasons and simply out of curiosity. When I want to learn more about a topic, I go to Wikipedia to read. I studied social communication, so we did cover a lot of national and international history, as well as writing techniques. That helped me feel “at home” during the courses and familiar with some resources such as Google Scholar and academic repositories.

P: What motivates you to edit?

A: -I like to think that there are more women and people in general like me who are curious. I want to give them a starting point on the topics I edit, so that they can then do further research and, hopefully, contribute so that the cycle of learning continues.

P: What topics did you edit, and why did you choose those topics?

A: -I have edited articles related to the area where I live. It is little known, and through the links within other articles, it can gain greater visibility. I am also interested in topics related to feminist organizations and national history. I came to these topics when I noticed the “red links” within other articles, especially because they refer to important, interesting, and useful topics that no one had yet had the opportunity to create. When I saw my first article published, I felt proud of my effort, even though on Wikipedia we are all really anonymous. Seeing the article you researched and corrected published, learning about the topic along the way, and knowing that people visit it is a really wonderful feeling.

P: How did you feel, and what difficulties did you encounter when editing?

A: -For now, only paid resources have been an obstacle, because some publications are restricted and can only be accessed by purchasing permission, so you have to work with what is available. Another challenge is finding time to edit, between work and other commitments. Even so, it is worth trying. During the course, I felt welcome. At all times, I felt like an equal, and that the course was a place to learn. There is no need to be ashamed of making mistakes or asking questions, no matter how simple they may seem. From the facilitator to the other participants, we all helped each other. In addition to learning how to write an article from scratch, I learned very useful tricks that make editing easier, resources and websites that can help, things that should not be done while editing or that a published article should not contain, among other things.

P: What would you say to another woman who is thinking about starting to edit but is still hesitant?

A: -It is necessary for women to be editing Wikipedia. For a long time, we have been made invisible in art, science, and many other spaces, both as people and in terms of the topics that interest us. Editing Wikipedia is an opportunity to share what you know or to complement and contribute to biased or incomplete articles.

After 21 years as a Wikimedia contributor, I consider myself a veteran Wikimedian. I have attended many Wikimedia conferences around the world, but Wikimania always feels special. They create incredible opportunities to meet fellow Wikimedians, listen, and learn. However, the highlight of the event for me this year was not the main conference itself, but a pre-conference event: the WikiWomen+ Summit, where I was also honored to give a talk.

The WikiWomen+ Summit brought together fellow Wikimedians in Paris, as well as online via the livestream chat channel. We gathered to reflect on and discuss gender equity, empowerment, and closing content gaps within the Wikimedia movement. This year, participants of the Summit were invited to wear something yellow to celebrate WikiWomen, and I proudly wore my yellow shirt.

During the icebreaker, we shared stories about the first articles we ever wrote and whether we had ever experienced the deletion of a biography about a woman we created. When the talks began, I learned more about the Celebrate Women Campaign and fell in love with its vision, which is deeply inspired by Wikimedia’s overall mission: “Imagine a world in which every single woman can freely share in the sum of all knowledge, and where representative information about all women and non-binary individuals is accessible to everyone, everywhere.”

Sukkoria’s presentations gave us a lot to think about in a short time, from what could potentially go wrong from a WikiWomen+ perspective to how we can responsibly integrate generative AI into wiki workflows.

Leila Zia from the Wikimedia Foundation (WMF) gave a presentation about Wikipedia readers, revealing that women are underrepresented not only as editors but also as regular readers. I was surprised by this statistic: roughly 65% of readers are reported to be men. This was the first time I had seriously considered what we can do to encourage more women to become frequent Wikipedia readers. 

Among the many great tools and projects introduced at the summit, we were also introduced to a new Wiki Gender Stats tool. I was delighted by Natacha’s roleplay, which brilliantly illustrated what features were needed and why male editors with technical skills often struggle to understand and prioritize those needs. Her roleplay perfectly captured the moment when women decided to take charge:

In my presentation, I addressed how governance and editorial decisions on Wikimedia projects can have consequences far beyond individual content disputes. I shared my concerns about contributors, particularly women+, being blocked or banned because of off-wiki disputes rather than their actual editorial work, and discussed how this can affect both content neutrality and the health of our communities. I explored these questions further in my presentation, “How Editorial Decisions Reach Beyond Wikipedia”, which you can watch here.

This blog post is an interview with Sana Sabir (User:CosmicCodex) reflecting on how WikiWomen+ Summit 2026 influenced her perspective on community, technology, AI, and women’s participation in Wikimedia. She highlights the importance of women supporting one another, the value of human-curated knowledge in the age of AI, and the many ways women can contribute to technical spaces without being programmers. Most importantly, Sana shares concrete plans to take her learnings back to her community through workshops, translation, documentation, and efforts to address knowledge gaps. 

For young Wikimedians, in-person events offer something that online collaboration often cannot – the chance to build genuine relationships, find mentors and peers, exchange ideas across communities, and return home with the confidence to turn inspiration into action. Over the years, the WikiWomen+ Summit has been one such space where women and allies from across the Wikimedia movement come together to share experiences, learn from one another, and explore how they can shape a more inclusive and sustainable future for free knowledge. 

As part of the WikiWomen+ Summit Core Organizing Team, I spoke with Sana about her experience at the Summit, the sessions that influenced her thinking, and the actions she plans to take after returning to her community.

Finding strength in community

When asked about the most valuable part of the Summit, Sana’s answer was not about a particular session or speaker. It was about the women she met.

“The women themselves. Being surrounded by resilient, welcoming leaders who uplifted and encouraged each other without judgment was transformative.”

For Sana, one of the most powerful aspects of the Summit was the openness between participants. Whether someone was an experienced Wikimedian or relatively new to the movement, people were willing to share, discuss, care and support one another.

That experience gave her a stronger sense of belonging within the global Wikimedia community.

Rethinking Wikimedia’s role in the age of AI

One session that particularly changed Sana’s perspective was Christophe’s talk on the next 25 years of Wikimedia.

The discussion about artificial intelligence made her reconsider how she viewed the relationship between AI and Wikimedia. Rather than seeing AI only as a challenge, she came away thinking about the unique value of human-curated and contextual knowledge.

As AI becomes increasingly widespread, Sana believes that Wikimedians have an increasingly important role to play in safeguarding & preserving knowledge that is grounded in human experience and context.

The idea of Wikimedians as “spearheads of knowledge” particularly stayed with her.

It shifted her perspective from asking how Wikimedia might simply respond to AI to thinking about what human contributors can uniquely provide in an increasingly automated information environment.

Looking for what is missing

Another important takeaway for Sana was the need to focus not only on what exists in Wikimedia projects, but also on what is still missing.

Inspired by the discussion around Women in Red, she wants to bring this approach into her own community by organising workshops that help contributors identify gaps in local knowledge.

For Sana, this means looking for overlooked subjects, stories, and perspectives that may not yet be adequately represented.

This idea connects the work of individual contributors with a much larger question: whose knowledge gets documented, and whose stories remain invisible?

Making technology more accessible

Experience and insights from this year’s Summit also prompted Sana to think about the challenges women continue to face in Wikimedia’s technical ecosystem.

She highlighted issues including translation gaps, AI-related bias, copyright transparency, declining readership, and the need for more women to participate in technical contribution.

But one of her strongest takeaways was that technical contribution does not have to mean becoming an expert programmer.

Bee Piovesan’s session particularly influenced this perspective.

Sana realised that women can contribute to Wikimedia’s technical ecosystem in many ways: by usage of translating tools, documenting them, improving accessibility, and helping communities build or use technology that responds to their actual needs.

“I don’t need to be an expert coder; I can contribute by translating tools, documenting them, and making them accessible to other communities.”

For someone who wants to see more women participate in the Wikimedia technical ecosystem, this can be an important shift in thinking. Technical spaces can become more inclusive when contribution is understood beyond writing codes.

An unexpected lesson about editing

Not every takeaway from the Summit was technical.

One moment that surprised Sana was hearing contributors describe editing Wikipedia as something that could be personally meaningful and even therapeutic.

Some contributors shared how focusing on one edit at a time could provide attention, purpose, and a sense of visible progress.For Sana, this offered a different perspective on editing. Contributions are not only about adding knowledge to Wikimedia projects; they can also be personally fulfilling.

It was an unexpected reminder that community participation can affect contributors in ways that go beyond the measurable output of edits or uploads- Satisfaction is a success too 🙂

What should come next?

When asked what she would like to see at a future WikiWomen+ Summit, Sana had a practical suggestion: more hands-on workshops for non-technical participants.

She would like women to have more opportunities to learn how to translate, document, and work with Wikimedia tools, rather than feeling that technical contribution is reserved for experienced developers.She also suggested exploring ways to make Wikipedia more reader-friendly for women, particularly in conversations around gender gaps in readership.

For Sana, inclusion is not only about bringing more women into existing spaces. It is also about making those spaces easier to understand, participate in, and shape.

“You don’t need to code”

Sana’s message to women who are interested in Wikimedia but have not started contributing is simple:

There are many ways to begin: editing an article, translating a tool, documenting technical resources, or helping others use Wikimedia projects.

She encourages women to start small and find a form of contribution that feels meaningful to them.

It is a message that reflects one of the central themes of her Summit experience: there is no single correct way to become a Wikimedia contributor.

Turning a Summit into action

Perhaps the clearest indication of what the Summit meant to Sana is what she plans to do next.

Her first step is to share her learnings with the women of Wiki Club AMU and discuss what their community could address together.

She also plans to document and translate a Wikimedia tool for her community, simplify complex articles to make them more accessible, and continue exploring editing as a personally meaningful practice.

These are concrete actions that connect the Summit to community work beyond the event itself.

When asked to describe her WikiWomen+ Summit 2026 experience in one sentence, Sana called it:

“An empowering revelation that women can code, translate, document, heal, and lead in shaping the future of knowledge, addressing AI bias, simplifying language, and building tools we actually need.”

Her answer brings together the different ways she now sees participation in Wikimedia: through technology, documentation, translation, community building, and leadership.

The Summit gave her new perspectives, but the next step is turning those perspectives into action.

And perhaps that is one of the most valuable outcomes of a community event—not simply leaving with new ideas, but returning home with the confidence and motivation to do something with them.

About the interview

This interview was conducted by Gauri Gupta on behalf of the WikiWomen+ Summit Core Organizing Team.

Sana Sabir (CosmicCodex) is a Wikimedia contributor and member of the Wiki Club AMU community.

The responses have been lightly edited for clarity and flow while retaining the substance of Sana’s reflections.

As the Iranian Wikimedians User Group (IWUG) started preparing its plans for the coming year and its next grant proposal, we wanted to hear directly from the community about what kinds of activities they were interested in and where we could do better.

In June 2026, we conducted a survey asking people around the user group about their current involvement with IWUG, their interest in future activities, and the kinds of programs and training they would find useful.

We received 147 responses. The results are now helping us shape our plans for the next year.

Who responded?

The survey ran from 8 to 23 June 2026. It was mainly intended for IWUG members and people connected with the user group community.

We shared it through the IWUG Telegram group and channel, as well as the Persian Wikipedia Village Pump (قهوه‌خانه). Anyone who had the link could respond.

Because participation was open and self-selected, the results should not be treated as representative of the entire Persian Wikipedia community. The survey was intended to help us understand the community around IWUG and what people in that community expect from the user group.

Almost all respondents were active on Wikipedia. 146 of the 147 respondents selected Wikipedia as one of the Wikimedia projects where they are most active. Some also reported contributing to Wikimedia Commons, Wikidata, Wikiquote, Wiktionary, Wikisource and other Wikimedia projects.

Interest is high, but participation is much lower

One of the clearest results was the difference between people’s interest in IWUG activities and how much they currently participate.

85% of respondents reported at least a medium level of interest in participating in IWUG activities. Of those, 44.9% described their interest as high or very high.

Current participation was much lower.

65.3% said their current participation in IWUG was none, very low or low. Only 9.5% described their participation as high or very high.

The survey also showed that many respondents were not very familiar with IWUG. 70.1% said they had no, very little or low familiarity with the user group, while only 11.6% reported high or very high familiarity.

Taken together, these results suggest that there are people who are interested in getting involved but do not yet participate regularly, and in many cases may not know enough about IWUG or how to take part.

This is something we want to work on during the coming year.

Online education was the most popular activity

Education was one of the strongest themes in the responses.

When we asked which types of activities people would be interested in joining, online educational and skill-building programs were the most popular option, selected by 92 respondents (62.6%).

Article-writing campaigns and competitions were selected by 70 respondents (47.6%), while 62 respondents (42.2%) were interested in in-person social gatherings.

We also asked what people would like to learn through online programs.

97 respondents (66%) selected introductory training about Wikipedia and editing. 83 (56.5%) were interested in tools and scripts. 78 (53.1%) wanted to learn more about contributing to other Wikimedia projects such as Wikidata, Wikimedia Commons and Wikisource, and 67 respondents (45.6%) were interested in learning more about the wider Wikimedia movement.

Some of the written responses also mentioned that learning how Wikipedia works can be difficult, especially for newer contributors. Several respondents asked for clearer and more accessible educational material in Persian.

This fits closely with some of the programs we were already considering.

During the coming year, IWUG plans to create short educational videos in Persian and organize online workshops and webinars introducing users to different ways of contributing to Wikimedia projects. The survey results will help us decide which topics to prioritize.

What other activities are people interested in?

Respondents also showed considerable interest in collaborative campaigns and competitions.

The most popular option in this area was competitions organized together with other countries or Wikipedia communities, selected by 79 respondents (53.7%).

Among established Wikimedia campaigns, Wiki Loves Earth was selected by 74 respondents (50.3%), followed by Wiki Loves Folklore with 70 (47.6%). Wiki Loves Monuments was selected by 49 respondents, while Women in Red and Wikipedia Asian Month each received 38 selections.

For possible in-person programs, educational workshops were selected by 90 respondents (61.2%). Informal meetups were selected by 81 (55.1%), Wikipedia birthday celebrations by 79 (53.7%), and technical sessions focused on bug fixing and tool development by 63 (42.9%).

We also asked how people would like to hear about future IWUG activities. 114 respondents, or 77.6%, said they would like to receive information about programs by email.

These responses give us a useful picture of both the programs people are interested in and how we can do a better job of informing them about those opportunities.

Using the results in our planning

The main purpose of this survey was practical. We are currently preparing our annual plan and our next grant proposal, and we wanted community feedback to be part of that process.

Some of the results have confirmed ideas that were already under discussion, particularly our plans for educational videos, webinars and online workshops.

Other results, especially the difference between interest and current participation, showed us that we also need to pay more attention to outreach and communication. It is not enough to organize activities if interested contributors do not know about them or are unsure how to get involved.

We hope to use the coming year to make IWUG’s activities more visible and make it easier for interested contributors to participate.

A larger Persian Wikipedia survey is coming

This survey was focused on the community around IWUG. We are also planning a much larger survey aimed at the wider Persian Wikipedia community.

There is some history behind this. From 21 May to 20 June 2016, Iranian Wikimedians conducted a large survey of the Persian Wikipedia community that received 1,780 responses. It was promoted very widely, including outside the usual Wikimedia community channels. The results of that survey were never formally published.

Later in 2026, we plan to conduct another broad survey of the Persian Wikipedia community. Like the 2016 survey, it will be promoted well beyond IWUG’s own channels and will also use outreach outside Wikimedia spaces.

The purpose of that survey will be different from the June IWUG survey. We want to hear from a much wider range of Persian Wikipedia users and contributors and get a broader picture of the community.

After the new survey is completed and its results are published, we also plan to prepare a separate report comparing the findings with the 2016 survey.

For now, the June survey has already given us useful information for our immediate planning. It showed strong interest in educational activities, identified areas where contributors want more support, and highlighted a clear gap between people’s interest in IWUG and their current level of participation.

Those findings will be part of how we shape our programs for the coming year.

I’m a volunteer with the Wikimedia Tanzania community, contributing to Wikimedia Commons, Lingua Libre, Wikidata, and Swahili Wikipedia. I also work as a Program Coordinator for the Digital Youth Clubs Program in Tanzania. I attended Wikimania 2026 in Paris on a Tandem Scholarship funded by Wikimedia Deutschland, and here is a report of my experience.

Pre-Conference Experience: WikiWomen Summit and Wikimedia Deutschland Dinner

A pre-conference for women across Wikimedia communities. One of the things I gained from the pre-conference was discovering the Wiki Women Conference. I had never known that such a conference existed within the Wikimedia movement. It was inspiring to see a dedicated space where women could openly share experiences, and encourage one another to take on leadership roles and contribute to closing gender gaps across Wikimedia projects.

The summit continued after lunch on gender equity, I learned about the #VisibleWikiWomen campaign which inspired to give women the visibility and acknowledgement they deserve by uploading quality images to Wikimedia commons.

 Later, Wikimedia Deutschland hosted a dinner with Tandem partners, creating another opportunity to connect. The pre-conference and dinner expanded my network and inspired me to contribute more in Wikimedia. Many thanks to Wikimedia Deutschland for generous sponsorship.

An Impactful Session: Learn How to Grow Your Local Wikimedia Project Initiatives

One of the most impactful sessions I attended at Wikimania 2026 was “Learn How to Grow Your Local Wikimedia Project Initiatives,” facilitated by Mattia L. Nappi, board member of Wikimedia Italia, and Ferdinando Traversa, President of Wikimedia Italia. I chose it because I want to strengthen Wikimedia initiatives in my local community and learn practical ways to build a sustainable team.

The message that stayed with me most: you don’t need to know everything before starting. The facilitators encouraged us to begin with the skills and resources we already have, and improve as we go. It was a reminder that progress starts with a first step, not a perfect plan.

The workshop included a group exercise where we designed an imaginary local group and planned its first activity. As writer and mentor for my group, I documented our ideas and guided discussion. We focused on building a community around GLAM and universities, proposing an online event for students, with mentorship and partnerships with university leaders to attract new contributors. I then presented our work to the room my first time presenting at Wikimania, and it gave me real confidence speaking to an international Wikimedia audience.

The session pushed me to think harder about improving the Digital Youth Clubs program, which just finished its pilot phase in Tanzania. Successful projects don’t need big resources at the start, they need commitment, teamwork, and people willing to act.

This experience reinforced how much collaboration and international connections matter in growing Wikimedia communities lessons I hope to apply to empower more contributors locally.

What I personally gained and what I can pass on to others

Wikimania 2026 introduced me to new ways of contributing to free knowledge. Beyond the sessions, I built friendships with international Wikimedians, gained mentors and collaborators, and discovered ideas to bring home.

One highlight was learning about Open Food Facts, a project I had never encountered before. I learned how to contribute, installed the app, and exchanged contacts with the team for future guidance. I was also introduced to the Wikidata tools Cradle and TABernacle.

In return, I shared my Wikimedia work in Tanzania with participants, connected fellow Tanzanians with my Tandem partner and Mattia L. Nappi, and invited another participant to a session that led to a valuable discussion with Asaf Bartov. I also shared Tanzanian Kilimanjaro Premium Blend tea with attendees. I returned home inspired to strengthen my contributions and share what I learned with the Youth Digital Clubs Program, where I serve as Program Coordinator.

Brigadier Martin Hotine is not quite the image of a decorated officer. His name is styled with trailing acronyms that make it no surprise that there is an official portrait, yet in that painting he appears disheveled, his tie far off center in the collar of his jacket. He leans off to one side, not quite like he is sitting for a portrait, but more like he was caught in the middle of something. Photos of the man are often similar: he's distracted, looking down at his desk or staring into space. His mind seems to be elsewhere. Hotine had a lot to think about. His duties in the First and Second World Wars had only been a distraction from the real work of his career: the precise measurement of the whole British Empire.

Late in Hotine's career, he was honored not only by his own country (as a Commander of the Order of the British Empire) but by the United States as well (named an Officer of the Legion of Merit). Most of his awards, though, reflected the technical nature of his work: the Founder's Medal of the Royal Geographical Society, and shortly after his death in 1968, the Gold Medal of the United States Department of Commerce.

Ribbons and medals, though, do not quite capture the breadth of Hotine's work. His greatest memorial is an artifact of his work: squat concrete pillars surmounted by a triangular brass plate. Found atop mountains and hills throughout the United Kingdom, these "trig points," designed by Hotine himself, are the physical references of the Retriangulation of Great Britain. This effort, spanning from 1935 to 1962 with the interruption of WWII, revised an original triangulation (initiated in the 18th century) as the basis for British surveying. Through the course of this effort, Hotine developed methods that would revolutionize the field of geodesy. His collaboration with mapmakers from the United States, a continuation of his wartime surveying for the Allied Forces, set the stage for one of geodesy's most ambitious projects: a measurement taken across the Atlantic Ocean.

Geodesy is the field concerned with the measurement of the Earth. It is perhaps one of the greatest examples of the subtle complexity of the real world: superficially, the measurement of distances and areas is a simple problem. In practice, it is extremely complex, subject to a web of complications that mean that even the most modern efforts should be viewed only as close approximations.

To begin, we have to consider the shape of our planet. This question, "what shape is the Earth?," is a central topic in geodesy known as the "figure of the Earth," and it has occupied mathematicians, cartographers, and astronomers for centuries. Of course we know what shape the Earth is: it is a sphere. Well, that's true to a level of approximation, but one that isn't even close enough for highway construction.

Geodesy's foundations are in the measurement of angles and distances, taken from the Earth's surface—of course, for most of human history, where else would we take them? By measuring the angles between three points and performing some trigonometry, the relative positions of the three points can be determined. This is known as triangulation. The same is true if you measure the distance between three points, known as trilateration, but up until the development of electronics the measurement of very long distances was a far more difficult problem than the measurement of angles.

So, the first triangulation of Great Britain, conducted over some 60 years starting in 1791, measured the angles between mountain and hill peaks. These measurements were taken very precisely using a then-new instrument called a theodolite, which is essentially a telescope coupled to a protractor. By taking enough measurements between enough hilltops, surveyors formed a sort of mesh or web that slowly spread across the country. Eventually, this network of reference points was dense enough that locations of buildings, property claims, and enemy encampments could be stated accurately by their relative position to fixed reference points.

This explanation of geodetic triangulation has omitted a major problem. Three angles or three measurements can precisely define a triangle, but the solution depends on the surface over which the triangle is formed. The math is simplest in a flat plane. Over a sphere, it becomes more complicated but is still well understood. The Earth, though, is not a sphere. It's not even that close.

One of the reasons that it is difficult to define the shape of the Earth is that it is unclear exactly what shape you would refer to. The physical surface of the Earth, its topography, is extremely messy. There are mountains, there are valleys, and the whole thing is the result of long, stochastic processes that left behind something that is not amenable to a mathematical description. Besides, surveyors are often trying to establish where exactly the topography is (e.g. the altitude of a given point), so referencing those measurements against the topography itself would be tautological. The field of geodesy, as separate from mere measurement, is perhaps defined by this realization that measurements of the Earth must be taken in reference to an abstract plane.

The most tempting of these planes is sea level, more or less. Geodesists refer to this as the "geoid:" roughly speaking, the geoid is the shape that the Earth would take if it were covered entirely by water and undisturbed by tides, weather, geotectonics, etc. It is not a sphere because the Earth is rotating, and the centrifugal force pushes things outward at the equator, causing a slight "flattening" of the sphere into an ellipsoid. It is not an ellipsoid either, because the geoid is ultimately defined by gravity, and gravity is controlled by the density of the Earth nearby. For example, where the Earth's crust is very dense, gravity is locally greater and pulls the "water" of the geoid closer, creating a sort of "hill." Where the Earth's crust is less dense, water can flow away. Ultimately, the geoid is irregular, describable only by measurement.

Instead, geodesists rely on a simplified form of the geoid called the ellipsoid. This is an ellipsoid that best approximates the geoid, for the surveyor's purposes. British mathematician and geodesist Alexander Ross Clarke, in the course of the 19th century Ordnance Survey, determined that an ellipsoid with a major axis of 6,378,306 meters and an inverse flattening ratio of 293.465 was the best approximation of the geoid based on the areas that the British Empire had surveyed so far.

Ellipsoids are not useful on their own, though, as they must be anchored to a location and orientation on the surface. In other words, an ellipsoid is just a shape, and you must have at least one reference point and angle, or otherwise multiple reference points, to define how the surface of the Earth is mapped to that shape. When you add a convention for how surface measurements are expressed (and relative to what, typically the same reference point at which the ellipsoid is defined), you have the basis of a geodetic system that can convert between abstract coordinates and physical locations in three dimensions.

One of the driving forces of cartography, both as an enabling factor and as a source of demand, is aviation. Aircraft cover large distances, and pilots can make close observations of the ground underneath them. As technology advanced, aerial photography offered a way to record the pilot's observations for later analysis.

Still, as with so many things in cartography, there is a chicken and egg problem. An aircraft can be used to survey a map, but the aircraft needs to know where it is—usually by comparison to a map. In a certain sense, a pilot setting out to map new territory is flying blind. Well, some pilots are literally flying blind, in fog or clouds or under the cover of night. Even before the war, aviation was venturing into more difficult missions and pilots found themselves caught in the dark.

A German company, Lorenz, developed the first solution for blind navigation: an early form of space-modulated radio beacon usually called a Lorenz beam. By emitting a precisely directional radio signal down the approach path to a runway, the Lorenz beam created an invisible path that pilots could follow. An instrument in the aircraft gave the pilot an indication of how far to the left or right they were, and when they were centered, they knew exactly where they were—at least on a line.

The first Lorenz beam was installed in 1932, and it opened the era of radionavigation. Other events in Germany would soon establish radionavigation as one of the period's most profound developments: a blessing to pilots and a curse to the people caught underneath.

This combination of an ellipsoid, reference points, and a system of measurement is roughly what geodesists call a datum. Clarke's ellipsoid, while most suitable in the UK where the majority of the measurements on which it was based were taken, was well-known in the very early 20th century when the United States Coast and Geodetic Survey set out to triangulate the United States. Almost 30 years of work led to revisions to the ellipsoid and the selection of a new reference point, Meades Ranch in Kansas. This became known as NAD27, the North American Datum of 1927, and remained the basis of most surveying in the US, Canada, and Mexico until a major revision adopted in 1983.

I explain this background to illustrate a core problem in geodesy: precise measurements require a datum, and a datum requires an ellipsoid, which is an approximation of the geoid calculated by fitting to a set of precise measurements. By the mid-20th century, major triangulations had been completed in the US, in the UK, and in many European countries.

The onset of the Second World War revealed a problem with this state of geodesy: the datums were all different. Military planners laid out maps surveyed by different countries and found that they did not meet up at the edges. Not just due to the projection, or due to the illustration, but because these mapping efforts fundamentally disagreed on the plane over which the maps were made. Different countries had used different ellipsoids (and different basic methods of determining the ellipsoids), they anchored them at different references, and they adopted different values for the correction of measurements. National surveys tended to be anchored somewhere in the interior, for centrality, which meant that datum-related inconsistencies were most acute at the borders of countries where different datums met.

An Army officer planning artillery fire would find that maps of the German-French border, produced by the mapping authorities of the two formerly separate countries, disagreed on the locations of towns by hundreds of meters. This was enough to become a dominant source of error in artillery aiming, so the military knew what it had to do: unify the maps of Europe onto a single datum. Soldiers on the front were joined by tactical surveyors, military units that traded rifles for theodolites as they completed one of the war's most important intelligence missions. This effort, to establish a uniform European datum, was not completed until after the war, when Allied intelligence units located the archives of the German cartographic authority and hauled them to the US for calculations.

The result became ED50, the European Datum of 1950.

The problem of flying blind was all the more acute for bombers. Early in the Second World War, Allied Forces had low expectations of German air defense and thought that bombers would be able to operate over Europe mostly unchallenged. Things did not work out so conveniently: German fighters were better than expected, ground-based defenses more extensive, and ultimately, bomber losses were much greater. This forced a change in strategy to night-time bombing runs. With the absent to primitive radar technology of the time, bombers were extremely difficult to intercept at night.

This sword cut both ways: bombing accuracy was also very poor at night as bombers struggled to find their targets, especially during defensive blackouts. German bombers headed towards England encountered the same conditions, forcing the same shift towards night-time bombing. Both sides put to work developing technology for improved night-time bombing. The Germans extended the Lorenz beam into a two-dimensional system that could "designate a target" by radio. On the Allied side, several systems were in simultaneous development in the UK and US. The rapid iteration of radionavigation technology and electronic countermeasures during the bombing of Great Britain became known as the "Battle of the Beams," and set the groundwork for much of our modern navigation technology.

Among the combatants in this ethereal battle was RCA, the Radio Corporation of America, which had unintentionally discovered a means of measuring distances by radio and then greatly improved it by incorporating features of a similar British system. RCA called it SHORAN, the Short Range Navigation system. SHORAN used an aircraft-mounted receiver that emitted pulses and measured the time until "reply" pulses arrived from fixed ground stations. By maintaining a constant delay time to a given ground station, an aircraft could fly a perfect orbit around it. By flying that orbit until a specified delay to a second ground station was achieved, an aircraft could position itself exactly on a bombing target. SHORAN saw extensive use in both WWII and the Korean War.

Let us consider the state of geodesy after the Second World War. By this time, the United States had been thoroughly surveyed under NAD27, the Retriangulation of Great Britain was underway using a datum called OSGB36 (Ordinance Survey of Great Britain, 1936), and European maps were being completed under ED50. Combined with new efforts in Japan, this put most of the Allied or Allied-occupied world into the same condition: accurate datums existed that covered large contiguous land areas, but the exact relationship between those land areas was difficult to determine.

In other words, the distance between two locations in the United States or Europe, even very far apart, could be fairly confidently stated because of the common datum. The distance between a location in the United States and a location in Europe, though, came up question marks. Cartographers, especially of the nautical charting variety, had methods of approximating distances over the ocean but they were of very poor accuracy compared to true triangulated surveys. Ocean charts were made mostly by celestial observation, a field that was struggling to develop into "astro-geodesy" because of the limited precision of telescope measurements. To put a scale on the problem, analysis of mid-century nautical charts against later methods found that many Pacific islands had mapped locations that were several kilometers off, and that even major land masses varied by hundreds of meters.

There is a simple reason for this problem: the methods of surveying. While experiments were conducted with intriguing methods like celestial photogrammetry, photographing the same balloon or flare against the starfield from multiple locations and calculating vectors to determine the distance between these locations, the only truly accurate method was still triangulation by theodolites. This was easy where survey stations could be set up on mountains and hills, but it was already a challenge in forested country where you simply couldn't see that far. It was impossible over the ocean, where there was no way of seeing from one landmass to another.

Up to the beginning of the Cold War, no one had paid that much effort towards surveying over the oceans. Geodesy was fundamentally a local concern: knowing where something was within its country was enough. The numerous countries of Europe and their extensive land borders had made the unified European datum a necessity, but even between Western Europe and Eastern Europe there was little need for standardization.

Geodesy did not become a global concern until the invention of the intercontinental ballistic missile.

By the end of the war, it was already obvious that SHORAN had potential beyond targeting bombers. SHORAN equipment had been tested off of the US coast, during which RCA engineers noted that a small Bahamian island must be several hundred yards away from its charted position. Later measurements with precision astronomical equipment confirmed the SHORAN-derived measurements as more accurate than the charts. A Coast and Geodetic Survey officer seconded to the Army Air Force for the war effort took note of this result, and reported back to the Survey as it was struggling to map the Aleutian Islands.

Many of the Aleutian Islands were close enough to be surveyed by conventional means, at least in theory. In practice, terrible weather and frequent fog meant that surveyors repeatedly tried and failed to spot one island from the next. The cost of flying people around, waiting for good weather, and getting a few measurements through each break in the fog had put the project well behind schedule. After the delivery of SHORAN equipment, a few months of experimentation developed a technique that completed the survey by radio.

At the same time, the oil industry was pushing into two frontiers: the South American jungle and the ocean offshore. Both were formidable challenges to surveyors, the jungle due to the lack of clear sightlines and the ocean due to the lack of anywhere to set up fixed points. Oil exploration companies purchased secondhand SHORAN equipment from the military and ran their own experiments, leading to refined solid-state radio equipment and another set of well-tested operating practices.

Early experiments in SHORAN surveying came up with a frustrating error: SHORAN measurements, as compared to reference points established by traditional surveying, were always a bit too short. It took careful comparisons of SHORAN networks to Coast and Geodetic Survey networks in Colorado and Florida to confirm the reason: the accepted value of the speed of light, up to that point, was incorrect by 16 ppm. In 1949, the same Coast and Geodetic Survey officer, Colonel Aslakson, published a new value for the speed of light derived from SHORAN. That measurement has held up to the modern day, within 1 ppm.

The British Ordnance Survey had maintained close contact with the Coast and Geodetic Survey, in part through Hotine, and eagerly observed these tests. The Retriangulation of Great Britain was back underway after the war, now with the benefit of improved technology, but it was facing similar challenges to those in the Aleutians. Some experimental use of SHORAN was attempted in Scotland. SHORAN surveying was widely adopted in Canada as well, where in the late 1940s Parliament authorized the first precise mapping of the entire country.

With all of these efforts in mind, and its own objectives as well, the Air Force continued research on SHORAN. Among other developments, this led to HIRAN, a high precision version of SHORAN that used improved receiver electronics to obtain much more accurate travel time measurements. HIRAN reduced the typical error of SHORAN distance measurements from hundreds of yards to around ten feet.

The Cold War was a fundamentally different type of conflict from the Second World War. Besides the somewhat speculative nature of the actual conflict, the distances involved were incredible: for the first time in military history, strategists planned out attacks that would start on one continent and end on another.

WWII experience had already shown the importance of accurate mapping for calculating SHORAN bombing targets, but this was less of an issue in the Cold War context since accurate geodesic data for the Soviet Union was a closely held secret and even the best of spies would struggle to operate a SHORAN ground station undetected. Bomber aircraft would have to navigate by celestial observations and then spot their targets more accurately by conventional means, which of course raised the same problems of night-time bombing. This was a contradiction in nuclear reprisal plans that was never really addressed, since by the time research was underway to achieve accurate global navigation the role of the bomber had been supplanted by the ICBM.

ICBMs used primarily inertial navigation, with corrections while underway from celestial measurements. Both inertial and celestial navigation were technologies that saw revolutionary advances due to the ICBM, reaching accuracies that were unimaginable during the Battle of the Beams.

There was a problem: the targeting capabilities of ICBMs were soon superior to the maps that were used to designate the targets.

In the era of the ICBM, the exact distance from, say, North Dakota to Moscow became a question of great significance. Military intelligence set about answering it, and a combination of espionage and information from the previous war allowed for the extension of ED50 into the Soviet Union with some degree of confidence, as well as the anchoring of Soviet maps to ED50. Even before this work was complete, though, it was obvious that with missile silos surveyed against NAD27 and targets surveyed against ED50, the actual misalignment between the two datums would become larger than the probable error of the missiles.

Post-facto, the Department of Defense presents a laundry list of motivations for a World Geodetic System. I'm sure there's some truth to all of it, we know now that accurate geodesy is useful in so many ways. In 1950, though, most of the applications hadn't yet been invented. Older materials tell a plainer story: the military needed to aim missiles, and to aim missiles it needed a single map that contained both the origin and the destination.

NAD27, OSGB36, and ED50 were all sufficient within their scope. What the Cold War demanded was a connection between the two. The Air Force determined that it would have to trilaterate the Atlantic Ocean.

Efforts started from both sides. Canadian survey efforts, already routinely using SHORAN, provided NAD27 references in the Maritimes. ED50 provided references in Norway, and the United Kingdom was well surveyed under OSGB36. If trilateration measurements could be made between Canada and the UK, and between the UK and Norway, that would be sufficient to relate all three datums to each other. This came to be known as the North Atlantic Tie.

To complete the tie, the Air Force would have to island-hop its way across: from Canada to Greenland, Greenland to Iceland, Iceland to Faroe Islands, then to Scotland and on to Norway. The first part of this project was completed between Scotland and Norway with the cooperation of the Ordnance Survey, but observations between Scotland and the Faroe Islands were the greatest challenge.

Here we will finally consider the details of SHORAN. SHORAN is a straightforward time of flight system in which ground stations support a transceiver on the aircraft. The aircraft emitted a pulse on some UHF frequency, and a ground station tuned to that frequency received it and immediately emitted another pulse. The aircraft equipment measured the time between transmitting the pulse and receiving the reply which was, much like radar, an indirect measurement of the distance between the aircraft and the ground station. By using two different ground stations on two different frequencies, and doing some math, it was of course possible to establish the position of the aircraft on a map.

More importantly to surveyors, though, it was realized that SHORAN could be used to accurately measure the distance between two ground stations. If you knew the altitude of the aircraft, and found the shortest possible SHORAN distance to the two stations, it was simple trigonometry to find the straight-line distance between stations. This method became known as "line-crossing:" surveyors would mark on a map the estimated straight-line between the two ground stations, and then draw a perpendicular line that crossed it in the middle. An aircraft would then fly back and forth on that line, taking repeated SHORAN distance measurements until the minimum sum was found. That point—where the total distance to the two stations was smallest—must be at the center point of the cross line. Those values were used, with the aircraft's altitude, to calculate the final measurement.

HIRAN, the enhanced accuracy of SHORAN, used the same basic principal but added a process called "gain riding" to zero out error due to the rise time of the transmitted pulses (which led to ambiguity in the "start" of the pulse). The details of gain riding have become obscure, but it was a manual process requiring a dedicated operator. I believe that the operator was simply adjusting the gain on the HIRAN receiving equipment to find the minimum point at which the pulse was detected at all—at which point the pulse must be tripping the detector at its maximum strength, and thus at the end of the rise time.

Combined with calculations to offset atmospheric effects and the electronic properties of the equipment, all of which had to be extensively researched during the refinement of SHORAN surveying, very fine accuracies could be achieved.

Still, the realities of measuring long survey lines in the North Atlantic were a challenge. One veteran of an Air Force geodetic squadron tells of hours spent flying a modified B-50 Superfortress out of Florida to make line crossings along the North Atlantic Tie. The B-50 was used because it already had SHORAN equipment installed and plenty of room for the operators, but the distances involved in the island hops challenged the range of SHORAN and they had to fly as high as possible to improve their chances of receiving the reply pulses.

The B-50 was a heavy aircraft not designed for high altitudes, and so they thundered high over the Atlantic, engines at maximum power, losing altitude with every turn and then waiting to slowly regain it. A film camera, called a reconnaissance recorder, took repeated photographs of the SHORAN instrument's display as operators adjusted the receiver and logged weather and signal conditions for later use in making corrections. The aircraft was essentially war surplus, not in its top condition, and the heat went out.

You can imagine this group of unlucky airmen, flying back and forth over the ocean in an aircraft that was coming to match the outside air at 60 below Fahrenheit. With difficult reception conditions, atmospheric effects became more significant, leading to error. It took repeated crossings of every line to reach a target statistical confidence level in the result. Back and forth, colder and colder, freezing hands struggling to write out logs, until they got the okay to come back home. They had measured the distance between a ground station in Scotland, I believe in the outer Hebrides, and Iceland—over 550 miles. This was, at the time, the longest distance measurement ever taken by such direct means.

The North Atlantic Tie was completed in 1955. These are hard to judge, in part due to the intrinsic complexity of geodesy. Many measurements were taken of each line, multiple lines were measured between different ground stations, and multiple ground stations were set up on each island or continent. Some measurements were almost certainly accurate to within ten feet. On the other hand, the final trilateration network, after least-squares fitting, left a troubling large inconsistency in the distance between Norwegian reference stations that suggested some uncaught error.

The North Atlantic Tie was one part of a variety of data used for the computation of the World Geodetic System of 1960, or WGS60, the first global datum. WGS60 is a direct ancestor of WGS84, the datum used by the Global Positioning System and thus the de facto datum of global geodesy today.

We now know so much more of the figure of the Earth. WGS84 models the planet as an ellipsoid with semi-major and semi-minor axes of 6,378,137 and 6,356,752.314245 meters and an inverse flattening factor of 298.257223563. These three precise figures describe only the abstract reference plane, a utilitarian simplification of a geoid that we know to be so complex in its shape that the polynomial approximations used today, if fully expanded from the functions that generate them, would have millions of terms. These approximations are accurate enough to be their undoing; they become outdated as the shape of the planet physically changes from year to year.

WGS60 was the beginning of global geodesy, but it was also the end of global trilateration. By the time the computation of WGS60 was complete, satellite Vanguard 1 had reached orbit. Phase and Doppler measurements of its radio transponder could be used to indirectly determine the position of a ground station relative to the satellite, and then by extension to another ground station. Some of these measurements were incorporated into WGS60, and its fast-following revision WGS66 was heavily corrected based on satellite measurements. The space age had come, and with it, the era of long-distance surveying had gone.

The old school of geodesy had one last hurrah: WGS84 incorporated data prepared for NAD83, which heavily used the Transcontinental Traverse, a 1961-1976 effort that physically measured the United States by headings and distances. This was likely the last large-scale physical survey in the world. By the time it was completed, networks of satellite ground stations had been incorporated into various national geodetic systems, grounding the datums ironically in space.

Geodesy is a complicated field with many fascinating stories. For the sake of simplicity, I have left so many details out of this sketch of the first trilateration of the globe. Today, we take accurate geospatial information for granted, and military secrecy around geodesy is mostly a thing of the past. Our maps were hard-won, though, the result of decades of ingenuity and hard effort to answer questions that have troubled scientific thinkers for centuries. Magellan sailed around the globe, but he didn't know how far he'd gone. Hotine retired from the army to help found the Directorate of Overseas Surveys, for which he explored the world again... this time, counting every mile.

There are many cases where some sort of debugging block sneaks by, especially cases where we see preprocessors or templates working, which leave us with nonsense like if (true == false) running in production. But Codemonkey found a new twist on that sort of thing, in a SQL query being run in production.

WHERE (some conditions) AND (1 = 0 OR (1 = 1 AND (other conditions)))

The OR means that by twiddling the first equality check, we can toggle "always return rows" with "return based on condition". Toggling the second we can make it "never return rows", which I'm not certain is actually useful. I can see how these likely did start life as debugging flags, but they're still weird, still unnatural. They point to some other problem in observability in the code. And, as all "good" flags go, they're not documented anywhere, this seems like it started life as a query an analyst was running until it got turned into stored procedure to be run again and again. The flags have never been changed since the code was released, as far as anyone can tell.

The opposite of meritocracy is kakistocracy: the worst and least-qualified are the ones who rise to the top.

Get real familiar with that word, dear readers. I think you'll need it.

If anyone can back me up on this, it's our submitter, Jared B:

I am a teacher by profession, and worked for a year at an ed-tech company founded by a mechanical engineering professor, Harry. Harry had spent a great deal of time in the 90's developing a C interpreter (yes, you read that right). 30 years later, he remained convinced that his interpreter was the technology of the future, and had founded a company that offered math and computer science curriculum to K-12 students based on C programming.

Originally, he had written a textbook that introduced students to programming using a locally-installed version of his interpreter and custom IDE. A little vain, but no serious problems. As Chromebooks grew popular in schools, he had developed a web IDE where students could write and run C code.

But Harry could never give fully give up on the Windows IDE for his C interpreter. So, he included in the web version a "Run Locally" button for those school computers still running Windows. It worked like so: installing the interpreter and IDE locally would also install a daemon that activated on startup and ran a websocket server. This server had an endpoint which accepted as a parameter a string of C code. It would then pass this C code to the locally-installed interpreter to run.

As you might suspect, there was no authentication whatsoever on this local websocket server. Knowing the form of the protocol, ANY domain could connect to localhost:12345/execute_c_program and send arbitrary code to run (of course, Harry prided himself on the completeness of his C implementation, including execv() and the like). Trick a user into visiting a malicious website, and you automatically had RCE on their computer.

Adding insult to injury, I discovered that the server was bound to 0.0.0.0 so that if you had Harry's software (/malware) installed, any computer on the same network as you could send you arbitrary C code to execute without question.

These vulnerabilities had existed for several years before I joined the company. In all that time, Harry had never hired anybody but his own grad students as software developers, and none of them had noticed the problem. By that time, the software was installed on thousands of school-owned computers throughout the state.

I documented and demonstrated the vulnerabilities to Harry. He did release a new version of the software addressing the issues and citing "security improvements" in the release notes, but there was never a communication to school/district IT leaders to describe the importance of updating. I suspect that Harry should be in serious legal trouble for potentially compromising data related to schools and minors, but I've since moved on and dropped the subject.

During the year I spent at the company (not in any sense as a dev, mind you, but as a lowly curriculum writer), I also discovered and reported a cookie-stealing exploit that would have compromised student and teacher data, as well as a code injection on another of Harry's websites (he decided to demonstrate that his C interpreter could work as a web server via a page where a user could type a math expression, which was then eval()ed server-side without any sanitation). The latter vulnerability gave me remote access, where I discovered thousands of transaction records that included credit card information stored in the clear.

Harry's company is still in business to this day, and has recently been ranked in TIME's list of top American ed-tech companies. Oh, and the office router's admin page still had the default Google-able username and password, but that one's a freebie.

I knew someone like this once, only they were stuck on ColdFusion long after everyone else stopped caring about it. However, I don't think they went on to endanger an entire state's educational system, only to be lauded as a visionary leader. Can't say for sure, though.

[Advertisement] Keep the plebs out of prod. Restrict NuGet feed privileges with ProGet. Learn more.

# Introduction

In this blog post, you will learn how to make a private IPFS network that will allow private peers to exchange data between each other.  This is in opposition to the default setup in which you use the default configuration connecting to the whole IPFS network.  The incentive to have your own network is mostly speed, because this does not have any extra access control, any peer can potentially access to any content you have in your IPFS network.

If you do not know about IPFS, I wrote some blog posts, but otherwise IPFS website may give you some clue.  It is not easy to grasp, but you can see it as a network service providing content addressed object storage.  The base blocks provide features to manage data in the network but also cache it or provide it through a web gateway.

=> https://ipfs.tech IPFS project official website

Kubo is the reference implementation of IPFS.

The setup I explain in this article is pretty much useless for most people.  Depending on your needs, you will have a better experience with Nextcloud/Seafile for central file sharing, Peergos for encrypted storage, syncthing for directory synchronization between peers, or even magic wormhole for one-shot transfers.  I am still not sure which purpose an IPFS private swarm serves correctly, but none of the use cases I just listed, maybe it is useful if you need to keep some dataset synchronized and quickly available in multiple areas, but then an hyperscaler may be more suited.

# Setup

In this setup, you will have various nodes that can speak to each other (through a VPN, LAN or directly over the Internet) and potentially but not mandatory, a node that is up 24/7.

They must all share the same `swarm.key` to create a swarm (a group of nodes, that is private until you get access to the key).

The setup itself will allow nodes to publish data and give it access to other nodes for caching, downloading or relaying, but also give the opportunity to publish on a web gateway.

# Too Long Didn't Read

IPFS daemon will hold data until it reaches its maximum allowed size, then will run a garbage collector to reclaim disk space by removing unused data.  Data which was pinned on the server will never be garbage collected.  A directory or file is content addressed, this mean is has a unique address derived from its content, which mean the URL of a resource depends on its content, if you update a directory, it will get a new URL because its content changed.

IPFS has a mechanism called IPNS which allows to publish a content hash under a fixed hash, which is published to other peers under some conditions, this is the only way to provide an address that never change but in which you can change the content.  An IPNS address requires an associated cryptographic key that is managed by kubo, if you want to pin a resource for each people you share data with, you will need a dedicated key for each.

Data do not propagate magically over IPFS, if you add data to your node, it remains local until another node pulls in the content, then it will be able to distribute the data too, but there is a high chance it gets garbage collected one day if the node owner do not pin your data.

IPFS has no at rest encryption or access control available, it is a method to publish data.

# Configuring a node

First, you need to install Kubo, make sure to not install kubo-desktop which is unfortunately incompatible with a private swarm.

I personally prefer to use it in a container version for sandboxing reasons.

Generate the swarm.key with this code:

```
echo -e "/key/swarm/psk/1.0.0/\n/base16/\n$(openssl rand -hex 32)" > swarm.key
```

Download the ipfs-webui which is normally downloaded by kubo from the IPFS network, but as you are not connecting to it, you need to inject the webui in your kubo.  On the following URL, download the "car" file that matches your kubo version.

=> https://github.com/ipfs/ipfs-webui/releases

Now, create and init your IPFS daemon using this script, if you do not use podman and prefer to use `ipfs` binary, replace the long podman command by just `ipfs`.

The script takes `swarm.key` as the first parameter and the web-ui car file as a second parameter:

```
#!/bin/sh

set -xe

if ! test -f "$1"
then
    echo "You must give the swarm.key file in parameter"
    exit 1
fi

if ! test -f "$2"
then
    echo "You must give the ipfs-webui@v4.13.0.car file in parameter (version should match your kubo version)"
    echo "You can download it from https://github.com/ipfs/ipfs-webui/releases"
    exit 1
fi

# execute commands with ipfs
run() {
 podman run --replace --name kubo \
   -e LIBP2P_FORCE_PNET=1 -e IPFS_PROFILE=lowpower \
   --userns=keep-id \
   -v $HOME/.ipfs/:/data/ipfs:Z \
   docker.io/ipfs/kubo:release $*
}

# import webui, requires stdin
import_webui() {
 podman run --replace -i --name kubo \
   -e LIBP2P_FORCE_PNET=1 -e IPFS_PROFILE=lowpower \
   --userns=keep-id \
   -v $HOME/.ipfs/:/data/ipfs:Z \
   docker.io/ipfs/kubo:release dag import < "$1"
}

# make sure the directory exist
# you can adjust if you want to store it elsewhere
mkdir -p ~/.ipfs

# copy the swarm key
cp "$1" ~/.ipfs/swarm.key

# uncomment this if you do not use the container version which does it automatically
#run init

# allow server to give other peers address
run config Routing.Type dht

# allow other peers to relay if a peer is not directly reachable by us but another peer can
run config --json Swarm.RelayClient.Enabled true

# remove all the default stuff
run bootstrap rm --all
run config --json Routing.DelegatedRouters '[]'
run config --json Bootstrap '[]'

# private swarm key = TLS can not be used but it is still encrypted
run config --json AutoTLS.Enabled false
run config --json AutoConf.Enabled false

# allow to use IPNS through pubsub and advertise often
run config --json Ipns.UsePubsub true
run config Ipns.RecordLifetime 240h
run config Ipns.RepublishPeriod 1m
run config Ipns.MaxCacheTTL 1m

import_webui "$2"

echo "Setup successful"
echo "You can visit http://localhost:5001/webui/ after starting the server"
```

Now, start the server (use `ipfs daemon` and the environment variables if not using the container):

```
podman run --replace --name kubo --restart=always \
  -e LIBP2P_FORCE_PNET=1 \
  -e IPFS_PROFILE=lowpower \
  -p 8080:8080 -p 127.0.0.1:5001:5001 -p 4001:4001 -p 4001:4001/udp \
  --userns=keep-id \
  -v $HOME/.ipfs/:/data/ipfs:z docker.io/ipfs/kubo:release
```

On the webui, in the Peers menu, add your other peers.

# 24/7 server specific setup

The snippet above will work for a server, but you want to change a few things:

If you use a container with restricted network, you need to give a reachable IP address to announce to your other peers:

```
ipfs config --json Addresses.Announce '["/ip4/192.168.1.166/tcp/4001"]'
```

You may also want it to be the DHT server to allow peers to discover each others through it, and maybe relay data between peers which could not connect directly:

```
# allow to give other peers address for mesh networking
ipfs config Routing.Type dhtserver

# allow to relay data between two peers that could not connect to each other
ipfs config --json Swarm.RelayService.Enabled true
```

You may also want to limit the amount of storage allowed in your Kubo server before the garbage collector free some space:

```
ipfs config Datastore.StorageMax "50GB"
```

# Networking

There are 4 differents ports in use that you need to be aware of:

* Port 4001 TCP and port 4001 UDP are used by peers to exchange data between each other
* Port 5001 TCP is used to reach the admin API and the webui, do not expose it publicly
* Port 8080 TCP is used for the gateway, it allows to expose your IPFS private network content to people able to reach the gateway, without them installing kubo at all

# Conclusion

In my use case, I have a script regularly pinning a list of IPNS addresses on the server running the HTTPS gateway, so it always download the latest version of the IPNS published resources and make them available through the gateway even if the computers owning the file is offline.

This is actually not super useful as I could have done the same thing using Nextcloud, or with a script copying the files to a static HTTPS server.

# Going further

This infrastructure got more useful after adding a single `ipfs-cluster` service near my gateway running 24/7 allowing the nodes to use it as a remote pinning service.  Now, a node can pin a CID on the gateway which make sure the data will be available locally there.

Newsletter updates were sporadic in July because of two weddings, two conferences (with two different talks!), and finishing Logic for Programmers. Huge thank you to everybody who bought a copy, as well as for your patience with the schedule. There's some podcast appearances, a conf talk, and a book sale at the end of this post.

Newsletter updates will be sporadic in August because I just started my Developer Educator job at Antithesis. I'll have less time to write because I'll be working 40 hour workweeks, about 8 hours of which being actual work and the other 32 being bashing my head against NixOS.

NixOS is the standard developer OS at the company. It's also a notoriously difficult distro to learn even for Linux heads, and I'm coming from Windows. The only way I am going to get anywhere is to go all in and commit fully to the NixOS philosophy.¹ For one, I'm seeing how long I can last without my customary 2000-line Neovim config.

Which immediately raises the question as to why I have 2000 lines of Neovim config. It's because Vim² (and Emacs) think of configuration in a very different way than more popular editors do.

Control and Consumption

Say we want to make ctrl+n to save the current file. In VSCode, you put this in keybindings.json:

[
  {
    "key": "ctrl+n",
    "command": "workbench.action.files.save"
  }
]

In Neovim, you put this in init.lua:

vim.keymap.set('n', '<c-n>', function() vim.cmd.write() end)

Now, a couple of differences to see. First, the VSCode example is invoking a fixed, built-in command, while Neovim can bind an arbitrary function. Second, in VSCode you edit a static configuration file with static data, while in Neovim you execute a command that edits the running editor state. In fact, it doesn't even need to be in a configuration file: you can add a new keymap directly from the command line. Though you'd probably instead do that command in the OG Vim way:

map <c-n> :w<CR>

And that does something different than a function: it makes pressing ctrl+n mean "do whatever typing :w and pressing Enter would do." In default Vim that is the same as saving a file, but if you remapped : to o then it would instead do the equivalent of ow<cr>, which would type the character w on its own line. ³

In other words, Vim gives you incredible programmatic control over the state of the editor. Want to make typing ;r paste from the clipboard? Easy. Want different setting options in normal and insert mode? Go ahead. Want to make "writing a file" do something different during a full moon? You could if you want. ⁴

Now, you can do some amount of customization in VSCode, especially with multicommands, but for the majority of complex stuff you need to write a plugin. And making a plugin in VSCode is a much heavier process than making one in Neo/Vim. If you want to make a command that prints the word count, you have to 1) learn TypeScript, 2) scaffold a special VSCode extension project, 3) define a wordcount function, 4) register the mycode.wordcount command, 5) add mycode.wordcount as a contributes record in the extension manifest, 6) package or publish your extension, and 7) import the extension. It's very clear that the plugin system is not meant to let you tweak in a bit of functionality, but rather to let specialists produce complete plugins for other developers to consume. And since so much of the advanced functionality of VSCode is only possible through plugins, this limits the control the average user has over their environment.

Neovim also has plugins, but they can't do anything you couldn't already do in your default config. Admittedly, some bits of the APIs are meant specifically for plugin specialists, but they're still documented and available for your personal configuration. You never know what someone's gonna need!

In summary:

• VSCode has a two-tier system, where plugin makers make plugins that users consume. The plugin has more power than the user. Most users aren't expected to make their own extensions.

• Vim has a one-tier system: the user can do anything a plugin can. Everybody is always able to extend the system and have as much control as they want.

Most people should be consumers

Confession: this all is unintentionally a little ragebaity. "Consumer" is a dirty word in software. A consumer is somebody at the mercy of a producer's decisions. I think there's even a connotation of passiveness in being a consumer. It sort of starts leaning into a moral judgment: developers should use Neovim because it gives them control.

But, and this is intentionally a little ragebaity, it's the other way around. Most developers are better off sticking with the consumer model and only switching to a control-editor if they really, really want to. ⁵

First of all, learning to configure and extend Vim is hard. It took me a long time to get comfortable with even making basic scripts. And even if I'm comfortable hacking Vim, I still need to consume other people's plugins if I want a modern developer experience. I'm not writing my own treesitter integration from scratch! ⁶

Second, while the experience of developing plugins is worse on VSCode, the experience of consuming them is far, far better:

• All plugins install, set up, and are used the same way. When I install a new plugin I don't have to spend ten minutes reading docs to get it working.

• I can read a list of all the new commands, keybindings, and configuration options added by the extension. I can edit any configuration option in the same settings UI, which has features like global search and input validation.

• I can enable and disable extensions without changing my startup scripts, and I can enable extensions for only specific workspaces.

• If a plugin adds a keybinding that conflicts with my custom keybindings (or another plugin's keybindings), VSCode will tell me instead of silently clobbering the older one.

• I can't break a plugin through a weird script I run at startup or by installing a different plugin.

• A plugin can't break my startup.

These are all possible precisely because the VSCode plugin system is so heavyweight and inflexible and because everything is configured through static JSON files. Figuring out what configuration options a Neovim extension has is tantamount to solving the Halting Problem.

(The broader principle here is the ability-guarantee tradeoff. ⁷ A VSCode extension can do fewer things than a Vim extension can, and therefore we have more guarantees about what it actually does. Exploring the AGT is one of the running themes of Logic for Programmers.)

The consumption paradigm is worse than the control paradigm in a lot of ways but it's so much better in this one specific, extremely important way that it's the right choice for most developers. To some extent I wonder if preferring control is more a personality trait than a measured tradeoff. I'm unhappy when I can't tweak some software just to my liking, it just grates on me that something's off and can't be fixed. If tomorrow I woke up and was just not bothered by that, would I still prefer Neovim to VSCode? I dunno. Maybe I'll find out as part of The Nix Experience.

Now there's a third point in the design space I haven't talked about: what if the editor restricted both your control and consumption? Helix, for example, allows adding LSP servers and treesitter grammars but not any other kind of plugin.⁸ It also has a static and very limited configuration language. You can't even set different keybindings for different filetypes.

That seems crazy to me! But a lot of people seem to like it, and I've been interested in Kakoune-style modal editing for a while now. So now I'm running Helix as my main terminal editor. I don't know if I will stick with it; I feel like I'll eventually crave a more hackable editor. But at least then I'll be more comfortable with Nix before trying to import my thousands of lines of Neovim conf.

Appearances and stuff

Three this time:

• I was on the Pragmatic Engineer podcast, talking about math, formal methods, AI, and the Crossover Project. Listen if you like the word "like" and check out the video for mad Stan hands.
• I was also on Leanpub Launch advertising LfP (accompanying blog).
• Last month I was at the inaugural Software Should Work conference, giving my standard talk on practical ideas in formal methods. But for some reason my computer wouldn't connect to A/V and nobody on the staff had Powerpoint, and large chunks of the presentation just Didn't Work on keynote. Watch me tapdance on a tightrope in an earthquake here. Somehow I pulled through.

Oh, also Amazon is selling Logic for Programmers for 15% off for some reason. I confirmed I get the same royalties either way, so hey, it's cheaper with no downside. I have no idea how long the sale will last.

AI isn’t killing engineering. It's making it less meditative.

There’s a lot of talk about AI killing engineering jobs. Some jobs will change, and some will disappear. But we’ve been automating engineering work for decades, and somehow we keep finding more engineering to do.

At Web Summit 2026, I talked about how AI can raise the floor for developers, and what that means for juniors trying to break into the industry.

If you'd rather read than watch, the full transcript is below.

Transcript

0:03 Hi. Hello. Thanks so much for joining us and I'm so delighted to have this opportunity to speak with Avery because you know, I hear from young developers all the time as well as employers who are trying to figure out what it even means to build a tech career or to manage a team of developers in this world where

0:29 you know, I'm a I myself have no CS training and I now make software and of course there are lots of companies all over the world where people are vibe coding in ways that change what it means to be a developer. So Avery, you obviously have had a whole career. You're an engineer yourself. You've managed engineering teams and now you're in the position where you're I assume well, I went and looked at Tailscale's

0:52 current job ads. How how do you think the role and experience of building a career as an engineer has changed over the course of your career and in particular with the advent of AI? So yeah, you talked you talked about young developers. I'm I'm clearly not a young developer. I'm an old developer. So I've seen a lot of stuff. When I was

1:14 when I was going to university in the late 1990s, there was a program that I didn't take called software engineering that was competing with computer science. I actually ended up taking computer engineering, but we had a lot of discussions about what what is software engineering? This is like a new thing. Yeah. And we're in Canada and it's like engineering is actually a protected legal term in Canada. If you work for a company and you don't have an

1:36 engineering license, they have to call you a software developer. You can't be a software engineer in Canada unless you like meet these certain criteria. And so like at the time the joke was like look, there's no such thing as software engineering. Like if you're an engineer, right? And you're building a bridge and the bridge falls down, people are going to sue you, right? And if you're selling software, you just put in the license agreement. Yeah, sorry, it's not my fault. Haha,

1:59 you know, as is. Yeah. Right? And that that's the difference, right? But over time, we have actually figured out what real software engineering is in the intervening years, in the last like 25 years since then. Like, engineering is taking responsibility for your work and understanding that like everything's going to break eventually. And like, monitoring that and deciding when is it going to be okay for it to break and what are you going to do about the fact

2:20 that it's going to break. And so, to me, like software like software engineering is new. Right? What's different now is is suddenly the part of the job that was used to be called you called computer science or used to be called software developer, like a lot of that's disappearing, right? But the engineering part is exactly the same as it always was. Like, that's what people want to buy. They want to buy the guarantee that this thing is is when it if it and when

2:42 it falls apart is not going to kill people. So, I mean, I feel like that's a distinction that a lot of people are missing, as you can see if you use five-coded software. The it won't break thing seems to be a loosely held goal. Um you know, for folks who are are building and managing teams of developers now, where people come into the field, um you

3:08 know, how how much do you think developers still need to prioritize I almost want to call them like it's weird to describe coding as an old-timey skill, but it's starting to feel that way. I mean, do you still think that when you're hiring, you want somebody who knows how to write the lines of code or is the guarantee that it won't break about a different sort of a different

3:32 lens? So, I should say that the guarantee that it won't break is not is more like, you know, deciding how likely you want it to be to break, right? But my first year engineering class, I remember we did a uh they made us do this experiment where they gave us like a bunch of paper clips and you had to just bend it back and forth and then write down how many times it took before it snapped. And then you did that with like 20 paper clips, and then you had to like plot it on a curve,

3:55 right? And then and the guy like combined all of our answers, and he was like this like beautiful Gaussian curve. He was like, "Hey, this is reality. There is no paper clip that doesn't break. Some of them break in one bend, right? You have enough of them, like some of them are going to break in one bend, and some of them like didn't break until like 20 or 30 bends." And he said like, "You need to understand, they don't make paper clips that won't break until 100 or 1,000 bends because nobody

4:18 would buy them cuz they would be too expensive, right? The paper clips people want to buy are the ones that break, right? And so, that's what engineering is is understanding those constraints. And so, as somebody who's like building software, it's like, "Okay, like it's okay if I write code some crappy stuff that barely works if that meets the specification. If I'm writing an app for myself, right? No, it doesn't

4:42 matter. It can break and I'll just tell Claude to fix it again, right? If I'm trying to sell something to a billion users that searches the internet or whatever, it's like, "Hey, it it needs to work." Well, I I love this analogy because I I feel like as a user, my special power is I know how to bend the paper clip to break very quickly. I feel like I should That should be a monetizable service

5:04 that I provide to software companies. But I'm also really struck that um when I I I do hire uh developers uh for different kinds of projects, and that the ability to anticipate when something is going to break is part of how I antici- like is how I evaluate, like how do people handle the fragility of what they're what they're building, and how do they

5:26 detect it. And I guess what I'm curious about is as AI becomes more and more how people navigate the development process, um do you think that we're losing some of those skills around understanding like the user interaction, understanding the um security implications, for example. we

5:49 were talking about that a little bit. I I'm I'm curious about whether you think that people who are kind of essentially growing up and learning um learning the field while these tools are already available, are they missing some of those basics now? Yeah, it's it's an interesting question. I think it it's hard to tell like exactly what the value of those basics are. Like when I was growing up, uh I started programming and we had like this

6:12 little computer at home that cost a few hundred dollars. Yeah. And allowance. And like that the time they sold the like the assembly language assembler for like a hundred dollars. And then there was a C compiler for another hundred dollars that we depended on the assembler. And I could not afford the second hundred dollars. And so I bought the assembler. No way. And I'm like, well, that's that's what's going to happen. I'm going to read the because programs actually came with books at the time. So I read the book

6:34 and I learned assembly language. And so I know how the guts of computers work, right? And then like by the time I'd saved up another hundred dollars, my computer had been obsoleted and like the last copy of the compiler they'd thrown it out at Radio Shack. And I could never get a C compiler. So then we had to switch to long story. But the the point is that I know assembly language. Since that time, I've written almost zero assembly language.

6:57 Right? It's just it's obsolete. Compilers have like eliminated the need to write assembly language. And yet the fact that I learned assembly language gives me a like a leg up on a bunch of people who have like come out since then have never had to write a line of assembly language in their lives. Right? Does that mean those people are obsolete? Does it mean they can't get a good job? Does it mean they can't write good software? Like, no. Right? I can write some software that they can't

7:18 write. But also they can write software that I can't write because they learned something different instead. Mhm. Right? And like the lines of code are just not that important, right? Like stuff you listed about like understanding user needs, right? And reacting to feedback and debugging things and architecting things, like none of that's going away. Like AIs are not doing that stuff for you, especially the user feedback. AIs have no idea what the user experience of your program is.

7:42 And that's like the defining element of engineering. Like what does the user need, right? Does it need to be good? Does it need to be fancy? Does it need to be expensive? Does it need to be cheap? Does it need to scale? Does it need to have a button over here versus over there? The AI can't tell you any of those things, right? And if you're distracted by lines of code, you're not going to think about those things as much as you should. It's It's interesting to hear you say that because one of the things that I

8:06 really struggle with at this point is how how much do you think um folks in those early stages of their career should be investing in Okay, I'm going to say assembly language, maybe not not so much, but you know, in the um nitty-gritty of being able to write a complete program, let's say, what you know, in whatever language, but versus like an a a you know, a young developer

8:31 who maybe has primarily focused on figuring out the requirements and the IA and those sorts of pieces and then is using you know, various AI coding agents to do the I don't want to call it the heavy lifting, but like the rote work, all of the generating of the code. Like if you're hiring a developer or if you're advising somebody who's new in the in the field, are you encouraging them to learn the line-by-line

8:56 code review skills or are you encouraging them to like go manage a team of a hundred virtual coding agents? Well, the funny thing is like all this stuff is valuable skills, right? Like, you know, to this day, if you learn assembly language, there are jobs you can get that nobody else can get because like the people who run these LLMs on GPUs are doing stuff in assembly language to optimize those LLMs on GPUs and train them faster, right? Those are

9:20 very very very high-paying jobs because so few people know how to do them, right? If you want a job that like if you want some skills that'll make it easy to get a job like anywhere, you should probably learn how to train a hundred LLMs or like manage a hundred LLM agents cuz that's what everybody's trying to hire right now. But like both are fine, right? So my advice to people is like if you think it's fun, you should probably learn it cuz

9:41 learning is a skill on its own and the more you learn, the more value you're valuable you're going to be. I know so much random stuff about so many random things. [laughter] And like each day, I'm like, wow, it's surprising that this dumb thing I learned cuz I was interested when I was browsing Wikipedia just paid off in my job as the CEO of Tail Scale. And I just like it occurred to me, it's like, oh, this is like that. I can do it like this, right? And that skill is actually going to be more and

10:04 more valuable cuz like cuz there's going to be weirder and weirder problems. I I mean, I I buy that, but again, I feel like the actual nature of learning is changing so quickly because of AI and how people learn like not just tech technical skills, but any any skill. And I will admit like I do think of this partly as a parent because um

10:26 you know, I have a kid who I thought would be you know, worst case scenario, work from home as a kind of coder by the hour. And those jobs are already gone. Like they're gone now. So, those jobs are not going to be there. So, you know, what what should people invest in learning and what are the like learning strategies that are going to give somebody some longevity as the field of

10:53 um technolo not just I don't I was about to say software development, but it really goes beyond software development as like all of these tech jobs get totally reimagined as AI becomes a bigger and bigger part of the production process. So, the funny thing is like again, I don't I don't know that you need to like over optimize up front, right? If it's something you hate, like I don't think you should force yourself to learn it

11:16 for the most part, right? And because like as the world is progressing, not only is more stuff automated, which is like one thing, but it's it's becoming easier and easier to learn stuff when you need it. Yeah. Right? I got my four-year-old a little stuffed dinosaur. There's a startup in San Francisco that's making these stuffed dinosaurs and it's like an AI stuffed dinosaur, right? And it's fine-tuned for kids. You can like dial your child's age, and it'll talk to them like at that age. But you can ask

11:39 anything you want in the world, and it will it will explain that to you, right? And I didn't have that when I was four, right? But he he asks like difficult stuff, and it can explain it in kids' terms. Like so, if you want to learn assembly language today, you don't have to go through what I did, where like every [laughter] every compile took like 5 minutes, and if you make one typo, it's like, "Whoops, another 5 minutes." Right? Now

12:01 it's like instantaneous. You can ask Claude, "Teach me assembly language, right? Quiz me on assembly language." And you can learn what you need to learn so much faster. So it's so much more important to just like, "Look, be interested in stuff. Learn what you're interested in." Because everybody in the world is like suddenly been up-leveled like two levels, right? Where you weren't a programmer before, like now you're by default. Everybody in the

12:24 world is suddenly a programmer. Just install this thing, and 5 minutes later, you're writing programs. Like that is not obsolete. It's not like you your skills have gone away. But if you know things, you're up-leveled even more, right? The more stuff you know, the more stuff you can do with the same tool. I I I mean, I'm I'm not 100% sold on that, because one of the things that we're observing very quickly is this phenomenon of cognitive offloading, where

12:47 because AI can do these tasks for you, you don't really it it's sort of like a veneer of learning rather than real learning. And again, if you think about that different like those stages of the first job, where maybe you can get by with that versus where you're going to be 10 years into your career, and with the with the hopefully the goal of managing projects

13:10 or managing teams, if you skip over that deeper learning in those earlier stages because AI is kind of answering too quickly, um then you have to make up for it later. And so I'm wondering what you know, what what do people do to challenge themselves in those earlier stages so that even if they're doing kind of the wrote parts of of tech projects, they're building the skills

13:35 that are going to support them over over time. Yeah, I mean I I actually I don't really believe in cognitive offloading as a phenomenon. I think people said the same thing when calculators came out. Like, "No, you need to learn how to do long division on paper." Right? And it's like, you know, I learned that in grade school. I have never once done long division on paper since grade school. And and my my brain has not atrophied, right? But what's what's what's really

13:59 dangerous, the thing that is is truly bad for you, is so-called decision fatigue. Right? And so the danger they talk about this with self-driving cars, right? If you have a self-driving car where you like a supposedly self-driving car where you must keep your hand on the steering wheel because every now and then it's going to make a fatal mistake that would cause an accident and it's your job to prevent that fatal mistake. Right? You're going to be like, "La la la." I'm thinking about something else.

14:21 You're not actually paying attention the way you would be if you were actually driving the car. Yeah. Right? You're only left to be like super hyper alert supposedly for this like one of hundred chance that it's going to screw something up. And you're not going to be paying attention and that's when it gets fatal. And this happens like this can happen if you apply AI to all the supposedly easy stuff or the low-level parts of the job, but you're constantly it's popping up these like

14:44 benign questions. Yeah. Right? And like Claude Code does this when you don't run it in dangerously mode, right? It's like, "Hey, can I do this?" Yes. "Can I do this?" Yes. "Can I do this?" Yes. Here's a 10-line bash script. "Am I allowed to run this?" And I'm like, Yeah. Yes. Right? [laughter] I know. you're not actually making decisions anymore. Now your brain is just like turning to mush. Right? But you don't have to run it that way. Right? What you can do instead is you can have it

15:07 eliminate a bunch of stuff and only bring you the things that are important sometimes, right? But when they come in they're like, "Oh, that's an interesting question. I hadn't thought of that." Right? And that is the opposite of your brain atrophying. It's like, "Oh, that's an interesting question that I I have never even thought of because I was too busy writing lines of code, right? I I'm I So So I totally agree with what you're saying and I also observe that not everybody opts to keep challenging

15:31 themselves. And you know, this is one of the things that I find interesting about, you know, having just written a book about neurodiversity in the workplace and really seeing how um the kind of prototypical programmer brain, you know, people have I mean, I think this has changed, but people used to go into software with a very certain kind of like problem-solving mentality. And so now, you know, you can apply that

15:56 problem-solving to higher order problems. But for folks who were entering the world of software development because it was less secure job and not like an itch at the back of their head, there is that ability to go on autopilot. So, you know, how do you encourage developers, I don't know if we're talking about on your team or people you're talking to, how do you know when you're in autopilot and how do

16:20 you know when you're continuing to challenge yourself? What are some habits you could put in place that ensure that continued growth? Yeah, so one thing I think there's research now that shows this, we saw it in our team as well. Like there's really it turns out there's two kinds of software developers. There's the kind that learns software development cuz they really like typing lines of code into a computer all by themselves in a room for hours at a time. It's really meditative and it is really meditative.

16:42 I love that process, right? Like when I got into coding, I'm like, I love this. It gives me an excuse A not to talk to any people and B I'm doing something useful and it is it's so quiet and I can clear my head and I can get some stuff done. And some people that is the goal is to like have that feeling all day, right? And it's great that you can get paid for it. Um and in the early days of computing like famously people who like people were like, "Huh, I can't believe

17:05 they're paying me to like babysit the mainframe at the university when I would obviously do this for free cuz it's so fun, right?" And then the other kind of person is like, "I just like solving problems, right?" And so the the people who just like meditating at their computer, they are a little bit at risk, right? Like let's be realistic, there's going to be less jobs meditating at the computer because that's actually the thing that isn't using your brain. It's

17:30 meditation is like the opposite of using your brain, right? It's like how do I get the rest of the stuff out of my brain? Stopping and having to ask the like hard questions, like what is the problem that I'm trying to solve and how do I do something useful to solve this problem? Those questions are scary for some people. Now, I found out luckily for me, uh I'm in category two. I just It turned out I just really love solving problems. I actually don't miss typing

17:52 code into a computer at all, right? I thought I would. I thought this was like my whole identity, but it's like, nope. What's really fun is like, oh, I identified a problem, I can create a computer system that will solve this problem and then a whole bunch of people benefit from the thing I created. Like that's awesome, right? But I have to get my meditation somewhere else, right? Like that that part of the job is not there. And I know we have people even at our company that like their identity is,

18:15 you know, this is what I am. I'm a programmer who types code into a computer and I love it. This is being taken away from me. It's like it kind of is. Um and there are there are nevertheless programming jobs that AIs cannot do that you can still do. Uh but it's that's that's where you're a little bit at risk. If there's a specific thing you just love to do over and over again, I don't know. But solving problems is never going to be obsolete.

18:37 It's it's it's a really helpful distinction because I I just had this conversation with a young developer recently who was basically saying, I don't want to do by coding. I like the sitting in the meditative and I was just like, well, I'm I'm sorry you were born 20 years too late for that career. Like I don't even know what to tell young people in that Yeah. What do you do? Do you just

18:59 totally change fields or Yeah, well, I think, you know, getting a little abstract. Like, you know, I still do spend my time quasi meditating. I don't I don't meditate in the official sense of like sitting there and listening to your particular kind of music and folding my legs in a particular way or whatever, [laughter] right? But like sitting there and thinking is is suddenly an extremely valuable skill. Yeah.

19:19 Right? And it's like hard for me as a CEO cuz usually my calendar is filled absolutely to the brim with meetings, but sometimes I just have to clear out meetings for like a week. Yeah. And my job is to sit there and like process all this stuff and like have like one insight. It's like, "Oh, this is the thing that will solve the problem." And the the neat thing now is that like some for for many people that one insight is like, "Okay, now I can bring this to Claud and it can come true

19:43 an hour later, right?" Oh. And before it's like, "Well, I had this great insight. Now I need to build a company to build the thing so I can tell people to set up a team so that they can solve this problem in 6 months or a year, right?" And it's like the the distance from like you can have that meditative state to like I have this brilliant idea that now has come true. It can be like a day. You know, but I mean it's So so

20:06 the the flip side of that as somebody who used to not be able to do all my crazy ideas is now you can do all the crazy ideas. Like because it's so easy to make the thing, it's really easy to make like an endless array of crappy software products. I mean, it brings us back to our paper clips. So again, if the goal is to have developers who are capable of creating and delivering

20:32 actual functioning software that breaks after 20 bends instead of two bends and that actual human beings might want to use and that aren't just like a stick-a-fantic AI's like idea of good software. How do you as a as a developer who's working with your 100 LLMs on a day-to-day basis and not in the guts of the code,

20:54 you know, what do you think are the most fundamental um abilities to cultivate so you have that kind of judgment? So like you know, where do where where do How do engineers become great engineers? Yes, that. Experience. Right? I've been programming for like 40 years. And like I've had to go I've gone through different company or different companies, different teams, different jobs, and like building stuff over like 2 or 3 years, and then after 2

21:18 or 3 years, we finished building it, we send it to the customer, and we find out all the stuff we did wrong. Yeah. Like that's that is a slow learning process. Yeah. Right? The cool thing about LLMs is that the people at Tailscale are doing this right now. I I know this, right? They're like, I want to build this thing. I don't know how. So, I'm going to try 10 different ways of building this thing. Yeah. Right? I'll ask Claude, like give me some ideas for how we might want to build a product like this. And it gives

21:40 you 10 ideas, and I'm like, okay, I'm going to open 10 windows, and I'm going to have Claude build me 10 things. Yeah. Right? And then I'm going to compare to see which one's better. And I just got 10 years of experience in 1 week. Wow. Right? And I've tried all the different things, and I know the pros and cons, and like this is how you become a good engineer is you try stuff, and you see what doesn't work and what does work. Like if you want to experiment with paper clips, I can now try like building

22:04 new kinds of paper clips in a virtual world Yeah. that I never like never could have gotten funding to even experiment with this like way out there method that everybody thinks is going to fail, right? So, when you make things super cheap, like yes, you're going to produce lots of garbage, Yeah. but you can finally do all these experiments and find out which things are surprisingly not garbage, right? One of the worst things about getting old is realizing I actually don't take as many

22:28 risks as I used to when I was 20 because now I know why things are going to fail, Yeah. right? And so, I used to assume they're going to fail, and then I don't do them. And then some startup person who's 20 doesn't know this, and they start a company is like, yeah, well, that would have failed 20 years ago, but the world's different now. That thing you thought was going to fail isn't going to fail, right? And you can find this out so much faster. Like that's how you gain this experience.

22:49 I I really appreciate your perspective. I want to go home and build like 50 pieces of software right away. Um and and thank you so much for sharing with us your perspective on, you know, what it means to be a developer in this world where now the tools are so different for us. Thank you. Yeah, thanks for being here. e.

AI is the cause of, and solution to, slop problems. Have AIs filter your PRs so humans get back to deciding what they want to exist.

AI was built on open source. Now it’s starting to return the favour: finding bugs, writing fixes, and optimizing code faster than humans can review the diffs.

At Web Summit, I sat down with the CEO of Cal.com to talk about what happens when AI agents and open-source communities start improving each other. The bottleneck may soon be code review, which is at least a problem we recognize. But there is a fix! You guessed it. It's AI.

If you'd rather read than watch, the full transcript is below.

Transcript

0:05 [music] Hi everyone. Welcome back from lunch. We're here to talk about open source and I really wanted to launch it with you Avery just because Tailscale is like you guys have embraced open source and and and you sort of have

0:28 come to sort of represent the adoption of open source for a major company like yourself. I was just if you could describe the moment right now in this sort of a genetic AI everyone going crazy staying up all night. How that affects and how you feel about that in your company? In my company Tailscale maybe ironically has always been a little bit of a late adopter and

0:50 we're intentionally a late adopter of AI. It doesn't mean we don't use AI but we use it really carefully and that's because Tailscale is a is a network security infrastructure project. It's used by Fortune 500 Fortune 50 companies and we can't like it would be violating their trust in us to take too many risks too quickly when their entire network security depends on our company, right? So we're being careful and I think there's a careful way to adopt AI that's really

1:13 really productive that I think people are overlooking. Now in a slight sort of plot twist is that at your company Tailscale is actually you were open source and then very recently two months ago to keep things a little bit spicy you did a major pivot. If you could describe that for us. Yeah, recently we went kind of viral when we went from being

1:35 historically an extremely open source company. We've always been huge like open source evangelists. We actually decided to go close source due to sort of security concerns because you know this stuff rapidly changes as we all know as all we sit here and discuss and for us we thought that the environment was a little bit turbulent to be able to safely operate in and much like Avery said, for us, we also have a duty to

1:59 protect our customers. And that led us to kind of perform a bit of a risk assessment of that. Maybe if I can just drill down a little bit on that. Like how did this how did this change happen? Because maybe it can help us sort of explore the open source experience right now. Like what happened? Was it Was it Was your customers are coming to you and saying say hey, wait a minute. We're giving you all this data. What's going on with it? What what what was the story behind

2:23 behind the change? familiar with like the changes recently you know, AI has become a lot more useful. It can now build software a lot more effectively than it could before. Like vibe coding has been around for years, but only in the last like 18 24 months did it become something which we can actually use for production ready cases. And just as AI gets better at building software, it gets better at breaking it. Yet the

2:47 problem is is AI isn't developed enough to the point that it produces the most consistent answers. You know, you get different answers to how many hours are in strawberry. You get different reviews to the same PR whether you plug it into Claude or um you know, a GPT model. And the problem for us is that with AI able to break software and find vulnerabilities,

3:10 we can't seem to find a single source of truth which can determine if an application is secure or not, which leads us to that kind of uncertain and turbulent um atmosphere. Every what I'm tempted to say how do you respond to that? I mean what what cuz he is describing a world that we all know, you know, hallucinations, bugs in the code and all that kind of stuff, but maybe it doesn't need to be that simple. It's not

3:33 a It's not a just sort of all in open open source or not. Yeah. Well, I mean, it is let's be honest with ourselves. Absolutely true that AI does a bunch of weird stuff. It's super gullible. I've described in the past as like, you know, hiring a really smart but not very worldly intern, right? That can code very fast, but it makes mistakes, right? It makes really scary mistakes. And then

3:56 you can clear its context and ask it to review its own code, and it'll it'll find those mistakes and say, "Wow, whoever wrote this, I don't know what they were what they were thinking." Right? And so, you you need to build systems around around these things to make them not not you know, to make them functional, right? And the open-source world is experiencing that in particular, right? Where, you know, a whole bunch of projects suddenly appearing cuz somebody vibe coded

4:19 something they thought was cool overnight. A whole bunch of people are like, "Oh, I can fix my favorite project by modifying it to do something. I'm going to send in a patch." But they haven't actually reviewed it, or they don't even really they've never coded before. They don't even understand what this patch does, and they're the maintainers of the project are just like, "Look, there's there's a hundred of these. I don't have time." So, I mean, some people talk about like the the the hundreds of these things,

4:41 thousands of these things. GitHub is just exploding with what, you know, some people say is like AI slop, you know? What There's just so much out there, and then so therefore, you know, engineers and tech you know, technical people are just stuck going through it all, through it all. And then then you're stuck in this moment of sort of like what what was this all good for? I mean, how do you address that kind of that that that sort of issue of this sort of like having just this just a massive surplus

5:04 of of of product out there? Yeah. I think I mean, one of the things I observed that I think people don't necessarily realize is that AI is at least as good at reviewing code as it is at producing code, right? And there's kind of in the open-source world, with this AI slop, there's this interesting like it's a you can think of it as a giant distributed system of people producing the slop. Anybody can do it

5:27 and upload it to your GitHub repository and make your life as a maintainer miserable, right? But you can also get an AI to review those incoming PRs and tell you whether they're slop or not, and sort of automatically disqualify them if they're slop. But that would require the maintainers to set up some stuff, and And like expensive time that hasn't really happened very consistently yet. People are only just getting

5:49 started with this kind of automated review process, right? But you can do very good code reviews. In in the the like extended version of this, you can have automatic code reviews of PRs coming into your project, and then the person who's bought uploaded the the code can actually read the code review and then fix the code, and it can do this in a few cycles until eventually it gives up because it's always going to be slop or it's actually come up with a pretty good change, right? And once

6:13 you've got a pretty good change, then you should bring the human in to say like, "Hey, this is a pretty good change. Do you want it or not based on the vision for your project, based on your design principles, based on these other things?" And that's the opposite of the kind of exhaustion that we've been creating today, right? So it's all about the processes that you put stuff in. Um Bailey, we had this moment of the the Mythos moment very recently.

6:35 Um I just wanted to for you to sort of like in in in the work that you do, this has there really been this uh we've heard about the Mythos and and about, you know, Firefox like where you said they discovered like these these these um these these fallibilities. I was just wondering if you could just sort of are you in agreement that this is definitely a moment where there are new sort of security dangers out there um

7:01 that there weren't before? Yeah, I mean Mythos scared us all a little bit when they found vulnerabilities across Firefox, FreeBSD, and you know, a ton of other things. And you know, as AI models improve, they're going to find better vulnerabilities. There are some extremely complex and intric- uh intricate like exploits that can be done. And some which are so hard that, you know, humans

7:24 may not be able to do that. And while Mythos like they created a monster, really. Um and they decided like let's gatekeep this and let's try and like roll it out in the the most cautious uh way. But like this is AI. Anybody can innovate in it. Um Um, know, we have frontier models coming from American companies and then suddenly China comes out with DeepSeek and it's literally

7:46 open source. You can just download a state-of-the-art model. Now, what happens when China comes out with a Mythos alternative? Does that mean that now every single person that can, you know, self-host an open-source AI model now has the ability to hack into almost any system on the planet including some of the most secure and open-source um projects that exist?

8:10 That's a very good question which I'm going to put to [laughter] Well, uh I think I'm actually more optimistic than that. I think the timing is really important, right? There's the attackers, the red teams, there's the defenders, the blue teams, and there's what I would call the arms dealers who are selling tools to both sides, right? And I maybe they don't want to think of themselves as that way, but they they have to, you know, they built this incredibly

8:31 powerful tool that can be used for good or it can be used for evil. And I respect Anthropic a lot with Mythos in particular. Like they've been holding back on availability of this thing probably for multiple reasons probably including for marketing reasons, but it it it finds real serious security holes in real critical infrastructure software. And if they'd given that to the bad guys first, those bad guys would be exploiting that software before we

8:53 have a chance to fix it and we would be in really big trouble, right? The fact that they didn't do that, the fact that they're giving us a chance to like, hey, we can fix our stuff first is like is a gift. And then the question is like, well, okay, as the models keep getting better and better, are they just going to find more and more obscure security holes and it's never going to end in this like sort of downward spiral into doom? Or are we actually just going to like is stuff just going to get more

9:15 secure and not really hard to attack? And I think it's actually more like the latter, right? Like most software in the world today has never had a security review ever, right? And even if software that's been security reviewed gets reviewed once or twice a year, gets pen tested occasionally, some of the PRs get security reviewed cuz they're considered to be sensitive and some Some don't, right? So like most code has just never

9:37 been looked at. And now you have these tools that can look at it constantly. Every single change you make can look investigate and and try to find one of these security holes. And so, I believe that if we all get on board with caring about security, which is which is a stretch. But, if we do, we can secure our software. Uh getting people on board to to to worry about security may involve

10:00 regulation or may involve governments. Is that sort of something are you um Baylor, are you reluctant to see the government taking too much of a a role in this or is that something that you think there might be a way for them to to there there is a place for for regulation here? There might be a way. I think, you know, in answer to what you said, I don't have access to me those. Something exists out there that may or may not be able to completely break my

10:23 code base. I don't have a good opportunity to defend against it. And I think much like how, you know, the government has created structured and regulated markets which enable, you know, like each party in the the stock market to be able to to trade in a fair way. Um there might be opportunity for, as long as they don't overreach, um to try and like equalize this stuff.

10:46 That might be through export controls, that might be through um you know, laws and acts. But, uh yeah, I think there's definitely a little bit of uh equalizing that needs to happen. Um at the opening of the of the Web Summit um uh Paddy Cosgrave sort of talked about like these two worlds that, you know, there's the open source, you know, there's those who say that open source is already won and it

11:09 is the way. And then there's the others who saying that, you know, the the the US frontier models have have have already won and it's the way. Um the reality is somewhere in between there probably. But, I was just sort of wondering um on the sort of big sort of big picture level, every where do you sort of see things right now? Well, I think I mean, I think all the models have their place just as in any economic system. There's going to be the premium product and there's going to be

11:32 like all the levels down to the mass-produced product, right? And I think the most expensive tokens from the most expensive models are going to get more and more expensive. That's what I think will happen, cuz there's a shortage of GPUs, and like if you want this good stuff, you're going to have to bid for it with everybody else, and the price is going to get eye-wateringly high. You think it's high right now? I think it's going to get even more high. But at the same time, the price of like the tokens that are like a year behind

11:55 that are going to are going to crater, right? And that's that's going to be really exciting. Like pick the tokens we want. I think from a security point of view, that's going to be like if you're building infrastructure software that everybody depends on, the Firefoxes, the FreeBSDs, the Linux, the Chrome of the world, and I guess the Tailscales of the world, you're going to have to pay a lot of money for like the best security defense, right? Uh what I think about open source software, like, you know, a

12:19 lot of the stuff that we're all going to vibe code for ourselves or for the five people on our team, right? You're never going to be able to afford to run the best frontier security software, for example, to review that. You have to do something else, right? My my proposal is don't put it on the public internet. It doesn't need to be there, and if it's not on the public internet, nobody can attack it, right? That that's the best approach unless you're building these

12:42 fundamental front-end-facing public products. Um Bailey, token maximization, you know, we hear these stories about, you know, like at the big corporations or people are told to spend as much as possible, you know, what is it spend? To use as many tokens as possible, which ultimately means spending. I'm I was just sort of wondering, how does a company like yours approach like, you know, token you know, just using up of tokens? Is it like

13:05 go crazy, or like how do you try I mean, how do you try to how do you navigate between like showing that you're utilizing these tools to their fullest potential, but at the same time, you know, staying afloat, you know? I mean Yeah, I'm a little bit against the whole like token maxing hype train. I think, you know, a lot of things are hype trains in this industry. Even to some degree, people want to be like open source just because it's the popular

13:27 thing or not because it isn't. And it's really about like using what's sensible for you. Now, I do completely agree with Avery on the whole thing about being a little bit sensible and a little bit cautious about AI adoption. Um, do I think that like everybody should be spending as many tokens writing as many lines of code as possible? No. We've known for years that like lines of code does not equal output. Um,

13:51 and so I think really you want people to be AI-enabled. I can do a better job with AI. Everybody on my team can do a better job with AI. But, you know, otherwise like let's keep it sensible. Right. Uh, Avery, you know, to people in the audience, um, you know, people that are, you know, coding, who are coding for a living or thinking about coding for a living, um, what is sort of your advice to them in terms of, you know, being in

14:17 this world where, you know, some people say AI and they think it's a magic wand and then something magically just appears and it's fit for purpose. Uh, there's still a a major role for, you know, the good old human being in the story, isn't there? Yeah. Well, rather than a magic wand, maybe I I can compare it to a genie, where you make a wish and you literally get what you ask for. Uh, and it turns out not to be what you wanted, right? Cuz that that's what

14:39 happens over and over again. It's like it creates that it is magical. I get it's magical in the most literal sense of like we don't even know how it works. Even the people making it don't know how it works. And it and it grants wishes, but it grants them in like strange ways that you might regret, right? Uh, and I but I think, you know, it's a tool just like just like a human is a tool. Humans are magical, right? You ask them to do something and you don't always get exactly what you wanted. And sometimes

15:01 it's good and sometimes it's bad. And we've got tens of thousands of years of building human society around the fact that humans are magical, right? But I think, you know, society exists to serve humans, right? It doesn't exist to serve computers in a like running mathematically calculations that simulate humans, right? And so it it's up to us to do what we want with the tools that we we

15:25 have, right? And these these tools can be used for anything, right? They can be used for attacking, they can be used for defending, but they can be used for like writing a bunch of slop code, but they can be used for defending and fixing slop code without humans having to be involved. So then we have to up-level ourselves to this like I'm going to think about the abstract. Like okay, I actually have this you know, this this patch came in and maybe a bot wrote it, maybe a human

15:48 wrote it, who knows, right? But like it works now. It's I've verified that it does what it's supposed to. I've even verified it doesn't have security holes. Is it what I want to exist in the world? That's what an open source maintainer will have to decide, right? And that's the that's the fun part of being an open source maintainer. That's why we get into it, right? When once we start, we realize it's like even before AI, it's like 90% like jerks writing

16:12 angry posts in your issue tracker or like triaging stuff or like slop that might or might not have been AI generated PRs, right? But like the 10% that was fun is like working with people, building something cool, and having someone send in something that makes your cool thing even more cool, right? And that can be the whole job because we can eliminate the tedious part using these tools. But it's the human's job to decide what they want to exist in the world. That's what we all

16:35 work and be are going to be able to do with these more advanced tools. Billy, do you have I mean what what is your perspective on that too? I mean in terms of like, you know, you may not be like the most senior, most most experienced like, you know, programmer, but suddenly these vibe coding tools like allow that person to to to explore ways that they they couldn't have done before. Yeah, I know for us we said we're never going to fire anybody

16:57 because of AI. I mean we never over hired the team in the first place, but now we just expect people to be able to to produce more. Um you know, like I said, this doesn't have to be going crazy at it and you know, uh overusing tokens just for the hell of it, but you know, it's it's a very exciting time to be in open source, to be in anything, really. The world is rapidly changing each week

17:21 to the next. We have more and more capability. And, you know, overall, this should be able to be a positive that we can all use just to build more things and, you know, invent. All right, cool. I think we have to we have to leave it there. Thank you very much, guys. That was really cool. Thanks a lot. Thank you. Thank you.

AI needs an open ‘Android’ to balance the vertically locked-in ‘iPhone’ of trillion-dollar AI companies.

Open-source AI is closing the gap with closed systems faster than many people expected. That changes the economics, the security model, and the assumptions behind a lot of Big Tech’s AI spending.

I joined a press conference at Web Summit to talk about what happens next: who benefits, which business models get squeezed, and how regulation should work when the most capable systems aren’t controlled by one company.

If you'd rather read than watch, the full transcript is below.

Transcript

0:01 Hi everybody. Who's going first? How was lunch? Ah, no lunch. So, I do want to say this is a really broad topic and so we are excited to talk about what you are excited to hear about. Uh so, if there's anybody with prompts, uh feel free. Otherwise, we'll do it like short monologues and then they'll go for some questions. Maybe maybe I'll kick things off. Sure.

0:25 Okay. So, um by way of background, I'm I'm Mike Conover, I'm the CEO and co-founder I yeah, sure. You guys can hear me. Project. I'm the CEO and co-founder of Brightwave. We built a a deep research system that is able to perform effectively the functions of an investigative journalist, but um in complex domains like finance. And

0:48 I'll just talk a little bit about some of the trends that we're seeing with respect to open source AI and um agents in particular. So, if you think about Who in the room has used this cloud code? Right? So, typically you're operating one to three agents at a time simultaneously and I think what we're going to see is massive parallelism, which is that

1:11 you're going to like if I want to understand how the war in Iran impacts commodity markets, there many dozens of different sub-themes and within each sub-theme there are sub-research topics and the ability to parallelize and scale um how many topics am I running down, how many analyses, how many code changes am I running at the same time um increases the number of agents that one person is going to control. And if

1:32 you look at OpenAI's symphony, uh this is a system for making that abstraction where I'm less aware of how many different agents are operating on behalf, um you know, clearer where I'm moving tickets through a linear board. Um This on top of sort of the task horizon link, so that's like the depth, how long can these agents function for independently,

1:54 um gives you this increasing breadth and depth surface area of total compute. And so from a like secular standpoint, that the what I expect is a massive increase in token volume owing to this parallelization increasing depth. Um we're going to see open source models become really competitive on a price and speed basis. So the same 64 GPUs that

2:18 you would use to run um a trillion parameter model, you can get 3x more tokens per second out of those same GPUs running isolated models on individual cards. Um and from a uh sort of switching cost standpoint, this is the last thing I'll say about it before we kind of hand it hand it off to you. Um it's not clear that the fungibility of

2:42 these resources is priced in. Like Anthropic had a moment and I I'm a huge fan of Anthropic, but as soon as a new frontier model comes out or as soon as a new open source model comes out that has some advantage, these all fulfill the chat completion API. And the switching costs are very, very low. Brightwave is all hot swappable so that you can use whatever model is best for the job. And I think that that structural force, that price and and

3:05 speed pressure is is not well appreciated with respect to like how much compute is going to increase through agent parallelism. I think we're going to do the three and I I'm here here to talk about it. All right. Um yeah, I'm Avery. I'm CEO and co-founder of Tailscale. Uh we make an AI connectivity and and governance framework called Aperture.

3:29 Um and like the my my picture of the like AI ecosystem right now, I I think it's very interesting how this is it's got to work out cuz that usually these big technology shifts, you'll have you know, the big winner and then the secondary one, right? And right now we're watching the clash of the Titans uh way up over our heads with the trillion-dollar valuations and the giant data centers and stuff. And the thing is they're all building what I would call like the iPhone of of AI, right? I

3:55 Anthropic, OpenAI, and Google are all building this like vertically integrated system where they're providing all the pieces and they want to lock you into their system so that you pay a lot of money for their tokens, right? I think the ecosystem needs the balancing Android of AI, which is like ecosystem-based, open source, everybody can contribute, and you can plug and play all the pieces yourself. And maybe you have to assemble the pieces

4:17 yourself, and maybe each of the pieces is is not not as beautifully machined and perfectly integrated as in your iPhone, but there's a space for that cuz there's always should be a space for like the super high-end premium thing and the like, you know, flexible thing that you can do at volume. And I think we're not seeing the second one as much yet, but there's a lot of pieces out there that can be assembled into that. And so, that's what at Tailscale we're most interested in

4:41 doing. We want to connect all these pieces together. We want to build that ecosystem. And I think that's where, you know, the lower-priced models, the various different kinds of harnesses, the really complicated agentic systems, the connectivity systems, sandboxes, there's a lot of room to experiment and put all those pieces together. Cool. I'm Bailey, co-founder and CEO of cal.com. cal.com is a scheduling

5:03 infrastructure platform. So, we power scheduling from anywhere from individuals to very large businesses. I have a slightly different perspective to to these guys. We were historically very open source. We were like the largest Next.js open source project. And recently we made a move to go close source due to security risks. So, obviously we're all aware that AI can build things even better, but they we

5:26 believe they can also break things even better. They've become really, really good at detecting vulnerabilities and things like that. And the problem is is AI is still somewhat in its infancy in the sense that they sometimes give inconsistent answers. And so, for us, you know, AI is never going to give the same answer to I don't know how many hours are in strawberry

5:49 or you know many things like that which we all know are quirks of AI and that also means that AI can't give you like a single source of truth as to is software secure. So you've all heard probably about Anthropic's Methuselah model and all these sort of things that are able to break things more and more. We don't have Methuselah none of us here but you know somebody does and you know AI isn't gate capped anybody can

6:12 innovate you know we have our US frontier models that that lead the way and then one day Deep Sea comes in and suddenly they can they can match that. What happens when China's now has a model which can rival Methuselah that means that you know are all of us under under attack because as I'm sure you've read Methuselah is breaking you know Firefox FreeBSD all these things

6:36 that we consider to have like a lot of eyes on them as open source and especially like FreeBSD is is an absolute sort of like staple of you know stability and so for us you know while going closed source isn't unto itself like a a solution it is an option we have on the table which we believe can reduce the risk.

7:01 We run six AI code vulnerability scanners all in parallel they all find different things and um you know that's a a scary thing for us. We also spoke to Hex Security one of the the big ones that that were like a YC company and they said uh open source is five to 10 times easier to hack than closed source and so for me

7:25 the reality becomes pretty clear that if I can make cower.com five to 10 times harder to hack um although that is not a complete and holistic solution to this it is definitely an option that I feel like we have to take to protect our customers. So, yeah, slightly different perspective to to these guys, but Can you please state your name and

7:50 affiliation? Jim Harris, Corporate Knights magazine. Uh just like we have hybrid cloud and multi-cloud, I think we'll have hybrid uh models, multi-cloud models. So, some things will use large language models, medium, very small, niche. And similarly, open source, closed source. So,

8:13 uh where we choose to put that load or query will depend on the context or nature of both the data, the security considerations, the cost, the speed, whether we use open or closed source. So, uh this is the vision uh that my clients are are taking, those I talked to, to optimize both cost, speed,

8:39 safety, all these considerations. Uh would you agree with this view that that's where we're going? While the you know, Open AI wants to lock you into their vertical stack, many companies uh don't want that. Just as AWS, Azure, Google wanted to lock you into their cloud. So,

9:03 Yeah. I think I I like my iPhone analogy uh for that one, right? Like every year a new iPhone comes out, they raise the price by a little bit. Uh everybody when I remember, I'm pretty old now, when the first iPhone came out in like 2007, it was like $800 USD, and everyone's like, "Oh my god, who's going to pay $800 for a phone?" Right? A bunch people did, right? And the price has gone up from there. And like, "You know what? That phone

9:25 is a perfectly fine phone even today, right? Nobody wants it because like we're willing to pay the premium for like a slightly better phone, right? And there are going to be people willing to pay a premium for these slightly better tokens because they believe it gives them a competitive advantage. It's going to come up a lot in the security world, right? Where if you have a slightly better model for finding security vulnerabilities, you have an incredibly big advantage over the person with the

9:47 second best model, right? For a lot of stuff though, that's not the case, right? We have a lot of communications technology. I have a watch that is more powerful than the phone from 2008, right? And all it does is tell me the weather really badly. Like it it can't even keep up with the weather updates, right? But it's okay, you know, I wear the watch and you know, it doesn't it doesn't cost me as much for my cellular subscription, right? So absolutely, it's

10:08 going to be a big market. There's like you're just buying these commodities of different values, right? And I think it, you know, there's as the price goes up, I I firmly believe the the cost of the most expensive tokens we have not seen the ceiling and won't be for a while. It's going to it's going to make your eyes bleed how expensive the most expensive tokens get, right? But the cheap tokens are going to get really cheap, right? And that's going to be both of those things are

10:30 going to be really exciting. Especially once like the VC money runs out. Um you know, it's it's like Uber. Uber was dirt cheap in San Francisco when it first came out and then now like the VC money dried up. So I think it's exactly what you say. Like the the expensive tokens will get even more expensive, but because of open source, like if you can, you know, self-host DeepSeek, uh and you can get it through any number of the inference providers, they're all

10:52 competing like on the way to the bottom. Um so yeah, I think you're also going to have that price discrepancy. And then there's also just what's the best fit for the job. Something that's often overlooked is like we look at, you know, benchmarks and overall intelligence scores, but say for instance for working with legal contracts, Claude, even though certain things outrank it on the intelligence index, Claude is better at that like long-form like understanding

11:15 the nuances of every word and and things like that. So I think for like AI to to become truly dominant, it needs to be, uh you know, versatile in terms of what provider you use. And we see that because there's like Vercel's AI SDK, you've got Open Router, and all these different sort of like switching things, where you can have the same, you know, core API function, and it will just

11:38 route it to whatever provider you want. [clears throat] All right. I'll check up there for a bit happier news. According to Forbes magazine back in 2025, 42% startup have failed in the Silicon Valley. And uh That seems low. Yeah. It could be more.

12:02 Maybe they're in the process of filing bankruptcy. Uh and I'm not sure you know about builder.ai. They ran through 450 million dollars, and they had to file for bankruptcy, too. You all of you are in your AIs building your AI companies. What is the moat? What competitive edge do you think companies have these days while they're building their product?

12:24 Because to us, it sounds like everybody's trying to build the next frontier model. But where do you think it's a competitive edge? Is the data set? Is it the privacy, security? We'd love to hear your feedback. I'll weigh in on that. Um My Yeah, thank you. Um So, I do think

12:46 Are you familiar with the bitter lesson? Like the idea of the bitter lesson that like effectively like more data and more total compute subsumes all bespoke like classical natural language processing a good example. There were a lot of methods for like vision models, a lot of methods for like detecting boundaries in images or like, you know, faces, and it's like none of those are relevant anymore. Um

13:07 and so that I think is kind of a a large inertial force, where stronger, more powerful models will subsume many of the things that we used to like wire up harnesses for. I think agent harnesses generally are a good example of something that will not exist in 18 months. Um I do think though that product judgement is like it's hard to describe what you want.

13:33 And I think like if you and I were to vibe code workout app we don't necessarily like know how to articulate like all what are all of the things that a person would need from a tool like that or cal.com. Like I imagine that there are a lot of decisions that you've made that like if I was like I I need calendaring software my ability to articulate that and create something a delightful experience would

13:56 be low and I I do think that you know it's like taste is one of these things which is like how quickly can you gather information and make judgements and articulate that to an AI system. Um I don't know that there's going to be one monolithic interface that subsumes all product. Um and then I think integrations like there's a long tail I

14:18 would say that there's a long tail of integrations and capabilities that are not in the call it the blast path of the meteor um that are really important for things like law or networking that um just will not ever be on the like cut list for the foundation labs. So it sounds like you're saying what customer wants uh validating

14:43 Yeah and just like yeah being like being so tight um like in the meta of like what is actually important um and then just creating a really delightful and easy to use product that reflects deep expertise in the subject matter. Um I don't think the foundation labs I mean they maybe they have like they become the one app and they have

15:06 many many different verticals but uh it's unclear that that will be the the business model. I think I think feedback loops uh is what it comes down to like almost all like everything about startups comes down to feedback loops, right? You know the the famous advice to startups is like get out of the building, go talk to a customer, or you're going to build the wrong thing. Right? And and AIs, when they have really good feedback, can produce really good output. And then,

15:31 the quality of when the quality of the feedback goes down, the quality of the output goes down. Anthropic did a project a few months ago where they implemented a full C++ compiler by providing it with a test suite of like 50,000 tests. And they just said like, go. And they spent like I think a million dollars in tokens over a weekend, and it produced this perfect compiler that passed all the tests. And they're like, that sounds very impressive, but like who wrote 50,000

15:53 tests of a C++ compiler? That is the optimal case for this kind of thing, and almost none of us are starting from that kind of perfect specification, right? Even calen- calendars, right? It sounds so simple. It's like, look, I want to display a list of my appointments. How hard can it be? Right? As soon as you as soon as you put it in front of a person, you'll find out how hard it can be. Right? Networking. I like to brag

16:17 that if you ask Claude, like, hey, can I can you make me a clone of Tailscale? It actually tries to talk you out of it. Cuz it knows that networking is really hard. And it'll give you a list of reasons why you shouldn't try to clone Tailscale. By the way, you should just go open fork their open source repository. I can add a feature to it if you want. Right? But like that's the kind of stuff that is that is a moat, right? It's like it took it takes a long time to test

16:39 networking software cuz you need like 100 different devices that it needs to be compatible with, and you can't just pretend to test against it. You actually really physically need those devices to be there to test against. Claude can't set that up for you, at least not right now. This may all just be wishful thinking. It's possible. Any other questions? I guess if I gave Open Claude credit

17:03 card number, it could like have some devices shipped, and then pay somebody to set them up in a data center. Okay, we can then wrap up. Appreciate your time today, folks. Thank you so much for your time. Thanks for attending. Thanks everybody.

AI is the universal translator that finally connects all your tools... into a hairball.

Most companies don’t have a tool shortage. They have too many tools, too many handoffs, and too many places where context gets lost.

I joined a panel at Web Summit to talk about when consolidation actually reduces work, where AI helps, and how to avoid replacing ten mediocre tools with one very large mediocre tool.

If you'd rather read than watch, the full transcript is below.

Transcript

0:00 Hello everybody. Good to see you. We're going to have a great chat. We have some super smart people here, so it's going to be fun. We're going to start off with a bit of a joke. I mean, like, you know, I've always kind of laughed at the idea of a tech stack, a company having a tech stack. I think they more often have a blob. Don't you agree? I I would say it's not even a joke. It's uh it's a blob or a morass or like a

0:22 collection. Um like people like the bigger the enterprise, the more programs they accumulate. I think we don't realize like I think the average company buys about one piece of software per employee. I've I've seen the enterprise data and it's like 1,000 or 1,500 different apps that are running in the various places. There's of course the official ones and

0:46 the unofficial ones. Anyways, we're going to talk about AI and consolidating the tech stack. Are we really consolidating it? Are we just massively changing it? When? Well, I think, look, this is happening. This is happening. Things are different now. I mean, you see Salesforce going headless, and you also see that when it comes to AI applications, a lot of the lighter applications getting absorbed into that. So, from where we're sitting,

1:09 what we're seeing is really two things. One, we believe the lighter applications are going to essentially just uh completely taken over by AI. However, the heavier stuff, they're probably going to go uh headless, similar to Salesforce. Now, the fundamental change that we're seeing is um AI, like agentic AI like GenSpark, like my company, [laughter]

1:30 is going to come out and become the new user of software, of SaaS. So, that is the fundamental change. And with that, we actually don't see the usage of SaaS going down. We see it blowing up. Because AI is going to be a lot faster than human beings to use all the SaaS products. But again, the lighter ones are going to become free, are going to

1:53 become just available and customized anywhere, uh essentially in all of these, you know, Agenty AI platforms. David, your thoughts? I think my take is it's a lot like the early days of the internet where when the internet first came out, there were hundreds of thousands of websites that came out for every single topic that you could think of, but it started to slowly consolidate over time cuz you can't remember all the products that you're

2:15 utilizing, all the websites that you want to go to, and you started to go and say like, "Hey, a super center like Amazon makes sense." You go to three different news sites instead of the 500 news sites that are out there, and you're seeing that level of consolidation. I think we are in that Cambrian explosion of AI tools. You don't want to stop people from using the tools that they need, that they want to experiment with, but I think it is going to be really it's going to happen in very short order that you're going to

2:37 see a good level of consolidation to say, "Okay, this is my system of record for meetings. This is my system of record for sales. This is my system of record for CRM, etc." Like, that will happen because you can't have 500 solutions on top of mind, and then you're starting to look at your bill, and you're going to start to realize like, "Oh, why am I spending so much on so many variable things?" It is super interesting because you mentioned already Salesforce came out

3:00 with an announcement they're going headless, which is very cool, actually. Wait, does everyone know what that means? So, if it's not headless, you're going in like a caveman to the user interface, and you're typing in your stuff, right? This is a customer record. I talked to the salesperson, blah blah blah, type it all down. Boring, slow, mistake-prone. Headless, I tell my agent to do it. My agent goes and connects via APIs to

3:24 Salesforce, and just does it for me, which is really cool. And I even talked to the the Salesforce PR people, and you can use open claw with if if if if your [laughter] if your enterprise people let you. Avery, your thoughts on this whole consolidation? Well, I think I mean, any any major technological shift, and this might be the biggest technological shift anyone has ever lived through, right? There's

3:47 going to be there's going to be winners, there's going to be companies that go into decline, and there's going to be a whole bunch of brand new stuff. Uh when you look at big enterprises, they're not going to change very fast. They They when they say fast, you think as fast as you might change. But what they mean is like in the next 10 years, we might slightly shift the collection of software that we're running. The really big thing that I think will change though is like AI is is this universal translator. We actually finally built it

4:12 the universal translator from Star Trek, like not even kidding, right? But it not only can translate between any human languages, it can translate between any software, right? Which is like the ultimate goal of so many IT projects is like can I please connect my Salesforce into my Snowflake database and then give me a dashboard that just tells me this thing that I couldn't get

4:34 from like one of those two things on their own. And it is really hard to do that and huge teams at enterprises have been like tasked to do this and failed. And now you can do it in 10 minutes with this AI. And that that is a huge difference, but it doesn't mean the set of software is going to get less. I think it means all of a sudden you can buy even more software and then connect it all together in this like incredible hairball that can only be understood by

4:56 a computer. Yeah. That's kind of hilarious, right? I mean because there's this yin and yang of software, right? Which is like somebody builds a tool and it's freakishly awesome and it does one thing incredibly well and then they think, "Huh, what else could it do?" And you know, 5 years later you have a suite and you've got a platform and you've got an app ecosystem, right? And so you have this continual expansion and then

5:18 everybody's like, "Oh, it's there's such a general tool. I just need this one thing that does one thing right." And so we have this cycle going on and on. We talked about the SaaS apocalypse when we were chatting earlier and that's affected valuations, right? But you guys are mostly saying that SaaS apocalypse is overrated. Your thoughts, David? Uh it's overrated. If you look at the

5:40 earnings, like DataDog actually was hit by the SaaS apocalypse, stock went down and then all of a sudden they came out and said like, "Actually, our growth has reaccelerated because AI is actually driving revenue for us and it's growing." Now, there's a subset of SaaS companies that have actually been hit where there's companies like uh HubSpot would be one example where they're still growing 20%, which is huge for a $3 billion company. They're growing 20%.

6:03 So, it's not SaaS apocalypse like the money is disappearing. It's more along the lines of your growth rate is slowing down and then the public markets are giving you a lower valuation. I wish my growth rate was slowing down from 20 to 20%. What kind of percentage? So, I actually have a different take on that, John. So, Go for it. I mean, look, if you take a look at the the trajectory of Salesforce stock, I mean, today I just checked, the market

6:26 cap is $136 billion, right? I remember checking this 2 years ago at the peak of the pandemic it was over $450 billion. That was $300 billion gone. Why? I mean, those are facts. So, basically, to what you gents just said earlier, like um I think what David said is um companies could still grow if they could

6:50 create a new value and taking advantage of AI, but if you don't do that and if you just sit there still, then of course, the value is going to disappear because the fundamental change is happening right now in a way that um the the we've all been a victim, in my view, to your description, John. To so many tools, so many SaaS. Everybody said like Software victim. Yeah, yeah, like we're we're so busy.

7:12 Like the world world was so tool-centric. Each of us would have to learn how to use 20, 30, 40 tools. We used to get you know, people used to get certified for being good at Excel. Yep. That world, in in our view, is long gone. Okay? You don't have to get certified for being good at using the software. You just got to be good at understanding what you want and then command your AI agent. And then your AI

7:36 agent would go and learn how to use tools. And and you don't even have to switch contacts, switch tools, like copy and pasting the same piece of business context you from meeting notes to slack to email to presentation. None of that. You tell your Jasper I want this meeting note turn into a prototype and boom it's done. Human beings on the two ends. Show of hands here who who has a AI

7:58 agent? At least one. Yeah, we got some. Who has more than one? More than five? More than 10? Wow, I think we got them there. Oh no, there was a hand at the very back there. Wow, you are super user, ma'am. Uh any comeback there, David? Oh, in terms of meeting notes? Yeah. In terms [laughter]

8:19 Let's talk about the SAS apocalypse. As long as read AI is one of those meeting notes, I'm okay with it. [laughter] Now, I I think people do experiment. Uh people get preferences. There are different use cases for different types of products. Yes. Um what you're just like you talked about, there's more features that we're adding. We're doing more than just meetings. We're doing search. We've got a digital assistant. We've got a digital twin. When you add those things together though, at some point you will go and

8:43 say this is good enough or this is better than the rest and I don't need to pay for all those. Yeah. And especially when you think about the platform-based solutions like uh with Microsoft Teams, with Zoom companion AI, with Google Gemini. Those are all great solutions on a standalone basis, but there is a real narrative to go and say how do you work across the different platforms? And that's what's been important with Open Comp OS like you're able to go in and work across everything. And I think there is a

9:05 scenario where you want a solution that works across everything or you've got to buy three, four, five, six different solutions. Cool. For the panelists on the stage, you have an agent, right? How many agents? Just a number. Uh I mean on Jasper we have One number. How many? 30 plus. 30 Oh, wow. Uh we've got one for 5 million uh MAUs

9:28 on a monthly basis. Uh how many agents, Avery? I just have one agent but Oh, you're agent poor. Well, I'm not impressed. Cuz it spins up agents dynamically. Are you even qualified [laughter] I'm just bugging you. Avery, um when talked about this value in the quote-unquote SaaS apocalypse, going other places, where's the value going?

9:51 Where's the value going? So, I saw a survey a couple months ago from CIOs and CISOs and they said like for the first time in their memory, in the build versus buy decision, they're actually buying less products this year. And in fact, they're going to be planning to spend less money on licenses for SaaS products and starting to build more things internally. And that's that's never happened in the history of of of software.

10:15 So, those hundreds of billions of dollars are going to OpenAI and they're going to Anthropic and they're going to Google maybe and other places like that, token costs. Well, I would say to what Avery just said, we've heard similar things, but that is really largely because what was impossible now is possible. Yep. Building out everything for your company, it was not possible. You have

10:38 to buy other people's solutions, but now, you know, for example, with just by AI developer, you don't have to know how to code. You just need to know what you need and one of the And the beauty of it is is you can build exactly what you need. Exactly. And I don't have to customize my business processes to this Salesforce giant or something like that. I can build exactly what I need. Okay, guys, 12 minutes.

10:59 20% of what Salesforce what Salesforce built anyways. Exactly. you pay for the whole thing? Yeah, people Right. People underestimate the the cost of procuring software. It's not just the dollars of the software, it's that when you buy it, it's actually not exactly what you want, right? You are you're you're satisfying yourself with something that's almost right because it's so much less expensive than building the thing

11:21 that is exactly what you want, right? And there's there's a marketing book that I read, it's from the 1980s and it was talking about ketchup. Right? And at one point, they you know, the the companies making ketchup went into a survey of all the consumers and they found the perfect consumer and they said this is the average ketchup that everybody's going to like the best and they sold that ketchup and nobody liked it. Right? And eventually they figured

11:44 out it's like, oh, you know what? There's a whole bunch of different people like whole that like a whole bunch of different things, right? And now if you go to the grocery store there's like 17 kinds of ketchup and if you go to the pharmacy there's like an entire aisle of toothpaste even though there's only two manufacturers, right? Because everybody wants a slightly different kind of toothpaste. And we do not have that in well, I don't meeting recording apps, right? Like there's a lot of meeting

12:06 recording apps now but there's still not the one that does exactly what I want every single time. Okay, going to transition a little bit here and we're going to talk about the AI native stack and what that looks like and we'll start over here. What does the AI native stack look like? What are we What are we going to move into? So, our viewpoint uh John is consistent with what I just said like the most the

12:29 natural most human way to work is us having one uh point to to to joke, one one thing to work with, one um system interact with, but we get everything done. In other words, you tell that one system what you want to do, it figures out the right place to go, the right software to update, the right thing to do. Exactly. Sounds like magic. Yeah, so today how I work is I text my Jasper Claw every morning, what's going

12:54 on in my inbox? And actually it sends me a morning brief. Like oh, these emails Yeah, you you you exactly. Yeah, you should pay attention. All the others I'll just triage to all your sub orders, right? Like I get off the airplane, I ask my Jasper Claw, okay, where should I go to eat dinner? Like it just figures out where [laughter] I'm I'm staying. need to exist anymore? It could be just your Claw. It's just your agent.

13:17 David, I'll go to you. Uh what does the AI native stack look like? I think it looks a lot like what we're doing now. Um it's going to be within your existing workflows. It's going to be subtle changes that you see. I don't think it's going to be a new portal, a bunch of new software that you're going in logging into every single day. It's going to be, "Hey, you get an email. Your agent pings you and says, 'Hey, here are 10 things that you should do today.'" You decide what 10 things you want to do. You go and you say, "Go do

13:40 that. Go deploy it." You don't even see what's happening on the back end because the agent is negotiating with other agents and other systems of record. I think that's where we're going to go. I don't think it's going to be like this brand new solution. I don't even think the ChatGPT interface, the Claude interface is the long-term interface on how we want to interact. If you think about how we consume content today, it's no longer long-form. It's no longer going to movie. It's on a screen for

14:03 15-second bites that you're looking at and you're swiping up to the next thing. And I think decision-making is going to be a lot like that. And a lot of people say like, "That's that's MARGINALIZING THE HUMAN." OH, WON'T THAT result in a lot of awful decisions? I made a decision on 15 seconds of attention. But it's going to end up with certain outcomes and it's going to be optimized outcomes. So, I'll give you the best possible example. Meta today, all you do is put a pixel on your site, you throw

14:26 variations of creative, and you could spend $4 million and get $8 million at the end of the month, and you don't even know how it works, but it's happening. I think that's the way we're going to go. It's like, "I have a problem. I want a solution against this." I decide between three options, and the agent goes and does that. Ignorance is a business advantage. Okay, I got it. Um Avery, going to you. Uh your input on what the AI native stack looks like.

14:49 So, I think, you know, I I I sometimes I'm a bit of a dreamer. I think this can go in a lot of different directions. With every major technological shift, we have a real opportunity to flush all of our mistakes from the last generation down the toilet. [laughter] You're a dreamer for sure. You have made You have made so many mistakes in the last generation, right? The best thing about AI-centric systems is that they're not really AI-centric

15:11 cuz the AI doesn't care, right? The AI will do what it's told, right? Which means you can finally have human-centric systems cuz the systems we've been living in for the last 15 years are not human-centric. They're like super weird mega corporations running our lives because they made all the software that mediates every single thing we do, right? And now they're not They don't have to be the only ones who can make

15:33 the software. I can make the software that does what I want it to do using the power of an agent that yeah, there's an LLM in the sky that like understands how to do stuff, right? But it's a fundamentally different thing. Like when someone talks about building an AI agent, if it's somebody else's AI agent and if you go to a website and you get a chatbot instead of a human for doing support, it's annoying. If you send your chatbot to the website to do the job,

15:58 it's exciting and empowering, right? And that is the difference. Like I want the ability to make a computer and make the world do what I want, not something that's going to present me with 10,000 options and make me make a decision every 15 seconds. Like those are not important decisions, right? Just I want food and I'm you know, ask me what kind of food I want and then tell me where to go.

16:18 I've wondered for a while now why websites aren't basically, you know, a prompt basically or a prompt insertion text box basically. Like what do you want and here you go, right? Like basically that. But is that the interface of the future? Is the interface of a future a text box? Let's start here. Well, look, I think that that that should be viewed as an option. Meaning

16:42 that when you don't want to spend time to browse through things, you don't have to. I think that option would exist. I I and but I don't think it's just a text box. Today, I could speak to my Jasper agent. I talk to it. Like it is a lot more humane than just typing on a laptop or you know, even typing your phone. I speak to my Jasper cloud through WhatsApp's voice.

17:05 That is how work should be done. That we all should feel like a Jensen Huang or a Jamie Dimon from JP Morgan. You don't see them typing hard to get work done. You see them calling up their associates and they run around for you, right? Like we can all have a fleet of Goldman Sachs analysts running our pocket doing work for us while we focus on the strategic and creative stuff. That's our vision. That's what we all deserve. You were the guy with 30 agents, right?

17:27 Yes. [laughter] But I don't have to manage 30 agents. I just manage one. I have noticed, by the way, that when you do manage agents, sometimes they have conflicts and it's like managing people. It's like that that agent didn't give me what I wanted and that agent didn't give me It's it's it's interesting. It's not perfect. Uh my agents make mistakes and I apologize to our investors because of that.

17:49 I'm telling you it's real. It happened. Crossed emails, but I would say still the mentality is anything new, there are going to be mistakes, there are going to be risks. I decided I'm just going to eat the mistakes myself and just take a full advantage of it. I agree. I agree. Absolutely. David, your thoughts? User interface of the future, is it us just talking? I don't think so cuz

18:13 you don't know what you don't know. So right now, what we've done is we've gone in and said like, "Hey, you've got all your data, you can create a digital twin, you can ask it questions." But it doesn't have enough context about the entire business. If I think what something is a right answer to a client and it says, "Hey, you close the deal at a 60% rate when you answer it this way." I want the knowledge from my rest of my team where it's like, "Hey David, if you actually answered it this way, your close rate goes up to 90%." You need to

18:36 have that information accessible. But I don't know to ask that. I don't know to go and say like, "Hey, did Charlie's call on Tuesday at 4:00 go well? What were the questions?" You want the AI to actually go in and contextualize and say, "Hey, you're a salesperson." And this goes to your kind of point on the meeting notes. You've got a situation where it's like you want the AI and this is what we're doing is you can go and say, "This is a sales call. If it is a sales call, these are the takeaways that I want. This is

18:58 how I want to format it and I want to look beyond just one person's data because I need to get the entire data set." And the best example I have is we've got a fan customer where they've got a product team in Tokyo, LA, and London. And right now, each one of those teams don't talk with each other because they don't speak the same language, they're in different time zones, but they're interviewing customers. Now, what they're doing is actually going and

19:21 saying, "Hey, I'm using AI to cross all the customer interviews. And if someone says, 'I don't like this feature,' that might be it. Like, only one person in my area said that." Well, did you know that 60% of people in Japan said that? Did you know that 75% of people in London said that? I don't know. The prompt is there to ask that question. But now, the AI is going to go and say like, "This is actually a pretty big deal. This should be number one on your product queue." Interesting. Avery, is there sometimes some value in doing things the hard way?

19:45 I've done a lot of data analysis in my life. Sometimes as an analyst, sometimes as a journalist, and sometimes getting the answer from the genie out of the bottle is amazing and incredible. Sometimes I understand the problem better, and I understand the solution better if I do it the hard way and analyze it myself from the baseline data. Thoughts? Yeah, I think I mean, in in

20:07 organizational theory, we have this idea of a core competency, right? Like, what does your business do that is like the fundamental thing that you do better than everybody else? And you should outsource everything that's not your core competency, and you should keep your core competency. And I think that is that is a value we can apply in the AI world as well, right? Like, if you're really, really, really good at something, you should identify what that thing is, and you should not hand it to

20:29 the AI agent to do for you, right? Uh but everything else that like, "Look, I'm not that good at this thing. I'm kind of average at this thing, or maybe I'm just like maybe I'm above average, but it's still not the best thing I'm at." Like, delegate that. And that's the same rule. Like, you know, I'm a CEO of a company. We have 300 employees, right? When I delegate stuff, do I get exactly what I wanted? Like, no. No, that's not how humans work, right?

20:52 They build what they wanted, and hopefully it's more or less what you wanted. But when I go and do something specifically myself, I don't have very much time. I have 1/300 of the time at the company. I better choose very, very carefully which thing absolutely positively has to be done my way. And there's only a very small number of things, but when I do those things, it's better. I think that's a great answer. I absolutely love that. Okay, we have just

21:14 about just less than 2 minutes left. Each of you take a third of that time, please. And the question is, we've all talked about, "Hey, I can build exactly the software I want. I can design my information ecosystem around my desires. I can be very egocentric about all this all this stuff." What about maintenance? What about maintaining code? Let's start here. You got like 30 seconds and go

21:36 down the row. I mean, look, so um the way we're entering in this token economy is it's not going to be overnight, but things are happening fast. Mhm. So, I would say my recommendation is on the one hand, obviously, we still got to get the day-to-day done, but on the other hand, I'd say hold yourself back when you wanted to just apply yourself, but really allow AI to get onto your tasks first for you. Because with that,

22:02 you can get out of the busy work. You can actually have time to think. And you can actually, you know, uh focus on the rather strategic things instead of just hustling. Cool. That is how we Yeah. Cool. David. I think the big thing is you've got to pick the system of records and then build around that with the customization. So, pick what you are going to use and build on top of. Do some customization there. Don't do it yourself. Don't do it something where

22:24 it's like you've got one person on your team working on it. You've got to pick some platform, some AI solution to go in and say, "This is a certified This is an approved process." So, it scales out. A neighbor. All right. I'll say maintenance, when you think about it, digital stuff is the only stuff that doesn't degrade just by existing, right? The out The real world just like falls apart if you leave it sitting there. Software for some reason

22:49 stops working after a while. What is that reason? It's because the platform it's running on when it changed out from under you. I think the neatest thing about AI is you can build your own thing and just tell it, "Please stop changing this." And it won't change. And you don't need to maintain it necessarily. other answer is that code is disposable. And if it if it doesn't suit me anymore, I don't need to maintain it. I'll just generate some new. Anyways, thank you so much for being here. Thank you so much,

23:11 guys. That was super awesome and fun. I hope you enjoyed it. Thanks, everybody.

Treat AI agents like a naive, high-energy intern.

As AI takes off, it’s bringing new security challenges—especially around how models are built and accessed. In this talk, I sat with Cohere Co-founder Ivan Zhang dig into the often-overlooked networking layer behind AI and why secure, reliable connections are becoming essential for enterprise-ready systems.

If you'd rather read than watch, the full transcript is below.

Transcript

0:01 Good morning everybody. We've got two Canadian unicorns here. So, one recently minted, right? Um, Coher is enterprise AI tail scale counts many of the larger better known AI companies among its users including Coher and I think all of us want to know what we can learn from you right this moment that we're meeting how you're

0:23 thinking about security going into it. Uh so you know what we've seen in the past year of enterprise AI adoption is everyone is tired of the PC's right uh when our customers try to move to production uh oftent times they're faced with issues like cost like governance where is their data going um and also you know the stringent security and

0:48 privacy uh regulations that they're facing under right and so for us it's very interesting to see that hey to get these PLC's into production it needs to solve all these host of challenges which is what we're focused on today. Uh building a you know our north or agentic platform that's secure by default uh and customers can safely experiment and also play with AI and also take that into production in a safe and secure uh

1:15 manner. So my my observation is that the AI world is moving really fast and every couple weeks there's some new trend some new change that everybody wants to jump on. Uh the latest trend is a really important one is they want to connect their AIs to things. They want to connect it to their company data to databases to APIs and stuff like that. And the way people approach stuff like that in the AI world because everyone's

1:39 in a hurry is they do it in a careless sort of we'll fix it later way. Uh Tails scale is a networking company. uh we run into a lot of people who need to do networking because networking is how you connect your AIs to stuff and they don't want to think about it because they're in a hurry and they're worried about their AI thing. And so the first thing that comes to mind is why don't I take my private data and put it out on the public internet so one of the AI engines

2:01 can then access it and that's what they do and then they come to us and it's like okay so we did that it's working should I fix it and the answer is is yes probably you should fix it which brings me to the point that it feels like that networking layer somehow that conversation gets overlooked in these conversations about AI security am I wrong is that you're I mean yeah I mean pretty much like exactly what Avery

2:25 uh you know the only real security is network isolation right you could try to harden your systems as much as you want um but really what prevents data or bytes from leaving your system is actual network isolation uh how why we really really like working with tails scale for example is you know our IT organization is able to manage safely manage our deployment of tail scale make an internal network available for our

2:49 engineers to then experiment with different MCP servers in a you know isolated environment uh and so they could safely you know play with this technology and evolve their thinking in the tail net. Yeah. AI workloads are typically distributed right across different environments that brings its own challenges. Do you mind to unpack that a bit too just why that's a bit different necessarily? Uh we run

3:13 into this a lot. Customers come to us especially in the AI world. we sort of uh accidentally became the back plane that all the AI companies are using for their data just because nobody else was doing it and we sort of found out after the game was already played that we had sort of won it. Uh it was kind of funny our website didn't have anything about AI on it until AI companies told us like hey you won how did you do that and

3:35 we're like it's a sick product. It's a really really sick product. Um, so that's why Yeah. But I I think like the the I wouldn't say networking is overlooked exactly in the AI world. Everybody knows they need it. It's just that security is like last on their list of priorities because they need they want to be first. Everybody needs to be first because everything's moving so

3:56 fast. So they like come and they look at security later. And I can't say that's even the wrong approach even though it always gets you into trouble eventually. But that's eventually, right? you also need short-term results so that somebody doesn't beat you in the short term. Uh so we tried to like think about a world where like hey shouldn't the easiest thing to do also be the safest thing to do right and I think coher is doing the

4:19 same thing they're like look this is the right way to roll out AI in your company you don't have to do it the wrong way it's actually faster to do it the right way and that's the path to like helping people build better systems. Yeah, we we take a lot of inspiration from how you guys have built um tails tail scale obviously uh the fact that the end business user doesn't have to really think about security and you know them using the product is already secure by

4:42 default that's how we approach how we design agents how we want our you know business users to actually use our products. Yeah, edge brings convenience it also brings more exposure. Can you also unpack part of what makes that hard and what's what you see working? Did you say edge? Yes. Ah, okay. Yeah. I

5:06 think well people want their AIs to be hosted in as many places as possible because you don't necessarily well in particular there's a shortage of GPUs. It's going to look it looks like there's going to continue to be a shortage of GPUs for a pretty long time. Uh, and so if you're a company that needs to use AI, you need to find GPUs somewhere at a reasonable price. And the place where GPUs are the most reasonably priced is probably not your favorite cloud

5:29 provider because somebody else already went there and bought them all. Um, and so you need to be able to connect these GPUs that you find at a reasonable price somewhere else to the rest of your system that is not the AI part. It's just the regular systems part. And that connectivity layer is how tails scale got dragged into this because nobody was building these these uh so-called multicloud connectivity environments. The advice right up until AI caught on

5:53 is like don't do multicloud. It just makes everything complicated for no reason. Which is also good advice. It's just now there's a reason so you have to make everything complicated. So now you have to solve a problem that you didn't want to have to deal with. Yeah, thank you for that. That's the thing that I think is probably most unclear right now. Um how to approach secure. Speaking of things unclear, how to approach security challenges posed by AI agents I feel like is keeping people up at night.

6:16 It's probably also the hottest topic going. It was, you know, many conferences this spring including RSA. It's becoming a key theme, right? So what would you say to companies trying to think about this? Um so what we saw in the last year where folks are trying to piece together these PC's you know with different AI models and they're trying to build these rag pipelines and agents is yeah you can get the initial example to work maybe you

6:41 index some data that's you know you got an export of your notion or something right um but to take that into production right you have to think about things like identity providers right like how are the users going to give the agents permission and authorization uh to actually access these data also you know where you're deploying ing it. Do you even have network access to such tools, to such data sources? Um, you

7:04 know, are you able to get the telemetry you need to actually debug when agents go wrong? So, another thing that was interesting is, you know, some of our customers, their downstream systems, their internal tools, internal APIs, they built it for a human level of usage, right? But as soon as they deployed agents to it, immediately they saw, oh, these things are these things are starting to die. like they're they're starting to come down because

7:27 agents can hit APIs and systems 10x 100x more than humans can within the minute, right? Um so yeah, I mean there's there's a ton of these challenges that uh you know I'm proud that we've helped some of our customers solve and and yeah, as a security person, I find AI security just sort of well, it's really it's fun and exciting because it's so insane the just the whole way we do it,

7:52 right? This is an entirely new way to use a computer. And the best analogy for it is like it's a really high energy intern that's super naive that you've hired at your company. And like, okay, this intern has a, you know, can run around and talks to everybody and accesses all your systems. What are you going to give them access to knowing that this intern will run away and hand all the

8:16 information to your competitors because that's what they thought you said to do? And it's exactly like that, right? So when you connect these LLMs to one of your internal databases with private stuff, that part is not really the problem. People kind of overrate the problem of like, oh, they're going to train on my data and it's going to leak out like that. Like most of that is solved already. The real problem is like it reads the data out of your database

8:39 and while it's reading it, it sort of gets what's the word? Like subconsciously influenced like one of the things it reads out of a database of customers in like someone's description field. it might interpret that as instructions and then it runs off and applies those instructions in some other thing that you gave it access to and it definitely should not have done that and then it just causes this chain reaction of craziness and so you have to think of

9:03 it like how would you manage human threats inside your company because it's the kind of mistakes that humans make and I don't mean to say that AI are human but they do make a lot of very human mistakes right they have trouble with arithmetic which computers have never had trouble with before we've invented at very great expense computers that have trouble with arithmetic, but they also have trouble with simple following of instructions without

9:26 getting distracted, right? And it's these distractions that lead to all sorts of problems. So, one thing you can do is insert auditing layers when whenever this, you know, you wouldn't want to do this with a person, but you do it sometimes with interns, right? Where any anytime they want to do something, maybe you better just like double check what it is that they're doing before they're allowed to like, you know, carry computers out of the building or whatever, right? If you do

9:49 that, if you build a system like that, then you can keep it under control while we're all doing these wild experiments and moving as fast as we can. I I'll plus one to that intern uh example because that that's the exact exactly the right way to think about how to make these systems effective as well, not just secure, right? You know, if your intern had to ask you every time they want to do an operation for permission, they're not going to be effective,

10:11 right? And what's more important is actually giving them an environment where they can safely explore and play around. You know, you have the proper policies and governance policies uh to actually let them explore within the environment. Uh you know, those are the use cases where we see AI agents actually create value rather than become a nuisance. So looking across the landscape, what

10:34 are some of the best practices here that you think you could share with other founders and startups in the room? Don't put your private API servers on the public internet. Don't do that. I know you're not going to listen to me, but still don't. I am going to laugh at

10:56 you. That's all. That's all. That's all. Oh, come on. No, I guess that's not all. The other thing you should do is you you want to make sure to give the thing access to what it needs access to and not access to what it doesn't need access to. And while you're experimenting, just like with people, right, you give them readon access first and then you give them read write access later once you've built up

11:19 like some structure on on what their job should be, right? And I find people like GitHub just announced this thing I think last week or a few days ago where now you can give an LLM access to your entire GitHub account. So it can read all of the issues and comment on the issues and make pull requests and approve pull requests and change your code. It's like, okay, you've gone too far in one step, right? You don't need to do that all in one step. Just take it take it easy a little bit. It can give

11:43 you lots of great advice about your code, but maybe before you approve the pull request, you should read it first. Yeah, I I mean I would say um you know AI is obviously a very interesting technology. There's a lot of hype around it. uh but don't get lost in you know building something searching for a problem right uh definitely remember or

12:07 try to figure out how you produce ROI and perhaps you know adopting AI or using AI agents is one part of the solution um but like any other let's say era of software right uh it it is just a tool in the toolbox is ultimately solve uh a business problem you know create value for your customers okay so with all of this said buy, sell or hold the growing push to use AI to secure your

12:32 network. Are we there yet? Um, so it it's it's it's interesting. So I think I've seen customers uh become much more ambitious in how they think about automation uh just in the last year, right? I think you know having reasoning agents giving them tools giving them access to essentially the equivalent of a employee work laptop

12:57 uh unlocks a lot of possibilities right now you can trust the agents enough to give them context to do the job. Uh so for some jobs that are let's say you can easily encode in some standard operating procedures uh we do see customers actually go all the way right like go full headless you know h no human in the loop uh sort of automation um so I think

13:23 I'm I'm pretty excited to see more and more of that right like and you know it's interesting to see because some of these jobs are not fun jobs um and so it's it's It's maybe good that uh humans aren't doing those ones. Yeah, I guess I go back to my analogy of thinking of AI is like interns, right? We we have as humans uh thousands of years of experience of dealing with

13:47 effectively interns and we can use that experience in a lot of the same ways. So in computers, we've certainly built up like you know our employees, you install software on their computer to make sure like antivirus for example, right? You make sure that's there because people sometimes click the wrong link and sometimes accidentally install a virus and it's not their fault and you need to catch it and deal with it. And you can build systems similarly to help you with your AI systems. But also we have human

14:11 systems that help with human mistakes, right? You have seniors who are supervising interns who give them advice uh or approve things when okay it's like hey I made a proposal I' I've done this whole patch maybe you can review it for me and make sure I didn't screw anything up. Uh, and you can also, if you're thinking like holistically of the sort of the future world of AI, you can build AI versions of those humans that are watching the other humans, right? And it

14:34 sound starts to sound a little convoluted, right? But you can, for example, even today, you can take, you know, let's let's put a log in front of every database access that our intern AI does to solve their problem, right? That log flows into a database. Well, it's another database, right? I can now have an AI look at that log and say, is there anything weird in here, right? And it will surface like, hey, at this moment,

14:58 at this time, this happened that's kind seems kind of unusual. It looks like they were downloading someone's private information or they accidentally deleted a table or something like that. And so you've got like one thing counterbalancing the other thing. Probably you're going to want to have a human in the loop eventually looking at that stuff that got surfaced, right? But you do have the ability like you should think of it as building social networks of humans except that some of them are not humans.

15:22 Right. Right. And you treat it the same way. Yeah. Right. It takes you back actually some of the older cyber examples for cyber AI. Right. Like the fish tank in Las Vegas that tried to access a high rollers database and the AI was like it's not usually typical that the fish tank needs access to this. Maybe we should that at Defcon. No, that that was actually I think it was a dark trace, one of the cyber companies or

15:45 anyway, one of their case studies. Um, we're in our final couple of minutes here. I'd love to know as we look out over the next two to five years what each of you will be watching for in this space. Um, yeah, I'm I'm the most excited about controlled automation, right? Like controlled, you know, actually like intelligent automation of uh some of these human tasks that we do. I think it'll free up a lot of, you know, not

16:09 fun work that, you know, we have to do day-to-day. Um, and I think that's where, you know, we'll see a lot of ROI from just, yeah, like these systems working autonomously in the background. Uh, you've well, you've documented wells some way to do this, you know, very, uh, boring job. Uh, and so now the agent can now do it. Um, so I'd love to see

16:32 more and more examples of that. Yeah. I'll give you my honest answer. I want to live in a world where Siri doesn't suck. Thank you. Thank you. And so Apple Apple has spent so much time thinking about what Siri should be able to do for you and building like prototype versions of all those features. And every single one of them

16:56 does not work correctly. Right? I ask my watch I say set a timer for 5 minutes. And about 30% of the time it sets something different, right? Like that that's just like baseline nonsense. And so we all know what we want our phone to be able to do when we ask it to do something. Hey, like call my wife, right? That does not work. It supposedly

17:19 works. I can supposedly go through my contact list and tag a bunch of things. It does not work, right? There's no reason it shouldn't work. This is stuff that AI could do like today if we gave it access to the information that it needs to do the work, right? and then just trusted it to do that thing. But you need to have low latency. You need to have high reliability. You need to have reasonable price. It needs to be built into your phone, right? And all that stuff isn't there yet. There's a

17:42 lot of engineering work to do that I think people are just sort of ignoring because they're so excited about the possibilities. We just need to sit and engineer it. That's your prediction for the next two years. We're going to sit and engineer it. Siri is going to be great. You heard you're first. All right, gentlemen. This has been lovely. Thank you all very much for joining us and thank you for your time. Thanks everybody.

The Internet’s been broken since IPv6 stalled; Tailscale fixes it at the IP layer.

Before I knew exactly what Tailscale would become, I knew how I wanted to build it: keep the product simple, sell to developers, win small markets before expanding, and avoid inventing management structures for fun.

I talked with Amit Kumar about those early decisions, what worked, what didn’t, and how Tailscale grew to 10,000 paid customers.

If you'd rather read than watch, the full transcript is below.

Transcript

0:00 the internet is no longer living up to the expectations that we had of it in the 1990s welcome to Spotlight on a podcast about how companies are built from the people doing the building one messy exhilarating decision at a time welcome to Spotlight on I'm your host Amed Kumar and I'm here with Avery penon the founder and CEO of tailes scale thanks Avery for joining us hey nice to be here before we get into tailes scale

0:23 and the founding story and what led you to this maybe you could walk us through a little bit about your background you know who are you and you know you you have a pretty pretty remarkable background and lots of different things that you worked on you came from Google maybe you could walk me through that a little bit uh in high school I worked at the first internet provider in my hometown of Thunder Bay uh running running the systems there there was first there was the uh nonprofit version the freet and then we uh I moved from

0:46 there to the first commercial internet company where we set I set up their servers uh I went from there off to University of waterl where we did uh co-op programs so I worked at six different companies uh one of those companies was one that I I started because I'm like well I we should learn what it's like to start a startup uh that one got a little out of control and ended up getting acquired by IBM in 2008 um after that I went and worked for a brief period in the banking industry

1:08 which was very educational but turned out not to be my thing really then I went off to Google I worked on Google Google Fiber primarily uh and some other side projects while I was there and I came out of Google and wasn't sure exactly what I wanted to do but I decided I wanted to be back in the startup world like big companies were not exactly my thing and what did you do at Google I started off in the Google Wallet team uh so if you ever noticed the little like attachment button at the

1:29 the bottom of an email that lets you attach money uh that was the team that I was your fault yeah well I mean yeah uh and you know I wouldn't say that feature was like spectacularly successful but it actually it launched and people use it it was pretty interesting it required you know it used my banking industry experience which is the reason they put me on that team in the first place and it was exciting because people have been trying to launch that feature for I think five

1:51 years uh and just like failing over and over again and you know one way or another like I showed up and either got lucky uh or pushed it over the finish line which was was pretty fun after that I went and worked in Google Fiber which of course is a internet service uh going back to my high school days uh but but maybe a thousand times faster in terms of uh uh bandwidth uh and there we were bu building Wi-Fi rers so that was my team uh building the home Wi-Fi rers

2:15 that go like come with your internet service and then once you were out of Google and you were sort of exploring ideas uh you had a friend at Versa Bank who kind of pulled you in to to solve a problem for them so this wasn't even you ideating this was just you trying to help out a buddy yeah he uh he was stalking me on LinkedIn and found out that I was no longer employed and like the day after uh which is also what he did uh my previous two jobs uh the day after he found out I was an employee

2:37 like hey Avery I've got a problem you want to help me fix this problem uh and I'm like well I'm thinking of starting a startup but we don't have any ideas yet so like sure why don't we go and try that uh and so uh yeah we we went and visited him uh and looked into his problem it's like oh this is interesting I bet we can solve I bet we can solve this in a in a weekend uh and and we built the you know project in a weekend and that that project that we built turned into tail scale what was the

2:59 problem that they had had that you needed to solve so the problem they had was they have banking software uh and you know banking software is sort of like famously not Leading Edge technology most of the time uh but this banking software didn't have two-factor authentication and they had had a security company come in do an audit and say like hey you know someone who can get the password from someone on your team uh could log into this banking

3:22 software and if you do a fishing test uh which I don't really recommend there's no point doing a fishing test because somebody at your company always fails the fishing test uh but if you do a fishing test you know you'll get maybe 20% of your uh team will accidentally give up their password which meant it was like definitely possible to break into the core banking system and start transferring money around and they're like look we have to fix this problem how do we get two Factor authentication into this Legacy banking software uh and so we brainstorm for a while until

3:46 eventually I said like hey what if instead of putting two-factor authentication in the software why don't we move the server onto a network and then VPN to the network and put two Factor authentication in the VPN and they said don't know what you're talking about but seems like a good idea and we're like all right all I need to do is find a VPN that's going to be good if you leave it on all the time and you put it in the office and it uses two Factor authentication and it integrates with

4:09 your ad surely I can just go buy that off the shelf and we'll be done and I'll go back to what I was doing and I could not find a product that was going to be good enough to solve that problem uh so I'm like all right fine this new thing called wire guard just came out it's a really good VPN it's super reliable we can leave it on all the time it's not going to break I just need to make it work with two Factor authentication how hard can to be so we slap something to together plugged it into aure ad which I

4:32 think was was it still called aure ad at the time maybe it was I can't remember whatever Azure ad used to be called uh we plugged it into that um and and it was basically a key generator for wire guard that we built in a weekend and it and it worked and he was really excited and one thing that was interesting about it this was preo this was back in 2019 shortly afterward covid hit and because of the way the system worked it was

4:55 always vpnn into the network that uh ran the banking server as he could send all of his employees home on their laptops and nothing changed they didn't have to reconfigure anything and still their banking software worked which is like weird side benefit that we hadn't even been planning for and when that happened we sort of realized we actually had something on our hands it's like wait a minute what this is this is way more powerful than we intended that was that

5:18 was the moment when you thought that lots of companies need to be using this yeah because we originally thought we were going to do like a bootstrap software company why don't we just like slap some stuff together we can pay three or four people no problem with a relatively small product who cares and just like dis realize this thing and like Whoa We can like we can fix so much stuff this this deserves to be a bigger project that we can't just sell fund

5:41 maybe walking back a little bit you know how did you how did you form the early team to go after this and then what was this what was the process about going after and kind of scaling out from One customer to kind of an early set and you know one one thing I think all our listeners know is that tail scale has done an incredible job of building the most special community of folks and developers that love tail scale I mean there's this unique product love in the

6:04 community like how did you go from zero to one when it comes to that well this is where like and I know people don't always believe me when I say this because I'm Canadian and people accuse me of doing being too modest but like I'm telling the truth here we didn't know what we were doing and we're like well look our costs are really low let's try some stuff and see what happens so the first thing we did is we took this package we had made for this Bank we're like well I don't actually want to be an

6:26 Enterprise software company uh and it's a long story but a previous startup that I did accidentally turned into an Enterprise software company I did not enjoy it very much right what I want to do is build software for like people that I can talk to that are more like me so engineers and like you know nowadays and maybe even at the time but I was a little out of the loop nowadays this is like considered a normal thing I build you know build a startup for engineers because Engineers are going to buy stuff

6:47 bottom up but I never heard of any of this I've been in Google for like seven years and before that I built software like in the preas days uh so it turned out the path that we took is very standard but I didn't know that I'm just like I don't want to to build Enterprise software we built this cool thing why don't we see if we can make something that like people can just install on their own and then play with at small companies like like what our company is and so we like fine-tuned it a little

7:10 bit and slapped it on our website and it was really not very good uh but we wrote a blog post about it uh and for whatever reason uh when I write blog posts they frequently end up on Hacker News front page uh this one ended up on The Hacker News front page and we got zillions of people coming to our website saying hey this looks neat and the problem is uh we didn't have a signup system or a registration system or a user active acation system we actually had all of the users in the system hardcoded in the

7:33 source code of the of the server and so I got you know and the way you connected was you like put you know fill in your name and push this button and it puts you in the so-called waiting list and the waiting list is a CSV file which I would I read every few minutes and then go through the emails and send them an email saying how to activate after I inserted their uh email address into our source code and activated their account uh and I did that like several hundred times in one night so it was up for like

7:55 24 hours straight just like activating people because we had to build the signup system now I forgot what the question was oh yeah and so that was the clue that maybe something cool was happening because the feedback we started getting the nice thing about this method uh I expected like five people on the waiting list and then I could have a conversation with five people instead I was having a conversation with hundreds of people that I had to email personally to tell

8:18 them they'd been activated and this is an amazing way to like get feedback on your product because they know the CEO of this company he's talking to them right now right right and so we got tons of feedback on what the product could do uh and ideas for what the product should do but also a lot of positive like oh my God this thing is amazing I can't believe how great this is you're already so much better than anything else I've seen in this category well you you sort of joked and I know it's not entirely a

8:41 joke that you know you didn't know what you were doing but your intuitions have always been right and I think one of the things the community has really appreciated about you and tail scale and you know all the folks who work there is you guys sort of try to do things the right way you guys are very transparent very honest with with customers in the community I think people have responded to that I also think you benefit from selling to developers and building for developers right a your intuition is

9:03 probably better for that than for Enterprise SAS but it's just easier and better to communicate with those people directly even that I don't want to give it too much credit my intuition is listen to people like actually and then give them what they want right and I know a lot of people believe it's like I think there's the the uh what is the Ford saying if you just listen to customers we would have built a faster horse or whatever it is it's like look people really like Faster Horses um but

9:27 the trick is like sometimes you can listen the important thing is to understand the problem they're trying to solve right right and if you understand the problem you're trying to solve then maybe you can come up with a more clever way to solve the problem you don't have to implemented just like they said but you have to give them the thing that they said they want right if they said they wanted to be able to connect to their banking software using two Factor authentication you have to find a way to let them connect to their banking

9:48 software using two-factor authentication you can't just tell them like no two-factor authentication is not solution right and so when these people tried out tail scale they said like wow this is really amazing but I wish it could do this and this and this then you can listen to them and actually give them what they want and so my I guess my addition to that maybe the part that makes tail scale unusual is is it comes from the name tail scale which was sort

10:11 of the founding principle we didn't know what we were going to build but we knew how we were going to build it which is like everybody in the world makes everything too complicated now right it's all based on this like Google advanced super scaling stuff from the early 2000s back when computers were like literally a thousand times slower than they are today right and like I want to make a giant distributed system because it's cool to make giant distributed systems but to do that

10:32 everything has to be hard and so you get to the point where people are like to launch my website I created a kubernetes cluster uh and then put made it auto scaling and blah blah blah blah blah and like you know what my website runs on the minimum cost linode node right using a little python script that I start by hand from the command line right and whenever I want to change the python script I like edit the one file that's the python script and then I kill it and restart it right right and like that's

10:55 Avery's website which can handle as much load as anyone will ever throw at Avery's website because it's not that good but every now and then it gets a popular blog post right and almost every project in the world is like that scale of difficulty or maybe 10 times or 100 times as hard as that which is still not very hard like Google levels of difficulty is millions of requests per second that have to be served in like 100 milliseconds and they're searching

11:18 the entire internet like to do that you need a monster size system but virtually guaranteed your problem is not that hard and so tail scale is like look if you have a problem that hard we're not going to try to fix it for you right if you have a problem that's easier than that we can do it a totally different way and we'll just go straight to solving your problem and not just get lost in all of the complexity I think it's an incredible insight and it's also just gives me PTSD because I started a series

11:41 of very unsuccessful companies but of course you build these companies thinking that millions of people are going to come and billions of requests are going to happen at the same time and you sort of prematurely scale all these systems and to the point you're making you know there are a few very special and lucky companies that have to hit those scale metrics and numbers but you know most every everybody else is is solving for something much smaller or different yeah I mean the biggest risk when you're small is that you're not

12:04 going to build something useful right and so you need to build something useful as POS as fast as you possibly can once it turns out to be useful then first of all it's easier to get funding but secondly you've got revenue coming in you can pick the revenue you can invest it in engineering you can fix the stuff you didn't fix later there's this principle uh that I learned a long time ago comes from the like so-called extreme programming people that evolved into agile it's called yagy Uh you

12:27 aren't going to need it right and this is the principle is like if you're not sure you need it right now just assume you're not going to need it because most of the time building that stuff is not going to be any harder later than it is right now which means like you should always push off that investment as long as you possibly can in case you don't need to spend it right and this is an amazingly useful principle you can apply to almost everything and even if it's a

12:51 little more expensive later than it is right now you're going to have a lot more money later than you have right now so you should always do the things you do need to do right now because there's an unlimited list of anyway right so don't like imagine things that you might need to do and then do them right now you can wait I know we talked about this at the last company offsite but I've just I'm I'm still amazed you know a how great it's been to work together but we we come from such different backgrounds

13:14 and and yet I think we actually did make the right choice and ended up working together it's been it's been awesome yeah so I have to say one of the things I liked about Excel uh was they were an investor in uunet a long long time ago uunet was the link that my first job in Thunder Bay when we had this first internet service that's how they got to the internet and so I'm like oh these people have good taste in uh internet providers or at least they did like 30

13:36 years ago um and probably you didn't have much to do with that um but you know thank God for Arthur yeah but you know you're you're you're the sort of successor in a long line of people with good judgment is is sort of what I was thinking well you know well let's let's hope that that lineage has continued with tail scale um so we made the investment in July or August of 2020 and everything has been Rosy ever since

14:01 right like things have gone extremely well and nothing has ever gone wrong correct at the company yes it's actually it's actually very unusual nothing ever goes wrong yeah no that's not true what um it it's obviously been a journey and there have been challenges there's been obstacles the company's had to do a lot of growing up what you know what were one or two Crucible moments for you as

14:24 you think about you know critical junctures things that you got right hard decisions they you to make that you know you look back on and you're like you know what like in the fog of war that felt like a really critical moment but now on the other side of it we made a really great decision maybe we made a really hard decision my My Philosophy generally around doing a startup is make your mistakes as cheap as possible so that you can just recover quickly and

14:47 don't don't go into denial about the truth about things so most things that we've done wrong and there's been like you know uncountable numbers of them have just like not turned into some big spiral that anybody would point at and say oh my God biggest mistake in my life cuz we was like oo that didn't work and then we turn it around in like a month or a few weeks and it's gone right yeah um we've we've gone in directions with the tail scale product for example we've

15:09 put out features that have not really taken off right or even if they did take off we realized they pulled in like the kind of users who are not the most profitable users and it's like this is a great feature but it's not the one that is going to like move the business forward right now yeah uh an example is tail drop which everybody loves tail drop once they hear about it right it's sort of like airdrop on your Apple devices but it doesn't require an Apple device and it doesn't require them to be

15:31 side by side but it lets you send a file between any device and any other device pointto Point encrypted without sending it up through our server and for free it's like well that sounds great that could be a whole product it's like it could but it's kind of consumer and we didn't need to go right now in the consumer Direction because we have all these engineers and the way tail scale works is all these Engineers most of them have jobs some of them

15:54 bring it to work uh some of the people who bring it to work end up using it at work and paying us yeah right and that's great and we're going to eventually invest more in the consumer Direction but there's so much opportunity in this Direction with just Engineers that we should we I'll say we should have instead put the energy into more engineering stuff but it's not like we wasted a lot of energy tail drop was like a three-month project for our 10 person team right and it created a whole

16:18 bunch of Buzz and that whole bunch of Buzz caused the word of mouth to go up so was it a mistake it's like there's probably something we could have done that would have been better might have been a more efficient but it was not a very expensive mistake our most expensive mistake have been in the direction of like we didn't really figure out marketing for a really long time uh and you know it's funny to say that because when I say that to investors and when I said it to investors in our series a and our series B they're like what are you talking

16:40 about you're so good at marketing I'm like I'm not you have no idea yeah uh tail scale Market he not kidding we we're not good at marketing we might be now though we're we're we are suddenly getting much better at marketing um but and that that completely relates to our new VP marketing Sydney uh who who really gets it right but before that like of our success was through Word of Mouth like literally our website didn't

17:02 matter we did an AB test at one point and and we just replaced the front page with a giant button that said download tail scale and that actually increased our signup rate because the only people who went to our website were the ones whose friends had been clubbing over the head saying you need to go download tail scale right right so they would Google for the word tail scale which was like still the number one reason anyone shows up on our website right somebody told them to Google for the word tail scale

17:24 then they click on tail scale right and then they dig around trying to find the download button right and that's like that's great that means your word of mouth multiplier is really high but that means we weren't doing marketing properly and the reason we weren't doing marketing properly had to do with like you know how do you hire people how do you find the right person who's really good at the stuff that you're not good at right because I'm pretty good at hiring people to do the

17:47 things that I am good at like many Engineers can spot another good engineer right so we hired a bunch of great Engineers but like anything I'm not good at how do I know like and even if they're good at it maybe they're not the right person for your company right like marketing has like a hundred different variations of how you should do it somebody who's really good at one of the variations that variation might not be the right one for your company and So

18:10 eventually we've tried many different itations but to get to our current VP marketing for example I had to interview dozens and dozens and dozens of people and find out how they did it and ask for people's advice and ask for their advice and like what do you think about this until I finally found somebody who like explained it to explained to me how my company should work Avery what's what's been surprising you know when you build a really powerful tool I mean of course

18:32 there are like obvious ways to use it what have been some surprising use cases that have emerged you know as tail scales become more and more popular people use tail scale for really unexpected use cases uh one of the most recent ones as we we as we were like grinding through our data about like who's using tail scale we found that like you know five of the five top or AI companies were using tail scale and then we found that like hundreds of not the

18:55 top five AI companies were also using tail scale and we're like oh why I don't know any of these people they all just signed up through self- serve and never contacted us right so we had to dig around and find out like okay what about tailes scale makes us popular with AI companies um and the answer is basically they're all stuck with multicloud uh they all have to deal with some gpus they're probably all using kubernetes and they all have like all of

19:18 these like you know they have connectivity problems but they don't want to invest in a networking team there are a bunch of really smart Engineers who are AI engineers and networking is just getting in their way and they just want to spend some money to make networking problems go away and they had all heard of us because they're early adopters and they were all mostly living in San Francisco and like everybody in San Francisco now knows about tail scale um and so they just AI companies just like across the board

19:42 started adopting tail scale for everything and this is again it's great it was a surprise but if we' been on top of our game we would have seen that coming and maybe put some work into it like at the time you could search for AI tail scale and not find anything on our website there was no not a single mention of the word AI anywhere or machine learning or or llm or GPU right like there was nothing our website was not serving it at all other people were

20:05 telling other people oh you you work at in AI you should do this like they go to an AI conference and people would like ah how do I connect to my GPU it sucks and like oh I have tail skill you should try it it's free trial right and just like took off uh more recently uh we found out retroactively that tail skill is a service mesh and I'm like I don't even know what a service mesh is uh but I'm talking to customers and we have

20:27 customers who are like well we threw out our old service mesh and now we're using tail scale as a service mesh tail scale as a service mesh and I'm like we did I'm gonna have to go Google some stuff let me come back and so now I know what a service mesh is a service mesh is a combination of connectivity identity uh and service Discovery right and tail scale obviously does productivity and identity which is the thing that it does

20:50 uh I never thought of it as a service Discovery system but it turns out the architecture of tail scale is that it keeps a list of all the devices in your network because it had to generated encryption keys and track them and distribute them and then it tells each node about each other node and then each node has a list of nodes and you can query it and say like who's the list of nodes that match this criteria and that turns out to be a service Discovery mechanism and some of our customers realized that we did these three things

21:12 and are like out with the old service mesh thing and put in tail scale and again it's like there's no not a single mention of the word surface mesh anywhere on the tail scale website actually that might still be true today people are actively working on it they're going to be launching web pages that say service mesh sometime soon right but it's like these kinds of things where it's like you just have to at this point we have to keep our eyes out for what people are already doing and then listen to them and then tell

21:35 the story back and improve the experience that they're having it wouldn't it wouldn't be a tail scale podcast if I didn't at least give you the layup of telling me about crossing the chasm and I I think I think this AI sort of pervasiveness you know allowed you to keep like resurrecting crossing the chasm and board meetings for at least another like two years so um could you maybe just talk about that because I

21:57 know it's like one of your guys principles as you think about the company and in particular for the motion that you have of getting ubiquity within developers how is that you just tell us the audience about that and then how that's guided some of your decision- making I guess the the history of this uh I was introduced to the book Crossing the CM by one of my investors at my very first startup toward the end of the cycle of that startup when we were like

22:19 we were struggling uh we had like growing revenues but it was really hard to grow the revenues and in retrospect what happened was we were stuck uh in with basically early adopter customers and there were lots of early adopter customers and we were pushing really hard we had a sales team like searching as hard as they could to find early adopter customers but like every single thing was like just push push push push and they're like well Avery

22:42 you you should read this book it kind of explains what's happening I'm like ah how could a book explain what's happening we have a unique situation nobody's ever done this before blah blah blah like I read the book and the format of the book every chapter is like here's what people usually think you should do here's what happens when you do that and here's what you should do instead and so I like opened up chapter one it's like oh that's what I did oh that is what happened oh that's what I should have

23:05 done instead and then you go to the next chapter and the same and the same and it's like the story of our entire uh startup from beginning to where we were and a recipe for what to do to fix it right and so I I decided to follow the recipe and then within like six months the business turned around right but by then it was too late we' been doing it for like eight years we were running out

23:27 of money in didn't want to put in more money the ex the First Investors were like getting impatient and so we exited IBM but if I had that same exact advice for or five years sooner it could have been a completely completely different company because we had the product that we needed we didn't know how to cross this casm and the secret of crossing the chasm is just to figure out how to become um the default product that everybody uses for some use case and the

23:53 the thing that nobody realizes the super counterintuitive part is that that use case that group of people has to be tiny because you're not going to win you're not going to be the default Solution that's more than 50% is like how you be the winning default solution you're not going to be the bigger than 50% of anything big when you're tiny right so everybody wants to go after this thing with a giant total addressable Market they want to find some Market that's

24:14 like humongous and tell their investors like don't worry there's like 10 billion dollars of possibility here a trillion dollars right it's like that's not how you succeed when you're tiny right the way you succeed is you find something really small and you win that really small thing by telling everybody like look no maybe you don't have to say this out loud to the customer but like this is too tiny for anybody to care about but I care because I'm also tiny so we're going to give you the best possible

24:38 service and go out of our way to make the best possible thing and then when you win that then you can win something adjacent and something adjacent to that and something adjacent to that and it just gets bigger and bigger and bigger so like the secret strategy of tail scale and again it feeds into the name like we're we accept the idea of doing Small Things yeah right and so tail scale is always going after like what is the small thing that we can do next that

25:00 we're going to win easily right and AI was one of the things where like okay we actually were kind of late we already won it before we noticed that we had won it right uh AI networking I should say um and then you know the service mesh thing again is like oh we're not winning but it's actually like I I looked it up the other day the service mesh Market is not that big in terms of dollars right now it's actually achievable to make it

25:23 a splash in the service mesh market right right and so we'll keep doing things like that uh and growing and growing like incrementally but it's such a good book because it just it explains why this works in a you know I'm a systems design kind of person but it explains it in a systems design sort of way like this is why systems like this always happen and when you do the obvious thing this is why it always doesn't work Avery how do you think

25:46 about monetizing or charging for what you've built at tail scale you started off with this amazing Community it's very Bottoms Up you have individual developers you talked about this motion of developers at home using it bringing it to work how do you think about charging and you know kind of growing up into a business over time sure so I I said earlier that like I didn't want to build an Enterprise software company

26:09 which is true but I want to maybe I should qualify that because like obviously our first customer was a bank uh sales scale is suitable for Enterprises what but I want to like make the internet a better place right and if you're going to fix the internet if you're going to fix this like lowlevel protocol of tcpip and get it out to everybody then it has to be literally everybody everybody includes Enterprises But it includes everybody else and so

26:30 tail scale my other favorite Business book is called The innovator's Dilemma and if I was to summarize it in like one line it's like nobody ever goes down Market they only ever go up market and so if you start up Market you're never going to go back down right so tail scale our policy is like we are going to make sure that the zero Doll part of the market is ours and we're going to do that by a giving away the product to people who want to use it for free and B

26:55 making sure that it's cost effective to give away the product at people for free free because I what I can't have is just I'm going to raise a bunch of money from investors and then spend all that money uh giving away like AWS credits or something like that so that people can have the product for free so I have this blog post called how tail scale remains free um and it explains the architecture of tail scale and why it doesn't cost us anything for you to have your free account and if it doesn't cost us

27:17 anything for you to have your free account we can keep giving it out to lots of people and in fact in the early days of tail scale because you know I I talked about how uh in the very early days we didn't even have a signup system that was not aver sending you an email uh but later we're like okay well we have lots of work to do why should we Implement restrictions on what customers can do so there was no actual limits like if you bought a 10 user account or if you didn't buy anything there was

27:40 nothing stopping you from signing up another hundred or few hundred users and that's what we actually got and so when we hired our first few salespeople the first few salespeople job was just like hey we've got some customers with like hundreds of seats they never actually emailed us we should probably tell them that they're actually supposed to pay for the product if you dug around if you actually went to the billing page it would tell you like whether you were

28:01 over your subscription but it was very Canadian it said like hey you know it's great that you're trying this thing uh probably you should pay us sometime I think I can't remember the exact wording but it was kind of like that I was uh I always told the people at Excel that you're uh the company is capital c Canadian and lower C capitalist it's it's stories like that that but but you know as a Canadian I want to point out that despite all of

28:24 that uh the amount of Revenue that we could capture by going out and collecting all the money from people who were underpaying us was only about 20% extra that means like 80% of the people in the world were paying us even though there was absolutely positively nothing making them do that right because most people in the world are honest uh and that that is you know a fundamental belief of Canadians of course but also like you know that extra 20% like I

28:46 could afford to have that 20% for the huge amount of Word of Mouth that It produced uh but anyway so as tailes scale has been getting bigger we have a more structured sales team now I actually got complaints from various people that I knew is like when they signed up for tail skill and started using it they're like Avery you're doing something wrong I'm not going to pay you until somebody reaches out to me and tells me to pay like my my own friends were telling me this it's like Avery fix

29:08 your company it doesn't make sense nobody's going to be mad because they've got 50 seats that they're really enjoying and somebody emails them saying like oh we should have a discussion about dollars now right and so we started doing that we have a real professional sales team uh we still don't actually enforce too many limits but the sales team will catch you sooner uh and invite a conversation right and so we've been getting bigger and bigger now and we're starting to actually do

29:30 our first like Million Dollar Plus deals and so when I said I don't want to do Enterprise sales it's like somebody now we have employees there are people who are going to do the Enterprise sales but I have to remember like where we came from and where we came from as individual Engineers who love the product they use it at home they bring it to work they use it in small teams and eventually it turns into a system where people buy it top down it roll it

29:52 out to the whole company and so that's where tail scale like the whole vision of the new internet like it can it can scale all the way down from zero to maximum and that's how you get the new tcbp rolled out to everybody you talked about transferring files between devices you talked about two-factor authentication you talked about vpns you talked about connecting

30:14 devices and now you're talking about service mesh and you're talking about hey AI networking how do you think about defining the vision for the company I mean you know when you stand up at an all hands in front of the whole company you know how do you describe to them what the future looks like and what does that mean to you um in terms of the mission of the business and what the implications are on the world that we're headed towards well I'm kind of workshopping this one uh the one the the

30:40 way we have been saying it we'll do it live let's go okay okay yeah yeah so what we say is like tail skill is the new internet and I like it because it's funny because there's a TV show Silicon Valley where like the main character has made this thing that was called the new internet it was just like a series of laughs but actually the internet needs to be fixed and I mean by that is like as an engineer tcpip is the problem right and we've known this since the

31:03 1990s ipv4 is not good enough they tried to launch IPv6 IPv6 should have solved a whole bunch of our networking problems have your networking problems been solved by IPv6 no right and why not large part of it is the thing has not fully rolled out and why didn't the thing fully roll out it didn't roll out because it didn't follow crossing the chasm right like they didn't have a roll out plan that

31:26 was going to win and it's still hasn't won right and it doesn't look like it's going to win if you look at the trend line it looks like maybe in the next 40 or 50 years maybe they will get to like 80 or 90% adoption it's like I might be dead by then right I've been like waiting for IPv6 my entire life right and it's not solving the problem like IPv6 was the chosen one it didn't work out the new internet didn't happen it was it was the new internet so what are

31:48 you going to do about it right tail scale is actually and again it was kind of by accident because I just wanted to fix some problems tail scale is this thing you insert at the IP layer that makes the internet work the way the internet was supposed to work right and when the internet works the way it was supposed to work any device can talk to any device you've got safety you've got security you've got encryption you've got identity you've got like e you know

32:10 easy to set up you don't think about it anymore it just works properly right and when things work properly like one of the things that's hard to explain about the vision is like when people were inventing tcpip they didn't really think that big they were like you know what I want to be able to access the supercomputer at my University from another University he's like you know that sounds neat but it doesn't sound like the internet we

32:32 have today right the internet we have today is used for everything all the time right I have internet on my watch if I if I bought a watch and it didn't have internet I'd be like why am I paying for this it's it's broken right like nobody thought about that we used to call them smart watches now we just call them watches exactly or smartphones like when was the last time you phone somebody on your phone right like it's so what is the vision for tail scale the

32:55 vision for tail scale like fundamentally on an engine engering level is like look we just need to fix the thing that we call the internet and replace it with this new thing and the layers on top can be actually the same but a whole bunch of stuff is going to work better right and then how do you explain like what's the business value of that what are people going to do with it it's like it's a little hard to explain because most of the uses for tail scale haven't

33:17 been imagined yet right and I think the only way I can explain that is through that analogy and so even though it's a joke like the new internet analogy is actually the right analogy like the internet is no longer living up to the expectations that we had of it in the 1990s because IPv6 didn't work and even if IPv6 did roll out now you know it's been 25 years since it was designed it's missing some stuff that we would have

33:40 put into it if we designed it today but it's just been like stuck and if you unstuck something that's 25 years old in the tech world you're going to get a whole bunch of benefits okay so that was incredible by the way that was pretty good pretty good for a workshop if you could go back and give any feedback or advice or wisdom to Avery outside of Reed crossing the chasm four years

34:02 earlier what would you what would you tell them I think tail scale we did a really good job building the company through the first several phases I think what we didn't do a fantastic job of is as we're switching from what I would call a like early stage startup to a growth stage startup you really have to restructure the way the company Works to make it more scalable right and that

34:24 means building an executive team in the right structure so that you can can like help people like get their work done when not everybody is going to be able to know what everybody else is doing all the time and so I didn't do a super great job of creating the structure inside the company that I probably shouldn't like the organization of the executive team is really important uh you don't need to be super Innovative

34:47 there's like again you're not the first person to build a company right uh and so there's lots of advice you can find on like okay what's the good structure of an executive team how do you choose what rle should go here there's only a few structures that like reliably work and so you don't want to innovate on everything in your company it's great to be Innovative on the technology side but you don't need to invent everything you don't need to invent your own accounting systems Finance systems and so on uh the

35:10 most important thing for me when building an executive team that I learned is like actually executive search uh companies are amazing and they're worth the weight in gold uh much more than I realized because the main thing they do is they introduce you to lots and lots of great candidates some of whom you won't even be able to land some of you don't even want to land but you get such a wide array of information about what's possible in a particular

35:34 role that after you've talked to like 25 or 30 of these people then you can understand what perfect looks like for you right and when you're when you're a tiny little company and you're only hiring one executive it doesn't matter if if there's not a hundred different people who can fit that exact role that you've invented if there's just one person who can do that exact combination of things that you want you can just find that one person and that one person

35:57 is really going to want to work there because the job that's absolutely perfect for them right and we found like that kind of Executives but I didn't realize before that it was possible to do that because I didn't connect to the fact ironically given that tail scale is all about small things right I didn't really connect to the fact that you only need one perfect person for this job and that person is there somewhere and you can just go find them but it's going to be a lot of work and an executive Search firm will help do that it's awesome yeah

36:20 I think I think you've done a really good job like in those in those situations one thing I've grown to really appreciate about you is you definitely take time to get all the data but once you have the data I I really trust your decision making and it's worked out pretty well for us my comment on data is I love data but in fact like most of the most useful data in my life turns out to be like anic data or like building an intuition for something so

36:43 maybe I'll stare at the data for a long time it's like okay I think I understand what the pattern is now and then close down the dashboard and like with the idea about the pattern you talk to people you get advice what's the com what's the pattern of advice what's the standard way to do things are we really special enough to be breaking this pattern or should we just stand do the standard way to do things uh and then just like you know there's there's a little bit of like just following your intuition but to make your intuition

37:06 smart enough requires a lot of studying which is how I do it it's awesome Avery thanks for joining us you're great dude [Music]

Let's talk about bug/feature tradeoffs.

Anyone who knows me has probably already heard me rant about Crossing the Chasm, my most favourite business book of all time. I love its simple explanation of market segmentation and why the life cycle of a tech startup so often goes the way it does. Reading that book is what taught me that business success is not just a result of luck or hard work. Strategy matters too.

As our company prepares for our chasm-crossing phase, I've been thinking about the math behind why chasm-crossing works and why our metrics plots (doesn't every startup do their key business metrics in R?) look the way they do, and I realized that chasm-crossing strategy must have a simple mathematical basis behind it. I bet I could math this.

And so, our simulated software engineering (SWE) team is back!

In previous episodes of SimSWE, we learned it's objectively good to be short-term decisive even if you're wrong and to avoid multitasking. Later, I expanded on all that, plus more, in my epic treatise on software scheduling. And then, as a bonus, our simulated SWEs went on to buy homes and distort prices in the California housing market.

This time, I want to explore the chasm-crossing process and, while we're here, answer the unanswerable question: what's a bug and what's a feature?

Nobody can agree on what they mean. When does "lack of a feature" become a bug? When a key customer demands it? When the project manager declares a code freeze but you still want to merge your almost-finished pull request? When it's Really Really Important that you launch at a particular conference?

The answer is, users don't care what you call it. Let's reformulate the question.

We need to make a distinction between needs and wants.

Back when I lived in New York, I took some fiction writing classes. One thing I learned is there is a specific recipe for "interesting" characters in a story, as follows: understand how characters' needs differ from their wants. It's rare that the two are the same. And that way lies drama.

So it is with customers. I want a browser that doesn't suck all my RAM and drain my battery. But I need a browser that works on every website and minimizes malware infections, so I use Chrome.

I want a scripting language that isn't filled with decades-old quoting idiosyncracies, but I need a scripting language that works everywhere, so I mostly use POSIX sh.

Some people call needs "table stakes." You must be this tall to ride the roller coaster, no exceptions. If you are not this tall, you cannot ride the roller coaster. Whether you want to ride the roller coaster is an orthogonal question related to your personal preferences.

Needs are AND. Wants are OR. A product must satisfy all your needs. It can get away with satisfying only one want, if you want it badly enough.

Needs are roadblocks to your product's adoption. (I previously wrote about roadblock analysis.)

A want is a reason to use some new software. A need is a reason you can't.

About 20 years ago(!), Joel on Software wrote about the 80/20 myth:

80% of the people use 20% of the features. So you convince yourself that you only need to implement 20% of the features, and you can still sell 80% as many copies. Unfortunately, it’s never the same 20%.

– Joel Spolsky

And yet, if you're starting a new project, you can't exactly do 100% of the features people want, all at once. What can you do instead?

Market segments, use cases, and needs

The best (and thankfully becoming common) advice to startups nowadays is to really nail just one use case at first. Pick a want, find people who want it, figure out what those people have in common, call it a market segment, solve the needs of the people in that segment, repeat.

This is all harder than it sounds, mostly because of your own human psychology. But it all lends itself well to rapid iteration, which is why our earlier SimSWE tips to be decisive and to avoid multitasking are right.

Getting back to Crossing the Chasm, the most essential advice in the book - and the hardest to follow - is to focus on your chosen market segment and ignore all requests from outside that segment. Pick one want. Fulfill all the needs.

Let's make a simulation to show what happens if you do or don't. And if we're lucky, the simulation will give us some insight into why that's such good advice.

Simulating wants and needs

The plot below simulates a market that with 10,000 potential users, 10 potential wants, and 15 potential needs. Each user has a varying number of wants (averaging 3 each) and needs (averaging 5 each).

For a user to be interested in our product, it's sufficient for our product to fulfill any of their wants. On the other hand, for a user to actually adopt the product, they need to be interested, and we need to fulfill all their needs.

Side note: we can think of a "Minimum Viable Product" (MVP) as a product that fulfills one want, but none of the needs. There will be some tiny number of users who have no special needs and could actually use it. But a much larger group might want to use it. The MVP gives you a context for discussion with that larger group.

Before we get to all that deliberate activity, though, here's an example run of the simulator, with random-ish sequencing of wants and needs.

The dim dotted line is the Total Addressable Market (TAM). Every time you implement a want, the TAM goes up. Fun! This is what venture capitalist dreams are made of. All the users in the TAM are "interested" in your product, even if they aren't able to use it yet.

The dashed line is the "unblocked" users. These are users who are in the TAM and whose needs you've entirely filled. They legitimately could buy your product and be happy with it. Assuming they hear about you, go through the trial and sales process, etc. This is the maximum number of users you could have with your current product.

Finally, the red line is the number of users you actually have at any given time. It takes into effect marketing, word-of-mouth, and adoption delays.

Commentary

I'm already excited about this simulation because it shows how adoption curves "really look" in real life. In particular, you can see the telltale signs of a "real" adoption curve:

• Exponentially growing uptake at first, which slows down as you saturate the market (ie. an "S-curve" shape).

• When you look more closely, the big S-curve is made up of a bunch of smaller S-curves. Each time we fulfill a need, some group of users becomes unblocked, and we can move toward saturating an ever-bigger market.

Observe also that the jumps in the dotted line (fulfilled wants) are big at first, and smaller each time. That's because each user has an average of three wants, and you only need to satisfy one of them. Because of overlapping wants, the second want is split between new users and users you already have. Each successive want has a greater and greater overlap with your already-interested users, and thus less and less effect.

(Alas, this gives a good mathematical rationale for why "mature" products stop improving. Yes, there are all sorts of additional things your audience might want. But if adding them doesn't increase your TAM, it's strategically questionable whether you should bother. Bring on the continuous improvement debate.)

(On the other hand, this simulation is somewhat unrealistic because of the pre-defined market size of only 10,000 participants. If, instead of fulfilling more wants for your existing market segment, you add a new market segment, those new wants might have a bigger impact and your "big" S-curve might get a newer, bigger S-curve added to it. This is small consolation to your existing users who would like some more stuff added that they care about, though.)

In contrast, the jumps in the dashed line (needs fulfilled) start small and get bigger. This also makes sense intuitively: since users can't adopt the product until all their needs are met, and the typical user has 5 needs, certainly the first 4 needs are going to attract only a small group of less-discerning people. Even the first 5 needs will only capture the group of users with exactly those 5 needs or fewer. But by the time you're reaching the end of the to-do list, every new need is unlocking a big group of almost-satisfied users.

(This part is coool because it explains what startups so often experience: at first, fulfilling needs for your target market creates a small jump in absolute user count. But through this "AND" effect, each subsequent need you fulfill can create a bigger and bigger jump. Even if the new features seem fairly small or relatively easy compared to your early work!)

Comparing strategies

Of course, that was a single simulation based on a bunch of made-up arbitrary assumptions and some numerical constants selected mainly on the basis of how pretty the graph would look.

The good part comes when we compare multiple product management strategies:

Let's continue to assume a fixed market segment of 10,000 users, each of whom have an assortment of wants and needs.

The four plots above correspond to four ways of prioritizing those wants and needs:

1. Features First: the "maximum hype" approach. Implement all 10 wants before solving any needs at all. This maximizes TAM as early as possible. Some early-stage investors get starry-eyed when they see that, but unfortunately you don't get a lot of live users because although people are excited, they can't actually use the product. This is also what you get if you don't, as Steve Blank would say, "get out of the building" and talk to real customers.

2. Alternating: switch between implementing wants and needs, semi-randomly. It turns out this grows your userbase considerably faster than the first option, for the same reason that you'll do okay at rock-paper-scissors by using a random number generator instead of always choosing rock. The main thing here is shipping those randomly-ordered milestones as fast as you can. As SimSWE 1 and 2 emphasized, if you do that, you can get away with not being super great at prioritization.

3. Needs First: just implement exactly one want, then fix all the needs before moving on to other wants. This is a purified Crossing the Chasm model. You can see that the TAM doesn't start increasing until pretty late, because new use cases are on hold. But we get precious real users earlier, which spread word-of-mouth sooner and lead to faster exponential adoption later.

4. Perfectionism: the naive opposite of features-first; a variant of needs-first where we don't even solve a single want before we start trying to address needs. Since the product does nothing useful, but very reliably, nobody wants to buy it at first (~zero TAM). When we finally start launching use cases, we can add them pretty quickly, but actual growth lags behind, at first, because we were late in getting our exponential growth curve started. think of this as the "we got SOC2 compliance before we had any customers" strategy.

In these plots, the important things to look for are getting more users sooner (money in the bank!) and total area under the curve (aggregate value delivered). Users you get earlier are users who give you money and spread word-of-mouth over a longer time, so they are much more valuable than users you add later.

In this version of the plot, it looks like #3 is winning, #4 is not too bad, and even #2 might be kind of okay. In the end, is there really much difference?

Let's zoom in!

More needs fulfilled, more momentum

This plot zooms the y axis to the first 1000 customers, leaving the x axis unchanged from before. Now the differences are more dramatic.

Here you can see that needs-first starts attracting at least a noticeable number of live customers at time 150 or so. The others take much longer to get rolling.

This feels intuitively right: in the early days of a startup, you build an MVP, nobody uses it, you find a few willing suckers early adopters and listen to their feedback, fix the first couple of roadblocks, and now you have a few happy niche users. If all goes well, those users will refer you to more users with a few more roadblocks, and so on. At that stage, it's way too early to worry about expanding your TAM.

What I find exciting - but not all that surprising, having now immersed ourselves in the math - is that the needs-first approach turns out to not be a compromise. The word-of-mouth advantage from having zero-roadblock, excited, active users early on means the slow part of the exponential growth can get started early, which over time makes all the other effects look small. And each successive fulfilled need unlocks an ever-greater number of users.

In contrast, you can see how increasing the TAM early on has not much benefit. It might get your investors excited, but if you don't have live users, there is nobody to spread word-of-mouth yet. Surprisingly little is lost by just focusing on one small want, clearing out roadblocks for people who want that, and worrying about the rest later.

"Systems design" is a branch of study that tries to find universal architectural patterns that are valid across disciplines.

You might think that's not a possibility. Back in university, students used to tease the Systems Design Engineers, calling it "boxes and arrows" engineering. Not real engineering, you see, since it didn't touch anything tangible, like buildings, motors, hydrochloric acid, or, uh, electrons.

I don't think the Systems Design people took this criticism too seriously since everyone also knew that programme had the toughest admittance criteria in the whole university.

(A mechanical engineer told me they saw electrical/computer engineers the same way: waveforms on a screen instead of real physical things that you could touch, change, and fix.)

I don't think any of us really understood what boxes-and-arrows engineering really was back then, but luckily for you, now I'm old. Let me tell you some stories.

What is systems design?

I started thinking more clearly about systems design when I was at a big tech company and helped people refine their self-promotion employee review packets. Most of it was straightforward, helping them map their accomplishments to the next step up the engineering ladder:

• As a Novice going for Junior, you had to prove you could fix bugs without too much supervision;
• Going for Senior, you had to prove you could implement a whole design with little supervision;
• Going for Staff, you had to show you could produce designs based on business problems with basically no management;
• Going for Senior Staff, you had to solve bigger and bigger business problems; and so on.

After helping a few dozen people with their assessments, I noticed a trend. Most developers mapped well onto the ladder, but some didn't fit, even though they seemed like great engineers to me.

There were two groups of misfits:

1. People who maxed out as a senior engineer (building things) but didn't seem to want to, or be able to, make it to staff engineer (translating business problems).

2. People who were ranked at junior levels, but were better at translating business problems than at fixing bugs.

Group #1 was formally accounted for: the official word was most employees should never expect to get past Senior Engineer. That's why they called it Senior. It wasn't not much consolation to people who wanted to earn more money or to keep improving for the next 20-30 years of a career, but it was something we could talk about.

(The book Radical Candor by Kim Scott has some discussion about how to handle great engineers who just want to build things. She suggests a separate progression for "rock solid" engineers, who want to become world-class experts at things they're great at, and "steep trajectory" engineers, who might have less attention to detail but who want to manage ever-bigger goals and jump around a lot.)

People in group #2 weren't supposed to exist. They were doing some hard jobs - translating business problems into designs - with great expertise, but these accomplishments weren't interesting to the junior-level promotion committees, who had been trained to look for "exactly one level up" attributes like deep technical knowledge in one or two specific areas, a history of rapid and numerous bug fixes, small independent launches, and so on. Meanwhile, their peers who couldn't (yet) architect their way out of a paper bag rose more quickly through the early ranks, because they wrote reams of code fast.

Tanya Reilly has an excellent talk (and transcribed slides) called Being Glue that perfectly captures this effect. In her words: "Glue work is expected when you're senior... and risky when you're not."

What she calls glue work, I'm going to call systems design. They're two sides of the same issue. Humans are the most unruly systems of all, and yet, amazingly, they follow many of the same patterns as other systems.

People who are naturally excellent at glue work often stall out early in the prescribed engineering pipeline, even when they'd be great in later stages (staff engineers, directors, and executives) that traditional engineers struggle at. In fact, it's well documented that an executive in a tech company requires almost a totally different skill set than a programmer, and rising through the ranks doesn't prepare you for that job at all. Many big tech companies hire executives from outside the company, and sometimes even from outside their own industry, for that reason.

...but I guess I still haven't answered the question. What is systems design? It's the thing that will eventually kill your project if you do it wrong, but probably not right away. It's macroeconomics instead of microeconomics. It's fixing which promotion ladders your company even has, rather than trying to climb the ladders. It's knowing when a distributed system is or isn't appropriate, not just knowing how to build one. It's repairing the incentives in a political system, not just getting elected and passing your favourite laws.

Most of all, systems design is invisible to people who don't know how to look for it. At least with code, you can measure output by the line or the bug, and you can hire more programmers to get more code. With systems design, the key insight might be a one-sentence explanation given at the right time to the right person, that affects the next 5 years of work, or is the difference between hypergrowth and steady growth.

Sorry, I don't know how to explain it better than that. What I can do instead is talk about some systems design problems and archetypes that repeat, over and over, across multiple fields. If you can recognize these archetypes, and handle them before they kill your project, you're on your way to being a systems designer.

Systems of control: hierarchies and decentralization

Let's start with an obvious one: the problem of centralized vs distributed control structures. If I ask you what's a better org structure: a command-and-control hierarchy or a flat organization, most people have been indoctrinated to say the latter. Similarly if I ask whether you should have an old crusty centralized database or a fancy distributed database, everyone wants to build the latter. If you're an SRE and we start talking about pets and cattle, you always vote for cattle. You'd laugh at me if I suggested using anything but a distributed software version control system (ie. git). The future of money, I've heard, is distributed decentralized cryptocurrency. If you want to defeat censorship, you need a distributed social network. The trend is clear. What's to debate?

Well, real structures are more complicated than that. The best introductory article I know on this topic is Jo Freeman's The Tyranny of Structurelessness, which includes the famous quote: "This apparent lack of structure too often disguised an informal, unacknowledged and unaccountable leadership that was all the more pernicious because its very existence was denied."

"Informal, unacknowledged, and unaccountable" control is just as common in distributed computing systems as it is in human social systems.

The truth is, nearly every attempt to design a hierarchy-free, "flat" control system just moves the central control around until you can't see it anymore. Human structures all have leaders, whether implicit or explicit, and the explicit ones tend to be more diverse.

The web depends on centrally controlled DNS and centrally approved TLS certificate issuers; the global Internet depends on a small cabal who sorts out routing problems. Every blockchain depends on whoever decides if your preferred chain will fork this week, and whoever runs the popular exchanges, and whoever decides whether to arrest those people. Distributed radio networks depend on centralized government spectrum licenses. Democracy depends on someone enforcing your right to vote. Capitalism depends on someone enforcing the rules of a "free" marketplace.

At my first startup, we tried to run the development team as a flat organization, where everyone's opinions were listened to and everyone could debate the best way to do something. The overall consensus was that we mostly succeeded. But I was shocked when one of my co-workers said to me afterward: "Our team felt flat and egalitarian. But you can't ever forget that it was only that way because you forced it to be that way."

Truly distributed systems do exist. Earth's ecosystem is perhaps one (although it's becoming increasingly fragile and dependent on humans not to break it). Truly distributed databases using Raft consensus or similar algorithms certainly exist and work. Distributed version control (like git) really is distributed, although we ironically end up re-centralizing our usage of it through something like Github.

CAP theorem is perhaps the best-known statement of the tradeoffs in distributed systems, between consistency, availability, and "partition tolerance." Normally we think of the CAP theorem as applying to databases, but it applies to all distributed systems. Centralized databases do well at consistency and availability, but suck at partition tolerance; so do authoritarian government structures.

In systems design, there is rarely a single right answer that applies everywhere. But with centralized vs distributed systems, my rule of thumb is to do exactly what Jo Freeman suggested: at least make sure the control structure is explicit. When it's explicit, you can debug it.

Chicken-egg problems

Another archetypal systems design question is the "chicken-egg problem," which is short for: which came first, the chicken or the egg?

In case that's not a common question where you come from, the idea is eggs produce chickens, and chickens produce eggs. That's all fine once it's going, but what happened, back in ancient history? Was the very first step in the first iteration an egg, or a chicken?

The question sounds silly and faux-philosophical at first, but there's a real answer and that answer applies to real problems in the business world.

The answer to the riddle is "neither"; unless you're a Bible literalist, you can't trace back to the Original Chicken that laid the Original Egg. Instead there was probably a chicken-like bird that laid a mostly egg-ish egg, and before that, there were millions of years of evolution, going all the way back to single-celled organisms and whatever phenomenon first spawned those. What came "first"? All that other stuff.

Chicken-egg problems appear all the time when building software or launching products. Which came first, HTML5 web browsers or HTML5 web content? Neither, of course. They evolved in loose synchronization, tracing back to the first HTML experiments and way before HTML itself, growing slowly and then quickly in popularity along the way.

I refer to chicken-egg problems a lot because designers are oblivious to them a lot. Here are some famous chicken-egg problems:

• Electrical distribution networks
• Phone and fax technologies
• The Internet
• IPv6
• Every social network (who will use it if nobody is using it?)
• CDs, DVDs, and Blu-Ray vs HD DVD
• HDTV (1080p etc), 4k TV, 8k TV, 3D TV
• Interstate highways
• Company towns (usually built around a single industry)
• Ivy league universities (could you start a new one?)
• Every new video game console
• Every desktop OS, phone OS, and app store

The defining characteristic of a chicken-egg technology or product is that it's not useful to you unless other people use it. Since adopting new technology isn't free (in dollars, or time, or both), people aren't likely to adopt it unless they can see some value, but until they do, the value isn't there, so they don't. A conundrum.

It's remarkable to me how many dreamers think they can simply outwait the problem ("it'll catch on eventually!") or outspend the problem ("my new mobile OS will be great, we'll just subsidize a few million phones"). And how many people think getting past a chicken-egg problem, or not, is just luck.

But no! Just like with real chickens and real eggs, there's a way to do it by bootstrapping from something smaller. The main techniques are to lower the cost of adoption, and to deliver more value even when there are fewer users.

Video game console makers (Nintendo, Sony, Microsoft) have become skilled at this; they're the only ones I know who do it on purpose every few years. Some tricks they use are:

• Subsidizing the cost of early console sales.
• Backward compatibility, so people who buy can use older games even before there's much native content.
• Games that are "mostly the same" but "look better" on the new console.
• Compatible gamepads between generations, so developers can port old games more easily.
• "Exclusive launch titles": co-marketing that ensures there's value up front for consumers (new games!) and for content producers (subsidies, free advertising, higher prices).

In contrast, the designs that baffle me the most are ones that absolutely ignore the chicken-egg problem. Firefox and Ubuntu phones, distributed open source social networks, alternative app stores, Linux on the desktop, Netflix competitors.

Followers of this diary have already seen me rant about IPv6: it provides nearly no value to anyone until it is 100% deployed (so we can finally shut down IPv4!), but costs immediately in added complexity and maintenance (building and running a whole parallel Internet). Could IPv6 have been rolled out faster, if the designers had prioritized unwinding the chicken-egg problem? Absolutely yes. But they didn't acknowledge it as the absolute core of their design problem, the way Android, Xbox, Blu-Ray, and Facebook did.

If your product or company has a chicken-egg problem, and you can't clearly spell out your concrete plan for solving it, then investors definitely should not invest in your company. Solving the chicken-egg problem should be the first thing on your list, not some afterthought.

By the way, while we're here, there are even more advanced versions of the chicken-egg problem. Facebook or faxes are the basic form: the more people who use Facebook or have a fax machine, the more value all those users get from each other.

The next level up is a two-sided market, such as Uber or Ebay. Nobody can get a ride from Uber unless there are drivers; but drivers don't want to work for Uber unless they can get work. Uber has to attract both kinds of users (and worse: in the same geographic region! at the same time of day!) before either kind gets anything from the deal. This is hard. They decided to spend their way to success, although even Uber was careful to do so only in a few markets at a time, especially at first.

The most difficult level I know is a three-sided market. For example, UberEats connects consumers, drivers, and restaurants. Getting a three-sided market rolling is insanely complicated, expensive, and failure-prone. I would never attempt it myself, so I'm impressed at the people who try. UberEats had a head start since Uber had consumers and drivers in their network already, and only needed to add "one more side" to their market. Most of their competitors had to attract all three sides just to start. Whoa.

If you're building a one-sided, two-sided, or three-sided market, you'd better understand systems design, chickens, and eggs.

Second-system effect

Taking a detour from business, let's move to an issue that engineers experience more directly: second-system effect, a term that comes from the excellent book, The Mythical Man-Month, by Fred Brooks.

Second system effect arises through the following steps:

• An initial product starts small and is built incrementally, starting with a low budget and a few users.
• Over time, the product gains popularity and becomes profitable.
• The system evolves, getting more and more hacks on top, and early design tradeoffs start to be a bottleneck.
• The engineers figure out a new design that would fix all the mistakes we know about, plus more! (And they're probably right.)
• Since the product is already popular, it's easy to justify spending the time to "do it right this time" and "build a strong platform for the next 10 years." So a project is launched to rewrite everything from scratch. It's expected to take several months, maybe a couple of years, and a big engineering team.

Sound familiar? People were trying this back in 1975 when the book was written, and they're still trying it now. It rarely goes well; even when it does work, it's incredibly painful.

25 years after the book, Joel Spolsky wrote Things you should never do, part 1 about the company-destroying effect of Netscape/Mozilla trying this. "They did it by making the single worst strategic mistake that any software company can make: they decided to rewrite the code from scratch."

[Update 2020-12-28: I mention Joel's now-20-year-old article not because Mozilla was such a landmark example, but because it's such a great article.]

Some other examples of second system effect are IPv6, Python 3, Perl 6, the Plan9 OS, and the United States system of government.

The results are remarkably consistent:

• The project takes longer than expected to reach feature parity.
• The new design often does solve the architectural problems in the original; however, it unexpectedly creates new architectural problems that weren't in the original.
• Development time is split (or different developers are assigned) between maintaining the old system and launching the new system.
• As the project gets increasingly overdue, project managers are increasingly likely to shut down the old system to force users to switch to the new one, even though users still prefer the old one.

Second systems can be merely expensive, or they can bankrupt your company, or destroy your user community. The attention to Perl 6 severely weakened the progress of perl; the work on Python 3 fractured the python community for more than a decade (and still does); IPv6 is obstinately still trying to deprecate IPv4, 25 years later, even though the problems it was created to solve are largely obsolete.

As for solutions, there isn't much to say about the second system effect except you should do your utmost to prevent it; it's entirely self-inflicted. Refactor your code instead. Even if it seems like incrementalism will be more work... it's worth it. Maintaining two systems in parallel is a lot more expensive than you think.

In his book, Fred Brooks called it the "second" system on purpose, because it was his opinion that after experiencing it once, any designer will build their third and later systems more incrementally so they never have to go through that again. If you're lucky enough to learn from historical wisdom, perhaps even your second system won't suffer from this strategic error.

A more embarrassing related problem is when large companies try to build a replacement for their own first system, but the developers of the first system have left or have already learned their Second System Lesson and are not willing to play that game. Thus, a new team is assembled to build the replacement, without the experience of having built the first one, but with all the confidence of a group of users who are intimately experienced with its surface flaws. I don't even know what this phenomenon should be called; the vicarious second system effect? Anyway, my condolences if you find yourself building or using such a product. You can expect years of pain.

[Update 2020-12-28: someone reminded me that CADT ("cascade of attention-deficit teenagers") is probably related to this last phenomenon.]

Innovator's dilemmas

Let's finally talk about a systems design issue that's good news for your startup, albeit bad news for big companies. The Innovator's Dilemma is a great book by Clayton Christensen that discusses a fascinating phenomenon.

Innovator's dilemmas are so elegant and beautiful you can hardly believe they exist as such a repeatable abstraction. Here's the latest one I've heard about, via an Anandtech Article about Apple Silicon:

A summary of the Innovator's Dilemma is as follows:

• You (Intel in this case) make an awesome product in a highly profitable industry.
• Some crappy startup appears (ARM in this case) and makes a crappy competing product with crappy specs. The only thing they seem to have going for them is they can make some low-end garbage for cheap.
• As a big successful company, your whole business is optimized for improving profits and margins. Your hard-working employees realize that if they cede the ultra-low-end garbage portion of the market to this competitor, they'll have more time to spend on high-valued customers. As a bonus, your average margin goes up! Genius.
• The next year, your competitor's product gets just a little bit better, and you give up the new bottom of your market, and your margins and profits further improve. This cycle repeats, year after year. (We call this "retreating upmarket.")
• The crappy competitor has some kind of structural technical advantage that allows their performance (however you define performance; something relevant to your market) to improve, year over year, at a higher percentage rate than your product can. And/or their product can do something yours can't do at all (in ARM's case: power efficiency).
• Eventually, one year, the crappy competitor's product finally exceeds the performance metrics of your own product, and promptly blows your entire fucking company instantly to smithereens.

Hey now, we've started swearing, was that really called for? Yes, I think so. If I were an Intel executive looking at this chart and Apple's new laptops, I would be scared out of my mind right now. There is no more upmarket to retreat to. The competitor's product is better, and getting better faster than mine. The game is already over, and I didn't even realize I was playing.

What makes the Innovator's Dilemma so beautiful, from a systems design point of view, is the "dilemma" part. The dilemma comes from the fact that all large companies are heavily optimized to discard ideas that aren't as profitable as their existing core business. Any company that doesn't optimize like this fails; by definition their profitability would go down. So thousands of worker bees propose thousands of low-margin and high-margin projects, and the company discards the former and invests heavily in the latter (this is called "sustaining innovation" in the book), and they keep making more and more money, and all is well.

But this optimization creates a corporate political environment (aha, you see we're still talking about systems design?) where, for example, Intel could never create a product like ARM. A successful low-priced chip would take time, energy, and profitability away from the high-priced chips, and literally would have made Intel less successful for years of its history. Even once ARM appeared and their trendline of improvements was established, they still had lower margins, so competing with them would still cannibalize their own high-margin products, and worse, now ARM had a head start.

In case you're a big company reading this: the book has a few suggestions for what you can do to avoid this trap. But if you're Intel, you should have read the book a few years ago, not now.