Normal view

There are new articles available, click to refresh the page.
Before yesterdayUncategorized

Ok the pitch is Transformers meets vapid teen relationship drama

2 May 2025 at 09:41

I keep notes for TV show concepts. Because well I don’t know why because, but that’s not my point here. My point is that they often include really mundane robots?

FOR INSTANCE:

A show like The O.C. (or maybe Normal People for a heavier take, if you want) but it’s the kids of the Decepticons and the kids of the Autobots and they’re hanging out on the beach in LA and the most dramatic thing that happens is that somebody gets off with somebody else’s boyfriend at a party, and it doesn’t matter that there’s a colossal war between their parents a thousand miles away. Not only does it not matter but the general vibe is that their parents are kind of boomer losers and stuck in the past for even caring about whatever it is they care about.

Like the meta-narrative is about how the beefs of the old generation make no sense in the frame of reference of the new, but the propelling stories are all low stakes teen relationship drama.


SEPARATELY BUT SIMILARLY:

I think the world is ready for a gritty Lovejoy reboot. I’m thinking, also bring it into the Transformers cinematic universe and every antique is potentially a robot in disguise /cheeky wink to camera

For those who don’t know Lovejoy, it’s a Brit light entertainment drama from the 80s/90s. Ian McShane (before he became a Respected Actor) plays the eponymous Lovejoy, a loveable rogue antiques dealer, of all things, Han Solo of the home counties, and the episodes have zero existential jeopardy.

It was also the first I time I saw TV characters break the fourth wall; Lovejoy would turn to camera for a cheeky aside from time to time, before even the original UK House of Cards did it, and I wouldn’t be surprised if Lovejoy has been secretly written by Tom Stoppard.


So I don’t what led to me imagining both of these in the Transformers universe.

Sure, growing up I was a big fan of The Transformers: The Movie (1986). Pop trivia: Orson Welles’ final movie performance was planet-eating robot Unicron.

But I never owned Transformers toys as a kid, they weren’t a big part of my life. I had a single knock-off Transformers toy and it was a robot that transformed into a campervan.

So I think the appeal is something more general: what happens when you take the extraordinary and un-sci-fi it?

Like, the most fascinating part of the original 1984 Transformers cartoon show (YouTube, that’s the whole first season) is that

And what I really want to see here is the government committee meetings and the PowerPoints, everything from the escalation of concern against general scepticism about this bizarre incursion into the regular reality of the world, to pragmatically and hurriedly launching a public information campaign and briefing an incredulous graphic designer.

This is 50% because I prefer stories that happen in a hidden layer of today’s world (c.f. James Bond got worse as it became fantasy) and 50% a kind of personal cultural anticipation for the world of robots that we’re stepping into.

Robots are just going to be really normal, you know?


btw there was a RoboCop TV show in 1994 that had weekly episodes and was written as a kind of everyday cyberpunk police procedural: "It’s love at first “byte” when Lippencott enters Virtual Reality and falls for Diana Powers, while RoboCop is blackmailed by the kidnapper of his wife Nancy into stealing a top secret weapon that can literally break hearts."

Like WHAT? I need to see this.


ANOTHER:

A Knight Rider reboot where the guy straps an Alexa onto a stolen Waymo then goes about hitting on girls and solving whatever limited set of situations you can solve with a robot car you half-assed in your garage, making sure someone gets to an interview on time or collecting a late DoorDash and such.


Something that smacks different about my concepts for TV shows, going back and unearthing them from my notes, is that - before - half the reason they made me laugh is because they could never happen.

But today, give it a year, and I’ll be able to put the prompt into an AI and get 30 minutes of plot and video to watch on the train.

Which does make me wonder: in the face of narrative abundance, what is the appeal of media? Like, it will still be appealing, I know that, but I’d like to understand how precisely.


Not all my TV show concepts are about robots.

Game of Thrones has an unsatisfyingly sloppy ending but it opens up an intriguing grimdark GoT TNG.

So reboot Game of Thrones a thousand years in the future where Bryn has fully become this ancient immortal time-travelling god-tyrant - which is how the series ended, if you don’t remember - keeping the whole world of Westeros in cultural stasis, and some people working for him finally realise he’s evil actually and they discover a lost cache of dragon eggs and start building the resistance.

But they have to take precautions so Bryn doesn’t retcon them out of existence - he breaks the fourth wall by having control over reboots and the TV commissioning process - and occasionally that happens and reality just shifts beneath their feet.

I would watch the heck out of that for, well, the first two episodes probably and then read all the season summaries on Wikipedia, which is how I tend to watch shows, pound for pound it’s a way more efficient way to consume TV, I commend it to you.


Auto-detected kinda similar posts:

Red caps and green beards

25 April 2025 at 17:45

Let’s take a meme’s eye view of MAGA fashion.

The hypothesis of the green beard (Wikipedia) is that there’s a way for some individual genes to signal their presence, which allows for selective social cooperation favouring that particular replicator – and the targets of this cooperation might be different to the usual blood-relationship cooperation that governs the (higher level) realm of individuals, and perhaps not even beneficial to the individuals themselves.

Richard Dawkins came up with this particular example of gene signalling in The Selfish Gene (1976): "I have a green beard and I will be altruistic to anyone else with green beard."

In the same book Dawkins also coined “meme” as the unit selfish replicator of culture; as a gene is in the biosphere so a meme is in the noosphere.

AND SO:

A shibboleth is a word that differentiates in-group from out-group. Here’s a list of shibboleths (Wikipedia).

btw an unlisted example: if a Brit says “college” rather than “uni” then they’re signalling they went to Oxbridge, and that’s a thumb-on-the-third-knuckle secret handshake if ever there was one.

Now, is this entirely voluntary by the individual or, from the meme’s point of view, could we see it as an expression of the selfish replicator? Potentially overriding other higher-level calculations for cooperation that we might regard as more “fair” or more “rational.”

i.e. shibboleths are green beards for memes?

Note that shibboleths don’t necessarily have to be kept secret.


Maybe shibboleths don’t even have to be words.

FOR INSTANCE: MEN’S FASHION.

At Bloomberg, by feisty men’s fashion history expert and X/Twitter skirmisher Derek Guy, the MAGA male aesthetic: "If you’ve noticed a certain look common to the manosphere, you’re not mistaken. A visual identity has taken hold."

Muscle men in tight shirts.

[Manfluencers] treated masculinity as a game: Confidence could be rehearsed, women were goals to be unlocked, and clothes were tools for climbing the socio-sexual hierarchy.

It’s this heady mix of performance, sexuality, power and hustle:

On YouTube and in early fitness forums, ex-soldiers, bodybuilders and amateur life coaches used their physiques as proof of transformation. David Goggins, Zyzz and Jocko Willink - fitness guys who doubled as motivational speakers - cast the body as both weapon and wisdom. Hustle culture took these ideas further. Tim Ferriss, known for his 4-Hour self-help books, wrote about cold plunges and productivity hacks.

What does it look like?

the beachside bravado of the Venice bodybuilders, the greed-soaked tailoring of 1980s finance and the tight-fitting clothes once labeled metrosexual. Today’s fixation on muscularity, discipline and traditional masculine aesthetics feels like a new chapter in that same historical cycle.

(Paywall-busting link.)

MEANWHILE:

Mar-a-Lago face (The Week): "copious use of Botox, a Miami-bronze tan, puffy lips and silky-smooth skin."

It’s a sculpted, high contrast facial appearance, the extreme opposite of anything that pretends to “natural.” A hyperreality adopted in the main, it seems, by conservative women in the US.

Take a look at this screen-grab from Vogue Business with many before/after comparisons (Bruce Sterling on tumblr).


What strikes me about both MAGA men’s fashion and Mar-a-Lago face is that they’re costly.

It’s the Make America Great Again red cap amped up: a conspicuous shibboleth that signals in-group allegiance but at the high cost of being excluded from other cooperation networks.

(I’m a Brit and I live in London. But I have a favourite red cricket cap, and I was very much informed I shouldn’t wear it when I last visited friends on the west coast.)

Net-net is the look actually good for the individual? I’m not sure.

So I think Mar-A-Lago face is a pretty good candidate for being a green beard meme.

Assuming it’s not a state sponsored fashion attack.


y’know,

I used to wonder about those fashion anecdotes like oh we do such-and-such because king whatever did it and everyone copied him and I was always like, reeeeeally?

Such as not buttoning the final button of your waistcoat (GQ): "when the future Edward VII was Prince of Wales he became so fat that he couldn’t do up the bottom button on his waistcoat so court followed suit to make him feel better about his body image."

Or the flower in the lapel buttonhole tradition which was invented by Prince Albert.

We can say “oh yeah, mimicry” or “oh yeah, fashion” but I feel like those words just paper over whatever the underlying mechanism might be.

Another: Henry VIII and codpieces.

A suit of the king’s armor, boasting a bulbous codpiece weighing more than two and a half pounds, is still on display at the Tower.

Life in 1530s, the dawning years of the Reformation: the brutal dissolution of the monasteries and repatriating power from Rome a.k.a. Make England Great Again.

So while the red cap and the manfluencer bulging shirt is the green beard peacocking of the MAGA meme, its method of reproduction is to entice you to join its complex of political and sexual power.

Analogy: cat virus Toxoplasma-infected men and women are rated as more attractive.


But why now?

I continue believe that the current political axis is not right vs left nor even authoritarian vs liberal, but oikos vs polis (2021). It’s the axis of differing cooperation rules:

  • oikos is “people like me” (individual choice)
  • polis is “rule of law” (delegated choice).

And although oikos might sound like it’s just mafia-style politics and corrupt nepotism, it is also high family values and neighbourhood spirit. We hold those values in high regard! This isn’t a bad/good axis, both approaches have their strengths and weaknesses.

But we can also observe this:

Codpieces and MAGA hats as successful green beard memes are parasitic on high oikos societies, and not possible when equality before the law dominates.

Which I think says something about how 1525 is like 2025, and why Wolf Hall was so resonant during the Brexit surge.


Anyway I did some light research to see if codpieces were making a comeback and long story short, my Amazon recommendations are a mess now, don’t judge me it was for science.


More posts tagged: fashion-statement (9), vibe-is-real (10).

Beach daydreams, lost at sea

16 April 2025 at 13:55

The gentle sound of the surf fills the soul.

No frequency left untouched. But never colouring outside the lines; never too loud.

It occupies the senses and allows detachment without dissociation. It lifts the mind just a fraction, a gliding ballroom aquaplane through thoughts – an environmentally stimulated free association.


The cricket ground in north London, Lord’s, the home of cricket, is genteel and perhaps not as shallow as other sports stadiums but shallower certainly than, say, the Gabba in Brisbane which is a bowl-like colosseum.

The ambience is known as the Lord’s hum, and like the waves on the shore it is fractionally hypnotising, a tonic that takes you out of the day to day and allows you to focus without concentrating and idle without becoming bored.

What better argument that humans used to be shore-dwelling fisher-folk, paddling the shores, browsing the rock pools, and gazing out to sea? A happy place.

Likewise: golf maybe? The low, rolling green grass, lush with life; trees to find animals to hunt and lakes for water. Looking out from the clubhouse, the links landscape must tug something ancient: we could live here.


I visited a swannery a few years back, 600 swans nesting and living wing to wing.

It’s uncanny to meander through the trees zig-zagging between nest after nest, and an actually a little alarming to see hundreds all together, synchronously bobbing on the inlet. (The swannery is on the coast.) Those alien eyes, the last dinosaurs.

I don’t know how the swanherd monks who established the swannery 700 years ago enticed these famously solitary birds to pack so closely.

But I wouldn’t be surprised if it were to do with the lapping waves on the shoreline.

The sound packs us in wool and allows us to pack together. Here on the sunbed I can hear people speaking a few meters away only as a murmur, and my family right next to me - who are reading, right now - when they speak, as clear as day.

What an advantage for a beach society of early humans, to live closely without stepping on each other’s toes.


Watching a fire, too, fills the eyes and releases the imagination to drift.

The flicker tells us stories. Not like the high-fructose corn syrup flicker of video which similarly draws us in but then submerges us. Flames leave space for our own personality and thoughts.

Or ASMR (previously), which replicates the surf but, like television drowns the self.

No, the back and forth of the waves is more homely hearth than YouTube; more cloud-gazing than TV.

Cloud-gazing being another ancient occupation that occupies without overwhelming.

How kind that keeping a watch on the weather should be so pleasant.

And yet another is stone-knapping. Watch a young kid pick up and crack two toys together, before they can speak even, feeling the heft in their hands, turning and smacking, turning and rapping, and you get an idea that these were the jobs of all of us for a million years, our brains contoured to quietly enjoy time spent in the practice of these vital activities, tumbled and smoothed on the riverbed of deep time.

Memories of the Pleistocene.


Language too.

Before we fished the shallows we were tree-dwellers.

Arboreal animals ascending trunks, branching out ever more precariously – before taking a leap – grabbing hold and working back to sure safety close in with another tree. Or flying through the canopy, branch after branch after branch after –

And doesn’t that sound like language, or rather whatever principle it is that underpins language and metaphor and science, or whatever model of the world it is that we hold inside us to understand it?

A load-bearing fossil of the ancestral forest every time we grasp for a word and reach out for a new idea.

Venturing out from one’s home tree… spying a leaf of a particular shape, generalising to the fruit of that tree, bringing to mind that harvest and everything it would mean, moving from bud to branch to trunk to whatever ur-trunk it is inside and beyond the specific tree, the ur-tree of that species, the Platonic type, and we become apes that are natural philosophers.

Broca’s area biologists, flying along chains of branching and leaf tip-interconnected concepts, making a leap from the final branch - a flight of fancy - across open air to somewhere new.


Even before we carried the trees in our brains we took the billion-years-ago proto-ocean in our blood and our cells.

We came onto the land and carried the salty sea with us in our bodies, delicate parcels wrapped with cell membranes, and the swell and the wash remembered with each beat of our hearts, the dark ocean that was once our home still inside us, rocking us in its invisible waves.

I can hear it, across a billion years of history, inside me, and across a short stretch of beach, under this bright blue sky and this square red beach umbrella, hot sun drying the sand on my feet, the lapping of the waves on the shore, the sea in my blood and my ears. It fills us.


Auto-detected kinda similar posts:

tl;dr I ran a marathon at the weekend and it was hard

10 April 2025 at 20:55

So I ran a marathon at the weekend.

I was going to say “my first marathon” - which it was, my first I mean - but that makes it sound like there are going to be many, which uh I don’t intend so much. Let’s see.

I expected a marathon to be hard. It was harder than I expected in a super interesting way.

Goals: (a) get round and (b) hit a target time of 3:45 if poss.

My time was 3 hours 40 minutes 12 seconds. I’m proud of that ngl (and wish I’d done better).

Why is a marathon hard? Energy.

26 miles is an interesting distance because it’s more than your body can do naturally without being genetically freakish.

I wouldn’t be me if I didn’t treat this whole thing as an excuse to get really nerdy about my body as a systems problem.

So I’ve been learning how energy works in the body, and as I understand it when you’re using your muscles you’ve got three energy sources.

  • Glucose. In your blood, comes from food. There’s barely any calories here – but if you eat, that’s where it goes first.
  • Fat. You can burn fat but it’s slow… which is a shame because 90% of your body’s stored energy is in the form of fat. So it contributes only about 10-20% to the mix, more as a race goes on. Really experienced runners will have more fat burning in the mix.
  • Glycogen. Glycogen is the main source of energy. It’s stored glucose, and converts to glucose before it can be used. Your body carries about 1,600–2,000 calories of glycogen.

Glycogen is stored mainly in the muscles. There are up to 1600 calories in the muscles, and it can only be used for those muscles. Also about 400 calories in the liver, and that can be transported anywhere.

One thing I learned about glycogen which I’d never really thought about is that energy has a weight in your body. Of course it does.

Glycogen is about 4 calories per 1 gram. But to store it, it has to be bound to water, another 3g per gram. So your 2000 calories of glycogen is about 2kg (4.4lb) weight, and you lose that as you run.

(That water aspect is itself fascinating. Using glycogen liberates water, so you self-hydrate to a degree.)

Now once you add all of this up, you’ve got a deficit.

Rule of thumb you burn 100 calories per mile. i.e. marathon = 2,600 calories.

(For comparison I eat a little under 1,800 calories/day.)

So this is mathematically not possible: your carrying capacity is max 2,000 calories glycogen, another couple hundred contributed by burning fat. Still 400 calories to find.

How do you close the gap?

  • Train and get biomechanically more efficient at moving
  • Train and get better at storing and converting glycogen (up to that 2,000 calories level)
  • Train while fasting to improve fat burning
  • Carb load: you can temporarily store more glycogen than usual, so load up with carbs 48 hours before a race
  • Eat on the way: running gels contribute 80 calories each of glucose into your blood, and that slows glycogen conversion. I had four gels on the way round. They take about 30 minutes apiece to hit your system.

Part of what’s going on during training is all these bodily adaptations and it’s amazing to see it happening. But yeah, you can’t do a marathon without doing all of the above, the distance is just enough to get you into that deficit territory. I used to think that 26 miles was a kinda arbitrary distance, and now I realise that it makes it a finely poised competition you can have with yourself.

I haven’t talked about electrolytes and water, and that’s because they never really popped up as issues for me during training.

But that’s what bit me.

Training and will

I think I’ve said before that training for me is about four things: heart, puff, muscles, will.

I help my “long will” by maintaining a training streak. (And part of what I’m training in that time is my “short will”, which is the will to keep going in the moment even when I’ve reverted to my stupid glucose-starved monkey brain.)

My streak: I’ve been training at 20 miles a week for about 10 months, since 27 May 2024, and that’s doubled as I’ve got closer to this race.

The main thing I’ve battled has been to come to terms with the fact that my days of personal bests are mostly behind me. Even when I’m at peak fitness right now, I’m a long way off where I was at peak fitness five years ago.

My first marathon was originally going to be back in 2020 and it was cancelled, being 3 weeks into the first pandemic lockdown. It was enormously disappointing and, for a variety of reasons, I was on for a great time back then.

Since then I’ve had knee injuries, Achilles injuries, IT band imbalances pulling my hip (and knee) off, months getting my puff back post bouts of Covid… and of course 5 years older.

btw I’ll mention that knee injury again.

But yeah, since May 2024 I’ve kept up this streak. I’ve taken part in one half marathon race in that period, which is a tricky moment in itself: training up to it is motivating, but extrinsic motivation eats intrinsic motivation for breakfast and it takes will to re-find motivation after the race is done. And my intrinsic motivation has had to shift from ever-increasing PBs to… something else. Enjoying being outside I guess!

Anyway I made it through training. I was pretty much done with it by the end tho.

The race itself

By the time I decided that I was going to do an actual race, and it seemed like I probably wouldn’t get injured again, and I’d be fit enough, all the high-profile marathons were full up. But Southampton had space, and you know what, that’s not so bad. It’s the city I grew up in.

The course looked hilarious. I’m running past places that I would hang out as a kid, going through parks I used to just kind of loiter in when I was 15, past the casino my mate worked at, through the stadium of the local football team, across the common I did cross-country at school, those kind of places. So it was a race of old haunts, a nice place to be.

The race went 81% well.

The start was a little chaotic and congested.

My half marathon pace felt very comfortable and so I was ticking along doing that.

What was tricky was that at 21 miles I felt pain in my knee, a bit like somebody jabbing into it HARD with a screwdriver and I couldn’t put any weight on it. My muscles had tightened up to the point it was pulling my knee out, a recurrence of that old knee injury. (If I’m being honest it had been niggling for two weeks and, if it hadn’t been for race day, I never would have risked a long run.)

So, after stopping a couple of times to stretch that out, cursing and grimacing, which at least kind of got me moving again, then cramp started.

My mental model of what occurred is that my knee made me stop, then my legs forgot their rhythm and fell into a new and dysfunctional pattern.

Now I’ve had cramp before, I suppose, but never like this.

Both calves. I can only describe cramp as having very hot animals like maybe like the size of mice forcing and crawling around under my skin all over my calves and the burning pain was extraordinary. I was barely able to walk so I would walk stubbornly and then start trying to jog a little, then the roaming boiling knots would come back and overwhelm me. Uphill was especially bad.

At a water station, a volunteer there checked in with me and he offered to help with my calves. He did something which I didn’t expect, which was not to massage them, but basically to run his hands up and down my calves, one at a time, not too hard, and that seemed to reboot them.

A superhero.

What is cramp anyway?

So I looked into cramp afterwards, and it is not, as I grew up thinking, lactate buildup.

Cramp is neuromuscular fatigue and (from my very, very partial understanding) it seems like what happens is the nerve junctions between the ends of the nerve and the muscles, these little junctions run out of the electrical juice the nerves sit in, and the juice is a combination of salt and water. The salt being your electrolytes of course, and they just dry out or stop conducting, and your nerves can’t communicate with the muscles properly, and your muscles go haywire interpreting the static.

It was wild after the race when I lay down and my wife was massaging my calves because of the cramp pain and she said to me “what are you doing with your legs?”

I was like “what do you mean?” – I couldn’t feel anything. Then I looked at my calves and I can only describe it as some kind of Cronenbergesque manifestation of worms crawling underneath my skin.

You see every so often those time-lapse videos of chemical reactions with bimodal stable states in a petri dish doing those wild, curved, radiating geometric patterns, Turing patterns. My muscles were doing the same, writhing underneath my skin, that’s all I could think of to look at it. This is what neuromuscular fatigue is, this misfiring.

It seems like when I was helped by that kind person at the water station, the movement he was doing on my calves was indeed like resetting the muscles – a new input stimulus to override the misfiring signals, and give my legs space to remember what they were supposed to be doing.

My model of what happened (stopping because of my knee, legs forgetting what to do) feels correct to me. My calves were overworked, compensating for my knee, then got confused. I have a picture of my legs as octopus tentacles, these semi-independent limbs that have their own rhythms and habits and - when push comes to shove - I have to negotiate with them.

Now the main factor, really, is that it was the second hot day of the year.

It was 20C by the time I finished, and I’ve done most of my training between -5C to +5C. It’s only recently hit 12C. So I’ve barely done any warm day training at all.

So I will have been dehydrated I’m sure, and lost more salt than I was expecting, those juicy nerve junctions parched and grainy. Then it all happened, this unfamiliar cascade.

I had enough energy though.

Now my monkey brain…

I hobbled for a mile, managed to get my knee stretched out, reset my calves, and found a pace slow enough that cramp wouldn’t overtake me but fast enough that I could still call it running, and red-lined the final three miles like gunning a car in second gear and made it over the finish line.

So would I do it again?

I went into the weekend saying this was going to be my one marathon. For closure because of the one I didn’t get to do before.

It was harder than I expected, even knowing it would be hard.

After I’d done my final 21 mile training run (you don’t go any further than that, then you taper for two weeks), I was like: I don’t know if I have another five miles left in me. It was the first time in my training that I felt doubt.

I assumed that I’d cover that 5 mile gap with a combo of energy and will. i.e. the ingredients that had got me this far.

But they weren’t what mattered. The experience beyond 21 miles was uncharted.

What this says to me is that when you go to extremes, new things happen?

There are new experiences to be found, when you go past your limits, which aren’t like the old ones scaled up. They’re something distinct. Unanticipated and unanticipatable.

I don’t know how to explain it better than that! It seems to me that this is true of so many things. The only way to know what it’s like to run mile 22 is to run 21 miles first, you can’t shortcut your way there.

The only way to know what you’ll write as your million-and-first word is first to write a million words.

Etc.

So would I do it again? I went into it saying no. One and done.

That said, lying there under the blue sky watching my muscles writhe around, in the park in Southampton city centre, there was a part of me going, I think I could have done better knowing what I know now about race strategy.

Actually I’m sure I could.

And there’s something I saw about myself, I think, at the end of the travel upriver, into the capillary-end heart of that journey, that place of pride and brutal will – and I can’t see it clearly from my vantage point of sitting at my laptop. I want to visit that place again.

Anyway.

Race time of 3:40, hit my goals, slightly better than actually. I got a medal (we all did). Where I came is number 249 out of 1,337. 62/199 if I look at my category, which is male veterans. Pleased, yes, and also would like to have done better.

And what an experience to experience my body as pure system, this kind of metabolic system which is just astounding and fascinating, and has so many nuances and edges; I’m learning a new landscape, a geography, this surface of a metabolic manifold of surprisingly few parameters really.

I feel so in tune with and in touch with myself. If I have any regrets about my running career it’s that I wish I’d started running earlier than my early 30s, and I wish I’d started taking it more seriously earlier than I did.

And that’s another lesson I can take more generally which is to try more and if I’m going to commit, then commit faster.

I’m on People and Blogs today

4 April 2025 at 14:00

I was interviewed for the 84th edition of People and Blogs, "the series where I ask interesting people to talk about themselves and their blogs."

So if you want to read about the backstory to this blog, or my creative process, or the technology (and my philosophy around it), then head on over. Lots of hyperlinks of course.

Read: P&B: Matt Webb.

A taster:

Given your experience, if you were to start a blog today, would you do anything differently?

If I were to start a blog today, I would start an email newsletter. And that would be a mistake.

Also: check out the archive. Loads of names you’ll recognise there. The questions are the same, every edition. Thank you Manu Moreale for the invitation to take part! What a great project.

I feel like I’ve talked about blogging a whole bunch, lately. What with the reflections on 25 years etc.


More posts tagged: meta (18).

Auto-detected kinda similar posts:

Between early computing and modern computing: some cultural histories

3 April 2025 at 19:00

My formative experiences of computing and the internet are now regarded as “history” haha

Which is BRILLIANT because I get to experience them all again, only via people who are doing the work to actually document and interpret it.

So, some links!


Cybercultural is by Richard MacManus who founded and edited ReadWriteWeb from 2003–2012. This was the period in which we all figured out that the web wasn’t a magazine, or a shop, or a TV, but a new kind of medium, with people and attention as the matter and energy of its physics. Anyway, he knows what he’s talking about, that’s what I mean.

I loved GeoCities in 1995: Building a Home Page on the Internet which has screenshots I’ve never seen, and the origin story of how and why_ they wanted to "give people the sense that they had a home on the internet."


The History of the Web by Jay Hoffman and updates twice monthly (you can get a newsletter). I’ve found the best way to explore is to scroll down the timeline and click on articles as you see them.

This history of Perl and Python, The Linguist and the Programmer, brought back so many memories. (And made remember hanging out on the perl usenet back in the day… Abigail!)

Then these two articles about early online publishing:

What I really appreciate about Hoffman’s writing is that there are many, many links in the Sources section for these articles. Thank you!


Folklore is an older project and still a great one: "Anecdotes about the development of Apple’s original Macintosh, and the people who made it."

I love reading about Mister Macintosh, a weird dumb idea (all the best ones are). From Steve Jobs:

Mr. Macintosh is a mysterious little man who lives inside each Macintosh. He pops up every once in a while, when you least expect it, and then winks at you and disappears again. It will be so quick that you won’t be sure if you saw him or not. We’ll plant references in the manuals to the legend of Mr. Macintosh, and no one will know if he’s real or not. …

One out of every thousand or two times that you pull down a menu, instead of the normal commands, you’ll get Mr. Macintosh, leaning against the wall of the menu.

Then you get the Roll Your Own Calculator Construction Set and the origin story of rounded rectangles.


50 Years of Text Games by Aaron A. Read was originally a Substack and you can still read it there: "A deep dive into text game history, from The Oregon Trail to A.I. Dungeon." One per year.

It’s more than that: it’s a lens to see the culture around computing over two important generations.

Every piece in the archive is gold. Two faves are 1990: LamdaMOO about multiplayer, co-created text gaming… and the social structures they had to discover. (Also the subject of the amazing and prescient My Tiny Life (1998) by Julian Dibbell, which I mention here.)

And 2017: Universal Paperclips about a super-intelligent AI that makes paperclips…

Buy the 50 Years of Text Games book. I own it. 100% worth getting a copy.


Benj Edwards, literally anything written by him. Not a project but a person, Edwards is a tech journalist and video game/tech historian. So go browse his archive of deep-dives and also on Ars Technica.

I really enjoyed Myst at 25: How it changed gaming, created addicts, and made enemies (and also it reminded me about Leisure Suit Larry… what an era).

This is a classic: Who Needs GPS? The Forgotten Story of Etak’s Amazing 1985 Car Navigation System – how do you build satnav with no satellites? In other articles Edwards extended the story to the earlier video games and to Google Maps; I connected the dots here.


So from time to time I lecture on folktales from early computing. (A highlight was doing the whole 3 x 1 hour series on successive nights at a tech conference. Evening entertainment!)

The reason I love the early web - outside my usual period - is because (a) I was there, a little bit, and (b) it’s when computing stopped being primarily about the military, and business, and academia and became just about… life.

Let’s define the period as the September that never ended and everything after: in 1993/94, commercial ISPs got people online en masse, and the flood of newbs couldn’t be absorbed by existing cyberculture. In previous Septembers, students in the new academic year had netiquette beaten into them eventually but… so we’re the visigoths that trashed Rome. I love it.

But mainly, (c), this reason: we were all still figuring it out.

So there are all these roads not taken that are none-the-less fascinating, and I use them as references all the time in my work and I would love to dig deeper, and discover what I missed at the time.

i.e. I would love to follow along if you have a history project that hits on Kai’s Power Tools or SmarterChild or Taligent or net.art or… you know what I mean. lmk?


More posts tagged: computing-history (10).

Auto-detected kinda similar posts:

Filtered for the rise of the well-dressed robots

28 March 2025 at 16:40

1.

Humanoid robots.

I feel like something that isn’t super well appreciated in civilian world is quite how quickly humanoid robots will arrive, and quite how good they will be.

See, there are a bunch of humanoid robotics companies now. Here are five robots:

  • Atlas by Boston Dynamicsdoing the hard engineering graft of movement for years. Look at the vid and the uncanny way the robot stands up.
  • Figurethe hot, glossy US startup building general purpose humanoids with a big mission (but actually going after factory jobs).
  • NEO by 1Xanother gorgeous general purpose humanoid, this one out of Norway and upholstered in plump beige fabric.
  • Digit by Agility Roboticshumanoid warehouse robots.
  • Walker S1 by UBTECHI don’t have great knowledge of the Chinese market, this is just one company. A big one though.

The way I understand it, there have been three major challenges with robots in the real world: mechanical engineering, perception, and instruction following.

Engineering has been solved for a while; perception mostly works, though not understanding. Instruction following, including contextual awareness, task sequencing, and safety… that was a work in progress. Solved at a stroke by gen-AI.

So, as of a couple years ago, there is clear line of sight to humanoid robots in the market. Research done, development phase: go.

Now we’re waiting for the shoe to drop.

2.

Robots and nails.

All physical automation gets called a robot now.

Here’s a new fully automated manicure machine (10beauty.co).

All their press refers to it as a robot. But it doesn’t have any legs. It’s a clever box you put your hand inside and it looks like a Bene Gesserit pain device, only instead of fear being the mind-killer, it softens your cuticles.

But this is why the humanoid form is good for robots: you shouldn’t need to develop a dedicated machine for manicures.

The friction for any technology during deployment phase is the build-out for integrations. For robots, integrations = operating in the physical world.

e.g. it will be hard and slow to bring AI-powered automation to warehouses and production lines because the warehouses and production lines will need to be re-tooled. Slow. Expensive.

Unless! Use the human form and you can pick up existing tools and press existing buttons.

At home, why buy a Roomba if your aging-in-place domestic robot assistant can pick up the exact same hoover that you own anyway? And carry it up and down the stairs too.

KINDA RELATED:

This robot hand has fingernails (Robotic Gizmos):

Thanks to this approach, robots can pick up coins and cards and achieve better tactile sensing. These fingernails also assist with dynamic manipulation.

Clever.

Also… I wonder if the fingernails can also break off and then be replaced?

There’s utility in having a sacrificial part of the body, when you’re interacting with the physical world, whether it’s nails on your fingers or the whiskers of a cat. (Cat whiskers are deep-wired tactile sensors that easily regrow when damaged. An intelligent design!)

3.

Economic takeoff.

Back in 2022, factories crossed the robot takeoff threshold.

…a nascent trend of offering robots as a service - similar to the subscription models offered by software makers, wherein customers pay monthly or annual use fees rather than purchasing the products - is opening opportunities to even small companies. That financial model is what led Thomson to embrace automation. The company has robots on 27 of its 89 molding machines and plans to add more. …

Thomson pays for the installed machines by the hour, at a cost that’s less than hiring a human employee … “I’m paying $10 to $12 an hour for a robot that is replacing a position that I was paying $15 to $18 plus fringe benefits.”

Grimace-emoji here.

This is when the market snowball effect kicks in… when cost-benefit is positive, even a little, that increases volume, which decreases cost, which makes cost-benefit more positive, which… and so it accelerates.

Figure publishes its Master Plan:

We believe humanoids will revolutionize a variety of industries, from corporate labor roles (3+ billion humans), to assisting individuals in the home (2+ billion), to caring for the elderly (~1 billion), and to building new worlds on other planets. However, our first applications will be in industries such as manufacturing, shipping and logistics, warehousing, and retail, where labor shortages are the most severe.

And all of this is what I mean when I say people are sleeping on humanoid robots coming into the world.

It’s not going to be like ChatGPT and that explosion into cultural awareness. More like the rise of Waymo and robot cars which are startlingly good and startlingly mundane.

And it might be a year or it might be five.

And before we know it, a ton of people are out of work, on top of all the people who no longer have call centre jobs, and 90% of the commercial artists, and there aren’t any jobs for them to go to, and that fact really, really ought to be the subject of a lot of policy thought, like today.

4.

Who will make clothes for robots?

Not meaning to be flippant but at least some of the future new jobs will be in robot fashion?

I look at those gorgeous upholstered NEO Gamma robots by 1X"NEO’s Knit Suit is soft to the touch and flexible for dynamic movements."

And tasteful beige knitwear wouldn’t survive a day in my home.

Even those high-capital-cost factory robots are going to have their elegant metal bodies scratched and dented.

So… they need clothes?

Protective workwear, sure. You need something washable and semi-sacrificial for physical work.

But also: the clothes that a robot wears are part of its brand. They feed into how we relate to it, how we remember it, how we feel about it.

Which sounds a lot like fashion?

I’m reminded of how the Apollo spacesuits were designed and manufactured by a commercial underwear brand:

The Apollo suits were blends of cutting-edge technology and Old World craftsmanship. Each suit was hand-built by seamstresses who had to be extraordinarily precise; a stitching error as small as 1/32 inch could mean the difference between a space-worthy suit and a reject. …

A division of the company that manufactured Playtex bras and girdles, ILC had engineers who understood a thing or two about rubber garments. They invented a bellowslike joint called a convolute out of neoprene reinforced with nylon tricot that allowed an astronaut to bend at the shoulders, elbows, knees, hips and ankles with relatively little effort. Steel aircraft cables were used throughout the suit to absorb tension forces and help maintain its shape under pressure.

So I don’t imagine 1X, Figure, Boston Dynamics and the rest will be creating their own robot clothes. Not their wheelhouse. They’ll partner with existing fashion brands.

Which means there’ll be a new need for new fashion designers. Fashion designers with experience in robot apparel, serving well-dressed artificial humans in head-to-toe haute couture.

Start brushing up your portfolios kids.


More posts tagged: fashion-statement (9), filtered-for (113), robots (10).

Auto-detected kinda similar posts:

Diane, I wrote a lecture by talking about it

20 March 2025 at 21:17

Diane, it’s Thursday and I’ve been figuring out how transcription fits into my everyday work. I had to make up a character to make it make sense, as I’ll say.

So I’m doing a little teaching this week and next. A two week mini module at AHO (Oslo) on designing interfaces for co-agency.

My lecture for next Monday has been… up in the air. I’ve been indecisive because the topic will prime how the students approach the brief for the week, so I want to get it right.

Clarity came (as always) during a run.

So, between picking up coffee and getting home, I roughed out the lecture outline.

It’s neat to talk into my Apple Watch using Whisper Memos. It’s a straightforward app: it "turns your ramblings into paragraphed articles, and emails them to you" – but the useful bit is that it works without my phone.

(I don’t carry my phone when I run and besides my hands are often too cold to type after.)

Then, when I got home, I said to Claude:

I’m writing a talk. please take this raw transcript and structure it into a high-level outline so I can work on it. do not add any of your own material, just structure my verbal notes

And pasted in my transcript from the email.

The result was great. I have a good outline that I can develop into a deck, and zero hallucinations.

btw this is not new! Lots of people talk to ChatGPT or whatever and get it to structure their work.

BUT: it’s not a very reproducible workflow, you know?

See, I’m not always writing a talk. I don’t like having to come up with a new and different prompt, each time, about what the AI should do with the transcript. I am lazy (proudly, virtuously). Don’t make me think.


The solution is to embed the instructions at the same time I’m talking to my watch.

Like, I want to be able to say at the end of my rambling about the lecture structure: hey uh take all of that and turn it into um a structured outline – but I can’t.

More generally, I want to be able to include instructions like “oh that point just now should actually be moved to the first section” and not have anything misinterpreted. I can’t just arbitrarily execute the entire transcript. Who knows what might happen.

It’s the usual out-of-band problem (Wikipedia): how do you include data and also data processing instructions on the same channel?

I thought about using an explicit delimiter. For instance the genius Cursorless voice interface uses a tongue click to fluidly enter command mode so you can easily multiplex speech and coding. (This YouTube video demonstrates the clicks and here’s how to set it up.)

But… tongue clicks didn’t feel right for this. The register change is too abrupt.

Diane, how else can I signify tick-tocking between data and instructions?


My generic prompt to Claude, used every time, is now:

you are Diane, my secretary. please take this raw verbal transcript and clean it up. do not add any of your own material. because you are Diane, also follow any instructions addressed to you in the transcript and perform those instructions

[paste in transcript]

Which means, when I’m talking through my lecture outline, I now finish by saying:

ok Diane I think that’s it. it’s a talk, so please structure all of that into a high level outline so I can work on it. thanks.

And I can mix in instructions like: oh Diane I meant to include that point in the last section. Please move it.

It works super well.


Why Diane?

Because I grew up watching Twin Peaks and FBI agent Dale Cooper spends much of the series talking into his pocket tape recorder:

Diane, it’s 8 A.M., Seattle, Washington. As you have no doubt surmised, by the clarity of this tape, I’ve purchased a new Micro-Mac pocket tape recorder. - ‘The big little recorder’ at Wally’s Rent-to-Own, 1145 North Hilltop. Where, as the sign says, ‘A bargain is a bargain, no matter what the cost.’ - for twenty-one dollars and eighty-nine cents, cash.

I decided to pass on the rent-to-own option, Diane. Leasing may be the fast track to an appearance of affluence, but equity will keep you warm at night. I have no doubt that this new model will prove to be an extremely useful tool in the investigatory process – where the most fleeting insight can be lost if your hardware isn’t as solid as you think it.

24 February 1989.

Here are all of Agent Cooper’s recordings to Diane.


I guess we’ll all be transcribing 24/7 one day. I can see already that it’s handy to be able to speak magic commands in regular chatter, to be executed at transcription time in the future.

e.g. "Robert WEIGHT 60.1 end Robert" is the example at that link there.

Addressing it to Diane is a better fit for my brain.


Auto-detected kinda similar posts:

Resistance from the tech sector

20 April 2025 at 00:00

As of late, most of us have been reading the news with a sense of anxious trepidation. At least, those of us who read from a position of relative comfort and privilege. Many more read the news with fear. Some of us are already no longer in a position to read the news at all, having become the unfortunate subjects of the news. Fascism is on the rise worldwide and in the United States the news is particularly alarming. The time has arrived to act.

The enemy wants you to be overwhelmed and depressed, to feel like the situation is out of your control. Propaganda is as effective on me as it is on you, and in my own home the despair and helplessness the enemy aims to engineer in us often prevails in my own life. We mustn’t fall for this gambit.

When it comes to resistance, I don’t have all of the answers, and I cannot present a holistic strategy for effective resistance. Nevertheless, I have put some thought towards how someone in my position, or in my community, can effectively apply ourselves towards resistance.

The fact of the matter is that the tech sector is extraordinarily important in enabling and facilitating the destructive tide of contemporary fascism’s ascent to power. The United States is embracing a technocratic fascism at the hands of Elon Musk and his techno-fetishist “Department of Government Efficiency”. Using memes to mobilize the terminally online neo-right, and “digitizing” and “modernizing” government institutions with the dazzling miracles of modern technology, the strategy puts tech, in its mythologized form – prophesied, even, through the medium of science fiction – at the center of a revolution of authoritarian hate.

And still, this glitz and razzle dazzle act obscures the more profound and dangerous applications of tech hegemony to fascism. Allow me to introduce public enemy number one: Palantir. Under the direction of neo-fascist Peter Thiel and in collaboration with ICE, Palantir is applying the innovations of the last few decades of surveillance capitalism to implementing a database of undesirables the Nazis could have never dreamed of. Where DOGE is hilariously tragic, Palantir is nightmarishly effective.

It’s clear that the regime will be digital. The through line is tech – and the tech sector depends on tech workers. That’s us. This puts us in a position to act, and compels us to act. But then, what should we do?

If there’s one thing I want you to take away from this article, something to write on your mirror and repeat aloud to yourself every day, it’s this: there’s safety in numbers. It is of the utmost importance that we dispense with American individualism and join hands with our allies to resist as one. Find your people in your local community, and especially in your workplace, who you can trust and who believe in what’s right and that you can depend on for support. It’s easier if you’re not going it alone. Talk to your colleagues about your worries and lean on them to ease your fears, and allow them to lean on you in turn.

One of the most important actions you can take is to unionize your workplace. We are long overdue for a tech workers union. If tech workers unionize then we can compel our employers – this regime’s instruments of fascist power – to resist also. If you’re at the bottom looking up at your boss’s boss’s boss cozying up with fascists, know that with a union you can pull the foundations of his power out from beneath him.

More direct means of resistance are also possible, especially for the privileged and highly paid employees of big tech. Maneuver yourself towards the levers of power. At your current job, find your way onto the teams implementing the technology that enables authoritarianism, and fuck it up. Drop the database by “mistake”. Overlook bugs. Be confidently wrong in code reviews and meetings. Apply for a job at Palantir, and be incompetent at it. Make yourself a single point of failure, then fail. Remember too that plausible deniability is key – make them work to figure out that you are the problem.

This sort of action is scary and much riskier than you’re probably immediately comfortable with. Inaction carries risks also. Only you are able to decide what your tolerance for risk is, and what kind of action that calls for. If your appetite for risk doesn’t permit sabotage, you could simply refuse to work on projects that aren’t right. Supporting others is essential resistance, too – be there for your friends, especially those more vulnerable than yourself, and support the people who engage in direct resistance. You didn’t see nuffin, right? If your allies get fired for fucking up an important digital surveillance project – you’ll have a glowing reference for them when they apply for Palantir, right?

Big tech has become the problem, and it’s time for tech workers to be a part of the solution. If this scares you – and it should – I get it. I’m scared, too. It’s okay for it to be scary. It’s okay for you not to do anything about it right now. All you have to do right now is be there for your friends and loved ones, and answer this question: where will you draw the line?

Remember your answer, and if and when it comes to pass… you will know when to act. Don’t let them shift your private goalposts until the frog is well and truly boiled to death.

Hang in there.

A Firefox addon for putting prices into perspective

4 April 2025 at 00:00

I had a fun idea for a small project this weekend, and so I quickly put it together over the couple of days. The result is Price Perspective.

Humor me: have you ever bought something, considered the price, and wondered how that price would look to someone else? Someone in the developing world, or a billionaire, or just your friend in Australia? In other words, can we develop an intuition for purchasing power?

The Price Perspective add-on answers these questions. Let’s consider an example: my income is sufficient to buy myself a delivery pizza for dinner without a second thought. How much work does it take for someone in Afghanistan to buy the same pizza? I can fire up Price Perspective to check:

The results are pretty shocking.

How about another example: say I’m looking to buy a house in the Netherlands. I fire up funda.nl and look at a few places in Amsterdam. After a few minutes wondering if I’ll ever be in an economic position to actually afford any of these homes (and speculating on if that day will come before or after I have spent this much money on rent over my lifetime), I wonder what these prices look like from the other side. Let’s see what it’d take for the Zuck to buy this apartment I fancy:

Well… that’s depressing. Let’s experiment with Price Perspective to see what it would take to make a dent in Zuck’s wallet. Let’s add some zeroes.

So, Zuckerberg over-bidding this apartment to the tune of €6.5B would cost him a proportion of his annual income which is comparable to me buying it for €5,000.

How about the reverse? How long would I have to work to buy, say, Jeff Bezos’s new mansion?

Yep. That level of wealth inequality is a sign of a totally normal, healthy, well-functioning society.

Curious to try it out for yourself? Get Price Perspective from addons.mozilla.org, tell it where you live and how much money you make in a year, and develop your own sense of perspective.

Using linkhut to signal-boost my bookmarks

27 March 2025 at 00:00

It must have been at least a year ago that I first noticed linkhut, and its flagship instance at ln.ht, appear on SourceHut, where it immediately caught my attention for its good taste in inspirations. Once upon a time, I had a Pinboard account, which is a similar concept, but I never used it for anything in the end. When I saw linkhut I had a similar experience: I signed up and played with it for a few minutes before moving on.

I’ve been rethinking my relationship social media lately, as some may have inferred from my unannounced disappearance from Mastodon.1 While reflecting on this again recently, in a stroke of belated inspiration I suddenly appreciated the appeal of tools like linkhut, especially alongside RSS feeds – signal-boosting stuff I read and found interesting.

The appeal of this reminds me of one of the major appeals of SoundCloud to me, back when I used it circa… 2013? That is: I could listen to the music that artists I liked were listening to, and that was amazing for discovering new music. Similarly, for those of you who enjoy my blog posts, and want to read the stuff I like reading, check out my linkhut feed. You can even subscribe to its RSS feed if you like. There isn’t much there today, but I will be filling it up with interesting articles I see and projects I find online.

I want to read your linkhut feed, too, but it’s pretty quiet there at the moment. If you find the idea interesting, sign up for an account or set up your own instance and start bookmarking stuff – and email me your feed so I can find some good stuff to subscribe to in my own feed reader.


  1. And simultaneous disappearance from BlueSky, though I imagine hardly anyone noticed given that I had only used it for a couple of weeks. When I set out to evaluate BlueSky for its merits from an OSS framing (findings: both surprisingly open and not open enough), I also took a moment to evaluate the social experience – and found it wanting. Then I realized that I also felt that way about Mastodon, and that was the end of that. ↩︎

Please stop externalizing your costs directly into my face

17 March 2025 at 00:00

This blog post is expressing personal experiences and opinions and doesn’t reflect any official policies of SourceHut.

Over the past few months, instead of working on our priorities at SourceHut, I have spent anywhere from 20-100% of my time in any given week mitigating hyper-aggressive LLM crawlers at scale. This isn’t the first time SourceHut has been at the wrong end of some malicious bullshit or paid someone else’s externalized costs – every couple of years someone invents a new way of ruining my day.

Four years ago, we decided to require payment to use our CI services because it was being abused to mine cryptocurrency. We alternated between periods of designing and deploying tools to curb this abuse and periods of near-complete outage when they adapted to our mitigations and saturated all of our compute with miners seeking a profit. It was bad enough having to beg my friends and family to avoid “investing” in the scam without having the scam break into my business and trash the place every day.

Two years ago, we threatened to blacklist the Go module mirror because for some reason the Go team thinks that running terabytes of git clones all day, every day for every Go project on git.sr.ht is cheaper than maintaining any state or using webhooks or coordinating the work between instances or even just designing a module system that doesn’t require Google to DoS git forges whose entire annual budgets are considerably smaller than a single Google engineer’s salary.

Now it’s LLMs. If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints like git blame, every page of every git log, and every commit in every repo, and they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses – mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure – actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic.

We are experiencing dozens of brief outages per week, and I have to review our mitigations several times per day to keep that number from getting any higher. When I do have time to work on something else, often I have to drop it when all of our alarms go off because our current set of mitigations stopped working. Several high-priority tasks at SourceHut have been delayed weeks or even months because we keep being interrupted to deal with these bots, and many users have been negatively affected because our mitigations can’t always reliably distinguish users from bots.

All of my sysadmin friends are dealing with the same problems. I was asking one of them for feedback on a draft of this article and our discussion was interrupted to go deal with a new wave of LLM bots on their own server. Every time I sit down for beers or dinner or to socialize with my sysadmin friends it’s not long before we’re complaining about the bots and asking if the other has cracked the code to getting rid of them once and for all. The desperation in these conversations is palpable.

Whether it’s cryptocurrency scammers mining with FOSS compute resources or Google engineers too lazy to design their software properly or Silicon Valley ripping off all the data they can get their hands on at everyone else’s expense… I am sick and tired of having all of these costs externalized directly into my fucking face. Do something productive for society or get the hell away from my servers. Put all of those billions and billions of dollars towards the common good before sysadmins collectively start a revolution to do it for you.

Please stop legitimizing LLMs or AI image generators or GitHub Copilot or any of this garbage. I am begging you to stop using them, stop talking about them, stop making new ones, just stop. If blasting CO2 into the air and ruining all of our freshwater and traumatizing cheap laborers and making every sysadmin you know miserable and ripping off code and books and art at scale and ruining our fucking democracy isn’t enough for you to leave this shit alone, what is?

If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts.

A holistic perspective on intellectual property, part 1

13 February 2025 at 00:00

I’d like to write about intellectual property in depth, in this first of a series of blog posts on the subject. I’m not a philosopher, but philosophy is the basis of reasonable politics so buckle up for a healthy Friday afternoon serving of it.

To understand intellectual property, we must first establish at least a shallow understanding of property generally. What is property?1 An incomplete answer might state that a material object I have power over is my property. An apple I have held in my hand is mine, insofar as nothing prevents me from using it (and, in the process, destroying it), or giving it away, or planting it in the ground. However, you might not agree that this apple is necessarily mine if I took it from a fruit stand without permission. This act is called “theft” — one of many possible transgressions upon property.

It is important to note that the very possibility that one could illicitly assume possession of an object is a strong indication that “property” is a social convention, rather than a law of nature; one cannot defy the law of gravity in the same way as one can defy property. And, given that, we could try to imagine other social conventions to govern the use of things in a society. If we come up with an idea we like, and we’re in a radical mood, we could even challenge the notion of property in society at large and seek to implement a different social convention.

As it stands today, the social convention tells us property is a thing which has an “owner”, or owners, to whom society confers certain rights with respect to the thing in question. That may include, for example, the right to use it, to destroy it, to exclude others from using it, to sell it, or give it away, and so on. Property is this special idea society uses to grant you the authority to use a bunch of verbs with respect to a thing. However, being a social convention, nothing prevents me from using any of these verbs on something society does not recognize as my property, e.g. by selling you this bridge. This is why the social convention must be enforced.

And how is it enforced? We could enforce property rights with shame: stealing can put a stain on one’s reputation, and this shame may pose an impediment to one’s social needs and desires, and as such theft is discouraged. We can also use guilt: if you steal something, but don’t get caught, you could end up remorseful without anyone to shame you for it, particularly with respect to the harm done to the person who suffered a loss of property as a result. Ultimately, in modern society the social convention of property is enforced with, well, force. If you steal something, society has appointed someone with a gun to track you down, restrain you, and eventually lock you up in a miserable room with bars on the windows.


I’d like to take a moment here to acknowledge the hubris of property: we see the bounty of the natural world and impose upon it these imagined rights and privileges, divvy it up and hand it out and hoard it, and resort to cruelty if anyone steps out of line. Indeed this may be justifiable if the system of private property is sufficiently beneficial to society, and the notion of property is so deeply ingrained into our system that it feels normal and unremarkable. It’s worth remembering that it has trade-offs, that we made the whole thing up, and that we can make up something else with different trade-offs. That being said, I’m personally fond of most of my personal property and I’d like to keep enjoying most of my property rights as such, so take from that what you will.2


One way we can justify property rights is by using them as a tool for managing scarcity. If demand for coffee exceeds the supply of coffee beans, a scarcity exists, meaning that not everyone who wants to have coffee gets to have some. But, we still want to enjoy scarce things. Perhaps someone who foregoes coffee will enjoy some other scarce resource, such as tea — then everyone can benefit in some part from some access to scarce resources. I suppose that the social convention of property can derive some natural legitimacy from the fact that some resources are scarce.3 In this sense, private property relates to the problem of distribution.

But a naive solution to distribution has flaws. For example, what of hoarding? Are property rights legitimate when someone takes more than they need or intend to use? This behavior could be motivated by an antagonistic relationship with society at large, such as as a means of driving up prices for private profit; such behavior could be considered anti-social and thus a violation of the social convention as such.

Moreover, property which is destroyed by its use, such as coffee, is one matter, but further questions are raised when we consider durable goods, such as a screwdriver. The screwdriver in my shed spends the vast majority of its time out of use. Is it just for me to assert property rights over my screwdriver when I am not using it? To what extent is the scarcity of screwdrivers necessary? Screwdrivers are not fundamentally scarce, given that the supply of idle screwdrivers far outpaces the demand for screwdriver use, but our modern conception of property has the unintended consequence of creating scarcity where there is none by denying the use of idle screwdrivers where they are needed.

Let’s try to generalize our understanding of property, working our way towards “intellectual property” one step at a time. To begin with, what happens if we expand our understanding of property to include immaterial things? Consider domain names as a kind of property. In theory, domain names are abundant, but some names are more desirable than others. We assert property rights over them, in particular the right to use a name and exclude others from using it, or to derive a profit from exclusive use of a desirable name.

But a domain name doesn’t really exist per-se: it’s just an entry in a ledger. The electric charge on the hard drives in your nearest DNS server’s database exist, but the domain name it represents doesn’t exist in quite the same sense as the electrons do: it’s immaterial. Is applying our conception of property to these immaterial things justifiable?

We can start answering this question by acknowledging that property rights are useful for domain names, in that this gives domain names desirable properties that serve productive ends in society. For example, exclusive control over a domain name allows a sense of authenticity to emerge from its use, so that you understand that pointing your browser to drewdevault.com will return the content that the person, Drew DeVault, wrote for you. We should also acknowledge that there are negative side-effects of asserting property rights over domains, such as domain squatting, extortionate pricing for “premium” domain names, and the advantage one party has over another if they possess a desirable name by mere fact of that possession, irrespective of merit.

On the balance of things, if we concede the legitimacy of personal property4 I find it relatively easy to concede the legitimacy of this sort of property, too.

The next step is to consider if we can generalize property rights to govern immaterial, non-finite things, like a story. A book, its paper and bindings and ink, is a material, finite resource, and can be thought of in terms that apply to material property. But what of the words formed by the ink? They can be trivially copied with a pen and paper, or transformed into a new medium by reading it aloud to an audience, and these processes do not infringe on the material property rights associated with the book. This process cannot be thought of as stealing, as the person who possesses a copy of the book is not asserting property rights over the original. In our current intellectual property regime, this person is transgressing via use of the idea, the intellectual property — the thing in the abstract space occupied by the story itself. Is that, too, a just extension of our notion of property?

Imagine with me the relationship one has with one’s property, independent of the social constructs around property. With respect to material property, a relationship of possession exists: I physically possess a thing, and I have the ability to make use of it through my possession of it. If someone else were to deny me access to this thing, they would have to resort to force, and I would have to resort to force should I resist their efforts.

Our relationship with intellectual property is much different. An idea cannot be withheld or seized by force. Instead, our relationship to intellectual property is defined by our history with respect to an idea. In the case of material property, the ground truth is that I keep it locked in my home to deny others access to it, and the social construct formalizes this relationship. With respect to intellectual property, such as the story in a book, the ground truth is that, sometime in the past, I imagined it and wrote it down. The social construct of intellectual property invents an imagined relationship of possession, modelled after our relationship with material property.

Why?

The resource with the greatest and most fundamental scarcity is our time,5 and as a consequence the labor which goes into making something is of profound importance. Marx famously argued for a “labor theory of value”, which tells us that the value inherent in a good or service is in the labor which is required to provide it. I think he was on to something!6 Intellectual property is not scarce, nor can it be possessed, but it does have value, and that value could ultimately be derived from the labor which produced it.

The social justification for intellectual property as a legal concept is rooted in the value of this labor. We recognize that intellectual labor is valuable, and produces an artifact — e.g. a story — which is valuable, but is not scarce. A capitalist society fundamentally depends on scarcity to function, and so through intellectual property norms we create an artificial scarcity to reward (and incentivize) intellectual labor without questioning our fundamental assumptions about capitalism and value.7 But, I digress — let’s revisit the subject in part two.

In part two of this series on intellectual property, I will explain the modern intellectual property regime as I understand it, as well as its history and justification. So equipped with the philosophical and legal background, part three will constitute the bulk of my critique of intellectual property, and my ideals for reform. Part four will examine how these ideas altogether apply in practice to open source, as well as the hairy questions of intellectual property as applied to modern problems in this space, such as the use of LLMs to file the serial numbers off of open source software.


If you want to dive deeper into the philosophy here, a great resource is the Stanford Encyclopedia of Philosophy. Check out their articles on Property and Ownership and Redistribution for a start, which expand on some of the ideas I’ve drawn on here and possess a wealth of citations catalogued with a discipline I can never seem to muster for my blog posts. I am a programmer, not a philosopher, so if you want to learn more about this you should go read from the hundreds of years of philosophers who have worked on this with rigor and written down a bunch of interesting ideas.


  1. In today’s article I will focus mainly on personal property (e.g. your shoes), private property (e.g. a house or a business), and intellectual property (e.g. a patent or a copyright). There are other kinds: public property, collective property, and so on, but to simplify this article we will use the layman’s understanding of “property” as commonly referring to personal property or private property, whichever is best supported by context, unless otherwise specified. In general terms all of these kinds of property refer to the rules with which society governs the use of things↩︎

  2. Marx, among others, distinguishes between personal property and private property. The distinction is drawn in that personal property can be moved – you can pick up a T-Shirt and take it somewhere else. Private property cannot, such as land or a house. Anyway, I’m not a Marxist but I do draw from Marxist ideas for some of my analysis of intellectual property, such as the labor theory of value. We’ll talk more about these ideas later on. ↩︎

  3. It occurred to me after writing this section that the selected examples of property and scarcity as applied to coffee and tea are begging for an analysis of the subject through the lens of colonialism, but I think my readers are not quite ready for that yet. ↩︎

  4. Not that I do, at least not entirely. I personally envision a system in which wealth is capped, hoarding is illegal, and everyone has an unconditional right to food, shelter, healthcare, and so on, and I’ll support reforming property rights in a heartbeat if that’s what it takes to get all of those things done. And, as the saying goes: if you see someone stealing groceries, you didn’t see anything. My willingness to accept property as a legitimate social convention is conditional on it not producing antisocial outcomes like homelessness or food insecurity. A system like this is considered a form of “distributive justice”, if you want to learn more. ↩︎

  5. And you’re spending some of it to read my silly blog, which I really feel is an honor. Thank you. ↩︎

  6. Marx loses me at historical determinism and the dominance of man over nature through dogmatic industrialization, among other things, but the labor theory of value is good shit. ↩︎

  7. Another tangent on the labor theory of value seems appropriate here. Our capitalist system is largely based on a competing theory, the “subjective theory of value”, which states that value is defined not by the labor required to provide a product or service, but by market forces, or more concretely by the subjective value negotiated between a buyer and seller. I admit this theory is compelling when applied to some examples, for example when explaining the value of a Pokemon card. When it comes to intellectual property, however, I find it very unsatisfying, given that a laissez-faire free market would presumably evolve a very different approach to intellectual property. As such I think that intellectual property as a concept depends at least a little bit on Marx for its legitimacy, which I find very funny. ↩︎

AI crap

29 August 2023 at 00:00

There is a machine learning bubble, but the technology is here to stay. Once the bubble pops, the world will be changed by machine learning. But it will probably be crappier, not better.

Contrary to the AI doomer’s expectations, the world isn’t going to go down in flames any faster thanks to AI. Contemporary advances in machine learning aren’t really getting us any closer to AGI, and as Randall Monroe pointed out back in 2018:


A panel from the webcomic “xkcd” showing a timeline from now into the distant
future, dividing the timeline into the periods between “AI becomes advanced
enough to control unstoppable swarms of robots” and “AI becomes self-aware and
rebels against human control”. The period from self-awareness to the indefinite
future is labelled “the part lots of people seem to worry about”; Randall is
instead worried about the part between these two epochs.

What will happen to AI is boring old capitalism. Its staying power will come in the form of replacing competent, expensive humans with crappy, cheap robots. LLMs are a pretty good advance over Markov chains, and stable diffusion can generate images which are only somewhat uncanny with sufficient manipulation of the prompt. Mediocre programmers will use GitHub Copilot to write trivial code and boilerplate for them (trivial code is tautologically uninteresting), and ML will probably remain useful for writing cover letters for you. Self-driving cars might show up Any Day Now™, which is going to be great for sci-fi enthusiasts and technocrats, but much worse in every respect than, say, building more trains.

The biggest lasting changes from machine learning will be more like the following:

  • A reduction in the labor force for skilled creative work
  • The complete elimination of humans in customer-support roles
  • More convincing spam and phishing content, more scalable scams
  • SEO hacking content farms dominating search results
  • Book farms (both eBooks and paper) flooding the market
  • AI-generated content overwhelming social media
  • Widespread propaganda and astroturfing, both in politics and advertising

AI companies will continue to generate waste and CO2 emissions at a huge scale as they aggressively scrape all internet content they can find, externalizing costs onto the world’s digital infrastructure, and feed their hoard into GPU farms to generate their models. They might keep humans in the loop to help with tagging content, seeking out the cheapest markets with the weakest labor laws to build human sweatshops to feed the AI data monster.

You will never trust another product review. You will never speak to a human being at your ISP again. Vapid, pithy media will fill the digital world around you. Technology built for engagement farms – those AI-edited videos with the grating machine voice you’ve seen on your feeds lately – will be white-labeled and used to push products and ideologies at a massive scale with a minimum cost from social media accounts which are populated with AI content, cultivate an audience, and sold in bulk and in good standing with the Algorithm.

All of these things are already happening and will continue to get worse. The future of media is a soulless, vapid regurgitation of all media that came before the AI epoch, and the fate of all new creative media is to be subsumed into the roiling pile of math.

This will be incredibly profitable for the AI barons, and to secure their investment they are deploying an immense, expensive, world-wide propaganda campaign. To the public, the present-day and potential future capabilities of the technology are played up in breathless promises of ridiculous possibility. In closed-room meetings, much more realistic promises are made of cutting payroll budgets in half.

The propaganda also leans into the mystical sci-fi AI canon: the threat of smart computers with world-ending power, the forbidden allure of a new Manhattan Project and all of its consequences, the long-prophesied singularity. The technology is nowhere near this level, a fact well-known by experts and the barons themselves, but the illusion is maintained in the interests of lobbying lawmakers to help the barons erect a moat around their new industry.

Of course, AI does present a threat of violence, but as Randall points out, it’s not from the AI itself, but rather from the people that employ it. The US military is testing out AI-controlled drones, which aren’t going to be self-aware but will scale up human errors (or human malice) until innocent people are killed. AI tools are already being used to set bail and parole conditions – it can put you in jail or keep you there. Police are using AI for facial recognition and “predictive policing”. Of course, all of these models end up discriminating against minorities, depriving them of liberty and often getting them killed.

AI is defined by aggressive capitalism. The hype bubble has been engineered by investors and capitalists dumping money into it, and the returns they expect on that investment are going to come out of your pocket. The singularity is not coming, but the most realistic promises of AI are going to make the world worse. The AI revolution is here, and I don’t really like it.

Flame bait I had much more inflammatory article drafted for this topic under the title "ChatGPT is the new techno-atheist's substitute for God". It makes some fairly pointed comparisons between the cryptocurrency cult and the machine learning cult and the religious, unshakeable, and largely ignorant faith in both technologies as the harbingers of progress. It was fun to write, but this is probably the better article.

I found this Hacker News comment and quoted it in the original draft: “It’s probably worth talking to GPT4 before seeking professional help [to deal with depression].”

In case you need to hear it: do not (TW: suicide) seek out OpenAI’s services to help with your depression. Finding and setting up an appointment with a therapist can be difficult for a lot of people – it’s okay for it to feel hard. Talk to your friends and ask them to help you find the right care for your needs.

Hello from Ares!

9 August 2023 at 00:00

I am pleased to be writing today’s blog post from a laptop running Ares OS. I am writing into an ed(1) session, on a file on an ext4 filesystem on its hard drive. That’s pretty cool! It seems that a lot of interesting stuff has happened since I gave that talk on Helios at FOSDEM in February.

A picture of my ThinkPad while I was editing this blog post

The talk I gave at FOSDEM was no doubt impressive, but it was a bit of a party trick. The system was running on a Raspberry Pi with one process which included both the slide deck as a series of raster images baked into the ELF file, as well as the GPU driver and drawing code necessary to display them, all in one package. This was quite necessary, as it turns out, given that the very idea of “processes” was absent from the system at this stage.

Much has changed since that talk. The system I am writing to you from has support for processes indeed, complete with fork and exec and auxiliary vectors and threads and so on. If I run “ps” I get the following output:

mercury % ps
1 /sbin/usrinit dexec /sbin/drv/ext4 block0 childfs 0 fs 0
2 /etc/driver.d/00-pcibus
3 /etc/pci.d/class/01/06/ahci
4 /etc/driver.d/00-ps2kb
5 /etc/driver.d/99-serial
6 /etc/driver.d/99-vgacons
7 /sbin/drv/ext4 block0
15 ed blog.md
16 ps

Each of these processes is running in userspace, and some of them are drivers. A number of drivers now exist for the system, including among the ones you see here a general-purpose PCI driver, AHCI (SATA), PS/2 keyboard, PC serial, and a VGA console, not to mention the ext4 driver, based on lwext4 (the first driver not written in Hare, actually). Not shown here are additional drivers for the CMOS real-time clock (so Ares knows what time it is, thanks to Stacy Harper), a virtio9pfs driver (thanks also to Tom Leb for the initial work here), and a few more besides.

As of this week, a small number of software ports exist. The ext4 driver is based on lwext4, as I said earlier, which might be considered a port, though it is designed to be portable. The rc shell I have been working on lately has also been ported, albeit with many features disabled, to Mercury. And, of course, I did say I was writing this blog post with ed(1) – I have ported Michael Forney’s ed implementation from sbase, with relatively few features disabled as a matter of fact (the “!” command and signals were removed).

This ed port, and lwext4, are based on our C library, designed with drivers and normal userspace programs in mind, and derived largely from musl libc. This is coming along rather well – a few features (signals again come to mind) are not going to be implemented, but it’s been relatively straightforward to get a large amount of the POSIX/C11 API surface area covered on Ares, and I was pleasantly surprised at how easy it was to port ed(1).

There’s still quite a lot to be done. In the near term, I expect to see the following:

  • A virtual filesystem
  • Pipes and more shell features enabled, such as redirects
  • More filesystem support (mkdir et al)
  • A framebuffer console
  • EFI support on x86_64
  • MBR and GPT partitions

This is more of the basics. As these basics unblock other tasks, a few of the more ambitious projects we might look forward to include:

  • Networking support (at least ICMP)
  • Audio support
  • ACPI support
  • Basic USB support
  • A service manager (not systemd…)
  • An installer, perhaps a package manager
  • Self-hosting builds
  • Dare I say Wayland?

I should also probably do something about that whining fan I’m hearing in the background right now. Of course, I will also have to do a fresh DOOM port once the framebuffer situation is improved. There’s also still plenty of kernel work to be done and odds and ends all over the project, but it’s in pretty good shape and I’m having a blast working on it. I think that by now I have answered the original question, “can an operating system be written in Hare”, with a resounding “yes”. Now I’m just having fun with it. Stay tuned!

Now I just have to shut this laptop off. There’s no poweroff command yet, so I suppose I’ll just hold down the power button until it stops making noise.

The rc shell and its excellent handling of whitespace

31 July 2023 at 00:00

This blog post is a response to Mark Dominus’ “The shell and its crappy handling of whitespace.

I’ve been working on a shell for Unix-like systems called rc, which draws heavily from the Plan 9 shell of the same name. When I saw Mark’s post about the perils of whitespace in POSIX shells (or derived shells, like bash), I thought it prudent to see if any of the problems he outlines are present in the shell I’m working on myself. Good news: they aren’t!

Let’s go over each of his examples. First he provides the following example:

for i in *.jpg; do
	cp $i /tmp
done

This breaks if there are spaces in the filenames. Not so with rc:

% cat test.rc
for (i in *.jpg) {
	cp $i subdir
}
% ls
a.jpg   b.jpg  'bite me.jpg'   c.jpg   subdir   test.rc
% rc ./test.rc 
% ls subdir/
a.jpg   b.jpg  'bite me.jpg'   c.jpg

He gives a similar example for a script that renames jpeg to jpg:

for i in *.jpeg; do
  mv $i $(suf $i).jpg
done

This breaks for similar reasons, but works fine in rc:

% cat test.rc  
fn suf(fname) {
	echo $fname | sed -e 's/\..*//'
}

for (i in *.jpeg) {
	mv $i `{suf $i}.jpg
}
% ls 
a.jpeg   b.jpeg  'bite me.jpeg'   c.jpeg   test.rc
% rc ./test.rc  
% ls 
a.jpg   b.jpg  'bite me.jpg'   c.jpg   test.rc

There are other shells, such as fish or zsh, which also have answers to these problems which don’t necessarily call for generous quoting like other shells often do. rc is much simpler than these shells. At the moment it clocks in at just over 3,000 lines of code, compared to fish at ~45,000 and zsh at ~144,000. Admittedly, it’s not done yet, but I would be surprised to see it grow beyond 5,000 lines for version 1.0.1

The key to rc’s design success in this area is the introduction of a second primitive. The Bourne shell and its derivatives traditionally work with only one primitive: strings. But command lines are made of lists of strings, and so a language which embodies the primitives of the command line ought to also be able to represent those as a first-class feature. In traditional shells a list of strings is denoted inline with the use of spaces within those strings, which raises obvious problems when the members themselves contain spaces; see Mark’s post detailing the errors which ensue. rc adds lists of strings as a formal primitive alongside strings.

% args=(ls --color /) 
% echo $args(1) 
ls
% echo $args(2) 
--color
% echo $#args 
3
% $args 
bin   dev  home  lost+found  mnt  proc  run   srv      swap  tmp  var
boot  etc  lib   media       opt  root  sbin  storage  sys   usr
% args=("foo bar" baz) 
% touch $args 
% ls 
 baz  'foo bar'

Much better, right? One simple change eliminates the need for quoting virtually everywhere. Strings can contain spaces and nothing melts down.

Let me run down the remaining examples from Mark’s post and demonstrate their non-importance in rc. First, regarding $*, it just does what you expect.

% cat yell.rc
#!/bin/rc
shift
echo I am about to run $* now!!!
exec $*
% ls *.jpg
'bite me.jpg'
% ./yell.rc ls *.jpg
I am about to run ls bite me.jpg now!!!
'bite me.jpg'

Note also that there is no need to quote the arguments to “echo” here. Also note the use of shift; $* includes $0 in rc.

Finally, let’s rewrite Mark’s “lastdl” program in rc and show how it works fine in rc’s interactive mode.

#!/bin/rc
cd $HOME/downloads
echo $HOME/downloads/`{ls -t | head -n1}

Its use at the command line works just fine without quotes.

% file `{lastdl} 
/home/sircmpwn/downloads/test image.jpg: JPEG image data, JFIF standard 1.01,
aspect ratio, density 1x1, segment length 16, baseline, precision 8,
5000x2813, components 3

Just for fun, here’s another version of this rc script that renames files with spaces to without, like the last example in Mark’s post:

#!/bin/rc
cd $HOME/downloads
last=`{ls -t | head -n1}
if (~ $last '* *') {
	newname=`{echo $last | tr ' \t' '_'}
	mv $last $HOME/downloads/$newname
	last=$newname
}
echo $HOME/downloads/$last

The only quotes to be found are those which escape the wildcard match testing for a space in the string.2 Not bad, right? Like Plan 9’s rc, my shell imagines a new set of primitives for shells, then starts from the ground up and builds a shell which works better in most respects while still being very simple. Most of the problems that have long plagued us with respect to sh, bash, etc, are solved in a simple package with rc, alongside a nice interactive mode reminiscent of the best features of fish.

rc is a somewhat complete shell today, but there is a bit more work to be done before it’s ready for 1.0, most pressingly with respect to signal handling and job control, alongside a small bit of polish and easier features to implement (such as subshells, IFS, etc). Some features which are likely to be omitted, at least for 1.0, include logical comparisons and arithmetic expansion (for which /bin/test and /bin/dc are recommended respectively). Of course, rc is destined to become the primary shell of the Ares operating system project that I’ve been working on, but I have designed it to work on Unix as well.

Check it out!


  1. Also worth noting that these line counts are, to some extent, comparing apples to oranges given that fish, zsh, and rc are written respectively in C++/Rust, C, and Hare. ↩︎

  2. This is a bit of a fib. In fact, globbing is disabled when processing the args of the ~ built-in. However, the quotes are, ironically, required to escape the space between the * characters, so it’s one argument rather than two. ↩︎

Alpine Linux does not make the news

25 July 2023 at 00:00

My Linux distribution of choice for several years has been Alpine Linux. It’s a small, efficient distribution which ships a number of tools I appreciate for their simplicity, such as musl libc. It has a very nice package manager, apk, which is fast and maintainable. The development community is professional and focuses on diligent maintenance of the distribution and little else. Over the years I have used it, very little of note has happened.

I run Alpine in every context; on my workstation and my laptops but also on production servers, on bare-metal and in virtual machines, on my RISC-V and ARM development boards, at times on my phones, and in many other contexts besides. It has been a boring experience. The system is simply reliable, and the upgrades go over without issue every other quarter,1 accompanied by high-quality release notes. I’m pleased to maintain several dozen packages in the repositories, and the community is organized such that it is easy for someone like me to jump in and do the work required to maintain it for my use-cases.

Red Hat has been in the news lately for their moves to monetize the distribution, moves that I won’t comment on but which have generally raised no small number of eyebrows, written several headlines, and caused intense flamewars throughout the internet. I don’t run RHEL or CentOS anywhere, in production or otherwise, so I just looked curiously on as all of this took place without calling for any particular action on my part. Generally speaking, Alpine does not make the news.

And so it has been for years, as various controversies come about and die off, be it with Red Hat, Ubuntu, Debian, or anything else, I simply keep running “apk upgrade” every now and then and life goes on uninterrupted. I have high-quality, up-to-date software on a stable system and suffer from no fuss whatsoever.

The Alpine community is a grassroots set of stakeholders who diligently concern themselves with the business of maintaining a good Linux distribution. There is little in the way of centralized governance;2 for the most part the distribution is just quietly maintained by the people who use it for the purpose of ensuring its applicability to their use-cases.

So, Alpine does not make the news. There are no commercial entities which are trying to monetize it, at least no more than the loosely organized coalition of commercial entities like SourceHut that depend on Alpine and do their part to keep it in good working order, alongside various users who have no commercial purpose for the system. The community is largely in unanimous agreement about the fundamental purpose of Alpine and the work of the community is focused on maintaining the project such that this purpose is upheld.

This is a good trait for a Linux distribution to have.


  1. Or more frequently on edge, which I run on my workstation and laptops and which receives updates shortly after upstream releases for most software. ↩︎

  2. There’s some. They mostly concern themselves with technical decisions like whether or not to approve new committers or ports, things like that. ↩︎

Seriously, don't sign a CLA

4 July 2023 at 00:00

SourceGraph is making their product closed source, abandoning the Apache 2.0 license it was originally distributed under, so once again we convene in the ritual condemnation we offer to commercial products that piss in the pool of open source. Invoking Bryan Cantrill once more:

Bryan Cantrill on OpenSolaris — YouTube

A contributor license agreement, or CLA, usually (but not always) includes an important clause: a copyright assignment. These agreements are provided by upstream maintainers to contributors to open source software projects, and they demand a signature before the contributor’s work is incorporated into the upstream project. The copyright assignment clause that is usually included serves to offer the upstream maintainers more rights over the contributor’s work than the contributor was offered by upstream, generally in the form of ownership or effective ownership over the contributor’s copyright and the right to license it in any manner they choose in the future, including proprietary distributions.

This is a strategy employed by commercial companies with one purpose only: to place a rug under the project, so that they can pull at the first sign of a bad quarter. This strategy exists to subvert the open source social contract. These companies wish to enjoy the market appeal of open source and the free labor of their community to improve their product, but do not want to secure these contributors any rights over their work.

This is particularly pathetic in cases like that of SourceGraph, which used a permissive Apache 2.0 license. Such licenses already allow their software to be incorporated into non-free commercial works, such is the defining nature of a permissive license, with relatively few obligations: in this case, a simple attribution will suffice. SourceGraph could have been made non-free without a CLA at all if this one obligation was met. The owners of SourceGraph find the simple task of crediting their contributors too onerous. This is disgusting.

SourceGraph once approached SourceHut asking about building an integration between our platforms. They wanted us to do most of the work, which is a bit tacky but reasonable under the reciprocal social contract of open source. We did not prioritize it and I’m glad that we didn’t: our work would have been made non-free.

Make no mistake: a CLA is a promise that a open source software project will one day become non-free. Don’t sign them.

What are my rights as a contributor?

If you sign away your rights by agreeing to a CLA, you retain all of the rights associated with your work.

By default, you own the copyright over your contribution and the contribution is licensed under the same software license the original project uses, thus, your contribution is offered to the upstream project on the same terms that their contribution was offered to you. The copyright for such projects is held collectively by all contributors.

You also always have the right to fork an open source project and distribute your improvements on your own terms, without signing a CLA – the only power upstream holds is authority over the “canonical” distribution. If the rug is pulled from under you, you may also continue to use, and improve, versions of the software from prior to the change in license.

How do I prevent this from happening to my project?

A CLA is a promise that software will one day become non-free; you can also promise the opposite. Leave copyright in the collective hands of all contributors and use a copyleft license.

Without the written consent of all contributors, or performing their labor yourself by re-writing their contributions, you cannot change the license of a project. Skipping the CLA leaves their rights intact.

In the case of a permissive software license, a new license (including proprietary licenses) can be applied to the project and it can be redistributed under those terms. In this way, all future changes can be written with a new license. The analogy is similar to that of a new project with a proprietary license taking a permissively licensed project and incorporating all of the code into itself before making further changes.

You can prevent this as well with a copyleft license: such a license requires the original maintainers to distribute future changes to the work under a free software license. Unless they can get all copyright holders – all of the contributors – to agree to a change in license, they are obligated to distribute their improvements on the same terms.

Thus, the absence of a CLA combined with the use of a copyleft license serves as a strong promise about the future of the project.

Learn more at writefreesoftware.org:

What should I do as a business instead of a CLA?

It is not ethical to demand copyright assignment in addition to the free labor of the open source community. However, there are some less questionable aspects of a contributor license agreement which you may uphold without any ethical qualms, notably to establish provenance.

Many CLAs include clauses which establish the provenance of the contribution and transfer liability to the contributor, such that the contributor agrees that their contribution is either their own work or they are authorized to use the copyright (for example, with permission from their employer). This is a reasonable thing to ask for from contributors, and manages your exposure to legal risks.

The best way to ask for this is to require contributions to be “signed-off” with the Developer Certificate of Origin.


Previously:

Social media and "parasocial media"

30 June 2023 at 00:00

A few months ago, as Elon Musk took over Twitter and instituted polices that alienated many people, some of these people fled towards federated, free software platforms like Mastodon. Many people found a new home here, but there is a certain class of refugee who has not found it to their liking.

I got to chatting with one such “refugee” on Mastodon today. NotJustBikes is a creator I enjoy watching on YouTube Invidious, who makes excellent content on urbanism and the design of cities. He’s based in my home town of Amsterdam and his videos do a great job of explaining many of the things I love about this place for general audiences. He’s working on building an audience, expanding his reach, and bringing his message to as many people as possible in the interest of bringing better infrastructure to everyone.

But he’s not satisfied with his move from Twitter to Mastodon, nor are some of his friends among the community of “urbanist” content creators. He yearns for an “algorithm” to efficiently distribute content to his followers, and Mastodon is not providing this for him.

On traditional “social media” platforms, in particular YouTube, the interactions are often not especially social. The platforms facilitate a kind of intellectual consumption moreso than conversation: conversations flow in one direction, from creator to audience, where the creator produces and the audience consumes. I think a better term for these platforms is “parasocial media”: they are optimized for creating parasocial relationships moreso than social relationships.

The fediverse is largely optimized for people having conversations with each other, and not for producing and consuming “content”. Within this framework, a “content creator” is a person only in the same sense that a corporation is, and their conversations are unidirectional, where the other end is also not a person, but an audience. That’s not the model that the fediverse is designed around.

It’s entirely reasonable to want to build an audience and publish content in a parasocial manner, but that’s not what the fediverse is for. And I think that’s a good thing! There are a lot of advantages in having spaces which focus on being genuinely “social”, rather than facilitating more parasocial interactions and helping creators build an audience. This limits the fediverse’s reach, but I think that’s just fine.

Within this model, the fediverse’s model, it’s possible to publish things, and consume things. But you cannot effectively optimize for building the largest possible audience. You will generally be more successful if you focus on the content itself, and not its reach, and on the people you connect with at a smaller scale. Whether or not this is right for you depends on your goals.

I hope you enjoyed this content! Remember to like and subscribe.

Burnout and the quiet failures of the hacker community

29 June 2023 at 00:00

This has been a very challenging year for me. You probably read that I suffered from burnout earlier in the year. In some respects, things have improved, and in many other respects, I am still haunted.

You might not care to read this, and so be it, take your leave if you must. But writing is healing for me. Maybe this is a moment for solidarity, sympathy, for reflecting on your own communities. Maybe it’s a vain and needlessly public demonstration of my slow descent into madness. I don’t know, but here we go.

Yesterday was my 30th birthday. 🎂 It was another difficult day for me. I drafted a long blog post with all of the details of the events leading up to my burnout. You will never read it; I wrote it for myself and it will only be seen by a few confidants, in private, and my therapist. But I do want to give you an small idea of what I’ve been going through, and some of the take-aways that matter for you and the hacker community as a whole.

Here’s a quote from yesterday’s unpublished blog post:

Trigger warnings: child abuse, rape, sexual harassment, suicide, pedophilia, torture.

You won’t read the full story, and trust me, you’re better off for that. Suffice to say that my life has been consumed with trauma and strife all year. I have sought healing, and time for myself, time to process things, and each time a new crisis has landed on my doorstep, most of them worse than the last. A dozen things went wrong this year, horribly wrong, one after another. I have enjoyed no peace in 2023.

Many of the difficulties I have faced this year have been beyond the scope of the hacker community, but several have implicated it in challenging and confronting ways.

The hacker community has been the home I never had, but I’m not really feeling at home here right now. A hacker community that was precious to me failed someone I love and put my friends in danger. Rape and death had come to our community, and was kept silent. But I am a principled person, and I stand for what is right; I spoke the truth and it brought me and my loved ones agonizing stress and trauma and shook our community to the core. Board members resigned. Marriages are on the rocks. When the dust settled, I was initially uncomfortable staying in this community, but things eventually started to get better. Until another member of this community, someone I trusted and thought of as a friend, confessed to me that he had raped multiple women a few years ago. I submitted my resignation from this community last night.

Then I went to GPN, a hacker event in Germany, at the start of June. It was a welcome relief from the stress I’ve faced this year, a chance to celebrate hacker culture and a warm reminder of the beauty of our community. It was wonderful. Then, on the last night, a friend took me aside and confided in me that they are a pedophile, and told me it was okay because they respected the age of consent in Germany – which is 14. What began as a wonderful reminder of what the hacker community can be became a PTSD episode and a reminder that rape culture is fucking everywhere.

I don’t want to be a part of this anymore. Our communities have tolerated casual sexism and misogyny and transphobia and racism and actual fucking rapists, and stamped down on women and queer people and brown people in our spaces with a smile on our face and a fucked-up facsimile of tolerance and inclusion as a cornerstone of the hacker ethic.

This destroys communities. It is destroying our communities. If there’s one thing I came to understand this year, it’s that these problems are pervasive and silent.

Here’s what you need to do: believe the victims. Stand up for what’s right. Have the courage to remove harmful people from your environment, especially if you’re a man and have a voice. Make people feel welcome, and seen. Don’t tolerate casual sexism in the hacker community or anywhere else. Don’t tolerate transphobia or homophobia. Don’t tolerate racists. If you see something, say something. And for fuck’s sake, don’t bitch about that code of conduct that someone wants to add to your community.1

I’m going to withdraw a bit from the in-person hacker community for the indefinite future. I don’t think I can manage it for a while. I have felt good about working on my software and collaborating with my free software communities online, albeit at a much-reduced capacity. I’m going to keep working, and writing, insofar as I find satisfaction in it. Life goes on.

Be there for the people you love, and love more people, and be there for them, too.


  1. And fuck Richard Stallman and his enablers, his supporters, and the Free Software Foundation’s leadership as a whole. Shame on you. Shame on you↩︎

Reforming the free software message

19 June 2023 at 00:00

Several weeks ago, I wrote The Free Software Foundation is dying, wherein I enumerated a number of problems with the Free Software Foundation. Some of my criticisms focused on the message: fsf.org and gnu.org together suffer from no small degree of incomprehensibility and inaccessibility which makes it difficult for new participants to learn about the movement and apply it in practice to their own projects.

This is something which is relatively easily fixed! I have a background in writing documentation and a thorough understanding of free software philosophy and practice. Enter writefreesoftware.org: a comprehensive introduction to free software philosophy and implementation.

The goals of this resource are:

  • Provide an accessible introduction to the most important principles of free software
  • Offer practical advice on choosing free software licenses from a free software perspective (compare to the OSS perspective at choosealicense.com).
  • Publish articles covering various aspects of free software in practice, such as how it can be applied to video games

More:

  • No particular association with any particular free software project or organization
  • No policy of non-cooperation with the open source movement

Compare writefreesoftware.org with the similar resources provided by GNU (1, 2) and you should get the general idea.

The website is itself free software, CC-BY-SA 4.0. You can check out the source code here and suggest any improvements or articles for the mailing list. Get involved! This resource is not going to solve all of the FSF’s problems, but it is an easy way to start putting the effort in to move the free software movement forward. I hope you like it!

Throwing in the towel on mobile Linux

16 June 2023 at 00:00

I have been tinkering with mobile Linux – a phrase I will use here to describe any Linux distribution other than Android running on a mobile device – as my daily driver since about 2019, when I first picked up the PinePhone. For about 3 years I have run mobile Linux as my daily driver on my phone, and as of a few weeks ago, I’ve thrown in the towel and switched to Android.

The distribution I ran for the most time is postmarketOS, which I was mostly quite happy with, running at times sxmo and Phosh. I switched to UBports a couple of months ago. I have tried a variety of hardware platforms to support these efforts, namely:

  • Pinephone (pmOS)
  • Pinephone Pro (pmOS)
  • Xiaomi Poco F1 (pmOS)
  • Fairphone 4 (UBports)

I have returned to LineageOS as my daily driver and closed the book on mobile Linux for the time being. What put the final nails in the coffin was what I have been calling out as my main concern throughout my experience: reliability, particularly of the telephony components.

Use-case Importance postmarketOS UBports LineageOS
Basic system reliability 5 2 4 5
Mobile telephony 5 3 3 5
Hotspot 4 5 3 5
2FA 4 4 1 5
Web browsing 4 5 2 4
Mobile banking 4 1 1 5
Bluetooth audio 3 4 2 4
Music player 3 4 1 3
Reading email 3 1 3 4
Navigation aid 3 2 1 5
Camera 3 3 3 5
Password manager 3 5 1 1
sysadmin 3 5 2 3
More on these use-cases and my experiences

Mobile banking: only available through a proprietary vendor-provided Android app. Tried to get it working on Waydroid; did not work on pmOS and almost worked on UBports, but Waydroid is very unreliable. Kind of shit but I don’t have any choice because my bank requires it for 2FA.

Web browsing: I can just run Firefox upstream on postmarketOS. Amazing! UBports cannot do this, and the available web browsers are not nearly as pleasant to use. I run Fennic on Android and it’s fine.

Music player: the music player on UBports is extremely unreliable.

Reading email: This is not entirely pmOS’s fault; I could have used my main client, aerc, which is a testament to pmOS’s general utility, but it is a TUI that is uncomfortable to use on a touchscreen-only device.

Password manager: pmOS gets 5/5 because I could use the password manager I wrote myself, himitsu, out of the box. Non-critical use-case because I could just type passwords in manually on the rare occasion I need to use one.

sysadmin: stuff like being able to SSH into my production boxes from anywhere to troubleshoot stuff.

Among these use-cases, there is one that absolutely cannot be budged on: mobile telephony. My phone is a critical communication device and I need to be able to depend on calls and SMS at all times, therefore the first two rows need to score 4 or 5 before the platform is suitable for my use. I remember struggling with postmarketOS while I was sick with a terrible throat infection – and I could not call my doctor. Not cool.

I really like these projects and I love the work that’s going into them. postmarketOS in particular: being able to run the same environment I run everywhere else, Alpine Linux, on my phone, is fucking amazing. The experience is impressively complete in many respects, all kinds of things, including things I didn’t expect to work well, work great. In the mobile Linux space I think it’s the most compelling option right now.

But pmOS really suffers from reliability issues – both on edge and on stable it seemed like every update broke some things and fixed others, so only a subset of these cool features was working well at any given moment. The breakage would often be minor nuisances, such as the media controls on my bluetooth headphones breaking in one update and being fixed in the next, or major showstoppers such as broken phone calls, SMS, or, in one case, all of my icons disappearing from the UI (with no fallback in most cases, leaving me navigating the UI blind).

So I tried UBports instead, and despite the general lack of good auxiliary features compared to pmOS, the core telephony was more reliable – for a while. But once issues started to appear, particularly around SMS, I could not tolerate it for long in view of the general uselessness of the OS for anything else. I finally gave it up and installed LineageOS.

Mobile Linux is very cool and the community has made tremendous, unprecedented progress towards realizing its potential, and the forward momentum is still strong. I’m excited to see it continue to improve. But I think that before anyone can be expected to use this as a daily driver, the community really needs to batten down the hatches and focus on one thing and one thing only: always, always being usable as a phone. I’ll be back once more reliability is in place.

How to go to war with your employer

12 June 2023 at 00:00

There is a power differential between you and your employer, but that doesn’t mean you can’t improve your working conditions. Today I’d like to offer a little bit of advice on how to frame your relationship with your employer in terms which empower you and afford you more agency. I’m going to talk about the typical working conditions of the average white-collar job in a neo-liberal political environment where you are mostly happy to begin with and financially stable enough to take risks, and I’m specifically going to talk about individual action or the actions of small groups rather than large-scale collective action (e.g. unions).

I wish to subvert the expectation here that employees are subordinate to their employers. A healthy employment relationship between an employee and employer is that of two entities who agree to work together on equal terms to strive towards mutual goals, which in the simplest form is that you both make money and in the subtleties also suggests that you should be happy doing it. The sense of “going to war” here should rouse in you an awareness of the resources at your disposal, a willingness to use them to forward your interests, and an acknowledgement of the fact that tactics, strategy, propaganda, and subterfuge are among the tools you can use – and the tools your employer uses to forward their own interests.

You may suppose that you need your employer more than they need you, but with some basic accounting we can get a better view of the veracity of this supposition. Consider at the most fundamental that your employer is a for-profit entity that spends money to make money, and they spend money on you: as a rule of thumb, they expect a return of at least your salary ×1.5 (accounting for overhead, benefits et al) for their investment in you, otherwise it does not make financial sense for them to employ you.

If you have finer-grained insights into your company’s financial situation, you can get a closer view of your worth to them by dividing their annual profit by their headcount, adjusted to your discretion to account for the difference in the profitability of your role compared to your colleagues. It’s also wise to run this math in your head to see how the returns from your employment are affected by conditions in the hiring market, layoffs, etc – having fewer employees increases the company’s return per employee, and a busier hiring market reduces your leverage. In any case, it should be relatively easy for you to justify, in the cold arithmetic of finance that businesses speak, that employees matter to the employer, and the degree to which solidarity between workers is a meaningful force amplifier for your leverage.

In addition to your fundamental value, there are some weak points in the corporate structure that you should be aware of. There are some big levers that you may already be aware of that I have already placed outside of the scope of this blog post, such as the use of collective bargaining, unionization, strikes, and so on, where you need to maximize your collective leverage to really put the screws to your employer. Many neo-liberal workplaces lack the class consciousness necessary to access these levers, and on the day-to-day scale it may be strategically wise to smarten up your colleagues on social economics in preparation for use of these levers. I want to talk about goals on the smaller scale, though. Suppose your goals are, for instance:

  • You don’t like agile/scrum and want to interact with it from the other end of a six foot pole and/or replace it with another system
  • Define your own goals and work on the problems you think are important at your own discretion moreso than at the discretion of your manager
  • Skip meetings you know are wasting your time
  • Set working hours that suit you or take time off on your terms
  • Work from home or in-office in an arrangement that meets your own wants/needs
  • Exercise agency over your tools, such as installing the software you want to use on your work laptop

You might also have more intimidating goals you want to address:

  • Demand a raise or renegotiating benefits
  • Negotiate a 4-day workweek
  • Replace your manager or move teams
  • Remove a problematic colleague from your working environment

All of these goals are within your power to achieve, and perhaps more easily than you expect.

First of all, you already have more agency than you know. Your job description and assigned tasks tells a narrow story of your role at the business: your real job is ultimately to make money for the business. If you install Linux on your work laptop because it allows you to work more efficiently, then you are doing your job better and making more money for the business; they have no right to object to this and you have a very defensible position for exercising agency in this respect. Likewise if you adapt the workflows around agile (or whatever) to better suit your needs rather than to fall in line with the prescription, if it makes you more productive and happy then it makes the business more money. Remember your real job – to make money – and you can adjust the parameters of your working environment relatively freely provided that you are still aligned with this goal.

Often you can simply exercise agency in cases like these, but in other cases you may have to reach for your tools. Say you don’t just want to have maintain a personal professional distance from agile, but you want to replace it entirely: now you need to talk to your colleagues. You can go straight to management and start making your case, but another option – probably the more effective one – is to start with your immediate colleagues. Your team also possesses a collective agency, and if you agree together, without anyone’s permission, to work according to your own terms, then so long as you’re all doing your jobs – making money – then no one is going to protest. This is more effective than following the chain of command and asking them to take risks they don’t understand. Be aware of the importance of optics here: you need not only to make money, but to be seen making money. How you are seen to be doing this may depend on how far up the chain you need to justify yourself to; if your boss doesn’t like it then make sure your boss’s boss does.

Ranked in descending order of leverage within the business: your team, your boss, you.

More individual-oriented goals such as negotiating a different working schedule or skipping meetings calls for different tools. Simple cases, such as coming in at ten and leaving at four every day, are a case of simple exercise of agency; so long as you’re making the company money no one is going to raise a fuss. If you want, for instance, a four day work-week, or to work from home more often, you may have to justify yourself to someone. In such cases you may be less likely to have your team’s solidarity at your disposal, but if you’re seen to be doing your job – making money – then a simple argument that it makes you better at that job will often suffice.

You can also be clever. “Hey, I’ll be working from home on Friday” works better than “can I work from home on Friday?” If you want to work from home every Friday, however, then you can think strategically: keeping mum about your final goal of taking all Fridays from home may be wise if you can start by taking some Fridays at home to establish that you’re still productive and fulfilling the prime directive1 under those terms and allow yourself to “accidentally” slip into a new normal of working home every Friday without asking until it’s apparent that the answer will be yes. Don’t be above a little bit of subversion and deception; your employer is using those tools against you too.

Then there are the big guns: human resources. HR is the enemy; their job is to protect the company from you. They can, however, be useful if you understand the risks they’re trying to manage and press the right buttons with them. If your manager is a dick, HR may be the tool to use to fix this, but you need to approach it the right way. HR does not give two fucks that you don’t like your manager, if your manager is making money then they are doing their job. What HR does give a fuck about is managing the company’s exposure to lawsuits.

They can also make your life miserable. If HR does not like you then you are going to suffer, so when you talk to them it is important to know your enemy and to make strategic use of them without making them realize you know the game. They present themselves as your ally, let them think you believe it’s so. At the same time, there is a coded language you can use that will get them to act in your interest. HR will perk up as soon as they smell “unsafe working conditions”, “sexual harassment”, “collective action”, and so on – the risks they were hired to manage – over the horizon. The best way to interact with HR is for them to conclude that you are on a path which ends in these problems landing on their desk without making them think you are a subversive element within the organization. And if you are prepared to make your knowledge of and willingness to use these tools explicit, all communication which suggests as much should be delivered to HR with your lawyer’s signature and only when you have a new job offer lined up as a fallback. HR should either view you as mostly harmless or look upon you with fear, but nothing in between.

These are your first steps towards class consciousness as a white-collar employee. Know your worth, know the leverage you have, and be prepared to use the tools at your disposal to bring about the outcomes you desire, and know your employer will be doing the same. Good luck out there, and don’t forget to actually write some code or whatever when you’re not busy planning a corporate coup.


  1. Making money, of course. ↩︎

Burnout

1 May 2023 at 00:00

It kind of crept up on me. One day, sitting at my workstation, I stopped typing, stared blankly at the screen for a few seconds, and a switch flipped in my head.

On the night of New Year’s Eve, my backpack was stolen from me on the train from Berlin to Amsterdam, and with it about $2000 worth of equipment, clothes, and so on. A portent for the year that was to come. I generally keep my private and public lives carefully separated, but perhaps I will offer you a peek behind the curtain today.

It seems like every week or two this year, another crisis presented itself, each manageable in isolation. Some were independent events, others snowballed as the same problems escalated. Gossip at the hackerspace, my personal life put on display and mocked. A difficult break-up in February, followed by a close friend facing their own relationship’s hurtful end. Another close friend – old, grave problems, once forgotten, remembered, and found to still be causing harm. Yet another friend, struggling to deal with depression and emotional abuse at the hands of their partner. Another friendship still: lost, perhaps someday to be found again.

Dependable Drew, an ear to listen, a shoulder to cry on, always knowing the right words to say, ready to help and proud to be there for his friends. Friends who, amidst these crises, are struggling to be there for him.

These events, set over the background of a world on fire.

One of the more difficult crises in my purview reached its crescendo one week ago, culminating in death. A selfish end for a selfish person, a person who had hurt people I love; a final, cruel cut to the wounds we were trying to heal.

I took time for myself throughout these endless weeks, looked after myself as best I could, and allowed my productivity to wane as necessary, unburdened by guilt in so doing. I marched on when I had the energy to, and made many achievements I’m proud of.

Something changed this week. I have often remarked that when you’re staring down a hard problem, one which might take years or even decades to finish, that you have two choices: give up or get to work. The years are going to pass either way. I am used to finding myself at the base of a mountain, picking up my shovel, and getting started. Equipped with this mindset, I have patiently ground down more than one mountain in my time. But this week, for the first time in my life, as I gazed upon that mountain, I felt intimidated.

I’m not sure what the purpose of this blog post is. Perhaps I’m sharing an experience that others might be able to relate to. Perhaps it’s healing in some way. Maybe it’s just indulgent.

I’m going to take the time I need to rest. I enjoy the company of wonderful colleagues at SourceHut, who have been happy to pick up some of the slack. I have established a formal group of maintainers for Hare and given them my blessing to work without seeking my approval. My projects will remain healthy as I take a leave. See you soon.

Who should lead us?

24 April 2023 at 00:00

Consider these two people, each captured in the midst of delivering a technical talk.

A picture of a young trans woman in a red dress A picture of a middle-aged white man in a red shirt

Based on appearances alone, what do you think of them?

The person on the left is a woman. She’s also pretty young, one might infer something about her level of experience accordingly. I imagine that she has led a much different life than I have, and may have a much different perspective, worldview, identity, and politics than I. Does she complain about sexism and discrimination in her work? Is she a feminist? Does she lean left or right on the political spectrum?

The person on the right looks like most of the hackers I’ve met. You’ve met someone who looks like this a thousand times. He is a man, white and middle-aged – that suggests a fair bit of experience. He probably doesn’t experience or concern himself with race or gender discrimination in the course of his work. He just focuses on the software. His life experiences probably map relatively well onto my own, and we may share a similar worldview and identity.

Making these assumptions is a part of human nature – it’s a useful shortcut in many situations. But they are assumptions based only on appearances. What are the facts?

The person on the right is Scott Guthrie, Vice President of Cloud and AI at Microsoft, giving a talk about Azure’s cloud services. He lives in an $11M house in Hunts Point, Washington. On the left is Alyssa Rosenzweig, main developer for the free software Panfrost GPU drivers and a trans woman, talking about how she reverse engineers proprietary graphics hardware.

You and I have a lot more in common with Alyssa than with Scott. The phone I have in my pocket right now would not work without her drivers. Alyssa humbles me with her exceptional talent and dedication, and the free software community is indebted to her. If you use ARM devices with free software, you owe something to Alyssa. As recently as February, her Wikipedia page was vandalized by someone who edited “she” and “her” to “he” and “him”.

Appearances should not especially matter when considering the merit of someone considered for a leadership role in our community, be it as a maintainer, thought leader, member of our foundations’ boards, etc. I am myself a white man, and I think I perform well in my leadership roles throughout the free software ecosystem. But it’s not my appearance that causes any controversy: someone with the approximate demographic shape of myself or Guthrie would cause no susurration when taking the stage.

It’s those like Alyssa, who aside from anything else is eminently qualified and well-deserving of her leadership role, who are often the target of ire and discrimination in the community. This is an experience shared by many people whose gender expression, skin color, or other traits differ from the “norm”. They’ve been telling us so for years.

Is it any wonder that our community is predominantly made up of white cisgendered men when anyone else is ostracized? It’s not because we’re predisposed to be better at this kind of work. It’s patently absurd to suppose that hackers whose identities and life experience differ from yours or mine cannot be good participants in and leaders of our movement. In actual fact, diverse teams produce better results. While the labor pool is disproportionately filled with white men, we can find many talented hackers who cannot be described as such. If we choose to be inspired by them, and led by them, we will discover new perspectives on our software, and on our movement and its broader place in the world. They can help us create a safe and inviting space for other talented hackers who identify with them. We will be more effective at our mission of bringing free software to everyone with their help.

Moreover, there are a lot of damned good hackers who don’t look like me, and I would be happy to follow their lead regardless of any other considerations.

The free software ecosystem (and the world at large) is not under threat from some woke agenda – a conspiracy theory which has been fabricated out of whole cloth. The people you fear are just people, much like you and I, and they only want to be treated as such. Asking them to shut up and get in line, to suppress their identity, experiences, and politics, to avoid confronting you with uncomfortable questions about your biases and privileges by way of their existence alone – it’s not right.

Forget the politics and focus on the software? It’s simply not possible. Free software is politics. Treating other people with respect, maturity, and professionalism, and valuing their contributions at any level, including leadership, regardless of their appearance or identity – that’s just part of being a good person. That is apolitical.


Alyssa gave her blessing regarding the use of her image and her example in this post. Thanks!

rc: a new shell for Unix

18 April 2023 at 00:00

rc is a Unix shell I’ve been working on over the past couple of weeks, though it’s been in the design stages for a while longer than that. It’s not done or ready for general use yet, but it is interesting, so let’s talk about it.

As the name (which is subject to change) implies, rc is inspired by the Plan 9 rc shell. It’s not an implementation of Plan 9 rc, however: it departs in many notable ways. I’ll assume most readers are more familiar with POSIX shell or Bash and skip many of the direct comparisons to Plan 9. Also, though most of the features work as described, the shell is a work-in-progress and some of the design I’m going over today has not been implemented yet.

Let’s start with the basics. Simple usage works much as you’d expect:

name=ddevault
echo Hello $name

But there’s already something important that might catch your eye here: the lack of quotes around $name. One substantial improvement rc makes over POSIX shells and Bash right off the bat is fixing our global shell quoting nightmare. There’s no need to quote variables!

# POSIX shell
x="hello world"
printf '%s\n' $x
# hello
# world

# rc
x="hello world"
printf '%s\n' $x
# hello world

Of course, the POSIX behavior is actually useful sometimes. rc provides for this by acknowledging that shells have not just one fundamental type (strings), but two: strings and lists of strings, i.e. argument vectors.

x=(one two three)
echo $x(1)  # prints first item ("one")
echo $x     # expands to arguments (echo "one" "two" "three")
echo $#x    # length operator: prints 3

x="echo hello world"
$x
# echo hello world: command not found

x=(echo hello world)
$x
# hello world

# expands to a string, list values separated with space:
$"x
# echo hello world: command not found

You can also slice up lists and get a subset of items:

x=(one two three four five)
echo $x(-4) # one two three four
echo $x(2-) # two three four five
echo $x(2-4) # two three four

A departure from Plan 9 rc is that the list operators can be used with strings for string operations as well:

x="hello world"
echo $#x     # 11
echo $x(2)   # e
echo $x(1-5) # hello

rc also supports loops. The simple case is iterating over the command line arguments:

% cat test.rc 
for (arg) {
	echo $arg
}
% rc test.rc one two three 
one
two
three

{ } is a command like any other; this can be simplified to for (arg) echo $arg. You can also enumerate any list with in:

list=(one two three)
for (item in $list) {
	echo $item
}

We also have while loops and if:

while (true) {
	if (test $x -eq 10) {
		echo ten
	} else {
		echo $x
	}
}

Functions are defined like so:

fn greet {
	echo Hello $1
}

greet ddevault

Again, any command can be used, so this can be simplified to fn greet echo $1. You can also add named parameters:

fn greet(user time) {
	echo Hello $user
	echo It is $time
}

greet ddevault `{date}

Note the use of `{script…} instead of $() for command expansion. Additional arguments are still placed in $*, allowing for the user to combine variadic-style functions with named arguments.

Here’s a more complex script that I run to perform sanity checks before applying patches:

#!/bin/rc
fn check_branch(branch) {
	if (test `{git rev-parse --abbrev-ref HEAD} != $branch) {
		echo "Error: not on master branch"
		exit 1
	}
}

fn check_uncommitted {
	if (test `{git status -suno | wc -l} -ne 0) {
		echo "Error: you have uncommitted changes"
		exit 1
	}
}

fn check_behind {
	if (test `{git rev-list "@{u}.." | wc -l} -ne 0) {
		echo "Error: your branch is behind upstream"
		exit 1
	}
}

check_branch master
check_uncommitted
check_behind
exec git pull

That’s a brief introduction to rc! Presently it clocks in at about 2500 lines of Hare. It’s not done yet, so don’t get too excited, but much of what’s described here is already working. Some other stuff which works but I didn’t mention include:

  • Boolean compound commands (x && y, x || y)
  • Pipelines, which can pipe arbitrary file descriptors (“x |[2] y”)
  • Redirects, also including arbitrary fds (“x >[2=1] file”)

It also has a formal context-free grammar, which is a work-in-progress but speaks to our desire to have a robust description of the shell available for users and other implementations. We use Ember Sawady’s excellent madeline for our interactive mode, which supports command line editing, history, ^r, and fish-style forward completion OOTB.

Future plans include:

  • Simple arithmetic expansion
  • Named pipe expansions
  • Sub-shells
  • switch statements
  • Port to ares
  • Find a new name, perhaps

It needs a small amount of polish, cleanup, and bugs fixed as well.

I hope you find it interesting! I will let you know when it’s done. Feel free to play with it in the meanwhile, and maybe send some patches?

The Free Software Foundation is dying

11 April 2023 at 00:00

The Free Software Foundation is one of the longest-running missions in the free software movement, effectively defining it. It provides a legal foundation for the movement and organizes activism around software freedom. The GNU project, closely related, has its own long story in our movement as the coding arm of the Free Software Foundation, taking these principles and philosophy into practice by developing free software; notably the GNU operating system that famously rests atop GNU/Linux.

Today, almost 40 years on, the FSF is dying.

Their achievements are unmistakable: we must offer them our gratitude and admiration for decades of accomplishments in establishing and advancing our cause. The principles of software freedom are more important than ever, and the products of these institutions remain necessary and useful – the GPL license family, GCC, GNU coreutils, and so on. Nevertheless, the organizations behind this work are floundering.

The Free Software Foundation must concern itself with the following ahead of all else:

  1. Disseminating free software philosophy
  2. Developing, publishing, and promoting copyleft licenses
  3. Overseeing the health of the free software movement

It is failing in each of these regards, and as its core mission fails, the foundation is investing its resources into distractions.

In its role as the thought-leaders of free software philosophy, the message of the FSF has a narrow reach. The organization’s messaging is tone-deaf, ineffective, and myopic. Hammering on about “GNU/Linux” nomenclature, antagonism towards our allies in the open source movement, maligning the audience as “useds” rather than “users”; none of this aids the cause. The pages and pages of dense philosophical essays and poorly organized FAQs do not provide a useful entry point or reference for the community. The message cannot spread like this.

As for copyleft, well, it’s no coincidence that many people struggle with the FSF’s approach. Do you, dear reader, know the difference between free software and copyleft? Many people assume that the MIT license is not free software because it’s not viral. The GPL family of licenses are essential for our movement, but few people understand its dense and esoteric language, despite the 16,000-word FAQ which supplements it. And hip new software isn’t using copyleft: over 1 million npm packages use a permissive license while fewer than 20,000 use the GPL; cargo sports a half-million permissive packages and another 20,000 or so GPL’d.

And is the free software movement healthy? This one gets an emphatic “yes!” – thanks to the open source movement and the near-equivalence between free software and open source software. There’s more free software than ever and virtually all new software contains free software components, and most people call it open source.

The FOSS community is now dominated by people who are beyond the reach of the FSF’s message. The broader community is enjoying a growth in the diversity of backgrounds and values represented, and the message does not reach these people. The FSF fails to understand its place in the world as a whole, or its relationship to the progressive movements taking place in the ecosystem and beyond. The foundation does not reach out to new leaders in the community, leaving them to form insular, weak institutions among themselves with no central leadership, and leaving us vulnerable to exploitation from growing movements like open core and commercial attacks on the free and open source software brand.

Reforms are sorely needed for the FSF to fulfill it basic mission. In particular, I call for the following changes:

  1. Reform the leadership. It’s time for Richard Stallman to go. His polemeic rhetoric rivals even my own, and the demographics he represents – to the exclusion of all others – is becoming a minority within the free software movement. We need more leaders of color, women, LGBTQ representation, and others besides. The present leadership, particularly from RMS, creates an exclusionary environment in a place where inclusion and representation are important for the success of the movement.
  2. Reform the institution. The FSF needs to correct its myopic view of the ecosystem, reach out to emerging leaders throughout the FOSS world, and ask them to take charge of the FSF’s mission. It’s these leaders who hold the reins of the free software movement today – not the FSF. If the FSF still wants to be involved in the movement, they need to recognize and empower the leaders who are pushing the cause forward.
  3. Reform the message. People depend on the FSF to establish a strong background in free software philosophy and practices within the community, and the FSF is not providing this. The message needs to be made much more accessible and level in tone, and the relationship between free software and open source needs to be reformed so that the FSF and OSI stand together as the pillars at the foundations of our ecosystem.
  4. Decouple the FSF from the GNU project. FSF and GNU have worked hand-in-hand over decades to build the movement from scratch, but their privileged relationship has become obsolete. The GNU project represents a minute fraction of the free software ecosystem today, and it’s necessary for the Free Software Foundation to stand independently of any particular project and focus on the health of the ecosystem as a whole.
  5. Develop new copyleft licenses. The GPL family of licenses has served us well, but we need to do better. The best copyleft license today is the MPL, whose terse form and accessible language outperforms the GPL in many respects. However, it does not provide a comprehensive answer to the needs of copyleft, and new licenses are required to fill other niches in the market – the FSF should write these licenses. Furthermore, the FSF should present the community with a free software perspective on licenses as a resource that project leaders can depend on to understand the importance of their licensing choice such that they understand the appeal of copyleft licenses without feeling pushed away from permissive approaches.

The free software movement needs a strong force uniting it: we face challenges from many sides, and today’s Free Software Foundation is not equal to the task. The FOSS ecosystem is flourishing, and it’s time for the FSF to step up to the wheel and direct its coming successes in the name of software freedom.

Writing Helios drivers in the Mercury driver environment

8 April 2023 at 00:00

Helios is a microkernel written in the Hare programming language and is part of the larger Ares operating system. You can watch my FOSDEM 2023 talk introducing Helios on PeerTube.

Let’s take a look at the new Mercury driver development environment for Helios.

As you may remember from my FOSDEM talk, the Ares operating system is built out of several layers which provide progressively higher-level environments for an operating system. At the bottom is the Helios microkernel, and today we’re going to talk about the second layer: the Mercury environment, which is used for writing and running device drivers in userspace. Let’s take a look at a serial driver written against Mercury and introduce some of the primitives used by driver authors in the Mercury environment.

Drivers for Mercury are written as normal ELF executables with an extra section called .manifest, which includes a file similar to the following (the provided example is for the serial driver we’ll be examining today):

[driver]
name=pcserial
desc=Serial driver for x86_64 PCs

[capabilities]
0:ioport = min=3F8, max=400
1:ioport = min=2E8, max=2F0
2:note = 
3:irq = irq=3, note=2
4:irq = irq=4, note=2
_:cspace = self
_:vspace = self
_:memory = pages=32

[services]
devregistry=

Helios uses a capability-based design, in which access to system resources (such as I/O ports, IRQs, or memory) is governed by capability objects. Each process has a capability space, which is a table of capabilities assigned to that process, and when performing operations (such as writing to an I/O port) the user provides the index of the desired capability in a register when invoking the appropriate syscall.

The manifest first specifies a list of capabilities required to operate the serial port. It requests, assigned static capability addresses, capabilities for the required I/O ports and IRQs, as well as a notification object which the IRQs will be delivered to. Some capability types, such as I/O ports, have configuration parameters, in this case the minimum and maximum port numbers which are relevant. The IRQ capabilities require a reference to a notification as well.

Limiting access to these capabilities provides very strong isolation between device drivers. On a monolithic kernel like Linux, a bug in the serial driver could compromise the entire system, but a vulnerability in our driver could, at worst, write garbage to your serial port. This model also provides better security than something like OpenBSD’s pledge by declaratively specifying what we need and nothing else.

Following the statically allocated capabilities, we request our own capability space and virtual address space, the former so we can copy and destroy our capabilities, and the latter so that we can map shared memory to perform reads and writes for clients. We also request 32 pages of memory, which we use to allocate page tables to perform those mappings; this will be changed later. These capabilities do not require any specific address for the driver to work, so we use “_” to indicate that any slot will suit our needs.

Mercury uses some vendor extensions over the System-V ABI to communicate information about these capabilities to the runtime. Notes about each of the _’d capabilities are provided by the auxiliary vector, and picked up by the Mercury runtime – for instance, the presence of a memory capability is detected on startup and is used to set up the allocator; the presence of a vspace capability is automatically wired up to the mmap implementation.

Each of these capabilities is implemented by the kernel, but additional services are available in userspace via endpoint capabilities. Each of these endpoints implements a particular API, as defined by a protocol definition file. This driver requires access to the device registry, so that it can create devices for its serial ports and expose them to clients.

These protocol definitions are written in a domain-specific language and parsed by ipcgen to generate client and server implementations of each. Here’s a simple protocol to start us off:

namespace io;

# The location with respect to which a seek operation is performed.
enum whence {
	# From the start of the file
	SET,
	# From the current offset
	CUR,
	# From the end of the file
	END,
};

# An object with file-like semantics.
interface file {
	# Reads up to amt bytes of data from a file.
	call read{pages: page...}(buf: uintptr, amt: size) size;

	# Writes up to amt bytes of data to a file.
	call write{pages: page...}(buf: uintptr, amt: size) size;

	# Seeks a file to a given offset, returning the new offset.
	call seek(offs: i64, w: whence) size;
};

Each interface includes a list of methods, each of which can take a number of capabilities and parameters, and return a value. The “read” call here, when implemented by a file-like object, accepts a list of memory pages to perform the read or write with (shared memory), as well as a pointer to the buffer address and size. Error handling is still a to-do.

ipcgen consumes these files and writes client or server code as appropriate. These are generated as part of the Mercury build process and end up in *_gen.ha files. The generated client code is filed away into the relevant modules (this protocol ends up at io/file_gen.ha), alongside various hand-written files which provide additional functionality and often wrap the IPC calls in a higher-level interface. The server implementations end up in the “serv” module, e.g. serv/io/file_gen.ha.

Let’s look at some of the generated client code for io::file objects:

// This file was generated by ipcgen; do not modify by hand
use helios;
use rt;

// ID for the file IPC interface.
export def FILE_ID: u32 = 0x9A533BB3;

// Labels for operations against file objects.
export type file_label = enum u64 {
	READ = FILE_ID << 16u64 | 1,
	WRITE = FILE_ID << 16u64 | 2,
	SEEK = FILE_ID << 16u64 | 3,
};

export fn file_read(
	ep: helios::cap,
	pages: []helios::cap,
	buf: uintptr,
	amt: size,
) size = {
	// ...
};

Each interface has a unique ID (generated from the FNV-1a hash of its fully qualified name), which is bitwise-OR’d with a list of operations to form call labels. The interface ID is used elsewhere; we’ll refer to it again later. Then each method generates an implementation which arranges the IPC details as necessary and invokes the “call” syscall against the endpoint capability.

The generated server code is a bit more involved. Some of the details are similar – FILE_ID is generated again, for instance – but there are some additional details as well. First is the generation of a vtable defining the functions implementing each operation:

// Implementation of a [[file]] object.
export type file_iface = struct {
	read: *fn_file_read,
	write: *fn_file_write,
	seek: *fn_file_seek,
};

We also define a file object which is subtyped by the implementation to store implementation details, and which provides to the generated code the required bits of state.

// Instance of an file object. Users may subtype this object to add
// instance-specific state.
export type file = struct {
	_iface: *file_iface,
	_endpoint: helios::cap,
};

Here’s an example of a subtype of file used by the initramfs to store additional state:

// An open file in the bootstrap filesystem
type bfs_file = struct {
	serv::io::file,
	fs: *bfs,
	ent: tar::entry,
	cur: io::off,
	padding: size,
};

The embedded serv::io::file structure here is populated with an implementation of file_iface, here simplified for illustrative purposes:

const bfs_file_impl = serv_io::file_iface {
	read = &bfs_file_read,
	write = &bfs_file_write,
	seek = &bfs_file_seek,
};

fn bfs_file_read(
	obj: *serv_io::file,
	pages: []helios::cap,
	buf: uintptr,
	amt: size,
) size = {
	let file = obj: *bfs_file;
	const fs = file.fs;
	const offs = (buf & rt::PAGEMASK): size;
	defer helios::destroy(pages...)!;

	assert(offs + amt <= len(pages) * rt::PAGESIZE);
	const buf = helios::map(rt::vspace, 0, map_flags::W, pages...)!: *[*]u8;

	let buf = buf[offs..offs+amt];
	// Not shown: reading the file data into this buffer
};

The implementation can prepare a file object and call dispatch on it to process client requests: this function blocks until a request arrives, decodes it, and invokes the appropriate function. Often this is incorporated into an event loop with poll to service many objects at once.

// Prepare a file object
const ep = helios::newendpoint()!;
append(fs.files, bfs_file {
	_iface = &bfs_file_impl,
	_endpoint = ep,
	fs = fs,
	ent = ent,
	cur = io::tell(fs.buf)!,
	padding = fs.rd.padding,
});

// ...

// Process requests associated with this file
serv::io::file_dispatch(file);

Okay, enough background: back to the serial driver. It needs to implement the following protocol:

namespace dev;
use io;

# TODO: Add busy error and narrow semantics

# Note: TWO is interpreted as 1.5 for some char lengths (5)
enum stop_bits {
	ONE,
	TWO,
};

enum parity {
	NONE,
	ODD,
	EVEN,
	MARK,
	SPACE,
};

# A serial device, which implements the file interface for reading from and
# writing to a serial port. Typical implementations may only support one read
# in-flight at a time, returning errors::busy otherwise.
interface serial :: io::file {
	# Returns the baud rate in Hz.
	call get_baud() uint;

	# Returns the configured number of bits per character.
	call get_charlen() uint;

	# Returns the configured number of stop bits.
	call get_stopbits() stop_bits;

	# Returns the configured parity setting.
	call get_parity() parity;

	# Sets the baud rate in Hz.
	call set_baud(hz: uint) void;

	# Sets the number of bits per character. Must be 5, 6, 7, or 8.
	call set_charlen(bits: uint) void;

	# Configures the number of stop bits to use.
	call set_stopbits(bits: stop_bits) void;

	# Configures the desired parity.
	call set_parity(parity: parity) void;
};

This protocol inherits the io::file interface, so the serial port is usable like any other file for reads and writes. It additionally defines serial-specific methods, such as configuring the baud rate or parity. The generated interface we’ll have to implement looks something like this, embedding the io::file_iface struct:

export type serial_iface = struct {
	io::file_iface,
	get_baud: *fn_serial_get_baud,
	get_charlen: *fn_serial_get_charlen,
	get_stopbits: *fn_serial_get_stopbits,
	get_parity: *fn_serial_get_parity,
	set_baud: *fn_serial_set_baud,
	set_charlen: *fn_serial_set_charlen,
	set_stopbits: *fn_serial_set_stopbits,
	set_parity: *fn_serial_set_parity,
}

Time to dive into the implementation. Recall the driver manifest, which provides the serial driver with a suitable environment:

[driver]
name=pcserial
desc=Serial driver for x86_64 PCs

[capabilities]
0:ioport = min=3F8, max=400
1:ioport = min=2E8, max=2F0
2:note = 
3:irq = irq=3, note=2
4:irq = irq=4, note=2
_:cspace = self
_:vspace = self
_:memory = pages=32

[services]
devregistry=

I/O ports for reading and writing to the serial devices, IRQs for receiving serial-related interrupts, a device registry to add our serial devices to the system, and a few extra things for implementation needs. Some of these are statically allocated, some of them are provided via the auxiliary vector. Our serial driver opens by defining constants for the statically allocated capabilities:

def IOPORT_A: helios::cap = 0;
def IOPORT_B: helios::cap = 1;
def IRQ: helios::cap = 2;
def IRQ3: helios::cap = 3;
def IRQ4: helios::cap = 4;

The first thing we do on startup is create a serial device.

export fn main() void = {
	let serial0: helios::cap = 0;
	const registry = helios::service(sys::DEVREGISTRY_ID);
	sys::devregistry_new(registry, dev::SERIAL_ID, &serial0);
	helios::destroy(registry)!;
	// ...

The device registry is provided via the aux vector, and we can use helios::service to look it up by its interface ID. Then we use the devregistry::new operation to create a serial device:

# Device driver registry.
interface devregistry {
	# Creates a new device implementing the given interface ID using the
	# provided endpoint capability and returns its assigned serial number.
	call new{; out}(iface: u64) uint;
};

After this we can destroy the registry – we won’t need it again and it’s best to get rid of it so that we can work with the minimum possible privileges at runtime. After this we initialize the serial port, acknowledge any interrupts that might have been pending before we got started, an enter the main loop.

com_init(&ports[0], serial0);

helios::irq_ack(IRQ3)!;
helios::irq_ack(IRQ4)!;

let poll: [_]pollcap = [
	pollcap { cap = IRQ, events = pollflags::RECV, ... },
	pollcap { cap = serial0, events = pollflags::RECV, ... },
];
for (true) {
	helios::poll(poll)!;
	if (poll[0].revents & pollflags::RECV != 0) {
		dispatch_irq();
	};
	if (poll[1].revents & pollflags::RECV != 0) {
		dispatch_serial(&ports[0]);
	};
};

The dispatch_serial function is of interest, as this provides the implementation of the serial object we just created with the device registry.

type comport = struct {
	dev::serial,
	port: u16,
	rbuf: [4096]u8,
	wbuf: [4096]u8,
	rpending: []u8,
	wpending: []u8,
};

fn dispatch_serial(dev: *comport) void = {
	dev::serial_dispatch(dev);
};

const serial_impl = dev::serial_iface {
	read = &serial_read,
	write = &serial_write,
	seek = &serial_seek,
        get_baud = &serial_get_baud,
        get_charlen = &serial_get_charlen,
        get_stopbits = &serial_get_stopbits,
        get_parity = &serial_get_parity,
        set_baud = &serial_set_baud,
        set_charlen = &serial_set_charlen,
        set_stopbits = &serial_set_stopbits,
        set_parity = &serial_set_parity,
};

fn serial_read(
	obj: *io::file,
	pages: []helios::cap,
	buf: uintptr,
	amt: size,
) size = {
	const port = obj: *comport;
	const offs = (buf & rt::PAGEMASK): size;
	const buf = helios::map(rt::vspace, 0, map_flags::W, pages...)!: *[*]u8;
	const buf = buf[offs..offs+amt];

	if (len(port.rpending) != 0) {
		defer helios::destroy(pages...)!;
		return rconsume(port, buf);
	};

	pages_static[..len(pages)] = pages[..];
	pending_read = read {
		reply = helios::store_reply(helios::CADDR_UNDEF)!,
		pages = pages_static[..len(pages)],
		buf = buf,
	};
	return 0;
};

// (other functions omitted)

We’ll skip much of the implementation details for this specific driver, but I’ll show you how read works at least. It’s relatively straightforward: first we mmap the buffer provided by the caller. If there’s already readable data pending from the serial port (stored in that rpending slice in the comport struct, which is a slice of the statically-allocated rbuf field), we copy it into the buffer and return the number of bytes we had ready. Otherwise, we stash details about the caller, storing the special reply capability in our cspace (this is one of the reasons we need cspace = self in our manifest) so we can reply to this call once data is available. Then we return to the main loop.

The main loop also wakes up on an interrupt, and we have an interrupt unmasked on the serial device to wake us whenever there’s data ready to be read. Eventually this gets us here, which finishes the call we saved earlier:

// Reads data from the serial port's RX FIFO.
fn com_read(com: *comport) size = {
	let n: size = 0;
	for (comin(com.port, LSR) & RBF == RBF; n += 1) {
		const ch = comin(com.port, RBR);
		if (len(com.rpending) < len(com.rbuf)) {
			// If the buffer is full we just drop chars
			static append(com.rpending, ch);
		};
	};

	if (pending_read.reply != 0) {
		const n = rconsume(com, pending_read.buf);
		helios::send(pending_read.reply, 0, n)!;
		pending_read.reply = 0;
		helios::destroy(pending_read.pages...)!;
	};

	return n;
};

I hope that gives you a general idea of how drivers work in this environment! I encourage you to read the full implementation if you’re curious to know more about the serial driver in particular – it’s just 370 lines of code.

The last thing I want to show you is how the driver gets executed in the first place. When Helios boots up, it starts /sbin/sysinit, which is provided by Mercury and offers various low-level userspace runtime services, such as the device registry and bootstrap filesystem we saw earlier. After setting up its services, sysinit executes /sbin/usrinit, which is provided by the next layer up (Gaia, eventually) and sets up the rest of the system according to user policy, mounting filesystems and starting up drivers and such. At the moment, usrinit is fairly simple, and just runs a little demo. Here it is in full:

use dev;
use fs;
use helios;
use io;
use log;
use rt;
use sys;

export fn main() void = {
	const fs = helios::service(fs::FS_ID);
	const procmgr = helios::service(sys::PROCMGR_ID);
	const devmgr = helios::service(sys::DEVMGR_ID);
	const devload = helios::service(sys::DEVLOADER_ID);

	log::printfln("[usrinit] Running /sbin/drv/serial");
	let proc: helios::cap = 0;
	const image = fs::open(fs, "/sbin/drv/serial")!;
	sys::procmgr_new(procmgr, &proc);
	sys::devloader_load(devload, proc, image);
	sys::process_start(proc);

	let serial: helios::cap = 0;
	log::printfln("[usrinit] open device serial0");
	sys::devmgr_open(devmgr, dev::SERIAL_ID, 0, &serial);

	let buf: [rt::PAGESIZE]u8 = [0...];
	for (true) {
		const n = match (io::read(serial, buf)!) {
		case let n: size =>
			yield n;
		case io::EOF =>
			break;
		};

		// CR => LF
		for (let i = 0z; i < n; i += 1) {
			if (buf[i] == '\r') {
				buf[i] = '\n';
			};
		};

		// echo
		io::write(serial, buf[..n])!;
	};
};

Each of the services shown at the start are automatically provided in usrinit’s aux vector by sysinit, and includes all of the services required to bootstrap the system. This includes a filesystem (the initramfs), a process manager (to start up new processes), the device manager, and the driver loader service.

usrinit starts by opening up /sbin/drv/serial (the serial driver, of course) from the provided initramfs using fs::open, which is a convenience wrapper around the filesystem protocol. Then we create a new process with the process manager, which by default has an empty address space – we could load a normal process into it with sys::process_load, but we want to load a driver, so we use the devloader interface instead. Then we start the process and boom: the serial driver is online.

The serial driver registers itself with the device registry, which means that we can use the device manager to open the 0th device which implements the serial interface. Since this is compatible with the io::file interface, it can simply be used normally with io::read and io::write to utilize the serial port. The main loop simply echos data read from the serial port back out. Simple!


That’s a quick introduction to the driver environment provided by Mercury. I intend to write a few more drivers soon myself – PC keyboard, framebuffer, etc – and set up a simple shell. We have seen a few sample drivers written pre-Mercury which would be nice to bring into this environment, such as virtio networking and block devices. It will be nice to see them re-introduced in an environment where they can provide useful services to the rest of userspace.

If you’re interested in learning more about Helios or Mercury, consult ares-os.org for documentation – though beware of the many stub pages. If you have any questions or want to get involved in writing some drivers yourself, jump into our IRC channel: #helios on Libera Chat.

When to comment that code

9 March 2023 at 00:00

My software tends to have a surprisingly low number of comments. One of my projects, scdoc, has 25 comments among its 1,133 lines of C code, or 2%, compared to the average of 19%.1 Naturally, I insist that my code is well-written in spite of this divergence from the norm. Allow me to explain.

The philosophy and implementation of code comments varies widely in the industry, and some view comment density as a proxy for code quality.2 I’ll state my views here, but will note that yours may differ and I find that acceptable; I am not here to suggest that your strategy is wrong and I will happily adopt it when I write a patch for your codebase.

Let’s begin with an illustrative example from one of my projects:

// Reads the next entry from an EFI [[FILE_PROTOCOL]] handle of an open
// directory. The return value is statically allocated and will be overwritten
// on the next call.
export fn readdir(dir: *FILE_PROTOCOL) (*FILE_INFO | void | error) = {
	// size(FILE_INFO) plus reserve up to 512 bytes for file name (FAT32
	// maximum, times two for wstr encoding)
	static let buf: [FILE_INFO_SIZE + 512]u8 = [0...];
	const n = read(dir, buf)?;
	if (n == 0) {
		return;
	};
	return &buf[0]: *FILE_INFO;
};

This code illustrates two of my various approaches to writing comments. The first comment is a documentation comment: the intended audience is the consumer of this API. The call-site has access to the following information:

  • This comment
  • The name of the function, and the module in which it resides (efi::readdir)
  • The parameter names and types
  • The return type

The goal is for the user of this function to gather enough information from these details to correctly utilize this API.

The module in which it resides suggests that this function interacts with the EFI (Extensible Firmware Interface) standard, and the user would be wise to pair a reading of this code (or API) with skimming the relevant standard. Indeed, the strategic naming of the FILE_PROTOCOL and FILE_INFO types (notably written in defiance of the Hare style guide), provide hints to the relevant parts of the EFI specification to read for a complete understanding of this code.

The name of the function is also carefully chosen to carry some weight: it is a reference to the Unix readdir function, which brings with it an intuition about its purpose and usage for programmers familiar with a Unix environment.

The return type also provides hints about the function’s use: it may return either a FILE_INFO pointer, void (nothing), or an error. Without reading the documentation string, and taking the name and return type into account, we might (correctly) surmise that we need to call this function repeatedly to read file details out of a directory until it returns void, indicating that all entries have been processed, handling any errors which might occur along the way.

We have established a lot of information about this function without actually reading the comment; in my philosophy of programming I view this information as a critical means for the author to communicate to the user, and we can lean on it to reduce the need for explicit documentation. Nevertheless, the documentation comment adds something here. The first sentence is a relatively information-sparse summary of the function’s purpose, and mainly exists to tick a box in the Hare style guide.3 The second sentence is the only real reason this comment exists: to clarify an important detail for the user which is not apparent from the function signature, namely the storage semantics associated with the return value.

Let’s now study the second comment’s purpose:

// size(FILE_INFO) plus reserve up to 512 bytes for file name (FAT32
// maximum, times two for wstr encoding)
static let buf: [FILE_INFO_SIZE + 512]u8 = [0...];

This comment exists to explain the use of the magic constant of 512. The audience of this comment is someone reading the implementation of this function. This audience has access to a different context than the user of the function, for instance they are expected to have a more comprehensive knowledge of EFI and are definitely expected to be reading the specification to a much greater degree of detail. We can and should lean on that context to make our comments more concise and useful.

An alternative writing which does not rely on this context, and which in my view is strictly worse, may look like the following:

// The FILE_INFO structure includes the file details plus a variable length
// array for the filename. The underlying filesystem is always FAT32 per the
// EFI specification, which has a maximum filename length of 256 characters. The
// filename is encoded as a wide-string (UCS-2), which encodes two bytes per
// character, and is not NUL-terminated, so we need to reserve up to 512 bytes
// for the filename.
static let buf: [FILE_INFO_SIZE + 512]u8 = [0...];

The target audience of this comment should have a reasonable understanding of EFI. We simply need to clarify that this constant is the FAT32 max filename length, times two to account for the wstr encoding, and our magic constant is sufficiently explained.

Let’s move on to another kind of comment I occasionally write: medium-length prose. These often appear at the start of a function or the start of a file and serve to add context to the implementation, to justify the code’s existence or explain why it works. Another sample:

fn init_pagetables() void = {
	// 0xFFFF0000xxxxxxxx - 0xFFFF0200xxxxxxxx: identity map
	// 0xFFFF0200xxxxxxxx - 0xFFFF0400xxxxxxxx: identity map (dev)
	// 0xFFFF8000xxxxxxxx - 0xFFFF8000xxxxxxxx: kernel image
	//
	// L0[0x000]    => L1_ident
	// L0[0x004]    => L1_devident
	// L1_ident[*]  => 1 GiB identity mappings
	// L0[0x100]    => L1_kernel
	// L1_kernel[0] => L2_kernel
	// L2_kernel[0] => L3_kernel
	// L3_kernel[0] => 4 KiB kernel pages
	L0[0x000] = PT_TABLE | &L1_ident: uintptr | PT_AF;
	L0[0x004] = PT_TABLE | &L1_devident: uintptr | PT_AF;
	L0[0x100] = PT_TABLE | &L1_kernel: uintptr | PT_AF;
	L1_kernel[0] = PT_TABLE | &L2_kernel: uintptr | PT_AF;
	L2_kernel[0] = PT_TABLE | &L3_kernel: uintptr | PT_AF;
	for (let i = 0u64; i < len(L1_ident): u64; i += 1) {
		L1_ident[i] = PT_BLOCK | (i * 0x40000000): uintptr |
			PT_NORMAL | PT_AF | PT_ISM | PT_RW;
	};
	for (let i = 0u64; i < len(L1_devident): u64; i += 1) {
		L1_devident[i] = PT_BLOCK | (i * 0x40000000): uintptr |
			PT_DEVICE | PT_AF | PT_ISM | PT_RW;
	};
};

This comment shares a trait with the previous example: its purpose, in part, is to justify magic constants. It explains the indices of the arrays by way of the desired address space, and a perceptive reader will notice that 1 GiB = 1073741824 bytes = 0x40000000 bytes.

To fully understand this, we must again consider the intended audience. This is an implementation comment, so the reader is an implementer. They will need to possess some familiarity with the behavior of page tables to be productive in this code, and they likely have the ARM manual up on their second monitor. This comment simply fills in the blanks for an informed reader.

There are two additional kinds of comments I often write: TODO and XXX.

A TODO comment indicates some important implementation deficiency; it must be addressed at some point in the future and generally indicates that the function does not meet its stated interface and is often accompanied by an assertion, or a link to a ticket on the bug tracker, or both.

assert(ep.send == null); // TODO: support multiple senders

This function should support multiple senders, but does not; an assertion here prevents the code from running under conditions it does not yet support and the TODO comment indicates that this should be addressed in the future. The target audience for this comment is someone who brings about these conditions and runs into the assertion failure.

fn memory_empty(mem: *memory) bool = {
	// XXX: This O(n) linked list traversal is bad
	let next = mem.next;
	let pages = 0u;
	for (next != FREELIST_END; pages += 1) {
		const addr = mem.phys + (next * mem::PAGESIZE): uintptr;
		const ptr = mem::phys_tokernel(addr): *uint;
		next = *ptr;
	};
	return pages == mem.pages;
};

Here we find an example of an XXX comment. This code is correct: it implements the function’s interface perfectly. However, given its expected usage, a performance of O(n) is not great: this function is expected to be used in hot paths. This comment documents the deficiency, and provides a hint to a reader that might be profiling this code in regards to a possible improvement.

One final example:

// Invalidates the TLB for a virtual address.
export fn invalidate(virt: uintptr) void = {
	// TODO: Notify other cores (XXX SMP)
	invlpg(virt);
};

This is an atypical usage of XXX, but one which I still occasionally reach for. Here we have a TODO comment which indicates a case which this code does not consider, but which must be addressed in the future: it will have to raise an IPI to get other cores to invalidate the affected virtual address. However, this is one of many changes which fall under a broader milestone of SMP support, and the “XXX SMP” comment is here to make it easy to grep through the codebase for any places which are known to require attention while implementing SMP support. An XXX comment is often written for the purpose of being easily found with grep.

That sums up most of the common reasons I will write a comment in my software. Each comment is written considering a target audience and the context provided by the code in which it resides, and aims to avoid stating redundant information within these conditions. It’s for this reason that my code is sparse on comments: I find the information outside of the comments equally important and aim to be concise such that a comment is not redundant with information found elsewhere.

Hopefully this post inspired some thought in you, to consider your comments deliberately and to be more aware of your ability to communicate information in other ways. Even if you chose to write your comments more densely than I do, I hope you will take care to communicate well through other mediums in your code as well.


  1. O. Arafat and D. Riehle, “The comment density of open source software code,” 2009 31st International Conference on Software Engineering - Companion Volume, Vancouver, BC, Canada, 2009, pp. 195-198, doi: 10.1109/ICSE-COMPANION.2009.5070980. ↩︎

  2. I hold this view weakly, but reverse of the norm: I consider a high comment density a sign that the code quality may be poor. ↩︎

  3. Which states that all exported functions that the module consumer is expected to use should have a comment, and that exported but undocumented symbols are exported to fulfill an implementation detail and not to provide a useful interface. ↩︎

Porting Helios to aarch64 for my FOSDEM talk, part one

20 February 2023 at 00:00

Helios is a microkernel written in the Hare programming language, and the subject of a talk I did at FOSDEM earlier this month. You can watch the talk here if you like:

A while ago I promised someone that I would not do any talks on Helios until I could present them from Helios itself, and at FOSDEM I made good on that promise: my talk was presented from a Raspberry Pi 4 running Helios. The kernel was originally designed for x86_64 (though we were careful to avoid painting ourselves into any corners so that we could port it to more architectures later on), and I initially planned to write an Intel HD Graphics driver so that I could drive the projector from my laptop. But, after a few days spent trying to comprehend the IHD manuals, I decided it would be much easier to port the entire system to aarch64 and write a driver for the much-simpler RPi GPU instead. 42 days later the port was complete, and a week or so after that I successfully presented the talk at FOSDEM. In a series of blog posts, I will take a look at those 42 days of work and explain how the aarch64 port works. Today’s post focuses on the bootloader.

The Helios boot-up process is:

  1. Bootloader starts up and loads the kernel, then jumps to it
  2. The kernel configures the system and loads the init process
  3. Kernel provides runtime services to init (and any subsequent processes)

In theory, the port to aarch64 would address these steps in order, but in practice step (2) relies heavily on the runtime services provided by step (3), so much of the work was ordered 1, 3, 2. This blog post focuses on part 1, I’ll cover parts 2 and 3 and all of the fun problems they caused in later posts.

In any case, the bootloader was the first step. Some basic changes to the build system established boot/+aarch64 as the aarch64 bootloader, and a simple qemu-specific ARM kernel was prepared which just gave a little “hello world” to demonstrate the multi-arch build system was working as intended. More build system refinements would come later, but it’s off to the races from here. Targeting qemu’s aarch64 virt platform was useful for most of the initial debugging and bring-up (and is generally useful at all times, as a much easier platform to debug than real hardware); the first tests on real hardware came much later.

Booting up is a sore point on most systems. It involves a lot of arch-specific procedures, but also generally calls for custom binary formats and annoying things like disk drivers — which don’t belong in a microkernel. So the Helios bootloaders are separated from the kernel proper, which is a simple ELF executable. The bootloader loads this ELF file into memory, configures a few simple things, then passes some information along to the kernel entry point. The bootloader’s memory and other resources are hereafter abandoned and are later reclaimed for general use.

On aarch64 the boot story is pretty abysmal, and I wanted to avoid adding the SoC-specific complexity which is endemic to the platform. Thus, two solutions are called for: EFI and device trees. At the bootloader level, EFI is the more important concern. For qemu-virt and Raspberry Pi, edk2 is the free-software implementation of choice when it comes to EFI. The first order of business is producing an executable which can be loaded by EFI, which is, rather unfortunately, based on the Windows COFF/PE32+ format. I took inspiration from Linux and made an disgusting EFI stub solution, which involves hand-writing a PE32+ header in assembly and doing some truly horrifying things with binutils to massage everything into order. Much of the header is lifted from Linux:

.section .text.head
.global base
base:
.L_head:
	/* DOS header */
	.ascii "MZ"
	.skip 58
	.short .Lpe_header - .L_head
	.align 4
.Lpe_header:
	.ascii "PE\0\0"
	.short 0xAA64                              /* Machine = AARCH64 */
	.short 2                                   /* NumberOfSections */
	.long 0                                    /* TimeDateStamp */
	.long 0                                    /* PointerToSymbolTable */
	.long 0                                    /* NumberOfSymbols */
	.short .Lsection_table - .Loptional_header /* SizeOfOptionalHeader */
	/* Characteristics:
	 * IMAGE_FILE_EXECUTABLE_IMAGE |
	 * IMAGE_FILE_LINE_NUMS_STRIPPED |
	 * IMAGE_FILE_DEBUG_STRIPPED */
	.short 0x206
.Loptional_header:
	.short 0x20b                     /* Magic = PE32+ (64-bit) */
	.byte 0x02                       /* MajorLinkerVersion */
	.byte 0x14                       /* MinorLinkerVersion */
	.long _data - .Lefi_header_end   /* SizeOfCode */
	.long __pecoff_data_size         /* SizeOfInitializedData */
	.long 0                          /* SizeOfUninitializedData */
	.long _start - .L_head           /* AddressOfEntryPoint */
	.long .Lefi_header_end - .L_head /* BaseOfCode */
.Lextra_header:
	.quad 0                          /* ImageBase */
	.long 4096                       /* SectionAlignment */
	.long 512                        /* FileAlignment */
	.short 0                         /* MajorOperatingSystemVersion */
	.short 0                         /* MinorOperatingSystemVersion */
	.short 0                         /* MajorImageVersion */
	.short 0                         /* MinorImageVersion */
	.short 0                         /* MajorSubsystemVersion */
	.short 0                         /* MinorSubsystemVersion */
	.long 0                          /* Reserved */

	.long _end - .L_head             /* SizeOfImage */

	.long .Lefi_header_end - .L_head /* SizeOfHeaders */
	.long 0                          /* CheckSum */
	.short 10                        /* Subsystem = EFI application */
	.short 0                         /* DLLCharacteristics */
	.quad 0                          /* SizeOfStackReserve */
	.quad 0                          /* SizeOfStackCommit */
	.quad 0                          /* SizeOfHeapReserve */
	.quad 0                          /* SizeOfHeapCommit */
	.long 0                          /* LoaderFlags */
	.long 6                          /* NumberOfRvaAndSizes */

	.quad 0 /* Export table */
	.quad 0 /* Import table */
	.quad 0 /* Resource table */
	.quad 0 /* Exception table */
	.quad 0 /* Certificate table */
	.quad 0 /* Base relocation table */

.Lsection_table:
	.ascii ".text\0\0\0"              /* Name */
	.long _etext - .Lefi_header_end   /* VirtualSize */
	.long .Lefi_header_end - .L_head  /* VirtualAddress */
	.long _etext - .Lefi_header_end   /* SizeOfRawData */
	.long .Lefi_header_end - .L_head  /* PointerToRawData */
	.long 0                           /* PointerToRelocations */
	.long 0                           /* PointerToLinenumbers */
	.short 0                          /* NumberOfRelocations */
	.short 0                          /* NumberOfLinenumbers */
	/* IMAGE_SCN_CNT_CODE | IMAGE_SCN_MEM_READ | IMAGE_SCN_MEM_EXECUTE */
	.long 0x60000020

	.ascii ".data\0\0\0"        /* Name */
	.long __pecoff_data_size    /* VirtualSize */
	.long _data - .L_head       /* VirtualAddress */
	.long __pecoff_data_rawsize /* SizeOfRawData */
	.long _data - .L_head       /* PointerToRawData */
	.long 0                     /* PointerToRelocations */
	.long 0                     /* PointerToLinenumbers */
	.short 0                    /* NumberOfRelocations */
	.short 0                    /* NumberOfLinenumbers */
	/* IMAGE_SCN_CNT_INITIALIZED_DATA | IMAGE_SCN_MEM_READ | IMAGE_SCN_MEM_WRITE */
	.long 0xc0000040

.balign 0x10000
.Lefi_header_end:

.global _start
_start:
	stp x0, x1, [sp, -16]!

	adrp x0, base
	add x0, x0, #:lo12:base
	adrp x1, _DYNAMIC
	add x1, x1, #:lo12:_DYNAMIC
	bl relocate
	cmp w0, #0
	bne 0f

	ldp x0, x1, [sp], 16

	b bmain

0:
	/* relocation failed */
	add sp, sp, -16
	ret

The specific details about how any of this works are complex and unpleasant, I’ll refer you to the spec if you’re curious, and offer a general suggestion that cargo-culting my work here would be a lot easier than understanding it should you need to build something similar.1

Note the entry point for later; we store two arguments from EFI (x0 and x1) on the stack and eventually branch to bmain.

This file is assisted by the linker script:

ENTRY(_start)
OUTPUT_FORMAT(elf64-littleaarch64)

SECTIONS {
	/DISCARD/ : {
		*(.rel.reloc)
		*(.eh_frame)
		*(.note.GNU-stack)
		*(.interp)
		*(.dynsym .dynstr .hash .gnu.hash)
	}

	. = 0xffff800000000000;

	.text.head : {
		_head = .;
		KEEP(*(.text.head))
	}

	.text : ALIGN(64K) {
		_text = .;
		KEEP(*(.text))
		*(.text.*)
		. = ALIGN(16);
		*(.got)
	}

	. = ALIGN(64K);
	_etext = .;

	.dynamic : {
		*(.dynamic)
	}

	.data : ALIGN(64K) {
		_data = .;
		KEEP(*(.data))
		*(.data.*)

		/* Reserve page tables */
		. = ALIGN(4K);
		L0 = .;
		. += 512 * 8;
		L1_ident = .;
		. += 512 * 8;
		L1_devident = .;
		. += 512 * 8;
		L1_kernel = .;
		. += 512 * 8;
		L2_kernel = .;
		. += 512 * 8;
		L3_kernel = .;
		. += 512 * 8;
	}

	.rela.text : {
		*(.rela.text)
		*(.rela.text*)
	}
	.rela.dyn : {
		*(.rela.dyn)
	}
	.rela.plt : {
		*(.rela.plt)
	}
	.rela.got : {
		*(.rela.got)
	}
	.rela.data : {
		*(.rela.data)
		*(.rela.data*)
	}

	.pecoff_edata_padding : {
		BYTE(0);
		. = ALIGN(512);
	}
	__pecoff_data_rawsize = ABSOLUTE(. - _data);
	_edata = .;

	.bss : ALIGN(4K) {
		KEEP(*(.bss))
		*(.bss.*)
		*(.dynbss)
	}

	. = ALIGN(64K);
	__pecoff_data_size = ABSOLUTE(. - _data);
	_end = .;
}

Items of note here are the careful treatment of relocation sections (cargo-culted from earlier work on RISC-V with Hare; not actually necessary as qbe generates PIC for aarch64)2 and the extra symbols used to gather information for the PE32+ header. Padding is also added in the required places, and static aarch64 page tables are defined for later use.

This is built as a shared object, and the Makefile mutilates reformats the resulting ELF file to produce a PE32+ executable:

$(BOOT)/bootaa64.so: $(BOOT_OBJS) $(BOOT)/link.ld
	$(LD) -Bsymbolic -shared --no-undefined \
		-T $(BOOT)/link.ld \
		$(BOOT_OBJS) \
		-o $@

$(BOOT)/bootaa64.efi: $(BOOT)/bootaa64.so
	$(OBJCOPY) -Obinary \
		-j .text.head -j .text -j .dynamic -j .data \
		-j .pecoff_edata_padding \
		-j .dynstr -j .dynsym \
		-j .rel -j .rel.* -j .rel* \
		-j .rela -j .rela.* -j .rela* \
		$< $@

With all of this mess sorted, and the PE32+ entry point branching to bmain, we can finally enter some Hare code:

export fn bmain(
	image_handle: efi::HANDLE,
	systab: *efi::SYSTEM_TABLE,
) efi::STATUS = {
    // ...
};

Getting just this far took 3 full days of work.

Initially, the Hare code incorporated a lot of proof-of-concept work from Alexey Yerin’s “carrot” kernel prototype for RISC-V, which also booted via EFI. Following the early bringing-up of the bootloader environment, this was refactored into a more robust and general-purpose EFI support layer for Helios, which will be applicable to future ports. The purpose of this module is to provide an idiomatic Hare-oriented interface to the EFI boot services, which the bootloader makes use of mainly to read files from the boot media and examine the system’s memory map.

Let’s take a look at the first few lines of bmain:

efi::init(image_handle, systab)!;

const eficons = eficons_init(systab);
log::setcons(&eficons);
log::printfln("Booting Helios aarch64 via EFI");

if (readel() == el::EL3) {
	log::printfln("Booting from EL3 is not supported");
	return efi::STATUS::LOAD_ERROR;
};

let mem = allocator { ... };
init_mmap(&mem);
init_pagetables();

Significant build system overhauls were required such that Hare modules from the kernel like log (and, later, other modules like elf) could be incorporated into the bootloader, simplifying the process of implementing more complex bootloaders. The first call of note here is init_mmap, which scans the EFI memory map and prepares a simple high-watermark allocator to be used by the bootloader to allocate memory for the kernel image and other items of interest. It’s quite simple, it just finds the largest area of general-purpose memory and sets up an allocator with it:

// Loads the memory map from EFI and initializes a page allocator using the
// largest area of physical memory.
fn init_mmap(mem: *allocator) void = {
	const iter = efi::iter_mmap()!;
	let maxphys: uintptr = 0, maxpages = 0u64;
	for (true) {
		const desc = match (efi::mmap_next(&iter)) {
		case let desc: *efi::MEMORY_DESCRIPTOR =>
			yield desc;
		case void =>
			break;
		};
		if (desc.DescriptorType != efi::MEMORY_TYPE::CONVENTIONAL) {
			continue;
		};
		if (desc.NumberOfPages > maxpages) {
			maxphys = desc.PhysicalStart;
			maxpages = desc.NumberOfPages;
		};
	};
	assert(maxphys != 0, "No suitable memory area found for kernel loader");
	assert(maxpages <= types::UINT_MAX);
	pagealloc_init(mem, maxphys, maxpages: uint);
};

init_pagetables is next. This populates the page tables reserved by the linker with the desired higher-half memory map, illustrated in the comments shown here:

fn init_pagetables() void = {
	// 0xFFFF0000xxxxxxxx - 0xFFFF0200xxxxxxxx: identity map
	// 0xFFFF0200xxxxxxxx - 0xFFFF0400xxxxxxxx: identity map (dev)
	// 0xFFFF8000xxxxxxxx - 0xFFFF8000xxxxxxxx: kernel image
	//
	// L0[0x000]    => L1_ident
	// L0[0x004]    => L1_devident
	// L1_ident[*]  => 1 GiB identity mappings
	// L0[0x100]    => L1_kernel
	// L1_kernel[0] => L2_kernel
	// L2_kernel[0] => L3_kernel
	// L3_kernel[0] => 4 KiB kernel pages
	L0[0x000] = PT_TABLE | &L1_ident: uintptr | PT_AF;
	L0[0x004] = PT_TABLE | &L1_devident: uintptr | PT_AF;
	L0[0x100] = PT_TABLE | &L1_kernel: uintptr | PT_AF;
	L1_kernel[0] = PT_TABLE | &L2_kernel: uintptr | PT_AF;
	L2_kernel[0] = PT_TABLE | &L3_kernel: uintptr | PT_AF;
	for (let i = 0u64; i < len(L1_ident): u64; i += 1) {
		L1_ident[i] = PT_BLOCK | (i * 0x40000000): uintptr |
			PT_NORMAL | PT_AF | PT_ISM | PT_RW;
	};
	for (let i = 0u64; i < len(L1_devident): u64; i += 1) {
		L1_devident[i] = PT_BLOCK | (i * 0x40000000): uintptr |
			PT_DEVICE | PT_AF | PT_ISM | PT_RW;
	};
};

In short, we want three larger memory regions to be available: an identity map, where physical memory addresses correlate 1:1 with virtual memory, an identity map configured for device MMIO (e.g. with caching disabled), and an area to load the kernel image. The first two are straightforward, they use uniform 1 GiB mappings to populate their respective page tables. The latter is slightly more complex, ultimately the kernel is loaded in 4 KiB pages so we need to set up intermediate page tables for that purpose.

We cannot actually enable these page tables until we’re finished making use of the EFI boot services — the EFI specification requires us to preserve the online memory map at this stage of affairs. However, this does lay the groundwork for the kernel loader: we have an allocator to provide pages of memory, and page tables to set up virtual memory mappings that can be activated once we’re done with EFI. bmain thus proceeds with loading the kernel:

const kernel = match (efi::open("\\helios", efi::FILE_MODE::READ)) {
case let file: *efi::FILE_PROTOCOL =>
	yield file;
case let err: efi::error =>
	log::printfln("Error: no kernel found at /helios");
	return err: efi::STATUS;
};

log::printfln("Load kernel /helios");
const kentry = match (load(&mem, kernel)) {
case let err: efi::error =>
	return err: efi::STATUS;
case let entry: uintptr =>
	yield entry: *kentry;
};
efi::close(kernel)!;

The loader itself (the “load” function here) is a relatively straightforward ELF loader; if you’ve seen one you’ve seen them all. Nevertheless, you may browse it online if you so wish. The only item of note here is the function used for mapping kernel pages:

// Maps a physical page into the kernel's virtual address space.
fn kmmap(virt: uintptr, phys: uintptr, flags: uintptr) void = {
	assert(virt & ~0x1ff000 == 0xffff800000000000: uintptr);
	const offs = (virt >> 12) & 0x1ff;
	L3_kernel[offs] = PT_PAGE | PT_NORMAL | PT_AF | PT_ISM | phys | flags;
};

The assertion enforces a constraint which is implemented by our kernel linker script, namely that all loadable kernel program headers are located within the kernel’s reserved address space. With this constraint in place, the implementation is simpler than many mmap implementations; we can assume that L3_kernel is the correct page table and just load it up with the desired physical address and mapping flags.

Following the kernel loader, the bootloader addresses other items of interest, such as loading the device tree and boot modules — which includes, for instance, the init process image and an initramfs. It also allocates & populates data structures with information which will be of later use to the kernel, including the memory map. This code is relatively straightforward and not particularly interesting; most of these processes takes advantage of the same straightforward Hare function:

// Loads a file into continuous pages of memory and returns its physical
// address.
fn load_file(
	mem: *allocator,
	file: *efi::FILE_PROTOCOL,
) (uintptr | efi::error) = {
	const info = efi::file_info(file)?;
	const fsize = info.FileSize: size;
	let npage = fsize / PAGESIZE;
	if (fsize % PAGESIZE != 0) {
		npage += 1;
	};

	let base: uintptr = 0;
	for (let i = 0z; i < npage; i += 1) {
		const phys = pagealloc(mem);
		if (base == 0) {
			base = phys;
		};

		const nbyte = if ((i + 1) * PAGESIZE > fsize) {
			yield fsize % PAGESIZE;
		} else {
			yield PAGESIZE;
		};
		let dest = (phys: *[*]u8)[..nbyte];
		const n = efi::read(file, dest)?;
		assert(n == nbyte);
	};

	return base;
};

It is not necessary to map these into virtual memory anywhere, the kernel later uses the identity-mapped physical memory region in the higher half to read them. Tasks of interest resume at the end of bmain:

efi::exit_boot_services();
init_mmu();
enter_kernel(kentry, ctx);

Once we exit boot services, we are free to configure the MMU according to our desired specifications and make good use of all of the work done earlier to prepare a kernel memory map. Thus, init_mmu:

// Initializes the ARM MMU to our desired specifications. This should take place
// *after* EFI boot services have exited because we're going to mess up the MMU
// configuration that it depends on.
fn init_mmu() void = {
	// Disable MMU
	const sctlr_el1 = rdsctlr_el1();
	wrsctlr_el1(sctlr_el1 & ~SCTLR_EL1_M);

	// Configure MAIR
	const mair: u64 =
		(0xFF << 0) | // Attr0: Normal memory; IWBWA, OWBWA, NTR
		(0x00 << 8);  // Attr1: Device memory; nGnRnE, OSH
	wrmair_el1(mair);

	const tsz: u64 = 64 - 48;
	const ips = rdtcr_el1() & TCR_EL1_IPS_MASK;
	const tcr_el1: u64 =
		TCR_EL1_IPS_42B_4T |	// 4 TiB IPS
		TCR_EL1_TG1_4K |	// Higher half: 4K granule size
		TCR_EL1_SH1_IS |	// Higher half: inner shareable
		TCR_EL1_ORGN1_WB |	// Higher half: outer write-back
		TCR_EL1_IRGN1_WB |	// Higher half: inner write-back
		(tsz << TCR_EL1_T1SZ) |	// Higher half: 48 bits
		TCR_EL1_TG0_4K |	// Lower half: 4K granule size
		TCR_EL1_SH0_IS |	// Lower half: inner sharable
		TCR_EL1_ORGN0_WB |	// Lower half: outer write-back
		TCR_EL1_IRGN0_WB |	// Lower half: inner write-back
		(tsz << TCR_EL1_T0SZ);	// Lower half: 48 bits
	wrtcr_el1(tcr_el1);

	// Load page tables
	wrttbr0_el1(&L0[0]: uintptr);
	wrttbr1_el1(&L0[0]: uintptr);
	invlall();

	// Enable MMU
	const sctlr_el1: u64 =
		SCTLR_EL1_M |		// Enable MMU
		SCTLR_EL1_C |		// Enable cache
		SCTLR_EL1_I |		// Enable instruction cache
		SCTLR_EL1_SPAN |	// SPAN?
		SCTLR_EL1_NTLSMD |	// NTLSMD?
		SCTLR_EL1_LSMAOE |	// LSMAOE?
		SCTLR_EL1_TSCXT |	// TSCXT?
		SCTLR_EL1_ITD;		// ITD?
	wrsctlr_el1(sctlr_el1);
};

There are a lot of bits here! Figuring out which ones to enable or disable was a project in and of itself. One of the major challenges, funnily enough, was finding the correct ARM manual to reference to understand all of these registers. I’ll save you some time and link to it directly, should you ever find yourself writing similar code. Some question marks in comments towards the end point out some flags that I’m still not sure about. The ARM CPU is very configurable and identifying the configuration that produces the desired behavior for a general-purpose kernel requires some effort.

After this function completes, the MMU is initialized and we are up and running with the kernel memory map we prepared earlier; the kernel is loaded in the higher half and the MMU is prepared to service it. So, we can jump to the kernel via enter_kernel:

@noreturn fn enter_kernel(entry: *kentry, ctx: *bootctx) void = {
	const el = readel();
	switch (el) {
	case el::EL0 =>
		abort("Bootloader running in EL0, breaks EFI invariant");
	case el::EL1 =>
		// Can boot immediately
		entry(ctx);
	case el::EL2 =>
		// Boot from EL2 => EL1
		//
		// This is the bare minimum necessary to get to EL1. Future
		// improvements might be called for here if anyone wants to
		// implement hardware virtualization on aarch64. Good luck to
		// this future hacker.

		// Enable EL1 access to the physical counter register
		const cnt = rdcnthctl_el2();
		wrcnthctl_el2(cnt | 0b11);

		// Enable aarch64 in EL1 & SWIO, disable most other EL2 things
		// Note: I bet someday I'll return to this line because of
		// Problems
		const hcr: u64 = (1 << 1) | (1 << 31);
		wrhcr_el2(hcr);

		// Set up SPSR for EL1
		// XXX: Magic constant I have not bothered to understand
		wrspsr_el2(0x3c4);

		enter_el1(ctx, entry);
	case el::EL3 =>
		// Not supported, tested earlier on
		abort("Unsupported boot configuration");
	};
};

Here we see the detritus from one of many battles I fought to port this kernel: the EL2 => EL1 transition. aarch64 has several “exception levels”, which are semantically similar to the x86_64 concept of protection rings. EL0 is used for userspace code, which is not applicable under these circumstances; an assertion sanity-checks this invariant. EL1 is the simplest case, this is used for normal kernel code and in this situation we can jump directly to the kernel. The EL2 case is used for hypervisor code, and this presented me with a challenge. When I tested my bootloader in qemu-virt, it worked initially, but on real hardware it failed. After much wailing and gnashing of teeth, the cause was found to be that our bootloader was started in EL2 on real hardware, and EL1 on qemu-virt. qemu can be configured to boot in EL2, which was crucial in debugging this problem, via -M virt,virtualization=on. From this environment I was able to identify a few important steps to drop to EL1 and into the kernel, though from the comments you can probably ascertain that this process was not well-understood. I do have a better understanding of it now than I did when this code was written, but the code is still serviceable and I see no reason to change it at this stage.

At this point, 14 days into the port, I successfully reached kmain on qemu-virt. Some initial kernel porting work was done after this, but when I was prepared to test it on real hardware I ran into this EL2 problem — the first kmain on real hardware ran at T+18.

That sums it up for the aarch64 EFI bootloader work. 24 days later the kernel and userspace ports would be complete, and a couple of weeks after that it was running on stage at FOSDEM. The next post will cover the kernel port (maybe more than one post will be required, we’ll see), and the final post will address the userspace port and the inner workings of the slidedeck demo that was shown on stage. Look forward to it, and thanks for reading!


  1. A cursory review of this code while writing this blog post draws my attention to a few things that ought to be improved as well. ↩︎

  2. PIC stands for “position independent code”. EFI can load executables at any location in memory and the code needs to be prepared to deal with that; PIC is the tool we use for this purpose. ↩︎

Should private platforms engage in censorship?

30 January 2023 at 00:00

Private service providers are entitled to do business with whom they please, or not to. Occasionally, a platform will take advantage of this to deny service to a particular entity on any number of grounds, often igniting a flood of debate online regarding whether or not censorship in this form is just. Recently, CloudFlare pulled the plug on a certain forum devoted to the coordinated harassment of its victims. Earlier examples include the same service blocking a far-right imageboard, or Namecheap cancelling service for a neo-Nazi news site.

In each of these cases, a private company elected to terminate service for a customer voluntarily, without a court order. Absent from these events was any democratic or judicial oversight. A private company which provides some kind of infrastructure for the Internet simply elected to unilaterally terminate service for a customer or class of customers.

When private companies choose with whom they do or do not do business with, this is an exercise of an important freedom: freedom of association. Some companies have this right limited by regulation — for instance, utility companies are often required to provide power to everyone who wants it within their service area. Public entities are required to provide their services to everyone — for instance, the US postal service cannot unilaterally choose not to deliver your mail. However, by default, private companies are generally allowed to deny their services to whomever they please.1

Are they right to?

An argument is often made that, when a platform reaches a given size (e.g. Facebook), or takes on certain ambitions (e.g. CloudFlare), it may become large and entrenched enough in our society that it should self-impose a role more analogous to a public utility than a private company. Under such constraints, such a platform would choose to host any content which is not explicitly illegal, and defer questions over what content is appropriate to the democratic process. There are a number of angles from which we can examine this argument.

For a start, how might we implement the scenario called for by this argument? Consider one option: regulation. Power companies are subject to regulations regarding how and with whom they do business; they must provide service to everyone and they are not generally allowed to shut off your heat in the cold depths of winter. Similarly, we could regulate digital platforms to require them to provide a soapbox for all legally expressible viewpoints, then utilize the democratic process to narrow this soapbox per society’s mutually-agreed-upon views regarding matters such as neo-Nazi propaganda.2

It’s important when making this argument to note that regulation of this sort imposes obligations on private businesses which erode their own right to free association; radical free speech for individuals requires radical curtailing of free association for businesses. Private businesses are owned and staffed by individuals, and requiring them to allow all legal forms of content on their platform is itself a limitation on their freedom. The staff of a newspaper may not appreciate being required by law to provide space in the editorials for KKK members to espouse their racist philosophy, but would nevertheless be required to typeset such articles under such an arrangement.

Another approach to addressing this argument is not to question the rights of a private business, but instead to question whether or not they should be allowed to grow to a size such that their discretion in censorship constitutes a disruption to society due to their scale and entrenched market position. Under this lens, we can suggest another government intervention that does not take the form of regulation, but of an application of antitrust law. With more platforms to choose from, we can explore more approaches to moderation and censorship, and depend on the market’s invisible hand to lead us true.

The free speech absolutist who makes similar arguments may find themselves in a contradiction: expanding free speech for some people (platform users) requires, in this scenario, curtailing freedoms for others (platform owners and staff). Someone in this position may concede that, while they support the rights of individuals, they might not offer the same rights to businesses who resemble utilities. The tools for implementing this worldview, however, introduce further contradictions when combined with the broader political profile of a typical free speech absolutist: calling for regulation isn’t very consistent with any “small government” philosophy; and those who describe themselves as Libertarian and make either of these arguments provide me with no small amount of amusement.

There is another flaw in this line of thinking which I want to highlight: the presumption that the democratic process can address these problems in the first place. Much of the legitimacy of this argument rests on the assumption that the ability for maligned users to litigate their grievances is not only more just, but also equal to the threat posed by hate speech and other concerns which are often the target of censorship on private platforms. I don’t think that this is true.

The democratic and judicial processes are often corrupt and inefficient. It is still the case that the tone of your skin has an outsized effect on the outcome of your court case; why shouldn’t similar patterns emerge when de-platformed racists are given their day before a judge? Furthermore, the pace of government interventions are generally insufficient. Could Facebook appeal a court for the right to remove the Proud Boys from their platform faster than they could organize an attack on the US Capitol building? And can lawmakers keep up with innovation at a pace sufficient to address new forms and mediums for communicating harmful content before they’re a problem?

We should also question if the democratic process will lead to moral outcomes. Minorities are, by definition, in the minority, and a purely democratic process will only favor their needs subject to the will of the majority. Should the rights of trans people to live free of harassment be subject to the pleasure of the cisgendered majority?

These systems, when implemented, will perform as they always have: they will provide disproportionately unfavorable outcomes for disadvantaged members of society. I am a leftist: if asked to imagine a political system which addresses these problems, I will first imagine sweeping reforms to our existing system, point out that the free market isn’t, lean in favor of regulation and nationalization of important industries, and seek to empower the powerless against the powerful. It will require a lot of difficult, ongoing work to get there, and I imagine most of this work will be done in spite of the protests of the typical free speech absolutist.

I am in favor of these reforms, but they are decades away from completion, and many will disagree on the goals and their implementation. But I am also a pragmatic person, and when faced with the system in which we find ourselves today, I seek a pragmatic solution to this problem; ideally one which is not predicated on revolution. When faced with the question, “should private platforms engage in censorship?”, what is the pragmatic answer?

To provide such an answer, we must de-emphasize idealism in favor of an honest examination of the practical context within which our decision-making is done. Consider again the status quo: private companies are generally permitted to exercise their right to free association by kicking people off of their platforms. A pragmatic framework for making these decisions examines the context in which they are made. In the current political climate, this context should consider the threats faced by many different groups of marginalized people today: racism is still alive and strong, what few LGBT rights exist are being dismantled, and many other civil liberties are under attack.

When someone (or some entity such as business) enjoys a particular freedom, the way they exercise it is meaningful. Inaction is a form of complicity; allowing hate to remain on your platform is an acknowledgement of your favor towards the lofty principles outlined in the arguments above in spite of the problems enumerated here and the realities faced by marginalized people today. A purely moral consideration thus suggests that exercising your right to free association in your role as a decision-maker at a business is a just response to this status quo.

I expect the people around me (given a definition of “around me” that extends to the staff at businesses I patronize) to possess a moral compass which is compatible with my own, and to act in accordance with it; in the absence of this I will express my discontent by voting with my feet. However, businesses in the current liberal economic regime often disregard morals in favor of profit-oriented decision making. Therefore, in order for the typical business behave morally, their decision-making must exist within a context where the moral outcomes align with the profitable outcomes.

We are seeing increasing applications of private censorship because this alignment is present. Businesses depend on two economic factors which are related to this issue: access to a pool of profitable users, and access to a labor pool with which to develop and maintain their profits. Businesses which platform bigots are increasingly finding public opinion turning against them; marginalized people and moderates tend to flee to less toxic spaces and staff members are looking to greener pastures. The free market currently rewards private censorship, therefore in a system wherein the free market reigns supreme we observe private censorship.

I reject the idea that it is appropriate for businesses to sideline morality in favor of profit, and I don’t have much faith in the free market to produce moral outcomes. For example, the market is responding poorly to the threat of climate change. However, in the case of private censorship, the incentives are aligned such that the outcomes we’re observing match the outcomes I would expect.

This is a complex topic which we have examined from many angles. In my view, freedom of association is just as important as freedom of speech, and its application to private censorship is not clearly wrong. If you view private censorship as an infringement of the principle of free speech, but agree that freedom of association is nevertheless important, we must resolve this contradiction. The democratic or judicial processes are an enticing and idealistic answer, but these are flawed processes that may not produce just outcomes. If I were to consider these tools to address this question, I’m going to present solutions from a socialist perspective which may or may not jibe with your sensibilities.

Nevertheless, the system as it exists today produces outcomes which approximate both rationality and justice, and I do not stand in opposition to the increased application of private censorship under the current system, flawed though it may be.


  1. There are some nuances omitted here, such as the implications of the DMCA “safe harbor” provisions. ↩︎

  2. Arguments on other issues also call for regulating digital platforms, such as addressing the impact that being binned by Google without recourse can have on your quality-of-life for users who are dependent on Google’s email services. Some nuance is called for; I will elaborate on this in future posts. ↩︎

My plans at FOSDEM: SourceHut, Hare, and Helios

24 January 2023 at 00:00

FOSDEM is right around the corner, and finally in person after long years of dealing with COVID. I’ll be there again this year, and I’m looking forward to it! I have four slots on the schedule (wow! Thanks for arranging these, FOSDEM team) and I’ll be talking about several projects. There is a quick lightning talk on Saturday to introduce Helios and tease a full-length talk on Sunday, a meetup for the Hare community, and a meetup for the SourceHut community. I hope to see you there!

Lightning talk: Introducing Helios

Saturday 12:00 at H.2215 (Ferrer)

Helios is a simple microkernel written in part to demonstrate the applicability of the Hare programming language to kernels. This talk briefly explains why Helios is interesting and is a teaser for a more in-depth talk in the microkernel room tomorrow.

Hare is a systems programming language designed to be simple, stable, and robust. Hare uses a static type system, manual memory management, and a minimal runtime. It is well-suited to writing operating systems, system tools, compilers, networking software, and other low-level, high performance tasks. Helios uses Hare to implement a microkernel, largely inspired by seL4.

BoF: The Hare programming language

Saturday 15:00 at UB2.147

Hare is a systems programming language designed to be simple, stable, and robust. Hare uses a static type system, manual memory management, and a minimal runtime. It is well-suited to writing operating systems, system tools, compilers, networking software, and other low-level, high performance tasks.

At this meeting we’ll sum up the state of affairs with Hare, our plans for the future, and encourage discussions with the community. We’ll also demonstrate a few interesting Hare projects, including Helios, a micro-kernel written in Hare, and encourage each other to work on interesting projects in the Hare community.

BoF: SourceHut meetup

Saturday 16:00 at UB2.147

SourceHut is a free software forge for developing software projects, providing git and mercurial hosting, continuous integration, mailing lists, and more. We’ll be meeting here again in 2023 to discuss the platform and its community, the completion of the GraphQL rollout and the migration to the EU, and any other topics on the minds of the attendees.

Introducing Helios

Sunday 13:00 at H.1308 (Rolin)

Helios is a simple microkernel written in part to demonstrate the applicability of the Hare programming language to kernels. This talk will introduce the design and rationale for Helios, address some details of its implementation, compare it with seL4, and elaborate on the broader plans for the system.

Setting a new focus for my blog

22 January 2023 at 00:00

Just shy of two months ago, I published I shall toil at a reduced volume, which addressed the fact that I’m not getting what I want from my blog anymore, and I would be taking an indefinite break. Well, I am ready to resume my writing, albeit with a different tone and focus than before.

Well, that was fast.

– Everyone

Since writing this, I have been considering what exactly the essential subject of my dissatisfaction with my writing has been. I may have found the answer: I lost sight of my goals. I got so used to writing that I would often think to myself, “I want to write a blog post!”, then dig a topic out of my backlog (which is 264 items long) and write something about it. This is not the way; much of the effort expended on writing in this manner is not spent on the subjects I care about most, or those which most urgently demand an expenditure of words.

The consequences of this misalignment of perspective are that my writing has often felt dull and rote. It encourages shallower takes and lends itself to the rants or unthoughtful criticisms that my writings are, unfortunately, (in)famous for. When I take an idea off of the shelf, or am struck by an idea that, in the moment, seemingly demands to be spake of, I often end up with a disappointing result when the fruit of this inspiration is published a few hours later.

Over the long term, these issues manifest as demerits to my reputation, and deservedly so. What’s more, when a critical tone is well-justified, the posts which utilize it are often overlooked by readers due to the normalization of this tone throughout less important posts. Take for instance my recent post on Rust in Linux. Though this article could have been written with greater nuance, I still find its points about the value of conservatism in software decision-making accurate and salient. However, the message is weakened riding on the coat-tails of my long history of less poignant critiques of Rust. As I resume my writing, I will have to take a more critical examination of myself and the broader context of my writing before reaching for a negative tone as a writing tool.

With these lessons in mind, I am seeking out stronger goals to align my writing with, in the hope that the writing is both more fulfilling for me, and more compelling for the reader. Among these goals I have identified two particularly important ones, whose themes resonate through my strongest articles throughout the years:

  1. The applicability of software to the just advancement of society, its contextualization within the needs of the people who use it, a deep respect for these people and the software’s broader impact on the world, and the use of free software to acknowledge and fulfill these needs.
  2. The principles of good software engineering, such that software built to meet these goals is reliable, secure, and comprehensible. It is in the service of this goal that I beat the drum of simplicity with a regular rhythm.

Naturally many people have important beliefs on these subjects. I simply aim to share my own perspective, and I find it rewarding when I am able to write compelling arguments which underline these goals.

There is another kind of blog post that I enjoy writing and plan to resume: in-depth technical analysis of my free software projects. I’m working on lots of interesting and exciting projects, and I want to talk about them more, and I think people enjoy reading about them. I just spent six weeks porting Helios to aarch64, for instance, and have an essay on the subject half-written in the back of my head. I would love to type it in and publish it.

So, I will resume writing, and indeed at a “reduced volume”, with a renewed focus on the message and its context, and an emphasis on serving the goals I care about the most. Hopefully I find it more rewarding to write in this manner, and you find the results more compelling to read! Stay tuned.

$ rm ~/sources/drewdevault.com/todo.txt

I shall toil at a reduced volume

1 December 2022 at 00:00

Over the last nine years I have written 300,000 words for this blog on the topics which are important to me. I am not certain that I have much left to say.

I can keep revisiting these topics for years, each time adding a couple more years of wisdom and improvements to my writing skills to present my arguments more effectively. However, I am starting to feel diminishing returns from my writing. It does not seem like my words are connecting with readers anymore. And, though the returns on my work seem to be diminishing, the costs are not. Each new article spurs less discussion than the last, but provides an unwavering supply of spiteful responses.

Software is still the same mess it was when I started writing and working, or perhaps even worse. You can’t overcome perverse incentives. As Cantrill once famously noted, the lawnmower can’t have empathy. The truth he did not speak is that we all have some Oracle in our hearts, and the lawnmower is the size of the entire industry.

I have grown tired of it. I will continue my work quietly, building the things I believe in, and remaining true to my principles. I do not yet know if this is a cessation or a siesta, but I do know that I will not write again for some time. Thank you for reading, and good luck in your endeavours. I hope you found something of value in these pages.

Here are some of the blog posts I am most proud of, should you want to revisit them today or the next time you happen upon my website:

Codegen in Hare v2

26 November 2022 at 00:00

I spoke about code generation in Hare back in May when I wrote a tool for generating ioctl numbers. I wrote another code generator over the past few weeks, and it seems like a good time to revisit the topic on my blog to showcase another approach, and the improvements we’ve made for this use-case.

In this case, I wanted to generate code to implement IPC (inter-process communication) interfaces for my operating system. I have designed a DSL for describing these interfaces — you can read the grammar here. This calls for a parser, which is another interesting topic for Hare, but I’ll set that aside for now and focus on the code gen. Assume that, given a file like the following, we can parse it and produce an AST:

namespace hello;

interface hello {
	call say_hello() void;
	call add(a: uint, b: uint) uint;
};

The key that makes the code gen approach we’re looking at today is the introduction of strings::template to the Hare standard library. This module is inspired by a similar feature from Python, string.Template. An example of its usage is provided in Hare’s standard library documentation:

const src = "Hello, $user! Your balance is $$$balance.\n";
const template = template::compile(src)!;
defer template::finish(&template);
template::execute(&template, os::stdout,
	("user", "ddevault"),
	("balance", 1000),
)!; // "Hello, ddevault! Your balance is $1000.

Makes sense? Cool. Let’s see how this can be applied to code generation. The interface shown above compiles to the following generated code:

// This file was generated by ipcgen; do not modify by hand
use errors;
use helios;
use rt;

def HELLO_ID: u32 = 0xC01CAAC5;

export type fn_hello_say_hello = fn(object: *hello) void;
export type fn_hello_add = fn(object: *hello, a: uint, b: uint) uint;

export type hello_iface = struct {
	say_hello: *fn_hello_say_hello,
	add: *fn_hello_add,
};

export type hello_label = enum u64 {
	SAY_HELLO = HELLO_ID << 16u64 | 1,
	ADD = HELLO_ID << 16u64 | 2,
};

export type hello = struct {
	_iface: *hello_iface,
	_endpoint: helios::cap,
};

export fn hello_dispatch(
	object: *hello,
) void = {
	const (tag, a1) = helios::recvraw(object._endpoint);
	switch (rt::label(tag): hello_label) {
	case hello_label::SAY_HELLO =>
		object._iface.say_hello(
			object,
		);
		match (helios::reply(0)) {
		case void =>
			yield;
		case errors::invalid_cslot =>
			yield; // callee stored the reply
		case errors::error =>
			abort(); // TODO
		};
	case hello_label::ADD =>
		const rval = object._iface.add(
			object,
			a1: uint,
			rt::ipcbuf.params[1]: uint,
		);
		match (helios::reply(0, rval)) {
		case void =>
			yield;
		case errors::invalid_cslot =>
			yield; // callee stored the reply
		case errors::error =>
			abort(); // TODO
		};
	case =>
		abort(); // TODO
	};
};

Generating this code starts with the following entry-point:

// Generates code for a server to implement the given interface.
export fn server(out: io::handle, doc: *ast::document) (void | io::error) = {
	fmt::fprintln(out, "// This file was generated by ipcgen; do not modify by hand")!;
	fmt::fprintln(out, "use errors;")!;
	fmt::fprintln(out, "use helios;")!;
	fmt::fprintln(out, "use rt;")!;
	fmt::fprintln(out)!;

	for (let i = 0z; i < len(doc.interfaces); i += 1) {
		const iface = &doc.interfaces[i];
		s_iface(out, doc, iface)?;
	};
};

Here we start with some simple use of basic string formatting via fmt::fprintln. We see some of the same approach repeated in the meatier functions like s_iface:

fn s_iface(
	out: io::handle,
	doc: *ast::document,
	iface: *ast::interface,
) (void | io::error) = {
	const id: ast::ident = [iface.name];
	const name = gen_name_upper(&id);
	defer free(name);

	let id: ast::ident = alloc(doc.namespace...);
	append(id, iface.name);
	defer free(id);
	const hash = genhash(&id);

	fmt::fprintfln(out, "def {}_ID: u32 = 0x{:X};\n", name, hash)!;

Our first use of strings::template appears when we want to generate type aliases for interface functions, via s_method_fntype. This is where some of the trade-offs of this approach begin to present themselves.

const s_method_fntype_src: str =
	`export type fn_$iface_$method = fn(object: *$object$params) $result;`;
let st_method_fntype: tmpl::template = [];

@init fn s_method_fntype() void = {
	st_method_fntype= tmpl::compile(s_method_fntype_src)!;
};

fn s_method_fntype(
	out: io::handle,
	iface: *ast::interface,
	meth: *ast::method,
) (void | io::error) = {
	assert(len(meth.caps_in) == 0); // TODO
	assert(len(meth.caps_out) == 0); // TODO

	let params = strio::dynamic();
	defer io::close(&params)!;
	if (len(meth.params) != 0) {
		fmt::fprint(&params, ", ")?;
	};
	for (let i = 0z; i < len(meth.params); i += 1) {
		const param = &meth.params[i];
		fmt::fprintf(&params, "{}: ", param.name)!;
		ipc_type(&params, &param.param_type)!;

		if (i + 1 < len(meth.params)) {
			fmt::fprint(&params, ", ")!;
		};
	};

	let result = strio::dynamic();
	defer io::close(&result)!;
	ipc_type(&result, &meth.result)!;

	tmpl::execute(&st_method_fntype, out,
		("method", meth.name),
		("iface", iface.name),
		("object", iface.name),
		("params", strio::string(&params)),
		("result", strio::string(&result)),
	)?;
	fmt::fprintln(out)?;
};

The simple string substitution approach of strings::template prevents it from being as generally useful as a full-blown templating engine ala jinja2. To work around this, we have to write Hare code which does things like slurping up the method parameters into a strio::dynamic buffer where we might instead reach for something like {% for param in method.params %} in jinja2. Once we have prepared all of our data in a format suitable for a linear string substitution, we can pass it to tmpl::execute. The actual template is stored in a global which is compiled during @init, which runs at program startup. Anything which requires a loop to compile, such as the parameter list, is fetched out of the strio buffer and passed to the template.

We can explore a slightly different approach when we generate this part of the code, back up in the s_iface function:

export type hello_iface = struct {
	say_hello: *fn_hello_say_hello,
	add: *fn_hello_add,
};

To output this code, we render several templates one after another, rather than slurping up the generated code into heap-allocated string buffers to be passed into a single template.

const s_iface_header_src: str =
	`export type $iface_iface = struct {`;
let st_iface_header: tmpl::template = [];

const s_iface_method_src: str =
	`	$method: *fn_$iface_$method,`;
let st_iface_method: tmpl::template = [];

@init fn s_iface() void = {
	st_iface_header = tmpl::compile(s_iface_header_src)!;
	st_iface_method = tmpl::compile(s_iface_method_src)!;
};

// ...

tmpl::execute(&st_iface_header, out,
	("iface", iface.name),
)?;
fmt::fprintln(out)?;

for (let i = 0z; i < len(iface.methods); i += 1) {
	const meth = &iface.methods[i];
	tmpl::execute(&st_iface_method, out,
		("iface", iface.name),
		("method", meth.name),
	)?;
	fmt::fprintln(out)?;
};

fmt::fprintln(out, "};\n")?;

The remainder of the code is fairly similar.

strings::template is less powerful than a more sophisticated templating system might be, such as Golang’s text/template. A more sophisticated templating engine could be implemented for Hare, but it would be more challenging — no reflection or generics in Hare — and would not be a great candidate for the standard library. This approach hits the sweet spot of simplicity and utility that we’re aiming for in the Hare stdlib. strings::template is implemented in a single ~180 line file.

I plan to continue polishing this tool so I can use it to describe interfaces for communications between userspace drivers and other low-level userspace services in my operating system. If you have any questions, feel free to post them on my public inbox, or shoot them over to my new fediverse account. Until next time!

In praise of Plan 9

12 November 2022 at 00:00

Plan 9 is an operating system designed by Bell Labs. It’s the OS they wrote after Unix, with the benefit of hindsight. It is the most interesting operating system that you’ve never heard of, and, in my opinion, the best operating system design to date. Even if you haven’t heard of Plan 9, the designers of whatever OS you do use have heard of it, and have incorporated some of its ideas into your OS.

Plan 9 is a research operating system, and exists to answer questions about ideas in OS design. As such, the Plan 9 experience is in essence an exploration of the interesting ideas it puts forth. Most of the ideas are small. Many of them found a foothold in the broader ecosystem — UTF-8, goroutines, /proc, containers, union filesystems, these all have their roots in Plan 9 — but many of its ideas, even the good ones, remain unexplored outside of Plan 9. As a consequence, Plan 9 exists at the center of a fervor of research achievements which forms a unique and profoundly interesting operating system.

One example I often raise to illustrate the design ideals of Plan 9 is to compare its approach to network programming with that of the Unix standard, Berkeley sockets. BSD sockets fly in the face of Unix sensibilities and are quite alien on the system, though by now everyone has developed stockholm syndrome with respect to them so they don’t notice. When everything is supposed to be a file on Unix, why is it that the networking API is entirely implemented with special-purpose syscalls and ioctls? On Unix, creating a TCP connection involves calling the “socket” syscall to create a magic file descriptor, then the “connect” syscall to establish a connection. Plan 9 is much more Unix in its approach: you open /net/tcp/clone to reserve a connection, and read the connection ID from it. Then you open /net/tcp/n/ctl and write “connect 127.0.0.1!80” to it, where “n” is that connection ID. Now you can open /net/tcp/n/data and that file is a full-duplex stream. No magic syscalls, and you can trivially implement it in a shell script.

This composes elegantly with another idea from Plan 9: the 9P protocol. All file I/O on the entire system uses the 9P protocol, which defines operations like read and write. This protocol is network transparent, and you can mount remote servers into your filesystem namespace and access their files over 9P. You can do something similar on Unix, but on Plan 9 you get much more mileage from the idea because everything is actually a file, and there are no magic syscalls or ioctls. For instance, your Ethernet interface is at /net/ether0, and everything in there is just a file. Say you want to establish a VPN: you simply mount a remote server’s /net/ether0 at /net/ether1, and now you have a VPN. That’s it.

The mountpoints are interesting as well, because they exist within a per-process filesystem namespace. Mounting filesystems does not require special permissions like on Unix, because these mounts only exist within the process tree that creates them, rather than modifying global state. The filesystems can also be implemented in userspace rather trivially via the 9P protocol, similar to FUSE but much more straightforward. Many programs provide a programmable/scriptable interface via a special filesystem such as this.

Userspace programs can also provide filesystems compatible with those normally implemented by kernel drivers, like /net/ether0, and provide these to processes in their namespace. For example, /dev/draw is analogous to a framebuffer device: you open it to write pixels to the screen. The window manager, Rio, implements a /dev/draw-like interface in userspace, then mounts it in the filesystem namespace of its children. All GUI programs can thus be run both on a framebuffer or in a window, without any awareness of which it’s using. The same is also true over the network: to implement VNC-like functionality, just mount your local /dev/draw and /dev/kbd on a remote server. Add /dev/audio if you like.

These ideas can also be built upon to form something resembling a container runtime, pre-dating even early concepts like BSD jails by several years, and implementing them much more effectively. Recall that everything really is just a file on Plan 9, unlike Unix. Access to the hardware is provided through normal files, and per-process namespaces do not require special permissions to modify mountpoints. Making a container is thus trivial: just unmount all of the hardware you don’t want the sandboxed program to have access to. Done. You don’t even have to be root. Want to forward a TCP port? Write an implementation of /net/tcp which is limited to whatever ports you need — perhaps with just a hundred lines of shell scripting — and mount it into the namespace.

The shell, rc, is also wonderful. The debugger is terribly interesting, and its ideas didn’t seem to catch on with the likes of gdb. The editors, acme and sam, are also interesting and present a unique user interface that you can’t find anywhere else. The plumber is cool, it’s like “what if xdg-open was good actually”. The kernel is concise and a pleasure to read. The entire operating system, kernel and userspace, can be built from source code on my 12 year old laptop in about 5 minutes. The network database, ndb, is brilliant. The entire OS is stuffed to the brim with interesting ideas, all of them implemented with elegance, conciseness, and simplicity.

Plan 9 failed, in a sense, because Unix was simply too big and too entrenched by the time Plan 9 came around. It was doomed by its predecessor. Nevertheless, its design ideas and implementation resonate deeply with me, and have provided an endless supply of inspiration for my own work. I think that everyone owes it to themselves to spend a few weeks messing around with and learning about Plan 9. The dream is kept alive by 9front, which is the most actively maintained fork of Plan 9 available today. Install it on your ThinkPad and mess around.

I will offer a caveat, however: leave your expectations at the door. Plan 9 is not Unix, it is not Unix-compatible, and it is certainly not yet another Linux distribution. Everything you’re comfortable and familiar with in your normal Unix setup will not translate to Plan 9. Come to Plan 9 empty handed, and let it fill those hands with its ideas. You will come away from the experience as a better programmer.

Notes from kernel hacking in Hare, part 3: serial driver

27 October 2022 at 00:00

Today I would like to show you the implementation of the first userspace driver for Helios: a simple serial driver. All of the code we’re going to look at today runs in userspace, not in the kernel, so strictly speaking this should be “notes from OS hacking in Hare”, but I won’t snitch if you don’t.

Note: In the previous entry to this series, I promised to cover the userspace threading API in this post. I felt like covering this instead. Sorry!

A serial port provides a simple protocol for transferring data between two systems. It generalizes a bit, but for our purposes we can just think of this as a terminal which you can use over a simple cable and a simple protocol. It’s a standard x86_64 feature (though one which has been out of style for a couple of decades now), and its simple design (and high utility) makes it a good choice for the first driver to write for Helios. We’re going to look at the following details today:

  1. The system’s initramfs
  2. The driver loader
  3. The serial driver itself

The initramfs used by Helios, for the time being, is just a tarball. I imported format::tar from the standard library, a module which I designed for this express purpose, and made a few minor tweaks to make it suitable for Helios' needs. I also implemented seeking within a tar entry to make it easier to write an ELF loader from it. The bootloader loads this tarball into memory, the kernel provides page capabilities to init for it, and then we can map it into memory and study it, something like this:

let base = rt::map_range(rt::vspace, 0, 0, &desc.pages)!;
let slice = (base: *[*]u8)[..desc.length];

const buf = bufio::fixed(slice, io::mode::READ);
const rd = tar::read(&buf);

Pulling a specific driver out of it looks like this:

// Loads a driver from the bootstrap tarball.
fn earlyload(fs: *bootstrapfs, path: str) *process = {
	tar::reset(&fs.rd)!;
	path = strings::ltrim(path, '/');

	for (true) {
		const ent = match (tar::next(&fs.rd)) {
		case io::EOF =>
			break;
		case let ent: tar::entry =>
			yield ent;
		case let err: tar::error =>
			abort("Invalid bootstrap.tar file");
		};
		defer tar::skip(&ent)!;

		if (ent.name == path) {
			// TODO: Better error handling here
			const proc = match (load_driver(&ent)) {
			case let err: io::error =>
				abort("Failed to load driver from boostrap");
			case let err: errors::error =>
				abort("Failed to load driver from boostrap");
			case let proc: *process =>
				yield proc;
			};
			helios::task_resume(proc.task)!;
			return proc;
		};
	};

	abort("Missing bootstrap driver");
};

This code finds a file in the tarball with the given path (e.g. drivers/serial), creates a process with the driver loader, then resumes the thread and the driver is running. Let’s take a look at that driver loader next. The load_driver entry point takes an I/O handle to an ELF file and loads it:

fn load_driver(image: io::handle) (*process | io::error | errors::error) = {
	const loader = newloader(image);
	let earlyconf = driver_earlyconfig {
		cspace_radix = 12,
	};
	load_earlyconfig(&earlyconf, &loader)?;

	let proc = newprocess(earlyconf.cspace_radix)?;
	load(&loader, proc)?;
	load_config(proc, &loader)?;

	let regs = helios::context {
		rip = loader.header.e_entry,
		rsp = INIT_STACK_ADDR,
		...
	};
	helios::task_writeregisters(proc.task, &regs)?;
	return proc;
};

This is essentially a standard ELF loader, which it calls via the more general “newprocess” and “load” functions, but drivers have an extra concern: the driver manifest. The “load_earlyconfig” processes manifest keys which are necessary to configure prior to loading the ELF image, and the “load_config” function takes care of the rest of the driver configuration. The remainder of the code configures the initial thread.

The actual driver manifest is an INI file which is embedded in a special ELF section in driver binaries. The manifest for the serial driver looks like this:

[driver]
name=pcserial
desc=Serial driver for x86_64 PCs

[cspace]
radix=12

[capabilities]
0:serial =
1:note = 
2:cspace = self
3:ioport = min=3F8, max=400
4:ioport = min=2E8, max=2F0
5:irq = irq=3, note=1
6:irq = irq=4, note=1

Helios is a capability-oriented system, and in order to do anything useful, each process needs to have capabilities to work with. Each driver declares exactly what capabilities it needs and receives only these capabilities, and nothing else. This provides stronger isolation than Unix systems can offer (even with something like OpenBSD’s pledge(2)) — this driver cannot even allocate memory.

A standard x86_64 ISA serial port uses two I/O port ranges, 0x3F8-0x400 and 0x2E8-0x2F0, as well as two IRQs, IRQ 3 and 4, together providing support for up to four serial ports. The driver first requests a “serial” capability, which is a temporary design for an IPC endpoint that the driver will use to actually process read or write requests. This will be replaced with a more sophisticated device manager system in the future. It also creates a notification capability, which is later used to deliver the IRQs, and requests a capability for its own cspace so that it can manage capability slots. This will be necessary later on. Following this it requests capabilities for the system resources it needs, namely the necessary I/O ports and IRQs, the latter configured to be delivered to the notification in capability slot 1.

With the driver isolated in its own address space, running in user mode, and only able to invoke this set of capabilities, it’s very limited in what kind of exploits it’s vulnerable to. If there’s a vulnerability here, the worst that could happen is that a malicious actor on the other end of the serial port could crash the driver, which would then be rebooted by the service manager. On Linux, a bug in the serial driver can be used to compromise the entire system.

So, the driver loader parses this file and allocates the requested capabilities for the driver. I’ll skip most of the code, it’s just a boring INI file parser, but the important bit is the table for capability allocations:

type capconfigfn = fn(
	proc: *process,
	addr: uint,
	config: const str,
) (void | errors::error);

// Note: keep these tables alphabetized
const capconfigtab: [_](const str, *capconfigfn) = [
	("cspace", &cap_cspace),
	("endpoint", &cap_endpoint),
	("ioport", &cap_ioport),
	("irq", &cap_irq),
	("note", &cap_note),
	("serial", &cap_serial),
	// TODO: More
];

This table defines functions which, when a given INI key in the [capabilities] section is found, provisions the requested capabilities. This list is not complete; in the future all kernel objects will be added as well as userspace-defined interfaces (similar to serial) which implement various driver interfaces, such as ‘fs’ or ‘gpu’. Let’s start with the notification capability:

fn cap_note(
	proc: *process,
	addr: uint,
	config: const str,
) (void | errors::error) = {
	if (config != "") {
		return errors::invalid;
	};
	const note = helios::newnote()?;
	defer helios::destroy(note)!;
	helios::copyto(proc.cspace, addr, note)?;
};

This capability takes no configuration arguments, so we first simply check that the value is empty. Then we create a notification, copy it into the driver’s capability space at the requested capability address, then destroy our copy. Simple!

The I/O port capability is a bit more involved: it does accept configuration parameters, namely what I/O port range the driver needs.

fn cap_ioport(
	proc: *process,
	addr: uint,
	config: const str,
) (void | errors::error) = {
	let min = 0u16, max = 0u16;
	let have_min = false, have_max = false;

	const tok = strings::tokenize(config, ",");
	for (true) {
		let tok = match (strings::next_token(&tok)) {
		case void =>
			break;
		case let tok: str =>
			yield tok;
		};
		tok = strings::trim(tok);

		const (key, val) = strings::cut(tok, "=");
		let field = switch (key) {
		case "min" =>
			have_min = true;
			yield &min;
		case "max" =>
			have_max = true;
			yield &max;
		case =>
			return errors::invalid;
		};

		match (strconv::stou16b(val, base::HEX)) {
		case let u: u16 =>
			*field = u;
		case =>
			return errors::invalid;
		};
	};

	if (!have_min || !have_max) {
		return errors::invalid;
	};

	const ioport = helios::ioctl_issue(rt::INIT_CAP_IOCONTROL, min, max)?;
	defer helios::destroy(ioport)!;
	helios::copyto(proc.cspace, addr, ioport)?;
};

Here we split the configuration string on commas and parse each as a key/value pair delimited by an equal sign ("="), looking for a key called “min” and another called “max”. At the moment the config parsing is just implemented in this function directly, but in the future it might make sense to write a small abstraction for capability configurations like this. Once we know the I/O port range the user wants, then we issue an I/O port capability for that range and copy it into the driver’s cspace.

IRQs are a bit more involved still. An IRQ capability must be configured to deliver IRQs to a notification object.

fn cap_irq(
	proc: *process,
	addr: uint,
	config: const str,
) (void | errors::error) = {
	let irq = 0u8, note: helios::cap = 0;
	let have_irq = false, have_note = false;

	// ...config string parsing omitted...

	const _note = helios::copyfrom(proc.cspace, note, helios::CADDR_UNDEF)?;
	defer helios::destroy(_note)!;
	const (ct, _) = rt::identify(_note)!;
	if (ct != ctype::NOTIFICATION) {
		// TODO: More semantically meaningful errors would be nice
		return errors::invalid;
	};

	const irq = helios::irqctl_issue(rt::INIT_CAP_IRQCONTROL, _note, irq)?;
	defer helios::destroy(irq)!;
	helios::copyto(proc.cspace, addr, irq)?;
};

In order to do this, the driver loader copies the notification capability from the driver’s cspace and into the loader’s cspace, then creates an IRQ with that notification. It copies the new IRQ capability into the driver, then destroys its own copy of the IRQ and notification.

In this manner, the driver can declaratively state which capabilities it needs, and the loader can prepare an environment for it with these capabilities prepared. Once these capabilities are present in the driver’s cspace, the driver can invoke them by addressing the numbered capability slots in a send or receive syscall.

To summarize, the loader takes an I/O object (which we know is sourced from the bootstrap tarball) from which an ELF file can be read, finds a driver manifest, then creates a process and fills the cspace with the requested capabilities, loads the program into its address space, and starts the process.

Next, let’s look at the serial driver that we just finished loading.

Let me first note that this serial driver is a proof-of-concept at this time. A future serial driver will take a capability for a device manager object, then probe each serial port and provision serial devices for each working serial port. It will define an API which supports additional serial-specific features, such as configuring the baud rate. For now, it’s pretty basic.

This driver implements a simple event loop:

  1. Configure the serial port
  2. Wait for an interrupt or a read/write request from the user
  3. On interrupt, process the interrupt, writing buffered data or buffering readable data
  4. On a user request, buffer writes or unbuffer reads
  5. GOTO 2

The driver starts by defining some constants for the capability slots we set up in the manifest:

def EP: helios::cap = 0;
def IRQ: helios::cap = 1;
def CSPACE: helios::cap = 2;
def IRQ3: helios::cap = 5;
def IRQ4: helios::cap = 6;

It also defines some utility code for reading and writing to the COM registers, and constants for each of the registers defined by the interface.

// COM1 port
def COM1: u16 = 0x3F8;

// COM2 port
def COM2: u16 = 0x2E8;

// Receive buffer register
def RBR: u16 = 0;

// Transmit holding regiser
def THR: u16 = 0;

// ...other registers omitted...

const ioports: [_](u16, helios::cap) = [
	(COM1, 3), // 3 is the I/O port capability address
	(COM2, 4),
];

fn comin(port: u16, register: u16) u8 = {
	for (let i = 0z; i < len(ioports); i += 1) {
		const (base, cap) = ioports[i];
		if (base != port) {
			continue;
		};
		return helios::ioport_in8(cap, port + register)!;
	};
	abort("invalid port");
};

fn comout(port: u16, register: u16, val: u8) void = {
	for (let i = 0z; i < len(ioports); i += 1) {
		const (base, cap) = ioports[i];
		if (base != port) {
			continue;
		};
		helios::ioport_out8(cap, port + register, val)!;
		return;
	};
	abort("invalid port");
};

We also define some statically-allocated data structures to store state for each COM port, and a function to initialize the port:

type comport = struct {
	port: u16,
	rbuf: [4096]u8,
	wbuf: [4096]u8,
	rpending: []u8,
	wpending: []u8,
};

let ports: [_]comport = [
	comport { port = COM1, ... },
	comport { port = COM2, ... },
];

fn com_init(com: *comport) void = {
	com.rpending = com.rbuf[..0];
	com.wpending = com.wbuf[..0];
	comout(com.port, IER, 0x00);	// Disable interrupts
	comout(com.port, LCR, 0x80);	// Enable divisor mode
	comout(com.port, DL_LSB, 0x01);	// Div Low:  01: 115200 bps
	comout(com.port, DL_MSB, 0x00);	// Div High: 00
	comout(com.port, LCR, 0x03);	// Disable divisor mode, set parity
	comout(com.port, FCR, 0xC7);	// Enable FIFO and clear
	comout(com.port, IER, ERBFI);	// Enable read interrupt
};

The basics are in place. Let’s turn our attention to the event loop.

export fn main() void = {
	com_init(&ports[0]);
	com_init(&ports[1]);

	helios::irq_ack(IRQ3)!;
	helios::irq_ack(IRQ4)!;

	let poll: [_]pollcap = [
		pollcap { cap = IRQ, events = pollflags::RECV },
		pollcap { cap = EP, events = pollflags::RECV },
	];
	for (true) {
		helios::poll(poll)!;
		if (poll[0].events & pollflags::RECV != 0) {
			poll_irq();
		};
		if (poll[1].events & pollflags::RECV != 0) {
			poll_endpoint();
		};
	};
};

We initialize two COM ports first, using the function we were just reading. Then we ACK any IRQs that might have already been pending when the driver starts up, and we enter the event loop proper. Here we are polling on two capabilities, the notification to which IRQs are delivered, and the endpoint which provides the serial driver’s external API.

The state for each serial port includes a read buffer and a write buffer, defined in the comport struct shown earlier. We configure the COM port to interrupt when there’s data available to read, then pull it into the read buffer. If we have pending data to write, we configure it to interrupt when it’s ready to write more data, otherwise we leave this interrupt turned off. The “poll_irq” function handles these interrupts:

fn poll_irq() void = {
	helios::wait(IRQ)!;
	defer helios::irq_ack(IRQ3)!;
	defer helios::irq_ack(IRQ4)!;

	for (let i = 0z; i < len(ports); i += 1) {
		const iir = comin(ports[i].port, IIR);
		if (iir & 1 == 0) {
			port_irq(&ports[i], iir);
		};
	};
};

fn port_irq(com: *comport, iir: u8) void = {
	if (iir & (1 << 2) != 0) {
		com_read(com);
	};
	if (iir & (1 << 1) != 0) {
		com_write(com);
	};
};

The IIR register is the “interrupt identification register”, which tells us why the interrupt occurred. If it was because the port is readable, we call “com_read”. If the interrupt occurred because the port is writable, we call “com_write”. Let’s start with com_read. This interrupt is always enabled so that we can immediately start buffering data as the user types it into the serial port.

// Reads data from the serial port's RX FIFO.
fn com_read(com: *comport) size = {
	let n: size = 0;
	for (comin(com.port, LSR) & RBF == RBF; n += 1) {
		const ch = comin(com.port, RBR);
		if (len(com.rpending) < len(com.rbuf)) {
			// If the buffer is full we just drop chars
			static append(com.rpending, ch);
		};
	};

	// This part will be explained later:
	if (pending_read.reply != 0) {
		const n = rconsume(com, pending_read.buf);
		helios::send(pending_read.reply, 0, n)!;
		pending_read.reply = 0;
	};

	return n;
};

This code is pretty simple. For as long as the COM port is readable, read a character from it. If there’s room in the read buffer, append this character to it.

How about writing? Well, we need some way to fill the write buffer first. This part is pretty straightforward:

// Append data to a COM port read buffer, returning the number of bytes buffered
// successfully.
fn com_wbuffer(com: *comport, data: []u8) size = {
	let z = len(data);
	if (z + len(com.wpending) > len(com.wbuf)) {
		z = len(com.wbuf) - len(com.wpending);
	};
	static append(com.wpending, data[..z]...);
	com_write(com);
	return z;
};

This code just adds data to the write buffer, making sure not to exceed the buffer length (note that in Hare this would cause an assertion, not a buffer overflow). Then we call “com_write”, which does the actual writing to the COM port.

// Writes data to the serial port's TX FIFO.
fn com_write(com: *comport) size = {
	if (comin(com.port, LSR) & THRE != THRE) {
		const ier = comin(com.port, IER);
		comout(com.port, IER, ier | ETBEI);
		return 0;
	};

	let i = 0z;
	for (i < 16 && len(com.wpending) != 0; i += 1) {
		comout(com.port, THR, com.wpending[0]);
		static delete(com.wpending[0]);
	};

	const ier = comin(com.port, IER);
	if (len(com.wpending) == 0) {
		comout(com.port, IER, ier & ~ETBEI);
	} else {
		comout(com.port, IER, ier | ETBEI);
	};

	return i;
};

If the COM port is not ready to write data, we enable an interrupt which will tell us when it is and return. Otherwise, we write up to 16 bytes — the size of the COM port’s FIFO — and remove them from the write buffer. If there’s more data to write, we enable the write interrupt, or we disable it if there’s nothing left. When enabled, this will cause an interrupt to fire when (1) we have data to write and (2) the serial port is ready to write it, and our event loop will call this function again.

That covers all of the code for driving the actual serial port. What about the interface for someone to actually use this driver?

The “serial” capability defined in the manifest earlier is a temporary construct to provision some means of communicating with the driver. It provisions an endpoint capability (which is an IPC primitive on Helios) and stashes it away somewhere in the init process so that I can write some temporary test code to actually read or write to the serial port. Either request is done by “call"ing the endpoint with the desired parameters, which will cause the poll in the event loop to wake as the endpoint becomes receivable, calling “poll_endpoint”.

fn poll_endpoint() void = {
	let addr = 0u64, amt = 0u64;
	const tag = helios::recv(EP, &addr, &amt);
	const label = rt::label(tag);
	switch (label) {
	case 0 =>
		const addr = addr: uintptr: *[*]u8;
		const buf = addr[..amt];
		const z = com_wbuffer(&ports[0], buf);
		helios::reply(0, z)!;
	case 1 =>
		const addr = addr: uintptr: *[*]u8;
		const buf = addr[..amt];
		if (len(ports[0].rpending) == 0) {
			const reply = helios::store_reply(helios::CADDR_UNDEF)!;
			pending_read = read {
				reply = reply,
				buf = buf,
			};
		} else {
			const n = rconsume(&ports[0], buf);
			helios::reply(0, n)!;
		};
	case =>
		abort(); // TODO: error
	};
};

“Calls” in Helios work similarly to seL4. Essentially, when you “call” an endpoint, the calling thread blocks to receive the reply and places a reply capability in the receiver’s thread state. The receiver then processes their message and “replies” to the reply capability to wake up the calling thread and deliver the reply.

The message label is used to define the requested operation. For now, 0 is read and 1 is write. For writes, we append the provided data to the write buffer and reply with the number of bytes we buffered, easy breezy.

Reads are a bit more involved. If we don’t immediately have any data in the read buffer, we have to wait until we do to reply. We copy the reply from its special slot in our thread state into our capability space, so we can use it later. This operation is why our manifest requires cspace = self. Then we store the reply capability and buffer in a variable and move on, waiting for a read interrupt. On the other hand, if there is data buffered, we consume it and reply immediately.

fn rconsume(com: *comport, buf: []u8) size = {
	let amt = len(buf);
	if (amt > len(ports[0].rpending)) {
		amt = len(ports[0].rpending);
	};
	buf[..amt] = ports[0].rpending[..amt];
	static delete(ports[0].rpending[..amt]);
	return amt;
};

Makes sense?

That basically covers the entire serial driver. Let’s take a quick peek at the other side: the process which wants to read from or write to the serial port. For the time being this is all temporary code to test the driver with, and not the long-term solution for passing out devices to programs. The init process keeps a list of serial devices configured on the system:

type serial = struct {
	proc: *process,
	ep: helios::cap,
};

let serials: []serial = [];

fn register_serial(proc: *process, ep: helios::cap) void = {
	append(serials, serial {
		proc = proc,
		ep = ep,
	});
};

This function is called by the driver manifest parser like so:

fn cap_serial(
	proc: *process,
	addr: uint,
	config: const str,
) (void | errors::error) = {
	if (config != "") {
		return errors::invalid;
	};
	const ep = helios::newendpoint()?;
	helios::copyto(proc.cspace, addr, ep)?;
	register_serial(proc, ep);
};

We make use of the serial port in the init process’s main function with a little test loop to echo reads back to writes:

export fn main(bi: *rt::bootinfo) void = {
	log::println("[init] Hello from Mercury!");

	const bootstrap = bootstrapfs_init(&bi.modules[0]);
	defer bootstrapfs_finish(&bootstrap);
	earlyload(&bootstrap, "/drivers/serial");

	log::println("[init] begin echo serial port");
	for (true) {
		let buf: [1024]u8 = [0...];
		const n = serial_read(buf);
		serial_write(buf[..n]);
	};
};

The “serial_read” and “serial_write” functions are:

fn serial_write(data: []u8) size = {
	assert(len(data) <= rt::PAGESIZE);
	const page = helios::newpage()!;
	defer helios::destroy(page)!;
	let buf = helios::map(rt::vspace, 0, map_flags::W, page)!: *[*]u8;
	buf[..len(data)] = data[..];
	helios::page_unmap(page)!;

	// TODO: Multiple serial ports
	const port = &serials[0];
	const addr: uintptr = 0x7fff70000000; // XXX arbitrary address
	helios::map(port.proc.vspace, addr, 0, page)!;

	const reply = helios::call(port.ep, 0, addr, len(data));
	return rt::ipcbuf.params[0]: size;
};

fn serial_read(buf: []u8) size = {
	assert(len(buf) <= rt::PAGESIZE);
	const page = helios::newpage()!;
	defer helios::destroy(page)!;

	// TODO: Multiple serial ports
	const port = &serials[0];
	const addr: uintptr = 0x7fff70000000; // XXX arbitrary address
	helios::map(port.proc.vspace, addr, map_flags::W, page)!;

	const (label, n) = helios::call(port.ep, 1, addr, len(buf));

	helios::page_unmap(page)!;

	let out = helios::map(rt::vspace, 0, 0, page)!: *[*]u8;
	buf[..n] = out[..n];
	return n;
};

There is something interesting going on here. Part of this code is fairly obvious — we just invoke the IPC endpoint using helios::call, corresponding nicely to the other end’s use of helios::reply, with the buffer address and size. However, the buffer address presents a problem: this buffer is in the init process’s address space, so the serial port cannot read or write to it!

In the long term, a more sophisticated approach to shared memory management will be developed, but for testing purposes I came up with this solution. For writes, we allocate a new page, map it into our address space, and copy the data we want to write to it. Then we unmap it, map it into the serial driver’s address space instead, and perform the call. For reads, we allocate a page, map it into the serial driver, call the IPC endpoint, then unmap it from the serial driver, map it into our address space, and copy the data back out of it. In both cases, we destroy the page upon leaving this function, which frees the memory and automatically unmaps the page from any address space. Inefficient, but it works for demonstration purposes.

And that’s really all there is to it! Helios officially has its first driver. The next step is to develop a more robust solution for describing capability interfaces and device APIs, then build a PS/2 keyboard driver and a BIOS VGA mode 3 driver for driving the BIOS console, and combine these plus the serial driver into a tty on which we can run a simple shell.

TOTP for 2FA is incredibly easy to implement. So what's your excuse?

18 October 2022 at 00:00

Time-based one-time passwords are one of the more secure approaches to 2FA — certainly much better than SMS. And it’s much easier to implement than SMS as well. The algorithm is as follows:

  1. Divide the current Unix timestamp by 30
  2. Encode it as a 64-bit big endian integer
  3. Write the encoded bytes to a SHA-1 HMAC initialized with the TOTP shared key
  4. Let offs = hmac[-1] & 0xF
  5. Let hash = decode hmac[offs .. offs + 4] as a 32-bit big-endian integer
  6. Let code = (hash & 0x7FFFFFFF) % 1000000
  7. Compare this code with the user’s code

You’ll need a little dependency to generate QR codes with the otpauth:// URL scheme, a little UI to present the QR code and store the shared secret in your database, and a quick update to your login flow, and then you’re good to go.

Here’s the implementation SourceHut uses in Python. I hereby release this code into the public domain, or creative commons zero, at your choice:

import base64
import hashlib
import hmac
import struct
import time

def totp(secret, token):
    tm = int(time.time() / 30)
    key = base64.b32decode(secret)

    for ix in range(-2, 3):
        b = struct.pack(">q", tm + ix)
        hm = hmac.HMAC(key, b, hashlib.sha1).digest()
        offset = hm[-1] & 0x0F
        truncatedHash = hm[offset:offset + 4]
        code = struct.unpack(">L", truncatedHash)[0]
        code &= 0x7FFFFFFF
        code %= 1000000
        if token == code:
            return True

    return False

This implementation has a bit of a tolerance added to make clock skew less of an issue, but that also means that the codes are longer-lived. Feel free to edit these tolerances if you so desire.

Here’s another one written in Hare, also public domain/CC-0.

use crypto::hmac;
use crypto::mac;
use crypto::sha1;
use encoding::base32;
use endian;
use time;

// Computes a TOTP code for a given time and key.
export fn totp(when: time::instant, key: []u8) uint = {
	const now = time::unix(when) / 30;
	const hmac = hmac::sha1(key);
	defer mac::finish(&hmac);

	let buf: [8]u8 = [0...];
	endian::beputu64(buf, now: u64);
	mac::write(&hmac, buf);

	let mac: [sha1::SIZE]u8 = [0...];
	mac::sum(&hmac, mac);

	const offs = mac[len(mac) - 1] & 0xF;
	const hash = mac[offs..offs+4];
	return ((endian::begetu32(hash)& 0x7FFFFFFF) % 1000000): uint;
};

@test fn totp() void = {
	const secret = "3N2OTFHXKLR2E3WNZSYQ====";
	const key = base32::decodestr(&base32::std_encoding, secret)!;
	defer free(key);
	const now = time::from_unix(1650183739);
	assert(totp(now, key) == 29283);
};

In any language, TOTP is just a couple of dozen lines of code even if there isn’t already a library — and there is probably already a library. You don’t have to store temporary SMS codes in the database, you don’t have to worry about phishing, you don’t have to worry about SIM swapping, and you don’t have to sign up for some paid SMS API like Twilio. It’s more secure and it’s trivial to implement — so implement it already! Please!


Update 2022-10-19 @ 07:45 UTC: A reader pointed out that it’s important to have rate limiting on your TOTP attempts, or else a brute force attack can be effective. Fair point!

Status update, October 2022

15 October 2022 at 00:00

After a few busy and stressful months, I decided to set aside October to rest. Of course, for me, rest does not mean a cessation of programming, but rather a shift in priorities towards more fun and experimental projects. Consequently, it has been a great month for Helios!

Hare upstream has enjoyed some minor improvements, such as from Pierre Curto’s patch to support parsing IPv6 addresses with a port (e.g. “[::1]:80”) and Kirill Primak’s improvements to the UTF-8 decoder. On the whole, improvements have been conservative. However, queued up for integration once qbe upstream support is merged is support for @threadlocal variables, which are useful for Helios and for ABI compatibility with C. I also drafted up a proof-of-concept for @inline functions, but it still needs work.

Now for the main event: Helios. The large-scale redesign and refactoring I mentioned in the previous status update is essentially complete, and the kernel reached (and exceeded) feature parity with the previous status quo. Since Helios has been my primary focus for the past couple of weeks, I have a lot of news to share about it.

First, I got back into userspace a few days after the last status update, and shortly thereafter implemented a new scheduler. I then began to rework the userspace API (uapi) in the kernel, which differs substantially from its prior incarnation. The kernel object implementations present themselves as a library for kernel use, and the new uapi module handles all interactions with this module from userspace, providing a nice separation of concerns. The uapi module handles more than syscalls now — it also implements send/recv for kernel objects, for instance. As of a few days ago, uapi also supports delivering faults to userspace supervisor processes:

A screenshot of a thread on Helios causing a page fault, then its parent
thread receives details of the fault and maps a page onto the address of the
attempted write. The child thread is resumed and is surprised to find that the
write succeeded (because a page was mapped underneath the write).

@test fn task::pagefault() void = {
	const fault = helios::newendpoint()!;
	defer helios::destroy(fault)!;

	const thread = threads::new(&_task_pagefault)!;
	threads::set_fault(thread, fault)!;
	threads::start(thread)!;

	const fault = helios::recv_fault(fault);
	assert(fault.addr == 0x100);

	const page = helios::newpage()!;
	defer helios::destroy(page)!;
	helios::map(rt::vspace, 0, map_flags::W | map_flags::FIXED, page)!;

	threads::resume(thread)!;
	threads::join(thread)!;
};

fn _task_pagefault() void = {
	let ptr: *int = 0x100: uintptr: *int;
	*ptr = 1337;
	assert(*ptr == 1337);
};

The new userspace threading API is much improved over the hack job in the earlier design. It supports TLS and many typical threading operations, such as join and detach. This API exists mainly for testing the kernel via Vulcan, and is not anticipated to see much use beyond this (though I will implement pthreads for the POSIX C environment at some point). For more details, see this blog post. Alongside this and other userspace libraries, Vulcan has been fleshed out into a kernel test suite once again, which I have been frequently testing on real hardware:

A picture of a laptop showing 15 passing kernel tests

Here’s an ISO you can boot on your own x86_64 hardware to see if it works for you, too. If you have problems, take a picture of the issue, boot Linux and email me said picture, the output of lscpu, and any other details you deem relevant.

The kernel now supports automatic capability address allocation, which is a marked improvement over seL4. The new physical page allocator is also much improved, as it supports allocation and freeing and can either allocate pages sparsely or continuously depending on the need. Mapping these pages in userspace was also much improved, with a better design of the userspace virtual memory map and a better heap, complete with a (partial) implementation of mmap.

I have also broken ground on the next component of the OS, Mercury, which provides a more complete userspace environment for writing drivers. It has a simple tar-based initramfs based on Hare’s format::tar implementation, which I wrote in June for this purpose. It can load ELF files from this tarball into new processes, and implements some extensions that are useful for driver loading. Consequently, the first Mercury driver is up and running:

Demo of a working serial driver

This driver includes a simple driver manifest, which is embedded into its ELF file and processed by the driver loader to declaratively specify the capabilities it needs:

[driver]
name=pcserial
desc=Serial driver for x86_64 PCs

[cspace]
radix=12

[capabilities]
0:endpoint =
1:ioport = min=3F8, max=400
2:ioport = min=2E8, max=2F0
3:note = 
4:irq = irq=3, note=3
5:irq = irq=4, note=3

The driver loader prepares capabilities for the COM1 and COM2 I/O ports, as well as IRQ handlers for IRQ 3 and 4, based on this manifest, then loads them into the capability table for the driver process. The driver is sandboxed very effectively by this: it can only use these capabilities. It cannot allocate memory, modify its address space, or even destroy any of these capabilities. If a bad actor was on the other end of the serial port and exploited a bug, the worst thing it could do is crash the serial driver, which would then be rebooted by the supervisor. On Linux and other monolithic kernels like it, exploiting the serial driver compromises the entire operating system.

The resulting serial driver implementation is pretty small and straightforward, if you’d like to have a look.

This manifest format will be expanded in the future for additional kinds of drivers, such as with details specific to each bus (i.e. PCI vendor information or USB details), and will also have details for device trees when RISC-V and ARM support (the former is already underway) are brought upstream.

Next steps are to implement an I/O abstraction on top of IPC endpoints, which first requires call & reply support — the latter was implemented last night and requires additional testing. Following this, I plan on writing a getty-equivalent which utilizes this serial driver, and a future VGA terminal driver, to provide an environment in which a shell can be run. Then I’ll implement a ramfs to host commands for the shell to run, and we’ll really be cookin’ at that point. Disk drivers and filesystem drivers will be next.

That’s all for now. Quite a lot of progress! I’ll see you next time.

In praise of ffmpeg

12 October 2022 at 00:00

My last “In praise of” article covered qemu, a project founded by Fabrice Bellard, and today I want to take a look at another work by Bellard: ffmpeg. Bellard has a knack for building high-quality software which solves a problem so well that every other solution becomes obsolete shortly thereafter, and ffmpeg is no exception.

ffmpeg has been described as the Swiss army knife of multimedia. It incorporates hundreds of video, audio, and image decoders and encoders, muxers and demuxers, filters and devices. It provides a CLI and a set of libraries for working with its tools, and is the core component of many video and audio players as a result (including my preferred multimedia player, mpv). If you want to do almost anything with multimedia files — re-encode them, re-mux them, live stream it, whatever — ffmpeg can handle it with ease.

Let me share an example.

I was recently hanging out at my local hackerspace and wanted to play some PS2 games on my laptop. My laptop is not powerful enough to drive PCSX2, but my workstation on the other side of town certainly was. So I forwarded my game controller to my workstation via USB/IP and pulled up the ffmpeg manual to figure out how to live-stream the game to my laptop. ffmpeg can capture video from KMS buffers directly, use the GPU to efficiently downscale them, grab audio from pulse, encode them with settings tuned for low-latency, and mux it into a UDP socket. On the other end I set up mpv to receive the stream and play it back.

ffmpeg \
  -f pulse \
  -i alsa_output.platform-snd_aloop.0.analog-surround-51.monitor \
  -f kmsgrab \
  -thread_queue_size 64 \   # reduce input latency
  -i - \
  # Capture and downscale frames on the GPU:
  -vf 'hwmap=derive_device=vaapi,scale_vaapi=1280:720,hwdownload,format=bgr0' \
  -c:v libx264 \
  -preset:v superfast \     # encode video as fast as possible
  -tune zerolatency \       # tune encoder for low latency
  -intra-refresh 1 \        # reduces latency and mitigates dropped packets
  -f mpegts \               # mux into mpegts stream, well-suited to this use-case
  -b:v 3M \                 # configure target video bandwidth
  udp://$hackerspace:41841

With an hour of tinkering and reading man pages, I was able to come up with a single command which produced a working remote video game streaming setup from scratch thanks to ffmpeg. ffmpeg is amazing.

I have relied on ffmpeg for many tasks and for many years. It has always been there to handle any little multimedia-related task I might put it to for personal use — re-encoding audio files so they fit on my phone, taking clips from videos to share, muxing fonts into mkv files, capturing video from my webcam, live streaming hacking sessions on my own platform, or anything else I can imagine. It formed the foundation of MediaCrush back in the day, where we used it to optimize multimedia files for efficient viewing on the web, back when that was more difficult than “just transcode it to a webm”.

ffmpeg is notable for being one of the first large-scale FOSS projects to completely eradicate proprietary software in its niche. Virtually all multimedia-related companies rely on ffmpeg to do their heavy lifting. It took a complex problem and solved it, with free software. The book is now closed on multimedia: ffmpeg is the solution to almost all of your problems. And if it’s not, you’re more likely to patch ffmpeg than to develop something new. The code is accessible and the community are experts in your problem domain.

ffmpeg is one of the foremost pillars of achievement in free software. It has touched the lives of every reader, whether they know it or not. If you’ve ever watched TV, or gone to a movie, or watched videos online, or listened to a podcast, odds are that ffmpeg was involved in making it possible. It is one of the most well-executed and important software projects of all time.

Does Rust belong in the Linux kernel?

3 October 2022 at 00:00

I am known to be a bit of a polemic when it comes to Rust. I will be forthright with the fact that I don’t particularly care for Rust, and that my public criticisms of it might set up many readers with a reluctance to endure yet another Rust Hot Take from my blog. My answer to the question posed in the title is, of course, “no”. However, let me assuage some of your fears by answering a different question first: does Hare belong in the Linux kernel?

If I should owe my allegiance to any programming language, it would be Hare. Not only is it a systems programming language that I designed myself, but I am using it to write a kernel. Like Rust, Hare is demonstrably useful for writing kernels with. One might even go so far as to suggest that I consider it superior to C for this purpose, given that I chose to to write Helios in Hare rather than C, despite my extensive background in C. But the question remains: does Hare belong in the Linux kernel?

In my opinion, Hare does not belong in the Linux kernel, and neither does Rust. Some of the reasoning behind this answer is common to both, and some is unique to each, but I will be focusing on Rust today because Rust is the language which is actually making its way towards mainline Linux. I have no illusions about this blog post changing that, either: I simply find it an interesting case-study in software engineering decision-making in a major project, and that’s worth talking about.

Each change in software requires sufficient supporting rationale. What are the reasons to bring Rust into Linux? A kernel hacker thinks about these questions differently than a typical developer in userspace. One could espouse the advantages of Cargo, generics, whatever, but these concerns matter relatively little to kernel hackers. Kernels operate in a heavily constrained design space and a language has to fit into that design space. This is the first and foremost concern, and if it’s awkward to mold a language to fit into these constraints then it will be a poor fit.

Some common problems that a programming language designed for userspace will run into when being considered for kernelspace are:

  • Strict constraints on memory allocation
  • Strict constraints on stack usage
  • Strict constraints on recursion
  • No use of floating point arithmetic
  • Necessary evils, such as unsafe memory use patterns or integer overflow
  • The absence of a standard library, runtime, third-party libraries, or other conveniences typically afforded to userspace

Most languages can overcome these constraints with some work, but their suitability for kernel use is mainly defined by how well they adapt to them — there’s a reason that kernels written in Go, C#, Java, Python, etc, are limited to being research curiosities and are left out of production systems.

As Linus recently put it, “kernel needs trump any Rust needs”. The kernel is simply not an environment which will bend to accommodate a language; it must go the other way around. These constraints have posed, and will continue to pose, a major challenge for Rust in Linux, but on the whole, I think that it will be able to rise to meet them, though perhaps not with as much grace as I would like.

If Rust is able to work within these constraints, then it satisfies the ground rules for playing in ring 0. The question then becomes: what advantages can Rust bring to the kernel? Based on what I’ve seen, these essentially break down to two points:1

  1. Memory safety
  2. Trendiness

I would prefer not to re-open the memory safety flamewar, so we will simply move forward with the (dubious) assumptions that memory safety is (1) unconditionally desirable, (2) compatible with the kernel’s requirements, and (3) sufficiently provided for by Rust. I will offer this quote from an unnamed kernel hacker, though:

There are possibly some well-designed and written parts which have not suffered a memory safety issue in many years. It’s insulting to present this as an improvement over what was achieved by those doing all this hard work.

Regarding “trendiness”, I admit that this is a somewhat unforgiving turn of phrase. In this respect I refer to the goal of expanding the kernel’s developer base from a bunch of aging curmudgeons writing C2 towards a more inclusive developer pool from a younger up-and-coming language community like Rust. C is boring3 — it hasn’t really excited anyone in decades. Rust is exciting, and its community enjoys a huge pool of developers building their brave new world with it. Introducing Rust to the kernel will certainly appeal to a broader audience of potential contributors.

But there is an underlying assumption to this argument which is worth questioning: is the supply of Linux developers dwindling, and, if so, is it to such and extent that it demands radical change?

Well, no. Linux has consistently enjoyed a tremendous amount of attention from the software development community. This week’s release of Linux 6.0, one of the largest Linux releases ever, boasted more than 78,000 commits by almost 5,000 different authors since 5.15. Linux has a broad developer base reaching from many different industry stakeholders and independent contributors working on the careful development and maintenance of its hundreds of subsystems. The scale of Linux development is on a level unmatched by any other software project — free software or otherwise.

Getting Rust working in Linux is certainly an exciting project, and I’m all for developers having fun. However, it’s not likely to infuse Linux with a much-needed boost in its contributor base, because Linux has no such need. What’s more, Linux’s portability requirements prevent Rust from being used in most of the kernel in the first place. Most work on Rust in Linux is simply working on getting the systems to cooperate with each other or writing drivers which are redundant with existing C drivers, but cannot replace them due to Rust’s limited selection of targets.4 Few to none of the efforts from the Rust-in-Linux team are likely to support the kernel’s broader goals for some time.

We are thus left with memory safety as the main benefit offered by Rust to Linux, and for the purpose of this article we’re going to take it at face value. So, with the ground rules set and the advantages enumerated, what are some of the problems that Rust might face in Linux?

There are a few problems which could be argued over, such as substantial complexity of Rust compared to C, the inevitable doubling of Linux’s build time, the significant shift in design sensibilities required to support an idiomatic Rust design, the fragile interface which will develop on the boundaries between Rust and C code, or the challenges the kernel’s established base of C developers will endure when learning and adapting to a new language. To avoid letting this post become too subjective or lengthy, I’ll refrain from expanding on these. Instead, allow me to simply illuminate these issues as risk factors.

Linux is, on the whole, a conservative project. It is deployed worldwide in billions of devices and its reliability is depended on by a majority of Earth’s population. Risks are carefully evaluated in Linux as such. Every change presents risks and offers advantages, which must be weighed against each other to justify the change. Rust is one of the riskiest bets Linux has ever considered, and, in my opinion, the advantages may not weigh up. I think that the main reason we’re going to see Rust in the kernel is not due to a careful balancing of risk and reward, but because the Rust community wants Rust in Linux, and they’re large and loud enough to not be worth the cost of arguing with.

I don’t think that changes on this scale are appropriate for most projects. I prefer to encourage people to write new software to replace established software, rather than rewriting the established software. Some projects, such as Redox, are doing just that with Rust. However, operating systems are in a difficult spot in this respect. Writing an operating system is difficult work with a huge scope — few projects can hope to challenge Linux on driver support, for example. The major players have been entrenched for decades, and any project seeking to displace them will have decades of hard work ahead of them and will require a considerable amount of luck to succeed. Though I think that new innovations in kernels are badly overdue, I must acknowledge that there is some truth to the argument that we’re stuck with Linux. In this framing, if you want Rust to succeed in a kernel, getting it into Linux is the best strategy.

But, on the whole, my opinion is that the benefits of Rust in Linux are negligible and the costs are not. That said, it’s going to happen, and the impact to me is likely to be, at worst, a nuisance. Though I would have chosen differently, I wish them the best of luck and hope to see them succeed.


  1. There are some other arguable benefits which mainly boil down to finding Rust to have a superior language design to C or to be more enjoyable to use. These are subjective and generally are not the most important traits a kernel hacker has to consider when choosing a language, so I’m leaving them aside for now. ↩︎

  2. A portrayal which, though it may have a grain of truth, is largely false and offensive to my sensibilities as a 29-year-old kernel hacker. For the record. ↩︎

  3. A trait which, I will briefly note, is actually desirable for a production kernel implementation. ↩︎

  4. Rust in GCC will help with this problem, but it will likely take several years to materialize and several more years to become stable. Even when this is addressed, rewriting drivers wholesale will be labor intensive and is likely to introduce more problems than solutions — rewrites always introduce bugs. ↩︎

Notes from kernel hacking in Hare, part 2: multi-threading

2 October 2022 at 00:00

I have long promised that Hare would not have multi-threading, and it seems that I have broken that promise. However, I have remained true to the not-invented-here approach which is typical of my style by introducing it only after designing an entire kernel to implement it on top of.1

For some background, Helios is a micro-kernel written in Hare. In addition to the project, the Vulcan system is a small userspace designed to test the kernel.

A picture of a laptop running Helios and showing the results of the Vulcan test suite

While I don’t anticipate multi-threaded processes playing a huge role in the complete Ares system in the future, they do have a place. In the long term, I would like to be able to provide an implementation of pthreads for porting existing software to the system. A more immediate concern is how to test the various kernel primitives provided by Helios, such as those which facilitate inter-process communication (IPC). It’s much easier to test these with threads than with processes, since spawning threads does not require spinning up a new address space.

@test fn notification::wait() void = {
	const note = helios::newnote()!;
	defer helios::destroy(note)!;

	const thread = threads::new(&notification_wait, note)!;
	threads::start(thread)!;
	defer threads::join(thread)!;

	helios::signal(note)!;
};

fn notification_wait(note: u64) void = {
	const note = note: helios::cap;
	helios::wait(note)!;
};

So how does it work? Let’s split this up into two domains: kernelspace and userspace.

Threads in the kernel

The basic primitive for threads and processes in Helios is a “task”, which is simply an object which receives some CPU time. A task has a capability space (so it can invoke operations against kernel objects), an virtual address space (so it has somewhere to map the process image and memory), and some state, such as the values of its CPU registers. The task-related structures are:

// A task capability.
export type task = struct {
	caps::capability,
	state: uintptr,
	@offset(caps::LINK_OFFS) link: caps::link,
};

// Scheduling status of a task.
export type task_status = enum {
	ACTIVE,
	BLOCKED, // XXX: Can a task be both blocked and suspended?
	SUSPENDED,
};

// State for a task.
export type taskstate = struct {
	regs: arch::state,
	cspace: caps::cslot,
	vspace: caps::cslot,
	ipc_buffer: uintptr,
	status: task_status,
	// XXX: This is a virtual address, should be physical
	next: nullable *taskstate,
	prev: nullable *taskstate,
};

Here’s a footnote to explain some off-topic curiosities about this code: 2

The most interesting part of this structure is arch::state, which stores the task’s CPU registers. On x86_64,3 this structure is defined as follows:

export type state = struct {
	fs: u64,
	fsbase: u64,

	r15: u64,
	r14: u64,
	r13: u64,
	r12: u64,
	r11: u64,
	r10: u64,
	r9: u64,
	r8: u64,
	rbp: u64,
	rdi: u64,
	rsi: u64,
	rdx: u64,
	rcx: u64,
	rbx: u64,
	rax: u64,

	intno: u64,
	errcode: u64,

	rip: u64,
	cs: u64,
	rflags: u64,
	rsp: u64,
	ss: u64,
};

This structure is organized in part according to hardware constraints and in part at the discretion of the kernel implementer. The last five fields, from %rip to %ss, are constrained by the hardware. When an interrupt occurs, the CPU pushes each of these registers to the stack, in this order, then transfers control to the system interrupt handler. The next two registers serve a special purpose within our interrupt implementation, and the remainder are ordered arbitrarily.

In order to switch between two tasks, we need to save all of this state somewhere, then load the same state for another task when returning from the kernel to userspace. The save/restore process is handled in the interrupt handler, in assembly:

.global isr_common
isr_common:
	_swapgs
	push %rax
	push %rbx
	push %rcx
	push %rdx
	push %rsi
	push %rdi
	push %rbp
	push %r8
	push %r9
	push %r10
	push %r11
	push %r12
	push %r13
	push %r14
	push %r15

	// Note: fsbase is handled elsewhere
	push $0
	push %fs

	cld

	mov %rsp, %rdi
	mov $_kernel_stack_top, %rsp
	call arch.isr_handler
_isr_exit:
	mov %rax, %rsp

	// Note: fsbase is handled elsewhere
	pop %fs
	pop %r15

	pop %r15
	pop %r14
	pop %r13
	pop %r12
	pop %r11
	pop %r10
	pop %r9
	pop %r8
	pop %rbp
	pop %rdi
	pop %rsi
	pop %rdx
	pop %rcx
	pop %rbx
	pop %rax

	_swapgs

	// Clean up error code and interrupt #
	add $16, %rsp

	iretq

I’m not going to go into too much detail on interrupts for this post (maybe in a later post), but what’s important here is the chain of push/pop instructions. This automatically saves the CPU state for each task when entering the kernel. The syscall handler has something similar.

This suggests a question: where’s the stack?

Helios has a single kernel stack,4 which is moved to %rsp from $_kernel_stack_top in this code. This is different from systems like Linux, which have one kernel stack per thread; the rationale behind this design choice is out of scope for this post.5 However, the “stack” being pushed to here is not, in fact, a traditional stack.

x86_64 has an interesting feature wherein an interrupt can be configured to use a special “interrupt stack”. The task state segment is a bit of a historical artifact which is of little interest to Helios, but in long mode (64-bit mode) it serves a new purpose: to provide a list of addresses where up to seven interrupt stacks are stored. The interrupt descriptor table includes a 3-bit “IST” field which, when nonzero, instructs the CPU to set the stack pointer to the corresponding address in the TSS when that interrupt fires. Helios sets all of these to one, then does something interesting:

// Stores a pointer to the current state context.
export let context: **state = null: **state;

fn init_tss(i: size) void = {
	cpus[i].tstate = taskstate { ... };
	context = &cpus[i].tstate.ist[0]: **state;
};

// ...

export fn save() void = {
	// On x86_64, most registers are saved and restored by the ISR or
	// syscall service routines.
	let active = *context: *[*]state;
	let regs = &active[-1];

	regs.fsbase = rdmsr(0xC0000100);
};

export fn restore(regs: *state) void = {
	wrmsr(0xC0000100, regs.fsbase);

	const regs = regs: *[*]state;
	*context = &regs[1];
};

We store a pointer to the active task’s state struct in the TSS when we enter userspace, and when an interrupt occurs, the CPU automatically places that state into %rsp so we can trivially push all of the task’s registers into it.

There is some weirdness to note here: the stack grows downwards. Each time you push, the stack pointer is decremented, then the pushed value is written there. So, we have to fill in this structure from the bottom up. Accordingly, we have to do something a bit unusual here: we don’t store a pointer to the context object, but a pointer to the end of the context object. This is what &active[-1] does here.

Hare has some memory safety features by default, such as bounds testing array accesses. Here we have to take advantage of some of Hare’s escape hatches to accomplish the goal. First, we cast the pointer to an unbounded array of states — that’s what the *[*] is for. Then we can take the address of element -1 without the compiler snitching on us.

There is also a separate step here to save the fsbase register. This will be important later.

This provides us with enough pieces to enter userspace:

// Immediately enters this task in userspace. Only used during system
// initialization.
export @noreturn fn enteruser(task: *caps::capability) void = {
	const state = objects::task_getstate(task);
	assert(objects::task_schedulable(state));
	active = state;
	objects::vspace_activate(&state.vspace)!;
	arch::restore(&state.regs);
	arch::enteruser();
};

What we need next is a scheduler, and a periodic interrupt to invoke it, so that we can switch tasks every so often.

Scheduler design is a complex subject which can have design, performance, and complexity implications ranging from subtle to substantial. For Helios’s present needs we use a simple round-robin scheduler: each task gets the same time slice and we just switch to them one after another.

The easy part is simply getting periodic interrupts. Again, this blog post isn’t about interrupts, so I’ll just give you the reader’s digest:

arch::install_irq(arch::PIT_IRQ, &pit_irq);
arch::pit_setphase(100);

// ...

fn pit_irq(state: *arch::state, irq: u8) void = {
	sched::switchtask();
	arch::pic_eoi(arch::PIT_IRQ);
};

The PIT, or programmable interrupt timer, is a feature on x86_64 which provides exactly what we need: periodic interrupts. This code configures it to tick at 100 Hz and sets up a little IRQ handler which calls sched::switchtask to perform the actual context switch.

Recall that, by the time sched::switchtask is invoked, the CPU and interrupt handler have already stashed all of the current task’s registers into its state struct. All we have to do now is pick out the next task and restore its state.

// see idle.s
let idle: arch::state;

// Switches to the next task.
export fn switchtask() void = {
	// Save state
	arch::save();

	match (next()) {
	case let task: *objects::taskstate =>
		active = task;
		objects::vspace_activate(&task.vspace)!;
		arch::restore(&task.regs);
	case null =>
		arch::restore(&idle);
	};
};

fn next() nullable *objects::taskstate = {
	let next = active.next;
	for (next != active) {
		if (next == null) {
			next = tasks;
			continue;
		};
		const cand = next as *objects::taskstate;
		if (objects::task_schedulable(cand)) {
			return cand;
		};
		next = cand.next;
	};
	const next = next as *objects::taskstate;
	if (objects::task_schedulable(next)) {
		return next;
	};
	return null;
};

Pretty straightforward. The scheduler maintains a linked list of tasks, picks the next one which is schedulable,6 then runs it. If there are no schedulable tasks, it runs the idle task.

Err, wait, what’s the idle task? Simple: it’s another state object (i.e. a set of CPU registers) which essentially works as a statically allocated do-nothing thread.

const idle_frame: [2]uintptr = [0, &pause: uintptr];

// Initializes the state for the idle thread.
export fn init_idle(idle: *state) void = {
	*idle = state {
		cs = seg::KCODE << 3,
		ss = seg::KDATA << 3,
		rflags = (1 << 21) | (1 << 9),
		rip = &pause: uintptr: u64,
		rbp = &idle_frame: uintptr: u64,
		...
	};
};

“pause” is a simple loop:

.global arch.pause
arch.pause:
	hlt
	jmp arch.pause

In the situation where every task is blocking on I/O, there’s nothing for the CPU to do until the operation finishes. So, we simply halt and wait for the next interrupt to wake us back up, hopefully unblocking some tasks so we can schedule them again. A more sophisticated kernel might take this opportunity to go into a lower power state, perhaps, but for now this is quite sufficient.

With this last piece in place, we now have a multi-threaded operating system. But there is one more piece to consider: when a task yields its time slice.

Just because a task receives CPU time does not mean that it needs to use it. A task which has nothing useful to do can yield its time slice back to the kernel through the “yieldtask” syscall. On the face of it, this is quite simple:

// Yields the current time slice and switches to the next task.
export @noreturn fn yieldtask() void = {
	arch::sysret_set(&active.regs, 0, 0);
	switchtask();
	arch::enteruser();
};

The “sysret_set” updates the registers in the task state which correspond with system call return values to (0, 0), indicating a successful return from the yield syscall. But we don’t actually return at all: we switch to the next task and then return to that.

In addition to being called from userspace, this is also useful whenever the kernel blocks a thread on some I/O or IPC operation. For example, tasks can wait on “notification” objects, which another task can signal to wake them up — a simple synchronization primitive. The implementation makes good use of sched::yieldtask:

// Blocks the active task until this notification is signalled. Does not return
// if the operation is blocking.
export fn wait(note: *caps::capability) uint = {
	match (nbwait(note)) {
	case let word: uint =>
		return word;
	case errors::would_block =>
		let note = note_getstate(note);
		assert(note.recv == null); // TODO: support multiple receivers
		note.recv = sched::active;
		sched::active.status = task_status::BLOCKED;
		sched::yieldtask();
	};
};

Finally, that’s the last piece.

Threads in userspace

Phew! That was a lot of kernel pieces to unpack. And now for userspace… in the next post! This one is getting pretty long. Here’s what you have to look forward to:

  • Preparing the task and all of the objects it needs (such as a stack)
  • High-level operations: join, detach, exit, suspend, etc
  • Thread-local storage…
    • in the Hare compiler
    • in the ELF loader
    • at runtime
  • Putting it all together to test the kernel

We’ll see you next time!


  1. Jokes aside, for those curious about multi-threading and Hare: our official stance is not actually as strict as “no threads, period”, though in practice for many people it might amount to that. There is nothing stopping you from linking to pthreads or calling clone(2) to spin up threads in a Hare program, but the standard library explicitly provides no multi-threading support, synchronization primitives, or re-entrancy guarantees. That’s not to say, however, that one could not build their own Hare standard library which does offer these features — and, in fact, that is exactly what the Vulcan test framework for Helios provides in its Hare libraries. ↩︎

  2. Capabilities are essentially references to kernel objects. The kernel object for a task is the taskstate struct, and there can be many task capabilities which refer to this. Any task which possesses a task capability in its capability space can invoke operations against this task, such as reading or writing its registers.

    The link field is used to create a linked list of capabilities across the system. It has a doubly linked list for the next and previous capability, and a link to its parent capability, such as the memory capability from which the task state was allocated. The list is organized such that copies of the same capability are always adjacent to one another, and children always follow their parents.

    The answer to the XXX comment in task_status is yes, by the way. Something to fix later. ↩︎

  3. Only x86_64 is supported for now, but a RISC-V port is in-progress and I intend to do arm64 in the future. ↩︎

  4. For now; in the future it will have one stack per CPU. ↩︎

  5. Man, I could just go on and on and on. ↩︎

  6. A task is schedulable if it is configured properly (with a cspace, vspace, and IPC buffer) and is not currently blocking (i.e. waiting on I/O or something). ↩︎

The phrase "open source" (still) matters

16 September 2022 at 00:00

In 1988, “Resin Identification Codes” were introduced by the plastic industry. These look exactly like the recycling symbol ♺, which is not trademarked or regulated, except that a number is enclosed within the triangle. These symbols simply identify what kind of plastic was used. The vast majority of plastic is non-recyclable, but has one of these symbols on it to suggest otherwise. This is a deceptive business practice which exploits the consumer’s understanding of the recycling symbol to trick them into buying more plastic products.

The meaning of the term “open source” is broadly understood to be defined by the Open Source Initiative’s Open Source Definition, the “OSD”. Under this model, open source has enjoyed a tremendous amount of success, such that virtually all software written today incorporates open source components.

The main advantage of open source, to which much of this success can be attributed, is that it is a product of many hands. In addition to the work of its original authors, open source projects generally accept code contributions from anyone who would offer them. They also enjoy numerous indirect benefits, through the large community of Linux distros which package and ship the software, or people who write docs or books or blog posts about it, or the many open source dependencies it is likely built on top of.

Under this model, the success of an open source project is not entirely attributable to its publisher, but to both the publisher and the community which exists around the software. The software does not belong to its publisher, but to its community. I mean this not only in a moral sense, but also in a legal sense: every contributor to an open source project retains their copyright and the project’s ownership is held collectively between its community of contributors.1

The OSD takes this into account when laying out the conditions for commercialization of the software. An argument for exclusive commercialization of software by its publishers can be made when the software is the result of investments from that publisher alone, but this is not so for open source. Because it is the product of its community as a whole, the community enjoys the right to commercialize it, without limitation. This is a fundamental, non-negotiable part of the open source definition.

However, we often see the odd company or organization trying to forward an unorthodox definition of the “open source”. Generally, their argument goes something like this: “open” is just an adjective, and “source” comes from “source code”, so “open source” just means source code you can read, right?

This argument is wrong,2 but it usually conceals the speaker’s real motivations: they want a commercial monopoly over their project.3 Their real reason is “I should be able to make money from open source, but you shouldn’t”. An argument for an unorthodox definition of “open source” from this perspective is a form of motivated reasoning.

Those making this argument have good reason to believe that they will enjoy more business success if they get away with it. The open source brand is incredibly strong — one of the most successful brands in the entire software industry. Leveraging that brand will drive interest to their project, especially if, on the surface, it looks like it fits the bill (generally by being source available).

When you get down to it, this behavior is dishonest and anti-social. It leverages the brand of open source, whose success has been dependent on the OSD and whose brand value is associated with the user’s understanding of open source, but does not provide the same rights. The deception is motivated by selfish reasons: to withhold those rights from the user for their own exclusive use. This is wrong.

You can publish software under any terms that you wish, with or without commercial rights, with or without source code, whatever — it’s your right. However, if it’s not open source, it’s wrong to call it open source. There are better terms — “source available”, “fair code”, etc. If you describe your project appropriately, whatever the license may be, then I wish you nothing but success.


  1. Except when a CLA is involved. A CLA is an explicit promise that the steward of an open source project will pull the rug out later and make the project proprietary. Never sign a CLA. Don’t ask contributors to sign one, either: consider the DCO instead. ↩︎

  2. This footnote used to explain why this argument is incorrect, but after five paragraphs I decided to save it for another time, like when the peanut gallery on Hacker News makes some form of this argument in the comments on this article. ↩︎

  3. Sometimes these arguments have more to do with the non-discrimination clause of the OSD. I have a different set of arguments for this situation. ↩︎

Status update, September 2022

15 September 2022 at 00:00

I have COVID-19 and I am halfway through my stockpile of tissues, so I’m gonna keep this status update brief.

In Hare news, I finally put the last pieces into place to make cross compiling as easy as possible. Nothing else particularly world-shattering going on here. I have a bunch of new stuff in my patch queue to review once I’m feeling better, however, including bigint stuff — a big step towards TLS support. Unrelatedly, TLS support seems to be progressing upstream in qbe. (See what I did there?)

powerctl is a small new project I wrote to configure power management states on Linux. I’m pretty pleased with how it turned out. It makes for a good case study on Hare for systems programming.

In Helios, I have been refactoring the hell out of everything, rewriting or redesigning large parts of it from scratch. Presently this means that a lot of the functionality which was previously present was removed, and is being slowly re-introduced with substantial changes. The key is reworking these features to take better consideration of the full object lifecycle — creating, copying, and destroying capabilities. An improvement which ended up being useful in the course of this work is adding address space IDs (PCIDs on x86_64), which is going to offer a substantial performance boost down the line.

Alright, time for a nap. Bye!

Notes from kernel hacking in Hare, part 1

7 September 2022 at 00:00

One of the goals for the Hare programming language is to be able to write kernels, such as my Helios project. Kernels are complex beasts which exist in a somewhat unique problem space and have constraints that many userspace programs are not accustomed to. To illustrate this, I’m going to highlight a scenario where Hare’s low-level types and manual memory management approach shines to enable a difficult use-case.

Helios is a micro-kernel. During system initialization, its job is to load the initial task into memory, prepare the initial set of kernel objects for its use, provide it with information about the system, then jump to userspace and fuck off until someone needs it again. I’m going to focus on the “providing information” step here.

The information the kernel needs to provide includes details about the capabilities that init has access to (such as working with I/O ports), information about system memory, the address of the framebuffer, and so on. This information is provided to init in the bootinfo structure, which is mapped into its address space, and passed to init via a register which points to this structure.1

// The bootinfo structure.
export type bootinfo = struct {
	argv: str,

	// Capability ranges
	memory: cap_range,
	devmem: cap_range,
	userimage: cap_range,
	stack: cap_range,
	bootinfo: cap_range,
	unused: cap_range,

	// Other details
	arch: *arch_bootinfo,
	ipcbuf: uintptr,
	modules: []module_desc,
	memory_info: []memory_desc,
	devmem_info: []memory_desc,
	tls_base: uintptr,
	tls_size: size,
};

Parts of this structure are static (such as the capability number ranges for each capability assigned to init), and others are dynamic - such as structures describing the memory layout (N items where N is the number of memory regions), or the kernel command line. But, we’re in a kernel – dynamically allocating data is not so straightforward, especially for units smaller than a page!2 Moreover, the data structures allocated here need to be visible to userspace, and kernel memory is typically not available to userspace. A further complication is the three different address spaces we’re working with here: a bootinfo object has a physical memory address, a kernel address, and a userspace address — three addresses to refer to a single object in different contexts.

Here’s an example of what the code shown in this article is going to produce:

A 64 by 64 grid of cells representing a page of physical memory. The first set
of cells are colored blue; the next set green; then purple; the remainder are
brown.

This is a single page of physical memory which has been allocated for the bootinfo data, where each cell is a byte. The bootinfo structure itself comes first, in blue. Following this is an arch-specific bootinfo structure, in green:

// x86_64-specific boot information
export type arch_bootinfo = struct {
	// Page table capabilities
	pdpt: cap_range,
	pd: cap_range,
	pt: cap_range,

	// vbe_mode_info physical address from multiboot (or zero)
	vbe_mode_info: uintptr,
};

After this, in purple, is the kernel command line. These three structures are always consistently allocated for any boot configuration, so the code which sets up the bootinfo page (the code we’re going to read now) always provisions them. Following these three items is a large area of free space (indicated in brown) which will be used to populate further dynamically allocated bootinfo structures, such as descriptions of physical memory regions.

The code to set this up is bootinfo_init, which is responsible for allocating a suitable page, filling in the bootinfo structure, and preparing a vector to dynamically allocate additional data on this page. It also sets up the arch bootinfo and argv, so the page looks like this diagram when the function returns. And here it is, in its full glory:

// Initializes the bootinfo context.
export fn bootinfo_init(heap: *heap, argv: str) bootinfo_ctx = {
	let cslot = caps::cslot { ... };
	let page = objects::init(ctype::PAGE, &cslot, &heap.memory)!;
	let phys = objects::page_phys(page);
	let info = mem::phys_tokernel(phys): *bootinfo;

	const bisz = size(bootinfo);
	let bootvec = (info: *[*]u8)[bisz..arch::PAGESIZE][..0];

	let ctx = bootinfo_ctx {
		page = cslot,
		info = info,
		arch = null: *arch_bootinfo, // Fixed up below
		bootvec = bootvec,
	};

	let (vec, user) = mkbootvec(&ctx, size(arch_bootinfo), size(uintptr));
	ctx.arch = vec: *[*]u8: *arch_bootinfo;
	info.arch = user: *arch_bootinfo;

	let (vec, user) = mkbootvec(&ctx, len(argv), 1);
	vec[..] = strings::toutf8(argv)[..];
	info.argv = *(&types::string {
		data = user: *[*]u8,
		length = len(argv),
		capacity = len(argv),
	}: *str);

	return ctx;
};

The first three lines are fairly straightforward. Helios uses capability-based security, similar in design to seL4. All kernel objects — such as pages of physical memory — are utilized through the capability system. The first two lines set aside a slot to store the page capability in, then allocate a page using that slot. The next two lines grab the page’s physical address and use mem::phys_tokernel to convert it to an address in the kernel’s virtual address space, so that the kernel can write data to this page.

The next two lines are where it starts to get a little bit interesting:

const bisz = size(bootinfo);
let bootvec = (info: *[*]u8)[bisz..arch::PAGESIZE][..0];

This casts the “info” variable (of type *bootinfo) to a pointer to an unbounded array of bytes (*[*]u8). This is a little bit dangerous! Hare’s arrays are bounds tested by default and using an unbounded type disables this safety feature. We want to get a bounded slice again soon, which is what the first slicing operator here does: [bisz..arch::PAGESIZE]. This obtains a bounded slice of bytes which starts from the end of the bootinfo structure and continues to the end of the page.

The last expression, another slicing expression, is a little bit unusual. A slice type in Hare has the following internal representation:

type slice = struct {
	data: nullable *void,
	length: size,
	capacity: size,
};

When you slice an unbounded array, you get a slice whose length and capacity fields are equal to the length of the slicing operation, in this case arch::PAGESIZE - bisz. But when you slice a bounded slice, the length field takes on the length of the slicing expression but the capacity field is calculated from the original slice. So by slicing our new bounded slice to the 0th index ([..0]), we obtain the following slice:

slice {
	data = &(info: *[*]bootinfo)[1]: *[*]u8,
	length = 0,
	capacity = arch::PAGESIZE - bisz,
};

In plain English, this is a slice whose base address is the address following the bootinfo structure and whose capacity is the remainder of the free space on its page, with a length of zero. This is something we can use static append with!3

// Allocates a buffer in the bootinfo vector, returning the kernel vector and a
// pointer to the structure in the init vspace.
fn mkbootvec(info: *bootinfo_ctx, sz: size, al: size) ([]u8, uintptr) = {
	const prevlen = len(info.bootvec);
	let padding = 0z;
	if (prevlen % al != 0) {
		padding = al - prevlen % al;
	};
	static append(info.bootvec, [0...], sz + padding);
	const vec = info.bootvec[prevlen + padding..];
	return (vec, INIT_BOOTINFO_ADDR + size(bootinfo): uintptr prevlen: uintptr);
};

In Hare, slices can be dynamically grown and shrunk using the append, insert, and delete keywords. This is pretty useful, but not applicable for our kernel — remember, no dynamic memory allocation. Attempting to use append in Helios would cause a linking error because the necessary runtime code is absent from the kernel’s Hare runtime. However, you can also statically append to a slice, as shown here. So long as the slice has a sufficient capacity to store the appended data, a static append or insert will succeed. If not, an assertion is thrown at runtime, much like a normal bounds test.

This function makes good use of it to dynamically allocate memory from the bootinfo page. Given a desired size and alignment, it statically appends a suitable number of zeroes to the page, takes a slice of the new data, and returns both that slice (in the kernel’s address space) and the address that data will have in the user address space. If we return to the earlier function, we can see how this is used to allocate space for the arch_bootinfo structure:

let (vec, user) = mkbootvec(&ctx, size(arch_bootinfo), size(uintptr));
ctx.arch = vec: *[*]u8: *arch_bootinfo;
info.arch = user: *arch_bootinfo;

The “ctx” variable is used by the kernel to keep track of its state while setting up the init task, and we stash the kernel’s pointer to this data structure in there, and the user’s pointer in the bootinfo structure itself.

This is also used to place argv into the bootinfo page:

let (vec, user) = mkbootvec(&ctx, len(argv), 1);
vec[..] = strings::toutf8(argv)[..];
info.argv = *(&types::string {
	data = user: *[*]u8,
	length = len(argv),
	capacity = len(argv),
}: *str);

Here we allocate a vector whose length is the length of the argument string, with an alignment of one, and then copy argv into it as a UTF-8 slice. Slice copy expressions like this one are a type-safe and memory-safe way to memcpy in Hare. Then we do something a bit more interesting.

Like slices, strings have an internal representation in Hare which includes a data pointer, length, and capacity. The types module provides a struct with this representation so that you can do low-level string manipulation in Hare should the task call for it. Hare’s syntax allows us to take the address of a literal value, such as a types::string struct, using the & operator. Then we cast it to a pointer to a string and dereference it. Ta-da! We set the bootinfo argv field to a str value which uses the user address of the argument vector.

Some use-cases call for this level of fine control over the precise behavior of your program. Hare’s goal is to accommodate this need with little fanfare. Here we’ve drawn well outside of the lines of Hare’s safety features, but sometimes it’s useful and necessary to do so. And Hare provides us with the tools to get the safety harness back on quickly, such as we saw with the construction of the bootvec slice. This code is pretty weird but to an experienced Hare programmer (which, I must admit, the world has very few of) it should make sense.

I hope you found this interesting! I’m going back to kernel hacking. Next up is loading the userspace ELF image into its address space. I had this working before but decided to rewrite it. Wish me good luck!


  1. %rdi, if you were curious. Helios uses the System-V ABI, where %rdi is used as the first parameter to a function call. This isn’t exactly a function call but the precedent is useful. ↩︎

  2. 4096 bytes. ↩︎

  3. Thanks to Rahul of W3Bits for this CSS. ↩︎

In praise of qemu

2 September 2022 at 00:00

qemu is another in a long line of great software started by Fabrice Bellard. It provides virtual machines for a wide variety of software architectures. Combined with KVM, it forms the foundation of nearly all cloud services, and it runs SourceHut in our self-hosted datacenters. Much like Bellard’s ffmpeg revolutionized the multimedia software industry, qemu revolutionized virtualisation.

qemu comes with a large variety of studiously implemented virtual devices, from standard real-world hardware like e1000 network interfaces to accelerated virtual hardware like virtio drives. One can, with the right combination of command line arguments, produce a virtual machine of essentially any configuration, either for testing novel configurations or for running production-ready virtual machines. Network adapters, mouse & keyboard, IDE or SCSI or SATA drives, sound cards, graphics cards, serial ports — the works. Lower level, often arch-specific features, such as AHCI devices, SMP, NUMA, and so on, are also available and invaluable for testing any conceivable system configurations. And these configurations work, and work reliably.

I have relied on this testing quite a bit when working on kernels, particularly on my own Helios kernel. With a little bit of command line magic, I can run a fully virtualised system with a serial driver connected to the parent terminal, with a hardware configuration appropriate to whatever I happen to be testing, in a manner such that running and testing my kernel is no different from running any other program. With -gdb I can set up gdb remote debugging and even debug my kernel as if it were a typical program. Anyone who remembers osdev in the Bochs days — or even earlier — understands the unprecedented luxury of such a development environment. Should I ever find myself working on a hardware configuration which is unsupported by qemu, my very first step will be patching qemu to support it. In my reckoning, qemu support is nearly as important for bringing up a new system as a C compiler is.

And qemu’s implementation in C is simple, robust, and comprehensive. On the several occasions when I’ve had to read the code, it has been a pleasure. Furthermore, the comprehensive approach allows you to build out a virtualisation environment tuned precisely to your needs, whatever they may be, and it is accommodating of many needs. Sure, it’s low level — running a qemu command line is certainly more intimidating than, say, VirtualBox — but the trade-off in power afforded to the user opens up innumerable use-cases that are simply not available on any other virtualisation platform.

One of my favorite, lesser-known features of qemu is qemu-user, which allows you to register a binfmt handler to run executables compiled for an arbitrary architecture on Linux. Combined with a little chroot, this has made cross-arch development easier than it has ever been before, something I frequently rely on when working on Hare. If you do cross-architecture work and you haven’t set up qemu-user yet, you’re missing out.

$ uname -a
Linux taiga 5.15.63-0-lts #1-Alpine SMP Fri, 26 Aug 2022 07:02:59 +0000 x86_64 GNU/Linux
$ doas chroot roots/alpine-riscv64/ /bin/sh
# uname -a
Linux taiga 5.15.63-0-lts #1-Alpine SMP Fri, 26 Aug 2022 07:02:59 +0000 riscv64 Linux

This is amazing.

qemu also holds a special place in my heart as one of the first projects I contributed to over email 🙂 And they still use email today, and even recommend SourceHut to make the process easier for novices.

So, to Fabrice, and the thousands of other contributors to qemu, I offer my thanks. qemu is one of my favorite pieces of software.

powerctl: A small case study in Hare for systems programming

28 August 2022 at 00:00

powerctl is a little weekend project I put together to provide a simple tool for managing power states on Linux. I had previously put my laptop into suspend with a basic “echo mem | doas tee /sys/power/state”, but this leaves a lot to be desired. I have to use doas to become root, and it’s annoying to enter my password — not to mention difficult to use in a script or to attach to a key binding. powerctl is the solution: a small 500-line Hare program which provides comprehensive support for managing power states on Linux for non-privileged users.

This little project ended up being a useful case-study in writing a tight systems program in Hare. It has to do a few basic tasks which Hare shines in:

  • setuid binaries
  • Group lookup from /etc/group
  • Simple string manipulation
  • Simple I/O within sysfs constraints

Linux documents these features here, so it’s a simple matter of rigging it up to a nice interface. Let’s take a look at how it works.

First, one of the base requirements for this tool is to run as a non-privileged user. However, since writing to sysfs requires root, this program will have to be setuid, so that it runs as root regardless of who executes it. To prevent any user from suspending the system, I added a “power” group and only users who are in this group are allowed to use the program. Enabling this functionality in Hare is quite simple:

use fmt;
use unix;
use unix::passwd;

def POWER_GROUP: str = "power";

// Determines if the current user is a member of the power group.
fn checkgroup() bool = {
	const uid = unix::getuid();
	const euid = unix::geteuid();
	if (uid == 0) {
		return true;
	} else if (euid != 0) {
		fmt::fatal("Error: this program must be installed with setuid (chmod u+s)");
	};

	const group = match (passwd::getgroup(POWER_GROUP)) {
	case let grent: passwd::grent =>
		yield grent;
	case void =>
		fmt::fatal("Error: {} group missing from /etc/group", POWER_GROUP);
	};
	defer passwd::grent_finish(&group);

	const gids = unix::getgroups();
	for (let i = 0z; i < len(gids); i += 1) {
		if (gids[i] == group.gid) {
			return true;
		};
	};

	return false;
};

The POWER_GROUP variable allows distributions that package powerctl to configure exactly which group is allowed to use this tool. Following this, we compare the uid and effective uid. If the uid is zero, we’re already running this tool as root, so we move on. Otherwise, if the euid is nonzero, we lack the permissions to continue, so we bail out and tell the user to fix their installation.

Then we fetch the details for the power group from /etc/group. Hare’s standard library includes a module for working with this file. Once we have the group ID from the string, we check the current user’s supplementary group IDs to see if they’re a member of the appropriate group. Nice and simple. This is also the only place in powerctl where dynamic memory allocation is required, to store the group details, which are freed with “defer passwd::grent_finish”.

The tool also requires some simple string munging to identify the supported set of states. If we look at /sys/power/disk, we can see the kind of data we’re working with:

$ cat /sys/power/disk 
[platform] shutdown reboot suspend test_resume 

These files are a space-separated list of supported states, with the currently enabled state enclosed in square brackets. Parsing these files is a simple matter for Hare. We start with a simple utility function which reads the file and prepares a string tokenizer which splits the string on spaces:

fn read_states(path: str) (strings::tokenizer | fs::error | io::error) = {
	static let buf: [512]u8 = [0...];

	const file = os::open(path)?;
	defer io::close(file)!;

	const z = match (io::read(file, buf)?) {
	case let z: size =>
		yield z;
	case =>
		abort("Unexpected EOF from sysfs");
	};
	const string = strings::rtrim(strings::fromutf8(buf[..z]), '\n');
	return strings::tokenize(string, " ");
};

The error handling here warrants a brief note. This function can fail if the file does not exist or if there is an I/O error when reading it. I don’t think that I/O errors are possible in this specific case (they can occur when writing to these files, though), but we bubble it up regardless using “io::read()?”. The file might not exist if these features are not supported by the current kernel configuration, in which case it’s bubbled up as “errors::noentry” via “os::open()?”. These cases are handled further up the call stack. The other potential error site is “io::close”, which can fail but only in certain circumstances (such as closing the same file twice), and we use the error assertion operator ("!") to indicate that the programmer believes this case cannot occur. The compiler will check our work and abort at runtime should this assumption be proven wrong in practice.

In the happy path, we read the file, trim off the newline, and return a tokenizer which splits on spaces. The storage for this string is borrowed from “buf”, which is statically allocated.

The usage of this function to query supported disk suspend behaviors is here:

fn read_disk_states() ((disk_state, disk_state) | fs::error | io::error) = {
	const tok = read_states("/sys/power/disk")?;

	let states: disk_state = 0, active: disk_state = 0;
	for (true) {
		let tok = match (strings::next_token(&tok)) {
		case let s: str =>
			yield s;
		case void =>
			break;
		};
		const trimmed = strings::trim(tok, '[', ']');

		const state = switch (trimmed) {
		case "platform" =>
			yield disk_state::PLATFORM;
		case "shutdown" =>
			yield disk_state::SHUTDOWN;
		case "reboot" =>
			yield disk_state::REBOOT;
		case "suspend" =>
			yield disk_state::SUSPEND;
		case "test_resume" =>
			yield disk_state::TEST_RESUME;
		case =>
			continue;
		};
		states |= state;
		if (trimmed != tok) {
			active = state;
		};
	};

	return (states, active);
};

This function returns a tuple which includes all of the supported disk states OR’d together, and a value which indicates which state is currently enabled. The loop iterates through each of the tokens from the tokenizer returned by read_states, trims off the square brackets, and adds the appropriate state bits. We also check the trimmed token against the original token to detect which state is currently active.

There’s two edge cases to be taken into account here: what happens if Linux adds more states in the future, and what happens if none of the states are active? In the former case, we have the continue branch of the switch statement mid-loop. Hare requires all switch statements to be exhaustive, so the compiler forces us to consider this edge case. For the latter case, the return value will be zero, simply indicating that none of these states are active. This is not actually possible given the invariants for this kernel interface, but we could end up in this situation if the kernel adds a new disk mode and that disk mode is active when this code runs.

When the time comes to modify these states, either to put the system to sleep or to configure its behavior when put to sleep, we use the following function:

fn write_state(path: str, state: str) (void | fs::error | io::error) = {
	const file = os::open(path, fs::flags::WRONLY | fs::flags::TRUNC)?;
	defer io::close(file)!;
	let buf: [128]u8 = [0...];
	const file = &bufio::buffered(file, [], buf);
	fmt::fprintln(file, state)?;
};

This code is working within a specific constraint of sysfs: it must complete the write operation in a single syscall. One of Hare’s design goals is giving you sufficient control over the program’s behavior to plan for such concerns. The means of opening the file — WRONLY | TRUNC — was also chosen deliberately. The “single syscall” is achieved by using a buffered file, which soaks up writes until the buffer is full and then flushes them out all at once. The buffered stream flushes automatically on newlines by default, so the “ln” of “fprintln” causes the write to complete in a single call.

With this helper in place, we can write power states. The ones which configure the kernel, but don’t immediately sleep, are straightforward:

// Sets the current mem state.
fn set_mem_state(state: mem_state) (void | fs::error | io::error) = {
	write_state("/sys/power/mem_sleep", mem_state_unparse(state))?;
};

The star of the show, however, has some extra concerns:

// Sets the current sleep state, putting the system to sleep.
fn set_sleep_state(state: sleep_state) (void | fs::error | io::error) = {
	// Sleep briefly so that the keyboard driver can process the key up if
	// the user runs this program from the terminal.
	time::sleep(250 * time::MILLISECOND);
	write_state("/sys/power/state", sleep_state_unparse(state))?;
};

If you enter sleep with a key held down, key repeat will kick in for the duration of the sleep, so when running this from the terminal you’ll resume to find a bunch of new lines. The time::sleep call is a simple way to avoid this, by giving the system time to process your key release event before sleeping. A more sophisticated solution could open the uinput devices and wait for all keys to be released, but that doesn’t seem entirely necessary.

Following this, we jump into the dark abyss of a low-power coma.

And that’s all there is to it! A few hours of work and 500 lines of code later and we have a nice little systems program to make suspending my laptop easier. I was pleasantly surprised to find out how well this little program plays to Hare’s strengths. I hope you found it interesting! And if you happen to need a simple tool for suspending your Linux machines, powerctl might be the program for you.

A review of postmarketOS on the Xiaomi Poco F1

25 August 2022 at 00:00

I have recently had cause to start looking into mainline Linux phones which fall outside of the common range of grassroots phones like the PinePhone (which was my daily driver for the past year). The postmarketOS wiki is a great place to research candidate phones for this purpose, and the phone I landed on is the Xiaomi Poco F1, which I picked up on Amazon.nl (for ease of return in case it didn’t work out) for 270 Euro. Phones of this nature have a wide range of support from Linux distros like postmarketOS, from “not working at all” to “mostly working”. The essential features I require in a daily driver phone are (1) a working modem and telephony support, (2) mobile data, and (3) reasonably good performance and battery life; plus of course some sane baseline expectations like a working display and touchscreen driver.

The use of mainline Linux on a smartphone requires a certain degree of bullshit tolerance, and the main question is whether or not the bullshit exceeds your personal threshold. The Poco F1 indeed comes with some bullshit, but I’m pleased to report that it falls short of my threshold and represents a significant quality-of-life improvement over the PinePhone setup I have been using up to now.

The bullshit I have endured for the Poco F1 setup can be categorized into two parts: initial setup and ongoing problems. Of the two, the initial setup is by far the worst. These phones are designed to run Android first, rather than the mainline Linux first approach seen in devices like the PinePhone and Librem 5. This means that it’s back to dealing with things like Android recovery, fastboot, and so on, during the initial setup. The most severe pain point for Xiaomi phones is unlocking the bootloader.

The only officially supported means of doing this is via a Windows-only application published by Xiaomi. A reverse engineered Java application supposedly provides support for completing this process on Linux. However, this approach comes with the typical bullshit of setting up a working Java environment, and, crucially, Xiaomi appears to have sabotaged this effort via a deliberate attempt to close the hole by returning error messages from this reverse engineered API which direct the user to the official tool instead. On top of this, Xiaomi requires you to associate the phone to be unlocked with a user account on their services, paired to a phone number, and has a 30-day waiting period between unlocks. I ultimately had to resort to a Windows 10 VM with USB passthrough to get the damn thing unlocked. This is very frustrating and far from the spirit of free software; Xiaomi earns few points for openness in my books.

Once unlocked, the “initial setup bullshit” did not cease. The main issue is that the postmarketOS flashing tool (which is just a wrapper around fastboot) seemed to have problems writing a consistent filesystem. I was required to apply a level of Linux expertise which exceeds that of even most enthusiasts to obtain a shell in the initramfs, connect to it over postmarketOS’s telnet debugging feature, and run fsck.ext4 to fix the filesystem. Following this, I had to again apply a level of Alpine Linux expertise which exceeds that of many enthusiasts to repair installed packages and get everything up to a baseline of workitude. Overall, it took me the better part of a day to get to a baseline of “running a working installation of postmarketOS”.

However: following the “initial setup bullshit”, I found a very manageable scale of “ongoing problems”. The device’s base performance is excellent, far better than the PinePhone — it just performs much like I would expect from a normal phone. PostmarketOS is, as always, brilliant, and all of the usual mainline Alpine Linux trimmings I would expect are present — I can SSH in, I easily connected it to my personal VPN, and I’m able to run most of the software I’m already used to from desktop Linux systems (though, of course, GUI applications range widely in their ability to accomodate touch screens and a portrait mobile form-factor). I transferred my personal data over from my PinePhone using a method which is 100% certifiably absent of bullshit, namely just rsyncing over my home directory. Excellent!

Telephony support also works pretty well. Audio profiles are a bit buggy, and I can often find my phone using my headphone output while I don’t have them plugged in instead of the speakers, having to resort to manually switching between them from time to time. However, I have never had an issue with the audio profiles being wrong during a phone call (the modem works, by the way); earpiece and speakerphone both work as expected. That said, I have heard complaints from recipients of my phone calls about hearing an echo of their own voice. Additionally, DTMF tones do not work, but the fix has already been merged and is expected in the next release of ModemManager. SMS and mobile data work fine, and mobile data works with a lesser degree of bullshit than I was prepared to expect after reading the pmOS wiki page for this device.

Another problem is that the phone’s onboard cameras do not work at all, and it seems unlikely that this will be solved in the near future. This is not really an issue for me. Another papercut is that Phosh handles the display notch poorly, and though pmOS provides a “tweak” tool which can move the clock over from behind the notch, it leaves something to be desired. The relevant issue is being discused on the Phosh issue tracker and a fix is presumably coming soon — it doesn’t seem particularly difficult to solve. I have also noted that, though GPS works fine, Mepo renders incorrectly and Gnome Maps has (less severe) display issues as well.

The battery life is not as good as the PinePhone, which itself is not as good as most Android phones. However, it meets my needs. It seems to last anywhere from 8 to 10 hours depending on usage, following a full night’s charge. As such, I can leave it off of the juice when I go out without too much fear. That said, I do keep a battery bank in my backpack just in case, but that’s also just a generally useful thing to have around. I think I’ve lent it to others more than I’ve used it myself.

There are many other apps which work without issues. I found that Foliate works great for reading e-books and Evince works nicely for PDFs (two use-cases which one might perceive as related, but which I personally have different UI expectations for). Firefox has far better performance on this device than on the PinePhone and allows for very comfortable web browsing. I also discovered Gnome Feeds which, while imperfect, accommodates my needs regarding an RSS feed reader. All of the “standard” mobile Linux apps that worked fine on the PinePhone also work fine here, such as Lollypop for music and the Porfolio file manager.

I was pleasantly surprised that, after enduring some more bullshit, I was able to get Waydroid to work, allowing me to run Android applications on this phone. My expectations for this were essentially non-existent, so any degree of workitude was a welcome surprise, and any degree of non-workitude was the expected result. On the whole, I’m rather impressed, but don’t expect anything near perfection. The most egregious issue is that I found that internal storage simply doesn’t work, so apps cannot store or read common files (though they seem to be able to persist their own private app data just fine). The camera does not work, so the use-case I was hoping to accommodate here — running my bank’s Android app — is not possible. However, I was able to install F-Droid and a small handful of Android apps that work with a level of performance which is indistinguishable from native Android performance. It’s not quite there yet, but Waydroid has a promising future and will do a lot to bridge the gap between Android and mainline Linux on mobile.

On the whole, I would rate the Poco F1’s bullshit level as follows:

  • Initial setup: miserable
  • Ongoing problems: minor

I have a much higher tolerance for “initial setup” bullshit than for ongoing problems bullshit, so this is a promising result for my needs. I have found that this device is ahead of the PinePhone that I had been using previously in almost all respects, and I have switched to it as my daily driver. In fact, this phone, once the initial bullshit is addressed, is complete enough that it may be the first mainline Linux mobile experience that I might recommend to others as a daily driver. I’m glad that I made the switch.

PINE64 has let its community down

18 August 2022 at 00:00

Context for this post:


I know that apologising and taking responsibility for your mistakes is difficult. It seems especially difficult for commercial endeavours, which have fostered a culture of cold disassociation from responsibility for their actions, where admitting to wrongdoing is absolutely off the table. I disagree with this culture, but I understand where it comes from, and I can empathise with those who find themselves in the position of having to reconsider their actions in the light of the harm they have done. It’s not easy.

But, the reckoning must come. I have been a long-time supporter of PINE64. On this blog I have written positively about the PinePhone and PineBook Pro.1 I believed that PINE64 was doing the right thing and was offering something truly revolutionary on the path towards getting real FOSS systems into phones. I use a PinePhone as my daily driver,2 and I also own a PineBook Pro, two RockPro64s, a PinePhone Pro, and a PineNote as well. All of these devices have issues, some of them crippling, but PINE64’s community model convinced me to buy these with confidence in the knowledge that they would be able to work with the community to address these flaws given time.

However, PINE64’s treatment of its community has been in a steady decline for the past year or two, culminating in postmarketOS developer Martijn Braam’s blog post outlining a stressful and frustrating community to participate in, a lack of respect from PINE64 towards this community, and a model moving from a diverse culture that builds working software together to a Manjaro mono-culture that doesn’t. PINE64 offered a disappointing response. In their blog post, they dismiss the problems Martijn brings up, paint his post as misguided at best and disingenuous at worst, and fail to take responsibility for their role in any of these problems.

The future of PINE64’s Manjaro mono-culture is dim. Manjaro is a very poorly run Linux distribution with a history of financial mismanagement, ethical violations, security incidents, shipping broken software, and disregarding the input of its peers in the distribution community. Just this morning they allowed their SSL certificates to expire — for the fourth time. An open letter, signed jointly by 16 members of the Linux mobile community, called out bad behaviors which are largely attributable to Manjaro. I do not respect their privileged position in the PINE64 community, which I do not expect to be constructive or in my best interests. I have never been interested in running Manjaro on a PINE64 device and once they turn their back on the lush ecosystem they promised, I no longer have any interest in the platform.

It’s time for PINE64 to take responsibility for these mistakes, and make clear plans to correct them. To be specific:

  • Apologise for mistreatment of community members.
  • Make a tangible commitment to honoring and respecting the community.
  • Rescind their singular commitment to Manjaro.
  • Re-instate community editions and expand the program.
  • Deal with this stupid SPI problem. The community is right, listen to them.

I understand that it’s difficult to acknowledge our mistakes. But it is also necessary, and important for the future of PINE64 and the future of mobile Linux in general. I call on TL Lim, Marek Kraus, and Lukasz Erecinski to personally answer for these problems.

There are three possible outcomes to this controversy, depending on PINE64’s response. If PINE64 refuses to change course, the community will continue to decay and fail — the community PINE64 depends on to make its devices functional and useful. Even the most mature PINE64 products still need a lot of work, and none of the new products are even remotely usable. This course of events will be the end of PINE64 and deal a terrible blow to the mobile FOSS movement.

The other option for PINE64 to change its behavior. They do this with grace, or without. If they crumble under public pressure and, for example, spitefully agree to re-instate community editions without accepting responsibility for their wrongdoings, it does not bode well for addressing the toxic environment which is festering in the PINE64 community. This may be better than the worst case, but may not be enough. New community members may hesitate to join, maligned members may not offer their forgiveness, and PINE64’s reputation will suffer for a long time.

The last option is for PINE64 to act with grace and humility. Acknowledge your mistakes and apologise to those who have been hurt. Re-commit to honoring your community and treating your peers with respect. Remember, the community are volunteers. They have no obligation to make peace, so it’s on you to mend these wounds. It will still be difficult to move forward, but doing it with humility, hand in hand with the community, will set PINE64 up with the best chance of success. We’re counting on you to do the right thing.


  1. The latter post, dated in May 2021, also mentions the u-Boot SPI issue that PINE64’s push-back on ultimately led Martijn to quit the PINE64 community. PINE64’s justification is “based on the fact that for years SPI was largely unused on PINE64 devices”, but people have been arguing that SPI should be used for years, too. ↩︎

  2. Though it has and has always had serious issues that would prevent me from recommending it to others. It still needs work. ↩︎

Status update, August 2022

16 August 2022 at 00:00

It is a blessedly cool morning here in Amsterdam. I was busy moving house earlier this month, so this update is a bit quieter than most.

For a fun off-beat project this month, I started working on a GameBoy emulator written in Hare. No promises on when it will be functional or how much I plan on working on it – just doing it for fun. In more serious Hare news, I have implemented Thread-Local Storage (TLS) for qbe, our compiler backend. Hare’s standard library does not support multi-threading, but I needed this for Helios, whose driver library does support threads. It will also presumably be of use for cproc once it lands upstream.

Speaking of Helios, it received the runtime components for TLS support on x86_64, namely the handling of %fs and its base register MSR in the context switch, and updates to the ELF loader for handling .tdata/.tbss sections. I have also implemented support for moving and copying capabilities, which will be useful for creating new processes in userspace. Significant progress towards capability destructors was also made, with some capabilities — pages and page tables in particular — being reclaimable now. Next goal is to finish up all of this capability work so that you can freely create, copy, move, and destroy capabilities, then use all of these features to implement a simple shell. There is also some refactoring due at some point soon, so we’ll see about that.

Other Hare progress has been slow this month, as I’m currently looking at a patch queue 123 emails backed up. When I’m able to sit down and get through these, we can expect a bunch of updates in short order.

SourceHut news will be covered in the “what’s cooking” post later today. That’s all for now! Thanks for tuning in.

How I wish I could organize my thoughts

10 August 2022 at 00:00

I keep a pen & notebook on my desk, which I make liberal use of to jot down my thoughts. It works pretty well: ad-hoc todo lists, notes on problems I’m working on, tables, flowcharts, etc. It has some limitations, though. Sharing anything out of my notebook online is an awful pain in the ass. I can’t draw a straight line to save my life, so tables and flowcharts are a challenge. No edits, either, so lots of crossed-out words and redrawn or rewritten pages. And of course, my handwriting sucks and I can type much more efficiently than I can write. I wish this was a digital medium, but there are not any applications available which can support the note-taking paradigm that I wish I could have. What would that look like?

Well, like this (click for full size):

A mock-up of an application. A4 pages are arranged ad-hoc on a grid.
Handwritten notes and drawings appear in red across the grid and over the pages.
A flowchart is shown outside of a page.

I don’t have the bandwidth to take on a new project of this scope, so I’ll describe what I think this should look like in the hopes that it will inspire another team to work on something like this. Who knows!

The essential interface would be an infinite grid on which various kinds of objects can be placed by the user. The most important of these objects would be pages, at a page size configurable by the user (A4 by default). You can zoom in on a page (double click it or something) to make it your main focus, zooming in automatically to an appropriate level for editing, then type away. A simple WYSIWYG paradigm would be supported here, perhaps supporting only headings, bold/italic text, and ordered and unordered lists — enough to express your thoughts but not a full blown document editor/typesetter.1 When you run out of page, another is generated next to the current page, either to the right or below — configurable.

Other objects would include flowcharts, tables, images, hand-written text and drawings, and so on. These objects can be placed free form on the grid, or embedded in a page, or moved between each mode.

The user input paradigm should embrace as many modes of input as the user wants to provide. Mouse and keyboard: middle click to pan, scroll to zoom in or out, left click and drag to move objects around, shift+click to select objects, etc. A multi-point trackpad should support pinch to zoom, two finger pan, etc. Touch support is fairly obvious. Drawing tablet support is also important: the user should be able to use one to draw and write free-form. I’d love to be able to make flowcharts by drawing boxes and arrows and having the software recognize them and align them to the grid as first-class vector objects. Some drawing tablets support trackpad and touch-screen-like features as well — so all of those interaction options should just werk.

Performance is important here. I should be able to zoom in and out and pan around while all of the objects rasterize themselves in real-time, never making the user suffer through stuttery interactions. There should also be various ways to export this content. A PDF exporter should let me arrange the pages in the desired linear order. SVG exporters should be able to export objects like flowcharts and diagrams. Other potential features includes real-time collaboration or separate templates for presentations.

Naturally this application should be free software and should run on Linux. However, I would be willing to pay a premium price for this tool — a one-time fee of as much as $1000, or subscriptions on the order of $100/month if real-time collaboration or cloud synchronization are included. If you’d like some ideas for how to monetize free software projects like this, feel free to swing by my talk on the subject in Italy early this September to talk about it.

Well, that’s enough dreaming for now. I hope this inspired you, and in the meantime it’s back to pen and paper for me.


  1. Though perhaps you could import pages from an external PDF, so you can typeset stuff in LaTeX or whatever and then work with those documents inside of this tool. Auto-reload from the source PDFs and so on would be a bonus for sure. ↩︎

Conciseness

26 July 2022 at 00:00

Conciseness is often considered a virtue among hackers and software engineers. FOSS maintainers in particular generally prefer to keep bug reports, questions on mailing lists, discussions in IRC channels, and so on, close to the point and with minimal faff. It’s not considered impolite to skip the formalities — quite the opposite. So: keep your faffery to a minimum. A quick “thanks!” at the end of a discussion will generally suffice. And, when someone is being direct with you, don’t interpret it as a slight: simply indulge in the blissful freedom of a discussion absent of faffery.

Code review at the speed of email

25 July 2022 at 00:00

I’m a big proponent of the email workflow for patch submission and code review. I have previously published some content (How to use git.sr.ht’s send-email feature, Forks & pull requests vs email, git-send-email.io) which demonstrates the contributor side of this workflow, but it’s nice to illustrate the advantages of the maintainer workflow as well. For this purpose, I’ve recorded a short video demonstrating how I manage code review as an email-oriented maintainer.

Disclaimer: I am the founder of SourceHut, a platform built on this workflow which competes with platforms like GitHub and GitLab. This article’s perspective is biased.

This blog post provides additional material to supplement this video, and also includes all of the information from the video itself. For those who prefer reading over watching, you can just stick to reading this blog post. Or, you can watch the video and skim the post. Or you can just do something else! When was the last time you called your grandmother?

With hundreds of hours of review experience on GitHub, GitLab, and SourceHut, I can say with confidence the email workflow allows me to work much faster than any of the others. I can review small patches in seconds, work quickly with multiple git repositories, easily test changes and make tweaks as necessary, rebase often, and quickly chop up and provide feedback for larger patches. Working my way through a 50-email patch queue usually takes me about 20 minutes, compared to an hour or more for the same number of merge requests.

This workflow also works entirely offline. I can read and apply changes locally, and even reply with feedback or to thank contributors for their patch. My mail setup automatically downloads mail from IMAP using isync and outgoing mails are queued with postfix until the network is ready. I have often worked through my patch queue on an airplane or a train with spotty or non-functional internet access without skipping a beat. Working from low-end devices like a Pinebook or a phone are also no problem — aerc is very lightweight in the terminal and the SourceHut web interface is much lighter & faster than any other web forge.

The centerpiece of my setup is an email client I wrote specifically for software development using this workflow: aerc.1 The stock configuration of aerc is pretty good, but I make a couple of useful additions specifically for development on SourceHut. Specifically, I add a few keybindings to ~/.config/aerc/binds.conf:

[messages]
ga = :flag<Enter>:pipe -mb git am -3<Enter>
gp = :term git push<Enter>
gl = :term git log<Enter>

rt = :reply -a -Tthanks<Enter>
Rt = :reply -qa -Tquoted_thanks<Enter>

[compose::review]
V = :header -f X-Sourcehut-Patchset-Update NEEDS_REVISION<Enter>
A = :header -f X-Sourcehut-Patchset-Update APPLIED<Enter>
R = :header -f X-Sourcehut-Patchset-Update REJECTED<Enter>

The first three commands, ga, gp, and gl, are for invoking git commands. “ga” applies the current email as a patch, using git am, and “gp” simply runs git push. “gl” is useful for quickly reviewing the git log. ga also flags the email so that it shows up in the UI as having been applied, which is useful as I’m jumping all over a patch queue. I also make liberal use of \ (:filter) to filter my messages to patches applicable to specific projects or goals.

rt and Rt use aerc templates installed at ~/.config/aerc/templates/ to reply to emails after I’ve finished reviewing them. The “thanks” template is:

X-Sourcehut-Patchset-Update: APPLIED

Thanks!

{{exec "{ git remote get-url --push origin; git reflog -2 origin/master --pretty=format:%h | xargs printf '%s\n' | tac; } | xargs printf 'To %s\n   %s..%s  master -> master'" ""}}

And quoted_thanks is:

X-Sourcehut-Patchset-Update: APPLIED

Thanks!

{{exec "{ git remote get-url --push origin; git reflog -2 origin/master --pretty=format:%h | xargs printf '%s\n' | tac; } | xargs printf 'To %s\n   %s..%s  master -> master'" ""}}

On {{dateFormat (.OriginalDate | toLocal) "Mon Jan 2, 2006 at 3:04 PM MST"}}, {{(index .OriginalFrom 0).Name}} wrote:
{{wrapText .OriginalText 72 | quote}}

Both of these add a magic “X-Sourcehut-Patchset-Update” header, which updates the status of the patch on the mailing list. They also include a shell pipeline which adds some information about the last push from this repository, to help the recipient understand what happened to their patch. I often make some small edits to request the user follow-up with a ticket for some future work, or add other timely comments. The second template, quoted_reply, is also particularly useful for this: it quotes the original message so I can reply to specific parts of it, in the commit message, timely commentary, or the code itself, often pointing out parts of the code that I made some small tweaks to before applying.

And that’s basically it! You can browse all of my dotfiles here to see more details about my system configuration. With this setup I am able to work my way through a patch queue easier and faster than ever before. That’s why I like the email workflow so much: for power users, no alternative is even close in terms of efficiency.

Of course, this is the power user workflow, and it can be intimidating to learn all of these things. This is why we offer more novice-friendly tools, which lose some of the advantages but are often more intuitive. For instance, we are working on user interface on the web for patch review, mirroring our existing web interface for patch submission. But, in my opinion, it doesn’t get better than this for serious FOSS maintainers.

Feel free to reach out on IRC in #sr.ht.watercooler on Libera Chat, or via email, if you have any questions about this workflow and how you can apply it to your own projects. Happy hacking!


  1. Don’t want to switch from your current mail client? Tip: You can use more than one 🙂 I usually fire up multiple aerc instances in any case, one “main” instance and more ephemeral processes for working in specific projects. The startup time is essentially negligible, so this solution is very cheap and versatile. ↩︎

The past and future of open hardware

25 July 2022 at 00:00

They say a sucker is born every day, and at least on the day of my birth, that certainly may have been true. I have a bad habit of spending money on open hardware projects that ultimately become vaporware or seriously under-deliver on their expectations. In my ledger are EOMA68, DragonBox Pyra, the Jolla Tablet — which always had significant non-free components — and the Mudita Pure, though I did successfully receive a refund for the latter two.1

There are some success stories, though. My Pine64 devices work great — though they have non-free components — and I have a HiFive Unmatched that I’m reasonably pleased with. Raspberry Pi is going well, if you can find one — also with non-free components — and Arduino and products like it are serving their niche pretty well. I hear the MNT Reform went well, though by then I had learned to be a bit more hesitant to open my wallet for open hardware, so I don’t have one myself. Pebble worked, until it didn’t. Caveats abound in all of these projects.

What does open hardware need to succeed, and why have many projects failed? And why do the successful products often have non-free components and poor stock? We can’t blame it all on the chip shortage and/or COVID: it’s been an issue for a long time.

I don’t know the answers, but I hope we start seeing improvements. I hope that the successful projects will step into a mentorship role to provide up-and-comers with tips on how they made their projects work, and that we see a stronger focus on liberating non-free components. Perhaps Crowd Supply can do some work in helping to secure investment2 for open hardware projects, and continue the good work they’re already doing on guiding them through the development and production processes.

Part of this responsibility comes down to the consumer: spend your money on free projects, and don’t spend your money on non-free projects. But, we also need to look closely at the viability of each project, and open hardware projects need to be transparent about their plans, lest we get burned again. Steering the open hardware movement out of infancy will be a challenge for all involved.

Are you working on a cool open hardware project? Let me know. Explain how you plan on making it succeed and, if I’m convinced that your idea has promise, I’ll add a link here.


  1. I reached out to DragonBox recently and haven’t heard back yet, so let’s give them the benefit of the doubt. EOMA68, however, is, uh, not going so well. ↩︎

  2. Ideally with careful attention paid to making sure that the resulting device does not serve its investors needs better than its users needs. ↩︎

Status update, July 2022

18 July 2022 at 00:00

Hello there! It’s been a hot July week in Amsterdam, and I expect hotter days are still to come. I wish air conditioning was more popular in Europe, but alas. This month of FOSS development enjoyed a lot of small improvements in a lot of different projects.

For Hare, I have introduced a number of improvements. I wrote a new standard library module for string templates, strings::template, and a new third-party library for working with pixel buffers, pixbuf. The templating is pretty simple — as is typical for the standard library — but allows a fairly wide range of formatting options. We’ll be extending this a little bit more in the future, but it will not be a complete solution like you see in things like Jinja2. Nevertheless, it makes some use-cases, like code generation, a lot cleaner, without introducing a weighty or complex dependency.

pixbuf is pretty neat, and is the first in a line of work I have planned for graphics on Hare. It’s similar to pixman, but with a much smaller scope — it only deals with pixel buffers, handling pixel format conversions and doing small operations like fill and copy. In the future I will add simple buffer compositing as well, and extending modules like hare-png to support loading data into these buffers. Later, I plan on writing a simple vector graphics library, capable at least of rendering TinyVG images and perhaps later TinySVG as well. I’m also working on hare-wayland again, to provide a place to display these buffers.

I also introduced format::tar, which will serve as the basis of initramfs-alike functionality for Helios. On the subject of Helios, much work has been completed. I have implemented a PCI driver and a small proof-of-concept AHCI driver (for reading from SATA disks). Alexey Yerin has also been hard at work on the RISC-V port, and has successfully implemented an e1000 ethernet driver which can send and receive ICMP (ping) packets. I also completed IRQ control for userspace, so that userspace device drivers can process interrupts, and used it to write a keyboard driver for a functional DOOM port. The full DOOM port required a fair bit of work — check out that blog post for the complete details. The idle thread was also added, so that all processes can be blocked waiting on interrupts, signals, endpoints, etc. Non-blocking send, receive, and wait syscalls were also added this month.

I’m working on splitting memory capabilities into separate device- and general-purpose capabilities, then adding support for destroying capabilities when they’re no longer required. I also implemented pre-emptive multi-tasking early this month, and the vulcan test suite now has several multi-threaded tests to verify IPC functionality. However, a couple of pieces are missing — the ability to create and work with new cspaces and vspaces — in order to spawn new processes. I’ll be focusing on these tasks in the coming weeks. With these pieces in place, we can start working on Mercury and Vulcan: the driver system.

I’ll save the SourceHut news for the “what’s cooking” post later today, so that’s all for now. Until next time!

The Fediverse can be pretty toxic

9 July 2022 at 00:00

Mastodon, inspired by GNU social, together with Pleroma, form the most popular components of what we know as the “Fediverse” today. All of them are, in essence, federated, free software Twitter clones, interoperable with each other via the ActivityPub protocol.

In many respects, the Fediverse is a liberating force for good. Its federated design distributes governance and costs across many independent entities, something I view as a very strong design choice. Its moderation tools also do a pretty good job of keeping neo-nazis out of your feeds and providing a comfortable space to express yourself in, especially if your form of expression is maligned by society. Large groups of Fediverse members have found in it a home for self-expression which is denied to them elsewhere on the basis of their sexuality, gender expression, politics, or other characteristics. It’s also essentially entirely free from commercial propaganda.

But it’s still just a Twitter clone, and many of the social and psychological ills which come with that are present in the Fediverse. It’s a feed of other people’s random thoughts, often unfiltered, presented to you without value judgement — even when a value judgement may be wise. Features like boosting and liking posts, chasing after follower counts and mini-influencers, these rig up dopamine reinforcement like any other social network does. The increased character limit does not really help; most posts are pretty short and no one wants to read an essay aggressively word-wrapped in a narrow column.

The Fediverse is an environment optimized for flame wars. Arguments in this medium are held under these constraints, in public, with the peanut gallery of followers from either side stepping in and out to reinforce their position and flame the opponents. Progress is measured in gains of ideological territory and in the rising and falling swells of participants dotting their comments throughout huge threads. You are not just arguing your position, but performing it to your audience, and to your opponent’s audience.

Social networks are not good for you. The Fediverse brought out the worst in me, and it can bring out the worst in you, too. The behaviors it encourages are plainly defined as harassment, a behavior which is not unique to any ideological condition. People get hurt on the Fediverse. Keep that in mind. Consider taking a look in the mirror and asking yourself if your relationship with the platform is healthy for you and for the people around you.

Porting Doom to Helios

1 July 2022 at 00:00

Doom was an incredibly popular video game by Id software which, six years following its release, was made open source under the GPLv2 license. Thanks to this release, combined with the solid software design and lasting legacy of backwards compatibility in C, Doom has been ported to countless platforms by countless programmers. And I recently added myself to this number :)

I’m working on a new kernel called Helios, and I thought that porting Doom would present a good opportunity for proving the kernel design — you never know if you have a good design until you try to use it for real. Doom is a good target because it does not require much to get working, but it is a useful (and fun) program to port. It calls for the following features:

  1. A working C programming environment
  2. Dynamic memory allocation
  3. A place to draw the screen (a framebuffer)
  4. Keyboard input

As I was working, I gradually came to understand that Helios was pretty close to supporting all of these features, and thought that the time to give Doom a shot was coming soon. In my last status update, I shared a picture of a Helios userspace program utilizing the framebuffer provided by multiboot, ticking one box. We’ve had dynamic memory allocation in userspace working since June 8th. The last pieces were a keyboard driver and a C library.

I started with the keyboard driver, since that would let me continue to work on Hare for a little bit longer, providing a more direct benefit to the long-term goals (rather than the short-term goal of “get Doom to work”). Since Helios is a micro-kernel, the keyboard driver is implemented in userspace. A PS/2 keyboard driver requires two features which are reserved to ring 0: I/O ports and IRQ handling. To simplify the interface to the essentials for this use-case, pressing or releasing a key causes IRQ 1 to be fired on the PIC, then reading from port 0x60 provides a scancode. We already had support for working with I/O ports in userspace, so the blocker here was IRQ handling.

Helios implements IRQs similarly to seL4, by using a “notification” object (an IPC primitive) which is signalled by the kernel when an IRQ occurs. I was pleased to have this particular blocker, as developing out our IPC implementation further was a welcome task. The essential usage of a notification involves two operations: wait and signal. The former blocks until the notification is signalled, and the later signals the notification and unblocks any tasks which are waiting on it. Unlike sending messages to endpoints, signal never blocks.

After putting these pieces together, I was able to write a simple PS/2 keyboard driver which echos pressed keys to the kernel console:

const irq1_notify = helios::newnotification()?;
const irq1 = helios::irqcontrol_issue(rt::INIT_CAP_IRQCONTROL, irq1_notify, 1)?;
const ps2 = helios::iocontrol_issue(rt::INIT_CAP_IOCONTROL, 0x60, 0x64)?;

for (true) {
	helios::wait(irq1_notify)?;
	const scancode = helios::ioport_in8(ps2, 0x60)?;
	helios::irq_ack(irq1)!;
};

This creates a notification capability to wait on IRQs, then creates a capability for IRQ 1 registered for that notification. It also issues an I/O port capability for the PS/2 ports, 0x60-0x64 (inclusive). Then it loops, waiting until an interrupt occurs, reading the scancode from the port, and printing it. Simple!

I now turned my attention to a C library for Doom. The first step for writing userspace programs in C for a new operating system is to produce a suitable C cross-compiler toolchain. I adapted the instructions from this OSdev wiki tutorial for my needs and produced the working patches for binutils and gcc. I started on a simple C library that included some assembly glue for syscalls, an entry point, and a couple of syscall wrappers. With great anticipation, I wrote the following C program and loaded it into Helios:

#include <helios/syscall.h>
#include <string.h>

int main() {
	const char *message = "Hello from userspace in C!\n";
	sys_writecons(message, strlen(message));
	return 0;
}
$ qemu-system-x86_64 -m 1G -no-reboot -no-shutdown \
	-drive file=boot.iso,format=raw \
	-display none \
	-chardev stdio,id=char0 \
	-serial chardev:char0
Booting Helios kernel
Hello from userspace in C!

Woohoo! After a little bit more work setting up the basics, I started rigging doomgeneric (a Doom fork designed to be easy to port) up to my cross environment and seeing what would break.

As it turned out, a lot of stuff would break. doomgeneric is designed to be portable, but it actually depends on a lot of stuff to be available from the C environment: stdio, libmath, string.h stuff, etc. Not too much, but more than I cared to write from scratch. So, I started pulling in large swaths of musl libc, trimming out as much as I could, and wriggling it into a buildable state. I also wrote a lot of shims to fake out having a real Unix system to run it in, like this code for defining stdout & stderr to just write to the kernel console:

static size_t writecons(FILE *f, const unsigned char *buf, size_t size) {
	sys_writecons(f->wbase, f->wpos - f->wbase);
	sys_writecons(buf, size);
	f->wend = f->buf + f->buf_size;
	f->wpos = f->wbase = f->buf;
	return size;
}

#undef stdout
static unsigned char stdoutbuf[BUFSIZ+UNGET];
hidden FILE __stdout_FILE = {
	.buf = stdoutbuf+UNGET,
	.buf_size = sizeof stdoutbuf-UNGET,
	.fd = 1,
	.flags = F_PERM | F_NORD,
	.lbf = '\n',
	.write = &writecons,
	.seek = NULL,
	.close = NULL,
	.lock = -1,
};
FILE *const stdout = &__stdout_FILE;
FILE *volatile __stdout_used = &__stdout_FILE;

#undef stderr
static unsigned char stderrbuf[UNGET];
hidden FILE __stderr_FILE = {
	.buf = stderrbuf+UNGET,
	.buf_size = 0,
	.fd = 2,
	.flags = F_PERM | F_NORD,
	.lbf = -1,
	.write = &writecons,
	.seek = NULL,
	.close = NULL,
	.lock = -1,
};
FILE *const stderr = &__stderr_FILE;
FILE *volatile __stderr_used = &__stderr_FILE;

The result of all of this hacking and slashing is quite a mess, and none of this is likely to be useful in the long term. I did this work over the course of a couple of afternoons just to get everything “working” enough to support Doom, but an actual useful C programming environment for Helios is likely some ways off. Much of the near-term work will be in Mercury, which will be a Hare environment for writing drivers, and we won’t see a serious look at better C support until we get to Luna, the POSIX compatibility layer a few milestones away.

Anyway, in addition to pulling in lots of musl libc, I had to write some original code to create C implementations of the userspace end for working with Helios kernel services. Some of this is pretty straightforward, such as the equivalent of the helios::ioport_issue code from the keyboard driver you saw earlier:

cap_t
iocontrol_issue(cap_t ctrl, uint16_t min, uint16_t max)
{
	uint64_t tag = mktag(IO_ISSUE, 1, 1);
	cap_t cap = capalloc();
	ipc_buffer->caddr = cap;
	struct sysret ret = sys_send(ctrl, tag, min, max, 0);
	assert(ret.status == 0);
	return cap;
}

A more complex example is the code which maps a page of physical memory into the current process’s virtual address space. In Helios, similar to L4, userspace must allocate its own page tables. However, these page tables are semantically owned by userspace, but they’re not actually reachable by userspace — the page tables themselves are not mapped into their address space (for obvious reasons, I hope). A consequence of this is that the user cannot examine the page tables to determine which, if any, intermediate page tables have to be allocated in order to perform a desired memory mapping. The solution is to try the mapping anyway, and if the page tables are missing, the kernel will reply telling you which table it needs to complete the mapping request. You allocate the appropriate table and try again.

Some of this workload falls on userspace. I had already done this part in Hare, but I had to revisit it in C:

struct sysret
page_map(cap_t page, cap_t vspace, uintptr_t vaddr)
{
	uint64_t tag = mktag(PAGE_MAP, 1, 1);
	ipc_buffer->caps[0] = vspace;
	return sys_send(page, tag, (uint64_t)vaddr, 0, 0);
}

static void
map_table(uintptr_t vaddr, enum pt_type kind)
{
	int r;
	cap_t table;
	switch (kind) {
	case PT_PDPT:
		r = retype(&table, CT_PDPT);
		break;
	case PT_PD:
		r = retype(&table, CT_PD);
		break;
	case PT_PT:
		r = retype(&table, CT_PT);
		break;
	default:
		assert(0);
	}
	assert(r == 0);

	struct sysret ret = page_map(table, INIT_CAP_VSPACE, vaddr);
	if (ret.status == -MISSING_TABLES) {
		map_table(vaddr, ret.value);
		map_table(vaddr, kind);
	}
}

void *
map(cap_t page, uintptr_t vaddr)
{
	while (1) {
		struct sysret ret = page_map(page, INIT_CAP_VSPACE, vaddr);
		if (ret.status == -MISSING_TABLES) {
			map_table(vaddr, ret.value);
		} else {
			assert(ret.status == 0);
			break;
		}
	}
	return (void *)vaddr;
}

Based on this work, I was able to implement a very stupid malloc, which rounds all allocations up to 4096 and never frees them. Hey! It works, okay?

uintptr_t base = 0x8000000000;

static cap_t
page_alloc()
{
	cap_t page;
	int r = retype(&page, CT_PAGE);
	assert(r == 0);
	return page;
}

void *
malloc(size_t n)
{
	if (n % 4096 != 0) {
		n += 4096 - (n % 4096);
	}
	uintptr_t ret = base;
	while (n != 0) {
		cap_t page = page_alloc();
		map(page, base);
		base += 4096;
		n -= 4096;
	}
	return (void *)ret;
}

There is also devmap, which you can read in your own time, which is used for mapping device memory into your address space. This is neccessary to map the framebuffer. It’s more complex because it has to allocate a specific physical page address into userspace, rather than whatever page happens to be free.

So, to revisit our progress, we have:

✓ A working C programming environment

✓ Dynamic memory allocation

✓ A place to draw the screen (a framebuffer)

✓ Keyboard input

It’s time for Doom, baby. Doomgeneric expects the porter to implement the following functions:

  • DG_Init
  • DG_DrawFrame
  • DG_GetKey
  • DG_SetWindowTitle
  • DG_SleepMs
  • DG_GetTicksMs

Easy peasy. Uh, except for that last one. I forgot that our requirements list should have included a means of sleeping for a specific period of time. Hopefully that won’t be a problem later.

I started with DG_Init, allocating the pieces that we’ll need and stashing the important bits in some globals.

int fb_width, fb_height, fb_pitch;
uint8_t *fb;
cap_t irq1_notify;
cap_t irq1;
cap_t ps2;

void DG_Init()
{
	uintptr_t vbeaddr = bootinfo->arch->vbe_mode_info;
	uintptr_t vbepage = vbeaddr / 4096 * 4096;
	struct vbe_mode_info *vbe = devmap(vbepage, 1) + (vbeaddr % 4096);
	fb_width = vbe->width;
	fb_height = vbe->height;
	fb_pitch = vbe->pitch;
	assert(vbe->bpp == 32);
	unsigned int npage = (vbe->pitch * vbe->height) / 4096;
	fb = devmap((uintptr_t)vbe->framebuffer, npage);

	irq1_notify = mknotification();
	irq1 = irqcontrol_issue(INIT_CAP_IRQCONTROL, irq1_notify, 1);
	ps2 = iocontrol_issue(INIT_CAP_IOCONTROL, 0x60, 0x64);
}

If the multiboot loader is configured to set up a framebuffer, it gets handed off to the kernel, and Helios provides it to userspace as mappable device memory, so that saves us from doing all of the annoying VBE crap (or heaven forbid, write an actual video driver). This lets us map the framebuffer into our process. Second, we do the same notification+IRQ+IOControl thing we did from the keyboard driver you saw earlier, except in C, so that we can process scancodes later.

Next is DG_DrawFrame, which is pretty straightforward. We just copy scanlines from the internal buffer to the framebuffer whenever it asks us to.

void DG_DrawFrame()
{
  for (int i = 0; i < DOOMGENERIC_RESY; ++i) {
    memcpy(fb + i * fb_pitch, DG_ScreenBuffer + i * DOOMGENERIC_RESX, DOOMGENERIC_RESX * 4);
  }
}

Then we have DG_GetKey, similar to our earlier keyboard driver, plus actually interpeting the scancodes we get, plus making use of a new non-blocking wait syscall I added to Helios:

int DG_GetKey(int *pressed, unsigned char *doomKey)
{
	struct sysret ret = sys_nbwait(irq1_notify);
	if (ret.status != 0) {
		return 0;
	}

	uint8_t scancode = ioport_in8(ps2, 0x60);
	irq_ack(irq1);

	uint8_t mask = (1 << 7);
	*pressed = (scancode & mask) == 0;
	scancode = scancode & ~mask;
	switch (scancode) {
	case K_AD05:
		*doomKey = KEY_ENTER;
		break;
	case K_AE08:
		*doomKey = KEY_UPARROW;
		break;
	case K_AD07:
		*doomKey = KEY_LEFTARROW;
		break;
	case K_AD08:
		*doomKey = KEY_DOWNARROW;
		break;
	case K_AD09:
		*doomKey = KEY_RIGHTARROW;
		break;
	case K_AB03:
		*doomKey = KEY_FIRE;
		break;
	case K_AB06:
		*doomKey = KEY_USE;
		break;
	case 1:
		*doomKey = KEY_ESCAPE;
		break;
	}

	return *doomKey;
}

Then, uh, we have a problem. Here’s what I ended up doing for DG_SleepMs:

uint32_t ticks = 0;

void DG_SleepMs(uint32_t ms)
{
	// TODO: sleep properly
	int64_t _ms = ms;
	while (_ms > 0) {
		sys_yield();
		ticks += 5;
		_ms -= 5;
	}
}

uint32_t DG_GetTicksMs()
{
	return ticks;
}

Some fellow on IRC said he’d implement a sleep syscall for Helios, but didn’t have time before I was ready to carry on with this port. So instead of trampling on his feet, I just yielded the thread (which immediately returns to the caller, since there are no other threads at this point) and pretend it took 5ms to do so, hoping for the best. It does not work! This port plays at wildly different speeds depending on the performance of the hardware you run it on.

I’m not too torn up about it, though. My goal was not to make a particularly nice or fully featured port of Doom. The speed is problematic, I hardcoded the shareware doom1.wad as the only supported level, you can’t save the game, and it crashes when you try to pick up the shotgun. But it does its job: it demonstrates the maturity of the kernel’s features thus far and provides good feedback on the API design and real-world utility.

If you’d like to try it, you can download a bootable ISO.

You can run it on qemu like so:

$ qemu-system-x86_64 -m 1G -no-reboot -no-shutdown \
		-drive file=doom.iso,format=raw \
		-display sdl \
		-chardev stdio,id=char0 \
		-serial chardev:char0

Enter to start, WASD to move, right shift to fire, space to open doors. It might work on real hardware, but the framebuffer stuff is pretty hacky and not guaranteed to work on most stuff, and the PS/2 keyboard driver will only work with a USB keyboard if you have legacy USB emulation configured in your BIOS, and even then it might not work well. YMMV. It works on my ThinkPad X230. Have fun!

GitHub Copilot and open source laundering

23 June 2022 at 00:00

Disclaimer: I am the founder of a company which competes with GitHub. I am also a long-time advocate for and developer of free and open source software, with a broad understanding of free and open source software licensing and philosophy. I will not name my company in this post to reduce the scope of my conflict of interest.

We have seen an explosion in machine learning in the past decade, alongside an explosion in the popularity of free software. At the same time as FOSS has come to dominate software and found its place in almost all new software products, machine learning has increased dramatically in sophistication, facilitating more natural interactions between humans and computers. However, despite their parallel rise in computing, these two domains remain philosophically distant.

Though some audaciously-named companies might suggest otherwise, the machine learning space has enjoyed almost none of the freedoms forwarded by the free and open source software movement. Much of the actual code related to machine learning is publicly available, and there are many public access research papers available for anyone to read. However, the key to machine learning is access to a high-quality dataset and heaps of computing power to process that data, and these two resources are still kept under lock and key by almost all participants in the space.1

The essential barrier to entry for machine learning projects is overcoming these two problems, which are often very costly to secure. A high-quality, well tagged data set generally requires thousands of hours of labor to produce,2 a task which can potentially cost millions of dollars. Any approach which lowers this figure is thus very desirable, even if the cost is making ethical compromises. With Amazon, it takes the form of gig economy exploitation. With GitHub, it takes the form of disregarding the terms of free software licenses. In the process, they built a tool which facilitates the large-scale laundering of free software into non-free software by their customers, who GitHub offers plausible deniability through an inscrutable algorithm.

Free software is not an unqualified gift. There are terms for its use and re-use. Even so-called “liberal” software licenses impose requirements on re-use, such as attribution. To quote the MIT license:

Permission is hereby granted […] subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

Or the equally “liberal” BSD license:

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

On the other end of the spectrum, copyleft licenses such as GNU General Public License or Mozilla Public License go further, demanding not only attribution for derivative works, but that such derived works are also released with the same license. Quoting GPL:

You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions:

[…]

You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy.

And MPL:

All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form.

Free software licenses impose obligations on the user through terms governing attribution, sublicensing, distribution, patents, trademarks, and relationships with laws like the Digital Millennium Copyright Act. The free software community is no stranger to the difficulties in enforcing compliance with these obligations, which some groups view as too onerous. But as onerous as one may view these obligations to be, one is nevertheless required to comply with them. If you believe that the force of copyright should protect your proprietary software, then you must agree that it equally protects open source works, despite the inconvenience or cost associated with this truth.

GitHub’s Copilot is trained on software governed by these terms, and it fails to uphold them, and enables customers to accidentally fail to uphold these terms themselves. Some argue about the risks of a “copyleft surprise”, wherein someone incorporates a GPL licensed work into their product and is surprised to find that they are obligated to release their product under the terms of the GPL as well. Copilot institutionalizes this risk and any user who wishes to use it to develop non-free software would be well-advised not to do so, else they may find themselves legally liable to uphold these terms, perhaps ultimately being required to release their works under the terms of a license which is undesirable for their goals.

Essentially, the argument comes down to whether or not the model constitutes a derivative work of its inputs. Microsoft argues that it does not. However, these licenses are not specific regarding the means of derivation; the classic approach of copying and pasting from one project to another need not be the only means for these terms to apply. The model exists as the result of applying an algorithm to these inputs, and thus the model itself is a derivative work of its inputs. The model, then used to create new programs, forwards its obligations to those works.

All of this assumes the best interpretation of Microsoft’s argument, with a heavy reliance on the fact that the model becomes a general purpose programmer, having meaningfully learned from its inputs and applying this knowledge to produce original work. Should a human programmer take the same approach, studying free software and applying those lessons, but not the code itself, to original projects, I would agree that their applied knowledge is not creating derivative works. However, that is not how machine learning works. Machine learning is essentially a glorified pattern recognition and reproduction engine, and does not represent a genuine generalization of the learning process. It is perhaps capable of a limited amount of originality, but is also capable of degrading to the simple case of copy and paste. Here is an example of Copilot reproducing, verbatim, a function which is governed by the GPL, and would thus be governed by its terms:

Source: Armin Ronacher via Twitter

The license reproduced by Copilot is not correct, neither in form nor function. This code was not written by V. Petkov and the GPL imposes much stronger obligations than those suggested by the comment. This small example was deliberately provoked with a suggestive prompt (this famous function is known as the “fast inverse square root”) and the “float Q_”, but it’s not a stretch to assume someone can accidentally do something similar with any particularly unlucky English-language description of their goal.

Of course, the use of a suggestive prompt to convince Copilot to print GPL licensed code suggests another use: deliberately laundering FOSS source code. If Microsoft’s argument holds, then indeed the only thing which is necessary to legally circumvent a free software license is to teach a machine learning algorithm to regurgitate a function you want to use.

This is a problem. I have two suggestions to offer to two audiences: one for GitHub, and another for free software developers who are worried about Copilot.

To GitHub: this is your Oracle v Google moment. You’ve invested in building a platform on top of which the open source revolution was built, and leveraging this platform for this move is a deep betrayal of the community’s trust. The law applies to you, and banking on the fact that the decentralized open source community will not be able to mount an effective legal challenge to your $7.5B Microsoft war chest does not change this. The open source community is astonished, and the astonishment is slowly but surely boiling over into rage as our concerns fall on deaf ears and you push forward with the Copilot release. I expect that if the situation does not change, you will find a group motivated enough to challenge this. The legitimacy of the free software ecosystem may rest on this problem, and there are many companies who are financially incentivized to see to it that this legitimacy stands. I am certainly prepared to join a class action lawsuit as a maintainer, or alongside other companies with interests in free software making use of our financial resources to facilitate a lawsuit.

The tool can be improved, probably still in time to avoid the most harmful effects (harmful to your business, that is) of Copilot. I offer the following specific suggestions:

  1. Allow GitHub users and repositories to opt-out of being incorporated into the model. Better, allow them to opt-in. Do not tie this flag into unrelated projects like Software Heritage and the Internet Archive.
  2. Track the software licenses which are incorporated into the model and inform users of their obligations with respect to those licenses.
  3. Remove copyleft code from the model entirely, unless you want to make the model and its support code free software as well.
  4. Consider compensating the copyright owners of free software projects incorporated into the model with a margin from the Copilot usage fees, in exchange for a license permitting this use.

Your current model probably needs to be thrown out. The GPL code incorporated into it entitles anyone who uses it to receive a GPL’d copy of the model for their own use. It entitles these people to commercial use, to build a competing product with it. But, it presumably also includes works under incompatible licenses, such as the CDDL, which is… problematic. The whole thing is a legal mess.

I cannot speak for the rest of the community that have been hurt by this project, but for my part, I would be okay with not pursuing the answers to any of these questions with you in court if you agreed to resolve these problems now.

And, my advice to free software maintainers who are pissed that their licenses are being ignored. First, don’t use GitHub and your code will not make it into the model (for now). I’ve written before about why it’s generally important for free software projects to use free software infrastructure, and this only re-enforces that fact. Furthermore, the old “vote with your wallet” approach is a good way to show your disfavor. That said, if it occurs to you that you don’t actually pay for GitHub, then you may want to take a moment to consider if the incentives created by that relationship explain this development and may lead to more unfavorable outcomes for you in the future.

You may also be tempted to solve this problem by changing your software licenses to prohibit this behavior. I’ll say upfront that according to Microsoft’s interpretation of the situation (invoking fair use), it doesn’t matter to them which license you use: they’ll use your code regardless. In fact, some proprietary code was found to have been incorporated into the model. However, I still support your efforts to address this in your software licenses, as it provides an even stronger legal foundation upon which we can reject Copilot.

I will caution you that the way you approach that clause of your license is important. Whenever writing or changing a free and open source software license, you should consider whether or not it will still qualify as free or open source after your changes. To be specific, a clause which outright forbids the use of your code for training a machine learning model will make your software non-free, and I do not recommend this approach. Instead, I would update your licenses to clarify that incorporating the code into a machine learning model is considered a form of derived work, and that your license terms apply to the model and any works produced with that model.

To summarize, I think that GitHub Copilot is a bad idea as designed. It represents a flagrant disregard of FOSS licensing in of itself, and it enables similar disregard — deliberate or otherwise — among its users. I hope they will heed my suggestions, and I hope that my words to the free software community offer some concrete ways to move forward with this problem.


  1. Shout-out to Mozilla Common Voice, one of the few exceptions to this rule, which is an excellent project that has produced a high-quality, freely available dataset of voice samples, and used it to develop free models and software for text-to-speech and speech recognition. ↩︎

  2. Typically exploitative labor from low-development countries which the tech industry often pretends isn’t a hair’s breadth away from slavery. ↩︎

Introducing the Himitsu keyring & password manager for Unix

20 June 2022 at 00:00

Himitsu is a new approach to storing secret information on Unix systems, such as passwords or private keys, and I released version 0.1 this morning. It’s available on Alpine Linux community and the Arch User Repository, with more distributions hopefully on the way soon.

So, what is Himitsu and what makes it special? The following video introduces the essential concepts and gives you an idea of what’s possible:

If you prefer reading to watching, this blog post includes everything that’s in the video.

What is Himitsu?

Himitsu draws inspiration from Plan 9’s factotum, but polished up and redesigned for Unix. At its core, Himitsu is a key/value store and a simple protocol for interacting with it. For example, a web login could be stored like so:

proto=web host=example.org user=jdoe password!=hunter2

Himitsu has no built-in knowledge of web logins, it just stores arbitrary keys and values. The bang (!) indicates that the password is a “secret” value, and the “proto” key defines additional conventions for each kind of secret. For proto=web, each key/value pair represents a form field on a HTML login form.

We can query the key store using the “hiq” command. For instance, we can obtain the example key above by querying for any key with “proto=web”, any “host”, “user”, and “password” value, and an optional “comment” value:

$ hiq proto=web host user password! comment?
proto=web host=example.org user=jdoe password!

You’ll notice that the password is hidden here. In order to obtain it, we must ask for the user’s consent.

$ hiq -d proto=web host user password! comment?

A screenshot of a GTK+ dialog confirming the operation

proto=web host=example.org user=jdoe password!=hunter2

You can also use hiq to add or delete keys, or incorporate it into a shell pipeline:

$ hiq -dFpassword host=example.org
hunter2

A simple, extensible protocol

The protocol is a simple line-oriented text protocol, which is documented in the himitsu-ipc(5) manual page. We can also use it via netcat:

$ nc -U $XDG_RUNTIME_DIR/himitsu
query host=example.org
key proto=web host=example.org user=jdoe password!
end
query -d host=example.org
key proto=web host=example.org user=jdoe password!=hunter2
end

The consent prompter also uses a standardized protocol, documented by himitsu-prompter(5). Based on this, you can implement new prompters for Qt, or the TTY, or any other technology appropriate to your system, or implement a more novel approach, such as sending a push notification to your phone to facilitate consent.

Additional frontends

Based on these protocols, a number of additional integrations are possible. Martijn Braam has written a nice GTK+ frontend called keyring:

A screenshot of the GTK+ frontend

There’s also a Firefox add-on which auto-fills forms for keys with proto=web:

Screenshot of himitsu-firefox

We also have a package called himitsu-ssh which provides an SSH agent:

$ hissh-import < ~/.ssh/id_ed25519
Enter SSH key passphrase: 
key proto=ssh type=ssh-ed25519 pkey=pF7SljE25sVLdWvInO4gfqpJbbjxI6j+tIUcNWzVTHU= skey! comment=sircmpwn@homura
$ ssh-add -l
256 SHA256:kPr5ZKTNE54TRHGSaanhcQYiJ56zSgcpKeLZw4/myEI sircmpwn@homura (ED25519)
$ ssh git@git.sr.ht
Hi sircmpwn! You've successfully authenticated, but I do not provide an interactive shell. Bye!
Connection to git.sr.ht closed.

I hope to see an ecosystem of tools built around Himitsu to grow. New frontends like keyring would be great, and new integrations like GPG agents would also be nice to see.

Zero configuration

Himitsu-aware software can discover your credentials and connection details without any additional configuration. For example, a mail client might look for proto=imap and proto=smtp and discover something like this:

proto=imap host=imap.migadu.com user=sir@cmpwn.com password! port=993 enc=tls
proto=smtp host=imap.migadu.com user=sir@cmpwn.com password! port=465 enc=tls

After a quick consent prompt, the software can load your IMAP and SMTP configuration and get connected without any manual steps. With an agent like himitsu-ssh, it could even connect without actually handling your credentials directly — a use-case we want to support with improvements to the prompter UI (to distinguish between a case where an application will view versus use your credentials).

The cryptography

Your key store is located at $XDG_DATA_HOME/himitsu/. The key is derived by mixing your password with argon2, and the resulting key is used for AEAD with XChaCha20+Poly1305. The “index” file contains a list of base64-encoded encrypted blobs, one per line, enumerating the keys in the key store.1 Secret keys are encrypted and stored separately in files in this directory. If you like the pass approach to storing your keys in git, you can easily commit this directory to a git repository, or haul it along to each of your devices with whatever other means is convenient to you.

Himitsu is written in Hare and uses cryptography primitives available from its standard library. Note that these have not been audited.

Future plans

I’d like to expand on Himitsu in the future. One idea is to store your full disk encryption password in Himitsu and stick a subset of your key store into the initramfs, which you unlock during early boot, pull FDE keys out of, and then pre-authorize the keyring for your desktop session - which you’re logged in to automatically on the basis that you were pre-authorized during boot.

We also want to add key sharing and synchronization tools. The protocol could easily be moved to TCP and authorized with your existing key store key (we could make an ed25519 key out of it, or generate and store one separately), so setting up key synchronization might be as simple as:

$ hiq -a proto=sync host=himitsu.sr.ht

You could also use Himitsu for service discovery — imagine a key ring running on your datacenter LAN with entries for your Postgres database, SMTP credentials, and so on.

There are some other ideas that we could use your help with:

  • himitsu-firefox improvements (web devs welcome!)
  • Chromium support (web devs welcome!)
  • Himitsu apps for phones (mobile devs welcome!)
  • More key management frontends (maybe a TUI?)
  • More security options — smart cards? U2F?
  • hare-ssh improvements (e.g. RSA keys)
  • PGP support
  • Anything else you can think of

Please join us! We hang out on IRC in #himitsu on Libera Chat. Give Himitsu a shot and let us know what you think.

Alright, back to kernel hacking. I got multi-tasking working yesterday!


  1. This offers an improvement over pass, for example, by not storing the list of entries in plain text. ↩︎

Status update, June 2022

15 June 2022 at 00:00

Hello again! I would like to open this post by acknowledging the response to my earlier post, “bleh”. Since it was published, I have received several hundred emails expressing support and kindness. I initially tried to provide these with thoughtful replies, then shorter replies, then I had to stop replying at all, but I did read every one. Thank you, everyone, for sending these. I appreciate it very much, and it means a lot to me.

I have actually had a lot more fun programming this month than usual, since I decided to spend more time on experimental and interesting projects and less time on routine maintenance or long-term developments. So, the feature you’ve been waiting for in SourceHut might be delayed, but in return, there’s cool progress on the projects that you didn’t even know you were waiting for. Of course, the SourceHut workload never dips below a dull roar, as I have to attend to business matters and customer support promptly, and keep a handle on the patch queue, and the other SourceHut staff and contributors are always hard at work — so there’ll be plenty to discuss in the “what’s cooking” later.

The bulk of my focus has been on the Helios kernel this month, a project I introduced a couple of days ago. I spent a lot of time furiously refactoring, reworking the existing kernel code for evalutaing features like page allocation and virtual address space management into capability-oriented kernel services that can be provided to userspace, then overhauling our startup code to provision a useful set of capabilities for the init process to take advantage of. I also implemented x86_64 I/O port services, which allowed for the first few drivers to be written in userspace — serial ports and simple VBE graphics. We also got interrupts working properly and brought up the PIT, which is another major step towards multi-tasking. I also implemented a new syscall ABI with error handling, and refactored a lot of the arch-specific code to make new ports easier. The kernel is in a much better state now than it was a month ago (and to think it’s only three months old!).

A picture of Helios drawing to a framebuffer

There was also a lot of progress on Himitsu, which I plan on presenting in a video and blog post in a few days time. The Firefox add-on actually works now (though some features remain to be done), and Alexey Yerin fixed several important bugs and contributed several new features. The user is now prompted to consent before deleting keys, and we have a new GTK+ prompter written in Python, which is much more reliable and feature-full thanks to Martijn Braam’s help (rewriting it in C again is a long-term TODO item for any interested contributor). I also made some progress towards what will ultimately become full-disk encryption support.

himitsu-keyring, a new GTK+ keyring manager from Martijn Braam

Hare also enjoyed many improvements this month. We have some new improvements to date/time support, including fixes for Martian time ;) I also mostly implemented cross-compiling, which you can try out with hare build -t riscv64 or something similar. The major outstanding pain point here is that the Hare cache is not arch-aware, so you need to rm -rf ~/.cache/hare each time you switch architectures for now. We now have complex number support, as well as improvements to encoding::json and net::uri.

A screenshot of a fractal rendered with the aid of Hare’s new complex number
support

That’s all for today. Until next time!

The Helios microkernel

13 June 2022 at 00:00

I’ve been working on a cool project lately that I’d like to introduce you to: the Helios microkernel. Helios is written in Hare and currently targets x86_64, and riscv64 and aarch64 are on the way. It’s very much a work-in-progress: don’t expect to pick this up and start building anything with it today.

A picture of a ThinkPad running Helios, demonstrating userspace memory allocation

Drawing some inspiration from seL4, Helios uses a capability-based design for isolation and security. The kernel offers primitives for allocating physical pages, mapping them into address spaces, and managing tasks, plus features like platform-specific I/O (e.g. reading and writing x86 ports). The entire system is written in Hare, plus some necessary assembly for the platform bits (e.g. configuring the GDT or IDT).

Things are still quite early, but I’m pretty excited about this project. I haven’t had this much fun hacking in some time :) We have several kernel services working, including memory management and virtual address spaces, and I’ve written a couple of simple drivers in userspace (serial and BIOS VGA consoles). Next up is preemptive multi-tasking — we already have interrupts working reliably, including the PIT, so all that’s left for multi-tasking is to actually implement the context switch. I’d like to aim for an seL4-style single-stack system, though some finageling will be required to make that work.

Again, much of the design comes from seL4, but unlike seL4, we intend to build upon this kernel and develop a userspace as well. Each of the planned components is named after celestial bodies, getting further from the sun as they get higher-level:

  • Helios: the kernel
  • Mercury: low-level userspace services & service bus
  • Venus: real-world driver collection
  • Gaia: high-level programming environment
  • Ares: a complete operating system; package management, GUI, etc

A few other components are planned — “Vulcan” is the userspace kernel testing framework, named for the (now disproved) hypothetical planet between Mercury and the Sun, and “Luna” is the planned POSIX compatibility layer. One of the goals is to be practical for use on real-world hardware. I’ve been testing it continuously on my ThinkPads to ensure real-world hardware support, and I plan on writing drivers for its devices — Intel HD graphics, HD Audio, and Intel Gigabit Ethernet at the least. A basic AMD graphics driver is also likely to appear, and perhaps drivers for some SoC’s, like Raspberry Pi’s VideoCore. I have some neat ideas for the higher-level components as well, but I’ll save those for later.

Why build a new operating system? Well, for a start, it’s really fun. But I also take most of my projects pretty seriously and aim for real-world usability, though it remains to be seen if this will be achieved. This is a hugely ambitious project, or, in other words, my favorite kind of project. Even if it’s not ultimately useful, it will drive the development of a lot of useful stuff. We’re planning to design a debugger that will be ported to Linux as well, and we’ll be developing DWARF support for Hare to facilitate this. The GUI toolkit we want to build for Ares will also be generally applicable. And Helios and Mercury together have a reasonably small scope and makes for an interesting and useful platform in their own right, even if the rest of the stack never completely materializes. If nothing else, it will probably be able to run DOOM fairly soon.

The kernel is a microkernel, so it is fairly narrow in scope and will probably be more-or-less complete in the foreseeable future. The next to-do items are context switching, so we can set up multi-tasking, IPC, fault handling, and userspace support for interrupts. We’ll also need to parse the ACPI tables and bring up PCI in the kernel before handing it off to userspace. Once these things are in place, the kernel is essentially ready to be used to write most drivers, and the focus will move to fleshing out Mercury and Venus, followed by a small version of Gaia that can at least support an interactive shell. There are some longer-term features which will be nice to have in the kernel at some point, though, such as SMP, IOMMU, or VT-x support.

Feel free to pull down the code and check it out, though remember my warning that it doesn’t do too much yet. You can download the latest ISO from the CI, if you want to reproduce the picture at the top of this post, and write it to a flash drive to stick in the x86_64 computer of your choice (boot via legacy BIOS). If you want to mess with the code, you could play around with the Vulcan system to get simple programs running in userspace. The kernel serial driver is write-only, but a serial driver written in userspace could easily be made to support interactive programs. If you’re feeling extra adventureous, it probably wouldn’t be too difficult to get a framebuffer online and draw some pixels — ping me in #helios on Libera Chat for a few words of guidance if you want to try it.

bleh

30 May 2022 at 00:00

A few weeks ago, the maintainer of a project on SourceHut stepped down from their work, citing harassment over using SourceHut as their platform of choice. It was a difficult day when I heard about that.

Over the past few weeks, I have been enduring a bit of a depressive episode. It’s a complex issue rooted in several different problems, but I think a major source of it is the seemingly constant deluge of hate I find myself at the receiving end of online. I had to grow a thick skin a long time ago, but lately it has not been thick enough. I am finding it increasingly difficult to keep up with my work.

Perhaps it this has something to do with the backlash, not just against me and my work, but against others who use and participate in that work. It’s not enough to dislike my programming language, but the skeptics must publicly denounce it and discourage others from using it. It’s irresponsible, if not immoral, to design a language without a borrow checker in 2022. SourceHut’s email-oriented approach might not be for everyone, and instead of simply not using it, skeptics must harass any projects that do. This kind of harassment is something I hear about often from many maintainers of projects on SourceHut. It breaks my heart and I feel helpless to do anything about it.

I’m also often dealing with harassment directed at me alone. When I complained this week about being DDoSed by a company with over a billion dollars in annual revenue, it was portrayed as righteous retribution and a sign of incompetence. I can’t even count the number of times someone has said they would refuse to use SourceHut (and that you, too, dear reader, should avoid it) on the sole basis that I’m involved with it. There is a steady supply of vile comments about me based on “facts” delivered from the end of a game of telephone in which every participant hates my guts, all easily believable without further research because I’m such a villainous character. Every project I work on, every blog post I write, even many of the benign emails to public lists or GitHub issues I open — the response is just vitriol.

I have made no shortage of mistakes, and there are plenty of hurt feelings which can be laid at my feet. I am regretful for my mistakes, and I have worked actively to improve. I think that it has been working. Perhaps that’s arrogant of me to presume, but I’m not sure what else to do. Must I resign myself to my fate for stupid comments I made years ago? I’m sorry, and I’ve been working to do better. Can I have another chance?

For some I think the answer is “no”. Many of my detractors just want me to shut up. No more blog posts, no new projects. Just go away, Drew.

Well, I can’t say it’s not working. This stuff gets to me. At times like this I have very little motivation to work. If you’re looking for a strategy to get me to shut up, just ensure that I have a constant flow of toxic comments to read.

I love writing code, at least most of the time. I believe in my principles and I enjoy writing software that embodies them. I love doing it, and I’m really good at it, and thousands of people are depending on my work.

I’m doing the work that I believe in, and working with people who share those values. I have worked very hard for that privilege. I’m sorry that it’s not good enough for many people. I’m just trying to do my best. And if you must harass anyone over it, at least harass me, and not anyone else. My inbox is at sir@cmpwn.com, and I promise that I will read your email and cry, so that no one else has to.

I’ll close by thanking those who have sent me positive notes. Some of these comments are very touching. If you’ve sent one of these, you have my thanks. Love you :)

Google has been DDoSing SourceHut for over a year

25 May 2022 at 00:00

Just now, I took a look at the HTTP logs on git.sr.ht. Of the past 100,000 HTTP requests received by git.sr.ht (representing about 2½ hours of logs), 4,774 have been requested by GoModuleProxy — 5% of all traffic. And their requests are not cheap: every one is a complete git clone. They come in bursts, so every few minutes we get a big spike from Go, along with a constant murmur of Go traffic.

This has been ongoing since around the release of Go 1.16, which came with some changes to how Go uses modules. Since this release, following a gradual ramp-up in traffic as the release was rolled out to users, git.sr.ht has had a constant floor of I/O and network load for which the majority can be attributed to Go.

I started to suspect that something strange was going on when our I/O alarms started going off in February 2021 (we eventually had to tune these alarms up above the floor of I/O noise generated by Go), correlated with lots of activity from a Go user agent. I was able to narrow it down with some effort, but to the credit of the Go team they did change their User-Agent to make more apparent what was going on. Ultimately, this proved to be the end of the Go team’s helpfulness in this matter.

I did narrow it down: it turns out that the Go Module Mirror runs some crawlers that periodically clone Git repositories with Go modules in them to check for updates. Once we had narrowed this down, I filed a second ticket to address the problem.

I came to understand that the design of this feature is questionable. For a start, I never really appreciated the fact that Go secretly calls home to Google to fetch modules through a proxy (you can set GOPROXY=direct to fix this). Even taking the utility at face value, however, the implementation leaves much to be desired. The service is distributed across many nodes which all crawl modules independently of one another, resulting in very redundant git traffic.

140 8a42ab2a4b4563222b9d12a1711696af7e06e4c1092a78e6d9f59be7cb1af275
 57 9cc95b73f370133177820982b8b4e635fd208569a60ec07bd4bd798d4252eae7
 44 9e730484bdf97915494b441fdd00648f4198be61976b0569338a4e6261cddd0a
 44 80228634b72777eeeb3bc478c98a26044ec96375c872c47640569b4c8920c62c
 44 5556d6b76c00cfc43882fceac52537b2fdaa7dff314edda7b4434a59e6843422
 40 59a244b3afd28ee18d4ca7c4dd0a8bba4d22d9b2ae7712e02b1ba63785cc16b1
 40 51f50605aee58c0b7568b3b7b3f936917712787f7ea899cc6fda8b36177a40c7
 40 4f454b1baebe27f858e613f3a91dfafcdf73f68e7c9eba0919e51fe7eac5f31b

This is a sample from a larger set which shows the hashes of git repositories on the right (names were hashed for privacy reasons), and the number of times they were cloned over the course of an hour. The main culprit is the fact that the nodes all crawl independently and don’t communicate with each other, but the per-node stats are not great either: each IP address still clones the same repositories 8-10 times per hour. Another user hosting their own git repos noted a single module being downloaded over 500 times in a single day, generating 4 GiB of traffic.

The Go team holds that this service is not a crawler, and thus they do not obey robots.txt — if they did, I could use it to configure a more reasonable “Crawl-Delay” to control the pace of their crawling efforts. I also suggested keeping the repositories stored on-site and only doing a git fetch, rather than a fresh git clone every time, or using shallow clones. They could also just fetch fresh data when users request it, instead of pro-actively crawling the cache all of the time. All of these suggestions fell on deaf ears, the Go team has not prioritized it, and a year later I am still being DDoSed by Google as a matter of course.

I was banned from the Go issue tracker for mysterious reasons,1 so I cannot continue to nag them for a fix. I can’t blackhole their IP addresses, because that would make all Go modules hosted on git.sr.ht stop working for default Go configurations (i.e. without GOPROXY=direct). I tried to advocate for Linux distros to patch out GOPROXY by default, citing privacy reasons, but I was unsuccessful. I have no further recourse but to tolerate having our little-fish service DoS’d by a 1.38 trillion dollar company. But I will say that if I was in their position, and my service was mistakenly sending an excessive amount of traffic to someone else, I would make it my first priority to fix it. But I suppose no one will get promoted for prioritizing that at Google.


  1. In violation of Go’s own Code of Conduct, by the way, which requires that participants are notified moderator actions against them and given the opportunity to appeal. I happen to be well versed in Go’s CoC given that I was banned once before without notice — a ban which was later overturned on the grounds that the moderator was wrong in the first place. Great community, guys. ↩︎

Status update, May 2022

16 May 2022 at 00:00

This was an exciting month: the Hare programming language is a secret no more! You can now try out the programming language I first teased over a year ago and tell me what you think. I hope you like it! I’m quite pleased with it so far.

One thing Hare has done is allow me to unshelve several projects which were blocked pending the availability of a suitable language to write them in. I have actually been working on several of these for a while now — and several more are to come later — but I couldn’t share them thanks to Hare’s policy of secrecy early in its development. Allow me to introduce you to a few projects!

Helios is a micro-kernel for x86_64, and ideally later for aarch64 and riscv64 as well (and possibly other targets as Hare grows additional ports). We have a few things working, such as paging and interrupts, and as of this morning we have entered userspace. Next up is rigging up syscalls and scheduling, then we’re going to start fleshing out an L4-inspired API and writing some drivers in userspace.

A screenshot showing Helios booting and entering userspace

Himitsu is a secret storage system. It can act as a password manager, but it also stores other arbitrary secret data, such as private keys. Each key is a set of key/value pairs, some of which can be secret. This allows you to store additional data alongside your password (such as your username or email for login), and also supports secret data other than passwords — like SSH keys. An extensible consent and agent protocols allow you to expand it to support a wide variety of use-cases for secure use of secrets.

btqd, or “bittorrent queue daemon”, is (going to be) a bittorrent daemon, but it is still very early in development. The design is essentially that of a process supervisor which manages a queue of torrents and fires up subprocesses to seed or leech for a set of active torrents. Each subprocess, such as btlc (bittorrent leech client), or btsc (bittorrent seed client), can also be used separately from the queue daemon. Further development is blocked on net::http, which is blocked on TLS support, for tracker announce requests. I may temporarily unblock this by shelling out to curl instead.

scheduled is also early in development. It is a replacement for crond (and also at(1)) which is redesigned from the ground up. I have never been thrilled with cron’s design — it’s very un-Unix like. scheduled will have better error handling and logging, a much more flexible and understandable approach to configuration, and a better approach to security, plus the ability to do ad-hoc scheduling from the command line. This was designed prior to date/time support landing in Hare, and was blocked for a while, but is now unblocked. However, it is not my highest priority.


Each of these projects will spawn more blog posts (or talks) going into greater depth on their design goals and rationale later on. For now, with the introductions out of the way, allow me to fill you in on the things which got done in this past month in particular.

I’ll keep the SourceHut news short, and expand upon it in the “what’s cooking” post later today. For my own part, I spent some time working on hut to add support for comprehensive account data import/export. This will allow you to easily take all of your data out of sourcehut and import it into another instance, or any compatible software — your git repos are just git repos and your mailing lists are just mbox files, so you could push them to GitHub or import them into GNU Mailman, for example. This work is also a step towards self-service account deletion and renaming, both prioritized for the beta.

Regarding Hare itself, there are many important recent developments. Over 300 commits landed this month, so I’ll have to leave some details out. An OpenBSD port is underway by Brian Callahan, and the initial patches have landed for the Hare compiler. The crypto module grew blowfish and bcrypt support, both useful mainly for legacy compatibility, as well as the more immediately useful x25519 and pem implementations. There is also a new encoding::json module,1 and a number of fixes and improvements have been steadily flowing in for regex, bufio, net, net::uri, and datetime, along with dozens of others.

For Himitsu, I developed hare-ssh this month to facilitate the addition of himitsu-ssh, which provides SSH tooling that integrates with Himitsu (check out the video above for a demo). The “hissh-import” command decodes OpenSSH private keys and loads them into the Himitsu keystore, and the “hissh-agent” command runs an SSH agent that performs authentication with the private keys stored in Himitsu. Future additions will include “hissh-export”, for getting your private keys back out in a useful format, and “hissh-keygen”, for skipping the import/export step entirely. Presently only ed25519 keys are supported; more will be added as the necessary primitives are added to Hare upstream.

I did some work on Helios this weekend, following a brief hiatus. I wrote a more generalized page table implementation which can manage multiple page tables (necessary to have separate address spaces for each process), and started rigging up the kernel to userspace transition, which I briefly covered earlier in the post. As of this morning, I have some code running in userspace — one variant attempts to cli, causing a general protection fault (as expected), and another just runs a busy loop, which works without any faults. Next steps are syscalls and scheduling.

That’s all the news for today. Hare! Woo! Thanks for reading, and be sure to check out — and maybe contribute to? — some of these projects. Take care!


  1. Which is likely to be moved to the extended library in the future. ↩︎

A Hare code generator for finding ioctl numbers

14 May 2022 at 00:00

Modern Unix derivatives have this really bad idea called ioctl. It’s a function which performs arbitrary operations on a file descriptor. It is essentially the kitchen sink of modern Unix derivatives, particularly Linux, in which they act almost like a second set of extra syscalls. For example, to get the size of the terminal window, you use an ioctl specific to TTY file descriptors:

let wsz = rt::winsize { ... };
match (rt::ioctl(fd, rt::TIOCGWINSZ, &wsz: *void)) {
case let e: rt::errno =>
	switch (e: int) {
	case rt::EBADFD =>
		return errors::invalid;
	case rt::ENOTTY =>
		return errors::unsupported;
	case =>
		abort("Unexpected error from ioctl");
	};
case int =>
	return ttysize {
		rows = wsz.ws_row,
		columns = wsz.ws_col,
	};
};

This code performs the ioctl syscall against the provided file descriptor “fd”, using the “TIOCGWINSZ” operation, and setting the parameter to a pointer to a winsize structure. There are thousands of ioctls provided by Linux, and each of them is assigned a constant like TIOCGWINSZ (0x5413). Some constants, including this one, are assigned somewhat arbitrarily. However, some are assigned with some degree of structure.

Consider for instance the ioctl TUNSETOWNER, which is used for tun/tap network devices. This ioctl is assigned the number 0x400454cc, but this is not selected arbitrarily. It’s assigned with a macro, which we can find in /usr/include/linux/if_tun.h:

#define TUNSETOWNER   _IOW('T', 204, int)

The _IOW macro, along with similar ones like _IO, _IOR, and _IOWR, are defined in /usr/include/asm-generic/ioctl.h. They combine this letter, number, and parameter type (or rather its size), and the direction (R, W, WR, or neither), OR’d together into an unsigned 32-bit number:

#define _IOC_WRITE	1U

#define _IOC_TYPECHECK(t) (sizeof(t))

#define _IOC(dir,type,nr,size) \
	(((dir)  << _IOC_DIRSHIFT) | \
	 ((type) << _IOC_TYPESHIFT) | \
	 ((nr)   << _IOC_NRSHIFT) | \
	 ((size) << _IOC_SIZESHIFT))

#define _IOW(type,nr,size)	_IOC(_IOC_WRITE,(type),(nr),(_IOC_TYPECHECK(size)))

It would be useful to define ioctl numbers in a similar fashion for Hare programs. However, Hare lacks macros, so we cannot re-implement this in exactly the same manner. Instead, we can use code generation.

Hare is a new systems programming language I’ve been working on for a couple of years. Check out the announcement for more detail.

Again using the tun interface as an example, our goal is to turn the following input file:

type sock_filter = struct {
	code: u16,
	jt: u8,
	jf: u8,
	k: u32,
};

type sock_fprog = struct {
	length: u16,
	filter: *sock_filter,
};

def TUNSETNOCSUM: u32 = @_IOW('T', 200, int);
def TUNSETDEBUG: u32 = @_IOW('T', 201, int);
def TUNSETIFF: u32 = @_IOW('T', 202, int);
def TUNSETPERSIST: u32 = @_IOW('T', 203, int);
def TUNSETOWNER: u32 = @_IOW('T', 204, int);
def TUNSETLINK: u32 = @_IOW('T', 205, int);
def TUNSETGROUP: u32 = @_IOW('T', 206, int);
def TUNGETFEATURES: u32 = @_IOR('T', 207, uint);
def TUNSETOFFLOAD: u32 = @_IOW('T', 208, uint);
def TUNSETTXFILTER: u32 = @_IOW('T', 209, uint);
def TUNGETIFF: u32 = @_IOR('T', 210, uint);
def TUNGETSNDBUF: u32 = @_IOR('T', 211, int);
def TUNSETSNDBUF: u32 = @_IOW('T', 212, int);
def TUNATTACHFILTER: u32 = @_IOW('T', 213, sock_fprog);
def TUNDETACHFILTER: u32 = @_IOW('T', 214, sock_fprog);
def TUNGETVNETHDRSZ: u32 = @_IOR('T', 215, int);
def TUNSETVNETHDRSZ: u32 = @_IOW('T', 216, int);
def TUNSETQUEUE: u32 = @_IOW('T', 217, int);
def TUNSETIFINDEX: u32 = @_IOW('T', 218, uint);
def TUNGETFILTER: u32 = @_IOR('T', 219, sock_fprog);
def TUNSETVNETLE: u32 = @_IOW('T', 220, int);
def TUNGETVNETLE: u32 = @_IOR('T', 221, int);
def TUNSETVNETBE: u32 = @_IOW('T', 222, int);
def TUNGETVNETBE: u32 = @_IOR('T', 223, int);
def TUNSETSTEERINGEBPF: u32 = @_IOR('T', 224, int);
def TUNSETFILTEREBPF: u32 = @_IOR('T', 225, int);
def TUNSETCARRIER: u32 = @_IOW('T', 226, int);
def TUNGETDEVNETNS: u32 = @_IO('T', 227);

Into the following output file:

type sock_filter = struct {
	code: u16,
	jt: u8,
	jf: u8,
	k: u32,
};

type sock_fprog = struct {
	length: u16,
	filter: *sock_filter,
};

def TUNSETNOCSUM: u32 = 0x400454c8;
def TUNSETDEBUG: u32 = 0x400454c9;
def TUNSETIFF: u32 = 0x400454ca;
def TUNSETPERSIST: u32 = 0x400454cb;
def TUNSETOWNER: u32 = 0x400454cc;
def TUNSETLINK: u32 = 0x400454cd;
def TUNSETGROUP: u32 = 0x400454ce;
def TUNGETFEATURES: u32 = 0x800454cf;
def TUNSETOFFLOAD: u32 = 0x400454d0;
def TUNSETTXFILTER: u32 = 0x400454d1;
def TUNGETIFF: u32 = 0x800454d2;
def TUNGETSNDBUF: u32 = 0x800454d3;
def TUNSETSNDBUF: u32 = 0x400454d4;
def TUNATTACHFILTER: u32 = 0x401054d5;
def TUNDETACHFILTER: u32 = 0x401054d6;
def TUNGETVNETHDRSZ: u32 = 0x800454d7;
def TUNSETVNETHDRSZ: u32 = 0x400454d8;
def TUNSETQUEUE: u32 = 0x400454d9;
def TUNSETIFINDEX: u32 = 0x400454da;
def TUNGETFILTER: u32 = 0x801054db;
def TUNSETVNETLE: u32 = 0x400454dc;
def TUNGETVNETLE: u32 = 0x800454dd;
def TUNSETVNETBE: u32 = 0x400454de;
def TUNGETVNETBE: u32 = 0x800454df;
def TUNSETSTEERINGEBPF: u32 = 0x800454e0;
def TUNSETFILTEREBPF: u32 = 0x800454e1;
def TUNSETCARRIER: u32 = 0x400454e2;
def TUNGETDEVNETNS: u32 = 0x54e3;

I wrote the ioctlgen tool for this purpose, and since it demonstrates a number of interesting Hare features, I thought it would make for a cool blog post. This program must do the following things:

  • Scan through the file looking for @_IO* constructs
  • Parse these @_IO* constructs
  • Determine the size of the type specified by the third parameter
  • Compute the ioctl number based on these inputs
  • Write the computed constant to the output
  • Pass everything else through unmodified

The implementation begins thusly:

let ioctlre: regex::regex = regex::regex { ... };
let typedefre: regex::regex = regex::regex { ... };

@init fn init() void = {
	ioctlre = regex::compile(`@(_IO[RW]*)\((.*)\)`)!;
	typedefre = regex::compile(`^(export )?type `)!;
};

@fini fn fini() void = {
	regex::finish(&ioctlre);
	regex::finish(&typedefre);
};

This sets aside two regular expressions: one that identifies type aliases (so that we can parse them to determine their size later), and one that identifies our @_IO* pseudo-macros. I also defined some types to store each of the details necessary to compute the ioctl assignment:

type dir = enum u32 {
	IO = 0,
	IOW = 1,
	IOR = 2,
	IOWR = IOW | IOR,
};

type ioctl = (dir, rune, u32, const nullable *types::_type);

Hare’s standard library includes tools for parsing and analyzing Hare programs in the hare namespace. We’ll need to use these to work with types in this program. At the start of the program, we initialize a “type store” from hare::types, which provides a mechanism with which Hare types can be processed and stored. The representation of Hare types varies depending on the architecture (for example, pointer types have different sizes on 32-bit and 64-bit systems), so we have to specify the architecture we want. In the future it will be necessary to make this configurable, but for now I just hard-coded x86_64:

const store = types::store(types::x86_64, null, null);
defer types::store_free(store);

The two “null” parameters are not going to be used here, but are designed to facilitate evaluating expressions in type definitions, such as [8 * 16]int. Leaving them null is permissible, but disables the ability to do this sort of thing.

Following this, we enter a loop which processes the input file line-by-line, testing each line against our regular expressions and doing some logic on them if they match. Let’s start with the code for handling new types:

for (true) {
	const line = match (bufio::scanline(os::stdin)!) {
	case io::EOF =>
		break;
	case let line: []u8 =>
		yield strings::fromutf8(line);
	};
	defer free(line);

	if (regex::test(&typedefre, line)!) {
		bufio::unreadrune(os::stdin, '\n');
		bufio::unread(os::stdin, strings::toutf8(line));
		loadtype(store);
		continue;
	};

	// ...to be continued...

If we encounter a line which matches our type declaration regular expression, then we unread that line back into the (buffered) standard input stream, then call this “loadtype” function to parse and load it into the type store.

fn loadtype(store: *types::typestore) void = {
	const tee = io::tee(os::stdin, os::stdout);
	const lex = lex::init(&tee, "<ioctl>");
	const decl = match (parse::decl(&lex)) {
	case let err: parse::error =>
		fmt::fatal("Error parsing type declaration:",
			parse::strerror(err));
	case let decl: ast::decl =>
		yield decl;
	};

	const tdecl = decl.decl as []ast::decl_type;
	if (len(tdecl) != 1) {
		fmt::fatal("Multiple type declarations are unsupported");
	};
	const tdecl = tdecl[0];
	const of = types::lookup(store, &tdecl._type)!;
	types::newalias(store, tdecl.ident, of);
};

Hare includes a Hare lexer and parser in the standard library, which we’re making use of here. The first thing we do is use io::tee to copy any data the parser reads into stdout, passing it through to the output file. Then we set up a lexer and parse the type declaration. A type declaration looks something like this:

type sock_fprog = struct {
	length: u16,
	filter: *sock_filter,
};

The types::lookup call looks up the struct type, and newalias creates a new type alias based on that type with the given name (sock_filter). Adding this to the type store will let us resolve the type when we encounter it later on, for example in this line:

def TUNGETFILTER: u32 = @_IOR('T', 219, sock_fprog);

Back to the main loop, we have another regex test to check if we’re looking at a line with one of these pseudo-macros:

let groups = match (regex::find(&ioctlre, line)!) {
case void =>
	fmt::println(line)!;
	continue;
case let cap: []regex::capture =>
	yield cap;
};
defer free(groups);

const dir = switch (groups[1].content) {
case "_IO" =>
	yield dir::IO;
case "_IOR" =>
	yield dir::IOR;
case "_IOW" =>
	yield dir::IOW;
case "_IOWR" =>
	yield dir::IOWR;
case =>
	fmt::fatalf("Unknown ioctl direction {}", groups[1].content);
};
const ioctl = parseioctl(store, dir, groups[2].content);

Recall that the regex from earlier is @(_IO[RW]*)\((.*)\). This has two capture groups: one for “_IO” or “_IOW” and so on, and another for the list of “parameters” (the zeroth “capture group” is the entire match string). We use the first capture group to grab the ioctl direction, then we pass that into “parseioctl” along with the type store and the second capture group.

This “parseioctl” function is kind of neat:

fn parseioctl(store: *types::typestore, d: dir, params: str) ioctl = {
	const buf = bufio::fixed(strings::toutf8(params), io::mode::READ);
	const lex = lex::init(&buf, "<ioctl>");

	const rn = expect(&lex, ltok::LIT_RUNE).1 as rune;
	expect(&lex, ltok::COMMA);
	const num = expect(&lex, ltok::LIT_ICONST).1 as i64;

	if (d == dir::IO) {
		return (d, rn, num: u32, null);
	};

	expect(&lex, ltok::COMMA);
	const ty = match (parse::_type(&lex)) {
	case let ty: ast::_type =>
		yield ty;
	case let err: parse::error =>
		fmt::fatal("Error:", parse::strerror(err));
	};

	const ty = match (types::lookup(store, &ty)) {
	case let err: types::error =>
		fmt::fatal("Error:", types::strerror(err));
	case types::deferred =>
		fmt::fatal("Error: this tool does not support forward references");
	case let ty: const *types::_type =>
		yield ty;
	};

	return (d, rn, num: u32, ty);
};

fn expect(lex: *lex::lexer, want: ltok) lex::token = {
	match (lex::lex(lex)) {
	case let err: lex::error =>
		fmt::fatal("Error:", lex::strerror(err));
	case let tok: lex::token =>
		if (tok.0 != want) {
			fmt::fatalf("Error: unexpected {}", lex::tokstr(tok));
		};
		return tok;
	};
};

Here we’ve essentially set up a miniature parser based on a Hare lexer to parse our custom parameter list grammar. We create a fixed reader from the capture group string, then create a lexer based on this and start pulling tokens out of it. The first parameter is a rune, so we grab a LIT_RUNE token and extract the Hare rune value from it, then after a COMMA token we repeat this with LIT_ICONST to get the integer constant. dir::IO ioctls don’t have a type parameter, so can return early in this case.

Otherwise, we use hare::parse::_type to parse the type parameter, producing a hare::ast::_type. We then pass this to the type store to look up technical details about this type, such as its size, alignment, storage representation, and so on. This converts the AST type — which only has lexical information — into an actual type, including semantic information about the type.

Equipped with this information, we can calculate the ioctl’s assigned number:

def IOC_NRBITS: u32 = 8;
def IOC_TYPEBITS: u32 = 8;
def IOC_SIZEBITS: u32 = 14; // XXX: Arch-specific
def IOC_DIRBITS: u32 = 2; // XXX: Arch-specific

def IOC_NRSHIFT: u32 = 0;
def IOC_TYPESHIFT: u32 = IOC_NRSHIFT + IOC_NRBITS;
def IOC_SIZESHIFT: u32 = IOC_TYPESHIFT + IOC_TYPEBITS;
def IOC_DIRSHIFT: u32 = IOC_SIZESHIFT + IOC_SIZEBITS;

fn ioctlno(io: *ioctl) u32 = {
	const typesz = match (io.3) {
	case let ty: const *types::_type =>
		yield ty.sz;
	case null =>
		yield 0z;
	};
	return (io.0: u32 << IOC_DIRSHIFT) |
		(io.1: u32 << IOC_TYPESHIFT) |
		(io.2 << IOC_NRSHIFT) |
		(typesz: u32 << IOC_SIZESHIFT);
};

And, back in the main loop, print it to the output:

const prefix = strings::sub(line, 0, groups[1].start - 1);
fmt::printfln("{}0x{:x};", prefix, ioctlno(&ioctl))!;

Now we have successfully converted this:

type sock_filter = struct {
	code: u16,
	jt: u8,
	jf: u8,
	k: u32,
};

type sock_fprog = struct {
	length: u16,
	filter: *sock_filter,
};

def TUNATTACHFILTER: u32 = @_IOW('T', 213, sock_fprog);

Into this:

def TUNATTACHFILTER: u32 = 0x401054d5;

A quick C program verifies our result:

#include <linux/ioctl.h>
#include <linux/if_tun.h>
#include <stdio.h>

int main() {
	printf("TUNATTACHFILTER: 0x%lx\n", TUNATTACHFILTER);
}

And:

TUNATTACHFILTER: 0x401054d5

It works!


Critics may draw attention to the fact that we could have saved ourselves much of this work if Hare had first-class macros, but macros are not aligned with Hare’s design goals, so an alternative solution is called for. This particular program is useful only in a small set of specific circumstances (and mainly for Hare developers themselves, less so for most users), but it solves the problem pretty neatly given the constraints it has to work within.

I think this is a nice case study in a few useful features available from the Hare standard library. In addition to POSIX Extended Regular Expression support via the regex module, the hare namespace offers many tools to provide Hare programs with relatively deep insights into the language itself. We can use hare::lex to parse the custom grammar for our pseudo-macros, use hare::parse to parse type declarations, and use hare::types to compute the semantic details of each type. I also like many of the “little things” on display here, such as unreading data back into the buffered stdin reader, or using io::tee to copy data to stdout during parsing.

I hope you found it interesting!

When will we learn?

12 May 2022 at 00:00

Congratulations to Rust for its first (but not its last) supply-chain attack this week! They join a growing club of broken-by-design package managers which publish packages uploaded by vendors directly, with no review step, and ship those packages directly to users with no further scrutiny.

Timeline of major incidents on npm/Crates/PyPI/etc

There are hundreds of additional examples. I had to leave many of them out. Here’s a good source if you want to find more.

Timeline of similar incidents in official Linux distribution repositories

(this space deliberately left blank)

Why is this happening?

The correct way to ship packages is with your distribution’s package manager. These have a separate review step, completely side-stepping typo-squatting, establishing a long-term relationship of trust between the vendor and the distribution packagers, and providing a dispassionate third-party to act as an intermediary between users and vendors. Furthermore, they offer stable distributions which can be relied upon for an extended period of time, provide cohesive whole-system integration testing, and unified patch distribution and CVE notifications for your entire system.

For more details, see my previous post, Developers: Let distros do their job.

Can these package managers do it better?

I generally feel that overlay package managers (a term I just made up for npm et al) are redundant. However, you may feel otherwise, and wonder what they could do better to avoid these problems.

It’s simple: they should organize themselves more like a system package manager.

  1. Establish package maintainers independent of the vendors
  2. Establish a review process for package updates

There’s many innovations that system package managers have been working on which overlay package managers could stand to learn from as well, such as:

  • Universal package signatures and verification
  • Reproducible builds
  • Mirrored package distribution

For my part, I’ll stick to the system package manager. But if you think that the overlay package manager can do it better: prove it.

Implementing an SSH agent in Hare

9 May 2022 at 00:00

Cross-posted from the Hare blog

In the process of writing an SSH agent for Himitsu, I needed to implement many SSH primitives from the ground up in Hare, now available via hare-ssh. Today, I’m going to show you how it works!

Important: This blog post deals with cryptography-related code. The code you’re going to see today is incomplete, unaudited, and largely hasn’t even seen any code review. Let me begin with a quote from the “crypto” module’s documentation in the Hare standard library:

Cryptography is a difficult, high-risk domain of programming. The life and well-being of your users may depend on your ability to implement cryptographic applications with due care. Please carefully read all of the documentation, double-check your work, and seek second opinions and independent review of your code. Our documentation and API design aims to prevent easy mistakes from being made, but it is no substitute for a good background in applied cryptography.

Do your due diligence before repurposing anything you see here.

Decoding SSH private keys

Technically, you do not need to deal with OpenSSH private keys when implementing an SSH agent. However, my particular use-case includes dealing with this format, so I started here. Unlike much of SSH, the OpenSSH private key format (i.e. the format of the file at ~/.ssh/id_ed25519) is, well, private. It’s not documented and I had to get most of the details from reverse-engineering the OpenSSH C code. The main area of interest is sshkey.c. I’ll spare you from reading it yourself and just explain how it works.

First of all, let’s just consider what an SSH private key looks like:

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAACmFlczI1Ni1jdHIAAAAGYmNyeXB0AAAAGAAAABDTIm/zSI
7zeHAs4rIXaOD1AAAAEAAAAAEAAAAzAAAAC3NzaC1lZDI1NTE5AAAAIE7qq/pMk9VrRupn
9j4/tNHclJnKgJAE1pfUecRNT1fAAAAAoEcx6mnJmFlYXx1eYztw6SZ5yuL6T1LWfj+bpg
7zNQBoqJW1j+Q17PUMtXj9wDDOQx+6OE7JT/RrK3Vltp4oXmFI4FgsYbE9RbNXSC2xvLaX
fplmx+eAOir9UTZGTIbOGy1cVho8LzDLLo4WiGYbpxtIvkJE72f0YdTm8RrNVkLlAy7ayV
uFcoq1JBrjIAa7UtqIr9SG8b76ALJZb9jPc3A=
-----END OPENSSH PRIVATE KEY-----

We can immediately tell that this is a PEM file (RFC 7468). The first step to read this file was to implement a decoder for the PEM format, which has been on our to-do list for a while now, and is also needed for many other use-cases. Similar to many other formats provided in the standard library, you can call pem::newdecoder to create a PEM decoder for an arbitrary I/O source, returning the decoder state on the stack. We can then call pem::next to find the next PEM header (-----BEGIN...), which returns a decoder for that specific PEM blob (this design accommodates PEM files which have several PEM segments concatenated together, or intersperse other data in the file alongside the PEM bits. This is common for other PEM use-cases). With this secondary decoder, we can simply read from it like any other I/O source and it decodes the base64-encoded data and returns it to us as bytes.

Based on this, we can examine the contents of this key with a simple program.

use encoding::hex;
use encoding::pem;
use fmt;
use io;
use os;

export fn main() void = {
	const dec = pem::newdecoder(os::stdin);
	defer pem::finish(&dec);

	for (true) {
		const reader = match (pem::next(&dec)) {
		case let reader: (str, pem::pemdecoder) =>
			yield reader;
		case io::EOF =>
			break;
		};
		const name = reader.0, stream = reader.1;
		defer io::close(&stream)!;

		fmt::printfln("PEM data '{}':", name)!;
		const bytes = io::drain(&stream)!;
		defer free(bytes);

		hex::dump(os::stdout, bytes)!;
	};
};

Running this program on our sample key yields the following:

PEM data 'OPENSSH PRIVATE KEY':
00000000  6f 70 65 6e 73 73 68 2d  6b 65 79 2d 76 31 00 00  |openssh-key-v1..|
00000010  00 00 0a 61 65 73 32 35  36 2d 63 74 72 00 00 00  |...aes256-ctr...|
00000020  06 62 63 72 79 70 74 00  00 00 18 00 00 00 10 d3  |.bcrypt.........|
00000030  22 6f f3 48 8e f3 78 70  2c e2 b2 17 68 e0 f5 00  |"o.H..xp,...h...|
00000040  00 00 10 00 00 00 01 00  00 00 33 00 00 00 0b 73  |..........3....s|
00000050  73 68 2d 65 64 32 35 35  31 39 00 00 00 20 4e ea  |sh-ed25519... N.|
00000060  ab fa 4c 93 d5 6b 46 ea  67 f6 3e 3f b4 d1 dc 94  |..L..kF.g.>?....|
00000070  99 ca 80 90 04 d6 97 d4  79 c4 4d 4f 57 c0 00 00  |........y.MOW...|
00000080  00 a0 47 31 ea 69 c9 98  59 58 5f 1d 5e 63 3b 70  |..G1.i..YX_.^c;p|
00000090  e9 26 79 ca e2 fa 4f 52  d6 7e 3f 9b a6 0e f3 35  |.&y...OR.~?....5|
000000a0  00 68 a8 95 b5 8f e4 35  ec f5 0c b5 78 fd c0 30  |.h.....5....x..0|
000000b0  ce 43 1f ba 38 4e c9 4f  f4 6b 2b 75 65 b6 9e 28  |.C..8N.O.k+ue..(|
000000c0  5e 61 48 e0 58 2c 61 b1  3d 45 b3 57 48 2d b1 bc  |^aH.X,a.=E.WH-..|
000000d0  b6 97 7e 99 66 c7 e7 80  3a 2a fd 51 36 46 4c 86  |..~.f...:*.Q6FL.|
000000e0  ce 1b 2d 5c 56 1a 3c 2f  30 cb 2e 8e 16 88 66 1b  |..-\V.</0.....f.|
000000f0  a7 1b 48 be 42 44 ef 67  f4 61 d4 e6 f1 1a cd 56  |..H.BD.g.a.....V|
00000100  42 e5 03 2e da c9 5b 85  72 8a b5 24 1a e3 20 06  |B.....[.r..$.. .|
00000110  bb 52 da 88 af d4 86 f1  be fa 00 b2 59 6f d8 cf  |.R..........Yo..|
00000120  73 70                                             |sp|

OpenSSH private keys begin with a magic string, “openssh-key-v1\0”, which we can see here. Following this are a number of binary encoded fields which are represented in a manner similar to the SSH wire protocol, most often as strings prefixed by their length, encoded as a 32-bit big-endian integer. In order, the fields present here are:

  • Cipher name (aes256-ctr)
  • KDF name (bcrypt)
  • KDF data
  • Public key data
  • Private key data (plus padding)

We parse this information like so:

export type sshprivkey = struct {
	cipher: str,
	kdfname: str,
	kdf: []u8,
	pubkey: []u8,
	privkey: []u8,
};

export fn decodesshprivate(in: io::handle) (sshprivkey | error) = {
	const pem = pem::newdecoder(in);
	const dec = match (pem::next(&pem)?) {
	case io::EOF =>
		return invalid;
	case let dec: (str, pem::pemdecoder) =>
		if (dec.0 != "OPENSSH PRIVATE KEY") {
			return invalid;
		};
		yield dec.1;
	};

	let magicbuf: [15]u8 = [0...];
	match (io::readall(&dec, magicbuf)?) {
	case size => void;
	case io::EOF =>
		return invalid;
	};
	if (!bytes::equal(magicbuf, strings::toutf8(magic))) {
		return invalid;
	};

	let key = sshprivkey { ... };
	key.cipher = readstr(&dec)?;
	key.kdfname = readstr(&dec)?;
	key.kdf = readslice(&dec)?;

	let buf: [4]u8 = [0...];
	match (io::readall(&dec, buf)?) {
	case size => void;
	case io::EOF =>
		return invalid;
	};
	const nkey = endian::begetu32(buf);
	if (nkey != 1) {
		// OpenSSH currently hard-codes the number of keys to 1
		return invalid;
	};

	key.pubkey = readslice(&dec)?;
	key.privkey = readslice(&dec)?;

	// Add padding bytes
	append(key.privkey, io::drain(&dec)?...);
	return key;
};

However, to get at the actual private key — so that we can do cryptographic operations with it — we first have to decrypt this inner data. Those three fields — cipher name, KDF name, and KDF data — are our hint. In essence, this data is encrypted by OpenSSH by using a variant of bcrypt as a key derivation function, which turns your password (plus a salt) into a symmetric encryption key. Then it uses AES 256 in CTR mode with this symmetric key to encrypt the private key data. With the benefit of hindsight, I might question these primitives, but that’s what they use so we’ll have to work with it.

Prior to starting this work, Hare already had support for AES and CTR, though they gained some upgrades during the course of this work, since using an interface for real-world code is the best way to evaluate its design. This leaves us to implement bcrypt.

bcrypt is a password hashing algorithm invented by OpenBSD based on the Blowfish cipher, and it is pretty badly designed. However, Blowfish was fairly straightforward to implement. I’ll spare you the details, but here’s the documentation and implementation for your consideration. I also implemented the standard bcrypt hash at crypto::bcrypt, whose implementation is here (for now). This isn’t especially relevant for us, however, since OpenSSH uses a modified form of bcrypt as a key derivation function.

The implementation the bcrypt KDF in Hare is fairly straightforward. To write it, I referenced OpenSSH portable’s vendored OpenBSD implementation at openbsd-compat/bcrypt_pbkdf.c, as well as the Go implementation in golang.org/x/crypto. Then, with these primitives done, we can implement the actual key decryption.

First, not all keys are encrypted with a passphrase, so a simple function tells us if this step is required:

// Returns true if this private key is encrypted with a passphrase.
export fn isencrypted(key: *sshprivkey) bool = {
	return key.kdfname != "none";
};

The “decrypt” function is used to perform the actual decryption. It begins by finding the symmetric key, like so:

export fn decrypt(key: *sshprivkey, pass: []u8) (void | error) = {
	assert(isencrypted(key));

	const cipher = getcipher(key.cipher)?;
	let ckey: []u8 = alloc([0...], cipher.keylen + cipher.ivlen);
	defer {
		bytes::zero(ckey);
		free(ckey);
	};

	let kdfbuf = bufio::fixed(key.kdf, io::mode::READ);
	switch (key.kdfname) {
	case "bcrypt" =>
		const salt = readslice(&kdfbuf)?;
		defer free(salt);
		const rounds = readu32(&kdfbuf)?;
		bcrypt_pbkdf(ckey, pass, salt, rounds);
	case =>
		return badcipher;
	};

The “KDF data” field I mentioned earlier uses a format private to each KDF mode, though at the present time the only supported KDF is this bcrypt one. In this case, it serves as the salt. The “getcipher” function returns some data from a static table of supported ciphers, which provides us with the required size of the cipher’s key and IV parameters. We allocate sufficient space to store these, create a bufio reader from the KDF field, read out the salt and hashing rounds, and hand all of this over to the bcrypt function to produce our symmetric key (and I/V) in the “ckey” variable.

We may then use these parameters to decrypt the private key area.

	let secretbuf = bufio::fixed(key.privkey, io::mode::READ);
	const cipher = cipher.init(&secretbuf,
		ckey[..cipher.keylen], ckey[cipher.keylen..]);
	defer cipher_free(cipher);

	let buf: []u8 = alloc([0...], len(key.privkey));
	defer free(buf);
	io::readall(cipher, buf)!;

	const a = endian::begetu32(buf[..4]);
	const b = endian::begetu32(buf[4..8]);
	if (a != b) {
		return badpass;
	};

	key.privkey[..] = buf[..];

	free(key.kdf);
	free(key.kdfname);
	free(key.cipher);
	key.kdfname = strings::dup("none");
	key.cipher = strings::dup("none");
	key.kdf = [];
};

The “cipher.init” function is an abstraction that allows us to support more ciphers in the future. For this particular cipher mode, it’s implemented fairly simply:

type aes256ctr = struct {
	st: cipher::ctr_stream,
	block: aes::ct64_block,
	buf: [aes::CTR_BUFSIZE]u8,
};

fn aes256ctr_init(handle: io::handle, key: []u8, iv: []u8) *io::stream = {
	let state = alloc(aes256ctr {
		block = aes::ct64(),
		...
	});
	aes::ct64_init(&state.block, key);
	state.st = cipher::ctr(handle, &state.block, iv, state.buf);
	return state;
};

Within this private key data section, once decrypted, are several fields. First is a random 32-bit integer which is written twice — comparing that these are equal to one another allows us to verify the user’s password. Once verified, we overwrite the private data field in the key structure with the decrypted data, and update the cipher and KDF information to indicate that the key is unencrypted. We could decrypt it directly into the existing private key buffer, without allocating a second buffer, but this would overwrite the encrypted data with garbage if the password was wrong — you’d have to decode the key all over again if the user wants to try again.

So, what does this private key blob look like once decrypted? The hare-ssh repository includes a little program at cmd/sshkey which dumps all of the information stored in an SSH key, and it provides us with this peek at the private data:

00000000  fb 15 e6 16 fb 15 e6 16  00 00 00 0b 73 73 68 2d  |............ssh-|
00000010  65 64 32 35 35 31 39 00  00 00 20 4e ea ab fa 4c  |ed25519... N...L|
00000020  93 d5 6b 46 ea 67 f6 3e  3f b4 d1 dc 94 99 ca 80  |..kF.g.>?.......|
00000030  90 04 d6 97 d4 79 c4 4d  4f 57 c0 00 00 00 40 17  |.....y.MOW....@.|
00000040  bf 87 74 0b 2a 74 d5 29  d0 14 10 3f 04 5d 88 c6  |..t.*t.)...?.]..|
00000050  32 fa 21 9c e9 97 b0 5a  e7 7e 5c 02 72 35 72 4e  |2.!....Z.~\.r5rN|
00000060  ea ab fa 4c 93 d5 6b 46  ea 67 f6 3e 3f b4 d1 dc  |...L..kF.g.>?...|
00000070  94 99 ca 80 90 04 d6 97  d4 79 c4 4d 4f 57 c0 00  |.........y.MOW..|
00000080  00 00 0e 73 69 72 63 6d  70 77 6e 40 74 61 69 67  |...sircmpwn@taig|
00000090  61 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |a...............|

We can see upfront these two 32-bit verification numbers I mentioned, and following this are several fields in a similar format to earlier — length-prefixed strings. The fields are:

  • Key type (“ssh-ed25519” in this case)
  • Public key (in a format specific to each key type)
  • Private key (in a format specific to each key type)
  • Comment
  • Padding up to the cipher’s block size (16)

This is a little bit weird in my opinion — the public key field is redundant with the unencrypted data in this file, and the comment field is probably not so secret as to demand encryption. I think these are just consequences of the file format being private to OpenSSH’s implementation; not much thought has gone into it and implementation details (like the ability to call the same “dump private key” function here as OpenSSH uses elsewhere) have probably leaked through.

We can decode this data with the following Hare code:

export fn decodeprivate(src: *sshprivkey) (key | error) = {
	assert(!isencrypted(src));
	const buf = bufio::fixed(src.privkey, io::mode::READ);

	let verify: [8]u8 = [0...];
	io::read(&buf, verify)!;
	const a = endian::begetu32(verify[..4]);
	const b = endian::begetu32(verify[4..8]);
	if (a != b) {
		return badpass;
	};

	const keytype = readstr(&buf)?;
	defer free(keytype);

	switch (keytype) {
	case "ssh-ed25519" =>
		let key = ed25519key { ... };
		decode_ed25519_sk(&key, &buf)?;
		return key;
	case =>
		// TODO: Support additional key types
		return badcipher;
	};
};

// An ed25519 key pair.
export type ed25519key = struct {
	pkey: ed25519::publickey,
	skey: ed25519::privatekey,
	comment: str,
};

fn decode_ed25519_pk(key: *ed25519key, buf: io::handle) (void | error) = {
	const l = readu32(buf)?;
	if (l != ed25519::PUBLICKEYSZ) {
		return invalid;
	};
	io::readall(buf, key.pkey)?;
};

fn decode_ed25519_sk(key: *ed25519key, buf: io::handle) (void | error) = {
	decode_ed25519_pk(key, buf)?;

	const l = readu32(buf)?;
	if (l != ed25519::PRIVATEKEYSZ) {
		return invalid;
	};
	io::readall(buf, key.skey)?;

	// Sanity check
	const pkey = ed25519::skey_getpublic(&key.skey);
	if (!bytes::equal(pkey, key.pkey)) {
		return invalid;
	};

	key.comment = readstr(buf)?;
};

Fairly straightforward! Finally, we have extracted the actual private key from the file. For this SSH key, in base64, the cryptographic keys are:

Public key:  Tuqr+kyT1WtG6mf2Pj+00dyUmcqAkATWl9R5xE1PV8A=
Private key: F7+HdAsqdNUp0BQQPwRdiMYy+iGc6ZewWud+XAJyNXJO6qv6TJPVa0bqZ/Y+P7TR3JSZyoCQBNaX1HnETU9XwA==

Signing and verification with ed25519

Using these private keys, implementing signatures and signature verification are pretty straightforward. We can stop reading the OpenSSH code at this point — RFC 8709 standardizes this format for ed25519 signatures.

use crypto::ed25519;
use io;

// Signs a message using the provided key, writing the message signature in the
// SSH format to the provided sink.
export fn sign(
	sink: io::handle,
	key: *key,
	msg: []u8,
) (void | io::error) = {
	const signature = ed25519::sign(&key.skey, msg);
	writestr(sink, "ssh-ed25519")?;
	writeslice(sink, signature)?;
};

// Reads an SSH wire signature from the provided I/O handle and verifies that it
// is a valid signature for the given message and key. If valid, void is
// returned; otherwise [[badsig]] is returned.
export fn verify(
	source: io::handle,
	key: *key,
	msg: []u8,
) (void | error) = {
	const sigtype = readstr(source)?;
	defer free(sigtype);
	if (sigtype != keytype(key)) {
		return badsig;
	};
	const sig = readslice(source)?;
	defer free(sig);

	assert(sigtype == "ssh-ed25519"); // TODO: other key types
	if (len(sig) != ed25519::SIGNATURESZ) {
		return badsig;
	};

	const sig = sig: *[*]u8: *[ed25519::SIGNATURESZ]u8;
	if (!ed25519::verify(&key.pkey, msg, sig)) {
		return badsig;
	};
};

This implementation writes and reads signatures in the SSH wire format, which is generally how they will be most useful in this context. This code will be expanded in the future with additional keys, such as RSA, once the necessary primitives are implemented for Hare’s standard library.

The SSH agent protocol

The agent protocol is also standardized (albeit in draft form), so we refer to draft-miller-ssh-agent-11 from this point onwards. It’s fairly straightforward. The agent communicates over an unspecified protocol (Unix sockets in practice) by sending messages in the SSH wire format, which, again, mainly comes in the form of strings prefixed by their 32-bit length in network order.

The first step for implementing net::ssh::agent starts with adding types for all of the data structures and enums for all of the constants, which you can find in types.ha. Each message begins with its length, then a message type (one byte) and a message payload; the structure of the latter varies with the message type.

I started to approach this by writing some functions which, given a byte buffer that contains an SSH agent message, either parses it or asks for more data.

export fn parse(msg: []u8) (message | size | invalid) = {
	if (len(msg) < 5) {
		return 5 - len(msg);
	};
	const ln = endian::begetu32(msg[..4]);
	if (len(msg) < 4 + ln) {
		return 4 + ln - len(msg);
	};

	const mtype = msg[4];
	const buf = bufio::fixed(msg[5..], io::mode::READ);
	switch (mtype) {
	case messagetype::REQUEST_IDENTITIES =>
		return request_identities;
	case messagetype::SIGN_REQUEST =>
		return parse_sign_request(&buf)?;
	case messagetype::ADD_IDENTITY =>
		return parse_add_identity(&buf)?;
	// ...trimmed for brevity, and also because it's full of TODOs...
	case =>
		return invalid.
	};
};

Each individual message payload includes its own parser, except for some messages (such as REQUEST_IDENTITIES), which have no payload. Here’s what the parser for SIGN_REQUEST looks like:

fn parse_sign_request(src: io::handle) (sign_request | invalid) = {
	return sign_request {
		key = readslice(src)?,
		data = readslice(src)?,
		flags = readu32(src)?: sigflag,
	};
};

Pretty straightforward! A more complex one is ADD_IDENTITY:

fn parse_add_identity(src: io::handle) (add_identity | invalid) = {
	const keytype = readstr(src)?;
	// TODO: Support more key types
	const key: ssh::key = switch (keytype) {
	case "ssh-ed25519" =>
		let key = ssh::ed25519key { ... };
		const npub = readu32(src)?;
		if (npub != len(key.pkey)) {
			return invalid;
		};
		io::readall(src, key.pkey)!;
		const npriv = readu32(src)?;
		if (npriv != len(key.skey)) {
			return invalid;
		};
		io::readall(src, key.skey)!;
		yield key;
	case =>
		return invalid;
	};
	return add_identity {
		keytype = keytype,
		key = key,
		comment = readstr(src)?,
	};
};

One thing I’m not thrilled with in this code is memory management. In Hare, libraries like this one are not supposed to allocate memory if they can get away with it, and if they must, they should do it as conservatively as possible. This implementation does a lot of its own allocations, which is unfortunate. I might refactor it in the future to avoid this. A more subtle issue here is the memory leaks on errors — each of the readslice/readstr functions allocates data for its return value, but if they return an error, the ? operator will return immediately without freeing them. This is a known problem with Hare’s language design, and while we have some ideas for addressing it, we have not completed any of them yet. This is one of a small number of goals for Hare which will likely require language changes prior to 1.0.

We have a little bit more code in net::ssh::agent, which you can check out if you like, but this covers most of it — time to move onto the daemon implementation.

Completing our SSH agent

The ssh-agent command in the hare-ssh tree is a simple (and non-production) implementation of an SSH agent based on this work. Let’s go over its code to see how this all comes together to make it work.

First, we set up a Unix socket, and somewhere to store our application state.

let running: bool = true;

type identity = struct {
	comment: str,
	privkey: ssh::key,
	pubkey: []u8,
};

type state = struct {
	identities: []identity,
};

export fn main() void = {
	let state = state { ... };
	const sockpath = "./socket";

	const listener = unix::listen(sockpath)!;
	defer {
		net::shutdown(listener);
		os::remove(sockpath)!;
	};
	os::chmod(sockpath, 0o700)!;
	log::printfln("Listening at {}", sockpath);

We also need a main loop, but we need to clean up that Unix socket when we terminate, so we’ll also set up some signal handlers.

	signal::handle(signal::SIGINT, &handle_signal);
	signal::handle(signal::SIGTERM, &handle_signal);

	for (running) {
		// ...stay tuned...
	};

	for (let i = 0z; i < len(state.identities); i += 1) {
		const ident = state.identities[i];
		ssh::key_finish(&ident.privkey);
		free(ident.pubkey);
		free(ident.comment);
	};

	log::printfln("Terminated.");
};

// ...elsewhere...
fn handle_signal(sig: int, info: *signal::siginfo, ucontext: *void) void = {
	running = false;
};

The actual clean-up is handled by our “defer” statement at the start of “main”. The semantics of signal handling on Unix are complex (and bad), and beyond the scope of this post, so hopefully you already grok them. Our stdlib provides docs, if you care to learn more, but also includes this warning:

Signal handling is stupidly complicated and easy to get wrong. The standard library makes little effort to help you deal with this. Consult your local man pages, particularly signal-safety(7) on Linux, and perhaps a local priest as well. We advise you to get out of the signal handler as soon as possible, for example via the “self-pipe trick”.

We also provide signalfds on platforms that support them (such as Linux), which is less fraught with issues. Good luck.

Next: the main loop. This code accepts new clients, prepares an agent for them, and hands them off to a second function:

		const client = match (net::accept(listener)) {
		case errors::interrupted =>
			continue;
		case let err: net::error =>
			log::fatalf("Error: accept: {}", net::strerror(err));
		case let fd: io::file =>
			yield fd;
		};
		const agent = agent::new(client);
		defer agent::agent_finish(&agent);
		run(&state, &agent);

This is a really simple event loop for a network daemon, and comes with one major limitation: no support for serving multiple clients connecting at once. If you’re curious what a more robust network daemon looks like in Hare, consult the Himitsu code.

The “run” function simply reads SSH agent commands and processes them, until the client disconnects.

fn run(state: *state, agent: *agent::agent) void = {
	for (true) {
		const msg = match (agent::readmsg(agent)) {
		case (io::EOF | agent::error) =>
			break;
		case void =>
			continue;
		case let msg: agent::message =>
			yield msg;
		};
		defer agent::message_finish(&msg);

		const res = match (msg) {
		case agent::request_identities =>
			yield handle_req_ident(state, agent);
		case let msg: agent::add_identity =>
			yield handle_add_ident(state, &msg, agent);
		case let msg: agent::sign_request =>
			yield handle_sign_request(state, &msg, agent);
		case agent::extension =>
			const answer: agent::message = agent::extension_failure;
			agent::writemsg(agent, &answer)!;
		case => abort();
		};
		match (res) {
		case void => yield;
		case agent::error => abort();
		};
	};
};

Again, this is non-production code, and, among other things, is missing good error handling. The handlers for each message are fairly straightforward, however. Here’s the handler for REQUEST_IDENTITIES:

fn handle_req_ident(
	state: *state,
	agent: *agent::agent,
) (void | agent::error) = {
	let idents: agent::identities_answer = [];
	defer free(idents);

	for (let i = 0z; i < len(state.identities); i += 1) {
		const ident = &state.identities[i];
		append(idents, agent::identity {
			pubkey = ident.pubkey,
			comment = ident.comment,
		});
	};

	const answer: agent::message = idents;
	agent::writemsg(agent, &answer)!;
};

The first one to do something interesting is ADD_IDENTITY, which allows the user to supply SSH private keys to the agent to work with:

fn handle_add_ident(
	state: *state,
	msg: *agent::add_identity,
	agent: *agent::agent,
) (void | agent::error) = {
	let sink = bufio::dynamic(io::mode::WRITE);
	ssh::encode_pubkey(&sink, &msg.key)!;
	append(state.identities, identity {
		comment = strings::dup(msg.comment),
		privkey = msg.key,
		pubkey = bufio::buffer(&sink),
	});
	const answer: agent::message = agent::agent_success;
	agent::writemsg(agent, &answer)?;
	log::printfln("Added key {}", msg.comment);
};

With these two messages, we can start to get the agent to do something relatively interesting: accepting and listing keys.

$ hare run cmd/ssh-agent/
[2022-05-09 17:39:12] Listening at ./socket
^Z[1]+  Stopped                    hare run cmd/ssh-agent/
$ bg
[1] hare run cmd/ssh-agent/
$ export SSH_AUTH_SOCK=./socket
$ ssh-add -l
The agent has no identities.
$ ssh-add ~/.ssh/id_ed25519
Enter passphrase for /home/sircmpwn/.ssh/id_ed25519: 
Identity added: /home/sircmpwn/.ssh/id_ed25519 (sircmpwn@homura)
2022-05-09 17:39:31] Added key sircmpwn@homura
$ ssh-add -l
256 SHA256:kPr5ZKTNE54TRHGSaanhcQYiJ56zSgcpKeLZw4/myEI sircmpwn@homura (ED25519)

With the last message handler, we can upgrade from something “interesting” to something “useful”:

fn handle_sign_request(
	state: *state,
	msg: *agent::sign_request,
	agent: *agent::agent,
) (void | agent::error) = {
	let key: nullable *identity = null;
	for (let i = 0z; i < len(state.identities); i += 1) {
		let ident = &state.identities[i];
		if (bytes::equal(ident.pubkey, msg.key)) {
			key = ident;
			break;
		};
	};
	const key = match (key) {
	case let key: *identity =>
		yield key;
	case null =>
		const answer: agent::message = agent::agent_failure;
		agent::writemsg(agent, &answer)?;
		return;
	};

	let buf = bufio::dynamic(io::mode::WRITE);
	defer io::close(&buf)!;
	ssh::sign(&buf, &key.privkey, msg.data)!;

	const answer: agent::message = agent::sign_response {
		signature = bufio::buffer(&buf),
	};
	agent::writemsg(agent, &answer)?;
	log::printfln("Signed challenge with key {}", key.comment);
};

For performance reasons, it may be better to use a hash map in a production Hare program (and, as many commenters will be sure to point out, Hare does not provide a built-in hash map or generics). We select the desired key with a linear search, sign the provided payload, and return the signature to the client. Finally, the big pay-off:

$ ssh git@git.sr.ht
[2022-05-09 17:41:42] Signed challenge with key sircmpwn@homura
PTY allocation request failed on channel 0
Hi sircmpwn! You've successfully authenticated, but I do not provide an interactive shell. Bye!
Connection to git.sr.ht closed.

Incorporating it into Himitsu

Himitsu was the motivation for all of this work, and I have yet to properly introduce it to the public. I will go into detail later, but in essence, Himitsu is a key-value store that stores some keys in plaintext and some keys encrypted, and acts as a more general form of a password manager. One of the things it can do (at least as of this week) is store your SSH private keys and act as an SSH agent, via a helper called himitsu-ssh. The user can import their private key from OpenSSH’s private key format via the “hissh-import” tool, and then the “hissh-agent” daemon provides agent functionality via the Himitsu key store.

The user can import their SSH key like so:

$ hissh-import < ~/.ssh/id_ed25519
Enter SSH key passphrase: 
key proto=ssh type=ssh-ed25519 pkey=pF7SljE25sVLdWvInO4gfqpJbbjxI6j+tIUcNWzVTHU= skey! comment=sircmpwn@homura

# Query the key store for keys matching proto=ssh:
$ hiq proto=ssh
proto=ssh type=ssh-ed25519 pkey=pF7SljE25sVLdWvInO4gfqpJbbjxI6j+tIUcNWzVTHU= skey! comment=sircmpwn@homura

Then, when running the agent:

(Yes, I know that the GUI has issues. I slapped it together in C in an afternoon and it needs a lot of work. Help wanted!)

Ta-da!

What’s next?

I accomplished my main goal, which was getting my SSH setup working with Himitsu. The next steps for expanding hare-ssh are:

  1. Expanding the supported key types and ciphers (RSA, DSA, etc), which first requires implementing the primitives in the standard library
  2. Implement the SSH connection protocol, which requires primitives like ECDH in the standard library. Some required primitives, like ChaCha, are already supported.
  3. Improve the design of the networking code. hare-ssh is one of a very small number of network-facing Hare libraries, and it’s treading new design ground here.

SSH is a relatively small target for a cryptography implementation to aim for. I’m looking forward to using it as a testbed for our cryptographic suite. If you’re interested in helping with any of these, please get in touch! If you’re curious about Hare in general, check out the language introduction to get started. Good luck!

Status update, April 2022

15 April 2022 at 00:00

This month marked my first time filing taxes in two countries, and I can assure you it is the worst. I am now a single-issue voter in the US: stop taxing expats! You can get some insight into the financials of SourceHut in the recently-published financial report. But let’s get right into the fun stuff: free software development news.

There was some slowdown from me this month thanks to all of the business and financial crap I had to put up with, but I was able to get some cool stuff done and many other contributors have been keeping things moving. I’ll start by introducing a new/old project: Himitsu.

Essentially, Himitsu is a secret storage system whose intended use-case is to provide features like password storage and SSH agent functionality. It draws much of its inspiration from Plan 9’s Factotum. You may have stumbled upon an early prototype on git.sr.ht which introduces the basic idea and included the start of an implementation in C. Ultimately I shelved this project for want of a better programming language to implement it with, and then I made a better programming language to implement it with. Over the past two weeks, I have implemented something similar to where the C codebase was left, in fewer than half the lines of code and much less than half the time. Here’s a little peek at what works now:

[12:18:31] taiga ~/s/himitsu $ ./himitsud 
Please enter your passphrase to unlock the keyring: 
[2022-04-15 12:18:56] himitsud running
^Z[1]+  Stopped                    ./himitsud
[12:18:57] taiga ~/s/himitsu $ bg
[1] ./himitsud
[12:18:58] taiga ~/s/himitsu $ nc -U ~/.local/state/himitsu/socket 
add proto=imap host=example.org user=sir@cmpwn.com password!="Hello world!"
^C
[12:19:12] taiga ~/s/himitsu $ ls ~/.local/share/himitsu/
2849c1d5-61b3-4803-98cf-fc57fe5f69a6  index  key
[12:19:14] taiga ~/s/himitsu $ cat ~/.local/share/himitsu/index
YNfVlkORDX1GmXIfL8vOiiTgBJKh47biFsUaKrqzfMP2xfD4B9/lqSl2Y9OtIpVcYzrNjBBZOxcO81vNQdgnvxQ+xaCKaVpQS4Dh6DyaY0/lpq6rfowTY5GwcI155KkmTI4z1ABOVkL4z4XDsQ2DEiqClcQE5/+CxsQ/U/u9DthLJRjrjw==
[12:19:19] taiga ~/s/himitsu $ fg
./himitsud
^C[2022-04-15 12:19:22] himitsud terminated
[12:19:22] taiga ~/s/himitsu $ ./himitsud 
Please enter your passphrase to unlock the keyring: 
Loaded key proto=imap host=example.org user=sir@cmpwn.com password!=2849c1d5-61b3-4803-98cf-fc57fe5f69a6
[2022-04-15 12:19:29] himitsud running
^C[2022-04-15 12:19:31] himitsud terminated
[12:19:33] taiga ~/s/himitsu $ find . -type f | xargs wc -l | tail -n1
  895 total

This project is progressing quite fast and I hope to have it working for some basic use-cases soon. I’ll do a dedicated blog post explaining how it works and why it’s important later on, though it will remain mostly under wraps until the language is released.

Speaking of the language, there were a number of exciting developments this month. Two major standard library initiatives were merged: regex and datetime. The regex implementation is simple, small, and fast, targeting POSIX ERE as a reasonably sane conservative baseline regex dialect. The datetime implementation is quite interesting as well, and it provides a pretty comprehensive API which should address almost all use-cases for timekeeping in our language with a robust and easy-to-use API. As a bonus, and a little flex at how robust our design is, we’ve included support for Martian time. I’m very pleased with how both of these turned out.

use datetime;
use fmt;
use os;
use time::chrono;

export fn main() void = {
	const now = datetime::in(chrono::MTC, datetime::now());
	fmt::printf("Current Martian coordinated time: ")!;
	datetime::format(os::stdout, datetime::STAMP, &now)!;
	fmt::println()!;
};

Other recent improvements include support for signal handling, glob, aes-ni, and net::uri. Work has slowed down on cryptography — please get in touch if you’d like to help. Many readers will be happy to know that there are rumblings about possibly going public soon; after a couple more milestones we’ll be having a meeting to nail down the most urgent priorities before going public and then we’ll get this language into your hands to play with.

I also started a bittorrent daemon in this language, but it’s temporarily blocked until we sort out HTTP/TLS. So, moving right along: SourceHut news? Naturally I will leave most of it for the “what’s cooking” post, but I’ll offer you a little tease of what we’ve been working on: GraphQL. We landed support this month for GraphQL-native webhooks in todo.sr.ht, as well as some new improvements to the pages.sr.ht GQL API. hg.sr.ht is also now starting to see some polish put on its GraphQL support, and some research is underway on GraphQL Federation. Very soon we will be able to put a bow on this work.

That’s all for today! Thanks again for reading and for your ongoing support. I appreciate you!

Announcing git snail-mail

1 April 2022 at 00:00

You’ve heard of git-over-email thanks to git send-email — now you can enjoy git snail-mail: a new tool making it easier than ever to print out git commits on paper and mail them to your maintainers.

Running git snail-mail HEAD~2.. prepares the last two commits for post and sends them directly to the system’s default printer. Configuration options are available for changing printer settings, paper size, and options for faxing or printing envelopes automatically addressed to the maintainers based on address info stored in your git config. Be sure to help the maintainers review your work by including a return envelope and a stamp!

And for maintainers, code review has never been easier — just get out your red marker and write your feedback directly on the patch! When you’re ready to import the patch into your repository, just place it on your scanner and run git scan-mail.

A picture of a patch printed out on paper

At least, this is what I’d like to say, but I ended up cancelling the project before it was ready for April Fool’s. After my friend kline (a staffer at Libera Chat) came up with this idea, I actually did write a lot of the code! Git is mostly written in Perl, but I could not really rouse the enthusiasm for implementing this idea in Perl. I did the prototype in $secretlang instead, and got it mostly working, but decided not to try to do some sneaky half-private joke release while trying to maintain the secrecy of the language.

Essentially how it works is this: I have a TeX template for patches:

\documentclass{article}
\usepackage[
	a4paper,
	top=1cm,
	bottom=1cm,
	left=1cm,
	right=1cm,
]{geometry}
\usepackage{graphicx}
\usepackage{fancyvrb}
\pagenumbering{gobble}

\begin{document}

\section*{implement os::exec::peek\{,any\}}

From: Bor Grošelj Simić \textless{}bor.groseljsimic@telemach.net\textgreater{} \\
Date: Fri, 25 Feb 2022 01:46:13 +0100

\VerbatimInput{input.patch}

\newpage
Page 1 of 2 \\
\includegraphics[]{./output-1.png}

\newpage
Page 2 of 2 \\
\includegraphics[]{./output-2.png}

\end{document}

This is generated by my git snail-mail code and then run through pdflatex to produce a file like this. It pipes it into lp(1) to send it to your printer and ta-da!

I chose not to make the commit selection work like git send-email, because I think that’s one of the most confusing parts of git send-email. Instead I just use a standard revision selection, so to print a single commit, you just name it, and to print a range of commits you use “..”. Here’s a peek at how that works:

fn get_commits(
	data: *texdata,
	workdir: str,
	range: str,
) (void | exec::error | exec::exit_status | io::error | fs::error) = {
	const fmt = `--format=%H%x00%s%x00%aN%x00%aE%x00%aD`;

	const pipe = exec::pipe();
	const cmd = exec::cmd("git", "show", "-s", fmt, range)?;
	exec::addfile(&cmd, os::stdout_file, pipe.1);
	const proc = exec::start(&cmd)?;
	io::close(pipe.1);
	const pipe = pipe.0;
	defer io::close(pipe);

	static let buffer: [os::BUFSIZ]u8 = [0...];
	const pipe = &bufio::buffered(pipe, buffer[..], []);

	let path = path::init();

	for (true) {
		const line = match (bufio::scanline(pipe)?) {
		case let line: []u8 =>
			yield strings::fromutf8(line);
		case io::EOF =>
			break;
		};

		// XXX: This assumes git always does the thing
		const tok = strings::tokenize(line, "\0");
		let commit = commitdata {
			sha = strings::next_token(&tok) as str,
			subject = strings::next_token(&tok) as str,
			author = strings::next_token(&tok) as str,
			email = strings::next_token(&tok) as str,
			date = strings::next_token(&tok) as str,
			...
		};

		path::set(&path, workdir)!;
		path::add(&path, commit.sha)!;
		commit.diff = strings::dup(path::string(&path));
		append(data.commits, commit);

		const file = os::create(commit.diff, 0o644)?;
		defer io::close(file);
		const parent = strings::concat(commit.sha, "^");
		defer free(parent);

		const cmd = exec::cmd("git", "diff", parent, commit.sha)?;
		exec::addfile(&cmd, os::stdout_file, file);
		const proc = exec::start(&cmd)?;
		const status = exec::wait(&proc)?;
		exec::check(&status)?;
	};

	const status = exec::wait(&proc)?;
	exec::check(&status)?;
};

The --format argument provided to git at the start here allows me to change the format of git-show to use NUL delimited fields for easily picking out the data I want. Point of note: this is minimum-effort coding for a joke, so there’s a lot of missing error handling and other lazy design choices here.

Anyway, I would have liked to have rewritten this in Perl and pitched it to the git mailing list for inclusion upstream, but alas, after prototyping in $secretlang I could not bring myself to rewrite it in Perl, and the joke fell flat. Not every idea pans out, but they’re still worth trying, anyway. If you want to see some joke projects I’ve made that actually work, check these out:

  • shit: a git implementation in POSIX shell
  • bfbot: a working IRC bot written in brainfuck
  • classic6: a working Minecraft server written in 6 hours
  • evilpass: a password strength checker that detects password reuse
  • tw: a Wayland compositor for your terminal in 80 lines of “code”

Take care!

It is important for free software to use free software infrastructure

29 March 2022 at 00:00

Disclaimer: I founded a project and a company that focuses on free software infrastructure. I will elect not to name them in this post, and will only recommend solutions I do not have a vested interest in.

Free and open source software (FOSS) projects need infrastructure. Somewhere to host the code, to facilitate things like code review, end-user support, bug tracking, marketing, and so on. A common example of this is the “forge” platform: infrastructure which pitches itself as a one-stop shop for many of the needs of FOSS projects in one place, such as code hosting and review, bug tracking, discussions, and so on. Many projects will also reach for additional platforms to provide other kinds of infrastructure: chat rooms, forums, social media, and more.

Many of these needs have non-free, proprietary solutions available. GitHub is a popular proprietary code forge, and GitLab, the biggest competitor to GitHub, is partially non-free. Some projects use Discord or Slack for chat rooms, Reddit as a forum, or Twitter and Facebook for marketing, outreach, and support; all of these are non-free. In my opinion, relying on these platforms to provide infrastructure for your FOSS project is a mistake.

When your FOSS project chooses to use a non-free platform, you give it an official vote of confidence on behalf of your project. In other words, you lend some of your project’s credibility and legitimacy to the platforms you choose. These platforms are defined by network effects, and your choice is an investment in that network. I would question this investment in and of itself, and the wisdom of offering these platforms your confidence and legitimacy, but there’s a more concerning consequence of this choice as well: an investment in a non-free platform is also a divestment from the free alternatives.

Again, network effects are the main driver of success in these platforms. Large commercial platforms have a lot of advantages in this respect: large marketing budgets, lots of capital from investors, and the incumbency advantage. The larger the incumbent platform, the more difficult the task of competing with it becomes. Contrast this with free software platforms, which generally don’t have the benefit of large amounts of investment or big marketing budgets. Moreover, businesses are significantly more likely to play dirty to secure their foothold than free software projects are. If your own FOSS projects compete with proprietary commercial options, you should be very familiar with these challenges.

FOSS platforms are at an inherent disadvantage, and your faith in them, or lack thereof, carries a lot of weight. GitHub won’t lose sleep if your project chooses to host its code somewhere else, but choosing Codeberg, for example, means a lot to them. In effect, your choice matters disproportionately to the free platforms: choosing GitHub hurts Codeberg much more than choosing Codeberg hurts GitHub. And why should a project choose to use your offering over the proprietary alternatives if you won’t extend them the same courtesy? FOSS solidarity is important for uplifting the ecosystem as a whole.

However, for some projects, what ultimately matters to them has little to do with the benefit of the ecosystem as a whole, but instead considers only the potential for their project’s individual growth and popularity. Many projects choose to prioritize access to the established audience that large commercial platforms provide, in order to maximize their odds of becoming popular, and enjoying some of the knock-on effects of that popularity, such as more contributions.1 Such projects would prefer to exacerbate the network effects problem rather than risk some of its social capital on a less popular platform.

To me, this is selfish and unethical outright, though you may have different ethical standards. Unfortunately, arguments against most commercial platforms for any reasonable ethical standard are available in abundance, but they tend to be easily overcome by confirmation bias. Someone who may loudly object to the practices of the US Immigration and Customs Enforcement agency, for example, can quickly find some justification to continue using GitHub despite their collaboration with them. If this example isn’t to your tastes, there are many examples for each of many platforms. For projects that don’t want to move, these are usually swept under the rug.2

But, to be clear, I am not asking you to use inferior platforms for philosophical or altruistic reasons. These are only one of many factors which should contribute to your decision-making, and aptitude is another valid factor to consider. That said, many FOSS platforms are, at least in my opinion, functionally superior to their proprietary competition. Whether their differences are better for your project’s unique needs is something I must leave for you to research on your own, but most projects don’t bother with the research at all. Rest assured: these projects are not ghettos living in the shadow of their larger commercial counterparts, but exciting platforms in their own right which offer many unique advantages.

What’s more, if you need them to do something differently to better suit your project’s needs, you are empowered to improve them. You’re not subservient to the whims of the commercial entity who is responsible for the code, waiting for them to prioritize the issue or even to care about it in the first place. If a problem is important to you, that’s enough for you to get it fixed on a FOSS platform. You might not think you have the time or expertise to do so (though maybe one of your collaborators does), but more importantly, this establishes a mentality of collective ownership and responsibility over all free software as a whole — popularize this philosophy and it could just as easily be you receiving a contribution in a similar manner tomorrow.

In short, choosing non-free platforms is an individualist, short-term investment which prioritizes your project’s apparent access to popularity over the success of the FOSS ecosystem as a whole. On the other hand, choosing FOSS platforms is a collectivist investment in the long-term success of the FOSS ecosystem as a whole, driving its overall growth. Your choice matters. You can help the FOSS ecosystem by choosing FOSS platforms, or you can hurt the FOSS ecosystem by choosing non-free platforms. Please choose carefully.

Here are some recommendations for free software tools that facilitate common needs for free software projects:

* Self-hosted only
† Partially non-free, recommended only if no other solutions are suitable

P.S. If your project is already established on non-free platforms, the easiest time to revisit this choice is right now. It will only ever get more difficult to move as your project grows and gets further established on proprietary platforms. Please consider moving sooner rather than later.


  1. I should note here that I’m uncritically presenting “popularity” as a good thing for a project to have, which aligns, I think, with the thought processes of the projects I’m describing. However, the truth is not quite so. Perhaps a topic for another day’s blog post. ↩︎

  2. A particularly egregious example is the Ethical Source movement. I disagree with them on many grounds, but pertinent to this article is the fact that they publish (non-free) software licenses which advocate for anti-capitalist sentiments like worker rights and ethical judgements such as non-violence, doing so on… GitHub and Twitter, private for-profit platforms with a myriad of published ethical violations. ↩︎

  3. I have made the arguments from this post to Libera staff many times, but they still rely on GitHub, Twitter, and Facebook. They were one of the motivations for writing this post. I hope that they have a change of heart someday. ↩︎

The Netherlands so far

24 March 2022 at 00:00

I moved to Amsterdam in July 2021, and now that I’ve had some time to settle in I thought I’d share my thoughts on how it’s been so far. In short: I love it here!

I did end up finding housing through the hacker community thanks to my earlier post, which was a great blessing. I am renting an apartment from a member of the Techinc hacker space, which I have joined as a member myself. One of my biggest fears was establishing a new social network here in the Netherlands, but making friends here has been easy. Through this hacker space and through other connections besides, I have quickly met many wonderful, friendly, and welcoming people, and I have never felt like a stranger in a strange land. For this I am very grateful.

There are many other things to love about this place. One of my favorite things about Amsterdam is getting around by bike. In Philadelphia, travelling by bicycle is signing up for a death wish. In the Netherlands, 27% of all trips utilize a bike, and in Amsterdam it’s as much as 38%. Cyclists enjoy dedicated cycling-first infrastructure, such as bike lanes separated entirely from the roads and dedicated bike-only longer-distance artery roads. The city is designed to reduce points of conflict between bikes and cars, and even when they have to share the road they’re almost always designed to slow cars down and give bikes priority. The whole country is very flat, too, though Dutch people will be quick to tell you about The Hill in their neighborhood, which is always no more than 2 meters tall.

Getting around without a bike is super pleasant as well. I have my choice of bus, tram, metro, long-distance train, or even free ferries across the river, all paid for with the same auto-recharging NFC card for a low price. Every line runs frequent stops, so during the day you’re generally not waiting more than 5 minutes to be picked up and at night you’re probably not going to be waiting more than 15 minutes at popular stops. When it gets really late, though, you might wait as much as 30 minutes. The inter-city trains are amazing — I can show up at any major station without a plan and there’s probably a train heading to where I want to go in less than 10 minutes. Compared to Amtrak, it’s simply mind boggling.

Little things no one here even thinks about have left an impression on me, too. I see street cleaners out all of the time, in a little squad where workers use leaf blowers and brooms to sweep trash and dirt from the sidewalks and squares into the streets where sweepers come through to pick it up. The trash and recycling bins are regularly collected, and when one of them in my neighborhood broke, it was replaced within days. There are some areas where trash does tend to accumulate, though, such as near benches in parks.

Isolated accumulations of trash aside, the parks are great. There’s a lot more of them throughout the city than you’d get in a typical American city. I live close to two large parks, Rembrandtpark and Vondelpark, plus the smaller Erasmuspark, all of which are less than 5 minutes of cycling away. I like to cycle there on cool summer days to read by the lakes or other water features, or on one of the lawns. These parks also typically have a lot of large cycling-only roads which act as little cycling highways throughout the city, which means many of my cycling routes take me through nature even for intra-city travel. Several of the parks also have public gym equipment available, with which you can get a pretty good outdoor work-out for free.

The layout of the neighborhoods is quite nice as well. I have not just one but four grocery stores within walking distance of my house, and I visit one multiple times per week to pick up food, just a 3 or 4 minute walk away from my place. Thanks to the ease of accessing good (and cheap) produce and other ingredients, my diet has improved quite a bit — something I didn’t expect when I moved here. I can’t get everything I want, though: finding genuinely spicy chili peppers is a challenge.

The infamous Dutch bureaucracy is not as bad as people made it out to be. Going through the immigration process was pretty stressful — as any process which could end with being kicked out of the country might be — but it was actually fairly straightforward for the kind of visa I wanted to get. Public servants here are more helpful and flexible than their reputation suggests.

Something which is proving to be a bit of a challenge, however, is learning Dutch. This surprised me given my existing background in languages; I thought it would be pretty easy to pick up. I was able to quickly learn the basics, and I can conduct many everyday affairs in Dutch, but I found it difficult to progress beyond this point with self-study alone. I enrolled in a formal class, which will hopefully help bridge that gap.

I could go on — experiences outside of Amsterdam and throughout the rest of Europe, the vibes of the FOSS community and other communities I’ve met, serendipitously meeting people I knew in America who also moved to Europe, and so on — but I think I’ll stop here for this post. Every time I’ve paused to reflect on my relocation abroad, I’ve come away smiling. So far, so good. Hopefully that doesn’t start to wear off!

Status update, March 2022

15 March 2022 at 00:00

Greetings! The weather is starting to warm up again, eh? I’m a bit disappointed that we didn’t get any snow this winter. Yadda yadda insert intro text here. Let’s get down to brass tacks. What’s new this month?

I mainly focused on the programming language this month. I started writing a kernel, which you can see a screenshot of below. This screenshot shows a simulated page fault, demonstrating that we have a working interrupt handler, and also shows something mildly interesting: backtraces. I need to incorporate this approach into the standard library as well, so that we can dump useful stack traces on assertion failures and such. I understand that someone is working on DWARF support as well, so perhaps we’ll soon be able to translate function name + offset into a file name and line number.

A redacted screenshot of a kernel showing a simulated page fault

I also started working on a PNG decoder this weekend, which at the time of writing can successfully decode 77 of the 161 PNG test vectors. I am quite pleased with how the code turned out here: this library is a good demonstration of the strengths of the language. It has simple code which presents a comprehensive interface for the file format, has a strong user-directed memory management model, takes good advantage of features like slices, and makes good use of standard library features like compress::zlib and the I/O abstraction. I will supplement this later with a higher level image API which handles things like pixel format conversions and abstracting away format-specific details.

Expand for a sample from image::png
use bufio;
use bytes;
use compress::zlib;
use errors;
use io;

export type idat_reader = struct {
	st: io::stream,
	src: *chunk_reader,
	inflate: zlib::reader,
	decoder: *decoder,
};

// Returns a new IDAT reader for a [[chunk_reader]], from which raw pixel data
// may be read via [[io::read]]. The user must prepare a [[decoder]] object
// along with a working buffer to store the decoder state. For information about
// preparing a suitable decoder, see [[newdecoder]].
export fn new_idat_reader(
	cr: *chunk_reader,
	decoder: *decoder,
) (idat_reader | io::error) = {
	assert(cr.ctype == IDAT, "Attempted to create IDAT reader for non-IDAT chunk");
	return idat_reader {
		st = io::stream {
			reader = &idat_read,
			...
		},
		src = cr,
		inflate = zlib::decompress(cr)?,
		decoder = decoder,
	};
};

fn idat_read(
	st: *io::stream,
	buf: []u8,
) (size | io::EOF | io::error) = {
	let ir = st: *idat_reader;
	assert(ir.st.reader == &idat_read);
	let dec = ir.decoder;
	if (dec.buffered != 0) {
		return decoder_copy(dec, buf);
	};

	if (dec.filter is void) {
		const ft = match (bufio::scanbyte(&ir.inflate)) {
		case io::EOF =>
			return idat_finish(ir);
		case let b: u8 =>
			yield b: filter;
		};
		if (ft > filter::PAETH) {
			return errors::invalid;
		};
		dec.filter = ft;
	};

	// Read one scanline
	for (dec.read < len(dec.cr)) {
		match (io::read(&ir.inflate, dec.cr[dec.read..])?) {
		case io::EOF =>
			// TODO: The rest of the scanline could be in the next
			// IDAT chunk. However, if there is a partially read
			// scanline in the decoder and no IDAT chunk in the
			// remainder of the file, we should probably raise an
			// error.
			return idat_finish(ir);
		case let n: size =>
			dec.read += n;
		};
	};

	applyfilter(dec);
	dec.read = 0;
	dec.buffered = len(dec.cr);
	return decoder_copy(dec, buf);
};

fn idat_finish(ir: *idat_reader) (io::EOF | io::error) = {
	// Verify checksum
	if (io::copy(io::empty, ir.src)? != 0) {
		// Extra data following zlib stream
		return errors::invalid;
	};
	return io::EOF;
};

@test fn idat_reader() void = {
	const src = bufio::fixed(no_filtering, io::mode::READ);
	const read = newreader(&src) as reader;
	let chunk = nextchunk(&read) as chunk_reader;
	const ihdr = new_ihdr_reader(&chunk);
	const ihdr = ihdr_read(&ihdr)!;

	let pixbuf: []u8 = alloc([0...], decoder_bufsiz(&ihdr));
	defer free(pixbuf);
	let decoder = newdecoder(&ihdr, pixbuf);

	for (true) {
		chunk = nextchunk(&read) as chunk_reader;
		if (chunk_reader_type(&chunk) == IDAT) {
			break;
		};
		io::copy(io::empty, &chunk)!;
	};

	const idat = new_idat_reader(&chunk, &decoder)!;
	const pixels = io::drain(&idat)!;
	defer free(pixels);
	assert(bytes::equal(pixels, no_filtering_data));
};

In SourceHut news, I completed our migration to Alpine 3.15 this month after a brief outage, including an upgrade to our database server, which is upgraded on a less frequent cadance than the others. Thanks to Adnan’s work, we’ve also landed many GraphQL improvements, mainly refactorings and other like improvements, setting the stage for the next series of roll-outs. I plan on transitioning back from focusing on the language to focusing on SourceHut for the coming month, and I expect to see some good progress here.

That’s all I have to share for today. Until next time!

It takes a village

14 March 2022 at 00:00

As a prolific maintainer of several dozen FOSS projects, I’m often asked how I can get so much done, being just one person. The answer is: I’m not just one person. I have enjoyed the help of thousands of talented people who have contributed to these works. Without them, none of the projects I work on would be successful.

I’d like to take a moment to recognize and thank all of the people who have participated in these endeavours. If you’ve enjoyed any of the projects I’ve worked on, you owe thanks to some of these wonderful people. The following is an incomplete list of authors who have contributed to one or more of the projects I have started:

A Mak
A. M. Joseph
a3v
Aaron Bieber
Aaron Holmes
Aaron Ouellette
Abdelhakim Qbaich
absrd
Ace Eldeib
Adam Kürthy
Adam Mizerski
Aditya Mahajan
Aditya Srivastava
Adnan Maolood
Adrusi
ael-code
agr
Aidan Epstein
Aidan Harris
Ajay R
Ajay Raghavan
Alain Greppin
Aleksa Sarai
Aleksander Usov
Aleksei Bavshin
Aleksis
Alessio
Alex Cordonnier
Alex Maese
Alex McGrath
Alex Roman
alex wennerberg
Alex Xu
Alexander ‘z33ky’ Hirsch
Alexander Bakker
Alexander Dzhoganov
Alexander Harkness
Alexander Johnson
Alexander Taylor
Alexandre Oliveira
Alexey Yerin
Aljaz Gantar
Alynx Zhou
Alyssa Ross
Amin Bandali
amingin
Amir Yalon
Ammar Askar
Ananth Bhaskararaman
Anders
Andreas Rammhold
Andres Erbsen
Andrew Conrad
Andrew Jeffery
Andrew Leap
Andrey Kuznetsov
Andri Yngvason
Andy Dulson
andyleap
Anirudh Oppiliappan
Anjandev Momi
Anthony Super
Anton Gusev
Antonin Décimo
aouelete
apt-ghetto
ARaspiK
Arav K
Ariadna Vigo
Ariadne Conill
Ariel Costas
Ariel Popper
Arkadiusz Hiler
Armaan Bhojwani
Armin Preiml
Armin Weigl
Arnaud Vallette d’Osia
Arsen Arsenović
Art Wild
Arthur Gautier
Arto Jonsson
Arvin Ignaci
ascent12
asdfjkluiop
Asger Hautop Drewsen
ash lea
Ashkan Kiani
Ashton Kemerling
athrungithub
Atnanasi
Aviv Eyal
ayaka
azarus
bacardi55
barfoo1
Bart Pelle
Bart Post
Bartłomiej Burdukiewicz
bbielsa
BearzRobotics
Ben Boeckel
Ben Brown
Ben Burwell
Ben Challenor
Ben Cohen
Ben Fiedler
Ben Harris
Benjamin Cheng
Benjamin Halsted
Benjamin Lowry
Benjamin Riefenstahl
Benoit Gschwind
berfr
bilgorajskim
Bill Doyle
Birger Schacht
Bjorn Neergaard
Björn Esser
blha303
bn4t
Bob Ham
bobtwinkles
boos1993
Bor Grošelj Simić
boringcactus
Brandon Dowdy
BrassyPanache
Brendan Buono
Brendon Smith
Brian Ashworth
Brian Clemens
Brian McKenna
Bruno Pinto
bschacht
BTD Master
buffet
burrowing-owl
Byron Torres
calcdude84se
Caleb Bassi
Callum Brown
Calvin Lee
Cameron Nemo
camoz
Campbell Vertesi
Cara Salter
Carlo Abelli
Cassandra McCarthy
Cedric Sodhi
Chang Liu
Charles E. Lehner
Charlie Stanton
Charmander
chickendude
chr0me
Chris Chamberlain
Chris Kinniburgh
Chris Morgan
Chris Morris
Chris Vittal
Chris Waldon
Chris Young
Christoph Gysin
Christopher M. Riedl
Christopher Vittal
chtison
Clar Charr
Clayton Craft
Clément Joly
cnt0
coderkun
Cole Helbling
Cole Mickens
columbarius
comex
Connor Edwards
Connor Kuehl
Conrad Hoffmann
Cormac Stephenson
Cosimo Cecchi
cra0zy
crondog
Cuber
curiousleo
Cyril Levis
Cédric Bonhomme
Cédric Cabessa
Cédric Hannotier
D.B
dabio
Dacheng Gao
Damien Tardy-Panis
Dan ELKOUBY
Dan Robertson
Daniel Bridges
Daniel De Graaf
Daniel Eklöf
Daniel Gröber
Daniel Kessler
Daniel Kondor
Daniel Lockyer
Daniel Lublin
Daniel Martí
Daniel Otero
Daniel P
Daniel Sockwell
Daniel V
Daniel V.
Daniel Vidmar
Daniel Xu
Daniil
Danilo
Danilo Spinella
Danny Bautista
Dark Rift
Darksun
Dave Cottlehuber
David Arnold
David Blajda
David Eklov
David Florness
David Hurst
David Kraeutmann
David Krauser
David Zero
David96
db
dbandstra
dece
delthas
Denis Doria
Denis Laxalde
Dennis Fischer
Dennis Schridde
Derek Smith
Devin J. Pohly
Devon Johnson
Dhruvin Gandhi
Di Ma
Dian M Fay
Diane
Diederik de Haas
Dillen Meijboom
Dimitris Triantafyllidis
Dizigma
Dmitri Kourennyi
Dmitry Borodaenko
Dmitry Kalinkin
dogwatch
Dominik Honnef
Dominique Martinet
Donnie West
Dorota Czaplejewicz
dudemanguy
Dudemanguy911
Duncaen
Dylan Araps
earnest ma
Ed Younis
EdOverflow
EIREXE
Ejez
Ekaterina Vaartis
Eli Schwartz
Elias Naur
Eloi Rivard
elumbella
Elyes HAOUAS
Elyesa
emersion
Emerson Ferreira
Emmanuel Gil Peyrot
Enerccio
Erazem Kokot
Eric Bower
Eric Drechsel
Eric Engestrom
Eric Molitor
Erik Reider
ernierasta
espkk
Ethan Lee
Euan Torano
EuAndreh
Evan Allrich
Evan Hanson
Evan Johnston
Evan Relf
Eyal Sawady
Ezra
Fabian Geiselhart
Fabio Alessandro Locati
Falke Carlsen
Fazlul Shahriar
Felipe Cardoso Resende
Fenveireth
Ferdinand Bachmann
FICTURE7
Filip Sandborg
finley
Flakebi
Florent de Lamotte
florian.weigelt
Francesco Gazzetta
Francis Dinh
Frank Smit
Franklin “Snaipe” Mathieu
Frantisek Fladung
François Kooman
Frode Aannevik
frsfnrrg
ftilde
fwsmit
Gabriel Augendre
Gabriel Féron
gabrielpatzleiner
Galen Abell
Garrison Taylor
Gauvain ‘GovanifY’ Roussel-Tarbouriech
Gaël PORTAY
gbear605
Genki Sky
Geoff Greer
Geoffrey Casper
George Craggs
George Hilliard
ggrote
Gianluca Arbezzano
gilbus
gildarts
Giuseppe Lumia
Gokberk Yaltirakli
Graham Christensen
Greg Anders
Greg Depoire–Ferrer
Greg Farough
Greg Hewgill
Greg V
Gregory Anders
Gregory Mullen
grossws
Grégoire Delattre
Guido Cella
Guido Günther
Guillaume Brogi
Guillaume J. Charmes
György Kurucz
Gökberk Yaltıraklı
Götz Christ
Haelwenn (lanodan) Monnier
Half-Shot
Hans Brigman
Haowen Liu
Harish Krupo
Harry Jeffery
Heghedus Razvan
Heiko Carrasco
heitor
Henrik Riomar
Honza Pokorny
Hoolean
Hristo Venev
Hubert Hirtz
hugbubby
Hugo Osvaldo Barrera
Humm
Hummer12007
Ian Fan
Ian Huang
Ian Moody
Ignas Kiela
Igor Sviatniy
Ihor Kalnytskyi
Ilia Bozhinov
Ilia Mirkin
Ilja Kocken
Ilya Lukyanov
Ilya Trukhanov
inwit
io mintz
Isaac Freund
Issam E. Maghni
Issam Maghni
István Donkó
Ivan Chebykin
Ivan Fedotov
Ivan Habunek
Ivan Mironov
Ivan Molodetskikh
Ivan Tham
Ivoah
ixru
j-n-f
Jaanus Torp
Jack Byrne
jack gleeson
Jacob Young
jajo-11
Jake Bauer
Jakub Kopański
Jakub Kądziołka
Jamelly Ferreira
James D. Marble
James Edwards-Jones
James Mills
James Murphy
James Pond
James Rowe
James Turner
Jan Beich
Jan Chren
Jan Palus
Jan Pokorný
Jan Staněk
JanUlrich
Jared Baldridge
Jarkko Oranen
Jasen Borisov
Jason Francis
Jason Miller
Jason Nader
Jason Phan
Jason Swank
jasperro
Jayce Fayne
jdiez17
Jeff Kaufman
Jeff Martin
Jeff Peeler
Jeffas
Jelle Besseling
Jente Hidskes
Jeremy Hofer
Jerzi Kaminsky
JerziKaminsky
Jesin
jhalmen
Jiri Vlasak
jman
Joe Jenne
johalun
Johan Bjäreholt
Johannes Lundberg
Johannes Schramm
John Axel Eriksson
John Chadwick
John Chen
John Mako
john muhl
Jon Higgs
Jonas Große Sundrup
Jonas Hohmann
Jonas Kalderstam
Jonas Karlsson
Jonas Mueller
Jonas Platte
Jonathan Bartlett
Jonathan Buch
Jonathan Halmen
Jonathan Schleußer
JonnyMako
Joona Romppanen
Joram Schrijver
Jorge Maldonado Ventura
Jose Diez
Josef Gajdusek
Josh Holland
Josh Junon
Josh Shone
Joshua Ashton
Josip Janzic
José Expósito
José Mota
JR Boyens
Juan Picca
Julian P Samaroo
Julian Samaroo
Julien Moutinho
Julien Olivain
Julien Savard
Julio Galvan
Julius Michaelis
Justin Kelly
Justin Mayhew
Justin Nesselrotte
Justus Rossmeier
Jøhannes Lippmann
k1nkreet
Kacper Kołodziej
Kaleb Elwert
kaltinril
Kalyan Sriram
Karl Rieländer
Karmanyaah Malhotra
Karol Kosek
Kenny Levinsen
kevin
Kevin Hamacher
Kevin Kuehler
Kevin Sangeelee
Kiril Vladimiroff
Kirill Chibisov
Kirill Primak
Kiëd Llaentenn
KoffeinFlummi
Koni Marti
Konrad Beckmann
Konstantin Kharlamov
Konstantin Pospelov
Konstantinos Feretos
kst
Kurt Kartaltepe
Kurt Kremitzki
kushal
Kévin Le Gouguec
Lane Surface
Langston Barrett
Lars Hagström
Laurent Bonnans
Lauri
lbonn
Leon Henrik Plickat
Leszek Cimała
Liam Cottam
Linus Heckemann
Lio Novelli
ljedrz
Louis Taylor
Lubomir Rintel
Luca Weiss
Lucas F. Souza
Lucas M. Dutra
Ludovic Chabant
Ludvig Michaelsson
Lukas Lihotzki
Lukas Märdian
Lukas Wedeking
Lukas Werling
Luke Drummond
Luminarys
Luna Nieves
Lyle Hanson
Lyndsy Simon
Lyudmil Angelov
M Stoeckl
M. David Bennett
Mack Straight
madblobfish
manio143
Manuel Argüelles
Manuel Mendez
Manuel Stoeckl
Marc Grondin
Marcel Hellwig
Marcin Cieślak
Marco Sirabella
Marian Dziubiak
Marien Zwart
Marius Orcsik
Mariusz Bialonczyk
Mark Dain
Mark Stosberg
Markus Ongyerth
MarkusVolk
Marten Ringwelski
Martijn Braam
Martin Dørum
Martin Hafskjold Thoresen
Martin Kalchev
Martin Michlmayr
Martin Vahlensieck
Matias Lang
Matrefeytontias
matrefeytontias
Matt Coffin
Matt Critchlow
Matt Keeter
Matt Singletary
Matt Snider
Matthew Jorgensen
Matthias Beyer
Matthias Totschnig
Mattias Eriksson
Matías Lang
Max Bruckner
Max Leiter
Maxime “pep” Buquet
mbays
MC42
meak
Mehdi Sadeghi
Mendel E
Merlin Büge
Miccah Castorina
Michael Anckaert
Michael Aquilina
Michael Forney
Michael Struwe
Michael Vetter
Michael Weiser
Michael Weiss
Michaël Defferrard
Michał Winiarski
Michel Ganguin
Michele Finotto
Michele Sorcinelli
Mihai Coman
Mikkel Oscar Lyderik
Mikkel Oscar Lyderik Larsen
Milkey Mouse
minus
Mitchell Kutchuk
mliszcz
mntmn
mnussbaum
Moelf
morganamilo
Moritz Buhl
Mrmaxmeier
mteyssier
Mukundan314
muradm
murray
Mustafa Abdul-Kader
mwenzkowski
myfreeweb
Mykola Orliuk
Mykyta Holubakha
n3rdopolis
Naglis Jonaitis
Nate Dobbins
Nate Guerin
Nate Ijams
Nate Symer
Nathan Rossi
Nedzad Hrnjica
NeKit
nerdopolis
ngenisis
Nguyễn Gia Phong
Niccolò Scatena
Nicholas Bering
Nick Diego Yamane
Nick Paladino
Nick White
Nicklas Warming Jacobsen
Nicolai Dagestad
Nicolas Braud-Santoni
Nicolas Cornu
Nicolas Reed
Nicolas Schodet
Nicolas Werner
NightFeather
Nihil Pointer
Niklas Schulze
Nils ANDRÉ-CHANG
Nils Schulte
Nixon Enraght-Moony
Noah Altunian
Noah Kleiner
Noah Loomans
Noah Pederson
Noam Preil
Noelle Leigh
NokiDev
Nolan Prescott
Nomeji
Novalinium
novenary
np511
nrechn
NSDex
Nuew
nyorain
nytpu
Nícolas F. R. A. Prado
oharaandrew314
Oleg Kuznetsov
Oliver Leaver-Smith
oliver-giersch
Olivier Fourdan
Ondřej Fiala
Orestis Floros
Oscar Cowdery Lack
Ossi Ahosalmi
Owen Johnson
Paco Esteban
Parasrah
Pascal Pascher
Patrick Sauter
Patrick Steinhardt
Paul Fenwick
Paul Ouellette
Paul Riou
Paul Spooren
Paul W. Rankin
Paul Wise
Pedro Côrte-Real
Pedro L. Ramos
Pedro Lucas Porcellis
Peroalane
Peter Grayson
Peter Lamby
Peter Rice
Peter Sanchez
Phil Rukin
Philip K
Philip Woelfel
Philipe Goulet
Philipp Ludwig
Philipp Riegger
Philippe Pepiot
Philz69
Pi-Yueh Chuang
Pierre-Albéric TROUPLIN
Piper McCorkle
pixelherodev
PlusMinus0
PoroCYon
ppascher
Pranjal Kole
ProgAndy
progandy
Przemyslaw Pawelczyk
psykose
punkkeks
pyxel
Quantum
Quentin Carbonneaux
Quentin Glidic
Quentin Rameau
R Chowdhury
r-c-f
Rabit
Rachel K
Rafael Castillo
rage 311
Ragnar Groot Koerkamp
Ragnis Armus
Rahiel Kasim
Raman Varabets
Ranieri Althoff
Ray Ganardi
Raymond E. Pasco
René Wagner
Reto Brunner
Rex Hackbro
Ricardo Wurmus
Richard Bradfield
Rick Cogley
rinpatch
Robert Günzler
Robert Johnstone
Robert Kubosz
Robert Sacks
Robert Vollmert
Robin Jarry
Robin Kanters
Robin Krahl
Robin Opletal
Robinhuett
robotanarchy
Rodrigo Lourenço
Rohan Kumar
Roman Gilg
ROMB
Ronan Pigott
ronys
Roosembert Palacios
roshal
Roshless
Ross L
Ross Schulman
Rostislav Pehlivanov
rothair
RoughB Tier0
Rouven Czerwinski
rpigott
Rune Morling
russ morris
Ryan Chan
Ryan Dwyer
Ryan Farley
Ryan Gonzalez
Ryan Walklin
Rys Sommefeldt
Réouven Assouly
S. Christoffer Eliesen
s0r00t
salkin-mada
Sam Newbold
Sam Whited
SatowTakeshi
Sauyon Lee
Scoopta
Scott Anderson
Scott Colby
Scott Leggett
Scott Moreau
Scott O’Malley
Scott Stevenson
sdilts
Sebastian
Sebastian Krzyszkowiak
Sebastian LaVine
Sebastian Noack
Sebastian Parborg
Seferan
Sergeeeek
Sergei Dolgov
Sergi Granell
sergio
Seth Barberee
Seán C McCord
sghctoma
Shaw Vrana
Sheena Artrip
Silvan Jegen
Simon Barth
Simon Branch
Simon Ruderich
Simon Ser
Simon Zeni
Siva Mahadevan
skip-yell
skuzzymiglet
Skyler Riske
Slowpython
Sol Fisher Romanoff
Solomon Victorino
somdoron
Sorcus
sourque
Spencer Michaels
SpizzyCoder
sqwishy
Srivathsan Murali
Stacy Harper
Steef Hegeman
Stefan Rakel
Stefan Schick
Stefan Tatschner
Stefan VanBuren
Stefan Wagner
Stefano Ragni
Stephan Hilb
Stephane Chauveau
Stephen Brennan
Stephen Brown II
Stephen Gregoratto
Stephen Paul Weber
Steve Jahl
Steve Losh
Steven Guikal
Stian Furu Øverbye
Streetwalrus Einstein
Stuart Dilts
Sudipto Mallick
Sumner Evans
Syed Amer Gilani
sykhro
Tadeo Kondrak
Taiyu
taiyu
taminaru
Tamir Zahavi-Brunner
Tancredi Orlando
Tanguy Fardet
Tarmack
Taryn Hill
tastytea
tcb
Teddy Reed
Tero Koskinen
Tharre
Thayne McCombs
The Depressed Milkman
TheAvidDev
TheMachine02
Theodor Thornhill
thermitegod
Thiago Mendes
thirtythreeforty
Thomas Bracht Laumann Jespersen
Thomas Hebb
Thomas Jespersen
Thomas Karpiniec
Thomas Merkel
Thomas Plaçais
Thomas Schneider
Thomas Weißschuh
Thomas Wouters
Thorben Günther
thuck
Till Hofmann
Tim Sampson
Tim Schumacher
Timidger
Timmy Douglas
Timothée Floure
Ting-Wei Lan
tiosgz
toadicus
Tobi Fuhrimann
Tobias Blass
Tobias Langendorf
Tobias Stoeckmann
Tobias Wölfel
Tom Bereknyei
Tom Lebreux
Tom Ryder
Tom Warnke
tomKPZ
Tommy Nguyen
Tomáš Čech
Tony Crisci
Torstein Husebø
Trannie Carter
Trevor Slocum
TriggerAu
Tudor Brindus
Tudor Roman
Tuomas Siipola
tuomas56
Twan Wolthof
Tyler Anderson
Uli Schlachter
Umar Getagazov
unlimitedbacon
unraised
User Name
v44r
Valentin
Valentin Hăloiu
Vasilij Schneidermann
Versus Void
vexhack
Vijfhoek
vil
vilhalmer
Vincent Gu
Vincent Vanlaer
Vinko Kašljević
Vitalij
Vitalij Mikhailov
Vlad Pănăzan
Vlad-Stefan Harbuz
Vyivel
w1ke
Wagner Riffel
wagner riffel
Wai Hon Law
wb9688
wdbw
Whemoon Jang
Wieland Hoffmann
Wiktor Kwapisiewicz
wil
Will Daly
Will Hunt
willakat
Willem Sonke
William Casarin
William Culhane
William Durand
William Moorehouse
William Wold
willrandship
Willy Goiffon
Wolf480pl
Wouter van Kesteren
Xaiier
xdavidwu
xPMo
y0ast
Yacine Hmito
yankejustin
Yasar
Yash Srivastav
Yong Joseph Bakos
Yorick van Pelt
yuiiio
yuilib
Yury Krivopalov
Yuya Nishihara
Yábir Benchakhtir
Yábir García
Zach DeCook
Zach Sisco
Zachary King
Zandr Martin
zccrs
Zetok Zalbavar
Zie
Zoltan Kalmar
Zuzana Svetlikova
Éloi Rivard
Érico Rolim
Štěpán Němec
наб
حبيب الامين

Each of these is a distinct person, with their own lives and aspirations, who took time out of those lives to help build some cool software. I owe everything to these wonderful, talented, dedicated people. Thank you, everyone. Let’s keep up the good work, together.

Why am I building a programming language in private?

13 March 2022 at 00:00

As many readers are aware, I have been working on designing and implementing a systems programming language. This weekend, I’ve been writing a PNG file decoder in it, and over the past week, I have been working on a simple kernel with it as well. I’m very pleased with our progress so far — I recently remarked that this language feels like the language I always wanted, and that’s mission accomplished by any definition I care to consider.

I started the project on December 27th, 2019, just over two years ago, and I have kept it in a semi-private state since. Though I have not given its name in public, the git repos, mailing lists, and bug trackers use sourcehut’s “unlisted” state, so anyone who knows the URL can see them. The website is also public, though its domain name is also undisclosed, and it is full of documentation, tutorials, and resources for developers. People can find the language if they want to, though at this stage the community only welcomes contributors, not users or onlookers. News of the project nominally spreads by word of mouth and with calls-to-action on this blog, and to date a total of 30 people have worked on it over the course of 3,029 commits. It is a major, large-scale project, secret though it may be.

And, though we’ve invested a ton of work into this project together, it remains as-of-yet unfinished. There is no major software written in our language, though several efforts are underway. Several of our key goals have yet to be merged upstream, such as date/time support, TLS, and regular expressions, though, again, these efforts are well underway. Until we have major useful projects written in our language, we cannot be confident in our design, and efforts in these respects do a great deal to inform us regarding any changes which might be necessary. And some changes are already in the pipeline: we have plans to make several major revisions to the language and standard library design, which are certain to require changes in downstream software.

When our community is small and private, these changes are fairly easy to reckon with. Almost everyone who is developing a project based on our language is also someone who has worked on the compiler or standard library. Often, the person who implements a breaking change will also send patches to various downstreams updating them to be compatible with this change, for every extant software project written in the language. This is a task which can be undertaken by one person. We all understand the need for these changes, participate in the discussions and review the implementations, and have the expertise necessary to make the appropriate changes to our projects.

Moreover, all of these people are also understanding of the in-development nature of the project. All users of our language are equipped with the knowledge that they are expected to help fix the bugs they identify, and with the skills and expertise necessary to follow-up on this fact. We don’t have to think about users who stumble upon the project, spend a few hours trying to use it, then encounter an under-developed part of the language and run out of enthusiasm. We still lack DWARF support, so debugging is a chore. Sometimes the compiler segfaults or aborts without printing a useful error message. It’s a work-in-progress, after all. These kinds of problems can discourage new learners very fast, and often requires the developers to offer some of their precious bandwidth to provide expert assistance. With the semi-private model, there are, at any given time, a very small number of people involved who are new to the language and require more hands-on support to help them through their problems.

A new programming language is a major undertaking. We’re building one with an explicit emphasis on simplicity and we’re still not done after two years. When most people hear about the project for the first time, I don’t want them to find a half-completed language which they will fail to apply to their problem because it’s not fleshed out for their use-case. The initial release will have comprehensive documentation, a detailed specification, and stability guarantees, so it can be picked up and used in production by curious users on day one. I want to fast-forward to the phase where people study it to learn how to apply it to their problems, rather than to learn if they can apply it to their problems.

Even though it is under development in private, this project is both “free software” and “open source”, according to my strict understanding of those terms as defined by the FSF and OSI. “Open source” does not mean that the project has a public face. The compiler is GPL 3.0 licensed, the standard library is MPL 2.0, and the specification is CC-BY-ND (the latter is notably less free, albeit for good reasons), and these details are what matter. Every person who has worked on the project, and every person who stumbles upon it, possesses the right to lift the veil of secrecy and share it with the world. The reason they don’t is because I asked them not to, and we maintain a mutual understanding regarding the need for privacy.

On a few occasions, someone has discovered the project and taken it upon themselves to share it in public places, including Hacker News, Lemmy, and 4chan. While this is well within your rights, I ask you to respect our wishes and allow us to develop this project in peace. I know that many readers are excited to try it out, but please give us some time and space to ensure that you are presented with a robust product. At the moment, we anticipate going public early next year. Thank you for your patience.

Thank you for taking the time to read my thoughts as well. I welcome your thoughts and opinions on the subject: my inbox is always open. If you disagree, I would appreciate it if you reached out to me to discuss it before posting about the project online. And, if you want to get involved, here is a list of things we could use help with — email me to volunteer if you have both the time and expertise necessary:

  • Cryptography
  • Ports for new architectures or operating systems
  • Image & pixel formats/conversions
  • SQL database adapters
  • Signal handling
  • JSON parsing & encoding
  • Compression and decompression
  • Archive formats

If you definitely don’t want to wait for the language to go public, volunteering in one of our focus areas is the best way to get involved. Get in touch! If not, then the release will come around sooner than you think. We’re depending on your patience and trust.


Update 2022-03-14

This blog post immediately generated detailed discussions on Hacker News and Lobsters in which people posted the language’s website and started tearing into everything they don’t like about it.

It’s not done yet, and the current state of the language is not representative of the project goals. This post was not a marketing stunt. It was a heartfelt appeal to your better nature.

You know, I have a lot on my plate. All of it adds up to a lot of stress. I had hoped that you would help relieve some of that stress by taking me seriously when I explained my motivations and asked nicely for you to leave us be. I was wrong.

Open Source is defined by the OSI's Open Source Definition

1 March 2022 at 00:00

The Open Source Initiative (OSI) publishes a document called the Open Source Definition (OSD), which defines the term “open source”. However, there is a small minority of viewpoints within the software community which wishes that this were not so. The most concerning among them are those who wish open source was more commercially favorable to themselves, and themselves alone, such as companies like Elastic.

I disagree with this perspective, and I’d like take a few minutes today to explore several of the most common arguments in favor of this view, and explain why I don’t agree with them. One of the most frustrating complications in this discussion is the context of motivated reasoning (relevant xkcd): most people arguing in favor of an unorthodox definition of “open source” have a vested interest in their alternative view.1 This makes it difficult to presume good faith. For example, say someone wants to portray their software as open source even if it prohibits commercial use by third parties, which would normally disqualify it as such. Their interpretation serves to re-enforce their commercialization plans, providing a direct financial incentive not only for them to promote this definition of “open source”, but also for them to convince you that their interpretation is valid.

I find this argument to be fundamentally dishonest. Let me illustrate this with an analogy. Consider PostgreSQL. If I were to develop a new program called Postgres which was similar to PostgreSQL, but different in some important ways — let’s say it’s a proprietary, paid, hosted database service — that would be problematic. The industry understands that “Postgres” refers to the popular open source database engine, and by re-using their name I am diluting the brand of Postgres. It can be inferred that my reasoning for this comes from the desire to utilize their brand power for personal commercial gain. The terms “Postgres” and “PostgreSQL” are trademarked, but even if they were not, this approach would be dishonest and ethically wrong.

So too are the attempts to re-brand “open source” in a manner which is more commercially exploitable for an individual person or organization equally dishonest. The industry has an orthodox understanding of the meaning of “open source”, i.e. that defined by the Open Source Initiative, which is generally well-understood through the proliferation of software licenses which are compatible with the OSD. When a project describes itself as “open source”, this is a useful short-hand for understanding that the project adheres to a specific set of values and offers a specific set of rights to its users and contributors. When those rights are denied or limited, the OSD no longer applies and thus neither does the term “open source”. To disregard this in the interests of a financial incentive is dishonest, much like I would be dishonest for selling “cakes” and fulfilling orders with used car tires with “cake” written on them instead.

Critics of the OSD frequently point out that the OSI failed to register a trademark on the term “open source”, but a trademark is not necessary for this argument to hold. Language is defined by its usage, and the OSD is the popular usage of the term “open source”, without relying on the trademark system. The existence of a trademark on a specific term is not required for language which misuses that term to be dishonest.

As language is defined by its usage, some may argue that they are as entitled as anyone else to put forward an alternative usage. This is how language evolves. They are not wrong, though I might suggest that their alternative usage of “open source” requires a substantial leap in understanding which might not be as agreeable to those who don’t stand to benefit financially from that leap. Even so, I argue that the mainstream definition of open source, that forwarded by the OSI, is a useful term that is worth preserving in its current form. It is useful to quickly understand the essential values and rights associated with a piece of software as easily as stating that it is “open source”. I am not prepared to accept a new definition which removes or reduces important rights in service of your private financial interests.

The mainstream usage of “open source” under the OSD is also, in my opinion, morally just. You may feel a special relationship with the projects you start and invest into, and a sense of ownership with them, but they are not rightfully yours once you receive outside contributions. The benefit of open source is in the ability for the community to contribute directly to its improvements — and once they do, the project is the sum of your efforts and the efforts of the community. Thus, is it not right that the right to commercial exploitation of the software is shared with that community? In the absence of a CLA,2 contributors retain their copyright as well, and the software is legally jointly owned by the sum of its contributors. And beyond copyright, the success of the software is the sum of its code along with the community who learns about and deploys it, offers each other support, writes blog posts and books about it, sells consulting services for it, and together helps to popularize it. If you wish to access all of these benefits of the open source model, you must play by the open source rules.

It’s not surprising that this would become a matter of contention among certain groups within the industry. Open source is not just eating the world, but has eaten the world. Almost all software developed today includes substantial open source components. The open source brand is very strong, and there are many interests who would like to leverage that brand without meeting its obligations. But the constraints of the open source definition are important, played a critical role in the ascension of open source in the software market, and worth preserving into the future.

That’s not to say that there isn’t room for competing ideologies. If you feel that the open source model does not work for you, then that’s a valid opinion to hold. I only ask that you market your alternative model honestly by using a different name for it. Software for which the source code is available, but which does not meet the requirements of the open source definition, is rightfully called “source available”. If you want a sexier brand for it, make one! “Open core” is also popular, though not exactly the same. Your movement has as much right to success as the open source movement, but you need to earn that success independently of the open source movement. Perhaps someday your alternative model will supplant open source! I wish you the best of luck in this endeavour.

A previous version of this blog post announced that I had submitted my candidacy for the OSI board. Due to unforseen circumstances, I will be postponing my candidacy until the next election. I apologise for the confusion.


  1. Am I similarly biased? I also make my living from open source software, but I take special care to place the community’s interests above my own. I advocate for open source and free software principles in all software, including software I don’t personally use or benefit from, and in my own software I don’t ask contributors to sign a CLA — keeping the copyrights collectively held by the community at large, and limiting my access to commercialization to the same rules of open source that are granted to all contributors to and users of the software I use, write, and contribute to. ↩︎

  2. Such CLAs are also unjust in my view. Tools like the Developer Certificate of Origin are better for meeting the need to establish the legitimate copyright of open source software without denying rights to its community. ↩︎

Plaid is an evil nightmare product from Security Hell

19 February 2022 at 00:00

Plaid is a business that has built a widget that can be embedded in any of their customer’s websites which allows their customers to configure integrations with a list of third-party service providers. To facilitate this, Plaid pops up a widget on their customer’s domain which asks the end-user to type in their username and password for the third-party service provider. If necessary, they will ask for a 2FA code. This is done without the third party’s permission, presumably through a browser emulator and a provider-specific munging shim, and collects the user’s credentials on a domain which is operated by neither the third party nor by Plaid.

The third-party service provider in question is the end-user’s bank.

What the actual fuck!

Plaid has weighed on my mind for a while, though I might have just ignored them if they hadn’t been enjoying a sharp rise in adoption across the industry. For decades, we have stressed the importance of double-checking the domain name and the little TLS “lock” icon before entering your account details for anything. It is perhaps the single most important piece of advice the digital security community has tried to bring into the public conciousness. Plaid wants to throw out all of those years of hard work and ask users to enter their freaking bank credentials into a third-party form.

The raison d’être for Plaid is that banks are infamously inflexible and slow on the uptake for new technology. The status quo which Plaid aims to disrupt (ugh), at least for US bank account holders, involves the user entering their routing number and account number into a form. The service provider makes two small (<$1) deposits, and when they show up on the user’s account statement a couple of days later, the user confirms the amounts with the service provider, the service provider withdraws the amounts again, and the integration is complete. The purpose of this dance is to provide a sufficiently strong guarantee that the account holder is same person who is configuring the integration.

This process is annoying. Fixing it would require banks to develop, deploy, and standardize on better technology, and, well, good luck with that. And, honestly, a company which set out with the goal of addressing this problem ethically would have a laudable ambition. But even so, banks are modernizing around the world, and tearing down the pillars of online security in exchange for a mild convenience is ridiculous.

A convincing argument can be made that this platform violates the Computer Fraud and Abuse Act. Last year, they paid out $58M in one of many lawsuits for scraping and selling your bank data. Plaid thus joins the ranks of Uber, AirBnB, and others like them in my reckoning as a “move fast and break laws” company. This platform can only exist if they are either willfully malignant or grossly incompetent. They’ve built something that they know is wrong, and are hoping that they can outrun the regulators.

This behavior is not acceptable. This company needs to be regulated into the dirt and made an example of. Shame on you Plaid, and shame on everyone involved in bringing this product to market. Shame on their B2B customers as well, who cannot, such as they may like to, offload ethical due-diligence onto their vendors. Please don’t work for these start-ups. I hold employees complicit in their employer’s misbehavior. You have options, please go make the world a better place somewhere else.

Status update, February 2022

15 February 2022 at 00:00

Hello once again! Another month of free software development goes by with lots of progress in all respects.

I will open with some news about godocs.io: version 1.0 of our fork of gddo has been released! Big thanks to Adnan Maolood for his work on this. I’m very pleased that, following our fork, we were not only able to provide continuity for godoc.org, but also to simplify, refactor, and improve the underlying software considerably. Check out Adnan’s blog post for more details.

In programming language news, we have had substantial progress in many respects. One interesting project I’ve started is a Redis protocol implementation:

const conn = redis::connect()!;
defer redis::close(&conn);

fmt::println("=> SET foo bar EX 10")!;
redis::set(&conn, "foo", "bar", 10: redis::ex)!;

Another contributor has been working on expanding our graphics support, including developing a backend for glad to generate OpenGL bindings, and a linear algebra library ala glm for stuff like vector and matrix manipulation. Other new modules include a MIME database and encoding::base32. Cryptography progress continued with the introduction of XTS mode for AES, which is useful for full disk encryption implementations, but has slowed while we develop bigint support for future algorithms like RSA. I have also been rewriting the language introduction tutorial with a greater emphasis on practical usage.

Before we move on from the language project: I need your help! I am looking for someone to help develop terminal support. This is fairly straightforward, though laborsome: it involves developing libraries in our language which provide the equivalents of something like ncurses (or, better, libtickit), as well as the other end like libvterm offers. Please email me if you want to help.

In SourceHut news, we have hired our third full-time engineer: Conrad Hoffmann! Check out the blog post for details. The first major effort from Adnan’s NLnet-sponsored SourceHut work also landed yesterday, introducing GraphQL-native webhooks to git.sr.ht alongside a slew of other improvements. pages.sr.ht also saw some improvements that allow users to configure their site’s behavior more closely. Check out the “What’s cooking” post later today for all of the SourceHut news.

That’s all for today, thanks for reading!

Framing accessibility in broader terms

13 February 2022 at 00:00

Upon hearing the term “accessibility”, many developers call to mind the HTML ARIA attributes and little else. Those who have done some real accessibility work may think of the WCAG guidelines. Some FOSS developers1 may think of AT-SPI. The typical user of these accessibility features is, in the minds of many naive developers, a blind person. Perhaps for those who have worked with WCAG, a slightly more sophisticated understanding of the audience for accessibility tools may include users with a greater variety of vision-related problems, motor impairments, or similar needs.

Many developers2 frame accessibility in these terms, as a list of boxes to tick off, or specific industry tools which, when used, magically create an accessible product. This is not the case. In truth, a much broader understanding of accessibility is required to create genuinely accessible software, and because that understanding often raises uncomfortable questions about our basic design assumptions, the industry’s relationship with accessibility borders on willful ignorance.

The typical developer’s relationship with accessibility, if they have one at all, is mainly concerned with making web pages work with screen readers. Even considering this very narrow goal, most developers have an even narrower understanding of the problem, and end up doing a piss-poor job of it. In essence, the process of doing accessibility badly involves making a web page for a sighted user, then using ARIA tags to hide cosmetic elements, adding alt tags, and making other surface-level improvements for users of screen readers. If they’re serious, they may reach for the WCAG guidelines and do things like considering contrast, font choices, and animations as well, but all framed within the context of adding accessibility band-aids onto a UI designed for sighted use.

A key insight here is that concerns like font choice and contrast involve making changes which are apparent to “typical” users as well, but we’ll expand on that in a moment. Instead of designing for people like you and then patching it up until it’s semi-functional for people who are not like you, a wise developer places themselves into the shoes of the person they’re designing for and builds something which speaks their design language. For visually impaired users, this might mean laying out information in a more logical sense than in a spatial sense.

Importantly, accessibility also means understanding that there are many other kinds of users who have accessibility needs.

For instance, consider someone who cannot afford a computer as nice as the one your developers are using. When your Electron crapware app eats up 8G of RAM, it may be fine on your 32G developer workstation, but not so much for someone who cannot afford anything other than a used $50 laptop from eBay. Waking up the user’s phone every 15 minutes to check in with your servers isn’t very nice for someone using a 5-year-old phone with a dying battery. Your huge JavaScript bundle, unoptimized images, and always-on network requirements are not accessible to users who are on low-bandwidth mobile connections or have a data cap — you’re essentially charging poorer users a tax to use your website.

Localization is another kind of accessibility, and it requires more effort than running your strings through gettext. Users in different locales speak not only different natural languages, but different design languages. Users of right-to-left languages like Arabic don’t just reverse their strings but also the entire layout of the page. Chinese and Japanese users are more familiar with denser UIs than the typical Western user. And subtitles and transcripts are important for Deaf users, but also useful for users who are consuming your content in a second language.

Intuitiveness is another important detail. Not everyone understands what your icons mean, for a start. They may not have the motor skill to hold their mouse over the button and read the tool-tip, either, and might not know that they can do that in the first place! Reliance on unfamiliar design language in general is a kind of inaccessible design. Remember the “save” icon? 💾 Flashing banner ads are also inaccessible for users with ADHD, and if we’re being honest, for everyone else, too. Software which is not responsive on many kinds of devices (touch, mouse and keyboard, different screen sizes, aspect ratios, orientations) is not accessible. Software which requires the latest and greatest technologies to use (such as a modern web browser) is also not accessible.

Adequate answers to these problems are often expensive and uncomfortable, so no one wants to think about them. Social-media-esque designs which are deliberately addictive are not accessible, and also not moral. The mountain of gross abstractions on which much software is built is cheap, but causes it to suck up all the user’s resources (RAM, CPU, battery, etc) on 10-year-old devices.3 And ads are inaccessible by design, but good luck explaining that to your boss.

It is a fool’s errand to aim for perfect accessibility for all users, but we need to understand that our design choices are excluding people from using our tools. We need to design our software with accessibility in mind from the ground up, and with a broad understanding of accessibility that acknowledges that simple, intuitive software is the foundation of accessibility which works for everyone, including you and me — and not retroactively adding half-assed tools to fundamentally unusable software. I want UI designers to be thinking in these terms, and less in terms of aesthetic properties, profitable designs, and dark patterns. Design with empathy first.

As someone who works exclusively in free software, I have to acknowledge the fact that free software is pretty pathetic when it comes to accessibility. In our case, this does not generally come from the perverse incentives that cause businesses to cut costs or even deliberately undermine accessibility for profit,4 but instead comes from laziness (or, more charitably, lack of free time and enthusiasm), and generally from free software’s struggles to build software for people who are not like its authors. I think that we can change this. We do not have the profit motive, and we can choose to take pride in making better software for everyone. Let’s do better.


  1. Vanishingly few. ↩︎

  2. Including me, once upon a time. ↩︎

  3. Not to mention that the model of wasteful consumerism required to keep up with modern software is destroying the planet. ↩︎

  4. Though I am saddened to admit that many free software developers, after years of exposure to these dark patterns, will often unwittingly re-implement them in free software themselves without understanding their sinister nature. ↩︎

Free software licenses explained: MIT

7 February 2022 at 00:00

This is the first in a series of posts I intend to write explaining how various free and open source software licenses work, and what that means for you as a user or developer of that software. Today we’ll look at the MIT license, also sometimes referred to as the X11 or Expat license.

The MIT license is:

This means that the license upholds the four essential freedoms of free software (the right to run, copy, distribute, study, change and improve the software) and all of the terms of the open source definition (largely the same). Further more, it is classified on the permissive/copyleft spectrum as a permissive license, meaning that it imposes relatively few obligations on the recipient of the license.

The full text of the license is quite short, so let’s read it together:

The MIT License (MIT)

Copyright (c) <year> <copyright holders>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

The first paragraph of the license enumerates the rights which you, as a recipient of the software, are entitled to. It’s this section which qualifies the license as free and open source software (assuming the later sections don’t disqualify it). The key grants are the right to “use” the software (freedom 0), to “modify” and “merge” it (freedom 1), and to “distribute” and “sell” copies (freedoms 2 and 3), “without restriction”. We also get some bonus grants, like the right to sublicense the software, so you could, for instance, incorporate it into a work which uses a less permissive license like the GPL.

All of this is subject to the conditions of paragraph two, of which there is only one: you must include the copyright notice and license text in any substantial copies or derivatives of the software. Thus, the MIT license requires attribution. This can be achieved by simply including the full license text (copyright notice included) somewhere in your project. For a proprietary product, this is commonly hidden away in a menu somewhere. For a free software project, where the source code is distributed alongside the product, I often include it as a comment in the relevant files. You can also add your name or the name of your organization to the list of copyright holders when contributing to MIT-licensed projects, at least in the absence of a CLA.1

The last paragraph sets the expectations for the recipient, and it is very important. This disclaimer of warranty is ubiquitous in nearly all free and open source software licenses. The software is provided “as is”, which is to say, in whatever condition you found it in, for better or worse. There is no expectation of warranty (that is to say, any support you receive is from the goodwill of the authors and not from a contractual obligation), and there is no guarantee of “merchantability” (that you can successfully sell it), fitness for a particular purpose (that you can successfully use it to solve your problem), or noninfringement (such as with respect to relevant patents). That last detail may be of particular importance: the MIT license disclaims all liability for patents that you might infringe upon by using the software. Other licenses often address this case differently, such as Apache 2.0.

MIT is a good fit for projects which want to impose very few limitations on the use or reuse of the software by others. However, the permissibility of the license permits behaviors you might not like, such as creating a proprietary commercial fork of the software and selling it to others without supporting upstream. Note that the right to sell the software is an inalienable requirement of the free software and open source definitions, but other licenses can level the playing field a bit with strategies like copyleft and virality, on the other end of the permissibility spectrum. I’ll cover some relevant licenses in the future.


  1. You should not sign a CLA which transfers your copyright to the publisher. ↩︎

Implementing a MIME database in XXXX

28 January 2022 at 00:00

This is a (redacted) post from the internal blog of a new systems programming language we’re developing. The project is being kept under wraps until we’re done with it, so for this post I’ll be calling it XXXX. If you are interested in participating, send me an email with some details about your background and I’ll get back to you.

Recently, I have been working on implementing a parser for media types (commonly called MIME types) and a database which maps media types to file extensions and vice-versa. I thought this would be an interesting module to blog about, given that it’s only about 250 lines of code, does something useful and interesting, and demonstrates a few interesting xxxx concepts.

The format for media types is more-or-less defined by RFC 2045, specifically section 5.1. The specification is not great. The grammar shown here is copied and pasted from parts of larger grammars in older RFCs, RFCs which are equally poorly defined. For example, the quoted-string nonterminal is never defined here, but instead comes from RFC 822, which defines it but also states that it can be “folded”, which technically makes the following a valid Media Type:

text/plain;charset="hello
 world"

Or so I would presume, but the qtext terminal “cannot include CR”, which is the mechanism by which folding is performed in the first place, and… bleh. Let’s just implement a “reasonable subset” of the spec instead and side-step the whole folding issue.1 This post will first cover parsing media types, then address our second goal: providing a database which maps media types to file extensions and vice versa.

Parsing Media Types

So, here’s what we’re going to implement today: we want to parse the following string:

text/plain; charset=utf-8; foo="bar baz"

The code for that I came up with for is as follows:

// Parses a Media Type, returning a tuple of the content type (e.g.
// "text/plain") and a parameter parser object, or [[errors::invalid]] if the
// input cannot be parsed.
//
// To enumerate the Media Type parameter list, pass the type_params object into
// [[next_param]]. If you do not need the parameter list, you can safely discard
// the object. Note that any format errors following the ";" token will not
// cause [[errors::invalid]] to be returned unless [[next_param]] is used to
// enumerate all of the parameters.
export fn parse(in: str) ((str, type_params) | errors::invalid) = {
	const items = strings::cut(in, ";");
	const mtype = items.0, params = items.1;
	const items = strings::cut(mtype, "/");
	if (len(items.0) < 1 || len(items.1) < 1) {
		return errors::invalid;
	};
	typevalid(items.0)?;
	typevalid(items.1)?;
	return (mtype, strings::tokenize(params, ";"));
};

This function accepts a string as input, then returns a tagged union which contains either a tuple of (str, type_params), or a syntax error.

I designed this with particular attention to the memory management semantics. xxxx uses manual memory management, and if possible it’s desirable to avoid allocating any additional memory so that the user of our APIs remains in control of the memory semantics. The return value is a sub-string borrowed from the “text/plain” part, as well as a tokenizer which is prepared to split the remainder of the string along the “;” tokens.

Inspiration for strings::cut comes from Go:

$ xxxxdoc strings::cut
// Returns a string "cut" along the first instance of a delimiter, returning
// everything up to the delimiter, and everything after the delimiter, in a
// tuple.
//
//      strings::cut("hello=world=foobar", "=") // ("hello", "world=foobar")
//      strings::cut("hello world", "=")        // ("hello world", "")
//
// The return value is borrowed from the 'in' parameter.
fn cut(in: str, delim: str) (str, str);

And strings::tokenize works like so:

$ xxxxdoc strings::tokenize
// Returns a tokenizer which yields sub-strings tokenized by a delimiter.
//
//      let tok = strings::tokenize("hello, my name is drew", " ");
//      assert(strings::next_token(tok) == "hello,");
//      assert(strings::next_token(tok) == "my");
//      assert(strings::next_token(tok) == "name");
//      assert(strings::remaining_tokens(tok) == "is drew");
fn tokenize(s: str, delim: str) tokenizer;

The RFC limits the acceptable characters for the media type and subtype, which we test with the typevalid function.

The user of this module often only cares about the media type and not its type parameters, so the tokenizer can be safely abandoned on the stack to get cleaned up when the stack frame exits if they don’t care about the rest.

This is enough to write a little test:

@test fn parse() void = {
	const res = parse("text/plain")!;
	assert(res.0 == "text/plain");

	const res = parse("image/png")!;
	assert(res.0 == "image/png");

	const res = parse("application/svg+xml; charset=utf-8; foo=\"bar baz\"")!;
	assert(res.0 == "application/svg+xml");
};
$ xxxx test mime::parse
mime::parse..................................... OK

1 passed; 0 failed; 1 tests completed in 0.10s

To handle the type parameters in the third case, we add this function:

// Returns the next parameter as a (key, value) tuple from a [[type_params]]
// object that was prepared via [[parse]], void if there are no remaining
// parameters, and [[errors::invalid]] if a syntax error was encountered.
export fn next_param(in: *type_params) ((str, str) | void | errors::invalid) = {
	const tok = match (strings::next_token(in)) {
	case let s: str =>
		if (s == "") {
			// empty parameter
			return errors::invalid;
		};
		yield s;
	case void =>
		return;
	};

	const items = strings::cut(tok, "=");
	// The RFC does not permit whitespace here, but whitespace is very
	// common in the wild. ¯\_(ツ)_/¯
	items.0 = strings::trim(items.0);
	items.1 = strings::trim(items.1);
	if (len(items.0) == 0 || len(items.1) == 0) {
		return errors::invalid;
	};

	if (strings::hasprefix(items.1, "\"")) {
		items.1 = quoted(items.1)?;
	};

	return (items.0, items.1);
};

This returns a (key, value) tuple and advances to the next parameter, or returns void if there are no further parameters (or, if necessary, an error). This is pretty straightforward: the tokenizer prepared by parse is splitting the string on ; tokens, so we first fetch the next token. We then use strings::cut again to split it over the = token, and after a quick trim to fix another RFC oversight, we can return it to the caller. Unless it’s using this pesky quoted-string terminal, which is where our implementation starts to show its weaknesses:

fn quoted(in: str) (str | errors::invalid) = {
	// We have only a basic implementation of quoted-string. It has a couple
	// of problems:
	//
	// 1. The RFC does not define it very well
	// 2. The parts of the RFC which are ill-defined are rarely used
	// 3. Implementing quoted-pair would require allocating a new string
	//
	// This implementation should handle most Media Types seen in practice
	// unless they're doing something weird and ill-advised with them.
	in = strings::trim(in, '"');
	if (strings::contains(in, "\\")
			|| strings::contains(in, "\r")
			|| strings::contains(in, "\n")) {
		return errors::invalid;
	};
	return in;
};

I think this implementation speaks for itself. It could be a bit faster if we didn’t do 3 × O(n) strings::contains calls, but someone will send a patch if they care. The completed test for this is:

@test fn parse() void = {
	const res = parse("text/plain")!;
	assert(res.0 == "text/plain");

	const res = parse("image/png")!;
	assert(res.0 == "image/png");

	const res = parse("application/svg+xml; charset=utf-8; foo=\"bar baz\"")!;
	assert(res.0 == "application/svg+xml");
	const params = res.1;
	const param = next_param(&params)! as (str, str);
	assert(param.0 == "charset" && param.1 == "utf-8");
	const param = next_param(&params)! as (str, str);
	assert(param.0 == "foo" && param.1 == "bar baz");
	assert(next_param(&params) is void);

	assert(parse("hi") is errors::invalid);
	assert(parse("text/ spaces ") is errors::invalid);
	assert(parse("text/@") is errors::invalid);

	const res = parse("text/plain;charset")!;
	assert(res.0 == "text/plain");
	assert(next_param(&res.1) is errors::invalid);
};

The Media Type database

The second part of this module is the Media Type database. This comes in two parts:

  1. An internal database which is populated by xxxx modules. For example, an image::png module might register the “image/png” mimetype with the internal MIME database, similar to protocol registration for net::dial.
  2. A system-provided database, usually via /etc/mime.types, which is more comprehensive, but may not be available at runtime.

I plan on doing the second part later on, so for now we’ll just focus on the first; most of the interesting bits are there anyway.

Again, special consideration is given to memory management here. The essence of a good xxxx program or API design can be ascertained from how well it handles memory management. As such, I have set aside separate lists to handle statically allocated MIME info (such as those provided by image::png et al) versus the forthcoming dynamically-allocated system database.

// A pair of a Media Type and a list of file extensions associated with it. The
// extension list does not include the leading '.' character.
export type mimetype = struct {
	mime: str,
	exts: []str,
};

// List of media types with statically allocated fields (though the list itself
// is dynamically allocated).
let static_db: []mimetype = [];

// List of media types with heap-allocated fields, used when loading mime types
// from the system database.
let heap_db: []mimetype = [];

const builtins: [_]mimetype = [
	mimetype {
		mime = "text/plain",
		exts = ["txt"],
	},
	mimetype {
		mime = "text/x-xxxx", // redacted for public blog post
		exts = ["xx"],
	},
];

@init fn init() void = {
	register(builtins...);
};

@fini fn fini() void = {
	for (let i = 0z; i < len(heap_db); i += 1) {
		free(heap_db[i].mime);
		strings::freeall(heap_db[i].exts);
	};
	free(heap_db);
	free(static_db);
};

The register function will be used from @init functions like this one to register media types with the internal database. This code has minimal allocations for the internal database, but we do actually do some allocating here to store the “static_db” slice. In theory we could eliminate this by statically provisioning a small number of slots to store the internal database in, but for this use-case the trade-off makes sense. There are use-cases where the trade-off does not make as much sense, however. For example, here’s how the command line arguments are stored for your program in the “os” module:

// The command line arguments provided to the program. By convention, the first
// member is usually the name of the program.
export let args: []str = [];

// Statically allocate arg strings if there are few enough arguments, saves a
// syscall if we don't need it.
let args_static: [32]str = [""...];

@init fn init_environ() void = {
	if (rt::argc < len(args_static)) {
		args = args_static[..rt::argc];
		for (let i = 0z; i < rt::argc; i += 1) {
			args[i] = strings::fromc(rt::argv[i]);
		};
	} else {
		args = alloc([], rt::argc);
		for (let i = 0z; i < rt::argc; i += 1) {
			append(args, strings::fromc(rt::argv[i]));
		};
	};

};

@fini fn fini_environ() void = {
	if (rt::argc >= len(args_static)) {
		free(args);
	};
};

A similar approach is also used on yyp’s RISC-V kernel for storing serial devices without any runtime memory allocations.

The internal database is likely to be small, but the system database is likely to have a lot of media types and file extensions registered, so it makes sense to build out an efficient means of accessing them. For this purpose I have implemented a simple hash map. xxxx does not have a built-in map construct, nor generics. The design constraints of xxxx are closer to C than to anything else, and as such, the trade-offs for first-class maps are similar to C, which is to say that they don’t make sense with our design. However, this use-case does not call for much sophistication, so a simple map will suffice.

use hash::fnv;

def MIME_BUCKETS: size = 256;

// Hash tables for efficient database lookup by mimetype or extension
let mimetable: [MIME_BUCKETS][]*mimetype = [[]...];
let exttable: [MIME_BUCKETS][]*mimetype = [[]...];

// Registers a Media Type and its extensions in the internal MIME database. This
// function is designed to be used by @init functions for modules which
// implement new Media Types.
export fn register(mime: mimetype...) void = {
	let i = len(static_db);
	append(static_db, mime...);
	for (i < len(static_db); i += 1) {
		const item = &static_db[i];
		const hash = fnv::string(item.mime);
		let bucket = &mimetable[hash % len(mimetable)];
		append(bucket, item);

		for (let i = 0z; i < len(item.exts); i += 1) {
			const hash = fnv::string(item.exts[i]);
			let bucket = &exttable[hash % len(exttable)];
			append(bucket, item);
		};
	};
};

A fixed-length array of slices is a common approach to hash tables in xxxx. It’s not a great design for hash tables whose size is not reasonably predictable in advance or which need to be frequently resized and rehashed, but it is pretty easy to implement and provides sufficient performance for use-cases like this. A re-sizable hash table, or tables using an alternate hash function, or the use of linked lists instead of slices, and so on — all of this is possible if the use-case calls for it, but must be written by hand.

Finally, we implement the look-up functions, which are very simple:

// Looks up a Media Type based on the mime type string, returning null if
// unknown.
export fn lookup_mime(mime: str) const nullable *mimetype = {
	const hash = fnv::string(mime);
	const bucket = &mimetable[hash % len(mimetable)];
	for (let i = 0z; i < len(bucket); i += 1) {
		const item = bucket[i];
		if (item.mime == mime) {
			return item;
		};
	};
	return null;
};

// Looks up a Media Type based on a file extension, with or without the leading
// '.' character, returning null if unknown.
export fn lookup_ext(ext: str) const nullable *mimetype = {
	ext = strings::ltrim(ext, '.');
	const hash = fnv::string(ext);
	const bucket = &exttable[hash % len(exttable)];
	for (let i = 0z; i < len(bucket); i += 1) {
		const item = bucket[i];
		for (let j = 0z; j < len(item.exts); j += 1) {
			if (item.exts[j] == ext) {
				return item;
			};
		};
	};
	return null;
};

For the sake of completeness, here are the tests:

@test fn lookup_mime() void = {
	assert(lookup_mime("foo/bar") == null);

	const result = lookup_mime("text/plain");
	assert(result != null);
	const result = result: *mimetype;
	assert(result.mime == "text/plain");
	assert(len(result.exts) == 1);
	assert(result.exts[0] == "txt");

	const result = lookup_mime("text/x-xxxx");
	assert(result != null);
	const result = result: *mimetype;
	assert(result.mime == "text/x-xxxx");
	assert(len(result.exts) == 1);
	assert(result.exts[0] == "xx");
};

@test fn lookup_ext() void = {
	assert(lookup_ext("foo") == null);
	assert(lookup_ext(".foo") == null);

	const result = lookup_ext("txt");
	assert(result != null);
	const result = result: *mimetype;
	assert(result.mime == "text/plain");
	assert(len(result.exts) == 1);
	assert(result.exts[0] == "txt");

	const result = lookup_ext(".txt");
	assert(result != null);
	const result = result: *mimetype;
	assert(result.mime == "text/plain");

	const result = lookup_ext("xx");
	assert(result != null);
	const result = result: *mimetype;
	assert(result.mime == "text/x-xxxx");
	assert(len(result.exts) == 1);
	assert(result.exts[0] == "xx");
};

There you have it! I will later implement some code which parses /etc/mime.types in @init and fills up the heap_db slice, and this lookup code should work with it without any additional changes.


  1. Any time we implement a “reasonable subset” of a specification rather than the whole specification, I add the module to the list of modules likely to be moved out of the standard library and into a standalone module at some point prior to release. Another module on this list is our XML parser. ↩︎

Pine64 should re-evaluate their community priorities

18 January 2022 at 00:00

Pine64 has a really interesting idea: make cheap hardware with low margins, get it into the hands of the FOSS community, and let them come up with the software. No one has ever done this before, at least not on this scale, and it’s a really neat idea! Pine64 is doing a lot to support the FOSS community bringing up its hardware, but I’m afraid that I have to ask them to do a bit more.

There’s a handful of different roles that need to be filled in on the software side of things to get this ecosystem going. Ordered from most to least important, these are (broadly speaking) as follows:

  1. Implementing and upstreaming kernel drivers, u-Boot support, etc
  2. Building out a robust telephony stack for Linux
  3. Building a mobile user interface for Linux
  4. Maintaining distros that tie it all together

Again, this is ordered from most to least important, but in practice, the ecosystem prioritizes them in reverse. Pine64 themselves contribute no labor to any of these focus areas, and though they provide some funding, they provide it from the bottom of this list up, putting most of it into distros and very little into the kernel, bootloaders, or telephony. This is nice, but… why fund the distros at all? Distros are not the ones getting results in these focus areas. Their job is to distribute the results of community efforts.

Don’t get me wrong, the distros do an important job and they ought to get the funding they need, but this is just creating fragmentation in the ecosystem. As one example, we could be installing the Linux distribution of our choice on the Pinebook Pro using a standard aarch64 UEFI ISO installer, just like we do for any other laptop, if someone spent a couple of weeks upstreaming the last 6 patches to mainline Linux and put together a suitable u-Boot payload to flash on the SPI flash chip. But, instead of one working solution for everyone, we have 20+ Linux distros publishing Pine64-specific images to flash to microSD cards.

The favorites, which is apparently Manjaro,1 compete for funding and then spend it each according to their own discretion working on the same problems. If we instead spent it on the focus areas directly, then Manjaro and all of the other distros would benefit from this work for free. The telephony stack is equally important, and equally sharable between distros, but isn’t really getting any dedicated funding. You can’t have a phone without telephony. The mobile UI is also important, but it’s the easiest part to build, and a working phone with a shitty UI is better than a phone with a pretty UI that doesn’t work.

The work is getting done, to be fair, but it’s getting done very slowly. Many of the distros targetting Linux for mobile devices have people working on the important focus areas, but as a matter of necessity: to accomplish their goals when no one else is working on these problems, they had to become experts and divide their limited volunteer time between distro maintenance and software development. As a result, they’ve become experts with specific allegiances and incentives, and though there’s some patch sharing and collaboration between distros, it’s done informally across a dozen independent organizational structures with varying degrees of collaboration based on a model which was stapled onto an inherently backwards system of priorities. In a system with limited resources (funding, developer time, etc), these inefficiencies can be very wasteful.

After I got my hands on the PineNote hardware, I quickly understood that it was likely going to suffer even moreso from this problem. A course change is called for if Pine64 wants to maximize their odds of success with their current and future product lines. I think that the best strategic decision would be to hire just one full-time software developer to specifically focus on development and upstreaming in Linux mainline, u-Boot mainline, ModemManager, etc, and on writing docs, collaborating with other projects, and so on. This person should be figuring out how to get generalized software solutions to unlock the potential of the hardware, focusing on getting it to the right upstreams, and distributing these solutions to the whole ecosystem.

It’s awesome that Pine64 is willing to financially support the FOSS community around their devices, and as the ones actually selling the devices, they’re the only entity in this equation with the budget to actually do so. Pine64 is doing some really amazing work! However, a better financial strategy is called for here. Give it some thought, guys.


  1. I will go on the record as saying that Manjaro Linux is a bad Linux distribution and a bad place to send this money. They have a history of internal corruption, a record of questionable spending, and a plethora of technical issues and problematic behavior in the FOSS ecosystem. What limited budget there is to go around was wasted in their hands. ↩︎

Status update, January 2022

17 January 2022 at 00:00

Happy New Year! I had a lovely time in Amsterdam. No one had prepared me for the (apparently infamous) fireworks culture of the Netherlands. I thought it was really cool.

Our programming language continues to improve apace. Our cryptography suite now includes Argon2, Salsa20/XSalsa20, ChaCha20/XChaCha20, and Poly1305, and based on these functions we have added libsodium-style high-level cryptographic utilities for AEAD and key derivation, with stream encryption, message signing and verification, and key exchange coming soon. We have also laid out the priorities for future crypto work towards supporting TLS, and on the way we expect to have ed25519/x25519 and Diffie-Hellman added soon. Perhaps enough to implement an SSH client?

I also implemented an efficient path manipulation module for the standard library (something I would really have liked to have in C!), and progress continues on date/time support. We also have a new MIME module (just for Media Types, not all of MIME) and I expect a patch implementing net::uri to arrive in my inbox soon. I also finished up cmsg support (for sendmsg and recvmsg), which is necessary for the Wayland implementation I’m working on (and was a major pain in the ass). I spent some time working with another collaborator, who is developing a RISC-V kernel in our language, implementing a serial driver for the SiFive UART, plus improving the device tree loader and UEFI support.

One of the standard library contributors also wrote a side-project to implement Ray Tracing in One Weekend in our language:

A ray-traced image of many small, colorful balls with three large spheres, two
of which have a mirrored surface that reflects the other
balls.

In other words, language development has been very busy in the past few weeks. Another note: I have prepared a lightning talk for FOSDEM which talks about the backend that we’re using: qbe. Check it out!

In SourceHut news, we have brought on a new full-time contributor, Adnan Maolood, thanks to a generous grant from NLNet. We also have another full-time software engineer starting on February 1st (on our own dime), so I’m very much looking forward to that. Adnan will be helping us with the GraphQL work, and the new engineer will be working similarly to Simon and I on FOSS projects generally (and, hopefully, with GraphQL et al as well). Speaking of GraphQL, I’m putting the finishing touches on the todo.sr.ht writable API this week: legacy webhooks. These are nearly done, and following this we need to do the security review and acceptance testing, then we can ship. Adnan has been hard at work on adding GraphQL-native webhooks to git.sr.ht, which should also ship pretty soon.

That’s all for today. Thanks for reading! I’ll see you again in another month.

The RISC-V experience

15 January 2022 at 00:00

I’m writing to you from a Sway session on Alpine Linux, which is to say from a setup quite similar to the one I usually write blog posts on, save for one important factor: a RISC-V CPU.

I’ll state upfront that what I’m using is not a very practical system. What I’m going to describe is all of the impractical hacks and workarounds I have used to build a “useful” RISC-V system on which I can mostly conduct my usual work. It has been an interesting exercise, and it bodes well for the future of RISC-V, but for all practical purposes the promise of RISC-V still lives in tomorrow, not today.

In December of 2018, I wrote an article about the process of bootstrapping Alpine Linux for RISC-V on the HiFive Unleashed board. This board was essentially a crappy SoC built around a RISC-V CPU: a microSD slot, GPIO pins, an ethernet port, a little bit of RAM, and the CPU itself, in a custom form-factor.1 Today I’m writing this on the HiFive Unmatched, which is a big step up: it’s a Mini-ITX form factor (that is, it fits in a standardized PC case) with 16G of RAM, and the ethernet, microSD, and GPIO ports are complemented with a very useful set of additional I/O via two M.2 slots, a PCIe slot, and a USB 3 controller, plus an SPI flash chip. I have an NVMe drive with my root filesystem on it and an AMD Radeon Pro WX 2100 GPU installed. In form, it essentially functions like a standard PC workstation.

I have been gradually working on bringing this system up to the standards that I expect from a useful PC, namely that it can run upstream Alpine Linux with minimal fuss. This was not really possible on the previous SiFive hardware, but I have got pretty close on this machine. I had to go to some lengths to get u-Boot to provide a working UEFI environment,2 and I had to patch grub as well, but the result is that I can write a standard Alpine ISO to a USB stick, then boot it and install Alpine onto an NVMe normally, which then boots itself with UEFI with no further fiddling. I interact with it through three means: the on-board UART via a micro-USB cable (necessary to interact with u-Boot, grub, or the early Linux environment), or ethernet (once sshd is up), or with keyboard, mouse, and displays connected to the GPU.

Another of the standards I expect is that everything runs with upstream free software, perhaps with a few patches, but not from a downstream or proprietary tree. I’m pleased to report that I am running an unpatched mainline Linux 5.15.13 build. I am running mainline u-Boot with one patch to correct the name of a device tree node to match a change in Linux upstream. I have a patched grub build, but the relevant patches have been proposed for grub upstream. I have a spattering of patches applied to a small handful of userspace programs and libraries, but all of them only call for one or two patches applied to the upstream trees. Overall, this is quite good for something this bleeding edge — my Pinephone build is worse.

I have enclosed the system in a mini-ITX case and set it down on top of my usual x86_64 workstation, then moved a few of my peripherals and displays over to it to use it as my workstation for the day.3 I was able to successfully set up almost all of my standard workstation loadout on it, with some notable exceptions. Firefox is the most painful omission — bootstrapping Rust is an utter nightmare4 and no one has managed to do it for Alpine Linux riscv64 yet (despite many attempts and lots of hours wasted), so anything which depends on it does not work. librsvg is problematic for the same reason; I had to patch a number of things to be configured without it. For web browsing I am using visurf, which is based on Netsurf, and which works for many of the lightweight websites that I generally prefer to use, but not for most others. For instance, I was unable to address an issue that was raised on GitLab today because I cannot render GitLab properly on this browser. SourceHut mostly works, of course, but it’s not exactly pleasant — I still haven’t found time to improve the SourceHut UI for NetSurf.

A picture of two computers stacked on on top of the other.
The lower computer is my typical x86_64 workstation, and the upper computer is the RISC-V machine. The USB ports on the side are not connected to the board, so I pulled a USB extension cord around from the back. This is mainly useful for rapid iteration when working on a RISC-V kernel that a colleague has been developing using our new programming language. I can probably get netboot working later, but this works for now.

Complicating things is the fact that my ordinary workstation uses two 4K displays. For example, my terminal emulator of choice is foot, but it uses CPU rendering and the 4K window is noticeably sluggish. Alacritty, which renders on the GPU, would probably fare better — but Rust spoils this again. I settled for st, which has acceptable performance (perhaps in no small part thanks to being upscaled from 1080p on this setup). visurf also renders on the CPU and is annoyingly slow; as a workaround I have taken to resizing the window to be much smaller while actively navigating and then scaling it back up to full size to read the final page.

CPU-bound programs can be a struggle. However, this system has a consumer workstation GPU plugged into its PCIe slot. Any time I can get the GPU to pick up the slack, it works surprisingly effectively. For example, I watched Dune (2021) today in 4K on this machine — buttery smooth, stunningly beautiful 4K playback — a feat that my Pinebook Pro couldn’t dream of. The GPU has a hardware HEVC decoder, and mpv and Sway can use dmabufs such that the GPU decodes and displays each frame without it ever having to touch the CPU, and meanwhile the NVMe is fast enough to feed it data at a suitable bandwidth. A carefully configured obs-studio is also able to record my 4K display at 30 FPS and encode it on the GPU with VAAPI with no lag, something that I can’t even do on-CPU on x86_64 very reliably. The board does not provide onboard audio, but being an audiophile I have a USB DAC available that works just fine.

I was able to play Armagetron Advanced at 120+ FPS in 4K, but that’s not exactly a demanding game. I also played SuperTuxKart, a more demanding game, at 1080p with all of the settings maxed out at a stable 30 FPS. I cannot test any commercial games, since I’m reasonably certain that there are no proprietary games that distribute a riscv64 build for Linux. If Ethan Lee is reading this, please get in touch so that we can work together on testing out a Celeste build.

My ordinary workday needs are mostly met on this system. For communication, my mail setup with aerc and postfix works just fine, and my normal Weechat setup works great for IRC.5 Much like any other day, I reviewed a few patches and spent some time working on a shell I’ve been writing in our new programming language. The new language is quite performant, so no issues there. I think if I had to work on SourceHut today, it might be less pleasant to work with Python and Go, or to work on the web UI without a performant web browser. Naturally, browsing Geminispace with gmnlm works great.

So, where does this leave us? I have unusually conservative demands of my computers. Even on high-end, top-of-the-line systems, I run a very lightweight environment, and that’s the way I like it. Even so, my modest demands stress the limits of this machine. If I relied more on a web browser, or on more GUI applications, or used a heavier desktop environment, or heavier programming environments, I would not be able to be productive on this system. Tomorrow, I expect to return to my x86_64 machine as my daily workstation and continue to use this machine as I have before, for RISC-V development and testing over serial and SSH. There are few use-cases for which this hardware, given its limitations, is adequate.

Even so, this is a very interesting system. The ability to incorporate more powerful components like DDR4 RAM, PCIe GPUs, NVMe storage, and so on, can make up for the slow CPU in many applications. Though many use-cases for this system must be explained under strict caveats, one use-case it certainly offers is a remarkably useful system with which to advance the development of the RISC-V FOSS ecosystem. I’m using it to work on Alpine Linux, on kernel hacking projects, compiler development, and more, on a CPU that is designed in adherence to an open ISA standard and runs on open source firmware. This is a fascinating product that promises great things for the future of RISC-V as a platform.


  1. Plus an expansion slot which was ultimately entirely useless. ↩︎

  2. I have u-Boot installed on a microSD card which the firmware boots to, which then runs grub, which runs Linux. I could theoretically install u-Boot to the SPI Flash and then I would not have to use a microSD card for this process, but my initial attempts were not met with success and I didn’t debug it any further. I think other people have managed to get it working, though, and someone is working on making Alpine handle this for you. In future hardware from SiFive I hope that they will install a working u-Boot UEFI environment on the SPI before shipping so that you can just install standard ISOs from a flash drive like you would with any other PC. ↩︎

  3. I use this machine fairly often for RISC-V testing, particularly for the new programming language I’m working on, but I usually just SSH into it instead of connecting my displays and peripherals to it directly. ↩︎

  4. Incidentally, my new language can be fully bootstrapped on this machine in 272 seconds, including building and running the test suite. For comparison, it takes about 10 seconds on my x86_64 workstation. Building LLVM on this machine, let alone Rust, takes upwards of 12+ hours. You can cross-compile it, but this is difficult and it still takes ages, and it’s so complicated and brittle that you’re going to waste a huge amount of time troubleshooting between every attempt. ↩︎

  5. There’s not a snowball’s chance in hell of using Discord or Slack on this system, for the record. ↩︎

Breaking down a small language design proposal

30 December 2021 at 00:00

We are developing a new systems programming language. The name is a secret, so we’ll call it xxxx instead. In xxxx, we have a general requirement that all variables must be initialized. This is fine for the simple case, such as “let x: int = 10”. But, it does not always work well. Let’s say that you want to set aside a large buffer for I/O:

let x: [1024]int = [0, 0, 0, 0, 0, // ...

This can clearly get out of hand. To address this problem, we added the “…” operator:

let x: [1024]int = [0...];
let y: *[1024]int = alloc([0...]);

This example demonstrates both stack allocation of a buffer and heap allocation of a buffer initialized with 1024 zeroes.1 This “…” operator neatly solves our problem. However, another problem occurs to me: what if you want to allocate a buffer of a variable size?

In addition to arrays, xxxx supports slices, which stores a data pointer, a length, and a capacity. The data pointer refers to an array whose length is equal to or greater than “capacity”, and whose values are initialized up to “length”. We have additional built-ins, “append”, “insert”, and “delete”, which can dynamically grow and shrink a slice.

let x: []int = [];
defer free(x);
append(x, 1, 2, 3, 4, 5); // x = [1, 2, 3, 4, 5]
delete(x[..2]);           // x = [3, 4, 5]
insert(x[0], 1, 2);       // x = [1, 2, 3, 4, 5]

You can also allocate a slice whose capacity is set to an arbitrary value, but whose length is only equal to the number of initializers you provide. This is done through a separate case in the “alloc” grammar:

use types;

let x: []int = alloc([1, 2, 3], 10);
assert(len(x) == 3);
assert((&x: *types::slice).capacity == 10);

This is useful if you know how long the slice will eventually be, so that you can fill it with “append” without re-allocating (which could be costly otherwise). However, setting the capacity is not the same thing as setting the length: all of the items between the length and capacity are uninitialized. How do we zero-initialize a large buffer in the heap?

Until recently, you simply couldn’t. You had to use a rather bad work-around:

use rt;

let sz: size = 1024;
let data: *[*]int = rt::malloc(sz * size(int)); // [*] is an array of undefined length
let x: []int = data[..sz];

This is obviously not great. We lose type safety, the initialization guarantee, and bounds checking, and we add a footgun (multiplying by the member type size), and it’s simply not very pleasant to use. To address this, we added the following syntax:

let sz: size = 1024;
let x: []int = alloc([0...], sz);

Much better! Arriving at this required untangling a lot of other problems that I haven’t mentioned here, but this isn’t the design I want to focus on for this post. Instead, there’s a new question this suggests: what about appending a variable amount of data to a slice? I want to dig into this problem to explore some of the concerns we think about when working on the language design.

The first idea I came up with was the following:

let x: []int = [];
append(x, [0...], 10);

This would append ten zeroes to “x”. This has a problem, though. Consider our earlier example of “append”:

append(x, 1, 2, 3);

The grammar for this looks like the following:2

A screenshot of the language spec showing the grammar for append expressions.

So, the proposed “append(x, [0…], 10)” expression is parsed like this:

slice-mutation-expression: append
    object-selector: x
    append-items:
        [0...]
        10

In other words, it looks like “append the values [0…] and 10 to x”. This doesn’t make sense, but we don’t know this until we get to the type checker. What it really means is “append ten zeroes to x”, and we have to identify this case in the type checker through, essentially, heuristics. Not great! If we dig deeper into this we find even more edge cases, but I will spare you from the details.

So, let’s consider an alternative design:

append(x, [1, 2, 3]);   // Previously append(x, 1, 2, 3);
append(x, [0...], 10);  // New feature

The grammar for this is much better:

A screenshot of the revised grammar for this design.

Now we can distinguish between these cases while parsing, so the first example is parsed as:

append-expression
    object-selector: x
    expression: [1, 2, 3]   // Items to append

The second is parsed as:

append-expression
    object-selector: x
    expression: [0...]    // Items to append
    expression: 10        // Desired length

This is a big improvement, but it comes with one annoying problem. The most common case for append in regular use in xxxx is appending a single item, and this case has worsened thanks to this change:

append(x, [42]);  // Previously append(x, 42);

In fact, appending several items at once is exceptionally uncommon: there are no examples of it in the standard library. We should try to avoid making the common case worse for the benefit of the uncommon case.

A pattern we do see in the standard library is appending one slice to another, which is a use-case we’ve ignored up to this point. This use-case looks something like the following:

append(x, y...);

Why don’t we lean into this a bit more?

let x: []int = [];
append(x, 42);              // x = [42]
append(x, [1, 2, 3]...);    // x = [42, 1, 2, 3]
append(x, [0...], 6);       // x = [42, 1, 2, 3, 0...]

Using the append(x, y...) syntax to generally handle appending several items neatly solves all of our problems. We have arrived at design which:

  • Is versatile and utilitarian
  • Addresses the most common cases with a comfortable syntax
  • Is unambiguous at parse time without type heuristics

I daresay that, in addition to fulfilling the desired new feature, we have improved the other cases as well. The final grammar for this is the following:

Formal grammar showing the final state of the design proposal

If you’re curious to see more, I’ve extracted the relevant page of the specification for you to read: download it here. I hope you found that interesting and insightful!

Note: Much of these details are subject to change, and we have future improvements planned which will affect these features — particularly with respect to handling allocation failures. Additionally, some of the code samples were simplified for illustrative purposes.


  1. You can also use static allocation, which is not shown here. ↩︎

  2. Disregard the second case of “append-values”; it’s not relevant here. ↩︎

Please don't use Discord for FOSS projects

28 December 2021 at 00:00

Six years ago, I wrote a post speaking out against the use of Slack for the instant messaging needs of FOSS projects. In retrospect, this article is not very good, and in the years since, another proprietary chat fad has stepped up to bat: Discord. It’s time to revisit this discussion.

In short, using Discord for your free software/open source (FOSS) software project is a very bad idea. Free software matters — that’s why you’re writing it, after all. Using Discord partitions your community on either side of a walled garden, with one side that’s willing to use the proprietary Discord client, and one side that isn’t. It sets up users who are passionate about free software — i.e. your most passionate contributors or potential contributors — as second-class citizens.

By choosing Discord, you also lock out users with accessibility needs, for whom the proprietary Discord client is often a nightmare to use.1 Users who cannot afford new enough hardware to make the resource-intensive client pleasant to use are also left by the wayside. Choosing Discord is a choice that excludes poor and disabled users from your community. Users of novel or unusual operating systems or devices (i.e. innovators and early adopters) are also locked out of the client until Discord sees fit to port it to their platform. Discord also declines service to users in countries under US sanctions, such as Iran. Privacy-concious users will think twice before using Discord to participate in your project, or will be denied outright if they rely on Tor or VPNs. All of these groups are excluded from your community.

These problems are driven by a conflict of interest between you and Discord. Ownership over your chat logs, the right to set up useful bots, or to moderate your project’s space according to your discretion; all of these are rights reserved by Discord and denied to you. The FOSS community, including users with accessibility needs or low-end computing devices, are unable to work together to innovate on the proprietary client, or to build improved clients which better suit their needs, because Discord insists on total control over the experience. Discord seeks to domesticate its users, where FOSS treats users as peers and collaborators. These ideologies are fundamentally in conflict with one another.

You are making an investment when you choose to use one service over another. When you choose Discord, you are legitimizing their platform and divesting from FOSS platforms. Even if you think they have a bigger reach and a bigger audience,2 choosing them is a short-term, individualist play which signals a lack of faith in and support for the long-term goals of the FOSS ecosystem as a whole. The FOSS ecosystem needs your investment. FOSS platforms generally don’t have access to venture capital or large marketing budgets, and are less willing to use dark patterns and predatory tactics to secure their market segment. They need your support to succeed, and you need theirs. Why should someone choose to use your FOSS project when you refused to choose theirs? Solidarity and mutual support is the key to success.

There are great FOSS alternatives to Discord or Slack. SourceHut has been investing in IRC by building more accessible services like chat.sr.ht. Other great options include Matrix and Zulip. Please consider these services before you reach for their proprietary competitors.

Perceptive readers might have noticed that most of these arguments can be generalized. This article is much the same if we replace “Discord” with “GitHub”, for instance, or “Twitter” or “YouTube”. If your project depends on proprietary infrastructure, I want you to have a serious discussion with your collaborators about why. What do your choices mean for the long-term success of your project and the ecosystem in which it resides? Are you making smart investments, or just using tools which are popular or that you’re already used to?

If you use GitHub, consider SourceHut3 or Codeberg. If you use Twitter, consider Mastodon instead. If you use YouTube, try PeerTube. If you use Facebook… don’t.

Your choices matter. Choose wisely.


  1. Discord had to be sued to take this seriously. Updated at 2021-12-28 15:00 UTC: I asked a correspondent of mine who works on accessibility to comment:

    I’ve tried Discord on a few occasions, but haven’t seriously tried to get proficient at navigating it with a screen reader. I remember finding it cumbersome to move around, but it’s been long enough since the last time I tried it, a few months ago, that I couldn’t tell you exactly why. I think the general problem, though, is that the UI of the desktop-targeted web app is complex enough that trying to move through it an element at a time is overwhelming. I found that the same was true of Slack and Zulip. I haven’t tried Matrix yet. Of course, IRC is great, because there’s a wide variety of clients to choose from.

    However, you shouldn’t take my experience as representative, even though I’m a developer working on accessibility. As you may recall, I have some usable vision, and I often use my computer visually, though I do depend on a screen reader when using my phone. I didn’t start routinely using a GUI screen reader until around 2004, when I started writing a screen reader as part of my job. And that screen reader was targeted at beginners using simple UIs. So it’s possible that I never really mastered more advanced screen reader usage.

    What I can tell you is that, to my surprise, Discord’s accessibility has apparently improved in recent years, and more blind people are using it now. One of my blind friends told me that most Discord functionality is very accessible and several blind communities are using it. He also told me about a group of young blind programmers who are using Discord to discuss the development of a new open-source screen reader to replace the current Orca screen reader for GNOME. ↩︎
  2. Discord appears to inflate its participation numbers compared to other services. It shows all users who have ever joined the server, rather than all users who are actively using the server. Be careful not to optimize for non-participants when choosing your tools. ↩︎

  3. Disclaimer: I am the founder of SourceHut. ↩︎

Please use me as a resource

25 December 2021 at 00:00

I write a lot of blog posts about my ideas,1 some of which are even good ideas. Some of these ideas stick, and many readers have attempted to put them into practice, taking on challenges like starting a business in FOSS or stepping up to be leaders in their communities. It makes me proud to see the difference you’re making, and I’m honored to have inspired many of you.

I’m sitting here on my soapbox shouting into the void, but I also want to work with you one-on-one. Here are some things people have reached out to me for:

  • Pitching their project/business ideas for feedback
  • Sharing something they’re proud of
  • Cc’ing me in mailing list discussions, GitHub/GitLab threads, etc, for input
  • Clarifying finer points in my blog posts
  • Asking for feedback on drafts of their own blog posts
  • Offering philosophical arguments about FOSS
  • Asking for advice on dealing with a problem in their community

I have my own confidants that I rely on for these same problems. None of us goes it alone, and for this great FOSS experiment to succeed, we need to rely on each other.

I want to be personally available to you. My email address is sir@cmpwn.com. I read every email I receive, and try to respond to most of them, though it can sometimes take a while. Please consider me a resource for your work in FOSS. I hope I can help!


  1. 84 in 2021, and counting. Wow! ↩︎

Sustainable creativity in a world without copyright

23 December 2021 at 00:00

I don’t believe in copyright. I argue that we need to get rid of copyright, or at least dramatically reform it. The public domain has been stolen from us, and I want it back. Everyone reading this post has grown up in a creative world defined by capitalism, in which adapting and remixing works — a fundamental part of the creative process — is illegal. The commons is dead, and we suffer for it. But, this is all we’ve ever known. It can be difficult to imagine a world without copyright.

When I present my arguments on the subject, the most frequent argument I hear in response is something like the following: “artists have to eat, too”. The answer to this argument is so mind-bogglingly obvious that, in the absence of understanding, it starkly illuminates just how successful capitalism has been in corrupting a broad human understanding of empathy. So, I will spell the answer out: why do we have a system which will, for any reason, deny someone access to food? How unbelievably cruel is a system which will let someone starve because they cannot be productive within the terms of capitalism?

My argument is built on the more fundamental understanding that the access to fundamental human rights such as food, shelter, security, and healthcare are not contingent on their ability to be productive under the terms of capitalism. And I emphasize the “terms of capitalism” here deliberately: how much creativity is stifled because it cannot be expressed profitably? The system is not just cruel, but it also limits the potential of human expression, which is literally the only thing that creative endeavours are concerned with.

The fact that the “starving artist” is such a common trope suggests to us that artists aren’t putting food on the table under the copyright regime, either. Like in many industries under capitalism, artists are often not the owners of the products of their labor. Copyright protects the rights holder, not the author. The obscene copyright rules in the United States, for example, are not doing much benefit for the artist when the term ends 70 years after their death. Modern copyright law was bought, paid for, and written by corporate copyright owners, not artists. What use is the public domain to anyone when something published today cannot be legally remixed by even our great-great-grandchildren?

Assume that we address both of these problems: we create an empathetic system which never denies a human being of their fundamental right to live, and we eliminate copyright. Creativity will thrive under these conditions. How?

Artists are free to spend their time at their discretion under the new copyright-free regime. They can devote themselves to their work without concern for whether or not it will sell, opening up richer and more experimental forms of expression. Their peers will be working on similar terms, freeing them to more frequent collaborations of greater depth. They will build upon each other’s work to create a rich commons of works and derivative works.

There’s no escaping the fact that derivation and remixing is a fundamental part of the creative process, and that copyright interferes with this process. Every artist remixes the works of other artists: this is how art is made. Under the current copyright regime, this practice ranges from grey-area to illegal, and because money makes right, rich and powerful artists aggressively defend their work, extracting rent from derivative works, while shamelessly ripping off works from less powerful artists who cannot afford to fight them in court. Eliminating copyright rids us of this mess and acknowledges that remixing is part of the creative process, freeing artists to build on each other’s work.

This is not a scenario in which artists stop making money, or in which the world grinds to a halt because no one is incentivized to work anymore. The right to have your fundamental needs met does not imply that we must provide everyone with a luxurious lifestyle. If you want a nicer house, more expensive food, to go out to restaurants and buy fancy clothes — you need to work for it. If you want to commercialize your art, you can sell CDs and books, prints or originals, tickets to performances, and so on. You can seek donations from your audience through crowdfunding platforms, court wealthy patrons of the arts, or take on professional work making artistic works like buildings and art installations for public and private sector. You could even get a side job flipping burgers or take on odd jobs to cover the costs of materials like paint or musical instruments — but not your dinner or apartment. The money you earn stretches longer, not being eaten away by health insurance or rent or electricity bills. You invest your earnings into your art, not into your livelihood.

Copyright is an absurd system. Ideas do not have intrinsic value. Labor has value, and goods have value. Ideas are not scarce. By making them artificially so, we sabotage the very process by which ideas are made. Copyright is illegitimate, and we can, and ought to, get rid of it.


Aside: I came across a couple of videos recently that I thought were pretty interesting and relevant to this topic. Check them out:

On commercial forks of FOSS projects

18 December 2021 at 00:00

The gaming and live streaming industry is a lucrative and rapidly growing commercial sector with a unique understanding of copyright and intellectual property, and many parties with conflicting interests and access to different economic resources.

The understanding of intellectual property among gamers and the companies which serve them differs substantailly from that of free software, and literacy in the values and philosophy of free software among this community is very low. It is then of little surprise that we see abuse of free software from this community, namely in the recent (and illegal) commercial forks of a popular FOSS streaming platform called OBS Studio by companies like TikTok.

These forks are in violation of the software license of OBS Studio, which is both illegal and unethical. But the “why” behind this is interesting for a number of reasons. For one, there is a legitimate means through which commercial entities can repurpose free software projects, up to and including reskinning and rebranding and selling them. The gaming community also has an unusual perspective on copyright which colors their understanding of the situation. Consider, for instance, the modding community.

Game modifications (mods) exist in a grey area with respect to copyright. Modding in general is entirely legal, though some game companies do not understand this (or choose not to understand this) and take action against them. Modders also often use assets of dubious provenance in their work. Many people believe that, because this is all given away for free, the use is legitimate, and though they are morally correct, they are not legally correct. Additionally, since most mods are free (as in beer),1 the currency their authors receive for their work is credit and renown. Authors of these mods tend to defend their work fiercely against its “theft”. Modders also tend to be younger, and grew up after the internet revolution and the commoditization of software.

On the other hand, the conditions under which free software can be “stolen” are quite different, because the redistribution, reuse, and modification of free software, including for commercial purposes, is an explicit part of the social and legal contract of FOSS. This freedom comes, however, with some conditions. The nature of these conditions varies from liberal to strict. For instance, software distributed with the MIT license requires little more than crediting the original authors in any derivative works. On the other end of this spectrum, copyleft licenses like the GPL family require that any derivative works of the original project are also released under the GPL license. OBS Studio uses the GPL license, and it is in this respect that all of these forks have made a legal misstep.

If a company like TikTok wants to use OBS Studio to develop its own streaming software, they are allowed to do this, though the degree to which they are encouraged to do this is the subject of some debate.2 However, they must release the source code for their modifications under the same GPL license. They can repurpose and rebrand OBS Studio only if their repurposed and rebranded version is made available to the free software community under the same terms. Then OBS Studio can take any improvements they like from the TikTok version and incorporate them into the original OBS Studio software, so that everyone shares the benefit — TikTok, OBS users, StreamLabs, and StreamElements alike, as well as anyone else who wants in on the game.

This happens fairly often with free software and often forms a healthy relationship by establishing an incentive and a pool of economic resources to provide for the upkeep and development of that software. Many developers of a project like this are often hired by such companies to do their work. Sometimes, this relationship is viewed more negatively, but that’s a subject for another post. It works best when all of the players view each other as collaborators, not competitors.

That’s not what happening here, though. What we’re seeing instead is the brazen theft of free software by corporations who believe that, because their legal budget exceeds the resources available to the maintainers, might makes right.

Free software is designed to be used commercially, but you have to do it correctly. This is a resource which is made available to companies who want to exploit it, but they must do so according to the terms of the licenses. It’s not a free lunch.


  1. I think that this is likely the case specifically to dis-incentivize legal action by the gaming companies (who would likely be wrong, but have a lot of money) or from the owners of dubiously repurposed assets (who would likely be right, and also have a lot of money). One notable exception is the Black Mesa mod, which received an explicit blessing from Valve for its sale. ↩︎

  2. For my part, I’m in the “this is encouraged” camp. ↩︎

Status update, December 2021

15 December 2021 at 00:00

Greetings! It has been a cold and wet month here in Amsterdam, much like the rest of them, as another period of FOSS progress rolls on by. I have been taking it a little bit easier this month, and may continue to take some time off in the coming weeks, so I can have a bit of a rest for the holidays. However, I do have some progress to report, so let’s get to it.

In programming language progress, we’ve continued to see improvement in cryptography, with more AES cipher modes and initial work on AES-NI support for Intel processors, as well as support for HMAC and blake2b. Improved support for linking with C libraries has also landed, which is the basis of a few third-party libraries which are starting to appear, such as bindings to libui. I have also started working on bindings to SDL2, which I am using to make a little tetromino game (audio warning):

I am developing this to flesh out the SDL wrapper and get a feel for game development in the new language, but I also intend to take it on as a serious project to make a game which is fun to play. I also started working on an IRC protocol library for our language, but this does not link to C.

Also, the reflection support introduced a few months ago has been removed.

My other main focus has been SourceHut, where I have been working on todo.sr.ht’s GraphQL API. This one ended up being a lot of work. I expect to require another week or two to finish it.

visurf also enjoyed a handful of improvements this month, thanks to some contributors, the most prolific of whom was Pranjal Kole. Thanks Pranjal! Improvements landed this month include tab rearranging, next and previous page navigation, and an improvement to all of the new-tab logic, along with many bug fixes and smaller improvements. I also did some of the initial work on command completions, but there is a lot left to do in this respect.

That’s all for today. Thanks for your continued support! Until next time.

Impressions of Linux Mint & elementary OS

14 December 2021 at 00:00

In a recent post, I spoke about some things that Linux distros need to do better to accommodate end-users. I was reminded that there are some Linux distros which are, at least to some extent, following my recommended playbook, and have been re-evaluating two of them over the past couple of weeks: Linux Mint and elementary OS. I installed these on one of my laptops and used it as my daily driver for a day or two each.

Both of these distributions are similar in a few ways. For one, both distros required zero printer configuration: it just worked. I was very impressed with this. Both distros are also based on Ubuntu, though with different levels of divergence from their base. Ubuntu is a reasonably good choice: it is very stable and mature, and commercially supported by Canonical.

I started with elementary OS, which does exactly what I proposed in my earlier article: charge users for the OS.1 The last time I tried elementary, I was less than impressed, but they’ve been selling the OS for a while now so I hoped that with a consistent source of funding and a few years to improve they would have an opportunity to impress me. However, my overall impressions were mixed, and maybe even negative.

The biggest, showstopping issue is a problem with their full disk encryption setup. I was thrilled to see first-class FDE support in the installer, but upon first boot, I was presented with a blank screen. It took me a while to figure out that a different TTY had cryptsetup running, waiting for me to enter the password. This is totally unacceptable, and no average user would have any clue what to do when presented with this. This should be a little GUI baked into the initramfs which prompts for your password on boot, and should be a regularly tested part of the installer before each elementary release ships.

The elementary store was also disappointing, though I think there’s improvements on the horizon. The catalogue is very sparse, and would benefit a lot by sourcing packages from the underlying Ubuntu repositories as well. I think they’re planning on a first-class Flatpak integration in a future release, which should improve this situation. I also found the apps a bit too elementary, haha, in that they were lacking in a lot of important but infrequently used features. In general elementary is quite basic, though it is also very polished. Also, the default wallpaper depicts a big rock covered in bird shit, which I thought was kind of funny.

There is a lot to like about elementary, though. The installer is really pleasant to use, and I really appreciated that it includes important accessibility features during the install process. The WiFi configuration is nice and easy, though it prompted me to set up online accounts before prompting me to set up WiFi. All of the apps are intuitive, consistently designed, and beautiful. I also noticed that long-running terminal processes I had in the background would pop-up a notification upon completion, which is a nice touch. Overall, it’s promising, but I had hoped for more. My suggestions to elementary are to consider that completeness is a kind of polish, to work on software distribution, and to offer first-class options for troubleshooting, documentation, and support within the OS.

I tried Linux Mint next. Several years ago, I actually used Mint as my daily driver for about a year — it was the last “normal” distribution I used before moving to Arch and later Alpine, which is what I use now. Overall, I was pretty impressed with Mint after a couple of days of use.

Let’s start again with the bad parts. The installer is not quite as nice as elementary’s, though it did work without any issues. At one point I was asked if I wanted to “enable multimedia codecs” with no extra context, which would confuse me if I didn’t understand what they were. I was also pretty pissed to see the installer advertising nonfree, predatory services like Netflix and YouTube to me — distributions have no business advertising this kind of shit. Mint also has encryption options, but it’s based on ecryptfs rather than LUKS, and I find that this is an inferior approach. Mint should move to full-disk encryption.

I also was a bit concerned about the organizational structure of Linux Mint. It’s unclear who is responsible for Linux Mint, how end-users can participate, or how donations are spent or how other financial concerns are addressed. I think that Linux Mint needs to be more transparent, and should also consider how its allegiance with proprietary services like Netflix acts as a long-term divestment from the FOSS ecosystem it relies on.

That said, the actual experience of using Linux Mint is very good. Unlike elementary OS, the OS feels much more comprehensive. Most of the things a typical user would need are there, work reliably, and integrate well with the rest of the system. Software installation and system upkeep are very easy on Linux Mint. The aesthetic is very pleasant and feels like a natural series of improvements to the old Gnome 2 lineage that Cinnamon can be traced back to, which has generally moved more in the direction that I would have liked Gnome upstream to. The system is tight, complete, and robust. Nice work.

In conclusion, Linux Mint will be my recommendation for “normal” users going forward, and I think there is space for elementary OS for some users if they continue to improve.


  1. I downloaded it for free, however, because I did not anticipate that I would continue to use it for more than a couple of days. ↩︎

How new Linux users can increase their odds of success

5 December 2021 at 00:00

The Linus Tech Tips YouTube channel has been putting out a series of videos called the Switching to Linux Challenge that has been causing a bit of a stir in the Linux community. I’ve been keeping an eye on these developments, and thought it was a good time to weigh in with my thoughts. This article focuses on how new Linux users can increase their odds for success — I have also written a companion article, “What desktop Linux needs to succeed in the mainstream”, which looks at the other side of the problem.

Linux is, strictly speaking, an operating system kernel, which is a small component of a larger system. However, in the common usage, Linux refers to a family of operating systems which are based on this kernel, such as Ubuntu, Fedora, Arch Linux, Alpine Linux, and so on, which are referred to as distributions. Linux is used in other contexts, such as Android, but the common usage is generally limited to this family of Linux “distros”. Several of these distros have positioned themselves for various types of users, such as office workers or gamers. However, the most common Linux user is much different. What do they look like?

The key distinction which sets Linux apart from more common operating systems like Windows and macOS is that Linux is open source. This means that the general public has access to the source code which makes it tick, and that anyone can modify it or improve it to suit their needs. However, to make meaningful modifications to Linux requires programming skills, so, consequentially, the needs which Linux best suits are the needs of programmers. Linux is the preeminent operating system for programmers and other highly technical computer users, for whom it can be suitably molded to purpose in a manner which is not possible using other operating systems. As such, it has been a resounding success on programmer’s workstations, on servers in the cloud, for data analysis and science, in embedded workloads like internet-of-things, and other highly technical domains where engineering talent is available and a profound level of customization is required.

The Linux community has also developed Linux as a solution for desktop users, such as the mainstream audience of Windows and macOS. However, this work is mostly done by enthusiasts, rather than commercial entities, so it can vary in quality and generally any support which is available is offered on a community-run, best-effort basis. Even so, there have always been a lot of volunteers interested in this work — programmers want a working desktop, too. Programmers also want to play games, so there has been interest in getting a good gaming setup working on Linux. In the past several years, there has also been a commercial interest with the budget to move things forward: Valve Software. Valve has been instrumental in developing more sophisticated gaming support on Linux, and uses Linux as the basis of a commercial product, the Steam Deck.1

Even so, I must emphasize the following point:

The best operating system for gaming is Windows.

Trying to make Linux do all of the things you’re used to from Windows or macOS is not going to be a successful approach. It is possible to run games on Linux, and it is possible to run some Windows software on Linux, but it is not designed to do these things, and you will likely encounter some papercuts on the way. Many advanced Linux users with a deep understanding of the platform and years of experience under their belt can struggle for days to get a specific game running. However, thanks to Valve, and the community at large, many games — but not all games — run out-of-the-box with much less effort than was once required of Linux gamers.

Linux users are excited about improved gaming support because it brings gaming to a platform that they already want to use for other reasons. Linux is not Windows, and offers an inferior gaming experience to Windows, but it does offer a superior experience in many other regards! If you are trying out Linux, you should approach it with an open mind, prepared to learn about what makes Linux special and different from Windows. You’ll learn about new software, new usability paradigms, and new ways of using your computer. If you just want to do all of the same things on Linux that you’re already doing on Windows, why switch in the first place? The value of Linux comes from what it can do differently. Given time, you will find that there are many things that Linux can do that Windows cannot. Leave your preconceptions at the door and seek to learn what makes Linux special.

I think that so-called “power users” are especially vulnerable to this trap, and I’ve seen it happen many times. A power user is someone who deeply understands the system that they’re using, knows about every little feature, knows all of the keyboard shortcuts, and integrates all of these details into their daily workflow. Naturally, it will take you some time to get used to a new system. You can be a power user on Linux — I am one such user myself — but you’re essentially starting from zero, and you will learn about different features, different nuances, and different shortcuts, all of which ultimately sums to an entirely different power user.

The latest LTT video in the Linux series shows the team going through a set of common computer tasks on Linux. However, these tasks do little to nothing to show off what makes Linux special. Watching a 4K video is nice, sure, and you can do it on Linux, but how does that teach you anything interesting about Linux?

Let me offer a different list of challenges for a new Linux user to attempt, hand-picked to show off the things which set Linux apart in my opinion.

  1. Learn how to use the shell. A lot of new Linux users are intimidated by the terminal, and a lot of old Linux users are understandably frustrated about this. The terminal is one of the best things about Linux! We praise it for a reason, intimidating as it may be. Here’s a nice tutorial to start with.
  2. Find and install packages from the command line. On Linux, you install software by using a “package manager”, a repository of software controlled by the Linux distribution. Think of it kind of like an app store, but non-commercial and without malware, adware, or spyware. If you are downloading Linux software from a random website, it’s probably the wrong thing to do. See if you can figure out the package manager instead!
  3. Try out a tiling window manager, especially if you consider yourself a power user. I would recommend sway, though I’m biased because I started this project. Tiling window managers change the desktop usability paradigm by organizing windows for you and letting you navigate and manipulate them using keyboard shortcuts alone. These are big productivity boosters.
  4. Compile a program from source. This generally is not how you will usually find and use software, but it is an interesting experience that you cannot do on Windows or Mac. Pick something out and figure out where the source code is and how to compile it yourself. Maybe you can make a little change to it, too!
  5. Help someone else out online. Linux is a community of volunteers supporting each other. Take what you’ve learned to /r/linuxquestions or your distro’s chat rooms, forums, wikis, or mailing lists, and make them a better place for everyone else. The real magic of Linux comes from the collaborative, grassroots nature of the project, which is something you really cannot get from Windows or Mac.

Bonus challenge: complete all of the challenges from the LTT video, but only using the command line.

All of these tasks might take a lot longer than 15 minutes to do, but remember: embrace the unfamiliar. You don’t learn anything by doing the things you already know how to do. If you want to know why Linux is special, you’ll have to step outside of your comfort zone. Linux is free, so there’s no risk in trying 🙂 Good luck, and do not be afraid to ask for help if you get stuck!


  1. Full disclosure: I represent a company which has a financial relationship with Valve and is involved in the development of software used by the Steam Deck. ↩︎

What desktop Linux needs to succeed in the mainstream

5 December 2021 at 00:00

The Linus Tech Tips YouTube channel has been putting out a series of videos called the Switching to Linux Challenge that has been causing a bit of a stir in the Linux community. I’ve been keeping an eye on these developments, and thought it was a good time to weigh in with my thoughts. This article focuses on what Linux needs to do better — I have also written a companion article, “How new Linux users can increase their odds of success”, which looks at the other side of the problem.

Linux is not accessible to the average user today, and I didn’t need to watch these videos to understand that. I do not think that it is reasonable today to expect a non-expert user to successfully install and use Linux for their daily needs without a “Linux friend” holding their hand every step of the way.

This is not a problem unless we want it to be. It is entirely valid to build software which is accommodating of experts only, and in fact this is the kind of software I focus on in my own work. I occasionally use the racecar analogy: you would not expect the average driver to be able to drive a Formula 1 racecar. It is silly to suggest that Formula 1 vehicle designs ought to accommodate non-expert drivers, or that professional racecar drivers should be driving mini-vans on the circuit. However, it is equally silly to design a professional racing vehicle and market it to soccer moms.

I am one of the original developers of the Sway desktop environment for Linux. I am very proud of Sway, and I believe that it represents one of the best desktop experiences on Linux. It is a rock-solid, high-performance, extremely stable desktop which is polished on a level that is competitive with commercial products. However, it is designed for me: a professional, expert-level Linux user. I am under no illusions that it is suitable for my grandmother.1

This scenario is what the incentives of the Linux ecosystem favors most. Linux is one of the best operating systems for professional programmers and sysadmins, to such an extraordinary degree that most programmers I know treat Windows programmers and sysadmins as the object of well-deserved ridicule. Using Windows for programming or production servers is essentially as if the race car driver from my earlier metaphor did bring a mini-van to the race track. Linux is the operating system developed by programmers, for programmers, to suit our needs, and we have succeeded tremendously in this respect.

However, we have failed to build an operating system for people who are not like us.

If this is not our goal, then that’s fine. But, we can build things for non-experts if we choose to. If we set “accessible to the average user” as a goal, then we must take certain steps to achieve it. We need to make major improvements in the following areas: robustness, intuitiveness, and community.

The most frustrating moments for a user is when the software they’re using does something inexplicable, and it’s these moments that they will remember the most vividly as part of their experience. Many Linux desktop and distribution projects are spending their time on shiny new features, re-skins, and expanding their scope further and further. This is a fool’s errand when the project is not reliable at its current scope. A small, intuitive, reliable program is better than a large, unintuitive, unreliable program. Put down the paint brush and pick up the polishing stone. I’m looking at you, KDE.2

A user-friendly Linux desktop system should not crash. It should not be possible to install a package which yeets gnome-desktop and dumps them into a getty. The smallest of interactions must be intuitive and reliable, so that when Linus drags files from the decompression utility into a directory in Dolphin, it does the right thing. This will require a greater degree of cooperation and unity between desktop projects. Unrelated projects with common goals need to be reaching out to one another and developing robust standards for achieving those goals. I’m looking at you, Gnome.

Linux is a box of loosely-related tools held together with staples and glue. This is fine when the user understands the tools and is holding the glue bottle, but we need to make a more cohesive, robust, and reliable system out of this before it can accommodate average end-users.

We also have a lot of work to do in the Linux community. The discussion on the LTT video series has been exceptionally toxic and downright embarrassing. There is a major problem of elitism within the Linux community. Given a hundred ways of doing things on Linux (✓), there will be 99 assholes ready to tell you that your way sucks (✓). Every Linux user is responsible for doing better in this regard, especially the moderators of Linux-adjacent online spaces. Wouldn’t it be better if we took pride in being a friendly, accessible community? Don’t flame the noobs.

Don’t flame the experts, either. When Pop!_OS removed gnome-desktop upon installing Steam, the Linux community rightly criticised them for it. This was a major failure mode of the system in one of its flagship features, and should have never shipped. It illuminates systemic failures in the areas I have drawn our attention to in this article such as robustness and intuitiveness, and Pop!_OS is responsible for addressing the problem. None of that excuses the toxic garbage which was shoveled into the inboxes of Pop!_OS developers and users. Be better people.

Beyond the toxicity, there are further issues with the Linux community. There are heaps and heaps of blogs shoveling out crappy non-solutions to problems noobs might be Googling, most of which will fuck up their Linux system in some way or another. It’s very easy to find bad advice for Linux, and very hard to find good advice for Linux. The blog spammers need to cut it out, and we need to provide better, more accessible resources for users to figure out their issues. End-user-focused Linux distributions need to take responsibility for making certain that their users understand the best ways to get help for any issues they run into, so they don’t go running off to the Arch Linux forums blindly running terminal commands which will break their Ubuntu installation.

End-user software also needs to improve in this respect. In the latest LTT video, Luke wanted to install OBS, and the right thing to do was install it from their package manager. However, the OBS website walks them through installing a PPA instead, and has a big blue button for building it from source, which is definitely not what an average end-user should be doing.

→ Related: Developers: Let distros do their job

One thing that we do not need to do is “be more like Windows”, or any other OS. I think that this is a common fallacy found in end-user Linux software. We should develop a system which is intuitive in its own right without having to crimp off of Windows. Let’s focus on what makes Linux interesting and useful, and try to build a robust, reliable system which makes those interesting and useful traits accessible to users. Chasing after whatever Windows does is not the right thing to do. Let’s be prepared to ask users to learn things like new usability paradigms if it illuminates a better way of doing things.

So, these are the goals. How do we achieve them?

I reckon that we could use a commercial, general-purpose end-user Linux distro. As I mentioned earlier, the model of developers hacking in their spare time to make systems for themselves does not create incentives which favor the average end-user. You can sell free software — someone ought to do so! Build a commercial Linux distro, charge $20 to download it or mail an install CD to the user, and invest that money in developing a better system and offer dedicated support resources. Sure, it’s nice that Linux is free-as-in-beer, but there’s no reason it has to be. I’ve got my own business to run, so I’ll leave that as an exercise for the reader. Good luck!


  1. However, I suspect that the LTT folks and other “gaming power-user” types would find Sway very interesting, if they approached it with a sufficiently open-minded attitude. For details, see the companion article. ↩︎

  2. There is at least one person at KDE working along these lines: Nate Graham. Keep it up! ↩︎

postmarketOS revolutionizes smartphone hacking

26 November 2021 at 00:00

I briefly mentioned postmarketOS in my Pinephone review two years ago, but after getting my Dutch SIM card set up in my Pinephone and having another go at using postmarketOS, I reckon they deserve special attention.

Let’s first consider the kind of ecosystem into which postmarketOS emerged: smartphone hacking in the XDA Forums era. This era was dominated by amateur hackers working independently for personal prestige, with little to no regard for the values of free software or collaboration. It was common to see hacked-together binary images shipped behind adfly links in XDA forum threads in blatant disregard of the GPL, with pages and pages of users asking redundant questions and receiving poor answers to the endless problems caused by this arrangement.

The XDA ecosystem is based on Android, which is a mess in and of itself. It’s an enormous, poorly documented ball of Google code, mixed with vendor drivers and private kernel trees, full of crappy workarounds and locked-down hardware. Most smart phones are essentially badly put-together black boxes and most smart phone hackers are working with their legs cut off. Not to mention that the software ecosystem which runs on the platform is full of scammers and ads and theft of private user information. Android may be Linux in implementation, but it’s about as far from the spirit of free software as you can get.

postmarketOS, on the other hand, is based on Alpine Linux, which happens to be my favorite Linux distribution. Instead of haphazard forum threads collecting inscrutable ports for dozens of devices, they have a single git repository where all of their ports are maintained under version control, complete with issue trackers and merge requests, plus a detailed centralized wiki providing a wealth of open technical info on their supported platforms. And, by virtue of being a proper Linux distribution, they essentially opt-out of the mess of predatory mobile apps and instead promote a culture of trusted applications which respect the user and are built by and for the community instead of by and for a corporation.

Where we once had to live with illegally closed-source forks of the Linux kernel, we now have a git repository in which upstream Linux releases are tracked with a series of auditable patches for supporting various devices, many of which are making their way into upstream Linux. Where we once had a forum thread with five wrong answers to the same question on page 112, we now have a bug report on GitLab with a documented workaround and a merge request pending review. Instead of begging my vendor to unlock my bootloader and using janky software reminiscent of old keygen hacks to flash a dubious Android image, I can build postmarketOS’s installer, pop it onto a microSD card, and two minutes I’ll have Linux installed on my Pinephone.

pmOS does not seek to elevate the glories of tiny individual hackers clutching their secrets close to their chest, instead elevating the glory of the community as a whole. It pairs perfectly with Pine64, the only hardware vendor working closely with upstream developers with the same vision and ideals. There is a promise for hope in the future of smart phones in their collaboration.

However, the path they’ve chosen is a difficult one. Android, for all of its faults, presents a complete solution for a mobile operating system, and upstream Linux does not. In my review, I said that software would be the biggest challenge of the Pinephone, and 2 years later, that remains the case. Work reverse engineering the Pine64 hardware is slow, there is not enough cooperation between project silos, and there needs to be much better prioritization of the work. To complete their goals, the community will have to work more closely together and narrow their attention in on the key issues which stand between the status quo and the completion of a useful Linux smartphone. It will require difficult, boring engineering work, and will need the full attention and dedication of the talented people working on these projects.

If they succeed in spite of these challenges, the results will be well worth it. postmarketOS and pine64 represent the foundations of a project which could finally deliver Linux on smartphones and build a robust mobile platform that offers freedom to its users for years to come.

My philosophy for productive instant messaging

24 November 2021 at 00:00

We use Internet Relay Chat (IRC) extensively at sourcehut for real-time group chats and one-on-one messaging. The IRC protocol is quite familiar to hackers, who have been using it since the late 80’s. As chat rooms have become more and more popular among teams of both hackers and non-hackers in recent years, I would like to offer a few bites of greybeard wisdom to those trying to figure out how to effectively use instant messaging for their own work.

For me, IRC is a vital communication tool, but many users of <insert current instant messaging software fad here>1 find it frustrating, often to the point of resenting the fact that they have to use it at all. Endlessly catching up on discussions they missed, having their workflow interrupted by unexpected messages, searching for important information sequestered away in a discussion which happened weeks ago… it can be overwhelming and ultimately reduce your productivity and well-being. Why does it work for me, but not for them? To find out, let me explain how I think about and use IRC.

The most important trait to consider when using IM software is that it is ephemeral, and must be treated as such. You should not “catch up” on discussions that you missed, and should not expect others to do so, either. Any important information from a chat room discussion must be moved to a more permanent medium, such as an email to a mailing list,2 a ticket filed in a bug tracker, or a page updated on a wiki. One very productive use of IRC for me is holding a discussion to hash out the details of an issue, then writing up a summary up for a mailing list thread where the matter is discussed in more depth.

I don’t treat discussions on IRC as actionable until they are shifted to another mode of discussion. On many occasions, I have discussed an issue with someone on IRC, and once the unknowns are narrowed down and confirmed to be actionable, ask them to follow-up with an email or a bug report. If the task never leaves IRC, it also never gets done. Many invalid or duplicate tasks are filtered out by this approach, and those which do get mode-shifted often have more detail than they otherwise might, which improves the signal-to-noise ratio on my bug trackers and mailing lists.

I have an extensive archive of IRC logs dating back over 10 years, tens of gigabytes of gzipped plaintext files. I reference these logs perhaps only two or three times a year, and often for silly reasons, like finding out how many swear words were used over some time frame in a specific group chat, or to win an argument about who was the first person to say “yeet” in my logs. I almost never read more than a couple dozen lines of the backlog when starting up IRC for the day.

Accordingly, you should never expect anyone to be in the know for a discussion they were not present at. This also affects how I use “highlights”.3 Whenever I highlight someone, I try to include enough context in the message so that they can understand why they were mentioned without having to dig through their logs, even if they receive the notification hours later.

Bad:

<sircmpwn> minus: ping
<sircmpwn> what is the best way to frob foobars?

Good:

<sircmpwn> minus: do you know how to frob foobars?

I will also occasionally send someone a second highlight un-pinging them if the question was resolved and their input is no longer needed. Sometimes I will send a vague “ping <username>” example when I actually want them to participate in the discussion right now, but if they don’t answer immediately then I will usually un-ping them later.4

This draws attention to another trait of instant messaging: it is asynchronous. Not everyone is online at the same time, and we should adjust our usage of it in consideration of this. For example, when I send someone a private message, rather than expecting them to engage in a real-time dialogue with me right away, I dump everything I know about the issue for them to review and respond to in their own time. This could be hours later, when I’m not available myself!

Bad:

<sircmpwn> hey emersion, do you have a minute?
*8 hours later*
<emersion> yes?
*8 hours later*
<sircmpwn> what is the best way to frob foobars?
*8 hours later*
<emersion> did you try mongodb?

Good:5

<sircmpwn> hey emersion, what's the best way to frob foobars?
<sircmpwn> I thought about mongodb but they made it non-free
*10 minutes later*
<sircmpwn> update: considered redis, but I bet they're one bad day away from making that non-free too
*8 hours later*
<emersion> good question
<emersion> maybe postgresql? they seem like a trustworthy bunch
*8 hours later*
<sircmpwn> makes sense. Thanks!

This also presents us a solution to the interruptions problem: just don’t answer right away, and don’t expect others to. I don’t have desktop or mobile notifications for IRC. I only use it when I’m sitting down at my computer, and I “pull” notifications from it instead of having it “push” them to me — that is, I glance at the client every now and then. If I’m in the middle of something, I don’t read it.

With these considerations in mind, IRC has been an extraordinarily useful tool for me, and maybe it can be for you, too. I’m not troubled by interruptions to my workflow. I never have to catch up on a bunch of old messages. I can communicate efficiently and effectively with my team, increasing our productivity considerably, without worrying about an added source of stress. I hope that helps!


  1. Many, many companies have tried, and failed, to re-invent IRC, usually within a proprietary walled garden. I offer my condolences if you find yourself using one of these. ↩︎

  2. Email is great. If you hate it you might be using it wrong↩︎

  3. IRC terminology for mentioning someone’s name to get their attention. Some platforms call this “mentions”. ↩︎

  4. I occasionally forget to… apologies to anyone I’ve annoyed by doing that. ↩︎

  5. I have occasionally annoyed someone with this strategy. If they have desktop notifications enabled, they might see 10 notifications while I fill their message buffer with more and more details about my question. Sounds like a “you” problem, buddy 😉 ↩︎

I will pay you cash to delete your npm module

16 November 2021 at 00:00

npm’s culture presents a major problem for global software security. It’s grossly irresponsible to let dependency trees grow to thousands of dependencies, from vendors you may have never heard of and likely have not critically evaluated, to solve trivial tasks which could have been done from scratch in mere seconds, or, if properly considered, might not even be needed in the first place.

We need to figure out a way to curb this reckless behavior, but how?

I have an idea. Remember left-pad? That needs to happen more often.

A LaTeX rendering of an equation which sets a reward (in dollars) to the logarithm of weekly downloads over lines of code in base 10 times one hundred

I’ll pay you cold hard cash to delete your npm module. The exact amount will be determined on this equation, which is designed to offer higher payouts for modules with more downloads and fewer lines of code. A condition of this is that you must delete it without notice, so that everyone who depends on it wakes up to a broken build.

Let’s consider an example: isArray. It has only four lines of code:

var toString = {}.toString;

module.exports = Array.isArray || function (arr) {
  return toString.call(arr) === '[object Array]';
};

With 51 million downloads this week, this works out to a reward of $710.

To prevent abuse, we’ll have to agree to each case in advance. I’ll review your module to make sure it qualifies, and check for any funny business like suspicious download figures or minified code. We must come to an agreement before you delete the module, since I will not be able to check the line counts or download numbers after it’s gone.

I may also ask you to wait to delete your module, so that the chaos from each deletion is separated by a few weeks to maximize the impact. Also, the reward is capped at $1,000, so that I can still pay rent after this.

Do we have a deal? Click here to apply →













































Alright, the gig is up: this is satire. I’m not actually going to pay you to delete your npm module, nor do I want to bring about a dark winter of chaos in the Node ecosystem. Plus, it wouldn’t actually work.

I do hope that this idea strikes fear in the hearts of any Node developers that read it, and in other programming language communities which have taken after npm. What are you going to do if one of your dependencies vanishes? What if someone studies the minified code on your website, picks out an obscure dependency they find there, then bribes the maintainers?

Most Node developers have no idea what’s in their dependency tree. Most of them are thousands of entries long, and have never been audited. This behavior is totally reckless and needs to stop.

Most of my projects have fewer than 100 dependencies, and many have fewer than 10. Some have zero. This is by design. You can’t have a free lunch, I’m afraid. Adding a dependency is a serious decision which requires consensus within the team, an audit of the new dependency, an understanding of its health and long-term prospects, and an ongoing commitment to re-audit them and be prepared to change course as necessary.


isArray license:

Copyright (c) 2013 Julian Gruber <julian@juliangruber.com>.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Python: Please stop screwing over Linux distros

16 November 2021 at 00:00

Linux distributions? Oh, those things we use to bootstrap our Docker containers? Yeah, those are annoying. What were you complaining about again?

The Python community is obsessed with reinventing the wheel, over and over and over and over and over and over again. distutils, setuptools, pip, pipenv, tox, flit, conda, poetry, virtualenv, requirements.txt, setup.py, setup.cfg, pyproject.toml… I honestly can’t even list all of the things you have to deal with. It’s a disaster.

An xkcd comic showing a convoluted graph of competing Python environments

This comic is almost 4 years old and it has become much worse since. Python is a mess. I really want to like Python. I have used it for many years and in many projects, including SourceHut, which was predominantly developed in Python. But I simply can’t handle it anymore, and I have been hard at work removing Python from my stack.

This has always been a problem with Python, but in the past few years everyone and their cousin decided to “solve” it by building another mess which is totally incompatible with all of the others, all of the “solutions” enjoying varying levels of success in the community and none of them blessed as the official answer.

I manage my Python packages in the only way which I think is sane: installing them from my Linux distribution’s package manager. I maintain a few dozen Python packages for Alpine Linux myself. It’s from this perspective that, throughout all of this turmoil in Python’s packaging world, I have found myself feeling especially put out.

Every one of these package managers is designed for a reckless world in which programmers chuck packages wholesale into ~/.pip, set up virtualenvs and pin their dependencies to 10 versions and 6 vulnerabilities ago, and ship their computers directly into production in Docker containers which aim to do the minimum amount necessary to make their user’s private data as insecure as possible.

None of these newfangled solutions addresses the needs of any of the distros, despite our repeated pleas. They all break backwards compatibility with our use-case and send our complaints to /dev/null. I have seen representatives from every Linux distro making repeated, desperate pleas to Python to address their concerns, from Debian to Arch to Alpine to NixOS, plus non-Linux distros like FreeBSD and Illumos. Everyone is frustrated. We are all struggling to deal with Python right now, and Python is not listening to us.

What is it about Linux distros that makes our use-case unimportant? Have we offered no value to Python over the past 30 years? Do you just feel that it’s time to shrug off the “legacy” systems we represent and embrace the brave new world of serverless cloud-scale regulation-arbitrage move-fast-and-break-things culture of the techbro startup?

Distros are feeling especially frustrated right now, but I don’t think we’re alone. Everyone is frustrated with Python packaging. I call on the PSF to sit down for some serious, sober engineering work to fix this problem. Draw up a list of the use-cases you need to support, pick the most promising initiative, and put in the hours to make it work properly, today and tomorrow. Design something you can stick with and make stable for the next 30 years. If you have to break some hearts, fine. Not all of these solutions can win. Right now, upstream neglect is destroying the Python ecosystem. The situation is grave, and we need strong upstream leadership right now.


P.S. PEP-517 and 518 are a start, but are very disappointing in how little they address distro problems. These PEPs are designed to tolerate the proliferation of build systems, which is exactly what needs to stop. Python ought to stop trying to avoid hurting anyone’s feelings and pick one. Maybe their decision-making framework prevents this, if so, the framework needs to be changed.


P.P.S. There are a lot of relevant xkcds that I wanted to add. Here’s the ones I left out:

Further reading: Developers: Let distros do their job

Status update, November 2021

15 November 2021 at 00:00

Hello again! Following a spooky month, we find ourselves again considering the progress of our eternal march towards FOSS world domination.

I’ll first address SourceHut briefly: today is the third anniversary of the opening of the public alpha! I have written a longer post for sourcehut.org which I encourage you to read for all of the details.

In other news, we have decided to delay the release of our new programming language, perhaps by as much as a year. We were aiming for February ‘22, but slow progress on some key areas such as cryptography and the self-hosting compiler, plus the looming necessity of the full-scale acceptance testing of the whole language and standard library, compound to make us unsure about meeting the original release plans. However, progress is slow but moving. We have incorporated the first parts of AES support in our cryptography library, and ported the language to FreeBSD. A good start on date/time support has been under development and I’m pretty optimistic about the API design we’ve come up with. Things are looking good, but it will take longer than expected.

visurf has enjoyed quite a bit of progress this month, thanks in large part to the help of a few new contributors. Nice work, everyone! We could still use more help, so please swing by the #netsurf channel on Libera Chat if you’re interested in participating. Improvements which landed this month include configuration options, url filtering via awk scripts, searching through pages, and copying links on the page with the link following tool.

Projects which received minor updates this month include scdoc, gmni, kineto, and godocs.io. That’s it for today! My focus for the next month will be much the same as this month: SourceHut GraphQL work and programming language work. See you in another month!

Breaking down Apollo Federation's anti-FOSS corporate gaslighting

5 November 2021 at 00:00

Gather around, my friends, for there is another company which thinks we are stupid and we enjoy having our faces spat in. Apollo Federation1 has announced that they will switch to a non-free license. Let’s find out just how much the Elastic license really is going to “protect the community” like they want you to believe.

Let’s start by asking ourselves, objectively, what practical changes can we expect from a switch from the MIT license to the Elastic License? Both licenses are pretty short, so I recommend quickly reading them yourself before we move on.

I’ll summarize the difference between these licenses. First, the Elastic license offers you (the recipient of the software) one benefit that MIT does not: an explicit license for any applicable patents. However, it also has many additional restrictions, such as:

  • No sublicensing (e.g. incorporating part of it into your own program)
  • No resale (e.g. incorporating it into Red Hat and selling support)
  • No modifications which circumvent the license key activation code
  • No use in a hosted or managed service

This is an objective analysis of the change. How does Apollo explain the changes?

Why the new license?

The Apollo developer community is at the heart of everything we do. As stewards of our community, we have a responsibility to prevent harm from anyone who intends to exploit our work without contributing back. We want to continue serving you by funding the development of important open-source graph technology for years to come. To honor that commitment, we’re moving Apollo Federation 2 to the Elastic License v2 (ELv2).

Taking them at their word, this change was motivated by their deep care for their developer community. They want to “honor their commitment”, which is to “fund the development of important open-source graph technology” and “prevent harm from anyone who intends to exploit our work without contributing back”.

This is a very misleading statement. The answer to the question stated by the header is “funding the development”, but they want us to first think that they’re keeping the community at the heart of this decision — a community that they have just withheld several rights from. Their wording also seeks to link the community with the work, “our work”, when the change is clearly motivated from a position where Apollo believes they have effective ownership over the software, sole right to its commercialization, and a right to charge the community a rent — enforced via un-circumventable license key activation code. The new license gives Apollo exclusive right to commercial exploitation of the software — so they can “exploit our work”, but the community itself cannot.

What’s more, the change does not fund “open-source graph technology” as advertised, because after this change, Apollo Federation is no longer open source. The term “open source” is defined by the Open Source Definition3, whose first clause is:

[The distribution terms of open-source software] shall not restrict any party from selling or giving away the software as a component of an aggregate software distribution containing programs from several different sources. The license shall not require a royalty or other fee for such sale.

The OSD elaborates later:

The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.

The rights attached to the program must apply to all to whom the program is redistributed without the need for execution of an additional license by those parties.

The Elastic license clearly does not meet this criteria.

Reading the Apollo announcement further, it continues to peddle this and other lies. The next paragraph attempts to build legitimacy for its peers in this anti-FOSS gaslighting movement:

Open-source licensing is evolving with the cloud. Many successful companies built on open-source technology (such as Elastic, MongoDB, and Confluent) have followed the path we’re taking to protect their communities and combine open, collaborative development with the benefits of cloud services that are easy to adopt and manage.

They continue to use “open-source” language throughout, and misleads us into believing that they’ve made this change to protect the community and empower developers.

When the Elastic License v2 was released, Elastic CEO Shay Banon called upon open-source companies facing a similar decision to “coalesce around a smaller number of licenses.” We’re excited to be part of this coalition of modern infrastructure companies building businesses that empower developers. […] Moving the Apollo Federation libraries and gateway to ELv2 helps us focus on our mission: empowering all of you.

It should be evident by now that this is complete horseshit. Let me peel away the bullshit and explain what is actually going on here in plain English.

Free and open source software can be commercialized — this is an essential requirement of the philosophy! However, it cannot be exclusively commercialized. Businesses which participate in the FOSS ecosystem must give up their intellectual property monopoly, and allow the commercial ecosystem to flourish within their community — not just within their own ledger. They have to make their hosted version better than the competitors, or seek other monetization strategies: selling books, support contracts, consulting, early access to security patches, and so on.

The community, allegedly at the heart of everything Apollo does, participates in the software’s development, marketing, and growth, and they are rewarded with the right to commercialize it. The community is incentivized to contribute back because they retain their copyright and the right to monetize the software. 634 people have contributed to Apollo, and the product is the sum of their efforts, and should belong to them — not just to the business which shares a name with the software. The community built their projects on top of Apollo based on the open source social contract, and gave their time, effort, and copyright for their contributions to it, and Apollo pulled the rug out from under them. In the words of Bryan Cantrill, this shameful, reprehensible behavior is shitting in the pool of open source.

The smashing success of the free and open source software movement, both socially and commercially, has attracted the attention of bad actors like Apollo, who want to capitalize on this success without meeting its obligations. This wave of nonfree commercial gaslighting is part of a pattern where a company builds an open-source product, leverages the open-source community to build a market for it and to directly improve the product via their contributions, then switches to a nonfree license and steals the work for themselves, fucking everyone else over.

Fuck Matt DeBergalis, Shay Banon, Jay Kreps, and Dev Ittycheria. These are the CEOs and CTOs responsible for this exploitative movement. They are morally bankrupt assholes and rent-seekers who gaslight and exploit the open source community for personal gain.

This is a good reminder that this is the ultimate fate planned by any project which demands a copyright assignment from contributors in the form of a Contributor License Agreement (CLA). Do not sign these! Retain your copyright over your contributions and contribute to projects which are collectively owned by their community — because that’s how you honor your community.


Previously:

If you are an Apollo Federation user who is affected by this change, I have set up a mailing list to organize a community-maintained fork. Please send an email to this list if you are interested in participating in such a fork.


  1. For those unaware, Apollo Federation is a means of combining many GraphQL2 microservices into one GraphQL API. ↩︎

  2. For those unaware, GraphQL is a standardized query language largely used to replace REST for service APIs. SourceHut uses GraphQL. ↩︎

  3. Beware, there are more gaslighters who want us to believe that the OSD does not define “open source”. This is factually incorrect. Advocates of this position usually have ulterior motives and, like Apollo, tend to be thinking more about their wallets than the community. ↩︎

GitHub stale bot considered harmful

26 October 2021 at 00:00

Disclaimer: I work for a GitHub competitor.

One of GitHub’s “recommended” marketplace features is the “stale” bot. The purpose of this bot is to automatically close GitHub issues after a period of inactivity, 60 days by default. You have probably encountered it yourself in the course of your work.

This is a terrible, horrible, no good, very bad idea.

A screenshot of an interaction with this bot

I’m not sure what motivates maintainers to install this on their repository, other than the fact that GitHub recommends it to them. Perhaps it’s motivated by a feeling of shame for having a lot of unanswered issues? If so, this might stem from a misunderstanding of the responsibilities a maintainer has to their project. You are not obligated to respond to every issue, implement every feature request, or fix every bug, or even acknowledge them in any way.

Let me offer you a different way of thinking about issues: a place for motivated users to collaborate on narrowing down the problem and planning a potential fix. A space for the community to work, rather than an action item for you to deal with personally. It gives people a place to record additional information, and, ultimately, put together a pull request for you to review. It does not matter if this process takes days or weeks or years to complete. Over time, the issue will accumulate details and workarounds to help users identify and diagnose the problem, and to provide information for the person that might eventually write a patch/pull request.

It’s entirely valid to just ignore your bug tracker entirely and leave it up to users to deal with themselves. There is no shame in having a lot of open issues — if anything, it signals popularity. Don’t deny your users access to an important mutual support resource, and a crucial funnel to bring new contributors into your project.

This is the approach I would recommend on GitHub, but for illustrative purposes I’ll also explain a slightly modified approach I encourage for SourceHut users. sr.ht provides mailing lists (and, soon, IRC chat rooms), which are recommended for first-line support and discussion about your project, including bug reports, troubleshooting, and feature requests, instead of filing a ticket (our name for issues). The mailing list gives you a space to refine the bug report, solicit extra details or point out an existing ticket, or clarifying and narrowing down feature requests. This significantly improves the quality of bug reports, eliminates duplicates, and better leverages the community for support, resulting in every single ticket representing a unique, actionable item.

I will eventually ask the user to file a ticket when the bug or feature request is confirmed. This does not imply that I will follow up with a fix or implementation on any particular time frame. It just provides this space I discussed before: somewhere to collect more details, workarounds, and additional information for users who experience a bug or want a feature, and to plan for its eventual implementation at an undefined point in the future, either from a SourceHut maintainer or from the community.

❌
❌