I'll roll up some responses to reader feedback here.

...

Someone asked if they could view the old code lessons. The one I put back online last year is where I do a terrible little TCP listener, compile it, start it in the background, and then connect to it with netcat. It's awkward as hell, but it's there if you really want to see it.

There is also the six part "protofeed" demo which showed how to fetch a feed in this protobuf-based scheme I rigged up. Spoiler: it's not there any more, since nobody was using it, so it's not very useful to follow the instructions on that thing.

Ironically, that fetcher program would run afoul of all kinds of badness if it was pointed at a production site. It doesn't do conditional requests, it doesn't know about Cache-Control headers, it won't recognize that a 429 is asking for throttling, and so on. I guess that's okay for something that was a proof of concept to show how to fetch something from the network and parse it, but *actual* feed readers get all of that stuff wrong, too.

...

Another reader asked if the Linux "hrtimer" glitch from the leap second was fixed. I have to assume that it was based on the fact that most people didn't have that same problem three years later when we had the one in my "leap smearing" story. My worries were about userspace stuff.

This is an opportunity for me to share just why I went to those lengths. In short, it was because of a lack of confidence in everyone everywhere doing the right thing in terms of time handling. If everyone uses monotonic clocks for measuring durations and otherwise is okay with wall time going backwards now and then, then there's no reason to smear it out. My own personal systems have never smeared a leap second. They just ride it out and keep on going.

I couldn't assume the correctness of such implementations at the company. Worse, even if I went and deliberately injected backwards-going time step operations and proved that it would crash some code, there was no guarantee of anything coming from it. I had found myself in a place at that company where some parts of it were completely unresponsive to the problems they were causing for themselves and sometimes for other people, and was starting to tire of the "bad cop" schtick. That's where I'd show up and go "your shit is broken" and they would do nothing to work with us (the whole team) to do something about it.

I just had this feeling that if we repeated the last UTC second of June 2015, we'd end up breaking something. What's kind of amazing is that later on that year, it actually happened.

Someone misconfigured the ailing NTP appliances to *not* apply the correction factor from GPS to UTC. This ended up forcing one appliance into shipping unadjusted GPS time to roughly half of production via NTP, and the difference at the time was something like 17 seconds. (This changes, and indeed, it's no longer 17.)

Anyway, I got to working on this after hearing about it and found roughly half the fleet running 17 seconds fast. It was completely unreasonable to try to "smear off" 17 seconds to get things back to normal - that would have taken weeks. I made the decision to fix the setting and then let every broken machine individually have its clock dragged backwards the 17 seconds to where things should be.

This broke stuff. Some kind of sharding mechanism deep inside the fabric of things was using wall time to determine something or other, and when it jumped back, it fired an assertion and killed the program. This nuked the web server (or whatever else was using that library).

So, basically, every single machine which had been poisoned with the bad time and which was running this library was going to crash exactly once and there wasn't really anything which could be done about it. It was something like 2 in the morning by this point and I opted to let it happen.

About the only good thing about this is that the adjustments happened at different intervals depending on the ntpd poll rate, so it's not like hundreds of thousands of machines all crashed their workloads at the same time. One would pop off here, then one there, and so on... over the span of an hour or two... until it was all done.

Thus, some services didn't really go down, but they did have a bad time with a bunch of failed/dropped requests which were on the affected systems.

That one was dubbed the "Back to the Future" SEV. At least one team made a screenshot of some display showing the 17 second offset the banner of the group where they talked about production issues.

Stuff like that is why I smeared it out. When you can't be sure of the correctness of implementations, and there are good chances that attempts to fix them will be ignored, rebuffed, or actively attacked, you have to "treat the whole situation as damage" and route around it. You remove the discontinuity in wall time to save them from themselves.

...

A reader asked for my take about "AI" and LLMs and all of this.

In the vein of the "annoyances" post from earlier in the month, I'll start by saying that I don't push any of that on you here, either. All of this stuff is straight off my keyboard with a sprinkling of ispell applied after the fact. Even that's of limited utility since there are a bunch of technical terms and not-really-words that I use for various reasons.

I think all of the hype and waste has generated an enormous mountain of useless nonsense that has attracted the absolute worst of the vampires and buzzards and bottom-feeders who are looking to exploit this stuff for their own benefit.

The LAST thing we needed was a better way to generate plausible-looking horse shit for random gullible people to consume unwittingly, but here we are, and it's only going to get worse.

I think a lot of this falls into "the Internet we deserve".

So no, I don't use anything of the sort, and I tell people not to quote any of that crap at me, or to send me screenshots of it pretending to be an answer to something, and that they need to find actual sources for their data. This has not made me the most popular person.

But hey, I've already said that I'm obviously out of touch with what most people are up to. My green-on-black terminal with nano in it that's writing up a bunch of plain text with a handful of triggers for callouts to other posts should be proof positive of that already. Hardly anyone else does things this way any more. That makes me the weirdo, not them. I know this. I'm okay with this.

Right, so, one of the things that can happen when you're trying to collect fresh data on the behaviors of something dynamic is that you get bogged down under the load of what happened previously. With the feed reader score project, this is what's been happening. A lot of clients were started up and pointed at it, and we gathered a lot of behavioral data.

The problem is that some of them are not changing, and having a few dozen of them call back every five minutes is not doing anyone any favors. So, I did what I promised I would do, and I updated the hostname.

If you are participating in the test and want to continue, go back to the original mail from me you got with the code(s), and load the instruction page. There, you will find the very slightly changed base hostname that can be used to construct your new unique feed URL. The keys are the same.

This also gives us the benefit of seeing what a fresh start looks like with the latest batch of feed reader software. Many of them have done a lot of excellent work these past six months, and they deserve to leave the historical baggage behind. I want to see where they are now, and this is how we get there.

For anyone wondering, I was looking at some of the reports before I cleared things out a few minutes ago. Some of the problem spots that I had mentioned in multiple report posts were still there. A lot of this was just people running old versions when they really need to upgrade. Some of it was just nobody at the wheel for the various clown services.

Seeing a whole bunch of unchanged behavior just reinforced the need to do a fresh start on this stuff. The people who have invested in improving their software deserve it.

Once there's a fresh set of data built up, I guess I'll write up another summary.

Oh, side note for anyone keeping track: this is not the "wildcard DNS" thing that I mentioned a few weeks back that would be needed to track down the *really* goofy stuff that polls all kinds of extra crap paths. That would require (more) actual work, and I'm not ready to do that just yet. (Plus, for that kind of effort, I might want to charge for it. Just saying.)

Every six months or so, this neat group called the International Earth Rotation Service issues a directive on whether there will be a leap second inserted at the end of that six month period. You usually find out at the beginning of January or the beginning of July, and thus would have a leap second event at the end of June or December, respectively.

Ten years ago, in January 2015, they announced a leap second would be added at the end of June 2015. The last one had been three years prior, and when it happened, it screwed things up pretty bad for the cat picture factory. They hit kernel problems, userspace problems, and worse.

This time, I was working there, and decided there would not be a repeat. The entire company's time infrastructure would be adjusted so it would simply slow down for about 20 hours before the event, and so it would become a whole second "slow" relative to the rest of the world. Then at midnight UTC, the rest of the world would go 58, 59, 60, 0, and we'd go 57, 58, 59, 0, and then we'd be in lock-step again.

So how do you do something like this? Well, you have to get yourself into a position where you can add a "lie" to the time standard. This company had a handful of these devices which had a satellite receiver for GPS on one side and an Ethernet port for NTP on the other with a decent little clock on the inside. I just had to get between those and everyone else so they would receive my adjusted time scale for the duration, then we could switch back when things were ready.

This is the whole "leap smearing" thing that you might have heard of if you run in "time nut" circles. Someone else came up with it and they had only published their formula for computing the lie over a spread of time. The rest of it was "left as an exercise for the reader", so to speak.

Work like this benefits from being highly visible, so I bought a pair of broadcast-studio style clocks which spoke NTP over Ethernet and installed them on my desk. One of them was pointed at the usual GPS->NTP infrastructure, and the other was pointed at the ntp servers running my hacked-up code which could have "lies" injected.

I'd start up a test and watch them drift apart. At first, you can't even tell, but after a couple of hours, you get to where one subtly updates just a bit before the other one. You can even see it in pictures: parts of one light up before the other.

Then at the end of the ramp, they're a full second apart, but they're still updating at the same time. It's just that one goes from 39 to 40 when the other goes from 40 to 41.

Back and forth I went with my test clocks, test systems, and a handful of guinea pig boxes that volunteered to subscribe to the hacked-up time standard during these tests. We had to find a rate-of-change that would be accepted by the ntp daemons all over the fleet. There's only so much of a change you can introduce to the rate of change itself, and that meant a lot of careful experimentation to find out just what would work.

We ended up with something like 20 hours to smear off a single second.

The end of June approached, and it was time to do a full-scale test. I wanted to be sure that we could survive being a second out of whack without having the confounding factor of the whole rest of the world simultaneously dealing with their own leap second stuff. We needed to know if we'd be okay, and the only way to know was to smear it off, hold a bit to see if anything happened, then *smear it back on*.

This is probably the first anyone outside the company's heard of it, but about a week before, I smeared off a whole second and left the ENTIRE company's infra (laptops and all) running a second slow relative to the rest of the world. Then we stayed there for a couple of hours if I remember correctly, and then went forward again and caught back up.

A week later, we did it for real and it just worked.

So, yes, in June 2015, I slowed down the whole company by a second.

Of course, here it is ten years later, and the guy in charge just sent it back fifty years. Way to upstage me, dude.

I've been thinking about things that annoy me about other web pages. Safari recently gained the ability to "hide distracting items" and I've been having great fun telling various idiot web "designers" to stuff it. Reclaiming a simple experience free of wibbly wobbly stuff has been great.

In doing this, I figured maybe I should tell people about the things I don't do here, so they realize how much they are "missing out" on.

I don't force people to have Javascript to read my stuff. The simplest text-based web browser going back about as far as you can imagine should be able to render the content of the pages without any trouble. This is because there's no JS at all in these posts.

I don't force you to use SSL/TLS to connect here. Use it if you want, but if you can't, hey, that's fine, too.

The last two items mean you could probably read posts via telnet as long as you were okay with skipping over all of the HTML <tag> <crap>. You might notice that the text usually word-wraps around 72, so it's not that much of a stretch.

I don't track "engagement" by running scripts in the post's pages that report back on how long someone's looked at it... because, again, no JS.

I don't set cookies. I also don't send unique values for things like Last-Modified or ETag which also could be used to identify individuals. You can compare the values you get with others and confirm they are the same.

I don't use visitor IP addresses outside of a context of filtering abuse.

I don't do popups anywhere. You won't see something that interrupts your reading to ask you to "subscribe" and to give up your e-mail address.

I don't do animations outside of one place. Exactly one post has something in it which does some graphical crap that changes by itself. It's way back in July 2011, and it's in a story ABOUT animating a display to show the absence of a value. It doesn't try to grab your attention or mislead you, and it's not selling anything.

I don't use autoplaying video or audio. There are a couple of posts where you can click on your browser's standard controls to start playback of a bit of audio that's related to the post. Those are also not used to grab your attention, mislead you, or sell something.

I don't try to "grab you" when you back out of a page to say "before you go, check out this other thing". The same applies to closing the window or tab: you won't get this "are you sure?" crap. If you want out, you get out *the first time*.

I don't pretend that posts are evergreen by hiding their dates. Everything has a clear date both in the header of the page and built into the URL. If it's out of date, it'll be pretty obvious.

I don't put crap in the pages which "follows you" down the page as you scroll. You want to see my header again? Cool, you can scroll back up to it if it's a particularly long post. I don't keep a "dick bar" that sticks to the top of the page to remind you which site you're on. Your browser is already doing that for you.

There are no floating buttons saying things like "contact me" or "pay me" or "check out this service I totally didn't just write this post to hawk on the red or orange sites". I don't put diagonal banner things across the corners. I don't blur it out and force you to click on something to keep going. TL;DR I don't cover up the content, period.

I don't mess with the scrolling of the page in your browser. You won't get some half-assed attempt at "smoothing" from anything I've done. You won't get yanked back up to the top just because you switched tabs and came back later.

I don't do some half-assed horizontal "progress bar" as you scroll down the page. Your browser probably /already/ has one of those if it's graphical. It's called the scroll bar. (See also: no animations.)

I don't litter the page with icons that claim to be for "sharing" or "liking" a post but which frequently are used to phone home to the mothership for a given service to report that someone (i.e., YOU) has looked at a particular page somewhere. The one icon you will find on all posts links to the "how-to" page for subscribing to my Atom feed, and that comes from here and phones home to nobody.

I don't use "invisible icons" or other tracker crap. You won't find evil 1x1s or things of that nature. Nobody's being pinged when you load one of these posts.

I don't load the page in parts as you scroll it. It loads once and then you have it. If you get disconnected after that point, you can still read the whole thing. There's nothing more to be done.

I don't add images without ALTs and/or accompanying text in the post which aims to describe what's going on for the sake of those who can't get at the image for whatever reason (and there are a great many). (Full disclosure: I wasn't always so good at writing the descriptions, and old posts that haven't been fixed yet are hit or miss.)

I don't do nefarious things to "outgoing links" to report back on which ones have been clicked on by visitors. A link to example.com is just <a href="http://example.com/">blah blah blah</a> with no funny stuff added. There are no ?tracking_args added or other such nonsense, and I strip them off if I find them on something I want to use here. If you click on a link, that's between you and your browser, and I'm none the wiser. I really don't want to know, anyway. I also don't mess with whether it opens in a tab or new window or whatever else.

I don't redirect you through other sites and/or domains in order to build some kind of "tracking" "dossier" on you. If you ask for /w/2024/12/17/packets/, you get that handed to you directly. (And if you leave off the trailing slash, you get a 301 back to that, because, well, it's a directory, and you really want the index page for it.)

I don't put godawful vacuous and misleading clickbait "you may be interested in..." boxes of the worst kind of crap on the Internet at the bottom of my posts, or anywhere else for that matter.

My pages actually have a bottom, and it stays put. If you hit [END] or scroll to the bottom, you see my footer and that's it. It won't try to jam more crap in there to "keep you engaged". That's it. If you want more stuff to read, that's entirely up to you, and you can click around to do exactly that.

I don't make any money just because someone lands on one of my posts. You won't find ads being injected by random terrible companies. In fact, keeping this stuff up and available costs me a chunk every month (and always has). I sell the occasional book and get the occasional "buy me a cup of tea or lunch" type of thing, and I do appreciate those. (I tried doing paid watch-me-code "lessons" years ago, but it really didn't go anywhere, and it's long gone now.)

I'm pretty sure everything that loads as part of one of my posts is entirely sourced from the same origin - i.e., http[s]://rachelbythebay.com/ something or other. The handful of images (like the feed icon or the bridge pic), sounds, the CSS, and other things "inlined" in a post are not coming from anywhere else. You aren't "leaving tracks" with some kind of "trust me I'm a dolphin" style third-party "CDN" service. You connect to me, ask for stuff, and I provide it. Easy.

I say "pretty sure" on the last one because there are almost 1500 posts now, and while my page generation stuff doesn't even allow for an IMG SRC that comes from another origin, there are some "raw" bits of HTML in a few old weird posts that break the usual pattern. I don't think I've ever done an IMG or SOURCE or LINK from off-site in a raw block, though.

I don't even WANT stuff coming from off-site, since it tends to break. I find that I can really only rely on myself to keep URLs working over time.

Phew! That's all I can think of for the moment.

I don't think people really appreciate what kind of mayhem some of their software gets up to. I got a bit of feedback the other night from someone who's been confounded by the site becoming unreachable. Based on running traceroutes, this person thinks that maybe it's carrier A or carrier B, or maybe even my own colocation host.

I would have responded to this person directly, but they didn't leave any contact info, so all I can do is write a post and hope it reaches them and others in the same situation.

It's not any of the carriers and it's not Hurricane Electric. It's my end, and it's not an accident. Hosts that get auto-filtered are usually running some kind of feed reader that flies in the face of best practices, and then annoys the web server, receives 429s, and then ignores those and keeps on going.

The web server does its own thing. I'm not even in the loop. I can be asleep and otherwise entirely offline and it'll just chug along without me.

A typical timeline goes like this:

• 00:04:51 GET /w/atom.xml, unconditional.
  Fulfilled with 200, 502 KB.
  
• 00:24:51 GET /w/atom.xml, unconditional.
  Rejected with 429.
  Advised (via Retry-After header) to come back in one day since they are unwilling or unable to do conditional requests.
  
• 00:44:51 GET /w/atom.xml, unconditional.
  Same 429 + Retry-After.
  
• 01:04:51 GET /w/atom.xml, unconditional.
  Just like last time.
  
• 01:24:51 GET /w/atom.xml, unconditional.
  Same thing, again.

Somewhere around here, the web server decided that it wasn't being listened to, and so it decided it was going to stop listening, too.

Some time after this, it will "forgive" and then things will work again, but of course, if there's still a bad feed reader running out there, it will eventually start this process all over again.

A 20 minute retry rate with unconditional requests is wasteful. That's three requests per hour, so 72 requests per day. That'd be about 36 MB of traffic that's completely useless because it would be the same feed contents over and over and over.

Multiply that by a bunch of people because it's a popular feed, and that should explain why I've been tilting at this windmill for a while now.

If you're running a feed reader and want to know what its behavior looks like, the "feed reader score" project thing I set up earlier this year is still running, and is just humming along, logging data as always.

You just point your reader at a special personalized URL, and you will receive a feed with zero nutritional content but many of your reader's behaviors (*) will be analyzed and made available in a report page.

It's easy... and I'm not even charging for it. (Maybe I should?)

...

(*) I say _many_ of the behaviors since a bunch of these things have proven that my approach of just handing people a bunch of uniquely-keyed paths on the same host is not nearly enough. Some of these feed readers just go and make up their own paths and that's garbage, but it also means my dumb little CGI program at /one/particular/path doesn't see it. It also means that when they drill / or /favicon.ico or whatever, it doesn't see it. I can't possibly predict all of their clownery, and need a much bigger hammer.

There's clearly a Second System waiting to be written here.

As usual, the requirements become known after you start doing the thing.

Hey there. Are you one of these "Fediverse" enthusiasts? Are you hard core enough to run an instance of some of this stuff? Do you run Pleroma? Is it version 2.7.0? If so, you probably should do something about that, like upgrading to 2.7.1 or something.

Based on my own investigations into really bad behavior in my web server logs, there's something that got into 2.7.0 that causes dumb things to happen. It goes like this: first, it shows up and does a HEAD. Then it comes back and does a GET, but it sends complete nonsense in the headers. Apache hates it, and it gets a 400.

What do I mean by nonsense? I mean sending things like "etag" *in the request*. Guess what, that's a server-side header. Or, sending "content-type" "and "content-length" *in the request*. Again, those are server-side headers unless you're sending a body, and why the hell would you do that on a GET?

I mean, seriously, I had real problems trying to understand this behavior. Who sends that kind of stuff in a request, right? And why?

This is the kind of stuff I was seeing on the inbound side:

raw_header {
  name: "user-agent"
  value: "Pleroma 2.7.0-1-g7a73c34d; < guilty party removed >"
}
raw_header {
  name: "date"
  value: "Thu, 05 Dec 2024 23:52:38 GMT"
}
raw_header {
  name: "server"
  value: "Apache"
}
raw_header {
  name: "last-modified"
  value: "Tue, 30 Apr 2024 04:03:30 GMT"
}
raw_header {
  name: "etag"
  value: "\"26f7-6174873ecba70\""
}
raw_header {
  name: "accept-ranges"
  value: "bytes"
}
raw_header {
  name: "content-length"
  value: "9975"
}
raw_header {
  name: "content-type"
  value: "text/html"
}
raw_header {
  name: "Host"
  value: "rachelbythebay.com"
}

Sending date and server? What what what?

Last night, I finally got irked enough to go digging around in their git repo, and I think I found a smoking gun. I don't know Elixir *at all*, so this is probably wrong on multiple levels, but something goofy seems to have changed with a commit in July, resulting in this:

  def rich_media_get(url) do
    headers = [{"user-agent", Pleroma.Application.user_agent() <> "; Bot"}]

    with {_, {:ok, %Tesla.Env{status: 200, headers: headers}}} <-
           {:head, Pleroma.HTTP.head(url, headers, http_options())},
         {_, :ok} <- {:content_type, check_content_type(headers)},
         {_, :ok} <- {:content_length, check_content_length(headers)},
         {_, {:ok, %Tesla.Env{status: 200, body: body}}} <-
           {:get, Pleroma.HTTP.get(url, headers, http_options())} do
      {:ok, body}

Now, based on my addled sense of comprehension for this stuff, this is just a guess, but it sure looks like it's populating "headers" with a user-agent, then fires that off as a HEAD. Then it takes the *incoming* headers, adds them to that, then turns the whole mess around and sends it as a GET.

Assuming I'm right, that would explain the really bizarre behavior.

There was another commit about a month later and the code changed quite a bit, including a telling change to NOT send "headers" back out the door on the second request:

  defp head_first(url) do
    with {_, {:ok, %Tesla.Env{status: 200, headers: headers}}} <-
           {:head, Pleroma.HTTP.head(url, req_headers(), http_options())},
         {_, :ok} <- {:content_type, check_content_type(headers)},
         {_, :ok} <- {:content_length, check_content_length(headers)},
         {_, {:ok, %Tesla.Env{status: 200, body: body}}} <-
           {:get, Pleroma.HTTP.get(url, req_headers(), http_options())} do
      {:ok, body}
    end
  end

Now both requests call a function (req_headers) which itself just supplies the user-agent as seen before.

What's frustrating is that the commit for this doesn't explain that it's fixing an inability to fetch previews of links or anything of the sort, and so the changelog for 2.7.1 doesn't say it either. This means users of the thing would have no idea if they should upgrade past 2.7.0.

Well, I'm changing that. This is your notification to upgrade past that. Please stop regurgitating headers at me. I know my servers are named after birds, but they really don't want to be fed that way.

...

One small side note for the devs: having version numbers and even git commit hashes made it possible to bracket this thing. Without those in the user-agent, I would have been stuck trying to figure it out based on the dates the behavior began, and that's never fun. The pipeline from "git commit" to actual users causing mayhem can be rather long.

So, whoever did that, thanks for that.

One of the more interesting things you can do with systemd is to use the "socket activation" feature: systemd itself opens a socket of some sort for listening, and then it hands it over to your program, inetd-style. And yes, I know by saying "inetd-style" that it's not even close to being a new thing. Obviously. This is about what else you can do with it.

Like in my previous tale about systemd stuff, you can add "deny" and "allow" rules which bring another dimension of filtering to whatever you're doing. That applies for the .socket files which are part of this socket activation thing. It can even forcibly bind it to a specific interface, i.e.:

[Socket]
ListenStream=443
IPAddressDeny=any
IPAddressAllow=192.0.2.0/24
BindToDevice=wg0

That gives you a socket which listens to TCP port 443 and which will do some bpf shenanigans to drop traffic unless the other end is in that specific /24. Then it also locks it down so it's not listening to the entire world, but instead is bound to this wg0 interface (which in this case means WireGuard).

This plus the usual ip[6]tables rules will keep things pretty narrowly defined, and that's just the way I like it.

I did this in a big way over the past year, and then never rebooted the box in question after installing such magic. Then earlier this week, I migrated that system's "personality" to new hardware and that meant boots and reboots here and there, and wasn't it weird how it was spending almost two minutes to reboot every time? What the hell, right?

Digging into the systemd journal turned up that some of the "wg" stuff wasn't coming up, and it sure looked like a dependency cycle. A depends on B, which depends on C, which depends on D, which depends on A again? If not for the thing eventually timing out, it wouldn't have EVER booted.

I'm thankful for that timeout, since the rest of the box came up and I was able to get into that little headless monster to work on the problem.

The problem is basically this: if you have a .socket rigged up in the systemd world, you by default pick up a couple of dependencies in terms of sequencing/ordering at boot time, and one of them is "sockets.target". Your foo.socket essentially has a "Before=sockets.target", which means that sockets.target won't succeed until you're up and running.

But, what if your foo.socket has a BindToDevice that's pointing at WireGuard? You now have a dependency on that wg0 thing coming up, and, well, at least on Debian, that gets interesting, because it ("wg-quick@wg0" or similar) wants basic.target to be done, and basic.target in turn wants sockets.target to happen first.

foo.socket waits on wg waits on basic waits on sockets waits on foo.socket. There's the cycle.

Getting out of this mess means breaking the cycle, and the way you do that is to remove the default dependencies from your .socket file, like this:

[Unit]
DefaultDependencies=no

After that, it's on you to set up the appropriate WantedBy, Wants, Before or After declarations on your .socket to make sure it's attached to the call graph somewhere.

I should mention that it took a LOT of rebooting, journal analysis, cursing, and generally complaining about things before I got to this point. If you're in a mess like this, "systemd-analyze dump <whatever>" is generally your friend, because it will point out the *implicit* dependencies which are just as important but which won't show up in your .socket or .service files. Then you get to sketch it out on paper, curse some more, and adjust things to not loop any more.

There doesn't seem to be a good way to spot this kind of problem before you step in it during a boot. It's certainly not the sort of thing which would stop you before you aimed a cannon directly at your foot. Apparently, "systemd-analyze verify <whatever>" will at least warn you that you have a cycle, but figuring out how you got there and what to do about it is entirely up to you. Also, if you don't remember to run that verify step, then obviously it's not going to help you. I only learned about it just now while writing up this post - far too late for the problem I was having.

I sure like the features, but the complexity can be a real challenge.

What in the name of clowntown is going on here?

ip - - [04/Dec/2024:23:18:21 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:22 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:22 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:23 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:23 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:23 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:23 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:24 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:24 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:24 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:24 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:25 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:25 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:25 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:25 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:26 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:26 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:26 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:26 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:26 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:27 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:27 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:27 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:27 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:28 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:28 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:28 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:28 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:29 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:29 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:29 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:29 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:30 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:30 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:30 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:30 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:30 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:31 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:31 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:31 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:31 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:32 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:32 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:32 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:32 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:32 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"

Note: the post index page isn't the feed. It has never been the feed. Even still, what's up with the unconditional requests, and several per second? What is that supposed to accomplish? In what world does that make sense?

Do you ever wonder if feed reader authors point their stuff at their own servers? You'd think they'd notice this kind of thing.

Oh, also, this will no longer work.

Yeah, I'm still here. I'm still processing things from last month. Obviously.

...

I just spent a few CPU hours on cracking something stupid and figured I'd share the results so nobody else goes through the same thing. Save some time and power, you know the thinking.

tukC0rEjV4g3g --> lvl7dbg

... and if this means anything to you, then you'll also be glad to know that the default value for "PermitRootLogin" of "prohibit-password" should be in place so nobody can get into your device that way. Test it yourself to be sure if you're worried. I sure did.

I know that if you go back far enough in these posts of mine, you will find some real crap in there. Sometimes that's because I had a position on something that turned out to not be very useful, or in some cases, actively harmful. This sucks, but that's life: you encounter new information and you are given the opportunity to change your mind, and then sometimes you actually do exactly that.

Recently, I realized that my position on something else has changed over time. It started when someone reached out to me a few weeks ago because they wanted me to get involved with them on some "employee metrics" product. It's some bullshit thing that has stuff like "work output" listing how many commits they've done, or comments, or whatever else. I guess they wanted me to shill for something of theirs, because from my posts, clearly I was such a fan of making that kind of tool, right?

I mean, sure, way back in 2004-2006, I was making all kinds of tools to show who was actually doing work and who was just sitting there doing nothing. I've written about a number of those tools, with their goofy names and the "hard truths" they would expose, showing who's a "slacker" and all of this.

When this company reached out, I did some introspection and decided that what I had done previously was the wrong thing to do, and I should not recommend it any more.

Why? It's surprisingly simple. It's the job of a manager to know what their reports are up to, and whether they're doing a good job of it, and are generally effective. If they can't do that, then they themselves are ineffective, and *that* is the sort of thing that is the responsibility of THEIR manager, and so on up the line. They shouldn't need me (or anyone else) to tell them about what's going on with their damn direct reports!

In theory, at least, that's how it's supposed to work. That's their job: actually managing people!

So, my new position on that sort of thing is: fuck them. Don't help them. Don't write tools like that, don't run tests to see if your teammates will take care of basic "service hygiene" issues, and definitely don't say anything substantive in a performance review. None of it will "move the needle" in the way you think it will, and it will only make life worse for you overall. "Peer reviews actually improve things" is about the biggest crock of shit that people in tech still believe in.

Once again, if management is too stupid to notice what's going on, they deserve every single thing that will happen when it finally "hits the fan".

Make them do their own damn jobs. You have enough stuff to do as it is.

...

I feel like giving this a second spin right here in case I failed to reach some of the audience with the first approach. Here's a purely selfish way of looking at things, for those who are so inclined.

Those tools I wrote 20 years ago didn't really indicate who was slacking at working tickets or whatever. What they *actually indicated* was that the management at Rackspace, by and large, had no clue what was going on right under their noses. And, hey, while that was true, that can be a dangerous thing to say! You want enemies? That's a great way to get them.

So, why expose yourself? Suppress the urge to point out who's slacking. It will only come back on you.

I know I have a weird and kind of skewed view of the world, since I expect nobody else to care about some things, and then the world goes and surprises me by jumping on it anyway. About six months ago, I expected the worst for my whole "let's improve the feed reader ecosystem" thoughts, and figured it would go unnoticed, and yet it's been anything but.

Clearly, I am miscalibrated for what the world is willing to do. So, with that in mind, I figured that I need to bring more things up in this context to see what the reaction is instead of just discarding it early due to my own doubts.

Tonight's concern goes like this: I was flipping through some old posts and ran across something from 2013 where I said "your simple (shell) script is someone else's bad day". It was inspired by the "onboarding" process for dev servers at FB which I was going through that week, and it was a total shit-show. It had plenty of the aforementioned scripts that you couldn't restart without making things worse. It took way too long for me to get my hands on a Linux box that had things set up so I could crank on stupid www stuff.

In re-reading that post the other night, it got me wondering what I could do in this space to try to raise awareness of various topics. Maybe I'd look at a few old posts, decide that there's a cluster of problems ("idempotence" in this case), and that maybe it would be a net win for humanity if more people were slightly better at it. Then I'd try to address that directly... somehow.

My goal would be to get more people in this life who, at least for that topic, look at everything they encounter and go "what if this fails, and is restarted". I basically would love to see more people going "what if" for that particular topic. Worry about the robot and the five quarts of oil. (Go look at the 2013 post if that made no sense.)

I don't know if that's plausible. I actually worry that it's not even possible for most people, and that caring about such things is some weird genetic corner case that is born, not made, and it would be unreasonable to expect it from the rest of the world.

This is where I need input. I have this feeling that such topics are not seen as important by reasonable people, and that they honestly never will be. Just because it matters to me is no indication that it will resonate anywhere else. That much has been made crystal clear over the past few years.

Wouldn't it be amazing, though? Just imagine it: groups of people heading out into the world armed with additional things to consider when designing something, and the resulting improvements in reliability for the things they work on down the road.

Then we'd have to continue on to some other fundamental topic, and try to move the needle on that, too.

Finally, I should mention another possible outcome: those same people go into the world with that knowledge, go to apply it, and run into the same brick walls that some of us have already been hitting because nobody else sees the point. They get all bitter and dejected, emit their own "delightful burnout sauce" (I love that line, btw), and start writing their own screeds on things like this, and we start over.

You can bake an amazing cake, but if they aren't hungry for cake, well, you might have just wasted a bunch of time.

Oh no, look out, it's another list of feed reader behaviors. This is based on my observations for those test keys which have been active in the past few days. Anything that stops polling eventually ages out of the report and won't show up here.

My thanks to those feed authors and contributors who have been cranking away to tweak things here and there. It's making a clear difference in the real world. I see more 304s and fewer 429s in general, and that makes me very happy.

Here is what I have to say for this time around, with some amount of aggregation applied. Some entries cover multiple test keys if the behavior is the same.

...

No UA. Sends bogus "" INM at startup, then makes up IMS values subsequently. Broken caching behavior that will miss updates in the real world.

Miniflux dev stuff. Seems a whole lot better now. Need to do some more pathological injections to see how they handle it.

Miniflux, older versions. Needs to upgrade. (Several different test keys).

SpaceCowboys Android RSS Reader 2.6.{31,32,33}. It's still doing unconditionals seemingly at random, and sometimes has bad timing. No idea what the hell is going on here.

FreshRSS dev stuff. Seems decent of late. This one also seems to be honoring Cache-Control "max-age" stuff, which is a bonus I never expected anyone to do. That's awesome. The useless referrers are also gone, so hooray! Lots of good stuff happening here.

FreshRSS/1.24.3. Didn't get tripped up on my last round of pathological header values. Will probably get even better when it picks up the latest dev work, like dumping the referrers (see earlier dev entry).

Audrey2 RSS reader, various versions. Seems pretty chill. Also seems to be honoring Cache-Control max-age! Great stuff.

Emacs Elfeed 3.4.1. No idea what controls the polling interval but it's doing fine. Some early weirdness that might be multiple instances running in parallel but nothing recently.

Slackbot. Broken beyond belief. Must have been some intern's project that's been left online to irritate people. Sends HEADs, gets 405s, sends more. Probably will force me to write "let's see what happens when you start getting 403s and 404s and 410s" and whatnot, and then I'll complain about it not honoring those, too. Watch and see.

Newsboat, various versions. Probably okay now in terms of caching behavior but I need to shove some more pathological cases at it to be sure.

feedmail.org/0. This one looks like it's trying desperately to honor the max-age thing, but sometimes it arrives a hair too early - on the order of about 100 msec here. So weird. How damn often is it considering these polls for it to be able to hit it that close? Does it ever rest?

SpaceCowboys RSS Reader, 2.6.{29,30,31,32,33}. This one doesn't seem to be doing the unconditionals or short timing any more. It did some early on but then it stopped the silliness.

Unread RSS Reader. Plugging away at a 21 minute interval, up from 15. At least they're conditional. (At least two distinct test keys.)

Some browser UA. Godawful timing, always unconditional. This would trip automated blocks on the real site, no doubt.

cry-reader v0.0. No idea how it picks its intervals, but they're always far enough apart and are always conditional, so that's great. Haven't managed to trip it up with any pathological header values.

Netvibes. Same super broken caching behavior, still.

SpaceCowboys ... etc ... 2.6.31. Never does IMS, only occasionally does INM, wildly unpredictable polling intervals. Not good behavior. No idea how there can be so much variance between these things (see above).

NextCloud-News dev. The caching behavior is getting better, and -huzzah- it has a real version number at last! This means it will be possible to treat all of those old versions as damaged and route around them. Would be nice if it didn't keep pulling / 3x and favicon 1x for every. single. feed. request. Those are wholly unnecessary.

NextCloud-News/1.0. This version is heavily broken in the caching department, but help is on the way (see dev version above). If you're running this, you should plan to upgrade to pick up those essential fixes.

NewsBlur. Usually okay but sometimes it does crazy stuff like 3 *unconditional* polls in two seconds. What? Why?! (UA is also crazy long since it's waving a dead chicken in the form of including a complete forgery of a Safari UA, (nested parens) and all).

Friendica, various versions. 100% unconditional requests. Occasionally shows up < 10s after the previous poll. Not good.

TTRSS. Not recommended.

ureq, various versions. Seems okay. Occasional short-spacing between polls, but at least they're conditional.

NetNewsWire. Multiple distinct test keys, but none of them have a version number so there's no way to know if anything's changed upstream. Since I started writing this list, one instance came back fast enough to demonstrate that it still has the buggy caching behavior.

haven, 100% unconditional, bumpy timing, unchanged from before.

inforss, multiple versions. Seems fine.

curl, various versions. It does conditionals properly and just has the usual 59m/60m fencepost scheduling thing.

Inoreader/1.0. This thing periodically flips to unconditional mode, and also has the 59m/60m thing going on. On the prod side of the house, it gets itself auto-blocked fairly often for bad behavior.

Feedbin. Seems fine now. (Three distinct test keys, same behavior.)

NetNewsWire. Probably still buggy - was about three weeks ago, at least. Need to see what happens when I do a fresh round of tests.

Liferea/1.15.3. Something is very wrong with the caching on this thing. It drops to unconditionals a lot, and recently started sending some bullshit 1970 IMS header with things like 5s and 14s intervals. No idea what the hell is going on there.

rawdog/2.24rc1. 138 days with no complaints from me.

NetNewsWire. Just like the last one: was showing signs of buggy cache behavior before, and probably will again when I next launch into the tests.

NewsBlur. Not quite as animated as the earlier instance but still uncorks an unconditional from time to time for some reason.

rss2email, various versions. Occasional unconditionals for some reason.

Newsboat/2.36.0. Jumpy timing, sometimes too fast. At least they're conditional.

CommaFeed, various versions. Has had caching bugginess in the past, and did during the last test. Probably still does.

Some browser UA. Keeps banging away with <1s unconditionals. Would definitely get auto-blocked in prod. Maybe multiple instances are running, but why would one always send unconditionals?

theoldreader.com. This seems to have switched behaviors to conditional polls in early September. I see a pair of too-quick polls since then, but even those were conditional so it's not that bad. This is also way better than before. Thank you for the fix, devs!

Generic "Go-http-client/1.1" UA. Seems fairly chill.

Rapids. This one turned a corner with some upgrades back in July and has been delightful since.

er0k feeds. Was upgraded at some point and then didn't trip over any of my "inject some crazy" stuff over the past month. Smooth sailing.

Emacs Elfeed 3.4.1. Mildly wacky behavior that suggests multiple instances banging away at the same key - either that, or it has really scary caching bugs. I hope it's the former and not the latter.

walrss/0.3.7. 145 days of complaint-free behavior. Nice.

Yarr/1.0. Just needs to tweak the 59m/60m fenceposting thing.

Some browser UA. This one settled down into a good polling interval and does conditional requests properly. Good enough.

Feedly. Still doing the unconditional requests daily for some reason. Fail. (Multiple distinct test keys, same behavior.)

feedparser/6.0.2. Just the 59m/60m thing. Otherwise is managing to do conditional (INM) requests so it's not the end of the world.

Reeder, various versions. This one acts strangely sometimes, and sends the wrong value in an INM header, like it got hit over the head and reverted to something from a few days *or weeks* before. Also sometimes sends unconditional requests. Could be multiple instances, so don't do that. One key = one instance of one reader. Anything else is chaos. (If this is you, ask me for more keys!)

Lots of mixed up stuff, but of late it's something claiming to be a browser that bangs away with unconditionals at 10 minute intervals. Would be auto-blocked in prod in a jiffy.

Yarr/1.0. Sends conditionals but has terrible timing - *way* too fast on those polls.

Bloggulus/0.4.0. Seems to be doing fine.

newsraft/0.27. This got some tweaks recently to deal with pathological ETag/LM behavior. Thanks for that!

feedparser/6.0.10. Still running too quickly: 29m/30m. They're conditional, but ... eh.

Mojolicious (Perl). 147 days of polls with no issues to report.

Some browser UA. Seems to be fine.

com.vanniktest.rssreader, various versions. Weird timing, as before. No idea what kind of scheduling it's using - repeats within 1-2s make no sense.

Broadsheet/0.1. Late update to this post: it's doing fine, but there's an unusual bit of wackiness showing up here. The URL it's using is checked into a github repo, and something or someone (not the author, and not Broadsheet) is pulling them. Maybe this is github trying to be "helpful". Joy.

[This entry originally said it was changing UAs and doing wild unconditional requests from time to time. It is not. Something else with the key is taking it for a ride. Raar.]

Occasional odd behavior - UA changes to something else, and it fires off multiple unconditional requests within a second. No idea if this is someone running multiple programs against the same key. (If this is you, ask for another key, or maybe get two, and stop using this one.)

feedbase-fetcher.pl/0.5. This settled into a good groove.

Artykul/1.0. The clown leak. I really need to just block them, or at least, start sending them really broken stuff to see what happens. Installing their app shouldn't have subscribed their backend to my early testing URL, but it did, and uninstalling it didn't stop it!

Okay, it seems I missed the mark yesterday while telling my story about a dumb little backup scheme, because a bunch of people have reached out to ask "why not sshfs"?

So, okay, some assumptions, first. I'm assuming from some basic reading that sshfs is one of those things that lets you take a filesystem that's on a distant host and mount it (with FUSE, sigh) on a local host. If that is in fact what it does, that is very much *not* what I want.

I don't want my stuff to be visible as a filesystem on the remote host. That is, I don't want the copies of my data to be sitting there, accessible to anyone who can get on that machine. I want it to be a amorphous blob of (obviously?) encrypted data. The only place where it should make sense is on the client machine after I've supplied the key.

Speaking of keys, that key material (like a typed passphrase) never leaves the client system, since all of the crypto stuff happens locally. If this stuff was a regular filesystem on the other end, then that host would need access to the key, and then it's no longer really under my control, right?

Given all that, let me come at this again.

I have a host. I want to back it up to something that's not in the same physical location, and ideally is rather far away. Having different power companies, ISPs, you name it? That's all a good thing in this case. (Also, "different fault lines" is always a consideration here.)

I have access to that host, but assume I don't have root access to it for whatever reason. I'm just a user on there, and I can ssh in, and I have a healthy amount of disk quota. I'm also welcome to use oodles of (network) bandwidth to do whatever I need to do.

I don't, however, want the contents of my backed-up data to be visible to anyone on that host, _even if_ they have root on it, whether legitimately or otherwise. The remote host is just being a dumb block device for me.

I don't want to have any extra daemons running that would provide network access to the blob o' bits that constitutes the encrypted image. It's a whole bunch of extra complications that I don't need or want, since ssh gives me a perfectly good pipe without changing anything fundamental about how that system works.

So, the end result is that I extend nbd over a wacky ssh connection to a Unix domain socket on the source machine, do my crypto magic to make it into a viable filesystem, then fsck it, mount it, and rsync my stuff onto it.

The only part of this I had to create was the aforementioned wacky ssh connection. nbdclient itself won't do a "ssh to far end, start nbdkit server, and then stay alive copying bytes around" thing. It's all about getting the connection up and handing it off to the kernel. It only sticks around if it's doing TLS mode and since that means running a persistent server on the far end, I'm not using it that way.

That's it. It's just one of those dumb things you build to make other stuff possible. Tools to make tools.

Sometimes I do dumb things to solve my own problems. This is one of those times. In this case, I wanted something that would give me access to a block device on a physically distant machine for backup purposes. I didn't want to do anything particularly fancy on the distant box, so rootly powers are out of the question. I just need disk space, a bit of bandwidth, and some CPU time every now and then.

Here's how it works. Perhaps you have heard of "nbd" if you're in the Linux world. It lets you load a kernel module on the client machine and then it'll turn a network type connection into a block device. That is in fact right in the name: "nbd" equals "network block device". Nice, right?

I wanted this, but didn't want it making a "bare" connection to the far end. By default, it's just a plain old TCP session to the other side. While you can rig it to use TLS, there are a bunch of problems with this. It means you need to leave a daemon running on the far end, and again, if you aren't root out there, that can be troublesome. It means the distant host ends up stuck with a listening port, and that's not always nice. It also forces you to deal with getting the whole certificate thing hooked up. Hope you're good at wrangling OpenSSL.

This seemed wholly unnecessary to me. I already have ssh access to the machines in question, so that right there will give me a relatively solid transport. I just needed to convince the nbd stuff to use it.

Now, let you stop you right here: if you're saying "SOCKS proxy" and/or "port forwarding", you have missed the part where I'd rather not have a persistent daemon on the far end that's listening on some port, *even if* it's "merely bound to loopback". That's still something running as me that presents an unnecessary orifice to unwelcome visitors. No thank you.

There's another way the nbd client can work: it will totally connect to a stream-style Unix domain socket. As long as whatever is on the other end of that socket speaks the right language, it doesn't matter where it is or what it is. Thus, I needed to make a Unix domain socket reach over the network to the other end of a ssh connection.

Here's what happened: I wrote some grungy plumbing stuff (my specialty) that sets up a Unix domain socket on my local machine. It does it in a directory where only I can access it, and it waits for a connection. Once it gets one, it forks, and the child fires up a ssh to the far end to invoke the nbd server (a trivial userspace thing I can leave in my personal bin directory).

Meanwhile, the parent process of the client sticks around and fires up a pair of threads to do a bucket-brigade thing. It takes any data from the Unix domain socket connection it just received and flings it at the ssh connection. The other one does the reverse.

This keeps going until one of the fds shows up as disconnected, at which point it shuts down everything and exits politely.

The other part of this is a currently a bit of *local* rootly scripting madness which points the NBD client at that socket, then does the crypto gunk to attach it, then fscks it, mounts it, and fires up rsync. Then after it's done, it undoes everything and declares victory.

It's not meant to be fast, and it's definitely not meant to be beautiful, but it does work. My only footprint on the far end is a giant blob of a file that is entirely meaningless to anyone without a suitable key. The data is only ever seen in a usable form here on the client machine which already has access to the original data by definition. I didn't have to install anything special on the far end, I don't have to run any daemons, and I didn't need root out there.

It's the type of thing that you can stand up and just let it run every now and then, kind of like Time Machine on a Mac. You hope you never need to use it, but it's there if you really get stuck somehow.

So, yeah, when my part of CA "falls into the ocean", I will bob up to the surface, swim to shore, and then walk to where my offsites are and restore from backups. You know, that whole thing.

Just basic sysadmin stuff. Nothing fancy.

I was successfully nerd-sniped a few days ago and figured I'd share my proposed solution with everyone just in case they could benefit from it. I've added a few of my own constraints based on expectations for how things could go wrong. So, if this seems familiar, maybe it is, but I've made it a little more complicated.

The situation is basically this: there's a large space with a bunch of dumb Linux boxes which are attached to displays of some sort. Different things are displayed depending on where it is in the space. This means the Linux boxes need some sense of identity to be able to tell the server "I need this particular set of stuff". They're "hands off" - nobody wants to log in to them to manage them.

So now let me present some of the issues:

Problem: relying on IP assignments is no good, since the DHCP mechanisms for that network can't be trusted. Things are sufficiently flaky to where you can't rely on it, and part of this is from the next item.

Problem: the hardware is wonky enough to where it gets replaced sometimes, so the MAC addresses also end up changing. This means any other on-board identifier (board id, cpuid, whatever) would also change. It would be nice if someone didn't have to keep updating mappings on the server every time the hardware got shuffled around.

Problem: it's a PITA to stamp out unique images for these things. The goal is to be able to stamp out a single image that's identical for all of them, and then have them all boot from that and figure out their identities some other way. (The same goes for netbooting, if you're thinking of that already.)

Problem: any other "ID marker" type solution that involves twiddling the set of installed hardware somehow means that you have to be sure each one ends up in a specific spot. If you have relatively unskilled people running around installing these things, this might not be what you want. It's easier to just say "make sure there's one plugged in at every space on this list" without worrying about matching specific units to specific spots, in other words.

Wouldn't it be cool if you could just grab a box, plug it in a given spot, and it would somehow start displaying the right set of media? That's what I was thinking when I came up with this idea.

So, how do you do it? My "solution" (which has its own caveats) comes in the form of LLDP.

For those who haven't encountered it yet, here's the scoop: certain higher-end Ethernet switches have the potential to _multi_cast identifying frames every so often. Any given station on such a switch might get a LLDP frame from it a couple of times per minute.

In that frame, you can see interesting things, like the MAC address(es) of the switch, its name, what OS it might be running (!), the management IP addresses (!!), and, finally, what the name of YOUR PORT is.

	Port Description TLV (4), length 4: eth2

        Port Description TLV (4), length 8: Gi1/0/24
          0x0000:  4769 312f 302f 3234

Yep, that's right, if you're on such a switch, you might well have something arriving on your interface every so often that says "oh by the way, you are connected to me on my Gi1/0/24". Another device on that same switch would get nearly the same announcement, but it might say "Gi1/0/48" or whatever else.

The point is: if you're that derpy little display machine, you now have a way to tell yourself apart from your "coworkers". You could build a tuple of (switch name, switch port) and kick that at the display server, and let it figure out what set of images/videos you're supposed to run.

Now, there's always a catch, and here it is: this assumes that you're not changing the mapping between ports on the switch and the actual RJ45s in the space. I'm assuming that port 23 on your switch will always be over there by the front door, port 24 will always be next to the bathrooms, and 25 will be by the patio door. You get the idea.

If those change, then yeah, you get to reconfigure everything, but at least it's still only on the server, and you don't have to deal with the dumb little display machines.

How do you see this? Well, tcpdump would be a start. A filter of "ether proto 0x88cc" will do the job, but make sure you stick a -v on there to make it actually expand the details (or it'll be pretty dull).

Some other things: a relatively low-end switch won't do anything in this space. It won't generate it, and it won't "absorb" it from other sources. So, if your listening post running tcpdump is plugged into such a beast, it might hear LLDP stuff coming from multiple other sources but it won't see anything from the switch itself. Yep, it's more than switches which potentially send these out. "Server class" systems with their own little management device ("BMC") glued onto them tend to do this, too.

Normally, such traffic would not cross the switch, but that requires a switch that actually "gets" it and knows to not forward it. Ordinary dumb switches will treat it just like any other bit of traffic and will send it on down the line.

I should also note that other protocols exist which serve similar purposes and it depends a lot on which switch vendor you've chosen. It might not be LLDP but it might be something else which ends in "DP".

If in doubt, sniff the network and find out!

I have a few more thoughts on the whole topic of colocation. First of all, Joel wrote in with a couple of tips beyond the basic "screwdriver and flashlight" that I mentioned. He says you should bring hearing protection, a step-stool or small ladder, and a jacket if you get cold. I like this thinking, and figured I'd expand on this for the benefit of those wondering what this all means.

First up, these places are LOUD. Everything you can imagine has fans on it. Obviously there are massive air handlers in the suites, but the (proper server class) computers and switches and everything else are also rocking a ton of fans. Some of them throttle back when the CPU load isn't too high, but a fair number of these things actually have rather high CPU load and so they never throttle back.

I mean, it's 2024, and people are writing CPU-bound computational stuff in languages that are interpreted, single-threaded, and slow as shit. OF COURSE they're running their CPUs as hard as they possibly can. But I digress.

So yes, it's loud as hell, and you could benefit from some kind of active protection. Just don't do what I did one time by cranking up the music in regular earbuds to cover the noise. Yeah, you might be able to hear your tunes, but you won't be hearing much else afterward. Stupid move, I know... now.

The step-stool or small ladder is not always a given. My particular cabinet isn't super tall and I'm able to reach all of it, but this might not apply for everyone else. Alternatively, you might need to use it to support something from below while doing an install, especially if you're flying solo.

The jacket is another one of those things that you might not appreciate until you've been on the inside. It's not like the 90s when everyone just had a giant room and did their very best to cool the entire thing down as far as possible. These days, there are "hot aisles" and "cold aisles": two rows of machines face each other across a walkway, and cold air is blown into there. Then the air goes through the machines thanks to all of those fans and ends up on the back side where it joins with hot air from yet another row that's also backed up to it. Finally, it's drawn into a chiller to start the process over again.

If it's a solid concrete floor type of setup, then the air will have to come from above, but if you're on a raised floor, it'll probably just emerge from beneath. If you're in that kind of setup, make note of what you wear before making a trip lest you turn into Marilyn Monroe.

(Yeah, I just gave some advice that only applies to a subset of the people who will ever read this and need it. Yep. That happened.)

The nature of things is that you will inevitably need to access your rack from both sides to reach certain parts of the equipment, so it will be really hot sometimes, and it'll be really cold other times. This is why I will amend that advice to "bring a jacket that zips".

Now, if you're in the Bay Area like me, you have hopefully long ago internalized the wisdom that "you will never be far from a light sweater", and so you *already* have one of those that goes everywhere with you. If so, you're set. Be ready to adjust it as appropriate.

Other stuff? Once you're past the "drop a Raspberry Pi in there" stage, you should not just jump on ebay and buy the first rack mounted server that looks like it'll work because you hate hardware and want it over with. This is because not all racks and not all servers are created equal, and you might find out the hard way that it won't work. I came super close to screwing up *hard* due to this.

I would recommend first going in with a tape measure to figure out exactly what you're working with. What are the posts like in the rack? Are they fixed in place or can they be adjusted? Are they round holes or square holes? How long can a server be while still fitting into the rack? Will the door(s) still be able to close and latch?

Bear in mind that it's not just the length of the machine, but you also have to include the loops of cables which emerge from it: power and Ethernet at the very least. You really don't want to force a regular power or network cable into a "hard 90" type of situation to make it all fit into the space. With a little research, you can get power cords which have their own built-in 90 degree turn on the end where it plugs into the machine that will let you claw back a bit of space, and likewise for the network stuff.

Or, you know, you can just buy a shorter machine and use normal cables.

Measure twice, buy once.

Another complication: if your provider gives you power by way of a "zero U" PDU (basically a big power strip that stands up vertically), that's both a plus and a minus. It's a plus in that you're not burning any rack units by definition: it's "zero U". But, it can be a problem because it still takes up space, and if you have anything really long in there, it'll probably bump into it. This constrains you to only using spots in the rack which are not blocked by the sneaky little PDU. It's just another 3-D Tetris problem for you to solve.

Late yesterday, I put up a post about how to get into colocation in about the crappiest way possible. I skipped a bunch of details just to get it out there. The inspiration was based on finding out just how many people have no idea that this business model even exists.

I used to work in a cousin of this space: managed and dedicated hosting, and here's how this all lined up:

Colocation (as mentioned yesterday) gives you some space, some power, bandwidth, a network allocation of some kind (and/or ability to route your own stuff), and hopefully some decent HVAC to keep everything cool. It's your hardware in their space, and if something breaks on that stuff, you get to fix it. What if a drive fails? That's on you. Your switch or router goes insane? Same deal: that's all you.

You might be able to get some "remote hands" service from the provider for very simple tasks: reboot something, take a picture of the (blue) screen, that kind of thing. Some also let you pay them to go and do other stuff with a higher degree of complexity. Read the fine print.

Managed hosting is where you get access to a box somewhere, and are usually given root on it. You can do about what you want to it, but beware of the "spheres of support" as we used to call it - the hosting company's support people will only go so far. You might want Debian, but they only do RHEL. You get the idea. You can probably ask them to do kernel upgrades, troubleshoot why it's being slow or seems to be down, install and configure certain things for you, and so on.

If the hardware fails, that's on them. They rip open the box and replace the part. You still have to figure out how to make your stuff work before, during, and after the event. They probably won't do any migrations for you, since everyone's setup is an odd little snowflake and only the customer has any chance of knowing how it all works.

Higher-end setups might offer hardware firewalls, load balancers, backup solutions, and more. Now, granted, it's been almost 20 years since I last did this myself - I assume someone's still doing this somewhere.

Dedicated servers are a step down from managed. You (probably) get root on a box somewhere, and you can do a lot of stuff on it. Outside of the absolute bare essentials provided by the hosting company (like a Red Hat Network entitlement), they aren't going to do much for you. They might have rigged up some magic hardware to let you power-cycle what's just a boring old whitebox without explicit out-of-band access. They might also have some netboot magic to let you shunt it into a "repair environment" if you really screw things up.

All of the servers used for hosting this site were of the "dedicated" persuasion initially: first at ServerBeach, then at SoftLayer (which became IBM). The SB machines were relatively crappy whiteboxes shoved into a bread rack somewhere, while the SL/IBM stuff was on a Real Rackmounted Server with out of band remote access and the whole bit. They handled the hardware, and I handled the software.

I mentioned some of the details last year when departing the IBM shit-show and the Texas shit-show at the same time. This ended the era of dedicated hosting for me and moved me into colocation.

These days, there's a cabinet in a nice conditioned space not too far away. I initially parked an old switch in there and a derpy little Raspberry Pi, just like yesterday's post implied. This was to give me a little "platform" while the rest of the stuff happened. A few days later, I got some "real" hardware, hung that in there, and then migrated everything to it from the SL/IBM box.

That was a bit over a year ago, and things are pretty much unchanged. This setup should just sit there and run and run and run until something breaks beyond that which can be handled in software. Then I get to drive out there and flip some parts around and make it go again.

It's not rocket science. This way of doing things has always been possible. You don't have to deal with clown providers pulling the rug out from under you. There are a few dimensions in which the provider can mess with me now:

1. They could have something terrible happen - think flooding or a fire. It could take the whole location offline for an extended period, and it could well destroy the equipment inside, such that merely relocating it to another spot would not be sufficient to revive things.

2. They could decide they're done with this kind of business model, and now they're going to boot out all of the little people who pay a pittance per month. Maybe they're going to sell out to the latest clown provider who wants to melt the ice caps and drown the polar bears with their "AI" crap. (This would also apply if the private equity vultures swoop in and try to hollow the place out before nuking it entirely.)

3. They could crank the prices up beyond that which I could afford.

4. They could do something that somehow makes the basics incompatible: they stop supplying AC power, conditioned air, or IP over Ethernet. (I know, this is ridiculous, but I might as well mention it.)

5. They could shack up with some really evil people such that I no longer want to do business with them.

My responses would be approximately this:

1. Acquire some other hardware, install a fresh OS and restore my stuff from backups, then park it somewhere else.

2, 3, 4, 5. Find a new spot, then schlep the hardware there.

...

If this is old hat to you, great! It means you're probably a grizzled 1990s sysadmin just like me, consarn it! This isn't for you, then.

This is for the newer folks who might not have realized that there's an alternative to paying tribute to one of the three churches of the Clown: M, G, or A. If you want to "get your stuff online", there are other ways... and there always have been!

I've been wanting to do this for a while: basically, to write a really snarky post about the bare minimum required to run your own hardware in a colocation environment. I'm talking about doing as little as possible, and possibly screwing up bigtime while you're at it.

Based on chats with some of my friends, it seems like this model is not well known. One of them had a situation a while back where everyone was now working from home due to COVID, and so they would all VPN back to the office park where the servers were, including their NAS "box-o-disks". It was obnoxiously slow and laggy and failed a lot, too.

I said he could probably find a spot to colocate that stuff given that he lives in a major metro area, and sure enough, he did. A few weeks later, everything was parked in there, and the resulting boost in connectivity (throughput and availability both) made everyone so much happier.

I think they were also able to dump the remaining cruft at the office park as a result. Win-win, right?

So, without further ado, here's the terrible list I just scribbled down rather quickly to get this going:

0. Scrounge up at least an old dumb Ethernet switch (or *gasp* a hub!) and some random-ass hardware that'll run Linux, like a Raspberry Pi, or some old PC box or whatever.

1. Install Linux on the box. Turn everything off but sshd. Turn off password access to sshd. If you just locked yourself out of sshd because you didn't install ssh keys first, STOP HERE. You are not ready for this.

If you somehow survived, continue as usual.

2. Find a place that you can physically access that will sell you some chunk of a cabinet, rack, shelf, or whatever else. You have to be able to get there somehow: walk, bike, drive, take the bus, whatever ... *while carrying equipment with you*. Keep that in mind.

If it's on the other side of the country (or planet), it might not be what you want, in other words.

3. Make a deal with the place. Prepare to throw some money at them.

4. Wait for them to set up your account. Get the networking details. Make sure they know you're just going to be running a stupid little switch and not a full-on router.

They'll probably carve out a tiny little block of v4 space where their router is one of the IP addresses, and you get to use the rest. They'll probably do the same with IPv6, only it'll be a /64 because why the hell not? (If they don't do IPv6, ask for your money back and run away.)

5. Configure the Linux box to match those details: static IP assignments! Yes! No DHCP here!

6. Figure out how to get your laptop to work while plugged into that same switch within your IP space, and WITHOUT any sort of wifi access while it's still at your house. You won't be able to see the outside world but you will be able to see your server. Filter out the world pre-emptively, because where you're going, you're not going to be behind a firewall.

7. Haul yourself and your stuff to the actual co-lo space: switch, Linux box, laptop, power cords, and Ethernet cables. Consider bringing a screwdriver and a flashlight (any halfway decent place will provide those for you, but you never know).

8. Plug everything in and turn it on. Connect to the server from your laptop. Verify that the server can get out to the Internet. Verify that you can get back to the server FROM the Internet. (That is, plug your switch into their network. Obviously.)

9. Go home and set up the rest from there - you know, DNS, loading stuff on the actual box, and so on.

...

To the usual broken people: knock it off. I know that you don't HAVE to use Linux, and can install a BSD, or Windoze, or whatever the hell else. I'm giving you A path, not THE path. Now shut up.

Hello again from the land of feed reader behavioral tests. I ran through the list of participants a couple of days ago and wrote up my results. This is only for those which had polled at least once in the past seven days relative to that point.

I'm going to group some of these together, but keep in mind that some behaviors are a function of however the user configures the program. Also, at this point I'm mostly focusing on their steady-state behavior, but any previously-reported goofiness at subscribe-time is still worthy of fixing for people so inclined.

Artykul's clown fetcher is still going. They're going to make me write a 404 generator and then a 410 generator, I just know it. Then I'm going to start complaining about them not honoring either of those. All unconditional, every 2-3 hours. Terrible.

feedbase-fetcher.pl/0.5 seems fine.

Broadsheet/0.1 seems fine.

com.vanniktech.rssreader, various versions has weird timing.

NextCloud-News/1.0 - all of these have broken IMS caching. There's a fix that's been submitted but it hasn't been merged, never mind rolled up into a release. Some old versions hammer the favicon.ico needlessly. I can only guess at who's running what, since they don't ever bump their version number. It's been 1.0 forever, and I assume this will continue unabated. This makes it really hard to direct particular version-based brokenness to an appropriate handler.

Some browser extension thing. Seems fine.

Mojolicious (Perl), seems fine.

NetNewsWire with buggy LM/ETag caching, as covered in a post a while back. I don't think this has been changed upstream yet. Also has terrible timing.

newsraft/0.25, which still has buggy ETag caching. It also has terrible timing.

Bloggulus/0.3.7, seems fine.

Miniflux/2.1.4 has buggy Last-Modified caching. Miniflux/2.2.0 does too. I thought it was squared away previously, but it's still doing crazy stuff when I do the pathological test cases to check for bugginess.

FreshRSS/1.24.0 has buggy Last-Modified + ETag caching. But, others running 1.24.2 and 1.24.3 have it fixed! The author actually reached out to let me know about this. Thanks!

Feedly/1.0 which went into a "once per day, always unconditional" thing months ago. Meh.

Reeder, various versions. Has the minor timing issue where sometimes it's 59 minutes, and other times it's an hour, as explained in a prior post. Call it the "cron alone is insufficient limiting for poll scheduling" thing, I guess?

feedparser/6.0.10, running too fast: 29/30 mins.

feedparser/6.0.2, doing the minor 59m/60m timing thing.

Yarr/1.0 with terrible timing.

Another Yarr/1.0 with the 59m/60m thing.

walrss/0.3.7, seems fine.

Emacs Elfeed 3.4.1 with some odd behaviors, like there are multiple instances running that aren't sharing the data on the last-modified/etag headers from the last poll. Another instance is doing the same thing.

er0k/feeds v0.2.0, seems fine.

Rapids, seems fine.

Go-http-client/1.1, seems fine, whatever this really is. Could use a more descriptive UA, if only to not run afoul of filtering (not me, but other places).

theoldreader.com has started doing If-Modified-Since and If-None-Match. This warranted a straight-up "holy shit" from me when I saw it in the list. I have no idea if this project had anything to do with it, and I really don't care. They're doing the right thing now, and that's great.

Tiny Tiny RSS, multiple instances from various users. Not recommended.

rss2email release-3.3.1, which keeps doing unconditional requests seemingly randomly. No idea why.

NewsBlur also randomly does unconditionals: 8 of 526 polls that way for one instance (and other counts for other instances). Also no idea why.

Feedbin, which seems to have fixed its ETag bugginess. Yay?

Inoreader/1.0. It has the 59m/60m thing, and also launches unconditionals roughly daily. Dumb.

Some curl-based thing (maybe even the CLI tool?) that's doing conditional requests nicely. Has the typical (minor) 59m/60m thing.

inforss/2.3.3.0, seems fine.

haven sends 100% unconditional requests. It also happens to have the 59m/60m thing, but the complete lack of conditional requests is the showstopper here.

ureq/2.9.1, seems fine.

Friendica/2024.08, also 100% unconditional requests.

SpaceCowboys Android RSS Reader/2.4.15 occasionally has weird timing, and it just goes unconditional randomly. No idea what's going on here, either. Another one is 2.6.31 and that seems fine, so I suspect there was a bugfix release in the middle.

Netvibes, with super broken LM+ETag caching.

cry-reader v0.0, seems fine.

Another undifferentiated browser extension thing, but this one is 100% unconditional, and wacky timing on top of that.

Blogtrottr/2.0, seems fine.

Unread RSS Reader, which is no longer unloading like mad onto the test server, but it has settled into a 15 minute poll interval. Too fast.

Slackbot/1.0, this is a relatively new one, and it's terrible. It's 100% unconditional polls with awful timing: always less than an hour between checks, sometimes much less. They also inexplicably send HEADs, and receive a 405 error from me, but of course come back and do it again a few minutes later.

The whole point of HEAD is if you want the metadata but have no interest in the content. If you want the content but only want a fresh copy if it's changed, that's why we have conditional requests, and again, If-Modified-Since has been in the RFCs since *1996*.

For sites that build the feed dynamically (i.e., not mine), HEAD represents the same amount of work: they get to build the feed, provide the metadata, and then throw it away.

Unless you actually want the metadata, and never anything BUT the metadata, then fine, send HEADs. Otherwise, forget it existed.

Audrey2 RSS Reader/0.7.1, seems fine.

rss2email/3.14, has the 59m/60m thing going on, otherwise it's fine.

...

I should note that if a feed reader's only problem is the 59:xx vs. 60 minute thing then they're actually doing pretty well. There are a lot of way worse things out there.

I'd love it if they could fix this, obviously, but that's just how I am.

This seems to be a thing now: someone finds out that you worked as an SRE ("site reliability engineer", something from the big G back in the day) somewhere, and now all you're good for is "devops" - that is, you're going to be the "ops bitch" for the "real" programmers. You are the consumer. They are the producer. They squeeze one out and you have to make it sing and dance. You keep things running and you shut the hell up. You wear the pager so they don't have to.

I've seen this from the hiring side of things: when we were trying to hire well-rounded people and put up a job posting with "SRE" in the title, all of the sudden we got a bunch of applications from people who basically *were* ops monkeys. They wanted to be that and do that. That was their life, and they enjoyed it. Those of us on the hiring side were taken aback by this and didn't want that kind of hire getting into the place.

Clearly, somewhere along the line, someone lost the thread, and it has completely destroyed any notion of what a SRE was supposed to be.

Just so we're operating on a level playing ground here, I'll lay down my own personal definition of the term, and what I expected from people in that role and what I expected from myself.

To me, a SRE is *both* a sysadmin AND a programmer, developer, whatever you want to call it. It's a logical-and, not an XOR.

By sysadmin, I mean "runs a mean Unix box, including fixing things and diving deeply when they break", and by the programmer/whatever part of it, I mean "makes stuff come into existence that wasn't there before". In particular, I expect someone to run the *right* things on those boxes, to find the actual problems and not just reboot stuff that looks squirrelly, and that they write good, solid code that's respectful of the systems and the network. They probably write programs to make the sysadmin part of the job run itself. Automation for the win.

That, to me, is my first order approximation of what a SRE should be.

Now, there must be some reason I'm writing about this, right? Yes, and there is. I put out some feelers to see about maybe working with a small company that's building some interesting things. They're wrangling Linux, C and C++, embedded stuff, networking, and they're almost certainly going to need a certain amount of pickiness regarding correctness and security to keep bad people from breaking into stuff. Also, they didn't have the usual lists of godawful clown software that most places rattle off that you'd be expected to work with.

In short, it was a place that's Actually Building Stuff and isn't just throwing their money at one of the usual clown vendors. That's rare!

So, I reached out... and heard nothing. Then, much later, I reached out a different way, and eventually heard back: they looked me over, figured I'm a SRE and they have devops people already, so, uh, no thanks.

That's it. That's the whole thing. The door is closed.

I'm obviously not happy with this situation. In sharing with some friends, they also mentioned "having to have this conversation" (that they are not just an ops monkey) a great many times in the course of looking for employment over the years.

Clearly, things have gone to hell, and unless you WANT that kind of ops-only life, you probably don't want to sell yourself this way.

Just to be annoying, I'm going to rattle off an example of something an ops monkey would never do. I wrote this C++ build tool, right? I've mentioned it a few times in various posts over the years, and there have been a few anemic web pages on my main server talking about what it is and how it works.

I won't go into the full history of it here. A quick description is: it knows how to build C++ using #includes to figure out dependencies, and so you need not write a build script, a Makefile, a CMakefile, or any other config language file. This goes both for the stuff inside the local source tree, and for external system library dependencies: stuff like libcurl, a pgsql or mysql client library, GNU Radio, libmicrohttpd, jansson, or basically anything else you might think of. It knows how to use pkg-config to get the details for some well-known targets, and you can add entries to a config file to map the "#include <proj/lib.h>" onto the pkg-config name for anything else.

So, again, that's all old news. I first wrote that over a decade ago and have been using it all this time, with small improvements over the years. What's new? Well, a couple of months back, I decided it was finally time to make the thing run its operations in parallel to take advantage of a multi-processor system.

It now scans the targets to determine the dependency graph in parallel, then kicks off compiles, then kicks off the linker. Everything that used to be serialized has now been made as parallel as possible.

Obviously, you can't just do this as a rash decision. There are any number of things which can go terribly wrong if you don't manage your data structures properly for safe cross-thread accesses. You need to be able to put things to sleep and have them wake up later without having them needlessly "spin" on the CPU. You need to do depth-first processing and then have it kick off the "parents" as the "children" finish up. You still have to catch errors and stop the process where appropriate, and you also have to make sure you don't just boil the damn machine by launching EVERYTHING at once. "Just use std::async", it is not!

To give some idea of the impact, touching a file that gets included into a whole bunch of stuff forces everything above it in the graph to get rebuilt. This used to take about 77 seconds on the same 2011-built 8-way workstation box I've had all along. With just the early work on parallelization, it became something more like 21 seconds.

77s -> 21s, just like that. That's a lot of win that was just waiting to be realized by using the hardware appropriately.

Yeah, I did that. Little old me. This is not the behavior of someone who just twiddles other people's config files for a living.

Raar, I'm cranky.

Yeah, it's another thing about feed readers. I don't blame you if you want to skip this one.

A reader (that is, a person!) reached out earlier and asked me to look at a bug report for a feed reader. It seems they passed along some of the details from one of my earlier posts, and it was closed with no action taken.

The program in question has cache problems, and it's something that's surprisingly common. A bunch of different programs do this, and it's interesting to wonder how they all came to this point.

Well, at least for some of them (the PHP-based ones), I think I know now: they're probably using the same library underneath, and it's been hacked to do some kind of "hash match" thing based on the body of the feed - like a md5sum or somesuch.

It goes something like this, apparently: fetch the feed for the first time. Then store the hash, the Last-Modified time, and the ETag.

Next time, send If-Modified-Since and If-None-Match using the stored values. But, then, if the hash of the body matches, return immediately and don't update anything else... *even if* the Last-Modified and/or ETag values changed! So, next time, they send the old values again, and it never gets any better.

Therein lies the shared bug: they're not designed around the notion of "always return the values you got last time from the web server". If they had been, they would not throw them away just because the hash matched.

So, when might the hash match when the LM or ETag values are the same?

Easy! When someone does this, or the equivalent:

webserver$ touch feed.xml

That will bump both of those fields and will leave the body unchanged.

Them not handling that case means they will start sending unconditionals until the body finally changes. That could be days or weeks later depending on how often something actually changes in the feed. Not good.

Many years ago, I was given one of those massive radio clocks that also reported indoor/outdoor temperature and humidity. They were apparently sold at Costco under the "SkyScan" brand, and had two parts: the (very large) indoor display which stood upright with a little plastic foot in the back, and a small sensor which was intended to be placed outside.

It was never quite 100%. It had issues receiving the 60 kHz WWVB clock signal and so it would get out of sync. It also apparently had its own rules for DST baked in, so when the US changed them in 2007, it would go stupid twice a year for a couple of weeks.

It had other problems. I once caught it saying that it was the 9th day of the 14th month, and oh hey, it's a Tuesday. Lousy Smarch weather or no, that's so unbelievably wrong I just had to stop using it.

However, I kept the sensor as more of a curiosity. In recent years I got into the whole business of tracking temperatures in certain interesting places, and this old sensor kept bugging me. The usual tools can see it doing its 433 MHz announcements but nobody seems to have decoded it, or at least, documented it. I've been taking a whack at it every now and then, and finally got it to a usable spot.

So, for the sake of the dozen or so people out there who might have one of these things nearly 20 years later and who want to use just the remote sensor, here's what I've learned about it.

The encoding is no frills: on is 1, off is 0. The on pulse or the off gap lasts about 3900 us (curiously close to 1 sec / 256, for what it's worth).

Transmissions are sent twice, twelve seconds apart. The first one has a particular bit set to 0, and the second one has it set to 1, so I've called it the transmission number. The whole process restarts every 60 seconds.

There appears to be some kind of preamble or sync which is just 10.

Bits 2-33 are followed by inverted versions of themselves in bits 34-65.

Humidity is BCD: 4 bits and another 4 bits, so [4] [7] for 47%.

The channel seems to be 3 bits but is only ever 1, 2, or 3.

Temperature is also BCD with 3 groups of 4 bits: [3] [2] [7] == 32.7C.

All of this has been determined empirically. There are timing diagrams for this and other similar devices on the usual "fccid" sites, but none of them match this scheme exactly.

So, from the top, 2 preamble bits (10), then...

4 + 4 humidity bits (BCD)

1 transmission bit

3 channel bits

1 sign bit (1 = negative temperature)

1 battery bit (1 = battery is low)

6 id bits

4 + 4 + 4 temperature bits (BCD)

... and then the part past the preamble repeats, inverted. This can be used to detect errors.

For the sake of grep bait: the SkyScan 81690 with a sensor bearing the FCC ID of OQH-000000-05-002, or perhaps OQH-CHUNGS-05-002. It might be a "C-8102" model, but again, all of the FCC stuff I've been able to find is not quite right.

I've been hearing from feed reader authors as a result of the whole "feed reader score" project. Last week, it was NetNewsWire, and this week I have an update for Unread Cloud.

The author put up a post which looks at what's been going on, and explains what happened with the poll frequency for a while on that one test instance. It also has some interesting graphs built from the raw "hits" table in the report.

I should point out how overwhelmingly reasonable everyone has been for this project. It's the sort of thing that could go over very badly - trust me, I've been in the position of "detecting badness" over the years at various jobs, and a lot of people do not appreciate it!

Instead, we've been slowly patching up a few small issues here and there across the collective fleet of feed readers, and hopefully this will benefit readers and writers both. Writers get lower "background radiation" for their servers, and readers get quicker updates without tripping rate limiters. Everyone wins.

To everyone, thank you for your efforts in this ecosystem.

Before I launch into responding to some reader feedback, there's a strange meta-note that I'll mention in case it twigs any memories for other people who run web servers. There's some really strange set of web robots that seem to go crawling for URLs that end in something like "feedback" or "contact" or "support" and "help" and stuff like that. They'll pull an index page and then fetch *all of them* in one fell swoop. It's pretty clear that they don't give a damn about looking halfway plausible. Real people don't request pages like that.

What's really strange is that nothing else happens after that point. Maybe they're looking for pages that have feedback forms, or comment fields, or something like that?

Fortunately, I have no such things on my posts.

...

An anonymous reader asked for my Github URL. I actually used it with a paid account for a while, but then stopped using it a long time ago. I'm talking waaaaay back in 2012 when they were pulling all kinds of stupid "let's trust your data to this overly dynamic web shit" on a regular basis.

I moved out 12 years ago and haven't gone back. It turns out that it's rather simple to just set up a repo on an accessible box and point the participant boxes at it - the ones where I do dev work, the ones that run automatic builds for me these days, and so on.

They've done some dubious things with "public keys" that I disagreed with, too. Someone actually went and made a "ssh whoami" type thing where if you connect to it and present the public key that's on a github profile, it greets you by that profile's name. Cute.

(Aside: There's a DEF CON talk out this week which talks about how people *still* don't really know that sshd will totally tell a client whether a key pair is allowed in given just the *public key*. You should find the slides for "SSHamble". They're good.)

Since then, they had a bunch of crazy stuff involving the treatment of their employees and all of that kind of crap, but I was already gone. Then, hey, an old adversary from the 90s popped its head up and gobbled them up. Yep, Microsoft bought Github in 2018. That's the kind of devious move that is entirely consistent with their old ways of operating.

This is where a bunch of people who weren't even alive in the 90s pop up and tell me how they're different, and thus prove a point I was telling people about 10 years back: "you're not going to recognize them eventually, and younger folks will have no idea of the evil they've foisted upon the world over the years".

Basically, if I was an evil sort and wanted to keep my thumb on the scale of what the "dirty free software hippies" were doing, buying the thing that's practically become synonymous with "git" itself for many people would be a great place to start! So, when they went and did that, well, that's just cake.

That'd be like IBM buying Red Hat... but wait, that happened too!

It's ironic that a fully-distributed-capable system like git ended up with its own "network effect" that brought people into a single site to that degree. Sure, there are others, but you have to know something about the scene to even realize they exist in the first place.

I bet if that had happened in 1998 instead of 2018, it'd be "Microsoft ActiveGit" or some crap like that by now. They would have totally rebranded it to try to fit with the rest of their corporate hell.

When it finally does happen, try to act surprised.

So... TL;DR there's a github profile out there where you'd expect to find one for me, but it has no activity, and for good reason.

...

The previous reader also asked if I had ever taken someone else's code base and turned it into something that "bears little resemblance to the original creature". I'm pondering this one, and, well, I don't think I have? It's not like I turned a wolf into an eel or something like that.

Okay, there was this internal web service up at a place with WAY too much Python. It was (of course) written in Python as well, and I've mentioned it before. It was a tool which was intended to make it easier for people to collaborate on outages... you know, "SEVs", or whatever you might call them where you are.

It had been written by someone as a side project of sorts, and that particular company had no love for such pure engineering work. In fact, that person was on the verge of getting in trouble with their manager for hacking on it in the first place.

So, when I showed up and realized we needed some different behavior out of this tool, I reached out to the nominal owner, learned of the situation, and then got their blessing to go and hack on it, and hack I did.

I put in enough to get it to where other people could see where I was trying to go with it, and then a couple of solid developers showed up and started cranking on it. They fixed a bunch of stupid things I had inflicted upon the code base, and got it to where it was a joy to use.

Also, I had an intern come through who did an amazing job on all of the "followup" side of things - you know, getting a report together, bringing it to a review meeting, collecting follow up tasks, and making sure those actually happen. I had been doing it by hand, and it was awful. When he got done with it, it was a breeze.

Did it look like the original tool? I guess superficially it kind-of did. It still had a list of comments and a way to add a new comment. It picked up features for both the direct users and the reliability people behind the scenes, but it was still nominally an "incident manager", even though I had rebranded it "SEVPanel".

It effectively went from a young wolf to an adult wolf. No eels or other aquatic wildlife mangling was involved. That's all.

The better question is probably *why* I did what I did and set to work on an existing thing instead of trying to come up with something new. That's a long and messy answer. It has chunks of "because it was a right pain in the ass to start a new project there" along with "I was getting enough pushback just introducing the term 'SEV'", so changing the *host name* of the service from incidentassistant.${COMPANY}.${TLD} to something else was right out. I never would have heard the end of that one.

It was so hard to start a service there... ("how hard was it?") ... that one weekend, someone parked a critical piece of infra atop our "bookmark" shortcut service because it was needed to support a certain piece of legislation that had been passed by a city. It was a matter of "put it right there right then" or the company would have been in capital-T Trouble. Also, the company would have looked like giant assholes to the people who needed that level of service, but don't despair - they delivered on that sort of thing plenty of other times that year.

In the rush, the only real solution was to stack it atop something solely intended to support in-browser shortcuts for employees. Why there? It was simple, it was easy to hack on, and if they broke it for a few days, eh, it wasn't the end of the world. It was literally a thing that saved you typing out longer URLs when you wanted a shortcut to a particular internal web page.

As for why it was so urgent, apparently management sat on it as long as possible, THEN dropped it on the engineers. Typical crap for them.

And hey, now you know a bit more about the context of my "edit in prod" story from a few years back.

...

Another reader asked for my opinion of Tiny Tiny RSS because they are planning on self-hosting it. I pretty much covered the range of behaviors that are showing up to the feed test site in my summary post from the other day.

It's definitely doing a bunch of "double tap" polls all over the place, and I have no idea why. To be clear, I don't really care all that much WHY it happens, just that it does, and it happens a lot. It's not like a single user went and hacked something broken into the source. That doesn't fly when it shows up from multiple users who don't even know about each other.

Nope, whatever's going wrong is some weird stuff from upstream.

Wow, okay, things are cooking today. I posted last night with the results of the feed readers which were still actively pinging the test feed. I've heard from several feed reader authors, and they're cranking away at stuff! They obviously care a lot about doing the right thing.

One of them even put up a post already, and so I feel obliged to share it:

NetNewsWire and Conditional GET Issues

That post actually has a link to the report page, so if you want to see what a real batch of data looks like, there you go.

There are some non-feed-reader bits of feedback I'd like to address, too.

...

One reader asked for my take on the whole CrowdStrike thing. This is where they shipped some update to a bunch of Windows machines and nuked them. I get the impression that this was something that ran rather low in the stack and had the ability to pretty much take control of the whole machine... or kill it entirely, as it turned out.

When stuff like this happens, I fall back to pointing out something that we tried to make into a "thing" at FB in the fall of 2014. There had been *far* too many outages in a really short period of time. It had started with "Call the Cops" (10 years ago now - August 1st!), and a month later we were getting tired of outages every Friday at 3 PM - I started calling it "SEV O'Clock".

The notion we tried to get known far and wide was "nothing goes everywhere at once". This means that code pushes, config changes, and even flag flips should happen piecemeal. Don't go from 0 to 100% in a single step. Take a bit and space it out. If you can't space it out, ask yourself why, then see if you can make it happen.

I've seen some amazing things almost happen. One company was in the quiet period prior to their IPO when the execs know when it'll happen but the rest of us are in the dark. Basically, it could happen *any day*, and the bigwigs are out schmoozing with the press. It's the last time you want to tank the site, service, or whatever.

Once, I overheard some people saying they were going to convert the entire site's user database from goofy not-SQL db flavor A to goofy not-SQL db flavor B in one fell swoop on Sunday night. It was Thursday. I asked if they had a way to back out. Nope, it was a one-way trip. I asked if they could do it in stages. Nope, it was all or nothing.

"So you want to do something that will affect everyone at once, with no way back, on a weekend... during the IPO road show?"

They reconsidered.

To be clear, I have no idea if the CrowdStrike thing rolled out as fast as possible or if it had stages. It seems like if it had a phased rollout, then it might've nuked a few machines in a few places, but then the alarms would have gone off and they would have hit the big red STOP button... right? All of those hosts disappearing right after getting our update means something, doesn't it?

Well, look at that. "Breach hull--all die." Even had it underlined.

Oh boy, it's been a while, so let's talk about feed readers. This whole scoring project has been running since the end of May, and much data has been collected. It's logged over 70K requests to tagged feed URLs in that period from 97 distinct keys - the unique IDs which should be used by a single install of a single program.

Two of them have sent over 6000 requests in the time they've been running. Another half-dozen or so have sent over 2000. Keep in mind that not all of these started at the end of May, either. While some of them go back to the beginning, quite a few are much newer than that.

I've gone through and summarized anything that's still checking in, using an arbitrary cutoff of "has polled in the past week".

Once again, while any one entry does not reflect a program since a user might've configured it really badly (or really well), the patterns become obvious pretty quickly when you see multiple instances of the same thing.

Unlike before, now I'm actually going to list the names and versions. It's time.

[IMS = If-Modified-Since header. INM = If-None-Match header.]

...

Netvibes. Has some kind of caching error - stuck on old IMS/INM values for the past month basically. I'm going to kick it again to see what happens, but it doesn't bode well. Has some dumb 3 second rapid-fire thing at feed init, too.

SpaceCowboys Android RSS Reader / 2.4.15 (previously aka "sweatpants", more or less, apparently). This thing has a polling interval that's all over the place and it likes to just drop the IMS/INM and do unconditional polls now and then. No idea why.

NextCloud-News/1.0. Super broken caching. Terrible mess.

NewsBlur. Rapid-fire thing at startup, otherwise seems fine.

Friendica/2024.06-dev. No conditionals, ever. Also bum rushes the server when first adding a feed by hitting a bunch of goofy not-the-feed URLs (which can't be seen in the report, but *I* can see it in the logs).

Roy and Rianne's Righteously Royalty-Free RSS Reader v 0.2. Seems to have settled down into a nice polling pattern. No complaints.

Tiny Tiny RSS/23.04. Nothing but unconditionals. But then it also does this < 1 second "OMG LET'S POLL AGAIN" thing. What's the point? Annoying.

ureq/2.9.1. Seems to be in a good pace/pattern now.

Feedbin. Hella broken IMS/INM caching, terrible pacing, regular forced unconditional polls, etc. They trip 429s in prod constantly. There are a bunch of these and they're all broken the same way, so I chalk that up to the program/service and not any one user's config.

NetNewsWire. Bigtime brokenness in their IMS/INM caching. No idea why, but it makes it generate unconditionals constantly.

haven. All unconditional requests. Pacing is all over the place.

Miniflux/2.1.4 (plus older versions previously). They had some kind of caching bug but it looks like it's good and dead with this version. Hooray for doing the right thing!

inforss/2.3.2.0. Good pacing, proper caching behavior. Nice.

Miniflux/2.1.3 and then 2.1.4. As above. Yay fixes.

Some random thing that's just curl. Seems generally okay. Does a cheeky thing every week where it drops the IMS/INM, perhaps to see if the web server admin is a clown. (Spoiler: I definitely am.)

Inoreader/1.0. All over the place with timing and unconditionals. Also shoves a useless referrer down the pipe every time, as if that is going to accomplish anything. Has a really nasty rapid-fire startup sequence that'll probably get it autoblocked before it can even get subscribed properly.

Feedbin. Bleh. See above.

Feedbin. Meh.

Miniflux/2.1.0 that needs to be upgraded to 2.1.4. Currently isn't showing signs of cache insanity because I haven't deliberately triggered it in a while.

NetNewsWire. Just like the earlier one, broken caching and crazy timing.

Liferea/1.15.3. Has a tendency to just drop everything and start making unconditional requests... one second apart? Not great.

rawdog/2.24rc1. Behavior is spot on. More like this, please.

NetNewsWire. Same bad caching and timing stuff.

Something with no User-Agent, or IMS, or INM headers. It's like... okay, what exactly is going on here? At least it polls super slowly.

NewsBlur. Just like the other one.

FreshRSS/1.24.0. This one latched bad IMS/INM values back in *May* and is still sending them. Hella broken.

Miniflux/2.1.3 and 2.1.4. Seems fine now that it's been upgraded.

Tiny Tiny RSS - multiple 24.05 and 24.06 builds. Slightly better than the 23.04 one mentioned earlier. This one does send conditional requests, but it also sends tons of unconditionals, and it does the "OMG 1 SECOND HAS ELAPSED SO I MUST GO GET IT AGAIN" thing. Not cool.

Tiny Tiny RSS 24.02 and 24.05 - various builds. Just like the previous entry.

CommaFeed/4.4.0, 4.4.1, 4.5.0, 4.6.0. This had some caching bugs for a while and was sending bad IMS values. It's been behaving lately. It's probably time for me to deliberately do some weird (but totally valid) behavior again.

Vienna/8052, 8219. Timing is all over the place, but at least it sends conditional requests.

Some claims-to-be-a-browser thing (probably a browser extension). Sends derpy 2000-01-01 IMS at startup for no good reason. Caching headers jump around and disappear sometimes, like maybe multiple instances are warring over the state storage or something. Far too many < 1s polls for my liking.

Feedbin. Feh.

Tiny Tiny RSS/24.04. As explained above, OMGs and all.

theoldreader.com. I'm pretty sure Google Reader's poller did conditional requests. This thing doesn't. Ever. They seem to get around blocks in production by polling every 6-7 hours and using multiple hosts.

Some unidentified "Go-http-client/1.1" that seems to have a decent pace and which sends INM headers. No idea what it is, but it seems to be polite about the job it's doing.

Rapids. This one got upgraded and is now doing both IMS and INM and looks great. Thanks to the author for the effort in changing it.

er0k feeds. This one also seems to have been upgraded at some point and seems to be keeping up just fine now. I definitely need to do some "inject some crazy" testing to see how it does.

NetNewsWire. Broken, as above.

walrss/0.3.7. Just works. Nice.

Yarr/1.0. Startup had a bunch of unconditionals but then it settled into sending IMS and INM headers. Timing seems a little squirrelly, like it's not actually waiting for at least an hour to elapse, but rather just starts on some schedule and is subject to however long the last run took.

Bad data. Different programs using the same key.

FreshRSS/1.24.0, 1.24.1. Bad caching, bad timing.

Some other "might be a web browser extension" thing. Does the same 2000-01-01 IMS rapid-fire stuff at startup. Sending caching values but it's running a bit quick.

Feedly/1.0. This one is just abusive. At some point it switched to sending nothing but unconditionals every 30 minutes. It's like, okay, this *fake* feed doesn't say much. So chill out, already. Comes from a bunch of different hosts, too. Annoying.

NetNewsWire. Bro-ken.

feedparser/6.0.2. Does cached requests fine, but the timing is squirrelly here just like some of the others. I wrote a post about why you can't just say 'start me every hour' and leave it at that a few weeks back.

Reeder, various builds. This thing is bizarre. It'll just flip and flop between INM values (things I served up as an ETag) from weeks in the past. Tell me that's not some kind of wicked caching bug. Also polls FAR too fast. Some programs have timing that is merely squirrelly. This is a whole cage of them.

Unusable data - multiple programs.

Unusable data - multiple programs.

Unread RSS Reader. Godawful poll timing. 6103 requests in 52 days is about one poll every 736 seconds _on average_, but they're hugely spread out. WTF? Put it this way: the list of unique intervals (nn seconds, nn minutes, ...) is *four pages tall* on my web browser.

Feedly. Also went to 30 minute all-unconditionals. Bah.

FreshRSS/1.23.1. Bad caching, like the others.

Yarr/1.0. Just like the other one: settled into sending IMS/INM but the timing is all over the place, and often far too fast.

Unusable data - multiple programs.

Miniflux/2.1.3 that needs to be upgraded to 2.1.4. As above.

Miniflux/2.1.3. Same.

Another unidentified "Go-http-client/1.1", but this one seems to have the weird timing issues like so many others.

newsraft/0.25. This one got into some bad places with stuck caching headers. Given the version number hasn't shifted, I assume it would happen again if I ran another test from my side. Wacky timing, too.

NetNewsWire. Broken record.

feedparser-perl/6.0.10 is doing conditionals but it's happening too quickly, and the timing is mighty inconsistent.

Mojolicious (Perl). Just works. No issues. Thanks.

NextCloud-News/1.0. Very broken caching. It's not even using the right timing to generate If-Modified-Since requests, and this has been a problem for years. This is the infamous "1800-01-01" IMS sender.

com.vanniktech.rssreader, various versions. Timing is all over the place, frequently far too fast - 2 seconds, 10 seconds? Also seems to "lose" the INM header now and then.

bdrss/4.0. Sends conditional requests properly. No idea how the timing works on this, but at least it's on the "long interval" side of things and not the "super fast polls over and over" like most of the others in this list. Maybe it throttles back when a feed is quiet? That's great, if so.

Broadsheet/0.1. Seems fine.

feedbase-fetcher.pl/0.5. Also seems fine.

Artykul/1.0. I installed an app on my Mac and it started checking in, and this stupid clown thing started checking in as well. The Mac app is long gone, but this albatross won't go away. Sends unconditional requests every single time, and it's visiting URLs you probably didn't intend to share with them! Wonderful. I wrote a post about this a while back.

...

There you go: a bunch of bright spots and then a bit of sadness.

Choose wisely.

I ran into a troubleshooting scenario the other day which ended up adding to the list of things that I need to check on when trying to figure out why packets seem to be disappearing. It went like this.

I showed up at a site where I'm running some weather station sensors and jumped on the console of one of the Linux boxes. My visit was about adding some sensors to some new areas, and I wanted to see how things were going. In particular, I wanted to see how the receiver on the local machine was doing, and what it had managed to log of late.

(Just imagine the port number is 1234 here.)

$ thermo_cli ::1 1234
rpc error: deadline exceeded while awaiting connection

... what? That made no sense. The thing was running.

I looked in 'ss' to make sure it was listening on that port and specifically was ready for IPv6 connections. It was.

LISTEN 0      1024         0.0.0.0:1234       0.0.0.0:*    users:(("thermo_server",pid=1141761,fd=4))                                                 
LISTEN 0      1024            [::]:1234          [::]:*    users:(("thermo_server",pid=1141761,fd=5))                                                 

I tried netcat instead... same thing. Instead of connecting, it just hung there. I looked in ip6tables... nothing. This host has no rules at the moment: nothing in 'filter', 'nat', 'mangle', etc. This should Just Work.

This site isn't running IPv6 natively due to a dumb ISP, but there are still ULA and link-local addresses, so I tried one of those from another host on the same network. That also went nowhere.

Looking in tcpdump, it was pretty clear: SYN comes in, nothing returns.

But, at the same time, I could change from that port to something like 22 and it would work fine. I'd get the usual banner from sshd.

Assuming it was something stupid I had done with my own code somehow, I ran another program as a test, then tried connecting to it over v6. It worked fine.

I straced it to make sure it was handing the IPv6 listener fd to poll(). It was. It wasn't getting any sort of notification from the kernel. But, actually, hold up, no SYN/ACK happened, so there's no way it would have gotten anywhere near userspace.

I stopped the service with systemctl and put up a netcat listening on the same TCP port (nc -6 -l -p 1234) ... and then I could connect to that just fine. So, it's not something magic about the port, somehow. It's just that port when it's going to the service which normally runs on it.

I started making a list to see what the patterns were. This box, this program, talking to ::1? Bad. Another box at this site, same program, also talking to ::1? Same problem.

Was it because this site has no v6 routing to the outside world? That makes no sense as to why ::1 wouldn't work, but, hey, one more thing to discard. I invented a fake route. Nothing happened (fortunately).

Next I started up a Debian VM on my laptop, hooked one of the radios to it, and started the receiver program on it by hand. It ran just fine, and accepted traffic over IPv6 to that same port without any trouble. It's on the same v6-route-less network as the other hosts, so what's up with that, right?

Maybe I did something stupid with the config file for the program, so I copied that across verbatim from one of the site hosts instead of just making a fresh one for testing. It didn't change things.

What if I dump the v4 listener on that port and just run the v6 listener? Nothing. What if I add a listener on another port? Nothing. Now that port also drops packets when I try to connect to it that way.

I don't know what it was about this last point, but somewhere around here, a couple of ideas finally connected in my head and I went "uh, systemd".

The failing instances were both running as systemd services. The successful instances (whether the thermo_server program, or my other test stuff) were just me doing ./foo from a shell.

That's when I thought about the hardening work I'd been applying to my systemd services of late. I've been taking away all kinds of abilities that they really don't need.

One of the newer tricks in systemd is that you can do "IPAddressDeny=" and then "IPAddressAllow" and keep a program from exchanging traffic with the rest of the world. For a program that's only ever supposed to talk to the local network, this was a good idea.

That's when I saw it: I had 127.0.0.0/8 and the local RFC 1918 networks on the Allow line, but *not* ::1, never mind the ULA prefix or the link-local v6 stuff. Adding ::1 and doing the usual daemon-reload && restart <service> thing fixed it.

Here's the deal: systemd implements that by injecting bpf program(s) when you ask it to filter traffic by IP addresses in the .service file. When this thing rejects traffic, it just drops it on the floor. It does this past the point where ip[6]tables would match it, and well before the point where it would generate a SYN/ACK or whatever else.

There are no counters associated with this, and it doesn't generate any messages in the syslog or whatever. The packets just disappear.

It's effectively equivalent to an ip[6]tables rule of "-j DROP", but at least those rules have byte and packet counters that you'd see incrementing when you're smashing your head against it. This just makes the packets disappear and nobody has any idea what's going on.

So, if you ever see traffic effectively being blackholed to the port or ports which are bound for a particular systemd-run service without anything showing up in your iptables (or let's face it, nftables) stuff, you'd better check to see if there are IPAddress* rules in the .service file. That might just explain it.

Hopefully you'll remember this and not waste a bunch of time like I just did.

I got some reader feedback the other day which amounted to "what happened to your /edu stuff" and "wasn't there a lot more in there?" These are legitimate questions, and I figured it's worth explaining what that was and then what happened to it.

Way back in 2013, I was post-one-big-company and pre-another, and got the idea in my head that people would want to watch screen captures of me writing code. I even had the thought that there might be some money to be made in the process.

I didn't like the fact that most of these "screen recordings" were literally videos of people's terminals (or IDEs, ick). They lost the ability to copy and paste from the screen since it was no longer character-based and instead was pixel-based. It also meant way more bandwidth to serve up such things.

Back then, I was on still on my ancient ServerBeach machine which ran its Ethernet port at 10 Mbps *half duplex*. I'm fairly sure they provisioned their network this way deliberately. It had the effect of limiting just how much load you put on their network since they didn't charge for bandwidth back then. (Now you know why YouTube used them for their video serving pre-Google, and why only their db stuff was on the Rackspace side of things!)

As a result, I didn't want to get into the business of serving videos over that anemic pipe. The link already slowed down way too much when someone would post an image-heavy post like the "Apple Maps sucks" series to HN or reddit or whatever.

That got me thinking, and I built something that would record my terminal as text, control sequences and all. Then I took a whack at writing a VT100 emulator, got far enough along and realized it had already been done (tty.js), and had a compatible license, so I grabbed a copy of that instead. Then I just chopped out all of the stuff that made it able to take input, and hard-wired it to my playback system.

Then I started putting up recordings of various dumb things, like me using my non-Makefile-based build system. The idea was that maybe if people saw me using a tool that didn't suck and which made my life better, they'd want a piece of it, too.

At some point, this craziness was linked up to my existing code which talked to Stripe, and so you could pay me a buck to add an item to your "account", and it would let you play back some "lessons".

Then I got hired to serve up cat pictures... or at least, to keep the serving of cat pictures working. All of my time and energy went into that, and things on this side of the world slowed way down and eventually ground to a halt. Weeks or months would pass with very little going on.

About two years into the cat-pic-wrangling gig, it was time to leave that ServerBeach machine behind in order to get IPv6 since they were still too clueless to offer it. All of the data was copied from one machine to another, but I didn't want to go to the work of rebuilding all of the CGI programs for RHEL 5, or validating that it actually worked. It required too much effort.

I just didn't have the energy to do much more than moving the web sites over and repointing DNS, and that included what it would have taken to make it work without the payment integration stuff - "free mode". The impact was that both /store and /edu were shut down.

That was pretty much it for /edu until someone reached out to me in May 2022 and said they were wondering what the page had been like, and if they could explore the old code. Fortunately for them, at that point in my life, I had cycles to spare, so I dug out some of the old stuff, chopped out the payment integration, and put some of the recordings of the old "protofeed" project online.

I didn't mention this anywhere, but if you happened to go there after that point, it would have Just Worked. It also means that some of the links in those old posts "came back to life", and that makes me happy on a certain nitpicky level that I don't expect most people to understand. (Cool [URLs] don't change, yadda yadda.)

So, if you haven't been keeping track, the virtual terminal is back online, and has been for about two years, but without new content.

If you like watching terrible code happen, you might enjoy it.

If you like badly-written web pages that tell you to reload to start over, you might really enjoy it.

Okay, let's talk about something other than feed readers for a moment. How about completely broken web browsers? Yeah, those.

This. This is a thing. Count the broken:

ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/css/main.css HTTP/1.1" 200 1651 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/feed.png HTTP/1.1" 200 689 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 

First up, why in the hell do you need to request the same link 12 times? No, scratch that, 15 times, since it does 3 more after getting the css and feed icon.

Then it goes for the favicon, and what clown decided that the right way to request "/favicon.ico" is to prepend the base path to it? This cursed thing that Microsoft foisted upon us back in the 90s is supposed to be at the top level. It's not part of individual directories. That would be stupid.

And yet, this thing decides to beat the shit out of the web server while trying to get it.

I used to wonder just what could be this stupid. The user-agents on these bad requests aren't particularly helpful. But, then one day, I got lucky and noticed that the first request of the set has one very interesting little detail in it (while the others do not):

FxiOS/127.1

FxiOS. That is, Firefox for iOS. That by itself was enough to get me looking, and oh, look what I found.

Request spamming when visiting a site

Request spam for favicon and apple-touch icons on iOS 16 + Firefox 105

Request Flooding when opening app

favicon.ico is in / , it looks for favicon.ico in every directory except / .

Firefox on iPhone makes a flood of requests for icons

Lovely. So, if you're using this garbage, know that you're probably leaving a trail of badness in your wake.

Yeah, I know, it's another post about feed reader stuff. I figured people are probably wondering what the results have been looking like. I also wanted to see things in aggregate, and fixing my mistakes turned out to really need a "30,000 foot view" approach.

So, here's a screen shot of the top third of my admin tool with the unique keys removed to give a taste of what we've all been collectively finding out.

This is showing the number of requests, how many were conditional or not, which ones presented a good If-Modified-Since value, one that was out of sequence ("oos"), or made up entirely. Then it does the same three counters for If-None-Match values.

Next up are the counters for cookies, referrers and query parameters which were needlessly presented for whatever reason, and finally it's whether the requests are showing up over IPv4 or IPv6.

I mostly rigged it this way so I could watch the "ims bogus" counts drop out as I fixed each set of entries in the database. (What a mess.)

There are more than a few keys which were issued but haven't been used, and that's what those grey rows are in the table. Even with that reduction, it still goes on much too far to present as a single screen shot - there are a *lot* of people taking part in this thing.

So, what can we learn from this view? The first row was just me running tests, so I deliberately tripped a bunch of badness to make it show up in the reports. Then there are a few boring ones which were more tests from me, and then it starts getting into actual programs.

See the one with 256 hits and 256 unconditional requests? That'd be the "leaked to the clown" situation from the other day. They're still showing up over and over to fetch that thing.

How about the one with 1258 bogus IMS values? That'd be a feed reader which is sending the feed update time, not the Last-Modified time. That's a nasty one since it looks conditional, but in practice it is not. Every one of those requests gets the full feed shipped out. (This is a rather popular program, so there are lots of these things hitting the real site all day every day. Groan.)

The really interesting parts of this are the ones which are consistently sending out-of-sequence If-Modified-Since and If-None-Match values. Out of sequence in this context means "they're sending a value, but it's an old one, not the most recent one served to them on their last hit". It seems we have managed to trip a LOT of caching bugs in these programs. They latch in some bad state, and just keep going like this forever. Not good!

One thing that has been gratifying about this is when feed reader authors have changed something in their code and then reached out to me for a fresh key. That lets them start from scratch with their new behavior and leave the old one behind. I'm always happy when this happens.

This also adds a new dimension that I'll have to start tracking, now that we have tests ending and others coming in to replace them: staleness. I'll only be looking at things which are still actively checking in so as not to penalize anyone who's improved their stuff. We will leave the old behaviors behind and focus on what's current.

That's the best you can hope for: a series of improvements.

A short note to my feed reader score project participants: I screwed up something fierce the other day. I set some bad dates for the Last-Modified header. It's all done in code, and it's not based on an actual file, so it's possible to set *nearly* any value in there.

It started when I either fat-fingered or stupid-houred this date the other day:

Thu, 08 Jun 2023 00:04:45 GMT

That was in the GET handler for the test feed for about a day. Then, when I "fixed" it, I set it to something a few days in the future:

Thu, 20 Jun 2024 00:04:45 GMT

It was only June 16th when I did that. Oops.

Anyway, this started showing up as all kinds of crazy If-Modified-Since values being presented by clients, and I thought those clients were taking the value from my end and were "clamping" it to the current time. Instead, nope, it was all on my side.

Would you believe that Apache httpd will do that all by itself? Yep. If you have a CGI program which emits a Last-Modified header in the future, it'll totally squish it down to the current date/time instead.

Further complicating the matter is that I logged the values that I *thought* were being served up, and then used them to run comparisons and generate warnings when they didn't match.

My sincere apologies to anyone who spent the last couple of days chasing after a problem that was totally created by me on this end.

Now I get to figure out how to clean up my mess. Fun!

With all of this feed reader stuff going on, I've learned a few more things about goings-on in this space. Some of it is just strange.

During the early development stage of this project, I installed a couple of apps from the Mac app store on my machine to do some testing. Each one of them was given a unique URL so I could tell them apart. This meant I started seeing traffic from my laptop to the test server, which is exactly how it was supposed to work.

But, one of those unique URLs started getting requests from some random "cloud" service. There was no indication this would happen when I plugged it into the app. It just appeared, and it was running in parallel with the requests from my actual laptop. In fact, I've since stopped running all of those programs, and they're *still* polling it - just under every three hours, and always unconditionally. Great. I bet it'll keep going approximately forever.

Nothing on the app's web page suggests this will happen. It just does. If anything, the "all articles are available offline and without an internet connection" blurb on their web page suggests the opposite: the laptop does a fetch and keeps it locally. What a concept!

So, if you were thinking about using that particular app to read some feed containing something relatively private, guess what, they're reading it too.

Timing things to happen at specific intervals is yet another way that we collectively find out that dealing with time is a hard problem. I've been noticing this while working on feed reader stuff, and I realized that it can apply to other problems.

It goes like this: say you want to have a process that runs at most once an hour. You are okay with it taking a little more than an hour between runs, but really don't want to go faster than that. Maybe you have an arrangement with a service provider to not poke them too often. Whatever.

So maybe you rig something up using cron, and it looks like this:

15 * * * * /home/me/bin/do_something

Then, every hour, at 15 minutes past, cron will run your program. Unfortunately, this by itself is not nearly enough to deliver on your arrangement. It's not even the problem you might imagine at first, which is that system clocks can be sloppy and can get pulled around by external forces.

Nope, this has to do with the time it takes to actually do the work, and not accounting for that when allowing the work to proceed again.

Back to our cron job. We'll say it gets installed at midnight, so 15 minutes later at 00:15:00, it starts a run. Maybe it does a lot of work and talks to many sites over the Internet. Some of them respond quickly, but others are slow. Maybe their DNS is taking forever to resolve the hostnames. Maybe another site is offline and is just dropping packets, so you sit there until a timeout fires on your end. It burns a good minute doing this.

At 00:16:00, it finally gets around to doing the "once an hour" work, and it happens relatively quickly. Then it finishes and goes to sleep.

About an hour later at 01:15:00, cron will run your program again. This time, maybe all of the earlier work happens much more quickly, and all of it completes in 15 seconds. That means you get around to your "once an hour" work at 01:15:15.

Oops. You were supposed to wait at least 3600 seconds - that's one hour - between requests, but you just ran it after only 3555 seconds.

The problem is that you you can't just rely on the start time of your program to know if enough time has elapsed since it last did some work which is supposed to be rate-limited. You have to actually track the time when the work *was attempted*, and then do the math of "elapsed = now - then" to see if enough time has gone by.

I tend to think of the timeline for this sort of thing as a series of fenceposts, like this:

start       action      end     (rest of the hour here)
    |          |        |
    v          v        v
----*----------*--------*---------------------------------------->

To avoid violating rate limits, you have to time things from when the action happens, not when the program starts up. If you want to really be paranoid about it, then you'll want to time it from when the program is all done with its work and is about to shut down (but this is a lot harder).

What ends up being much easier is to just remember whenever the work last started and/or finished, even if it didn't succeed. It should never select a target for refreshing until it has been idle for long enough. The program must never assume "well, I'm running again, so it must be time to do my thing". What if the box just rebooted, or any of a number of other possibilities? What then?

Here's an easy way to know if a program is on the right track: could it be run in a tight loop without causing a giant mess for other people?

$ while true; do run-my-stuff; done

If you can run something in a loop like that and not have it beat the crap out of whatever it's supposed to periodically talk to, then you're probably headed in the right direction. It also means that if the program gets into a start-crash-restart loop some day, maybe it won't unleash a hellstorm on whatever it happens to talk to.

Running a program in an infinite loop like that might chew a lot of resources on the local machine, but that's (relatively) okay. It's your machine. Feel free to burn your own resources. Where it becomes troublesome is when it reaches out and starts burning those of other people.

As usual, the details are important here.

Well, it's been an interesting couple of days. I got all kinds of feedback submitted from people who wanted to participate in the feed reader score service project. I spent quite a while answering every one of those and issuing keys to anyone who asked.

The whole time, the data has been coming in, and we've been able to start seeing some interesting patterns in the noise. There are a fair number of people who have absolutely perfect feed readers. They show up at reasonable intervals, send the right header(s), and just work with no fuss. It's fantastic.

For those not currently participating, here's how it works. I send you a link to a web page with the instructions and a unique key that's just a bunch of random hex digits. The web page tells you how to turn that into a unique URL that can be handed to a feed reader.

Then, there's another key-based URL which shows the report. I provided a text dump of what it looked like in Thursday's post, and that's all it was at the time. I've since put in the effort to make it actually generate a Real Web Page, complete with actual HTML and all of this stuff (imagine that). Now it looks more like this:

I've removed the identifying marks (the key at the top, and the user-agent string at the bottom).

This program got a little frisky at startup and actually sent multiple identical unconditional requests for some reason. I don't know why, but it's not good behavior.

It also did some requests for some other made-up URLs that weren't supplied by the user (who was talking to me at the time), and I don't mean favicon or those apple-touch-icon things. I mean it took the URL provided by the user and glued on /other /stuff. That kind of stuff doesn't show up here, but I could add it later.

It's strange. Bum rushing a web server isn't a good first impression.

...

Separately, an anonymous reader asked if I would be willing to put up stats for the posts so they could see which ones were particularly popular. I actually don't really analyze that. I can tell on a basic level when things are popular because the "tail -f" of the logs really starts moving, and I can see an uptick on the bandwidth utilization.

But other than that, I have only my own random impressions of what's popular and what isn't based on remembering activity levels and the flavor of whatever feedback it might generate (or not). So, there's no data to provide, and I really don't want to build such a dataset anyway.

A post is a post. If it works out, that's great, but if not, eh. I have no particular reason to "tune" them. I don't have ads to serve, impressions to generate, or eyeballs to sell out to the highest bidder. It's just a whole mess of text.

Most of what I use my logging for is to handle abuse. Analytics? Feh.

The one exception is from 2020 when I noticed that a fair number of bug-tracking systems were leaking URLs (many internal) to specific posts about particular nerdly analyses of failure modes like "don't setenv in multi-threaded code" and "malloc(1213486160)" and all of this. The post I did about that links to the older posts and gives a little info on them and a few scant details about the incoming links. Such referrer data is almost completely dead now, since I just get bare hostnames without any paths. It's rare to see much more than that, so I don't expect a repeat of that post, well, ever.

They also asked for "like/dislike" and maybe more (FB style?) "reactions". I'm not likely to do that, either, for the same reason as why I don't have any kind of public comments: managing that kind of thing is serious work. If it's not managed and kept in the right groove, it will turn into a very bad place at the times of the week when people with lives are out living them, and THE ONE can run amok with nobody to tell them to shut the hell up. You know exactly which forums I'm talking about.

Also, I'm pretty sure that having me trying to operate the forum and yet also moderate it would quickly turn into yet another case of "don't do both" as I have few qualms about dropping the banhammer on bad behavior... but that tends to divide communities. (Been there, done that.)

Finally, I like the fact that all of this stuff is a forest of flat files. There are no connections to any sort of database in the serving path for the posts or the feed. That would be defeated if I added something to dip into a table and look to see how many "likes" it had before shoving the post out the door.

The entire /w/ path in the document root is about 151 MB at the moment. flicker (the current web server) is a monster that besides being physically massive also happens to have 128 GB of RAM in it. That means it can fit the entire thing into memory and basically keep it there. That means there isn't even a concern about how fast the "disks" are, since it only touches that stuff once per item.

You might have noticed that things are usually pretty snappy. That's a big part of why: it's doing as little work as possible.

If there was dynamic stuff going on, that wouldn't work nearly as well. For me to cross that particular divide, it's going to have to be for a very good reason.

The "feed reader score" service that I mentioned earlier this week is now up and running. Several people reached out to me and I have sent them their unique codes and links to the instructions so they can get started.

As of about a day ago, we are now logging metadata on the requests, and as of a couple of hours ago, the reports are starting to be built. I'm still adding features to it, but it's already pretty clear that not all feed readers are built to the same standards.

This is a sample report for the sort of real programs that I see plenty on the actual site all day every day:

Time span analyzed: about 17 hours (63734.284600 sec)
-------------------------------------
Number of log entries: 34
-------------------------------------
Conditional requests: 0
-------------------------------------
❗ Unconditional requests: 34
One is normal for feed initialization.
-------------------------------------
Good If-Modified-Since values: 0
-------------------------------------
❗ Bogus If-Modified-Since values: 34
-------------------------------------
Good If-None-Match values: 0
-------------------------------------
User agent: (34) <redacted>
-------------------------------------
Useless cookies: none!
-------------------------------------
Useless referrers: none!
-------------------------------------
Useless query parameters: none!
-------------------------------------
(Still evolving.  Check back later.)

I've removed the user-agent info here so you'll just have to guess at which software this might be.

As you can see, in the short time it's been reporting in, it always sends requests with broken If-Modified-Since values. This makes it get a full copy of the feed every time, and that makes it an unconditional request.

The average polling interval appears to be about 17-18 minutes. If it ran all day, it would make about 45 requests and would use about 25 MB of data that's all redundant since nothing's actually changing.

So, it begins! If you want to participate, send me some feedback and let me know. Also, to that one "wants to be a good citizen" person who indicated interest but didn't supply an e-mail address, can you try that again? Thanks.

Wow, okay, the "feed reader score" post has been making some waves. There have been some good discussions about it, and very few of the usual failure modes for those forums.

I heard from a bunch of people who are feed reader authors or just feed reader users who care deeply about doing the right thing. They actually want to point their program at my proposed "score" site to see what it's doing, and so they can find out if it's behaving badly somehow.

I like hearing this! This kind of stuff warms my heart.

So, I've started taking steps to invest in this project. To that end, I ran out, had a couple of slices of pizza and some root beer, then came back and started digging in my parts box. I grabbed the old Raspberry Pi B+ (star of the armv6 post from last fall) and started bringing it up to date.

I got it all configured as a proper outside-facing box, then I actually *hopped in the car* earlier and drove it out to the data center to plug it in. It's there now, just waiting for me to start doing stuff with it.

Why a separate box? Well, I've done this kind of stuff before, and it's easy for a "test target" to turn into a smoking crater. This way, if it goes really badly, I can just turn it off and it won't affect the other boxes out there. It isn't affected by whatever rate-limiting I might do on the "real" web server.

Also, it caps the amount of resources that can be spent by this project - it's a goofy little box that'll barely move a few tens of Mbps across its godawful NIC. It's not going to be haul-ass fast or anything like that, but it doesn't have to be. It just has to log the incoming requests. Something else gets to do the analysis to figure out whether the observed behavior is good, bad, or just plain meh.

To be clear, the "feed" this thing will be serving is not going to have any real posts in it. You won't be able to read my latest stuff by subscribing to it. Also, I will warn everyone right now that I fully expect to have to yank the entire sub-zone out of the domain multiple times after people ruin one, and I have to NXDOMAIN them early.

The URLs you get for testing are not expected to be long-term durable, in other words. It all depends on how much abuse it gets. Considering you can figure out what a feed reader is doing after a day or two at most, that's probably not going to be a problem.

Why am I writing about this? I'm trying something different: basically, if I talk about it first, will that light a fire under me to make stuff happen sooner? This is not normally how I operate. I normally just show up with something already done. This kind of project is just big enough to where that won't work.

I guess it's time to ignite a petroleum product at zero-hundred hours.

It's been well over a year since I started serving 429s to clients which are hitting the feed too often. Since then, much has happened, and most of it is generally good news.

I've heard from users and authors alike of feed software. Sometimes the users have filed bug reports and/or feature requests and have gotten positive results from the project (or vendor). Other times, the authors of such software have gotten in touch, did some digging, found a few nuances of how their libraries work, and improved the situation.

Some of them are trying but are still not quite making it right.

Here's some of what's been going on.

...

At least one reader improved to not send a date from 1800. Unfortunately, it's now sending the wrong value in its If-Modified-Since headers. Instead of sending the value it obtained from the Last-Modified header on the past fetch, it's using the value from the "updated" header in the feed itself.

These are different layers of the system, and you can't mix their values together. They're close, but not exactly the same. This is how it works.

I get the impression from stalking their issues that they don't really control their HTTP requests very well because it's done by some other library. That's where the 1800 thing came from in the first place. It sounds like yet another case of using libraries that really don't do their jobs properly.

...

There's another one which hits every 2 minutes without fail, and there's no way to change it. I've even installed that app on a test account and verified this myself - it's hard-coded. As a result, it's the one feed user-agent I've had to block outright. It doesn't stop requesting things even when it hits a brick wall of 403s. Clearly, I need to use a bigger hammer.

...

A fair number of people are sending conditional requests, but are doing it every 5 or 10 minutes. This is ridiculous. I don't write that often, and never have. Polling more is not going to get you anywhere, and indeed, will now get you delayed so you get your updates much later than the well-behaved people. Knock it off.

It seems like most of these come from things which appear to just be jammed into web browsers as some kind of extension. From hearing from at least one developer, it seems like they don't do conditional requests as a matter of course. This, despite being part of a web browser ecosystem which has understood the notion of a conditional request and caching things locally for nearly three decades. Amazing.

...

A while back, I added a "Retry-After: " header to the feed. Anyone who gets a 429 will also get intel on when they should try back. It's in seconds, so it'll be something like 3600 or 86400 depending on which kind of request was sent in the first place.

There are feed services which will actually reset their countdowns every time someone trips a 429. I'm not doing that. Yet.

This is why noticing and honoring that header matters.

...

Oh, here's a new thing: goofy programs that try to "guess" the feed URL. I see all kinds of stupid requests to paths that might have a feed on it. This is a new level of density on the part of the authors of those programs.

Here's the thing. I've had metadata in the top of every single /w/ post *and* its index since some time in 2012. It looks like this:

<link rel="alternate" type="application/atom+xml" href="/w/atom.xml">

If you view source on this post or any other on the web, you'll see it up there, just hanging out.

I did that way back then because browsers used to care about RSS and Atom, and they'd put that little yellow feed icon somewhere in the top bar when they spotted this sort of thing in a page. At least in the case of Firefox, you could click on it, and it would throw the target URL to a helper of your choice.

I wrote a feed reader system at the time (remember fred?), and indeed, I could click on that icon and it would flip the feed URL over to my "subscribe to new feed" handler. It was easy.

Then, something happened, and browsers gave up on feeds, and the icon disappeared. I kept it there anyway, figuring people would make use of it. It's still the right way to programmatically find out where to get an Atom feed for the content you're looking at.

So what's with all of the groping around in the dark with made-up URLs?

...

This one blows my mind. I put together a page which has the feed URL on it as just plain text, not a link. I've seen people paste it into their feed reader and include spaces and even newlines. Seriously!

I know this because I get requests for things like "/w/atom.xml%20" over and over from feed readers which obviously don't notice they get a 404 every time.

...

Now we get to the part where I pitch a way forward, and nobody takes me up on the offer. The idea is basically this: I get some kind of commitment and support from the people who do feed reader stuff, and in turn, I build a new kind of web site which amounts to a "feed reader correctness score".

It would probably work like this: you load up a page and it hands you a special (fake) feed URL that is keyed to you and you alone. You plug it into your feed reader program through whatever flow and it will keep track of every single request to that keyed URL.

Then, after it had collected data for a while, a report would eventually become available. Just off the top of my head, the kinds of things it might say could look like this:

* Poll history: 46 checks in the past 48 hours (average 62 minutes)

* Request types: (1) unconditional (45) conditional

* If-Modified-Since timestamps: (45) matches (0) made up from whole cloth

* ETag hashes: (45) matches (0) made up from whole cloth

* Useless cookies sent: none!

* Useless referrers sent: none!

* Useless CGI arguments sent: none!

* User-agents: (40) FooGronk/1.0 +http://fg.example.org/ (6) FooGronk/1.01 +http://fg.example.org/

That's the kind of stuff I'd expect to see from a nigh-perfect reader. It connects at a reasonable pace, it sends headers with correct values, and it doesn't send along stuff like cookies that I never set in the first place.

But, okay, this is nothing but vaporware unless someone actually wants it, is willing to support it, and will commit to take actions based what it says.

There's a bigger lesson here: don't measure stuff if nobody's going to take actions based on the results. It only ever ends in misery. I wanted to write a separate post about this very topic, but figured I'd give a preview of it right here.

Okay world, surprise me. Do the right thing.

Oh, yesterday was a barrel of laughs. I've said a lot that I hate hardware, and it's pretty clear that hardware hates me right back.

I have this old 2012-ish Mac Mini which has long since stopped getting OS updates from Apple. It's been through a lot. I upgraded the memory on it at some point, and maybe four years ago I bought one of those "HDD to SSD" kits from one of the usual Mac rejuvenation places. Both of those moves gave it a lot of life, but it's nothing compared to the flexibility I got by moving to Debian.

Then a couple of weeks ago, the SSD decided to start going stupid on me. This manifested as smartd logging some complaint and then also barking about not having any way to send mail. What can I say - it's 2024 and I don't run SMTP stuff any more. It looked like this:

Apr 29 07:52:23 mini smartd[1140]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 29 07:52:23 mini smartd[1140]: Sending warning via /usr/share/smartmontools/smartd-runner to root ...
Apr 29 07:52:23 mini smartd[1140]: Warning via /usr/share/smartmontools/smartd-runner to root produced unexpected output (183 bytes) to STDOUT/STDERR:
Apr 29 07:52:23 mini smartd[1140]: /etc/smartmontools/run.d/10mail:
Apr 29 07:52:23 mini smartd[1140]: Your system does not have /usr/bin/mail.  Install the mailx or mailutils package

Based on the "(pending)" thing, I figured maybe it would eventually reallocate itself and go back to a normal and quiet happy place. I ran some backups and then took a few days to visit family. When I got back, it was still happening, so I went to the store and picked up a new SSD, knowing full well that replacing it was going to suck.

Thus began the multi-hour process of migrating the data from the failing drive to the new one across a temporary USB-SATA rig that was super slow. Even though I was using tar (and not dd, thank you very much), it still managed to tickle the wrong parts of the old drive, and it eventually freaked out. ext4 dutifully failed into read-only mode, and the copy continued.

I was actually okay with this because it meant I didn't have to go to any lengths to freeze everything on the box. Now nothing would change during the copy, so that's great! Only, well, it exposed a neat little problem: Debian's smartmontools can't send a notification if it's pointed at a disk that just made the filesystem fail into read-only mode.

Yes, really, check this out.

May 14 20:04:47 mini smartd[1993]: Sending warning via /usr/share/smartmontools/smartd-runner to root ...
May 14 20:04:47 mini smartd[1993]: Warning via /usr/share/smartmontools/smartd-runner to root produced unexpected output (92 bytes) to STDOUT/STDERR:
May 14 20:04:47 mini smartd[1993]: mktemp: failed to create file via template ‘/tmp/tmp.XXXXXXXXXX’: Read-only file system
May 14 20:04:47 mini smartd[1993]: Warning via /usr/share/smartmontools/smartd-runner to root: failed (32-bit/8-bit exit status: 256/1)

There it is last night attempting to warn me that things are still bad (and in fact have gotten worse) ... and failing miserably. What's going on here? It comes from what they have in that smartd-runner script. Clearly, they meant well, but it has some issues in certain corner cases.

This is the entirety of that script:

#!/bin/bash -e

tmp=$(mktemp)
cat >$tmp

run-parts --report --lsbsysinit --arg=$tmp --arg="$1" \
    --arg="$2" --arg="$3" -- /etc/smartmontools/run.d

rm -f $tmp

Notice run-parts. It's an interesting little tool which lets you run a bunch of things that don't have to know about each other. This lets you drop stuff into the /etc/smartmontools/run.d directory and get notifications without having to modify anything else. When you have a bunch of potential sources for customizations, a ".d" directory can be rather helpful.

But, there's a catch: smartd (well, smartd_warning.sh) fires off this giant multi-line message to stdout when it invokes that handler. The handler obviously can't consume stdin more than once, so it first socks it away in a temporary file and then hands that off to the individual notifier items in the run.d path. That way, they all get a fresh copy of it.

Unfortunately, mktemp requires opening a file for writing, and it tends to use a real disk-based filesystem (i.e., whatever's behind /tmp) to do its thing. It *could* be repointed somewhere else with either -p or TMPDIR in the environment (/dev/shm? /run/something?), but it's not.

This is another one of those "oh yeah" or "hidden gotcha" type things. Sometimes, the unhappy path on a system is *really* toxic. Things you take for granted (like writing a file) won't work. If you're supposed to operate in that situation and still succeed, it might take some extra work.

As for the machine, it's fine now. And hey, now I have yet another device I can plug in any time I want to make smartd start doing stuff. That's useful, right?

...

One random side note: you might be wondering how I have messages from the systemd journal about it not being able to write to the disk. I was storing this stuff to another system as it happened, and it's in my notes, but I just pulled this back out of journalctl right now, and it hit me while writing this. Now I'm wondering how I have them, too!

Honestly, I have no idea how this happened. Clearly, I have some learning to do here. How do you have a read-only filesystem that still manages to accept appends to the systemd journal? Where the hell does that thing live?

The box has /, /boot, /boot/efi, and swap. / (dm-1) went readonly. The journals are in /var/log/journal, which is just part of /.

If a tree falls in a forest and nobody's around...

...

Late update: yeah, okay, I missed something here. I'm obviously looking at the new SSD on the machine now, right? That SSD got a copy of whatever was readable from the old one, which turned out to be the entire system... *including* the systemd journal files.

Those changes weren't managing to get flushed to the old disk with the now-RO filesystem, but they were apparently hanging out in buffers and were available for reading... or something? That makes sense, right?

So, any time I copied something from the failing drive, I was scooping up whatever it could read from that filesystem. The telling part is that while these journals do cover the several hours it took to copy all of the stuff through that USB 2->SATA connection, they don't include the system shutdown. Clearly, that happened *after* the last copy ran. Obviously.

Now, if those journal entries had made it onto the original disk, then it would mean that I have a big hole in my understanding of what "read-only filesystem" means even after years of doing this. That'd be weird, right?

Just to be really sure before sending off this update, I broke out the failing SSD and hooked it up to that adapter again, then went through the incantations to mount it, and sure enough:

-rw-r-----+ 1 root systemd-timesync 16777216 May 14 17:06 system.journal

The last entry in that log is this:

May 14 17:06:38 mini kernel: ata1: EH complete

There we go. Not so spooky after all.

It's time for some responses to reader feedback.

One person mentions that running git log with "--full-history" should show the change that's buried within the commit that also was a merge. Unfortunately, that's not it. I tried that and a bunch of other things before landing on "git show" the other night. They did mention that it might be some other flag.

That plus help from a friend who's clearly been down this road before turned up the right answer: you probably want to add -c or -cc. Now, if you go looking for those in the actual manual for git log, you'll trip over the usual problem that searching for "-c" matches some other term that starts with c, in this case, clear-decorations. So, if you then search for " -c" (note: leading space only), then you'll find this first:

Note that unless one of --diff-merges variants (including short -m, -c, and --cc options) is explicitly given, merge commits will not show a diff, even if a diff format like --patch is selected, nor will they match search options like -S. The exception is when --first-parent is in use, in which case first-parent is the default format.

... and now you're getting somewhere.

Why this isn't the default, I have no idea. You're asking it to show a patch, so show a patch, already!

...

I saw some other comments saying that I should report some of the wacky brokenness that I tripped over, and I need to respond to that.

It's been made very clear to me that plenty of people don't give a shit about what I report as "broken" in their projects. This isn't necessarily about the world of free software/open source, either. It's happened plenty of times at actual jobs when I was there specifically to find badness.

Case in point: someone apparently had a conversation along the lines of "hey, it would be bad if the <redacted spooky people> got their hands on this stuff and it didn't work, so let's have someone who isn't one of the devs test this out first". Then they got a hold of me and ask me to Do some Stuff with their products.

I got my hands on the hardware and then the software, and in so doing, found all kinds of crazy problems in it, almost like nobody had actually tried to use the full extent of the hardware with their driver software.

I started asking questions of the engineers, trying to figure out what they have done. At some point they went radio-silent and I started getting responses from their boss. At that point I thought "it's good to be a contractor" and referred it to MY boss - the person who hired me for the gig. It was clearly an internal matter and not anything for me to deal with. (One of the rare perks of being a contractor, gotta say.)

So, back to what happened the other night. Did I find some badness in pkg-config or pkgconf or whatever variant they're using? I probably did! Could it be exploited? Maybe? Probably? Do I care that much? Not really. Why? Because *far too few people* actually give a shit about this kind of correctness.

This is fairly common when you set me loose in an ecosystem: I find all kinds of dumb "mosquito bite" things that make things less than pleasant for someone who isn't expecting it. They're the things that experienced users know how to route around because they start accepting it as "normal".

If my "trouble reports" tend to be seen as more of a burden than a value, why would I keep generating them?

For those few people out there who truly care about this sort of thing, obviously I love and cherish you and people who do what you do. Please don't think I'm painting you with the same brush. Just realize that you are very much in the minority, and I have no idea if I'm going to encounter one of you (unlikely) or one of the others (very likely) when I file a report.

Another thing: I'm not paying any of these people, so why should they listen to me? There are exactly zero consequences for them if my stuff goes ignored. They don't work for me, and it's not like I could fire them or somehow use some other leverage in order to make things happen.

...

I believe that really good products have at least one person behind them who's spotting all of the goofiness and is having some success in getting those things ironed out. It's not enough to just notice them, and it's not enough to do the kind of gunky eng work it takes to find out why something happened or how it broke.

Whoever does this work has to actually have traction in the organization so that these problems will actually be fixed. Otherwise, the first two parts have no real value, and only serve to burn out whoever's making the attempts.

I'd love to do this kind of work for a place that truly cares about it, to the point of providing air cover to those who are tasked with finding the problems. If the company doesn't actually care, that's "fine". Don't hire people who are tasked with trying to improve those things, then if there's a hypocrisy gap in your engineering culture.

Also, I get that companies change, and what's valued today is not what will be valued tomorrow. It seems like the least a good manager would do is *explicitly* say as much when that point arrives, so you can cut and run instead of trying to push back an unstoppable tide. When the company shifts from engineering, reliability and delighting the customers to growth at all costs and exploiting the customers, it's probably time to go.

Is anyone actually doing this now? I'd love to hear about it if so.

...

Finally, to the handful of people who say that the results of ./configure "will change all the time" because of the kernel or libc version changing... that's just wrong. You're making that up based on a technicality that does not actually track with real-world use of our systems.

I'm going to guess that most of the people who are saying this are going based on the fact that you CAN change the C library or kernel arbitrarily, but probably never have. Meanwhile, I was there way back in the day when we first got ELF kernels in the 1.3 world and had to install a new compiler in order to link it in the first place, among plenty of other godawful things we had to do back in the 90s. I've been there, and I'm telling you, this is not the way of the world right now. Systems just don't change that often. When they do change to that extent, it tends to be associated with a major version bump (unless someone's mighty clueless).

The whole point of having things set up for you by the OS builder is that it takes the kernel, C library, and other localizations into account. If someone is changing those things to where syscalls and library calls are shifting materially, they are essentially signing up to maintain their own OS variant, and should update the "how you do things on this OS" data at the same time if it affects such things.

This *already* happens. Companies take a base OS (like CentOS), and then they run a far newer kernel and/or glibc atop it for their own reasons. This forces a whole raft of other changes to take advantage of the new stuff: new versions of perf, strace, ethtool, mcelog or whatever else. I know this because I did this to help out as a certain employer went from CentOS 5 to 6 to 7 over the span of several years.

Their build systems already know how to build stuff on this hybrid environment, particularly to use the overlay glibc instead of the system one. So, someone is maintaining this, and they track whatever changes when it actually matters.

I keep seeing people saying that the answer to my complaints about autoconf is to rub *more* autoconf on the problem. I don't like this. In the general vein of "this should not be that hard", I decided to revisit something from two years ago and tried to use my build tool to generate my stuff on a fresh BSD-flavored install. (The exact flavor is unimportant here, and mentioning it by name would only trigger the weenies in the crowd, so I won't.)

I wanted to prove to myself that yes, my stuff can Just Build on other (i.e., not Linux or Mac) systems without resorting to the kinds of stuff that I wish we had collectively left behind in the 90s.

The OS itself was fine. The install process on a throwaway VM image was quick and painless. I knew how to get my usual tools installed - bash, nano, that kind of thing. Unlike last time, I opted to not do X and just focused on getting my stuff to build.

But then I made a mistake: I told it to install "protobuf" since I use that library in my build tool. That actually installed "protobuf-24.4,1" which is some insane version number I'd never seen before. All of my other systems are all running 3.x.x type versions.

Now, realize, I didn't know this was a mistake yet, and kept on going. I did manage to bootstrap my build tool into a usable binary, and then started a "build the world" process, at which point it blew up. It was complaining about not being able to find a library called "google/protobuf/arena" inside my personal source tree.

This made no sense, so I started digging, and found out that the "protoc" compiler in that version of the software spits out code like this:

#include "google/protobuf/thing1.h"
#include "google/protobuf/thing2.h"
#include "google/protobuf/thing3.h"

... you get the idea. It's a third-party library that's installed at the system level, and yet it's using "" like it's all chummy and hanging out with your code in your local repo. Yeah, no. That's wrong. They should be <> includes, like this:

#include <google/protobuf/thing1.h>
#include <google/protobuf/thing2.h>
#include <google/protobuf/thing3.h>

What's weird is... it *is* that way on all of my other machines - my Mac with Macports and my Debian/Raspbian boxes all generate those #includes with <> like they're supposed to, and everything Just Works.

I won't lie. This really made me angry at first. I was like, okay, they did yet another stupid thing upstream, and now everyone else is going to have to work around it. It got me thinking thoughts like "just how hard would it be to NOT use protobuf, anyway". I figured that this abomination would eventually filter down to Macports and Debian's apt repo and whatnot, and then I'd have to deal with it, or toss it.

After a few minutes of cooling off, it occurred to me that I could do something super-duper obnoxious: wrap protoc, and run a nasty little sed command afterward to flip the "" to <>. So I did that, and things proceeded. Awful.

Of course, then I ran into some other fun problems with my code, like IPPROTO_* definitions not being available. I have a wrapper for getaddrinfo() and it uses IPPROTO_TCP in the .ai_protocol field. I had all of the #includes that the man pages say to have for using that function, but that's not enough on this particular system.

I assume that there's some transitive #include on Macs and on glibc-flavored Linuxes that drags this in for me, but on this one BSD it doesn't work that way. The fix was simple enough, and mighty stupid:

#include <netinet/in.h>

And no, that's not listed in their getaddrinfo(3) manual page, even though IPPROTO_UDP and _TCP are both explicitly mentioned in it. Dig around online and you'll find this tripping up other people. That's the extent of my self-inflicted damage that had to be fixed to make it build: lack of a few #includes.

Stuff like this is why I tend to wall off calls into the C library with a bunch of compatibility gunk and then use my own interfaces above that.

At some point during this, I decided to go back into the protobuf git repo to see just when they decided to dump the angle brackets in favor of the double-quotes, and that's when I hit another wall of stupid. Apparently it's possible to change a git repo in such a way that "git log -p" will never show it. Did you know that? Before yesterday, I definitely didn't.

Here's how I discovered this: obviously, there was code that would do the <> stuff at some point. The last version of it I could find looked like this:

  std::string left = "\"";
  std::string right = "\"";
  if (use_system_include) {
    left = "<";
    right = ">";
  }
  return left + name + right;

It seems simple enough, if a little goofy: return "input" unless use_system_include gets set a few lines up, in which case it should return <input>. No big deal, right?

But... that code exists nowhere in the repo as it stands now. Silly naive me, I thought I could just "git log -p" and do a / search in less for "use_system_include" to find the commit which dropped it. I wanted to learn why they did this, because maybe they had a good reason, or basically, if I complained about it, what I would be up against.

I found nothing.

This started a terrible sequence where I started checking out different commits from the tree to see what it looked like at various points in the past. I got it down to a commit that contained the above code, and then one commit past that dropped it.

This must be it, right? I should be able to "git log -p" and see it, right? Nope.

commit d85c9944c55fb38f4eae149979a0f680ea125ecb (HEAD)
Merge: 7764c864b 0264866ce
Author: <removed because it's not their fault>
Date:   Mon Sep 19 14:10:44 2022 -0700

    Sync from Piper @475378801
    
    PROTOBUF_SYNC_PIPER

The next line in the git log output is the next commit. There's no "body" to this commit. It's just a "Merge:" and two other commits.

7764c864b and 0264866ce, right? I should be able to sync to those with git checkout and see which one dropped it, yeah? Well, I'll spare you the effort and just say that BOTH OF THEM have the old code in it.

So... this commit somehow drops the code even though it's merging two "ancestral commits" that both contain it, and there's no diff shown.

Confusing, right?

I don't know how I finally figured this out, but after a whole lot of cursing and thrashing, I found "git show <commit>" will FINALLY give me the results I want, ish. It contains the change which dumped the <> code and put in the new stuff.

--  std::string left = "\"";
--  std::string right = "\"";
--  if (use_system_include) {
--    left = "<";
--    right = ">";
--  }
--  return left + name + right;
++  return absl::StrCat("\"", basename, "\"");

There's no explanation or other context. Presumably that all got squashed out when it was exported from whatever they use internally.

"Why" is gone. I just have "when", and that's not very interesting: it was merged in September 2022, ho hum. That just means that whenever Linux distributions and Macports catch up with at least that point, I'm going to have to deal with this for real.

Oh, there's one more bit of batshittery which needs to be mentioned here. My stuff uses pkg-config to find out how to compile and link against these libraries, right? Well, when it was using this oh-so-new protobuf version, the commands it was running were so long, it was scrolling off my standard 80x25 terminal.

$ pkg-config --cflags protobuf | wc -c
    4326

Yep! 4 KB of cflags. Here's just the top part of it:

# pkg-config --cflags protobuf
-I/usr/local/include -DPROTOBUF_USE_DLLS -Wno-float-conversion 
-DNOMINMAX -Wno-float-conversion -DNOMINMAX -Wno-float-conversion 
-DNOMINMAX -Wno-float-conversion -DNOMINMAX -Wno-float-conversion 
-DNOMINMAX -Wno-float-conversion -DNOMINMAX -Wno-float-conversion 
-DNOMINMAX -Wno-float-conversion -DNOMINMAX -Wno-float-conversion 

... and it just goes on like this. It actually worked, though!

Finally, remember when I said that I made a problem by installing their "protobuf" package without realizing it? Yeah, it turns out they actually also have "protobuf3" which is a nice sane version just like the ones on my other machines, #include <...> and all. So, I removed the bad one, installed this other one, and dropped my sed hack.

What a night.

I have a story about paper cuts when using a little Linux box.

One of my sites has an older Raspberry Pi installed in a spot that takes some effort to access. A couple of weeks ago, it freaked out and stopped allowing remote logins. My own simple management stuff was still running and was reporting that something was wrong, but it wasn't nearly enough detail to find out exactly what happened.

I had to get a console connected to it in order to find out that it was freaking out about its filesystem because something stupid had apparently happened to the SD card. I don't know exactly why it wouldn't let me log in. Back in the old days, you could still get into a machine with a totally dead disk as long as enough stuff was still in the cache - inetd + telnetd + login + your shell, or sshd + your shell and (naturally) all of the libraries those things rely on. I guess something happened and some part of the equation was missing. There are a LOT more moving parts these days, as we've been learning with the whole xz thing. Whatever.

So I rebooted it, and went about my business, and it wasn't until a while later that I noticed the thing's clock was over a day off. chrony was running, so WTF, right? chrony actually said that it had no sources, so it was just sitting there looking sad.

This made little sense to me, given that chrony is one of the more clueful programs which will keep trying to resolve sources until it gets enough to feel happy about using them for synchronization. In the case of my stock install, that meant it was trying to use 2.debian.pool.ntp.org.

I tried to resolve it myself on the box. It didn't work. I queried another resolver (on another box) and it worked fine. So now what, on top of chrony not working, unbound wasn't working too?

A little context here: this box was reconfigured at some point to run its own recursive caching resolver for the local network due to some other (*cough* TP-Link *cough*) problems I had last year. It was also configured to *only* use that local unbound for DNS resolution.

This started connecting some of the dots. chrony wasn't setting the clock because it couldn't resolve hosts in the NTP pool. It couldn't resolve hosts because unbound wasn't working. But, okay, why wasn't unbound working?

Well, here's the problem - it *mostly* was. I could resolve several other domains just fine. It's just that ntp.org stuff wasn't happening.

(This is where you start pointing at the screen if this has happened to you before.)

So, what would make only some domains not resolve... but not all of them... on a box... with a clock that's over a day behind?

Yeah, that's about when it fit together. I figured they must be running DNSSEC on that zone (or some part of it), and it must have a "not-before" constraint on some aspect of it. I've been down this road before with SSH certificates, so why not DNS?

I added another resolver to resolv.conf, then chrony started working, and that brought the time forward, and then unbound started resolving the pool, and everything else returned to normal.

By "everything else", I also mean WireGuard. Did you know that if your machine gets far enough out of sync, that'll stop working, too? I had no idea that it apparently includes time in its crypto stuff, but what other explanation is there?

Backing up, let's talk about what happened, because most of this is on me.

I have an old Pi running from an SD card. It freaked out. It took me about a day and a half to get to where it was so I could start working on fixing it.

This particular Pi doesn't have a real-time clock. The very newest ones (5B) *do*, but you have to actually buy a battery and connect it. By default, they are in the same boat. This means when they come up, they use some nonsense time for a while. I'm not sure exactly what that is offhand, because...

systemd does something of late where it will try to put the clock back to somewhere closer to "now" when it detects a value that's too far in the past. I suspect it just digs around in the journal, grabs the last timestamp from that, and runs with it. This is usually pretty good, since if you're just doing a commanded reboot, the difference is a few seconds, and your time sync stuff fixes the rest not long thereafter.

But, recall that the machine sat there unable to write to its "disk" (SD card) for well over a day, so that's the timestamp it used. If I had gotten there sooner, I guess it wouldn't have been so far off, but that wasn't an option.

Coming up with time that far off made unbound unable to resolve the ntp.org pool servers, and that made chrony unable to update the clock... which made unbound unable to resolve the pool servers... which...

My own configuration choice which pointed DNS resolution only at localhost did the rest.

So, what now? Well, first of all, I gave it secondary and tertiary resolvers so that particular DNS anomaly won't be repeated. Then I explicitly gave chrony a "peer" source of a nearby host (another Pi, unfortunately) which might be able to help it out in a pinch even if the link to the outside isn't up for whatever reason.

There's a certain problem with thinking of these little boxes as cheap. They are... until they aren't. To mangle a line from jwz, a Raspberry Pi is only cheap if your time has no value.

As usual, this post is not a request for THE ONE to show up. If you are THE ONE, you don't make mistakes. We know. Shut up and go away.

I've gotten a few bits of feedback asking for my thoughts and/or reactions to the whole "xz backdoor" thing that happened over the past couple of days. Most of my thoughts on the matter apply to autoconf and friends, and they aren't great.

I don't have to cross paths with those tools too often these days, but there was a point quite a while back when I was constantly building things from source, and a ./configure --with-this --with-that was a given. It was a small joy when the thing let me reuse the old configure invocation so I didn't have to dig up the specifics again.

I got that the whole reason for autoconf's derpy little "recipes" is that you want to know if the system you're on supports X, or can do Y, or exactly what flavor of Z it has, so you can #ifdef around it or whatever. It's not quite as relevant today, but sure, there was once a time when a great many Unix systems existed and they all had their own ways of handling stuff, and no two were the same.

So, okay, fine, at some point it made sense to run programs to empirically determine what was supported on a given system. What I don't understand is why we kept running those stupid little shell snippets and little bits of C code over and over. It's like, okay, we established that this particular system does <library function foobar> with two args, not three. So why the hell are we constantly testing for it over and over?

Why didn't we end up with a situation where it was just a standard thing that had a small number of possible values, and it would just be set for you somewhere? Whoever was responsible for building your system (OS company, distribution packagers, whatever) could leave something in /etc that says "X = flavor 1, Y = flavor 2" and so on down the line.

And, okay, fine, I get that there would have been all kinds of "real OS companies" that wouldn't have wanted to stoop to the level of the dirty free software hippies. Whatever. Those same hippies could have run the tests ONCE per platform/OS combo, put the results into /etc themselves, and then been done with it.

Then instead of testing all of that shit every time we built something from source, we'd just drag in the pre-existing results and go from there. It's not like the results were going to change on us. They were a reflection of the way the kernel, C libraries, APIs and userspace happened to work. Short of that changing, the results wouldn't change either.

But no, we never got to that point, so it's still normal to ship a .tar.gz with an absolute crap-ton of dumb little macro files that run all kinds of inscrutable tests that give you the same answers that they did the last time they ran on your machine or any other machine like yours, and WILL give the same answers going forward.

That means it's totally normal to ship all kinds of really crazy looking stuff, and so when someone noticed that and decided to use that as their mechanism for extracting some badness from a so-called "test file" that was actually laden with their binary code, is it so surprising that it happened? To me, it seems inevitable.

Incidentally, I want to see what happens if people start taking tarballs from various projects and diff them against the source code repos for those same projects. Any file that "appears" in the tarball that's allegedly due to auto[re]conf being run on the project had better match something from the actual trees of autoconf, automake, ranlib, gettext, or whatever else goofy meta-build stuff is being used these days.

$ find . -type f | sort | xargs sha1sum
7d963e5f46cd63da3c1216627eeb5a4e74a85cac  ./ax_pthread.m4
c86c8f8a69c07fbec8dd650c6604bf0c9876261f  ./build-to-host.m4
0262f06c4bba101697d4a8cc59ed5b39fbda4928  ./getopt.m4
e1a73a44c8c042581412de4d2e40113407bf4692  ./gettext.m4
090a271a0726eab8d4141ca9eb80d08e86f6c27e  ./host-cpu-c-abi.m4
961411a817303a23b45e0afe5c61f13d4066edea  ./iconv.m4
46e66c1ed3ea982b8d8b8f088781306d14a4aa9d  ./intlmacosx.m4
ad7a6ffb9fa122d0c466d62d590d83bc9f0a6bea  ./lib-ld.m4
7048b7073e98e66e9f82bb588f5d1531f98cd75b  ./lib-link.m4
980c029c581365327072e68ae63831d8c5447f58  ./lib-prefix.m4
d2445b23aaedc3c788eec6037ed5d12bd0619571  ./libtool.m4
421180f15285f3375d6e716bff269af9b8df5c21  ./lt~obsolete.m4
f98bd869d78cc476feee98f91ed334b315032c38  ./ltoptions.m4
530ed09615ee6c7127c0c415e9a0356202dc443e  ./ltsugar.m4
230553a18689fd6b04c39619ae33a7fc23615792  ./ltversion.m4
240f5024dc8158794250cda829c1e80810282200  ./nls.m4
f40e88d124865c81f29f4bcf780512718ef2fcbf  ./po.m4
f157f4f39b64393516e0d5fa7df8671dfbe8c8f2  ./posix-shell.m4
4965f463ea6a379098d14a4d7494301ef454eb21  ./progtest.m4
15610e17ef412131fcff827cf627cf71b5abdb7e  ./tuklib_common.m4
166d134feee1d259c15c0f921708e7f7555f9535  ./tuklib_cpucores.m4
e706675f6049401f29fb322fab61dfae137a2a35  ./tuklib_integer.m4
41f3f1e1543f40f5647336b0feb9d42a451a11ea  ./tuklib_mbstr.m4
b34137205bc9e03f3d5c78ae65ac73e99407196b  ./tuklib_physmem.m4
f1088f0b47e1ec7d6197d21a9557447c8eb47eb9  ./tuklib_progname.m4
86644b5a38de20fb43cc616874daada6e5d6b5bb  ./visibility.m4
$ 

... there's no build-to-host.m4 with that sha1sum out there, *except* for the bad one in the xz release. That part was caught... but what about every other auto* blob in every other project out there? Who or what is checking those?

And finally, yes, I'm definitely biased. My own personal build system has a little file that gets installed on a machine based on how the libs and whatnot work on it. That means all of the Macs of a particular version of the OS get the same file. All of the Debian boxes running the same version get the same file, and so on down the line.

I don't keep asking the same questions every time I go to build stuff. That's just madness.

There was this team which was running a pretty complicated data storage, leader election and "discovery" service. They had something like 3200 machines and had something like 300 different clusters/cells/ensembles/...(*) running across them. This service ran something kind of like etcd, only not that.

The way it worked was that a bunch of "participant" machines would start an election process, and then they'd decide who was going to lead them for a while. That leader got to handle all of the write traffic and it did all of the usual raft/paxos-ish spooky coordination stuff amongst the participants, including updating the others, and dealing with hosts that go away and come back later, and so on. It's all table stakes for this kind of service.

This group of clusters had started out relatively simple but had grown into a monster over the years. Nobody probably expected them to have hundreds of clusters and thousands of machines, but they now did, and they were having trouble keeping track of everything. There were constant outages, and since they were so low in the stack, when they broke, lots of other stuff broke.

I wanted to know just what the ground truth looked like and so started something really stupid from my development machine. It would take a list of their servers and would crawl them, interrogating the TCP ports on which the service ran. This was only about 10 ports per machine, so while it sounded obnoxiously high, it was still possible for prototyping purposes.

On these ports, there were simple text-based commands which could be sent, and it would return config information about what that particular instance was running. It was possible to derive the identity of the cluster from that. Given all of this and a scrape of the entire fleet, it was possible to see which host+port combinations were actually supporting any given cluster, and thus see how well they were doing.

Early results from this terrible manual scraping started showing promise. Misconfigurations were showing up all over the place - clusters that are supposed to have 5 hosts but only have 3 in practice with the other two missing in action somewhere, clusters with non-standard host counts, clusters in the wrong spots, and so on.

To get away from the "printf | nc the world in cron" thing, we wound up writing this dumb little agent thing that would run on all of the ~3200 hosts. It would do the same crawling, but it would happen over loopback so it was a good bit faster by removing long hauls over the production network from the equation. It also took the load of polling ~32000 ports off my singular machine, and was inherently parallel.

It was now possible to just query an agent and get a list of everything running on that box. It would refresh things every minute, so it was far more current than my terrible script which might run every couple of hours (since it was so slow). This made things even better, and so we needed an aggregator.

We did some magic to make each of these agents create a little "beacon" somewhere any time they were run. Our "aggregator" process would start up and would subscribe to the spot where beacons were being created. It would then schedule the associated host for checks, where it would speak to the agent on that host and ask for a copy of its results.

So now we had an agent on every one of the ~3200 hosts, each polling 10 local ports, plus an aggregator that talked to the ~3200 agents and refreshed the data from them.

Finally, all of the data was available in one place with a single query that was really fast. The next step was to write a bunch of simple "dashboard" web pages which allowed anyone to look at the entire fleet, or to narrow it down by certain parameters - a given cluster (of these servers), a given region, data center, whatever.

With all of this visible with just a few clicks, it was pretty clear that we needed something more to actually find the badness for us. It was all well and good to go clicking around while knowing what things are supposed to look like, but there were supposed to be rules about this sort of thing: this many hosts in a cluster, no more than N hosts per failure domain, and more.

...

Failure domains are a funny thing. Let's say you have five hosts which form a quorum and which are supposed to be high-availability. You'd probably want to spread them around, right? If they were serving clients from around the world, maybe you'd put them in different locations and never put two in the same spot? If something violated that, how would you know?

Here's an example of bad placement. We had this one cluster which was supposed to be spread out throughout an entire region which was composed of multiple datacenter buildings, each with multiple (compute) clusters in it, with different racks and so on down the line. But, because it had been turned up early in the life of that region when only a handful of hosts had existed, all of them were in the same two or three racks.

Worse still, those racks were physically adjacent. Put another way, if the servers had arms and hands, they could have high-fived each other across the hot and cold aisles in the datacenter suite. That's how close together they were. One bad event in a certain spot would have wiped out all of their data.

We had to write a schema which would let us express limits for a given cluster - how many regions it should be in, the maximum number of members per host, rack, (compute) cluster, building, region, etc. Then we wrote a tool to let us create rules, and then started using that to churn out rulesets. Next we came up with some tools which would fetch the current state of affairs (from the agent/aggr combo) and compare it to the rulesets. Anything out of "compliance" would show up right away.

...

Then there was the problem of managing the actual ~3200 hosts. With a footprint that big, there's always something happening. A location gets turned up and new hosts appear. Another location is taken down after the machines get too old and those hosts go away. We kept having outages where a decom would be scheduled, and then someone far away would run a script with a bunch of --force type commands, and it would just yank the machines and wipe them. It had no regard for what they were actually doing, and they managed to clobber a bunch of stuff this way. It just kept happening.

This is when I had to do something that does not scale. I said to the decom crew that they should treat any host owned by this team as off limits because we do not have things under control. That means never *ever* running a decom script against these hosts while they are still owned by the team.

I further added that while we're working to get things under control, if for some reason a decom is blocked due to this decree of mine, they are to contact me, any time of day or night, and I will get them unblocked... somehow. I figured it was my way of showing that I had "skin in the game" for making such a stupid and unreasonable demand.

I've often said that the way to get something fixed is to make sure someone is in the path of the badness so they will feel it when something screws up. This was my way of doing exactly that.

We stopped having decom-related outages. We instead started having these "fire drill" type events where one or two people on the team (and me) would have to drop what they were doing and spend a few hours manually replacing machines in various clusters to free them up.

Obviously, this couldn't stand, and so we started in on another project. This one was more of a "fleet manager", where a dumb little service would keep track of which machines the team owned, and it would store a series of bits for each one that I called "intents".

There were only three bits per host: drain, release, freeze. Not all combinations were valid.

If no bits were set on a host, that meant it was intended for production use. If it has a server on it, that's fine. If someone needs a replacement, it's potentially available (assuming it meets the other requirements, like being far enough away from the other participants).

If the "drain" bit was set, that meant it was not supposed to be serving. Any server on it should be taken off by replacing it with an available host which itself isn't marked for "drain" (or worse).

The "release" bit meant that if a host no longer had anything running on it, then it should be released back to the machine provisioning system. In doing this, the name of the machine changed, and thus the ownership (and responsibility) for it left the team, and it was no longer our problem. The people doing decoms would take it from there.

"Freeze" was a special bit which was intended as a safety mechanism to stop a runaway automation system. If that bit was set on a host, none of the tools would change anything on it. It's one of those things where you should never need to use it, but you'll be sorry if you don't write it and then need it some day.

"Drain" + "release" meant "keep trying to kick instances off this host and don't add any new ones", and then "once it becomes empty, give it back".

Other combinations of the bits (like "release" without "drain") were invalid and were rejected by the automation.

I should note that this was meant to be level-triggered, meaning on every single pass, if a host had a bit set and yet wasn't matching up with that intent or those intents, something should try to drain it, or give it away, or whatever. Even if it failed, it should try again on the next pass, and failures should be unusual and thus reported to the humans.

...

Then there was also the pre-existing system which took config files and used it to install instances on machines. This system worked just fine, but it only did that part of the process. It didn't close the loop and so many parts of the service lifecycle wound up unmanaged by it.

Looking back at this, you can now see that we could establish a bunch of "sets" with the data available.

Configs: "where we told it to run"

Agent + aggregator: "where it's actually managing to run"

Checker: "what rules these things should be obeying"

Fleet manager: "which machines should be serving (or not), which machines we should hang onto (or give back)".

Doing different operations on those sets yielded different things.

[configs] x [agent/aggr] = hosts which are doing what they are supposed to be doing, hosts which are supposed to be serving but aren't for some reason, and hosts which are NOT supposed to be running but are running it anyway. It would find sick machines, failures in the config system, weird hand-installed hack jobs in dark corners, and worse.

[agent/aggr] x [checker] = clusters which are actually spread out correctly, and clusters which are actually spread out incorrectly, (possibly because of bad configs, but could be any reason).

[agent/aggr] x [fleet manager] = hosts which are serving where that's okay, hosts which need to be drained until empty, and hosts which are now empty and can be given back.

[configs] x [checker] = are out-of-spec clusters due to the configs telling them to be in the wrong spot, or is something else going on? You don't really need to do this one, since if the first one checks out, then you know that everything is running exactly what it was told to run.

[configs] x [fleet manager] = if you ever get to a point where you completely trust that the configs are being implemented by the machines (because some other set operations are clear), then you could find mismatches this way. You wouldn't necessarily have to resort to the empirical data, and indeed, could stop scanning for it.

For that matter, the whole port-scanning agent/aggr combination shouldn't have needed to exist in theory, but in practice, independent verification was needed.

I should point out that my engagement with this team was not viewed kindly by management, and my reports about what had been going on ultimately got me in trouble more than anything else. It's kind of amazing, considering I was working with them as a result of a direct request for reliability help, but shooting the messenger is nothing new. This engagement taught me that a lot of so-called technical problems are in fact rooted in human issues, and those usually come from management.

There's more that happened as part of this whole process, but this post has gotten long enough.

...

(*) I'm using "clusters" here to primarily refer to the groups of 5, 7, or 9 hosts which participated in a quorum and kept the state of the world in sync. Note that there's also the notion of a "compute cluster", which is just a much larger group of perhaps tens of thousands of machines (all with various owners), and that does show up in this post in a couple of places, and is called out explicitly when it does.

Oh dear, it's popcorn for breakfast yet again. Another outage in a massive set of web sites.

It's been about 10 years, so let's talk about the outage that marks the point where I started feeling useful in that job: Friday, August 1, 2014. That's the one where FB went down and people started calling 911 to complain about it, and someone from the LA County sheriff's office got on Twitter to say "knock it off, we know and it's not an emergency".

Right, so, it's been well-documented what happened that day, even on the outside world - SRECon talks, a bunch of references in papers, you name it. It was time for "push", and as it was being seeded, that process pretty much consumed all of the available memory (and swap) on the smallest machines.

Then there was this program which ran on every box as root, and its job was to run a bunch of awful subprocesses, capture their outputs, parse them somewhat, and ship the results to a time series database or a logging system. This program is the one that had the infamous bug in it where it would call fork() and saved the return value, but didn't check it for failure: the -1 retval.

So, later on, it went to kill this "child process" that never started, and did the equivalent of 'kill -9 -1', and on Linux, that whacks everything but yourself and pid 1 (init). Unsurprisingly, this took down the web server and pretty much everything else. This was pre-systemd on CentOS 5 machines running Upstart, so the only things that "came back" were the "respawn" entries in inittab, like [a]getty on the text consoles.

This is how we were able to fire up a remote console on one of the affected machines and log in and see that there was basically init, the shell that had just been started, and this fbagent process which was responsible for assassinating the entire system that morning.

The rest of the story has also been told, which is where it took me a couple of weeks to figure out why we kept losing machines this way, and when I did, I found the source had already been patched. Another engineer unrelated to the fbagent project had been hitting the same problem, decided to go digging, found the "-1" pid situation leaking through, and fixed it.

Even though the fix was committed, it wasn't shipped, because this binary was big and scary and ran as root on (then) hundreds of thousands of machines, and the person who usually shipped it was on vacation getting married somewhere. As a result, the old version stayed in prod for much longer than it otherwise would have, complete with the hair-trigger bug that would nuke every process on the machine.

All it needed was something that would screw up fork, and on that morning, it finally happened.

What hasn't really been told is that the memory situation had been steadily getting worse on those machines that whole summer. We had been watching it creep up, and kept trying to make things happen, but by and large, few people really cared. Also, people had been adding more and more crap to what the web servers would run. Back in those days, you could just tell your endpoint to run arbitrary code, and it basically would, right there on the web server!

Case in point: people had started running ffmpeg on our web servers. They decided that was an AWESOME place to transcode videos. By doing that, they didn't have to build out their own "tier" of machines to do that work, which would have meant requesting resources, and all of that other stuff. Instead, they just slipped that into a release and slowly turned up the percentage knob until it was everywhere.

ffmpeg is no small thing. One instance could pull nine CPU cores and use 800 MB of memory - that's actual memory, not just virtual mappings. Also, this made requests run really long, and when that happened, the "treadmill" in the web server couldn't happen sufficiently quickly.

What's the treadmill? Well, when you have memory allocations for a bunch of requests that then finish, you have to garbage-collect them eventually. My understanding is that the treadmill essentially worked by waiting until every request that had been active at the same time was also gone, and then it would free up the resources.

This is a little confusing so think about it this way. These machines were true multitasking, so they'd possibly have 100 or more web server threads running, each potentially servicing a request. Let's say requests A-M were running and then request N started up and allocated some memory. The memory allocated by N would only be freed once not only N was done, but A-M too, since they had overlapped it in time. If any of them were sticking around for a while, then N's resources couldn't be freed until that first one exited.

Given this, it's not too hard to see that really long-running requests effectively limit how often the "treadmill" can run, and thus how often the server will release memory for use in other things.

Also, there were other things going on which were just really expensive endpoints which could chew a gig of memory all by themselves. This was NOT scalable. You simply couldn't sustain that on these systems.

Basically, if you were to make a time-traveling phone call to me a few weeks before "Call the Cops" happened, and ask me what I was worried about, "web tier chewing memory and going into swap" probably would have been pretty high on the list.

To give some idea of how long this had been going on, that year, July 4th (a national holiday) fell on a Friday, so we had a "three day weekend". When this happened, the site didn't get pushed. This mattered because push would usually get the machines to free up a bunch of memory at once and generally become less-burdened.

A regular two-day weekend would leave things looking pretty thin by the time Monday's push rolled around, but a three-day weekend made things a lot worse... and this was a full month before everything finally broke.

So, yeah, the site broke that morning, but it's not like it was too surprising. The signs had been visible for quite a while in advance. Imagine standing on top of a massive dam and you start seeing one leak, then two, then four, and so on. You try to get help but it's just not happening.

Of course, once the dam actually fails, then somehow you find the resources to get people caring about dam maintenance. It's funny how that works.

It's that time again, when code written in the past four years shows up in our lives and breaks something. Still, while you're enjoying the clown show of game companies telling people to manually set the clocks on their consoles and people not being able to fill up their cars, keep one thing in mind:

Only half of the fun of a leap year happens on February 29th.

The rest of it happens in ten months, when a bunch more code finds out that it's somehow day 366, and promptly flips out. Thus, instead of preparing to party, those people get to spend the day finding out why their device is being stupid all of the sudden.

So, if you got through today unscathed, but are somehow counting days in the year somewhere, you now have about 305 days to make sure you don't have your own Zune bug buried in your own code.

...

One more random thought on the topic: some of today's kids will be around to see what happens in 2100. That one will be all kinds of fun to see who paid attention to their rules and who just guessed based on a clean division by four.

Maybe 15 years ago, I heard that a certain cell phone camera would lose the ability to autofocus for about two weeks, then it would go back to working for another two weeks, and so on. It had something to do with the time (<some unit> since the epoch), the bits in use, and a fun little thing called sign extension.

I got some of this from a leaflet that was posted around where I worked at the time. It was posted in areas where the public could see it, so I figure it's fair game.

Here's a nice little test program to show what I'm talking about:

#include <stdio.h>

static unsigned long set_bit_a(int bit) {
  return 1 << bit;
}

static unsigned long set_bit_b(int bit) {
  return 1U << bit;
}

int main() {
  printf("sizeof(unsigned long) here: %zd\n", sizeof(unsigned long));

  for (int i = 0; i < 32; ++i) {
    printf("1 << %d : 0x%lx | 0x%lx\n", i, set_bit_a(i), set_bit_b(i));
  }

  return 0;
}

This does something mildly interesting when run on a 64 bit system:

$ bin/exp/signext 
sizeof(unsigned long) here: 8
1 << 0 : 0x1 | 0x1
1 << 1 : 0x2 | 0x2
1 << 2 : 0x4 | 0x4
1 << 3 : 0x8 | 0x8
...
1 << 28 : 0x10000000 | 0x10000000
1 << 29 : 0x20000000 | 0x20000000
1 << 30 : 0x40000000 | 0x40000000
1 << 31 : 0xffffffff80000000 | 0x80000000

Meanwhile, the same code on a 32 bit machine is relatively boring:

$ ./t
sizeof(unsigned long) here: 4
1 << 0 : 0x1 | 0x1
1 << 1 : 0x2 | 0x2
1 << 2 : 0x4 | 0x4
1 << 3 : 0x8 | 0x8
...
1 << 28 : 0x10000000 | 0x10000000
1 << 29 : 0x20000000 | 0x20000000
1 << 30 : 0x40000000 | 0x40000000
1 << 31 : 0x80000000 | 0x80000000

Gotta love it.

In the vein of my "flash" story from a few years ago, here's one about "smurf".

Back around 1997, there was something new going around in the realm of net abuse: "smurfing" a target. This one involved a nice little trick that let you send out a relatively small amount of traffic and let someone else turn it into a much larger amount of traffic, and then that response would be directed onto your target.

This required two bits of cooperation from the environment. First, you had to be able to transmit a packet of some sort with the source address set to your target. Yes, this does mean "spoofing" the source address, and any responsible ISP should filter that nonsense on egress, but back then it was all too infrequent.

Then, you also had to send it to either the network or the broadcast address of some particularly juicy network that was laden with hosts that would reply to that sort of thing.

For example, let's say you had a network 192.0.2.0 with the netmask 255.255.255.0 (a /24). Then in that case, .0 would be the network and .255 would be the broadcast address. Back in those days, firing a packet at either of those would usually make the router spew it out to the *Ethernet* broadcast address, and so it would hit every host on that subnet which could then decide to reply or not.

So, just imagine a packet "from" your target, seemingly addressed to dozens or hundreds of machines, which then all answer at once. The attacker sends out a single ~1500 byte ping (for example), and the victim receives that multiplied by however many hosts decide to reply - not great!

There were some things which could be done about this. Routers eventually got a config knob that let you turn off "directed-broadcast" or similar, so anything arriving from the outside for a network or broadcast address would just be dropped on the floor. Unix boxes of different flavors also started gaining the ability to have packet filtering rules. (This took far too long in some cases.)

Besides that, people running networks could follow various best practices and not let traffic that's claiming to be from somewhere else in the world egress from their network. Any packet like that is either a misconfiguration on someone's part (possibly yours), or maybe some dummy trying to attack someone else. Either way, it needs to be tracked down and dealt with.

Sometimes people would use this kind of stuff as a reason to "block all ICMP", and then they would just create other problems like breaking path MTU discovery, and that would cause connections to hang with large packets and weird non-Ethernet-MTU-sized links.

Another related attack tool back then was called "fraggle" and it did the same sort of directed-broadcast shenanigans, but which used UDP instead of ICMP. The effect was the same, and it also got around anyone who thought that filtering all ICMP was somehow a good idea.

The old days weren't always good days.

It's another round of feedback, because there's been a lot going on.

I must admit that I did not expect my post about diffing LDAP to have such a response. I honestly just wanted to tell a story about some mildly rebellious activity I had seen happen and then had decided to do myself, and it turned into a whole thing. Lots of people wrote in to say that they have also been doing it, and others have started as a result of that post! That was not my intent but the net effect is definitely pleasing to me, so it all worked out.

Now, in response to some specific comments - a few people wrote in to say that the "epitaphs" internal page/service at Google (yep) now allows you to line some stuff up before you leave. That way, when you leave and your entry shows up, it'll have something on it that you submitted directly, and you don't have to "bounce through a friend" or whatever.

I think part of this is that people don't realize just how long it's been since I was plugged in to that ecosystem. I left in May of 2011 - almost thirteen years ago now! I thought it was broken badly enough to leave all the way back then, and this was after being there for about four and a half years. It still amazes me when I find out that people willingly go there, but I've had to tell myself to shut up about that and just advise them to "get in, get paid, and get the hell out".

Seriously, get in, take their money, and go. Whatever tech darling status they had was gone a LONG time ago. I dare say that I watched it curl up and die from the inside. I can't even imagine what could possibly be left inside there now that so much time has passed.

It occurs to me that sufficiently young people who are just now entering the industry fresh out of school (or whatever) have no idea what it used to be like. They've only known the current versions of things, and probably figure it's as bad as everywhere else, so why not, right? I guess it's hard to argue with that. Just never make the mistake of thinking that it's special somehow. Those days are gone gone gone and they aren't coming back. Companies which are that massive just can't deliver that kind of environment.

...

In response to the WPA3 stuff and badness happening after 11 hours, Ewen (and a few other people) wrote in and said that I should have been looking at minutes, not seconds. 39960 / 60 gives 666 minutes. Oops. Yeah, I guess I missed the forest for the trees there. 666 minutes would do the job, for sure. \m/ rock and roll?

...

Other people said they were diffing far more than just the list of unixnames in LDAP. They used it to detect people getting promoted when their titles changed, and other things like that. I honestly didn't care about detecting that, and I don't think that the LDAP (really AD behind the scenes) system I was poking at even stored such things.

A fair number of companies have a glossed-over view of things for their permission systems that don't reflect whatever HR has for those same people. Everyone in LDAP might be a "software engineer", but in the actual HR system they might have 100 different varieties for "new grad" and "testing" and "server" and "app" and all of these other dumb things that they think they need. That means you might not see anything change when people get promoted, change teams, or shift around between different parts of the company.

While I'm talking about titles, I will mention one thing: it's interesting that certain companies talk about why they hide levels for random "ICs" (individual contributors, i.e., not managers), but then go ahead and make a big deal out of managerial titles.

Seriously, one company in particular had everyone be some sort of Software Engineer or Production Engineer or something like that without saying that this person was a 3, or a 4, or a 5, or whatever on up the line.

Meanwhile, that same company let you see that a given person was a Manager (5, 6), a Director (7, 8), or a VP (9, 10) with just a glance at their profile page.

The same sort of visibility was not afforded to the ICs at those same higher levels. You had to "just know" that so and so was "one of the 10s" or whatever.

Also, for anyone who hasn't already seen my thoughts on the matter somehow: you are not your level, and your level is NOT an indication of basically anything more than how much they like you. It is only loosely linked to your ability at the bottommost rungs of the "career ladder", and only when management is being forced to adhere to it. If the right people like you, your level will rise. If they don't, it will stagnate. Your technical abilities are *almost* completely disconnected from there.

There is one notable exception I can mention though: if you are somehow able to do something that nobody else can/will do, they will "put up with you" as long as the amount of whatever you bring in outweighs the costs of you being, well, you. But, once you start asking for things or try to do stuff that goes against what they personally want, the balance will tilt, and once it goes past center into the other side, they won't give a damn about what you can deliver any more.

I should note this has little to do with what the business needs at that point. They're probably in it for themselves, and they don't care that chasing someone out is not the right thing for the business. Indeed, they will probably bail out for greener pastures a few years later.

...

Some people from a few very large tech companies that are currently doing layoffs have pointed out that their "epitaphs" or equivalent isn't always accurate. There are groups of people who will be "off limits" and so won't come up in the reports. Obviously, once management has gotten to that level of involvement with the day to day operations of such a tool, it can be considered compromised.

That's pretty much a given: things start out as a simple hack, then grow into a small community that knows about it, and sometimes end up becoming well-known and even legitimized. But, more often than not, these same systems will be co-opted by whoever's running the show in terms of hiring and firing, and it'll stop providing useful data.

At that point, I guess you have a choice: you can try to build another thing from scratch for yourself, or you can admit that the company is too far down the road of corporate lockdown hell, and live with the fact that it'll never be accessible the way it had been.

...

Finally, there's at least one person who was visibly annoyed by the fact that I said "uid" and "(unix account name)" in the same breath. Guess what? That particular LDAP (again, really AD) system I was dumping DID use a field named "uid" to put in the unixnames. Sure, there were also *numeric* uids in the Unix sense, but that lived elsewhere in other fields, like, oh, uidNumber. Surprise surprise.

I just love it when there's this assumption that I must be screwing it up by default. To that person: ask yourself if you'd assume that about every random post on people's web sites that might mention such a thing, or only certain ones. If it's only certain ones, I bet you know which ones, and why.

It's obvious. You're making it clear that you're part of the problem. Knock that shit off.

And, as for me making mistakes, hell yes I make mistakes. I screw up all kinds of stuff and have to go back and fix it and/or explain what happened where appropriate. Anyone who follows the feed will tell you that old posts will "mysteriously" spring back to life with a recent modified time and a handful of tweaks applied. Most of those fixes come about from people sending feedback and going "hey I think X might be Y instead". It happens all the time, and I'm talking about 13 years of posts here.

If you think I made a mistake back in that post by saying "uid" and "unix account name" instead of "numeric unix account number", you could just hit the feedback button and say as much. But you know that it's way more impactful to assume that I'm a dumbass and don't know what the hell I'm talking about and do that on a big orange sewer.

You do know what "projection" means, right?

Okay, enough of that.

One common element of the larger places where I've worked is that they tend to have a directory service of some sort that keeps track of who's an employee and who isn't. You can learn some interesting things by periodically dumping that list and then running comparisons against the previous dump.

A certain company had this rolled up into an internal service called "epitaphs" where an entry for a person would appear a day or two after they "disappeared from LDAP" - meaning, they left the company. Then other people who still worked there could add comments like "went back to school", "moved to Idaho to raise sheep", that kind of thing.

This had an interesting side-effect that you couldn't write to your own "epitaph" because by definition you had to already be gone from the company for your page to exist. Someone else who knew you had to add it. I actually received an e-mail to that effect one time: "I'm leaving, so when it shows up, please add XYZ". I was pleased that they trusted me to do that, and a few days later, I pasted it in as requested.

Another place I worked didn't have anything quite like this. There was the "internal profile" where you could see that so and so worked at the company from <date> to <date>, but there wasn't any sort of periodic update available. I decided to roll my own. It didn't take much in the way of effort, really. A cron job on my dev server (a physical box in a datacenter with access to my home directory) woke up a couple of times every day and dumped the entire list to a file. Then it compared it to the last one, crunched it down to just the uid (unix account name) field, and appended the results to a log file.

Over time, various other people learned about this, and since I had left it world-readable, they were able to leave up a "tail -f <path>" to keep tabs on it, and sometimes something surprising would show up during the day. People would sometimes just vanish. Other times, there were bizarre things going on that added a bit of context.

The log entries looked like this:

Thu Feb 08 18:26:42 PST 2024 : uid: <someone>

That was enough to let you go digging and find out more if you actually gave a damn about why that particular person no longer worked there. Otherwise, it didn't flood you with useless data.

One time, I pasted in a line like that into an IRC channel and that <someone> popped up and said "yeah, I don't work here any more". It turned out their account had been deactivated, but they still had a client connected. When I mentioned their account name, they got a notification, flipped to that window, and replied. We had a few minutes to chat about it.

It was weird saying farewell to someone that way. Normally, the electronic lines of communication are severed early on. I think what happened here is that the IRC servers only checked auth at connect-time, and then nothing went back to make sure that sessions remained associated with current employees. (It's a bit of a hard problem.)

Another time, some manager type said they were going to be late for a meeting because of some "dumb manager thing" they had to do. Sure enough, a few minutes into that meeting, a line scrolled across showing the deactivation of an account of one of their direct reports. Obviously, they had to go into one of those HR meetings where they showed someone the door.

I'd say the best time to start doing this is when you start at a company, or when that company grows big enough to actually have LDAP or whatever. That means the second-best time would be today.

Incidentally, the 'comm' tool is great for this sort of thing.

comm -2 -3 <(grep ^uid: old | sort) <(grep ^uid: new | sort)

... and there you go.

Now, this sort of thing is not perfect. If you don't catch errors, the first time it fails to dump and yet diffs a full list against an empty list, it'll look like everyone quit. This is not what you want. Also, once you work at a big enough company, there WILL be days when some automation will run amok and "fire" everyone, and every account will be deactivated. This will happen more than once if you stay there long enough.

Incidentally, if someone gets mad about you running this sort of thing, you probably don't want to work there anyway. On the other hand, if you're able to build such tools without IT or similar getting "threatened" by it, then you might be somewhere that actually enjoys creating interesting and useful stuff. Treasure such places. They don't tend to last.

It's time for me to respond to some recent feedback. As usual, this is a mix of topics and the responses are pretty much off the cuff, so strap in and hold on tight.

...

At least one person mentioned the 11 hour WPA3 problem on my Raspberry Pis and asked if I was experiencing clock drift. This is kind of funny to me since I've been picky about keeping clocks synced in my personal and professional lives these past few years. So, no, not really. All of those Pis have chrony installed, and it's doing a great job of keeping their clocks disciplined.

I was the crazy person who spent $300 of my own money to buy a GPS-to-NTP box when the unmaintained corporate infrastructure at a certain job was down to its last "proper" time server and was in danger of failing itself. It never came to that, but we got mighty close that winter. If it had fallen over, then I would have "backfed" time into production from the corporate network using a little GPS antenna puck in the window by my desk. How crazy is that?

I've gone to lengths to make things right, put it that way.

...

Also regarding the WPA3/Pi stuff, someone said "if it is truly cursed, it will be 11 hours and 360 seconds". That would be, what... 11:06:00? I don't think that joke landed with me. I was hoping it would involve "666" somehow, but (11 * 60) + 360 is 39960.

They did mention that an 11:06:40 period is a rollover from 9999999 to 10000000 jiffies with Hz set at 250. That is, if you tick at 250 Hz, after 11 hours, you'll be at 9900000 ticks, and another 100000 ticks past that is 400 seconds, hence the 06:40 part.

Of course, for that to break something, some clown would have to be expressing the time as ASCII digits, and then breaking when it got "too wide". I mean, it happens. It happened to KDE back in September 2001 when time_t went from 999999999 to 1000000000. That was a "fun" one to deal with.

...

Niels asks how I keep my posts "so level-headed". I guess that's in the eye of the beholder, to be honest. I've had situations where I've deliberately taken every bit of emotion out of my actions, and *still* had people calling it out after the fact.

After a few decades of this, I've concluded that a lot of these results were decided long before I opened my mouth or started writing. They basically have a problem with the overall concept of me saying stuff, and exactly what gets said doesn't matter a whole lot. They just use the words I choose in order to sprinkle it throughout their takedowns.

The way you can know this is happening is when the same words come from someone else who has a different *ahem* "presence" in the same space, and then nothing bad happens, or perhaps it's taken as good. That happens too, and it's really irritating since it serves to confirm to you that it may be the 2020s, but it's really just a number in terms of people being narrow-minded and generally pig-headed.

...

A reader writes that I should try wpa_supplicant 2.10 on my RPI, and, well, I'm sorry to say that I've done that and then some, and it didn't help. Check it out - wpa_supplicant as built from upstream (hostap's git repo) *does* bring the link up... for about 10 seconds. Then it throws an error and kills it. It looks like this:

Jan 28 13:32:24 rpi5b NetworkManager[847]: <info>  [1706477544.7638] device (p2p-dev-wlan0): supplicant management interface state: associating -> associated
...
Jan 28 13:32:34 rpi5b wpa_supplicant[1209]: wlan0: Authentication with <AP> timed out.

It's a consistent 10 seconds. I was feeling like torturing myself that afternoon, so I started screwing around with the code. It took a while to find where things were happening, but I finally just extended the timeout. It would hold on there a bit longer, then it would die after the new timeout.

So, finally, I just commented out the part where it complains about that and tears down the link, and you know what? It stayed up. So, I went back a little bit, and just let it bring the link up, then suspended it with ^Z and went about my day.

Three hours later, it was fine. Two hours after that, same thing. I kept checking on it into the night. Then, finally, it died early the next morning, even with wpa_supplicant *still suspended*. Here's what it looked like from the AP side:

Mon Jan 29 01:19:23 2024 daemon.info hostapd[6371]: ath1: STA <pi 5's mac address> IEEE 802.11: disassociated

And yeah, that's about 11 hours after I started the last experiment. (I should mention that having to wait 11 hours to verify things absolutely sucks.)

What I gather from this and what a few people have told me so far is that the actual association is kept running by the device itself, and whatever you have running on the host machine (iwd or wpa_supplicant) is basically along for the ride.

Also, I could swear I found a patch from someone to one of those projects that basically says "it's gonna die after 12 hours anyway, so just let it happen and restart the connection then". I can't find it while writing this, but it doesn't matter. Here's what's wrong with "just let it fail": NetworkManager has a limit for exactly how much crap it'll accept, and after the default of 3 retries, it'll just leave the link down.

Sure, you can override this, but now you're stuck with Yet Another Behavioral Patch for any of your machines which might be affected by this. Have fun with that.

Oh, and, obviously, forget about actually DOING anything over that link while it's being restarted. If you care about reliability, this is not a tenable situation.

Once again, if you care about having decent "modern" (as in 2020) wifi on a Pi, go get yourself a little USB barnacle that's supported upstream and go on with life. Or, better yet, just use hardwired Ethernet and forget that the thing even has a radio on board.

I'll pour one out for the sanity of anyone who doesn't have a choice in the matter. At least you're not suffering by yourself.

As my assortment of dumb little home-grown utility programs grew over the years, I found myself needing to know when a given binary was built. Sometimes things exist outside the realm of a packaging system, and so the binary itself needs to convey that metadata from build time.

I've seen some places solve for this by having giant placeholder strings baked into their binaries that they then reach in and "stamp" later, turning the "XXXXXXXX" or whatever into "Built by foo@foo.blah.evilcorp on ...". While that approach mostly worked, it was too spooky for me and I decided to stay away from it.

My system is something that uses a little nuance of C++ that I've mentioned a couple of times already. It's not the cleanest thing and it does involve a bit of groaning, but it works. In case anyone else wants to try it in their projects, here's how I set it up.

First, I have this buildinfo/base.h, and in it, I define a struct called Details, and it has all of the fields I care about - times, hostnames, usernames, commit hashes, that kind of thing.

There's also this:

extern std::optional<Details> details_;

Yes, that is globally visible, but it's inside a namespace so the crap factor is reduced somewhat. It's a necessary evil.

I also have a buildinfo/base.cc and it actually creates that variable:

std::optional<Details> details_;

There's also a GetBuildDetails function which will return the value of details_ if one exists, or a suitable error if not.

Now, you might be saying "it'll never have a value, so it'll always be an error", and you're mostly right. Just from what I've described so far, that in fact is the case. buildinfo/base.{cc,h} rolls up into buildinfo/base.o, and that gets linked into my programs during a normal development type build. If one of those programs calls the GetBuildDetails() function, then yes, they get the "sorry, nobody home" error response.

But, I have a way to inject the build info when I do a "production" build. This kind of build has slightly different config settings in my build system, and one of them tells it to "stamp" the binary.

The way this works is where the evil starts slipping in. On stamped builds, my build system writes out a file called buildinfo/overlay.cc. This file #includes "buildinfo/base.h" to pick up the definition of Details and the 'extern' for details_ itself. Then it rattles off a bunch of variables and their values (build time, build host, build user, ...) then it defines a class called Overlay.

Overlay's constructor has one job: it reaches into details_ and populates a bunch of fields with the values from those earlier variables.

Then the thing that actually makes it run shows up, and here's the "spooky action at a distance":

static Overlay overlay;

Just by having that line in the file, it will cause the program to create an instance of that class shortly before it reaches main() as long as it's linked into the final binary. When that constructor runs, it will populate details_, and then any code run from the rest of the program will see the build info.

This is convoluted, so I'll restate it here for clarity: it's the difference between linking "a.o", "b.o" and "c.o" into "prog", or "a.o", "b.o", "c.o" *and* "overlay.o" into "prog". If you don't link in that extra object, the sneaky stuff never happens, and it stays unpopulated. Using a std::optional wrapper saves us from the jankiness of using a bare pointer... or worse.

There are some bonuses from using the intermediate variables instead of just having a bunch of .field = "val" type things in the part where details_ gets initialized. For one, if those variables are not set to static, then they'll be visible to things like debuggers and certain other tools. Then you can do something like this:

$ gdb lavalamp_server -q
Reading symbols from lavalamp_server...
(gdb) print buildinfo::kBuildTime
$1 = 1707186971
(gdb) 

That's pretty neat, right? Analysis of a binary at rest? You can even do this without going through a debugger if you really want to.

Finally, how does the build tool handle this? It's a bit more of the special-case stuff for something that isn't just an ordinary build. If "stamping" of binaries has been requested for the active build type, then it generates a fresh overlay.cc and compiles it to overlay.o.

Then, in the link stage, if it's in "stamp mode", it inserts that object file as just another dependency of the build target as if it had been discovered by way of a '#include "some_dir/some_lib.h"' or whatever. This adds it to the list of objects passed to the linker, and a fresh binary pops out a few moments later.

I'm a fan of this technique since it only really adds two places in the build system where things go off and act a little strangely: the init sequence of the build tool when it's first built, and the link sequence of the target(s) to make sure it gets "injected".

For anyone who's worrying about "repeatable builds" or somesuch, I will point out that nothing's stopping you from having yet another build type which is otherwise as described above but which puts known placeholder data into the details_ variable. In that world, you should be able to go through the entire process and still get the same output, even on a different date, on a different box, and as another username.

I no longer wonder about which version of a binary is in "prod".

What a night. I hit upon something that got WPA3 working on some Raspberry Pi systems and excitedly put up a post to share the good news. Then I went away for a while, and this morning found something new: the damn things won't stay connected for more than 11 hours. All three of them failed in the same order that I changed them over.

The timeline is something like this:

01:03:51 NetworkManager [...]: new IWD device state is connected
[...]
12:04:39 iwd[...]: Received Deauthentication event, reason: 0, from_ap: false

Now that's something, considering none of the tunable aspects of the WPA3/SAE setup on my network are set to 11 hours. But, what's this? You do a search for "received deauthentication event" and "11 hours" and what do you find but a cursed Infineon developer community post on that very topic.

It's from October 2021 (!) and it's from someone who's using the same sort of chipset (Broadcom/Cypress/Infineon CYW stuff) NOT on a Pi, and they get the same "deauth" thing when on a WPA3 network. If then drop down to WPA2, it stops. The thread runs for close to a year, and then just stops cold in August 2022 with no resolution.

So, what'll it be? WPA2 mode and not being able to get onto any networks that have gone to 6E or 7, or WPA3 mode and having it fall off the network every 11 hours?

Here, I'll make a prediction: someone will say "it works most of the time, so that's fine, we won't be fixing this". Considering the level of crap people put up with in their tech these days, that'll probably be the steady-state.

My conclusion: this entire ecosystem is deeply cursed.

Okay, it's been several months since I last wrote about WPA3 on Raspberry Pi hardware. Now, I have some good news: it now mostly works, assuming you're willing to do a little tinkering. You no longer have to wrangle custom firmwares and binary blobs into place. That's been done for you.

One important thing here: I'm only talking about Raspbian/Raspberry Pi OS here, and then only bookworm (12). If you're running something else, none of this may apply. For all I know, it might have been working all along if your distribution figured it out sooner.

So then, if you have a 3B+, 4B or 5B on bookworm, get ready to rock.

Every so often I look at the list of package updates coming down the pipe through apt for my systems, and usually groan at the usual round of CVE patches. But, this time I saw something rather different. It's a change to "firmware-brcm80211", and what's this? Something specific to the CYW43455 that the 3B+ and later use? Oh, that's interesting, right?

  * brcm80211: cypress: Use a generic CYW43455 firmware
    - Version: 7.45.234 (4ca95bb CY) CRC: 212e223d Date: Thu 2021-04-15 03:06:00 PDT Ucode Ver: 1043.2161 FWID 01-996384e2

I wondered if this would stop the madness, and so applied it and rebooted. "iw phy" showed the good news - at the very bottom, it now supports both SAE_OFFLOAD and SAE_OFFLOAD_AP. This means it can actually do SAE... but you're going to have to say goodbye to wpa_supplicant in favor of iwd.

If you switched over to NetworkManager for consistency with the rest of the bookworm changes, this is not a big deal. If you're running your network some other way, you get to figure this out yourself.

The steps work out like this: apt update, then apt upgrade so you get the firmware-brcm80211 from January 17th (or later, once this post gets old, I guess). Then apt install iwd.

Assuming you've been running your Pi in a crappy non-WPA3 network, go into nmtui or equivalent and disable that network. In a minute or two, you're not going to need it any more.

Then disable wpa_supplicant and drop a bit of config into /etc/NetworkManager/conf.d/iwd.conf:

[device]
wifi.backend=iwd

Reboot ... or do equivalent wrangling of drivers and binary crap to get it to unload and reload the fresh stuff. Then run "iw phy" again. If it says SAE_OFFLOAD and SAE_OFFLOAD_AP, you're ready to proceed. If not, well, something's wrong, and you should turn on the crappy old non-WPA3 network again.

Assuming it worked, then go into nmtui or whatever and tell it to activate the actual WPA3-only network. Paste in your PSK and let it go, and a few seconds later you should be in business.

That's it. That's all it takes.

Now then, what about the 3B, you might ask. It's on a different chip that doesn't even do 5 GHz, so changing the 43455 firmware wouldn't help any. It seems to be a 43430 instead, and I have no idea if there's any chance of getting a similar firmware change for it. Obviously, if this changes, I'll post something about it, too.

I can't comment on the other models, like the Zeroes and the other weird little boards they have. I only have access to these relatively normal models, and that's what I was able to work on.

Go forth and have better networks!

January 24, 2024: This post has an update.

It's a new year, so let's talk about some more time-related shenanigans. This one comes from the world of writing C++ for multiple platforms. A couple of weeks ago, I was looking at some of my code and couldn't remember why it did something goofy-looking. It's a utility function that runs stat() on a target path and returns the mtime as a std::chrono::system_clock::time_point. This is nicer than using a time_t since it has sub-second precision.

The trick is getting it out of a "struct stat" and into that time_point. The integer part is simple enough: you use from_time_t on the tv_sec field. But then you have to get the nanoseconds (tv_nsec) from that struct into your time_point. What do you do?

The "obvious" answer sounds something like this: add std::chrono::nanoseconds(foo.tv_nsec) to your time_point. It even works in a few places! It just doesn't work everywhere. On a Mac, it'll blow up with a nasty compiler error. Good luck trying to make sense of this the first time you see it:

exp/tp.cc:14:6: error: no viable overloaded '+='
  tp += std::chrono::nanoseconds(2345);
  ~~ ^  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__chrono/time_point.h:65:73: note: candidate function not viable: no known conversion from 'duration<[...], ratio<[...], 1000000000>>' to 'const duration<[...], ratio<[...], 1000000>>' for 1st argument
    _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR_SINCE_CXX17 time_point& operator+=(const duration& __d) {__d_ += __d; return *this;}

Nice, right? It tells you that there's something wrong, but the chances of someone figuring that out quickly are pretty slim. For the benefit of anyone else who encounters this, it's basically this: a system_clock::time_point on that platform isn't fine enough to represent nanoseconds, and they're keeping you from throwing away precision.

To make it happy, you have to jam it through a duration_cast and just accept the lack of precision - you're basically shaving off the last three digits, so instead of something like 0.111222333 seconds, your time will appear as 0.111222 seconds. The nanoseconds are gone.

I assume you might find other platforms out there which don't support microseconds or even milliseconds, and so you'd hit the same problem with trying to "just add" them to system clock time point.

At any rate, here's a little bit of demo code to show what I'm talking about. As-is, it'll run on Linux boxes and Macs, and it'll show slightly different results.

#include <stdio.h>

#include <chrono>

int main() {
  std::chrono::system_clock::time_point tp =
      std::chrono::system_clock::from_time_t(1234567890);

  // Okay.
  tp += std::chrono::milliseconds(1);

  // No problem here so far.
  tp += std::chrono::microseconds(1);

  // But... this fails on Macs:
  // tp += std::chrono::nanoseconds(123);

  // So you adapt, and this works everywhere.  It slices off some of that
  // precision without any hint as to why or when, and it's ugly too!

  tp += std::chrono::duration_cast<std::chrono::system_clock::duration>(
      std::chrono::nanoseconds(123));

  // Something like this swaps the horizontal verbosity for vertical
  // stretchiness (and still slices off that precision).

  using std::chrono::duration_cast;
  using std::chrono::system_clock;
  using std::chrono::nanoseconds;

  tp += duration_cast<system_clock::duration>(nanoseconds(123));

  // This is what you ended up with:

  auto tse = tp.time_since_epoch();

  printf("%lld\n", (long long) duration_cast<nanoseconds>(tse).count());

  // Output meaning when split up:
  //
  //        sec        ms  us  ns
  //
  // macOS: 1234567890 001 001 000  <-- 000 = loss of precision (246 ns)
  //
  // Linux: 1234567890 001 001 246  <-- 246 = 123 + 123 (expected)
  //

  return 0;
}

To bring this full-circle, that's why I have that ugly thing in my code to handle the addition of the tv_nsec field. Without it, the code doesn't even compile on a Mac.

Stuff like this is why comments can be very important after the fact.

It's time for some end of the year feedback. I get a bunch of comments and questions from people through my contact page, and sometimes this is the only way to reply. Other times, a response is also suitable for a wider audience.

...

Igor asks:

Have you played around with cling? Seems you may have the knowledge to break it or request the developers to enhance it with some useful feature(s). Like suppose doing scripting in it?

I didn't even know what cling was until I read this and went looking. It seems to be a clang/LLVM-based C++ interpreter that's interactive. I guess I don't have a need for something like that. As far as deliberately breaking things, I have no doubt that my luck would lead to any number of bad outcomes. However, I have enough of that in my life already without seeking out new stuff just for the sake of having something to break.

Now, if there was a good reason for me to do it, that would be another story. (I am available for mercenary purposes, put it that way.)

...

An anonymous reader asks:

What's the specs of this new web server?

It's only new to me. It's actually fairly old. I think it's roughly 2014 vintage. I bought it from one of those vendors who resell old servers. I figured "it's a server, not a toothbrush" and so the notion of using some random old box was not a problem for me. It runs Linux and is plenty quick, so I'm happy enough.

I guess this is a good point to tell the story of actually installing this thing. Despite working at places with millions of Linux boxes over the years, I'd never hung a server. I'd done routers and dialup boxes, but those were all relatively short. They'd hang from the front posts and were plenty happy with life that way.

flicker, on the other hand, is a monster. It's so long the cabinet doors almost didn't close. I had no idea that length was a dimension that might be a problem. I clearly didn't realize just what I was getting when that order was placed, and when it showed up in a giant box, I started thinking "what have I done?"...

Initially, the machine barely fit into the cabinet with the power cord being squished on one end and the network cables being squished on the other. I came back about a month later and swapped it to a new power cord that has a 90 degree bend built into the server-side plug. This clawed back about an inch on that side and let the whole thing slide back a little bit which took the pressure off the Ethernet cables.

I took the now-permanently-crimped (and damaged) power cable, cut it in half so nobody would find it and try to use it, and tossed it.

It was the sort of thing that anyone with experience would point at and go "ha, you screwed up", and indeed, I did. It turned out to not be fatal to the project of moving to colocation, but it was mighty close.

Next time, I'll pay attention to this sort of thing.

Lesson learned: go see the cabinet and measure it before ordering something that's almost three feet long!

(It's an Ivy Bridge flavored dual-socket hex core Xeon with 128 GB of memory and a couple of SSDs. It also has a far faster pipe than the previous box. It's a complete monster for how I use it. Oh, and it's not in Texas, which has turned out to be very important.)

...

Regarding my "char buf[1048576]" thing from the other day where I blew the stack, a reader says:

You can use boost::thread instead. It lets you specify the stack size.

Honestly though, I don't *want* to specify the stack size. I'm fine with defaults. Also, any answer which involves "boost" means I'm asking the wrong question. I don't ever want to get stuck with something like that. There's been too much drama in my life from boost dependencies in years gone by.

The obvious way around the "problem" for me is to do something like a vector<char>, push it out to whatever size, and then let read() use it as a buffer. This is pretty much what I end up doing any time I have to deal with old-school C library functions from my C++ code.

Like, okay, mkdtemp(). It wants a char* template, like "/tmp/test_thingy.XXXXXXXX". It *reaches into* that space and alters it, changing the Xs to some random gunk that turned out to be unique - this is how you avoid /tmp race attacks. This is more annoying than it sounds.

You might think "oh I know, I'll use a string, then hand the .c_str() pointer from it to mkdtemp". But, no, c_str() gives you a *const* pointer, and mkdtemp is going to violate that const-ness. Your compiler will stop you from doing this. You then must either lie to it and do some nasty casting, or you decide that's not going to work and find another way.

This is why I have a bunch of stupid shims to deal with these scenarios. It's just that I never really shimmed read() before - the old hacky way of having a buffer on the stack was never a problem.

As for "why is that buffer so big", well, I like to cut down on the number of syscalls required to inhale a file: a bigger buffer eats the whole thing in a single go. A tiny buffer would need to keep coming back until it finished. This sort of thing used to matter more when machines were much slower.

Besides, I hate spammy syscall situations. read() or write() with a single byte? Argh!

...

From another reader:

hey are you still running wg barebones on your mac? I just made the mistake of installing and got the 'cant uninstall' behavior with additional 'nothing happens when you dbl click on app icon' . just wanted to compare notes

Yep, I still am doing this on my personal machine and also those of family members (in my duties as "holiday sysadmin"), having removed the "app store" version a while back. I get feedback from random people who find that three year old post about once a quarter, and they all seem to have the same problems. I take that as a pretty strong signal to not try it again just yet.

I should note that if you're using Macports, upgrade to Sonoma, and then try to build wireguard-tools or several other packages, it'll fail due to various Apple-related stupidity. I'll do another post about that eventually.

...

Finally, a bunch of people wrote in about the "clang + C++ + original armv6 Raspberry Pi = unusable binaries" to say that it explained some oddities they had been seeing on their own stuff, so I think that turned out okay. That's really what I aim for in writing these posts: visibility into problems, and confirmation that you aren't alone in experiencing something. Sometimes it's technical, but other times it's about squishy meatspace stuff. Both are valid.

I need to remind people how easy it is to forget just one of the many gotchas of working on this ridiculous computer stuff. One missed nugget of data at a critical moment can leave you scratching your head and going "WTF" for longer than would otherwise seem reasonable.

Here's something that happened to me last week. I was working on a stupid little utility that runs on my machines and lets me keep tabs on what's going on with systemd. If it gets unhappy because any of the services have stopped running, then this thing will let me know about it. For the handful of systems I have to worry about, it gets the job done.

Now, since I'm in "holiday mode", I'm largely working on my laptop instead of sshing back to a Linux box somewhere else. This laptop is a Mac, so it's mostly compatible with what I'm doing. Obviously, it doesn't run systemd, but that wouldn't stop me from tidying up a tool in test mode. I was working on this thing, and noticed it started blowing up in strange places. Also, it was a really strange "bus error". To me, that says "binaries on NFS" or "unaligned access on some architectures". I'm not doing either sort of thing here.

gdb was not really an option at that moment for various annoying reasons so I resorted to "debug via printf" - putting little notes to say "I got here" and whatnot. They kept changing. I'd think I had it nailed down, and it would move!

Eventually, I got it down to something truly odd: it was blowing up in a worker thread, and it was the point where that thread started up and read in a config file from the disk. The line of code looked something like this, where I call into one of my own helper libraries:

auto raw = file::ReadFileToString(kDefaultConfigPath);

Okay, I said to myself, let's find out what's going on in that function and started sprinkling my "I got here" notes into there. One of those notes was at the very top of that function and just said "got into ReadFileToString". It never ran.

I removed the call to that function. It stopped crashing.

So, what's in that function that's so spooky? Well, it opens a file descriptor, does the usual sanity checks on it, and then creates a buffer that it'll pass to read()... and herein lies the problem:

  char buf[1048576];

Yep, just having that there was blowing the stack, and the bus error is how it manifested in that particular arrangement of function calls within the worker thread.

That's right, if you're already pressed for stack space and then enter a function with something like that, you might just explode. Here's a contrived example with an even bigger buffer to demonstrate it with just a single innocent-seeming function call:

mac$ cat bs.cc 
#include <stdio.h>

#include <memory>
#include <thread>

static void do_thing() {
  char buf[1048576 * 8];
  buf[0] = '\0';
}

int main(int argc, char** argv) {
  if (argc != 1) {
    printf("running in worker thread\n");

    auto worker = std::make_unique<std::thread>(&do_thing);
    worker->join();
    return 0;
  }

  printf("running in main\n");
  do_thing();
  return 0;
}

The fun part is that on a Mac, the flavor of error changes between "bus error" and "segmentation fault" just by shoveling it into a thread.

mac$ ./bs 
running in main
zsh: segmentation fault  ./bs
mac$ ./bs foo
running in worker thread
zsh: bus error  ./bs foo
mac$ 

Nice, right? Further complicating matters is that on a boring old x86_64 Linux box, it gets reported as a segmentation fault both ways.

linux$ ./bs
running in main
Segmentation fault
linux$ ./bs foo
running in worker thread
Segmentation fault

A simple twiddling of the ulimits will change the behavior ever so slightly:

linux$ ulimit -s unlimited
linux$ ./bs
running in main
linux$ ./bs foo
running in worker thread
Segmentation fault
linux$ 

Fun fun fun. Obviously, I need to rethink the way I manage my buffers.

Sometimes, inside a company, you find someone who's just so good at what they do and who has the fire burning inside them to always do the right thing. It's easy to fall into a spot where you just go to them first instead of chasing down the actual person holding the pager that week. At some point you have to tell yourself to at least *try* "going through channels" to give them a break.

But still, you wind up with some tremendous stories from when they came through and saved the day. I love collecting these stories and I periodically share them here. This is another one of those times.

Back up about a decade. There was something new happening where people were starting to get serious about compressing their HTTP responses to cut down on bandwidth and latency both: fewer packets = fewer ACKs = less waiting in general. You get the idea.

A new version of the app for one particular flavor of mobile device had just been built which could handle this particular flavor of compression. It was going out to alpha and then beta testers, so it wasn't full-scale yet. When it made a request, it included a HTTP header that said "hey, web server, I can handle the new stuff, so please use it when you talk back to me".

On my side of the world, we didn't know this right away. All we knew was that our web servers had started dying. It was one here, then one there, then some more over in this other spot, and a few more back in the first place, and it was slowly creeping up. This wasn't great.

We eventually figured out that it was crashing in this new compression code. It had been added to the web server's binary code at some point before, and it obviously had a problem, but I don't think we had a good way to turn it off from our side. So, every time one of these new clients showed up with a request, their header switched on the new code for that response, and when it ran, the whole thing blew up.

When the web server hit the bad code, it not only killed the request from the alpha/beta app, but it also took down every other one that same machine was serving at that moment. Given that these systems could easily be doing dozens of requests simultaneously, so this was no small thing! Lots of people started noticing.

That's when one of those amazing people I mentioned earlier stepped in. He knew how to wrangle the proxies which sat between the outside world and our web servers. It had a scripting language which could be used to apply certain transforms to the data passing through it without going through a whole recompile & redeploy process for the actual proxies.

What he did was quick and decisive: it was a rule to drop the "turn on the new compression" header on incoming HTTP requests. With those stripped from the request, the web server wouldn't go down the branch into the new (bad) code, and wouldn't explode. We stopped losing web servers, and we were now in a situation where the pressure was off and we could work on the actual crash problem.

I should mention that we were unable to just switch off the new feature in the clients. The way that clients found out what features to run in the first place was by talking to the web servers. They'd get an updated list of what to enable or disable, and would proceed that way. But, if the web server crashed every time they talked to it, they would never get an update.

That's why this little hack was so effective. It broke the cycle and let us regain control of the situation. Otherwise, as the app shipped out to more and more people, we would have had a very bad day as every query killed the web servers.

And yes, we do refer to such anomalies as a "query of death". They tend to be insidious, such that when they show up, they take down a whole multitenant node and all of the other requests too. Then they inevitably get retried, find another node and nuke that one too. Pretty soon, you have no servers left.

I have a bunch of Raspberry Pi systems all over the place, goofy things that they are. They do dumb and annoying jobs in strange locations. I even have one of the older models, which is called just the B+. You can think of it as the "1B+" but apparently it was never officially branded the 1.

If you have one of these, or perhaps an original Pi Zero hanging around, you might find that C++ programs built with clang don't work any more. I ran into this as soon as I started trying to take binaries from my "build host" (a much faster Pi 4B) to run them on this original beast. It throws an illegal instruction.

This used to work in the old version (bullseye). It now breaks in the current one (bookworm). I figured, okay, maybe it's doing some optimization because it was built on the 4B. So, I went and did a build on the B+ natively. It also broke.

So I backed off another level to a much simpler reproduction case: just declare main() and return. That still broke.

Looking this up, there are a bunch of screwy dead-end forum posts where people go back and forth asserting this package is installed and that's making the compiler go stupid, or it's because they did the "lite" install vs. the "recommended" install, or who knows what.

I wanted to do better than that, so this afternoon I picked up a brand new SD card, blew the whole "desktop + recommended" OS image onto it, booted *that*, then installed clang, and...

raspberrypi:~/prog$ cat t.cc
#include <stdio.h>

int main() {
  return 0;
}
raspberrypi:~/prog$ clang++ -Wall -o t t.cc
raspberrypi:~/prog$ ./t
Illegal instruction

Awesome. It can compile something it can't even run. What's the bad instruction? gdb will answer that in a jiffy.

(gdb) disassemble
Dump of assembler code for function main:
   0x004005a4 <+0>:	sub	sp, sp, #4
=> 0x004005a8 <+4>:	movw	r0, #0

movw. That's not in armv6l, apparently. So yeah, this compiler is effectively cross-compiling for armv7 (or something) by default. That's not very useful.

You can work around this by grabbing the compiler by the lapels and saying "build for armv6, punk", and it will give you a working binary:

raspberrypi:~/prog$ clang++ --target=armv6-unknown-linux-gnueabihf -Wall -o t t.cc
raspberrypi:~/prog$ ./t
raspberrypi:~/prog$ 

How and why did it get to that point? I can only imagine it's some default that got bumped from version 11 to version 12, and somehow nobody noticed? I guess nobody still runs these old things anywhere?

So weird.

A little over 20 years ago, Verisign did something mighty evil: they effectively typosquatted every single unregistered domain in the .com and .net top-level domains. They could do this because they controlled those from the registry side of things, and it was trivial to slam something that would make it resolve.

Reactions from people like me who had systems to run and spam to block were swift and universally negative due to all of the collateral damage it caused. Here's one situation it created: sometimes you'd have a user who thought they were being clever, and they'd put something like "nospam" in the from address in their e-mail client. Thus, they'd try to send mail as luser@nospam-example.com instead of just @example.com.

Before Verisign pulled that crap, the mail servers of the day would have just rejected it as an invalid domain that didn't resolve (in DNS). Once it was online, it WOULD resolve, and so the mail system would accept it. Utter chaos.

Or, how about all of the random people who would mistype something? Instead of getting a failure to resolve error from their browser and/or organization's web proxy, it would just pop out to that stupid Verisign site.

It should surprise nobody that a bunch of us sour sysadmin types did things about it rather quickly over the next day or so. I still have the snippet of cruft to drop into a sendmail.cf in my notes from back then. I got this from Usenet, apparently:

Local_check_mail
R$*                     $: $>canonify $1
R$*<@$*.>               $: $1<@$2> strip the trailing . if present
R$*<@$+>                $: $(verisign $2 $)
R64.94.110.11           $#error $: "550 Real domain name required for sender address"

Incidentally, if any of you can still parse that syntax in your heads just by reading it, I'm so sorry. Take it from me - that ability does go away eventually, but you have to stop supplying it with new data.

Anyway, what that did was to exploit the limitless programmability of sendmail's config language to resolve the supplied domain. If it came back to 64.94.110.11, it would reject it right then and there. That was the IP address they were using for their little marketdroid-fueled fever dream, and I bet there are *still* systems blocking it 20 years later (and they probably have no idea why).

Over the next couple of days, a bunch of other things happened. ISC wrote a "delegation-only" feature for BIND (aka named, the occasional remote sudo implementation). You could use this to say that the "com." or "net." zones were only allowed to provide delegation to other zones. That is, within the bailiwick of "com.", it could say that "example.com." has a nameserver at <foo> with IP address <bar>, but that's it. It couldn't come right out and say that something had an A record outright.

Now, this worked great, but it wouldn't have been terribly difficult for them to sidestep it. They could have had it delegate all of those things down to some other level which then would have had a blanket answer for any incoming questions. Fortunately, this did not happen.

This whole bout of stupidity lasted about three weeks, and then it disappeared. They've never done it again, but plenty of other providers of recursive resolver action have pulled it since then as a matter of course. Screw up the way DNS works in order to fellate the advertisers? Brilliant!

Remember this? That was 2019.

I find it strangely fulfilling that this event has garnered a Wikipedia article in the years since. Hopefully people will never forget what happened back in 2003 with the DNS.

Perhaps you've been running Ubiquiti stuff for a while, and you've been disappointed by their stock issues, their goofy software issues, and the general lack of quality all the way around. Maybe you turned your eyes to the TP-Link Omada ecosystem. I'm here to warn you that the grass is not greener on that side of the fence. It may in fact be spray-painted.

First, some context. I'm the family sysadmin - not by choice, but because nobody else would do it. When I visit family, I have to fix their stuff. There are some gearhead types and I do my best to make them happy. Various ISPs are starting to sell services that are well above 1 Gbps. This is typically symmetric fiber stuff.

That's the situation with one of the sites I support, and their existing Ubiquiti stuff from years gone by became a bottleneck once they had that installed. Obviously, they want to get this greater-than-gig performance wherever possible. That means a derpy Windoze box or two, and that brought on a whole hellscape of dealing with resource conflicts the likes of which I hadn't seen in 20 years.

But no, this isn't about that. This is about TP-Link. I was pointed at this ecosystem as a possible escape from the clowntown that is Ubiquiti, so that's what I bought this time around: one of their gateway boxes (calling it a router would be too kind), a switch, and a hardware controller for local control - none of that cloud crap here, thanks.

It's been a new bit of stupid every week with this stuff. First of all, the switch is really best suited for a closet at a business, not anywhere in someone's home. It has dinky little fans that run pretty hard all the time, with all the noise that entails. People who replace them invariably get fan errors and then the thing eats itself within a year. (Maybe the switches fail by themselves either way - the jury is still out on that.)

The latest build of their controller software flat out does not work on Safari. I mean, sure, it loads up, and then the browser starts doing something indicative of a horrible big-O blowup factor somewhere in their Javascript. It'll hang for a minute at a time any time you move the pointer around. Or, it'll prompt you to download a file CONSTANTLY. Like, WTF kind of content-type brain damage are you doing? It doesn't happen in Firefox or Chrome, apparently, but it still goes to show that they gave zero fucks about even TRYING TO LOGIN in Safari when they were developing it. You know, the browser that every Mac and iOS device ships with?

So, you have to roll back the controller to get out of this mess. Doing that wipes your config. Fortunately for me, I discovered this during my early shakedown testing at my own residence before hauling it out to the site, and there was no actual config to lose.

Next up, their NAT implementation is just plain obnoxious. Typically with this kind of stuff, if you fire a packet from the inside to the outside, the source address gets changed from RFC 1918 or whatever you're using internally to whatever you have on the outside. That much works. What also happens here on the TP-Link ecosystem is that they mangle your source port, too. This affects both UDP and TCP.

Why does this matter? It makes NAT hole-chopping tricks much harder to pull off. Normally, you can do such fun things as configuring WireGuard to punch through from either side by lining up the ports exactly. This will let two sites connect to each other without going through a third fixed spot. This is very handy if that third spot goes down and you need an OMFG backdoor into your networks!

This does not work if the source ports change. At that point, you have to resort to all kinds of nasty birthday paradox type stuff to figure it out, and that requires Actual Work to pull it off and keep it working. Me, I don't want to put TailScale everywhere. But I digress.

Last week, something very bad happened that I haven't managed to troubleshoot since I'm remote and can only do limited things from here. HomeKit stuff stopped working. By that, I mean that viewing the home from off the local wifi said the usual "no hubs online" thing. But, stranger still, HomeKit *clients* on that wifi also couldn't connect *outward* to other spots! They, too, got the same thing about no hubs about other HomeKit locations... even when those locations were actually fine and worked for other people.

The only commonality was crossing that Omada-powered network. I had some luck in this case since there's a Mac out there which I can hop into and beat into submission, and beat I did. I figured maybe it was something goofy about the routing to Apple's cloud stuff, and started shunting all of the traffic through a tunnel. Nothing helped ... until I also switched DNS resolution on that Mac to something I controlled instead of using whatever resolver is inside the TP-Link gateway box.

Once I did that, it started working again. Even after I turned off the tunneling, it kept going. This was enough for me. I stood up unbound on a couple of Raspberry Pis out there and changed the DHCP config to make sure clients would resolve things through them instead of the ER8411 gateway. It took a while, but eventually, everything stopped being stupid.

Now, big caveat here: I don't know 100% that it was the resolver in the thing. I wasn't on site, and could only do so much without kicking myself out of the network, since my access came in through those very devices. Also, my troubleshooting abilities are limited with this crap for yet another reason I'll get to later.

Then there's what happened this morning. One of my Pis behind this setup decided it wasn't going to run one of its WireGuard links. The other link on the same interface (going to another external host) was fine. The other link on the other interface was fine. The other Pi's two links were also fine.

It was just this one particular association that wasn't working. So, into tcpdump I went yet again, looking at it from both sides of the link. The exchange I saw from inside looked like this over and over:

their_internal_ip.AAAAA -> my_external_box.BBBBB: udp data
(no reply)

But, from the outside world, it looked like this:

their_external_ip.CCCCC -> my_external_box.BBBBB: udp data
my_external_box.BBBBB -> their_external_ip.CCCCC: udp data
their_external_ip -> my_external_box: ICMP port CCCCC unreachable

So yeah, even though it had JUST sent traffic to me from that port, upon reply, the gateway box was rejecting it. This to me says "really terrible IP connection tracking setting and/or implementation that dropped the association and is somehow not picking it back up".

This WG link has a keepalive on both ends. There's no excuse for this. It should be established in the firewall as soon as a packet goes out, as one did above. But the ICMP error indicates otherwise.

Note that the port-unreachable error is not coming from the Pi itself. The Pi was only sending actual traffic and had no idea why it wasn't getting any responses. WG won't switch source ports by itself, so it just keeps smacking its head into the wall ... over and over and over.

And this brings me to the final point of frustration: I wanted to ssh into the damn gateway to to see what they were doing to screw things up so badly. It took a while to find the knob to enable ssh, and once that was on, I found the ultimate insult: it's a completely neutered interface. You can't do anything useful. It's busybox, the "ip" utility, and something that apparently lets you point it at a controller for when the adoption process doesn't work.

su? sudo? Forget about it. You don't even have /proc in there - so no ps, no w. You can't run dmesg because it doesn't exist (and they probably lock down the kernel ring buffer anyway). You are a luser, and you will never be able to do anything useful from this setup.

When pressed, tech support tells people that such things are unsupported when using the controller - that is, the dumb pointy clicky web-based UI that takes the setup and pushes it out to the devices. You know, the one that broke on Safari in the latest version. They're locking you out _on purpose_.

Finally, I haven't run into this one yet since the ISP for this site is still in the dark ages in terms of providing access to the ENTIRE Internet, but it sounds like they don't do any sort of IPv6 firewalling. So, if your ISP switches that on and you suddenly get an allocation, look out world! It's the wild west on your network!

So, let's recap the suckiness here.

0. The switch is stupidly noisy.

1. Their latest version of the controller just does not work in Safari.

2. You can't easily roll back the controller when it does suck. You'd better save the config from the old version before you upgrade, just in case you ever have to go back. And, if you never ran that particular old version, you're doubly screwed.

3. Their NAT implementation mangles source ports needlessly. Sure, some scenarios call for it. They do it constantly.

4. *Something* broke HomeKit comms really badly, and switching recursive DNS services for clients away from whatever the gateway box provides fixed it. It's probably some terrible DNS forwarder implementation but I have no way to be sure at this point.

5. The NAT apparently dropped an assoc this morning and never put it back. I couldn't get my tunnel going until I restarted it to pick a new source port on the client. Completely ridiculous.

6. Forget about ssh to troubleshoot things. The hood is welded shut. You will never know what's really going on when one of the other items decides to rear up and bite you in a sensitive place.

7. They apparently have no IPv6 firewalling based on what other people have reported in various places. (This is the only one I haven't actually encountered myself... yet.)

So now what? I'm honestly looking at returning to what I was doing in the 90s: building my own Linux boxes with enough horsepower to handle the networks in question. It worked then and it'll work again now. Things will still break, but at least I'll be able to use my actual experience to do something useful about it. Right now, I can do nothing. My hands are tied.

Why did I think these clowns had any idea what to do? I've been both inside and outside of this world, and it's pretty clear that they do not. Just look at how awful these products really are.

Thanks to a bit of anonymous feedback this morning, I have some good news about the Raspberry Pi WPA3 thing. Apparently the good folks over at the Asahi Linux project have taken up the cause of fixing the upstream kernel situation. It seems this will happen by throwing out some existing implementation that didn't work anyway, and perhaps my posts were confirmation that they were in fact crap.

This is great! Hopefully this will lower the barriers for regular people who just want things to work and don't want to patch kernels and drivers and maintain their own forks of things.

So, yay for that person cleaning up the mess of some dumb big companies. Thanks for pushing on this and not letting the suck stop you.

...

For the record, I've tried this on a 3B, 3B+, 4B, and now a 5B. Did I buy the 5 expecting the wifi to suck? I sure did. Did I expect to write a post bagging on it? You know it. I did my research to see if anyone else had mentioned it, and when that came up empty, I hit the store and picked one up, then came home, tested it, and wrote the post.

While waiting for this fix to come down the pipe to be usable on your systems, there are any number of alternatives. Unfortunately, they all amount to a barnacle that consumes one of your USB ports, but they do work. They tend to actually behave better with tools like Kismet, they do WPA3, and they don't make the kernel panic when you look at them funny!

If this is you, hit the USB-Wifi main menu and start digging around. Note in particular the "plug and play" list. I grabbed some weird $40 Alfa thing I had never heard of before based on a recommendation from the list. It worked great for sniffing things and generally screwing around. It also ran WPA3 as a client just fine.

This is the Linux experience I remember from the 90s: poring over compatibility lists and making sure you buy the right thing every time. That's why it's so vexing that the Pi people would keep shipping this thing in this state. You don't want your customers to keep buying these wifi barnacles, do you?

January 24, 2024: This post has an update.

About a year ago, I wrote about trying to make WPA3 (wireless security) work on a Raspberry Pi 4, among other things. It didn't work then and it still doesn't work now.

In the past couple of weeks, they released the Pi 5, and it's been making the rounds through the usual people, but somehow, nobody's talking about whether it'll do WPA3 or not. So, I'll break the silence and save everyone a lot of work: it still has the same CYW43455 wifi+bluetooth chip as the Pi 4, so it has the same limitations: no WPA3 support, at least, not right now. Maybe some day, someone will do something about the driver situation, but given that nothing has changed in almost a year since my last post, I'm not going to hold my breath.

This Broadcom / Infineon / Cypress wifi situation has other ramifications beyond just the Raspberry Pi ecosystem. Let's say you are like me and you bought one of the "early 2015" Macbook Pros in 2017 since the then-current ones had the terrible new keyboard and still hadn't figured out the whole USB-C thing yet. You didn't get Ventura (13) and are stuck on Monterey (12), never mind Sonoma (14).

So, maybe you thought "I know, I'll install Linux on this thing since it still has some life left in it". Once you do that, you will discover that you are in the same crappy wifi situation. You won't be able to join a WPA3 network in Linux, either. The hardware is clearly able to support it since it worked as a macOS install, but it's just not going to happen on Linux. It's the same sort of software problem.

None of this is news, but the updates about this mess seem to not get enough visibility. Note: "seriously unmaintained and years behind on features and firmware integration". Wonderful.

This puts the Raspberry Pi squarely in the same bucket as "random Internet of Shit devices" when it comes to wifi compatibility. You are going to have to keep a terrible auxiliary wireless network around for a very long time in order to support them as-is.

Going forward, you can just ignore the built-in hardware and buy a dumb little USB wifi adapter which is actually supported by regular kernels. Then you just plug it in, configure things, and go on with your life. (Or, you know, pull actual cabling to it, and live life with the stability of hardwired Ethernet.)

It's kind of amazing that this situation persists, given how the Pi is supposedly intended for entry-level people who need a low-cost platform to learn about stuff. What happens when they encounter a WPA3-only network? They do exist, and they will only become more numerous over time.

November 7, 2023: This post has an update.

There are more than a few bear traps in the larger Unix environment that continue to catch people off-guard. One of the perennial favorites is thread safety, particularly as it applies to the environment manipulation functions under glibc. The usual warning is that if you run multiple threads, you'd best not call setenv, because if someone else calls getenv, you run a decent chance of segfaulting.

The last time I talked about this was 2017 and I said something goofy like "see you in 2022" at the bottom. Well, it's a little late, but it's come up again, and this time it's biting people who aren't even using C or C++!

This one is in Go!

Last time, I said that sometimes people run afoul of this by using mktime() which calls getenv()... and then sometimes pick *that* up by using libzip. Here, it's a little different. getaddrinfo() calls getenv(). Did you know that? Before a few minutes ago, I sure didn't! Check out the resolv.conf (5) man page - it looks at LOCALDOMAIN and RES_OPTIONS.

getaddrinfo() is pretty much required if you want to connect to anything beyond mere legacy IPv4 gunk, so it's not like you can avoid it. You're probably going to call it quite often if you are opening connections over the IPv6 Internet.

If you're on Linux, and you're using glibc, you're probably a passenger on this boat. Try not to drill any more holes.

Thanks to Evan for the tip on this one (and good luck with the fix).

Back in March, I wrote about how the HTML generation worked for all of these posts. In short, it was mostly "loose approximations of HTML via printf", and it was terrible. It generally worked for the past 12 years, but I knew how wrong it was.

One thing I didn't mention in that post was just how bad the feed generation had become. I was doing a CDATA thing and was just spewing out the HTML inside of that. In theory, if I had put a "]]>" in a post, it probably would have broken the entire feed.

Then there are the finer points of the metadata for the feed. It was using an id of "tag:rachelbythebay.com,writing-2011" which I could have sworn was fine at the time I picked it, but which turns out to actually be illegal for that scheme. I fixed that for both the feed and the entries themselves, so if you see duplicates of the last 100 posts, or the entire feed shows up somewhere else, that might be why.

Those and many more things caused the w3c Atom validator to scream *quite* loudly about it being broken. A lot of people sent me feedback about this over the past few months.

That's all gone. A few minutes ago, I threw the last switch to finally cut over the entirety of the /w/ files to the new stuff. This meant that every single index.html has been regenerated. Quite a few corrections have been applied at the same time. It took me a very long time to go through all of these posts and convert my raw HTML shenanigans into meaningful commands that will be parsed by the generator.

Every view of things now actually makes sense. I can now write <foo> as an example in a post and it will come out escaped properly on the output side. I can put an & in the post without having to literally type in &amp. Yes, I'd been having to manually do &lt; and &gt; and all of this... if I remembered. If not, well, there'd be a "live" tag hanging out in the post!

Or, there'd be a broken tag. Last week's post about ASCII protocol buffers and config files actually had a "<pre" without a ">" in it, and then it just went into the contents. I bet you didn't see "syntax = proto2" in that thing as a result, but trust me, it was there!

There are a bunch of other little stylistic changes in here. The feed icon in the banner of both the top index and the individual posts no longer links to the feed itself, but rather a page that explains what to do next. This is a cheesy way to sidestep the "someone clicked on the feed in their browser and used up their unconditional request" thing for a little bit.

Anyone who was using a phone or other smaller device probably noticed that if you loaded a sufficiently old post, it wouldn't quite fit the screen properly whereas newer ones would. This is because I had added a little "viewport" magic in the headers at some point, but had only applied it to the template file (!), and never rebuilt all of the old posts. This also meant the post footers were all slightly different, depending on when it was last rebuilt. Now all of the posts have that viewport thing.

I also adjusted the line-height based on some feedback from at least one reader who commented that it would make it easier to read. I find it hard to argue with that, and didn't see anything bad about it, so that's in there too.

You should also notice that preformatted blocks now sport a different background color and a border to set them apart from the rest of the post.

There are probably some other things I've forgotten, too.

I'm sure there are going to be some anomalies, so if you see something that seems broken to you, go ahead and fire off some feedback. I do appreciate it.

While I don't go on the Orange Site any more, I still make enough trips through the larger space of similar sites to get some idea of what people are talking about. Last week, the topic of interest seemed to be YAML and how evil it is. I can't argue with that. Every time I've crossed paths with it, I've been irritated by both it and whoever decided to use it for their stuff.

The discussions invariably start talking about alternatives, and frequently end up on JSON. This is unfortunate.

I've mentioned this before in passing, but have never given it a whole post. Today, it graduates to having a whole post about the topic.

The topic is: ASCII-form protocol buffers used as config files.

This was a tip given to me something like 17 years ago when I was "on the inside", and it's turned out very well. Protocol buffers have a canonical ASCII representation, and it accepts comments, too! You get the benefits of not having to write a scanner or lexer combined with a system in which everything is explicitly specified, right down to the data types.

Here's an example of such a file:

# **** Contains auth data: must be thermo:thermo 0660 or better ****

db_conninfo: "host=localhost dbname=foo user=xyz_role password= ...

# barn (hardwired)
server_info {
  host: "172.25.161.10"
  port: "18099"
}

# barn (backup wireless on IoS network)
server_info {
  host: "172.25.225.10"
  port: "18099"
}

# office (broken 20230825)
# server_info {
#   host: "172.25.161.17"
#   port: "18099"
# }

sensor_location {
  name: "loft"
  model: "Acurite-Tower"
  id: "1563"
  channel: "A"
}

sensor_location {
  name: "entry"
  model: "Acurite-Tower"
  id: "2375"
  channel: "B"
}

There. That's not terrible, right? It has a bunch of common stuff that gets repeated as needed for my different servers and sensors. There's also a string that gets handed to Postgres to connect to the database. And yes, notice the comments everywhere.

Over in protobuf-land, this is what the .proto file looks like for that config format:

syntax = "proto2";

package thermo;

message LoggerConfig {
  message ServerInfo {
    required string host = 1;         // 192.168.31.67
    required string port = 2;         // 18099
  }

  message SensorLocation {
    required string name = 1;             // room
    required string model = 2;            // Acurite-Tower
    required string id = 3;               // 1015
    required string channel = 4;          // C
  }

  required string db_conninfo = 1;

  repeated ServerInfo server_info = 2;
  repeated SensorLocation sensor_location = 3;

There's one important bit here: I'm using "required" here since this is a config file format and NOT something that will be passed around over the network. It lets me cheat on the field presence checks, and this is the one case where it's acceptable to me.

If you're using protobuf for anything that gets handed around to something else (RPC, files that get written by the program, ...), whether across space *or time* (i.e., future instances of yourself), use optional and explicitly test for the presence of fields you need in your own code. You have been warned.

How does the program use it? First, it reads the entire config file into a single string. Then it creates a LoggerConfig (the outermost message) and tells the TextFormat flavor of protobuf to ParseFromString into that new message. If that returns true, then we're in business.

I can now do things like hand config.db_conninfo() to Postgres or iterate over config.server_info() or config.sensor_location() to figure out who to talk to and what sensors to care about.

Is it perfect? Definitely not. It's software, which means it will never truly stop sucking, like all other software. It's a dependency that will now follow you, your code, and your binaries around like an albatross. It's yet another shared library that has to be installed wherever you want to run.

But, hey, if you're already paying the price of using protobuf in your projects for some other reason, then why not use it for config storage, too?

In short: I should have looked at my notes instead of relying purely on my memory of a random event from four years ago.

Right, so, the other day I wrote a post eviscerating the Philips (Signify) Hue situation, in which they are heading full steam into enshittification. I said that I didn't want to use Home Assistant becuase of Javascript and a "curl | sh" attitude.

Yeah, that's where I screwed up. That's not them. That's actually Homebridge, aka homebridge.io.

People have been commenting that HA is Python, not JS, or something like that. I had to go back to my notes from January 2019 to set this straight, and, well, they go like this:

I was trying to find a way to run some existing "real" security cameras without going full-on cloud mayhem. The users in question wanted to see it in the HomeKit ecosystem, so I went looking for solutions of that sort. The idea was to get this thing running, document the protocol, and then figure out if I could do it some other way.

So I end up on their wiki for installing this on a Raspberry Pi. At that point, I had one just sitting around collecting dust, and figured "what's the harm in trying". First thing up, they wanted me to "curl some-url | sudo bash".

Hell no, I'm not doing that. There are so many things wrong with that philosophy. The whole point of cutting actual releases is that you get people to cluster around a handful of known "artifacts" (you know, tarballs and the like), and then you can work up some kind of reputation based on that. If that actual release ends up in some distribution like Debian, you can be sure that exact version is being seen by a fair number of people.

curl | sh basically says "I don't give a damn" and "give me whatever you want" at the same time.

I figured I could at least grab the script, read it, and parse it myself, then run the commands by hand. This meant I had to add their apt repository. Ugh. First up? "apt-get install -y nodejs" ... oh boy.

But wait, no, then it went on from there.

I tried "npm install -g homebridge" but that wasn't happening. It wouldn't go until I added "--unsafe-perm". Oh, gee, that's not sketchy *at all*.

At this point I was glad this was all happening on what was effectively a throwaway machine. It did get installed, and started up, then displayed a QR code, and once past a few warnings, it did in fact show up in Homekit.

Of course, without plugins, it wouldn't do anything, so that meant going back for MORE node stuff. I found 72 pages of them on npmjs.com.

I picked something really stupid to export a temperature value as a test. I put a config stanza in config.json to try to activate it, and nope, it started dying. So I took the config-sample.json and copied that in its place, and that much worked, and told me what I needed to do: the blob of JSON crap from the plugin's page is supposed to be an entry in the accessories array of the config.json. That was not at all obvious.

I didn't want to do much more with this, and that's about where it ended.

So, yeah, bagging on Home Assistant for being JS and making you install it with sketchy curl pipelines? That was a mistake. I got it mixed up in my head with Homebridge based on something I did four years ago.

I loaded up the HA page on github, and oh hey, Python.

Mmm, yeah. Righto. Okay then.

How about them Knicks?

If you've gotten into the home automation thing in the past few years, it's possible you set up some Philips Hue devices along the way. This was an ecosystem which had a bunch of bulbs, switches, outlets and a hub that spoke Zigbee on one side and Ethernet on the other. It was pretty much no-nonsense, never dropped commands, and just sat there and worked. Also, it integrated with the Apple Homekit ecosystem perfectly.

Unfortunately, the idiot C-suite phenomenon has happened here too, and they have been slowly walking down the road to full-on enshittification. I figured something was up a few years ago when their iOS app would block entry until you pushed an upgrade to the hub box. That kind of behavior would never fly with any product team that gives a damn about their users - want to control something, so you start up the app? Forget it, we are making you placate us first! How is that user-focused, you ask? It isn't.

Their latest round of stupidity pops up a new EULA and forces you to take it or, again, you can't access your stuff. But that's just more unenforceable garbage, so who cares, right? Well, it's getting worse.

It seems they are planning on dropping an update which will force you to log in. Yep, no longer will your stuff Just Work across the local network. Now it will have yet another garbage "cloud" "integration" involved, and they certainly will find a way to make things suck even worse for you.

If you ever saw the South Park episode where they try to get the cable company to do something on their behalf and the cable company people just touch themselves inappropriately upon hearing the lamentations of their customers, well, I suspect that's what's going on here. The management of these places are fundamentally sadists, and they are going to auger all of these things into the ground to make their short-term money before flying the coop for the next big thing they can destroy.

What can you do about it? Before you say "Home AssistantHomebridge", let me stop you right there. Javascript plus a "curl | sudo sh" attitude to life equals "yeah no, I am never touching this thing".

Instead, I have a simpler workaround, assuming you just have lights and "smart outlets" in your life. Get a hold of an Ikea Dirigera hub. Then delete the units from the Hue Hub and add them to the Ikea side of things. It'll run them just fine, and will also export them to HomeKit so that much will keep working as well.

I will warn you that Ikea isn't perfect here, either. They won't plumb through the Hue light/motion/temp sensors or the remote controllers to HomeKit. This means you lose any motion sensor data, the light level, and the temperature of that room. You also lose the ability to do custom behaviors with those buttons, like having one turn something on and then automatically switch it off a few minutes later. (Don't laugh - this is perfect for making kitchen appliances less sketchy when unattended.)

Also, there's no guarantee that Ikea won't hop on the train to sketchville and start screwing over their users as well.

My hope is that someone with good taste and some sensibility in terms of their technology choices will make something that does Zigbee on one side, Homekit on the other, and is at least as flexible as the Hue setup that existed originally. Until then, it's going to be yet another shit show.

And people wonder why I don't trust these things.

October 3, 2023: This post has an update.

I'm a lazy programmer sometimes. Let me tell you a story about something I wrote earlier this year that's not exactly the finest set of programs ever produced. It's about the whole feed rate limiting thing on my web server.

I've been getting reports of people who run into the block even when they didn't do anything wrong. They didn't start up something that polled every two seconds and pulled the full ~500K feed every single time, for example (and yes, this has happened at least once).

No, the problem goes like this - someone sees the little orange feed icon up there (on the web view, that is) and clicks on it and gets a screenful of XML. My server also goes "okay, you just got the feed". Then they take the URL, hand it over to their feed reader, and it reaches out and tries to make the same request. My server says "hey wait a minute you clown, you JUST GOT IT", and rejects it with a 429.

See the problem? It can't tell the difference between a pairing of a one-time human request + their feed reader's startup sequence and someone who's actively hammering the thing. It's because the thing is relatively stupid. It knows about IP addresses, request types (conditional or not), and elapsed times. That's it.

In order to support some kind of "you're going to make a handful of closely-spaced unconditional requests at startup but will be good thereafter" leniency, it would have to actually have some thought put into it. Now you're talking about more of a "token bucket" system, or something else of that sort where it does some time-based accounting and allows for "bursty" behavior at first. That means tracking a lot more than just "the last time you got a full copy of the feed".

But you know what? That's work. It's not fun, it's not interesting, and it doesn't do me any favors besides avoiding receiving feedback messages from confused users of feed readers. So, I've been lazy, and I haven't done it. I've instead done a bunch of other things which also had to be done and had slightly better contexts.

I'll admit something else: I don't have a ready solution to this. I've never written a burst-handling inflow system before. It would be different if I could just reach back into my head and go "oh yeah, this is just one of those things from XYZ project". But nope, this time there's nothing in the past to "borrow" in the present.

Also, this feels more like a "moving average" type of problem, which then means *actual math*, and that's just not my bag, normally. So, I find reasons to do something else. Repeat as necessary.

Again, most feed readers and their users are doing just fine. This is something I have to do in order to deal with the pathological cases who are small in number but large in impact. I suspect that just a handful of them take up way more resources than all of the normal, happy, good people put together.

As with so many technologies, they would all be unnecessary if not for the people who are causing the problems. It's why some of us get wistful for the "old days" when the net was far smaller and the amount of bad behavior was accordingly tiny - eternal September and all that.

Do I want to write rate-limiters? Hell no. I'd rather do anything else.

One problem with working in a customer support environment is that you tend to lose track of just how many tasks you've completed. After a few hours, most of them get pretty fuzzy, and by the end of the week, only the most notable ones stand out. A month later, it's even worse than that. This is just how it goes when there's so much quantity passing by.

This is why I tried to take notes about a handful of them as they happened. After a certain point, the memories start losing "cohesion" (whatever) and then it might as well be "fiction inspired by real life".

A fair number of my posts are sourced from these notes. It's how I can still give some details all these years later without making them up.

Here's something that came in one night almost 20 years ago while working web hosting tech support.

A customer wrote in. They opened an "emergency: emergency" ticket, which is usually reserved for "OMFG my server is down please fix0r" type events. It actually had a HTML blink tag baked into the very string so it would blink in our browsers. It was hard to miss.

It was a Monday night. What they said, more or less: "Three things. I have a Tuesday deadline. I'm stuck in (some airport) because of the weather problems from Hurricane Jeanne in Atlanta. I can't connect to port 22 because the wireless in the airport seems to firewall it off."

"So, if not for that, I wouldn't call this 'emergency'. Also, I can't get to webmin to add another port myself. So, can you open up another sshd on port NNNN (since I know that gets through) so I can get to the machine?"

They ended this with a "Thank you" with a bunch of exclamation points and even a 1. (Whether they were trying to be KIBO or B1FF, I may never know.)

They opened this ticket at 7:40. About five minutes later, one of our frontline responders saw it in the queue (probably noticed the *blinking*), mentioned it out loud, and after a short discussion assigned it to one of the people on the floor.

At 7:50, we responded, stating that some iptables magic (shown in the ticket) had been done to let sshd answer on port NNNN in addition to port 22. Also, there was a note added to clarify that this was made persistent, such that it would persist across reboots. The customer was then asked to try connecting and to let us know if that didn't work out.

Why iptables instead of a second ssh daemon? It was way faster, for one thing, and time was of the essence. You could run the two commands: one to add the rule, and one to make it persistent, and then the customer is good to go. Standing up a second sshd instance on a separate port back in those days would have meant wrangling init scripts to make a second version that points at a slightly different config file. Also, it would create something of a maintenance issue down the road as that forked config would quickly become forgotten.

Sure, someone could have run "sshd -p NNNN", but then they'd have to make sure it kept running, and if it got whacked somehow (reboot?), the customer would be screwed again with their deadline looming.

Also, in terms of cleanup, the customer could just flip the -I (insert) in the iptables command to -D (delete) and save it to make it disappear for good later. Tidying the second-sshd thing would have more fiddly.

In any case, the customer came back a few minutes later, thanked us for the work, and promised to clean it up when they were clear of the problem. We didn't hear back, so things apparently worked out.

I hope they made their deadline.

I had this really bizarre telephone for some years in the 90s and 2000s. While mine is long gone now, I figured I'd talk about a little to establish that yes, this thing did exist, and to also hopefully inspire some Youtube types to find one and dissect it in a video.

It was called the FV 1000, dubbed a "Freedom Phone" model by Southwestern Bell, and it was a giant plastic piece of awful. It had certainly *sounded* cool when it was described in that electronics clearance catalog (Damark, maybe), but actually using the thing was another story entirely.

You see, it was supposed to be a "voice phone" ... as in voice-activated. While I got mine somewhere around 1990, I've been able to find evidence of it existing as far back as December 1987. So, imagine how mind-blowing that was back then: "wow! dialing the phone with my voice! In the 80s!".

Yeah well, Siri it was not.

Here's how it worked. It had no number buttons and no dial (you know, the spinny bit on a rotary phone). On the front, it just had some cursor keys (left, right, down, up) and a "store" button. The actual handset was unexpectedly lightweight and had a button at the top behind the earpiece. (I think there was also a reset button under a flip-up door, for what it's worth.)

Oh, and even more confusingly, you didn't get a dial tone when you picked it up. Picking it up off the very flimsy "hookswitch" presented you with a locally-generated tone that meant "okay, I'm waiting for you to talk to me now".

What you had to do was push the button down and say a command word like "DIAL", then wait for it to do the "ke-bwoop" confirmation noise and show "DIAL" on the single-line display. Then you'd read out the numbers one by one, waiting after each one for it to confirm. "1" *wait* "2" *wait* "0" *wait* "2" *wait* "4" *wait* "5" *wait* "6" ... you get the idea.

Then at the end, I think you just released the button and it would then "execute" the "command" you had just painstakingly built one word at a time. You'd hear the dial tone at last, and it would actually dial the number, and then it would connect things through and in theory you could talk like normal.

If this description is making you think "this thing sounds really slow", you'd be right. So, okay, naturally it had some memory features, right? Of course it did. You could store things like "Home 1" or "Neighbor 1" or "Office 1". Oh, and by the way, those names were immutable. You couldn't call it "Mrs. Brown" or "Mr. Chilman". It was "Neighbor 1" and "Neighbor 2" for you. Hope you remembered who was who! Enjoy flipping up that little door to see the labels you hand-wrote!

They did let you adjust the on-screen display for any given memory location, so while it might need to be TOLD "Neighbor 1", you could make it *display* "Mrs. Brown" or whatever... if it would fit.

It didn't always hear you correctly. When that happened, you had to say "BACKSPACE" and wait for it to acknowledge with another *ke-bwoop* noise. If you wanted to cancel, you had to be careful how you went about it, since letting go of the button would make it execute whatever you had told it so far.

In a stunning preview of today's event-driven half-assed GUI programs, you could actually get it out of sync with the "on hook" / "off hook" state that it maintained locally. I mentioned that it made that "I'm ready" tone when it's off the hook, right? Well, if you jostled it with just the right timing, you could manage to get it to be very much on the hook and yet still making the damn noise from the earpiece.

It wasn't super loud, but late at night when everything else was quiet, you could hear that tone coming from the phone as it was just waiting for you to push the button and give it some commands. The "fix" was to jostle it some more until it realized that you were not in fact trying to make it do something on your behalf.

One side note: for those thinking "this must have been amazing for people who can hear but can't see"... probably not so much. It didn't read back the numbers you told it to dial. So, if it mis-heard one digit as another, you'd never know. It only showed it on the display, so if you couldn't see it... oops.

Apparently the list price for this thing at the end of 1987 was $450... or about $1200 today. Imagine spending that much cash on some tech and then realizing it was annoying, flimsy, and generally unreliable.

Oh, wait, I guess we all do that pretty much constantly now. Never mind.

I've built more than a few projects which use protocol buffers somewhere in them to store data or otherwise schlep it around - in files, over the network, and that kind of thing. A friend heard about this and wanted to write an implementation in another language and so I supplied the details. Everything seemed to be going fine, but then we started getting *really weird* errors when he tried to point his new client at my server process.

Just trying to get the outermost "envelope" thing to pass would fail. This made no sense. We finally had to get down to individual bytes from the network dump to try to sort it out. Then we tried to encode "the same thing" and got two different results. His end was generating "1f 0a 0b (string)" and mine was doing "0a 0b (string)".

Where was this extra 1f coming from? We started trying to unravel it according to the rules of protobuf: the tag of a record is a varint which comes from the field number and wire type and blah blah blah... and I won't even bother with the details here since that was also a dead end. It decoded to "field 3, type 7" but there isn't a type 7. There are just 0-5. So, again, WTF? What is this "invalid wire type 7" thing? (And yes, that string in this post is entirely deliberate.)

My friend is good at this sort of thing, and so started digging in deeper... and it started looking like a length byte. It's like, wait, what? Hold on. protobufs do not work that way! They don't have their own framing. That's why recordio was invented, and countless other ways to bundle them up so you know what type they are, how long they are, and all of that other stuff. The actual binary encoding of the protobuf itself is bare bones! So what's up with this length byte?

So then we started looking at this protobuf library he had selected, and sure enough, the author decided it was a good idea to prepend the message with the message length encoded as a varint.

WHY? Oh, why?!

And yes, it turns out that other people have noticed this anomaly. It's screwed up encoding and decoding in their projects, unsurprisingly. We found a (still-open) bug report from 2018, among others. They all manifest slightly differently, so not everyone realizes that it's all from the same root cause.

The fix was dubious, but it did work: you skip the "helper" function that's breaking things. That gives you just the proper bytes, and then everything is happy.

That's how I got both a "second source" for speaking my goofy RPC language and another story about wacky broken libraries at the same time.

Earlier, someone wrote in saying "the RSS feed seems to be broken", but didn't leave any contact info. All I have is an IP address and whatever the web server logged. Let's just see what we got here, and see what's broken and what isn't.

xx:01:29 : requests feed over http with no conditional headers (If-Modified-Since, If-None-Match). Receives entire feed as a result. Expected behavior for first fetch.

xx:03:53 : requests feed over http again... with no conditional headers. It's been a hair over two minutes. The server rejects it with a 429 "Too Many Requests". Strong signal for a broken feed reader.

xx:20:37 : third request over http with no conditional headers (so it wants the unchanged feed again, 19 minutes later). Rejected the same way.

Then things pivot and hit the "secure" side of the site. I treat them differently for the purposes of throttling, or very bad things would happen to feed aggregator places which subscribe to both versions of the feed.

xx:21:46 : requests feed over https. Sends no conditional headers. Receives entire feed since it's specifically rigged to not care about the earlier http traffic.

xx:22:04 : requests feed over https again. Sends no conditional headers despite receiving entire feed 18 seconds earlier. Is rejected with a 429.

xx:22:11 : requests feed over https a third time, again with no conditional headers. Is rejected with a 429 again.

xx:22:18 : fourth request over https, again no conditional headers, is again thrown in the bit bucket with a 429.

xx:23:13 : fifth request over https, still no conditional headers. Gets 429.

I'd say things are working perfectly... here. There's a feed reader involved which doesn't send conditional requests, doesn't throttle on a 429, and doesn't surface HTTP failure codes to the user, but I have no control over that.

HTTP 429 means slow your roll.

I got some anonymous feedback a while back asking if I could do an article on how to become a systems engineer. I'm not entirely sure that I can, and part of that is the ambiguity in the request. To me, a "systems engineer" is a Real Engineer with actual certification and responsibilities to generally not be a clown. That's so far from the industry I work in that it's not even funny any more.

Seriously though, if you look up "systems engineering" on Wikipedia, it talks about "how to design, integrate and manage complex systems over their life cycles". That's definitely not my personal slice of the world. I don't think I've ever taken anything through a whole "life cycle", whatever that even means for software.

In the best case scenario, I suppose some of my software has gotten to where it's "feature complete" and has nothing obviously wrong with it. Then it just sits there and runs, and runs, and runs. Then, some day, I move on to some other gig, and maybe it keeps running. I've never had something go from "run for a long time" to "be shut down" while I was still around.

This is not to say that I haven't had long-lived stuff of mine get shut down. I certainly have. It's just that it's all tended to happen long enough after I left that it wasn't me managing that part of the "life cycle", so I heard about it second- or third-hand and much much later.

If anything, some things have lived far too long. My workstation at the web hosting support gig started its life with me in 2004 as a pile of parts that had formerly been a dedicated server. It had a bunch of dumb tools that I wrote and other people found useful. It should have been used to inspire the "real" programmers at that company to code up replacements, but seemingly did not. That abomination lived until *at least* 2011, or five years after I moved on from that company. None of that stuff was intended to run long-term, but someone kept tending it for years and years. It was awful.

But, okay, let's be charitable here. Maybe the feedback isn't asking for that exact definition, but rather something more like "how to get a job sort-of like the things I've done over the years". That's the kind of thing I definitely could take a whack at answering, assuming you like caveats.

I think it goes something like this: you start from the assumption that when you see something, you wonder why it is the way it is. Then maybe you observe it and maybe do a little research to figure out how it came to be the thing you see in front of you. This could go for just about anything: a telephone, a scale, a crusty old road surface, a forgotten grove of fruit trees, you name it. By research, I mean maybe you go poking around: try to open that scale with a screwdriver, get out of the car and walk down the old road, or turn over some of the dirt in the field to see if you can find any identifying marks.

I should also point out that this goes for trying to understand how people and groups of people came to be the way they are, too, but most tend to not respond well to being opened with screwdrivers, walked on, or turned over in the dirt. (And if they do, well, don't yuck their yum.)

Anyway, if you start from this spot, then maybe you start coming up with some hypotheses for how something happened, and then sort of mentally file that away for later. Or, maybe you even write it down. Then as more data comes down the pipe over the years, you revisit those thoughts and notes and refine them. Some notions are discarded (and noted as to why), but others are reinforced and evolved.

Do this for a while, and sooner or later you might have some working models. They might not necessarily be the actual explanation for why something is the way it is, but it gives you a starting point.

Then, one day, something breaks, and you end up getting involved. It might be a high-level system that's new to you, but it has some low-level stuff deep inside, and you recognize some of that. One of those low-level things had a history of doing a certain thing, and that never changed. They might've built a whole obscure system over top of it, but the fundamentals are still there, and they still break the same way. You go and look, and sure enough, some obscure thing has happened. Nobody else saw something like this before, and so when you point it out and flip it back to sanity to restore the rest of the system, they look at you like you just pulled off some deep magic.

The question is: did you, really? It's all relative. If you've been poking and prodding at things and have remembered the results of these experiments from over the years, it's not really new to you. It's just one of many events and might not be anything particularly special by itself. It just happened to be important on this occasion.

Some people will accept this explanation. Others will refuse it and will insist that you are a magician for fixing "the unfixable". A few others will know exactly what you did because they did it themselves once upon a time.

Then there are the one or two in every sufficiently large crowd who will see that you are being celebrated for knowing and utilizing some obscure factoid, and they will make it their mission to wreck your world. Basically, they have to make your random happenstance about them somehow, and so they make it about how it hurt them and how they need to get back at you. If this sounds pathological, it's because it is, and unfortunately you will encounter this at any company which doesn't have the ability to screen out the psychos.

This also goes for the web as a whole. Having something you've done be (temporarily!) elevated to a point of visibility somewhere public will just set these people off. This, too, is enabled by having forums which don't notice this and deal with their pests.

Now, for some examples of obscure knowledge that paid off, somehow.

pid = fork(); ... kill(pid, SIGKILL); ... but they didn't check for -1. "kill -9 -1" as root nukes everything on the box. This takes down the cat pictures for a couple of hours one morning because it turns out you need web servers to run a web site. Somehow, the bit in the kill(1) man page about "it indicates all processes except the kill process itself and init" stuck in my head. Also, the bit in the fork(2) man page that says "on failure, -1 is returned in the parent".

malloc(1213486160) is really malloc(0x48545450) is really malloc("HTTP"). I think this came from years of digging around in hex dumps and noticing that the letters in ASCII tend to bunch together (this is entirely deliberate). Seeing four of them in a row in the same range with nothing going over 0x7f suggested SOME WORD IN ALL CAPS. It was.

The fact I had seen some of this stuff before is just linked to some chance events in my life, combined with doing this kind of ridiculous work for a rather long time now. There are plenty of other times when something broke (or was generally flaky) and I had no idea what it could possibly be, and had to work up from first principles.

For someone who's just getting started, it's a given that you haven't seen many of these events yet. Don't feel too badly about it. If you keep doing it, you'll build up your own library of wacky things that could only be earned by slogging away at the job for years and years.

Also, if you think this is nuts and choose another path, I don't blame you. This *is* nuts, and it's entirely reasonable to seek something that doesn't require years of arcane experiences to somehow become effective.

Welcome to the new hosting situation. Over the past month or so, I've been working to move this web page and some of my other stuff to a new spot. As of this morning, it's done, and this is being served from the new machine. Say hello to flicker.rachelbythebay.com.

So, what happened? Well, a cute little company called SoftLayer turned into a massive monster called IBM. They still had acceptable rates and actually offered IPv6 (barely), but their corporate brain damage only got worse every passing year.

They had definite "left hand, right hand" moments, like when I went to turn up a new machine in February 2020 and they didn't offer a kickstart of RHEL 8. It's like, hello, you bought Red Hat six months before. RHEL 8 itself had been out for nearly a year at that point, and indeed, had made it to 8.1 by then. So, I had to do CentOS 8, and then they hosed us all royally that year. That's when I stuck more pins in my IBM voodoo doll and migrated to Rocky.

Then there was the day in January 2022 when I was doing some work on the machine and noticed that it needed a firmware update or something. I figured, okay, fine, I'll take the downtime and let their automatic doodad do exactly that. It's really late and nobody should care. I queued it up and powered it down (per their instructions).

I watched from the remote screen monitor as the automatic updater powered it up and got it to boot over the network into Windows (!) in order to run some nasty thing that popped up CMD windows and worse. I went off to do something else to distract myself. One hour turned into two, then into three, and support started saying "oh, it'll be a total of four hours". Great. The worst part was the complete lack of updates during this process. They just kept flailing.

I finally said "please just abort this and put my machine back up". I told them that it failed, and they should not attempt to troubleshoot their automation system on my machine. They should admit that it failed, put me back up, and leave me alone for a while until I can figure out what happens next. They finally got someone who was paying attention to do exactly this, and the machine went back up.

We scheduled it to happen the next night during another four-hour window. They started it, worked for about an hour, then called it and decided to go with a chassis swap. Yep, they pulled my drive out and jammed it into another box (and I was fine with this). Since I'm not a complete clown, it came back up by itself and figured everything out and kept going. How about that.

So, if you noticed multiple hours of the site being down on January 3rd, 4th and 5th of 2022, that's why!

What else with them? Their customer support is completely boneheaded sometimes. They had this "VPN" thing so you could tunnel into your "privatenet" which has the IPMI/remote KVM interface for your server(s). I would do that when doing a kernel upgrade in case I screwed up and needed to rescue things. I'd get that working *first* before doing the reboot just out of paranoia. I've yet to need it, but old habits die hard.

One day, it just stopped working. I filed a ticket asking them what I should be doing, since their documentation web page (and I provided the URL) said to use X, but X wasn't working. Is there a new hostname, or can you fix the thing?

They came back and said, oh, use this documentation web page.

It was the same page I had put in the request, unchanged.

Several days went by. Finally, I "thanked" them for "providing the same URL that I had provided them in the first place", and closed the ticket with a thumbs-down.

In the meantime, I had managed to find another way in by guessing how their hostname scheme worked, and got my work done and rebooted into the new kernel. They never really fixed the docs as far as I know, and they are probably still pointing people at a long-dead VPN endpoint.

But no, that wasn't it, either. The machine was physically in Texas. That particular hive of hate and villainy is talking about making ISPs restrict access to certain kinds of web pages. That's obviously about consumer-side stuff, but they could probably find ways to extend that to the *hosting* side of it, too. Also, screw them and feeding their tax base. I started looking for replacement options in other locales.

At this point, I noticed that all IBM would sell me was something that was much less box for much more money. I'm talking a slower processor, less memory, and all of that stuff, and the monthly bill would go up. Screw. That.

And then I got my final sign from the universe: they're "modernizing" and so DAL05 (my location) will be shutting down in April 2024. I didn't even notice this until I happened to be in their "portal" to do some unrelated work. Did they mail me? No. Did they call me? No. I just happened to notice it while in there one day.

Well, that's the last sign I needed, and I pulled the trigger on a colocation cabinet a few days later. That then started the whole crazy mess of getting a server, pulling together the network equipment, installing it *physically* (this was hard!), installing it *logically*, and then migrating everything.

Late Friday night into Saturday morning, I started flipping things over and kept an eye on them. A few minutes ago, I turned off the web server on the old machine. I figure if your DNS provider is crazy enough to clamp my 900 second TTL up to something over 12 hours, you deserve to talk to a brick wall of RSTs for a while.

So here we are. I now have a server I can physically lay hands on, albeit with a little driving involved. I got it used, and it's a real beast, but it does work. I'm also hearing from early testers that it's significantly faster for them. I thought it was just because I moved it about 40 milliseconds closer to me, but it just seems snappier for them, too. How about that?

I probably screwed up at least one thing with this migration like I did last time, so if you spot something amiss, please do holler. All of the URLs should still be working and all of that stuff. I already know the mtimes all reset, so a bunch of pages look new when they have the same content - that was unavoidable.

That's the story of one more bird in the flock.

I got a bit of feedback the other day from Nate asking if I had dot files. I certainly do. I assume what they meant is if I have particular customizations, and then if I would care to share them. I definitely have a bunch of particular changes, and as for sharing them, why not. It lets me get a bunch of shots in at things that have become annoying over the years, and that means it's perfect for stirring up the hornet nests with a Friday night post.

Starting on my daily driver box that runs Debian, then:

I have a .bashrc that has a bunch of dumb two- or three-letter aliases which amount to 'ssh (otherbox)'. For some reason that is lost to time, they all start with the letter m, and then the second letter sometimes reflects the name of the target system - "mm" takes me to my Mac Mini (which also runs Debian), for instance.

The stock PS1 bugged me a bit, so I mangled it down to this:

export PS1='\h:\w\$ '

... which turns into "hostname:/some/path$ ", in other words.

I think I've had a prompt like that on my personal machines basically forever - probably back to 1994 if not before. That's fine when I'm just running things as myself. If I run sudo, I get the stock setting which ends up looking like this:

root@hostname:/some/path#

... and that's fine, too. Making it look a bit different when rootly powers are in force is a good thing.

The next one is switching off another annoyance:

alias ls='/bin/ls -N'

The -N switch to ls says "print entry names without quoting" ... and it's the difference between having just the filename shown, spaces or no, and having it 'wrapped like this'. The way I see it, if you're printing quotes there, they'd better be part of the damn name. It reminds me of the time they started doing crazy UTF-8 "smart quotes" in their error messages and I didn't know it had changed. Cue me going "WTF is this gunk in this filename?" and thinking we had major corruption in the system somewhere.

I'd probably put up with the quoting if it didn't bump everything else out to the right another column. Two spaces between the time and the filename? Heresy!

The next two are filed under "everyone sucks at setting colors in Unix tools so stop adding it to everything". The first one is for sar:

export S_COLORS=never

... and the second one is something I found a little later which seems to be something that might work across multiple programs (assuming they've been patched to recognize it):

export NO_COLOR="eat flaming death you [elided]"

You can guess what the rest of it says. The actual value doesn't matter. Just having it set does the job. The value I put there is just to make me feel better every time I have to fight to get back to the perfectly working system I've had.

That's it for .bashrc. Next, I have .gitconfig which is mostly boring. There's a [user] section which has name= and email=, and those are set to about what you would expect.

I have pull.rebase set to true because that's always what I would use anyway when doing a pull, and it started whining at some point. So I put this in here to make it keep doing what I wanted. This is because I don't do branches and other goofiness and just want a nice simple continuous timeline for my commits.

I also have init.defaultBranch set to main because, eh, why not? I've designed enough systems based on the old broken naming schemes and don't need any more.

I have a .gdbinit. Why? Same old story: the default now sucks. It has one line:

set style_enabled off

It's amazing just how awful it is when it changes colors every time it hits a ( or " or whatever. How do people deal with that stuff? So bad. It's so nasty.

Next up, .nanorc, and this one is a three-ring circus. Basically, for the longest time, I didn't need one of these. Now, I add about one line on average every three or four years because - again - things keep changing for the worse.

Here's where things are now:

syntax "all" ".*"
color yellow "^$"
unset locking
set emptyline
set breaklonglines

The first two have been with me for quite a while now, and serve to disable syntax highlighting across the board. Again, not my thing.

Line three stops it from pooping out stupid ~ files everywhere. Not wanted, not needed, didn't ask for it, was forced upon me, had to murder it with a setting.

Lines four and five just put back behaviors that they dumped in 4.0: the blank line right below the status bar at the top, and the wordwrap that happens when you hit a certain column. I use that all the time, like, well, *right now* writing this post. It hard-wraps at 72, because OF COURSE it does.

Next is my .Xresources which provides a way to disable some obnoxious behavior in urxvt without having to recompile it. For the longest time, I'd chop it out and drop a custom binary into my bin directory. Then I realized it could be tamed without such mangling, and here we are:

URxvt.perl-ext:
URxvt.perl-ext-common:

This has the effect of making it so a double-click highlights the whole word, and a third click highlights the whole line *even if* someone's holding a LISP convention on that particular row of the terminal.

Then I have a .xsessionrc which needs to exist because I now log in through xdm, and the window manager (fluxbox) ends up inheriting *that* environment. Yep, it doesn't get a .bashrc type thing applied to it. (Not gonna lie - this took a while to figure out. Quick, which of .bashrc, .bash_profile, .profile et al get run for any given type of login you do to a box? Text mode, X *and* ssh all matter.) Anyway, that means I have to twiddle my PATH in there, or the commands that fluxbox runs for me won't find anything in those extra directories.

That is, I like my .fluxbox/menu entries to be short and sweet, like "term". That's a small stupid script in my bin directory. If that's not in my PATH then I'd have to spell out the whole /home/blahblah thing, and that's just idiotic.

Speaking of fluxbox, that has a dot directory, and a startup script in there to set a few things up properly.

xset b off
xset r rate 250 30
xset dpms 1900 2000 2100
xscreensaver &

Line one turns off the console beep - not that my machine has a PC squeaker any more, but I think some things try to be "helpful" by sending a beep into the system audio path. That can be really obnoxious, like when I'm deliberately holding down a key for whatever reason and get to the beginning of the line.

Line two is about getting that key-repeat going at a speed I like. If I end up on a machine where that's not fast enough, it becomes obvious pretty quickly, and I have to go adjust things. Not every situation allows for things like ^W to eat a word or ^U to eat the whole line, and so holding down backspace to change the wording of something is what I want.

Likewise, if I want to put a "-------------" divider somewhere, I don't want to wait for it to get going. It looks like that means "wait 250 ms before repeating, and then repeat at 30 Hz", but I had to look it up because it's been set like that for as long as I can recall.

Or maybe I want to hold down the cursor key to scroll something, or just move somewhere else on the line. Same thing.

Annoyingly, this seems to be set in the keyboard itself and not on anything local to the machine, so if I have to replug the keyboard for some reason, I have to run that again or it'll be stuck in stock molasses mode. This feels like a regression from the PS/2 days but I haven't bothered plugging in one of my old model Ms to verify this.

Line three just sets up the power-saver specifics on the monitor. Those don't usually matter too much since I have a hotkey that explicitly locks things and then forces it to go to sleep right away, and I push that when I'm done using this thing.

Line four, well, that's my dose of jwz, and that's what actually keeps the screen locked, as opposed to the legions of craptacular also-ran "lock" programs that always end up sucking and failing open. I can't imagine how many years in total my screens have been protected by xscreensaver in "lock" mode.

The rest of that file just starts my three Window Maker-era widgets and those aren't important or even interesting. There's a clock/calendar, the CPU load, and something to twiddle the system volume for when I have speakers or headphones connected.

That's about it. I don't use .plan or .project files any more since I haven't run fingerd for decades, and besides, my machines are all just me and nobody else, and so a local finger is also not a thing. (Oh, get your minds out of the gutter. It's the "ratting out to the cops" sense of "finger".)

Want to see the last time I used that stuff? Here's the file in my homedir archive from the last machine which had that running:

-r-------- 1 rkroll rkroll 34 Apr 28  1996 .plan

See, told you it's been decades. All I did was rip off a line that I had seen in someone else's file that was intended to sow confusion:

Segmentation fault (core dumped).

The idea is that you'd think that the far-end finger process crashed, or the far-end finger daemon, or maybe even *your local finger client*, and then you'd run around trying to figure it out. Then you'd eventually realize what was going on and shoot a nerf dart at whoever wasted your time.

Ah, the '90s.

I used to work at a place that had an internal task tracking system. Big deal, you think. Lots of places do that. Well, at this particular company, it was sometimes a pit of sorrow into which you would issue a request and never hear from it again... unless you went to some lengths.

Let's back up a little to set the stage. It's June of some year quite a while back, and it's about 9:30 at night. I guess I'm on call for the "last line of defense" debugging team, and I get pinged by the manager type who's wrangling an outage. It seems this team did some kind of code push and now they were completely down in some cluster: "0 online users" type of thing.

The incident manager asked me to find out what the load balancers were doing to healthcheck the systems in question, so I went looking. It turned out they were getting a small HTTP request on port 80, sort of like this: "GET /status HTTP/1.0".

But... the ones in the broken cluster weren't listening on port 80.

I asked if they did port takeover stuff (where one process can actively hand the listening socket to the one that's going to replace it), but then noticed they were running Java, and figured not. That kind of stuff was only really seen in some of the C++ backend processes at this gig.

I asked if maybe they had restarted in such a way that they tried to bind port 80 with the new process before the old one had shut down. Crickets.

Anyway, lacking a response from the engineer in question, I kept going, and found that poking another one of their instances in another (healthy) location would get a "I am alive" type response. To me, that seemed like a smoking gun: no response to HC = no love from the load balancer = no users.

A few minutes had gone by with no reply from the engineer, so I just used my Magic Powers on the box to just kick one of the java instances in the head to see what would happen. A few minutes later, it restarted, and now it was listening on port 80 and answering the health checks. Unsurprisingly, it started receiving requests from the load balancer, and the number of users started creeping upward.

I suggested a follow-up task: their stuff should kill itself when it can't get all of the ports it needs. Also, the thing that runs tasks should be set to check all of the ports too, and if it can't get satisfaction, *it* should kill the task.

Now, okay, granted, this is hacky. The program should be able to survive port 80 not being immediately available. It should also have some way to "hand off" from the other process. Or, you know, it could bind to a new port every time and just update its entry in the service directory. There are lots of ways to go about fixing this. However, considering the context, I went for the lowest-hanging fruit. You don't want to ask people to boil the ocean when they're having trouble making tea.

Anyway, they started restarting tasks and the service slowly returned to normal. I dropped offline and went to bed.

Two months went by. They kept having outages related to not having healthchecks. They were all preventable. They'd happen right when I was getting ready to go home, and so I'd miss my bus and have to wait an hour for the next one. That kind of crap.

I started counting the HC-related outages. Two, three, four.

At some point, I was over it, and dropped a post-it note on the desk of the head of engineering, pointing at the task to fix this thing and pleading for them to get involved. I was through being the "bad cop" for this particular one. It was time for management to deal with it.

Another month went by. Then, one day in late September, someone popped up in our IRC channel saying that they had turned on health checks and now had to test them, and could we help? I happened to be there, grabbed one of their machines, and promptly screwed up one of their java processes so it would just hang. (I forget what I did, but SIGSTOP seems plausible.)

The task runner thing noticed and set it to unhealthy. About three minutes later, it killed it, and then restarted it. Four minutes after that, it was still restarting, and maybe another four minutes after that, it was finally alive again and taking requests.

I informed this person that it did in fact work, but it took something like ten minutes to cycle back up to being useful again. They thanked me for checking and that was the last I heard about it. Apparently they were fine with this.

Considering this method of accessing things sent more people to the service than the *web site* did, you'd think it'd be kind of important. But, no, they were just rolling along with it.

Then today, I read a post about someone who found that their system had something like 1.2 GB strings full of backslashes because they were using JSON for internal state, and it kept escaping the " characters, so it turned into \\\\\\\\\\\\\\\\\\\\\\\\\" type of crap. That part was new to me, but the description of the rest of it seemed far too familiar.

And I went... hey, I think I know that particular circus!

Turns out - same circus, same elephants, different shit.

I've been kind of quiet these past few weeks. Part of that has been from plowing a bunch of work into getting serious about how all of the /w/ posts get generated. I figure if I'm going to start leaning on people to not do goofy things with their feed readers, the least I can do is make sure I'm not sending them broken garbage.

To really explain this, I need to back up to 2011 when this whole thing was just getting off the ground. I started writing here in order to keep the momentum from the writing I had been doing inside the company I was about to leave. I figured that anything was better than nothing, and so those posts were all done by hand. The posts themselves were hand-formatted: I'd type it up, slap on the header and footer, and then it'd get a link from the top-level index page.

Then people asked for an Atom feed, and I delivered on that too... ALSO doing it by hand at first. Yeah, that was about as awful as you can possibly imagine. Obviously that could not stand, but it did get me through the first couple of days and posts, and then my little generator came together and it picked up most of the load for me.

But there's a dirty little secret here: this generator has been little more than a loop that slaps HTML paragraph ("p") tags around everything. It doesn't really understand what's going on, and any time it sees a blank line, it assumes one ended and another one just began.

If you've ever looked at the source of some of the more complicated posts with embedded tables, audio players, PRE blocks or anything else of the sort, you've probably wondered what kind of crazy I was smoking. Now you know why. The only reason it works at all is because the web as a whole is terrible and browsers have had to adapt to our collective human clownery. HTML parsers tend to ignore the botched tags, and it generally looks right anyway.

I still find myself doing stupid things to work around the nuances of the ridiculous state machine that I created. If you've seen PRE blocks where for some reason there are lines with a single space in them, this is why! A blank line would trip the "stick on a /p and then a p" thing, but a line with a single space would not. So, I've been doing that.

Worse still, see how I'm calling it /p and p? I'm not using the actual angle brackets? Yeah, that's because there's no entity encoding in this thing at the moment. I'd have to manually do the whole "ampersand l t semicolon" thing... and HAVE been doing this all this time. I don't feel like doing that at the moment. (Because I'd have to fix it when it's time to convert this very post, but I'm getting ahead of myself.)

Both publog (the thing that is responsible for what you're seeing now) and my own diary software share a similar heritage, and I've been bitten by the lack of proper handling of this stuff over the years. For whatever reason, I decided it was time to do something about it, and finally got traction with an approach around the time of the new year.

Here's what's coming: every single post will be run through a generator that actually functions like a "real" parser - tokens and rules and put_backs and all of this! It's not just a "am I in a paragraph right now" state machine. It'll accumulate text, and when it's ready to emit a paragraph, it will do that with all of the rules it's been told about, like how to handle attributes, their values, AND when (and what) to escape/encode in the actual body of the tag/container.

This also goes for some of the "commands" that have been part of the input files all this time. When I include an image, I've been doing a special little thing that says "generate the IMG SRC gunk with the right path for this file with this height and width". This lets me ensure that the http and https feeds don't get cross-protocol URLs, among other things. The "this post has an update" lines and the backwards links to older posts also work this way.

This HAD been working with a bunch of nasty stuff that was basically building HTML from strings. You know the type, right? You print the left bracket, IMG SRC=, then you have to do a \" to get a literal " in there without ending the string... and then you end the string. Then you add the filename, and start another string and put a \" in it to cap off the SRC attribute of the IMG tag, and so on and so forth...

I'm kind of wondering who's reading this and thinks I'm a clown vs. how many people are reading this and are just nodding their heads like "yeah, totally, that's how we do HTML all over the place". But I digress.

Now, actually doing this has meant coding it up, but it's also meant going back and converting all of the damn posts, too. Any place where I had raw HTML shenanigans going on (like doing my own "ampersand + l + t + semicolon" stuff) had to be found and changed back to the actual character I want there. The program itself will do that encoding for me now. It's nice to have it, but it's a chore to go and do it without breaking anything, like a place where I WANT the literal gunk there.

With almost 5.5 MB of input text across 1400 posts, that was a non-trivial amount of work. I would not be surprised if I missed things that will pop up down the road and which will need to be hammered back down.

So yes, for a while, it will be "same clown, different circus". But, at least this time, I'll be trying to emit the right stuff.

I haven't set a date or anything for this. There's this possibility of also trying to solve some other dumb problems that also vex certain (broken) feed readers at the same time, and I haven't decided whether to block the rollout of the one thing on the rollout of the other one. This matters because I'd rather not rewrite every single /w/YYYY/MM/DD/whatever/index.html page multiple times. Ideally, they'll only change the one time. (What can I say, I care about these things.)

...

While waiting on that, if you're a feed reader author, you can at least check on a few things. You aren't honestly taking the "updated" time from inside the feed and using that in the HTTP transaction (If-Modified-Since), right? Right?? You know those are two totally different things from different layers of the stack, and aren't interchangeable, right? The IMS value should come from the "Last-Modified" header I sent you in the first place.

Right, Akregator? Right, NextCloud-News?

It's crazy how long it took me to figure out why they were sending me reasonable-looking "IMS" values that I had never handed out. It wasn't until I looked inside the actual feed that the penny dropped.

Want to know how the sausage is made and why this happens? Okay, settle in.

The web pages and the feed files (yep, plural: http and https) are made by running the generator on my laptop. The wall time on that system winds up being used in the "updated" fields in the XML gunk that is the Atom feed. The files also get a mtime that's about the same... on the laptop. More on that in a bit.

This writes to a directory tree that's a git repo, and a few moments later there's a git add + git commit + git push that captures the changes and schleps it off to my usual git storage space.

Later on, I jump on snowgoose (that's my current web server machine) and have it pull from that same git storage space into a local directory and then rsync the new stuff out of that tree into the various document roots - there are multiple web sites on this box.

If you didn't know this already, git does not preserve mtimes. The mtimes on files it writes out are just "now", whatever that may be. It's usually a minute or two later than when I did the generation on my laptop, just because I don't usually push to "production" right away. I usually eyeball things on an internal machine first.

Now, rsync DOES preserve mtimes, but it's preserving values that aren't particularly interesting. They are just the time when "git pull" ran on the web server and brought in the new/updated versions of the files. It's not the same time that the actual feed was updated on my laptop.

Apache uses the mtime on the files, so it's handing out "Last-Modified: (whatever)" based on that "git pull". This is not going to match the "updated" XML blob in the feed itself.

So, what I get to consider is whether I want to go nuclear on this and come up with something that will actually *SET* the mtimes explicitly and make sure they stay set all the way to the document root, no matter where it is.

Besides the broken feed fetchers, there's another reason to care about this sort of thing. What if I get a second web server, and put it behind a load balancer? Requests could be served by one or the other. Imagine if the two web heads did their "git pull" at two different times. Clients would get one Last-Modified value from server #1 and another value from server #2. Chaos! Madness! Insanity!

Now, I don't have a second web server, and in fact have no plans to do that unless people want to start throwing a LOT of money at me to run one in a colocation rack somewhere. But, it's the principle of the thing: controlling important values explicitly instead of leaving them to chance, *especially* since I'm expecting other people to do their part with those same values.

It's funny, right. I never thought I'd miss XHP until I started doing this project, and I didn't even do that many (internal) web pages at FB - just the ones I absolutely needed because nothing else would do.

In recent times, I've been realizing more and more just how much a screwed up management situation can lead to screwed up technical situations. I've written a bit about this in the past few months, and got to thinking about a specific anecdote from not too long ago.

I was working on a team which was supposed to be the "last line of defense" for outages and other badness like that. We kept having issues with this one service run by this team which ran on every system in the fleet and was essential for keeping things going (you know, the cat pics). We couldn't figure out why it kept happening.

Eventually, I wound up transferring from my "fixer" team and into the organization which contained the team in question, and my first "tour of duty" was to embed with that team to figure out what was going on. What I found was interesting.

The original team had been founded some years before, but none of those original members were still there. They had moved on to other things inside the company. There was one person who had joined the team while the original people were still there, and at this point, he was the only one left who had "overlapped" with the original devs.

What I found was that this one person who had history going back to when the "OGs" were still around was basically carrying the load of the entire team. Everyone else was very new, and so it was up to him.

I got to know him, and found out that he wasn't batshit or even malicious. He was just under WAY too much load, and was shipping insanity as a result. Somehow, we managed to call timeout and got them to stop shipping broken things for a while. Then I got lucky and intercepted a few of the zanier ideas while he was still under the stupid-high load, and we got some other people to step up and start spreading the load around.

I pitched in too, like trying to help some of the irked customers of the team and do some general "customer service" work. My thinking was that if I could do some "firewall" type work on behalf of the team, it would give them some headroom so they could relax and figure out how to move forward.

This pretty much worked. The surprise came later, when the biannual review cycle started up and the "calibration sessions" got rolling. They wanted to give this person some bullshit sub-par rating. I basically said that if they give him anything less than "meets expectations", I would be royally pissed off, since it wasn't his fault.

What's kind of interesting is that they asked the same question of one of my former teammates (who had also been dealing with the fallout from these same reliability issues), and he said the same thing! We didn't know we had both been asked about it until much later. We hadn't even discussed the situation with the overloaded engineer. It was just apparent to both of us.

With both of us giving the same feedback, they took it seriously, and didn't hose him over on the review. He went on to do some pretty interesting stuff for monitoring and other new stuff (including bouncing it off the rest of the team first), and eventually shoved off for (hopefully) happier shores.

The service, meanwhile, got way better at not breaking things. The team seemed to gel in a way that it hadn't before. It even pulled through a truly crazy Friday night event that you'd think would have caused a full site outage, but didn't. Everyone came together and worked the problem. The biggest impact was that nobody internally could ship new features for a couple of hours while we figured it out and brought things back to normal. The outside world never noticed.

Not long after that event, I considered the team "graduated" and that I no longer needed to embed with them, and went off to the next wacky team in that particular slice of the company's infra organization.

This was never a tech problem. It was one guy with 3 or 4 people worth of load riding on his shoulders who was doing his very best but was still very much human and so was breaking down under the stress. They tried to throw him under the bus post-facto, but we wouldn't stand for it. This was a management problem for letting it happen in the first place.

See how it works?

I saw a post about a year ago talking about the "five whys" technique of trying to figure out what caused something to fail. It was using a car scenario for an example, and it went something like this:

The car didn't start... because the battery is dead... because the alternator wasn't charging it... because the alternator belt broke... because the belt was beyond its useful life but wasn't replaced... because it wasn't maintained according to recommended schedule.

That's about five levels, and it pretty much stopped there. I figure, well, you can go beyond that, and in the case of the infra stuff at a big enough company, you probably need to if you intend to actually try to fix something.

So, that's been my life: trying to roll back through the series of actions (or lack of actions) to see how things happened, and then trying to do something about it. The problem is that if you do this long enough, eventually the problems start leaving the tech realm and enter the squishy human realm.

Perhaps you've heard of the OSI model of networking, where you have seven layers as a way to talk about what's going on in the "stack". I've seen some brilliantly snarky T-shirts that talk about "layer eight" and sometimes beyond as things like "corporate politics" and "management" and all of that good stuff.

It turns out that when you start doing this root-cause analysis and really keep after it, the "squishy human realm" is actually the no-longer-hypothetical "layer eight" from those T-shirts.

In our "car" example, you might discover that management is forcing people to ignore the maintenance schedule while saying things like "it'll work, trust me". Or, they're doing even worse things, like ignoring safety codes that have been written in blood.

For those of us in tech, we tend to get off much more lightly than people who do Actual Stuff in the Real World (like cars). Chasing down our problems means you start getting into things like "empire-building manager is hiring anyone with a pulse in order to look more important by having more direct reports". Maybe you chase that one down and you get to "manager of manager is also into this whole thing, and benefits from the equation".

That might lead into "the entire company is obsessed with hiring even though the tech equivalent of the Drake equation says there is no way they can find anywhere near that many qualified people in the entire world".

What that does that look like? Well, some people have no business working on certain kinds of systems, whether as a transient situation, or a permanent one. Transient situations are a lack of training. Permanent ones might come from attitudes or a genuine lack of ability for whatever reason. Having the wrong person on the job is supposed to be noticed and handled by the manager. If they don't, that's a failure.

Now, the team's manager (M1) also has a manager (M2) of some kind, and M2 is supposed to be making sure M1 can actually, well, manage! If they can't tell if that's happening or not, that too is a failure.

In some situations, you come to realize that a whole bunch of bad things happen due to non-technical causes, and they are some of the hardest things that you might ever need to remove from an organization. Unlike the line workers, management is in a whole different world in which the "reality distortion field" matters most. You either generate a big enough one yourself, or you slot into someone else's. If you are opposed to it, you are rejected.

I guess this is my way of warning anyone who fancies themselves a troubleshooter and who really, truly, wants to get to the bottom of things. If you do this long enough, expect to start discovering truly unsatisfying situations that cannot be resolved.

Also, I will remind anyone who wants to try to tilt at such a windmill that if you are given responsibility without the power to make any changes, then you have just become the scapegoat. I said this in a post way back in February 2013, and I *still* fell into that damn trap in 2017 within a particularly broken organization.

Finally, in this same vein, I wanted to share something that a reader sent to me a while back, and that I found to be brilliant and amazing (I still do, but I did then, too): People can read their manager's mind.

In particular, pay attention to where it says corollary 1 and starts talking about the "insane employee". The whole "personal offense" thing? Yeah, if you have the ability to not become that person, try to avoid it. Alternatively, if you're cursed with the tendency to fall into those things, try not to give yourself a hard time when someone terrible takes advantage of you for the nth time.

Hang in there.

Sometimes, on a lazy weekend afternoon, I use apt-get to pull down the source of something and start grepping it for things that are bound to be interesting. This is one of those afternoons, and I found something silly. While looking for uses of time_t in bash, I found a variable called "time_since_start". Uh oh.

bash supports a dynamic variable called "SECONDS" (bet you didn't know that - I didn't), and it's documented as "the number of seconds since shell invocation". I'm sorry to say, that's not quite true. You can totally make it go negative since it's based on wall time. Just set the system clock back.

root@rpi4b:/tmp# systemctl stop chrony
root@rpi4b:/tmp# echo $SECONDS
11
root@rpi4b:/tmp# date -s "2023-01-01 00:00:00Z"
Sat 31 Dec 2022 04:00:00 PM PST
root@rpi4b:/tmp# echo $SECONDS
-2500987

That's an extreme demonstration, but backwards-going wall time happens every time we have a leap second. Granted, we're in a long dry spell at the moment, but it'll probably happen again in our lifetimes. The difference there is just one second, but it could break something if someone relies on that value in a shell script.

Or, how about if the machine comes up with a really bad time for some reason (did your hardware people cheap out on the BOM and leave off the 25 cent real-time clock on the brand new multi-thousand-dollar server?), the shell gets going, and later chrony (or whatever) fixes it? Same deal, only then it might not be a second. It might be much more.

In the case where the machine comes up with a past date and then jumps forward, SECONDS on a still-running shell from before it's fixed will be far bigger than it should be. I'm pretty sure every Raspberry Pi thinks it's time=0 for a few moments when it first comes up because there's no RTC on the board. Run "last reboot" on one to see what I mean.

I should also mention that bash does other similar things to (attempt to) see how much time has passed. Have you ever noticed that it'll sometimes say "you have new mail", for those rare people who actually use old-school mail delivery? It only checks when enough time has elapsed. I imagine a "negative duration" would mean no more checks.

The lesson here is that wall time is not to be used to measure durations. Any time you see someone subtracting wall times (i.e., anything from time() or gettimeofday()), worry. Measure durations with a monotonic clock if your device has one. The actual values are a black box, but you can subtract one from the other and arrive at a count of how many of their units have elapsed... ish.

Be sure to pay attention to which monotonic clock you use if you have a choice and there's any possibility the machine can go to sleep. "Monotonic time I have been running" and "monotonic time since I was booted" are two different things on such devices.

Here's today's bonus "smash head here" moment. From the man pages for clock_gettime on a typical Linux box:

CLOCK_MONOTONIC: "This clock does not count time that the system is suspended."

Here's the same bit on a current (Ventura) Mac:

CLOCK_MONOTONIC: "...and will continue to increment while the system is asleep."

Ah yes, portability. The cause of, and solution to, all of life's software issues.

I find some funny rabbit holes sometimes. Tonight, it went like this. Ubiquiti released a new version of the software for their USG devices because they had this thing where their dhcpv6-pd implementation could be exploited to run arbitrary commands by someone sitting in the right spot on the network (i.e., out your "WAN" port).

It's been a good while since they put out a new build for these devices, and I wanted to know what else changed. I find that when companies supposedly ship a "just a security fix" patch, they usually end up shipping far more, and probably break stuff too. (I'm still bitter about the 2020-002 "security update" for Macs.)

Anyway, it got me thinking: can you diff these things? Turns out, you sure can. It's two squashfs filesystems, so mount 'em and diff 'em, and dig through the results, and... hey.

/opt/vyatta/sbin/dhcpv6-pd-response.pl:
 
         if (defined $domain) {
             $domain =~ s/\.\s+$//;
+            $domain =~ s/[^A-Za-z0-9.]+/-/g;
             $dn = $domain;
         } else {
             $dn = "";

Yeah. That's what changed. So there's that. *facepalm*. Then I got bored and kept looking through the output to see what else happened. That's when I saw that the entirety of /etc/shadow changed. A bunch of numeric values changed, like this:

-root:!:18920:0:99999:7:::
+root:!:19369:0:99999:7:::

I had to look it up to be sure, but that's the "date of last password change". Divide them by 365 and you'll realize one of them is about 51 years, and the other one is about 53 years. So, 2021, and 2023 - the dates of the previous release and the new release, respectively. Their release process obviously rebuilds the shadow file.

But that's not the end of the rabbit hole. Thinking of last week's time post, I started looking at that number. It's so small. It fits into 16 bits (but not 15). I wondered what sort of type they were using to hold it. Into the shadow source I went.

The first thing I found was something called strtoday(). It looks like this (adjusted a bit to fit here):

long strtoday (const char *str) {
        time_t t;
[...]
        t = get_date (str, NULL);
        if ((time_t) - 1 == t) {
                return -2;
        }
        /* convert seconds to days since 1970-01-01 */
        return (long) (t + DAY / 2) / DAY;
}

Uh huh. It returns a long. On a 32 bit machine, a long is 4 bytes, and it's still going to be 4 bytes even after glibc does their "time_t is now 64 bits" thing that's coming down the pipe eventually. longs aren't going to change.

So, when does this break? It turns out... 12 hours BEFORE everything else blows up. "DAY" is defined in the source as (24L*3600L), so 86400 - the number of seconds in a day. It's taking half of that (so 43200 - 12 hours worth of seconds) and is adding it to the value it gets back from get_date. That makes it blow up 12 hours early.

2038-01-18 15:14:08Z is when that code will start returning negative numbers. That'll be fun and interesting.

Remember, the actual "end times" for signed 32 bit time_t is 12 hours later: 2038-01-19 03:14:08Z.

The lesson here is: if you take a time and do math on it and shove it into another data type, you'd better make sure it won't overflow one of those types that *won't* be extended between now and then.

...

$ cat t.cc
#include <stdio.h>
#include <sys/time.h>
 
#include <cinttypes>
 
#define DAY (24L*3600L)
 
long strtoday_tt(time_t t) {
  return (long) (t + DAY / 2) / DAY;
}
 
int main() {
  printf("2147440447 -> %ld\n", strtoday_tt(2147440447));
  printf("2147440448 -> %ld\n", strtoday_tt(2147440448));
  return 0;
}
$ ./t
2147440447 -> 24855
2147440448 -> -24855

Questions, questions, questions. Sometimes I have answers. Here we go with more reader feedback.

...

M.D. asks if I could share a story about when a company "did something really wrong".

OK, how about yet another case of "real names" gone wrong? I'm talking about the thing where some derpy programmer writes a filter to "exclude naughty words" and ends up rejecting people with names that HAPPEN to match a four-letter English word. My canonical example is "Nishit" because that's what actually happened at another job back around 2009.

But that's old news. I'm talking about "yet another case". This one is from right at the end of 2019. It seems like someone decided they were going to "move a metric" from the support queue. There were a TINY NUMBER of customers who had deliberately signed up with obviously offensive names. They were being handled by reasonable bags of mostly water (i.e., people) who could look at it and figure out what to keep and what to purge.

Well, the derpy programmer with a goal to hit by the end of the quarter apparently struck again, and they wrote a thing to ban anyone who matched a short list. Then they ran it, and - surprise - it banned a bunch of real people who weren't doing anything wrong. Of course, those people have probably had many problems on other services, and now THIS company was the latest one to show how damn stupid and unfeeling it could be.

"My last name really is 'Cocks'. How would you like me to proceed?"

Unsurprisingly, this pissed off a bunch of people and generated a blip in the news cycle. Internally, it was brought to the weekly review meeting that I was somehow still running at that point. Someone was there and presented the case, and it was pretty clear they were going through the motions because we called them on the carpet.

For some really stupid reason (literally every other senior engineer and manager was at some offsite planning thing that morning, and *someone* had to run this meeting), I was the most senior person in the room, and so I felt I had to ask them the question as "the voice of the company" (whatever that even means):

"Can I get you to promise to never do this again?"

They wouldn't commit to it. I got no reply. They just looked at me. Conclusion: this will definitely happen again. Nobody gave a damn about what happened to the customers, and how bad the whole thing looked.

Afterwards, I talked to some friends who had worked in the trenches in customer support. They knew what was happening in terms of the "trouble reports" that would come in from people using the app. They had a good feel for what was actually a problem and what was clearly "OKR scamming".

Near as we can figure, they decided to code this up because it would let them claim to have automated some class of tickets that were being filed. It's like, sure, it would in fact remove the handful of tickets that get filed about this. It would also generate a godawful amount of hurt (and bad PR and so on) a few hours or days later, and would have to be turned off. But, the person managed to ship the feature, and so they can get their bonus, or promotion, or whatever.

Of course, karma is a bitch. A few months later, COVID hit and the company started laying people off in droves. I bet all of those people are gone now. Unfortunately, this also means anyone who learned a lesson from this event is probably gone, too. Hmph.

For anyone who's in today's "lucky 10,000" set, have some additional reading on this topic.

...

A reader responded to my "mainless" program post and asked if you could avoid the segfault by putting "_exit()" into the destructor or similar. They're totally right. If you make the program bail out instead of trying to carry along into uncharted territory, it will in fact exit cleanly. You already have unistd.h in my example, so go for it.

...

Another reader commented on the same post, asking just "The file name is a bit worrying. Is everything alright?". I guess they missed the link to knowyourmeme.com, or are generally unfamiliar with the notion of "things that aren't really trolling but are sort of funny kind of".

When it comes to that kind of stuff, that lizard gets me.

Hhhhhehe.

...

Jeff asks if I'm on Mastodon. I am not using that (or any other aspect of the "Fediverse"). I am also not on Twitter. I used to have a "business" account out there for a while, but never really used it, and deleted it last year when it became apparent where Twitter was headed. I stand by that decision.

I should mention that there are some dubious parts of the whole Fediverse thing, as someone who runs a site that occasionally gets links shared around. Posting a link to one of these things basically summons a giant swarm of (bot) locusts who all want to generate a link preview at the same time. I was going to write a post about this a while back, but it would be short and kind of threadbare, so I'll just mention it here instead.

Now, since all of this /w/ stuff is just a bunch of flat files on disk, all of it gets served out of memory and it's like ho hum. I can't run out of database connections or anything like that. I'm basically limited by the level of bandwidth I pay for on my switch ports. Eventually, they all have what they came for and it stops. But, for a minute or two, it can be interesting to watch.

There's a certain "cadence" to a web server's activity logs as seen in "tail -f". When this happens, it scrolls so fast you can barely read it. You definitely notice.

This is far from a new issue. A report that it can collectively be used "as a DDOS tool" was filed back in August 2017, when it was a far smaller problem.

It should not surprise anyone who's been doing this kind of thing for a while that the bug report in question (which I won't link here, but which people will still look up and try to brigade) has been closed since 2018.

Did anyone ever see the classic Simpsons episode where someone falls down a well, and at the end of the episode they "solve the problem" by sticking a "CAUTION WELL" sign in the ground?

"That should do it." - Groundskeeper Willie

I can hear his voice in my head any time I see a bug like that.

Want to inflict terrible things on other programmers who show up later to do maintenance work? Write C++ code that doesn't need a main(). Then write C++ code that doesn't *have* a main(). Yes.

I mentioned this quite a while back, calling it "spooky action at a distance" in code, but looking at that now, it seems like it was a very long and drawn out demonstration.

Instead, tonight, I present a far simpler version. Usual disclaimers apply: may summon Ancient Ones who will haunt your soul. Probably won't work on all systems or compilers. It didn't work for me until I gave it -O2, and even then, it still gives a magnificent segfault.

At any rate, enjoy the evil.

$ cat hhhehehehe.cc
#include <unistd.h>
 
class wat {
 public:
  wat() { write(1, "wat\n", 4); }
};
 
static wat wat_;
 
// no main.  nothing else.
$ g++ -O2 -Wall -nostartfiles -o hhhehehehe hhhehehehe.cc
/usr/bin/ld: warning: cannot find entry symbol _start; [...]
$ ./hhhehehehe 
wat
Segmentation fault
$ 

That's it.

Given that we're now within 15 years of the signed 32-bit time_t craziness, I decided to start playing around with my own stuff to see how things are doing. I wanted to see what would break and what would work.

One thing I particularly wanted to see was how my smaller systems would work. It's basically a given that my 64 bit Linux boxes are going to be fine since time_t is already wider, and it won't explode in 2038. But that's far from the whole story. 32 bit machines still exist, and are more common than some would think thanks to the existence of things like Raspberry Pis.

Unless you deliberately install the 64-bit flavor of Raspbian, you're going to get a 32-bit system. With the version of glibc it's currently running, you will hit the wall. It's easy enough to try - you'll notice that you can't actually set the clock that far ahead:

root@rpi4b:/tmp# date -s "2038-01-19 03:14:08 UTC"
date: invalid date ‘2038-01-19 03:14:08 UTC’

So, okay, put on your "time to do evil" hat, set it one second earlier, and wait for the fun to happen. Starting from scratch again, it does this:

root@rpi4b:/tmp# systemctl stop chrony
root@rpi4b:/tmp# date -s "2038-01-19 03:14:07 UTC"
Mon 18 Jan 2038 07:14:07 PM PST
root@rpi4b:/tmp# 
Message from syslogd@rpi4b at Jan 18 19:14:07 ...
 systemd[1]: Failed to run main loop: Invalid argument

Broadcast message from systemd-journald@rpi4b (--- XXXX-XX-XX XX:XX:XX):

systemd[1]: Failed to run main loop: Invalid argument


Message from syslogd@rpi4b at Jan 18 19:14:07 ...
 systemd[1]: Freezing execution.

Broadcast message from systemd-journald@rpi4b (--- XXXX-XX-XX XX:XX:XX):

systemd[1]: Freezing execution.

Yee haw! Look at that sucker burn. I particularly dig the XX-XX stuff. It's like a cartoon character who's been knocked out.

Now, before you whip out the pitchforks, keep in mind that systemd is just the messenger here. It's just working with what it's been given.

Also, the system is actually still up here. systemd has just basically checked out and is not going to do much more for you. It's not even going to take an ordinary "reboot" since that's really just a request to init (pid 1, so systemd again) to reboot the box. You're going to need to use "reboot -f" and suffer whatever badness might happen to stuff on the box. It's like pulling the plug, so have fun with that.

What happened? If you dig around in the remains, you will find that an assertion in systemd fired. It's refusing to continue unless clock_gettime() returns 0. Clearly, it returned something else. systemd saw this not-zero value and decided to protect itself by effectively stopping.

So you think "I know, I'll try this again, and strace pid 1 this time, and see what was in fact returned". You get something like this right before it croaks:

... what? It returned 0? Yes... and no. Look at it closely.

clock_gettime*64* returned 0. But systemd called clock_gettime. strace is showing you the system call... but that system call happens by way of a C library function which in this case is being provided by glibc 2.31. If you were to open up glibc's source code and go digging around for clock_gettime(), you'd find this:

  ret = __clock_gettime64 (clock_id, &tp64);

  if (ret == 0)
    {
      if (! in_time_t_range (tp64.tv_sec))
        {
          __set_errno (EOVERFLOW);
          return -1;
        }

First call the (64-bit capable) syscall. Then assuming that succeeds (and it does, per strace), then see if it'll fit in a (32-bit) time_t. It won't, so set errno to EOVERFLOW, and return -1.

That's what systemd gets, and so it blows up.

glibc is saying "I can't fit this into that, so I'm failing this call".

This is wrapped in a bunch of preprocessor #if tests such that it only runs when __TIMESIZE isn't set to 64, but guess what? On this particular combination of hardware and software, __TIMESIZE is in fact 32. Grovel around in the headers if you like and follow the bouncing ball starting here:

... or just write something dumb to printf(..., __TIMESIZE) and see.

To be clear, this is glibc 2.31 on the 32 bit build of Raspbian/Raspberry Pi OS 11 (bullseye) on a Pi 4B. Newer versions of the OS will almost certainly not behave this way, since glibc itself is marching down the road to having 64-bit time even on 32-bit machines. Once that's done and rolled up into a release, expect this to go away.

...

And yes, NetBSD and OpenBSD tore off this band-aid about 10 years ago, and it's a done deal now. I know. Cheers to that.

In the past, I've written a few complaints about poorly-behaved feed fetchers. It's been a little over a year, and the situation is about the same. There are still a few people out there who think it's cool to poll every minute, or every 2 minutes, or whatever. It's not cool. It's useless. I don't update this thing anywhere near that often, so what's the point of wasting those resources?

There have been some bright spots. At least one person switched on If-Modified-Since headers and even put a little comment in their User-Agent header to let me know about it. That was above and beyond, so thank you to whoever that is.

But, there are still plenty of misbehaving feed readers out there, so it's time to talk about carrots and sticks.

The carrot basically is: if you have a well-behaved feed reader, you will continue to be able to discover a new post on my feed in a reasonable amount of time. This is most people. Most people do it right. Thank you for that.

The stick is: if you do not, you will not. It will take considerably longer to notice something's different out here.

What constitutes a well-behaved feed reader? My primary concern is about not having to serve the full feed to someone who has no reason to pull it again. This means making conditional requests - your client tells my server the last version of things it saw, and my server goes "okay, nothing's different" or (once in a while, after an update) "oh cool, here's the latest".

How do you do this? Ideally, you just run your feed reader and it figures it out. But, trust me, from looking at the feature requests and code bases for far too many of these things this past week, it seems like that's not very common.

This is how the tech part of it works, lest anyone claim it's too hard to implement. My server sends out a number of headers when you fetch the feed. Two of them are potentially applicable here. Right now, they look something like (but not exactly like) this:

Last-Modified: Fri, 06 Jan 2023 00:00:00 GMT
ETag: "xxxxx-yyyyyyyyyyyy"

Well-behaved HTTP clients can store those values when they do a fetch, and then return either or both of them in their subsequent requests. The first one turns into If-Modified-Since, and the other one turns into If-None-Match. Note that second one actually requires the "" around it or it won't work. (Yeah, I know. Not my doing.)

If-Modified-Since: Fri, 06 Jan 2023 00:00:00 GMT

... and/or...

If-None-Match: "xxxxx-yyyyyyyyyyyy"

Now, your HTTP client software should take this as some kind of argument to some well-defined setting and you should probably not be setting headers directly, but we're still smashing rocks together for a protocol that's 30+ years old. But I digress.

(Side note: this means your feed reader has to maintain some state per feed. You can't just statelessly fetch a URL until the end of time. That's incredibly boneheaded.)

Just take what you got before and hand it back as shown above. If nothing's changed, you'll get a 304 HTTP code back, and that means "nothing new". It's a short, simple transaction, and uses very little in the way of resources.

If the feed has been updated, say, because I wrote a new post, or did an update or typo fix or whatever to an existing one, then you'll automatically get that returned as a 200 along with a new set of headers. It's your feed reader's responsibility to remember one or both of those fields and then use them later on.

From my point of view, a request with a proper "IMS" or "INM" header is considered a conditional request. I look relatively kindly upon those. Those tend to come from people who want to do the right thing.

A request with neither "IMS" nor "INM" headers is unconditional, and I'm not such a fan of those. I understand that everyone's going to fetch something "fresh" now and then. That's a given. You have to prime the pump somehow. I don't care about that.

But when someone requests the full feed and makes no attempts to conserve, and they do it over and over again, like every 2 seconds? That's when I sit down and start coding. And code I did. That's why I'm writing this post. Poorly-behaved feed readers will no longer get timely updates.

I should note that one particular feed reader sends "Wed, 01 Jan 1800 00:00:00 GMT" and that's utter bullshit. You made that up and you know it. Nobody ever served you a page with that value. See, this is actually known pathological behavior. Sending that does not count as conditional.

Bad clients get a 429. That means slow your roll.

Bonus note for pedants: yes, it's still possible to be abusive with perfectly-formed conditional requests. Please don't try to find out where that point is. Just remember, I don't post that often. You don't need to poll that often.

I was doing some light nerd reading at lunch the other day and ran across someone who had encountered trouble with a program that was using TCP_NODELAY when perhaps it shouldn't. TCP_NODELAY is one of those things which turns off Nagle's algorithm, which is usually used to batch up a bunch of small writes so you don't spam the network with tons of tiny packets. (If this sounds familiar to long-time readers, it's because it starred in a post that made the rounds in the fall of 2020.)

All of those packets have overhead. It's not quite the same problem that it was when we had 10 megabit shared-media networks with collisions out the wazoo, but it's still not great to just waste bandwidth and CPU time on things that aren't latency-sensitive.

The problem comes when you have a program that has a bunch of stuff to put on the wire, and yet it does it with individual calls to write(). Instead of pushing (say) ~2 KB at the network with a single call, it instead spins through the buffer, writing each one individually. Now you have 2000 packets flying around, all with their headers and everything else as overhead. Having the kernel batch this up is basically saving the world from broken code.

I saw this and it reminded me of a similar bit of damage in my own life. I have some projects where I am forced to wrap another program and listen to its stdout. It doesn't have a library form, so the only way to make use of it is to go through this whole rigamarole. I get to create a pipe, then fork and have the child connect stdout to that pipe and exec the program in question. The parent process then sits there listening to the pipe for updates.

I realized that my program (the reader) was waking up FAR too often. I should be getting updates every 30-45 seconds, but it would wake up a couple of thousand times in that interval. WTF? Well, it turns out that for whatever reason, it writes to stdout (more or less) a byte at a time.

Seriously. I had to see this for myself, and attached to it with strace. It pretty much looked like this:

708589 22:46:24.174856 write(1, "\"", 1) = 1 <0.000039>
708589 22:46:24.175018 write(1, "i", 1) = 1 <0.000041>
708589 22:46:24.175187 write(1, "d", 1) = 1 <0.000040>
708589 22:46:24.175339 write(1, "\"", 1) = 1 <0.000041>
708589 22:46:24.175506 write(1, " : ", 3) = 3 <0.000048>
708589 22:46:24.175666 write(1, "12345", 5) = 5 <0.000041>
708589 22:46:24.175814 write(1, ", ", 2) = 2 <0.000041>
708589 22:46:24.175981 write(1, "\"", 1) = 1 <0.000041>
708589 22:46:24.176138 write(1, "c", 1) = 1 <0.000039>
708589 22:46:24.176279 write(1, "h", 1) = 1 <0.000040>
708589 22:46:24.176443 write(1, "a", 1) = 1 <0.000041>
708589 22:46:24.176596 write(1, "n", 1) = 1 <0.000040>
708589 22:46:24.176732 write(1, "n", 1) = 1 <0.000040>
708589 22:46:24.176875 write(1, "e", 1) = 1 <0.000043>
708589 22:46:24.177045 write(1, "l", 1) = 1 <0.000070>
708589 22:46:24.177331 write(1, "\"", 1) = 1 <0.000030>
708589 22:46:24.177454 write(1, " : ", 3) = 3 <0.000029>

That's 17 lines from a much longer log. Those 17 lines alone were displayed in about three milliseconds, and had many many more above and below. Here I was, trying to see what kind of data it was sending to me, and it was spamming me with syscalls.

If you look closely, you can see it's not quite one byte per write() but it's pretty close. Numbers go all at once for whatever reason, and those " : " strings are another curiosity.

I pretty much forgot about this until the TCP_NODELAY post crossed my path, and then was reminded of this. Clearly, short writes are pretty common.

I wonder though, do people not strace programs any more? If this was my project and I was trying to figure something out, all of the vertical scrolling would drive me crazy. When it's spewing out more than my scrollback buffer will let me access, something is wrong! I'd go to lengths to try to batch it up a little.

The one part of this process I control is the reader, so that side has some sanity-enabling hacks added to it. I wait for up to a second between checks, and even then, I call select() with a 250 msec timeout. This gives the syscall-spamming writer program a chance to finish writing to the pipe before I go and read it. This raises the chances that I'll get the whole thing from a single read() call. Their program can spin making thousands of syscalls per event. Mine makes about four: futex, futex, select, read.

...

Full disclosure: I thought about writing a program to spit the body of this post out as a series of individual bytes handed to write(), and then the post itself would have just been the output from running it in strace. Everyone would have had to read it vertically, and I bet it would have been seriously annoying for basically anyone, except the rare people who have lived through it and would find it oddly hilarious.

But, I'm trying to make a point here, so I decided to make it accessible to people who tend to read things in terms of words and sentences. I left the crazy up in the first two words of the title instead.

Oh boy. Yesterday's post has riled people up. A lot of them don't like what I said about the ACME protocol. I get the impression a few of these people haven't looked at the problem that needs to be solved in a different light.

How about we give that a shot now? Let's just go through the steps for getting a secure web site going, and ignore the specifics of the protocol for the moment.

First, the baseline assumptions: there's a key. There's a certificate signing request which references that key. Then there's the certificate itself with a signature which attaches it to the "web of trust" (ehhh...) that is largely accepted by most clients. Okay?

1. You generate a key. It'll eventually need to be installed on the web server somewhere. It's just a blob of gunk that's been encoded in ASCII with that whole --- BEGIN/END blah blah --- stuff at either end. You should probably generate it with a certain algorithm and with a certain complexity (i.e., number of bits).

2. You generate the CSR. It needs to read from that key in order to pick up its identity, but the CSR itself doesn't contain sensitive key material. The CSR has a bunch of fields that almost nobody uses: country, organization name and unit name, an e-mail address, and so on. Some of them used to mean something in a different age, but you'll probably find that a CA ignores all of them and generates something else entirely in the resulting certificate. Case in point: the cert for my site has CN = rachelbythebay.com and that's it.

3. You tell the CA that you want a cert by sending them that CSR.

4. The CA says "ok, prove you own this domain, punk" and gives you a URL or two that you can populate with some magic string in order to do exactly that. Or, it gives you some DNS entries for that domain that you can create with a magic string - maybe the same string as the URL, maybe not.

5. You stand up the magic URL with the magic data, or drop in a DNS entry with the magic data.

6. You either poke the CA to say "ok, go look", or (much more likely) you just sit there feeling stupid until they happen to retry and notice. They usually don't provide much in the way of being able to trigger this on-demand, so this can add a bunch of delay to the process.

7. The CA eventually sees the right thing in the right place and says "okay, this can't be chance, so they must actually control the document root and/or DNS or whatever", and that's enough for them. They issue a cert for that key and domain name. (In the old days, they'd start looking at DUNS numbers and stuff like that instead as verification.)

8. You check back and notice it is in fact available. It would be nice if it poked you somehow, but odds are, they won't, or at least, they won't do it faster than you could check for it. You download the cert which is yet another --- BEGIN --- ... --- END --- ... blob of gunk.

9. You install the key and cert on the web server and swing the config around to reference it.

10. You kick the web server in whatever fashion it requires to make it start using the new key/cert and stop using the old one - reload, restart, reconfigure, whatever.

11. You load the site up and verify that it actually works, and that it's using the new cert and not the old one.

Now, when you're talking to the CA, it's likely they are going to want you to authenticate yourself. Notice though, at no point did I say that you should now throw down and start doing your own crypto (*cough* JWT) in order to generate some kind of "proof" of some "claim" to convince them that you are in fact who you say you are.

You could just, you know, pass them an opaque token that they issued to you when you logged in... just like people have been doing with the web for years and years. Ever call Stripe from inside a program? You get an API key. How about Wavefront? Another API key. Gandi? API key. It gets set in the headers. Now they know who you are. Done.

When you're working in these terms, it's a matter of building up a request in a format they desire. It'll probably be a POST because sane people don't create things with just a GET. So you fire some form data containing the CSR at them, or maybe you send them a blob of JSON, and stick your auth token in a header. This goes over https to them, and so it's as secure as anything else you'll be doing with this stuff.

See how much more accessible that is? See how evil rolling your own crypto can be?

It's evil. Very evil. Don't enable their goofy auth schemes and don't roll your own crypto.

No, really.

How long since the last alg=none JWT vulnerability? It's 14 days as I write this, and this is January 2023. 2023!

BTW, to the person that said these protocols are "... an opportunity to pull in as many 'standards' as one can ..."? You nailed it. That's what I meant when I said "web kool-aid" in the original post.

People sometimes ask me why I don't use Let's Encrypt, and it's a long story. It has a lot to do with just how damn evil the protocol is. It looks like it was created by people who had been drinking FAR too much of the web kool-aid, since it's chock full of terrible things. It should be a small amount of drama to start a process, receive a magic string, sock it away somewhere at a magic path, then poke the validator and say "go for it". Then you just check back and see whether it worked or not.

Actually doing this with the ACME stuff is terrifying. I first looked at this several years ago, and not only was the protocol bad, but the implementations I checked out were also miserable. One of them had line widths in excess of 200 characters. Many puppies paid the price when that was created, and the fact that nobody else cares just boggles the mind.

But, I kept reading. This thing winds up running openssl (as in, the CLI toolset in /usr/bin or whatever) and then grovels around in the output with regexes. Then it turns it into a jwk, and this is where the "web kool-aid" thing shows up. It's part of the protocol, so it's not like they had any choice, but it's just one more awful thing you now have to worry about supporting.

But somehow this gets turned into JSON, and then that gets a SHA-256 hash, and then the base64 encoding of that turns into a thumbprint? So it's a SHA of a text representation of something that can be reordered or reformatted, and this is supposed to be useful?

Then it runs openssl again to read the CSR, and so on and so forth.

Anyway, that's about where I got to it after first encountering it in 2018, and then after reconsidering it in 2020 when I found myself with a bunch of extra time on my hands due to the lockdown.

But what about now? This isn't about Let's Encrypt. This is about me finding a supposed alternative and going through the same process of due diligence to understand just what I'd be getting into. Somehow, a couple of weeks ago, I found this other site which claimed to be better than LE and which used relatively simple HTTP requests without a bunch of funny data types.

I went through all of their API docs. You call an endpoint and tell it which domain or domains you want, and feed in the CSR. You also tell it how many days it should last: 3 or 12 months, more or less. Once you do this, it tells you what to do in order to perform verification. They do the usual techniques: put this magic blob in DNS, put a magic blob in your document root somewhere, or confirm that you can receive e-mail at a certain address.

Your end goes and sets this up, then calls back and says "okay, I'm ready, go look". This starts the process rolling, and assuming it all checks out, in theory you'll get a certificate issued pretty soon after that point.

Sounds great, right? I thought so, and created an account at this point to take a whack at it. That's when I saw it started asking questions about what kind of account did I want. Did I want a free account, or to pay this much, or that much, or this other even bigger amount? What? They didn't mention this before.

This is when the fine print finally appeared. This service only lets you mint 90 day certificates on the free tier. Also, you can only do three of them. Then you're done. 270 days for one domain or 3 domains for 90 days, and then you're screwed. Isn't that great?

Oh, and finally? You can't do the less-insane API on the free tier. Yep. You have to pay up for that. Gotcha, sucker!

I immediately deleted my account and marked the experience as one worthy of warning others.

At least now I can point at this post when people ask why I'm still using an old-school certificate on my site. It's deliberate.

January 4, 2023: This post has an update.

Just a hair over 10 years ago, I wrote a post lamenting the fact that my WiFi-only original iPad (which was new at the time) was probably a mistake. After all, if I took it outside my house and away from the one wireless network it knew, it was now cut off from the Internet. If someone stole it or if I left it behind somewhere, there would be no way to track it down or wipe it.

Well, times change, and now that's no longer a concern. Pretty much anything that Apple sells nowadays that is intended to be portable is also trackable, even if it's "off" (whatever that even means now). Obviously I'm talking about AirTags, but the phones, watches, iPads, and yes, even laptops have the ability to be found in the same way if they are new enough.

When I thought about this at the time, I figured maybe they'd do some kind of one-way wifi and satellite-based "push" system where you could send "kill signals" for stolen devices. Yeah, that was a pretty bad call. It didn't end up even close to that. Instead, now, just every other Apple device in the vicinity has the ability to hear a beacon from a missing device and report about where it was found.

That plus the whole activation lock thing where a device won't let you use it unless the previous owner releases it from their iCloud account hopefully put a pretty big dent in the utility of stealing these things. Even with that, you still hear about people knocking over Apple stores and stealing armloads of devices for some reason. It must be for the non-serialized parts, since the rest of them will surely tattle if they are ever connected to the "mothership" again.

Looking at it another way, I no longer worry about this kind of thing. It's as if they saw something that might keep people from wanting to buy a certain class of device and then did something about it. How about that?

I heard about a neat bug once that was caused by the interaction of some hardware that was missing some electronics and some software which was just doing what it was told. It had to do with the "access of last resort" you'd use on a machine that was otherwise dead to the world: the console.

Imagine a datacenter with tens of thousands of Linux boxes running. Sometimes, they break and fall off the network. Fortunately, they have a "mini-me" type thing attached which then allows you access to a serial console. It's not quite the same as being there with a monitor and keyboard plugged into the box, but it's frequently enough to dig out of a real mess without getting in a car (or worse).

It seemed that people had been trying to fire up the console on their systems and weren't getting the expected results. What's supposed to happen is that they connect, hit ENTER once or twice, and it should pop up something like this in reply:

Linux x.y.z (something-arcane.foo.bar.company.example)

login:

They'd hit ENTER and at best, nothing would happen. Sometimes, it would just be a jumbled mess. Obviously, if the machine was unreachable over the network, we couldn't decode it, so it took a bit to find one that had a broken console but which was still available over the network.

What we found was interesting. The thing that actually puts up that login prompt is a process called getty (or some variant, like "agetty"). Its job is just to sit there and handle the serial line, read your login name, and get you connected to a login process to carry on from there.

For this to work, agetty and the serial port on the host have to agree with the serial port on the client in terms of baud rates (and other things too, but let's keep this story simple). If you get one out of sync, the other end will have no idea what you're talking about.

If you've never crossed paths with this before, imagine you're a dog that can only hear whistles at a specific set of pitches: one high, one low. Someone who uses the wrong set of frequencies won't make much sense to you. Baud rates are a little like that.

Somehow, the getty on these machines had gotten into a state where it wasn't running at the same baud rate as the actual system which was providing remote access to the serial console. We knew there was a feature in getty that would look for a "serial break" (imagine a really long low whistle in the dog analogy) and it would cause it to rotate through a list of baud rates.

This feature was probably intended to avoid a chicken-and-egg situation where you plug a terminal into a serial port on some Unix box and can't talk to it because it's at some rate that you can't change to. So, you keep poking it with BREAKs until it comes around to something that you can reach, and then you proceed from there.

We didn't have the ability to jam a BREAK down the line from the remote console client system, so what gives? These were two subsystems that were part of a much larger rackmounted beast, so it's not like there was an old-school serial cable running between them. They were probably just traces on a board somewhere. Something didn't add up.

This is when I heard some really neat troubleshooting from someone who actually understood this stuff (i.e., not me): they knew that on other hardware, they had installed buffers between the two systems to keep the electrical low state at boot time from triggering the BREAK behavior.

Unfortunately, they hadn't done this same thing on this particular type of hardware. It was missing the necessary electronics magic (they called it a "pullup") to keep things from getting out of hand when the controller restarted. Oops.

Their solution was to disable that behavior in their getty config. Since the server was hardwired to the only client it would ever have, there was no reason for it to honor a BREAK to do that sort of thing.

That was it. The machines probably still have the same electrical situation to this day and send all kinds of wild crap down the line when their controllers reboot, but at least their gettys won't care.

If you're someone who's never done serial stuff on a vaguely Unixy box and you're bored over the holidays, maybe this is your time to check it out. Find a box with a serial port (good luck!), plop a getty on it, then wire it up to another box with a serial port (more luck to you!) and see if you can get them talking to each other.

Failing that, check out the magic of someone who already did that and then some. Enjoy!

If you've been doing the wifi thing for a while, you've probably followed the successive rounds of "security" that get layered on top. Back more than 20 years now, it was WEP, the so-called "wired equivalent privacy". That claimed to be 64 or 128 bits, but was closer to 40 or 104 due to the whole 24-bit "IV" thing, and a whole bunch of dumb problems with the crypto generally meant it was weaker than that in practice. Collect enough packets and burn some CPU power and the network is yours.

Then we got WPA, and then WPA2, and now the new hotness is WPA3. You might have noticed this last one in the settings of your newer network stuff and thought "hey, maybe I can benefit from it". Maybe you can, but a lot of it comes down to just how much you're willing to abandon.

Perhaps you have a thing for Raspberry Pis. They gained the ability to do 2.4 GHz wifi natively when the 3B came out, and picked up the 5 GHz band with the 3B+, so now you can have reasonable connectivity anywhere you can find power. The trouble is that the stock hardware and software absolutely will not do a true WPA3 network.

By "true WPA3", I mean a network that's only speaking WPA3 in SAE mode, which requires protected management frames (802.11w), and which does not support any kind of WPA2 fallback. This is a network that you can scan with something like Kismet and it'll say "WPA3-SAE" and nothing else. A stock RPi will absolutely fail to connect to them. This has been known for years and yet still persists.

If you spend far too much time digging around through the bug reports and forum posts, you may discover the angle of starting from the Linux kernel source, applying Infineon patches, fixing compilation errors, and then installing new Cypress firmware as well. Assuming you're willing to go through all of that, then yes, you may find yourself able to join it up to a WPA3-only network.

Wonderful. You now get to track this abomination of a kernel yourself, since you'll now be off whatever upstream decides to push out - security fixes, bug fixes, new features, or whatever else. Have fun!

In the Apple ecosystem, things are a little better. Support is pretty good for such things, and you should find that any Mac or iPhone made in the past few years should work just fine with a WPA3-only network. Even a first generation HomePod can handle it.

But, there's a catch, at least on the Macs. This assumes you are running in normal mode, i.e., booting from your SSD or whatever and running macOS in the usual way. If something happens to your machine and you need network recovery mode, it'll just fail to associate with the WPA3-only wireless network.

At that point, you'd better hope you have another network around that still has WPA2 mode available. Otherwise, you're kind of stuck. These (laptop) machines haven't had built-in Ethernet ports for many many years so that's not an easy option, either.

I should point out that if you get the bright idea to plug your ailing Mac into a Thunderbolt 3 dock with an Ethernet port with the intent of having it "phone home" for recovery mode that way, you will find that it does not work. It seems that whatever drivers are necessary to notice and/or use that NIC just don't exist in that world, just like how WPA3 support is also somehow missing.

If you have an old Apple Thunderbolt Ethernet adapter for some reason, and also have the requisite USB-C TB3 to mini-DP type TB2 dongle, then you just got lucky. That much will actually be recognized in recovery mode, and you can bootstrap into network recovery mode without standing up a WPA2 network.

Some day, these things will be fixed and this whole post will be a sour footnote in history, but for the moment I figured I'd warn people before they blew too much time trying to make this stuff work.

Let's try a thought experiment: we're going to design a little program that provides a service on a vaguely Unix-flavored box. It's designed to periodically source information over the Internet from hosts that may be close or far away, and then it keeps a local copy for itself and others to use.

You might have it use some kind of config file where it is told the hostnames of the servers it's going to access. Maybe you've set up a pool, such that any given attempt at resolving foo.service.example yields a different IP address every time, and there are bunches of them.

server 0.foo.service.example
server 1.foo.service.example
server 2.foo.service.example
server 3.foo.service.example

When would you make it resolve the host down to an IP address? It seems like you might want it to happen when your program starts up. Given the above config, it would find four entries, would turn that into four IP addresses, and then would get busy trying to sync data from them.

But, I haven't told you the whole story. What if you designed your program in a day and age where the network was just assumed to "always be there"? There was no such thing as consumer-grade Internet and home connections. You'd probably write it to do the name-to-IP resolution stuff once and then never again.

Consider what happens when a system with that design runs into the reality of running on goofy consumer-grade hardware with goofy consumer-grade Internet connections, raw crappy power from the local utility, and all of the other entropy sources you can think of. It's probably not going to behave well.

Such a system would start when the machine started and would attempt to get its IP addresses. Then it would take the success or failure and would use whatever it happened to get. If it got nothing, then that's it. It would just sit there staring at its own shoes for eternity, or at least until the next wonky utility power situation restarted the cycle.

This is what happens when you run ntpd on a dumb little consumer "router" for home Internet connections. Chances are good that both the router box and the cable modem, DSL bridge (or whatever else) will both restart at the same time. It's also a good bet that the router might manage to boot and start ntpd before the actual Internet connection comes up.

That means ntpd will find itself on a network with no routing to the outside world, and then it will try to resolve things and will fail. Then it will just sit there being useless until something or someone comes along and kicks it.

This happens on Unifi gateway devices, and it will bite you *right now* if the order of things happens to line up as described above.

So, if you find yourself with a machine that's attempting to run, say, systemd-timesyncd against a local USG or something like that and it's not syncing, you probably fell into this trap. Nothing in ntpd is going to wake it up and try to rectify the situation.

The Unifi + ntpd situation is effectively edge-triggered: the "rising edge" of the box starting up sends it off to do a bunch of setup stuff. If it works, you're good, but if it fails, you're screwed.

Let's try a different approach, then. You are a server. Your job is to talk to other servers periodically. You have been given some config directives to help you find them. Until you have "enough" servers to talk to, you keep trying to add more. This means attempting DNS resolution, and then if that succeeds, trying to talk to them and see if they are sane. If they are, then you keep them around and potentially use them as a source of data. If you aren't, you evict them and start the process over to get another one.

This situation is more of a level-triggered one. The system in question is going to keep trying to get to where it needs to be. It's able to start up in a broken environment and then eventually recover once the rest of the world starts doing its job again. It won't just go on vacation because everyone else hasn't shown up for work yet. Now, obviously it needs a little care because retrying in a tight loop is also bad. There's an art to doing retries (backoff, jitter, that sort of thing), and it also needs to be considered.

It's a big difference in how things work, and once you start thinking about systems this way, you'll start noticing all of the little race conditions and timing anomalies which trip up edge-triggered stuff in everyday life. Any time you've had to reset something in "the right order" or otherwise run something back through a series of other states in order to make it all "sync up", you probably were fighting with that.

Isn't it nice when systems know what they're supposed to be doing, and then keep working towards it until they succeed?

TL;DR use chrony.

If you read back through some of my posts from 2021, you might discover something which basically says "run an IRC network or get involved with the chatting on it, but try not to do both". This was a reflection on my own youthful stupidity, and a plea to others to not make the same mistakes (as many of my posts tend to be).

Imagine my surprise when I got to thinking about some of the recent bits of the tech / politics / malignant narcissist news cycle of late, and realized that it could apply there too.

"Hey (owner), what about X?"

"Raar!"

[keyboard furiously clicking in the background]

*service unavailable*

Sound familiar?

44 billion dollar tech companies: run them or use them but not both.

This one came in a request from a reader. They want to know my feelings about trying to "... avoid a company contributing to the downfall of humanity". This one's tough, particularly given my own history.

I worked for a web hosting company that had a dubious history of keeping spammers around far too long. Then while I was there, they had the so-called "adware" vendor. They got mad if you called it "spyware". I honestly thought it was random trash that people were installing on their own machines and so that's what they wanted. I only found out recently that it apparently was distributed by way of Internet Exploder drive-by ActiveX/whatever shenanigans. So, if you ran that cursed browser and landed on a page with their stuff in it, you got owned.

Now, that customer didn't last forever. They got whacked by AUP after a bit, but they were still there for a good... six months or so? And we definitely got bonuses in our paychecks when they upgraded their configs because we had managed to solve a bunch of their scaling problems. Yes, we made them more efficient, and *they got bigger* as a result, and those of us on the support teams directly benefited in a paycheck or two.

Then I worked for a place that was doing web search and had gotten into the business of providing free web-based e-mail that was pretty good. They had also started doing a few other things. They had a few simple *well delineated* ads on their result pages (and maybe a few other places), and that was it. Lots of people were like "you should go work there", so I tried it, and somehow I got in.

During my tenure there, they went and ate a company that I had a real beef with as a spam-fighting sysadmin for a bunch of users before the web hosting job. I'm convinced it's actually karma: eight years before, I had dinner with some people, including someone I had never met before. When I found out where he worked, I asked him something like "what's it like working for an evil company like Doubleclick". Yeah, I actually said that. *facepalm*

When the legalities of the merger were finished in 2008, I too worked for that evil company by extension. By absorbing it instead of killing it, we became them (see also: Collabra). The name was different, but the internal damage was done. This lead to all kinds of other crazy shit that came down the line, all in the name of fellating the advertisers, like Emerald Sea, aka Google Plus. That whole thing.

They were trying to do all kinds of crazy stuff, like you'd be browsing around and it'd say "hey, this looks like your Twitter page, so would you like to link it to your profile?" - and it's like holy crap, the company has crossed the line, then dug it up and set the pit on fire. Just because you CAN make a dossier on someone with your damn crawling infra doesn't mean you DO IT. That's where they were going. Full on creeper land, with the immense power of their infrastructure.

Then I decided to go somewhere else that (as far as I could tell) existed because people willingly put their data there. They uploaded pics and posted about going places and doing things. All of the data was sent to the site. The site didn't go out and scrape it off the web. I was okay with this. I didn't use the site myself, but I figured that made me the weirdo, not the (then) billion-something people who did. Clearly, they find it useful, so what do I care?

Of course, while I toiled in the infra mines at this company, all kinds of truly evil shit was going on, including the installation of a fascist regime in my country, the apparent genocide in at least one other country, and so on. It's like, someone even asked me about supporting the not-quite-UTF-8 language stuff they used in that country. Now I wonder exactly what all was enabled by virtue of being able to support that encoding! (Seriously, you know who you are. Is that what happened? Did that work let the bad people break loose out there?)

Then there's the joint which tried to look like they were all about smarter use of cars, but which probably added to overall congestion. They didn't want the key people who actually do the real work to be employees and went to the mat with heavy lobbying to make it happen during an election cycle. They also pulled out of a good-sized urban area in a very large state when the city put up requirements for background checks.

This is just the obvious stuff. I haven't even mentioned any of the "how they treat their employees" incidents from these places. Every company has at least a couple of these that I've actually witnessed, and far more that I heard about from trustworthy sources.

Sometimes I think about the fact that I've made some bad things more reliable so they can go about doing evil more efficiently, quickly, or just at all. It sucks.

I said this in 2013: "If your resources or reputation could be used to harm people, you owe it to them to jealously guard it lest it fall into the wrong hands." I still think this has happened too many times.

However, I no longer think that people are capable of guarding it to keep the vampires out. The only way to keep something with great power from being exploited might be to keep it from existing in the first place.

But what do I know, right?

Not all of my stories are long. I have plenty of shorter ones.

...

Did you hear the one about the company that posted a bunch of videos to advertise something, and the backend screwed it up? Somehow, one of them got mixed up with someone else's video upload.

They were trying to advertise a sports league. They got something else entirely.

"... the video was switched to a dolphin swimming in a pool."

But it gets better: the customer asked the response team to not fix it, and instead to leave it alone because "engagement with their [ad] was going up".

This sounds hilarious, but what if it hadn't been another public video? Then it would have been a privacy disaster.

Crosslinked videos? That's bad news.

...

Or, how about the time a rare lightning storm rolled through, and a bolt hit some of the nearby electric infrastructure? It knocked half the campus offline, and everyone was forced to find other places to be, including the chefs who moved dinner to the one cafe building which still had power.

Someone put "zeus" in the root cause.

This one is actually funnier if you worked there, because there was a real service called zeus, and it caused all kinds of outages once upon a time.

...

Backhoes love to find fiber optic cables. Sysadmins know that the best thing to bring with you on a trip into the wild (in which you may get stranded somewhere) is a length of fiber. That way, you can bury it, and when the backhoe arrives to dig it up, you can hitch a ride back to civilization with the operator.

One fine day, a backhoe found a nice fat fiber run somewhere in the world. The updates from the scene were not encouraging.

"Cable is thirteen feet down and beside a creek. Water keeps filling up the space. Working to find a shallower access point. In the mean time, a larger backhoe has been requested. ETA 30 minutes."

You know the line from Jaws? We're gonna need a bigger boat? They're gonna need a bigger backhoe.

I used to work at a place that sold ads. One of the things this company wanted was for the employees to try it out and see what it was like to actually use the ads product themselves. It's the usual "dogfooding" thing you hear about sometimes.

To that end, they issued a $250 credit every month. You just had to go to a certain internal web page and click a button, and it would credit it to your account. Every time the calendar rolled over to a new month, you could go click it again.

They told us all about this during our first day or two of classes - the infernally-named "onboarding". I noticed something during this: our presenter hadn't claimed their credit yet, so they went and did it for real right in front of us. They went to load up the page and it bombed - something in the code blew up and it didn't work. They reloaded it and then it worked, and they now had $250 of virtual ad money in their account.

Some weeks later, a new month started and I wanted to get in there and give it a shot. I went to start it up, and it blew up, just like what happened in my class. But hey, this time I had a computer of my own, and access to the source code, and even a tiny bit of experience poking at frontend stuff courtesy of some of the introductory tasks they assigned to new employees. Why not take a whack at it? This place is supposed to be all about fixing random stuff even if it's "not yours" - the "nothing is someone else's problem" posters all over the place implied it, at least.

I loaded it up on my dev environment and got cracking. Sure enough, something was wrong with it, and the first time through, it would blow up. It was something dumb like the code was throwing an exception but the exception handling path was making the wrong sort of log call so that would then blow up the whole request. I fixed the logging so we'd actually get to see what the exception was, and that'd give us a chance to fix any real problems. Simple enough, right? I sent the change to the last person to touch the code... who had just touched it that morning, oddly enough. They thanked me and it was applied.

Then I tried to get my credit, and this time it blew up again, but now it logged what was wrong. I could see this on the dev environment. It was something about calling some function with the wrong number of parameters.

The code itself did something like this:

if (condition) old_func(a, b, c, d, e) else new_func(a, b, c, d, e);

The problem is that new_func didn't take 5 arguments. It took 4. I read through the code and found that it didn't need a "d" argument any more, and so I just changed the arg list to (a, b, c, e). I figured it was a simple oversight by the person who had just changed it.

Then I ran it for myself, clicked the button, got the "your credit is now in your account" message, and was pleased. I asked a friend to try it too and it worked for them as well.

It turned out this very if-then-else part was what had been added that morning, and so I again sent that person the code for review, and they again thanked me and accepted it. I went off to go do other not-frontendy things, and the code went out to the internal web servers a little while later.

A few hours later, someone reached out online: we have to turn off the ads credit thing. It's giving away WAY too much money. How much? Twenty-five thousand dollars. $25,000. Not $250.

What happened? The thing had been passing the credit amount as pennies to "old_func", so it was passing in 25000, because 25000 pennies is in fact 250 dollars. But... new_func took dollars, not pennies. So, 25000 in that context was 25 thousand dollars!

I had been at the company something like six weeks and had changed a line of source code to fix a bug (logging), to uncover another bug (wrong argument count), to enable yet another bug (wrong units, and zero type safety) that gave 25 grand worth of funny money to anyone who clicked! And I had clicked! And I got a friend to click! And other people got it too!

What happened? They just turned off the feature until they could fix it. Those of us who had way too much credit in our accounts turned off our ads so as not to actually consume any of the "bad money", and kept them off until they reversed it out of our accounts. Then we were clear to go back to dogfooding.

And no, nobody was fired for this.

This is yet another reason why I say bare numbers can be poison in a sufficiently complicated system. If that function had demanded a type called "dollars" and the caller had another one called "pennies", it simply would not have passed the type checker/compiler. But, this was before those days, so it sailed right through.

Here's another story from way back that somehow hasn't been told yet. It's from fairly early in my days of working tech support for a web hosting company. I had been there less than two months and one night, things got pretty interesting.

It was around midnight, and our monitoring system went crazy. It popped up hundreds of alerts. We noticed and started looking into it. The boxes were all answering pings, but stuff like ssh was really slow. FTP and SMTP and similar things would connect but wouldn't yield a banner.

Someone realized they were all in one datacenter, so we called up networking to see what was up. They said none of their alarms had gone off. So, uh, great.

Somehow during this, the question of DNS was raised. One customer's box was grabbed randomly, and my usual "run w after login" was showing the IP address of the support network's external (NAT) interface instead of the usual nat-vlanXYZ.company.domain thing (which comes from a PTR). That was weird, and it was an early hint of what was wrong. Running DNS queries from there also failed - "host" this, "dig" that. Even forcing the queries to the two recursive nameservers that customers were supposed to use from that datacenter didn't work.

Next, it was time to see what this had to do with the daemons. With tcpdump running, I'd poke port 25 and watch as sendmail (or whatever) lobbed a bunch of queries to the usual DNS resolvers but got no reply. This would go on for a while, and if you waited long enough, it would eventually stop attempting those lookups, and you'd finally get the usual SMTP banner. The same applied for other services which worked more or less like that.

This made me suspect those resolver boxes, and sure enough, they couldn't be reached with traceroute or ping or really anything else for that matter. Our manager called down to someone again and told them what was happening, but somehow they didn't get on it right away.

Some time passed, and the customers started noticing - the phone calls started and lots of tickets were being created. Eventually, someone who ran the internal infrastructure responded and things started clearing out.

This was my first time in that sort of situation, and I regret to say that I participated in a "maybe it was a transient error, since things seem fine again now" response storm. I mean, it *was* transient in that it happened and then it stopped, and things DID seem fine again then, but it just feels so wrong and dishonest.

One interesting customer during all of this had the whole thing figured out while it was still going on. They managed to do the same thing we had, and noticed that both of the recursive nameserver boxes for that datacenter were toast. I'm sorry to say they got the same form-letter response.

What's even more amazing is that this customer came back with "hey cool, it's up now, just wanted to mention it in case it helped out", and was mostly shocked by the speed of our response. I guess we "got away with it" in that sense.

I found out much later that they were just two physical boxes for that whole datacenter, and apparently they had no monitoring. Nice, right?

Now, telling the customers that? That would have been epic, and it would have started a conflagration of credit memos the likes of which hadn't been seen since the time our friend "Z" "slipped" with the cutters while cleaning up the racks.

I was new on the job. I went with it. Later on, I would try to find ways to convey information without resorting to such slimy tactics. Mostly, I tried to get out of tech support, and eventually succeeded.

So, to that specific customer out there those 18 years ago, you were right. To all of those technical customers out there who think they're being told a line, this is to say that sometimes you are in fact being told a line. They're afraid of sharing the truth.

If you're in a spot where you can tell the truth about what happened and not get in trouble for that, even to a customer, consider yourself lucky.

...

Side note: the reason the alerting blew up was that the poller would only wait so long for the daemons to send their usual banner. The daemons, meanwhile, were waiting on their DNS resolution attempts to fail. The monitoring system's poller timeout was shorter than the DNS timeout, so when DNS went down, everything went into an alert status.

While this was going on, the Windows techs (as in, the people who supported customers running Windows boxes) were giving us grief because "only the Linux boxes are showing up with alerts". Apparently the Windows machines didn't tend to do blocking DNS calls in line with the banner generation, or timed things out sooner, or who knows what, but it allowed their monitoring polls to succeed.

"Windows boxes are fine... seems like Linux can't get the job done!"

They were mostly joking about it (as we tended to do with each other), but it was an interesting difference at the time.

Just a short note: I'm not Mosquito Capital, and I'm not sure who it is. I can make some educated guesses based on the lingo, and I'm pretty sure I worked with them in the past.

It doesn't really matter who they are though, since what they are saying is spot-on and you should pay attention if you're in this kind of business.

Go read. It's good. Be sure to hit "more replies" when you get to #29.

Today, I'll share a story about someone from marketing who was trying to make things happen and really stepped in it. It goes like this: a bunch of us engineer types are invited to a meeting with someone from marketing. We don't know the first thing about marketing in general, or even the particulars of it at this company. But they want to chat and hey, it's an excuse to not "work", so why not go see what they want?

We'll call this person A. A gets up there and it's pretty clear they're the sort of loose, goofy and hopefully funny... almost hippie type... you've heard about. It makes sense. You need to be creative for that job, and they're the creative type. So far, so good.

A proceeds to tell a story about this big "tent-pole" movie that was coming out. All of the nerds were going to want to see this latest movie in a series that went back a couple of decades. Plus, it'd successfully slopped over into non-nerd life, such that a bunch of regular people would also be excited about it coming out.

Marketing saw this coming and proceeded to reach out to the company that worked on the movies. They wanted to do a partnership where the company's product would change for the weekend of the movie's opening. Instead of having the usual little icons for whatever it is they did (dog walking, pizza delivery, you know the type of company), there'd be icons of characters and certain well-known and well-loved vehicles from the franchise zipping around.

All they had to do was get it written and out the door in advance of the movie. The marketing folks sat down with the engineering peeps and laid it out. They were told right then and there "no can do". Even though they had all kinds of time in which to do this kind of "theming" of the app, they knew they couldn't do it. That's how bad it was at this company.

Anyway, A told us this story, and used it as a basis for asking if anyone could help solve the fundamental disconnect that kept the company from doing cool things. It was basically a "this is why we can't have nice things" story, but at the same time it was a cry for help. Since they were at the mercy of the software people who had to add the actual icons and toggles and features into the app (and backend stuff), they were stuck.

Towards the end, A told the audience a story about what happens if you have a product that might be good in theory but has terrible marketing. As part of this part of the talk, they put up a slide. You might've been wondering what the hell was with the image I put at the top of this post. Well, that's what was on the slide.

For people who are using text to speech or otherwise can't see the image, I'll describe it here: it's a bog-standard conference room with a projector screen, and the only thing on the screen is "clownpenis.fart" - like a URL, only not. There are also some white blocks where I cut out a few people who would otherwise be identifiable (including the speaker).

When I saw this, I didn't get it. My own reaction was along the lines of "heh" from my inner 12 year old and also "what?" and "I guess I'm missing the reference". Someone quickly informed me that it was a callback to a Saturday Night Live skit that aired around 2000. I had stopped watching SNL by then, and so that's why it was off my radar. Easy enough.

I had to go back and watch it - you can still find it online if you're interested. It's about an investment firm that is solid and has been around forever, but is only now (2000, remember) "getting online" - establishing a "web presence", if you will. They waited too long and so got the "last domain name available" - the aforementioned clownpenis.fart. This ran on national TV, albeit late on a Saturday night some 22 years ago. (And yes, that's the voice of Jerry from Rick and Morty.)

Basically, the presenter was trying to get people to connect the dots by using something funny that they had seen in the past. It didn't work. Oh, did it ever not work.

For the next month, all you could hear about in the company was that A did the wrong thing, and shouldn't have said that, and they should apologize, and what are we going to do about this sort of communications issue, and so on.

Personally, I filed it under "stupid", not "hostile", but that was me at that point in time. I had plenty of "buffer space" in my life.

What's kind of amazing is that nobody ever mentioned the "we can't have nice things because eng is fundamentally unable to do the simplest, stupidest adjustments to our 'chrome' which would surprise and delight our customers". That went COMPLETELY unnoticed and was largely forgotten.

The company continued to be unable to have nice things.

I've been pondering a hypothesis for a while now. It has to do with the way that groups of people cluster together, particularly when certain topics are involved. This is a simplification for the sake of trying to better understand the things I witness every day.

First, there are the actual states you could be in:

(A) I understand the situation.

(B) I don't understand the situation.

Then there's the way you are taking it:

(1) I accept that I am in whatever state that I am in.

(2) I don't accept that I am in whatever state that I am in.

Put these together and now we're talking quadrants. Four possibilities. Which one are you?

A1: This person understands, and accepts that they understand it. They might be living it (whatever "it" is in this case) and have no choice in the matter. They just *are*.

B1: This person doesn't understand, but they accept the fact that they do not in fact get it. They can work at understanding things at their own pace. They might also appreciate that they may never fully "get it" if they don't actually live it. They're usually not the problem.

A2: This person understands, but doesn't accept that they actually do. This is basically impostor syndrome. This person can be counseled and treated with care, and sometimes might be "brought out of their shell" to start believing in themselves. Also not the problem.

B2: This person doesn't understand, and can't accept it. They usually lash out at other people as a result. Even reading things like this enrages them, and they turn to their local hive of other insecure people to yell into the echo chamber. They're usually too stupid to realize that lashing out identifies them *clearly* as members of this set. They could just shut the fuck up and then they'd at least LOOK like a member of some other set. Tiny in membership, but appears way bigger by way of being incredibly loud assholes. Also possibly tiny in member-ship.

And, sure, there are other dimensions. There always are. Life is complex. A reasonable person can read this approximation and not be threatened by it.

Shut up, B2s.

...

The thing is, I came up with this to talk about tech work. There are some people who are terrible at this stuff, but are okay with that. Then there are the ones who are terrible, *and* they are giant assholes about it. You can spot that second group really easily because they will self-identify any time a sufficiently skilled individual does something and it crosses their path. It's narcissistic injury on their part. Go read up on it.

It's kind of useful since you have this group that makes things terrible for other people, and there they are, jumping up and down, waving flags and screaming to get attention because they will never get attention for excelling, since, well, they can't. It's like, gee, you don't have to kick out most of the people. Just get the few assholes in that group - they'll even help you find them! Your group will be far healthier for it.

Again, I came up with this some years ago in private to try to get a handle on why a handful of people at companies turn into such giant problems for other people, and how they can multiply if they are allowed to remain in a group setting. You know, like certain web forums.

It just so happens to also work for some other things in life. If you got through the first part and thought I was talking about one of those things, then you probably agree without even realizing it. How about that.

For the record, I'm still here. I'm still pissed off at the state of affairs in my country, and I'm still taking my usual summer break, so there's been no writing.

I appreciate that several people have checked in, naturally.

As for the anonymous rando that said something about me going back to a certain broken pink mustache factory, well, shit, that's funny.

No, that's not happening. I haven't talked to anyone from there (other than friends who are still there, naturally) in months. Nobody's ever reached out about "coming back". I have no desire to go back. That's the stuff of nightmares, like you're back in high school and can't find the exit door. *shudder*

Besides, they couldn't afford me, anyway.

As for my eventual return to regular writing, I'm not so sure I want to just put things online and open to everyone any more. For one thing, fueling the idiot mixer that is HN and usually reddit is not one of my goals in life. I'd rather put up some kind of wall ... or something ... and make it so that casual haters can't just come in and read stuff.

I'm not talking about what happens when you click through to jwz.org from a HN referrer, but I certainly understand *why* that's there.

I'd rather continue reaching the good people in the world to provide stories, entertainment and technical data while starving out the Friday night crew that has nothing better to do than unload on the Internet.

For some people, this has meant "Patreon". I'm not sure I want to do it that way, but something's got to give here. I'm done being a free resource for those who would serve to make life miserable for half of humanity.

In response to recent changes in the look and feel of the site...

Q: Did someone complain about the old design with the blue and white?

A: No. I did this in response to ridiculous bullshit going on in my country. We've rolled back to 1972 in terms of basic human decency, and I'm not happy about it. Obviously. So I've forced the design back to a terrible place as well.

Q: Why not go whole-hog and make it look like a real terminal with phosphor effects and run it at 75 baud?

A: This is not a fucking game. I am not celebrating 1972.

Q: Can you fix the colors so I can read things?

A: No. I want you to feel the pain here. It's small and stupid but it's where I am right now. I'm one person with a keyboard. What do you want me to do? Just pretend everything is okay? Everything is NOT OKAY.

Q: Is this all you're going to do?

A: Probably not. I see no reason to continue helping people who would treat others as a piece of meat, or worse.

Q: What does that mean?

A: I don't know yet. Wait and see.

Amber on black. You want 1972, you get 1972.

Some readers have written in with questions about my photo geotagging post from last week. One common question is whether the place name has ended up in the file's metadata somehow. The answer is: I don't think so. I did an "export as original" on the photo in question and ran it through a bunch of exif dumper tools and didn't find anything that suggested a name like that.

The EXIF data looks like this:

Create Date         : 2014:06:16 13:02:15.202
Date/Time Original  : 2014:06:16 13:02:15.202
GPS Altitude        : 5.2 m Above Sea Level
GPS Latitude        : 37 deg 29' 6.13" N
GPS Longitude       : 122 deg 8' 53.30" W
Circle Of Confusion : 0.004 mm
Field Of View       : 57.2 deg
Focal Length        : 4.1 mm (35 mm equivalent: 33.0 mm)
GPS Position        : 37 deg 29' 6.13" N, 122 deg 8' 53.30" W
Hyperfocal Distance : 1.89 m
Light Value         : 15.4
Lens ID             : iPhone 5 back camera 4.12mm f/2.4

(Side note: that's an interesting term, huh? Today I learned.)

Anyway, it's just a bunch of numbers, as you would expect. Something in the actual Photos app on the Mac and the equivalent thing on my phone is translating it to a name.

What's kind of nutty is that the same picture still shows "Facebook - Headquarters" when viewed on my phone. Really. Check it out:

So, not only is there some mapping going on, but the phone and the computers (both of them) are looking at two different sources of data. I have to assume the phone has it cached, while the Macs must have flushed it and picked up the new value in recent times.

Or, who knows, maybe Apple is running multiple backends with disjoint geographical data sources. It wouldn't be the first time they had terrible map data, right?

So here's another fun problem: how do you do a "fourth dimensional" geo-tag (that is, adding a time system) without revealing all of the places a person's been and when they were there? In other words, how do you do that without compromising privacy?

The best I can figure so far is that you'd send back a list of ALL of the place names for a given area and let the device figure out which times apply to which photos, and just discard the rest. Also, it should probably be "zoomed out" pretty far, such that only very coarse bounds are given to the server. Just return all of the mappings for all of the polygons or whatever inside some giant swath of space, and do all of the nitty gritty stuff on their device.

Otherwise, hey, it becomes pretty easy to track people after the fact.

I like finding inspirations in unusual places. Several summers back, a friend shared a link to a story about apt-get starting Dwarf Fortress (!) when they tried to upgrade packages, or something like that. The general consensus was that it involved something intending to invoke df (as in /usr/bin/df, the command on the box) and getting the game instead of the "disk free" info. It was one of those things where you read about it and just groan, but you don't dig in because it's not biting you directly and you have your own problems.

I gave it my usual eyeroll because it fell squarely into the bucket of "running a subprocess to get info when a library call will do". It also had a bonus garnish of using something that looked in the PATH and so could be mislead into running the wrong thing somehow.

This chat lead to the inevitable topic of that time someone wrote an uninstaller that did something like "rm -rf $THING/", and sometimes $THING would be empty, so fun things happened. This generally put us in the mindset of "how can things go wrong", and "how can they can *be made* to go wrong, if you wanted to do that". The fact that a PATH was coming into play suggested system() or popen(), both of which run through a shell and thus introduce the possibility of injecting your own commands.

So there we were with a large corporate source tree sitting there in front of us. It contained lots of C++ and C code, besides other things. With all of the people writing code over the years, had anyone managed to make the same mistake? It wouldn't exactly be hard to grep it for popen, and so we did. Three minutes later, we found this "network helper" service that ran on the whole fleet, and what did it do?

FILE* output = popen(command.c_str(), "r");

Well, that's fun. What's this? It's listening to the network, and accepts RPCs? It'll run traceroute and ping for us? And it takes parameters like "destination"? So, all we had to do was find a host running it, fabricate a RPC with a destination including a little bonus, like "127.0.0.1; touch /tmp/0wned", and fire it off.

It happily ran touch and dropped the file in /tmp, thus proving we could get it to execute anything we wanted as that user. Oh yeah, and the file was owned by root, because this thing was running as root. Score! Root anywhere you want it!

With that, we now had a security incident to handle. On that particular day, the usual security peeps were busy some other thing that had broken early that morning. It turned out that this "feature" had been running in prod for a year, so letting it chill for a few more hours didn't change much. We let them put out that fire and then handed them the next flaming dumpster full of "win".

So, somehow, a share of a reddit post that linked to something on Ask Ubuntu turned into a way to get root on any machine in the company. That's the kind of free-association stuff that would happen. Good times.

I used to work at Facebook. That was both the name of the service with all of the cat pictures and the name of the company that paid me every two weeks. The cat picture part still has the same name, but the parent company does not. It's called Meta now. I left well before any of that renaming happened.

As a result, I have plenty of pictures that predate that change on my Macs and other Apple devices. They're mostly geotagged, but something curious has happened to them: the name of the place has shifted. It's no longer "Facebook HQ". It's now "Meta - Headquarters". This is what it looks like in the Apple photos app:

Given that this picture was taken in 2014, it was clearly not Meta back then. I know, the usual "well actually" people are warming up their keyboards right about now: "it used to be called that, and it's the same company, so it's fine" and so on and so forth. I don't like that, but okay, whatever, let's say we accept that for the moment because it IS the same company with a different name. Companies do that all the time.

What if that picture of the ground had been taken in that same spot in 2004? Would it still make sense to call it "Meta - Headquarters"? I hope you wouldn't say that. Back then, that space was inhabited by Sun Microsystems, a company that very much is not the same as Facebook. (This is well before Oracle ate them - that was 2009-2010.)

What happens in another couple of years when Meta is the next smoking crater in the tech landscape and then some other company tries to become the next unicorn in the mud flats of Menlo Park? Or how about a couple of decades past then when that whole area is underwater? Will my pictures say something like "San Francisco Bay"?

This why I would say that perhaps we need some time bounding on these hyperlocal place names. Now, I realize this is no small thing. It's one of those big-O blowup factors, and that's annoying for all involved. Still, if you take the very long view on these things, something is going to have to happen eventually. Otherwise, our grandkids will have pictures that we took that make no sense at all.

There should also be some actual humanity applied here. Place names are non-trivial, and many of them have captured a large amount of hateful and just plain ignorant behavior. That's why you can't just automatically build up a list of "this place was called this at this time". It needs people in the loop to make thoughtful decisions about how to handle the more interesting ones.

Case in point: Palisades Tahoe. I also have pictures that I took there many years ago. I am more than fine with them being rendered with its current name. I know it wasn't called that when I was there. Give it an asterisk if you must, but really, even that probably isn't needed.

That's what I mean when I say that we should be careful about this.

June 20, 2022: This post has an update.

Not everything at a big company involves writing code. Sometimes, you have to do a bunch of meta stuff in order to manage the projects. These are two stories of doing exactly that.

...

Have you ever seen what happens when a team has a long list of "to-do" items and has no idea where to start? Have you had to sit there and witness the circular conversations of people trying to attach priority values like P0, P1, P2 to those items? Did they then still have no idea where to begin?

"AB-123 is a P0..."

"Oh yeah, definitely" "Yep" "Absolutely is"

"But we can't do that because XX-996 isn't done yet..."

"XX-996 is a P2 so it can't be done before a P0..."

... sound familiar?

I went through something like this indirectly. There was a nearby team which was closely affiliated with what I was doing but wasn't plugged into my management chain. They were stuck on this sort of thing, and their manager asked if I would join them in sorting out their priorities.

This was one of those situations where you had a bunch of people trying to improve reliability for a service. For whatever reason, they couldn't crack this particular organizational nut. This is what ended up happening in the meeting room that day.

Since we had everyone in the same room at the same time (a luxury these days, I admit), I just asked if I could take a whack at the problem, grabbed a marker and headed for the board. I asked for someone to give me one of the things they wanted to see happen. It went up on the board with a short yet memorable title and had a circle drawn around it. I asked for another, and it too went on the board somewhere else.

This kept going until we had 10 or 15 items up there that people thought the team should be doing for one reason or another. Then I started from one end of the board, picked an item, and asked if it depended on any of the others. It turned out that this one did in fact have a dependency, so it was "blocked" by some other item on the board. I drew an arrow from the first one to the second.

This kept going until we had visited all of the "nodes" on the board, and we had something interesting to look at. There were definitely some nodes which were blocked by all kinds of other things, but there were also several which were blocked by nothing. This was completely independent of whatever P-this, P-that values people had been attaching to these tasks. I hinted that their priorities don't mean anything if you can't make progress, so take the ones which are possible to do right now, and go work on them. If you keep doing this, you will eventually get to your supposed P0 and it will happen.

Really, the situation unfolded like a giant Makefile to me. This is where you say, okay, to build my_project, I need to build httpclient.o and logging.o. To build httpclient.o, I need to have libcurl on the system. I don't have libcurl on the system yet, so I can't build httpclient yet. But I *can* build logging, since it doesn't depend on anything. So you can either work on logging, or get libcurl going. Or, if you have two workers, do both at the same time. Hooray.

To me, the "priorities" were largely ornamental. It seemed like you might use them as a tie-breaker. For example, let's say you had one chain of dependencies blocking a so-called P0 task (highest priority, supposedly), and another blocking a P3 (good luck). You might try to put more effort into getting the former done rather than the latter, and maybe start on that first. It's a hinting system, not a mandate.

...

There's another fun thing a few of us did at that same job around the same time. Several like-minded people had decided we needed a better way to ship the corporate-built system software which ran on every machine in the fleet - the "Widely Distributed Binaries", if you will. Three of us decided to just go for it.

Since nobody had really done this before, we knew there was no way we'd ever get a "road map" laid out up front. We knew what success looked like, but we also knew that we'd have to adapt to the conditions encountered along the way. For that reason, we started off by shipping binaries by hand, then kept notes of what parts of the process were awful. Once we thought we had a handle on what it was and how it could be made better, we'd automate that step. Then we'd move one level up the stack and do it again. This continued until the thing pretty much ran itself... and we had a halfway-decent list of "stuff you need to have to ship WDBs sanely".

I've talked about that before, but what I haven't shared is how we dealt with the "oh, what about if we did this thing" ideas that came up along the way. We came up with a method of handling those, and I like to think it was quite humane and didn't make any promises we couldn't keep. Here's how it worked.

Since there were three of us working on the project, and our unixnames (which were also our IRC nicknames) were three different initials (A, B, and R), we used that to our advantage. We'd hop on IRC and chat about things, and as we came up with ideas, we'd add another item from our own set. That is, my first idea was R1, then R2, then R3, while Mr. B's ideas went B1, B2, B3, and Mr. A's ideas were A1, A2, A3, and so on.

This let us all pitch ideas out there while giving them a unique identifier so they didn't get lost, and without having to coordinate with each other in order to not collide while incrementing the counter. All of the ideas went onto our Wiki page, and that's about as concrete as they got for a while. We were then all able to see what everyone else was thinking about, look for duplicated ideas, overlap, and generally look for things that would help (or hurt) other things.

None of these items were commitments. They were simply ideas. Also, just because you had the idea didn't mean you had to do anything with it. The only reason we tagged them the way we did is so they could be uniquely identified, assigned in a non-blocking fashion, and so you'd know who to ask for clarification if the gist of it wasn't clear from the wiki page and/or IRC chat logs.

"Hey B, on B5, where were you thinking about storing that thing?"

"Yeah I wanted that in svn."

"Oh yeah right, that makes much more sense to me now, thanks!"

... you get the idea.

At some point, we started stitching them together, such that A1 might depend on A5, but then A5 depended on R2. After doing this for a while, we ended up with several different types of list items.

Some items didn't block anything and weren't blocked. You could pick one and go on hack on it at any time, but at the same time, you weren't going to unblock anyone or anything else by doing it. It might make for a nice afternoon of work on a self-contained project.

There were items which were blocked by one or more other items. They tended to be rather large.

Then there were items which themselves blocked one or more items.

The ones which were blockers but which were not themselves blocked were the obvious places to start working on stuff if you were looking for something to do. This is because if you started on something that was blocked, you'd hit the blocker! Pretty obvious if you think about it, right?

By drawing this out as a bunch of chains (think labeled circles with arrows pointing at other circles) courtesy of the Wiki's built-in "dot"/graphviz renderer, we could just look at the ends of them to figure out what was possible to crank on *right now*. We could also see which items would have the most impact in terms of unblocking future work.

At the point that someone decided they were going to do something, we'd create a task for it. A "task" was the unit of currency in this company's bug tracking software. That meant it got a number and a nice little shortcut URL, and so the wiki would be updated to link to it. Anyone who was curious could follow that link into the tasks tool to see what was up.

I will note that the graph had been arranged to show things with different colors and rendering styles based on whether they were blocked or not, and whether they had been completed or not.

From time to time, we'd go through and prune back the graph in order to remove the nodes which had been completed and were just taking up space. Watching things get struck through and then disappear from the list and the graph felt great.

One important bit here: some of those items never left the list. For all I know, they might still be there on some corporate wiki page buried inside the belly of that company. Those items were just ideas that someone had one day, but as the service went into production, it turned out we didn't need it that badly, or had the wrong idea altogether. Then, that was the end of that. Some of them were actually struck through on the list without being completed along with a note explaining why, just so we didn't have to keep thinking about them.

This system of doing things made me happy because it didn't leave us holding a bag full of promises. Just talking about something didn't commit us to spending time on it later. By disconnecting the talk from the commitment, this gave us the freedom to talk about all kinds of stuff without worry of somehow ending up with tons of stupid and useless items down the road. We did still make commitments, of course - that's what the tasks were for.

How do you keep from having a soul-crushing load of tasks/tickets/whatever that will never go away? You don't create one until you are damn sure that it needs to exist.

Caveat: if you have so many things going on that you run the risk of "losing" something due to it only being a line item on a wiki page somewhere, you probably don't want to try this on your project. (You probably also want to run away from that project, since there's far too much going on there.)

I've received some comments from readers asking me about my distribution choices, and specifically what I thought about the various flavors of BSD. It seems like a fair question. It's been a long time since I ran any of them for any "serious work", and it's likely much has changed. In any case, it seemed like a good opportunity to mess around and see what's out there.

So, over the span of about a week, I tested out the latest offerings: FreeBSD 13.0-RELEASE, NetBSD 9.2, DragonFly BSD 6.2.1_REL and OpenBSD 7.1. This all happened in identically-configured VMs on a Macbook I have that's just kind of sitting around not doing much otherwise. The primary goal was to build as much of my C++ code that I could. The secondary goal was to try to rig up an X environment that looks and works like what I tend to use on my normal workstation: fluxbox, urxvt, bash, a couple of specific (and ancient) Dock/Wharf/Slit/whatever widgets, and so on.

I had actually tried this out on my own about a year ago and things didn't go too well. This time, things are a lot better. I have a bunch of random dumb things or amusements to share.

Now, since I'm talking about operating systems, and then potentially comparing them to each other, I run the risk of bringing out all of the people who have nothing better to do but smear poop all over the Internet on a Friday night. So, I'm going to steal a trick from another post: I'm going to list a bunch of things that are strange or just broken, but I'm not going to say which one it was. If you've used it, you might recognize it. Otherwise, eh.

One of the installers is really worried about hard drive geometry. It's odd to be asked about such things in 2022, to be honest. I'm talking about the whole "cylinders, heads, sectors" thing that you used to do. (I'm a little surprised it didn't ask me to type in a list of badblocks from the sticker on top of the drive.)

One of the package installation tools that you're supposed to use to bring in extra stuff (like, you know, bash...) has this giant security warning when you install it. Both the binary and its shared library use tmpnam. It looks like you can race it to the actual path, so that's not great. It's marked as obsolete and there are other ways to accomplish the same goal. It's not a good look when you're getting started and this shows up. You just set your first foot into the jungle and heard "click".

There's an installer which uses its own color scheme which involves a background color other than black. That's fine. But, when it's installing things, it clears the screen... and it clears it back to black. Then a moment later it repaints it as that other color and starts putting text on it. This means every time it flips between files (and this happens a lot), it flashes. Blink. Blink. Blink.

One system said I could only have an 8 character username. It's like, personally, that's not a problem, but I have had many users back in the good old days of running multi-user systems who would not appreciate that. 1996 called and it wants its 8-character username restriction back. (At least it's not using DES for the encrypted password with that wonderful 7 character limit, eh?)

At least two different systems installed dependencies which don't compile with -Wshadow (a flag I usually have enabled for my builds). In both cases, the dependency conflicts *with itself*. One of them is protobuf, and the other one is boost. This is the fault of whoever wrote that code, NOT the port maintainers, but it's still goofy and annoying. Since I use -Werror, I have to compromise on something in order to build when this kind of thing is present.

Also in the realm of dependencies from ports, a fair number of these things ship pkg-config hint files (blahblah.pc) that miss out on stuff you actually need to make it compile and/or link. This is stuff like providing "-L/some/dir/foo" for the direct libs and not providing "-L/some/dir" for the dependency libs. This is also usually not the fault of the port maintainers.

Case in point: mariadb, which is what you use if you want "mysqlclient" type behavior. It relies on -lcrypto and -lssl, both of which are from OpenSSL. It's a port, so it's in a non-system path too. You have to do -L/top/level/lib to pick them up. But, you *also* need to do -L/top/level/lib/mysql to make -lmariadb work.

I'm able to deal with all of this due to how my builds work, but anyone using a build system which believes that 'pkg-config --libs foobar' is the gospel truth is going to have a bad time.

GNU Radio is particularly bad about not telling you when it needs -llog4cpp or -lboost_system. This is *not* specific to any of the BSDs. They also get this wrong on Debian and a bunch of other places. You're better off using "ldd" on any GR lib you intend to link to in order to be sure you have everything covered. Or, you know, keep trying to link and add -l flags for what's missing every time it yells, and keep going until it stops yelling. Clowntown, party of one, your table is now ready.

One of the installers offered a nice "auto" disk setup system, and then promptly sliced up the disk space a bunch of different ways. Everything was fine until maybe 30 minutes later when it was time to install GNU Radio. It was just trucking along until... it ran out of disk space. What happened? The path for ports was given its own filesystem for some reason, and it was 4 GB.

Meanwhile, on that same system, two other filesystems were given 10 GB and 3 GB and had ZERO utilization. This makes no sense. ports are going to take up tons of room since very little comes from the system itself, and you have to augment it.

I found myself having to either go back and start over, or figure out how to wrangle a bunch of existing partitions to reallocate the space. I wound up punting, since it's not worth the effort.

There's this vmware-user-suid-wrapper thing which pokes the X server to tell it how big the "screen" is. This is what you need to make it follow the size of the VM's output, like when you resize the window, push it to an external display, pull it back onto the laptop's native screen, or whatever. If you have this running, then it Just Works. Otherwise, you have to figure it out yourself and hack xrandr calls into your X startup and your xdm/whatever startup, and just never resize or move things. Most of them have it. One does not. If you're running as a VM and not on real hardware, this might get annoying.

It's easy to download the wrong file for one of them. It's a disk image instead of a bootable ISO-9660 image. I'm not even sure when you'd want the first one. You'll probably download it (since it's first on the list), wonder why you keep getting the PXE boot screen on a VM, then double back, *facepalm*, and download the right one. Hope you have a fast connection.

There's a system which asks if you want it to autoconfigure the network, and then it happily sets the hostname to "localhost". It then asks you if everything is okay. If you say no, you don't get a chance to fix the one bad part. Instead, you start over where your choices are effectively "do everything by yourself" or "take everything we autodetect". I said, fine, you can be localhost. Have fun with that.

One of the installers put a ^A in the search domains in /etc/resolv.conf. I still have no idea where that came from, or what it means in that context. I've certainly never seen that before. It's quite a surprise when you first see it. I didn't keep it running long enough to see if it started spewing bad queries at the root servers like some broken configs have done in the past.

It's kind of funny to see which systems let you su from your newly-added local account and which ones balk because you're not (yet) in group wheel. It's like, oh yeah, /that/ whole thing. Better log in as root and fix your account, peasant.

Speaking of su, once I switched my personal account to bash, I also twiddled PS1 to put some hostname/path info into my prompt. This is pretty important when messing with a bunch of machines in parallel so you don't apply a change to the wrong one. On this system and this system only, doing "su" kept my PS1 around... and the root shell displayed it literally. It looked mighty ugly. That's one way to get me to do "su -", I guess, but... really?

Filesystem choices seem to have a lot of magic associated with them. They tend to ask you questions for which you probably have no good answer, at least, not as a new user, or at least one who hasn't used it for many years. There's also one that steers you into a specific recommended filesystem and then gives you two different warnings afterward. One of them is essentially "small filesystems can fill up very quickly". Part of me goes "yeah, and water is wet". The rest of me (battle-scarred from far too much Unix use) goes "oh great, another ENOSPCfs, that'll run out of metadata or something, just what we needed".

Keyboard selection is just bizarre in some cases. Quick, do you want emacs, iso, pc-ctrl, or unix?

In terms of my goals, it went like this:

Primary: build personal C++ code: mostly done. Some of the GNU Radio ports didn't have full hardware support like gr-osmosdr (for rtlsdr sticks, among other things) and gr-uhd for USRPs. I have both kinds of hardware, so the systems without one or the other wouldn't be able to use it unless I went and built support for it myself. Considering that was the status quo pre-Debian for me, that's not a massive problem. It's just not as clean as having someone else do it for you in a nice neat binary form.

I had to turn off -Wshadow in a few places to work around some dumb broken libraries that conflict with themselves.

One of the systems has a build of gcc which hates "non-trivial designated initializers". This basically means you can't do the new fancypants { .foo = bar } C++ stuff on a struct unless you initialize ALL of the fields. This is the code which gave it fits:

struct sockaddr_in in = {
    .sin_family = AF_INET,
    .sin_port = htons(port),
    .sin_addr = in_addr
};

What's wrong? Well, this system has other members in their struct sockaddr_in, and I'm not mentioning them in that initializer. I could explicitly mention them and set them to something, but then this code would fail to compile on other systems where those members don't exist (they aren't in POSIX.1). Not great.

One of them is already on GNU Radio 3.9 where they've done another round of API changes that broke compatibility with existing stuff. It's all stupid minor stuff, like boost::shared_ptr turning into std::shared_ptr (yay) and gr::fft::window taking over some types from gr::filter::firdes. It's relatively simple to fix, but it also means that the same code won't Just Build across all of the systems. Welcome to #ifdef hell (or worse).

If the stuff I had to do didn't cross paths with the missing hardware support, I'd probably be fine on that system. The "turn off -Wshadow" stuff is a harder pill to swallow. I'd probably end up hand-hacking those headers to make them less goofy so I could switch it back on.

Secondary: run my usual X setup. Also largely OK. One of them doesn't have wmcpuload so I'd have to figure out how to make that work on there. This can be annoying at times, since every system has a slightly different way of figuring out just how busy the CPUs are. It's really nice when someone else has gotten there first. I'd probably do without it until one of those lazy afternoons rolled around where literally nothing else needed to be done.

The missing vmware-user-suid-wrapper thing in that one case is mildly annoying, but if I'm running on a VM, I've already compromised on some of my sanity. It's on me to get onto native hardware to get rid of the hacks and unavoidable jank.

Conclusion? All systems have bits of weirdness. Most of it stems from the varying degrees of "hands-off-ness" which applies to basically anything that's not in the (relatively tiny) base systems. Installers are all over the place. Some of them could probably benefit from setting some people loose who have no previous context with their systems to see where the rough spots are.

With enough practice, you can fix any of these problems or otherwise "get in front of them" so the system doesn't get mis-installed the first place. The question is whether you want to. Is that you what you wanted to spend your time on?

I could use any of them in a pinch to get my work done. Lacking such a pinch, however, I choose not to. That says more about me than them.

I've been mentioning Raspberry Pis in a few of my recent posts. I keep finding weird things in these systems. The question is: why am I suddenly wrangling these odd little boxes? The answer involves a story of heating and air conditioning.

First, I need to back up and tell a story about something that happened perhaps 10 years ago. I heard a terrible noise coming from the inside unit where the blower is. It sounded like water draining out, and I half-expected the ceiling to just open up and flood the place. That didn't happen, but that was the end of my hot air. From that point on, it would no longer produce heat for the space.

I reported it to the maintenance people at that building. They actually told me "to push the up button". You know the button on the thermostat that makes it go from 70 to 71 to 72? The actual target temperature? Yes. They told me to push that... and did no further investigation. The fact that it had made a terrible noise right before stopping had no impact on them. The fact I had been operating that system normally for five years at that point and knew how a damn thermostat worked had no impact on them.

I finally had to plead my case to the admin staff, and they came out with actual HVAC technicians. I wasn't there when it happened, but I found out later that there was an actual mechanical type problem with the thing. In other words, it was nothing the thermostat would have ever been able to "fix".

I could have put that thing on 90 and it would have just blown the air around. Obviously. I knew that, but they refused to take me at face value until I went over their heads. WTF? (I mean, I know exactly why. But if I put it in writing then a bunch of 12 year olds are going to attack me. So I'm going to talk around it and about 50% of the adults in the room will know what I'm talking about. It's absolutely true.)

Jump forward to a few months ago. I have a different space in a different building, and this time it was the air conditioner not doing its thing. I don't want some kind of crap happening again. Now, granted, this time it'd be a "down arrow" thing, but I'm not having that. I wanted hard data that they could not refute.

To that end, I obtained several of those "weather station" type wireless sensors. Normally people get them when they buy a so-called "atomic clock" (it's a longwave radio, thank you very much) "weather station" that shows both the inside and outside temperature and may in fact synchronize time from a 60 kHz transmission. You park one part (with the big display) inside, and the other part goes outside.

The outside parts tend to break a lot, so there is a thriving business selling replacements. I picked up a few of them and got busy. One of them was unapologetically zip-tied to a vent. Another one was parked right next to the system thermostat, and a third was placed outside.

There's this program called rtl_433 which will use a cheap $20 SDR (software defined radio) USB stick to receive and decode signals. I took that, convinced it to emit some output that wasn't entirely terrible, and got to wrapping it for my own purposes. Then I installed it on a pair of Raspberry Pis and put a RPC server on top.

Why multiple Pis? Well, a couple of reasons. First of all, I wanted some diversity in my radio receivers. By putting them in different spots, I could probably get a good decode from one even when something kept it from reaching the other. It also lets me update, upgrade or even reboot (!) them as long as I do it one at a time. The Raspberry Pi systems just sit there listening to the (433 MHz) radio, decoding whatever they can. If it looks like a sensor, then it keeps that info in memory and remembers when it heard it. Then, if something queries it over the network, it coughs up all of the data. Each sensor has an "id" and "channel", plus the actual temperature and humidity values, and finally there's an age value.

My Debianized Mac Mini runs another piece of this system. Every 15 seconds or so, it checks in with the Pis and asks them what's up. In theory, both of them will have the same data set, but in practice, it can be slightly different. This is fine. It knows this will happen, and it just keeps the newest sample for all of the sensors it actually cares about. Now you know why I have that "age" field in there!

And yes, "sensors it cares about" is important. Since this is an unlicensed band and these things are rather popular, my radios frequently pick up other plausible-looking transmissions from nearby sources. They aren't consistent, but they do exist.

If we have good data that isn't too old for sensors we care about, then it flushes those data points as rows in a Postgres database on the Mac mini. Then it goes to sleep for another 15 seconds or so. Easy enough.

Getting to this stage quickly was important. Any temperature variations you don't measure and log are gone forever. Any data points you don't get off the air are gone forever. Any time you aren't polling the "radio server" on the Pis, those samples disappear forever. The most important thing for me was getting a long stream of uninterrupted data so I could make my case.

Why measure both the room and the vent? That's easy. When the AC is on, there should be a considerable drop from one to the other. An insufficient drop means either the system is broken, or it's somehow so incredibly hot outside that it can't bleed off any heat when it pumps through the coils out there. That's why I have the third sensor outside: it tells me what the temperature is right here at the building - not at one of the airports, and not in some ideal case. It's right here buried in the same urban "heat island" that the outside half of the HVAC system is in. (And no, the outside sensor is not in a spot where the outside system can influence it.)

With just this, I could show that there was no drop at all despite the (hyperlocal) outside temperature being totally reasonable. It should have been giving me cold air. It wasn't. It was all there in the numbers. I guess they realized they had to take me seriously, since I received a plan to replace the entire unit. This would involve ripping a giant hole in the ceiling, pulling the existing unit, then putting in a brand new unit.

Now, this happened during these "supply chain" shananigans, so this took *months* to receive, and all the while, I was just sitting here, logging away, busily collecting data points. I eventually got tired of looking at raw logs and then later, running SQL queries, so I started on some visualizations.

My first approach was to write a very simple web page that basically did the 1995 "meta refresh" trick. It would IMG SRC a CGI program. That CGI program just hit up the database, asked it for the last N hours, rendered it as a graph in a PNG, and then shipped it to stdout. That gave me a nice graphical view with all three sensors using the same scale, and it was easy to see what was (and wasn't) happening.

That worked okay, but it was annoying. Reloading the whole page 1995-style meant it flickered as the whole thing came back in every time. It had a fixed width and height, and it basically only worked on my usual web browser window on my one machine. If I loaded it from anywhere else, then it looked wrong.

It was a really crappy renderer, but it was a start. It looked like this:

That's what the space looks like when it just "freewheels". My guess for the periodic spikes is a defrost cycle on a nearby freezer, but I never bothered to prove that conclusively. As for the bigger cycles, that's just whatever happened with people coming and going, the sun shining on the windows or not, and things of that nature.

Aside from some small improvements (like vertical bars for the hours), that's about what I ran for a couple of months. Then, I got this weird notion one day: what if it was rendered client-side to a canvas in JavaScript? That would let it adapt to whatever size the page happened to be, and it could figure out how far back to go while maintaining a reasonable density - that is, how many seconds cook down into each horizontal pixel?

So, when I mentioned I was doing stuff in JS a while back, that too was not an exaggeration. I was in fact writing that, because there's really no choice in the matter. If you want to do this kind of stuff in a browser, it's this or nothing.

Anyway, this is what things have turned into (showing outside):

The fun part about this is utterly abusing the web server on that Mac Mini by grabbing the corner of my browser window and whipping it around. That thing repaints all over the place and generates bunches of requests for freshly aggregated data at the new settings. I haven't bothered rate-limiting or debouncing any of it since it's just me using it, and I can generate all the requests I want.

You can't see it from this second screenshot, but I even went as far as to do some mouseover magic so it will set the TITLE of the canvas to the temperature value at whatever X-offset I'm over. So, if I spot some weird peak and want to know that value, pointing at it and waiting a moment for the tooltip will answer that question right away.

The actual replacement happened a while back, and the space is now being managed properly yet again. I haven't stopped monitoring it, because, eh, why not. It's still fun to look at, and besides, it could happen again.

...

In terms of the moving parts here, it looks like this:

Two Raspberry Pis: one 3B, one 3B+, that I just had hanging around. Stock Raspbian installs, albeit with a whole lot of "WTF is this? Buh-bye" removals having been applied. I've tossed a whole bunch of packages that had no business being on there.

Two RTLSDR sticks that I also had hanging around: one per Pi.

rtl_433, which is available from apt as "rtl-433". It's configured to spit out JSON since that was the least obnoxious output I could get from it. (It's still annoying. Ask me about numbers vs. chars for sensor channels sometime.)

My own "thermo_server" which does the pipe/fork/dup2/exec thing to wrap rtl_433, and then sits there parsing output and storing it in memory in one thread. Then my existing RPC gunk serves that data up to authorized clients. It uses jansson to chew on the JSON since the code to use that already existed from other projects.

Over on the Mac Mini: it's a Debian box, as mentioned previously. It has postgres and Apache. It also runs my "thermo_logger" which knows to go poke the "thermo_server" processes over the network (with the RPC gunk) every so often. Then it flushes usable data to the database: INSERT INTO x ... whatever. Easy enough.

A chunk of HTML and another blob of JS that generates requests to the server and renders the data points as a reasonable graph looking thing. There's also a bit of CSS to make it render just so.

A CGI program unimaginatively called "data" which actually takes those requests from the JS callout, hits up the database, and then throws it at the requester. It too uses jansson, because JSON, because web browsers. It's basically the one place where it makes a little bit of sense.

...

And so, yeah, there it is: I wrote a temperature monitoring system to keep from being treated badly by maintenance people. Funny how that works.

I've been using my Raspberry Pis for some data collection purposes, and noticed some of them doing a few things which seemed suboptimal. There's this process called "rngd" which really likes to wake up every so often and announce stuff about what it's been up to. It's usually about every hour, and it tends to push a dozen lines to the syslog, none of which I care about.

So, on one of them, in the interest of not polluting the logs, and not burning the SD card with tons of writes that add zero value to my life, I decided to turn it off with "-S0". Per the manual:

Controls the interval in which statistics are output to syslog. Normally every hour, the maximum being one week (604800 seconds). Set to 0 to disable automatic statistics logging, e.g. to avoid hard disc spin-up in embedded systems (sending a SIGUSR1 signal will still cause statistics to be logged).

I did that back in February, and indeed, it stopped logging. I was pretty happy. I left the other ones alone for the time being.

Well, a few hours ago, I was poking around at these things, and noticed that rngd on that one was running about 15% CPU while it wasn't doing anything on the other ones. This kind of thing irked me, so I attached to it with strace, and this was my reward:

In short, WTF, over. Why are you spinning over and over and over calling sleep... with no delay? Check that out: tv_sec is the number of seconds to sleep. tv_nsec is the number of nanoseconds to sleep. Calling it with a struct where both are set to 0 will make it return right away, unsurprisingly.

Why would you do that? Then why would you keep doing it? And why would you keep doing it forever, such that you had 15% of the system's CPU?

I went to attach to it with gdb to find out exactly where this call to sleep was coming from, but there was a problem. It seems that nobody in Raspbian land cares about having debug symbols for their packages. Normally in the world of Debian, you can stick something extra in your apt sources file, do an update, and you'll have a whole new world of -dbgsym packages available to you. If those exist in Raspbian, I sure can't find them.

So I had no symbols beyond those for glibc itself. I dumped a core anyway, knowing that I'd probably wind up killing the process at some point and would want to come back to it later. I'm glad I did, because it ended up helping solve the mystery later.

The next hour or so was spent trying to reproduce this in a version I built from source. It wouldn't go into that loop. Instead, it would start up and go into a week-long sleep. The few spots where it called sleep in the source didn't seem to be causing this. I got really annoyed with this and was rather close to calling it a night, but then I decided to start over from first principles.

The original process was long dead so I could no longer grab it with strace or gdb, but honestly, neither of those would have anything new to tell me. I did still have the core file, so that much was still there, and I managed to load it up to see this:

Notice that the thread in question is what gdb calls #1, which is LWP 422, aka PID 422. If you look at my strace from before, all of those calls to sleep are coming from pid 422. This told me that it wasn't one of the other threads causing the badness, but rather the main thread.

All of that noodling around with running it from source had taught me that the program's main thread just spins up some worker threads and then goes into this loop where it calls sleep and maybe does some stats stuff. It looks like this:

        sleepinterval = arguments->stats_interval ?
            arguments->stats_interval : STATS_INTERVAL_MAX;
        sleeptime = sleepinterval;
        while (!gotsigterm) {
                sleeptime = sleep(sleeptime);
                if ((arguments->stats_interval && sleeptime == 0) ||
                    gotsigusr1 || gotsigterm) {
                        dump_rng_stats();
                        sleeptime = sleepinterval;
                        gotsigusr1 = 0;
                }
        }

Looking at the code, sleepinterval gets set from a ternary expression that either takes the value of stats_interval if it's nonzero, or it sets it to STATS_INTERVAL_MAX which turns out to be 604800 seconds. That's a week worth of seconds, and that 604800 was showing up whenever I ran the one I had built from source. No surprise there.

Given that strace and gdb had proven that the looping code came from main, and this is where main ends up, how could this possibly sleep for 0 seconds? After all, it resets it inside that janky-looking "if" down there inside the loop, and it sets it to sleepinterval, and sleepinterval is 604800, so all good, right?

Well, no, actually.

It's around this time that I noticed that it's not a bare call to sleep. It's actually using the return value from sleep. I don't see this very much and missed it at first.

	sleeptime = sleep(sleeptime);

This whole thing. So... for us to ever sleep(0), then sleeptime would have to be 0. But we always reset it higher, don't we? No, we don't.

if ((arguments->stats_interval && sleeptime == 0) ||
	  gotsigusr1 || gotsigterm) {

Unrolling that branch goes like this.

Is arguments->stats_interval true? It's set to 0, so no, it isn't. We don't even look at the rest of that first one as a result. gotsigusr1 and gotsigterm are also not set since we didn't get a signal. None of those OR possibilities are true, so we DO NOT take the branch. We don't dump_rng_stats, and we don't reset sleeptime to sleepinterval.

That means sleeptime is going to stay set to whatever it was when it got set coming out of sleep. Could it be set to zero? Absolutely. The return value from sleep is "how long you had left to wait when it exited for whatever reason". This is because sleep can be interrupted, and this way you could restart it. Worse is better, PC losering, all of that stuff.

But what happens if you reach the time you asked it to wait? Well then, it returns zero.

So... all we have to do is get all the way through that sleep without getting one of those signals, and it'll return 0, and then never get reset, so it'll get called with 0 again, and again, and again, and again, ................. in a very tight loop, burning CPU the whole way.

The trick, and the reason why it's so damn hard to reproduce, is that you have to wait a full week for that first timeout to hit! Yep. Imagine that kind of patience.

I do not have that kind of patience, so to test my hypothesis, I lowered STATS_INTERVAL_MAX to 30 seconds and started up the new build in strace. It sat there for 30 seconds, and then promptly started chewing CPU in a very tight sleep loop.

So, yeah, if you run Debian's rngd with -S0, it'll be perfectly fine for a week or so, and then it'll go into a tight loop that'll tie up one of your CPUs from then until the process is stopped for some reason. Lovely.

If you've tried running it this way on your Raspberry Pi for similar card-preserving reasons, you might want to go look and see if you have a rngd that's eating a core. If your uptime is more than a week, I bet it is.

Sheesh.

...

Side note: this is the first thing I saw upon diving into the code. I was looking for places that might call sleep, and found a function called random_sleep(). Check this out.

         gettimeofday(&start, NULL);
         while (!gotsigterm && timeout_usec > 0 &&
                      poll(&pfd, 1, timeout_usec) < 0 &&
                      errno != EINTR) {
                gettimeofday(&now, NULL);
                timeout_usec -= elapsed_time(&start, &now);
                start = now;
        }

Right off the bat, I saw "gettimeofday" and thought "I picked the wrong week to stop something something". They're calculating elapsed time... with the WALL CLOCK. That's what gettimeofday() gives you. If you want a monotonic clock, you need to use clock_gettime() and specify the right one. This is just asking for trouble if the system clock gets changed while it's running.

Put it this way: it's a good thing they explicitly check for timeout_usec being a positive number, since if you pass poll a negative timeout, that means "infinity". Neat, right?

I think I'm going to stop there before I find something else.

Okay, wow, my post about an empty file has generated a lot of feedback, mostly for the throwaway line at the bottom about an empty file being the smallest version of /bin/true possible.

First up, Rob Pike wrote in with a link to a tweet of his. Thanks, Rob! That's exactly the kind of feedback which blows my mind: when someone who Definitely Knows shows up and provides the authoritative answer.

A bunch of other folks wrote in to either mention the tweet, link to the tweet, or pin it on the lawyers (and, in one case, GNU). It all works out to be right, so everybody wins.

Garrett wrote in with a link to a great story about another zero-byte file from the world of CP/M called The Infinitely Profitable Program. (Look for the part about "go.com" if you are impatient.)

I also heard from the author of this piece about POSIX true(1) who pointed out some of the references, including one by John Chambers called The /bin/true Command and Copyright. So, it seems people have put some actual work into this figuring out just how it all came to be.

...

One person mentioned that there have been disagreements in the Linux realm about filesystem operations, atomicity, and dropping zero byte files all over the place. I definitely saw some of this in the job in question. I've had this story on the back burner for a while, and now seems like the perfect opportunity to share it, so here we go.

Besides zero-byte files, another problem we saw was that machines would *mix up file contents* when they got rebooted during write operations. This would usually manifest as RPM crap being written to files that had no business containing RPM information.

I found out about it from a different angle. One of the teams I supported owned this "proxy" (really a cache, but the name stuck) that ran on all of the boxes. You'd ask it to get config items for you, and it would connect to the config backend store and would set up "watches" on them. Then, whenever it changed, it would feed you the update.

At some point, someone decided to make the "proxy" remember what had been explicitly requested on the box, and so they had it dump out a list of the items requested, one per line, in a *plain text file*. This worked okay, but then it crossed paths with the filesystem corruption monster. Their file picked up all kinds of binary crap that had no connection to anything in the config store.

The next time the machine came up, it started their "proxy", and it tried to do the "warm up" mode. It read through that file, and every time it saw a newline in that binary crap, it treated it as a path. Then it created a thread to go and load it from the backend. Now, since this was absolute garbage, there's no way that fetch would ever succeed. They assumed that only valid paths would show up in here, so they had these things retry until it did succeed. And, of course, there's no way a bunch of gibberish will succeed, so the threads didn't go away.

Now for the final rub: every one of these threads made the process that much bigger. This is the VSZ you see in tools like ps - not physical memory, but rather the overall *virtual* size of it. It can be many times bigger than the actual physical allocation (RSS) of a process.

In this case, for some reason, some well-intentioned individual had tried setting a limit on how big the program could get. They used ulimit (aka setrlimit) to do this. The problem is that RSS is *NOT* one of the things you can control with u/rlimits. There's a setting which looks a lot like it would work for such things, but it limits *VSZ* only.

So, when all of this finally got smooshed together, you had a "proxy" that would come up, try to start tons of threads that never went away, would balloon its VSZ, and then the limit would kick in and it would start failing memory allocations. That is, C++ would throw a "bad alloc" exception for things like "new", and the program would die because nobody ever expects *that* to fail.

I never found out just how much cross-pollination was happening between files on that company's machines. For all we know, it may have resulted in cat picture #1's contents being written to cat picture #2's file, or vice-versa. That would be a pretty epic privacy failure, wouldn't it?

As best I can recall, it was something crazy involving the kernel's attempts to flush its buffers to the disk when shutting down. Somehow, buffer A wouldn't necessarily flush to file A, and so on down the line. There are undoubtedly people who remember this and know exactly what happened, but I am not one of them.

Fixing all of these warts involved making the kernel not mix up its buffers, and actually removing the well-intentioned but ultimately useless "ulimit -v" that someone had placed in the startup script. As for the plain text file, well, I tried pitching them on making it into something strongly (and not "stringly") typed with framing or some other way to detect corruption, but the devs didn't go for it. Sigh.

Yesterday's post about "pipefail" also involved some systems which treated an empty file as a valid file. This turns out to be surprisingly common. I think people might be shocked at exactly how much you can get away with, and how many different things will accept such a beast.

One of the failure modes I used to see with certain package-based systems at a job with a large number of computers is that sometimes it would just fail to populate it with actual data. This might've been due to a reboot during the package install process, or maybe the packager got killed for some reason, but it meant all kinds of badness would happen.

In particular, I saw times where it would drop a file on the disk in the right place, with the right permissions (readable + executable) but totally empty! Then, later on, something or other would ask if the package was installed, and as far as the package database was concerned, it was. Also, then the cron job would try to run one of these binaries... and it wouldn't fail. Nope, it was perfectly happy running exactly nothing.

Yes, if you weren't already aware of this, an empty file that's executable will totally "run" to "completion" and will even yield an exitcode of 0. If you are checking that exitcode for success or failure, it's going to think everything is just fine.

No, really, look:

linux$ touch empty
linux$ chmod a+rx empty
linux$ ls -l empty
-rwxr-xr-x 1 rkroll rkroll 0 Apr  6 12:07 empty
linux$ ./empty
linux$ echo $?
0

Awesome, right? This works just fine on Macs as well. I don't have quite the assortment of shell accounts on other flavors of Unix that I did back in the '90s to test further, but I have to assume it'll probably work most of the time. It's probably something funny involving how the interpreter is spun up - looking for #! and that kind of stuff.

I mean, what would the error be, anyway?

Does this mean the smallest /bin/true is actually an empty file?

Here's a bash shell scenario for people who like these things. Let's say you have a script like this:

#!/bin/bash
reader | writer

... and "reader" fails and yields no data, and then "writer" runs and writes a nice fat blank to the database system. This then causes all kinds of fun times down the road.

So, this gets used as an opportunity to write one of those awful corporate outage bulletins which is more about hawking the company's wares than getting to the bottom of a problem. And, in it, the key is "we forgot pipefail".

It's like, you forgot more than that. pipefail isn't going to stop the second thing from running. Seriously. It's still going to run. Just look.

Let me introduce our cast of characters here. First, we have "writer":

#!/bin/bash
echo "writer is reading stdin"
cat > output
echo "writer is done reading stdin"
ls -l output

It's simple and stupid: it says hello, inhales its data, and tells us what it got. It simulates whatever "read stdin until EOF" you'd have in this scenario.

Next up, here is "reader-that-works":

#!/bin/bash
echo "here's some data."

This one is nice and easy.

Now here's its buddy, the one that's going to fail (and exit 1), "reader-that-fails":

#!/bin/bash
exit 1

No surprises there. Now, we have the script that'll simulate the happy path called "run-good":

#!/bin/bash
set -o pipefail
./reader-that-works | ./writer

And finally, the script that simulates the unhappy path called "run-bad":

#!/bin/bash
set -o pipefail
./reader-that-fails | ./writer

Note that both of these runners have pipefail enabled already.

Let's run the happy path.

/tmp/duh:$ ./run-good
writer is reading stdin
writer is done reading stdin
-rw-r--r-- 1 rkroll rkroll 18 Apr  5 13:51 output
/tmp/duh:$ cat output
here's some data.
/tmp/duh:$ 

Okay, so, now the other one:

/tmp/duh:$ ./run-bad
writer is reading stdin
writer is done reading stdin
-rw-r--r-- 1 rkroll rkroll 0 Apr  5 13:52 output

What's this? The second part of the pipeline still ran? Of course it did. It's *already running* at the point that the reader fails. Its stdin is hooked to the stdout of the other thing, Unix-centipede style. It *has to be there running already*, or the reader couldn't run in all situations! It would fill the pipe of its stdout and then it would block.

This isn't MS-DOS, where foo | bar involves running foo, sending the output to a temp file, then running bar and getting the input from that same temp file. There's parallel execution here, and it's way too late to abort a pipeline if only two processes are involved.

This like is one of those "interview questions" people ask about "being the shell" and trying to understand how it does what it does. One snag they throw at people in these interviews is why you don't want to have a shell command that greps for something in a file and then writes back to that same file at the same time.

That is, something like this (don't do this):

grep -v noise my_file > my_file

You think "that'll remove 'noise' from the file. And it's like... it will... sorta... by removing *everything* from the file! The shell will happily open my_file for writing as part of setting up the execution environment for the subprocess and will clobber whatever's there.

...

And, you know what, the worst part about this is that none of this knowledge should even apply. The fact we're talking about shell scripts for something critical means that the battle for reliability was lost a long time ago.

Apple's AirPlay is something that's useful when it's constrained to a person and their own devices and the devices of their visiting friends (who have been granted access to the host's network). It lets them send music and/or video to a speaker or TV and have it display for the benefit of everyone.

Unfortunately, at least the Apple TV does this goofy thing where it also (by default, I'm pretty sure) advertises itself to anyone nearby. This means if you are in a relatively high-density environment like an apartment building, a condo tower, or just have one of those houses with paper-thin walls and no side yards, you are probably going to see your neighbor's devices.

Now, while it's kind of "fun" to occasionally pick their device and make it do the HDMI-CEC thing to turn on their TV to a "enter passcode" screen in the middle of the night, it's also obnoxious to have to skip over them in the list of devices. When you're on a Mac, there's this dropdown bar, and you have to make sure you pick the right one. Quick, there are FOUR "Living Room" devices. Which one is yours?

But wait, there's more. This is the part I want to rally some of my more snarky readers to go and try. It looks like our devices try to connect to the IP address of the foreign device, just in case we happen to be on the same network as it.

I want someone to get an ATV, configure it with some "interesting" IP address (although, I don't even know what that would be in 2022, honestly... it's not like the old days), set it up in that mode where it offers itself to anyone nearby, and then get it near a whole bunch of Apple people. Maybe wait for the next WWDC or something.

Anyway, in theory, ALL of those devices will start trying to establish TCP connections to that IP address on port 7000... over the Internet. Do it with enough devices nearby and it'll look like a DDOS, or something. Maybe that'll piss off enough people to make something happen.

Alternatively, you could now realize you have a way to find out the (external, if behind a NAT) IP address of an Apple device near you. All you have to do is convince it to try to connect outward to a real host you control, and then see who comes a-knocking. All of those weenie advertisers who were trying to collect marketing data based on MAC addresses doing wifi and BT scans will be absolutely drooling over this.

If you've ever watched the traffic from your Apple devices leaving your network for the Internet and wondered why in the hell it's trying to connect to RFC 1918 space you don't even use... this is probably why. It's almost certainly something one of your neighbors uses, and your devices are trying to hump it repeatedly.

Stupid.

I have an ulterior motive in writing this. Maybe Apple will finally do something about this so we can just see our own devices in the AirPlay chooser and not have to dodge a bunch of lookalikes. It's like, come on already, I realize they must all live in mansions where nobody else is nearby, but the rest of us are packed in like sardines and get tons of stupid extra devices showing up constantly. It gets worse every Christmas.

C'mon Apple. Stop the AirPlay insanity.

A long time ago, I tried categorizing the people you encounter in tech support as one of three major types: the ones who investigate problems and come up with solutions, the ones who use those solutions in response to problems coming up, and the ones who think they're an investigator but are actually terrible and screw things up on the customer's machine. Pretty soon, it's full of awful crap all because this person really did not understand what's going on with anything, and just barreled on ahead anyway.

It's this "let's just run around doing crazy stuff that is totally not indicated by ANY kind of data" thing that brings me to today's story.

I was eating dinner. My phone started ringing, and I let it go to voicemail. Whatever it was could wait. Later, I checked on things, and sure enough, it was the new "webmaster" from a gig where I was the sysadmin. His message started out in a most remarkable fashion: "both box1 and box2 have crashed", continued with some more blather, and then added that he had rebooted box2 (with the reset button, that is).

box1 was the mail server for the whole operation. box2 was one of the web servers. Either of those being down would be a big deal. Still, I didn't buy it. These things didn't just die like that, particularly at the same time.

I calmly walked back to my home office, flipped over to my "work" session on box1 that always stayed logged in, and hit ENTER. It was alive. I called him, and he asked "what about box2?", so I loaded up a web page on that box in Mozilla (yeah, this was a long time ago...) and it was fine. I poked box3 for good measure. It was also fine.

Later on, I got an interesting e-mail.

Sorry to have bothered you about that so-called outage. There's something strange going on with network 1. While I was getting an IP address on network 2, I asked (name) for a fixed IP for my workstation on network 1. He gave me x.x.x.16. It connects to box4 and box5, but not with box1 and box2. I used x.x.x.1 as the gateway. I went back to dhcp and everything is reachable again. I'll ask (name) about it tomorrow.

Yes, this person did something to his workstation that ended up changing its IP address, and upon trying to connect outward from it, failed to reach some of my machines. Because he was physically present and I wasn't, he got to one of them and pushed its reset button on the front panel. The box got shot in the head for no good reason. Poor little box.

What happened is that he had been given the "poisoned" IP address. Once upon a time, (name) had set up this network scanning thing that kept filling up my logs with crap. I asked (name) to knock it off, since I actually watched those things and tended to act on the signal it provided. Crapflooding my logs made it harder to find the actual crazy stuff like infected client systems (and worse).

So, after he failed to leave my machines alone, I just filtered out his scanner on box1 and box2. He could shoot packets at me all day long and they'd just drop into the bit bucket. It's like, cool, cool, have fun with that. I bet you don't even notice. I had set it up nearly two years earlier according to my notes in the firewall config file.

# (date two years in the past): idiot box that won't stop scanning me
DROP * x.x.x.16 * * *

When this "webmaster" got a hold of the IP address, the two machines which had existed back when this happened were still dropping it with a very simple and stupid iptables rule. Any machine that had been "born" past whatever point they stopped the scanning never got the rule because there was nothing to drop, and that's why box3, box4 and box5 didn't act the same way.

Let's see. You have physical access to a bunch of servers that are not yours. You do something to your workstation. The next thing you know, you can't reach some of those servers from that workstation. Do you undo what you did to your workstation? No. Do you find another machine? No. Do you ask someone else to also try hitting it? No. Do you try to hop into a machine that IS responding, and then try to poke one of the "dead" machines from it? No. Do you notice the difference between a host that truly is down and one that is just dropping your packets, i.e. ICMP host-unreachables from the router versus... you know, nothing? No.

What do you do? You let yourself into the server room and start pushing front-panel reset buttons thinking it'll do something useful.

It takes a certain kind of individual to go and do things like that.

Have you ever wondered where the time goes during a "laptop" coding interview? This is one of those things where you bring in your own little laptop computer (or, they supply one, I guess) and you are expected to solve some problem on the spot. They give you maybe 60-90 minutes to do this, and they want to see it work, and have tests, and all of that good stuff.

The interviewer hands you a piece of paper and may or may not discuss it. Then they set you loose and disappear for a while. It's just you and your little machine, and maybe some music if you thought ahead.

Where does the time go? The last time I went through it, it went something like this in my head:

(looks at paper) huh, okay, so, what are they giving me here. I have to go get some text files from some Dropbox thing. Okay, get on their guest network, go scarf down their files. Two input files, two output files. I'm supposed to take the input and turn it into the output by running some algorithm they explain on rest of the sheet of paper.

The file is line-oriented, such that the first line is a number, and that says how many data lines will follow. Then they have that many more lines, each with two numbers split by a space. This didn't come from a DOS machine so it's not CRLF, at least. It didn't come from an old-school Mac so it's not a bare CR, either. It all ends with just a line feed, typical Unix stuff.

Right, so, I have to figure out some method to solve the actual problem they gave me, and then I have to write it. But, I also need all of the stuff required to actually inhale this file and get the data to run through that "engine" I end up writing. Joy.

So I guess I can just write something terrible to use fgets to read through this and give me a bunch of lines, then chop it up in some really terrible way, mash it through strtoul or something to make values out of them, and then probably call into my "engine" with the two arguments. Joy.

Right, so, let's get going on the base level stuff and hope the algorithm starts setting into place while I do this. I kinda-sorta have some idea for how to handle it but will need to try it and see how it goes, and I can't do that until I get something to actually throw numbers at it.

Let's get to it, then. Hum, well, I don't have any of my personal library stuff here, so I guess I get to code it all up the old-school way. Also, what are they going to be building this on? I have no idea where they are in terms of compilers. Let's assume C++98 just to be safe.

Okay, need to get the file, then read the file, so...

int main(int argc, char** argv) {
  if (argc != 2) {
    fprintf(stderr, "usage: %s <file>\n", argv[0]);
    exit(1);
  }

... and of course this means we need stdio.h for fprintf and stdlib.h for exit right there, so add those #includes at the top.

Keep going. Open the thing and yell if it's not there. I normally have my own LOG stuff but not here. Failing that I'd usually fprintf something to stderr, give it a message and a format string and then jam strerror(errno) into it, but time is short. perror is yucky but it's faster, so screw it, use that.

  FILE* f = fopen(argv[1], "r");
 
  if (!f) {
    perror("open");
    exit(1);
  }

And now we need errno.h, too. Go #include that.

Time to get that first line. Let's just read it and totally ignore the fact that it could EOF on this read and we might not get anything, because, again, pressed for time here. Crap, this is going to suck.

And, uh, crap, fgets leaves the newline on the end, so I have to tie that off. Again, I guess I'll just assume I got something more than an empty line here and assume I can subtract 1 from strlen without going into the weeds. This is exactly why I have all of this fancypants read-file-to-line stuff at home that was so painstakingly written!

  char buf[1024];
 
  fgets(buf, sizeof(buf), f);
  buf[strlen(buf) - 1] = '\0';

Dirty. So dirty. And now we need a number from that. strtoul would be the right sort of thing, then all of the error checking you're supposed to do, but that's a ton of work, and again, none of my home library set is here to save me (which already has that work done in it). Screw it, atoi, and hope nothing comes along which doesn't work with that.

  int input_lines = atoi(buf);

[ Time check: it's already taken me 17 minutes just *writing this post* to get this far down. I'm going based on my notes and the code from that day several years ago. Tick, tick, tick. ]

Better say what I got for debugging purposes. Use stderr so it doesn't corrupt the output.

  fprintf(stderr, "XXX input lines: %d\n", input_lines);

Now I guess it's time to start a loop that reads this crap and splits it up and stores it somewhere. So, uhm...

  map<int, int> input_jobs;  // key: start time, val: run time
 
  while (fgets(buf, sizeof(buf), f)) {
    if (strlen(buf) < 1) {
      continue;
    }
 
    buf[strlen(buf) - 1] = '\0';
 
    char* sp = strchr(buf, ' ');
 
    if (!sp) {
      fprintf(stderr, "bad line (no space): %s\n", buf);
      exit(1);
    }
 
    *sp = '\0';
    char* len_str = sp + 1;
 
    int start_at = atoi(buf);
    int run_for = atoi(len_str);
 
    input_jobs[start_at] = run_for;
  }
 
  fclose(f);

SO MUCH BADNESS HERE. Gotta go add stuff up top: #include <map> and using std::map. I really hope they don't give me a duplicate start time or I'll squish the previous one. (I think they said something like "this won't happen", but why would you ever trust that?)

Again with the blindly assuming we have usable data for atoi... and the bad line thing with no space thing, well, that's also not supposed to happen, but again, I don't trust them that much. I'm willing to barrel through some stupidity, but this would do terrible things when it gets a NULL back from strchr and then tries to dereference it later, so instead of writing yet another terrible '80s C pointer hole, let's just die.

Oh yeah, I guess I should make sure they gave me what they told me they'd give me, since I'm reading to EOF.

Go back in above the loop...

  int actual = 0;

... and in the loop ...

  ++actual;

... and after the loop:

  if (actual != input_lines) {
    fprintf(stderr, "expected %d got %d\n", input_lines, actual);
    exit(1);
  }

Right, so now I'll know if that's screwy for some reason at least.

[ Time check: 27 minutes elapsed in writing this post. ]

Now I guess I get to iterate through the data I read and do something useful with it. Oh yeah, by reading it in first and not "streaming" it, I've basically committed myself to an O(n) memory situation at best, and it'll probably end up worse than that. So this temporary "read into a map" thing will probably have to go at some point, and I'll call into the "add something I just parsed" from inside the for loop. Joy.

Anyway. Iterate. But, okay, C++98, so old-school iteration, none of that "for const auto" stuff.

  map<int, int>::const_iterator ii;
 
  int job_count = 0;
  for (ii = input_jobs.begin(); ii != input_jobs.end(); ++ii) {
    const int& start_time = ii->first;
    const int& run_time = ii->second;
    add_job(start, time, run_time, job_count++);
  }

And obviously now we need something up above main that'll do this add_job, so let's write a stub to get that started.

static void add_job(int start_time_raw, int run_time, int job_number) {
  fprintf(stderr, "XXX add_job %d | %d | %d\n\n",
          start_time_raw, run_time, job_number);
  // XXX write me
}

Okay. I'm going to stop there for now. Realize that what you're seeing is the *final form* of that code. There's a bunch of stuff that happened in-between. For example, see how it's called "start_time_raw" in the add_job function? That's because the start time value from the file turned out to be "HHMM". That is, 12:00 noon was in the file as "1200". 1:45 PM was in the file as "1345".

I had initially parsed it as something in the range 0-1439, because, well, given a system that lets you schedule stuff at a given time during the day with one-minute precision, personally, I'd be using the "number of minutes since midnight" as the base. They weren't. They were using human-readable time less the usual colon we put in the middle for readability!

How did I know 1439? I've written schedulers before, and a normal civil day that doesn't have any DST transitions in it is 1440 minutes long: 24 * 60. I also know from writing those same schedulers that a week of those same non-DST days is 10080 minutes long, because that's just 24 * 60 * 7, and you run into that number a lot. It gets baked into your head. Or at least, it got baked into *my* head. What can I say, I'm weird like that.

So, it's start_time_raw because it hasn't been split up and turned into a value that'll make sense inside an array. The first bits of code in the eventual add_job() ended up doing something terrible to change HHMM into "minutes since midnight" in the range 0-1439, like this:

  int start_hour = start_time_raw / 100;
  int start_minute = start_time_raw % 100;
 
  int start_time = start_hour * 60 + start_minute;

Awful, right? No sanity-checking. It just mods and divides and multiplies and adds and off it goes.

Now, at this point, I haven't done one damn thing towards actually working on the algorithm, and so I haven't even talked about it here because I wanted to point out how much time goes into just getting the foundation in place to let you actually do anything with the data.

Getting a filename, opening the file, reading the contents, turning them from text into numbers, putting them somewhere sensible and then starting to iterate through them takes some amount of time. If you know exactly what you need when you sit down and start typing, you can probably rip through it in, I don't know, maybe 15-20 minutes?

But, can you do it in an interview, away from your usual setup, on a laptop keyboard, with the laptop screen, in a funny chair, on a weird desk, with the clock ticking, and someone (in theory) keeping tabs on you? Also, you have the possibility of not getting the job riding on this. You probably need the job to make money to keep your house or apartment, feed you and your loved ones, and all of that good stuff.

But hey, no pressure.

I went through this, and I had nothing at stake. If I didn't get the job, big deal. I didn't need the job. I'm also significantly older than your average new grad and have been worn down into not putting too much of my self-worth into what these interviewers might think of me or my code.

Also, I've done this whole "argc, argv, fgets into buf, squash the newline, do it in a loop, chop it up on a space and deal with the results" in C probably over a hundred times in my life up to this point. I know all of the parts. I also know what parts can go wrong. I had to make a decision of what to skip and what to keep in terms of sanity-checking in the name of saving time. I didn't have to spend time trying to make that work. Someone who hasn't been down that road yet wouldn't have it so easy.

But... even so, I was still sweating like crazy, and felt like shit when it was over... even though, again, nothing was riding on this. If I didn't get the job then I wouldn't end up coworkers with some of my friends who wanted me there and had invited me to interview there. I'd just go back home and keep writing snarky tales about the industry, like this right here.

Now imagine you're 22, you need the job, and all of that stuff. You have actual things at stake. Imagine how you'd feel, and what that would do to your problem-solving skills in the moment with that level of stress.

Incidentally, when it came to this problem and I finally started working on the "engine" and the algorithm, I picked the wrong implementation. I wound up focusing on the wrong "axis" in terms of how I imagined the solution working, and caught myself in a dead-end corner. Ironically, my experience doing a scheduler (for people) previously had pointed me at trying to use that as the solution, and it didn't work for what they wanted here.

It wasn't after disconnecting from the problem and spending some time away from the computer that it came to me and I figured it out.

What's that? You don't have the time to disconnect and then come back to it, because you're being timed, and they're coming to check in on you soon? Too bad! Good luck figuring out where you went wrong.

That. That right there is where the time goes.

...

Pre-emptive sidebar: "she should have used Python". Just go away. You are *so* missing the point.

Cutting to the chase: text editors *and* tabs vs. spaces! Ooooh!

I have Macs in my life. They regularly receive updates. Over the past couple of days, they got Monterey version 12.3. This has seemed largely uneventful... until I went to edit something using my little wrapper script.

I've had this thing called "e" for a while now. It's a stupid little script which serves to do a couple of things. It starts my editor with a given file, and it kicks it with the right arguments to make it disable line wrapping and to make the [TAB] key actually put in two spaces.

Why have a script and not a .nanorc? Well, I think this pre-dates them having that file. Also, there is one nuance: I used to work on Makefiles a lot, like before I did my no-Makefile C++ build tool, and in that context, you *actually want* no-joke ^I tab characters. That's the only separator it'll allow when you write the subsequent parts of a rule.

So, my dumb script would use "-w -E -T 2" most of the time, except when it was running on a Makefile, at which point it would skip those args. Simple enough, right?

I've been using this script on Linux boxes and Macs alike for a long time, and it just broke on the Macs. I now get a complaint about "E" and "T" not being valid arguments, and then it spits out "possible starting arguments for pico editor". Wait, what?

pico is back? Yes indeed, on Macs, as of 12.3, pico is in fact back. To make sure I wasn't imagining things, I tracked down a machine that hadn't been upgraded yet and checked there. Sure enough, it was in fact nano and not pico.

macOS before 12.3 had pico as a symlink pointing to nano.

macOS 12.3 has nano as a symlink... pointing to pico!

For those not in this world, pico was the composer part of the pine mail client split off into its own thing. It had some weird licensing issues for years, and generally wasn't the easiest thing to find. Perhaps due to that, nano was started. It used to be pretty close in terms of features and behaviors, but nano diverged at some point. It picked up the ability to turn the [TAB] key into some number of spaces as if you whacked the space bar that many times. pico did not.

At some point, I switched from literal tab characters in my source code to blobs of spaces, probably due to using that style at work and not hating it. So, somewhere along the line I actually started depending on nano behavior without realizing it.

The macOS 12.3 update silently switched the base system from nano 2.0.6 to pico 5.09, and so if you use those switches, you're out of luck. My advice if you care about this kind of thing is to go get nano from Macports or whatever and forget about using the stock editor. That's what I'm going to do. At least I don't have to reprogram my fingers since "e foo" is still "e foo", no matter what it ends up calling underneath.

As for the lack of a feature in pico itself, it's kind of strange, really. If you go digging into the source, they have this random.c with a function called tab which claims to "simulate tab stop" using spaces. That said, I can't find any place where it would be called with the magic values which would make it behave that way.

This is also old-school C that goes back into the '80s and I don't feel much like spending too much time on it, so there might be some way.

It wouldn't surprise me if this switch is just more of Apple removing GPLed things from their OS - pico seems to use the Apache license. They dumped bash for zsh (MIT license) at some point (Big Sur, I think), and now this? Okay then.

Here's a very old and stupid trick you could do with some filesystems in some situations back in the day... and might still be able to do in a few places today.

Let's say you have an account on a vaguely Unixy box that you don't run, and you are looking to exploit a hole to get root. You notice there's some program that is installed suid root and which has a terrible security record. It's chock full of awful practices and it seems like every month there's a new hole in it and admins have to race to patch it before the script kiddies of the world use it for great evil.

The problem is that while version X might eventually have an exploit, it'll get removed and it'll be replaced by version Y without that particular hole. By the time you get your hands on a tool to break into X, it'll be gone.

But... what if you could keep a copy of X around, you know, for later?

What you'd do is create a hard link pointing at the file you wanted to preserve, storing it in a spot which was writable by your account. Then, say, /usr/bin/sudo (because that never has security issues...) gets replaced at some point with a new file and you get to keep the old version. When you get the tool to pop it, off you go!

This is one of those reasons why old-school multi-user systems used to put stuff like /usr on one filesystem and things like home directories on another. That way, they couldn't hard link to arbitrary crap like that and extend the lifespan of a potentially vulnerable file. Of course, they could also extend the lifespan of someone else's stuff and screw up their quota, so there was that...

Of course, these days, it's not quite so easy on the more modern systems you find. Debian 11 here won't let me do it - it's EPERM city. Rocky 8 (so CentOS 8 and RHEL 8 too) also won't do it. They both have some gunk in /usr/lib/sysctl.d to lock this down. But, my old Slackware 14.2 install of my workstation let me do it the last time I tried about a year ago. How about that, huh? I bet it'd still work if I dug out the old drives and booted them up.

So, if you have older machines in your lives with potentially shady characters running things on them, you might want to check them out. You might learn something new and interesting.

Incidentally, this is the magic sauce: /proc/sys/fs/protected_hardlinks Read the man page for proc(5) to learn more about it.

What's kind of funny is that there was a discussion about this on the linux-kernel mailing list back in November 2003 according to my original notes on the subject. So, I went looking for it, and sure enough, here it is, the thread I read a long time ago. It's interesting to note that as far as I can tell, nothing actually changed upstream. This left paranoid admins like me with no option but to use the (most excellent) Openwall patches to lock down this kind of stuff. I used that on my machines at work which had non-staff users... just in case.

It appears this /proc knob showed up in Linux 3.6, and it looks like that dates to 2012. Hmm. 2003... and 2012. That's a long time. Obviously, mad props are due the person who braved the lkml and got it to finally happen upstream.

This whole situation is scary. Read more about it.

Back in January, you might have seen my post about a very large BBS which seemed to solve for the problem of "how do we host a bunch of users" by having one physical box per user. It might have been one physical box per *two* users, maybe, but it's hard to say.

Anyway, I have some pictures like this from my own life. They show a lot of tangled-up wires and a bunch of cruft like that, but in general, we didn't have one box per user. We had one decently-sized box for everybody... at first.

Now, keep in mind digital cameras weren't easy to come by, so I have very few actual pictures. I'll have to do a lot of explaining to make up for the lack of detail and alternate angles. Clearly, I had no notion of ever using these pictures to tell stories 25+ years later.

It's the summer of 1995. I was asked to build a BSD/OS box that would be "the box" for this brand new network a local school district was building out. The people in charge wanted me to set it up like the box that had been at my high school for a couple of years. It ended up being a Pentium 90 with a Digiboard for eight external serial ports. We bought 4 consumer-grade Sportster v.34 modems to drive those dialups. Later, we bought four more for a grand total of eight analog dialin ports.

Some friends at a local company had some bandwidth to spare (a T1 to Alternet somewhere, as I recall), so they parked a Zoom modem on one of their boxes and plugged it into a phone line. We got an identical model to avoid cross-brand issues (Zoom modems were awful that way...) and used it to dial them. That was our Internet pipe for the first year or so: v.34 (~28800 bps) into them, into their T1.

This is more or less what it looked like:

In this picture, you can see the "glorious" ASA Pentium 90 with both kinds of floppy drives and a CD-ROM (for no particular reason) and a DAT (DDS) tape drive for running backups. On the wall, you can see where *9* different analog phone lines were punched down into jacks: one for each dialin port, and one for the dial *out* to our "ISP".

There are also a pair of Ethernet ports and cables present: one orange, one white. One went to the machine shown here, and another went to some desktop machine not shown. It's hard to tell, but they *are* labeled. Go us.

Sitting on its side is the cursed Zoom modem being used as our uplink.

You can't really see them here, but on top of the case there is a black "Digibox" that fans out all of the serial ports and then eight of those Sportster modems, each with a flat satin cord running back to the wall, and a fairly thick serial cable running back to that Digibox.

There are also a pair of power strips hidden back there with OH so many of those stupid AC adapter "wall wart" bricks: one per modem.

(You'll see some of this insanity in a later picture.)

This box right here did *everything* for a while: dialups, shell accounts, SLIP and PPP access, DNS, mail (incoming and outgoing), it ran the first few web pages, it monitored the rest of the network, and so on and so forth. It just sat there and ran and ran and ran. Every week we'd feed it a tape, and this turned out to be a good thing, since one of the drives died on us and we had to restore from that tape.

That was life back then.

Let's jump forward to 1997. We bought a system based around a 200 MHz Pentium Pro to replace the original Pentium 90 from 1995. The plan went like this: I get BSD/OS 3.0 running on the new machine (a HP Vectra Xu, for anyone who cares), get all of the users and whatnot ready to go, and then swing them over in about 5-10 minutes of downtime.

This took some doing involving port forwarding, NFS mounts and all sorts of other demonry I'd rather not get into... but it worked. Most users never noticed this. What they did notice is what happened a few hours later.

After moving the Digiboard (and thus, all of the modems) from the old machine to the new machine, I started stress-testing it. I had some test boxes I could get to dial in and nail stuff up. Part of the point of this upgrade was to get PAP going so our users could dial in and do PPP without those damned chat scripts - you know, wait for "ogin", send the username, wait for "assword", chuckle like a 12 year old, then send the password, that kind of thing. I wanted to see it in action.

So, I got some boxes to dial in, and started flood pinging them and generally pushing lots of bits at them. The box hung. I rebooted it and tried again. It hung again.

The day went on like this, and then the days that followed. We tried all kinds of things. We worked with the OS vendor (BSDI). We went this way and that with kernel patches and config changes. Nothing helped. Finally, I moved the damn Digiboard back into the old box and just upgraded it to 3.0 so we'd have PAP dialups. It was perfectly fine, so it wasn't BSD/OS 3.0. It was something about that HP Vectra. We never did figure that out.

As a result, the old box didn't get to retire in 1997. It stayed online, and since it had all of those modems hooked to it, it couldn't move from that spot, either. We just put the old and the new machines side by side and left 'em there.

Along the way, we also picked up a dedicated machine to use as both a web server and a web proxy since we found out the hard way that some of our IP space could not be routed over large chunks of the Internet. That, too, was another HP Vectra running BSD/OS, and so it ended up stuffed in there with the other two.

The situation looked like this:

Now you can see some of the context that you couldn't see before. Note the stack of modems there: some on the Digibox, some on the new Vectra. If you look closely, you can see the BSD/OS release box wedged between Vectra #1 and #2. There's also a massive battery backup there so we wouldn't die when the site power went out.

At this point, the 1995 server was running dialups, so it was handling SLIP and PPP logins. It also did this goofy thing I rigged up where if you dialed in interactively (like with a terminal program), it would flip you to the new machine to do your shell stuff on there. Technically people *had* shells on the old box too, but I didn't want them using it. It would have been too confusing.

Maintaining these accounts in parallel was annoying, so people tended to not get accounts on the dialin side of things unless they needed one and asked for it. No, we weren't doing YP/NIS.

The 1997 server was doing most of what the original box had been doing, minus the dialups and the web site. You can see it's also running X here with whatever version of Netscape would have been current.

We had fortunately gotten our own T1 to the outside world, so the Zoom dialout is long gone, for anyone wondering what happened to that thing.

There was a fair amount of heat coming out of this corner, as you might imagine.

Let's jump forward four more years to 2001. The modem companies had come out with this stuff that benefited from the fact that most phone lines were digital most of the way until the last few thousand feet to the actual customer. They used this to create *almost* 56K connections in the form of their own K56Flex and X2 proprietary stuff, and then the eventual V.90 standard. The trick was that to support this as a dialup host, you had to have digital lines into your pool. You couldn't do analog on both ends and still have this work.

We wanted that to let our users do (near) 56K from home. We also wanted more than eight dialups and didn't want to add yet another Digibox, still another stack of those stupid modems, and the associated ports that would have to be strung into the wall and punched down into those jacks. So, we pulled the trigger and bought a dialup rack.

This monster box fit in a 19" rack and took these vertical blades. You'd put in modem cards which would then accept a PRI or T1 in the back. This would give you 23 or 24 lines per card depending on what sort of service you fed it. Then you also had a "server" card that basically did the equivalent of what my old 1995 machine did: it authenticated people (via RADIUS this time) and ran PPP for them, and forwarded packets between those lines and the local network. Finally there was this monitoring card for keeping track of the entire setup.

This monster was installed in 1999 but I didn't get a picture of it until 2001. Here it is at that point:

It's the big black box taking up the bottom half of the frame here. The two boards at left are the actual modems (HiPer DSPs, for those USR nerds out there), and the two towards the right are the others described earlier (the ARC and NMC, for anyone who remembers and cares). The last one there at far right is a power supply.

(Also, looking at this picture now, I have no idea why the Bay router up top only had two of its four screws installed. Sketchy.)

This thing took days to get setup because the telco people were incompetent. Yes, I'm just going to say it. They had no idea what the hell we wanted to do, despite us telling them "we want PRI and are okay with 46 dialups, because we don't want robbed-bit signalling, so we get that slight bump in line rate" for months leading up to that point. But, once they finally figured out how to provision their circuits, it was in, it worked, and that was that.

This meant the original machine could finally retire. I don't remember what happened to all of the modems. I think maybe we doled them out to the school board members so they could dial in from home, or something. They had pretty rough lives up to that point, so it wasn't a great deal even for a "free" modem.

The Digiboard and Digibox probably would have been useful for some projects involving lots of little serial connections, but I have no idea what happened to it, either.

Anyway, this is a very limited view of some of the crap I used to run. I never ran a multi-line BBS, but I did wrangle some of the same stuff for other reasons.

Knowing what I know now, would I have done things differently? You bet. Live-swapping the mail/etc server in the middle of the day because I was an hourly employee and so doing it at night was not a good choice? What utter crap that was. Having only the one digiboard and so not being able to test it in the new box without taking down the old one? Also utter crap. Buying Sportsters instead of Couriers for the original 8 analog lines? Using HP Vectras for ANYTHING? Not having a redundant power supply in the Total Control box? Not fully securing that Bay router in the rack?

You get the idea... so don't send me hatemail about how much of a shitty operation it was. I fully admit it was a joke.

I sometimes write the beginnings of posts but don't post them. I still save them in case conditions change or other things happen and make me want to follow up on the topic. In this case, I wrote something about offices before the whole COVID lockdown thing happened and people largely stopped going to offices every day. With that in mind, let's proceed.

Offices tend to have break rooms. If you're lucky, it'll have a clean source of water and a safe place to stash your bagged lunch. If you're of the coffee-drinking persuasion, then maybe it'll have a coffee pot that's kept somewhat clean and never violated by really evil disgruntled people. There might even be a place to sit down for a while if you normally stand all day, or walk around if you normally sit.

However, tech companies decided to go past this at some point and brought about this whole notion of "microkitchens". This is where they'd put their branding on the coolers (instead of "Coca-Cola" or "Pepsi") and would load it up with every kind of fizzy sugary liquid they could get their hands on. They'd have racks and racks of shelving packed with all sorts of interesting crunchy, sugary and carb-loaded snacks. They'd also have professional coffee and espresso machines so that people could serenade the whole office with their steaming skills.

Invariably, conditions change, and the company wants to recoup some of the dollars they are spending on this stuff. Some of the micro kitchens go away and others shrink, and people complain. I certainly did... the first time it happened. They had claimed that those benefits were part of my salary and are why they offered me what they did. I wrote about this a few years later, and was taken to task for it by the Internet.

By the time my next Big Tech job happened, I had no delusions about the food situation. It would be there for a while, and then it would shrink, and indeed, over the next four and a half years, it did in fact do exactly that. I just smiled at the people who were going through it for the first time.

Every single thing that happened at company G then happened at company F. It just took a while, and didn't always happen in the same order.

Long after that, I foolishly signed up to do this sort of thing again, but this time I had planned to handle these things myself. People would occasionally rope me into having lunch on site, but I tried to get OUT of the office as much as possible. This company's office was in the middle of a real (if troubled) big city and was therefore walkable to a great many things. I could get pizza, fried chicken, a bar burger, soup and a salad with a cookie, another kind of pizza, soul food... all on foot. There was also a Philz Coffee right across the street for those afternoons when things really sucked and I just wanted to get away for a while and recharge before trying again.

So, what did I write back in 2019, while in that gig, then? This:

...

The best time to take away the free berries in the break room (or free sodas, or snacks, or candy, or kombucha on tap, or ...) is yesterday. This is always true.

The sooner you take it away, the sooner it will "age out" of the company history, and go from a sore point, to a joke (the "no-berry" all-hands meeting room), to a legend ("why is it named that?"), and then finally "those people were just entitled and stupid".

This is because most of the people who were around for it leave, and many more people join. Look at the average tenure of the employees in your company to get an idea of what the "half-life" is for the population. The people don't miss it because they never had it in the first place.

So, my take on kitchens and break rooms? Have fresh water available, preferably both cold and not-cold (not necessarily hot).

Pay your employees enough so they can buy snacks and not feel bad about it. Don't put your company in a f'in *mud flat* so they can actually LEAVE THE OFFICE to spend that money... on snacks. Or lunch. Or coffee.

Don't pack employees in so tightly that you can't drive away for lunch because there will be nowhere to park when you return.

Appreciate the fact that giving them a chance to shift gears may just bring the kinds of fresh perspectives you're not going to get from having wage slaves who don't leave the building once in nine hours.

Of course, if you WANT assembly-line workers who just glue things together and don't build lovely things or solve hard problems by using their innate tool-using capacity they have as healthy, happy people, then don't do any of what I said. Instead, build the hell out of those mud flats and keep 'em locked up all day long, then take away the stuff when they don't act appreciative.

Before long, you'll have washed out anyone who cares about that kind of thing and will be left with the nice, sort of "grey goo" that can be molded to do whatever you want, you lovely little sociopath, you.

Have fun with that.

Another day, another bit of local flavor, but this time it's something I wish I didn't have to write about.

In August 2012, I wrote about seeing a maintenance truck rolling down the Caltrain line in Mountain View. It had a special attachment to let it stay on the rails. At no point did the arms come down for the crossing. It managed to approach, then the people inside looked around and went through, and the arms never budged. There were no flashing lights or bells.

Based on my understanding of how those crossings work, that says "whatever holds that truck to the rails does not manage to connect them in whatever way that a train does to trigger the crossing arms".

This is not great. Then today happened. Earlier this morning, Caltrain hit *two* different trucks on the rails north of Millbrae. Then it caught fire. A bunch of people were hurt. More were shaken up.

These weren't random trucks from some random idiots gallavanting up and down the lines. These were their own construction trucks which have been working to install overhead power for the future electric trains.

I'm assuming their construction trucks don't trigger the "short the rails together to signal various things" logic, either. This would mean that crossing arms stay up... and I have to assume it also means the signals along the way stay green instead of going red.

We will presumably hear more about whatever the signals said, and whether the train went past a red signal or not. I just have a bad feeling that they saw a green and kept on going.

Here's a little bit of Silicon Valley local flavor for a change. We have this freeway that runs from San Jose to San Francisco called US 101, or around here, just "101". (Southern California has it too, but they call it "the" 101. That's how we detect invaders.) Anyway, this thing is usually eight or more lanes up and down the peninsula, and it's had a "diamond" or car pool lane on the #1 (innermost/"fast") lane for many years. You had to have a certain number of people in the car (or some other conditions met) during certain hours (AM and PM rush, weekdays), or you'd be in for a hefty ticket.

This started changing recently, and now parts of the corridor have turned it into a toll lane all day on weekdays. This is a deal where you can drive in the lane and pay for the privilege if you're by yourself, pay a discounted fee if you have one other person with you, or it's free if at least two other people are riding along (plus a bunch of other variations I won't get into here - motorcycles, clean air vehicles, you name it).

How does the system know what to charge you? Well, you either have an old-style toll transponder that always bills you as a solo driver, or you have a new-style transponder with a 1/2/3+ switch that tells the road how to handle you.

Yes, the billing is entirely a function of whether you have that thing in your car, and how you configured it for that particular driving session. You could totally lie to it and set it to "2" for a discount or "3+" for the complete freebie ride. But, then you have to worry about getting caught and getting a ticket.

Think about what this does to enforcement. Now, instead of just "is there a vehicle in the lane", "is it the right time during a weekday", and "how many people are in the car", now it's also a matter of "do they have their transponder set properly".

How do you suppose they're going to tell what position you had that thing set to when your car rolled under the reader? The answer turns out to be relatively simple: they put a sign over each lane, and it just flashes up a "2" or a "3" depending on what it heard from your transponder's response.

Check this out.

This is a scene from Mountain View where a car just passed underneath a reader and the "2" lit up a second or two later. If there's nobody else in that car, CHP could roll up on them and give them a ticket.

I already told a story about how you can hose your machine by pointing the Linux kernel at a custom core dump handler and then have that handler fail to function. This'll clog up the pipe (really), and then everything else will just stack up behind it. Pretty soon, every crashing process will be stuck just the same.

What I left out of that story is how it got to that point one time.

Once upon a time, in a big company environment, someone decided that they wanted to provide a service to other things running on the same Linux box. But, instead of either writing Yet Another Service (tm) that would go through release processes and all of that good stuff, they went for a different design entirely.

Yes, this service lived in *client* programs. Every time a program built using the usual corp RPC transport stuff started up, it would try to bind to a certain TCP port. If it succeeded, it said "well, I guess I'm the winner" and proceeded to start up this service on that port. Otherwise, the idea was that it would just then connect to that port and talk to whoever got there first.

There are so many problems with this. You have the issue of a "server" that appears out of thin air depending on who or what starts first on the machine. It takes up CPU time and memory that gets "charged" back to the process that "won", instead of being its own thing that tracks back to whoever wrote this thing in the first place.

Imagine if the first car that got the freeway every morning after a certain time had to follow everyone to their offices and make coffee for them. That's the kind of random, completely nonsensical and difficult to troubleshoot thing I'm talking about.

Better still, apparently this thing they did was buggy. As the story goes, it started threads to do this stuff, and it managed to have some circular references such that it would never get a refcount of 0, and so it would never actually go away. Better still, the thing which attempted to grab the port had an infinite timeout, so it would just try forever. It never actually failed, so it never gave up and shut down.

This meant every program built after a certain commit in the tree had this misfeature in it, and could not shut down cleanly by itself. You'd have to shoot it in the head with a strong enough signal to make it go away.

Now, given that basically everything eventually used this library, imagine what happened when the core dump helper thing got a hold of it. It too got stuck, and then failed to shut down. This meant the kernel never managed to finish its core dump sequence when something on the box crashed. That then backed up everything behind it.

Basically, this bug had been breaking other programs up to that point, but when it reached the core dump helper program, that's when it turned really bad. It took that kind of visibility to bring enough people to bear on the problem and finally shut it down for good.

My own heuristic for picking up on this kind of thing is that every thread but one is in exit(), and the last one is in some pipe_* function pretty deep inside the kernel... and probably a few stack frames past do_coredump or similar.

If it happens to you, hopefully you'll remember this.

If a web site makes an RSS or Atom feed available, it's not a bad idea to poll it from time to time. Actually doing that poll like a good netizen (remember that concept?) takes a little attention to detail.

First of all, the feed probably doesn't change that often. Depending on who it is and what you're following, you might get an update once a day, or multiple times a day if it's something really special. Everyone and everything else is going to be somewhere less than that.

Given the reality of the situation, does it make sense to poll every 10 seconds? Probably not, right? Someone set up something like that and left it running for a couple of months until I finally noticed it and dealt with it myself.

Another thing about URLs is that some of them are well-behaved and stay the same until they are deliberately changed. They don't change just because you requested a copy. A good feed certainly behaves this way.

Given this, you can keep track of the "Last-Modified" header when you get a copy of the feed. Then, you can turn around and use that same value in an "If-Modified-Since" header the next time you come to look for an update. If nothing's changed, the web server will notice it's the same as what it already has, and it'll send you a HTTP 304 code telling you to use your local (cached) copy. In other words, there is no reason for you to download another ~640 KB of data right now.

Alternatively, many web servers (mine included) support this thing called "ETag", and it amounts to a blob that you just return in your requests. If it hasn't changed, you get a nice small 304. Otherwise, you'll get the same content as always. It's effectively another way to do the "IMS" behavior described above.

Besides that, a well-behaved feed will have the same content as what you will get on the actual web site. The HTML might be slightly different to account for any number of failings in stupid feed readers in order to save the people using those programs from themselves, but the actual content should be the same. Given that, there's an important thing to take away from this: there is no reason to request every single $(*&^$*(&^@#* post that's mentioned in the feed.

*exhale* ... okay, let's continue.

If you pull the feed, don't pull the posts. If you pull the posts, don't pull the feed. If you pull both, you're missing the whole point of having an aggregated feed!

Then there are the user-agents who lie about who they are or where they are coming from because they think it's going to get them special treatment somehow. I mean, it probably will... some day... but not in the way they wanted. It'll be more like "welcome to the IP filter" and less like "oh here, bypass the paywall that this site never had in the first place".

Let's tally up some of the bad behaviors here:

• Loading the feed way too often, like every 10 seconds, when it never updates anywhere near that often
  
• Not using If-Modified-Since (ETag), and so always download a full copy of the feed even when nothing has changed
  
• Scraping individual posts after pulling a copy of the feed even though they have the same damn content
  
• Doing this while claiming to be "GoogleBot"... guess what, that actually *draws attention to* your process. It's pretty easy to remember "66.249" and notice things which are definitely not that. Also, any hosts that reverse to "googleusercontent.com" in particular are NOT the actual crawler. (And yes, that Google netblock is a /19, not a /16. It's a reasonable approximation if you're just eyeballing a log.)
  
• Sending referrers which make no sense is just bad manners.

Would you believe most of the bad ones actually do more than one of these bad behaviors at the same time? It's not enough to just hit the site every 10 seconds, for example, but it'll also score a copy of the entire feed, too. 640 KB every 10 seconds for weeks and weeks until someone notices and pulls the plug (or I block it). Really. This is not a good use of resources!

What's really amazing is that when asked to slow it down and actually do the If-Modified-Since thing, it turned out they *couldn't* send the header. Yes! That is the state of the art in feed reader technology these days: download the full thing every time.

If-Modified-Since. It's in RFC 1945, aka the HTTP/1.0 spec... from *May 1996*. It's been over 25 years, and it's time to start using this thing.

Here's a concept: "ssh to prod and edit directly on the machine" is generally bad news.

I heard from a third party that someone was doing this at $COMPANY on one of my services, and I just thought (and said to them in chat) something like "ugh" or "oy". I left it at that. I did not reach out to the actual person who was doing it because I didn't care that much. This particular company had many, many problems and this one was comparatively minor. There was no reason to go out and deliberately stir things up, considering how poorly many people tend to handle being told "don't do that" in general.

I should point out that I didn't know this person and had no idea how they would respond to such feedback - it might have been fine! They might be a well-adjusted person who can take a request like that and not turn it around into a full-on nuclear attack. But, I didn't know either way.

Some time passed, and it turned out this person somehow heard about my random groan when it had been mentioned to me by that third party. I assume the third party talked to them at some point, or maybe it just wound its way through the grapevine and made it back to this person.

At some point, I actually had a meeting with this developer for some other reason, and what happened was kind of amazing. First thing, he brought it up and asked me about it. I said something like, "well, I didn't want to get into it, but since you asked", and proceeded into something along the lines of "obviously it would be best if you didn't do that - sshing in and twiddling Python files on the actual production AWS instance of the service".

I didn't stop there, though.

"But... I suspect you are doing it for some reason?"

He was, and said something like "we can't do (other method) because of (dumb thing inflicted upon all of us by $COMPANY's crap infra) and (other dumb company thing)".

My response was... I hope... reasonable: "So as much as I would love to say 'don't do that!!!', I'm not going to stand in the way of you literally doing your job. So please proceed with care, and keep an eye open for some day when (first dumb thing) and (second dumb thing) aren't problems any more and you need not do this any more."

This really happened. That person's hands were tied, and there was no useful way to do anything about it. The "infra" which had accreted at this company forced us into any number of terrible patterns, and this was the path of least resistance that still worked. He was biasing for useful outcomes and to his credit was being *very* careful about it.

It was either "edit in prod" or endure DAY LONG development cycles: make a change, wait 24 hours, come back tomorrow, and try it again. That's how broken it was.

In other words, this was someone who had been asked to give haircuts to hyperactive children and had only been handed a chainsaw, and still managed to actually DO it without harming or scaring anyone. That takes some serious effort (and skill).

The best part about this was that we had a totally reasonable conversation about it. It's the kind of thing which had become a rare commodity in those days at that place.

Months later, someone asked me what I would have done if I had been in his shoes and hit the same problem. My answer was something evil involving "punch a TCP port through ssh tunnels and/or socats to make it look like I'm coming from the prod machine". I would have done that because it's one of the things I know how to do.

Would I expect someone else to go to that level of insanity every single time they just want to get their job done? Absolutely not.

Editing files in prod to develop them? That's bad news. Deliberately rigging tunnels to spoof a connection from prod? That's also bad news. Having to do them to get your job done? You might be in the bad place.

Quite a while back, I wrote a post that basically went like this: if someone has a problem with someone else or something that person is doing, they're going to find a way to get to "no". They will find some reason, some excuse, or some problem with it. They'll bring up a point and you'll show that this is not in fact a thing. They will then pivot again and again.

At the time, I was mostly referring to the open sewers that are certain web forums on certain days of the week and certain hours of those days. For some reason, when the regular people are out doing whatever they do, the haters have nothing better to do than run unchecked on the web. With no sensible people there to set them straight, they feed off each other, and pretty soon you have some straight-up nuclear waste in those forums.

But, as I said, that was back then, and this time I'm talking about something else: when it happens in person with a coworker. I had an experience like this a few years back, and figured I'd share how it goes for the sake of others who might not have encountered it yet, or who have and thought they were alone in this.

It goes something like this. The topic of something or other that you intend to build has come up. You've mentioned it multiple times in different venues with very little uptake from the rest of the team. It's pretty clear they are more concerned with other things. Time passes, and now it comes up in conversation with this coworker.

Coworker: There are no docs for [this thing you're doing].

You: Yes there are.

Coworker: You could make a wiki page.

You: There *IS* a wiki page. [finds page, sends link]

Coworker: This is just a list.

You: Scroll down.

Coworker: But this isn't a design plan.

You: I posted that to the team group two months ago. [finds post, sends link to that too]

Coworker: You should put it in the wiki.

You: If I put it in the wiki you would have said to put it in the group, and if I put it in the group you would have said I should have put it in the wiki.

Basically, with them, something's always wrong, and if you refute their points, they will pivot to find something else. This will go on for a while, and at no point will they ever reach a point of being able to understand what's going on. They chose their outcome before they even started the conversation, and they're on rails, headed for that outcome and no other.

Here's how you can tell. A reasonable person might also raise concerns, but it sure wouldn't go the same way.

Coworker: [goes and does a cursory search, finds the group post, then finds the wiki page, and reads all of it first]

Coworker: I have (specific concern) about [thing you're doing]. I couldn't find anything about it in that group post or on the wiki.

You: Oh, crap, yeah, that's a thing. Huh, okay, we have to figure that out.

or ...

You: Check it out, it's kind of buried in the wiki, behind this thing you have to flip open for it to be visible. It's not great but the page is getting too big. Sorry about that. Maybe we can reorganize it somehow.

or ...

You: Huh, I thought we talked about that in the group post? Guess not, okay, well, whatever, we're here now, tell me more?

... you get the idea.

The point is: reasonable people approach such situations entirely differently. They tend to do *at least* the basics of looking around a little for some answers before showing up and declaring that the person in question has put something out there without documentation.

Basically, that declaration from the first scenario ("there are no docs for X") actually means "you did not manage to do whatever arcane steps are required to forcibly insert this into my brain, such that I somehow know about it without doing ANY ACTUAL WORK ON MY OWN to learn about it". And, as a bonus, it's your fault they don't know about it. Short of fully supplicating yourself before them, you will *never* please them. It is not possible.

Now, sure, there are projects in this life where people don't write a damn thing about them and which definitely have no documentation attached. This is not that scenario. Consider the players here. If I was involved with something, do you think there'd be any lack of writing on the topic, given how much I obviously like seeing myself in print? Of course not. I write early and often when there's something like that going on. It's just my nature.

That guy can take a long walk off a short pier. Good riddance.

This is a short one since it's somewhere between a random bit of "grab some popcorn" amusement and a "huh, that might be a problem some day". In short, it looks like every Raspberry Pi filesystem installed from any given release image probably ends up with the same UUID.

When is a universally unique identifier not universally unique? When it gets stamped into a filesystem that is then cloned onto a bunch of storage devices and used as-is.

For instance, if you downloaded the "lite" version of Raspbian between roughly November 8th and January 28th, you probably got a file called 2021-10-30-raspios-bullseye-armhf-lite.zip. Inside, there's an image, and the idea is that you dump that straight to your SD card, cram it in the Pi, and off you go. The second partition on that image is an ext4 filesystem which was created on some build machine somewhere. The UUID on it never gets "evolved" to something else after the install.

So, you now know that every single install from that image, no matter where it is, now has the same UUID value for that filesystem unless someone or something has come along to change it after the fact, and that UUID is 568caafd-bab1-46cb-921b-cd257b61f505. It even shows up in a few places on the web, and that'd normally be a neat trick for such a thing.

You can see it inside the image itself here:

# losetup /dev/loop0 2021-10-30-raspios-bullseye-armhf-lite.img
# partx -a /dev/loop0
# tune2fs -l /dev/loop0p2 | grep UUID
Filesystem UUID:          568caafd-bab1-46cb-921b-cd257b61f505
# 

Or, if you actually installed a few of these things like I did over the past couple of months, you can just 'tune2fs -l /dev/mmcblk0p2' and see the same value in there.

What does this mean? For most people, probably nothing, hence my initial description. However, if you are the sort who likes to track filesystems in terms of their UUIDs instead of a device name (sda, sdb, ...) or anything else that tends to be brittle, weird and jumpy in this wacky modern world, this might trip you up some day!

Don't end up with a pile of SD cards on your desk and think "I know, I'll use the filesystem UUIDs to track which one goes where", in other words.

The good news is that tune2fs will happily make up a new value and will replace it for you if you so choose. The bad news is that anything you did which references that filesystem by the old "568c..." value will break when you do that. So, be sure of how your mounting stuff works before you go messing with such things!

If it's not breaking anything for you, it's probably best to just let it be. Untangling a boot mess on these weird little boxes is anything but fun.

Note: if you're actually on a recent Raspberry Pi, odds are good that your /etc/fstab will be using *partition* UUIDs (PARTUUID=...) and not those of the filesystem. If in doubt, use blkid to see what's what.

A reader wrote in asking what I thought about the whole GitHub "copilot" thing. If you haven't heard of it yet, it's apparently this thing that was trained on a whole mess of repositories and their code, and you can "prompt" it and it'll spit out code that may or may not do what you want. It doesn't seem to know about licensing (BSD, GPL, MIT, that sort of thing), or plagiarism, or any of that other good stuff.

When this hit the news last summer, I had some thoughts about it but held back at the time. Now, maybe it's just the caffeine talking, but I feel like going for it. So, here we go.

This is actually a story about somewhere I used to work, and someone who was on one of my teams back in the day. One day, I heard that one of the people in a far-away office had created some kind of tool that would let you look at the status of different machines - that is, Linux boxes. I don't remember the finer points of it or why this person even bothered, but they had this thing and wanted to get the code reviewed in order to check it in.

It was written in a language which was not common for that company, particularly back then. Most backend stuff was written in C++, with a significant amount in Java. A lot of people did dumb glue things in Python (and paid the price long-term, but I digress). We even had people hacking on site integrity stuff in Haskell and doing linters in OCaml. Frontend stuff was largely derived from PHP. This project was in Go, and so was quite the unusual thing to see in this environment.

Maybe that's why I got to looking at it more closely. It just felt "off". Something about the code didn't feel right beyond the fact that it was in a new-to-me language, in a new-to-the-company language, and for a brand new project. Maybe it didn't really match up with the problem it was supposed to solve at the company.

Finally, I just grabbed some bits of code that looked fairly specific and hit up the web to see if they came from anywhere. I figured it might be Stack Overflow or something. It wasn't.

This stuff was cribbed straight from some GitHub project. It had no attribution, and it had no license. This was a problem in so many ways. This company was rather particular about how and when it would add external code to the tree. They had deep pockets and were worried about license violations, lawsuits, and (worse still) patent violations and the associated lawsuits from *those*.

There was this whole thing you'd do in order to bring in an external library. First, it had to be suitably licensed. You'd run it by the local "open source czar" who would approve it. Then you'd add it to this "third-party" part of the tree with all kinds of info about where it came from and how it was licensed. It would then be built that way, and you could pull it in from there.

None of this happened here. This code was ripped off wholesale and was just jammed straight into the project.

Now do you see why I tell this story in relation to a question about Copilot? It's like, yes, there's now a coder tool which does this. But... there have been "coder tool" *people* who have been ripping off code and jamming it deep into their company's code bases for a very long time. This might make it easier to do and even harder to prove when it does, but it is by no means a new situation.

There are some interesting implications to the way that you have to handle references to other bits of code when you write a C++ header file. There are things you can do which turn out to be suboptimal, and until or unless you work someplace with a coding style document or get some mentor who's already been there, you're likely to make some mistakes.

There's this thing we tend to do when we need a class to own an instance of another class. Maybe that class is going to do some work for us, or will hold state for us, or maybe it's expensive to set up, so we want it to hang around for a while. Who knows, right?

Anyway, a common way to do this is to set up the outer class such that it has a pointer to the inner one. You *could* have it just own it directly, but then that introduces problems.

That is, you don't want this (in your foo.h, inside the Foo class):

 private:
  Bar bar_;

What kind of problems? Well, for one thing, you now have to #include bar.h in your foo.h at that point, since it's "really there". The compiler needs to know how big a Bar is at the point it's dealing with the rest of your Foo class. It can only do that if you #include it.

Of course, bar.h probably has its own dependencies which get #included, and so now you end up with this evil web of slow compiles. It sucks.

* ... Scooby-Doo wavy lines... *

Hey, it's me from two days after this post was originally written. It turns out that's not actually true. You *can* in fact pass in arguments even when the Bar is "baked in" to your class. You just have to do it when you get constructed, like this:

Foo::Foo(int something)
  : bar_(something) { ... }

So there you go. You can do it that way if you want. Obviously, you need to know exactly how you intend to construct it fairly early on in your lifetime. If that isn't known until later on, you're still kind of screwed, and wind up doing something else like one of the later methods.

I'm doing an inline update here and striking through the old stuff so (1) you can see my mistake (2) I can own it and (3) nobody gets the wrong idea from my mistake sitting out in the open forever.

Thanks to Jens for the first comment reporting this. No thanks to anyone posting in the usual places calling me stupid for not knowing this. It's like, no shit, I don't know everything. Good thing THE ONE does.

* ... Scooby-Doo wavy lines out ... *

What used to happen next is that someone would say, okay, let's make it a pointer instead, and this would be their next iteration:

 private:
  Bar* bar_;

This by itself meant you now have to call new on it somewhere in the actual body of the class to create it, and then you also had to call delete to release it too! Otherwise, you'd leak it any time someone deleted *you*. But, hey, now you could send arguments to its constructor, since they'd get jammed in when you did the "new".

The switch to a pointer here did allow one other neat trick: you could now drop the #include of bar.h and just forward-declare it, like this:

class Bar;

(And yes, you might've had to wrap it in some namespace { ... } stuff if it wasn't in the same namespace as you.)

That's nice, but of course now you have the manual memory management thing going on, and that truly sucks. For a long time, that was the best we could do, then people started rigging their own smart pointer wrapper classes. I picked up "scoped_ptr<>" from my employer at the time, and after leaving that company, rigged it up in my own stuff at home. I used that for years until C++ evolved a bit more and eventually gained unique_ptr and shared_ptr. Then I switched to those.

This means we can all collectively do this now:

 private:
  std::unique_ptr<Bar> bar_;

You have to #include <memory> for the unique_ptr itself, but you still don't need to bring in bar.h in your header file, so the compiles stay relatively uncomplicated. You get automatic destruction when you go out of scope, so that's nice too. You just have to wrap your head around not doing stupid things with the lifetimes of your stuff, same as always. (If you're using .get() on a unique_ptr and old C library stuff isn't involved somewhere, WTF, over.)

Another side-effect of coming forward a version or two in the C++ standards is that you get things like std::make_unique and std::make_shared, so there will be no more of this "call new directly" crap. This means you can now put "new" on the list of things that your linter can scream about, and start banning it from the code base.

That's pretty much where things are now: we do these smart pointers because we want things to suck less than bare pointers, and we do pointers at all because it avoids this whole multi-suck thing of #include hell, terrible compile hell, and default constructor hell. (Clearly, C++ programmers have many hells.)

Honestly, it's kind of stupid that you have to do any of this, but when you remember the history of how things got to this point, it's kind of understandable. It still sucks but at least you can tell how it happened.

It would be a lot better if this kind of stuff was handled some other way that didn't make you deal with so much of the housekeeping. It's like, yes, class Foo is going to hang onto a Bar. That instance of Bar needs to exist as long as Foo does. We might want to configure that Bar somehow. Oh, and we might even need to *swap out* the Bar somehow.

Yep, that's the last part here: once you moved from "a direct Bar in your class" to a pointer and beyond, you got the ability to replace it with something similar. The usual reason for this is when you're doing a mock, like with gmock. You inherit from Bar and make a MockBar, and it has all of the right curves and angles in the right places, so it just "fits" where everything else is expecting a Bar.

I've been down this road many times. It usually involves creating the outermost class, then reaching into it from inside the test (hello, private member access) to change some of the smart pointers to point at a MockWhatever I had created in the test class. This was back in the days of using scoped_ptr so we didn't have the make_whatever yet. It looked like this:

  Foo foo;                               // the thing we're testing
  MockBar* mock_bar = new MockBar(...)   // what we're injecting
  foo.bar_.reset(mock_bar);              // and here we inject it

Since the Foo constructor usually created a Bar for itself, this meant you'd build one and immediately toss it out. If Bar's constructor didn't actually do any work, this was relatively safe (and quick). Bar's constructor might've been written like Foo's constructor which just creates instances of classes it intends to use later. There were classes which did non-trivial work in their constructors (fools!) and so all of this got very costly. Tests would run for *multiple seconds* instead of mere milliseconds (or less).

You'd end up needing to some truly crazy things to work around this. One way was to change your code, and so now Foo's constructor would do *nothing*, and then Foo::Init would actually create things it needed (like a Bar). This also meant you couldn't really test Foo::Init, since it would go off and create those real versions of the objects and you couldn't intercept them.

Another way was to have multiple constructors: one that normal people called which did all of the sub-class creation stuff, and another one that was private (again, hello to the friend class stuff) that accepted a Bar and whatever else you cared to pass in. Then your test could just "slip in the back door" to create a Foo. This did mean your actual constructor went untested unless you went to lengths to do that directly, and then that generally wasn't great, either.

Then there was still another method where Foo would have a function like MakeBar() which would *create a Bar on the spot* and would return it to the caller. But, instead of calling it directly, you'd then have a glorified function pointer referencing it. We didn't have std::function at that point, but the company had its own magic called Closures and Callbacks, so you'd do it that way.

It looked a little like this:

Foo::Foo()
  : get_bar_(MakeBar) {}

Bar* Foo::MakeBar(...) { return new Bar(...); }

Then, in Init (or whatever part of the code needed a Bar), it would do this:

  bar_ = get_bar_();

get_bar_, then, was the glorified function pointer.

So, still with me? What happened next is that the test class would *reach into Foo* (private access yadda yadda) and would reset its get_bar_ pointer to instead call something inside the test class. Yes, really.

  Foo foo;
  foo.get_bar_ = NewPermanentCallback(TestClass::MakeMockBar);

[Don't hold me to the exact syntax of their NewPermanentCallback stuff. It's been over a decade, okay? Just think "it created a thing that would call another thing".]

Now you could call Foo::Init, and it would call its local get_bar_ pointer which would then follow the redirection into the test class. The test class would return a pointer to the MockBar it had created for the occasion.

This was seriously annoying, but it was pretty much all you could do when you had code that created instances of things "mid-stream" (like within a member function) and you needed to mock them out. That is, if you had Foo::DoStuff that created a KernelInstaller and kept it local to its own scope, you couldn't do any of the top-level stuff described earlier. Your only hope for injecting a mock was to rework it to call a thing that generated a KernelInstaller, and then you'd intercept *that*.

Awful, right? Just describing it is a chore. Doing it was worse.

I haven't even gotten into the realm of whatever inefficiencies you'd pick up by bouncing through a smart pointer instead of doing something else. I'm sure it exists and matters on some level, and there are undoubtedly people who have to care about it. I've managed to not have to worry about that quite yet. I don't want to discount it as a problem because it surely matters to someone out there, but handling it is an even bigger problem.

One related thing to the whole Mock injection situation is that you have to make everything in Bar "virtual" so that it can be overridden by the MockBar child class. That's another potential layer of indirection with whatever performance impact that might add.

Apparently, if that's your world (it's not mine...) then if you're going to "inject a mock", you end up having a MockBar which looks *EXACTLY* like a Bar, and then you do some build system trickery to wrangle that one into place when you go to build your test.

That is, in normal life, foo.o and bar.o turn into server_bin, but in test life, foo.o and mock_bar.o turn into test_foo. The obvious problem with this is that you have to be very careful about not breaking that interface. Maybe mock_bar.cc would #include bar.h instead of having a lookalike mock_bar.h? Also, it would still have to be *named* Bar inside mock_bar.cc, because that's what the linker is going to be looking for.

At the employer in question, they called this scenario the "high-performance mock", and I am very happy that I never had to use it.

So, taken all together, I like to think of this body of knowledge as an "invisible hand" (in the economics sense) that reached backwards in time and influenced a great many design decisions. If you intended to test a bunch of things thoroughly, you had to start from a bunch of design elements that left the door open for you later. Trying to retrofit something in order to make it testable later was anything but fun.

Yesterday, I put forth a bunch of API possibilities for the problem of needing to return a value but also indicating the lack of a value and being able to comment on why. In it, the notion of a "wrapper" around a string was presented, such that it might look like "Result<string>", and those brackets started looking like a pain. There was also the possibility of having something like "ResultString", but didn't that open the door to having a bunch of code duplication?

That's about where I was with my own projects about a year ago. I wanted the Result behavior, but didn't want to deal with templates and those brackets everywhere, or the duplication of code behind the scenes. Still, after some messing around, I came up with a proof of concept that only worked for strings, and started converting bits of my code to it. As things would get some attention for maintenance purposes, they'd get converted from one of the other ways to this style.

After this had existed for a while, the general happy feeling of having this stuff in place became infectious. I wanted it to cover other things with it, too. I wanted it to handle something a bit more complicated than a mere string, and so decided to try my hand at the template-wrangling metaprogramming cruft that would be necessary to make it go.

What ended up happening is that now there's a "result_template.h" which defines a "ResultT" that can be wrapped around another type, so you get "ResultT<string>" or "ResultT<vector<string>>" or whatever else. But, I wanted to insulate the rest of the code from all of those damn brackets, so any time a type like this has been created, it gets its own little header file.

Now, resultstr.h includes result_template.h (and <string>, naturally) and then just does "typedef ResultT<std::string> ResultStr;". That's it.

ResultU16 wraps a uint16_t. ResultU64 wraps a uint64_t. ResultVecStr wraps std::vector<std::string>. All of them have the same key behavior: you have to initialize them to either a value or an error, and then you have to check them before attempting to use that value.

It looks like this in practice:

ResultStr feedback = post.GetVar("feedback");
 
if (!feedback()) {
  // call the thing that spits out a HTTP 400 error
  return false;
}
 
if (feedback.value().length() > kMaxStrLen) {
  // spit out a HTTP/400 here too
  return false;
}

// use feedback.value() in prepared statement for INSERT

Without that call to feedback(), the call to value() later on would abort the program. This means it's pretty difficult to ship code that does the wrong thing, since it will never actually work. You'd have to ship code that merely compiled and didn't run. I mean, sure, people do that kind of crap, but there's only so much I'm aiming to fix with this.

There are some other bits like that in here, too: calling error() is only permitted after doing that check... *and* having it indicate a problem. If there's no problem and you call .error(), it'll abort the program. Likewise, if there IS an error, and you call .value(), it'll also abort the program.

So, to reiterate:

Someone initializes it to a value or an error and returns it to you.

As the caller, you have to check for success or you can't call anything else on it. If you're not going to check, all you can do is throw it away.

If you get back a successful result, then you can call .value(). But, if you call .error(), it will kill your program.

If you get back an error result, then you can call .error(). If you call .value() on that same object, it will kill your program.

And yes, it's possible for an evil person to write something that throws away the value of the check and just goes on to use .value() directly, like "if they don't want to deal with it", or "it makes the code messy". I imagine this would become common with some of the people who starred in one of my previous stories if they encountered this kind of Result scheme in their career.

Personally, I think I'd treat "deliberately throwing away the safety" along the same lines as how I'd treat an employee who deliberately blocked a fire door for their own gains. I don't think they'd stay an employee for very long, in other words.

Anyway, this is the sort of "spring cleaning" I've been doing of late. It's a neat concept I picked up from reading about other programming languages, and I encourage other people to give it a spin. You might like what it does to your code.

If you agree with the notion that you need to be able to tell the difference between the absence of a value and a value itself, then this has some impacts on the code you write. If you want to keep 'em separated (so they can come out and play), then it gets you looking at your API designs in certain ways.

These are my reactions to some of the ways a value can be returned (or not).

Imagine we're talking about a helper function that is intended to be run inside a program that's handling the access to a web page. It's probably a CGI program, but it could also be buried inside some much bigger thing, too. This is all about C++ styles, but you can probably see slices of other languages in here, too.

string* UserIP() -- Oh great. It's a pointer to a string, so that means it got allocated somewhere in there, and we have to worry about the lifetime of that buffer. Guess it's being created from scratch, and then we have to free (delete) it? But wait, what if the allocation fails? I guess it can return a NULL or nullptr or something. Also, what happens if it can't find the IP address? I guess it could return a NULL/nullptr/whatever for that, too. I sure hope everyone who uses this checks for it...

string UserIP() -- No more lifetime worries here, but oops, uh, well, what if we don't know what it is and need to provide an error? If it's a CGI situation, maybe REMOTE_ADDRESS or whatever wasn't defined for some reason. What then? Do we throw? Yuck! Do we return some magic value? Double yuck! Don't say return the string "unknown", since what if there's a host on the network named that, and you end up using that as a real value somehow? You just know someone would do it. Just ask the guy who has "NULL" as his license plate.

string UserIP(string* errmsg) -- Oh geez, I'm going to be sick. Sure, now they can "set an error message", but now how do you know when to look at errmsg vs. the return value?

bool GetUserIP(string* ip) -- Well, this is a little better, right? It's kind of wonky because now you're wrangling outvars and the user could do something stupid like handing you a bad pointer, plus that means THEY are having to think in terms of addresses and pointers and that's probably not great, either. But hey, at least you can point out that something failed. Too bad you can't say what.

bool GetUserIP(string* ip, string* errmsg) -- Okay, so now we can include a message if we want to provide some color commentary on what happened, but geez, now it's TWO outvars. Imagine what the call sites are going to look like.

Result<string> UserIP() -- Now this is getting interesting. You can do some fun stuff, like having a flag that only gets set once the Result is checked for correctness. Only then is .value() allowed to be used. If you call .value() first, YOU DIE. This should catch you much earlier in development, since even "happy path" cases will blow up if it's not being used correctly. You also have an .error() to return some nice message in the other case. It's a pity about the whole <> thing though. Kind of nasty-looking... and annoying to keep typing over and over and over.

ResultString UserIP() -- Same idea as before with fewer brackets, but oh dear, did someone code this up as a one-off? Now what happens when we want to do this with other types? Do we copy and paste the magic "you must check for correctness before calling value()" gunk into ResultInt, ResultFloat, ResultDouble, and so on? How do we keep them from diverging in the event of a bug fix that only touches one? Don't we really need the ability to do generic programming and templates?

string UserIP() -- Wait, didn't we do this already? Not quite. This time, we're in a slightly different language with different assumptions. The native "string" type now includes all of the correctness checking of ResultString (aka Result<string>), so now you have to check ALL OF THEM. Now it looks simple enough here, but you have to check everywhere. EVERYWHERE. Only then can you use .value() from it. How much extra CPU and memory goes into this for the cases where it'll never be "missing" and will always have some usable value? In fact, how often do we have the one case instead of the other, anyway? What percentage of our strings are "yep, always gonna be something" versus "might have failed completely"?

It's complicated, right? To win a little in one place, you have to spend a little more in another place.

February 21, 2022: This post has an update.

In yesterday's post, I mentioned taking my old Mac Mini out of the OS X/macos ecosystem and bringing it into Debian, and then what kind of hilarity followed when I attempted to install a Javascript linter (and it tried to drag in 180+ node-related packages).

This and some other mentions have prompted some questions, mostly along the lines of "Debian? What happened to Slackware and green-on-black?"... and I guess I can answer some of those.

Readers are advised that this post will probably containdefinitely contains an even higher than usual amount of navel-gazing as I dish out the details of my setup. My usual disclaimer of "this works for me and probably won't work for you, so if you have different tastes, then great, go with that" definitely applies here.

Right, so, with that out of the way, let's get into a discussion of what I'm actually using to get things done these days.

First of all, the one that most people actually notice is the actual box which is serving rachelbythebay.com, which at the moment is a box called snowgoose. I had installed it in early 2020 as CentOS 8 because it had advertised a final end-of-life date of 2029. I figured that was awesome. Of course, as we know now, they pulled the rug out from under us that year and decided that it would die in 2021 instead.

This got a whole bunch of projects off the ground to do something about it, and one of the several which survived is Rocky Linux. It pretty much is the CentOS for 2021 and beyond, and apparently has some of the same people involved, too.

I followed this project with interest, and fretted about what would happen if there wasn't an easy way to migrate. Fortunately, they did a bang-up job and came through with a migration script, and I ran it. Chances are, most people have no idea I ever did this migration, because nothing changed on the site itself. People who are looking really closely for the finer details (like Netcraft...) would have noticed that the "Server:" line emitted by my Apache changed subtly at one point. That's the only outward-facing difference as far as I can tell.

Basically, if I didn't tell people that it happened on September 22, 2021, would they have known? Apparently not. There was even a reboot involved just to verify that everything was solid in terms of the kernels and boot loader action. Everything worked great.

This migration didn't happen on a whim. I actually tortured a local machine into installing CentOS 8 and then migrating to Rocky in order to see what would happen and to have some idea of what to expect. The whole thing happened that afternoon.

Obviously, I am grateful to the Rocky Linux folks for delivering on something that saved a lot of people a whole lot of work.

...

But, okay, that's my server. People want to know about my machines at home, since that's where Slackware entered the picture. This is where I'll just drop the bomb: I stopped using Slackware in January. This was no small thing. I'd experimented with it late in 1993, did my first real install to a dedicated disk in 1994, and had at least one workstation box running it full time (having erased the DOS install) since 1996.

It was a nice, simple distribution that stayed out of my way. I used it for a handful of things, and it did those things well enough for my needs.

The problem is that the world didn't sit still. In particular, the software-defined radio world of GNU Radio keeps changing, and it keeps dragging its dependencies along with it. I found that keeping it current on my Slackware box had become too much of a burden. In particular, I ran afoul of VOLK (something they spun off a while back) requiring cmake "3.8 or higher", and the version of Slackware I was running had 3.5.2.

Now, I can build my own cmake. I've *built* my own cmake. I could have done this. But, I also noticed that cmake-using people tend to lie about what their stuff actually needs. I'd managed to build other things just fine by hacking their stuff to set the actual "required version" to 3.5.2.

Of course, once I had done that, then this happened:

GNU Radio also has heavy dependencies on Python, and there's no way it was ever going to play nicely with the ancient version my machine had. Also, there's no way I was going to try to wrangle a Python 3.whatever install onto this box.

While this was happening, I had put Debian on my Mac Mini as sort of a long-term shakedown test to see how it did. I even went as far as installing X and my usual set of X-related things to see how it would turn out. It all went pretty well, and soon I found myself actively building things over there since so many of the hurdles had been removed.

Continued development on my SDR-related stuff meant that it got ahead of what the GNU Radio install on my main workstation (the Slackware box) could support, and before long, I had to block those targets from my "build the world" tool. I could no longer work on them directly on this machine, since they relied on GNU Radio APIs that only existed in the newer versions, and those newer versions wanted too much done in order to happen on my version of Slackware.

Now, I do mean this was a long-term test of Debian. The Mini got it in February 2021, and my main workstation got it in January 2022. That's how long I remained open to the possibility of staying on Slackware.

Towards the end of 2021, it appeared that Slackware 15 would be coming out soon, and I had to ask myself some hard questions, like if I wanted to bother with it. In being brutally honest about the situation, I realized that it would come down to "install almost every package" or "play whack-a-mole with the inter-package dependencies". I didn't want to do either.

Back in the 90s, I actually had a pretty good handle on which packages required other packages in the Slackware tree. I could probably tell you that if you wanted to install such and such, you probably also would need this other thing, or it'd complain about a missing library or something. I couldn't (and didn't) do this for the KDE and GNOME stuff, naturally, but for things in the "AP" or "D" or "N" sets (applications, development, and networking, respectively), I had a pretty good feel for it. (You would hope, given that it was my job to keep a few dozen Slackware boxes running back then.)

At some point, even that stopped working. Somewhere in the early 2000s, there were just too many interconnected packages. I'd watch a system being installed and see all kinds of random stuff going by that I had never seen before - "WTF is pango? expat? Huh?" - you get the idea. I realized that the strategy had turned into "install *everything* from the L (library) set", and that's just awful.

There once was a time when you could do a "custom install" and be thoughtful about which packages were installed, and end up with a working system. These times are not those times. Those times are gone.

Having failed to convince myself to install the world, run into random issues from things I hadn't realized were needed, or throw down and code something up to solve my own dependencies, instead I created and took the fourth option: leave.

So, one day about a month ago, I headed out to one of the few computer stores left in the Bay Area and picked up a couple of SSDs - from two different manufacturers, so as to avoid "same batch" woes, naturally. Then I came back here, got 'em mounted in the case, did an install, and started pulling things across.

It didn't take too long to get back to where I wanted to be. Now, there have been growing pains and certain challenges, like when apt wanted to remove almost everything on the system one time. At this point, I think I have a handle on how their state machine works now, and can avoid doing anything particularly stupid to myself. I even upgraded the mini from Debian 10 to 11 without killing it. Hooray!

As for the green-on-black and whatnot, that's the thing: very little has changed about my day-to-day on the machine. It's running the same window manager (fluxbox) with the same theme, the same terminal (urxvt), the same color scheme (yes, really, green-on-black), the same AfterStep/WindowMaker era widgets I've relied on for years, and so on. Even the fonts are the same.

What's different? Well, I can run my current SDR stuff on here again. I can do other things like screw around with gqrx because Qt5 is here, too. Previously I had to do that in a "lab VM" on some other machine. Certain things also have gotten easier, like anything involving audio.

What's also notable is what *isn't* on here. If I see a package that looks useless, I can give it a tug and see what depends on it. A lot of times, it's just more useless stuff, and so I cast it all out. Do I want bluetooth on here? Nope. Gone. DOS filesystem utilities? Nope, seeya. modem manager? WPA supplicant? laptop tools? Don't need 'em. This is a box that isn't a laptop, has no modems, and has no wireless cards. So, I toss that out and shrink my "exploitable footprint" slightly. (Every daemon, suid or sgid program, or other funny hotpluggable thing with magic privileges is a potential problem.)

Being able to safely remove packages like that is great stuff.

Is Debian perfect? Nope. But it works well enough. Is Slackware broken? Nope. It's true to itself, still, after all these years. The difference is that I don't feel like that works for me any more.

I mean, I also used to run my own mail services. After 30 years of that (not an exaggeration: 1990 to 2020), I finally gave in and moved it to being hosted somewhere. It's the same deal: I paid my dues, and now I'm going to focus my efforts on something else.

...

As for the rest of my "daily driver" situation, it would be dishonest to not mention the ever-present Macs in my life. My desk has two monitors, two keyboards, and two mice. One set is for the aforementioned newly-Debianized Linux workstation, and the other one is for my MacBook Air. The old MacBook Pro sits off to the side, not doing much.

I tend to do all of my "real work" on the Linux box: writing, programming, managing other systems that are far away, etc. Basically, if it involves a terminal, I'm doing it right here ... *even if* that means sshing BACK to the Mac. Terminal stuff on the Mac just feels wrong, and it's a constant source of friction when trying to do actual work. It's been like this for a long time, and I stopped trying to figure it out or fight it many years ago.

At the moment, the Mac is fine for my web browsing, e-mail, photos, and music. This comes and goes. There was a point when it looked like both OS X and iOS were on a straight shot to usability hell, and I wasn't sure what would happen next. Somehow, things shifted again, and they stopped being so terrible for me, and I moved along. It took a while, though.

...

Finally, for the sake of anyone who really cares about the finer points, here's the full dump of way more crap than you actually ever wanted to know.

"Real work" machine: 2011 pile of parts originally specced out for a SDR project back in the day. Gigabyte motherboard, 8-way i7-2600 at 3.4 GHz. 16 GB of memory. A pair of 2 TB SSDs, hooked up via SATA (hey, it's a 2011 build). HP 2509b monitor. Logitech K120 keyboard that's seen so much use, it has nail indentations on some of the keycaps. Some random Logitech (wired) mouse of a similar vintage - apparently a M500 or M500s.

OS: Debian Linux 11 (bullseye). X. fluxbox. Flux theme. Toolbar set to auto-hide. urxvt. -bg black, -fg green, 7x14 font. wmCalClock and wmcpuload in the upper right corner. Other widgets may join the permanent set.

Dev environment: aforementioned green-on-black urxvts starting bash. nano editor. C++ (14). My own little build tool that understands that '#include "..."' is enough to figure out what needs what. No build scripts, no Makefiles, no cmake, no Java services, no funny stuff. No IDEs (gasp!).

Frequently used libraries: protobuf, jansson, gtest, gmock, gnuradio, osmosdr, uhd, libmp3lame, mysqlclient, libpq (Postgres client), libgd.

"Rest of the world" machine: 2020 Intel-based MacBook Air - the first one with the fixed keyboard, that really really sucked at Thunderbolt for basically all of 2020 (and looks to have regressed -more on that in another post). Last one before they went to their own "sillikin". 4-way i7-something or other at 1.2 GHz. 16 MB of memory and 2 TB SSD, no doubt soldered to the board. OWC Thunderbolt 3 dock. Apple Thunderbolt monitor ca 2014. Same models of keyboard and mouse as the Linux box.

macOS Monterey, 12.2. Oops, guess I should patch that (but apparently it won't fix the Thunderbolt stupidity that came back in 12.2). Macports.

Desk: something bought at IKEA in East Palo Alto about 15 years ago. Looks like a "Jonas" based on a sticker I just found. Whatever that is.

Chair: whatever I could find at the office supply store when the last one decided to fail. These things are all crap. It's not the least bit special. It has a seat, a back, some wheels, and some adjustable parts. Just like everything else. It, too, will fall apart eventually.

Music: I still buy CDs when I can. Airpods Pro. Usually playing my own library (which is largely from those CDs, else iTunes store), not streaming. (I also had a "land line" until 2012 or so. My lawn: get off of it.)

And, in case you couldn't get enough, the other two notable machines in my life:

"Distant server in a rack I'll never see": Some kind of SuperMicro cruft. 8-way Xeon E-2174G at 3.8 GHz. 16 GB of memory. 1 TB no-joke hard drive, presumably SATA. Pair of NICs running at 100 Mbps to the outside world, because I don't need more and the redundancy was free.

Rocky Linux 8.5. Apache httpd. MySQL. A bunch of the compiled versions of the C++ crap I write on the "real work" machine. CGI.

"Workhorse Mac mini": mid-2011 Mac Mini, 4-way i5-2415M @ 2.3 GHz. 16 GB memory. 1 TB SSD, SATA.

Debian Linux 11. Apache httpd. Postgres. A bunch of other dumb compiled things for projects I haven't written much about yet, like why I was wrangling Javascript the other night. More CGI.

I'll skip the Raspberry Pis since they have no personalities to speak of, but they'll show up in some future post where I talk about what they have to do with SDRs, Postgres, Apache, and yes, a bit of Javascript.

...

And there you have it. I resisted doing this kind of post for a long time. I hope it was worth running afoul of the haters.

Oh sure, everyone laughs about things having a lot of dependencies, but do they ever really think about the implications of having hundreds of teeny little libraries, all moving independently? This is how it manifests in the real world.

I put Debian on my Mac mini last year after it finally reached the point of no longer receiving even security fixes from Apple. It's over 10 years old, so I'm more than fine with that. It makes a nice little Linux box, and so it won't be turning into e-waste any time soon. I'm very happy with the situation.

My relative inexperience with Debian systems means I'm constantly being delighted to discover that such and such has already been packaged up and is just an apt command away. Things that I would have done by hand in the years up to this point now have someone else worrying about them.

Unfortunately, the packagers can only work with what they've been given, and in the case of certain ecosystems, that means an explosion of crazy. Case in point: the other night, I was working on some Javascript code (unavoidable for the problem at hand), wanted to see how much I had screwed up, and thought "hmm, there was this jshint thing that would show me where I did something stupid back in the day". I went looking and didn't find that, but did find something called "eslint". I figured, okay, es, ECMAScript, that's the fancy name for this stuff now, should work, right?

This is what happened next:

... huh. The story basically ends there. I'm going to handle this some other way.

Oh dear. The software fungus where bits of code take the literal characters "HTTP" as a number has expanded. Now it has a variant: 4740180.

What's that? 4740180 is 0x485454... that's "HTT".

Where does this one come from? Go and gRPC, apparently!

Again, it comes down to "HTTP/1.1 400 Bad Request" being parsed as part of some binary protocol.

I found this while searching the web for "1213486160" to see how things have been going in the past few years. Unfortunately, it's still biting people all over the place. There are far too many posts asking people to upgrade their software, or restart things, or do any number of useless rubber-chicken-waving techniques which ultimately will accomplish nothing.

When you see a funny-looking number, figure out where it comes from. That's the only way to have any chance of understanding what's actually going on.

Yep, it's time for another round of reader feedback.

An anonymous reader writes in response to the "process infection" story:

comment: Hi! Loved your recent post on environment inheritence, and the note about system's approach reminded me of something relevant that Pottering recently wrote aboit the on the Fedora mailing list (highlighted in LWN's article on Fedora and pkexec @ https://lwn.net/Articles/883547/

The thing with setuid/setgid is that the invoked privileged process inherits a lot of implicit state and context that people aren't really aware of or fully understand. i.e. it's not just env vars and argv[], it's cgroup memberships, audit fields, security contexts, open fds, child pids, parent pids, cpu masks, IO/CPU scheduling priorities, various prctl() settings, tty control, signal masks + handlers, … and so on. And it's not even clear what gets inherited as many of these process properties are added all the time.

If you do privileged execution of specific operations via IPC you get the guarantee that whatever is done, is done from a well-defined, pristine execution environment, without leaking context implicitly. The IPC message is the *full* vulnerable surface, and that's as minimal as it can get. And that's great.

[Yep, that's a nested blockquote. Because why not.]

Oh wow, thank you for mentioning that, unnamed reader! I definitely read that post on LWN at some point in the past couple of days when the pkexec thing started going around. It must have really worked its way into my head, and then eventually stirred up the memory of everything getting "nicely infected" at $OLD_JOB.

It's really a nasty situation if you think about it. There is a ridiculous amount of state out there, and nobody really knows what all of it is. We try to fool ourselves with these namespace hacks on Linux that some people call "containers", but they aren't the full story and probably never will be.

Every time I talk about trying to partition a machine at this level, I can hear my dad the retired mainframe programmer laughing at us all. Those machines knew a thing or two about actually dividing themselves into useful smaller pieces. Right now, on these dumb little (x86[_64]) boxes, we're still just playing with toys by comparison.

Imagine how nasty it is to have to fork + exec to start a subprocess just so you can run something and find out how it turns out. It runs afoul of multi-threaded locking and all kinds of other things that you aren't "allowed" to run post-fork and pre-exec, lest you trip over one of the many, many potholes and bear traps we've left behind.

Just think if you could say "something, please run X for me, and let me poke it with long sticks over stdin/stdout, and let me know when it finishes, and how it finished". Kind of like inetd, but for purely local invocations of various things, right?

At that point, would you even need to run everything as the same uid? I bet you wouldn't. You could wall off your web browser from your mail client from your music player from your USB-powered mug warmer controlled from... you get the idea.

Anyway, it just goes to show that sometimes it takes a good LWN post to trigger a memory to get me to post about something myself.

...

Regarding the SWC BBS picture, another reader had this to say:

That picture of the BBS that ran on all those PCs just reminded that in the same period, the well BBS ran 64 phone lines off of one sequent with 1.6 GB of RAM. Sure, it cost 25,000 1990 dollars, and they used it as their primary heat source for their office in Sausalito, but it strikes me as a much better setup than a bunch of PCs running DOS.

That brings back some memories. Now, I never called it, but even I had somehow managed to hear about it through the grapevine of what passed for a social environment back then in those dialup days. For those who haven't crossed paths with this thing, it stood for (stands for?) "Whole Earth 'Lectronic Link".

So at a glance, $25K of 1990 dollars sounds crazy high, right? You could buy a really nice car for that much money back then. For reference purposes, apparently that translates to about $53-54K here 32 years later.

Anyway, it sounds like a lot, but would it have been, really? All of those PCs in there must have cost a pretty penny too. Unless they were buying the absolute cheapest crap at wholesale prices, I have to imagine that having that many computers around would have cost a similar amount.

Maybe what happened is that it crept up on them. Maybe they had four or five, then five or ten... then a dozen... then two... then thirty... fifty... and the next thing you know, you are wrangling a "cluster" of MS-DOS machines. THE HORROR!

I'd have to dig around old magazines to get a feel for how much this stuff cost back in the day. I mean, I know what my family's machine cost, and it was not cheap. In the fall of 1989, we got a 386DX-25 with 4 MB of memory (expandable to 8), an 80 MB SCSI (!) drive, both sorts of high-density floppy (5.25", 3.5"), a VGA card and matching monitor, and a keyboard. It came out to an astonishing four thousand dollars... and 32 cents.

I actually kept the details from that original order form, so here's how it broke out.

• 386-25: $1649
• 80 MB (disk + SCSI controller): $519
• 1.44 MB FDD: $89 (assuming the other one was in the base price)
• VGA (assuming board + monitor): $569
• Math coprocessor (20 MHz): $479
• 4 MB memory: $399

That's $3704, and if you multiply that by 1.08, you get $4000.32 on the nose. I guess the local tax rate was 8% back then. (My dad picked it out, and he apparently spared no expense. Wild.)

Now, for a node of a BBS, you could skip the fancypants display, and go with the absolute cheapest POS you could get which would still let the machine boot. (It has to have SOME video card to placate the BIOS.) Grab some garbage CGA or MDA card from a machine that's going to be scrapped. It's not going to be hooked to a monitor for very long so who cares?

You wouldn't need either sort of floppy drive, so drop those from the bill of materials. (Yeah, the "BOM". You pick up some interesting terms when working at some of these places. Don't say this one out loud in an airport or on a plane, okay?)

I'd hope that you wouldn't need a math coprocessor to run a BBS, even if it had some nutty door games, so drop that right there.

If it was netbooting, then maybe you could do without a hard drive too? Those Netware-friendly NICs could have PROMs put on them to boot straight off the network, so no hard drive and no SCSI controller.

At this point you basically have a motherboard, a processor, some amount of memory, a really crappy video card, and a serial port, right? Maybe you could go really cheap on those? Maybe you don't need a 386, and can use a 286 or even an 8088. How much would it cost, anyway?

Here's the catch: to come in under $25K (the cost of the WELL mentioned above) with the 64 machines shown in the picture, each one would have to cost no more than $390 each. (390 x 64 ~= 25000). Could you get below that?

Then you find out that they either had twice as many machines, or were doubling up on some with multitasking. That either means now you have to make 128+ lines in that $25K (so about $195 per node), or you have to somehow get enough CPU + RAM to make the multitasking "work" on them.

All of this is just me guessing out loud, because feedback posts are entirely off the cuff with no real planning. This is as close as you'll get to me doing improv for you here, I guess.

I wonder how much money actually went into some of this stuff.

Oh, before I forget: that whole thing where it creeps up on you, and you go from having 5 of them, to 10, to 20, to 50...? Yeah, that's pretty much what happens when you let your software people ship whatever they want to "THE CLOUD" and let it just autoscale forever.

If you don't have someone like my friend the badass capacity engineer keeping people honest at your company, your laziest software people will bleed you to death, and the cloud vendors will love them for it.

...

Next up, a bit of older feedback that I'm only now getting around to, sorry about that. In response to the New Year's Day post about 2022 overflowing signed 32 bit integers, a reader says:

comment: This could be another instance:

(crvownersclub link)

Although, on page six someone quotes Honda as saying “We have escalated the NAVI Clock Issue to our Engineering Team and they have informed us that you will experience issue from Jan 2022 thru August 2022 and then it will auto-correct.”, which doesn’t fit. Others however say that Honda told them that a fix would be available sometime before August. Game of telephone and all that.

Fun times!

This one just blows my mind. I read through the posts and tried to make sense of it and tried to figure out just what they could have possibly screwed up to make this happen, and how the "auto-correct" could possibly happen as well. I still don't quite understand.

I'm hoping that someone will take this apart and will explain what's actually going on here. I'm sure it'll be another tale of awful, awful things about timekeeping that leads a bunch of people to drink (as usual). That's pretty much what you expect with that kind of story.

I hadn't heard about this one prior to this feedback, so thank you to that reader for mentioning it!

I once worked at a place that had an interesting little problem stemming from the way they managed machines and some of the fun side-effects processes can have on each other. This is back in the pre-systemd days of running services, such that things tended to be started or restarted by someone or something that itself ran as root. That is, if you started something, it might inherit one or more things from your environment. Likewise, if you ran something with an unusual environment, it might bestow those "gifts" onto anything it happened to touch.

This story involves a lot of weird little things going back a ways. Anyone who used to run OpenSSH before it was a stock part of their Linux distribution probably ran into the problem where you'd apply an upgrade to glibc and then OpenSSH would stop letting people log in. This ultimately turned out to be a mismatch between the dynamically linked bits of glibc in the running sshd and the NSS stuff for authentication on the disk that it would grab on the fly.

After locking themselves out of machines too many times, people generally learned that after you mess with a glibc upgrade, you restart the sshd listener on port 22. Of course, they also had to learn to not whack EVERY sshd at the same time, lest they kneecap the very thing that was letting them on the box, but that's another problem entirely.

Okay, so after whatever summer it was when we had to keep patching these things every few weeks (2002? 2003?), people probably had this drilled into them. Keep that in mind as we go forward.

So we had this system management stuff that worked by having a whole bunch of little scripts that would run to take care of things. It ran everywhere on the fleet several times an hour, and it would fetch updates to those scripts, then run them and upgrade (or downgrade) packages, restart things, and generally keep the boxes up to date.

For various reasons, it ran "niced" to 19, meaning the Linux scheduler was told to take care of other processes which wanted the CPU before it came along. It also ran "ioniced", which told Linux that it should try to take care of other processes who wanted to do disk operations before scheduling this thing.

Normally, that was fine, but sometimes this had an interesting effect.

One time, a glibc update went out. This management stuff noticed and upgraded the packages. Then, because of the NSS shenanigans from over a decade before with OpenSSH, their "postinstall" script for the glibc package included a "service sshd restart". This forced it to stop and restart the listener on port 22 so it would still be able to let people log in.

This sshd got started inside the "doubly niced" environment, and so it too was running with very low CPU and disk priorities. You'd think that might not be a huge deal if it was just the occasional human who would ssh in to these otherwise identical boxes for troubleshooting or whatever. It probably would have been fine, but there wouldn't have been a story if not for the next part.

This company also had a home-rolled "deploy" program that worked... by sshing into the box and running stuff. So, if this machine was supposed to serve cat pictures, it might ssh in and run commands to install the cat picture storage stuff, and then it would run the command to start it up.

It should come as no surprise that the cat picture server *also* started with really loose priorities even though it was the very reason for the machine to exist. If anything, it should have been pushed to the front, since that program was really the ONLY thing that machine should have been worrying about. Everything else was just auxiliary cruft for housekeeping and whatnot.

Imagine trying to troubleshoot this. You find out that machines are getting slow, but only after you do a fresh deploy to them, and then only sometimes. But, once it happens, it starts happening *everywhere*. Then, if you reboot the machine (SIGH... but it happens), then it just vanishes and stops happening until maybe months go by, then it happens all over again.

Someone eventually realized that the cat picture program was being niced/reniced down to nothing, and that it inherited it from sshd, which inherited it from the management stuff, which only restarted it because it was trying to avoid a bigger disaster of having all logins fail from that point forward.

As for why "the processes didn't notice and then undo the nice/ionice values", think about it. Everyone assumes they're going to get started at the usual baseline/default values. Nobody ever expects that they might get started down in the gutter and have to ratchet themselves back out of it. Why would they even think about that?

Truly, this was a way for processes to infect other processes, and this inspired the kind of naughty name one of my coworkers gave it: "unixherp". Yep.

Incidentally, since we don't really start persistent server-type processes directly in the systemd world, I don't think it would happen any more. You run systemctl to say "start X", and then it goes off and does it in its own context. The intent is conveyed without any of the extra funk you might have picked up along the way. How about that.

Okay, so yesterday I posted a picture of an old multi-line bulletin board system which had apparently taken over an entire bedroom with dozens of computers. I mentioned that I intended to come back to the topic to throw in my own two cents, and so here we go.

When you looked at that picture, which way did you react? Were you one of the people who looked at it and thought it was remarkable, and wished you could "drive" such a setup? All of those machines, all needing to be built out and set up, just so your stuff could scale.

Or, were you someone who found it horrific and wished there was a simpler way? Did you wonder if it would be possible to pack all of that into one or maybe two boxes, instead? Did you want a way to negate all of that complexity while still providing the same level of service?

I'll admit: when I first saw that picture in 1993, I was in awe of the scale. It had never occurred to me what kind of layout you would need to run some of these systems at that level of scale. Some small part of me probably wanted to at least experience that, but I never actually went that far.

Now, things are different. I look at that and just sigh at the lost opportunities. The actual amount of work being done on behalf of any of the users who are connected is tiny. If we ignore the fact that the actual BBS software and related accessories (door games...) forced them to run DOS and just focused on the other elements, it becomes possible to see an alternative.

If you wanted to solve for the base problem of having a bunch of people dialing into a machine that would send and receive files, read and write messages, and yes, even playing some games, that did not require a basement full of computers, not even in 1993. Universities, ISPs, and even a couple of forward-thinking high schools were doing exactly that at that time.

There are *large* parallels from those two approaches to what I've been seeing more of as the years have gone by. There are people who go into a situation assuming that they are going to want a whole cluster of machines to do their thing. It doesn't matter that what they are actually doing should be a tiny amount of complexity and it should all fit on one box... or maybe two for some redundancy. They've decided it will have a measurable "footprint" and by god, they're going to make it happen that way.

Meanwhile, someone else who just wants it done comes up with a better approach that actually cares about efficiency, fits it onto the one or two boxes mentioned before, stands it up, and goes on with life.

Part of this has to do with what kind of tools you choose to use. If you are sticking with the 2022 equivalent of MS-DOS and machines that do exactly one thing at a time, then yeah, you are going to have a whole fleet of systems all sitting there busy-waiting on something stupid.

Or, you can make better choices and just have a boring little setup that sits there and crunches through anything you might possibly throw at it. Oh, sure, eventually your business might get big enough to where you have to scale it up, but *IF* (not when) that happens, then you can worry about it.

Don't recreate the basement full of PCs when the problem can actually be solved with a single box sitting in a cabinet somewhere. Put away the urge to check off all of those boxes on your next job search by making your current employer pay for your wasteful shenanigans. Do the right thing and screw around with performance art on your own time.

...

Epilogue: in the hours after writing yesterday's post, I spotted a couple of things which seemed to hit this same point, and so I will share some links with you.

"How do I make the cluster though?" "Don't"

... and ...

"1 VM, 1 Zone, 1 process programming"

This is not to say I agree with everything in all of those posts, but I do like the way they remind people that one or two decent machines can probably do the job... assuming you don't just burn money by building things badly.

Finally, I'll also use this opportunity to again share the link to the COST paper.

I love this quote:

You can have a second computer once you've shown you know how to use the first one.

-- Paul Barham, quoted in the COST paper

After all, how can you have any pudding if you don't eat your meat?