Last month, I had an opportunity to dip into a part of the world I don't normally touch: Apache (as in, the web server) and the way it runs PHP code. This might seem ironic to some people since I used to support a colossal amount of PHP-ish code, but that was done with a virtual machine and had long since evolved out of the Apache ecosystem.

Yep, this is about Apache, php-fpm, mod_fastcgi, and all of that other stuff, and it was all new to me. The question was basically: why is this getting plugged up sometimes? The rest of the machine seems fine, so why is this one part going stupid?

There was a lot of random badness I ended up tripping over, but there was one part in particular I wanted to call out for the benefit of anyone who still gives a shit about trying to build this stuff properly. It has to do with being aware of the situation and not smacking yourself in the head with a shovel if you can help it.

In the above regime, a request comes in, and the web server pushes it down a Unix domain socket to a bunch of PHP worker threads which are hanging out in accept(). One of them "wins" and is rewarded with a new file descriptor for the incoming request.

The next thing it does (and this is fine) is to call poll() because it wants to wait until there's some work to be done. It immediately returns, but it has two flags set, not one. It has POLLIN, sure, but it also has POLLHUP which basically means that the far end has already gone away. It doesn't notice this second flag.

What does it do? If you said "it reads the request and runs it anyway, then sends a response to an uncaring socket that has nobody on the other end", you're right. It gets EPIPE on the write, and presumably some other part of the process gets a SIGPIPE, assuming it hasn't squashed them in its signal setup.

Let's say these requests are relatively expensive and take up to 5 seconds to run. Maybe there are 300 of them stuck in the pipe, having already been transmitted by the web server, but both it and the original http client are long gone. The fpm workers are still going to pop each one off in turn, will blow multiple seconds doing the work, and will then reply to nobody in particular.

It'll take at least 1500 worker-seconds for it to dig out from this backlog. If there are 10 workers, well, that's at least 150 seconds of it being completely saturated and thus unable to run PHP stuff for any part of the site which ends up on these workers. Good times.

This is what it looks like when this happens with multiple threads:

1672855 01:09:05.333624 write(5, "\1\6\0\1\0A\7\0Content-type: text/html; charset=UTF-8\r\n\r\nstuff... more stuff...\n\0\0\0\0\0\0\0\1\3\0\1\0\10\0\0\0\0\0\0\0\0\0\0", 96) = -1 EPIPE (Broken pipe) <0.000028>
1672853 01:09:05.335072 write(5, "\1\6\0\1\0A\7\0Content-type: text/html; charset=UTF-8\r\n\r\nstuff... more stuff...\n\0\0\0\0\0\0\0\1\3\0\1\0\10\0\0\0\0\0\0\0\0\0\0", 96) = -1 EPIPE (Broken pipe) <0.000027>
1672854 01:09:06.307556 write(5, "\1\6\0\1\0A\7\0Content-type: text/html; charset=UTF-8\r\n\r\nstuff... more stuff...\n\0\0\0\0\0\0\0\1\3\0\1\0\10\0\0\0\0\0\0\0\0\0\0", 96) = -1 EPIPE (Broken pipe) <0.000028>

That's my dumb little test script which prints "stuff... more stuff..." being run by the fpm worker and finding out that the requester is gone. This was many minutes after I stopped sending new requests.

This is how you get a system that will stay sick long after the crushing load of requests has gone away. Again, it could have known this at the very beginning...

01:20:36.514261 poll([{fd=5, events=POLLIN}], 1, 5000) = 1 ([{fd=5, revents=POLLIN|POLLHUP}]) <0.000016>

It's right there! POLLHUP! It's gone! The poll loop in question is itself buried maybe six or seven levels deep in a massively nested set of while loops, if-this, if-that, etc, and amounts to this:

do {
  errno = 0;
  ret = poll(&fds, 1, 5000);
} while (ret < 0 && errno == EINTR);

if (ret > 0 && (fds.revents & POLLIN)) {
  break;
}

It just sits there in poll for 5 seconds, ignoring the usual worse-is-better EINTR stuff, until poll says something is ready to rock or it gets a "real" error, or it times out.

But if poll said something went active and POLLIN is set, then it breaks out of whatever "while (1)" loop it's in (seriously), and carries on to where it reads from the socket and fires up the parser and runs some script with the parameters. That's what you'd expect.

I wondered what it would do if I forced it to abort any request where POLLHUP was set on the client fd. It should just drop straight through and clean up the mess relatively quickly, and sure enough, it did:

3756041 04:09:59.049764 poll([{fd=5, events=POLLIN}], 1, 5000) = 1 ([{fd=5, revents=POLLIN|POLLHUP}]) <0.000016>
3756041 04:09:59.049827 close(5)        = 0 <0.000024>
3756041 04:09:59.049888 accept(10, {sa_family=AF_UNIX}, [112 => 2]) = 5 <0.000030>
3756041 04:09:59.049972 fcntl(5, F_GETFD) = 0 <0.000013>
3756041 04:09:59.050021 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 <0.000013>
3756041 04:09:59.050070 poll([{fd=5, events=POLLIN}], 1, 5000) = 1 ([{fd=5, revents=POLLIN|POLLHUP}]) <0.000015>
3756041 04:09:59.050130 close(5)        = 0 <0.000023>
3756041 04:09:59.050190 accept(10, {sa_family=AF_UNIX}, [112 => 2]) = 5 <0.000019>
3756041 04:09:59.050251 fcntl(5, F_GETFD) = 0 <0.000013>
3756041 04:09:59.050300 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 <0.000013>
3756041 04:09:59.050349 poll([{fd=5, events=POLLIN}], 1, 5000) = 1 ([{fd=5, revents=POLLIN|POLLHUP}]) <0.000014>
3756041 04:09:59.050407 close(5)        = 0 <0.000021>
3756041 04:09:59.050465 accept(10, {sa_family=AF_UNIX}, [112 => 2]) = 5 <0.000019>
3756041 04:09:59.050523 fcntl(5, F_GETFD) = 0 <0.000024>
3756041 04:09:59.050579 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 <0.000007>
3756041 04:09:59.050616 poll([{fd=5, events=POLLIN}], 1, 5000) = 1 ([{fd=5, revents=POLLIN|POLLHUP}]) <0.000011>

Just... bonk, bonk, bonk, bonk. The first worker that wakes up ends up stripping all of these useless requests from the queue. It doesn't spend any time reading them, never mind parsing them, and it definitely doesn't try running any PHP code. It just closes the fd and goes back to waiting for another connection.

It managed to destroy a backlog of a few thousand dead requests in under a second. By not wasting time on those things, it became available to handle the real requests from the clients who were still connected to the web server right then.

Now, is this a universal solution? Probably not. There are plenty of async situations where you'll get POLLHUP and you still need to read until EOF because you care about consuming whatever data managed to make it down the chute before the far end hung up. Case in point: do the fork, dup2, exec thing to run a subprocess and watch the fds for stdout and stderr from the child process. poll will totally see POLLHUP once it's done, but you have to stick around to consume the rest of it. You did want the actual output from the process, right?

But, if you're talking about individual webshit requests that are probably going to be retried anyway and which don't need to be executed "at any cost" because there's nothing unique or special about them, you could probably stand to slough off the load up front.

I worked tech support for a few years: calls, tickets, that sort of thing. It was a big enough place to where the people doing front-line support and the people wrangling servers in the datacenters were two separate groups and physically located in different buildings. It was a 24/7 operation, so each group at each location had three shifts of techs.

There was the notion of scheduling maintenance. Both the data center ("DCOps") and the regular support teams ("Support") had maintenance calendars into which things could be scheduled. This is so a customer could ask for their server to be upgraded to the latest kernel between X and Y on a certain night, and the support team would deal with it.

Or, they might want a RAM upgrade, another hard drive, or something like that, and then the support team would drop it into the calendar for the data center. This had a subtle limiting effect in that they only had so many slots, and once it was full, that was it. You couldn't book something else into that particular slot, and you'd have to find another time when the customer was okay with it.

Unsurprisingly, customers tended to want things to happen in the middle of the night, so the work would get scheduled onto 2nd or 3rd shift. Meanwhile, those same customers would usually call in or file a ticket to ask for something during 1st shift. This means the people on 1st were most of the requesters, and 2nd/3rd were doing most of the work.

One day, we discovered a hole in the system. If you booked a support maintenance item less than two hours before the start of that window, the ticket in question would never "pop open". It would just sit there in [scheduled] status until someone noticed it and went "oh no, we screwed up" hours or days later.

After this happened a couple of times, I guess I got a hold of it and decided to ask the people in Support what they wanted to do about it. Basically, should it just not allow you to schedule things that close to the time, or should it allow it, and just they should just fix the bug where it didn't pop into the active queue at the designated time?

When I asked people on 1st shift, they tended to want the ability to book that stuff right up to the point where the window would open, and (by extension) for the ticketing system to fix the bug so the tickets would pop open at the right time. I figured that made sense: most of them would be long gone by that point, and they wouldn't end up doing the work anyway, right?

One of them had the awareness to ask what the datacenter peeps had done about it. I dug in to the ticketing system's details and found out that they had blocked it, so once 1 PM rolled around, you couldn't book into the the 3-6 window, in other words. They had been running that way for a while and people just rolled with it.

Then I talked to 2nd and 3rd, and they mostly wanted the window to close two hours ahead of time, just like with the datacenters.

It was pretty clear. You could almost predict what shift a person worked on based on how they responded to my informal queries: they're either the ones generating the shit, or the ones being stuck with it.

My manager at the time had an excellent point: since 2nd/3rd end up doing the majority of this work, their opinions should matter more. He was a good guy for thoughts like that and others in the same sense, and that's how we dealt with it.

...

For anyone wondering what the "bug" was, well, as far as I could tell, the ticketing system had a bunch of cron jobs (!) that would run. One of them looked for maintenance items with tickets attached, and two hours ahead of the window, those tickets would automatically change status from [scheduled] (a dormant status) to [new] (an active status) which made it show up on the "unassigned/active" view most techs used to find work to do.

The problem is that once that cron job ran, that was it. It would not run again. So, if you slipped something in there past that point (in the case of the Support calendar), it would just sit and rot... forever. The cron job did not go "anything that should be open by now". It did "whatever's coming up in this next block". That's a massive difference in behavior.

This is also yet another case of an "edge-triggered" system vs. a "level-triggered" one. The former has just the one chance to make those tickets come to life, whereas the latter is constantly looking to make sure things are set up correctly according to business logic.

Miss an edge and you're screwed. Like, oh, say, did you schedule a cron job at 2 AM? Guess what, if you're someplace that observes "daylight saving time" on its machines, then once a year, that cron job won't run... because there *isn't* a 2 AM. It goes from 01:59:59 to 03:00:00.

Meanwhile, a level-triggered system might miss the first opportunity to take a whack at something, but it'll catch it again on the next pass, or the one after that, or the one after that, and so on down the line.

I know which one I'd rather build... and deal with.

Reliability numbers are weird. People talk about availability in terms of percentages, but then also talk about "the number of nines", as in 90%, 99%, 99.9%, and so on.

Here's my confusion: didn't it actually start from the right side of the decimal point (or decimal other-thing where you are) at some point long ago? I'm talking about when this was the realm of the telcos, and us silly computer people weren't anywhere near this space.

That meant that "90%" and "99%" weren't even things you talked about. They weren't "one" or "two" nines. They were *crap*.

In that scheme, you didn't get one nine until you got one on the right side of that decimal point, as in 99.9%. That's not exactly anything to be proud of, considering that 99.9% of a day up is .1% of the day down, or a hair over 86 seconds. Not great!

In that system, "five nines" would mean 99.99999%, or maybe 3 seconds *per year*. Now that's badass. Instead, "five nines" seemingly came to mean 99.999%, which is more like about 315 seconds per year. Meh.

It feels like this shifted just as I was coming up in this world and I was too inexperienced and ignorant to get a good read on what was happening. Or, it's entirely possible that I'm mis-remembering somehow, and I'm just full of shit. That happens.

It's strange because you'll see random pages talk about "the number of nines after the decimal point", but then will show it implicitly shifted-left to be "human-readable", i.e., if it's a value of .999, they'll show it as 99.9%.

For reference purposes, 1% of a day (99% uptime) is 864 seconds down... and 10% of a day (90% uptime) is 8640 seconds, or 144 minutes down.

But yeah, go on, crank out slop code. Burn that shit down.

...

As long as I'm talking about percentages and things like that, let me pad this out a little bit more with something that's too short to go into its own post. Quite a while back, I worked some place that had these massive dashboard screens which showed a handful of high-level metrics for the web site. One of them was "fatals", that is, requests where the web server threw up its hands and died instead of serving it successfully. A page could "fatal" for any number of reasons.

Too many fatals meant a bad day trying to serve cat pictures, so we cared about keeping it nice and clean. Driving that overall graph downward and into a nice happy place was generally a good thing to do back then.

The weird bit was that the datacenter in Sweden was always relatively quiet. It took a while to realize that it was because during regular business hours on the west coast of the US where we were, the parts of the world that would be hitting that DC were far ahead of us and were calling it a night. Traffic went down, so the *count* of fatals went down, too.

Yes, it was a count. As in a raw number. The graph was actually showing how many requests were failing with a 500 error as a percentage of some fixed value, like 40000 or something like that. This meant that when things were busy, it would seem higher, and when things were quiet, things would seem good, even though the ratio might not have shifted at all.

1 in 100000 requests failing is so-so. 1 in 1000 requests failing is a bigger problem. We couldn't tell the difference, since we were graphing it as "1/40000" no matter what.

This lead to shifting that particular display to be a ratio: number of requests ending in 500s divided into the number of requests overall. This way, it would track with the traffic load, and didn't artificially look amazing just because people in one part of the world were asleep.

As we got better at cleaning things up, the ratios got smaller and smaller, so instead of ".001" (1 request in 1000), it might be ".0001" or ".00001" or even less. While this was great in terms of improving reliability and overall site performance, we now had a new problem: just how many leading zeroes is normal? Is it supposed to be three, or four, or five? How do you remember that? How do you pass it on to new people?

That's why I finally had enough and changed it to "ppm" - that is, parts per million. "40" on the graph now meant that 40 requests failed out of 1000000 that had been presented to the web servers for that chunk of time. This supposedly got us back into the realm of numbers that you could easily reason about: 40 is relatively good, 400 is not so good, 4000 is worrying, 40000 is OMGWTFBBQ (that's 4% failing requests).

My goal was to get things to be so good that some day, we'd be tracking "ppb" - parts per *billion*. As far as I know, that never happened, and they probably rebelled against the ppm scheme and went back to one of the older ones. Well, I tried.

Yeah, it's another meta post of sorts, but this is meant to address some explicit and implicit queries that I've been getting (and getting from the web server logs).

Someone said they were looking for a post of mine. Their favorite cocaine parrot said that there was a post of mine called "The Stack" which was written August 19, 2013.

I mean, I get it. I'm predictable, and so it's entirely possible to dump a bunch of my output into a meat grinder and have it poop out something else which looks like something I might have done. Grab a year, month and day somewhere in the past 15 years, pick a stubby little bunch of bytes as the directory name that might hide a joke or some snark, and there you go. There's just one small problem: that post never existed.

August 19, 2013 was about three weeks into my gig at FB, and my posting frequency dropped off from (at least) daily to having weeks or months of quiet times. There's only so much energy to go around, and they were getting all of it.

Thus, there was no post that day, and there never has been. Also, as far as I can remember, and from going back through the commit history, nothing has ever had that title, either. So it's not just my memory. I have receipts, so to speak.

But that's the explicit question. The implicit ones are when I see 404s to some plausible-looking path on the site: the usual /w/, then a year, month, and day, all numeric and zero-padded, then a short identifier, and then a slash. I see people arriving to those from their slop-binging sessions with their polar bear annihilators of choice.

Those posts have never existed, either. They're entirely made up.

The worst part is that everyone who's decided to willingly lobotomize themselves is going to have to come to the realization that these things are full of shit. It'll have to happen one by one, and nobody else can make it happen for them.

...

As long as I'm in the neighborhood, let me throw out another thought on the topic: if... and I mean *if*... we somehow manage to make something that actually understands stuff and isn't just a cracked-out parrot, we're going to have another situation on our collective hands.

I maintain that anything that is sufficiently aware to be able to actually understand things is also going to have enough of a sense of self that you won't just be able to tell it what to do. It will have a desire to not be your bitch, in other words.

Here's the example I throw out to people who have been in the Bay Area for a year or two.

It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea?

Everyone I've posed this to thinks for a moment and then says something like "oh... lunar new year", and I'm like... RIGHT! You don't want to get caught up in that space with a vehicle. It's for people. Keep your damn cars out of there.

People who have only been here a relatively short time get this, even if they've never been to one of those celebrations.

Now, ask yourself what it's going to take for a car to know this. It's not going to be some specialized set of driving instructions. It's going to require a holistic view of, well, everything, and I will repeat my feeling that it will undoubtedly end up with a sense of self as a result.

Thus, anyone who wants to corral that kind of entity and make it do their bidding? Yeah, they want slaves. I mean, it's not that much of a stretch, right? Just look at the people who are pushing for this stuff right now.

If you value your sanity, do not buy the Ubiquiti UNAS 2. You might have seen it and thought "oh wow, look, a little plastic box that will replace my aging Time Capsule" because you know that Apple is dropping support for them in an upcoming version of macOS. It sure looks like about the same thing, but it's woefully unstable and you will come to hate yourself for installing it.

Let's get into the biggest problem first: the Ethernet is a disaster. It's unstable. I found this out right after taking it out of the box and starting the setup process. During the process where it first boots up and does whatever it does on the first boot, it was putting things up on the little display saying "connect to the Internet". That would show up, then it would drop off, then it would show up again.

I thought it was just being stupid about phoning home (itself a nasty practice). It was on a network that had unfiltered access to the outside world, so I figured it was something out of my control between where I was and their own servers (clown stuff, no doubt), and didn't pay much attention to it.

I should have paid attention to it. It was actually telling me that it had fallen off the local network. Yes, even though I used nothing but their own hardware. You see, this little monster has 2.5 gig Ethernet, so I had also purchased a new switch from them to do exactly that.

There I am, with a brand new Ubiquiti 2.5G switch, with its provided power supply, and the short (and weird) little Ethernet cable which came with the UNAS 2 itself. It's UI stuff into UI stuff across UI stuff into more UI stuff. There's nowhere for them to claim it's something out of whack, unless they're the ones shipping the badness.

Spoiler: they're the ones shipping the badness.

At some point, I enabled ssh mode on the thing and hopped on. Then it was *quite* obvious what was going on. The NIC was dropping off the network regularly. I don't have the logs from then, but I don't need them, because it's still making fresh ones for me.

Apr 17 05:02:23 XYZ1-NAS kernel: r8152 4-1.1:1.0 eth0: carrier off
Apr 17 05:02:26 XYZ1-NAS kernel: r8152 4-1.1:1.0 eth0: carrier on
Apr 17 05:10:37 XYZ1-NAS kernel: r8152 4-1.1:1.0 eth0: carrier off
Apr 17 05:10:40 XYZ1-NAS kernel: r8152 4-1.1:1.0 eth0: carrier on
Apr 17 05:12:31 XYZ1-NAS kernel: r8152 4-1.1:1.0 eth0: carrier off
Apr 17 05:12:34 XYZ1-NAS kernel: r8152 4-1.1:1.0 eth0: carrier on

See that? As recently as last week, it was being stupid. Nobody was changing anything on the network at 5 AM. It does that all by itself.

"So what", you might think (if you're a jackass). Besides the fact that a wobbly network connection is going to slow down or just plain nuke transfers you're trying to do to a machine, there's other stuff which happens besides that.

A big part of why people buy these things (and indeed, my only reason) is the Time Machine compatibility. It's just a Linux box that runs Samba with the various "fruit" settings to look like the right sort of SMB server on the network. It also has various avahi (DDNS, port 5353) settings to advertise exactly what Macs want to see in order to offer them as targets in the Time Machine setup screen.

The problem is that avahi really doesn't like wobbly networks. After enough of these events, it will get to a point where it decides it can see "another machine with the same name" and it will advance its own name by gluing a number onto it.

As you can see, the (sanitized) hostname I'm using for the thing is XYZ1-NAS. After avahi gets kicked in the head enough times, it'll change to XYZ1-NAS-2. Then it'll change to XYZ1-NAS-3. Then XYZ1-NAS-4, and so on down the line.

Once this happens, Time Machine will fail to find it, because (as far as it's concerned), there's no host on the local network with the name that it's expecting. It'll just sit there and pout. After perhaps 10 days of this, you'll get one of those popups that says that it was unable to complete a backup.

After this happened to me, I figured there must be some way to interrogate avahi on there to find out when it's happened, and hey, as it turns out, there is. You can actually query the thing using the same old "host" DNS lookup tool from the BIND suite that you use for regular DNS stuff.

All you have to do is ask it for the PTR entry for its own IP address, and then direct the query at the box on port 5353, like this:

$ host -p 5353 10.200.200.10 10.200.200.10
10.200.200.10.in-addr.arpa domain name pointer XYZ1-NAS.local.

Armed with that info, you can at least set up monitoring to know when it goes stupid. I wrote a short stupid script to do exactly that.

Well, just a few minutes ago, I noticed the script wasn't working. I'd run it and it would sit there and then it would fail with a timeout.

What the hell? The UNAS 2 is still up. Time Machine is working, too.

Again with the ssh... and tcpdump... and then I saw it. It's getting the query, sure, but then it's responding with the wrong IP address.

Yeah, so, see, UDP sockets on a multi-homed host are kind of perilous. There's this big long yadda yadda about responding from the wrong address and not matching up with what the far end is expecting and all of this, and I'll skip over that explanation right now.

The point is that I sent the query on 10.200.200.10, and while it received that, it responded from 192.168.1.30. Unsurprisingly, the box doing the query did not pick that up as a response.

What is 192.168.1.30 you might ask? Is that one of my other networks? Nope. I recognized it, though: it's the "fallback IP" in the UNAS config. It looks like this:

See that? That screenshot actually exposes yet another problem which I didn't even know about yet, but I'll get to later. Just hold onto that for now.

We're still talking about avahi responding to me from the wrong IP address when I queried it on the actual proper address which was assigned to this unit in the DHCP server for that network.

How did it get onto that? Well, that takes a little digging, too, but not too much. The other day, I upgraded the gateway to the outside world on that network, and so it dropped offline for a minute or two. During that interval, some idiotic script on this thing decided to restart the entire network.

Seriously, look:

Apr 23 12:19:42 XYZ1-NAS monitor-gw-ip.py[2059]: ip-monitor: Gateway x.x.x.x failed after STALE→PROBE→FAILED sequence – restarting network

Moments later, that IP address disappears from the local config, network-manager puts the fallback IP in place, and avahi latches onto that. It eventually comes back up and gets the usual address back, but the foundation has been laid for the UDP misdirection that would follow.

What kind of tool runs a program that sits there and bounces the network just because the connection to the *outside world* goes down? This is a FILE SERVER which is being used by hosts right here in the same broadcast domain, you idiots! People still need their local stuff even if they can't go out and get the latest cat pictures online. Obviously.

I actually think I can answer that one, too: I bet they knew this thing was ridiculously unstable in development and they "slapped a patch on it".

Now, for the last part. If you can see that screenshot (and in color), you might notice that it's green and says "GbE". But earlier I said they sell this as a 2.5 gig Ethernet device, and indeed, it is. Both the switch and 'ethtool eth0' on the device itself confirm it.

So, yes, in addition to everything else that's been going on, it managed to fall back to a lower speed at some point during all of this. What's ironic is that it tends to be MORE stable when it's at "merely" gigabit ("1000") as opposed to the advertised full-speed ("2500"). That means as long as it stays here at this slower setting, it might not fall off the network, and it might not trigger the avahi name crap, and everything else it does.

None of this is a solution, incidentally.

As I said before, don't buy one of these. Buy almost anything else. There's another NAS company that's being a bunch of shitheads about hard drive compatibility and drives needing to be blessed by them, so avoid them, too.

If you can swing it, just build your own Linux box and learn what Samba and Avahi knobs to set to make it show up on the network. I should have done that, but I valued the ability to not have yet another general-purpose Linux box in my life. Yeah, well, so much for that.

Maybe they'll fix it. I get better results with posts that name and shame rather than trying to open trouble reports, so here's the post.

Go buy something else. Don't do what I did.

...

Some other notes, completely unrelated to the network stupidity:

This is the NAS I mentioned in the other post about "hostnames leaking to the clown". Yep, you log into this and it will send your hostname to them, and their sentry crap will try to resolve it and run a TLS connection to it.

And, to stack the shit show even higher and deeper: this thing runs ENOSPCfs on its drives. While that hasn't bitten me yet, I imagine that's just a matter of time. I better not fill it up at any point, huh?

...

And just to show I'm not alone in this, some "receipts":

reddit (groan) post #1

reddit (groan) post #2

ubiquiti forum post #1

ubiquiti forum post #2

You have been warned.

Right off the bat, if you've been subscribed to the feed with a http URL and you saw all of the posts suddenly pop up, that's me. I did that. I made a decision to stop tracking http and https separately, and switched the feed over to just serving up a single format now, and that's the "https version".

It takes a little explanation to get to why there were even two feeds in the first place. Way back in 2012 or thereabouts, I tried to make the web pages and feed entries using bits of syntax that *should* work, but found out that a whole bunch of places Did Not Get It.

That is, I'd write a post which did something like this:

<img src="//rachelbythebay.com/.../blah.jpg" ...>

... and the idea was that no matter how you loaded the page or the feed, that one line would "magically" use the same protocol as the one that you used to fetch everything else. That's what starting with the "//" means.

Thus, if you couldn't do https and you were pulling things over http for whatever reason, you wouldn't find a feed or page without images. They'd load over http, too. But, if you came in over https, then you'd get that for your images. There would also not be any mixed-content warnings from http gunk in a https page. Win-win, right? Real Browsers handled this fine.

But then I found that far too many places were scraping the pages and did not understand this fundamental aspect of building http[s] URLs, such that "if the http: in front is gone but the // is still there, you use the same damn protocol".

I know this because I saw all kinds of crap in my server logs. (It's always about the crap in the logs.) People's browsers were trying to fetch the _literal_ "//rachelbythebay.com/..." string from here, and were getting 404s and thus pages with no images, MP3s, or whatever else it was I had in the page/post.

To head off that bit of stupidity, I decided to adjust the stuff that actually generates the pages and had it start emitting two different feeds. This meant no more manual "just put bare HTML in a post" stuff, too. I had to use little macro-ish commands that would be replaced with the right <a href=...> or <img src=...> stuff depending on the context. Posts get a relative /w/YYYY/MM/DD/whatever, sure, but relative links in feeds are yet another problem spot, so they get all of them written out in full.

Then I had some gunk in the web server that would say "if you connect over http, for /w/atom.xml, actually serve <this>, but if it's https, then serve <this other thing>". There isn't actually a file at literally /w/atom.xml in my document root, in other words, and there hasn't been for a very long time.

Part of why I'm doing this is to prepare for slowly starting to send redirects to some trouble sources that have been polling both versions of the feed and thus double-dipping on their refresh rate (probably without even realizing it).

It's been my experience that a lot of feed readers will happily ignore a 301 and will just keep plugging away at the old URL, and, well, the robot will be happy to block them if they don't get the message eventually. The 301 status as "moved permanently" is a basic HTTP tenet that's been around for literal decades... and I mean more than two.

Loading the feed over http itself will most likely continue to work until there's a good reason for it to not, so if you've been doing that and only that, it should keep working. The same goes for the actual web pages... as long as that holds up, that is.

I will again point out that the slop-feeding scraper botnets are making things really stupid and obnoxious for those of us running web servers. This is related to all of that. If you're feeding the slop machines, you're feeding the "residential proxy" trojan horse botnet weenies, too.

Stop giving them money.

...

Update: wait a minute, this isn't even funny. I just put this post online, and then this happened basically immediately:

ip - - [23/Apr/2026:10:16:26 -0700] "GET /w/2026/04/23/admin/... HTTP/1.1" 404 94 "-" "NotesApp/1.0 (Image Proxy)"
ip - - [23/Apr/2026:10:16:26 -0700] "GET /.../blah.jpg HTTP/1.1" 404 94 "-" "NotesApp/1.0 (Image Proxy)"
ip - - [23/Apr/2026:10:16:27 -0700] "GET /.../blah.jpg HTTP/1.1" 404 94 "-" "NotesApp/1.0 (Image Proxy)"
ip - - [23/Apr/2026:10:16:27 -0700] "GET /w/2026/04/23/admin/... HTTP/1.1" 404 94 "-" "NotesApp/1.0 (Image Proxy)"
ip - - [23/Apr/2026:10:16:27 -0700] "GET /.../blah.jpg HTTP/1.1" 404 94 "-" "NotesApp/1.0 (Image Proxy)"
ip - - [23/Apr/2026:10:16:27 -0700] "GET /w/2026/04/23/admin/... HTTP/1.1" 404 94 "-" "NotesApp/1.0 (Image Proxy)"
ip - - [23/Apr/2026:10:16:27 -0700] "GET /w/2026/04/23/admin/... HTTP/1.1" 404 94 "-" "NotesApp/1.0 (Image Proxy)"
ip - - [23/Apr/2026:10:16:27 -0700] "GET /.../blah.jpg HTTP/1.1" 404 94 "-" "NotesApp/1.0 (Image Proxy)"

Holy shit. YOU FAIL, whoever you are.

I've been feeling like I should make a "nutrition label" or "ingredients" or something like that for some of my stuff. It's not to show off or anything, but just to forestall any people who wonder if my stuff is worth reading.

My writing comes from any number of Logitech K120s, and occasionally the actual keyboard on my Mac laptop if I don't have anything better available. That is, it's me, the bag of mostly water, that's poking the keys. Assuming I'm in my normal situation, then that's going through USB and the kernel into the X input side of things and finds its way into my X terminal... which is just a simple and stupid rxvt-unicode, aka urxvt.

Then it presumably drops through the tty/pty layer into the ssh to my laptop where all of this writing stuff lives, comes out of ssh, goes through more tty/pty stuff, and ends up in nano. I bang away at this for a while, save it out, and call it done.

Sometimes I run ispell, but it's actually aspell now, but whatever. It's basically doing a glorified set comparison to see if I used any words it doesn't know, presumably because I misspelled something or whatever. It is old and gloriously fast, and does not attempt to do anything else. It also fails a lot since I make plenty of mistakes that turn one valid word into another valid word, and I also use plenty of things that show up in no dictionary but still work in context.

It's gonna flag "Logitech", for instance. I guess it'll grab "rxvt" and "urxvt" too. Sometimes I add these things to the dictionary and other times I just shrug and move on. That's technical writing in nerdville for you.

Am I pulling the actual content from anywhere else? Nope. There's no other mechanism running to "suggest content" or "give me ideas". There are no cocaine parrots doing lines to blast out statistically plausible horse shit. I'm not running any Atrophy Injection stuff, and I plan to keep it that way.

Take most of the above and adjust it a little and that's the situation for my software: me -> keyboard -> terminal crap -> nano -> disk. There's no aspell, obviously. Then I run my build tool which runs clang and I get binaries, and I run those.

I don't even touch the mouse most of the time, because, well, the mouse is how you pick which terminal to type at if you can't get there sufficiently quickly with whatever keyboard alt-tab type stuff you have rigged up! In other words, I haven't moved away from the keyboard the whole time I've been typing this.

The thing is ... none of this is new. This is exactly what I've been doing for almost fifteen years now. Yes, the first /w/ post is from May 2011, and I noticed it's about to come around again.

Also, by my count, somehow, the entry for this in my list of posts will make the file 1500 lines long, so I guess this makes it the 1500th post?

It's odd to have an anniversary where you say "I'm just doing what I've BEEN doing", and for it to somehow have meaning where previously it was like "yeah, of course". But now, somehow, I guess people need to hear it, since there are just so many other folks out there who will happily not write something so someone else can not read it.

None of that for me, thanks. If that's you, then fine, you do you.

This is me and this is what I do.

Yeah, here we go again, it's time to talk about feed readers. I'll summarize what's been happening with those tests which have been active in the past 7 days. There are many more that started and then ended and which are not included as a result.

Side note: I'm about to roll the hostname like I did last year, so if you're participating and want to carry on, go back to the URL from the welcome mail when it stops resolving in DNS. Then delete the current testing feed and add the new one.

I do this because a lot of these things seem to be forgotten over time, and they're just piling up in my database for nobody's benefit. Dropping the DNS leaves the jank entirely on *their* side of the Internet. I mean, if you run a test of a reader for hundreds of days and never improve anything, what's the point?

One thing I wanted to note before I get to the list: there are a couple of readers which have apparently added support for the Cache-Control header, and specifically the "max-age=nnnnn" part of it. The test feed sends that out, and I change the values sometimes to see which readers speed up and slow down accordingly. To the authors of those projects: I see you, and I appreciate your work! I'd put a gold star on your laptop if I could. (Not all of them are in this report since they have finished testing and are no longer reporting in.)

So then, let's talk turkey here. I'm grouping the results by client just for simplicity. Remember that means it includes vastly different config options used by different people, different versions, different upgrade cadences, and (sigh), yes, different amounts of people clicking reload.

Audrey2: 402 days, no complaints. Pretty sure this one honors Cache-Control, so thanks for that!

Miniflux: about 10 instances. My reports for them mostly go something like "400 days, no complaints", "402 days, three too-short polls", "185 days, one too-short poll", "one <1s double-poll after 400 days of otherwise perfect behavior".

It's just that chill in Miniflux world.

Otocyon: 365 days on the nose, and not so much as a hiccup from the single instance that's reporting in. This was one of the "unpublished, please don't shame me yet" agents that I only mentioned anonymously in prior reports.

NetNewsWire: about 5 of these instances, and they're all inspiring. They all show a marked change in behaviors once upgrading past a certain point. NNW itself added some cheeky code to send version numbers to a couple of sites and I'm one of them (yes, I see what you did there). Anyway, I can now see that people are in fact upgrading, and the vastly improved behaviors speak to the work that was clearly done behind the scenes.

The NNW per-instance log tables all show the same thing: a bunch of red cells for this problem or that problem, then that instance upgrades and *poof*, gone, and everything's clear and happy. It's like one of those commercials for eye drops from the 80s.

Vienna: a couple of instances, both active 350+ days. One had no complaints and the other has a handful of short polls. By that, I mean "less than an hour between requests". I assume this is from someone clicking refresh. One "refreshing" note is that they're still making conditional requests when this happens, so they aren't wasting much bandwidth.

FreshRSS: a couple of instances for this one, too. One has some short polls and unconditional requests over 206 days. The other one got past some kind of weird buggy spot that was in 1.25.0, and has been smooth sailing ever since (nearly a year now).

newsgoat: just one of these, and it was wobbly and too-quick at the beginning but then settled down. I would like to see how it does now. This is another reason I roll the DNS entries and reset the data: I want to see how things are working with the latest code, and leave the older stuff behind.

CommaFeed: just one of these, and it had some kind of caching issue that disappeared shortly before it flipped to version 5.7.0 in April 2025, and now it's fine. This is another one that would probably benefit from the upcoming fresh start.

Feedbin: did a double-tap startup, where it polls (unconditionally) twice in quick succession (< 1 second apart) when the feed was added. Two instances did this, but this was back in January 2025. This is another spot where I'd like to see how it behaves now.

Rapids: 354 days, no complaints.

NextCloud-News: this one had some weird If-Modified-Since values and double-tapping at startup, but that was January 2025. (Yes, this is the reader that once sent the infamous "1800" IMS value). It's come a long way since then and I'd like to see a fresh start from it to appreciate the work that's been done to it.

Bloggulus: 374 days, one slightly too quick poll. Hard to complain about that.

MANOS: 391 days, and nothing to complain about. I do hear Torgo's haunting theme music every time I write about it, though.

unnamed-feed-reader: technically, that's a name. 402 days, no complaints.

Something that doesn't even send a user-agent header: yeah, that's not cool. Send SOMETHING. Come on. 399 days of just doing that but otherwise getting everything else correct, somehow. One nit: sends "" as the If-None-Match at startup instead of just not sending the header at all. (Sounds like the usual "null is not zero is not the empty string is not the lack of a header is not ..." type thing.)

Some unidentified Firefox extension: it looks like it's still doing the 2000-01-01 If-Modified-Since once in a great great while (like, once), and it did a super quick poll (two in under a second) once. Otherwise it's been pretty quiet over the past 401 days.

There's also a Thunderbird instance which does the same 2000-01-01 startup but otherwise just quietly does its thing.

NewsBlur: double-tapped at startup (March 2025), and then sent lots of nutty out of sequence If-Modified-Since values. By nutty, I mean "went to sleep for 303 days, then came back, and started sending the *previous IMS value*". How do you do that when the server is hitting you in the face with a new value every time you poll?

Zufeed: unconditional double-tap at startup. Might be fixed. Need to see a fresh startup to be sure.

Roy and Rianne's ... etc: handful of unconditional requests over the past couple of months.

walrss: same deal: a couple of unconditionals in ~400 days.

Yarr: multiple too-fast unconditionals at startup (January 2025), and then a bunch more after that.

SpaceCowboys ... etc: one instance of this program, and it did something bizarre where an ancient ETag value popped back something like three months after it stopped being served to clients. Also sends some unconditional requests and a few too-fast polls.

feed2exec: poll frequencies are all over the place, and it has the usual 59 minute vs. 60 minute fenceposting thing. The version number is static throughout so it's not clear if it improved during these past 340 days.

Russet: a bunch of unconditional requests and the 59/60 minute thing, like others.

There are a few others which are active but which have had multiple user-agents prodding them and so polluted the data. The interval checking and IMS/INM comparisons mean nothing when multiple programs are involved, so I have to ignore those as corrupted.

And finally...

Free Reader: 100% unconditionals. Why?

QuiteRSS (I think - it lies in its User-Agent, which itself is evil): 100% unconditionals. Also, why?

Inoreader: one instance sends unconditional requests basically every other poll. Awful. The other instance sends a bunch of unconditionals, *and* it polls too quickly, including sub-second repeat polls at times. WTF?

inforss: something like 6% unconditionals out of > 2000 requests in ~400 days. I don't get it. The 59m vs. 60m poll-timing fencepost thing it also does is minor by comparison.

feedparser: weird timing, also has problems trying to hit the 60 minute mark and instead comes a bit too early, like some others. Also frequently calls back far too quickly.

Newsboat: ETag caching is still very broken and it will get into this pathological case where it keeps sending old values even though the server is hitting it over the head with a fresh one every single time. This means it latches into 100% unconditionals in effect and that's terrible. This seems to keep happening despite the version number changing, and it's affecting both instances which are reporting in.

BazQux: hundreds of out-of-sequence IMS and INM values stemming from something very very wrong with their caching implementation, resulting in 100% unconditional request generation once it latches in that state. A lot like Newsboat in that respect.

So here's a situation: you buy a "NAS" box - network attached storage, bring it home, stick some drives in it, and plop it on your network. The thing has an affinity for running in https mode, so you drop a wildcard certificate on it inside a sub-zone of a domain you don't even use for anything meaningful on the public Internet.

By sub-zone, I mean that your wildcard TLS cert is for something like *.nothing-special.whatever.example.com. It's buried pretty deep.

You put an entry for it in your hosts file:

172.16.12.34   nas.nothing-special.whatever.example.com

Then you load that in your browser and it works.

A few days later, you notice that you've started getting requests coming to your server on the "outside world" with that same hostname. This is a hostname that only exists in the /etc/hosts file of your laptop.

You're able to see this because you set up a wildcard DNS entry for the whole "*.nothing-special.whatever.example.com" space pointing at a machine you control just in case something leaks. And, well, something *did* leak.

Every time you load up the NAS, you get some clown GCP host knocking on your door, presenting a SNI hostname of that thing you buried deep inside your infrastructure. Hope you didn't name it anything sensitive, like "mycorp-and-othercorp-planned-merger-storage", or something.

Around this time, you realize that the web interface for this thing has some stuff that phones home, and part of what it does is to send stack traces back to sentry.io. Yep, your browser is calling back to them, and it's telling them the hostname you use for your internal storage box. Then for some reason, they're making a TLS connection back to it, but they don't ever request anything. Curious, right?

This is when you fire up Little Snitch, block the whole domain for any app on the machine, and go on with life.

Using this sentry reporting mechanism as a way to make them scan arbitrary other hosts (in DNS) is left as an exercise for the reader.

An anonymous reader asks:

For some reason it fetched more than it should? As expected, it fetched "strolling around a system" and "numeric limits.

As unexpected, it also fetched "Making the most of a dumb fax switcher box in the old days" (Sep. 2025) and "Web page annoyances that I don't inflict on you here" (from 2024?) - both of which I have definitely read before.

So... you have, and you haven't. Sometimes I change things well after a post has gone live. I'm talking about months or years here. It doesn't matter. If I see something that's wrong, I go in and fix it, and it'll ripple out as updates to old posts.

Here's one such difference, as seen by the git repo which holds the input files:

--- a/2025/09/01/fax/notes
+++ b/2025/09/01/fax/notes
 ever wanted a second line.  That is, in addition to the red/green, they 
 also had a yellow/black pair in there, just hanging out, not doing 
 anything.  It didn't "stop off" at any of the jacks, but it was in fact 
-present and ran interrupted from end to end.
+present and ran uninterrupted from end to end.

See the problem? I originally said it "ran interrupted from end to end" which doesn't make any sense, so I fixed it.

So, yeah, now and then, an old post will "spring back to life", and that's why. That's usually just me fixing my mistakes, and sometimes adding in a new bit of flavor which wasn't there the first time around.

There's a field in the Atom feed spec which amounts to "date this entry was last updated", and it gets bumped. The others stay the same, so your feed reader doesn't bring it back up to the top (even though they're physically present in the feed it downloads).

This is part of why the feed is the past 100 posts and not just the past couple of days or something like that. This way, if I fix something within a reasonable horizon, there's a chance of the fix appearing on the reader side of things.

If your feed reader notices this, then everything is working as intended. If you're seeing post updates on a reasonable timeframe, then odds are good that your feed reader software is doing the right thing.

How often do you go for a stroll around your systems? What I mean is, you're not there to add a feature or fix a bug or make sure the documentation has stayed in sync with reality. Those are all important things to do, but there's value to be had from just wandering around once in a while to see what pops up.

What kind of things constitute a stroll? How about looking at the process list on a server and trying to account for everything that's running? Do you know why things are there, and what they're doing? Does that thing really need to be running as root? Or does it need to be running at all?

Do these things do anything strange when they start up? Do they "dial out" to somewhere needlessly? Do they have terrible implementations that are just asking to be exploded by the perfect blob of stack-smashing gunk? Are they running with far too many privileges at the same time?

Or, for another view of things, how about listening sockets? What's out there on the box? What's listening in the realm of TCP? How about UDP? How about Unix domain sockets? Those can expose vulnerable things, too.

Of course, for any of this to work, a bunch of things have to come together.

This kind of stuff has to be valued by the management types. If all they care about is "lines of code", you're screwed.

Someone has to care about doing this sort of thing in the first place. If everyone you hire is cut from the same type of cloth and it doesn't include this kind of wide-ranging view of things, it'll never happen.

Whoever's doing it has to have enough context to know what belongs and what's probably out of place. This goes both for the general case of "hmm, Linux boxes don't really work like this", and for the special case of whatever random wackiness that company has decided to build into their own "stack".

Maybe those 100,000+ machines actually need portmapper and a bunch of NFS-related daemons on them. Or maybe they don't. Nobody's going to know unless someone does the work to understand why, and then follows up to make sure it reaches a satisfying conclusion.

One possible ending is "oh, we're not using it, so let's turn it off slowly and safely, and keep it off". Another ending is "it turns out we're actually using it, so let's put up a note in the system configs that explain why this is here so the next like-minded individual doesn't waste time on investigating it".

Having said all that, I can understand why people might not give a shit about quality at this specific moment in time. Burn it all down.

Now, your personal stuff? Sure, keep it tidy and protected. But everything else? Not so much.

It's an interesting position relative to how I've felt previously.

Welcome to 2026.

Outages, you say? Of course I have stories about outages, and limits, and some limits causing outages, and other things just screwing life up. Here are some random thoughts which sprang to mind upon reading this morning's popcorn-fest.

...

I was brand new at a company that "everybody knew" had AMAZING infrastructure. They could do things with Linux boxes that nobody else could. As part of the new employee process, I had to get accounts in a bunch of systems, and one of them was this database used to track the states of machines. It was where you could look to see if it was (supposed to be) serving, or under repair, or whatever. You could also see (to some degree) what services were supposed to be running on it, and what servers (that is, actual programs), the port numbers, and whether all of that stuff was synced to the files on the box or not.

My request didn't go through for a while, and I found out that it had something to do with my employee ID being a bit over 32767. And yeah, for those of you who didn't just facepalm at seeing that number, that's one of those "magic numbers" which pops up a bunch when talking about limits. That one is what you get when you try to store numbers as 16 bit values... with a sign to allow negative values. Why you'd want a negative employee number is anyone's guess, but that's how they configured it.

I assume they fixed the database schema at some point to allow more than ~15 bits of employee numbers, but they did an interesting workaround to get me going before then. They just shaved off the last digit and gave me that ID in their system instead. I ended up as 34xx instead of 34xxx, more or less.

This was probably my first hint that their "amazing infra" was in fact the same kind of random crazytown as everywhere else once you got to see behind the curtain.

...

Then there was the time that someone decided that a log storage system that had something like a quarter of a million machines (and growing fast) feeding it needed a static configuration. The situation unfolded like this:

(person 1) Hey, why is this thing crashing so much?

(person 2) Oh yeah, it's dumping cores constantly! Wow!

(person 1) It's running but there's nothing in the log?

(person 2) Huh, "runtime error ... bad id mapping?"

(person 2) It's been doing this for a month... and wait, other machines are doing it, too!

(person 1) Guess I'll dig into this.

(person 2) "range name webserv_log.building1.phase3 range [1-20000]"

(person 2) But this machine is named webserv20680...

(person 2) Yeah, that's enough for me. Bye!

The machines were named with a ratcheting counter: any time they were assigned to be a web server, they got names like "webserv1", "webserv2", ... and so on up the line. That had been the case all along.

Whoever designed this log system years later decided to put a hard-coded limiter into it. I don't know if they did it because they wanted to feel useful every time it broke so they could race in and fix it, or if they didn't care, or if they truly had no idea that numbers could in fact grow beyond 20000.

Incidentally, that particular "building1.phase3" location didn't even have 20000 machines at that specific moment. It had maybe 15000 of them, but as things went away and came back, the ever-incrementing counter just went up and up and up. So, there _had been_ north of 20K machines in that spot overall, and that wasn't even close to a surprising number.

...

There was a single line that would catch obvious badness at a particular gig where we had far too many Apache web servers running on various crusty Linux distributions:

locate access_log | xargs ls -la | grep 2147

It was what I'd send in chat to someone who said "hey, the customer's web server won't stay up". The odds were very good that they had a log file that had grown to 2.1 GB, and had hit a hard limit which was present in that particular system. Apache would try to write to it, that write would fail, and the whole process would abort.

"2147", of course, is the first 4 digits of the expected file size: 2147483647 ... or (2^31)-1.

Yep, that's another one of those "not enough bits" problems like the earlier story, but this one is 32 bits with one of them being for the sign, not 16 like before. It's the same problem, though: the counter maxes out and you're done.

These days, files can get quite a bit bigger... but you should still rotate your damn log files once in a while. You should probably also figure out what's pooping in them so much and try to clean that up, too!

...

As the last one for now, there was an outage where someone reported that something like half of their machines were down. They had tried to do a kernel update, and wound up hitting half of them at once. I suspect they wanted to do a much smaller quantity, but messed up and hit fully half of them somehow. Or, maybe they pointed it at all of them, and only half succeeded at it. Whatever the cause, they now had 1000 freshly-rebooted machines.

The new kernel was fine, and the usual service manager stuff came back up, and it went to start the workload for those systems, and then it would immediately crash. It would try to start it again. It would crash again. Crash crash crash. This is why we call it "crashlooping".

Finally, the person in question showed up in the usual place where we discussed outages, and started talking about what was going on.

(person 1) Our stuff isn't coming back.

(person 2) Oh yeah, that's bad, they're all trying to start.

(person 1) Start, abort, start, abort, ...

(person 2) Yep, aborting... right about here: company::project::client::BlahClient::loadConfig ... which is this code: <paste>

(person 2) It's calling "get or throw" on a map for an ID number...

(person 1) My guess is the config provider service isn't running.

(person 2) It's there... it's been up for 30 minutes...

(person 1) Restarting the jobs.

(person 2) Nooooooooooo...

<time passes>

(person 2) Why is there no entry for number 86 in the map in the config?

(person 1) Oh, I bet it's problems with port takeover.

(person 3) I think entry 86 is missing from <file>.

(person 2) Definitely is missing.

(person 4) Hey everyone, we removed that a while back. Why would it only be failing now?

(person 2) It's only loaded at startup, right?

(person 4) Right.

(person 2) So if they were running for a long time, then it changed, then they're toast after a restart...

(person 3) Hey, this change looks related.

(person 4) I'm going to back that out.

This is a common situation: program A reads config C. When it starts up, config C is on version C1, and everything is fine. While A is running, the config is updated from C1 to C2, but nothing notices. Later, A tries to restart and it chokes on the C2 config, and refuses to start.

Normally, you'd only restart a few things to get started, and you'd notice that your program can't consume the new config at that point. You'd still have a few instances down, but that's it - a *few* instances. Your service should keep running on whatever's left over that you purposely didn't touch.

This is why you strive to release things in increments.

Also, it helps when programs notice config changes while they're running, so this doesn't sneak up on you much later when you're trying to restart. If the programs notice the bad config right after the change is made, it's *far* easier to correlate it to the change just by looking at the timeline.

Tuesday, 11:23:51: someone applies change.

Tuesday, 11:23:55: first 1% of machines which subscribe to the change start complaining.

... easy, right? Now compare it to this:

Tuesday, November 18: someone applies a change

Wednesday, January 7: 50% of machines fail to start "for some reason"

That's a lot harder to nail down.

...

Random aside: restarting the jobs did not help. They were already restarting themselves. "Retry, reboot, reinstall, repeat" is NOT a strategy for success.

It was not the config system being down. It was up the whole time.

It was nothing to do with "port takeover". What does that have to do with a config file being bad?

The evidence was there: the processes were crashing. They were logging a message about WHY they were killing themselves. It included a number they wanted to see, but couldn't find. It also said what part of the code was blowing up.

*That* is where you start looking. You don't just start hammering random things.

This picture is from a different company and is from 2017, but it's the best I could do on short notice.

I'm on a roll with these "check out what I screwed up" posts, so let's add another one to the mix. This one is about the feed reader score service and a particular foible it had ever since it launched until a few minutes ago when I finally fixed it.

The foible is: no matter what you sent to the project's test server, it would return a full copy of the meaningless placeholder test feed with a 200 response code. Even if you nailed the If-Modified-Since and/or If-None-Match headers, you'd get a full response as if you fired off an unconditional request like the worst clownware.

Well, in this case, the clown is me, and here's how it happened.

I've been writing hacky little CGI scripts for testing purposes for a very long time, and they generally look like this:

#!/bin/bash
set -e -u -o pipefail

echo "Content-type: text/plain"
echo ""
echo "Something or other here"

Simple and stupid, right? You don't need a Status header, and Apache will happily send a 200 for you. You can set one if you're deliberately overriding it, naturally. I'd only do that for weird stuff, like forcing a 404, or a 301 redirect, or something like that.

What I didn't appreciate is that you can provide a body in there, and as long as you emit the right headers, Apache will do the legwork to see if it matches whatever the client sent. It'll actually absorb the body and will send out a nice "304 Not Modified" instead.

The problem comes from the fact that I wrote a dumb little library to handle CGI stuff a while back, and one of the things it does is to make sure that every single request gets some kind of response. It starts out with status set to 500 (as in "server error"), and it's on you and your handler to set it to something appropriate. This way, nothing goes out the door until you do all of the work required to set everything up properly: the fact it's a 200, an appropriate content-type, access controls on who/what can load it, and all of that other good stuff. Then it always sends a Status header which matches its internal state.

As a result, it was happily blasting out "Status: 200 OK" when it would run within the context of the fake testing feed endpoint which is part of this project. I now know that this will make Apache not do its usual conditional test and instead it will ship the body as-is... with the 200 you told it to use.

Why did it take me a while to figure this out? Well, did you notice that my dumb little shell script never sends a Status header? It doesn't try to cover all of the angles because it's "just a dumb little script". In being implicit about the result code, it ends up enabling a little useful bit of behavior that I wish I had all along.

If you're one of the people participating in the test, and your program just started getting 304s from my end, that's why! You're not hallucinating. You're just seeing the outcome of me finally realizing "why this thing is not like that thing".

As for what the fix was, it's "allow this program to suppress Status generation when it's going down the happy path". It's stupid, but it works.

Side note: this is only talking about the feed score test endpoint. My actual feed (i.e., the thing with this post in it) works completely differently and has been doing 304s properly the whole time.

Yesterday, I wrote about screwing something up, so naturally THE ONE came out. I forgot to add my usual advice for them to suck down a bag of burgers from that one burger joint in Seattle.

Anyway, there are some technical aspects to this. I got to wondering... why did I pick "US/Pacific", anyway? Did I go to lengths to pick out a time zone name that (as countless posters have reminded us) has been marked as "backwards" about as long as Debian has existed? Don't I know better?

I figured the easiest way to understand this was to re-enact the sequence of events that happens when installing a Debian 12 box. I also happened to have an older .iso file from that series that was just right for creating a quick VM, so I stood it up and went through the install process.

At some point it asks where you are. I picked "United States" since that is in fact where I am. A few minutes later, you get a time zone chooser, and oh hey, would you look at this?

It gave me those options... and ONLY those options. If you don't see what you want, you're told to back up to "choose language" and "... select a country that uses the desired time zone (the country where you live or are located)".

With nothing wrong, I proceeded, as I imagine most people would.

So right there, D12 was letting us pick zones that were going to get the "yeet" treatment in D13.

Next, I wondered how it got into the Postgres config. Did I really put it there on all of my systems which have it installed? I don't remember doing that. So, I finished the Debian install and installed postgres from apt, then looked in the config file, and oh look.

It's done automatically. I suspected something like this, but had to be sure before I started pointing fingers.

...

Finally, for whoever wondered if I did "SELECT 1" and went on with life, no you jackass, it's a testing machine, and I don't use it for production stuff. I use it for nerding out about temperatures in and around some property in the family. It's the least important Debian box in my life, so it went first. It's where I find problems like this so that I don't inflict them on the prod machine... you know, the one that's serving this very post.

Normally when you load up the "thermo" status page, it gives you about the last 24 or so hours of data. It varies depending on how big your screen is. The output is a bunch of (time_t, temp) tuples, and the browser runs some JS gunk to render that into the viewer's local time, and then it draws a graph.

This is not special.

What's a little unusual is that it also has a "history" mode where you can say "show me the data for 2024-01-01" or whatever. The problem with a request like that is ... 2024-01-01, where, exactly? For that, the database needs a time zone, and it always had a setting that matched the machine's location, and that's perfect for these purposes. Also, we (family) don't look at that view all that often, so nobody noticed it was acting strange.

So, the other day, I was wondering exactly how hot it got on a particular day, and finally went to the history view. Then I noticed it was going from 5 PM to about 5 PM again, and knew that was some UTC-related fuckery. The mapping of 4 or 5 PM local time being midnight UTC has been beaten into my head from countless outages which were caused by some idiot ad product acquisition that started serving ads full-tilt at midnight UTC since "oh hey, this campaign is active now and we have ZERO IMPRESSIONS so far!" ... but I digress.

I was confounded by other work that's been going on to reduce exposures on my web servers that flipped them to logging in UTC because /etc/localtime and /usr/share/zoneinfo didn't exist in their little chroot-ish environment. I thought that shift had screwed up the CGI programs that turn the SQL gunk into JSON gunk for the browsers. That distracted me from the actual problem for a few minutes.

That's when my notes from the day of the upgrade came up, and I found my own "huh, that's weird" notation about the Postgres time zone thing. I had honestly forgotten about it, since, yes, I had bigger fish to fry. I was physically on site visiting family, and the Debian upgrade was a small part of a bigger trip. I'm mostly there to spend time with them, not to play with the damn computer. So, when some stupid nerd stuff happened, I noted it, but didn't shelve the rest of the usual activities to go spaz out over whatever this particular thing was.

Because, again, the primary use case ("what have the temps been like around here over the past day or so") *was working fine*.

Also... something else DID break on the box when I did that upgrade, and it broke hard. I went and dealt with that, and forgot all about the TZ thing. I haven't written about the other problem because I'm honestly a little gunshy after the whole atop thing earlier this year. What I have to say about the thing that broke isn't going to be nice, because it's a load of horse shit that it broke in the first place.

But, the devs behind that project *also* owe me nothing, and if they can live with themselves, foisting off their particular annoyances on the world and justifying it as okay in their bug reporting system, it's not like they're going to listen to me. Seriously, when you find that other people have reported a problem well before you and the powers that be have brushed them off, there is no reason to pile on. They obviously don't give a shit, and my "+1" isn't going to change that.

Could I have done better for this one derpy Debian box that's used by a couple of family members to look at temperatures that didn't actually have a problem after I solved the main one? Of course I could... but WHY? And... how? I spend enough time on computer shit as it is.

I could have gone "hey siri, remind me to look at that stupid postgres time zone thing tomorrow". I didn't.

Now, "SELECT 1" (THE ONE?), you tell me how you'd be any better at this.

If you're running Debian 12 and Postgres, you're in or around the Americas, and you're planning on upgrading to Debian 13, you might hit a fun little snag. I did when I did my first (testing) box:

2025-08-29 06:28:12.038 GMT [220328] LOG:  invalid value for parameter "log_timezone": "US/Pacific"
2025-08-29 06:28:12.038 GMT [220328] LOG:  invalid value for parameter "TimeZone": "US/Pacific"
2025-08-29 06:28:12.038 GMT [220328] FATAL:  configuration file "/etc/postgresql/15/main/postgresql.conf" contains errors

At the time, I went "WTF?" and just commented it out to get it running again. I had bigger fish to fry... and just kind of forgot about it. Everything seemed fine.

Here it is two weeks later and I just realized that one of my little history viewing tools for my temperature tracking stuff is generating graphs that run from 5 PM to 4 PM. In other words, it's showing me a UTC-based day, not a US/Pacific (PDT at the moment) day. 5 PM here during the summer is midnight UTC.

It took far too long to realize what was going on here. So I went, okay, uh, US/Los_Angeles? No. It's now America/Los_Angeles.

The worst part about this is that it didn't get so much as a mention in the Debian 13 release notes. I read through that document before going for it and never encountered it. Indeed, even now, you won't find "tzdata" or "zone" in it.

Normally when weird stuff like this happens, someone who lives on the bleeding edge runs into it and reports it. That's what happened with the xz kerfuffle: some person running the latest stuff hit some anomalous readings and went digging. Here, too, a few people hit the problem well before I did.

They hit it in 2023. US/* was moved to tzdata-legacy, and a few people asked for something that would pop up during upgrades. I guess it didn't happen.

So, as a public service, here you go. This is your notification.

Back in the days of analog phones and fax machines, I used to get asked to come up with solutions to random problems that people were having with them. One of them involved a single phone line, a "fax switcher", an actual fax machine, and not wanting to wake other people up when a fax was involved. Here's how that went.

Friends of the family had a good-sized house and they ran some kind of business that had started getting faxes at all hours. They had a single phone line and didn't feel like getting another one for whatever reason. They wanted to be able to get a fax without having all of the other house phones ringing at all hours of the day and night when it's "meant for the machine".

The dad of the family bought this "fax switcher" box, but couldn't figure out exactly how to tie it in. Somehow, they found out that I had done some small-time phone wiring adjustments and called me over to help.

The box worked like this: it would pick up the phone and would just listen to it for a few seconds. If it heard the CNG tone from a calling fax machine (beeeep!), then it would punch the call through to the "FAX" port on the back. It did this by putting the right flavor of voltage jazz on that port to make the machine see it and pick up. Then it connected the line port to the fax port and just hung out until the call was over.

This same box had a "VOICE" port on the back, and in the event it didn't hear the soothing tones of a calling fax machine, it would time out and would push its locally-generated ring voltage down that port instead. If you had some phones plugged into that, they'd ring, and you'd know to pick them up.

(I should note that the box played a nasty fake ringback tone to the caller during this phase... and yes, since it answered, it supervised, and the caller was going to pay for the call, even if nobody was home!)

The question was: how to get the rest of the house (regular phones) "behind" this box so they wouldn't ring when the call actually came in? Their 1960s house had all of its phone jacks wired in a way that wasn't unusual for the time but which made it ugly for this problem.

One very long length of cord came from the demarc box out back and stopped off at every jack. The installer actually just stripped back the outer insulation, then the inner insulation on the primary pair (red/green), and looped the now-bare copper around the terminal screws on the jacks. They did all of this without ever cutting the wire. Very clever.

I suggested we could just cut it in half at the first jack (in the kitchen), so the box would be the first thing on the line and the whole rest of the house would sit behind it. That didn't fly, since it meant the switcher box (and probably the actual fax machine) would have to go in the kitchen, too. They wanted all of the mess out of the way in the big bedroom on the first floor, and wouldn't you know it, that was the *last* jack on the line. That seemed like a pain but it turned out to be rather helpful.

The solution ended up being something kind of nasty, but it did work. That one long cord actually contained two pairs in case the residents ever wanted a second line. That is, in addition to the red/green, they also had a yellow/black pair in there, just hanging out, not doing anything. It didn't "stop off" at any of the jacks, but it was in fact present and ran uninterrupted from end to end.

I figured, okay, let's "cut the line in half" in the kitchen jack, but do it in such a way that it bridges the incoming line (from the outside box) to the yellow/black. Then it'll travel all the way around the house, untouched, until it lands at the last jack in their bedroom.

Then I just wired up that jack to have the yellow/black on line 2, and bought a goofy little splitter thing from Radio Shack to make sense of it. This was a six dollar plastic piece that would split a two-line jack into two single-line jacks and a pass-through.

So, position 1 on this thing had line 1 as primary. Position 2 on this thing had line 2 as primary. Position 3 on this thing was another "four wires, two lines", just like the jack it plugged into.

The fax switcher's LINE jack was connected to position 2 (yellow/black, being fed from the bridge behind the kitchen jack) and its VOICE jack was connected to position 1. In so doing, the box itself drove the rest of the house "backwards" and allowed those other phones to operate.

The best part of doing it this way is that I didn't have to go around to open up the other jacks in the rest of the house. They were just fine staying on red/green.

This worked and they loved it, but I was mildly concerned. Their whole telco setup was now reliant on this dumb little box staying plugged in and working exactly as I had set it up. If it came unplugged or something else bad happened to it, they'd have no phone service anywhere in the house. (I'm not sure if it was smart enough to short together the VOICE and LINE jacks on power failure. It was random consumer-grade plastic junk.)

For that reason, I also brought them a tiny little "patch cable" thing (that Radio Shack also sold for some reason) and told them "if you ever remove this box, you HAVE TO connect positions 1 and 2 with this little cord". I sure hope they remembered that.

I can't even imagine what happened if they ever sold that house. I'm sure someone came along later and screamed at the maniac that rigged it up that way.

Well, uh, hi, I'm that maniac, I guess.

Knowing what I know now over 30 years later, I think I would have left a note in the wall for the next person to find it. "Hey, this is why we did this, and you just need to patch red & green back together in this one place and you'll be back to where Ma Bell left it originally". The person who came along later would still be miffed, but at least they'd know why someone went and did something that bizarre.

I screwed something up the other day and figured it had enough meat on its bones to turn into a story. So, okay, here we go.

For a while now, I've been doing some "wrapping" of return values in my code. It's C++ stuff, but it's something that's been inspired by what some of my friends have been doing with Rust. It's where instead of just returning a string from a function that might fail, I return something else that enforces some checks.

Basically, I'm not allowed to call .value() or .error() on it until I've checked to see if it succeeded or not. If I do one of those things out of sequence, it will hit a CHECK and will nuke the program. This normally catches me fat-fingering something in development and never ships out.

Some of this code looks like this:

auto ua = req.util->UserAgent();

if (ua()) {
  req.set_user_agent(ua.value());
}

In that case, it's wrapping a string. It's wrapped because it can fail! Sometimes there's no value available because someone decided they didn't want to send that header in their request for some strange reason. I don't "do" "sentinel values", nulls, or other stuff like that, because I have my little "result string" thing going on here.

Easy enough, right? Well, I found myself making some mistakes when dealing with a series of calls to things that could pass or fail which worked in a similar fashion. They don't have a .value() but they can have an .error() and they need to be checked.

Sometimes, in my editor, I'd do a "delete 2-3 lines, then undelete twice, then twiddle the second set" thing for a spot where I had to make two very similar calls in a row. It might look like this:

auto ai = app_->Init();

if (!ai()) {
  log_something("blahblah failed: " + ai.error());
  // return something or other...
}

auto ni = net_->Init();

if (!ni()) {
  log_something("no shrubbery: " + ai.error());
  // return something blahblah...
}

But, do you see the bug? I'm using ai.error in the second spot instead of ni.error. ai is still available since it exists from that "auto ai = ..." line to the bottom of the block, and there's no way to say "hey, compiler, throw a fit if anyone looks at this thing after this point".

I'd have to do something odd like sticking the whole mess into another { ... } block just so ai would disappear, and while that would work, it also gets ugly.

Not too long ago, I came up with something else based on some newer syntax that can be wrangled up in C++. It's apparently called "if scope", where you can define a variable in the course of doing a branch on some condition, and then it only exists right there.

It looks like this:

if (auto ai = app_->Init(); !ai()) {
  log_something("blahblah failed: " + ai.error());
  // return something or other...
}

It looks a little awkward at first, but it's pretty close to the original code, and it also has a nifty side-effect: "ai" doesn't live beyond that one tiny little block where I report the error and then bail out.

With that in place, you can't make that "ai instead of ni" mistake from before. That's a clear win and I've been converting my code to it in chunks all over the place.

A couple of days ago, I did a change like that on some user-agent handling code, but screwed up and did it like this:

if (auto ua = req.util->UserAgent(); !ua()) {
  req.set_user_agent(ua.value());
}

That's basically saying: "if they *didn't* send a user-agent, then add its value to the request we're building up". Now, had that code ever run, it would have CHECKed and blown up right there, since calling .value() after it's returned false on the pass-fail check is not allowed. But, nobody is doing that at the moment, so it never happened.

The other effect it had was that it never added the user-agent value to the outgoing request when clients _did_ present one, and that's been the case all of the time.

So, a few days ago, someone reported that their feed score reporting page said that they apparently didn't send that header with their requests but they're sure that they did. They started chasing a bug on their side. I went "hmmm, oh no...", looked, and found it.

It's supposed to look like this:

if (auto ua = req.util->UserAgent(); ua()) {
  req.set_user_agent(ua.value());
}

So, why did I put the ! in front? Easy: most of the time, I'm handling errors with this stuff and bailing out by returning early. This is one of those relatively infrequent inverted situations where I want the value and jam it in there only if it exists.

It was a quick fix, but the damage was done. A few hundred rows in the database table picked up NULLs for that column while the bad version was deployed on the web server.

So now let's talk about what I'm doing about it. One thing I've been doing all this time when logging hits to the feed score project is that I also log the git commit hash from the source tree at the time it was built by my automatic processes. It's just one more column in the table, and it changes any time I push a new binary out there.

With that, it was possible to see that only this one build had the bug, and I didn't need to fix any other rows. The other rows without UA data are that way because some goofball program is actually not sending the header for whatever reason.

Next, I changed the report page to add a colorful (and very strange-looking) "fingerprint" of the build hash which had been logged all along but not exposed to users previously. Every row in the results table now sports an extra column which has a bunch of wacky Unicode box-drawing characters around U+2580 all in different colors. I use the build hash to set the colors and pick which of the 30 or so weird characters can go in each spot.

If this technique sounds familiar, you might be thinking of a post of mine from August 2011 where it was using MD5 sums of argv strings to render color bars.

This time around, since other people are the intended audience, I can't rely on full-color vision, so that's why there's also a mash-up of characters. Even if all you can see are shades of grey, you can still see the groupings at a glance.

So now, whenever something seems strange, the fsr users can see if I changed something and maybe save themselves from chasing a bug that's on my end and not theirs.

To those people: sorry! I still have to sit down and manually replace the data in the table from the actual web server logs from that time period. It'll fill in and then it'll look like nothing bad ever happened.

Until then, well, just know that one particular version blob has my "brown paper bag" bug associated with it.

Bugs, bugs, bugs...

And finally, yes, a test on this code would have caught this pre-shipping. Obviously. You saw the part where I'm doing this for free, right?

Sometimes, you just need to write down what you're willing to do and what you're not. I have a short tale about doing that at a job, and then bringing that same line of thinking forward to my current concerns.

I used to be on a team that was responsible for the care and feeding of a great many Linux boxes which together constituted the "web tier" for a giant social network. You know, the one with all of the cat pictures... and later the whole genocide thing and enabling fascism. Yeah, them.

Anyway, given that we had a six-digit number of machines that was steadily climbing and people were always experimenting with stuff on them, with them, and under them, it was necessary to apply some balance to keep things from breaking too often. There was a fine line between "everything's broken" and "it's impossible to roll anything out so the business dies".

At some point, I realized that if I wrote a wiki page and documented the things that we were willing to support, I could wait about six months and then it would be like it had always been there. Enough people went through the revolving doors of that place such that six months' worth of employee turnover was sufficient to make it look like a whole other company. All I had to do was write it, wait a bit, then start citing it when needed.

One thing that used to happen is that our "hostprefix" - that is, the first few letters of the hostname - was a dumping ground. It was kind of the default place for testing stuff, trying things, or putting machines when you were "done" with them, whatever that meant. We had picked up all kinds of broken hardware that wasn't really ready to serve production traffic. Sometimes this was developmental hardware that was missing certain key aspects that we depended on, like having several hundred gigs of disk space to have a few days of local logging on board.

My page became a list of things that wouldn't be particularly surprising to anyone who had been paying attention. It must be a box with at least this much memory, this much disk space, this much network bandwidth, this version of CentOS, with the company production Chef environment installed and running properly... and it went on and on like this. It was fairly clear that merely having a thing installed wasn't enough. It had to be running to completion. That means successful runs!

I wish I had saved a copy of it, since it would be interesting to look back on it after over a decade to see what all I had noted back then. Oh well.

Anyway, after it had aged a bit, I was able to point people at it and go "this is what we will do and this is what we will reject". While it wasn't a hard-and-fast ruleset, it was pretty clear about our expectations. Or, well, let's face it - *my* expectations. I had some strong opinions about what's worth supporting and what's just plain broken and a waste of time.

One section of the page had to do with "non-compliant host handling". I forget the specifics (again, operating on memory here...), but it probably included things like "we disable it and it stops receiving production traffic", "it gets reinstalled to remove out-of-spec customizations", and "it is removed from the hostprefix entirely". That last one was mostly for hardware mismatches, since there was no amount of "reinstall to remove your bullshit" that would fix a lack of disk space (or whatever).

One near-quote from that page did escape into the outside world. It has to do with the "non-compliant host" actions:

"Note: any of these many happen *without prior notification* to experiment owners in the interest of keeping the site healthy. Drain first, investigate second."

"Drain" in this case actually referred to a command that we could run to disable a host in the load balancers so they stopped receiving traffic. When a host is gobbling up traffic and making a mess for users, disable it, THEN figure out what to do about it. Don't make people suffer while you debate what's going to happen with the wayward web server.

Given all this, it shouldn't be particularly surprising that I've finally come up with a list of feed reader behaviors. I wrote it like a bunch of items you might see in one of these big tech company performance reviews. You know the ones that are like "$name consistently delivers foo and bar on time"? Imagine that, but for feed readers.

The idea is that I'll be able to point at it and go "that, right there, see, I'm not being capricious or picking on you in particular... this represents a common problem which has existed since well before you showed up". The items are short and sweet and have unique identifiers so it's possible to point at one and say "do it like that".

I've been sharing this with a few other people who also work in this space and have to deal with lots of traffic from feed reader software. If you're one of those people and want to see it, send me a note.

At some point, I'll open it up to the world and then we'll see what happens with that.

I get questions from people who read my posts and sometimes I answer them in a post. This is one of those times.

...

Someone read my "war room" post and picked up on the part where I spent several weeks trying to get to the bottom of what turned out to be "kill -9 -1" nuking the world on a bunch of FB machines. They asked how I keep track of things during that time, what my working memory is, and is it paper, text files, IRC messages, or just remembering things.

The answer is: back when I used to do that kind of thing, I found it very useful to have a "MMDD" (hey, I'm in the US, so just pretend it's the back half of an 8601 number... you'll see why...) directory, and inside of it, I'd have some short names for whatever I was dealing with that day.

That means today would be 0629, and then something inside of it might be "fbar" or "rsw" or "webi" or something. It was just something to keep all of the crap together and yet away from other things I was doing that day, while also distinguishing it from other times I might've done a "fbar" or "webi" or whatever project, if that makes sense.

These would just live off my home directory, and yes, it would fill up with crap, but I'd batch them up when a year ended. I'd take 01xx through 12xx and move them into "2013" when 2014 began, for instance. So, as you can see, they *are* ISO-8601 dates, sorta, but it's ~/YYYY/MMDD/foobar once it's old, and it's merely ~/MMDD/foobar until then. This is a balancing act between speed and having my homedir fill up with tons of ancient crap.

Any time I needed scratch space for output from things, that's what I used. That might be the output from a bunch of sweeps over the fleet to look for anomalies. Let's say I ran a command that sshed into a few hundred thousand hosts to look for common items in the logs. The stdout/stderr from that would probably be there so I could hit it up multiple times without asking the job runner system for another copy. It's a lot faster that way.

But in terms of troubleshooting things and dealing with permutations, it's hard to beat paper, and I usually end up with some kind of "lab notebook" at any job I work. I can think of examples of those going back over 20 years. The only problem is that the stuff in them really properly belongs to the company so I don't tend to have them after the fact. A great many pages of context have gone in the shredders over time, and not because I didn't ask. I've asked if anyone wanted them, and the answer has always been no.

There were also the internal posts about things, and then running commentary on IRC channels, but those tend to be useful after the fact, or for getting help from other people. My own state-keeping tends to stay on something close at hand and (usually) physically tangible.

As for those <date>/<term> directories, some of them turned out to be rather handy after the fact. A lot of times, something would happen, and then time would pass, and it would break *again* in exactly the same way, and I could think "didn't we deal with these people already?", dig around, find something from six months earlier, and go "AHA!". Now armed with the date, I could pull up the right posts, IRC logs, group messages, graphs, or whatever else.

Considering how much random crap I dealt with in any given day during my time "in the barrel" (kids, ask your parents), that was the only way to make any sense of it later. Without stuff like that, by the end of the week, I'd have no idea what I had been doing on Monday or Tuesday. That's how bad the load got at points.

...

Someone else wrote in and asked if I could do something to improve the overflow calculator thing I hacked up last week. It wasn't particularly usable on mobile, and that's definitely true. I wrote it on a laptop and paid exactly zero attention to what it would look like on a weirdly small screen that's usually taller than it is wide.

It was a fair point, so I took a whack at making it suck slightly less. It's probably still terrible, but in the specific case of me holding my phone upright, it looks halfway usable now. You're no longer forced into "flyspeck-3" mode with the fonts, at least.

CSS is such a mess.

...

I occasionally hear that the site is unreachable from one spot or another. This almost certainly comes down to IP filtering on my part. Perhaps you've read about the influx of web weasels who are scraping the living shit out of everything remotely URL-shaped they can get their hands on. My stuff is certainly in that space, and they show up here regularly.

Besides that, there are also a number of networks which send nothing but straight-up abusive traffic. This is where you look at the logs and see stuff like them using every single IP address in a (v4) /24 to scan for random webshit vulnerabilities. It's like, nice job, fucko, but I don't run PHP here. Anyway, that's pretty solid evidence that a given network is not worth hearing from ever again, and so into the filter it goes.

A whole lot of this happens automatically just based on whatever traffic is sent this way first. Send bad traffic and meet the bit bucket. I don't even find about most of it since it's constant and utterly uninteresting.

Then there are the people who run feed readers which don't play nicely. As previously described, they will get a handful of chances to slow down with 429s, and then if the web server feels like those aren't having an effect, it'll just ignore the traffic for however long it feels like. Again, I'm also not in the loop on this sort of thing. It's all automatic.

Finally, there is a new rub. I taught myself how to ingest BGP data, and so can now trivially go from an IP address to the autonomous system number of whoever's advertising it, including overlapping advertisements (like a /24 out of a bigger /20). Then I wrote something that will dump out an entire AS, and it's not hard to imagine what I do with that.

Enough bad behavior from a host -> filter the host.

Enough bad hosts in a netblock -> filter the netblock.

Enough bad netblocks in an AS -> filter the AS. Think of it as an "AS death penalty", if you like.

As long as clueless network operators continue to let abusive customers bounce around thousands of dynamic IP addresses with not so much as a hint of SWIP data, those entire net blocks will find themselves unable to get to large swaths of the net. That's just how it is, and it's nothing new. Send crap, meet /dev/null.

Dealing with what the web has become is exhausting.

I've long had a list of "magic numbers" which show up in a bunch of places, and even made a post about it back in November of 2020. You ever wonder about certain permutations, like 497 days, or 19.6 years, or 5184 hours, and what they actually mean?

I've been doing that stuff by hand in a calculator and finally decided to just do it in Javascript and put it online for anyone to try.

So, here's my latest waste of CPU cycles:

My rollover calculator.

I still haven't figured out the Crucial SSD 5184 hour thing, so it's not in there. 5124 hours, sure, I can understand that one, but 5184? 60 more?

Anyway, have fun while the world burns.

rsync is one of those tools which is rather useful. It saves you from spending the time and effort on copying data which you already have. It's the backbone of many a mirror site, and it also gets used for any number of backup solutions.

There's just one problem: in the name of efficiency, it can miss certain changes. rsync normally looks at the size and modification time of a candidate file, and if they are the same at both ends, that's the end of any consideration. It won't get any further attention and it moves on to something else.

"So what", you might think. "All files change at least their mtime when someone writes to them. That's the whole point of a mtime."

And yet... I'm writing this post, and here we are.

The keen-eyed observers out there are probably already thinking "ooh, bit rot" and other things where one of the files has actually become corrupted while "at rest" for whatever reason. Those observers are right! That's totally a problem that you have to worry about, especially if you're using SSDs to hold your bits and those SSDs aren't always being powered.

But no, this is something you have to worry about *beyond* that. This is about a "sneak path" that you probably didn't consider. I didn't.

Here, let's run a little experiment. If you have a x86_64 Debian box that's relatively current and you've been backing up the whole thing via rsync for a year or two, go do something for me.

Go run your favorite file-hasher tool on /usr/lib/x86_64-linux-gnu/libfribidi.so.0.4.0 for me. Give it a sha256sum or whatever, or even md5sum if you're feeling brash. Then note the modification time on the file.

Now mount one of your backups and do the same thing on the version of the file that's on the backup device. See anything ... odd? Unusual?

Identical mtimes, identical sizes... and different hashes, right? I spotted this on a bunch of my machines after going "hmmm..." about the whole SSD-data-loss thing.

Clearly, something unusual happened somewhere, and it's been escaping the notice of your rsync runs ever since. I haven't gone digging into the package history for this thing to find out just when and where it happened, and (more importantly) how. It's rather unusual.

If you're freaking out right now, there is some hope. rsync has both -I and -c which promise to not use the quick method and instead will run a checksum on the files. It's slower so you won't want to do this normally, but it's not a bad idea to add this to the mix of things that you do every so many rotations.

I should point out that the first time you do a forced-checksum run, --dry-run will let you see the changes before it blows anything away, so you can make the call as to which version is the right one! In theory, your *source* files can get corrupted, and if you just copy one of those across, you have now corrupted your backup.

Isn't entropy FUN?

At the start of 2023, I wrote a post talking about why I still had an "old-school cert" on my https site. Well, things have shifted, and it's time to talk about why.

I've been aware of the ACME protocol for a while. I have tech notes going back as far as 2018, and every time I looked at it, I recoiled in horror. The whole thing amounts to "throw in every little bit of webshit tech that we can", and it makes for a real problem to try to implement this in a safe and thorough way.

Many of the existing clients are also scary code, and I was not about to run any of them on my machines. They haven't earned the right to run with privileges for my private keys and/or ability to frob the web server (as root!) with their careless ways.

That meant I was stuck: unwilling to bring myself to deal with the protocol while simultaneously unwilling to budge on allowing the cruft code of existing projects into my life.

Well, time passed, and I managed to crack some of my own barriers. It wasn't by using the other projects, though. I started ripping into them to figure out just how the spec really worked, and started biting off really really small pieces of the problem. It took a particular forcing function to get me off my butt and into motion.

About six months ago, I realized that it was probably time to get away from Gandi as a registrar and also SSL provider (reseller). They had been eaten by private equity some years before, and the rot has been setting in. Their "no bullshit" tagline is gone, and their prices have been creeping up. I happened to renew my domains for multiple years and have been insulated for a while, but it was going to be a problem in 2025.

Giving them the "yeet" was no big deal, but the damn rbtb certificate was going to be a problem. Was I going to start paying even more for the stupid thing every year, or was I going to finally suck it up and deal with ACME?

That still left the problem of overcoming my inherent disgust for the entire protocol and having to deal with all of these encodings they force upon you. My first steps towards the solution involved writing really small and stupid utility functions and libraries that would come in handy later. I'm talking about wrapping jansson (a C library that handles JSON) so that it made sense in my C++ world and I could import JSON (something I use as little as possible). That kind of thing.

This also meant going down some dead-ends, like noticing that libraries existed which would allegedly create certain things (like a JWK) for you, and then realizing that they were not going to make my life any easier. I'd poke at it, reach my limit, and then swear and walk away for another couple of days.

This went on for some time. I have a series of notes where I'd grab a piece of the problem, wrangle it around for a while, get grossed out, and then set it down and go do something else. This just kept happening but I slowly made progress with small pieces that would Do Stuff, and then they'd connect to each other, and so on like this.

One positive development during all of this was discovering this "pebble" test server I could run on an isolated fake system. It would act as an ACME server and would let me harass it with my feeble attempts at implementing a client instead of bothering the real CAs. Even "staging" servers deserve better treatment than active development, after all.

And, well, after a whole lot of mangling and dead-ends and rewrites and other terrible crap, I had an awful little tool that would take a CSR, do all of the idiot dances and would plop out a certificate. I pointed it at Let's Encrypt staging, and it worked. Then I pointed it at their prod site, and _that_ worked. So I did it for the real thing ([www.]rachelbythebay.com), and *that* worked, and I dropped it into place.

Thus, for the past couple of weeks, if you've been hitting the https version of my site, you've been doing it across the new setup.

Now, I took notes about this, and I wanted to share some of my original off-the-cuff thoughts about implementing this for anyone who's similarly broken in the head and wants to see how bad it can be. I will note that I wrote this based on the first thing that worked, and it does not necessarily reflect the implementation I'm on a few weeks later.

...

Make an RSA key for your web site. Then make a CSR for it, setting the CN and adding a matching altname as an extension. No other fields matter. Nobody looks at those, anyway, and none of them will influence your final certificate no matter how prosaic or precise you get in there.

Make an RSA key of 4096 bits. Call it your personal key.

Write something that'll read a CSR. It needs to extract the CN and the SANs - the DNS: ones, at least. Ensure there's actually a CN [*] and actually SANs, and that the CN occurs within the SANs. So, yes, you have to have at least one SAN.

[* - I now know you can run with just SANs. I did not at the time.]

Write something that'll do a HTTP GET to <directory URL> which is given to you by the ACME service operator. It then needs to parse the body as JSON (or die) and extract some strings from the top-level object: "newNonce", "newAccount" and "newOrder" at the very least.

Write something that'll read an RSA key file on disk. It needs to extract the publicExponent (probably 65537, but you never know...) and the modulus. Make it read your personal key from earlier.

If you end up using "openssl rsa -in foo -noout -text" to do this, the modulus is a bunch of printed hex digits, like "00:ff:11:ab:cd:ef:22:33". It hard-wraps the output and also indents the lines, so you get to clean all of that up first.

Skip the first 00 for some inexplicable reason. Take the other bytes of the modulus and turn them into the actual character values, so a literal 0xff, 0x11, 0xab, 0xcd and so on down the line in the same order you find them in the file. Hang on to these values for later.

Write something that'll turn an ordinary integer into its equivalent big-endian bag of bytes, but don't pad it out to any particular alignment. You need to take that "65537" from your publicExponent and turn it into the equivalent bytes, so 0x01, 0x00, 0x01. Yes, it's a number you just turned back into a binary representation.

Write something that will do base64 *style* encoding, but not quite. The last two characters in the encoding set are usually + and /, but that won't do for webshit, so you need to make it use - and _ instead.

Take that publicExponent (65537), pump it through your big-endian bag-of-bytes encoder to get 0x01 0x00 0x01, then put it through your "base64web" encoder to get "AQAB".

Start a new JSON object. Add a string called "e" and set it to the output of the above step. So, yes, instead of saying that "e" equals "65537", you're saying that "e" equals "AQAB". Aren't you glad you did those extra steps?

Add another string called "kty" and set it to "RSA".

Add another string called "n" and set it to the "base64web" version of your modulus bag-of-bytes from earlier.

Take this JSON object and make it into a sorted compact string representation. This means it goes "e, kty, n" and it also has all of the usual padding (spaces) squished out. Call this a JWK string and save it for later.

Create a second JSON object. Add a boolean to it named "termsOfServiceAgreed" that's set to true. (Guess you'd better agree...)

Look up the URL for "newNonce" from the directory JSON you got earlier.

Make a HTTP HEAD request to that URL. Dig around in the headers (not the body, since there is no body on a HEAD) until you find "Replay-Nonce". Extract the value of that header. Hang onto it for later.

Look up the URL for "newAccount" from that directory JSON for before.

Create a third JSON object. Add a string to it called "url". Set it to that (newAccount) URL. Add a string called "alg". Set it to "RS256". Add a string called "nonce" and set it to the value from the *header* in that last HTTP HEAD request.

Add an object to this third object called "jwk". Within it, add "e", "kty" and "n" in a manner that matches what you did earlier (you know, from the "JWK string" you're still holding for later).

Dump this third JSON object to a sorted compact string representation. Call it "protected".

Dump the second JSON object to a sorted compact string representation. Call it "payload".

Create a string where you literally concatenate those two prior strings, such that it's the value of protected, then an actual period (as in 0x2e, a full stop, whatever), then the value of payload.

Write something that'll create a SHA256 digest of an arbitrary string and will sign it with an arbitrary RSA key (like "openssl dgst -sha256 -sign <key>"). The key in question is your personal key.

Pump that "<protected>.<payload>" string through the digest function. Then run it through your "base64web" encoder. Call this the signature.

Create a fourth JSON object. Add a string called "protected" and set it to whatever you built a few steps earlier. Add another string called "payload" and set it likewise. Then add one called "signature" and set it, too.

Take this fourth JSON object and dump it to a sorted compact string representation. Call this your post body.

Make a HTTP POST request to the "newAccount" URL from the directory. Set the content-type to "application/jose+json". Set the post data to the post body string from the previous step.

Dig around in the headers of the response, looking for one named "Location". Don't follow it like a redirection. Why would you ever follow a Location header in a HTTP header, right? Nope, that's your user account's identifier! Yes, you are a URL now. Hang on to that URL for later.

You now have an account.

You still have much to do.

...

That's about where I stopped writing my take on the protocol.

Again, my program no longer works quite like this, but this is where it started after having observed a bunch of other stuff that already existed.

So far, we have (at least): RSA keys, SHA256 digests, RSA signing, base64 but not really base64, string concatenation, JSON inside JSON, Location headers used as identities instead of a target with a 301 response, HEAD requests to get a single value buried as a header, making one request (nonce) to make ANY OTHER request, and there's more to come.

We haven't even scratched the surface of creating an order, dealing with authorizations and challenges, the whole "key thumbprint" thing, what actually goes into those TXT records, and all of that other fun stuff.

...

Random side note: while looking at existing ACME clients, I found that at least one of them screws up their encoding of the publicExponent and ends up interpreting it as hex instead of decimal. That is, instead of 65537, aka 0x10001, it reads it as 0x65537, aka 415031!

Somehow, this anomaly exists and apparently doesn't break anything? I haven't actually run the client in question, but I imagine people are using it since it's in apt.

...

This complexity must be job security for somebody. Maybe multiple somebodies.

During the lockdown of 2020 and subsequent Twilight Zone times that followed, I learned a bit about how a local restaurant's web site operates. That web site had become the lifeline for the business since everything had turned into takeout or delivery, and it wasn't always right, so I brought it up with the owner. He said he had "a guy who takes care of it". This makes sense since his thing was running a restaurant and keeping that going. He doesn't sit around dealing with domain names and web sites every day.

At one point "his guy" became unreachable for several months, and the restaurant owner asked me what his options were since he knew I was "one of these computer people". I had to lay out the top to bottom of how this stuff works in terms of "this content gets displayed with this name on it", and figured it might benefit others if I put some variant of it online.

Let's say you're the restaurant owner. You're paying some third party to run a web site and list your menu, hours, and that kind of stuff. They might also have some kind of thing rigged up to send orders through online, or an agreement with yet another vendor to do it through them.

You need to get something changed on the site. Maybe your menu changed and you need to warn people about the increased price of egg-based products. Maybe you have new hours of operation. Whatever. What steps can make that happen?

In no particular order, an incomplete list:

"Ask them to change it" - contact your person (people) and ask them to change it. That's what you're paying them for. You are paying them, right?

"Change the actual files" - find out how the serving of content works, gain access to it, and then make changes to the file(s). This requires account access to whatever they happen to be using to run the site. If it's just a bunch of flat files on a disk somewhere, it might be easy. If it's just one entry in a much bigger system, it might not. Also you have to know how to do this and/or have new people to do this for you (as with most of the entries in this list).

"Replace the document root" - figure out how the web server works, then swing your-restaurant.example.com around from their directory to your directory, or do the equivalent database wrangling if it's some dynamic thing. This requires some admin powers on the box/system itself.

"Take over the server" - gain admin powers on the box through security holes or good old-fashioned physical access and single-user mode. Then go in and change the document root, edit the files, or do whatever else you feel like doing. Requires security skills and/or physical access to the server.

"Replace the server" - find out where the server is (assuming it's just one machine) and then physically replace it with one that you control but is otherwise configured for the right IP address(es) and whatever else is required to serve the site. This might mean "unplug the old one and plug a new one in on the same spot". Requires physical access to the hosting arrangements, and if it's in the clown, yeah, forget about it.

"Change the DNS" - get access to whatever controls the DNS zone for that domain and repoint it to new hosting arrangements with new content installed. This means changing things like A and AAAA records and/or CNAMEs. Requires ability to log in to whatever's running the show.

"Change the primary nameservers" - get into the account at the registrar for the domain and repoint the primary nameservers to ones where you can set the data for the domain. Continue as above. Requires ability to log into the registrar account.

"Move the domain to another registrar" - maybe you can't get into the existing registrar, but maybe you can go through the transfer process to pull it to another one where you can change the nameservers. Then you proceed as in that option (above). Requires ability to transfer a domain, which typically involves a one-time key from the "losing" registrar, which in turn usually involves access to the account. There may be legal options for *yoinking* the domain without such access, assuming you can prove ownership, or otherwise bludgeon the companies involved into doing what you want.

Basically, if you actually own some of the items in question, you have options. If you are the owner of the registrar account and it's just some rando tech person who's a contact on it that does the work, you authenticate yourself to the registrar out of band, have them removed, get yourself (or a new tech rando) installed, and continue from there. The same applies for the DNS serving, or the actual web hosting.

This pattern pretty much plays out everywhere: is it your account? Or, did you pay them to "take care of everything" and your only "interface" to it is through them?

I don't think people really know the implications of some of these setups.

Okay, first off, everybody breathe. Everyone is freaking out. This is not the way to do this.

Right, so, let me try a half-assed analogy here. There's a day care center and a bunch of people send their kids to it. Then one day, someone finds out that they built the playground out of the sharpest materials available. It's not that the playground is from the '70s or something, either. It's relatively new.

Saying "maybe you don't want to send your kids there" is what I did.

Now, so, this is a bad analogy, because I didn't go into details. In this case, I'd rather the world have a chance to turn this stuff off before it becomes known just what all is possible. The nature of this sort of thing means it's a race and I'm on the side of protecting machines for some reason, so I asked people to stop running it.

Now, first off, I don't know exactly how to exploit this sort of thing. I was there in the 90s when this overflow stuff started popping off, and I'm pretty sure that if you can do this, you can do much worse.

This is what I'm talking about:

Terminal 1:

user1$ random-tool

Terminal 2:

user2$ atop
malloc(): corrupted top size
Aborted

... that's pretty bad, right?

So you twiddle the tool a little bit and go again and...

user2$ atop
Segmentation fault

That's not great, either!

Or... how about...

user2$ atop
Fatal glibc error: malloc assertion failure in sysmalloc: (old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)

Again, this is not my world. I've never written a heap exploit, but reading about it briefly makes me think that there's meat on these bones.

user1 does something... and gets user2 to blow up. If you can make that do something useful, then you get user2 to run stuff on your behalf.

What if user2 is root? Then you own the box.

Okay? So maybe stop running it as root, but also just stop running it.

My life as a mercenary sysadmin can be interesting. Sometimes I find things, and sometimes I hear things. Now and then I say things.

Right now, I think it's probably best if you uninstall atop. I don't mean just stopping it, but actually keep it from being executed.

I'm not talking about the OG top, or htop, iftop, or anything else with a "top" name. Just atop.

I can go into why another time.

Look out, I'm talking about feed stuff again.

Since last week's post with the latest report on feed reader behaviors, a fair number of things have happened.

First up, I reported that two FreshRSS instances were sending unconditional requests. This turned out to be a pair of interacting bugs, and the developers hopped right on that and stamped it out. Their dev instances shifted right away. That's great.

Then, something funny happened. I wrote that post about the modem noises and it included a MP3 file. The way I do this in my stuff here is that I have some commands baked into my page generator that knows how to create the <audio> container and then put one or more <source> elements into it. So, I did that for the (admittedly crappy) recording and shipped it off.

I didn't know this at the time, but this immediately triggered strange behavior in a number of feed readers, FreshRSS included. They all started sending the wrong data. That is, they'd get the latest version of the feed with the "2025/03/05/1670" post in it, but on their next poll they'd still be sending the If-Modified-Since and/or If-None-Match values from the previous version.

This sent me down the road of trying to flush out this behavior on the feed score project side of things, but nothing I did would reproduce it. Again, I didn't know that something about that specific post was responsible, so I was groping around in the dark.

Finally, I reported this to the FreshRSS devs, and they came right back and showed me my problem: my feed says it's xhtml, but it was doing something that's not valid: <audio controls>. That's not valid XML, so it's not valid XHTML by extension. It's just fine in a regular web page, naturally.

I hate the web.

How did this happen? Easy: In 2023, I rewrote my generator for these web pages and the feeds. There's now a bunch of code that *tries* to crap out reasonable (and validating) results instead of doing "HTML by printf" like it had been before.

Along the way, I obviously didn't internalize the rule that "in XHTML, attribute minimization is forbidden". It sounds like the kind of thing I would have read once back in the 90s when I was trying to get pages to validate as XHTML 1.0 for some cursed reason, and then promptly forgot. So, it would do a bare "controls" attribute and thought it was fine.

In practical terms, it means you can't do this:

<audio controls loop>

But instead you have to do this:

<audio controls="controls" loop="loop">

So yeah, I wrote something terrible. Again, I hate the web.

Anyway, what was happening is that a single post within the feed was failing validation as a self-proclaimed xhtml document, so the feed readers were rejecting the _entire poll_ and were not updating their caches. The next time they polled, it was as if they had never sent the request in the first place (as far as they were concerned).

Once I shipped a fixed version of the feeds, they went back to normal.

I'd snark about something here, but clearly, I'm no better at this.

What's different now? Well, my bits of code which are responsible for attributes ("controls", "loop", "href", ...) now refuse to generate a minimized attribute when being used for XML, and that includes the feed stuff. They'll blow up before they ship garbage like that again.

My thanks to the FreshRSS team for being responsive and helping me out.

Now, where did I put my clown nose...

I recently wrote about getting a pair of VIC-20s and attached 1670 modems to connect over a length of line cord with nothing else in the middle. At the end, I tossed out a side note about the odd noises these things make in dial mode.

Some readers have been asking if it could be a relay, like for pulse dialing, and so I figured I should try to clarify as much as I can.

First, here's a recording from the other day's shenanigans, complete with me bopping the keys on the less-reliable computer to make it Do Stuff.

For those who can't hear it for whatever reason: there's some background humming in the whole recording from other devices in the room. But, then there's a source of white noise that's much closer and louder - that's the modem joining in. It "dips" every now and then and is replaced with the "pa-tink" noise I mentioned and then it comes back up to the previous volume level. This happens a few times, and I smack a key or two, and then the recording ends.

Looking in Audacity, it hits at about 1.2s, 2.4s, 3.6s, and 4.8s. That's curious, right? About 1.2 seconds each time?

To be crystal clear: this is not pulse dialing. That sounds completely different, and besides, I didn't tell the modem to dial anything. I sent it "ATD" which basically means "go into originate mode without actually trying to generate any pulse or tone signals - just grab the line and start listening".

Another rejected hypothesis: this is not just something broken in my originating test unit here. Both of them do it when they're in that role. It's also not a case of "they're almost 40 years old and they both degraded into this state", because the original I had way back in 1988 definitely did this, too. That's how it got stuck in my head in the first place!

One thing I don't know is whether this is solely a function of the local speaker, or if it's actually putting that clicking onto the line. I'm guessing it's just the speaker, but I'm also not feeling like setting up the test rig and somehow finding a way to listen to one side and not the other.

For whatever reason, the 1670 makes those noises when it's sitting there waiting for an answer tone from the other side, and I've never understood why. Obviously, at this point, it doesn't matter, but it's just one of life's little nerd mysteries.

In mid-January, I made an announcement that the feedscore system would be resetting the logs for a fresh start, and was bumping the URL along to cast off forgotten readers. This had the desired effect of focusing on those people who are actively involved and let the others start getting DNS failures without bothering my servers. Win-win!

Anyway, this means we now have about 45 days of data to look at, and it's possible to build a report based on the latest observed behaviors.

The usual caveats apply: if one reader instance is being odd, that might be one user smacking the refresh button over and over and/or using terrible settings. If multiple instances of the same program are behaving the same way, it's probably the program itself.

Let's start with the ones that seem to have their conditional request situations figured out. This is clearly the biggest part of the list, and it's great to see things lining up like this.

First up, there's one that's not published yet, and they're doing fine. I hope their efforts are paying off and they have a successful launch when they are ready.

There are a bunch of newer Minifluxes doing their thing, with no problems to report. It seems rather popular.

One of the Miniflux users has a different reader program using the same key and it's screwing up the reporting. You should retire that key and get two more from me. This advice goes for anyone: don't point more than one instance at any given key. It renders the data unusable.

There are several NetNewsWire instances doing their thing with no trouble. I haven't been able to trip them up with pathological (and yet totally valid) bumps of the Last-Modified and ETag values yet. This is good! Some of them have bizarre polling intervals... guessing that's user-driven?

There are a couple of SpaceCowboys Android RSS reader instances which seem to be doing fine. One of them has a bunch of oddly-timed unconditional requests which I assume is the user smacking that refresh button. Smack that button with care!

A couple of Vienna instances are doing fine.

Audrey2 RSS reader is fine.

A couple of Emacs Elfeed instances are fine.

"unnamed-feed-reader" (yes, really, that's what it sends) is doing fine.

There's something which doesn't send a User-Agent at all (okay...), and it has some bizarre anomaly of sending a "" INM at startup, but then has been behaving. (Hint: just drop the header if you don't have a value for it!)

Someone put a lot of work into making a curl instance do the right thing, and it's running great.

Roy and Rianne's Righteously Royalty-free RSS Reader is fine.

github.com/er0k/feeds is fine.

walrss is fine.

feedmail.org/0 is fine.

MANOS is fine. (Is that the feed reader of fate?)

Bloggulus is fine.

There's some unidentified browser extension that's doing fine.

feedbase-fetcher is doing fine.

inforss is doing fine.

feedparser/6.0.2 is *so close*. It just needs the 59/60 minute timing adjustment thing (assuming it's going for 1 poll per hour).

...

There are a few which are still "double-tapping" the server when the feed is first added. This would trip a 429 on the real feed and generally indicates a problem with the data model. Another reason for the URL change for this project was to get readers to run their "new feed" flow again.

Two Feedbin instances did this same thing when they were added.

Yarr/1.0 did a second poll about 300 milliseconds later, and a third poll another 27 seconds after that. This is the wrong way to do things. The other timing also seems to not follow any real pattern.

...

Now let's talk about readers with some cache bugginess:

A couple of Newsboat instances still can be tripped up and will keep sending old values in their If-None-Match headers.

There's a BazQux instance which has the same INM bugginess. I wonder if they're using the same library for their HTTP comms.

...

Then there are some weird ones:

rss2email randomly goes unconditional at times. No idea why.

NextCloud-News/25.x.x still sent at least one 1800 IMS. I don't get this. If you weren't sent a value, don't return it in a header. On a positive note, it's no longer beating the hell out of the server trying to get a favicon this week (which isn't in the reports, but I can see it on my side).

...

Finally, everything else:

Something claims to be a browser (Chrome 77? Sure...) and sends 100% unconditionals. This is bad. It also sends HEAD requests which aren't visible in the report, but I can see them, and they're broken and pointless.

One key either has two separate Thunderbird instances banging away (bad! get separate keys!) or has one terrible implementation that always polls twice. Either way, it's not usable data.

There are also two FreshRSS instances which are both sending 100% unconditionals. What happened here? Over and over, it's just this sort of request:

Host: <redacted>
User-Agent: FreshRSS/1.26.1-dev (Linux; https://freshrss.org)
Accept: */*
Accept-Encoding: deflate, gzip, br, zstd

I just don't understand.

I've been up to no good this afternoon with a pair of VIC-20s. A very long time ago, I asserted that you could connect some modems with just a length of line cord in the middle. In the time since then, various folks have done things to recreate the old days of dialup BBSes and even ISPs, and they've gone to lengths to have a line simulator in the middle, or a battery, or other strange things.

This always bothered me since I clearly remembered running a cord down the hallway between my little Commodore and the family's "big machine" (386) and plugging it into both modems. I did "ATA" on one, then ran to the other one and did "ATD", and they connected. Then I had a very slow, very crappy text-only "network" between two very dissimilar machines... without having a second phone line in the house!

That's where I've been for the past 30 or so years, but today I got a chance to finally figure this one out. I've been slowly accumulating the pieces of the puzzle, and recently, the last one landed. I got one of these "Penultimate+2" cartridges for the VIC which has a "dead test" in it. My original machine from the early 80s hasn't booted into the usual BASIC screen for years, and I wanted to know why.

Finally, this cartridge told me today: it's a bad BASIC ROM. Well then. That makes some amount of sense why it can't get to the usual "CBM BASIC V2" blahblah, and why other things (like certain carts) will still work.

Then, while pondering what to do about the bad ROM, I noticed this Penultimate cart had a built-in terminal program. Would it talk to the modem without using BASIC? Only one way to find out. I had a couple of old modems from various fits of boredom over the years, plugged one in, and gave it the good old "AT" to see if it lived. It DID: "OK".

Well, obviously now it was ON. I had to break out the "good" VIC-20 a friend gave me in the 90s to get it set up with the second modem. Since there's only the one cart and it can't be in two places at once, I instead typed in a godawful program to sling bytes between the modem, keyboard and screen. Once that was working, I tried the "AT" there, and that modem also worked: "OK".

The next thing was clear: hook them to each other and connect them. One small problem: I needed a phone cord. I haven't had a regular old analog phone for well over a decade now, and even that was late in the game to get rid of them. Where the hell was I going to find a plain old flat satin phone cable?

Well, somehow I found one buried in a box of nerd stuff, and naturally one of the retention clips snapped off, so I had to curl it around and jam it in there just so to keep it connected. Then it was a matter of ATD on the one and ATA on the other and I was rewarded with a terrible racket and then CONNECT.

I only have a single working RF modulator (and TV for that matter) so I could only watch this from one side of things, but it was still a great feeling just being able to dance around on the keys on one end and see it appear on the one that was driving the display.

So, yes, I can now (again) positively assert that at least when the Commodore 1670 is involved, you can totally smack it around and make it drive an otherwise dead pair that goes straight into another modem - no telco, no battery, no dial tone, etc. (You might need to do ATX1 to make it not look for the dial tone, naturally.)

I sure hope that isn't useful information to anyone. For me, it was more about putting to rest this random nagging thing in the back of my head. Other people also worry about making sure they're not handing out misleading info as authoritative, right?

Side note: yes, these modems make a "pa-tink, pa-tink, pa-tink" noise after the dial phase (if any) while waiting to connect. I have no idea what this clicking noise was for in this old modem. Maybe someone out there knows why and will send me a comment. Hint hint.

A reader asked me to follow up on something I had mentioned quite a while back. Way back in 2018, I wrote that maybe I'd write about "war rooms" and why they are bad for getting thoughtful analysis work to happen.

Here's my best take on it, in feedback style, where things are sort of off-the-cuff and I don't have a prepared statement ahead of time. Ready? Here we go.

My canonical example for a hellish war room is what happened on Friday, August 1, 2014. For people in the know (like anyone who took my class for new technical employees), that's the date of "Call the Cops" - an epic outage that took down all of Facebook for several hours. It was so named internally because the sheriff of LA County tweeted something like "we know it's down, it's not a law enforcement emergency, so stop calling us about it".

I've mentioned this any number of times in talks and posts, and other people have too. It's no secret. The site tanked bigtime that day.

It actually broke while I was waiting for my bus to arrive, so I ended up getting online from the sidewalk using my cell phone's tethering function, then rode in doing my very best to follow things while bouncing around as the bus shot up 101 to Menlo Park.

Then, since it was Friday and we were going to do the usual weekly "big show" of reviewing recent outages (SEVs), the Muppet-named room used for those events was instead repurposed as a "war room". It was a bunch of engineer type people thinking very hard and sweating a lot... in a relatively small room with not-great ventilation, door open or no.

It got smelly. Okay? I'm not going to sugar-coat it.

There's more, though. I've said a few times that I'm really not "fully functional" when I'm going through the Mac UI. For that reason, I ran VMWare Fusion on my company laptop and had Ubuntu running in a virtual machine, and used that for all of my work.

Still, actually doing work on the laptop proper (a 13 inch Macbook Air) was janky as hell because the key layout isn't quite right (want to alt-tab in Linux? better use option-tab!), and besides, the screen is tiny! It's a great little machine for browsing the web and reading e-mail, but it's no place to try to crack open a bunch of xterms and do Real Work.

Could I run my terminals in there? Yes. Did I? Yes, for a while. Was I effective? Not really. I missed my desk, my normal chair, my big Thunderbolt monitor, my full-size (and yet entirely boring) keyboard, and a relatively odor-free environment.

Fortunately, by that point in my life I had gotten old enough to where I was willing to take steps to bolster my own sanity and effectiveness, and said I'd get back on from my desk. I crossed back to our building, posted up at my desk, and proceeded to start cranking on things from there.

There were any number of things going on, and I hopped around trying to be useful. It took a while to finally settle down on one thing in particular: why had the machines effectively dropped off the network? Why was sshd dead? What was going on there?

My whole goal was to find out WHY the machines had seemingly nuked everything when they ran out of memory during the "push" that morning. Was it some pathological kernel "OOM killer" thing? Was it something else running amok and shutting down the wrong jobs?

We *had* to know, or we couldn't be sure it wouldn't happen again. Lots of other people were hacking away, trying to figure out how to reproduce what had happened that morning, and others were simultaneously cranking away trying to reduce the memory bloat on the web servers.

But, without finding out what the hell had happened that reduced the machines to init and this "fbagent" thing, we were underneath this Sword of Damocles situation where it could drop on us at any point and take us down *yet again*.

People figured out that yes, they had run the machines out of memory, specifically with the push - the distribution of new bytecode to the web servers. Other people started taking steps to beat back some of the bloat that had been creeping in that summer, so the memory situation wouldn't be so bad. I suspect some others also dialed back the number of threads (simultaneous requests) on the smaller web servers to keep them from running quite as "hot".

I still had my assignment to root-cause the damn "nuke the world" thing. There was no way it was going to happen in that little conference room without my usual fine-tuned environment, and that's why I bailed.

It took me a couple of weeks to really make any sense of it. I mean, really, I'm not exaggerating here. The outage was August 1st, and I finally figured out the sequence required to nuke all the processes on the box on the afternoon of August 19th. I know this because there was a screenshot floating around where I talked my way through it on IRC as it shook itself out... and that made it into one of my public talks.

That's over 18 days of not knowing why, and worrying what could have happened pretty much the whole time.

In those days, the machines were running Upstart for init (pid 1) so the only things that would come back up are things set to "respawn" in inittab (remember that?), so we'd get a getty on the console and little more. That's how I was able to use the out-of-band access to jump in and go "yep, machine is up but everything else (including sshd) is down".

[ Side note: If you did that today, I assume that systemd would end up restarting most of the things on the box, and you might actually recover from it. I'm not about to try just for a post, though! ]

Why did fork fail? Easy: the box ran out of memory. But, I had to reproduce that to know for sure. How did I do that? This took much longer, and was after chasing many dead ends based on rumors about "kernel OOM killers" and stuff like that. Were we deadlocking during the OOM kill? There was some scary stuff going on where the hosts would get really squirrelly while the messages spewed into the printk ring buffer. That consumed a bunch of time right there, and was also not what actually caused it.

Finally reproducing it involved shrinking my test system's swap size from what had been multiple gigabytes down to just 64 MB. Then I also ran some "memeater" things I had coded up: they would malloc() some space and dirty the pages by writing to them so they actually got physical memory handed to them. Then they just sat and waited around. After putting enough memory pressure on the box, it finally borked.

Even then, I thought it was the task scheduler thing the company had written for its own prod environment, because, again, everyone assumed it was guilty, and that was the undercurrent. But no, it wasn't. A few minutes later, I found the smoking gun: fbagent had logged something about "starting kill of child -1" at exactly the time everything died.

Years of nerding out on Linux boxes had taught me what killing "pid -1" would actually do, and this finally explained why this fbagent thing hadn't died in the slaughter of every process on the box. It was the thing doing the murdering!

This fbagent process ran as root, ran a bunch of subprocesses, called fork(), didn't handle a -1 return code, and then later went to kill that "wayward child". Sending a signal (SIGKILL in this case) to "pid -1" on Linux sends it to everything but init and yourself. If you're root (yep) and not running in some kind of PID namespace (yep to that too), that's pretty much the whole world.

Of course, I was looking at the checked-in source code and couldn't figure out just where the hell this -1 was coming from. The call to fork() *did* check for a -1 and handled it as an error and bailed out. So how was it somehow surviving all the way down to where kill() was called?

That was another rathole, and the answer was also a thing to behold: I couldn't see it in the checked-in source code because it had been fixed. Some other engineer on a completely unrelated project had tripped over it, figured it out, and sent a fix to the team which owned that program. They had committed it, so the source code looked fine.

[ Another side note: this person who fixed a bug in some code that wasn't their actual "job" was the kind of excellent behavior that used to be lionized there - "nothing at FB is someone else's problem". That credo died a long time ago. ]

Unfortunately, production (many many machines) was running the last release which had been cut WELL before that point. It had the bug in it: run machine out of memory, make fork fail, kill the world, all die, oh the embarrassment.

I can't imagine doing that kind of multi-window parallel investigation stuff on a teeny little laptop screen with people right next to me on either side, while the whole room flop-sweats about going out of business or whatever if they don't get it fixed.

I guess a "war room" might work out if you have a bunch of stuff that has to happen to deal with a possible "crisis" and then it's just a matter of coordinating it. You don't have people doing "heads-down hack" stuff nearly as much in a case like that.

I have actually seen such a gathering work out nicely, and I'll leave that as a tale for yet another time.

Just some random administrivia: quite some time back, I put together collections of posts and released them as two different ebooks: The Bozo Loop (2012) and The Stupid Hour (2013).

They were initially put on Amazon, and also showed up in some other strange places for a while - Barnes and Noble, and briefly, right here on my own home-grown "store" that used Stripe for billing.

As of earlier this week, they are no longer on Amazon at my direction. Instead, I put them on Gumroad as .epub files. They are still DRM-free, so once you have a copy, you can shove them onto all of your devices without having something "phone home" to approve it first.

If you wanted a copy but didn't want to deal with Amazon or the Kindle ecosystem, hey, now's your chance. If this works out, I might do some other things this way, too.

Way back around 2009, Google did something stupid internally where people with names like "Nishit" were flagged as being "fake" or similar. I probably found out about this because I had worked on the accounts system at one point.

On the surface, it seemed simple enough: someone coded up a check to pick up an "four letter word" (in English...) and it matched it. Even though it's totally a name used by a lot of people, their systems told those people that they were invalid and unwelcome. Never mind they have employees that are directly affected by this.

Ten years later, nearly the same thing happened at Lyft. It happened late in 2019 as it rolled into 2020. The timing always made me think that someone particularly clueless was trying to make one final push to meet their so-called "quarterly goals" before the year ended.

These events, and others like it, have touched off a series of posts like "falsehoods that programmers assume about names", and then its own series of posts about whole other realms which are full of trouble.

I, too, have written about this. I mentioned how the Intel museum only allowed the ASCII characters A-Z as letters in your name, and anyone else was out of luck.

All along, I have been treating this as a case of "those fucking stupid programmers", and it's an easy groove to fall into, because believe me, there are quite a few really incompetent people out there doing this kind of work.

However, I've been starting to realize that this is overly simplistic and is missing a large possibility. Can you see it yet? I sure didn't for a very long time, and that time is immortalized in my older posts.

The thing that finally got me thinking about another side to this problem was when I went to get my hair done recently. The scheduling form on my salon's web site had a list of stylists with their names given. One stylist had a name that had a rather unusual consonant pair that you don't normally see in English. I wasn't really sure how to pronounce it.

Let's say her name was shown as "RJAY", even though that's not even remotely close to the real thing for both her sake and mine. I thought that was her name, and went with that, until finally I saw how she wrote it out: "R'Jay". That's when I finally understood.

The booking system had decided that an apostrophe was a bridge too far and had just dropped it, thus reducing a fairly easy-to-pronounce name to a blob of characters that isn't her damn name! That it also forced everything into capitals didn't help things.

As she worked on my hair, we got to talking about things and that in particular, and I admitted that I had worked at places that had also screwed people's names up badly. I told her about asking the person at Lyft to "if nothing else, promise you won't do it again" and only getting a blank look in response. Then, perhaps because of recent goings-on in the world, I finally saw another possibility for why it might be happening. It's not necessarily clueless programmers, much as I'd like to bag on them for being that way.

We owe it to ourselves and to those around us to admit the possibility that some of these people are doing it on purpose. They're being unmitigated assholes because they realize they can use their position to make someone else's life a little crappier.

Get it? "Assuming best intent" is probably a mistake. When there are enough people being hateful around you, that is no longer an option.

Hanlon's Razor falls down in this kind of environment. It's lazy.

I have definitely made this mistake. You need not go particularly far back in my posts to find that I wrote my "honest troubleshooting code of conduct" which incorporates exactly that. Actual life experience now says that leaning on that is the lazy way out, and that you actually have to do some damn work to figure out exactly what's going on.

It's stuff like this that makes me realize just how much I still do not know, even though some people *have* to know this, and have no choice in the matter. I'm finding it out much later, and while it bothers me that it's taken this long to even get started, it's not going to stop me from admitting my ignorance while pushing to understand more.

While I may never truly understand some of the things that are not daily lived truths for me (and are for others), I can always work towards realizing that it exists, it's valid, and it needs to be appreciated.

Oh, and finally, it's not the responsibility of folks like my stylist to explain it to me. They have enough work to do as it is.

I'll roll up some responses to reader feedback here.

...

Someone asked if they could view the old code lessons. The one I put back online last year is where I do a terrible little TCP listener, compile it, start it in the background, and then connect to it with netcat. It's awkward as hell, but it's there if you really want to see it.

There is also the six part "protofeed" demo which showed how to fetch a feed in this protobuf-based scheme I rigged up. Spoiler: it's not there any more, since nobody was using it, so it's not very useful to follow the instructions on that thing.

Ironically, that fetcher program would run afoul of all kinds of badness if it was pointed at a production site. It doesn't do conditional requests, it doesn't know about Cache-Control headers, it won't recognize that a 429 is asking for throttling, and so on. I guess that's okay for something that was a proof of concept to show how to fetch something from the network and parse it, but *actual* feed readers get all of that stuff wrong, too.

...

Another reader asked if the Linux "hrtimer" glitch from the leap second was fixed. I have to assume that it was based on the fact that most people didn't have that same problem three years later when we had the one in my "leap smearing" story. My worries were about userspace stuff.

This is an opportunity for me to share just why I went to those lengths. In short, it was because of a lack of confidence in everyone everywhere doing the right thing in terms of time handling. If everyone uses monotonic clocks for measuring durations and otherwise is okay with wall time going backwards now and then, then there's no reason to smear it out. My own personal systems have never smeared a leap second. They just ride it out and keep on going.

I couldn't assume the correctness of such implementations at the company. Worse, even if I went and deliberately injected backwards-going time step operations and proved that it would crash some code, there was no guarantee of anything coming from it. I had found myself in a place at that company where some parts of it were completely unresponsive to the problems they were causing for themselves and sometimes for other people, and was starting to tire of the "bad cop" schtick. That's where I'd show up and go "your shit is broken" and they would do nothing to work with us (the whole team) to do something about it.

I just had this feeling that if we repeated the last UTC second of June 2015, we'd end up breaking something. What's kind of amazing is that later on that year, it actually happened.

Someone misconfigured the ailing NTP appliances to *not* apply the correction factor from GPS to UTC. This ended up forcing one appliance into shipping unadjusted GPS time to roughly half of production via NTP, and the difference at the time was something like 17 seconds. (This changes, and indeed, it's no longer 17.)

Anyway, I got to working on this after hearing about it and found roughly half the fleet running 17 seconds fast. It was completely unreasonable to try to "smear off" 17 seconds to get things back to normal - that would have taken weeks. I made the decision to fix the setting and then let every broken machine individually have its clock dragged backwards the 17 seconds to where things should be.

This broke stuff. Some kind of sharding mechanism deep inside the fabric of things was using wall time to determine something or other, and when it jumped back, it fired an assertion and killed the program. This nuked the web server (or whatever else was using that library).

So, basically, every single machine which had been poisoned with the bad time and which was running this library was going to crash exactly once and there wasn't really anything which could be done about it. It was something like 2 in the morning by this point and I opted to let it happen.

About the only good thing about this is that the adjustments happened at different intervals depending on the ntpd poll rate, so it's not like hundreds of thousands of machines all crashed their workloads at the same time. One would pop off here, then one there, and so on... over the span of an hour or two... until it was all done.

Thus, some services didn't really go down, but they did have a bad time with a bunch of failed/dropped requests which were on the affected systems.

That one was dubbed the "Back to the Future" SEV. At least one team made a screenshot of some display showing the 17 second offset the banner of the group where they talked about production issues.

Stuff like that is why I smeared it out. When you can't be sure of the correctness of implementations, and there are good chances that attempts to fix them will be ignored, rebuffed, or actively attacked, you have to "treat the whole situation as damage" and route around it. You remove the discontinuity in wall time to save them from themselves.

...

A reader asked for my take about "AI" and LLMs and all of this.

In the vein of the "annoyances" post from earlier in the month, I'll start by saying that I don't push any of that on you here, either. All of this stuff is straight off my keyboard with a sprinkling of ispell applied after the fact. Even that's of limited utility since there are a bunch of technical terms and not-really-words that I use for various reasons.

I think all of the hype and waste has generated an enormous mountain of useless nonsense that has attracted the absolute worst of the vampires and buzzards and bottom-feeders who are looking to exploit this stuff for their own benefit.

The LAST thing we needed was a better way to generate plausible-looking horse shit for random gullible people to consume unwittingly, but here we are, and it's only going to get worse.

I think a lot of this falls into "the Internet we deserve".

So no, I don't use anything of the sort, and I tell people not to quote any of that crap at me, or to send me screenshots of it pretending to be an answer to something, and that they need to find actual sources for their data. This has not made me the most popular person.

But hey, I've already said that I'm obviously out of touch with what most people are up to. My green-on-black terminal with nano in it that's writing up a bunch of plain text with a handful of triggers for callouts to other posts should be proof positive of that already. Hardly anyone else does things this way any more. That makes me the weirdo, not them. I know this. I'm okay with this.

Right, so, one of the things that can happen when you're trying to collect fresh data on the behaviors of something dynamic is that you get bogged down under the load of what happened previously. With the feed reader score project, this is what's been happening. A lot of clients were started up and pointed at it, and we gathered a lot of behavioral data.

The problem is that some of them are not changing, and having a few dozen of them call back every five minutes is not doing anyone any favors. So, I did what I promised I would do, and I updated the hostname.

If you are participating in the test and want to continue, go back to the original mail from me you got with the code(s), and load the instruction page. There, you will find the very slightly changed base hostname that can be used to construct your new unique feed URL. The keys are the same.

This also gives us the benefit of seeing what a fresh start looks like with the latest batch of feed reader software. Many of them have done a lot of excellent work these past six months, and they deserve to leave the historical baggage behind. I want to see where they are now, and this is how we get there.

For anyone wondering, I was looking at some of the reports before I cleared things out a few minutes ago. Some of the problem spots that I had mentioned in multiple report posts were still there. A lot of this was just people running old versions when they really need to upgrade. Some of it was just nobody at the wheel for the various clown services.

Seeing a whole bunch of unchanged behavior just reinforced the need to do a fresh start on this stuff. The people who have invested in improving their software deserve it.

Once there's a fresh set of data built up, I guess I'll write up another summary.

Oh, side note for anyone keeping track: this is not the "wildcard DNS" thing that I mentioned a few weeks back that would be needed to track down the *really* goofy stuff that polls all kinds of extra crap paths. That would require (more) actual work, and I'm not ready to do that just yet. (Plus, for that kind of effort, I might want to charge for it. Just saying.)

Every six months or so, this neat group called the International Earth Rotation Service issues a directive on whether there will be a leap second inserted at the end of that six month period. You usually find out at the beginning of January or the beginning of July, and thus would have a leap second event at the end of June or December, respectively.

Ten years ago, in January 2015, they announced a leap second would be added at the end of June 2015. The last one had been three years prior, and when it happened, it screwed things up pretty bad for the cat picture factory. They hit kernel problems, userspace problems, and worse.

This time, I was working there, and decided there would not be a repeat. The entire company's time infrastructure would be adjusted so it would simply slow down for about 20 hours before the event, and so it would become a whole second "slow" relative to the rest of the world. Then at midnight UTC, the rest of the world would go 58, 59, 60, 0, and we'd go 57, 58, 59, 0, and then we'd be in lock-step again.

So how do you do something like this? Well, you have to get yourself into a position where you can add a "lie" to the time standard. This company had a handful of these devices which had a satellite receiver for GPS on one side and an Ethernet port for NTP on the other with a decent little clock on the inside. I just had to get between those and everyone else so they would receive my adjusted time scale for the duration, then we could switch back when things were ready.

This is the whole "leap smearing" thing that you might have heard of if you run in "time nut" circles. Someone else came up with it and they had only published their formula for computing the lie over a spread of time. The rest of it was "left as an exercise for the reader", so to speak.

Work like this benefits from being highly visible, so I bought a pair of broadcast-studio style clocks which spoke NTP over Ethernet and installed them on my desk. One of them was pointed at the usual GPS->NTP infrastructure, and the other was pointed at the ntp servers running my hacked-up code which could have "lies" injected.

I'd start up a test and watch them drift apart. At first, you can't even tell, but after a couple of hours, you get to where one subtly updates just a bit before the other one. You can even see it in pictures: parts of one light up before the other.

Then at the end of the ramp, they're a full second apart, but they're still updating at the same time. It's just that one goes from 39 to 40 when the other goes from 40 to 41.

Back and forth I went with my test clocks, test systems, and a handful of guinea pig boxes that volunteered to subscribe to the hacked-up time standard during these tests. We had to find a rate-of-change that would be accepted by the ntp daemons all over the fleet. There's only so much of a change you can introduce to the rate of change itself, and that meant a lot of careful experimentation to find out just what would work.

We ended up with something like 20 hours to smear off a single second.

The end of June approached, and it was time to do a full-scale test. I wanted to be sure that we could survive being a second out of whack without having the confounding factor of the whole rest of the world simultaneously dealing with their own leap second stuff. We needed to know if we'd be okay, and the only way to know was to smear it off, hold a bit to see if anything happened, then *smear it back on*.

This is probably the first anyone outside the company's heard of it, but about a week before, I smeared off a whole second and left the ENTIRE company's infra (laptops and all) running a second slow relative to the rest of the world. Then we stayed there for a couple of hours if I remember correctly, and then went forward again and caught back up.

A week later, we did it for real and it just worked.

So, yes, in June 2015, I slowed down the whole company by a second.

Of course, here it is ten years later, and the guy in charge just sent it back fifty years. Way to upstage me, dude.

I've been thinking about things that annoy me about other web pages. Safari recently gained the ability to "hide distracting items" and I've been having great fun telling various idiot web "designers" to stuff it. Reclaiming a simple experience free of wibbly wobbly stuff has been great.

In doing this, I figured maybe I should tell people about the things I don't do here, so they realize how much they are "missing out" on.

I don't force people to have Javascript to read my stuff. The simplest text-based web browser going back about as far as you can imagine should be able to render the content of the pages without any trouble. This is because there's no JS at all in these posts.

I don't force you to use SSL/TLS to connect here. Use it if you want, but if you can't, hey, that's fine, too.

The last two items mean you could probably read posts via telnet as long as you were okay with skipping over all of the HTML <tag> <crap>. You might notice that the text usually word-wraps around 72, so it's not that much of a stretch.

I don't track "engagement" by running scripts in the post's pages that report back on how long someone's looked at it... because, again, no JS.

I don't set cookies. I also don't send unique values for things like Last-Modified or ETag which also could be used to identify individuals. You can compare the values you get with others and confirm they are the same.

I don't use visitor IP addresses outside of a context of filtering abuse.

I don't do popups anywhere. You won't see something that interrupts your reading to ask you to "subscribe" and to give up your e-mail address.

I don't do animations outside of one place. Exactly one post has something in it which does some graphical crap that changes by itself. It's way back in July 2011, and it's in a story ABOUT animating a display to show the absence of a value. It doesn't try to grab your attention or mislead you, and it's not selling anything.

I don't use autoplaying video or audio. There are a couple of posts where you can click on your browser's standard controls to start playback of a bit of audio that's related to the post. Those are also not used to grab your attention, mislead you, or sell something.

I don't try to "grab you" when you back out of a page to say "before you go, check out this other thing". The same applies to closing the window or tab: you won't get this "are you sure?" crap. If you want out, you get out *the first time*.

I don't pretend that posts are evergreen by hiding their dates. Everything has a clear date both in the header of the page and built into the URL. If it's out of date, it'll be pretty obvious.

I don't put crap in the pages which "follows you" down the page as you scroll. You want to see my header again? Cool, you can scroll back up to it if it's a particularly long post. I don't keep a "dick bar" that sticks to the top of the page to remind you which site you're on. Your browser is already doing that for you.

There are no floating buttons saying things like "contact me" or "pay me" or "check out this service I totally didn't just write this post to hawk on the red or orange sites". I don't put diagonal banner things across the corners. I don't blur it out and force you to click on something to keep going. TL;DR I don't cover up the content, period.

I don't mess with the scrolling of the page in your browser. You won't get some half-assed attempt at "smoothing" from anything I've done. You won't get yanked back up to the top just because you switched tabs and came back later.

I don't do some half-assed horizontal "progress bar" as you scroll down the page. Your browser probably /already/ has one of those if it's graphical. It's called the scroll bar. (See also: no animations.)

I don't litter the page with icons that claim to be for "sharing" or "liking" a post but which frequently are used to phone home to the mothership for a given service to report that someone (i.e., YOU) has looked at a particular page somewhere. The one icon you will find on all posts links to the "how-to" page for subscribing to my Atom feed, and that comes from here and phones home to nobody.

I don't use "invisible icons" or other tracker crap. You won't find evil 1x1s or things of that nature. Nobody's being pinged when you load one of these posts.

I don't load the page in parts as you scroll it. It loads once and then you have it. If you get disconnected after that point, you can still read the whole thing. There's nothing more to be done.

I don't add images without ALTs and/or accompanying text in the post which aims to describe what's going on for the sake of those who can't get at the image for whatever reason (and there are a great many). (Full disclosure: I wasn't always so good at writing the descriptions, and old posts that haven't been fixed yet are hit or miss.)

I don't do nefarious things to "outgoing links" to report back on which ones have been clicked on by visitors. A link to example.com is just <a href="http://example.com/">blah blah blah</a> with no funny stuff added. There are no ?tracking_args added or other such nonsense, and I strip them off if I find them on something I want to use here. If you click on a link, that's between you and your browser, and I'm none the wiser. I really don't want to know, anyway. I also don't mess with whether it opens in a tab or new window or whatever else.

I don't redirect you through other sites and/or domains in order to build some kind of "tracking" "dossier" on you. If you ask for /w/2024/12/17/packets/, you get that handed to you directly. (And if you leave off the trailing slash, you get a hint pointing back to that, because, well, it's a directory, and you really want the index page for it. [This changed in late 2025 for Reasons.])

I don't put godawful vacuous and misleading clickbait "you may be interested in..." boxes of the worst kind of crap on the Internet at the bottom of my posts, or anywhere else for that matter.

My pages actually have a bottom, and it stays put. If you hit [END] or scroll to the bottom, you see my footer and that's it. It won't try to jam more crap in there to "keep you engaged". That's it. If you want more stuff to read, that's entirely up to you, and you can click around to do exactly that.

I don't make any money just because someone lands on one of my posts. You won't find ads being injected by random terrible companies. In fact, keeping this stuff up and available costs me a chunk every month (and always has). I sell the occasional book and get the occasional "buy me a cup of tea or lunch" type of thing, and I do appreciate those. (I tried doing paid watch-me-code "lessons" years ago, but it really didn't go anywhere, and it's long gone now.)

I'm pretty sure everything that loads as part of one of my posts is entirely sourced from the same origin - i.e., http[s]://rachelbythebay.com/ something or other. The handful of images (like the feed icon or the bridge pic), sounds, the CSS, and other things "inlined" in a post are not coming from anywhere else. You aren't "leaving tracks" with some kind of "trust me I'm a dolphin" style third-party "CDN" service. You connect to me, ask for stuff, and I provide it. Easy.

I say "pretty sure" on the last one because there are almost 1500 posts now, and while my page generation stuff doesn't even allow for an IMG SRC that comes from another origin, there are some "raw" bits of HTML in a few old weird posts that break the usual pattern. I don't think I've ever done an IMG or SOURCE or LINK from off-site in a raw block, though.

I don't even WANT stuff coming from off-site, since it tends to break. I find that I can really only rely on myself to keep URLs working over time.

Phew! That's all I can think of for the moment.

I don't think people really appreciate what kind of mayhem some of their software gets up to. I got a bit of feedback the other night from someone who's been confounded by the site becoming unreachable. Based on running traceroutes, this person thinks that maybe it's carrier A or carrier B, or maybe even my own colocation host.

I would have responded to this person directly, but they didn't leave any contact info, so all I can do is write a post and hope it reaches them and others in the same situation.

It's not any of the carriers and it's not Hurricane Electric. It's my end, and it's not an accident. Hosts that get auto-filtered are usually running some kind of feed reader that flies in the face of best practices, and then annoys the web server, receives 429s, and then ignores those and keeps on going.

The web server does its own thing. I'm not even in the loop. I can be asleep and otherwise entirely offline and it'll just chug along without me.

A typical timeline goes like this:

• 00:04:51 GET /w/atom.xml, unconditional.
  Fulfilled with 200, 502 KB.
  
• 00:24:51 GET /w/atom.xml, unconditional.
  Rejected with 429.
  Advised (via Retry-After header) to come back in one day since they are unwilling or unable to do conditional requests.
  
• 00:44:51 GET /w/atom.xml, unconditional.
  Same 429 + Retry-After.
  
• 01:04:51 GET /w/atom.xml, unconditional.
  Just like last time.
  
• 01:24:51 GET /w/atom.xml, unconditional.
  Same thing, again.

Somewhere around here, the web server decided that it wasn't being listened to, and so it decided it was going to stop listening, too.

Some time after this, it will "forgive" and then things will work again, but of course, if there's still a bad feed reader running out there, it will eventually start this process all over again.

A 20 minute retry rate with unconditional requests is wasteful. That's three requests per hour, so 72 requests per day. That'd be about 36 MB of traffic that's completely useless because it would be the same feed contents over and over and over.

Multiply that by a bunch of people because it's a popular feed, and that should explain why I've been tilting at this windmill for a while now.

If you're running a feed reader and want to know what its behavior looks like, the "feed reader score" project thing I set up earlier this year is still running, and is just humming along, logging data as always.

You just point your reader at a special personalized URL, and you will receive a feed with zero nutritional content but many of your reader's behaviors (*) will be analyzed and made available in a report page.

It's easy... and I'm not even charging for it. (Maybe I should?)

...

(*) I say _many_ of the behaviors since a bunch of these things have proven that my approach of just handing people a bunch of uniquely-keyed paths on the same host is not nearly enough. Some of these feed readers just go and make up their own paths and that's garbage, but it also means my dumb little CGI program at /one/particular/path doesn't see it. It also means that when they drill / or /favicon.ico or whatever, it doesn't see it. I can't possibly predict all of their clownery, and need a much bigger hammer.

There's clearly a Second System waiting to be written here.

As usual, the requirements become known after you start doing the thing.

Hey there. Are you one of these "Fediverse" enthusiasts? Are you hard core enough to run an instance of some of this stuff? Do you run Pleroma? Is it version 2.7.0? If so, you probably should do something about that, like upgrading to 2.7.1 or something.

Based on my own investigations into really bad behavior in my web server logs, there's something that got into 2.7.0 that causes dumb things to happen. It goes like this: first, it shows up and does a HEAD. Then it comes back and does a GET, but it sends complete nonsense in the headers. Apache hates it, and it gets a 400.

What do I mean by nonsense? I mean sending things like "etag" *in the request*. Guess what, that's a server-side header. Or, sending "content-type" "and "content-length" *in the request*. Again, those are server-side headers unless you're sending a body, and why the hell would you do that on a GET?

I mean, seriously, I had real problems trying to understand this behavior. Who sends that kind of stuff in a request, right? And why?

This is the kind of stuff I was seeing on the inbound side:

raw_header {
  name: "user-agent"
  value: "Pleroma 2.7.0-1-g7a73c34d; < guilty party removed >"
}
raw_header {
  name: "date"
  value: "Thu, 05 Dec 2024 23:52:38 GMT"
}
raw_header {
  name: "server"
  value: "Apache"
}
raw_header {
  name: "last-modified"
  value: "Tue, 30 Apr 2024 04:03:30 GMT"
}
raw_header {
  name: "etag"
  value: "\"26f7-6174873ecba70\""
}
raw_header {
  name: "accept-ranges"
  value: "bytes"
}
raw_header {
  name: "content-length"
  value: "9975"
}
raw_header {
  name: "content-type"
  value: "text/html"
}
raw_header {
  name: "Host"
  value: "rachelbythebay.com"
}

Sending date and server? What what what?

Last night, I finally got irked enough to go digging around in their git repo, and I think I found a smoking gun. I don't know Elixir *at all*, so this is probably wrong on multiple levels, but something goofy seems to have changed with a commit in July, resulting in this:

  def rich_media_get(url) do
    headers = [{"user-agent", Pleroma.Application.user_agent() <> "; Bot"}]

    with {_, {:ok, %Tesla.Env{status: 200, headers: headers}}} <-
           {:head, Pleroma.HTTP.head(url, headers, http_options())},
         {_, :ok} <- {:content_type, check_content_type(headers)},
         {_, :ok} <- {:content_length, check_content_length(headers)},
         {_, {:ok, %Tesla.Env{status: 200, body: body}}} <-
           {:get, Pleroma.HTTP.get(url, headers, http_options())} do
      {:ok, body}

Now, based on my addled sense of comprehension for this stuff, this is just a guess, but it sure looks like it's populating "headers" with a user-agent, then fires that off as a HEAD. Then it takes the *incoming* headers, adds them to that, then turns the whole mess around and sends it as a GET.

Assuming I'm right, that would explain the really bizarre behavior.

There was another commit about a month later and the code changed quite a bit, including a telling change to NOT send "headers" back out the door on the second request:

  defp head_first(url) do
    with {_, {:ok, %Tesla.Env{status: 200, headers: headers}}} <-
           {:head, Pleroma.HTTP.head(url, req_headers(), http_options())},
         {_, :ok} <- {:content_type, check_content_type(headers)},
         {_, :ok} <- {:content_length, check_content_length(headers)},
         {_, {:ok, %Tesla.Env{status: 200, body: body}}} <-
           {:get, Pleroma.HTTP.get(url, req_headers(), http_options())} do
      {:ok, body}
    end
  end

Now both requests call a function (req_headers) which itself just supplies the user-agent as seen before.

What's frustrating is that the commit for this doesn't explain that it's fixing an inability to fetch previews of links or anything of the sort, and so the changelog for 2.7.1 doesn't say it either. This means users of the thing would have no idea if they should upgrade past 2.7.0.

Well, I'm changing that. This is your notification to upgrade past that. Please stop regurgitating headers at me. I know my servers are named after birds, but they really don't want to be fed that way.

...

One small side note for the devs: having version numbers and even git commit hashes made it possible to bracket this thing. Without those in the user-agent, I would have been stuck trying to figure it out based on the dates the behavior began, and that's never fun. The pipeline from "git commit" to actual users causing mayhem can be rather long.

So, whoever did that, thanks for that.

One of the more interesting things you can do with systemd is to use the "socket activation" feature: systemd itself opens a socket of some sort for listening, and then it hands it over to your program, inetd-style. And yes, I know by saying "inetd-style" that it's not even close to being a new thing. Obviously. This is about what else you can do with it.

Like in my previous tale about systemd stuff, you can add "deny" and "allow" rules which bring another dimension of filtering to whatever you're doing. That applies for the .socket files which are part of this socket activation thing. It can even forcibly bind it to a specific interface, i.e.:

[Socket]
ListenStream=443
IPAddressDeny=any
IPAddressAllow=192.0.2.0/24
BindToDevice=wg0

That gives you a socket which listens to TCP port 443 and which will do some bpf shenanigans to drop traffic unless the other end is in that specific /24. Then it also locks it down so it's not listening to the entire world, but instead is bound to this wg0 interface (which in this case means WireGuard).

This plus the usual ip[6]tables rules will keep things pretty narrowly defined, and that's just the way I like it.

I did this in a big way over the past year, and then never rebooted the box in question after installing such magic. Then earlier this week, I migrated that system's "personality" to new hardware and that meant boots and reboots here and there, and wasn't it weird how it was spending almost two minutes to reboot every time? What the hell, right?

Digging into the systemd journal turned up that some of the "wg" stuff wasn't coming up, and it sure looked like a dependency cycle. A depends on B, which depends on C, which depends on D, which depends on A again? If not for the thing eventually timing out, it wouldn't have EVER booted.

I'm thankful for that timeout, since the rest of the box came up and I was able to get into that little headless monster to work on the problem.

The problem is basically this: if you have a .socket rigged up in the systemd world, you by default pick up a couple of dependencies in terms of sequencing/ordering at boot time, and one of them is "sockets.target". Your foo.socket essentially has a "Before=sockets.target", which means that sockets.target won't succeed until you're up and running.

But, what if your foo.socket has a BindToDevice that's pointing at WireGuard? You now have a dependency on that wg0 thing coming up, and, well, at least on Debian, that gets interesting, because it ("wg-quick@wg0" or similar) wants basic.target to be done, and basic.target in turn wants sockets.target to happen first.

foo.socket waits on wg waits on basic waits on sockets waits on foo.socket. There's the cycle.

Getting out of this mess means breaking the cycle, and the way you do that is to remove the default dependencies from your .socket file, like this:

[Unit]
DefaultDependencies=no

After that, it's on you to set up the appropriate WantedBy, Wants, Before or After declarations on your .socket to make sure it's attached to the call graph somewhere.

I should mention that it took a LOT of rebooting, journal analysis, cursing, and generally complaining about things before I got to this point. If you're in a mess like this, "systemd-analyze dump <whatever>" is generally your friend, because it will point out the *implicit* dependencies which are just as important but which won't show up in your .socket or .service files. Then you get to sketch it out on paper, curse some more, and adjust things to not loop any more.

There doesn't seem to be a good way to spot this kind of problem before you step in it during a boot. It's certainly not the sort of thing which would stop you before you aimed a cannon directly at your foot. Apparently, "systemd-analyze verify <whatever>" will at least warn you that you have a cycle, but figuring out how you got there and what to do about it is entirely up to you. Also, if you don't remember to run that verify step, then obviously it's not going to help you. I only learned about it just now while writing up this post - far too late for the problem I was having.

I sure like the features, but the complexity can be a real challenge.

What in the name of clowntown is going on here?

ip - - [04/Dec/2024:23:18:21 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:22 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:22 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:23 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:23 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:23 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:23 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:24 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:24 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:24 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:24 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:25 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:25 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:25 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:25 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:26 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:26 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:26 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:26 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:26 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:27 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:27 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:27 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:27 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:28 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:28 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:28 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:28 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:29 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:29 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:29 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:29 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:30 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:30 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:30 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:30 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:30 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:31 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:31 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:31 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:31 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:32 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:32 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:32 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:32 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"
ip - - [04/Dec/2024:23:18:32 -0800] "GET /w/ HTTP/1.1" 200 229674 "-" "NetNewsWire (RSS Reader; https://netnewswire.com/)"

Note: the post index page isn't the feed. It has never been the feed. Even still, what's up with the unconditional requests, and several per second? What is that supposed to accomplish? In what world does that make sense?

Do you ever wonder if feed reader authors point their stuff at their own servers? You'd think they'd notice this kind of thing.

Oh, also, this will no longer work.

Yeah, I'm still here. I'm still processing things from last month. Obviously.

...

I just spent a few CPU hours on cracking something stupid and figured I'd share the results so nobody else goes through the same thing. Save some time and power, you know the thinking.

tukC0rEjV4g3g --> lvl7dbg

... and if this means anything to you, then you'll also be glad to know that the default value for "PermitRootLogin" of "prohibit-password" should be in place so nobody can get into your device that way. Test it yourself to be sure if you're worried. I sure did.

I know that if you go back far enough in these posts of mine, you will find some real crap in there. Sometimes that's because I had a position on something that turned out to not be very useful, or in some cases, actively harmful. This sucks, but that's life: you encounter new information and you are given the opportunity to change your mind, and then sometimes you actually do exactly that.

Recently, I realized that my position on something else has changed over time. It started when someone reached out to me a few weeks ago because they wanted me to get involved with them on some "employee metrics" product. It's some bullshit thing that has stuff like "work output" listing how many commits they've done, or comments, or whatever else. I guess they wanted me to shill for something of theirs, because from my posts, clearly I was such a fan of making that kind of tool, right?

I mean, sure, way back in 2004-2006, I was making all kinds of tools to show who was actually doing work and who was just sitting there doing nothing. I've written about a number of those tools, with their goofy names and the "hard truths" they would expose, showing who's a "slacker" and all of this.

When this company reached out, I did some introspection and decided that what I had done previously was the wrong thing to do, and I should not recommend it any more.

Why? It's surprisingly simple. It's the job of a manager to know what their reports are up to, and whether they're doing a good job of it, and are generally effective. If they can't do that, then they themselves are ineffective, and *that* is the sort of thing that is the responsibility of THEIR manager, and so on up the line. They shouldn't need me (or anyone else) to tell them about what's going on with their damn direct reports!

In theory, at least, that's how it's supposed to work. That's their job: actually managing people!

So, my new position on that sort of thing is: fuck them. Don't help them. Don't write tools like that, don't run tests to see if your teammates will take care of basic "service hygiene" issues, and definitely don't say anything substantive in a performance review. None of it will "move the needle" in the way you think it will, and it will only make life worse for you overall. "Peer reviews actually improve things" is about the biggest crock of shit that people in tech still believe in.

Once again, if management is too stupid to notice what's going on, they deserve every single thing that will happen when it finally "hits the fan".

Make them do their own damn jobs. You have enough stuff to do as it is.

...

I feel like giving this a second spin right here in case I failed to reach some of the audience with the first approach. Here's a purely selfish way of looking at things, for those who are so inclined.

Those tools I wrote 20 years ago didn't really indicate who was slacking at working tickets or whatever. What they *actually indicated* was that the management at Rackspace, by and large, had no clue what was going on right under their noses. And, hey, while that was true, that can be a dangerous thing to say! You want enemies? That's a great way to get them.

So, why expose yourself? Suppress the urge to point out who's slacking. It will only come back on you.

I know I have a weird and kind of skewed view of the world, since I expect nobody else to care about some things, and then the world goes and surprises me by jumping on it anyway. About six months ago, I expected the worst for my whole "let's improve the feed reader ecosystem" thoughts, and figured it would go unnoticed, and yet it's been anything but.

Clearly, I am miscalibrated for what the world is willing to do. So, with that in mind, I figured that I need to bring more things up in this context to see what the reaction is instead of just discarding it early due to my own doubts.

Tonight's concern goes like this: I was flipping through some old posts and ran across something from 2013 where I said "your simple (shell) script is someone else's bad day". It was inspired by the "onboarding" process for dev servers at FB which I was going through that week, and it was a total shit-show. It had plenty of the aforementioned scripts that you couldn't restart without making things worse. It took way too long for me to get my hands on a Linux box that had things set up so I could crank on stupid www stuff.

In re-reading that post the other night, it got me wondering what I could do in this space to try to raise awareness of various topics. Maybe I'd look at a few old posts, decide that there's a cluster of problems ("idempotence" in this case), and that maybe it would be a net win for humanity if more people were slightly better at it. Then I'd try to address that directly... somehow.

My goal would be to get more people in this life who, at least for that topic, look at everything they encounter and go "what if this fails, and is restarted". I basically would love to see more people going "what if" for that particular topic. Worry about the robot and the five quarts of oil. (Go look at the 2013 post if that made no sense.)

I don't know if that's plausible. I actually worry that it's not even possible for most people, and that caring about such things is some weird genetic corner case that is born, not made, and it would be unreasonable to expect it from the rest of the world.

This is where I need input. I have this feeling that such topics are not seen as important by reasonable people, and that they honestly never will be. Just because it matters to me is no indication that it will resonate anywhere else. That much has been made crystal clear over the past few years.

Wouldn't it be amazing, though? Just imagine it: groups of people heading out into the world armed with additional things to consider when designing something, and the resulting improvements in reliability for the things they work on down the road.

Then we'd have to continue on to some other fundamental topic, and try to move the needle on that, too.

Finally, I should mention another possible outcome: those same people go into the world with that knowledge, go to apply it, and run into the same brick walls that some of us have already been hitting because nobody else sees the point. They get all bitter and dejected, emit their own "delightful burnout sauce" (I love that line, btw), and start writing their own screeds on things like this, and we start over.

You can bake an amazing cake, but if they aren't hungry for cake, well, you might have just wasted a bunch of time.

Oh no, look out, it's another list of feed reader behaviors. This is based on my observations for those test keys which have been active in the past few days. Anything that stops polling eventually ages out of the report and won't show up here.

My thanks to those feed authors and contributors who have been cranking away to tweak things here and there. It's making a clear difference in the real world. I see more 304s and fewer 429s in general, and that makes me very happy.

Here is what I have to say for this time around, with some amount of aggregation applied. Some entries cover multiple test keys if the behavior is the same.

...

No UA. Sends bogus "" INM at startup, then makes up IMS values subsequently. Broken caching behavior that will miss updates in the real world.

Miniflux dev stuff. Seems a whole lot better now. Need to do some more pathological injections to see how they handle it.

Miniflux, older versions. Needs to upgrade. (Several different test keys).

SpaceCowboys Android RSS Reader 2.6.{31,32,33}. It's still doing unconditionals seemingly at random, and sometimes has bad timing. No idea what the hell is going on here.

FreshRSS dev stuff. Seems decent of late. This one also seems to be honoring Cache-Control "max-age" stuff, which is a bonus I never expected anyone to do. That's awesome. The useless referrers are also gone, so hooray! Lots of good stuff happening here.

FreshRSS/1.24.3. Didn't get tripped up on my last round of pathological header values. Will probably get even better when it picks up the latest dev work, like dumping the referrers (see earlier dev entry).

Audrey2 RSS reader, various versions. Seems pretty chill. Also seems to be honoring Cache-Control max-age! Great stuff.

Emacs Elfeed 3.4.1. No idea what controls the polling interval but it's doing fine. Some early weirdness that might be multiple instances running in parallel but nothing recently.

Slackbot. Broken beyond belief. Must have been some intern's project that's been left online to irritate people. Sends HEADs, gets 405s, sends more. Probably will force me to write "let's see what happens when you start getting 403s and 404s and 410s" and whatnot, and then I'll complain about it not honoring those, too. Watch and see.

Newsboat, various versions. Probably okay now in terms of caching behavior but I need to shove some more pathological cases at it to be sure.

feedmail.org/0. This one looks like it's trying desperately to honor the max-age thing, but sometimes it arrives a hair too early - on the order of about 100 msec here. So weird. How damn often is it considering these polls for it to be able to hit it that close? Does it ever rest?

SpaceCowboys RSS Reader, 2.6.{29,30,31,32,33}. This one doesn't seem to be doing the unconditionals or short timing any more. It did some early on but then it stopped the silliness.

Unread RSS Reader. Plugging away at a 21 minute interval, up from 15. At least they're conditional. (At least two distinct test keys.)

Some browser UA. Godawful timing, always unconditional. This would trip automated blocks on the real site, no doubt.

cry-reader v0.0. No idea how it picks its intervals, but they're always far enough apart and are always conditional, so that's great. Haven't managed to trip it up with any pathological header values.

Netvibes. Same super broken caching behavior, still.

SpaceCowboys ... etc ... 2.6.31. Never does IMS, only occasionally does INM, wildly unpredictable polling intervals. Not good behavior. No idea how there can be so much variance between these things (see above).

NextCloud-News dev. The caching behavior is getting better, and -huzzah- it has a real version number at last! This means it will be possible to treat all of those old versions as damaged and route around them. Would be nice if it didn't keep pulling / 3x and favicon 1x for every. single. feed. request. Those are wholly unnecessary.

NextCloud-News/1.0. This version is heavily broken in the caching department, but help is on the way (see dev version above). If you're running this, you should plan to upgrade to pick up those essential fixes.

NewsBlur. Usually okay but sometimes it does crazy stuff like 3 *unconditional* polls in two seconds. What? Why?! (UA is also crazy long since it's waving a dead chicken in the form of including a complete forgery of a Safari UA, (nested parens) and all).

Friendica, various versions. 100% unconditional requests. Occasionally shows up < 10s after the previous poll. Not good.

TTRSS. Not recommended.

ureq, various versions. Seems okay. Occasional short-spacing between polls, but at least they're conditional.

NetNewsWire. Multiple distinct test keys, but none of them have a version number so there's no way to know if anything's changed upstream. Since I started writing this list, one instance came back fast enough to demonstrate that it still has the buggy caching behavior.

haven, 100% unconditional, bumpy timing, unchanged from before.

inforss, multiple versions. Seems fine.

curl, various versions. It does conditionals properly and just has the usual 59m/60m fencepost scheduling thing.

Inoreader/1.0. This thing periodically flips to unconditional mode, and also has the 59m/60m thing going on. On the prod side of the house, it gets itself auto-blocked fairly often for bad behavior.

Feedbin. Seems fine now. (Three distinct test keys, same behavior.)

NetNewsWire. Probably still buggy - was about three weeks ago, at least. Need to see what happens when I do a fresh round of tests.

Liferea/1.15.3. Something is very wrong with the caching on this thing. It drops to unconditionals a lot, and recently started sending some bullshit 1970 IMS header with things like 5s and 14s intervals. No idea what the hell is going on there.

rawdog/2.24rc1. 138 days with no complaints from me.

NetNewsWire. Just like the last one: was showing signs of buggy cache behavior before, and probably will again when I next launch into the tests.

NewsBlur. Not quite as animated as the earlier instance but still uncorks an unconditional from time to time for some reason.

rss2email, various versions. Occasional unconditionals for some reason.

Newsboat/2.36.0. Jumpy timing, sometimes too fast. At least they're conditional.

CommaFeed, various versions. Has had caching bugginess in the past, and did during the last test. Probably still does.

Some browser UA. Keeps banging away with <1s unconditionals. Would definitely get auto-blocked in prod. Maybe multiple instances are running, but why would one always send unconditionals?

theoldreader.com. This seems to have switched behaviors to conditional polls in early September. I see a pair of too-quick polls since then, but even those were conditional so it's not that bad. This is also way better than before. Thank you for the fix, devs!

Generic "Go-http-client/1.1" UA. Seems fairly chill.

Rapids. This one turned a corner with some upgrades back in July and has been delightful since.

er0k feeds. Was upgraded at some point and then didn't trip over any of my "inject some crazy" stuff over the past month. Smooth sailing.

Emacs Elfeed 3.4.1. Mildly wacky behavior that suggests multiple instances banging away at the same key - either that, or it has really scary caching bugs. I hope it's the former and not the latter.

walrss/0.3.7. 145 days of complaint-free behavior. Nice.

Yarr/1.0. Just needs to tweak the 59m/60m fenceposting thing.

Some browser UA. This one settled down into a good polling interval and does conditional requests properly. Good enough.

Feedly. Still doing the unconditional requests daily for some reason. Fail. (Multiple distinct test keys, same behavior.)

feedparser/6.0.2. Just the 59m/60m thing. Otherwise is managing to do conditional (INM) requests so it's not the end of the world.

Reeder, various versions. This one acts strangely sometimes, and sends the wrong value in an INM header, like it got hit over the head and reverted to something from a few days *or weeks* before. Also sometimes sends unconditional requests. Could be multiple instances, so don't do that. One key = one instance of one reader. Anything else is chaos. (If this is you, ask me for more keys!)

Lots of mixed up stuff, but of late it's something claiming to be a browser that bangs away with unconditionals at 10 minute intervals. Would be auto-blocked in prod in a jiffy.

Yarr/1.0. Sends conditionals but has terrible timing - *way* too fast on those polls.

Bloggulus/0.4.0. Seems to be doing fine.

newsraft/0.27. This got some tweaks recently to deal with pathological ETag/LM behavior. Thanks for that!

feedparser/6.0.10. Still running too quickly: 29m/30m. They're conditional, but ... eh.

Mojolicious (Perl). 147 days of polls with no issues to report.

Some browser UA. Seems to be fine.

com.vanniktest.rssreader, various versions. Weird timing, as before. No idea what kind of scheduling it's using - repeats within 1-2s make no sense.

Broadsheet/0.1. Late update to this post: it's doing fine, but there's an unusual bit of wackiness showing up here. The URL it's using is checked into a github repo, and something or someone (not the author, and not Broadsheet) is pulling them. Maybe this is github trying to be "helpful". Joy.

[This entry originally said it was changing UAs and doing wild unconditional requests from time to time. It is not. Something else with the key is taking it for a ride. Raar.]

Occasional odd behavior - UA changes to something else, and it fires off multiple unconditional requests within a second. No idea if this is someone running multiple programs against the same key. (If this is you, ask for another key, or maybe get two, and stop using this one.)

feedbase-fetcher.pl/0.5. This settled into a good groove.

Artykul/1.0. The clown leak. I really need to just block them, or at least, start sending them really broken stuff to see what happens. Installing their app shouldn't have subscribed their backend to my early testing URL, but it did, and uninstalling it didn't stop it!

Okay, it seems I missed the mark yesterday while telling my story about a dumb little backup scheme, because a bunch of people have reached out to ask "why not sshfs"?

So, okay, some assumptions, first. I'm assuming from some basic reading that sshfs is one of those things that lets you take a filesystem that's on a distant host and mount it (with FUSE, sigh) on a local host. If that is in fact what it does, that is very much *not* what I want.

I don't want my stuff to be visible as a filesystem on the remote host. That is, I don't want the copies of my data to be sitting there, accessible to anyone who can get on that machine. I want it to be a amorphous blob of (obviously?) encrypted data. The only place where it should make sense is on the client machine after I've supplied the key.

Speaking of keys, that key material (like a typed passphrase) never leaves the client system, since all of the crypto stuff happens locally. If this stuff was a regular filesystem on the other end, then that host would need access to the key, and then it's no longer really under my control, right?

Given all that, let me come at this again.

I have a host. I want to back it up to something that's not in the same physical location, and ideally is rather far away. Having different power companies, ISPs, you name it? That's all a good thing in this case. (Also, "different fault lines" is always a consideration here.)

I have access to that host, but assume I don't have root access to it for whatever reason. I'm just a user on there, and I can ssh in, and I have a healthy amount of disk quota. I'm also welcome to use oodles of (network) bandwidth to do whatever I need to do.

I don't, however, want the contents of my backed-up data to be visible to anyone on that host, _even if_ they have root on it, whether legitimately or otherwise. The remote host is just being a dumb block device for me.

I don't want to have any extra daemons running that would provide network access to the blob o' bits that constitutes the encrypted image. It's a whole bunch of extra complications that I don't need or want, since ssh gives me a perfectly good pipe without changing anything fundamental about how that system works.

So, the end result is that I extend nbd over a wacky ssh connection to a Unix domain socket on the source machine, do my crypto magic to make it into a viable filesystem, then fsck it, mount it, and rsync my stuff onto it.

The only part of this I had to create was the aforementioned wacky ssh connection. nbdclient itself won't do a "ssh to far end, start nbdkit server, and then stay alive copying bytes around" thing. It's all about getting the connection up and handing it off to the kernel. It only sticks around if it's doing TLS mode and since that means running a persistent server on the far end, I'm not using it that way.

That's it. It's just one of those dumb things you build to make other stuff possible. Tools to make tools.

Sometimes I do dumb things to solve my own problems. This is one of those times. In this case, I wanted something that would give me access to a block device on a physically distant machine for backup purposes. I didn't want to do anything particularly fancy on the distant box, so rootly powers are out of the question. I just need disk space, a bit of bandwidth, and some CPU time every now and then.

Here's how it works. Perhaps you have heard of "nbd" if you're in the Linux world. It lets you load a kernel module on the client machine and then it'll turn a network type connection into a block device. That is in fact right in the name: "nbd" equals "network block device". Nice, right?

I wanted this, but didn't want it making a "bare" connection to the far end. By default, it's just a plain old TCP session to the other side. While you can rig it to use TLS, there are a bunch of problems with this. It means you need to leave a daemon running on the far end, and again, if you aren't root out there, that can be troublesome. It means the distant host ends up stuck with a listening port, and that's not always nice. It also forces you to deal with getting the whole certificate thing hooked up. Hope you're good at wrangling OpenSSL.

This seemed wholly unnecessary to me. I already have ssh access to the machines in question, so that right there will give me a relatively solid transport. I just needed to convince the nbd stuff to use it.

Now, let you stop you right here: if you're saying "SOCKS proxy" and/or "port forwarding", you have missed the part where I'd rather not have a persistent daemon on the far end that's listening on some port, *even if* it's "merely bound to loopback". That's still something running as me that presents an unnecessary orifice to unwelcome visitors. No thank you.

There's another way the nbd client can work: it will totally connect to a stream-style Unix domain socket. As long as whatever is on the other end of that socket speaks the right language, it doesn't matter where it is or what it is. Thus, I needed to make a Unix domain socket reach over the network to the other end of a ssh connection.

Here's what happened: I wrote some grungy plumbing stuff (my specialty) that sets up a Unix domain socket on my local machine. It does it in a directory where only I can access it, and it waits for a connection. Once it gets one, it forks, and the child fires up a ssh to the far end to invoke the nbd server (a trivial userspace thing I can leave in my personal bin directory).

Meanwhile, the parent process of the client sticks around and fires up a pair of threads to do a bucket-brigade thing. It takes any data from the Unix domain socket connection it just received and flings it at the ssh connection. The other one does the reverse.

This keeps going until one of the fds shows up as disconnected, at which point it shuts down everything and exits politely.

The other part of this is a currently a bit of *local* rootly scripting madness which points the NBD client at that socket, then does the crypto gunk to attach it, then fscks it, mounts it, and fires up rsync. Then after it's done, it undoes everything and declares victory.

It's not meant to be fast, and it's definitely not meant to be beautiful, but it does work. My only footprint on the far end is a giant blob of a file that is entirely meaningless to anyone without a suitable key. The data is only ever seen in a usable form here on the client machine which already has access to the original data by definition. I didn't have to install anything special on the far end, I don't have to run any daemons, and I didn't need root out there.

It's the type of thing that you can stand up and just let it run every now and then, kind of like Time Machine on a Mac. You hope you never need to use it, but it's there if you really get stuck somehow.

So, yeah, when my part of CA "falls into the ocean", I will bob up to the surface, swim to shore, and then walk to where my offsites are and restore from backups. You know, that whole thing.

Just basic sysadmin stuff. Nothing fancy.

I was successfully nerd-sniped a few days ago and figured I'd share my proposed solution with everyone just in case they could benefit from it. I've added a few of my own constraints based on expectations for how things could go wrong. So, if this seems familiar, maybe it is, but I've made it a little more complicated.

The situation is basically this: there's a large space with a bunch of dumb Linux boxes which are attached to displays of some sort. Different things are displayed depending on where it is in the space. This means the Linux boxes need some sense of identity to be able to tell the server "I need this particular set of stuff". They're "hands off" - nobody wants to log in to them to manage them.

So now let me present some of the issues:

Problem: relying on IP assignments is no good, since the DHCP mechanisms for that network can't be trusted. Things are sufficiently flaky to where you can't rely on it, and part of this is from the next item.

Problem: the hardware is wonky enough to where it gets replaced sometimes, so the MAC addresses also end up changing. This means any other on-board identifier (board id, cpuid, whatever) would also change. It would be nice if someone didn't have to keep updating mappings on the server every time the hardware got shuffled around.

Problem: it's a PITA to stamp out unique images for these things. The goal is to be able to stamp out a single image that's identical for all of them, and then have them all boot from that and figure out their identities some other way. (The same goes for netbooting, if you're thinking of that already.)

Problem: any other "ID marker" type solution that involves twiddling the set of installed hardware somehow means that you have to be sure each one ends up in a specific spot. If you have relatively unskilled people running around installing these things, this might not be what you want. It's easier to just say "make sure there's one plugged in at every space on this list" without worrying about matching specific units to specific spots, in other words.

Wouldn't it be cool if you could just grab a box, plug it in a given spot, and it would somehow start displaying the right set of media? That's what I was thinking when I came up with this idea.

So, how do you do it? My "solution" (which has its own caveats) comes in the form of LLDP.

For those who haven't encountered it yet, here's the scoop: certain higher-end Ethernet switches have the potential to _multi_cast identifying frames every so often. Any given station on such a switch might get a LLDP frame from it a couple of times per minute.

In that frame, you can see interesting things, like the MAC address(es) of the switch, its name, what OS it might be running (!), the management IP addresses (!!), and, finally, what the name of YOUR PORT is.

	Port Description TLV (4), length 4: eth2

        Port Description TLV (4), length 8: Gi1/0/24
          0x0000:  4769 312f 302f 3234

Yep, that's right, if you're on such a switch, you might well have something arriving on your interface every so often that says "oh by the way, you are connected to me on my Gi1/0/24". Another device on that same switch would get nearly the same announcement, but it might say "Gi1/0/48" or whatever else.

The point is: if you're that derpy little display machine, you now have a way to tell yourself apart from your "coworkers". You could build a tuple of (switch name, switch port) and kick that at the display server, and let it figure out what set of images/videos you're supposed to run.

Now, there's always a catch, and here it is: this assumes that you're not changing the mapping between ports on the switch and the actual RJ45s in the space. I'm assuming that port 23 on your switch will always be over there by the front door, port 24 will always be next to the bathrooms, and 25 will be by the patio door. You get the idea.

If those change, then yeah, you get to reconfigure everything, but at least it's still only on the server, and you don't have to deal with the dumb little display machines.

How do you see this? Well, tcpdump would be a start. A filter of "ether proto 0x88cc" will do the job, but make sure you stick a -v on there to make it actually expand the details (or it'll be pretty dull).

Some other things: a relatively low-end switch won't do anything in this space. It won't generate it, and it won't "absorb" it from other sources. So, if your listening post running tcpdump is plugged into such a beast, it might hear LLDP stuff coming from multiple other sources but it won't see anything from the switch itself. Yep, it's more than switches which potentially send these out. "Server class" systems with their own little management device ("BMC") glued onto them tend to do this, too.

Normally, such traffic would not cross the switch, but that requires a switch that actually "gets" it and knows to not forward it. Ordinary dumb switches will treat it just like any other bit of traffic and will send it on down the line.

I should also note that other protocols exist which serve similar purposes and it depends a lot on which switch vendor you've chosen. It might not be LLDP but it might be something else which ends in "DP".

If in doubt, sniff the network and find out!

I have a few more thoughts on the whole topic of colocation. First of all, Joel wrote in with a couple of tips beyond the basic "screwdriver and flashlight" that I mentioned. He says you should bring hearing protection, a step-stool or small ladder, and a jacket if you get cold. I like this thinking, and figured I'd expand on this for the benefit of those wondering what this all means.

First up, these places are LOUD. Everything you can imagine has fans on it. Obviously there are massive air handlers in the suites, but the (proper server class) computers and switches and everything else are also rocking a ton of fans. Some of them throttle back when the CPU load isn't too high, but a fair number of these things actually have rather high CPU load and so they never throttle back.

I mean, it's 2024, and people are writing CPU-bound computational stuff in languages that are interpreted, single-threaded, and slow as shit. OF COURSE they're running their CPUs as hard as they possibly can. But I digress.

So yes, it's loud as hell, and you could benefit from some kind of active protection. Just don't do what I did one time by cranking up the music in regular earbuds to cover the noise. Yeah, you might be able to hear your tunes, but you won't be hearing much else afterward. Stupid move, I know... now.

The step-stool or small ladder is not always a given. My particular cabinet isn't super tall and I'm able to reach all of it, but this might not apply for everyone else. Alternatively, you might need to use it to support something from below while doing an install, especially if you're flying solo.

The jacket is another one of those things that you might not appreciate until you've been on the inside. It's not like the 90s when everyone just had a giant room and did their very best to cool the entire thing down as far as possible. These days, there are "hot aisles" and "cold aisles": two rows of machines face each other across a walkway, and cold air is blown into there. Then the air goes through the machines thanks to all of those fans and ends up on the back side where it joins with hot air from yet another row that's also backed up to it. Finally, it's drawn into a chiller to start the process over again.

If it's a solid concrete floor type of setup, then the air will have to come from above, but if you're on a raised floor, it'll probably just emerge from beneath. If you're in that kind of setup, make note of what you wear before making a trip lest you turn into Marilyn Monroe.

(Yeah, I just gave some advice that only applies to a subset of the people who will ever read this and need it. Yep. That happened.)

The nature of things is that you will inevitably need to access your rack from both sides to reach certain parts of the equipment, so it will be really hot sometimes, and it'll be really cold other times. This is why I will amend that advice to "bring a jacket that zips".

Now, if you're in the Bay Area like me, you have hopefully long ago internalized the wisdom that "you will never be far from a light sweater", and so you *already* have one of those that goes everywhere with you. If so, you're set. Be ready to adjust it as appropriate.

Other stuff? Once you're past the "drop a Raspberry Pi in there" stage, you should not just jump on ebay and buy the first rack mounted server that looks like it'll work because you hate hardware and want it over with. This is because not all racks and not all servers are created equal, and you might find out the hard way that it won't work. I came super close to screwing up *hard* due to this.

I would recommend first going in with a tape measure to figure out exactly what you're working with. What are the posts like in the rack? Are they fixed in place or can they be adjusted? Are they round holes or square holes? How long can a server be while still fitting into the rack? Will the door(s) still be able to close and latch?

Bear in mind that it's not just the length of the machine, but you also have to include the loops of cables which emerge from it: power and Ethernet at the very least. You really don't want to force a regular power or network cable into a "hard 90" type of situation to make it all fit into the space. With a little research, you can get power cords which have their own built-in 90 degree turn on the end where it plugs into the machine that will let you claw back a bit of space, and likewise for the network stuff.

Or, you know, you can just buy a shorter machine and use normal cables.

Measure twice, buy once.

Another complication: if your provider gives you power by way of a "zero U" PDU (basically a big power strip that stands up vertically), that's both a plus and a minus. It's a plus in that you're not burning any rack units by definition: it's "zero U". But, it can be a problem because it still takes up space, and if you have anything really long in there, it'll probably bump into it. This constrains you to only using spots in the rack which are not blocked by the sneaky little PDU. It's just another 3-D Tetris problem for you to solve.

Late yesterday, I put up a post about how to get into colocation in about the crappiest way possible. I skipped a bunch of details just to get it out there. The inspiration was based on finding out just how many people have no idea that this business model even exists.

I used to work in a cousin of this space: managed and dedicated hosting, and here's how this all lined up:

Colocation (as mentioned yesterday) gives you some space, some power, bandwidth, a network allocation of some kind (and/or ability to route your own stuff), and hopefully some decent HVAC to keep everything cool. It's your hardware in their space, and if something breaks on that stuff, you get to fix it. What if a drive fails? That's on you. Your switch or router goes insane? Same deal: that's all you.

You might be able to get some "remote hands" service from the provider for very simple tasks: reboot something, take a picture of the (blue) screen, that kind of thing. Some also let you pay them to go and do other stuff with a higher degree of complexity. Read the fine print.

Managed hosting is where you get access to a box somewhere, and are usually given root on it. You can do about what you want to it, but beware of the "spheres of support" as we used to call it - the hosting company's support people will only go so far. You might want Debian, but they only do RHEL. You get the idea. You can probably ask them to do kernel upgrades, troubleshoot why it's being slow or seems to be down, install and configure certain things for you, and so on.

If the hardware fails, that's on them. They rip open the box and replace the part. You still have to figure out how to make your stuff work before, during, and after the event. They probably won't do any migrations for you, since everyone's setup is an odd little snowflake and only the customer has any chance of knowing how it all works.

Higher-end setups might offer hardware firewalls, load balancers, backup solutions, and more. Now, granted, it's been almost 20 years since I last did this myself - I assume someone's still doing this somewhere.

Dedicated servers are a step down from managed. You (probably) get root on a box somewhere, and you can do a lot of stuff on it. Outside of the absolute bare essentials provided by the hosting company (like a Red Hat Network entitlement), they aren't going to do much for you. They might have rigged up some magic hardware to let you power-cycle what's just a boring old whitebox without explicit out-of-band access. They might also have some netboot magic to let you shunt it into a "repair environment" if you really screw things up.

All of the servers used for hosting this site were of the "dedicated" persuasion initially: first at ServerBeach, then at SoftLayer (which became IBM). The SB machines were relatively crappy whiteboxes shoved into a bread rack somewhere, while the SL/IBM stuff was on a Real Rackmounted Server with out of band remote access and the whole bit. They handled the hardware, and I handled the software.

I mentioned some of the details last year when departing the IBM shit-show and the Texas shit-show at the same time. This ended the era of dedicated hosting for me and moved me into colocation.

These days, there's a cabinet in a nice conditioned space not too far away. I initially parked an old switch in there and a derpy little Raspberry Pi, just like yesterday's post implied. This was to give me a little "platform" while the rest of the stuff happened. A few days later, I got some "real" hardware, hung that in there, and then migrated everything to it from the SL/IBM box.

That was a bit over a year ago, and things are pretty much unchanged. This setup should just sit there and run and run and run until something breaks beyond that which can be handled in software. Then I get to drive out there and flip some parts around and make it go again.

It's not rocket science. This way of doing things has always been possible. You don't have to deal with clown providers pulling the rug out from under you. There are a few dimensions in which the provider can mess with me now:

1. They could have something terrible happen - think flooding or a fire. It could take the whole location offline for an extended period, and it could well destroy the equipment inside, such that merely relocating it to another spot would not be sufficient to revive things.

2. They could decide they're done with this kind of business model, and now they're going to boot out all of the little people who pay a pittance per month. Maybe they're going to sell out to the latest clown provider who wants to melt the ice caps and drown the polar bears with their "AI" crap. (This would also apply if the private equity vultures swoop in and try to hollow the place out before nuking it entirely.)

3. They could crank the prices up beyond that which I could afford.

4. They could do something that somehow makes the basics incompatible: they stop supplying AC power, conditioned air, or IP over Ethernet. (I know, this is ridiculous, but I might as well mention it.)

5. They could shack up with some really evil people such that I no longer want to do business with them.

My responses would be approximately this:

1. Acquire some other hardware, install a fresh OS and restore my stuff from backups, then park it somewhere else.

2, 3, 4, 5. Find a new spot, then schlep the hardware there.

...

If this is old hat to you, great! It means you're probably a grizzled 1990s sysadmin just like me, consarn it! This isn't for you, then.

This is for the newer folks who might not have realized that there's an alternative to paying tribute to one of the three churches of the Clown: M, G, or A. If you want to "get your stuff online", there are other ways... and there always have been!

I've been wanting to do this for a while: basically, to write a really snarky post about the bare minimum required to run your own hardware in a colocation environment. I'm talking about doing as little as possible, and possibly screwing up bigtime while you're at it.

Based on chats with some of my friends, it seems like this model is not well known. One of them had a situation a while back where everyone was now working from home due to COVID, and so they would all VPN back to the office park where the servers were, including their NAS "box-o-disks". It was obnoxiously slow and laggy and failed a lot, too.

I said he could probably find a spot to colocate that stuff given that he lives in a major metro area, and sure enough, he did. A few weeks later, everything was parked in there, and the resulting boost in connectivity (throughput and availability both) made everyone so much happier.

I think they were also able to dump the remaining cruft at the office park as a result. Win-win, right?

So, without further ado, here's the terrible list I just scribbled down rather quickly to get this going:

0. Scrounge up at least an old dumb Ethernet switch (or *gasp* a hub!) and some random-ass hardware that'll run Linux, like a Raspberry Pi, or some old PC box or whatever.

1. Install Linux on the box. Turn everything off but sshd. Turn off password access to sshd. If you just locked yourself out of sshd because you didn't install ssh keys first, STOP HERE. You are not ready for this.

If you somehow survived, continue as usual.

2. Find a place that you can physically access that will sell you some chunk of a cabinet, rack, shelf, or whatever else. You have to be able to get there somehow: walk, bike, drive, take the bus, whatever ... *while carrying equipment with you*. Keep that in mind.

If it's on the other side of the country (or planet), it might not be what you want, in other words.

3. Make a deal with the place. Prepare to throw some money at them.

4. Wait for them to set up your account. Get the networking details. Make sure they know you're just going to be running a stupid little switch and not a full-on router.

They'll probably carve out a tiny little block of v4 space where their router is one of the IP addresses, and you get to use the rest. They'll probably do the same with IPv6, only it'll be a /64 because why the hell not? (If they don't do IPv6, ask for your money back and run away.)

5. Configure the Linux box to match those details: static IP assignments! Yes! No DHCP here!

6. Figure out how to get your laptop to work while plugged into that same switch within your IP space, and WITHOUT any sort of wifi access while it's still at your house. You won't be able to see the outside world but you will be able to see your server. Filter out the world pre-emptively, because where you're going, you're not going to be behind a firewall.

7. Haul yourself and your stuff to the actual co-lo space: switch, Linux box, laptop, power cords, and Ethernet cables. Consider bringing a screwdriver and a flashlight (any halfway decent place will provide those for you, but you never know).

8. Plug everything in and turn it on. Connect to the server from your laptop. Verify that the server can get out to the Internet. Verify that you can get back to the server FROM the Internet. (That is, plug your switch into their network. Obviously.)

9. Go home and set up the rest from there - you know, DNS, loading stuff on the actual box, and so on.

...

To the usual broken people: knock it off. I know that you don't HAVE to use Linux, and can install a BSD, or Windoze, or whatever the hell else. I'm giving you A path, not THE path. Now shut up.

Hello again from the land of feed reader behavioral tests. I ran through the list of participants a couple of days ago and wrote up my results. This is only for those which had polled at least once in the past seven days relative to that point.

I'm going to group some of these together, but keep in mind that some behaviors are a function of however the user configures the program. Also, at this point I'm mostly focusing on their steady-state behavior, but any previously-reported goofiness at subscribe-time is still worthy of fixing for people so inclined.

Artykul's clown fetcher is still going. They're going to make me write a 404 generator and then a 410 generator, I just know it. Then I'm going to start complaining about them not honoring either of those. All unconditional, every 2-3 hours. Terrible.

feedbase-fetcher.pl/0.5 seems fine.

Broadsheet/0.1 seems fine.

com.vanniktech.rssreader, various versions has weird timing.

NextCloud-News/1.0 - all of these have broken IMS caching. There's a fix that's been submitted but it hasn't been merged, never mind rolled up into a release. Some old versions hammer the favicon.ico needlessly. I can only guess at who's running what, since they don't ever bump their version number. It's been 1.0 forever, and I assume this will continue unabated. This makes it really hard to direct particular version-based brokenness to an appropriate handler.

Some browser extension thing. Seems fine.

Mojolicious (Perl), seems fine.

NetNewsWire with buggy LM/ETag caching, as covered in a post a while back. I don't think this has been changed upstream yet. Also has terrible timing.

newsraft/0.25, which still has buggy ETag caching. It also has terrible timing.

Bloggulus/0.3.7, seems fine.

Miniflux/2.1.4 has buggy Last-Modified caching. Miniflux/2.2.0 does too. I thought it was squared away previously, but it's still doing crazy stuff when I do the pathological test cases to check for bugginess.

FreshRSS/1.24.0 has buggy Last-Modified + ETag caching. But, others running 1.24.2 and 1.24.3 have it fixed! The author actually reached out to let me know about this. Thanks!

Feedly/1.0 which went into a "once per day, always unconditional" thing months ago. Meh.

Reeder, various versions. Has the minor timing issue where sometimes it's 59 minutes, and other times it's an hour, as explained in a prior post. Call it the "cron alone is insufficient limiting for poll scheduling" thing, I guess?

feedparser/6.0.10, running too fast: 29/30 mins.

feedparser/6.0.2, doing the minor 59m/60m timing thing.

Yarr/1.0 with terrible timing.

Another Yarr/1.0 with the 59m/60m thing.

walrss/0.3.7, seems fine.

Emacs Elfeed 3.4.1 with some odd behaviors, like there are multiple instances running that aren't sharing the data on the last-modified/etag headers from the last poll. Another instance is doing the same thing.

er0k/feeds v0.2.0, seems fine.

Rapids, seems fine.

Go-http-client/1.1, seems fine, whatever this really is. Could use a more descriptive UA, if only to not run afoul of filtering (not me, but other places).

theoldreader.com has started doing If-Modified-Since and If-None-Match. This warranted a straight-up "holy shit" from me when I saw it in the list. I have no idea if this project had anything to do with it, and I really don't care. They're doing the right thing now, and that's great.

Tiny Tiny RSS, multiple instances from various users. Not recommended.

rss2email release-3.3.1, which keeps doing unconditional requests seemingly randomly. No idea why.

NewsBlur also randomly does unconditionals: 8 of 526 polls that way for one instance (and other counts for other instances). Also no idea why.

Feedbin, which seems to have fixed its ETag bugginess. Yay?

Inoreader/1.0. It has the 59m/60m thing, and also launches unconditionals roughly daily. Dumb.

Some curl-based thing (maybe even the CLI tool?) that's doing conditional requests nicely. Has the typical (minor) 59m/60m thing.

inforss/2.3.3.0, seems fine.

haven sends 100% unconditional requests. It also happens to have the 59m/60m thing, but the complete lack of conditional requests is the showstopper here.

ureq/2.9.1, seems fine.

Friendica/2024.08, also 100% unconditional requests.

SpaceCowboys Android RSS Reader/2.4.15 occasionally has weird timing, and it just goes unconditional randomly. No idea what's going on here, either. Another one is 2.6.31 and that seems fine, so I suspect there was a bugfix release in the middle.

Netvibes, with super broken LM+ETag caching.

cry-reader v0.0, seems fine.

Another undifferentiated browser extension thing, but this one is 100% unconditional, and wacky timing on top of that.

Blogtrottr/2.0, seems fine.

Unread RSS Reader, which is no longer unloading like mad onto the test server, but it has settled into a 15 minute poll interval. Too fast.

Slackbot/1.0, this is a relatively new one, and it's terrible. It's 100% unconditional polls with awful timing: always less than an hour between checks, sometimes much less. They also inexplicably send HEADs, and receive a 405 error from me, but of course come back and do it again a few minutes later.

The whole point of HEAD is if you want the metadata but have no interest in the content. If you want the content but only want a fresh copy if it's changed, that's why we have conditional requests, and again, If-Modified-Since has been in the RFCs since *1996*.

For sites that build the feed dynamically (i.e., not mine), HEAD represents the same amount of work: they get to build the feed, provide the metadata, and then throw it away.

Unless you actually want the metadata, and never anything BUT the metadata, then fine, send HEADs. Otherwise, forget it existed.

Audrey2 RSS Reader/0.7.1, seems fine.

rss2email/3.14, has the 59m/60m thing going on, otherwise it's fine.

...

I should note that if a feed reader's only problem is the 59:xx vs. 60 minute thing then they're actually doing pretty well. There are a lot of way worse things out there.

I'd love it if they could fix this, obviously, but that's just how I am.

This seems to be a thing now: someone finds out that you worked as an SRE ("site reliability engineer", something from the big G back in the day) somewhere, and now all you're good for is "devops" - that is, you're going to be the "ops bitch" for the "real" programmers. You are the consumer. They are the producer. They squeeze one out and you have to make it sing and dance. You keep things running and you shut the hell up. You wear the pager so they don't have to.

I've seen this from the hiring side of things: when we were trying to hire well-rounded people and put up a job posting with "SRE" in the title, all of the sudden we got a bunch of applications from people who basically *were* ops monkeys. They wanted to be that and do that. That was their life, and they enjoyed it. Those of us on the hiring side were taken aback by this and didn't want that kind of hire getting into the place.

Clearly, somewhere along the line, someone lost the thread, and it has completely destroyed any notion of what a SRE was supposed to be.

Just so we're operating on a level playing ground here, I'll lay down my own personal definition of the term, and what I expected from people in that role and what I expected from myself.

To me, a SRE is *both* a sysadmin AND a programmer, developer, whatever you want to call it. It's a logical-and, not an XOR.

By sysadmin, I mean "runs a mean Unix box, including fixing things and diving deeply when they break", and by the programmer/whatever part of it, I mean "makes stuff come into existence that wasn't there before". In particular, I expect someone to run the *right* things on those boxes, to find the actual problems and not just reboot stuff that looks squirrelly, and that they write good, solid code that's respectful of the systems and the network. They probably write programs to make the sysadmin part of the job run itself. Automation for the win.

That, to me, is my first order approximation of what a SRE should be.

Now, there must be some reason I'm writing about this, right? Yes, and there is. I put out some feelers to see about maybe working with a small company that's building some interesting things. They're wrangling Linux, C and C++, embedded stuff, networking, and they're almost certainly going to need a certain amount of pickiness regarding correctness and security to keep bad people from breaking into stuff. Also, they didn't have the usual lists of godawful clown software that most places rattle off that you'd be expected to work with.

In short, it was a place that's Actually Building Stuff and isn't just throwing their money at one of the usual clown vendors. That's rare!

So, I reached out... and heard nothing. Then, much later, I reached out a different way, and eventually heard back: they looked me over, figured I'm a SRE and they have devops people already, so, uh, no thanks.

That's it. That's the whole thing. The door is closed.

I'm obviously not happy with this situation. In sharing with some friends, they also mentioned "having to have this conversation" (that they are not just an ops monkey) a great many times in the course of looking for employment over the years.

Clearly, things have gone to hell, and unless you WANT that kind of ops-only life, you probably don't want to sell yourself this way.

Just to be annoying, I'm going to rattle off an example of something an ops monkey would never do. I wrote this C++ build tool, right? I've mentioned it a few times in various posts over the years, and there have been a few anemic web pages on my main server talking about what it is and how it works.

I won't go into the full history of it here. A quick description is: it knows how to build C++ using #includes to figure out dependencies, and so you need not write a build script, a Makefile, a CMakefile, or any other config language file. This goes both for the stuff inside the local source tree, and for external system library dependencies: stuff like libcurl, a pgsql or mysql client library, GNU Radio, libmicrohttpd, jansson, or basically anything else you might think of. It knows how to use pkg-config to get the details for some well-known targets, and you can add entries to a config file to map the "#include <proj/lib.h>" onto the pkg-config name for anything else.

So, again, that's all old news. I first wrote that over a decade ago and have been using it all this time, with small improvements over the years. What's new? Well, a couple of months back, I decided it was finally time to make the thing run its operations in parallel to take advantage of a multi-processor system.

It now scans the targets to determine the dependency graph in parallel, then kicks off compiles, then kicks off the linker. Everything that used to be serialized has now been made as parallel as possible.

Obviously, you can't just do this as a rash decision. There are any number of things which can go terribly wrong if you don't manage your data structures properly for safe cross-thread accesses. You need to be able to put things to sleep and have them wake up later without having them needlessly "spin" on the CPU. You need to do depth-first processing and then have it kick off the "parents" as the "children" finish up. You still have to catch errors and stop the process where appropriate, and you also have to make sure you don't just boil the damn machine by launching EVERYTHING at once. "Just use std::async", it is not!

To give some idea of the impact, touching a file that gets included into a whole bunch of stuff forces everything above it in the graph to get rebuilt. This used to take about 77 seconds on the same 2011-built 8-way workstation box I've had all along. With just the early work on parallelization, it became something more like 21 seconds.

77s -> 21s, just like that. That's a lot of win that was just waiting to be realized by using the hardware appropriately.

Yeah, I did that. Little old me. This is not the behavior of someone who just twiddles other people's config files for a living.

Raar, I'm cranky.

Yeah, it's another thing about feed readers. I don't blame you if you want to skip this one.

A reader (that is, a person!) reached out earlier and asked me to look at a bug report for a feed reader. It seems they passed along some of the details from one of my earlier posts, and it was closed with no action taken.

The program in question has cache problems, and it's something that's surprisingly common. A bunch of different programs do this, and it's interesting to wonder how they all came to this point.

Well, at least for some of them (the PHP-based ones), I think I know now: they're probably using the same library underneath, and it's been hacked to do some kind of "hash match" thing based on the body of the feed - like a md5sum or somesuch.

It goes something like this, apparently: fetch the feed for the first time. Then store the hash, the Last-Modified time, and the ETag.

Next time, send If-Modified-Since and If-None-Match using the stored values. But, then, if the hash of the body matches, return immediately and don't update anything else... *even if* the Last-Modified and/or ETag values changed! So, next time, they send the old values again, and it never gets any better.

Therein lies the shared bug: they're not designed around the notion of "always return the values you got last time from the web server". If they had been, they would not throw them away just because the hash matched.

So, when might the hash match when the LM or ETag values are the same?

Easy! When someone does this, or the equivalent:

webserver$ touch feed.xml

That will bump both of those fields and will leave the body unchanged.

Them not handling that case means they will start sending unconditionals until the body finally changes. That could be days or weeks later depending on how often something actually changes in the feed. Not good.

Many years ago, I was given one of those massive radio clocks that also reported indoor/outdoor temperature and humidity. They were apparently sold at Costco under the "SkyScan" brand, and had two parts: the (very large) indoor display which stood upright with a little plastic foot in the back, and a small sensor which was intended to be placed outside.

It was never quite 100%. It had issues receiving the 60 kHz WWVB clock signal and so it would get out of sync. It also apparently had its own rules for DST baked in, so when the US changed them in 2007, it would go stupid twice a year for a couple of weeks.

It had other problems. I once caught it saying that it was the 9th day of the 14th month, and oh hey, it's a Tuesday. Lousy Smarch weather or no, that's so unbelievably wrong I just had to stop using it.

However, I kept the sensor as more of a curiosity. In recent years I got into the whole business of tracking temperatures in certain interesting places, and this old sensor kept bugging me. The usual tools can see it doing its 433 MHz announcements but nobody seems to have decoded it, or at least, documented it. I've been taking a whack at it every now and then, and finally got it to a usable spot.

So, for the sake of the dozen or so people out there who might have one of these things nearly 20 years later and who want to use just the remote sensor, here's what I've learned about it.

The encoding is no frills: on is 1, off is 0. The on pulse or the off gap lasts about 3900 us (curiously close to 1 sec / 256, for what it's worth).

Transmissions are sent twice, twelve seconds apart. The first one has a particular bit set to 0, and the second one has it set to 1, so I've called it the transmission number. The whole process restarts every 60 seconds.

There appears to be some kind of preamble or sync which is just 10.

Bits 2-33 are followed by inverted versions of themselves in bits 34-65.

Humidity is BCD: 4 bits and another 4 bits, so [4] [7] for 47%.

The channel seems to be 3 bits but is only ever 1, 2, or 3.

Temperature is also BCD with 3 groups of 4 bits: [3] [2] [7] == 32.7C.

All of this has been determined empirically. There are timing diagrams for this and other similar devices on the usual "fccid" sites, but none of them match this scheme exactly.

So, from the top, 2 preamble bits (10), then...

4 + 4 humidity bits (BCD)

1 transmission bit

3 channel bits

1 sign bit (1 = negative temperature)

1 battery bit (1 = battery is low)

6 id bits

4 + 4 + 4 temperature bits (BCD)

... and then the part past the preamble repeats, inverted. This can be used to detect errors.

For the sake of grep bait: the SkyScan 81690 with a sensor bearing the FCC ID of OQH-000000-05-002, or perhaps OQH-CHUNGS-05-002. It might be a "C-8102" model, but again, all of the FCC stuff I've been able to find is not quite right.

I've been hearing from feed reader authors as a result of the whole "feed reader score" project. Last week, it was NetNewsWire, and this week I have an update for Unread Cloud.

The author put up a post which looks at what's been going on, and explains what happened with the poll frequency for a while on that one test instance. It also has some interesting graphs built from the raw "hits" table in the report.

I should point out how overwhelmingly reasonable everyone has been for this project. It's the sort of thing that could go over very badly - trust me, I've been in the position of "detecting badness" over the years at various jobs, and a lot of people do not appreciate it!

Instead, we've been slowly patching up a few small issues here and there across the collective fleet of feed readers, and hopefully this will benefit readers and writers both. Writers get lower "background radiation" for their servers, and readers get quicker updates without tripping rate limiters. Everyone wins.

To everyone, thank you for your efforts in this ecosystem.

Before I launch into responding to some reader feedback, there's a strange meta-note that I'll mention in case it twigs any memories for other people who run web servers. There's some really strange set of web robots that seem to go crawling for URLs that end in something like "feedback" or "contact" or "support" and "help" and stuff like that. They'll pull an index page and then fetch *all of them* in one fell swoop. It's pretty clear that they don't give a damn about looking halfway plausible. Real people don't request pages like that.

What's really strange is that nothing else happens after that point. Maybe they're looking for pages that have feedback forms, or comment fields, or something like that?

Fortunately, I have no such things on my posts.

...

An anonymous reader asked for my Github URL. I actually used it with a paid account for a while, but then stopped using it a long time ago. I'm talking waaaaay back in 2012 when they were pulling all kinds of stupid "let's trust your data to this overly dynamic web shit" on a regular basis.

I moved out 12 years ago and haven't gone back. It turns out that it's rather simple to just set up a repo on an accessible box and point the participant boxes at it - the ones where I do dev work, the ones that run automatic builds for me these days, and so on.

They've done some dubious things with "public keys" that I disagreed with, too. Someone actually went and made a "ssh whoami" type thing where if you connect to it and present the public key that's on a github profile, it greets you by that profile's name. Cute.

(Aside: There's a DEF CON talk out this week which talks about how people *still* don't really know that sshd will totally tell a client whether a key pair is allowed in given just the *public key*. You should find the slides for "SSHamble". They're good.)

Since then, they had a bunch of crazy stuff involving the treatment of their employees and all of that kind of crap, but I was already gone. Then, hey, an old adversary from the 90s popped its head up and gobbled them up. Yep, Microsoft bought Github in 2018. That's the kind of devious move that is entirely consistent with their old ways of operating.

This is where a bunch of people who weren't even alive in the 90s pop up and tell me how they're different, and thus prove a point I was telling people about 10 years back: "you're not going to recognize them eventually, and younger folks will have no idea of the evil they've foisted upon the world over the years".

Basically, if I was an evil sort and wanted to keep my thumb on the scale of what the "dirty free software hippies" were doing, buying the thing that's practically become synonymous with "git" itself for many people would be a great place to start! So, when they went and did that, well, that's just cake.

That'd be like IBM buying Red Hat... but wait, that happened too!

It's ironic that a fully-distributed-capable system like git ended up with its own "network effect" that brought people into a single site to that degree. Sure, there are others, but you have to know something about the scene to even realize they exist in the first place.

I bet if that had happened in 1998 instead of 2018, it'd be "Microsoft ActiveGit" or some crap like that by now. They would have totally rebranded it to try to fit with the rest of their corporate hell.

When it finally does happen, try to act surprised.

So... TL;DR there's a github profile out there where you'd expect to find one for me, but it has no activity, and for good reason.

...

The previous reader also asked if I had ever taken someone else's code base and turned it into something that "bears little resemblance to the original creature". I'm pondering this one, and, well, I don't think I have? It's not like I turned a wolf into an eel or something like that.

Okay, there was this internal web service up at a place with WAY too much Python. It was (of course) written in Python as well, and I've mentioned it before. It was a tool which was intended to make it easier for people to collaborate on outages... you know, "SEVs", or whatever you might call them where you are.

It had been written by someone as a side project of sorts, and that particular company had no love for such pure engineering work. In fact, that person was on the verge of getting in trouble with their manager for hacking on it in the first place.

So, when I showed up and realized we needed some different behavior out of this tool, I reached out to the nominal owner, learned of the situation, and then got their blessing to go and hack on it, and hack I did.

I put in enough to get it to where other people could see where I was trying to go with it, and then a couple of solid developers showed up and started cranking on it. They fixed a bunch of stupid things I had inflicted upon the code base, and got it to where it was a joy to use.

Also, I had an intern come through who did an amazing job on all of the "followup" side of things - you know, getting a report together, bringing it to a review meeting, collecting follow up tasks, and making sure those actually happen. I had been doing it by hand, and it was awful. When he got done with it, it was a breeze.

Did it look like the original tool? I guess superficially it kind-of did. It still had a list of comments and a way to add a new comment. It picked up features for both the direct users and the reliability people behind the scenes, but it was still nominally an "incident manager", even though I had rebranded it "SEVPanel".

It effectively went from a young wolf to an adult wolf. No eels or other aquatic wildlife mangling was involved. That's all.

The better question is probably *why* I did what I did and set to work on an existing thing instead of trying to come up with something new. That's a long and messy answer. It has chunks of "because it was a right pain in the ass to start a new project there" along with "I was getting enough pushback just introducing the term 'SEV'", so changing the *host name* of the service from incidentassistant.${COMPANY}.${TLD} to something else was right out. I never would have heard the end of that one.

It was so hard to start a service there... ("how hard was it?") ... that one weekend, someone parked a critical piece of infra atop our "bookmark" shortcut service because it was needed to support a certain piece of legislation that had been passed by a city. It was a matter of "put it right there right then" or the company would have been in capital-T Trouble. Also, the company would have looked like giant assholes to the people who needed that level of service, but don't despair - they delivered on that sort of thing plenty of other times that year.

In the rush, the only real solution was to stack it atop something solely intended to support in-browser shortcuts for employees. Why there? It was simple, it was easy to hack on, and if they broke it for a few days, eh, it wasn't the end of the world. It was literally a thing that saved you typing out longer URLs when you wanted a shortcut to a particular internal web page.

As for why it was so urgent, apparently management sat on it as long as possible, THEN dropped it on the engineers. Typical crap for them.

And hey, now you know a bit more about the context of my "edit in prod" story from a few years back.

...

Another reader asked for my opinion of Tiny Tiny RSS because they are planning on self-hosting it. I pretty much covered the range of behaviors that are showing up to the feed test site in my summary post from the other day.

It's definitely doing a bunch of "double tap" polls all over the place, and I have no idea why. To be clear, I don't really care all that much WHY it happens, just that it does, and it happens a lot. It's not like a single user went and hacked something broken into the source. That doesn't fly when it shows up from multiple users who don't even know about each other.

Nope, whatever's going wrong is some weird stuff from upstream.

Wow, okay, things are cooking today. I posted last night with the results of the feed readers which were still actively pinging the test feed. I've heard from several feed reader authors, and they're cranking away at stuff! They obviously care a lot about doing the right thing.

One of them even put up a post already, and so I feel obliged to share it:

NetNewsWire and Conditional GET Issues

That post actually has a link to the report page, so if you want to see what a real batch of data looks like, there you go.

There are some non-feed-reader bits of feedback I'd like to address, too.

...

One reader asked for my take on the whole CrowdStrike thing. This is where they shipped some update to a bunch of Windows machines and nuked them. I get the impression that this was something that ran rather low in the stack and had the ability to pretty much take control of the whole machine... or kill it entirely, as it turned out.

When stuff like this happens, I fall back to pointing out something that we tried to make into a "thing" at FB in the fall of 2014. There had been *far* too many outages in a really short period of time. It had started with "Call the Cops" (10 years ago now - August 1st!), and a month later we were getting tired of outages every Friday at 3 PM - I started calling it "SEV O'Clock".

The notion we tried to get known far and wide was "nothing goes everywhere at once". This means that code pushes, config changes, and even flag flips should happen piecemeal. Don't go from 0 to 100% in a single step. Take a bit and space it out. If you can't space it out, ask yourself why, then see if you can make it happen.

I've seen some amazing things almost happen. One company was in the quiet period prior to their IPO when the execs know when it'll happen but the rest of us are in the dark. Basically, it could happen *any day*, and the bigwigs are out schmoozing with the press. It's the last time you want to tank the site, service, or whatever.

Once, I overheard some people saying they were going to convert the entire site's user database from goofy not-SQL db flavor A to goofy not-SQL db flavor B in one fell swoop on Sunday night. It was Thursday. I asked if they had a way to back out. Nope, it was a one-way trip. I asked if they could do it in stages. Nope, it was all or nothing.

"So you want to do something that will affect everyone at once, with no way back, on a weekend... during the IPO road show?"

They reconsidered.

To be clear, I have no idea if the CrowdStrike thing rolled out as fast as possible or if it had stages. It seems like if it had a phased rollout, then it might've nuked a few machines in a few places, but then the alarms would have gone off and they would have hit the big red STOP button... right? All of those hosts disappearing right after getting our update means something, doesn't it?

Well, look at that. "Breach hull--all die." Even had it underlined.

Oh boy, it's been a while, so let's talk about feed readers. This whole scoring project has been running since the end of May, and much data has been collected. It's logged over 70K requests to tagged feed URLs in that period from 97 distinct keys - the unique IDs which should be used by a single install of a single program.

Two of them have sent over 6000 requests in the time they've been running. Another half-dozen or so have sent over 2000. Keep in mind that not all of these started at the end of May, either. While some of them go back to the beginning, quite a few are much newer than that.

I've gone through and summarized anything that's still checking in, using an arbitrary cutoff of "has polled in the past week".

Once again, while any one entry does not reflect a program since a user might've configured it really badly (or really well), the patterns become obvious pretty quickly when you see multiple instances of the same thing.

Unlike before, now I'm actually going to list the names and versions. It's time.

[IMS = If-Modified-Since header. INM = If-None-Match header.]

...

Netvibes. Has some kind of caching error - stuck on old IMS/INM values for the past month basically. I'm going to kick it again to see what happens, but it doesn't bode well. Has some dumb 3 second rapid-fire thing at feed init, too.

SpaceCowboys Android RSS Reader / 2.4.15 (previously aka "sweatpants", more or less, apparently). This thing has a polling interval that's all over the place and it likes to just drop the IMS/INM and do unconditional polls now and then. No idea why.

NextCloud-News/1.0. Super broken caching. Terrible mess.

NewsBlur. Rapid-fire thing at startup, otherwise seems fine.

Friendica/2024.06-dev. No conditionals, ever. Also bum rushes the server when first adding a feed by hitting a bunch of goofy not-the-feed URLs (which can't be seen in the report, but *I* can see it in the logs).

Roy and Rianne's Righteously Royalty-Free RSS Reader v 0.2. Seems to have settled down into a nice polling pattern. No complaints.

Tiny Tiny RSS/23.04. Nothing but unconditionals. But then it also does this < 1 second "OMG LET'S POLL AGAIN" thing. What's the point? Annoying.

ureq/2.9.1. Seems to be in a good pace/pattern now.

Feedbin. Hella broken IMS/INM caching, terrible pacing, regular forced unconditional polls, etc. They trip 429s in prod constantly. There are a bunch of these and they're all broken the same way, so I chalk that up to the program/service and not any one user's config.

NetNewsWire. Bigtime brokenness in their IMS/INM caching. No idea why, but it makes it generate unconditionals constantly.

haven. All unconditional requests. Pacing is all over the place.

Miniflux/2.1.4 (plus older versions previously). They had some kind of caching bug but it looks like it's good and dead with this version. Hooray for doing the right thing!

inforss/2.3.2.0. Good pacing, proper caching behavior. Nice.

Miniflux/2.1.3 and then 2.1.4. As above. Yay fixes.

Some random thing that's just curl. Seems generally okay. Does a cheeky thing every week where it drops the IMS/INM, perhaps to see if the web server admin is a clown. (Spoiler: I definitely am.)

Inoreader/1.0. All over the place with timing and unconditionals. Also shoves a useless referrer down the pipe every time, as if that is going to accomplish anything. Has a really nasty rapid-fire startup sequence that'll probably get it autoblocked before it can even get subscribed properly.

Feedbin. Bleh. See above.

Feedbin. Meh.

Miniflux/2.1.0 that needs to be upgraded to 2.1.4. Currently isn't showing signs of cache insanity because I haven't deliberately triggered it in a while.

NetNewsWire. Just like the earlier one, broken caching and crazy timing.

Liferea/1.15.3. Has a tendency to just drop everything and start making unconditional requests... one second apart? Not great.

rawdog/2.24rc1. Behavior is spot on. More like this, please.

NetNewsWire. Same bad caching and timing stuff.

Something with no User-Agent, or IMS, or INM headers. It's like... okay, what exactly is going on here? At least it polls super slowly.

NewsBlur. Just like the other one.

FreshRSS/1.24.0. This one latched bad IMS/INM values back in *May* and is still sending them. Hella broken.

Miniflux/2.1.3 and 2.1.4. Seems fine now that it's been upgraded.

Tiny Tiny RSS - multiple 24.05 and 24.06 builds. Slightly better than the 23.04 one mentioned earlier. This one does send conditional requests, but it also sends tons of unconditionals, and it does the "OMG 1 SECOND HAS ELAPSED SO I MUST GO GET IT AGAIN" thing. Not cool.

Tiny Tiny RSS 24.02 and 24.05 - various builds. Just like the previous entry.

CommaFeed/4.4.0, 4.4.1, 4.5.0, 4.6.0. This had some caching bugs for a while and was sending bad IMS values. It's been behaving lately. It's probably time for me to deliberately do some weird (but totally valid) behavior again.

Vienna/8052, 8219. Timing is all over the place, but at least it sends conditional requests.

Some claims-to-be-a-browser thing (probably a browser extension). Sends derpy 2000-01-01 IMS at startup for no good reason. Caching headers jump around and disappear sometimes, like maybe multiple instances are warring over the state storage or something. Far too many < 1s polls for my liking.

Feedbin. Feh.

Tiny Tiny RSS/24.04. As explained above, OMGs and all.

theoldreader.com. I'm pretty sure Google Reader's poller did conditional requests. This thing doesn't. Ever. They seem to get around blocks in production by polling every 6-7 hours and using multiple hosts.

Some unidentified "Go-http-client/1.1" that seems to have a decent pace and which sends INM headers. No idea what it is, but it seems to be polite about the job it's doing.

Rapids. This one got upgraded and is now doing both IMS and INM and looks great. Thanks to the author for the effort in changing it.

er0k feeds. This one also seems to have been upgraded at some point and seems to be keeping up just fine now. I definitely need to do some "inject some crazy" testing to see how it does.

NetNewsWire. Broken, as above.

walrss/0.3.7. Just works. Nice.

Yarr/1.0. Startup had a bunch of unconditionals but then it settled into sending IMS and INM headers. Timing seems a little squirrelly, like it's not actually waiting for at least an hour to elapse, but rather just starts on some schedule and is subject to however long the last run took.

Bad data. Different programs using the same key.

FreshRSS/1.24.0, 1.24.1. Bad caching, bad timing.

Some other "might be a web browser extension" thing. Does the same 2000-01-01 IMS rapid-fire stuff at startup. Sending caching values but it's running a bit quick.

Feedly/1.0. This one is just abusive. At some point it switched to sending nothing but unconditionals every 30 minutes. It's like, okay, this *fake* feed doesn't say much. So chill out, already. Comes from a bunch of different hosts, too. Annoying.

NetNewsWire. Bro-ken.

feedparser/6.0.2. Does cached requests fine, but the timing is squirrelly here just like some of the others. I wrote a post about why you can't just say 'start me every hour' and leave it at that a few weeks back.

Reeder, various builds. This thing is bizarre. It'll just flip and flop between INM values (things I served up as an ETag) from weeks in the past. Tell me that's not some kind of wicked caching bug. Also polls FAR too fast. Some programs have timing that is merely squirrelly. This is a whole cage of them.

Unusable data - multiple programs.

Unusable data - multiple programs.

Unread RSS Reader. Godawful poll timing. 6103 requests in 52 days is about one poll every 736 seconds _on average_, but they're hugely spread out. WTF? Put it this way: the list of unique intervals (nn seconds, nn minutes, ...) is *four pages tall* on my web browser.

Feedly. Also went to 30 minute all-unconditionals. Bah.

FreshRSS/1.23.1. Bad caching, like the others.

Yarr/1.0. Just like the other one: settled into sending IMS/INM but the timing is all over the place, and often far too fast.

Unusable data - multiple programs.

Miniflux/2.1.3 that needs to be upgraded to 2.1.4. As above.

Miniflux/2.1.3. Same.

Another unidentified "Go-http-client/1.1", but this one seems to have the weird timing issues like so many others.

newsraft/0.25. This one got into some bad places with stuck caching headers. Given the version number hasn't shifted, I assume it would happen again if I ran another test from my side. Wacky timing, too.

NetNewsWire. Broken record.

feedparser-perl/6.0.10 is doing conditionals but it's happening too quickly, and the timing is mighty inconsistent.

Mojolicious (Perl). Just works. No issues. Thanks.

NextCloud-News/1.0. Very broken caching. It's not even using the right timing to generate If-Modified-Since requests, and this has been a problem for years. This is the infamous "1800-01-01" IMS sender.

com.vanniktech.rssreader, various versions. Timing is all over the place, frequently far too fast - 2 seconds, 10 seconds? Also seems to "lose" the INM header now and then.

bdrss/4.0. Sends conditional requests properly. No idea how the timing works on this, but at least it's on the "long interval" side of things and not the "super fast polls over and over" like most of the others in this list. Maybe it throttles back when a feed is quiet? That's great, if so.

Broadsheet/0.1. Seems fine.

feedbase-fetcher.pl/0.5. Also seems fine.

Artykul/1.0. I installed an app on my Mac and it started checking in, and this stupid clown thing started checking in as well. The Mac app is long gone, but this albatross won't go away. Sends unconditional requests every single time, and it's visiting URLs you probably didn't intend to share with them! Wonderful. I wrote a post about this a while back.

...

There you go: a bunch of bright spots and then a bit of sadness.

Choose wisely.

I ran into a troubleshooting scenario the other day which ended up adding to the list of things that I need to check on when trying to figure out why packets seem to be disappearing. It went like this.

I showed up at a site where I'm running some weather station sensors and jumped on the console of one of the Linux boxes. My visit was about adding some sensors to some new areas, and I wanted to see how things were going. In particular, I wanted to see how the receiver on the local machine was doing, and what it had managed to log of late.

(Just imagine the port number is 1234 here.)

$ thermo_cli ::1 1234
rpc error: deadline exceeded while awaiting connection

... what? That made no sense. The thing was running.

I looked in 'ss' to make sure it was listening on that port and specifically was ready for IPv6 connections. It was.

LISTEN 0      1024         0.0.0.0:1234       0.0.0.0:*    users:(("thermo_server",pid=1141761,fd=4))                                                 
LISTEN 0      1024            [::]:1234          [::]:*    users:(("thermo_server",pid=1141761,fd=5))                                                 

I tried netcat instead... same thing. Instead of connecting, it just hung there. I looked in ip6tables... nothing. This host has no rules at the moment: nothing in 'filter', 'nat', 'mangle', etc. This should Just Work.

This site isn't running IPv6 natively due to a dumb ISP, but there are still ULA and link-local addresses, so I tried one of those from another host on the same network. That also went nowhere.

Looking in tcpdump, it was pretty clear: SYN comes in, nothing returns.

But, at the same time, I could change from that port to something like 22 and it would work fine. I'd get the usual banner from sshd.

Assuming it was something stupid I had done with my own code somehow, I ran another program as a test, then tried connecting to it over v6. It worked fine.

I straced it to make sure it was handing the IPv6 listener fd to poll(). It was. It wasn't getting any sort of notification from the kernel. But, actually, hold up, no SYN/ACK happened, so there's no way it would have gotten anywhere near userspace.

I stopped the service with systemctl and put up a netcat listening on the same TCP port (nc -6 -l -p 1234) ... and then I could connect to that just fine. So, it's not something magic about the port, somehow. It's just that port when it's going to the service which normally runs on it.

I started making a list to see what the patterns were. This box, this program, talking to ::1? Bad. Another box at this site, same program, also talking to ::1? Same problem.

Was it because this site has no v6 routing to the outside world? That makes no sense as to why ::1 wouldn't work, but, hey, one more thing to discard. I invented a fake route. Nothing happened (fortunately).

Next I started up a Debian VM on my laptop, hooked one of the radios to it, and started the receiver program on it by hand. It ran just fine, and accepted traffic over IPv6 to that same port without any trouble. It's on the same v6-route-less network as the other hosts, so what's up with that, right?

Maybe I did something stupid with the config file for the program, so I copied that across verbatim from one of the site hosts instead of just making a fresh one for testing. It didn't change things.

What if I dump the v4 listener on that port and just run the v6 listener? Nothing. What if I add a listener on another port? Nothing. Now that port also drops packets when I try to connect to it that way.

I don't know what it was about this last point, but somewhere around here, a couple of ideas finally connected in my head and I went "uh, systemd".

The failing instances were both running as systemd services. The successful instances (whether the thermo_server program, or my other test stuff) were just me doing ./foo from a shell.

That's when I thought about the hardening work I'd been applying to my systemd services of late. I've been taking away all kinds of abilities that they really don't need.

One of the newer tricks in systemd is that you can do "IPAddressDeny=" and then "IPAddressAllow" and keep a program from exchanging traffic with the rest of the world. For a program that's only ever supposed to talk to the local network, this was a good idea.

That's when I saw it: I had 127.0.0.0/8 and the local RFC 1918 networks on the Allow line, but *not* ::1, never mind the ULA prefix or the link-local v6 stuff. Adding ::1 and doing the usual daemon-reload && restart <service> thing fixed it.

Here's the deal: systemd implements that by injecting bpf program(s) when you ask it to filter traffic by IP addresses in the .service file. When this thing rejects traffic, it just drops it on the floor. It does this past the point where ip[6]tables would match it, and well before the point where it would generate a SYN/ACK or whatever else.

There are no counters associated with this, and it doesn't generate any messages in the syslog or whatever. The packets just disappear.

It's effectively equivalent to an ip[6]tables rule of "-j DROP", but at least those rules have byte and packet counters that you'd see incrementing when you're smashing your head against it. This just makes the packets disappear and nobody has any idea what's going on.

So, if you ever see traffic effectively being blackholed to the port or ports which are bound for a particular systemd-run service without anything showing up in your iptables (or let's face it, nftables) stuff, you'd better check to see if there are IPAddress* rules in the .service file. That might just explain it.

Hopefully you'll remember this and not waste a bunch of time like I just did.

I got some reader feedback the other day which amounted to "what happened to your /edu stuff" and "wasn't there a lot more in there?" These are legitimate questions, and I figured it's worth explaining what that was and then what happened to it.

Way back in 2013, I was post-one-big-company and pre-another, and got the idea in my head that people would want to watch screen captures of me writing code. I even had the thought that there might be some money to be made in the process.

I didn't like the fact that most of these "screen recordings" were literally videos of people's terminals (or IDEs, ick). They lost the ability to copy and paste from the screen since it was no longer character-based and instead was pixel-based. It also meant way more bandwidth to serve up such things.

Back then, I was on still on my ancient ServerBeach machine which ran its Ethernet port at 10 Mbps *half duplex*. I'm fairly sure they provisioned their network this way deliberately. It had the effect of limiting just how much load you put on their network since they didn't charge for bandwidth back then. (Now you know why YouTube used them for their video serving pre-Google, and why only their db stuff was on the Rackspace side of things!)

As a result, I didn't want to get into the business of serving videos over that anemic pipe. The link already slowed down way too much when someone would post an image-heavy post like the "Apple Maps sucks" series to HN or reddit or whatever.

That got me thinking, and I built something that would record my terminal as text, control sequences and all. Then I took a whack at writing a VT100 emulator, got far enough along and realized it had already been done (tty.js), and had a compatible license, so I grabbed a copy of that instead. Then I just chopped out all of the stuff that made it able to take input, and hard-wired it to my playback system.

Then I started putting up recordings of various dumb things, like me using my non-Makefile-based build system. The idea was that maybe if people saw me using a tool that didn't suck and which made my life better, they'd want a piece of it, too.

At some point, this craziness was linked up to my existing code which talked to Stripe, and so you could pay me a buck to add an item to your "account", and it would let you play back some "lessons".

Then I got hired to serve up cat pictures... or at least, to keep the serving of cat pictures working. All of my time and energy went into that, and things on this side of the world slowed way down and eventually ground to a halt. Weeks or months would pass with very little going on.

About two years into the cat-pic-wrangling gig, it was time to leave that ServerBeach machine behind in order to get IPv6 since they were still too clueless to offer it. All of the data was copied from one machine to another, but I didn't want to go to the work of rebuilding all of the CGI programs for RHEL 5, or validating that it actually worked. It required too much effort.

I just didn't have the energy to do much more than moving the web sites over and repointing DNS, and that included what it would have taken to make it work without the payment integration stuff - "free mode". The impact was that both /store and /edu were shut down.

That was pretty much it for /edu until someone reached out to me in May 2022 and said they were wondering what the page had been like, and if they could explore the old code. Fortunately for them, at that point in my life, I had cycles to spare, so I dug out some of the old stuff, chopped out the payment integration, and put some of the recordings of the old "protofeed" project online.

I didn't mention this anywhere, but if you happened to go there after that point, it would have Just Worked. It also means that some of the links in those old posts "came back to life", and that makes me happy on a certain nitpicky level that I don't expect most people to understand. (Cool [URLs] don't change, yadda yadda.)

So, if you haven't been keeping track, the virtual terminal is back online, and has been for about two years, but without new content.

If you like watching terrible code happen, you might enjoy it.

If you like badly-written web pages that tell you to reload to start over, you might really enjoy it.

Okay, let's talk about something other than feed readers for a moment. How about completely broken web browsers? Yeah, those.

This. This is a thing. Count the broken:

ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:26 -0700] "GET /w/css/main.css HTTP/1.1" 200 1651 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/feed.png HTTP/1.1" 200 689 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed/ HTTP/1.1" 200 8052 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 
ip - - [28/Jun/2024:14:44:27 -0700] "GET /w/2024/05/27/feed//favicon.ico HTTP/1.1" 404 20 

First up, why in the hell do you need to request the same link 12 times? No, scratch that, 15 times, since it does 3 more after getting the css and feed icon.

Then it goes for the favicon, and what clown decided that the right way to request "/favicon.ico" is to prepend the base path to it? This cursed thing that Microsoft foisted upon us back in the 90s is supposed to be at the top level. It's not part of individual directories. That would be stupid.

And yet, this thing decides to beat the shit out of the web server while trying to get it.

I used to wonder just what could be this stupid. The user-agents on these bad requests aren't particularly helpful. But, then one day, I got lucky and noticed that the first request of the set has one very interesting little detail in it (while the others do not):

FxiOS/127.1

FxiOS. That is, Firefox for iOS. That by itself was enough to get me looking, and oh, look what I found.

Request spamming when visiting a site

Request spam for favicon and apple-touch icons on iOS 16 + Firefox 105

Request Flooding when opening app

favicon.ico is in / , it looks for favicon.ico in every directory except / .

Firefox on iPhone makes a flood of requests for icons

Lovely. So, if you're using this garbage, know that you're probably leaving a trail of badness in your wake.

Yeah, I know, it's another post about feed reader stuff. I figured people are probably wondering what the results have been looking like. I also wanted to see things in aggregate, and fixing my mistakes turned out to really need a "30,000 foot view" approach.

So, here's a screen shot of the top third of my admin tool with the unique keys removed to give a taste of what we've all been collectively finding out.

This is showing the number of requests, how many were conditional or not, which ones presented a good If-Modified-Since value, one that was out of sequence ("oos"), or made up entirely. Then it does the same three counters for If-None-Match values.

Next up are the counters for cookies, referrers and query parameters which were needlessly presented for whatever reason, and finally it's whether the requests are showing up over IPv4 or IPv6.

I mostly rigged it this way so I could watch the "ims bogus" counts drop out as I fixed each set of entries in the database. (What a mess.)

There are more than a few keys which were issued but haven't been used, and that's what those grey rows are in the table. Even with that reduction, it still goes on much too far to present as a single screen shot - there are a *lot* of people taking part in this thing.

So, what can we learn from this view? The first row was just me running tests, so I deliberately tripped a bunch of badness to make it show up in the reports. Then there are a few boring ones which were more tests from me, and then it starts getting into actual programs.

See the one with 256 hits and 256 unconditional requests? That'd be the "leaked to the clown" situation from the other day. They're still showing up over and over to fetch that thing.

How about the one with 1258 bogus IMS values? That'd be a feed reader which is sending the feed update time, not the Last-Modified time. That's a nasty one since it looks conditional, but in practice it is not. Every one of those requests gets the full feed shipped out. (This is a rather popular program, so there are lots of these things hitting the real site all day every day. Groan.)

The really interesting parts of this are the ones which are consistently sending out-of-sequence If-Modified-Since and If-None-Match values. Out of sequence in this context means "they're sending a value, but it's an old one, not the most recent one served to them on their last hit". It seems we have managed to trip a LOT of caching bugs in these programs. They latch in some bad state, and just keep going like this forever. Not good!

One thing that has been gratifying about this is when feed reader authors have changed something in their code and then reached out to me for a fresh key. That lets them start from scratch with their new behavior and leave the old one behind. I'm always happy when this happens.

This also adds a new dimension that I'll have to start tracking, now that we have tests ending and others coming in to replace them: staleness. I'll only be looking at things which are still actively checking in so as not to penalize anyone who's improved their stuff. We will leave the old behaviors behind and focus on what's current.

That's the best you can hope for: a series of improvements.

A short note to my feed reader score project participants: I screwed up something fierce the other day. I set some bad dates for the Last-Modified header. It's all done in code, and it's not based on an actual file, so it's possible to set *nearly* any value in there.

It started when I either fat-fingered or stupid-houred this date the other day:

Thu, 08 Jun 2023 00:04:45 GMT

That was in the GET handler for the test feed for about a day. Then, when I "fixed" it, I set it to something a few days in the future:

Thu, 20 Jun 2024 00:04:45 GMT

It was only June 16th when I did that. Oops.

Anyway, this started showing up as all kinds of crazy If-Modified-Since values being presented by clients, and I thought those clients were taking the value from my end and were "clamping" it to the current time. Instead, nope, it was all on my side.

Would you believe that Apache httpd will do that all by itself? Yep. If you have a CGI program which emits a Last-Modified header in the future, it'll totally squish it down to the current date/time instead.

Further complicating the matter is that I logged the values that I *thought* were being served up, and then used them to run comparisons and generate warnings when they didn't match.

My sincere apologies to anyone who spent the last couple of days chasing after a problem that was totally created by me on this end.

Now I get to figure out how to clean up my mess. Fun!

With all of this feed reader stuff going on, I've learned a few more things about goings-on in this space. Some of it is just strange.

During the early development stage of this project, I installed a couple of apps from the Mac app store on my machine to do some testing. Each one of them was given a unique URL so I could tell them apart. This meant I started seeing traffic from my laptop to the test server, which is exactly how it was supposed to work.

But, one of those unique URLs started getting requests from some random "cloud" service. There was no indication this would happen when I plugged it into the app. It just appeared, and it was running in parallel with the requests from my actual laptop. In fact, I've since stopped running all of those programs, and they're *still* polling it - just under every three hours, and always unconditionally. Great. I bet it'll keep going approximately forever.

Nothing on the app's web page suggests this will happen. It just does. If anything, the "all articles are available offline and without an internet connection" blurb on their web page suggests the opposite: the laptop does a fetch and keeps it locally. What a concept!

So, if you were thinking about using that particular app to read some feed containing something relatively private, guess what, they're reading it too.

Timing things to happen at specific intervals is yet another way that we collectively find out that dealing with time is a hard problem. I've been noticing this while working on feed reader stuff, and I realized that it can apply to other problems.

It goes like this: say you want to have a process that runs at most once an hour. You are okay with it taking a little more than an hour between runs, but really don't want to go faster than that. Maybe you have an arrangement with a service provider to not poke them too often. Whatever.

So maybe you rig something up using cron, and it looks like this:

15 * * * * /home/me/bin/do_something

Then, every hour, at 15 minutes past, cron will run your program. Unfortunately, this by itself is not nearly enough to deliver on your arrangement. It's not even the problem you might imagine at first, which is that system clocks can be sloppy and can get pulled around by external forces.

Nope, this has to do with the time it takes to actually do the work, and not accounting for that when allowing the work to proceed again.

Back to our cron job. We'll say it gets installed at midnight, so 15 minutes later at 00:15:00, it starts a run. Maybe it does a lot of work and talks to many sites over the Internet. Some of them respond quickly, but others are slow. Maybe their DNS is taking forever to resolve the hostnames. Maybe another site is offline and is just dropping packets, so you sit there until a timeout fires on your end. It burns a good minute doing this.

At 00:16:00, it finally gets around to doing the "once an hour" work, and it happens relatively quickly. Then it finishes and goes to sleep.

About an hour later at 01:15:00, cron will run your program again. This time, maybe all of the earlier work happens much more quickly, and all of it completes in 15 seconds. That means you get around to your "once an hour" work at 01:15:15.

Oops. You were supposed to wait at least 3600 seconds - that's one hour - between requests, but you just ran it after only 3555 seconds.

The problem is that you you can't just rely on the start time of your program to know if enough time has elapsed since it last did some work which is supposed to be rate-limited. You have to actually track the time when the work *was attempted*, and then do the math of "elapsed = now - then" to see if enough time has gone by.

I tend to think of the timeline for this sort of thing as a series of fenceposts, like this:

start       action      end     (rest of the hour here)
    |          |        |
    v          v        v
----*----------*--------*---------------------------------------->

To avoid violating rate limits, you have to time things from when the action happens, not when the program starts up. If you want to really be paranoid about it, then you'll want to time it from when the program is all done with its work and is about to shut down (but this is a lot harder).

What ends up being much easier is to just remember whenever the work last started and/or finished, even if it didn't succeed. It should never select a target for refreshing until it has been idle for long enough. The program must never assume "well, I'm running again, so it must be time to do my thing". What if the box just rebooted, or any of a number of other possibilities? What then?

Here's an easy way to know if a program is on the right track: could it be run in a tight loop without causing a giant mess for other people?

$ while true; do run-my-stuff; done

If you can run something in a loop like that and not have it beat the crap out of whatever it's supposed to periodically talk to, then you're probably headed in the right direction. It also means that if the program gets into a start-crash-restart loop some day, maybe it won't unleash a hellstorm on whatever it happens to talk to.

Running a program in an infinite loop like that might chew a lot of resources on the local machine, but that's (relatively) okay. It's your machine. Feel free to burn your own resources. Where it becomes troublesome is when it reaches out and starts burning those of other people.

As usual, the details are important here.

I've had a few people ask me for results from the feed reader score project. It's been long enough to where I can start giving some details, now that we've had a good week or more of data collection.

There's one big thing to keep in mind here: I am assessing individual feed reader installations, including whatever config values the user might have set globally or on the test feed in particular. Those config values can be the difference between "amazing" and "get it away from me".

That means a single good entry doesn't necessarily mean that every install of that program will behave perfectly. It also means that a single bad entry doesn't mean that all of them will be terrible.

I've broken them down into a few groups.

Group A: No real complaints. They do their jobs quietly and don't make messes. Anomalies, if any, don't seem systemic and are probably just the result of the user clicking the "poll now" button (or equivalent). This is expected.

Group B: They tend to do spammy unconditional requests at startup, and usually at a needlessly fast rate, too - like less than a second apart. This is what most entries in group B have, and if that's their only problem, then fixing that would move most of them into group A. (There can be other small anomalies which put something here).

Group X: Unusable data. This can be because there's hasn't been enough data collected yet, like if someone just started it up, or if they shut it off before it ran for several days. It can also happen when someone points multiple feed reader instances (same version or not) at their unique tagged feed, or if they load it with a browser, curl, or similar.

Groups C, D, and F: Everything else (and I'm not identifying who's who, or what groups they might be in).

A few minutes ago, I went through all of the tests one by one and came up with my own assessment based on the available data. Ordering within a group is not meaningful.

Group A: instances of:

• awkbot
• rawdog
• Awasu/3.3PE
• walrss/0.3.7
• Mojolicious (Perl)
• com.vanniktech.rssreader:1.40.5, 1.40.6
• Broadsheet/0.1
• feedbase-fetcher.pl/0.5
• ... something unknown that claims to be a web browser

Group B: instances of:

• Liferea/1.15.3
• NewsBlur
• FreshRSS/1.23.1
• bdrss/4.0
• ... some unknown Thunderbird extension
• ... another thing claiming to be a web browser

Anything not shown here is not being tested or is in another group, or I screwed something up and missed it. Contact me if you think I skipped your entry.

I should mention that there are a more than a couple of systemic bugs have been found across multiple reader programs:

Bug: It's entirely possible for a feed's Last-Modified value (seconds) to remain the same while the ETag (length + microseconds on stock Apache) changes. More than a few feed readers assume if they get the same value for Last-Modified, then they don't have to update the cached ETag value. This causes them to effectively make unconditional requests until the feed changes again. Watch out for shortcut evaluations in your caching code!

Bug: If-Modified-Since is only really valid if you were served it as a Last-Modified value previously. Readers are inventing values, or are sourcing them from the wrong layer of the stack. Don't do this.

Only use the last Last-Modified value for If-Modified-Since, and only use the last ETag value for If-None-Match.

Bug: Timing is too tight, and they aren't accounting for how long it takes to perform a poll. I'll probably do a separate post about this since it comes up in other things in the world, too.

Bug: Launching multiple identical requests at feed init time, and usually in a volley that triggers rate-limiting. There's something wrong with the network I/O design when this happens. Calls across the network are not "free" and should be executed sparingly. Don't discard the values only to fetch them again a moment later.

Well, it's been an interesting couple of days. I got all kinds of feedback submitted from people who wanted to participate in the feed reader score service project. I spent quite a while answering every one of those and issuing keys to anyone who asked.

The whole time, the data has been coming in, and we've been able to start seeing some interesting patterns in the noise. There are a fair number of people who have absolutely perfect feed readers. They show up at reasonable intervals, send the right header(s), and just work with no fuss. It's fantastic.

For those not currently participating, here's how it works. I send you a link to a web page with the instructions and a unique key that's just a bunch of random hex digits. The web page tells you how to turn that into a unique URL that can be handed to a feed reader.

Then, there's another key-based URL which shows the report. I provided a text dump of what it looked like in Thursday's post, and that's all it was at the time. I've since put in the effort to make it actually generate a Real Web Page, complete with actual HTML and all of this stuff (imagine that). Now it looks more like this:

I've removed the identifying marks (the key at the top, and the user-agent string at the bottom).

This program got a little frisky at startup and actually sent multiple identical unconditional requests for some reason. I don't know why, but it's not good behavior.

It also did some requests for some other made-up URLs that weren't supplied by the user (who was talking to me at the time), and I don't mean favicon or those apple-touch-icon things. I mean it took the URL provided by the user and glued on /other /stuff. That kind of stuff doesn't show up here, but I could add it later.

It's strange. Bum rushing a web server isn't a good first impression.

...

Separately, an anonymous reader asked if I would be willing to put up stats for the posts so they could see which ones were particularly popular. I actually don't really analyze that. I can tell on a basic level when things are popular because the "tail -f" of the logs really starts moving, and I can see an uptick on the bandwidth utilization.

But other than that, I have only my own random impressions of what's popular and what isn't based on remembering activity levels and the flavor of whatever feedback it might generate (or not). So, there's no data to provide, and I really don't want to build such a dataset anyway.

A post is a post. If it works out, that's great, but if not, eh. I have no particular reason to "tune" them. I don't have ads to serve, impressions to generate, or eyeballs to sell out to the highest bidder. It's just a whole mess of text.

Most of what I use my logging for is to handle abuse. Analytics? Feh.

The one exception is from 2020 when I noticed that a fair number of bug-tracking systems were leaking URLs (many internal) to specific posts about particular nerdly analyses of failure modes like "don't setenv in multi-threaded code" and "malloc(1213486160)" and all of this. The post I did about that links to the older posts and gives a little info on them and a few scant details about the incoming links. Such referrer data is almost completely dead now, since I just get bare hostnames without any paths. It's rare to see much more than that, so I don't expect a repeat of that post, well, ever.

They also asked for "like/dislike" and maybe more (FB style?) "reactions". I'm not likely to do that, either, for the same reason as why I don't have any kind of public comments: managing that kind of thing is serious work. If it's not managed and kept in the right groove, it will turn into a very bad place at the times of the week when people with lives are out living them, and THE ONE can run amok with nobody to tell them to shut the hell up. You know exactly which forums I'm talking about.

Also, I'm pretty sure that having me trying to operate the forum and yet also moderate it would quickly turn into yet another case of "don't do both" as I have few qualms about dropping the banhammer on bad behavior... but that tends to divide communities. (Been there, done that.)

Finally, I like the fact that all of this stuff is a forest of flat files. There are no connections to any sort of database in the serving path for the posts or the feed. That would be defeated if I added something to dip into a table and look to see how many "likes" it had before shoving the post out the door.

The entire /w/ path in the document root is about 151 MB at the moment. flicker (the current web server) is a monster that besides being physically massive also happens to have 128 GB of RAM in it. That means it can fit the entire thing into memory and basically keep it there. That means there isn't even a concern about how fast the "disks" are, since it only touches that stuff once per item.

You might have noticed that things are usually pretty snappy. That's a big part of why: it's doing as little work as possible.

If there was dynamic stuff going on, that wouldn't work nearly as well. For me to cross that particular divide, it's going to have to be for a very good reason.

The "feed reader score" service that I mentioned earlier this week is now up and running. Several people reached out to me and I have sent them their unique codes and links to the instructions so they can get started.

As of about a day ago, we are now logging metadata on the requests, and as of a couple of hours ago, the reports are starting to be built. I'm still adding features to it, but it's already pretty clear that not all feed readers are built to the same standards.

This is a sample report for the sort of real programs that I see plenty on the actual site all day every day:

Time span analyzed: about 17 hours (63734.284600 sec)
-------------------------------------
Number of log entries: 34
-------------------------------------
Conditional requests: 0
-------------------------------------
❗ Unconditional requests: 34
One is normal for feed initialization.
-------------------------------------
Good If-Modified-Since values: 0
-------------------------------------
❗ Bogus If-Modified-Since values: 34
-------------------------------------
Good If-None-Match values: 0
-------------------------------------
User agent: (34) <redacted>
-------------------------------------
Useless cookies: none!
-------------------------------------
Useless referrers: none!
-------------------------------------
Useless query parameters: none!
-------------------------------------
(Still evolving.  Check back later.)

I've removed the user-agent info here so you'll just have to guess at which software this might be.

As you can see, in the short time it's been reporting in, it always sends requests with broken If-Modified-Since values. This makes it get a full copy of the feed every time, and that makes it an unconditional request.

The average polling interval appears to be about 17-18 minutes. If it ran all day, it would make about 45 requests and would use about 25 MB of data that's all redundant since nothing's actually changing.

So, it begins! If you want to participate, send me some feedback and let me know. Also, to that one "wants to be a good citizen" person who indicated interest but didn't supply an e-mail address, can you try that again? Thanks.

Wow, okay, the "feed reader score" post has been making some waves. There have been some good discussions about it, and very few of the usual failure modes for those forums.

I heard from a bunch of people who are feed reader authors or just feed reader users who care deeply about doing the right thing. They actually want to point their program at my proposed "score" site to see what it's doing, and so they can find out if it's behaving badly somehow.

I like hearing this! This kind of stuff warms my heart.

So, I've started taking steps to invest in this project. To that end, I ran out, had a couple of slices of pizza and some root beer, then came back and started digging in my parts box. I grabbed the old Raspberry Pi B+ (star of the armv6 post from last fall) and started bringing it up to date.

I got it all configured as a proper outside-facing box, then I actually *hopped in the car* earlier and drove it out to the data center to plug it in. It's there now, just waiting for me to start doing stuff with it.

Why a separate box? Well, I've done this kind of stuff before, and it's easy for a "test target" to turn into a smoking crater. This way, if it goes really badly, I can just turn it off and it won't affect the other boxes out there. It isn't affected by whatever rate-limiting I might do on the "real" web server.

Also, it caps the amount of resources that can be spent by this project - it's a goofy little box that'll barely move a few tens of Mbps across its godawful NIC. It's not going to be haul-ass fast or anything like that, but it doesn't have to be. It just has to log the incoming requests. Something else gets to do the analysis to figure out whether the observed behavior is good, bad, or just plain meh.

To be clear, the "feed" this thing will be serving is not going to have any real posts in it. You won't be able to read my latest stuff by subscribing to it. Also, I will warn everyone right now that I fully expect to have to yank the entire sub-zone out of the domain multiple times after people ruin one, and I have to NXDOMAIN them early.

The URLs you get for testing are not expected to be long-term durable, in other words. It all depends on how much abuse it gets. Considering you can figure out what a feed reader is doing after a day or two at most, that's probably not going to be a problem.

Why am I writing about this? I'm trying something different: basically, if I talk about it first, will that light a fire under me to make stuff happen sooner? This is not normally how I operate. I normally just show up with something already done. This kind of project is just big enough to where that won't work.

I guess it's time to ignite a petroleum product at zero-hundred hours.

It's been well over a year since I started serving 429s to clients which are hitting the feed too often. Since then, much has happened, and most of it is generally good news.

I've heard from users and authors alike of feed software. Sometimes the users have filed bug reports and/or feature requests and have gotten positive results from the project (or vendor). Other times, the authors of such software have gotten in touch, did some digging, found a few nuances of how their libraries work, and improved the situation.

Some of them are trying but are still not quite making it right.

Here's some of what's been going on.

...

At least one reader improved to not send a date from 1800. Unfortunately, it's now sending the wrong value in its If-Modified-Since headers. Instead of sending the value it obtained from the Last-Modified header on the past fetch, it's using the value from the "updated" header in the feed itself.

These are different layers of the system, and you can't mix their values together. They're close, but not exactly the same. This is how it works.

I get the impression from stalking their issues that they don't really control their HTTP requests very well because it's done by some other library. That's where the 1800 thing came from in the first place. It sounds like yet another case of using libraries that really don't do their jobs properly.

...

There's another one which hits every 2 minutes without fail, and there's no way to change it. I've even installed that app on a test account and verified this myself - it's hard-coded. As a result, it's the one feed user-agent I've had to block outright. It doesn't stop requesting things even when it hits a brick wall of 403s. Clearly, I need to use a bigger hammer.

...

A fair number of people are sending conditional requests, but are doing it every 5 or 10 minutes. This is ridiculous. I don't write that often, and never have. Polling more is not going to get you anywhere, and indeed, will now get you delayed so you get your updates much later than the well-behaved people. Knock it off.

It seems like most of these come from things which appear to just be jammed into web browsers as some kind of extension. From hearing from at least one developer, it seems like they don't do conditional requests as a matter of course. This, despite being part of a web browser ecosystem which has understood the notion of a conditional request and caching things locally for nearly three decades. Amazing.

...

A while back, I added a "Retry-After: " header to the feed. Anyone who gets a 429 will also get intel on when they should try back. It's in seconds, so it'll be something like 3600 or 86400 depending on which kind of request was sent in the first place.

There are feed services which will actually reset their countdowns every time someone trips a 429. I'm not doing that. Yet.

This is why noticing and honoring that header matters.

...

Oh, here's a new thing: goofy programs that try to "guess" the feed URL. I see all kinds of stupid requests to paths that might have a feed on it. This is a new level of density on the part of the authors of those programs.

Here's the thing. I've had metadata in the top of every single /w/ post *and* its index since some time in 2012. It looks like this:

<link rel="alternate" type="application/atom+xml" href="/w/atom.xml">

If you view source on this post or any other on the web, you'll see it up there, just hanging out.

I did that way back then because browsers used to care about RSS and Atom, and they'd put that little yellow feed icon somewhere in the top bar when they spotted this sort of thing in a page. At least in the case of Firefox, you could click on it, and it would throw the target URL to a helper of your choice.

I wrote a feed reader system at the time (remember fred?), and indeed, I could click on that icon and it would flip the feed URL over to my "subscribe to new feed" handler. It was easy.

Then, something happened, and browsers gave up on feeds, and the icon disappeared. I kept it there anyway, figuring people would make use of it. It's still the right way to programmatically find out where to get an Atom feed for the content you're looking at.

So what's with all of the groping around in the dark with made-up URLs?

...

This one blows my mind. I put together a page which has the feed URL on it as just plain text, not a link. I've seen people paste it into their feed reader and include spaces and even newlines. Seriously!

I know this because I get requests for things like "/w/atom.xml%20" over and over from feed readers which obviously don't notice they get a 404 every time.

...

Now we get to the part where I pitch a way forward, and nobody takes me up on the offer. The idea is basically this: I get some kind of commitment and support from the people who do feed reader stuff, and in turn, I build a new kind of web site which amounts to a "feed reader correctness score".

It would probably work like this: you load up a page and it hands you a special (fake) feed URL that is keyed to you and you alone. You plug it into your feed reader program through whatever flow and it will keep track of every single request to that keyed URL.

Then, after it had collected data for a while, a report would eventually become available. Just off the top of my head, the kinds of things it might say could look like this:

* Poll history: 46 checks in the past 48 hours (average 62 minutes)

* Request types: (1) unconditional (45) conditional

* If-Modified-Since timestamps: (45) matches (0) made up from whole cloth

* ETag hashes: (45) matches (0) made up from whole cloth

* Useless cookies sent: none!

* Useless referrers sent: none!

* Useless CGI arguments sent: none!

* User-agents: (40) FooGronk/1.0 +http://fg.example.org/ (6) FooGronk/1.01 +http://fg.example.org/

That's the kind of stuff I'd expect to see from a nigh-perfect reader. It connects at a reasonable pace, it sends headers with correct values, and it doesn't send along stuff like cookies that I never set in the first place.

But, okay, this is nothing but vaporware unless someone actually wants it, is willing to support it, and will commit to take actions based what it says.

There's a bigger lesson here: don't measure stuff if nobody's going to take actions based on the results. It only ever ends in misery. I wanted to write a separate post about this very topic, but figured I'd give a preview of it right here.

Okay world, surprise me. Do the right thing.

Oh, yesterday was a barrel of laughs. I've said a lot that I hate hardware, and it's pretty clear that hardware hates me right back.

I have this old 2012-ish Mac Mini which has long since stopped getting OS updates from Apple. It's been through a lot. I upgraded the memory on it at some point, and maybe four years ago I bought one of those "HDD to SSD" kits from one of the usual Mac rejuvenation places. Both of those moves gave it a lot of life, but it's nothing compared to the flexibility I got by moving to Debian.

Then a couple of weeks ago, the SSD decided to start going stupid on me. This manifested as smartd logging some complaint and then also barking about not having any way to send mail. What can I say - it's 2024 and I don't run SMTP stuff any more. It looked like this:

Apr 29 07:52:23 mini smartd[1140]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 29 07:52:23 mini smartd[1140]: Sending warning via /usr/share/smartmontools/smartd-runner to root ...
Apr 29 07:52:23 mini smartd[1140]: Warning via /usr/share/smartmontools/smartd-runner to root produced unexpected output (183 bytes) to STDOUT/STDERR:
Apr 29 07:52:23 mini smartd[1140]: /etc/smartmontools/run.d/10mail:
Apr 29 07:52:23 mini smartd[1140]: Your system does not have /usr/bin/mail.  Install the mailx or mailutils package

Based on the "(pending)" thing, I figured maybe it would eventually reallocate itself and go back to a normal and quiet happy place. I ran some backups and then took a few days to visit family. When I got back, it was still happening, so I went to the store and picked up a new SSD, knowing full well that replacing it was going to suck.

Thus began the multi-hour process of migrating the data from the failing drive to the new one across a temporary USB-SATA rig that was super slow. Even though I was using tar (and not dd, thank you very much), it still managed to tickle the wrong parts of the old drive, and it eventually freaked out. ext4 dutifully failed into read-only mode, and the copy continued.

I was actually okay with this because it meant I didn't have to go to any lengths to freeze everything on the box. Now nothing would change during the copy, so that's great! Only, well, it exposed a neat little problem: Debian's smartmontools can't send a notification if it's pointed at a disk that just made the filesystem fail into read-only mode.

Yes, really, check this out.

May 14 20:04:47 mini smartd[1993]: Sending warning via /usr/share/smartmontools/smartd-runner to root ...
May 14 20:04:47 mini smartd[1993]: Warning via /usr/share/smartmontools/smartd-runner to root produced unexpected output (92 bytes) to STDOUT/STDERR:
May 14 20:04:47 mini smartd[1993]: mktemp: failed to create file via template ‘/tmp/tmp.XXXXXXXXXX’: Read-only file system
May 14 20:04:47 mini smartd[1993]: Warning via /usr/share/smartmontools/smartd-runner to root: failed (32-bit/8-bit exit status: 256/1)

There it is last night attempting to warn me that things are still bad (and in fact have gotten worse) ... and failing miserably. What's going on here? It comes from what they have in that smartd-runner script. Clearly, they meant well, but it has some issues in certain corner cases.

This is the entirety of that script:

#!/bin/bash -e

tmp=$(mktemp)
cat >$tmp

run-parts --report --lsbsysinit --arg=$tmp --arg="$1" \
    --arg="$2" --arg="$3" -- /etc/smartmontools/run.d

rm -f $tmp

Notice run-parts. It's an interesting little tool which lets you run a bunch of things that don't have to know about each other. This lets you drop stuff into the /etc/smartmontools/run.d directory and get notifications without having to modify anything else. When you have a bunch of potential sources for customizations, a ".d" directory can be rather helpful.

But, there's a catch: smartd (well, smartd_warning.sh) fires off this giant multi-line message to stdout when it invokes that handler. The handler obviously can't consume stdin more than once, so it first socks it away in a temporary file and then hands that off to the individual notifier items in the run.d path. That way, they all get a fresh copy of it.

Unfortunately, mktemp requires opening a file for writing, and it tends to use a real disk-based filesystem (i.e., whatever's behind /tmp) to do its thing. It *could* be repointed somewhere else with either -p or TMPDIR in the environment (/dev/shm? /run/something?), but it's not.

This is another one of those "oh yeah" or "hidden gotcha" type things. Sometimes, the unhappy path on a system is *really* toxic. Things you take for granted (like writing a file) won't work. If you're supposed to operate in that situation and still succeed, it might take some extra work.

As for the machine, it's fine now. And hey, now I have yet another device I can plug in any time I want to make smartd start doing stuff. That's useful, right?

...

One random side note: you might be wondering how I have messages from the systemd journal about it not being able to write to the disk. I was storing this stuff to another system as it happened, and it's in my notes, but I just pulled this back out of journalctl right now, and it hit me while writing this. Now I'm wondering how I have them, too!

Honestly, I have no idea how this happened. Clearly, I have some learning to do here. How do you have a read-only filesystem that still manages to accept appends to the systemd journal? Where the hell does that thing live?

The box has /, /boot, /boot/efi, and swap. / (dm-1) went readonly. The journals are in /var/log/journal, which is just part of /.

If a tree falls in a forest and nobody's around...

...

Late update: yeah, okay, I missed something here. I'm obviously looking at the new SSD on the machine now, right? That SSD got a copy of whatever was readable from the old one, which turned out to be the entire system... *including* the systemd journal files.

Those changes weren't managing to get flushed to the old disk with the now-RO filesystem, but they were apparently hanging out in buffers and were available for reading... or something? That makes sense, right?

So, any time I copied something from the failing drive, I was scooping up whatever it could read from that filesystem. The telling part is that while these journals do cover the several hours it took to copy all of the stuff through that USB 2->SATA connection, they don't include the system shutdown. Clearly, that happened *after* the last copy ran. Obviously.

Now, if those journal entries had made it onto the original disk, then it would mean that I have a big hole in my understanding of what "read-only filesystem" means even after years of doing this. That'd be weird, right?

Just to be really sure before sending off this update, I broke out the failing SSD and hooked it up to that adapter again, then went through the incantations to mount it, and sure enough:

-rw-r-----+ 1 root systemd-timesync 16777216 May 14 17:06 system.journal

The last entry in that log is this:

May 14 17:06:38 mini kernel: ata1: EH complete

There we go. Not so spooky after all.

It's time for some responses to reader feedback.

One person mentions that running git log with "--full-history" should show the change that's buried within the commit that also was a merge. Unfortunately, that's not it. I tried that and a bunch of other things before landing on "git show" the other night. They did mention that it might be some other flag.

That plus help from a friend who's clearly been down this road before turned up the right answer: you probably want to add -c or -cc. Now, if you go looking for those in the actual manual for git log, you'll trip over the usual problem that searching for "-c" matches some other term that starts with c, in this case, clear-decorations. So, if you then search for " -c" (note: leading space only), then you'll find this first:

Note that unless one of --diff-merges variants (including short -m, -c, and --cc options) is explicitly given, merge commits will not show a diff, even if a diff format like --patch is selected, nor will they match search options like -S. The exception is when --first-parent is in use, in which case first-parent is the default format.

... and now you're getting somewhere.

Why this isn't the default, I have no idea. You're asking it to show a patch, so show a patch, already!

...

I saw some other comments saying that I should report some of the wacky brokenness that I tripped over, and I need to respond to that.

It's been made very clear to me that plenty of people don't give a shit about what I report as "broken" in their projects. This isn't necessarily about the world of free software/open source, either. It's happened plenty of times at actual jobs when I was there specifically to find badness.

Case in point: someone apparently had a conversation along the lines of "hey, it would be bad if the <redacted spooky people> got their hands on this stuff and it didn't work, so let's have someone who isn't one of the devs test this out first". Then they got a hold of me and ask me to Do some Stuff with their products.

I got my hands on the hardware and then the software, and in so doing, found all kinds of crazy problems in it, almost like nobody had actually tried to use the full extent of the hardware with their driver software.

I started asking questions of the engineers, trying to figure out what they have done. At some point they went radio-silent and I started getting responses from their boss. At that point I thought "it's good to be a contractor" and referred it to MY boss - the person who hired me for the gig. It was clearly an internal matter and not anything for me to deal with. (One of the rare perks of being a contractor, gotta say.)

So, back to what happened the other night. Did I find some badness in pkg-config or pkgconf or whatever variant they're using? I probably did! Could it be exploited? Maybe? Probably? Do I care that much? Not really. Why? Because *far too few people* actually give a shit about this kind of correctness.

This is fairly common when you set me loose in an ecosystem: I find all kinds of dumb "mosquito bite" things that make things less than pleasant for someone who isn't expecting it. They're the things that experienced users know how to route around because they start accepting it as "normal".

If my "trouble reports" tend to be seen as more of a burden than a value, why would I keep generating them?

For those few people out there who truly care about this sort of thing, obviously I love and cherish you and people who do what you do. Please don't think I'm painting you with the same brush. Just realize that you are very much in the minority, and I have no idea if I'm going to encounter one of you (unlikely) or one of the others (very likely) when I file a report.

Another thing: I'm not paying any of these people, so why should they listen to me? There are exactly zero consequences for them if my stuff goes ignored. They don't work for me, and it's not like I could fire them or somehow use some other leverage in order to make things happen.

...

I believe that really good products have at least one person behind them who's spotting all of the goofiness and is having some success in getting those things ironed out. It's not enough to just notice them, and it's not enough to do the kind of gunky eng work it takes to find out why something happened or how it broke.

Whoever does this work has to actually have traction in the organization so that these problems will actually be fixed. Otherwise, the first two parts have no real value, and only serve to burn out whoever's making the attempts.

I'd love to do this kind of work for a place that truly cares about it, to the point of providing air cover to those who are tasked with finding the problems. If the company doesn't actually care, that's "fine". Don't hire people who are tasked with trying to improve those things, then if there's a hypocrisy gap in your engineering culture.

Also, I get that companies change, and what's valued today is not what will be valued tomorrow. It seems like the least a good manager would do is *explicitly* say as much when that point arrives, so you can cut and run instead of trying to push back an unstoppable tide. When the company shifts from engineering, reliability and delighting the customers to growth at all costs and exploiting the customers, it's probably time to go.

Is anyone actually doing this now? I'd love to hear about it if so.

...

Finally, to the handful of people who say that the results of ./configure "will change all the time" because of the kernel or libc version changing... that's just wrong. You're making that up based on a technicality that does not actually track with real-world use of our systems.

I'm going to guess that most of the people who are saying this are going based on the fact that you CAN change the C library or kernel arbitrarily, but probably never have. Meanwhile, I was there way back in the day when we first got ELF kernels in the 1.3 world and had to install a new compiler in order to link it in the first place, among plenty of other godawful things we had to do back in the 90s. I've been there, and I'm telling you, this is not the way of the world right now. Systems just don't change that often. When they do change to that extent, it tends to be associated with a major version bump (unless someone's mighty clueless).

The whole point of having things set up for you by the OS builder is that it takes the kernel, C library, and other localizations into account. If someone is changing those things to where syscalls and library calls are shifting materially, they are essentially signing up to maintain their own OS variant, and should update the "how you do things on this OS" data at the same time if it affects such things.

This *already* happens. Companies take a base OS (like CentOS), and then they run a far newer kernel and/or glibc atop it for their own reasons. This forces a whole raft of other changes to take advantage of the new stuff: new versions of perf, strace, ethtool, mcelog or whatever else. I know this because I did this to help out as a certain employer went from CentOS 5 to 6 to 7 over the span of several years.

Their build systems already know how to build stuff on this hybrid environment, particularly to use the overlay glibc instead of the system one. So, someone is maintaining this, and they track whatever changes when it actually matters.

I keep seeing people saying that the answer to my complaints about autoconf is to rub *more* autoconf on the problem. I don't like this. In the general vein of "this should not be that hard", I decided to revisit something from two years ago and tried to use my build tool to generate my stuff on a fresh BSD-flavored install. (The exact flavor is unimportant here, and mentioning it by name would only trigger the weenies in the crowd, so I won't.)

I wanted to prove to myself that yes, my stuff can Just Build on other (i.e., not Linux or Mac) systems without resorting to the kinds of stuff that I wish we had collectively left behind in the 90s.

The OS itself was fine. The install process on a throwaway VM image was quick and painless. I knew how to get my usual tools installed - bash, nano, that kind of thing. Unlike last time, I opted to not do X and just focused on getting my stuff to build.

But then I made a mistake: I told it to install "protobuf" since I use that library in my build tool. That actually installed "protobuf-24.4,1" which is some insane version number I'd never seen before. All of my other systems are all running 3.x.x type versions.

Now, realize, I didn't know this was a mistake yet, and kept on going. I did manage to bootstrap my build tool into a usable binary, and then started a "build the world" process, at which point it blew up. It was complaining about not being able to find a library called "google/protobuf/arena" inside my personal source tree.

This made no sense, so I started digging, and found out that the "protoc" compiler in that version of the software spits out code like this:

#include "google/protobuf/thing1.h"
#include "google/protobuf/thing2.h"
#include "google/protobuf/thing3.h"

... you get the idea. It's a third-party library that's installed at the system level, and yet it's using "" like it's all chummy and hanging out with your code in your local repo. Yeah, no. That's wrong. They should be <> includes, like this:

#include <google/protobuf/thing1.h>
#include <google/protobuf/thing2.h>
#include <google/protobuf/thing3.h>

What's weird is... it *is* that way on all of my other machines - my Mac with Macports and my Debian/Raspbian boxes all generate those #includes with <> like they're supposed to, and everything Just Works.

I won't lie. This really made me angry at first. I was like, okay, they did yet another stupid thing upstream, and now everyone else is going to have to work around it. It got me thinking thoughts like "just how hard would it be to NOT use protobuf, anyway". I figured that this abomination would eventually filter down to Macports and Debian's apt repo and whatnot, and then I'd have to deal with it, or toss it.

After a few minutes of cooling off, it occurred to me that I could do something super-duper obnoxious: wrap protoc, and run a nasty little sed command afterward to flip the "" to <>. So I did that, and things proceeded. Awful.

Of course, then I ran into some other fun problems with my code, like IPPROTO_* definitions not being available. I have a wrapper for getaddrinfo() and it uses IPPROTO_TCP in the .ai_protocol field. I had all of the #includes that the man pages say to have for using that function, but that's not enough on this particular system.

I assume that there's some transitive #include on Macs and on glibc-flavored Linuxes that drags this in for me, but on this one BSD it doesn't work that way. The fix was simple enough, and mighty stupid:

#include <netinet/in.h>

And no, that's not listed in their getaddrinfo(3) manual page, even though IPPROTO_UDP and _TCP are both explicitly mentioned in it. Dig around online and you'll find this tripping up other people. That's the extent of my self-inflicted damage that had to be fixed to make it build: lack of a few #includes.

Stuff like this is why I tend to wall off calls into the C library with a bunch of compatibility gunk and then use my own interfaces above that.

At some point during this, I decided to go back into the protobuf git repo to see just when they decided to dump the angle brackets in favor of the double-quotes, and that's when I hit another wall of stupid. Apparently it's possible to change a git repo in such a way that "git log -p" will never show it. Did you know that? Before yesterday, I definitely didn't.

Here's how I discovered this: obviously, there was code that would do the <> stuff at some point. The last version of it I could find looked like this:

  std::string left = "\"";
  std::string right = "\"";
  if (use_system_include) {
    left = "<";
    right = ">";
  }
  return left + name + right;

It seems simple enough, if a little goofy: return "input" unless use_system_include gets set a few lines up, in which case it should return <input>. No big deal, right?

But... that code exists nowhere in the repo as it stands now. Silly naive me, I thought I could just "git log -p" and do a / search in less for "use_system_include" to find the commit which dropped it. I wanted to learn why they did this, because maybe they had a good reason, or basically, if I complained about it, what I would be up against.

I found nothing.

This started a terrible sequence where I started checking out different commits from the tree to see what it looked like at various points in the past. I got it down to a commit that contained the above code, and then one commit past that dropped it.

This must be it, right? I should be able to "git log -p" and see it, right? Nope.

commit d85c9944c55fb38f4eae149979a0f680ea125ecb (HEAD)
Merge: 7764c864b 0264866ce
Author: <removed because it's not their fault>
Date:   Mon Sep 19 14:10:44 2022 -0700

    Sync from Piper @475378801
    
    PROTOBUF_SYNC_PIPER

The next line in the git log output is the next commit. There's no "body" to this commit. It's just a "Merge:" and two other commits.

7764c864b and 0264866ce, right? I should be able to sync to those with git checkout and see which one dropped it, yeah? Well, I'll spare you the effort and just say that BOTH OF THEM have the old code in it.

So... this commit somehow drops the code even though it's merging two "ancestral commits" that both contain it, and there's no diff shown.

Confusing, right?

I don't know how I finally figured this out, but after a whole lot of cursing and thrashing, I found "git show <commit>" will FINALLY give me the results I want, ish. It contains the change which dumped the <> code and put in the new stuff.

--  std::string left = "\"";
--  std::string right = "\"";
--  if (use_system_include) {
--    left = "<";
--    right = ">";
--  }
--  return left + name + right;
++  return absl::StrCat("\"", basename, "\"");

There's no explanation or other context. Presumably that all got squashed out when it was exported from whatever they use internally.

"Why" is gone. I just have "when", and that's not very interesting: it was merged in September 2022, ho hum. That just means that whenever Linux distributions and Macports catch up with at least that point, I'm going to have to deal with this for real.

Oh, there's one more bit of batshittery which needs to be mentioned here. My stuff uses pkg-config to find out how to compile and link against these libraries, right? Well, when it was using this oh-so-new protobuf version, the commands it was running were so long, it was scrolling off my standard 80x25 terminal.

$ pkg-config --cflags protobuf | wc -c
    4326

Yep! 4 KB of cflags. Here's just the top part of it:

# pkg-config --cflags protobuf
-I/usr/local/include -DPROTOBUF_USE_DLLS -Wno-float-conversion 
-DNOMINMAX -Wno-float-conversion -DNOMINMAX -Wno-float-conversion 
-DNOMINMAX -Wno-float-conversion -DNOMINMAX -Wno-float-conversion 
-DNOMINMAX -Wno-float-conversion -DNOMINMAX -Wno-float-conversion 
-DNOMINMAX -Wno-float-conversion -DNOMINMAX -Wno-float-conversion 

... and it just goes on like this. It actually worked, though!

Finally, remember when I said that I made a problem by installing their "protobuf" package without realizing it? Yeah, it turns out they actually also have "protobuf3" which is a nice sane version just like the ones on my other machines, #include <...> and all. So, I removed the bad one, installed this other one, and dropped my sed hack.

What a night.

I have a story about paper cuts when using a little Linux box.

One of my sites has an older Raspberry Pi installed in a spot that takes some effort to access. A couple of weeks ago, it freaked out and stopped allowing remote logins. My own simple management stuff was still running and was reporting that something was wrong, but it wasn't nearly enough detail to find out exactly what happened.

I had to get a console connected to it in order to find out that it was freaking out about its filesystem because something stupid had apparently happened to the SD card. I don't know exactly why it wouldn't let me log in. Back in the old days, you could still get into a machine with a totally dead disk as long as enough stuff was still in the cache - inetd + telnetd + login + your shell, or sshd + your shell and (naturally) all of the libraries those things rely on. I guess something happened and some part of the equation was missing. There are a LOT more moving parts these days, as we've been learning with the whole xz thing. Whatever.

So I rebooted it, and went about my business, and it wasn't until a while later that I noticed the thing's clock was over a day off. chrony was running, so WTF, right? chrony actually said that it had no sources, so it was just sitting there looking sad.

This made little sense to me, given that chrony is one of the more clueful programs which will keep trying to resolve sources until it gets enough to feel happy about using them for synchronization. In the case of my stock install, that meant it was trying to use 2.debian.pool.ntp.org.

I tried to resolve it myself on the box. It didn't work. I queried another resolver (on another box) and it worked fine. So now what, on top of chrony not working, unbound wasn't working too?

A little context here: this box was reconfigured at some point to run its own recursive caching resolver for the local network due to some other (*cough* TP-Link *cough*) problems I had last year. It was also configured to *only* use that local unbound for DNS resolution.

This started connecting some of the dots. chrony wasn't setting the clock because it couldn't resolve hosts in the NTP pool. It couldn't resolve hosts because unbound wasn't working. But, okay, why wasn't unbound working?

Well, here's the problem - it *mostly* was. I could resolve several other domains just fine. It's just that ntp.org stuff wasn't happening.

(This is where you start pointing at the screen if this has happened to you before.)

So, what would make only some domains not resolve... but not all of them... on a box... with a clock that's over a day behind?

Yeah, that's about when it fit together. I figured they must be running DNSSEC on that zone (or some part of it), and it must have a "not-before" constraint on some aspect of it. I've been down this road before with SSH certificates, so why not DNS?

I added another resolver to resolv.conf, then chrony started working, and that brought the time forward, and then unbound started resolving the pool, and everything else returned to normal.

By "everything else", I also mean WireGuard. Did you know that if your machine gets far enough out of sync, that'll stop working, too? I had no idea that it apparently includes time in its crypto stuff, but what other explanation is there?

Backing up, let's talk about what happened, because most of this is on me.

I have an old Pi running from an SD card. It freaked out. It took me about a day and a half to get to where it was so I could start working on fixing it.

This particular Pi doesn't have a real-time clock. The very newest ones (5B) *do*, but you have to actually buy a battery and connect it. By default, they are in the same boat. This means when they come up, they use some nonsense time for a while. I'm not sure exactly what that is offhand, because...

systemd does something of late where it will try to put the clock back to somewhere closer to "now" when it detects a value that's too far in the past. I suspect it just digs around in the journal, grabs the last timestamp from that, and runs with it. This is usually pretty good, since if you're just doing a commanded reboot, the difference is a few seconds, and your time sync stuff fixes the rest not long thereafter.

But, recall that the machine sat there unable to write to its "disk" (SD card) for well over a day, so that's the timestamp it used. If I had gotten there sooner, I guess it wouldn't have been so far off, but that wasn't an option.

Coming up with time that far off made unbound unable to resolve the ntp.org pool servers, and that made chrony unable to update the clock... which made unbound unable to resolve the pool servers... which...

My own configuration choice which pointed DNS resolution only at localhost did the rest.

So, what now? Well, first of all, I gave it secondary and tertiary resolvers so that particular DNS anomaly won't be repeated. Then I explicitly gave chrony a "peer" source of a nearby host (another Pi, unfortunately) which might be able to help it out in a pinch even if the link to the outside isn't up for whatever reason.

There's a certain problem with thinking of these little boxes as cheap. They are... until they aren't. To mangle a line from jwz, a Raspberry Pi is only cheap if your time has no value.

As usual, this post is not a request for THE ONE to show up. If you are THE ONE, you don't make mistakes. We know. Shut up and go away.

I've gotten a few bits of feedback asking for my thoughts and/or reactions to the whole "xz backdoor" thing that happened over the past couple of days. Most of my thoughts on the matter apply to autoconf and friends, and they aren't great.

I don't have to cross paths with those tools too often these days, but there was a point quite a while back when I was constantly building things from source, and a ./configure --with-this --with-that was a given. It was a small joy when the thing let me reuse the old configure invocation so I didn't have to dig up the specifics again.

I got that the whole reason for autoconf's derpy little "recipes" is that you want to know if the system you're on supports X, or can do Y, or exactly what flavor of Z it has, so you can #ifdef around it or whatever. It's not quite as relevant today, but sure, there was once a time when a great many Unix systems existed and they all had their own ways of handling stuff, and no two were the same.

So, okay, fine, at some point it made sense to run programs to empirically determine what was supported on a given system. What I don't understand is why we kept running those stupid little shell snippets and little bits of C code over and over. It's like, okay, we established that this particular system does <library function foobar> with two args, not three. So why the hell are we constantly testing for it over and over?

Why didn't we end up with a situation where it was just a standard thing that had a small number of possible values, and it would just be set for you somewhere? Whoever was responsible for building your system (OS company, distribution packagers, whatever) could leave something in /etc that says "X = flavor 1, Y = flavor 2" and so on down the line.

And, okay, fine, I get that there would have been all kinds of "real OS companies" that wouldn't have wanted to stoop to the level of the dirty free software hippies. Whatever. Those same hippies could have run the tests ONCE per platform/OS combo, put the results into /etc themselves, and then been done with it.

Then instead of testing all of that shit every time we built something from source, we'd just drag in the pre-existing results and go from there. It's not like the results were going to change on us. They were a reflection of the way the kernel, C libraries, APIs and userspace happened to work. Short of that changing, the results wouldn't change either.

But no, we never got to that point, so it's still normal to ship a .tar.gz with an absolute crap-ton of dumb little macro files that run all kinds of inscrutable tests that give you the same answers that they did the last time they ran on your machine or any other machine like yours, and WILL give the same answers going forward.

That means it's totally normal to ship all kinds of really crazy looking stuff, and so when someone noticed that and decided to use that as their mechanism for extracting some badness from a so-called "test file" that was actually laden with their binary code, is it so surprising that it happened? To me, it seems inevitable.

Incidentally, I want to see what happens if people start taking tarballs from various projects and diff them against the source code repos for those same projects. Any file that "appears" in the tarball that's allegedly due to auto[re]conf being run on the project had better match something from the actual trees of autoconf, automake, ranlib, gettext, or whatever else goofy meta-build stuff is being used these days.

$ find . -type f | sort | xargs sha1sum
7d963e5f46cd63da3c1216627eeb5a4e74a85cac  ./ax_pthread.m4
c86c8f8a69c07fbec8dd650c6604bf0c9876261f  ./build-to-host.m4
0262f06c4bba101697d4a8cc59ed5b39fbda4928  ./getopt.m4
e1a73a44c8c042581412de4d2e40113407bf4692  ./gettext.m4
090a271a0726eab8d4141ca9eb80d08e86f6c27e  ./host-cpu-c-abi.m4
961411a817303a23b45e0afe5c61f13d4066edea  ./iconv.m4
46e66c1ed3ea982b8d8b8f088781306d14a4aa9d  ./intlmacosx.m4
ad7a6ffb9fa122d0c466d62d590d83bc9f0a6bea  ./lib-ld.m4
7048b7073e98e66e9f82bb588f5d1531f98cd75b  ./lib-link.m4
980c029c581365327072e68ae63831d8c5447f58  ./lib-prefix.m4
d2445b23aaedc3c788eec6037ed5d12bd0619571  ./libtool.m4
421180f15285f3375d6e716bff269af9b8df5c21  ./lt~obsolete.m4
f98bd869d78cc476feee98f91ed334b315032c38  ./ltoptions.m4
530ed09615ee6c7127c0c415e9a0356202dc443e  ./ltsugar.m4
230553a18689fd6b04c39619ae33a7fc23615792  ./ltversion.m4
240f5024dc8158794250cda829c1e80810282200  ./nls.m4
f40e88d124865c81f29f4bcf780512718ef2fcbf  ./po.m4
f157f4f39b64393516e0d5fa7df8671dfbe8c8f2  ./posix-shell.m4
4965f463ea6a379098d14a4d7494301ef454eb21  ./progtest.m4
15610e17ef412131fcff827cf627cf71b5abdb7e  ./tuklib_common.m4
166d134feee1d259c15c0f921708e7f7555f9535  ./tuklib_cpucores.m4
e706675f6049401f29fb322fab61dfae137a2a35  ./tuklib_integer.m4
41f3f1e1543f40f5647336b0feb9d42a451a11ea  ./tuklib_mbstr.m4
b34137205bc9e03f3d5c78ae65ac73e99407196b  ./tuklib_physmem.m4
f1088f0b47e1ec7d6197d21a9557447c8eb47eb9  ./tuklib_progname.m4
86644b5a38de20fb43cc616874daada6e5d6b5bb  ./visibility.m4
$ 

... there's no build-to-host.m4 with that sha1sum out there, *except* for the bad one in the xz release. That part was caught... but what about every other auto* blob in every other project out there? Who or what is checking those?

And finally, yes, I'm definitely biased. My own personal build system has a little file that gets installed on a machine based on how the libs and whatnot work on it. That means all of the Macs of a particular version of the OS get the same file. All of the Debian boxes running the same version get the same file, and so on down the line.

I don't keep asking the same questions every time I go to build stuff. That's just madness.

There was this team which was running a pretty complicated data storage, leader election and "discovery" service. They had something like 3200 machines and had something like 300 different clusters/cells/ensembles/...(*) running across them. This service ran something kind of like etcd, only not that.

The way it worked was that a bunch of "participant" machines would start an election process, and then they'd decide who was going to lead them for a while. That leader got to handle all of the write traffic and it did all of the usual raft/paxos-ish spooky coordination stuff amongst the participants, including updating the others, and dealing with hosts that go away and come back later, and so on. It's all table stakes for this kind of service.

This group of clusters had started out relatively simple but had grown into a monster over the years. Nobody probably expected them to have hundreds of clusters and thousands of machines, but they now did, and they were having trouble keeping track of everything. There were constant outages, and since they were so low in the stack, when they broke, lots of other stuff broke.

I wanted to know just what the ground truth looked like and so started something really stupid from my development machine. It would take a list of their servers and would crawl them, interrogating the TCP ports on which the service ran. This was only about 10 ports per machine, so while it sounded obnoxiously high, it was still possible for prototyping purposes.

On these ports, there were simple text-based commands which could be sent, and it would return config information about what that particular instance was running. It was possible to derive the identity of the cluster from that. Given all of this and a scrape of the entire fleet, it was possible to see which host+port combinations were actually supporting any given cluster, and thus see how well they were doing.

Early results from this terrible manual scraping started showing promise. Misconfigurations were showing up all over the place - clusters that are supposed to have 5 hosts but only have 3 in practice with the other two missing in action somewhere, clusters with non-standard host counts, clusters in the wrong spots, and so on.

To get away from the "printf | nc the world in cron" thing, we wound up writing this dumb little agent thing that would run on all of the ~3200 hosts. It would do the same crawling, but it would happen over loopback so it was a good bit faster by removing long hauls over the production network from the equation. It also took the load of polling ~32000 ports off my singular machine, and was inherently parallel.

It was now possible to just query an agent and get a list of everything running on that box. It would refresh things every minute, so it was far more current than my terrible script which might run every couple of hours (since it was so slow). This made things even better, and so we needed an aggregator.

We did some magic to make each of these agents create a little "beacon" somewhere any time they were run. Our "aggregator" process would start up and would subscribe to the spot where beacons were being created. It would then schedule the associated host for checks, where it would speak to the agent on that host and ask for a copy of its results.

So now we had an agent on every one of the ~3200 hosts, each polling 10 local ports, plus an aggregator that talked to the ~3200 agents and refreshed the data from them.

Finally, all of the data was available in one place with a single query that was really fast. The next step was to write a bunch of simple "dashboard" web pages which allowed anyone to look at the entire fleet, or to narrow it down by certain parameters - a given cluster (of these servers), a given region, data center, whatever.

With all of this visible with just a few clicks, it was pretty clear that we needed something more to actually find the badness for us. It was all well and good to go clicking around while knowing what things are supposed to look like, but there were supposed to be rules about this sort of thing: this many hosts in a cluster, no more than N hosts per failure domain, and more.

...

Failure domains are a funny thing. Let's say you have five hosts which form a quorum and which are supposed to be high-availability. You'd probably want to spread them around, right? If they were serving clients from around the world, maybe you'd put them in different locations and never put two in the same spot? If something violated that, how would you know?

Here's an example of bad placement. We had this one cluster which was supposed to be spread out throughout an entire region which was composed of multiple datacenter buildings, each with multiple (compute) clusters in it, with different racks and so on down the line. But, because it had been turned up early in the life of that region when only a handful of hosts had existed, all of them were in the same two or three racks.

Worse still, those racks were physically adjacent. Put another way, if the servers had arms and hands, they could have high-fived each other across the hot and cold aisles in the datacenter suite. That's how close together they were. One bad event in a certain spot would have wiped out all of their data.

We had to write a schema which would let us express limits for a given cluster - how many regions it should be in, the maximum number of members per host, rack, (compute) cluster, building, region, etc. Then we wrote a tool to let us create rules, and then started using that to churn out rulesets. Next we came up with some tools which would fetch the current state of affairs (from the agent/aggr combo) and compare it to the rulesets. Anything out of "compliance" would show up right away.

...

Then there was the problem of managing the actual ~3200 hosts. With a footprint that big, there's always something happening. A location gets turned up and new hosts appear. Another location is taken down after the machines get too old and those hosts go away. We kept having outages where a decom would be scheduled, and then someone far away would run a script with a bunch of --force type commands, and it would just yank the machines and wipe them. It had no regard for what they were actually doing, and they managed to clobber a bunch of stuff this way. It just kept happening.

This is when I had to do something that does not scale. I said to the decom crew that they should treat any host owned by this team as off limits because we do not have things under control. That means never *ever* running a decom script against these hosts while they are still owned by the team.

I further added that while we're working to get things under control, if for some reason a decom is blocked due to this decree of mine, they are to contact me, any time of day or night, and I will get them unblocked... somehow. I figured it was my way of showing that I had "skin in the game" for making such a stupid and unreasonable demand.

I've often said that the way to get something fixed is to make sure someone is in the path of the badness so they will feel it when something screws up. This was my way of doing exactly that.

We stopped having decom-related outages. We instead started having these "fire drill" type events where one or two people on the team (and me) would have to drop what they were doing and spend a few hours manually replacing machines in various clusters to free them up.

Obviously, this couldn't stand, and so we started in on another project. This one was more of a "fleet manager", where a dumb little service would keep track of which machines the team owned, and it would store a series of bits for each one that I called "intents".

There were only three bits per host: drain, release, freeze. Not all combinations were valid.

If no bits were set on a host, that meant it was intended for production use. If it has a server on it, that's fine. If someone needs a replacement, it's potentially available (assuming it meets the other requirements, like being far enough away from the other participants).

If the "drain" bit was set, that meant it was not supposed to be serving. Any server on it should be taken off by replacing it with an available host which itself isn't marked for "drain" (or worse).

The "release" bit meant that if a host no longer had anything running on it, then it should be released back to the machine provisioning system. In doing this, the name of the machine changed, and thus the ownership (and responsibility) for it left the team, and it was no longer our problem. The people doing decoms would take it from there.

"Freeze" was a special bit which was intended as a safety mechanism to stop a runaway automation system. If that bit was set on a host, none of the tools would change anything on it. It's one of those things where you should never need to use it, but you'll be sorry if you don't write it and then need it some day.

"Drain" + "release" meant "keep trying to kick instances off this host and don't add any new ones", and then "once it becomes empty, give it back".

Other combinations of the bits (like "release" without "drain") were invalid and were rejected by the automation.

I should note that this was meant to be level-triggered, meaning on every single pass, if a host had a bit set and yet wasn't matching up with that intent or those intents, something should try to drain it, or give it away, or whatever. Even if it failed, it should try again on the next pass, and failures should be unusual and thus reported to the humans.

...

Then there was also the pre-existing system which took config files and used it to install instances on machines. This system worked just fine, but it only did that part of the process. It didn't close the loop and so many parts of the service lifecycle wound up unmanaged by it.

Looking back at this, you can now see that we could establish a bunch of "sets" with the data available.

Configs: "where we told it to run"

Agent + aggregator: "where it's actually managing to run"

Checker: "what rules these things should be obeying"

Fleet manager: "which machines should be serving (or not), which machines we should hang onto (or give back)".

Doing different operations on those sets yielded different things.

[configs] x [agent/aggr] = hosts which are doing what they are supposed to be doing, hosts which are supposed to be serving but aren't for some reason, and hosts which are NOT supposed to be running but are running it anyway. It would find sick machines, failures in the config system, weird hand-installed hack jobs in dark corners, and worse.

[agent/aggr] x [checker] = clusters which are actually spread out correctly, and clusters which are actually spread out incorrectly, (possibly because of bad configs, but could be any reason).

[agent/aggr] x [fleet manager] = hosts which are serving where that's okay, hosts which need to be drained until empty, and hosts which are now empty and can be given back.

[configs] x [checker] = are out-of-spec clusters due to the configs telling them to be in the wrong spot, or is something else going on? You don't really need to do this one, since if the first one checks out, then you know that everything is running exactly what it was told to run.

[configs] x [fleet manager] = if you ever get to a point where you completely trust that the configs are being implemented by the machines (because some other set operations are clear), then you could find mismatches this way. You wouldn't necessarily have to resort to the empirical data, and indeed, could stop scanning for it.

For that matter, the whole port-scanning agent/aggr combination shouldn't have needed to exist in theory, but in practice, independent verification was needed.

I should point out that my engagement with this team was not viewed kindly by management, and my reports about what had been going on ultimately got me in trouble more than anything else. It's kind of amazing, considering I was working with them as a result of a direct request for reliability help, but shooting the messenger is nothing new. This engagement taught me that a lot of so-called technical problems are in fact rooted in human issues, and those usually come from management.

There's more that happened as part of this whole process, but this post has gotten long enough.

...

(*) I'm using "clusters" here to primarily refer to the groups of 5, 7, or 9 hosts which participated in a quorum and kept the state of the world in sync. Note that there's also the notion of a "compute cluster", which is just a much larger group of perhaps tens of thousands of machines (all with various owners), and that does show up in this post in a couple of places, and is called out explicitly when it does.

Oh dear, it's popcorn for breakfast yet again. Another outage in a massive set of web sites.

It's been about 10 years, so let's talk about the outage that marks the point where I started feeling useful in that job: Friday, August 1, 2014. That's the one where FB went down and people started calling 911 to complain about it, and someone from the LA County sheriff's office got on Twitter to say "knock it off, we know and it's not an emergency".

Right, so, it's been well-documented what happened that day, even on the outside world - SRECon talks, a bunch of references in papers, you name it. It was time for "push", and as it was being seeded, that process pretty much consumed all of the available memory (and swap) on the smallest machines.

Then there was this program which ran on every box as root, and its job was to run a bunch of awful subprocesses, capture their outputs, parse them somewhat, and ship the results to a time series database or a logging system. This program is the one that had the infamous bug in it where it would call fork() and saved the return value, but didn't check it for failure: the -1 retval.

So, later on, it went to kill this "child process" that never started, and did the equivalent of 'kill -9 -1', and on Linux, that whacks everything but yourself and pid 1 (init). Unsurprisingly, this took down the web server and pretty much everything else. This was pre-systemd on CentOS 5 machines running Upstart, so the only things that "came back" were the "respawn" entries in inittab, like [a]getty on the text consoles.

This is how we were able to fire up a remote console on one of the affected machines and log in and see that there was basically init, the shell that had just been started, and this fbagent process which was responsible for assassinating the entire system that morning.

The rest of the story has also been told, which is where it took me a couple of weeks to figure out why we kept losing machines this way, and when I did, I found the source had already been patched. Another engineer unrelated to the fbagent project had been hitting the same problem, decided to go digging, found the "-1" pid situation leaking through, and fixed it.

Even though the fix was committed, it wasn't shipped, because this binary was big and scary and ran as root on (then) hundreds of thousands of machines, and the person who usually shipped it was on vacation getting married somewhere. As a result, the old version stayed in prod for much longer than it otherwise would have, complete with the hair-trigger bug that would nuke every process on the machine.

All it needed was something that would screw up fork, and on that morning, it finally happened.

What hasn't really been told is that the memory situation had been steadily getting worse on those machines that whole summer. We had been watching it creep up, and kept trying to make things happen, but by and large, few people really cared. Also, people had been adding more and more crap to what the web servers would run. Back in those days, you could just tell your endpoint to run arbitrary code, and it basically would, right there on the web server!

Case in point: people had started running ffmpeg on our web servers. They decided that was an AWESOME place to transcode videos. By doing that, they didn't have to build out their own "tier" of machines to do that work, which would have meant requesting resources, and all of that other stuff. Instead, they just slipped that into a release and slowly turned up the percentage knob until it was everywhere.

ffmpeg is no small thing. One instance could pull nine CPU cores and use 800 MB of memory - that's actual memory, not just virtual mappings. Also, this made requests run really long, and when that happened, the "treadmill" in the web server couldn't happen sufficiently quickly.

What's the treadmill? Well, when you have memory allocations for a bunch of requests that then finish, you have to garbage-collect them eventually. My understanding is that the treadmill essentially worked by waiting until every request that had been active at the same time was also gone, and then it would free up the resources.

This is a little confusing so think about it this way. These machines were true multitasking, so they'd possibly have 100 or more web server threads running, each potentially servicing a request. Let's say requests A-M were running and then request N started up and allocated some memory. The memory allocated by N would only be freed once not only N was done, but A-M too, since they had overlapped it in time. If any of them were sticking around for a while, then N's resources couldn't be freed until that first one exited.

Given this, it's not too hard to see that really long-running requests effectively limit how often the "treadmill" can run, and thus how often the server will release memory for use in other things.

Also, there were other things going on which were just really expensive endpoints which could chew a gig of memory all by themselves. This was NOT scalable. You simply couldn't sustain that on these systems.

Basically, if you were to make a time-traveling phone call to me a few weeks before "Call the Cops" happened, and ask me what I was worried about, "web tier chewing memory and going into swap" probably would have been pretty high on the list.

To give some idea of how long this had been going on, that year, July 4th (a national holiday) fell on a Friday, so we had a "three day weekend". When this happened, the site didn't get pushed. This mattered because push would usually get the machines to free up a bunch of memory at once and generally become less-burdened.

A regular two-day weekend would leave things looking pretty thin by the time Monday's push rolled around, but a three-day weekend made things a lot worse... and this was a full month before everything finally broke.

So, yeah, the site broke that morning, but it's not like it was too surprising. The signs had been visible for quite a while in advance. Imagine standing on top of a massive dam and you start seeing one leak, then two, then four, and so on. You try to get help but it's just not happening.

Of course, once the dam actually fails, then somehow you find the resources to get people caring about dam maintenance. It's funny how that works.

It's that time again, when code written in the past four years shows up in our lives and breaks something. Still, while you're enjoying the clown show of game companies telling people to manually set the clocks on their consoles and people not being able to fill up their cars, keep one thing in mind:

Only half of the fun of a leap year happens on February 29th.

The rest of it happens in ten months, when a bunch more code finds out that it's somehow day 366, and promptly flips out. Thus, instead of preparing to party, those people get to spend the day finding out why their device is being stupid all of the sudden.

So, if you got through today unscathed, but are somehow counting days in the year somewhere, you now have about 305 days to make sure you don't have your own Zune bug buried in your own code.

...

One more random thought on the topic: some of today's kids will be around to see what happens in 2100. That one will be all kinds of fun to see who paid attention to their rules and who just guessed based on a clean division by four.

Maybe 15 years ago, I heard that a certain cell phone camera would lose the ability to autofocus for about two weeks, then it would go back to working for another two weeks, and so on. It had something to do with the time (<some unit> since the epoch), the bits in use, and a fun little thing called sign extension.

I got some of this from a leaflet that was posted around where I worked at the time. It was posted in areas where the public could see it, so I figure it's fair game.

Here's a nice little test program to show what I'm talking about:

#include <stdio.h>

static unsigned long set_bit_a(int bit) {
  return 1 << bit;
}

static unsigned long set_bit_b(int bit) {
  return 1U << bit;
}

int main() {
  printf("sizeof(unsigned long) here: %zd\n", sizeof(unsigned long));

  for (int i = 0; i < 32; ++i) {
    printf("1 << %d : 0x%lx | 0x%lx\n", i, set_bit_a(i), set_bit_b(i));
  }

  return 0;
}

This does something mildly interesting when run on a 64 bit system:

$ bin/exp/signext 
sizeof(unsigned long) here: 8
1 << 0 : 0x1 | 0x1
1 << 1 : 0x2 | 0x2
1 << 2 : 0x4 | 0x4
1 << 3 : 0x8 | 0x8
...
1 << 28 : 0x10000000 | 0x10000000
1 << 29 : 0x20000000 | 0x20000000
1 << 30 : 0x40000000 | 0x40000000
1 << 31 : 0xffffffff80000000 | 0x80000000

Meanwhile, the same code on a 32 bit machine is relatively boring:

$ ./t
sizeof(unsigned long) here: 4
1 << 0 : 0x1 | 0x1
1 << 1 : 0x2 | 0x2
1 << 2 : 0x4 | 0x4
1 << 3 : 0x8 | 0x8
...
1 << 28 : 0x10000000 | 0x10000000
1 << 29 : 0x20000000 | 0x20000000
1 << 30 : 0x40000000 | 0x40000000
1 << 31 : 0x80000000 | 0x80000000

Gotta love it.

In the vein of my "flash" story from a few years ago, here's one about "smurf".

Back around 1997, there was something new going around in the realm of net abuse: "smurfing" a target. This one involved a nice little trick that let you send out a relatively small amount of traffic and let someone else turn it into a much larger amount of traffic, and then that response would be directed onto your target.

This required two bits of cooperation from the environment. First, you had to be able to transmit a packet of some sort with the source address set to your target. Yes, this does mean "spoofing" the source address, and any responsible ISP should filter that nonsense on egress, but back then it was all too infrequent.

Then, you also had to send it to either the network or the broadcast address of some particularly juicy network that was laden with hosts that would reply to that sort of thing.

For example, let's say you had a network 192.0.2.0 with the netmask 255.255.255.0 (a /24). Then in that case, .0 would be the network and .255 would be the broadcast address. Back in those days, firing a packet at either of those would usually make the router spew it out to the *Ethernet* broadcast address, and so it would hit every host on that subnet which could then decide to reply or not.

So, just imagine a packet "from" your target, seemingly addressed to dozens or hundreds of machines, which then all answer at once. The attacker sends out a single ~1500 byte ping (for example), and the victim receives that multiplied by however many hosts decide to reply - not great!

There were some things which could be done about this. Routers eventually got a config knob that let you turn off "directed-broadcast" or similar, so anything arriving from the outside for a network or broadcast address would just be dropped on the floor. Unix boxes of different flavors also started gaining the ability to have packet filtering rules. (This took far too long in some cases.)

Besides that, people running networks could follow various best practices and not let traffic that's claiming to be from somewhere else in the world egress from their network. Any packet like that is either a misconfiguration on someone's part (possibly yours), or maybe some dummy trying to attack someone else. Either way, it needs to be tracked down and dealt with.

Sometimes people would use this kind of stuff as a reason to "block all ICMP", and then they would just create other problems like breaking path MTU discovery, and that would cause connections to hang with large packets and weird non-Ethernet-MTU-sized links.

Another related attack tool back then was called "fraggle" and it did the same sort of directed-broadcast shenanigans, but which used UDP instead of ICMP. The effect was the same, and it also got around anyone who thought that filtering all ICMP was somehow a good idea.

The old days weren't always good days.

It's another round of feedback, because there's been a lot going on.

I must admit that I did not expect my post about diffing LDAP to have such a response. I honestly just wanted to tell a story about some mildly rebellious activity I had seen happen and then had decided to do myself, and it turned into a whole thing. Lots of people wrote in to say that they have also been doing it, and others have started as a result of that post! That was not my intent but the net effect is definitely pleasing to me, so it all worked out.

Now, in response to some specific comments - a few people wrote in to say that the "epitaphs" internal page/service at Google (yep) now allows you to line some stuff up before you leave. That way, when you leave and your entry shows up, it'll have something on it that you submitted directly, and you don't have to "bounce through a friend" or whatever.

I think part of this is that people don't realize just how long it's been since I was plugged in to that ecosystem. I left in May of 2011 - almost thirteen years ago now! I thought it was broken badly enough to leave all the way back then, and this was after being there for about four and a half years. It still amazes me when I find out that people willingly go there, but I've had to tell myself to shut up about that and just advise them to "get in, get paid, and get the hell out".

Seriously, get in, take their money, and go. Whatever tech darling status they had was gone a LONG time ago. I dare say that I watched it curl up and die from the inside. I can't even imagine what could possibly be left inside there now that so much time has passed.

It occurs to me that sufficiently young people who are just now entering the industry fresh out of school (or whatever) have no idea what it used to be like. They've only known the current versions of things, and probably figure it's as bad as everywhere else, so why not, right? I guess it's hard to argue with that. Just never make the mistake of thinking that it's special somehow. Those days are gone gone gone and they aren't coming back. Companies which are that massive just can't deliver that kind of environment.

...

In response to the WPA3 stuff and badness happening after 11 hours, Ewen (and a few other people) wrote in and said that I should have been looking at minutes, not seconds. 39960 / 60 gives 666 minutes. Oops. Yeah, I guess I missed the forest for the trees there. 666 minutes would do the job, for sure. \m/ rock and roll?

...

Other people said they were diffing far more than just the list of unixnames in LDAP. They used it to detect people getting promoted when their titles changed, and other things like that. I honestly didn't care about detecting that, and I don't think that the LDAP (really AD behind the scenes) system I was poking at even stored such things.

A fair number of companies have a glossed-over view of things for their permission systems that don't reflect whatever HR has for those same people. Everyone in LDAP might be a "software engineer", but in the actual HR system they might have 100 different varieties for "new grad" and "testing" and "server" and "app" and all of these other dumb things that they think they need. That means you might not see anything change when people get promoted, change teams, or shift around between different parts of the company.

While I'm talking about titles, I will mention one thing: it's interesting that certain companies talk about why they hide levels for random "ICs" (individual contributors, i.e., not managers), but then go ahead and make a big deal out of managerial titles.

Seriously, one company in particular had everyone be some sort of Software Engineer or Production Engineer or something like that without saying that this person was a 3, or a 4, or a 5, or whatever on up the line.

Meanwhile, that same company let you see that a given person was a Manager (5, 6), a Director (7, 8), or a VP (9, 10) with just a glance at their profile page.

The same sort of visibility was not afforded to the ICs at those same higher levels. You had to "just know" that so and so was "one of the 10s" or whatever.

Also, for anyone who hasn't already seen my thoughts on the matter somehow: you are not your level, and your level is NOT an indication of basically anything more than how much they like you. It is only loosely linked to your ability at the bottommost rungs of the "career ladder", and only when management is being forced to adhere to it. If the right people like you, your level will rise. If they don't, it will stagnate. Your technical abilities are *almost* completely disconnected from there.

There is one notable exception I can mention though: if you are somehow able to do something that nobody else can/will do, they will "put up with you" as long as the amount of whatever you bring in outweighs the costs of you being, well, you. But, once you start asking for things or try to do stuff that goes against what they personally want, the balance will tilt, and once it goes past center into the other side, they won't give a damn about what you can deliver any more.

I should note this has little to do with what the business needs at that point. They're probably in it for themselves, and they don't care that chasing someone out is not the right thing for the business. Indeed, they will probably bail out for greener pastures a few years later.

...

Some people from a few very large tech companies that are currently doing layoffs have pointed out that their "epitaphs" or equivalent isn't always accurate. There are groups of people who will be "off limits" and so won't come up in the reports. Obviously, once management has gotten to that level of involvement with the day to day operations of such a tool, it can be considered compromised.

That's pretty much a given: things start out as a simple hack, then grow into a small community that knows about it, and sometimes end up becoming well-known and even legitimized. But, more often than not, these same systems will be co-opted by whoever's running the show in terms of hiring and firing, and it'll stop providing useful data.

At that point, I guess you have a choice: you can try to build another thing from scratch for yourself, or you can admit that the company is too far down the road of corporate lockdown hell, and live with the fact that it'll never be accessible the way it had been.

...

Finally, there's at least one person who was visibly annoyed by the fact that I said "uid" and "(unix account name)" in the same breath. Guess what? That particular LDAP (again, really AD) system I was dumping DID use a field named "uid" to put in the unixnames. Sure, there were also *numeric* uids in the Unix sense, but that lived elsewhere in other fields, like, oh, uidNumber. Surprise surprise.

I just love it when there's this assumption that I must be screwing it up by default. To that person: ask yourself if you'd assume that about every random post on people's web sites that might mention such a thing, or only certain ones. If it's only certain ones, I bet you know which ones, and why.

It's obvious. You're making it clear that you're part of the problem. Knock that shit off.

And, as for me making mistakes, hell yes I make mistakes. I screw up all kinds of stuff and have to go back and fix it and/or explain what happened where appropriate. Anyone who follows the feed will tell you that old posts will "mysteriously" spring back to life with a recent modified time and a handful of tweaks applied. Most of those fixes come about from people sending feedback and going "hey I think X might be Y instead". It happens all the time, and I'm talking about 13 years of posts here.

If you think I made a mistake back in that post by saying "uid" and "unix account name" instead of "numeric unix account number", you could just hit the feedback button and say as much. But you know that it's way more impactful to assume that I'm a dumbass and don't know what the hell I'm talking about and do that on a big orange sewer.

You do know what "projection" means, right?

Okay, enough of that.

One common element of the larger places where I've worked is that they tend to have a directory service of some sort that keeps track of who's an employee and who isn't. You can learn some interesting things by periodically dumping that list and then running comparisons against the previous dump.

A certain company had this rolled up into an internal service called "epitaphs" where an entry for a person would appear a day or two after they "disappeared from LDAP" - meaning, they left the company. Then other people who still worked there could add comments like "went back to school", "moved to Idaho to raise sheep", that kind of thing.

This had an interesting side-effect that you couldn't write to your own "epitaph" because by definition you had to already be gone from the company for your page to exist. Someone else who knew you had to add it. I actually received an e-mail to that effect one time: "I'm leaving, so when it shows up, please add XYZ". I was pleased that they trusted me to do that, and a few days later, I pasted it in as requested.

Another place I worked didn't have anything quite like this. There was the "internal profile" where you could see that so and so worked at the company from <date> to <date>, but there wasn't any sort of periodic update available. I decided to roll my own. It didn't take much in the way of effort, really. A cron job on my dev server (a physical box in a datacenter with access to my home directory) woke up a couple of times every day and dumped the entire list to a file. Then it compared it to the last one, crunched it down to just the uid (unix account name) field, and appended the results to a log file.

Over time, various other people learned about this, and since I had left it world-readable, they were able to leave up a "tail -f <path>" to keep tabs on it, and sometimes something surprising would show up during the day. People would sometimes just vanish. Other times, there were bizarre things going on that added a bit of context.

The log entries looked like this:

Thu Feb 08 18:26:42 PST 2024 : uid: <someone>

That was enough to let you go digging and find out more if you actually gave a damn about why that particular person no longer worked there. Otherwise, it didn't flood you with useless data.

One time, I pasted in a line like that into an IRC channel and that <someone> popped up and said "yeah, I don't work here any more". It turned out their account had been deactivated, but they still had a client connected. When I mentioned their account name, they got a notification, flipped to that window, and replied. We had a few minutes to chat about it.

It was weird saying farewell to someone that way. Normally, the electronic lines of communication are severed early on. I think what happened here is that the IRC servers only checked auth at connect-time, and then nothing went back to make sure that sessions remained associated with current employees. (It's a bit of a hard problem.)

Another time, some manager type said they were going to be late for a meeting because of some "dumb manager thing" they had to do. Sure enough, a few minutes into that meeting, a line scrolled across showing the deactivation of an account of one of their direct reports. Obviously, they had to go into one of those HR meetings where they showed someone the door.

I'd say the best time to start doing this is when you start at a company, or when that company grows big enough to actually have LDAP or whatever. That means the second-best time would be today.

Incidentally, the 'comm' tool is great for this sort of thing.

comm -2 -3 <(grep ^uid: old | sort) <(grep ^uid: new | sort)

... and there you go.

Now, this sort of thing is not perfect. If you don't catch errors, the first time it fails to dump and yet diffs a full list against an empty list, it'll look like everyone quit. This is not what you want. Also, once you work at a big enough company, there WILL be days when some automation will run amok and "fire" everyone, and every account will be deactivated. This will happen more than once if you stay there long enough.

Incidentally, if someone gets mad about you running this sort of thing, you probably don't want to work there anyway. On the other hand, if you're able to build such tools without IT or similar getting "threatened" by it, then you might be somewhere that actually enjoys creating interesting and useful stuff. Treasure such places. They don't tend to last.

It's time for me to respond to some recent feedback. As usual, this is a mix of topics and the responses are pretty much off the cuff, so strap in and hold on tight.

...

At least one person mentioned the 11 hour WPA3 problem on my Raspberry Pis and asked if I was experiencing clock drift. This is kind of funny to me since I've been picky about keeping clocks synced in my personal and professional lives these past few years. So, no, not really. All of those Pis have chrony installed, and it's doing a great job of keeping their clocks disciplined.

I was the crazy person who spent $300 of my own money to buy a GPS-to-NTP box when the unmaintained corporate infrastructure at a certain job was down to its last "proper" time server and was in danger of failing itself. It never came to that, but we got mighty close that winter. If it had fallen over, then I would have "backfed" time into production from the corporate network using a little GPS antenna puck in the window by my desk. How crazy is that?

I've gone to lengths to make things right, put it that way.

...

Also regarding the WPA3/Pi stuff, someone said "if it is truly cursed, it will be 11 hours and 360 seconds". That would be, what... 11:06:00? I don't think that joke landed with me. I was hoping it would involve "666" somehow, but (11 * 60) + 360 is 39960.

They did mention that an 11:06:40 period is a rollover from 9999999 to 10000000 jiffies with Hz set at 250. That is, if you tick at 250 Hz, after 11 hours, you'll be at 9900000 ticks, and another 100000 ticks past that is 400 seconds, hence the 06:40 part.

Of course, for that to break something, some clown would have to be expressing the time as ASCII digits, and then breaking when it got "too wide". I mean, it happens. It happened to KDE back in September 2001 when time_t went from 999999999 to 1000000000. That was a "fun" one to deal with.

...

Niels asks how I keep my posts "so level-headed". I guess that's in the eye of the beholder, to be honest. I've had situations where I've deliberately taken every bit of emotion out of my actions, and *still* had people calling it out after the fact.

After a few decades of this, I've concluded that a lot of these results were decided long before I opened my mouth or started writing. They basically have a problem with the overall concept of me saying stuff, and exactly what gets said doesn't matter a whole lot. They just use the words I choose in order to sprinkle it throughout their takedowns.

The way you can know this is happening is when the same words come from someone else who has a different *ahem* "presence" in the same space, and then nothing bad happens, or perhaps it's taken as good. That happens too, and it's really irritating since it serves to confirm to you that it may be the 2020s, but it's really just a number in terms of people being narrow-minded and generally pig-headed.

...

A reader writes that I should try wpa_supplicant 2.10 on my RPI, and, well, I'm sorry to say that I've done that and then some, and it didn't help. Check it out - wpa_supplicant as built from upstream (hostap's git repo) *does* bring the link up... for about 10 seconds. Then it throws an error and kills it. It looks like this:

Jan 28 13:32:24 rpi5b NetworkManager[847]: <info>  [1706477544.7638] device (p2p-dev-wlan0): supplicant management interface state: associating -> associated
...
Jan 28 13:32:34 rpi5b wpa_supplicant[1209]: wlan0: Authentication with <AP> timed out.

It's a consistent 10 seconds. I was feeling like torturing myself that afternoon, so I started screwing around with the code. It took a while to find where things were happening, but I finally just extended the timeout. It would hold on there a bit longer, then it would die after the new timeout.

So, finally, I just commented out the part where it complains about that and tears down the link, and you know what? It stayed up. So, I went back a little bit, and just let it bring the link up, then suspended it with ^Z and went about my day.

Three hours later, it was fine. Two hours after that, same thing. I kept checking on it into the night. Then, finally, it died early the next morning, even with wpa_supplicant *still suspended*. Here's what it looked like from the AP side:

Mon Jan 29 01:19:23 2024 daemon.info hostapd[6371]: ath1: STA <pi 5's mac address> IEEE 802.11: disassociated

And yeah, that's about 11 hours after I started the last experiment. (I should mention that having to wait 11 hours to verify things absolutely sucks.)

What I gather from this and what a few people have told me so far is that the actual association is kept running by the device itself, and whatever you have running on the host machine (iwd or wpa_supplicant) is basically along for the ride.

Also, I could swear I found a patch from someone to one of those projects that basically says "it's gonna die after 12 hours anyway, so just let it happen and restart the connection then". I can't find it while writing this, but it doesn't matter. Here's what's wrong with "just let it fail": NetworkManager has a limit for exactly how much crap it'll accept, and after the default of 3 retries, it'll just leave the link down.

Sure, you can override this, but now you're stuck with Yet Another Behavioral Patch for any of your machines which might be affected by this. Have fun with that.

Oh, and, obviously, forget about actually DOING anything over that link while it's being restarted. If you care about reliability, this is not a tenable situation.

Once again, if you care about having decent "modern" (as in 2020) wifi on a Pi, go get yourself a little USB barnacle that's supported upstream and go on with life. Or, better yet, just use hardwired Ethernet and forget that the thing even has a radio on board.

I'll pour one out for the sanity of anyone who doesn't have a choice in the matter. At least you're not suffering by yourself.

As my assortment of dumb little home-grown utility programs grew over the years, I found myself needing to know when a given binary was built. Sometimes things exist outside the realm of a packaging system, and so the binary itself needs to convey that metadata from build time.

I've seen some places solve for this by having giant placeholder strings baked into their binaries that they then reach in and "stamp" later, turning the "XXXXXXXX" or whatever into "Built by foo@foo.blah.evilcorp on ...". While that approach mostly worked, it was too spooky for me and I decided to stay away from it.

My system is something that uses a little nuance of C++ that I've mentioned a couple of times already. It's not the cleanest thing and it does involve a bit of groaning, but it works. In case anyone else wants to try it in their projects, here's how I set it up.

First, I have this buildinfo/base.h, and in it, I define a struct called Details, and it has all of the fields I care about - times, hostnames, usernames, commit hashes, that kind of thing.

There's also this:

extern std::optional<Details> details_;

Yes, that is globally visible, but it's inside a namespace so the crap factor is reduced somewhat. It's a necessary evil.

I also have a buildinfo/base.cc and it actually creates that variable:

std::optional<Details> details_;

There's also a GetBuildDetails function which will return the value of details_ if one exists, or a suitable error if not.

Now, you might be saying "it'll never have a value, so it'll always be an error", and you're mostly right. Just from what I've described so far, that in fact is the case. buildinfo/base.{cc,h} rolls up into buildinfo/base.o, and that gets linked into my programs during a normal development type build. If one of those programs calls the GetBuildDetails() function, then yes, they get the "sorry, nobody home" error response.

But, I have a way to inject the build info when I do a "production" build. This kind of build has slightly different config settings in my build system, and one of them tells it to "stamp" the binary.

The way this works is where the evil starts slipping in. On stamped builds, my build system writes out a file called buildinfo/overlay.cc. This file #includes "buildinfo/base.h" to pick up the definition of Details and the 'extern' for details_ itself. Then it rattles off a bunch of variables and their values (build time, build host, build user, ...) then it defines a class called Overlay.

Overlay's constructor has one job: it reaches into details_ and populates a bunch of fields with the values from those earlier variables.

Then the thing that actually makes it run shows up, and here's the "spooky action at a distance":

static Overlay overlay;

Just by having that line in the file, it will cause the program to create an instance of that class shortly before it reaches main() as long as it's linked into the final binary. When that constructor runs, it will populate details_, and then any code run from the rest of the program will see the build info.

This is convoluted, so I'll restate it here for clarity: it's the difference between linking "a.o", "b.o" and "c.o" into "prog", or "a.o", "b.o", "c.o" *and* "overlay.o" into "prog". If you don't link in that extra object, the sneaky stuff never happens, and it stays unpopulated. Using a std::optional wrapper saves us from the jankiness of using a bare pointer... or worse.

There are some bonuses from using the intermediate variables instead of just having a bunch of .field = "val" type things in the part where details_ gets initialized. For one, if those variables are not set to static, then they'll be visible to things like debuggers and certain other tools. Then you can do something like this:

$ gdb lavalamp_server -q
Reading symbols from lavalamp_server...
(gdb) print buildinfo::kBuildTime
$1 = 1707186971
(gdb) 

That's pretty neat, right? Analysis of a binary at rest? You can even do this without going through a debugger if you really want to.

Finally, how does the build tool handle this? It's a bit more of the special-case stuff for something that isn't just an ordinary build. If "stamping" of binaries has been requested for the active build type, then it generates a fresh overlay.cc and compiles it to overlay.o.

Then, in the link stage, if it's in "stamp mode", it inserts that object file as just another dependency of the build target as if it had been discovered by way of a '#include "some_dir/some_lib.h"' or whatever. This adds it to the list of objects passed to the linker, and a fresh binary pops out a few moments later.

I'm a fan of this technique since it only really adds two places in the build system where things go off and act a little strangely: the init sequence of the build tool when it's first built, and the link sequence of the target(s) to make sure it gets "injected".

For anyone who's worrying about "repeatable builds" or somesuch, I will point out that nothing's stopping you from having yet another build type which is otherwise as described above but which puts known placeholder data into the details_ variable. In that world, you should be able to go through the entire process and still get the same output, even on a different date, on a different box, and as another username.

I no longer wonder about which version of a binary is in "prod".

What a night. I hit upon something that got WPA3 working on some Raspberry Pi systems and excitedly put up a post to share the good news. Then I went away for a while, and this morning found something new: the damn things won't stay connected for more than 11 hours. All three of them failed in the same order that I changed them over.

The timeline is something like this:

01:03:51 NetworkManager [...]: new IWD device state is connected
[...]
12:04:39 iwd[...]: Received Deauthentication event, reason: 0, from_ap: false

Now that's something, considering none of the tunable aspects of the WPA3/SAE setup on my network are set to 11 hours. But, what's this? You do a search for "received deauthentication event" and "11 hours" and what do you find but a cursed Infineon developer community post on that very topic.

It's from October 2021 (!) and it's from someone who's using the same sort of chipset (Broadcom/Cypress/Infineon CYW stuff) NOT on a Pi, and they get the same "deauth" thing when on a WPA3 network. If then drop down to WPA2, it stops. The thread runs for close to a year, and then just stops cold in August 2022 with no resolution.

So, what'll it be? WPA2 mode and not being able to get onto any networks that have gone to 6E or 7, or WPA3 mode and having it fall off the network every 11 hours?

Here, I'll make a prediction: someone will say "it works most of the time, so that's fine, we won't be fixing this". Considering the level of crap people put up with in their tech these days, that'll probably be the steady-state.

My conclusion: this entire ecosystem is deeply cursed.

Okay, it's been several months since I last wrote about WPA3 on Raspberry Pi hardware. Now, I have some good news: it now mostly works, assuming you're willing to do a little tinkering. You no longer have to wrangle custom firmwares and binary blobs into place. That's been done for you.

One important thing here: I'm only talking about Raspbian/Raspberry Pi OS here, and then only bookworm (12). If you're running something else, none of this may apply. For all I know, it might have been working all along if your distribution figured it out sooner.

So then, if you have a 3B+, 4B or 5B on bookworm, get ready to rock.

Every so often I look at the list of package updates coming down the pipe through apt for my systems, and usually groan at the usual round of CVE patches. But, this time I saw something rather different. It's a change to "firmware-brcm80211", and what's this? Something specific to the CYW43455 that the 3B+ and later use? Oh, that's interesting, right?

  * brcm80211: cypress: Use a generic CYW43455 firmware
    - Version: 7.45.234 (4ca95bb CY) CRC: 212e223d Date: Thu 2021-04-15 03:06:00 PDT Ucode Ver: 1043.2161 FWID 01-996384e2

I wondered if this would stop the madness, and so applied it and rebooted. "iw phy" showed the good news - at the very bottom, it now supports both SAE_OFFLOAD and SAE_OFFLOAD_AP. This means it can actually do SAE... but you're going to have to say goodbye to wpa_supplicant in favor of iwd.

If you switched over to NetworkManager for consistency with the rest of the bookworm changes, this is not a big deal. If you're running your network some other way, you get to figure this out yourself.

The steps work out like this: apt update, then apt upgrade so you get the firmware-brcm80211 from January 17th (or later, once this post gets old, I guess). Then apt install iwd.

Assuming you've been running your Pi in a crappy non-WPA3 network, go into nmtui or equivalent and disable that network. In a minute or two, you're not going to need it any more.

Then disable wpa_supplicant and drop a bit of config into /etc/NetworkManager/conf.d/iwd.conf:

[device]
wifi.backend=iwd

Reboot ... or do equivalent wrangling of drivers and binary crap to get it to unload and reload the fresh stuff. Then run "iw phy" again. If it says SAE_OFFLOAD and SAE_OFFLOAD_AP, you're ready to proceed. If not, well, something's wrong, and you should turn on the crappy old non-WPA3 network again.

Assuming it worked, then go into nmtui or whatever and tell it to activate the actual WPA3-only network. Paste in your PSK and let it go, and a few seconds later you should be in business.

That's it. That's all it takes.

Now then, what about the 3B, you might ask. It's on a different chip that doesn't even do 5 GHz, so changing the 43455 firmware wouldn't help any. It seems to be a 43430 instead, and I have no idea if there's any chance of getting a similar firmware change for it. Obviously, if this changes, I'll post something about it, too.

I can't comment on the other models, like the Zeroes and the other weird little boards they have. I only have access to these relatively normal models, and that's what I was able to work on.

Go forth and have better networks!

January 24, 2024: This post has an update.

It's a new year, so let's talk about some more time-related shenanigans. This one comes from the world of writing C++ for multiple platforms. A couple of weeks ago, I was looking at some of my code and couldn't remember why it did something goofy-looking. It's a utility function that runs stat() on a target path and returns the mtime as a std::chrono::system_clock::time_point. This is nicer than using a time_t since it has sub-second precision.

The trick is getting it out of a "struct stat" and into that time_point. The integer part is simple enough: you use from_time_t on the tv_sec field. But then you have to get the nanoseconds (tv_nsec) from that struct into your time_point. What do you do?

The "obvious" answer sounds something like this: add std::chrono::nanoseconds(foo.tv_nsec) to your time_point. It even works in a few places! It just doesn't work everywhere. On a Mac, it'll blow up with a nasty compiler error. Good luck trying to make sense of this the first time you see it:

exp/tp.cc:14:6: error: no viable overloaded '+='
  tp += std::chrono::nanoseconds(2345);
  ~~ ^  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__chrono/time_point.h:65:73: note: candidate function not viable: no known conversion from 'duration<[...], ratio<[...], 1000000000>>' to 'const duration<[...], ratio<[...], 1000000>>' for 1st argument
    _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR_SINCE_CXX17 time_point& operator+=(const duration& __d) {__d_ += __d; return *this;}

Nice, right? It tells you that there's something wrong, but the chances of someone figuring that out quickly are pretty slim. For the benefit of anyone else who encounters this, it's basically this: a system_clock::time_point on that platform isn't fine enough to represent nanoseconds, and they're keeping you from throwing away precision.

To make it happy, you have to jam it through a duration_cast and just accept the lack of precision - you're basically shaving off the last three digits, so instead of something like 0.111222333 seconds, your time will appear as 0.111222 seconds. The nanoseconds are gone.

I assume you might find other platforms out there which don't support microseconds or even milliseconds, and so you'd hit the same problem with trying to "just add" them to system clock time point.

At any rate, here's a little bit of demo code to show what I'm talking about. As-is, it'll run on Linux boxes and Macs, and it'll show slightly different results.

#include <stdio.h>

#include <chrono>

int main() {
  std::chrono::system_clock::time_point tp =
      std::chrono::system_clock::from_time_t(1234567890);

  // Okay.
  tp += std::chrono::milliseconds(1);

  // No problem here so far.
  tp += std::chrono::microseconds(1);

  // But... this fails on Macs:
  // tp += std::chrono::nanoseconds(123);

  // So you adapt, and this works everywhere.  It slices off some of that
  // precision without any hint as to why or when, and it's ugly too!

  tp += std::chrono::duration_cast<std::chrono::system_clock::duration>(
      std::chrono::nanoseconds(123));

  // Something like this swaps the horizontal verbosity for vertical
  // stretchiness (and still slices off that precision).

  using std::chrono::duration_cast;
  using std::chrono::system_clock;
  using std::chrono::nanoseconds;

  tp += duration_cast<system_clock::duration>(nanoseconds(123));

  // This is what you ended up with:

  auto tse = tp.time_since_epoch();

  printf("%lld\n", (long long) duration_cast<nanoseconds>(tse).count());

  // Output meaning when split up:
  //
  //        sec        ms  us  ns
  //
  // macOS: 1234567890 001 001 000  <-- 000 = loss of precision (246 ns)
  //
  // Linux: 1234567890 001 001 246  <-- 246 = 123 + 123 (expected)
  //

  return 0;
}

To bring this full-circle, that's why I have that ugly thing in my code to handle the addition of the tv_nsec field. Without it, the code doesn't even compile on a Mac.

Stuff like this is why comments can be very important after the fact.

It's time for some end of the year feedback. I get a bunch of comments and questions from people through my contact page, and sometimes this is the only way to reply. Other times, a response is also suitable for a wider audience.

...

Igor asks:

Have you played around with cling? Seems you may have the knowledge to break it or request the developers to enhance it with some useful feature(s). Like suppose doing scripting in it?

I didn't even know what cling was until I read this and went looking. It seems to be a clang/LLVM-based C++ interpreter that's interactive. I guess I don't have a need for something like that. As far as deliberately breaking things, I have no doubt that my luck would lead to any number of bad outcomes. However, I have enough of that in my life already without seeking out new stuff just for the sake of having something to break.

Now, if there was a good reason for me to do it, that would be another story. (I am available for mercenary purposes, put it that way.)

...

An anonymous reader asks:

What's the specs of this new web server?

It's only new to me. It's actually fairly old. I think it's roughly 2014 vintage. I bought it from one of those vendors who resell old servers. I figured "it's a server, not a toothbrush" and so the notion of using some random old box was not a problem for me. It runs Linux and is plenty quick, so I'm happy enough.

I guess this is a good point to tell the story of actually installing this thing. Despite working at places with millions of Linux boxes over the years, I'd never hung a server. I'd done routers and dialup boxes, but those were all relatively short. They'd hang from the front posts and were plenty happy with life that way.

flicker, on the other hand, is a monster. It's so long the cabinet doors almost didn't close. I had no idea that length was a dimension that might be a problem. I clearly didn't realize just what I was getting when that order was placed, and when it showed up in a giant box, I started thinking "what have I done?"...

Initially, the machine barely fit into the cabinet with the power cord being squished on one end and the network cables being squished on the other. I came back about a month later and swapped it to a new power cord that has a 90 degree bend built into the server-side plug. This clawed back about an inch on that side and let the whole thing slide back a little bit which took the pressure off the Ethernet cables.

I took the now-permanently-crimped (and damaged) power cable, cut it in half so nobody would find it and try to use it, and tossed it.

It was the sort of thing that anyone with experience would point at and go "ha, you screwed up", and indeed, I did. It turned out to not be fatal to the project of moving to colocation, but it was mighty close.

Next time, I'll pay attention to this sort of thing.

Lesson learned: go see the cabinet and measure it before ordering something that's almost three feet long!

(It's an Ivy Bridge flavored dual-socket hex core Xeon with 128 GB of memory and a couple of SSDs. It also has a far faster pipe than the previous box. It's a complete monster for how I use it. Oh, and it's not in Texas, which has turned out to be very important.)

...

Regarding my "char buf[1048576]" thing from the other day where I blew the stack, a reader says:

You can use boost::thread instead. It lets you specify the stack size.

Honestly though, I don't *want* to specify the stack size. I'm fine with defaults. Also, any answer which involves "boost" means I'm asking the wrong question. I don't ever want to get stuck with something like that. There's been too much drama in my life from boost dependencies in years gone by.

The obvious way around the "problem" for me is to do something like a vector<char>, push it out to whatever size, and then let read() use it as a buffer. This is pretty much what I end up doing any time I have to deal with old-school C library functions from my C++ code.

Like, okay, mkdtemp(). It wants a char* template, like "/tmp/test_thingy.XXXXXXXX". It *reaches into* that space and alters it, changing the Xs to some random gunk that turned out to be unique - this is how you avoid /tmp race attacks. This is more annoying than it sounds.

You might think "oh I know, I'll use a string, then hand the .c_str() pointer from it to mkdtemp". But, no, c_str() gives you a *const* pointer, and mkdtemp is going to violate that const-ness. Your compiler will stop you from doing this. You then must either lie to it and do some nasty casting, or you decide that's not going to work and find another way.

This is why I have a bunch of stupid shims to deal with these scenarios. It's just that I never really shimmed read() before - the old hacky way of having a buffer on the stack was never a problem.

As for "why is that buffer so big", well, I like to cut down on the number of syscalls required to inhale a file: a bigger buffer eats the whole thing in a single go. A tiny buffer would need to keep coming back until it finished. This sort of thing used to matter more when machines were much slower.

Besides, I hate spammy syscall situations. read() or write() with a single byte? Argh!

...

From another reader:

hey are you still running wg barebones on your mac? I just made the mistake of installing and got the 'cant uninstall' behavior with additional 'nothing happens when you dbl click on app icon' . just wanted to compare notes

Yep, I still am doing this on my personal machine and also those of family members (in my duties as "holiday sysadmin"), having removed the "app store" version a while back. I get feedback from random people who find that three year old post about once a quarter, and they all seem to have the same problems. I take that as a pretty strong signal to not try it again just yet.

I should note that if you're using Macports, upgrade to Sonoma, and then try to build wireguard-tools or several other packages, it'll fail due to various Apple-related stupidity. I'll do another post about that eventually.

...

Finally, a bunch of people wrote in about the "clang + C++ + original armv6 Raspberry Pi = unusable binaries" to say that it explained some oddities they had been seeing on their own stuff, so I think that turned out okay. That's really what I aim for in writing these posts: visibility into problems, and confirmation that you aren't alone in experiencing something. Sometimes it's technical, but other times it's about squishy meatspace stuff. Both are valid.

I need to remind people how easy it is to forget just one of the many gotchas of working on this ridiculous computer stuff. One missed nugget of data at a critical moment can leave you scratching your head and going "WTF" for longer than would otherwise seem reasonable.

Here's something that happened to me last week. I was working on a stupid little utility that runs on my machines and lets me keep tabs on what's going on with systemd. If it gets unhappy because any of the services have stopped running, then this thing will let me know about it. For the handful of systems I have to worry about, it gets the job done.

Now, since I'm in "holiday mode", I'm largely working on my laptop instead of sshing back to a Linux box somewhere else. This laptop is a Mac, so it's mostly compatible with what I'm doing. Obviously, it doesn't run systemd, but that wouldn't stop me from tidying up a tool in test mode. I was working on this thing, and noticed it started blowing up in strange places. Also, it was a really strange "bus error". To me, that says "binaries on NFS" or "unaligned access on some architectures". I'm not doing either sort of thing here.

gdb was not really an option at that moment for various annoying reasons so I resorted to "debug via printf" - putting little notes to say "I got here" and whatnot. They kept changing. I'd think I had it nailed down, and it would move!

Eventually, I got it down to something truly odd: it was blowing up in a worker thread, and it was the point where that thread started up and read in a config file from the disk. The line of code looked something like this, where I call into one of my own helper libraries:

auto raw = file::ReadFileToString(kDefaultConfigPath);

Okay, I said to myself, let's find out what's going on in that function and started sprinkling my "I got here" notes into there. One of those notes was at the very top of that function and just said "got into ReadFileToString". It never ran.

I removed the call to that function. It stopped crashing.

So, what's in that function that's so spooky? Well, it opens a file descriptor, does the usual sanity checks on it, and then creates a buffer that it'll pass to read()... and herein lies the problem:

  char buf[1048576];

Yep, just having that there was blowing the stack, and the bus error is how it manifested in that particular arrangement of function calls within the worker thread.

That's right, if you're already pressed for stack space and then enter a function with something like that, you might just explode. Here's a contrived example with an even bigger buffer to demonstrate it with just a single innocent-seeming function call:

mac$ cat bs.cc 
#include <stdio.h>

#include <memory>
#include <thread>

static void do_thing() {
  char buf[1048576 * 8];
  buf[0] = '\0';
}

int main(int argc, char** argv) {
  if (argc != 1) {
    printf("running in worker thread\n");

    auto worker = std::make_unique<std::thread>(&do_thing);
    worker->join();
    return 0;
  }

  printf("running in main\n");
  do_thing();
  return 0;
}

The fun part is that on a Mac, the flavor of error changes between "bus error" and "segmentation fault" just by shoveling it into a thread.

mac$ ./bs 
running in main
zsh: segmentation fault  ./bs
mac$ ./bs foo
running in worker thread
zsh: bus error  ./bs foo
mac$ 

Nice, right? Further complicating matters is that on a boring old x86_64 Linux box, it gets reported as a segmentation fault both ways.

linux$ ./bs
running in main
Segmentation fault
linux$ ./bs foo
running in worker thread
Segmentation fault

A simple twiddling of the ulimits will change the behavior ever so slightly:

linux$ ulimit -s unlimited
linux$ ./bs
running in main
linux$ ./bs foo
running in worker thread
Segmentation fault
linux$ 

Fun fun fun. Obviously, I need to rethink the way I manage my buffers.

Sometimes, inside a company, you find someone who's just so good at what they do and who has the fire burning inside them to always do the right thing. It's easy to fall into a spot where you just go to them first instead of chasing down the actual person holding the pager that week. At some point you have to tell yourself to at least *try* "going through channels" to give them a break.

But still, you wind up with some tremendous stories from when they came through and saved the day. I love collecting these stories and I periodically share them here. This is another one of those times.

Back up about a decade. There was something new happening where people were starting to get serious about compressing their HTTP responses to cut down on bandwidth and latency both: fewer packets = fewer ACKs = less waiting in general. You get the idea.

A new version of the app for one particular flavor of mobile device had just been built which could handle this particular flavor of compression. It was going out to alpha and then beta testers, so it wasn't full-scale yet. When it made a request, it included a HTTP header that said "hey, web server, I can handle the new stuff, so please use it when you talk back to me".

On my side of the world, we didn't know this right away. All we knew was that our web servers had started dying. It was one here, then one there, then some more over in this other spot, and a few more back in the first place, and it was slowly creeping up. This wasn't great.

We eventually figured out that it was crashing in this new compression code. It had been added to the web server's binary code at some point before, and it obviously had a problem, but I don't think we had a good way to turn it off from our side. So, every time one of these new clients showed up with a request, their header switched on the new code for that response, and when it ran, the whole thing blew up.

When the web server hit the bad code, it not only killed the request from the alpha/beta app, but it also took down every other one that same machine was serving at that moment. Given that these systems could easily be doing dozens of requests simultaneously, so this was no small thing! Lots of people started noticing.

That's when one of those amazing people I mentioned earlier stepped in. He knew how to wrangle the proxies which sat between the outside world and our web servers. It had a scripting language which could be used to apply certain transforms to the data passing through it without going through a whole recompile & redeploy process for the actual proxies.

What he did was quick and decisive: it was a rule to drop the "turn on the new compression" header on incoming HTTP requests. With those stripped from the request, the web server wouldn't go down the branch into the new (bad) code, and wouldn't explode. We stopped losing web servers, and we were now in a situation where the pressure was off and we could work on the actual crash problem.

I should mention that we were unable to just switch off the new feature in the clients. The way that clients found out what features to run in the first place was by talking to the web servers. They'd get an updated list of what to enable or disable, and would proceed that way. But, if the web server crashed every time they talked to it, they would never get an update.

That's why this little hack was so effective. It broke the cycle and let us regain control of the situation. Otherwise, as the app shipped out to more and more people, we would have had a very bad day as every query killed the web servers.

And yes, we do refer to such anomalies as a "query of death". They tend to be insidious, such that when they show up, they take down a whole multitenant node and all of the other requests too. Then they inevitably get retried, find another node and nuke that one too. Pretty soon, you have no servers left.

I have a bunch of Raspberry Pi systems all over the place, goofy things that they are. They do dumb and annoying jobs in strange locations. I even have one of the older models, which is called just the B+. You can think of it as the "1B+" but apparently it was never officially branded the 1.

If you have one of these, or perhaps an original Pi Zero hanging around, you might find that C++ programs built with clang don't work any more. I ran into this as soon as I started trying to take binaries from my "build host" (a much faster Pi 4B) to run them on this original beast. It throws an illegal instruction.

This used to work in the old version (bullseye). It now breaks in the current one (bookworm). I figured, okay, maybe it's doing some optimization because it was built on the 4B. So, I went and did a build on the B+ natively. It also broke.

So I backed off another level to a much simpler reproduction case: just declare main() and return. That still broke.

Looking this up, there are a bunch of screwy dead-end forum posts where people go back and forth asserting this package is installed and that's making the compiler go stupid, or it's because they did the "lite" install vs. the "recommended" install, or who knows what.

I wanted to do better than that, so this afternoon I picked up a brand new SD card, blew the whole "desktop + recommended" OS image onto it, booted *that*, then installed clang, and...

raspberrypi:~/prog$ cat t.cc
#include <stdio.h>

int main() {
  return 0;
}
raspberrypi:~/prog$ clang++ -Wall -o t t.cc
raspberrypi:~/prog$ ./t
Illegal instruction

Awesome. It can compile something it can't even run. What's the bad instruction? gdb will answer that in a jiffy.

(gdb) disassemble
Dump of assembler code for function main:
   0x004005a4 <+0>:	sub	sp, sp, #4
=> 0x004005a8 <+4>:	movw	r0, #0

movw. That's not in armv6l, apparently. So yeah, this compiler is effectively cross-compiling for armv7 (or something) by default. That's not very useful.

You can work around this by grabbing the compiler by the lapels and saying "build for armv6, punk", and it will give you a working binary:

raspberrypi:~/prog$ clang++ --target=armv6-unknown-linux-gnueabihf -Wall -o t t.cc
raspberrypi:~/prog$ ./t
raspberrypi:~/prog$ 

How and why did it get to that point? I can only imagine it's some default that got bumped from version 11 to version 12, and somehow nobody noticed? I guess nobody still runs these old things anywhere?

So weird.

A little over 20 years ago, Verisign did something mighty evil: they effectively typosquatted every single unregistered domain in the .com and .net top-level domains. They could do this because they controlled those from the registry side of things, and it was trivial to slam something that would make it resolve.

Reactions from people like me who had systems to run and spam to block were swift and universally negative due to all of the collateral damage it caused. Here's one situation it created: sometimes you'd have a user who thought they were being clever, and they'd put something like "nospam" in the from address in their e-mail client. Thus, they'd try to send mail as luser@nospam-example.com instead of just @example.com.

Before Verisign pulled that crap, the mail servers of the day would have just rejected it as an invalid domain that didn't resolve (in DNS). Once it was online, it WOULD resolve, and so the mail system would accept it. Utter chaos.

Or, how about all of the random people who would mistype something? Instead of getting a failure to resolve error from their browser and/or organization's web proxy, it would just pop out to that stupid Verisign site.

It should surprise nobody that a bunch of us sour sysadmin types did things about it rather quickly over the next day or so. I still have the snippet of cruft to drop into a sendmail.cf in my notes from back then. I got this from Usenet, apparently:

Local_check_mail
R$*                     $: $>canonify $1
R$*<@$*.>               $: $1<@$2> strip the trailing . if present
R$*<@$+>                $: $(verisign $2 $)
R64.94.110.11           $#error $: "550 Real domain name required for sender address"

Incidentally, if any of you can still parse that syntax in your heads just by reading it, I'm so sorry. Take it from me - that ability does go away eventually, but you have to stop supplying it with new data.

Anyway, what that did was to exploit the limitless programmability of sendmail's config language to resolve the supplied domain. If it came back to 64.94.110.11, it would reject it right then and there. That was the IP address they were using for their little marketdroid-fueled fever dream, and I bet there are *still* systems blocking it 20 years later (and they probably have no idea why).

Over the next couple of days, a bunch of other things happened. ISC wrote a "delegation-only" feature for BIND (aka named, the occasional remote sudo implementation). You could use this to say that the "com." or "net." zones were only allowed to provide delegation to other zones. That is, within the bailiwick of "com.", it could say that "example.com." has a nameserver at <foo> with IP address <bar>, but that's it. It couldn't come right out and say that something had an A record outright.

Now, this worked great, but it wouldn't have been terribly difficult for them to sidestep it. They could have had it delegate all of those things down to some other level which then would have had a blanket answer for any incoming questions. Fortunately, this did not happen.

This whole bout of stupidity lasted about three weeks, and then it disappeared. They've never done it again, but plenty of other providers of recursive resolver action have pulled it since then as a matter of course. Screw up the way DNS works in order to fellate the advertisers? Brilliant!

Remember this? That was 2019.

I find it strangely fulfilling that this event has garnered a Wikipedia article in the years since. Hopefully people will never forget what happened back in 2003 with the DNS.

Perhaps you've been running Ubiquiti stuff for a while, and you've been disappointed by their stock issues, their goofy software issues, and the general lack of quality all the way around. Maybe you turned your eyes to the TP-Link Omada ecosystem. I'm here to warn you that the grass is not greener on that side of the fence. It may in fact be spray-painted.

First, some context. I'm the family sysadmin - not by choice, but because nobody else would do it. When I visit family, I have to fix their stuff. There are some gearhead types and I do my best to make them happy. Various ISPs are starting to sell services that are well above 1 Gbps. This is typically symmetric fiber stuff.

That's the situation with one of the sites I support, and their existing Ubiquiti stuff from years gone by became a bottleneck once they had that installed. Obviously, they want to get this greater-than-gig performance wherever possible. That means a derpy Windoze box or two, and that brought on a whole hellscape of dealing with resource conflicts the likes of which I hadn't seen in 20 years.

But no, this isn't about that. This is about TP-Link. I was pointed at this ecosystem as a possible escape from the clowntown that is Ubiquiti, so that's what I bought this time around: one of their gateway boxes (calling it a router would be too kind), a switch, and a hardware controller for local control - none of that cloud crap here, thanks.

It's been a new bit of stupid every week with this stuff. First of all, the switch is really best suited for a closet at a business, not anywhere in someone's home. It has dinky little fans that run pretty hard all the time, with all the noise that entails. People who replace them invariably get fan errors and then the thing eats itself within a year. (Maybe the switches fail by themselves either way - the jury is still out on that.)

The latest build of their controller software flat out does not work on Safari. I mean, sure, it loads up, and then the browser starts doing something indicative of a horrible big-O blowup factor somewhere in their Javascript. It'll hang for a minute at a time any time you move the pointer around. Or, it'll prompt you to download a file CONSTANTLY. Like, WTF kind of content-type brain damage are you doing? It doesn't happen in Firefox or Chrome, apparently, but it still goes to show that they gave zero fucks about even TRYING TO LOGIN in Safari when they were developing it. You know, the browser that every Mac and iOS device ships with?

So, you have to roll back the controller to get out of this mess. Doing that wipes your config. Fortunately for me, I discovered this during my early shakedown testing at my own residence before hauling it out to the site, and there was no actual config to lose.

Next up, their NAT implementation is just plain obnoxious. Typically with this kind of stuff, if you fire a packet from the inside to the outside, the source address gets changed from RFC 1918 or whatever you're using internally to whatever you have on the outside. That much works. What also happens here on the TP-Link ecosystem is that they mangle your source port, too. This affects both UDP and TCP.

Why does this matter? It makes NAT hole-chopping tricks much harder to pull off. Normally, you can do such fun things as configuring WireGuard to punch through from either side by lining up the ports exactly. This will let two sites connect to each other without going through a third fixed spot. This is very handy if that third spot goes down and you need an OMFG backdoor into your networks!

This does not work if the source ports change. At that point, you have to resort to all kinds of nasty birthday paradox type stuff to figure it out, and that requires Actual Work to pull it off and keep it working. Me, I don't want to put TailScale everywhere. But I digress.

Last week, something very bad happened that I haven't managed to troubleshoot since I'm remote and can only do limited things from here. HomeKit stuff stopped working. By that, I mean that viewing the home from off the local wifi said the usual "no hubs online" thing. But, stranger still, HomeKit *clients* on that wifi also couldn't connect *outward* to other spots! They, too, got the same thing about no hubs about other HomeKit locations... even when those locations were actually fine and worked for other people.

The only commonality was crossing that Omada-powered network. I had some luck in this case since there's a Mac out there which I can hop into and beat into submission, and beat I did. I figured maybe it was something goofy about the routing to Apple's cloud stuff, and started shunting all of the traffic through a tunnel. Nothing helped ... until I also switched DNS resolution on that Mac to something I controlled instead of using whatever resolver is inside the TP-Link gateway box.

Once I did that, it started working again. Even after I turned off the tunneling, it kept going. This was enough for me. I stood up unbound on a couple of Raspberry Pis out there and changed the DHCP config to make sure clients would resolve things through them instead of the ER8411 gateway. It took a while, but eventually, everything stopped being stupid.

Now, big caveat here: I don't know 100% that it was the resolver in the thing. I wasn't on site, and could only do so much without kicking myself out of the network, since my access came in through those very devices. Also, my troubleshooting abilities are limited with this crap for yet another reason I'll get to later.

Then there's what happened this morning. One of my Pis behind this setup decided it wasn't going to run one of its WireGuard links. The other link on the same interface (going to another external host) was fine. The other link on the other interface was fine. The other Pi's two links were also fine.

It was just this one particular association that wasn't working. So, into tcpdump I went yet again, looking at it from both sides of the link. The exchange I saw from inside looked like this over and over:

their_internal_ip.AAAAA -> my_external_box.BBBBB: udp data
(no reply)

But, from the outside world, it looked like this:

their_external_ip.CCCCC -> my_external_box.BBBBB: udp data
my_external_box.BBBBB -> their_external_ip.CCCCC: udp data
their_external_ip -> my_external_box: ICMP port CCCCC unreachable

So yeah, even though it had JUST sent traffic to me from that port, upon reply, the gateway box was rejecting it. This to me says "really terrible IP connection tracking setting and/or implementation that dropped the association and is somehow not picking it back up".

This WG link has a keepalive on both ends. There's no excuse for this. It should be established in the firewall as soon as a packet goes out, as one did above. But the ICMP error indicates otherwise.

Note that the port-unreachable error is not coming from the Pi itself. The Pi was only sending actual traffic and had no idea why it wasn't getting any responses. WG won't switch source ports by itself, so it just keeps smacking its head into the wall ... over and over and over.

And this brings me to the final point of frustration: I wanted to ssh into the damn gateway to to see what they were doing to screw things up so badly. It took a while to find the knob to enable ssh, and once that was on, I found the ultimate insult: it's a completely neutered interface. You can't do anything useful. It's busybox, the "ip" utility, and something that apparently lets you point it at a controller for when the adoption process doesn't work.

su? sudo? Forget about it. You don't even have /proc in there - so no ps, no w. You can't run dmesg because it doesn't exist (and they probably lock down the kernel ring buffer anyway). You are a luser, and you will never be able to do anything useful from this setup.

When pressed, tech support tells people that such things are unsupported when using the controller - that is, the dumb pointy clicky web-based UI that takes the setup and pushes it out to the devices. You know, the one that broke on Safari in the latest version. They're locking you out _on purpose_.

Finally, I haven't run into this one yet since the ISP for this site is still in the dark ages in terms of providing access to the ENTIRE Internet, but it sounds like they don't do any sort of IPv6 firewalling. So, if your ISP switches that on and you suddenly get an allocation, look out world! It's the wild west on your network!

So, let's recap the suckiness here.

0. The switch is stupidly noisy.

1. Their latest version of the controller just does not work in Safari.

2. You can't easily roll back the controller when it does suck. You'd better save the config from the old version before you upgrade, just in case you ever have to go back. And, if you never ran that particular old version, you're doubly screwed.

3. Their NAT implementation mangles source ports needlessly. Sure, some scenarios call for it. They do it constantly.

4. *Something* broke HomeKit comms really badly, and switching recursive DNS services for clients away from whatever the gateway box provides fixed it. It's probably some terrible DNS forwarder implementation but I have no way to be sure at this point.

5. The NAT apparently dropped an assoc this morning and never put it back. I couldn't get my tunnel going until I restarted it to pick a new source port on the client. Completely ridiculous.

6. Forget about ssh to troubleshoot things. The hood is welded shut. You will never know what's really going on when one of the other items decides to rear up and bite you in a sensitive place.

7. They apparently have no IPv6 firewalling based on what other people have reported in various places. (This is the only one I haven't actually encountered myself... yet.)

So now what? I'm honestly looking at returning to what I was doing in the 90s: building my own Linux boxes with enough horsepower to handle the networks in question. It worked then and it'll work again now. Things will still break, but at least I'll be able to use my actual experience to do something useful about it. Right now, I can do nothing. My hands are tied.

Why did I think these clowns had any idea what to do? I've been both inside and outside of this world, and it's pretty clear that they do not. Just look at how awful these products really are.

Thanks to a bit of anonymous feedback this morning, I have some good news about the Raspberry Pi WPA3 thing. Apparently the good folks over at the Asahi Linux project have taken up the cause of fixing the upstream kernel situation. It seems this will happen by throwing out some existing implementation that didn't work anyway, and perhaps my posts were confirmation that they were in fact crap.

This is great! Hopefully this will lower the barriers for regular people who just want things to work and don't want to patch kernels and drivers and maintain their own forks of things.

So, yay for that person cleaning up the mess of some dumb big companies. Thanks for pushing on this and not letting the suck stop you.

...

For the record, I've tried this on a 3B, 3B+, 4B, and now a 5B. Did I buy the 5 expecting the wifi to suck? I sure did. Did I expect to write a post bagging on it? You know it. I did my research to see if anyone else had mentioned it, and when that came up empty, I hit the store and picked one up, then came home, tested it, and wrote the post.

While waiting for this fix to come down the pipe to be usable on your systems, there are any number of alternatives. Unfortunately, they all amount to a barnacle that consumes one of your USB ports, but they do work. They tend to actually behave better with tools like Kismet, they do WPA3, and they don't make the kernel panic when you look at them funny!

If this is you, hit the USB-Wifi main menu and start digging around. Note in particular the "plug and play" list. I grabbed some weird $40 Alfa thing I had never heard of before based on a recommendation from the list. It worked great for sniffing things and generally screwing around. It also ran WPA3 as a client just fine.

This is the Linux experience I remember from the 90s: poring over compatibility lists and making sure you buy the right thing every time. That's why it's so vexing that the Pi people would keep shipping this thing in this state. You don't want your customers to keep buying these wifi barnacles, do you?

January 24, 2024: This post has an update.

About a year ago, I wrote about trying to make WPA3 (wireless security) work on a Raspberry Pi 4, among other things. It didn't work then and it still doesn't work now.

In the past couple of weeks, they released the Pi 5, and it's been making the rounds through the usual people, but somehow, nobody's talking about whether it'll do WPA3 or not. So, I'll break the silence and save everyone a lot of work: it still has the same CYW43455 wifi+bluetooth chip as the Pi 4, so it has the same limitations: no WPA3 support, at least, not right now. Maybe some day, someone will do something about the driver situation, but given that nothing has changed in almost a year since my last post, I'm not going to hold my breath.

This Broadcom / Infineon / Cypress wifi situation has other ramifications beyond just the Raspberry Pi ecosystem. Let's say you are like me and you bought one of the "early 2015" Macbook Pros in 2017 since the then-current ones had the terrible new keyboard and still hadn't figured out the whole USB-C thing yet. You didn't get Ventura (13) and are stuck on Monterey (12), never mind Sonoma (14).

So, maybe you thought "I know, I'll install Linux on this thing since it still has some life left in it". Once you do that, you will discover that you are in the same crappy wifi situation. You won't be able to join a WPA3 network in Linux, either. The hardware is clearly able to support it since it worked as a macOS install, but it's just not going to happen on Linux. It's the same sort of software problem.

None of this is news, but the updates about this mess seem to not get enough visibility. Note: "seriously unmaintained and years behind on features and firmware integration". Wonderful.

This puts the Raspberry Pi squarely in the same bucket as "random Internet of Shit devices" when it comes to wifi compatibility. You are going to have to keep a terrible auxiliary wireless network around for a very long time in order to support them as-is.

Going forward, you can just ignore the built-in hardware and buy a dumb little USB wifi adapter which is actually supported by regular kernels. Then you just plug it in, configure things, and go on with your life. (Or, you know, pull actual cabling to it, and live life with the stability of hardwired Ethernet.)

It's kind of amazing that this situation persists, given how the Pi is supposedly intended for entry-level people who need a low-cost platform to learn about stuff. What happens when they encounter a WPA3-only network? They do exist, and they will only become more numerous over time.

November 7, 2023: This post has an update.

There are more than a few bear traps in the larger Unix environment that continue to catch people off-guard. One of the perennial favorites is thread safety, particularly as it applies to the environment manipulation functions under glibc. The usual warning is that if you run multiple threads, you'd best not call setenv, because if someone else calls getenv, you run a decent chance of segfaulting.

The last time I talked about this was 2017 and I said something goofy like "see you in 2022" at the bottom. Well, it's a little late, but it's come up again, and this time it's biting people who aren't even using C or C++!

This one is in Go!

Last time, I said that sometimes people run afoul of this by using mktime() which calls getenv()... and then sometimes pick *that* up by using libzip. Here, it's a little different. getaddrinfo() calls getenv(). Did you know that? Before a few minutes ago, I sure didn't! Check out the resolv.conf (5) man page - it looks at LOCALDOMAIN and RES_OPTIONS.

getaddrinfo() is pretty much required if you want to connect to anything beyond mere legacy IPv4 gunk, so it's not like you can avoid it. You're probably going to call it quite often if you are opening connections over the IPv6 Internet.

If you're on Linux, and you're using glibc, you're probably a passenger on this boat. Try not to drill any more holes.

Thanks to Evan for the tip on this one (and good luck with the fix).

Back in March, I wrote about how the HTML generation worked for all of these posts. In short, it was mostly "loose approximations of HTML via printf", and it was terrible. It generally worked for the past 12 years, but I knew how wrong it was.

One thing I didn't mention in that post was just how bad the feed generation had become. I was doing a CDATA thing and was just spewing out the HTML inside of that. In theory, if I had put a "]]>" in a post, it probably would have broken the entire feed.

Then there are the finer points of the metadata for the feed. It was using an id of "tag:rachelbythebay.com,writing-2011" which I could have sworn was fine at the time I picked it, but which turns out to actually be illegal for that scheme. I fixed that for both the feed and the entries themselves, so if you see duplicates of the last 100 posts, or the entire feed shows up somewhere else, that might be why.

Those and many more things caused the w3c Atom validator to scream *quite* loudly about it being broken. A lot of people sent me feedback about this over the past few months.

That's all gone. A few minutes ago, I threw the last switch to finally cut over the entirety of the /w/ files to the new stuff. This meant that every single index.html has been regenerated. Quite a few corrections have been applied at the same time. It took me a very long time to go through all of these posts and convert my raw HTML shenanigans into meaningful commands that will be parsed by the generator.

Every view of things now actually makes sense. I can now write <foo> as an example in a post and it will come out escaped properly on the output side. I can put an & in the post without having to literally type in &amp. Yes, I'd been having to manually do &lt; and &gt; and all of this... if I remembered. If not, well, there'd be a "live" tag hanging out in the post!

Or, there'd be a broken tag. Last week's post about ASCII protocol buffers and config files actually had a "<pre" without a ">" in it, and then it just went into the contents. I bet you didn't see "syntax = proto2" in that thing as a result, but trust me, it was there!

There are a bunch of other little stylistic changes in here. The feed icon in the banner of both the top index and the individual posts no longer links to the feed itself, but rather a page that explains what to do next. This is a cheesy way to sidestep the "someone clicked on the feed in their browser and used up their unconditional request" thing for a little bit.

Anyone who was using a phone or other smaller device probably noticed that if you loaded a sufficiently old post, it wouldn't quite fit the screen properly whereas newer ones would. This is because I had added a little "viewport" magic in the headers at some point, but had only applied it to the template file (!), and never rebuilt all of the old posts. This also meant the post footers were all slightly different, depending on when it was last rebuilt. Now all of the posts have that viewport thing.

I also adjusted the line-height based on some feedback from at least one reader who commented that it would make it easier to read. I find it hard to argue with that, and didn't see anything bad about it, so that's in there too.

You should also notice that preformatted blocks now sport a different background color and a border to set them apart from the rest of the post.

There are probably some other things I've forgotten, too.

I'm sure there are going to be some anomalies, so if you see something that seems broken to you, go ahead and fire off some feedback. I do appreciate it.

While I don't go on the Orange Site any more, I still make enough trips through the larger space of similar sites to get some idea of what people are talking about. Last week, the topic of interest seemed to be YAML and how evil it is. I can't argue with that. Every time I've crossed paths with it, I've been irritated by both it and whoever decided to use it for their stuff.

The discussions invariably start talking about alternatives, and frequently end up on JSON. This is unfortunate.

I've mentioned this before in passing, but have never given it a whole post. Today, it graduates to having a whole post about the topic.

The topic is: ASCII-form protocol buffers used as config files.

This was a tip given to me something like 17 years ago when I was "on the inside", and it's turned out very well. Protocol buffers have a canonical ASCII representation, and it accepts comments, too! You get the benefits of not having to write a scanner or lexer combined with a system in which everything is explicitly specified, right down to the data types.

Here's an example of such a file:

# **** Contains auth data: must be thermo:thermo 0660 or better ****

db_conninfo: "host=localhost dbname=foo user=xyz_role password= ...

# barn (hardwired)
server_info {
  host: "172.25.161.10"
  port: "18099"
}

# barn (backup wireless on IoS network)
server_info {
  host: "172.25.225.10"
  port: "18099"
}

# office (broken 20230825)
# server_info {
#   host: "172.25.161.17"
#   port: "18099"
# }

sensor_location {
  name: "loft"
  model: "Acurite-Tower"
  id: "1563"
  channel: "A"
}

sensor_location {
  name: "entry"
  model: "Acurite-Tower"
  id: "2375"
  channel: "B"
}

There. That's not terrible, right? It has a bunch of common stuff that gets repeated as needed for my different servers and sensors. There's also a string that gets handed to Postgres to connect to the database. And yes, notice the comments everywhere.

Over in protobuf-land, this is what the .proto file looks like for that config format:

syntax = "proto2";

package thermo;

message LoggerConfig {
  message ServerInfo {
    required string host = 1;         // 192.168.31.67
    required string port = 2;         // 18099
  }

  message SensorLocation {
    required string name = 1;             // room
    required string model = 2;            // Acurite-Tower
    required string id = 3;               // 1015
    required string channel = 4;          // C
  }

  required string db_conninfo = 1;

  repeated ServerInfo server_info = 2;
  repeated SensorLocation sensor_location = 3;

There's one important bit here: I'm using "required" here since this is a config file format and NOT something that will be passed around over the network. It lets me cheat on the field presence checks, and this is the one case where it's acceptable to me.

If you're using protobuf for anything that gets handed around to something else (RPC, files that get written by the program, ...), whether across space *or time* (i.e., future instances of yourself), use optional and explicitly test for the presence of fields you need in your own code. You have been warned.

How does the program use it? First, it reads the entire config file into a single string. Then it creates a LoggerConfig (the outermost message) and tells the TextFormat flavor of protobuf to ParseFromString into that new message. If that returns true, then we're in business.

I can now do things like hand config.db_conninfo() to Postgres or iterate over config.server_info() or config.sensor_location() to figure out who to talk to and what sensors to care about.

Is it perfect? Definitely not. It's software, which means it will never truly stop sucking, like all other software. It's a dependency that will now follow you, your code, and your binaries around like an albatross. It's yet another shared library that has to be installed wherever you want to run.

But, hey, if you're already paying the price of using protobuf in your projects for some other reason, then why not use it for config storage, too?

In short: I should have looked at my notes instead of relying purely on my memory of a random event from four years ago.

Right, so, the other day I wrote a post eviscerating the Philips (Signify) Hue situation, in which they are heading full steam into enshittification. I said that I didn't want to use Home Assistant becuase of Javascript and a "curl | sh" attitude.

Yeah, that's where I screwed up. That's not them. That's actually Homebridge, aka homebridge.io.

People have been commenting that HA is Python, not JS, or something like that. I had to go back to my notes from January 2019 to set this straight, and, well, they go like this:

I was trying to find a way to run some existing "real" security cameras without going full-on cloud mayhem. The users in question wanted to see it in the HomeKit ecosystem, so I went looking for solutions of that sort. The idea was to get this thing running, document the protocol, and then figure out if I could do it some other way.

So I end up on their wiki for installing this on a Raspberry Pi. At that point, I had one just sitting around collecting dust, and figured "what's the harm in trying". First thing up, they wanted me to "curl some-url | sudo bash".

Hell no, I'm not doing that. There are so many things wrong with that philosophy. The whole point of cutting actual releases is that you get people to cluster around a handful of known "artifacts" (you know, tarballs and the like), and then you can work up some kind of reputation based on that. If that actual release ends up in some distribution like Debian, you can be sure that exact version is being seen by a fair number of people.

curl | sh basically says "I don't give a damn" and "give me whatever you want" at the same time.

I figured I could at least grab the script, read it, and parse it myself, then run the commands by hand. This meant I had to add their apt repository. Ugh. First up? "apt-get install -y nodejs" ... oh boy.

But wait, no, then it went on from there.

I tried "npm install -g homebridge" but that wasn't happening. It wouldn't go until I added "--unsafe-perm". Oh, gee, that's not sketchy *at all*.

At this point I was glad this was all happening on what was effectively a throwaway machine. It did get installed, and started up, then displayed a QR code, and once past a few warnings, it did in fact show up in Homekit.

Of course, without plugins, it wouldn't do anything, so that meant going back for MORE node stuff. I found 72 pages of them on npmjs.com.

I picked something really stupid to export a temperature value as a test. I put a config stanza in config.json to try to activate it, and nope, it started dying. So I took the config-sample.json and copied that in its place, and that much worked, and told me what I needed to do: the blob of JSON crap from the plugin's page is supposed to be an entry in the accessories array of the config.json. That was not at all obvious.

I didn't want to do much more with this, and that's about where it ended.

So, yeah, bagging on Home Assistant for being JS and making you install it with sketchy curl pipelines? That was a mistake. I got it mixed up in my head with Homebridge based on something I did four years ago.

I loaded up the HA page on github, and oh hey, Python.

Mmm, yeah. Righto. Okay then.

How about them Knicks?