I recently shared a war story about how we had load problems on a new web server but not an older one because of a missing firewall rule. In a comment, Aristotle Pagaltzis asked how we'd realized that we had a missing firewall rule. This is a good question, because the sequence of events involves a certain amount of luck and coincidence. So here's that story, which may be a useful example of system administration in action in practice (ie, it's kind of messy).

When we first deployed our new web server for a highly in-demand data set, Apache wasn't configured for very many concurrent connections and was immediately overloaded with HTTP requests, but we mostly shrugged. Because this upset our monitoring system and I did want to monitor the web server at least a bit, I turned up the Apache connection limits to an absurd number. This was mostly enough, and the web server's outgoing bandwidth jumped up to 1G wire rates, which at the time I thought was fine. Soon after that, an apparently unrelated machine began to have NFS performance problems, and after some work we realized that it was on the same 1G switch as the highly active web server and its network traffic was getting crowded out.

In a rush to limit the web server's bandwidth usage so the much more important other machine would stop having NFS problems, I hastily added some tc based bandwidth limits by hand, and by this I mean that I typed 'tc' commands in a shell session. A few days later, we worked out how to make mod_qos based limits work in Apache itself. One of the limits we imposed and fiddled with was a limit on the number of concurrent connections from a single IP address, which had a much bigger effect than I expected. The Apache error log showed that some IPs were hitting it quite a lot, and our metrics system said the number of concurrent Apache requests dropped dramatically afterward (to about half what they'd been before).

After the dust settled from the immediate crisis, we needed to decide if we were going to make the tc-based limits a permanent part of the machine's configuration (making it the first machine officially using tc in production, and we'd have to write something to install them on system startup) or if we'd rely entirely on mod_qos. While considering the tradeoffs involved, I remembered that we had a general purpose 'block brute force things' system on our perimeter firewall, so we should probably make it apply to HTTP and HTTPS requests to our new web server for extra insurance. Immediately after I did this, our metrics system showed another major drop in concurrent Apache connections (shrinking by half again).

(The perimeter firewall's system has a list of IP addresses that it applies the HTTP and HTTPS limits to, and the new web server, with a new IP address, wasn't in the list (until I added it).)

That major drop from the firewall rule was what sparked my realization of what was different that would explain why our main web server mostly hadn't been overwhelmed by this traffic, because that was the only thing out of all of our rate limiting changes that our main web server had in place. Our main web server had no tc limits and we'd turned off mod_qos several years ago and never revisited that change.

PS: We decided to keep both the Apache mod_qos limits and the tc based limits, partly for extra insurance. We've decided that we really don't want this particular web server to run over the bandwidth limits, so having two mechanisms that limits it means they'd both have to fail.

(We're not yet worried enough to look into FreeBSD PF's features for bandwidth limits, partly because if we got it wrong on the firewall, we could affect a lot more than this machine.)

I mentioned recently that we'd moved a very popular data set from our main web server to a new one that only handled that data set. On the new web server, we found it necessary to set an absurdly high Apache connection limit of 4,000 concurrent requests, because Apache could run out otherwise (and could even run out at 4,000, it was just less often).

Our main web server has a much lower concurrent connection limit but it didn't experience these problems (and not because we'd imposed connection limits in Apache; we'd turned that off in December of 2022). But as we (I) wrangled with the new web server to get it to stop running out of connections and so on, and even after I got mod_qos working on it, I never paused to ask myself why the new web server had so many problems with this when the main web server had run for years without explosions (or at least very infrequent ones).

(Part of this was because the main web server had started exploding sometimes; that was why we'd moved this data set to its own server. But even those explosions had been less severe than what I was seeing.)

The answer is that for years, our perimeter firewall has had per-IP connection rate limits for HTTP and HTTPS connection to our main web server (among other per-IP connection rate limits, for example for SSH connections). These predate the modern popularity of this data set and were added to stop other abuse, but it turns out that a lot of the connection volume for this data set was coming from a few IPs that were opening up a ton of rapid-fire and often simultaneous connections. Once we applied per-IP limits on both the number of simultaneous connections you could have (in mod_qos) and the rate at which you could make connections (in the firewall), the new web server's connection count dropped like a stone (but people still kept pulling data from it as fast as they could).

In retrospect, the change in the web server's behavior when we moved this data set to a new host was a signal. We'd had only occasional problems on the old web server host one (despite it being actively used for other things) and we had constant ones on the new server (dedicated only to the data set). So we could have asked what was different, and then investigated, and then found the perimeter firewall issue. But on the other hand, this is sort of hindsight bias speaking. Such changes in behavior are a signal, but as system administrators we're drowning in signals and we have to sort out what's meaningful and what's either a coincidence or a consequence of something else (for example, a sudden increase in demand for this data set, which would have also explained why we were suddenly seeing problems even on the main web server).

PS: There's some recent evidence that there was a real but temporary shift in demand for this data set, in large part from people (or software) that make extremely inefficient requests. If these people are done now, or have improved their software, that would be nice. Perhaps all of the connection blocking and limited bandwidth have encouraged them to download things only once and then keep local caches.

An exciting change landed in the development version of Prometheus recently, making PromQL arithmetic expressions in time durations a standard feature instead of an experimental one. For me, duration arithmetic expressions by themselves aren't the truly interesting part. What's really exciting is that as part of this change, Prometheus has added some new PromQL functions, especially step().

(Although step() is gated behind an 'experimental PromQL functions' feature flag today, it will be made available as a standard function as part of duration expressions becoming a standard feature.)

For people who are familiar with Grafana, step() is the PromQL version of Grafana's $__interval interpolation variable. When you're in a range query, step() is how big the range step is, which was previously unavailable in PromQL even though Prometheus obviously knew this information. Prometheus has also added range(), which gives you the full size of the range duration (and the *_of() functions for further selection if, for example, the step() might be too small). Since PromQL now allows arithmetic expressions in durations, you can use step() in them, allowing you to write PromQL expressions like 'rate(your_metric[step()])', where the duration will automatically adjust to whatever the range step is.

Where step() is especially handy for me is when I'm doing ad-hoc graphs directly in Prometheus's web query interface (instead of wrestling with Grafana Explore). Previously I had to go through various increasingly elaborate processes to find out what the step value was for a given time range, so I could plug it into rate() or various *_over_time() things or the like. Now I can just ask for 'rate(...[step()])' and it will all work out right, and it will keep working right as I zoom the time scale in or out.

In theory one could replace various Grafana uses of $__interval in PromQL queries with step(). In practice this probably isn't worth it, unless you're running into problems with $__interval for some reason or maybe when you're writing completely new queries for new dashboards and so on. The minor advantage of using step() even in Grafana is that you can easily copy the query out of Grafana and put it directly into Prometheus or a query tool to see exactly what you're getting.

To use step() and friends in Prometheus 3.13, you need to enable some feature flags. But since this is going to be in the next version of Prometheus (unless something goes wrong and the Prometheus developers have to back out this change), I think it's pretty safe to turn on the necessary feature flags and start using step() and friends now, at least in ad-hoc poking around. Even if you have to stop using step() later, it will improve your experience today.

(This elaborates on a Fediverse post of mine.)

Today I shared a brief war story on the Fediverse:

Today's new and exciting failure mode for a DHCP server handing out leases to dynamic clients: have something on your network that answers pings for absolutely every IP address (yes, it was broken). The ISC DHCP server pings what it thinks is a free IP before handing it out (to be sure), so if something answers all of those pings you have no 'free' IPs on your network and no one gets a dynamic IP.

(Technically this was like a day or two ago.)

The direct symptom of this in your ISC DHCP server logs is some log lines that look like this:

dhcpd[1656384]: Reclaiming abandoned lease 172.17.101.132.
[...]
dhcpd[1656384]: ICMP Echo reply while lease 172.17.101.132 valid.
dhcpd[1656384]: Abandoning IP address 172.17.101.132: pinged before offer

As ISC dhcpd documents (for example in dhcpd.conf's discussion of the 'ping-check' statement), by default dhcpd will ping an IP it's about to dynamically allocate to make sure it's unused. If something answers, dhcpd more or less gives up on the IP address (this doesn't happen for statically assigned IPs, at least according to the dhcpd.conf manual page). The consequence of this is that if you have such a 'screaming' machine, one that's answering ICMP pings for all IP addresses, dhcpd will conclude that your dynamic IP address pool is entirely exhausted and no dynamic client will be able to lease a new IP. For extra fun, apparently some clients will not accept a DHCP IP if there seems to be something else using it.

(I'm not sure what happens when clients are renewing leases.)

Such a screaming machine is obviously broken in some way and you need it off your network, but unless you get lucky, tracking it down may be hard. We were lucky that the machine was using its real MAC to answer all of the pings (which showed up all over the DHCP server's ARP table, among other places) and that MAC was registered with some useful and accurate additional information. Without that we would have been reduced to tracing through switch ARP tables (for switches smart enough to report that) and eventually unplugging sections of this particular network, which would have been pretty disruptive.

This particular network is port isolated, but that doesn't help here. Our DHCP server has to be able to reach the entire network and the entire network reach it, so its ARP requests flood through the network and anyone can answer them (and then its ICMP ping will be routed through the switch fabric to whatever port answered).

In a comment on my entry about how sometimes it's the network that's the problem, I was asked what networking workstations typically have on our networks (and if we still use 100 MBit for low speed things). The simple answer is that it's somewhat mixed but mostly 1G Ethernet, often with 1G uplinks from the switches that the office network jacks are connected to. But there's some stories involved.

As a university department, we have been in our current buildings for what is mostly multiple decades, and in some cases since the early 1980s (cf). In the oldest building (with our old machine room), the department's presence predates twisted pair network wiring entirely (there is still disconnected thicknet wiring around our offices); our other most used building predates Cat-6 cabling. As a result, all of this older space is wired with Cat-5 twisted pair and is only good for 1G Ethernet.

We started out using all of this wiring at 100 MBit (including our connection to the university backbone), and even as time went by we had a mixture of 100 MBit and 1G switches and connections. Due to cost reasons and sometimes wiring density problems, we only slowly migrated people's office network jacks from 100 MBit to 1G as we progressively replaced switches in our multi-switch setup. However, we did eventually move everything to 1G, including in our machine rooms (partly because our old 100 MBit switches were getting unreliable). I think most of our '1G' switches in wiring closets and so on are purely 1G, with no 10G uplink, so wired workstations on those 1G network jacks are sharing the uplink bandwidth.

Areas in buildings do get renovated from time to time, including parts of the department's space. For years, these renovations have typically included running new Cat-6 (or Cat-6A) wiring that is properly certified and tested for 10G-T (it's hard to significantly renovate space without needing to tear out the old wiring and then put in replacement wiring). Sometimes the people who are funding the space renovation really do want their desktops to have 10G connections available, and in that case the renovation funding will also include a collection of 10G-T switches for the other end of the office network jacks to plug in to (however many switches needed for however many true 10G office ports, which is often all of them). Otherwise, while we have 10G-T capable wiring in the walls, we only plug our end of the wires (in wiring closets and machine rooms) into 1G switches and everyone gets 1G.

(Newly built out space in new buildings is all wired with Cat-6A, of course, and we try to get funding for 10G-T switches to go with it. There's been some of this over the recent enough past, as the university does acquire a certain amount of new space over time.)

All of this is the theoretical answer. The practical answer is that a lot of people are using laptops (possibly with docking stations) or desktops with wireless built in, and many people default to using the wireless connection even if there's a network jack or three in their office. Typically, the only people who use actual wired connections are those that need their machines on one of our special research group private networks, instead of on our general 'random desktops and laptops' network. In some areas of the department, there's almost no use of wired network jacks because almost everyone is using laptops and the wireless.

(This makes wireless into critical network infrastructure.)

Here in 2026, I wouldn't try to use 100 MBit for anything, even things that don't need 1G, because I wouldn't trust it. In theory your 1G network interfaces and switches still support it (and maybe your 10G ones too, but don't count on it). In practice, it's been a fairly long time since 100 MBit hardware was in common use. Any switches or the like that you still have lying around are old, and these code paths and physical capabilities in network ports aren't likely to be heavily tested. You want to be running your networks in common and well supported setups, and today that means 1G is the minimum speed you should target.

(And if you have cabling that seems to only support 100 MBit, it's probably broken. Network cables do go bad over time.)

We've recently been having mysterious problems getting some of our backups to perform well. Also I recently wrote about how we'd wound up with a web server that frequently saturated its outgoing 1G interface with traffic (and it was a feature that it didn't have a faster network link). These two things turn out to not be as unrelated as we'd like, and there's a story or two there.

The problematic backups aren't done through our usual Amanda-based backup system; instead, a particular central machine pulls /var/log and other relevant files from FreeBSD and OpenBSD hosts (either through rsync or direct SSH access) and then writes them over NFS to one of our NFS fileservers (and then they get backed up by the Amanda backups). The problems manifested as terrible NFS performance; the central machine's load average would go to 50 or 70, everything NFS related on it would be slow, and so on. While we like NFS in general we've had quite a share of problems with it (cf, also), so we immediately assumed that this was yet another instance of an NFS problem. First we reduced the IO load from rsyncs in various ways (eg), then we spent quite a while digging at various NFS metrics, both from our metrics system and from the live system during a problem. Nothing really seemed to be the problem; the fileservers had perfectly good performance, the master machine had perfectly terrible actual NFS performance, and especially it didn't seem to be able to write the backup data very fast, in the range of a few MBytes a second.

Since I was pretty sure that the central machine could write to its local disk acceptably fast, I considered switching to a more complex backup scheme where we first rsync'd things to the local disk, packed up this directory tree into a tar archive, then scp'd it to the relevant fileserver, thereby bypassing any NFS write issues. To check that this would work acceptably fast, I started by scp'ing a test file from the central machine to the fileserver. To my surprise, the scp ran at only about 2 MBytes/s. More testing showed that scp's from this central machine to anywhere ran at 2 MBytes/sec at most, which did rather explain the NFS write problems (much like the lack of a CPU explains a server's failure to power on). At this point a penny dropped in my mind.

We're a university department, which means that we don't necessarily have the newest, shiniest stuff around and we keep things in service for a long time. One of the things we've kept in service is basic 1G switches, because quite a lot of servers don't need more than 1G (and as mentioned, sometimes it's a feature that a server only has 1G). But when I say 'basic 1G switches' I mean switches that are so old that all of their ports are 1G, including the port we use for 'uplinking' them into our overall 10G switch fabric (in contrast to modern 1G switches, many of which have one or two SFP+ ports that can run at 10G, or even a 10G-T port or two). This is fine for our normal 1G servers, which don't generate or receive much traffic even in aggregate, but it breaks down badly the moment you put a high volume 1G server on such a switch. For example, our new high volume web server, which was not only saturating its own outgoing 1G interface but was also saturating the 1G uplink from the switch it was connected to, a switch which unfortunately also had this critical central machine connected to it.

There's a programmer saying that if you think you have a compiler bug, you don't, you have a regular bug that you haven't spotted yet. This saying is almost always true, and it generalizes to other areas, like kernel bugs (also) or hardware problems, or networking problems. At our scale, modern networks are reliable, so if we don't have anything obvious wrong (for example, our monitoring system hasn't alerted us that some machine is unexpected at 100 MBit/sec), our network is almost certainly working fine. So for days it didn't occur to us to actually check. Of course the network was working fine, the network is always working fine. It obviously had to be NFS, especially since NFS has been flaky for us in the past. Except that sometimes it is the network and there's even a good explanation for it that becomes obvious once you realize it's the network.

This whole experience has given me some things to think about. On the one hand, years ago I wrote about how an obvious problem isn't necessarily obvious. There are a ton of things that can be wrong and you have to winnow through them somehow. On the other hand, I'm pretty sure that if we'd engaged in systematic troubleshooting from the ground up, we'd have found this pretty early on. For example, the USE method would have had us look at usage, saturation, and critically 'errors', which might well have caused us to look at TCP retransmits on the central machine (which were decidedly high).

Being systematic about troubleshooting is generally a good thing, but at the same time it's tedious. If the problem had really been a NFS problem (as it has been in the past) and I'd followed the USE method from the ground up, I'd have spent a chunk of time verifying that yes, the network was performing fine on both machines (along with the other things I looked at it, like NFS server metrics). Possibly what I should try to do is start out with likely guesses and then when they come up dry (eg, there are no obvious reasons for a NFS performance problem) and I'm getting frustrated, fall back to the USE method or something similar, even though it's possibly tedious.

Suppose that you're generating and transferring a file over ssh, for example to create a tar archive of something on a remote server and save it locally:

ssh [...] rem-server 'cd /var/log && tar -cf - .' >/backup/file.tar

If you're experiencing IO problems in this backup process, an interesting question is what buffer size ssh uses for its writes, perhaps because you'd like to make a few large writes to disk (for example, at the natural 128 KByte block size for your ZFS fileservers, even when you're writing over NFS) instead of a bunch of smaller ones.

The ssh manual page doesn't document anything about this, and there's no options to control it in either ssh or ssh_config(5). On Linux, running over TCP from a remote machine, the answer appears to be that ssh will normally do 32 KByte writes (or 64 KByte writes under some circumstances, possibly if the network is fast enough). It's possible ssh will write smaller buffers if the remote command generating the output can't keep up at full network bandwidth, and in general I suspect that there are a lot of things that can change this.

If this is important to you (and I'm not convinced it's important to us), you need to re-buffer the output from ssh in some way. The traditional way is to use dd, but you need to pick the right options to have dd reblock its input. There are other programs floating around but I don't know if any of them are standard.

(Since I looked it up, there's at least the Debian buffer package and mbuffer. There are probably others out there as well that I can't dig up in casual Internet searches.)

Finding out this information requires some way to trace ssh's system call activity. On Linux this is most easily done with 'strace', and you can narrow down all of the system calls that ssh does with 'strace -e trace-fd=5 ...'. You might think that you want to trace file descriptor 1 (standard output), but in fact current versions of OpenSSH ssh rewires its standard output on to file descriptor 5 and make file descriptor 1 point to /dev/null (which can be very confusing when you first encounter it).

(This is one of those things that I look into, don't find much, and then want to write down my negative results anyway for future use.)

Suppose, not hypothetically, that you use rsync to push an encrypted backup file from the machine it's created on to a fileserver, where it will be backed up by your regular backup system. Because this encrypted backup file is backed up every day, you only need one copy of it on the filesystem, so you use and reuse a fixed name for the file. In other words, we're using rsync somewhat as if it was scp, but with better control over what remote files can be written and (not) read.

When you push (or pull) a file over rsync, rsync normally attempts to optimize what gets transferred by looking for common blocks in the file (how big a 'block' is depends on the file size, or you can fix it with the '--block-size' option, as covered in rsync(1)). This is a nice potential bandwidth saving, but it creates CPU and IO load on both ends as each of them checks their version of the file. In our specific case, we know that there aren't going to be common blocks; since the whole file is encrypted, it's basically random noise. Recently this backup process had some IO load problems, and today in the process of working on this I discovered rsync's '-W' option (also known as '--whole-file'). As the manual page explains, this 'disables rsync's delta-transfer algorithm'; in other words, it stops looking for common pieces between the two versions of the file. Rsync simply sends the whole file (and the receiver writes the whole file).

Since we know that today's encrypted file has no blocks in common with yesterday's encrypted file (well, it had better not if the encryption is working right), '-W' is exactly what we want to stop the receiving rsync daemon from doing unnecessary work (specifically, unnecessary IO). Effectively it turns the 'file copy' part of rsync into scp (although not literally; rsync will normally write the new version of the file to a temporary file and then replace the old version). Now that I know about -W, I'm going to be looking at some of our other uses of rsync to see if we might want to use it more widely.

(For example, we use rsync to back up /var/log from some FreeBSD hosts, and I'm pretty sure that's a good candidate for -W too.)

If you're using '-W', you want to avoid using '--checksum' and instead rely on the default 'quick check' of file size and modification time. This is because using --checksum requires rsync to read and checksum the whole file before the transfer starts (which is something that the manual page warns you about).

One lesson I've taken from today's experience is that when I use rsync, I should think about what I want to optimize (and what can actually be optimized). Rsync's default behavior is to optimize transfer bandwidth, but sometimes you have enough transfer bandwidth and you want to optimize for lower IO, lower CPU, or both (which is sort of our case for this encrypted backup file, with the extra issue that we know rsync can't reduce the transfer size). Alternately, sometimes you really want to squeeze the bandwidth and maybe '-S' and '-z' (and perhaps others) are what you want, even though you'll do more work on both ends.

(It's possible that rsync already has a clever encoding for runs of zero bytes and so '-S' doesn't save you any transfer bandwidth. I haven't tested.)

(This elaborates on a Fediverse post of mine.)

There are a variety of ways to automatically or semi-automatically install systems, especially Linux systems and especially over the network. People have built a whole raft of them over the years, often with relatively impressive capabilities. A number of them have significant levels of automation and control, including for things we might want like sophisticated automatic disk setup. Despite that, we're interested in approximately none of them. As a practical matter, we only want to use the standard, native install systems for whatever we're running, which in this case is the Ubuntu server installer. There are two reasons for this.

The obvious reason is that only native installers and install methods are officially supported by the Linux distribution (or whatever other free Unix we're using), and they're also the ones that are most used in practice. The native installers aren't always perfect, but other people work (often quite a lot) on making them work well, we can find plenty of people using them, and so on. If we use another installer, at the very least we're without support from the distribution. We're also likely to be off the beaten path, which means fewer people running into issues before us, fewer bug reports being filed and fixed, and so on.

The less obvious reason is that we do a certain amount of direct manual installation of servers for Ubuntu, FreeBSD, OpenBSD, and so on. The easiest way to do this is to use whatever native installer the distribution ships on their install ISO images. In the case of Ubuntu we customize this a bit, but none of our customizations are essential; we can and have installed systems straight from the main Ubuntu ISO images (and then possibly imported our customizations afterward). We could still do this even if we normally used a third party install system, but less would carry over between the two environments and we'd want to learn and stay familiar with both of them.

These issues aren't absolute blocks on using a third party installer system; we could deal with both of them. But we'd want to be getting something important from such a system, something that was enough of a gain to be worth the extra costs. I can think of situations which would be worth it but they haven't come up so far (at our modest scale and with only Ubuntu really in the picture for semi-automated installs, network installs, and so on).

(This is related to why it took us a relatively long time to build up a network install system, cf. There are specialized automated network install systems for Linux, but they run into this third party issue. Our current network install system for Ubuntu uses our Ubuntu ISO images that we also use for local installs, and that's a feature even though it creates complications.)

Once upon what's now a long while ago, we had a bunch of SunFire X2100 and X2200 1U servers (for example, they were used in our first generation ZFS fileserver). One of the reasons that I loved these servers back in the days was that for free, they came with a full IPMI/BMC setup that supported both KVM over IP and virtual media. I installed any number of servers this way from the comfort of my office, using the KVM over IP as if I was at the console (because I effectively was) and the IPMI virtual media to feed a local ISO image to the server I was installing. In time those servers went the way of all servers (which is to say, into the e-waste dumpster, although it took a while) and none of our later servers gave me that '(re)install from the office' experience with their limited BMCs. For years, we lived with doing in person physical installs and reinstalls of our servers (well, I lived with that, my co-workers don't care as much).

Recently we wound up with a bunch of servers in another building and a need to reinstall them all with Ubuntu 26.04. This pushed me into learning about UEFI network boot, working out how to network boot our customized Ubuntu ISO image, and then the realization that if we were reinstalling an existing, running server we could use a somewhat simpler kexec-based method. This has given us a reinstall experience (and sometimes an install experience) that looks a fair bit like the SunFire X2100 BMC based experience, and can be done from comfort of our office (instead of a noisy machine room a block or two away). My co-workers like it enough that we're talking about using our new network reinstall system even for servers in our main machine room.

On the one hand, it's nice that we can now have this experience with basically any system (it's nicer if it supports network booting, but reinstalls work without that). On the other hand, the experience is a lot more fragile than the BMC-based experience. Our network installs only works because the Ubuntu server installer supports access over SSH, it requires either a running system on the server or a whole collection of network booting infrastructure (including that it be enabled on the server). If the installer blows up (as it does too often in 26.04), you can't necessarily get access to the system or force a power cycle, and you may be stuck needing to go visit the server in person.

(The most practical version of the network install experience also relies on the server having enough RAM to download the ISO image. This seems to require 8 GB of RAM in common situations in my testing, which is far more than our SunFire X2100s normally had.)

This change from hardware support on limited machines to general software that more or less replicates the earlier experience (but imperfectly) feels like a pattern that's happened repeatedly. One advantage of the software version is that it generalizes, for example to virtual machines.

(Virtual machines can have a BMC-like experience, but it's all with a bespoke environment that's specific to that VM system. Network installs are generic across physical servers, virtual machines, and so on.)

PS: Of course the SunFire BMC used a Java applet, which was somewhat painful (cf). Modern BMCs thankfully use straightforward web stuff.

One of my standard little scripts is something I call sshup, which waits for a machine to be 'up' by periodically checking to see if its SSH port is responding. As mentioned in my original entry on sshup, I actually have two versions of this script and recently I discovered that the difference is quietly important.

One version of the script uses Netcat, on our Ubuntu machines. The specific Netcat command line it uses is:

nc -w3 -q3 -z "$1" ssh >/dev/null 2>&1

(I didn't used to need the redirection, but Debian reverted a patch and Ubuntu 24.04 picked up the change.)

The technical effect of this is to try to connect to the SSH TCP port and exit once it's done that (or been rejected or timed out), with an appropriate exit status.

The other version doesn't use Netcat, but in Netcat terms what it does is the equivalent of:

nc -N "$1" ssh </dev/null >/dev/null >2&1

The technical effect of this is to try to connect to the server (if possible), immediately shut down the sending side of the TCP conversation, and wait until the server closes the connection completely.

If the server is working properly, these two versions have the same result. But if the server isn't working, the answer these give is different, with the second version being more useful. The problem with the first version is that it only checks if the kernel is willing to let you make a TCP connection to the SSH port; it doesn't check if the SSH daemon is actually responding (normally, the SSH daemon will print a banner, then see that there's nothing more to read from the network and close the connection). If the server is broken sufficiently so that the kernel will accept the connection but the SSH daemon won't run, the first version will tell me that the machine is up and the second one will correctly tell me I can't SSH in to the machine.

(Recently a server hung during shutdown in exactly this way, which is a story for another entry.)

This difference is also relevant for health and monitoring checks for services (and famously so). Connecting to a TCP port is the very basic step in a health check; after that is checking that you received a banner or a canned response from a service (for HTTP services, perhaps a '200 Okay' from a health check endpoint), and after that is checking that you can do something meaningful that's part of the service's regular activity. Of course the higher level you go in checking the service, the more specific your health check is to the service (whether you implement the check in code or in a configuration file that says what to look for). But at the same time, it's more meaningful because it comes closer and closer to what actually matters.

(But there's a tradeoff in how close you come versus how much work and so on it is. You make that tradeoff for individual services based partly on your knowledge of how that service works and what's likely to fail. For my 'is a machine up' checking script, seeing the SSH server banner is good enough basically all of the time. And if my sshup script says the machine is up but then I can't actually log in, that's telling me something useful too.)

Recently I wrote an entry about a simple but difficult wish I had for a certain sort of terminal pager that handled emoji. In comments, people suggested that the venerable less would do what I wanted, which was something I'd already tried and discovered it had behavior I didn't want. Except, well, let me quote my eventual comment:

It turns out that my testing of less's behavior was being contaminated by my usual $LESS settings, and I think that 'less -XRn' does more or less what I want (I was being fooled partly because my usual $LESS includes 'c', well, technically 'C').

I've been using less for a long time and for most of that time I've had the same set of less settings, burned into a $LESS environment variable set in my dotfiles, and also the same set of alternate less key bindings (where N and P move through files instead of their normal actions). I had long since forgotten the specifics of my $LESS settings when I tested less; my settings were just how less worked. And so I was fooled into initially thinking that less didn't work for what I wanted.

(I remember N and P partly because when I use less as root or on a machine I haven't particularly set up my account on, I don't have them and I miss them.)

My $LESS setting is far from the only environment variable I have set in my dotfiles. Without looking through them carefully I couldn't tell you what I have set, why I set them that way, and whether or not they're still correct or useful. The same is true for the configuration files for various programs, where I'm sure that surprising things lurk.

(This isn't the first time I've had old personal dotfiles cause problems.)

This is of course just the personal manifestation of a general problem we have in system administration. We set up a lot of settings and configurations that make sense and work at the time, and then we forget about them because they're in the background, just doing what we expect. They can readily become our mental image of how the program or system behaves, because it's certainly how it behaves for us. Then someday either they clash with what we want to do (as my standard $LESS did) or they don't work any more because things changed out from underneath them (cf).

I don't have any good answers to this. Some people will advocate not changing any settings from the defaults, but not only is that giving yourself countless little papercuts over time, it doesn't even necessarily protect you; programs can change their default behavior so something you've become accustomed to doesn't work any more.

(It's good to document the settings and why you made them, but that only helps with half of the problem. It's still easy to forget that you even have custom settings. Should you budget an hour every six months or whatever to look over and check your settings? Well, you can put it on the calendar, but I don't think it's going to work in practice, not after the first few times, because it's boring.)

One of the broadly accepted things among at least system administrators is that our own code is a liability (some programmers may resist this idea somewhat more). If your systems run on a collection of locally developed, bespoke programs, scripts, and so on, someone has to maintain and update all of that (I sort of wrote about this long ago, also). If you use standard programs instead (or at least as much standard code as possible), hopefully someone else does that, and you can choose which programs and what code to use based on how well maintained it seems to be at the moment.

(This breaks down when you're large enough, but I'm talking about more modest sized organizations or groups.)

It's recently struck me that the obvious extension of this is that configuration and configuration files are also a cost and a liability (partly as a result of thinking about the staff time cost of our email system). This is obvious when a configuration file embeds what is effectively code, such as Prometheus alerting rules (which have a whole array of clever tricks that you wind up learning), because code in any form is still code. But it goes beyond such embedded code into any configuration that you have to maintain. If you need a complex Bind configuration file or a complex DHCP setup or a tangled Apache configuration (perhaps for a very special server), some day they're probably going to have to be changed or at least understood again.

We can't operate with no configuration and we shouldn't try to. Under the right circumstances, code and configuration are both more of an asset than a liability. But I think we should be cautious about it. Just because we can create a complex configuration that does a lot doesn't mean that we should. It's tempting to create a complex configuration that does lots of things to meet your needs and support all the features you want, and at least for me there's an appealing problem solving aspect to it. But the result can be something that has long term costs and consequences.

Have I created some overly complicated configurations at work? Probably yes. These configurations have given us useful or important features, but they've also saddled us with complexity that we're now stuck with because we (the sysadmins) and other people have come to depend on the features the configurations create.

(Ultimately, one issue is that as a system administrator, it's hard to persuade myself to leave a problem unsolved. Simpler configurations usually mean living without things and we often put in the things for good reason, so simple configurations would require telling more people "you can't have that because we don't feel it's a good idea to support it, although we could".)

In theory, network booting with UEFI is fairly straightforward. Unlike the earlier BIOS based network booting, UEFI defines network booting in the standard itself (although it's not required). In practice, vendors of x86 servers have found any number of creative ways to put stumbling blocks in your way in their firmware (aka 'BIOS'). Here's an incomplete list of reasons that your server might not be willing to do a UEFI network boot the way you expect it.

• The firmware might not even be in UEFI mode, in which case at best it's trying to do BIOS PXE boot, which requires a different set of support infrastructure on your boot server that may not be there (cf, also).

• There's no UEFI network booting option (or options) in the server's UEFI boot order.
• The UEFI network boot option is trying to boot from the wrong interface (on a server with multiple interfaces).
• The firmware only supports UEFI network booting using the built-in 1G network interfaces, not with your 10G-T add-in PCIe card that is actually connected to your network.

• UEFI network booting is enabled in the boot order, but your firmware settings have disabled network booting for all network devices that the firmware will try. Depending on the firmware, this might be a master option that applies to all onboard network interfaces or it might be a per-interface setting.

  (In some firmware this isn't even in the regular 'BIOS' setup screens, but is instead in a completely separate top level configuration system for (some) network devices.)

• Your firmware settings have the network devices in 'legacy' network boot mode (ie, PXE booting) despite your firmware being in UEFI mode, so the firmware won't touch them for UEFI network booting. Sometimes the firmware's setup screens will vaguely describe this as the device firmware being in 'legacy' or 'UEFI' modes, and not specifically mention network booting.

  You might ask why vendors would allow this configuration mismatch. I have no answer, I just know that at least one vendor does have UEFI firmware that will let you do this.

If the server's firmware has a specific 'PXE/UEFI network boot' option that it offers to you on startup (along with things like 'enter setup' or a regular boot menu), this boot option may or may not provide any useful information if the firmware decides it has no valid UEFI network device to boot from. You may be in a situation where you pound on F12 (or whatever key is applicable) but nothing happens and you drop through to whatever 'boot failed' option the firmware has available.

(If the firmware is feeling especially hostile, it will drop you into the UEFI shell [PDF], and see also the Arch wiki page.)

Similarly, if the server offers you a general one time boot menu, but when you pick it there's no UEFI network boot options, this could mean either that the general UEFI boot ordering in the firmware doesn't have network booting included or that the firmware thinks there's no valid UEFI network devices.

There are typically at least two ways to disable (UEFI) network booting on servers: you can take it out of the boot order, or you can tell the often built in network devices to not do network booting. If you've historically disabled network booting on your servers because you don't use it and typically all it did was add extra time to server boots as the firmware poked at network devices, you may have done either or both of these and now need to reverse either or both.

(In non-UEFI, BIOS MBR boot mode, network device firmware has historically fiddled around with PXE stuff even if the BIOS boot sequence wasn't going to wind up trying PXE booting. Or at least it did on some of our servers, so we often turned that off.)

All of this can leave you hunting all over your server firmware's setup screens in order to find whatever magic bit that's making the system not boot from the network the way you want it to. Good luck, but start by checking UEFI boot order and (network) device settings.

In a comment on my entry on universities, email, and the issues of running things in house, I mentioned that our departmental email system has a non-trivial cost in both hardware and staff time. For the hardware costs I can easily count servers, but things are more fuzzy for the staff time side of things so I'm not going to try to come up with a number (especially because our email system itself is somewhat complex because we have a number of unusual features, such as our simple mailing lists). The bigger issue is that the basic time cost to maintain your email system is in some sense an illusion, or at least an inadequate measure.

The reality of modern email is that to run a modern email system you need to know about a lot of stuff, and worse the stuff that you need to know about keeps evolving. You can't set up a mail system and then walk away from it apart from software and hardware upgrades; instead, you have to keep on top of a perpetually changing and evolving landscape of anti-spam systems and especially what you need to keep your outgoing email being delivered. SPF, DKIM, DMARC, DMARC alignment, and so on are merely today's names; in a few years there will be more, different things that are necessary to know and deal with, which will require you to modify your mail system as part of its obvious time requirements.

Obtaining this knowledge and keeping up on developments in email is often more or less invisible time. Without it, your system may work today (especially if you set it up from a good guide) but things get riskier as time goes on. At best, you notice when outgoing email starts bouncing; at worse, you have no idea until people start complaining to you (if you're lucky) that email they send isn't getting received properly. Then, if you're lucky, you can find another guide for the current new reality of email; otherwise, you're in for research and learning on your own. To copy from an old story, your time spent might be half an hour to make the eventual change to your mail system but a day of research to know what to change and why.

(A similar thing exists with anti-spam and anti-malware handling for incoming email. In a modest sized environment (such as ours) you have no realistic choice but to outsource much of that to some free or paid piece of software, but that still leaves you to monitor the overall situation, determine when your current software is falling behind, and then find and configure new software.)

Of course you need to develop and maintain a certain amount of expertise with your mail software, but that's usually easier to keep current. Your mail software typically makes changes less frequently than the overall Internet mail environment does, and those changes will often be explicitly documented in release notes, news entries, and changelogs. Still, depending on how much complexity you opt for in your mail system, this may take a significant amount of time initially (learning Exim was in no way an overnight thing for me, and then I had to design and build our configuration because Exim is a construction kit).

(We have what I certainly hope is an unusually complicated mail system (also, also).)

If you have a multi-person system administration team and only one person from it goes through all of this and learns all about email (both things like DKIM and things like your local mail setup), what you've got is a single point of expertise. If that person is on vacation and a mail problem comes up, you kind of have a problem. To avoid this, you'll need more staff time for more people to be at least somewhat up to speed on email stuff and how to deal with problems. At one level this is no different than any other system you operate, but those systems probably evolve slower than Internet mail does and require less time to keep up on.

None of this is insurmountable. But it's not trivial either. Maintaining your own mail system (in the broad sense) is going to take a non-trivial amount of staff time for one or more people to keep up on Internet mail developments, monitor your mail system for signs of problems like too much incoming spam and malware getting through or too little outgoing mail reaching its destinations, and troubleshoot issues. This need for maintained expertise is part of why it can be simpler to outsource email to specialists who deal with it a lot.

(Also, as you deal with a larger population of people using your systems, you also need to worry about compromised accounts sending out bad stuff.)

One of the things we operate for the research side of the department is an old fashioned general purpose web server, where everyone has a home page area of their own in the traditional '/~<login>/' style (cf). This web server has been there for a very long time, and one of the decisions that was made very early on was that for security reasons, the web server would not NFS mount people's regular home directories from our fileservers.

The traditional Apache way to do '/~<login>/' home pages is to have some location under your home directory that's exposed as your web home page area; the traditional name for this is 'public_html'. One alternative is to relocate this to a separate directory tree, but this directory tree is flat, which makes it awkward to have different pools of disk space for different people (which is absolutely required for us). Since we didn't want to use people's regular home directories for security reasons and we couldn't put everyone in one directory, we did the obvious hack: people have a different, special home directory on the web server. These home directories are in special 'webdir' filesystems on our fileservers, and these webdir filesystems are the only NFS filesystems that the web server NFS mounts.

The result is that everyone actually has two home directories in two different filesystems (although those two filesystems will come from the same ZFS pool). They have their regular home directory filesystem, which is accessible on our login and compute servers but not the web server, and their 'webdir' home directory, which is accessible everywhere. To make this more convenient to people, we create a 'public_html' symlink in people's regular home directories that points to the 'public_html' in their webdir home directory. If people have personally run web servers, these and their support files also live in the 'webdir' home directory, for relatively obvious reasons.

(We have a special short name form of people's home directories, so on the web server this short form points to their web home directory. The public_html symlink combined with this means that '/u/<login>/public_html' always refers to your web home page directory tree no matter what machine you're on.)

Because everyone's web home directory filesystem is in the same ZFS pool as their normal home directory filesystem, the web server still depends on all of our ZFS fileservers. Since our web server is reasonably active (also, also), it tends to react very rapidly to any NFS fileserver hiccups.

PS: The web home directory security decision predates me, so I don't know why it was made, but in my view it's a perfectly sensible decision. In general you should probably assume that your web server can be coaxed into reading and disclosing any Unix file that it has filesystem level access to. If you don't like the implications of this, you need to arrange for it to have access to fewer files. A dedicated set of filesystems is one relatively straightforward way to do that.

In a comment on this recent entry, I was asked if we still considered Unix logins to be a serious security boundary. This is a sensible question; there are a horde of Linux local privilege escalation vulnerabilities going around right now (and one FreeBSD one for spice), and in general (some) security people have been saying for years that once an attacker had local code execution, the game was over. Our answer is that yes, we consider it a serious security boundary, and if that situation ever changed we'd need a drastically different system environment from our current environment.

Our current environment has shared NFS fileservers where people keep all their files and data, shared login servers for both general usage and compute, a (shared) SLURM computer cluster, and a reasonably flexible shared web server environment where people can run programs. While some people are still using our login servers interactively, others are running software (such as VSCode) that connects to them somewhat behind the scenes and uses them to run tools. All of this is critically dependent on the security provided by Unix logins; if Unix logins weren't a real security boundary any more, anyone on any of these machines could read other people's files or run programs as them.

Since these machines are all shared machines with multiple people logged in at once, switching to Kerberos authenticated NFS wouldn't solve the problem. If we assume that attackers can merely become any other person, then they can gain access to the Kerberos tickets of anyone else who's currently logged in and access their files. If we assume that attackers can compromise root, then all bets are off and once a person has used that machine it can't be trusted for any future use (since the attacker could have compromised programs to capture the login credentials of future people logging in).

Basically, if you lose the security boundaries of Unix logins, you lose shared machines. You need to create a new environment without sharing (or with sharing boundaries that people can't break out of). Today, it appears that the only way to do that securely is a separate virtual machine for each person, with Kerberos authentication to our NFS fileservers (given some of the Linux security issues, containers are clearly not good enough). I'm not sure how you manage a SLURM cluster in this environment, but it certainly wouldn't be the straightforward way we do it today.

This would be a drastic change for people here and it would also be a significant increase in resource requirements (since realistic virtual machines are much more heavyweight than even full login sessions). We couldn't leave 'your' virtual machine (or machines) running all the time (we have too many people using our systems for that), so you'd have to use some web interface to request it be started with some resource allocation. Managing, maintaining, and updating these virtual machine images and running VMs would be at least a bit painful, and people would probably experience more disruption in their activities. Some things would become effectively impossible, such as running CGIs on our web server.

In a comment on my entry on universities, email, and the issues of running things in house, I mentioned that our departmental email system has a non-trivial cost in hardware alone to keep going. To better illustrate that, I'll describe all of the servers that our email system currently requires (because it's more than one). Some of these servers exist for historical reasons and may go away at some point, but many of them don't.

Currently, we have:

• A server as our external mail gateway (our DNS MX target). This is separate from other mail servers because it's much simpler to configure and operate this way.

• A server for the (FOSS) anti-spam and anti-virus software we use (and everyone needs some version of). This could be folded into the mail gateway server (and it was in our recent backup MX, but we weren't sure about the software's resource usage and system impact when we set it up. Keeping it separate also means we can move it to a new OS version for more up to date software without having to worry about any changes in new versions of the mailer that the mail gateway runs.

• A server for our central mail machine that handles all aspects of email to local addresses, which for various reasons (cf) can include sending email to the outside world. This machine doesn't store any email locally; instead, to simplify slightly, email lives on our general purpose NFS fileservers.

• A separate server to handle forwarding known spam to outside email addresses. We're required to support this by people using our email system and we found it necessary to put this work on a separate machine.

• A server to handle unauthenticated mail submission from inside our networks. Separating mail submission from the central mail machine makes for a simpler configuration for both (eg), and we historically started with only an unauthenticated mail submission machine.

• A fairly powerful server to handle IMAP and authenticated SMTP submission, which these days also has /var/mail (where all our inboxes live) on local storage and thus also acts as a NFS server.

• A server for a webmail frontend (to our IMAP server). We put this on a separate server than IMAP for multiple reasons, including resource usage and that it decouples the OS and packaged software version requirements of our webmail (for instance, certain versions of PHP and Apache) from everything else.

We've found it very important for practical reasons to use separate IP addresses for different sorts of outgoing email (also). We can do this on a single machine (and we do), but in many ways it's simpler to use separate machines for different sorts of email. It's also simpler to handle things like rate limits if we use different machines for things that need different rate limits.

All of these servers rely on existing elements of our general infrastructure, such as our general purpose NFS fileservers, our local DNS resolvers, and our system of propagating account information. I hope that at some point in the future our IMAP server machine will also wind up relying on our local OIDC identity provider (and indirectly on the LDAP server it uses), but that's currently not possible in practice. I'm mentioning these because a stand-alone mail environment would require some equivalent of all of them; you have to store mailboxes somewhere, get account and authentication information, do DNS resolution, and so on.

Most of these servers are 'basic' 1U servers, which these days means that they have 16 GB to 32 GB of RAM, a mirrored pair of SATA SSDs, a reasonable CPU, and traditionally cost a few thousand dollars each if bought new (their prices are probably higher at the moment). These specifications are good enough that we don't have to worry about the exact resource requirements of each server's job (although we made sure to give the anti-spam software machine 32 GB of RAM and a decent CPU). If we used smaller machines we'd have to be more careful; I'm pretty sure that not all of these roles would be happy with only 8 GB of RAM in practice (much less 4 GB). Basic 1U servers used to be cheaper, and these days we've got a stock of older servers that are good enough for these jobs. But if we were setting up a green field environment from scratch and had to buy all of these new, five or six servers (possibly plus a spare) would be a non-trivial cost.

(Because we're using the same sort of servers for these as we use for everything else, there's no dedicated spare for specific machines; we have spare server hardware in general.)

The one server that is an exception is our IMAP server. The current version has 64 GB, four relatively large SATA SSDs, a decent CPU, and 10G-T networking, and because it's so important we have a spare server ready to be pressed into use immediately in case of a hardware failure. The current hardware is old enough that we'd like to replace it, this time with more memory (so more things get cached) and NVMe SSDs instead of SATA ones. Unfortunately, in the current environment the price quotes we got are jaw dropping and unpleasant (especially since we have to buy two of the basic server to have a spare, although we don't need two sets of the NVMe drives).

All of this serves a department with somewhat over a thousand active people, about 1.5 TBytes of inboxes (if we talk about the likely uncompressed size; since we use ZFS for /var/mail, we have compression turned on), and an inbound mail volume that is probably around 10,000 messages a day. As mail system sizes go, this is modest.

(We have several thousand inboxes (and Unix accounts to go with them), but many of them are inactive for various reasons. The size distribution of inboxes is also extremely uneven, as you might guess.)

(Publication of this entry was delayed by me getting distracted and forgetting to actually publish it last night. I didn't realize it was still sitting in my drafts area until I noticed the stray editor window just now.)

We had a weekend long power shutdown this past weekend in the building with our main machine room. As is our custom, we powered off the servers before hand (on Friday evening, with some surprises), and then turned them back on this morning. This isn't the first time we've gone through such a power shutdown (although usually they're shorter), and over time we've written checklists and lessons learned for these things. This time was no exception, so I wrote checklists for both powering down everything and powering it back up (well in advance for once). Then we collectively looked at my nice, detailed, step by step power on checklist and ripped it up.

The issue is that powering things up in our environment is not really an orderly, step by step process. A lot of our systems are both core things and relatively independent of each other, and while there are ordering dependencies (our fileservers have to be up before any NFS client, for example, and the DNS resolvers need to be up before the fileservers), they're small and at the start. Even in the ordering there's a lot that can be done at once, such as booting up all of the fileservers at once.

This structure, or lack of it, doesn't particularly fit in the traditional checklist format and process, which sort of assumes that you have a real order to things. Our power up process is more anarchic than that; at best it proceeds in stages, and even then there are multiple stages that can be done at once (such as turning on most of the firewalls and turning on the fileservers; neither depends on the other). Adding to the mix is the potential need to either troubleshoot things like failed PDUs or non-booted switches, or to decide to defer them to later.

This isn't the first time I've written up a power up list and had it more or less abandoned in practice (and our retrospective 'what actually happened' worklogs even talked about it). This is just the first time I've really admitted it up front.

I'm not sure what the best form of documentation is for our orderly cold start power up requirements, but it's certainly not a detailed checklist or anything that claims to be a linear narrative. Maybe what we want to do is simply list what everything requires, starting from the machines that don't require anything. Then everyone involved can look at what has all its requirements satisfied and go for it.

(A complication is that there are also some things that are ideally started early but if they're having problems it's not critical. For example, it's nice to have our central syslog server up early to collect everyone's logs right from the start, but it's not essential in the way that, say, our NFS fileservers or our local DNS resolvers are.)

One of the recent commotions in my corner of the tech sphere was over an incident where a piece of software deleted a company's production database and all of its backups. The software got all of the backups too because, I'll quote:

[Their SaaS provider] stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — [...]

A lot of people were horrified, but I had some sympathies with the SaaS provider. An important thing about backups is how backups work depends on what you're trying to recover from, and for certain sorts of disasters and recoveries, this decision is perfectly sensible. For a SaaS company, they also depend on customer support needs and what customers are going to want, and the decision can also make sense from that perspective.

In this case, the obvious question is whether the SaaS provider is trying to protect customers from loss of data in the volume or from deliberate deletion of the volume. If what you're protecting people from is an accidental 'DROP TABLE' or an accidental 'rm' (or an accidental overwrite of something important), then in volume backups such as ZFS snapshots make perfect sense. We use ZFS snapshots ourselves for this purpose on some filesystems (although they're not our only form of backups). As a bonus, restores are much faster than external backups. However, backups tied directly to the volume aren't a good ideal if what you're protecting people against is deletion of the volume itself.

(The SaaS provider itself might be concerned about loss of the volume from things other than deliberate deletion, but this isn't a concern customers want to have; they want to pretend that the SaaS provider has 100% reliable handling of volumes until they delete them. Of course, this can lead to unpleasant customer surprises if something goes wrong, which is why wise customers have completely external backups so they don't have to trust the SaaS provider and the SaaS provider's cloud vendor. The people this happened to were not wise customers, but if you've heard of this incident, you already knew that.)

If a SaaS provider wants to potentially protect people from deliberate deletion of a volume, there are a bunch of tradeoffs. For example, you're probably charging people for out of volume backups in some way, which means that if people really want to delete an unused volume, they also want to delete its backups so they're not being charged for those either. If you surface an option for 'also delete backups of this volume' so that people deleting volumes can handle the situation right away and aren't surprised by charges later, what you're surfacing is an easy total data loss option; people will reflexively say "yes" and wipe out their backups too.

(After all, typically people who delete volumes think they're doing the right thing at the time. Software agents don't think but they're generally going to behave in the same way.)

The harder you make it to delete volume backups, the more you're going to annoy some of your customers who really do want to delete their volume backups (or perhaps many of your customers, since you'd hope that almost all volume deletions are customers making the right choice and they probably don't want the backups either). At a certain point, a SaaS provider might take a rational look at their data on what people are deleting and what they're recovering from (and customer support calls), and conclude that hard to delete volume backups aren't worth it because customers don't use the extra resilience and are annoyed by the side effects of it. Perhaps you can design both your systems and your charging to get around this, but it's more product development work and if you're a SaaS company, you have a lot of other product development work you could be doing and that other work may have much higher value to your company.

(Convenient, easily accessible in volume backups may also have side effects. The space consumption side effects of ZFS backups are why we don't use them pervasively for all of our fileserver ZFS filesystems.)

Locally we use external backups, but this is because we're operating physical storage and so we have to be concerned about all sorts of catastrophic things happening to it. Our external backups are slower to restore from for in-volume damage like deleted files, but we have to make that tradeoff because we absolutely have to be able to recover from a total loss of a ZFS filesystem, ZFS pool, or an entire fileserver (or our entire machine room).

We have a whole building, weekend long power shutdown in the building with our machine room that officially starts tomorrow (Saturday) morning at 5am, which is the motivation for our newly added temporary backup MX. Because we like to be in control of both the shutdown and the startup of our machines, we turn machines off in advance for scheduled outages (there's not much we can do about unscheduled ones). For various reasons we did the shutdown earlier this evening.

(One reason to start machines under controlled circumstances is that sometimes hardware fails, things go wrong, or you discover unfortunate aspects of your environment (also). At least these days we've mostly learned lessons from previous power shutdowns and startups, although there are aspects I hadn't fully absorbed and will write about later.)

During the shutdown, something surprising happened, which is that all of our ZFS fileservers came back to life. We definitely ran 'poweroff' on each of them and they were off the network for some amount of time, but then my co-workers doing work in the machine room noticed that they were all powered back on. We ran 'poweroff' on the rebooted servers and they shut down properly, rather than rebooting, so that part's not the problem. After some discussion we decided to deal with the immediate problem by pulling their power plugs, so they can't come back on even if something on board wants them to (all of these servers have BMCs).

One of the things we did between the fileservers shutting down and them coming back up is that I ran fping to scan the subnet they're on, to see if we'd missed shutting down any machines (and this fping run showed that none of them were on the network at the time). The host I ran fping from was on the same network and would have still had the MAC addresses of the fileservers in its ARP cache, so it could have directly unicast packets to the MAC.

One theory we have is that this triggered some sort of 'Wake on LAN' power up behavior. I wasn't pinging with a WoL 'Magic Packet', but as covered in sources like the Linux ethtool(8) manual pages, your hardware may potentially support a whole host of WoL mechanisms, including 'unicast messages'. This sounds like it might cause a server to wake up if its network interface receives a packet to its hardware MAC. Such as, for example, an ICMP ping packet that didn't need an ARP because the sending host already knew the target's MAC.

(I can't find much documentation on what these Wake on LAN options mean, but see eg here, this chipset documentation, or FreeBSD's ifconfig and its 'wol' options.)

When the power shutdown is over and we bring the fileservers back up on Monday, we'll be looking at what 'ethtool' reports as their Wake on LAN settings. Since they have fully capable BMCs, we may want to force all of them to have no Wake on LAN active at all. Certainly it seems undesired to have them potentially powering up based on just receiving packets, since there's a whole host of ways they could receive traffic.

PS: We haven't seen this in past power shutdowns, but our fileserver hardware was refreshed between the last one and now.

I recently mentioned that we'd built a backup MX server due to concerns prompted by a scheduled power outage. In a comment on that entry, Greg A. Woods said something that I broadly agree with:

I think backup MX hosts are, generally speaking, a bad idea in modern times (even going back a couple of decades).

[...]

The added maintenance overhead and headache of keeping a full-time backup MX host running and reliably forwarding ALL email it collects, and reliably rejecting all email it should reject, isn't usually worth the bother.

One reason that we implemented a backup MX is that this isn't our experience. Our backup MX was easy to build and is essentially trivial to keep in reliable operation. However, this isn't because we have some special trick to running backup MXes; instead, it's because we have a general mail architecture that enables it.

Many, many years ago we moved from a mail architecture that was essentially monolithic to one that had an external MX gateway that was stuck in front of our central mail server. This transition involved creating what I call a 'white-box' mailer environment, where knowledge of things like valid local addresses and domains was materialized in text files and reusable in many contexts. Our spam and virus filtering is also done with FOSS components, which we can more or less run as many copies of as we like.

So our backup MX is essentially a clone of our regular external MX gateway machine, except that it has the MTA and the anti-spam stuff on the same machine (and we may do this for the next version of the external MX gateway, now we know more about how much load the anti-spam stuff creates). The backup MX server uses the same white-box mail information that our external MX gateway machine does, and we arranged for it to sit in a network environment where it could deliver accepted mail straight to our central mail server (instead of later delivering it to the normal external MX gateway, which would have added more hops and more redundant spam checking).

(All of the changes from the regular external MX gateway were things that we already had in operation on other machines and needed only modest tweaks to deal with the unique parts of this one.)

This is only possible because we already had all of the pieces. We have a general framework for installing and operating servers, we had an external MX gateway separate from the main mail system, that external MX gateway didn't rely on internal services to do things like validate addresses, and we didn't have commercial software involved that might have had license restrictions that prevented us from running an extra copy on our new backup MX.

We're also making life easier on ourselves by only running this backup MX temporarily, and with a configuration for valid email addresses, spam settings, and so on that is effectively frozen because all of the machines and services that could change any of that are powered off. That way we don't have to worry about what happens if the network connection between the backup MX and us gets blocked and the backup MX starts drifting out of sync on what email addresses are valid and so on.

If we hadn't already moved from a monolithic black-box mailer environment to a multi-machine white box one, building and running a backup MX host would have had all of the issues that Greg A. Woods identified. The existence of some of these issues is part of why spammers like to probe your backup MX. Also, in general I still agree with my old entry on the case against a full time backup MX, although modern email makes me nervous about the potential for aggressive mail delivery timeouts.

(In my old terminology, what we've built is technically a redundant MX. But that's a happy accident of the available network connectivity where this machine is going to be located for the power outage, and it could have had to deliver mail to our regular external MX gateway.)

There are at least three ways that x86 machines can try to boot from the network; BIOS PXE boot, UEFI PXE boot, and UEFI HTTP boot. All of them start by the machine asking a DHCP server for what it should boot, and all of them require different answers from the DHCP server. If you want to support more than one network booting option, your DHCP server needs to give each sort of client the right answer for it, which generally means you have to tell the DHCP server how to tell the types of clients apart.

(If you have all modern machines you can probably get away with only supporting UEFI PXE booting, which will simplify your life slightly.)

The DHCP server we use is the standard and now old-fashioned ISC DHCP server. There are a variety of guides for how to configure your ISC DHCP server for multiple types of network booting, but for various reasons I'm writing my own. This one is actually tested in real use (I've booted machines all three ways from this configuration).

When DHCP clients send out network booting requests, they include two important pieces of information, their "vendor class identifier" and their 'architecture'; these are DHCP option code 60 and DHCP option code 93 respectively. The vendor class identifier is a string and the architecture is a 16-bit integer. ISC DHCP has names for both options, vendor-class-identifier and pxe-system-type respectively (cf), although the latter appears to be recent enough that a lot of Internet writeups think you have to define it yourself in your dhcpd.conf, eg:

option pxe-arch code 93 = unsigned integer 16;

Since I didn't read up on all of this before this entry, my dhcpd.conf contains this superstition and I haven't (yet) tested a version without it.

If all you care about is UEFI x86 systems, you can use the vendor class identifier to tell apart UEFI PXE booting and UEFI HTTP booting. In PXE booting, it starts with 'PXEClient', and in HTTP booting, it starts with 'HTTPClient'. This results in a configuration snippet that looks like this:

class "pxeclients" {
  # TFTP
  match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
  next-server X.Y.Z.Q;
  filename "/grub/shimx64.efi";
}
class "httpclients" {
  match if substring (option vendor-class-identifier, 0, 10) = "HTTPClient";
  # the v-c-i in the reply is required
  option vendor-class-identifier "HTTPClient";
  filename "http://X.Y.Z.Q/grub/shimx64.efi";
}

If you also want to handle BIOS PXE systems, you need something more complicated, because both BIOS PXE and UEFI PXE have a vendor class identifier that starts with 'PXEClient'. You can be more precise by matching more of the vendor class identifier because it also includes an 'Arch:XXXXX' string (cf), but I think it's simpler to switch to using the 'architecture' number (which is what the 'Arch:' part is telling you anyway). The official list of architecture types is IANA's Processor Architecture Types, and one thing to know when reading it is that 'x64' is 64-bit x86, not Itanium. In practice with x86, what you'll see is 0x00 (BIOS PXE), 0x07 (UEFI PXE), and 0x10 (UEFI HTTP). In your dhcpd.conf, this looks like:

if (option pxe-arch = 00:10) {
  # The v-c-i is required
  option vendor-class-identifier "HTTPClient";
  filename "http://X.Y.Z.Q/grub/shimx64.efi";
} else if (option pxe-arch = 00:07) {
  next-server X.Y.Z.Q;
  filename "/grub/shimx64.efi";
} else {
  next-server X.Y.Z.Q;
  filename "/pxe/lpxelinux.0";
}

(Technically I should check pxe-arch for the last clause.)

I believe you can use the official 'pxe-system-type' here instead of my self-defined version, but I'm copying this example straight from my known-working dhcpd.conf. Also, as covered in dhcp-eval, possibly this would be more clearly written as a switch statement. I may experiment with both changes later, but this is what's working for me today.

(See also my entry on the various steps of a network install from an Ubuntu server ISO, which discusses the shimx64.efi and lpxelinux.0 bits a bit more.)

Over on the Fediverse I indulged in some snark:

Network inventory hot and grumpy take: Yep, it's not great that sysadmins and network people don't necessarily have a hardware and network inventory, unlike modern software development where famously everyone knows exactly what their entire dependency tree is and why it's there and has full trust in it staying that way.

(That is sarcasm.)

Let's get this out of the way right at the start: inventories are hard. I don't just mean network inventories or machine inventories or software inventories or dependency inventories. I mean any and all inventories, everywhere. For example, some real businesses periodically take a day or two off from doing business in order to check and reconcile their inventory with actual physical reality. It's ordinary to have a business's website say they have something in stock at a location, but when you go to the location, the people there can only shrug and tell you they have no idea where the theoretically in-stock item is, if it even exists.

(I can also assure you that an inventory of other physical items, even very important ones like keys, can become completely hopeless. One reason lots of people like reprogrammable electronic locks is that you can make your inventory be the authoritative state of the world. Of course, this will also lead to you discovering ways in which your inventory did not reflect reality, as people turn up who should have access but aren't in your lock inventory.)

One reason that all inventories are hard is that they're an attempt to keep two (or more) things in sync with each other, those being the inventory itself and the physical or software reality. Not coincidentally, in our field the most accurate inventories tend to be the ones that are built on self-reporting. Unfortunately there is only so much information that can be accurately self-reported. For example, a machine intrinsically knows that it exists and has certain hardware and software states, but it doesn't intrinsically know why it exists. If you try to make a machine 'self report' why it exists, this is generally going to be the machine echoing back to you something that you told it earlier.

This also relies on being able to get a self report from machines or whatever else is of interest. A machine or a piece of software or whatever that doesn't generate a self report is mostly invisible. Generally self reporting is something that has to be added to machines, software, and other things of interest, and if this isn't complete, that creates gaps in a self reported inventory. You can fill these gaps in the inventory by hand, but then you're trying to keep two things in sync with each other.

The less you can trust self reporting, the harder inventories get. We see this in the perpetual struggle of default deny firewalls, which can be seen as an inventory of allowed network traffic except that we can't allow things to self-report that they should be allowed. This creates a burden of inventory maintenance in the form of firewall rule updates (which is often made more annoying by organizational structure, where you can't update the 'inventory' yourself but have to wait for other people to do it before you can do things).

Ultimately, maintaining an inventory takes work. If you want that work to happen, you must budget time for that work and you must make that work rewarded. If your organization's structure of rewards and demerits makes it clear that maintaining an inventory is not as important as other things, well, you will get what you'd expect.

(Locally, we do budget time to maintain several sorts of inventories, but at the same time many of them are imperfect. Partly there is a trade off between the amount of time spent maintaining inventories and their accuracy, and partly people make mistakes, which is another reason why things self reporting themselves is better if you can manage it.)

Over on the Fediverse, I had a question:

To the sysadmin population of the Fediverse: do people have any numbers on how long common mail senders will retry sending mail if your MX is unreachable? Once upon a time people retried for many days, but my impression is that quite a few places now stop trying and bounce the email after quite short intervals, like a day.

(Boosts and practical experiences welcome, like "my MX was down for three days and I still got all that email sent from GMail".)

The context for my sudden curiosity is that there's a scheduled all-weekend, whole building power outage at the start of May for the building with our machine room. It seems likely that basically all of our systems will be down for roughly two and a half days, and longer if things go wrong, and this obviously includes our incoming email gateway.

As I mentioned, in the old days you could definitely expect mail systems to retry for more than a long weekend and so we wouldn't really worry about it. But I'm not sure about that in practice any more, hence my sudden curiosity. Based on replies to that post (and some additional research), common Unix MTA software still seems reasonably okay on retry durations; postfix and sendmail default to five days, while exim more or less defaults to four. RFC 5321 recommends four to five days in section 4.5.4.1, for what an RFC on SMTP mail is worth these days.

Unfortunately, what matters in practice is how the dominant sources of your email behave, and generally those aren't going to be people running normal Unix mailers in normal configurations. A lot of our email comes from GMail and Office 365, who are obviously using custom mail systems. Office365 covers email from both people at other universities or organizations that use Office 365 and people using the university's central email system to send email to people in my department. It's also possible that configurations vary between organizations using Office365.

There are also all of the people and organizations sending out newsletters, notifications, and so on through the various mailing list service providers, like Amazon SES. These organizations may well have shorter retry times than for individual human-generated email from GMail, Office365, and so on. Another category of email is activity notification emails from places like Github, which people may also want to (eventually) get.

(We have access to an alternate location with a different network and power setup, so we could deploy a backup MX machine to there. There are some potential drawbacks to that, but we may do it as a precaution.)

(Once again it's been a while since the last little script.)

Every so often I find myself in a situation where I have a bunch of lines with multiple columns and I want to either add up all of the numbers in one column (for example, to get total transfer volume from Apache log files) or add up all of the numbers in one column grouped by the value of a second column. This leads to two scripts, which I call 'addup' and 'sumup'.

Addup is a simple awk script that adds up all the values from some column:

#!/bin/sh
# add up column N
awk '{sum += $('$1') } END {print sum}'

(Looking at this now, I should use printf and specify a format to avoid scientific notation. A more sophisticated version would do things like allow you to set the column separator character(s) rather than just using the awk default of whitespace, but so far I haven't needed anything more.)

My version of sumup is more complicated than I've described, partly it either counts up how many times each value happened for a particular field or it computes a sum of another field for the particular field. This sounds abstract, so let me make it more concrete. Suppose that you have a file of lines that look like:

300 thing1
800 thing2
900 thing1
100 thing3
[...]

Sumup can either tell you how many times each of the second field occurs, or sum up the value of the first field for each of the values of the second field (giving you 1200 for thing1, 800 for thing2, and 100 for thing3 in this simple case).

The actual sumup that I currently use is a Python program, partly so that I can conveniently print output sorted by the breakdown field. However, my older awk-based version is:

#!/bin/sh
# sum up field $1 by field $2
# if no $2 is provided, it just counts by one.
(
if [ -n "$2" ]; then 
        awk '{sums[$'$1'] += $'$2'} END {for (i in sums) print sums[i], i}'
else
        awk '{sums[$'$1'] += 1} END {for (i in sums) print sums[i], i}'
fi
) | sort -nr

My memory is that this version works fine, although it's been a while since I used it.

If there are relatively widely available Unix utilities that will do these jobs, I'm not aware of them, although I wouldn't be surprised if they've emerged by now.

PS: Looking at the sort of things I do with these tools, I should also write an 'avgup', although that strays into the lands of statistical analysis where I may also want things like the median.

I mentioned recently that at work, we're retaining old systems that we would have normally discarded. We're doing this for the obvious reason that new servers have become increasingly expensive, due to escalating prices of RAM (especially DDR5 RAM) and all forms of SSDs, especially as new servers might really require us to buy ones that support U.2 NVMe instead of SATA SSDs (because I'm not sure how available SATA SSDs are these days).

Our servers are generally fairly old anyways, so our retention takes two forms. The straightforward one is that we're likely going to slow down completely pushing old servers out of service. Instead, we'll keep them on the shelf for if we want test or low importance machines, and along with that we're probably going to be more careful about which generation of hardware we use for new machines. We've traditionally simply used the latest hardware any time we turn over a machine (for example, updating it to a new Ubuntu version), but this time around a bunch of those will reuse what we consider second generation hardware or even older hardware for machines where we don't care too much if it's down for a day or two.

The second form of retention is that we're sweeping up older hardware that other groups at the university are disposing of, when in the past we'd have passed on the offer or taken only a small number of machines. For example, we just inherited a bunch of Supermicro servers and Lenovo P330 desktops (both old enough that they use DDR4 RAM), and in the past we'd have taken only a few of each at most. These inherited servers are likely to be used as part of what we consider 'second generation' hardware, equivalent to Dell R340s and R240s (and perhaps somewhat better in practice), so we'll use them for somewhat less important machines but ones where we still actually care.

(A couple of the inherited servers have already been reused as test servers.)

The hardware we're inheriting is perfectly good hardware and it'll probably work reliably for years to come (and if not, we have a fair number of spares now). But it's hardware with several years of use and wear already on it, and there's nothing special about it that makes it significantly better than the sort of second generation hardware we already have. However, we're looking at a future where we may not be able to afford to get new general purpose 1U servers and our current server fleet is all we'll have for a few years, even as some of them break or increasingly age out. So we're hoarding what we can get, in case. Maybe we won't need them, but if we do need them and we pass them up now, we'll really regret it.

(The same logic applies to the desktops. We don't have any immediate, obvious use for them, but at the same time they're not something we could get a replacement for if we pass on them now. We'll probably put a number of them to use for things we might not have bothered with it we had to get new machines; for example, I may set one up as a backup for my vintage 2017 office desktop.)

I suspect that there will be more of this sort of retention university-wide, whether or not the retained hardware gets used in the end. We're not in a situation where we can assume a ready supply of fresh hardware, so we'd maybe better hold on to what we have if it still works.

Back in 2022, I wrote about how old our servers were at the time, partly because they're older than you might expect, and today I want to update that with our current situation. My group handles the general departmental infrastructure for the research side of the department (the teaching side is a different group), and we've tended to keep servers for quite a while. Research groups are a different matter; they often have much more modern servers and turn them over much faster.

As in past installments, our normal servers remain Dell 1U servers. What we consider our current generation are Dell R350s, which it looks like we got about two years ago in 2024 (and are now out of production). We still have plenty of Dell R340s and R240s in production, which were our most recent generation in 2022. We still have some Dell R230s and even R210 IIs in production in less important server roles. We also have a fair number of Supermicro servers in production, of assorted ages and in assorted roles (including our fileservers and our giant login server, which is now somewhat old).

(On a casual look, the Dell R210 IIs are all for machines that we consider decidedly unimportant; they're still in service because we haven't had to touch them. Our current view is that R350s are for important servers, and R340s and R240s are acceptable for less important ones.)

In a change from 2022, we turned over the hardware for our fileservers somewhat recently, 'modernizing' all of our ZFS filesystems in the process. The current fileservers have 512 GBytes of RAM in each, so I expect that we'll run this hardware for more than five years unless prices drop drastically back to what they were when we could afford to get a half-dozen machines with a combined multiple terabytes of (DDR5) RAM.

(Today, a single machine with 128 GBytes of DDR5 RAM and some U.2 NVMe drives came out far more expensive than we hoped (and the prices forced us to lower the amount of RAM we were targeting).)

Our SLURM cluster is quite a mix of machines. We have both CPU-focused and GPU-focused machines, and on both sides there's a lot of hand-built machines stuffed into rack cases. On the GPU side, the vendor servers are mostly Dell 3930s; on the CPU side, they're mostly Supermicro servers. A significant number of these servers are relatively old by now; the 3930s appear to date from 2019, for example. We have updated the GPUs somewhat but we mostly haven't bothered to update the servers otherwise, as we assume people mostly want GPU computation in GPU SLURM nodes. Even the CPU nodes are not necessarily the most modern; half of them (still) have Threadripper 2990WX CPUs (launched in 2018, and hand built into the same systems as in 2022). With RAM prices being the way they are, it's unlikely that we'll replace these CPU nodes with anything more recent in the near future.

With current hardware prices being what they are (and current and future likely funding levels), I don't think we're likely to get a new generation of 1U servers in the moderate future. We have one particular important server getting a hardware refresh soon, but apart from that we'll run servers on the hardware we have available today. This may mean we have to accept more hardware failures than usual (our usual amount of server hardware failures is roughly zero), but hopefully we'll have a big enough pool of old spare servers to deal with this.

(I expect us to reuse a lot more old servers than we traditionally have. For instance, our first generation of Linux ZFS fileservers date from 2018 but they've been completely reliable and they have a lot of disk bays and decent amounts of RAM. Surely we can find uses for that.)

PS: If I'm doing the math correctly, we have roughly 10 TBytes of DDR4 RAM of various sizes in machines that report DMI information to our metrics system, compared to roughly 6 TBytes of DDR5 RAM. That DDR5 RAM number is unlikely to go up by much any time soon; the DDR4 number probably will, for various reasons beyond the scope of this entry. This doesn't include our old fileserver hardware, which is currently turned off and not in service (and so not reporting DMI information about their decent amount of DDR4 RAM).

In yesterday's entry I described a lingering duplicate system and how it had taken us a long time to get rid of it, but I got too distracted by the story to write down the general thoughts I had on how this sort of thing happens and keeps happening (also, the story turned out to be longer than I expected). We've had other long running duplicate systems, and often they have more or less the same story as yesterday's disk space usage tracking system.

The first system built is a basic system. It's not a bad system, but it's limited and you know it. You can only afford to gather disk usage information once a day and you have nowhere to put it other than in the filesystem, which makes it easy to find and independent of anything else but also stops it updating when the filesystem fills up. Over time you may improve this system (cheaper updates that happen more often, a limited amount of high resolution information), but the fundamental issues with it stick around.

After a while it becomes possible to build a different, better system (you gather disk usage information every few minutes and put it in your new metrics system), or maybe you just realize how to do a better version from scratch. But often the initial version of this new system has its own limitations or works a bit differently or both, or you've only implemented part of what you'd need for a full replacement of the first system. And maybe you're not sure it will fully work, that it's really the right answer, or if you'll be able to support it over the long term (perhaps the cardinality of the metrics will be too overwhelming).

(You may also be wary of falling victim to the "second system effect", since you know you're building a second system.)

Usually this means that you don't want to go through the effort and risk of immediately replacing the old system with the new system (if it's even immediately possible without more work on the new system). So you use the new system for new stuff (providing dashboards of disk space usage) and keep the old system for the old stuff (the officially supported commands that people know). The old system is working so it's easier to have it stay "for now". Even if you replace part of the use of the old system with the new system, you don't replace all of it.

(If your second system started out as only a partial version of the old system, you may also not be pushed to evolve it so that it could fully replace the old system, or that may only happen slowly. In some ways this is a good thing; you're getting practical experience with the basic version of the new system rather than immediately trying to build the full version. This is a reasonable way to avoid the "second system effect", and may lead you to find out that in the new system you want things to operate differently than the old one.)

Since both the old system and the new system are working, you now generally have little motivation to do more work to get rid of the old system. Until you run into clear limitations of the old system, moving back to only having one system is (usually) cleanup work, not a priority. If you wanted to let the new system run for a while to prove itself, it's also easy to simply lose track of this as a piece of future work; you won't necessarily put it on a calendar, and it's something that might be months or a year out even in the best of circumstances.

(The times when the cleanup is a potential priority are when the old system is using resources that you want back, including money for hardware or cloud stuff, or when the old system requires ongoing work.)

A contributing factor is that you may not be sure about what specific behaviors and bits of the old system other things are depending on. Some of these will be actual designed features that you can perhaps recover from documentation, but others may be things that simply grew that way and became accidentally load bearing. Figuring these out may take careful reverse engineering of how the system works and what things are doing with it, which takes work, and when the old system is working it's easier to leave it there.

We have been operating a fileserver environment for a long time now, back before we used ZFS. When you operate fileservers in a traditional general Unix environment, one of the things you need is disk usage information. So a very long time ago, before I even arrived, people built a very Unix-y system to do this. Every night, raw usage information was generated for each filesystem (for a while with 'du'), written to a special system directory in the filesystem, and then used to create a text file with a report showing currently usage and the daily and weekly change in everyone's usage. A local 'report disk usage' script would then basically run your pager on this file.

After a while, we we able to improve this system by using native ZFS commands to get per-user 'quota' usage information, which made it much faster than the old way (we couldn't do this originally because we started with ZFS before ZFS tracked this information). Later, this made it reasonable to generate a 'frequent' disk usage report every fifteen minutes (with it keeping a day's worth of data), which could be helpful to identify who had suddenly used a lot of disk space; we wrote some scripts to use this information, but never made them as public as the original script. However, all of this had various limitations, including that it stopped updating once the filesystem had filled up.

Shortly after we set up our Prometheus metrics system and actually had a flexible metrics system we could put things into, we started putting disk space usage information into it, giving us more fine grained data, more history (especially fine grained history, where we'd previously only had the past 24 hours), and the ability to put it into Grafana graphs on dashboards. Soon afterward it became obvious that sometimes the best way to expose information is through a command, so we wrote a command to dump out current disk usage information in a relatively primitive form.

Originally this 'getdiskusage' command produced quite raw output because it wasn't really intended for direct use. But over time, people (especially me) kept wanting more features and options and I never quite felt like writing some scripts to sit on top of it when I could just fiddle the code a bit more. Recently, I added some features and tipped myself over a critical edge, where it felt like I could easily re-do the old scripts to get their information from 'getdiskusage' instead of those frequently written files. One thing led to another and so now we have some new documentation and new (and revised) user-visible commands to go with it.

(The raw files were just lines of 'disk-space login', and this was pretty close to what getdiskusage produced already in some modes.)

However, despite replacing the commands, we haven't yet turned off the infrastructure on our fileservers that creates and updates those old disk usage files. Partly this is because I'd want to clean up all the existing generated files rather than leave them to become increasingly out of date, and that's a bit of a pain, and partly it's because of inertia.

Inertia is also a lot of why it took so long to replace the scripts. We've had the raw capability to replace them for roughly six years (since 'getdiskusage' was written, demonstrating that it was easily possible to extract the data from our metrics system in a usable form), and we'd said to each other that we wanted to do it for about that long, but it was always "someday". One reason for the inertia was that the existing old stuff worked fine, more or less, and also we didn't think very many people used it very often because it wasn't really documented or accessible. Perhaps another reason was that we weren't entirely sure we wanted to commit to the new system, or at least to exact form we first implemented our disk space metrics in.

One of my long standing gripes with Debian and Ubuntu is, well, I'll quote myself on the Fediverse:

I understand that Debian wants me to use 'apt' instead of apt-get, but the big reason I don't want to is because you can't turn off that progress bar at the bottom of your screen (or at least if you can it's not documented). That curses progress bar is something that I absolutely don't want (and it would make some of our tooling explode, yes we have tooling around apt-get).

Over time, I've developed opinions on what I want to see tools do for progress reports and other text output, and what I feel is increasingly too clever in tools that makes them more and more inconvenient for me. Today I'm going to try to run down that taxonomy, from best to worst.

1. Line by line output in plain text with no colours.
2. Represent progress by printing successive dots (or other characters) on the line until finally you print a newline. This is easy to capture and process later, since the end result is a newline terminated line with no control characters.

3. Reporting progress by printing dots (or other characters) and then backspacing over them to erase them later. Pagers like less have some ability to handle backspaces, but this will give you heartburn in your own programs.

4. Reporting progress by repeatedly printing a line, backspacing over it, and reprinting it (as apt-get does). This produces a lot more output, but I think less and anything that already deals with backspacing over things will generally be able to handle this. I believe apt-get does this.

5. Any sort of line output with colours (which don't work in my environment, and when they do work they're usually unreadable). Any sort of terminal codes in the output make it complicated to capture the output with tools like script and then look over them later with pagers like less, although less can process a limited amount of terminal codes, including colours.

6. Progress bar animation on one line with cursor controls and other special characters. This looks appealing but generates a lot more output and is increasingly hard for programs like less to display, search, or analyze and process. However, your terminal program of choice is probably still going to see this as line by line output and preserve various aspects of scrollback and so on.

7. Progress output that moves the cursor and the output from its normal line to elsewhere on screen, such as at the bottom (as 'apt autoremove' and other bits of 'apt' do). Now you have a full screen program; viewing, reconstructing, and searching its output later is extremely difficult, and its output will blow up increasingly spectacularly if it's wrong about your window size (including if you resize things while it's running) or what terminal sequences your window responds to. Terminal programs and terminal environments such as tmux or screen may well throw up their hands at doing anything smart with the output, since you look much like a full screen editor, a pager, or programs like top. In some environments this may damage or destroy terminal scrollback.

   An additional reason I dislike this style is that it causes output to not appear at the current line. When I run your command line program, I want your program to print its output right below where I started it, in order, because that's what everything else does. I don't want the output jumping around the screen to random other locations. The only programs I accept that from are genuine full screen programs like top. Programs that insist on displaying things at random places on the screen are not really command line programs, they are TUIs cosplaying being CLIs.

8. Actual full screen output, as a text UI, with the program clearing the screen and printing status reports all over the place. Fortunately I don't think I've seen any 'command line' programs do this; anything that does tends to be clearly labeled as a TUI program, and people mostly don't provide TUIs for command line tools (partly because it's usually more work).

My strong system administrator's opinion is that if you're tempted to do any of these other than the first, you should provide a command line switch to turn these off. Also, you should detect unusual settings of the $TERM environment variable, like 'dumb' or perhaps 'vt100', and automatically disable your smart output. And you should definitely disable your smart output if $TERM isn't set or you're not outputting to a (pseudo-)terminal.

(Programs that insist on fancy output no matter what make me very unhappy.)

Back when we had just started with our current metrics and dashboards adventure, I wrote about how sometimes the simplest version of a graph is a text table. Today I will extend that further: sometimes the simplest version of a text table is to have a command that prints it out, rather than making people look at a web page.

We recently had a major power outage at work, and in the aftermath not all of our machines came back. One of my co-workers is an extreme early bird and he came in to the university about as early as it's possible to on the TTC, and started work on troubleshooting what was going on. One of the things he needed to know was what machines were still down, so he could figure out any common elements to them (and see what machines were stubbornly not coming back on even though they ought to be).

We have Grafana dashboards for this, and the information about what machines are down is present in some of them in tabular form. But it's a table embedded in a widget in a web page, and you need a browser to look at it, which you may not have from the server console of some server you just powered up. Since I like command line tools, at one point I wrote some little scripts that make queries to our Prometheus server with curl and run the result through 'jq' to extract things. One of them is called 'promdownhosts' and it prints out what you'd expect. Initially this was just something I used, but several years ago I mentioned my collection of these scripts to my co-workers and we wound up making them group scripts in a central location.

(I initially wrote this script and a few others for use during our planned power outages and other downtimes, because it was a convenient way of seeing what we hadn't yet turned on or might have missed.)

Early in the morning of that Tuesday, bringing machines back up after the power outage and finding dead PDUs, my co-worker used the 'promdownhosts' script extensively to troubleshoot things. One of the nice aspects of it being a script was that he could put the names of uninteresting machines in a file and then exclude them easily with things like 'promdownhosts | fgrep -v -f /tmp/ignore-these' (something that's much harder to do in a web page dashboard interface, especially if the designer hasn't thought of that). And in general, the script made (and makes) this information quite readily accessible in a compact format that was quick to skim and definitely free of distractions.

Not everything can be presented this way, in a list or a table printed out in plain text from a command line tool. Sometimes tables on a web page are the better option, and it's good to have options in general; sometimes we want to look at this information along with other information too. As I've found out the hard way sometimes, there's only so much information you can cram into a plain text table before the result is increasingly hard to read.

(I have a command that summarizes our current Prometheus alerts and its output is significantly harder to read because I need it to be compact and there's more information to present. It's probably only really suitable for my use because I understand all of its shorthand notations, including the internal Prometheus names for our alerts.)

Early last Tuesday there was a widespread power outage at work, which took out power to our machine rooms for about four hours. Most things came back up when the power was restored, but not everything. One of the things that had happened was that one of our rack PDUs had failed. Fixing this took a surprising amount of work.

We don't normally think about our PDUs very much. They sit there, acting as larger and often smarter versions of power bars, and just, well, work. But both power bars and PDUs can fail eventually, and in our environment rack PDUs tend to last long enough to reach that point. We may replace servers in the racks in our machine rooms, but we don't pull out and replace entire racks all that often. The result is that a rack's initial PDU is likely to stay in the rack until it fails.

(This isn't universal; there are plenty of places that install and remove entire racks at a time. If you're turning over an entire rack, you might replace the PDU at the same time you're replacing all of the rest of it. Whole rack replacement is certainly going to keep your wiring neater.)

A rack PDU failing not a great thing for the obvious reason; it's going to take out much or all of the servers in the rack unless you have dual power supplies on your servers, each connected to a separate PDU. For racks that have been there for a while and gone through a bunch of changes, often it will turn out to be hard to remove and replace the PDU. Maintaining access to remove PDUs is often not a priority either in placing racks in your machine room or in wiring things up, so it's easy for things to get awkward and encrusted. This was one of the things that happened with our failed PDU on last Tuesday; it took quite some work to extract and replace it.

(Some people might have pre-deployed spare PDUs in each rack, but we don't. And if those spare PDUs are already connected to power and turned on, they too can fail over time.)

We're fortunate that we already had spare (smart) PDUs on hand, and we had also pre-configured a couple of them for emergency replacements. If we'd had to order a replacement PDU, things would obviously have been more of a problem. There are probably some research groups around here with their own racks who don't have a spare PDU, because it's an extra chunk of money for an unlikely or uncommon contingency, and they might choose to accept a rack being down for a while.

I have a habit of writing little scripts at work for my own use (perhaps like some number of my readers). They pile up like snowdrifts in my $HOME/adm, except they don't melt away when their time is done but stick around even when they're years obsolete. Every so often I mention one of them to my co-workers; sometimes my co-workers aren't interested, but sometimes they find the script appealing and have me put it into our shared location for 'production' scripts and programs. Sometimes, these production-ized scripts have turned out to be very useful.

(Not infrequently, having my co-workers ask me to move something into 'production' causes me to revise it to make it less of a weird hack. Occasionally this causes drastic changes that significantly improve the script.)

When I say that I mentioned my scripts to my co-workers, that makes it sound more intentional than it often is. A common pattern is that I'll use one of my scripts to get some results that I share, and then my co-workers will ask how I did it and I'll show them the command line, and then they'll ask things like "what is this ~cks/adm/<program> thing' and 'can you put that somewhere more accessible, it sounds handy'. I do sometimes mention scripts unprompted, if I think they're especially useful, but I've written a lot of scripts over time and many of them aren't of much use for anyone beside me (or at least, I think they're too weird to be shared).

If you have your own collection of scripts, maybe your co-workers would find some of them useful. It probably can't hurt to mention some of them every so often. You do have to mention specific scripts; in my experience 'here is a directory of scripts with a README covering what's there' doesn't really motivate people to go look. Mentioning a specific script with what it can do for people is the way to go, especially if you've just used the script to deal with some situation.

(One possible downside of doing this is the amount of work you may need to do in order to turn your quick hack into something that can be operated and maintained by other people over the longer term. In some cases, you may need to completely rewrite things, preserving the ideas but not the implementation.)

PS: Speaking from personal experience, don't try to write a README for your $HOME/adm unless you're the sort of diligent person who will keep it up to date as you add, change, and ideally remove scripts. My $HOME/adm's README is more than a decade out of date.

Suppose, not hypothetically, that you have a system that uses GNU Tar for its full and incremental backups (such as Amanda). Or maybe you use GNU Tar directly for this. If you have an incremental backup tar archive, you might be interested in one or both of two questions, which are in some ways mirrors of each other: what files were deleted between the previous incremental and this incremental, or what's the state of the directory tree as of this incremental (if it and all previous backups it depends on were properly restored).

(These questions are of deep interest to people who may have deleted some amount of files but they're not sure exactly what files have been deleted.)

Handling deleted files is one of the challenges of incremental backups, with various approaches. How GNU Tar handles deleted files is sort of documented in Using tar to perform incremental dumps and Dumpdir, but the documentation doesn't explain it specifically. The simple version is that GNU Tar doesn't explicitly record deletions; instead, every incremental tar archive carries a full listing of the directory tree, covering both things that are in this incremental archive and things that come from previous ones. To deduce deleted files, you have to compare two listings of the directory tree.

(As part of this full listing, an incremental tar archive records every directory, even unchanged ones.)

You can get at these full listings with 'tar --list --incremental --verbose --verbose --file ...', but tar prints them in an inconvenient format. You don't get a directory tree, the way you do with plain 'tar -t'; instead you get the Dumpdir contents of each directory printed out separately, and it's up to you to post-process the results to assemble a directory tree with full paths and so on. People have probably written tools to do this, either from tar's output or by directly reading the GNU Tar incremental tar archive format.

In my view, GNU Tar's approach is sensible and it comes with some useful properties (although there are tradeoffs). Conveniently, you can reconstruct the full directory tree as of that point in time from any single incremental archive; you don't have to go through a series of them to build up the picture. This probably also makes things somewhat more resilient if you're missing some incremental archives in the middle, since at least you know what's supposed to be there but you don't have any copy of. Finding where a single file was deleted is better than it would be if there were explicit deletion records, since you can do a binary search across incrementals to find the first one where it doesn't appear. The lack of explicit deletion reports does make it inconvenient to determine everything that was deleted between two successive incrementals, but on the other hand you can determine what was deleted (or added) between any two tar archives without having to go through every incremental between them.

(You could say that GNU Tar incremental archives have a snapshot of the directory tree state instead of carrying a journal of changes to the state.)

I've been logging in to Unix machines for what is now quite a long time. When I started, it was traditional for your login process to be noisy. The login process itself would tell you last login details and the 'message of the day' ('motd'), and people often made their shell .profile or .login report more things, so you could see things like:

Last login: Tue Feb 10 22:16:14 2026 from 128.100.X.Y
 22:22:42 up 1 day, 11:22,  3 users,  load average: 0.40, 2.95, 3.30
cks cks cks
[output from fortune elided]
: <host> ;

(There is no motd shown here but it otherwise hits the typical high points, including a quote from fortune. People didn't always use 'fortune' itself but printing a randomly selected quote on login used to be common.)

Many years ago I modified my shell environment on our servers so that it wouldn't report the currently logged in users, show the motd, or tell me my last login. But I kept the 'uptime' line:

$ ssh cs.toronto.edu
 22:26:05 up 209 days,  5:26, 167 users,  load average: 0.47, 0.51, 0.60
: apps0.cs ;

Except, I typically didn't see that. I see this only on full login sessions, and when I was in the office I typically used special tools (also, also, also) that didn't actually start a login session and so didn't show me this greeting banner. Only when I was at home did I do SSH logins (with tooling) and so see this, and I didn't do that very much (because I didn't normally work from home, so I had no reason to be routinely opening windows on our servers).

As a long term result of that 2020 thing I work from home a lot more these days and so I open up a lot more SSH logins than I used to. Recently I was thinking about how to make this feel nicer, and it struck me that one of the things I found quietly annoying was that line from 'uptime' (to the point that sometimes my first action on login was to run 'clear', so I had a clean window). It was the one last thing cluttering up 'give me a new window on host X' and making the home experience visibly different from the office experience.

So far I've taken only a small step forward. I've made it so that I skip running 'uptime' if I'm logging in from home and the load on the machine I'm logging in to is sufficiently low to be uninteresting (which is often the case). As I get used to (or really, accept) this little change, I'll probably slowly move to silence 'uptime' more often.

When I think about it, making this change feels long overdue. Printing out all sorts of things on login made sense in a world where I logged in to places relatively infrequently. But that's not the case in my world any more. My terminal windows are mostly transient and I mostly work on servers that I have to start new windows on, and right from very early I made my office environment not treat them as login sessions, with the full output and everything (if I cared about routinely seeing the load on a server, that's what xload was for (cf)).

(I'm bad about admitting to myself that my usage has shifted and old settings no longer make sense.)

Over on the Fediverse, I mentioned that we once missed noticing that there had been a power failure. Naturally there is a story there (and this is the expanded version of what I said in the Fediverse thread). A necessary disclaimer is that this was all some time ago and I may be mangling or mis-remembering some of the details.

My department is spread across multiple buildings, one of which has my group's offices and our ancient machine room (which I believe has been there since the building burned down and was rebuilt). But for various reasons, this building doesn't have any of the department's larger meeting rooms. Once upon a time we had a weekly meeting of all the system administrators (and our manager), both my group and all of the Points of Contact, which amounted to a dozen people or so and needed one of the larger meeting rooms, which was of course in a different building than our machine room.

As I was sitting in the meeting room during one weekly meeting, fiddling around, I tried to get my Linux laptop on either our wireless network or our wired laptop network (it's been long enough that I can't remember which). This was back in the days when networking on Linux laptops wasn't a 100% reliable thing, especially wireless, so I initially assumed that my inability to get on the network was the fault of my laptop and its software. Only after a bit of time and also failing on both wired and wireless networking did I ask to see if anyone else (with a more trustworthy laptop) could get on the network. As a ripple of "no, not me" spread around the room, we realized that something was wrong.

(This was in the days before smartphones were pervasive, and also it must have been before the university-wide wireless network was available in that meeting room.)

What was wrong turned out to be a short power failure that had been isolated to the building that our machine room was in. Had people been in their offices, the problem would have been immediately obvious; we'd have seen all networking fail, and the people in the building would have seen the lights go out and so on. But because the power issue hit at exactly the time that we were all in our weekly meeting in a different building, we missed it.

(My memory is that by the time we'd reached the machine room the power was coming back, but obviously we had a variety of work to do to clean the situation up so that was it for the meeting.)

For extra irony, the building we were meeting in was right next to our machine room's building, and the meeting room had a window that literally looked across the alleyway at our building. At least that made it quick and easy to get to the machine room, because we could just walk across the bridge that connects the two buildings.

PS: In our environment, this is such a rare collection of factors that it's not worth trying to set up some sort of alerting for it, especially today in a world with pervasive smartphones (where people outside the meeting room can easily send some of us messages, even with the network down).

(Also, these days we don't normally have such big meetings any more and if we did, they'd be virtual meetings and we'd definitely notice bits of the network going down, one way or another.)

As covered recently, the normal way to check simple services from outside in a Prometheus environment is with Prometheus Blackbox, which is somewhat complicated to understand. One of its abstractions is a prober, a generic way of checking some service using HTTP, DNS queries, a TCP connection, and so on. The TCP prober supports conducting a query-response dialog once you connect, but currently (as of Blackbox 0.28.0) it doesn't directly expose metrics that tell you where your TCP probe with a query-response set failed (and why), and sometimes you'd like to know.

A somewhat typical query-response probe looks like this:

  smtp_starttls:
    prober: tcp
    tcp:
      query_response:
        - expect: "^220"
        - send: "EHLO something\r"
        - expect: "^250-STARTTLS"
        - expect: "^250 "
        - send: "STARTTLS\r"
        - expect: "^220"
        - starttls: true
        - expect: "^220"
        - send: "QUIT\r"

To understand what metrics we can look for on failure, we need to both understand how each important option in a step can fail, and what metrics they either set on failure or create when they succeed.

• starttls will fail if it can't successfully negotiate a TLS connection with the server, possibly including if the server's TLS certificate fails to verify. It sets no metrics on failure, but on success it will set various TLS related metrics such as the probe_ssl_* family and probe_tls_version_info.

• send will fail if there is an error sending the line, such as the TCP connection closing on you. It sets no metrics on either success or failure.

• expect reads lines from the TCP connection until either a line matches your regular expression, it hits EOF, or it hits a network error. If it hit a network error, including from the other end abruptly terminating the connection in a way that raises a local error, it sets no metrics. If it hit EOF, it sets the metric probe_failed_due_to_regex to 1; if it matched a line, it sets that metric to 0.

  One important case of 'network error' is if the check you're doing times out. This is internally implemented partly by putting a (Go) deadline on the TCP connection, which will cause an error if it runs too long. Typical Blackbox module timeouts aren't very long (how long depends on both configuration settings and how frequent your checks are; they have to be shorter than the check interval).

  If you have multiple 'expect' steps and you check fails at one of them, there's (currently) no way to find out which one it failed at unless you can determine this from other metrics, for example the presence or absence of TLS metrics.

• expect_bytes fails if it doesn't immediately read those bytes from the TCP connection. If it failed because of an error or because it read fewer bytes than required (including no bytes, ie an EOF), it sets no metrics. If it read enough bytes it sets the probe_failed_due_to_bytes metric to either 0 (if they matched) or 1 (if they didn't).

In many protocols, the consequences of how expect works means that if the server at the other end spits out some error response instead of the response you expect, your expect will skip over it and then wait endlessly. For instance, if the SMTP server you're probing gives you a SMTP 4xx temporary failure response in either its greeting banner or its reply to your EHLO, your 'expect' will sit there trying to read another line that might start with '220'. Eventually either your check will time out or the SMTP server will, and probably it will be your check (resulting in a 'network error' that leaves no traces in metrics). Generally this means you can only see a probe_failed_due_to_regex of 1 in a TCP probe based module if the other end cleanly closed the connection, so that you saw EOF. This tends to be pretty rare.

(We mostly see it for SSH probes against overloaded machines, where we connect but then the SSH daemon immediately closes the connection without sending the banner, giving us an EOF in our 'expect' for the banner.)

If the probe failed because of a DNS resolution failure, I believe that probe_ip_addr_hash will be 0 and I think probe_ip_protocol will also be 0.

If the check involves TLS, the presence of the TLS metrics in the result means that you got a connection and got as far as starting TLS. In the example above, this would mean that you got almost all of the way to the end.

I'm not sure if there's any good way to detect that the connection attempt failed. You might be able to reasonably guess that from an abnormally low probe_duration_seconds value. If you know the relevant timeout values, you can detect a probe that failed due to timeout by looking for a suitably high probe_duration_seconds value.

If you have some use of the special labels action, then the presence of a probe_expect_info metric means that the check got to that step. If you don't have any particular information that you want to capture from an expect line, you can use labels (once) to mark that you've succeeded at some expect step by using a constant value for your label.

(Hopefully all of this will improve at some point and Blackbox will provide, for example, a metric that tells you the step number that a query-response block failed on. See issue #1528, and also issue #1527 where I wish for a way to make an 'expect' fail immediately and definitely if it receives known error responses, such as a SMTP 4xx code.)

Prometheus Blackbox is somewhat complicated to understand. One of its fundamental abstractions is a 'prober', a generic way of probing some service (such as making HTTP requests or DNS requests). One prober is the 'tcp' prober, which makes a TCP connection and then potentially conducts a conversation with the service to verify its health. For example, here's a ClamAV daemon health check, which connects, sends a line with "PING", and expects to receive "PONG":

  clamd_pingpong:
    prober: tcp
    tcp:
      query_response:
        - send: "PING\n"
        - expect: "PONG"

The conversation with the service is detailed in the query_response configuration block (in YAML). For a long time I thought that this was what it looks like here, a series of entries with one directive per entry, such as 'send', 'expect', or 'starttls' (to switch to TLS after, for example, you send a 'STARTTLS' command to the SMTP or IMAP server).

However, much like an earlier case with Alertmanager, this is not actually what the YAML syntax is. In reality each step in the query_response YAML array can have multiple things. To quote the documentation:

 [ - [ [ expect: <string> ],
       [ expect_bytes: <string> ],
       [ labels:
         - [ name: <string>
             value: <string>
           ], ...
       ],
       [ send: <string> ],
       [ starttls: <boolean | default = false> ]
     ], ...
 ]

When there are multiple keys in a single step, Blackbox handles them in almost the order listed here: first expect, then labels if the expect matched, then expect_bytes, then send, then starttls. Normally you wouldn't have both expect and expect_bytes in the same step (and combining them is tricky). This order is not currently documented, so you have to read prober/query_response.go to determine it.

One reason to combine expect and send together in a single step is that then send can use regular expression match groups from the expect in its text. There's an example of this in the example blackbox.yml file:

  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        # cks: note use of ${1}, from PING
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"

The 'labels:' key is something added in v0.26.0, in #1284. As shown in the example blackbox.yml file, it can be used to do things like extract SSH banner information into labels on a metric:

  ssh_banner_extract:
    prober: tcp
    timeout: 5s
    tcp:
      query_response:
      - expect: "^SSH-2.0-([^ -]+)(?: (.*))?$"
        labels:
        - name: ssh_version
          value: "${1}"
        - name: ssh_comments
          value: "${2}"

This creates a metric that looks like this:

probe_expect_info {ssh_comments="Ubuntu-3ubuntu13.14", ssh_version="OpenSSH_9.6p1"} 1

At the moment there are some undocumented restrictions on the 'labels' key (or action or whatever you want to call it). First, it only works if you use it in a step that has an 'expect'. Even if all you want to do is set constant label values (for example to record that you made it to a certain point in your steps), you need to expect something; you can't use 'labels' in a step that otherwise only has, say, 'send'. Second, you can only have one labels in your entire query_response section; if you have more than one, you'll currently experience a Go panic when checking reaches the second.

This is unfortunate because Blackbox is currently lacking good ways to see how far your query_response steps got if the probe fails. Sometimes it's obvious where your probe failed, or irrelevant, but sometimes it's both relevant and not obvious. If you could use multiple labels, you could progressively set fixed labels and tell how far you got by what labels were visible in the scrape metrics.

(And of course you could also record various pieces of useful information that you don't get all at once.)

Sidebar: On (not) condensing expect and send together

My personal view is that I normally don't want to condense 'expect' and 'send' together into one step entry unless I have to, because most of the time it inverts the relationship between the two. In most protocols and protocol interactions, you send something and expect a response; you don't receive something and then send a response to it. In my opinion this is more naturally written in the style:

      query_response:
      - expect: "something"
      - send: "my request"
      - expect: "reply to my request"
      - send: "something else"
      - expect: "reply to something else"

Than as:

      query_response:
      - expect: "something"
        send: "my request"
      - expect: "reply to my request"
        send: "something else"
      - expect: "reply to something else"

What look like pairs (an expect/send in the same step) are not actually pairs; the 'expect' is for a previous 'send' and then 'send' pairs with the next 'expect' in the next step. So it's clearer to write them all as separate steps, which doesn't create any expectations of pairing.

The news of the day is that Microsoft had a significant outage inside their Microsoft 365 infrastructure. We noticed when we stopped being able to deliver email to the university's institutional email system, which was a bit mysterious in the usual way of today's Internet:

The joys of modern email: "Has Microsoft decided to put all of our email on hold or are they having a global M365 inbound SMTP email incident?"

(For about the last hour and a half, if it's an incident someone is having a bad day.)

We didn't find out immediately when this happened (and if our systems had been working right, we wouldn't have found out when I did, but that's another story). Initially I was going to write an entry about whether or not we should use our monitoring system to monitor external services that other people run, but it turns out that we do try to monitor whether we can do a SMTP conversation to the university's M365-hosted institutional email. There were several things that happened with this monitoring.

The first thing that happened is that the alerts related to it rotted. The university once had a fixed set of on-premise MX targets and we monitored our ability to talk to them and alerted on it. Then the university moved their MX targets to M365 and our old alerts stopped applying, so we commented them out and never added any new alerts for any new checking we were doing.

One of the reasons for that is that we were doing this monitoring through Prometheus Blackbox, and Blackbox is not ideal for monitoring Microsoft 365 MX targets. The way M365 does redundancy in their inbound mail servers for your domain is not by returning multiple DNS MX records, but by returning one MX record for a hostname that has multiple IP addresses (and the IP addresses may change). What a mailer will do is try all of the IP addresses until one responds. What Blackbox does is it picks one IP address and then it probes the IP address; if the address fails, there is no attempt to check the other IP addresses. Failing if one IP of many is not responding is okay for casual checks, but you don't necessarily want to alert on it.

(I believe that Blackbox picks the first IP address in the DNS A record, but this depends on how the Go standard library and possibly your local resolver behaves. If either sort the results, you get the first A record in the sorted result.)

The final issue is that we weren't necessarily checking enough of the SMTP conversation. For various reasons, we decided that all we could safely and confidently check was that the university's mail system accepted a testing SMTP MAIL FROM from our subdomain; we didn't check that it also accepted a SMTP RCPT TO. I believe that during part of this Microsoft 365 incident, the inbound M365 SMTP servers would accept our SMTP MAIL FROM but report an error at the RCPT TO (although I can't be sure). Certainly if we want to have a more realistic check of 'is email to M365 working', we should go as far as a SMTP RCPT TO.

(During parts of the incident, DNS lookups didn't succeed for the MX target. Without detailed examination I can't be sure of what happened in the other cases.)

Overall, Blackbox is probably the wrong tool to check an external mail target like M365 if we're serious about it and want to do a good job. At the moment it's not clear to me if we should go to the effort to do better, since it is an external service and there's nothing we can do about problems (although we can let people know, which has some value, but that's another entry).

PS: You can get quite elaborate in a mail deliverability test, but to some degree the more elaborate you get the more pieces of infrastructure you're testing, and you may want a narrow test for better diagnostics.

When querying Spamhaus DNS blocklists, either their public mirrors or through a DQS account, the DNS blocklists can potentially return error codes in 127.255.255.0/24 (also). Although Exim has a variety of DNS blocklist features, it doesn't yet let you match return codes based on CIDR netblocks. However, it does have a magic way of doing this.

The magic way is to stick '!&0.255.255.0' on the end of the DNS blocklist name. This is a negated DNS (blocklist) matching conditions, specifically a negated bitmask (a 'bitwise-and'). The whole thing looks like:

deny dnslists = zen.spamhaus.org!&0.255.255.0

What this literally means is to consider the lookup to have failed if the resulting IP address matches '*.255.255.*'. Because Exim already requires successful lookup results to be in 127.0.0.0/8, this implicitly constrains the entire result to not match 127.255.255.*, which is what we want.

As covered in Additional matching conditions for DNS lists, Exim can match DNS blocklist results by a specific IP or a bitmap, the latter of which is written as, eg, '&0.255.255.0'. When you match by bitmap, the IP address is anded with the bitmap and the result must be the same as the bitmap (meaning that all bits set in the bitmask are set in the IP address):

(ip & bitmask) == bitmask

(You can consider both the IP and the bitmask as 32-bit numbers, or you can consider each octet separately in both, whichever makes it easier.)

There's no way to say that the match succeeds if the result of and'ing the IP and the bitmask is non-zero (has any bits set). For small number of bits, you can sort of approximate that by using multiple bitmasks. For example, to succeed if either of the two lowest bits are set:

a.example&0.0.0.1,0.0.0.2

(The 'lowest bit' here is the lowest bit of the rightmost octet.)

If you negate a bitmask condition by writing it as '!&', the lookup is considered to have failed if the '&<bitmask>' match is successful, which is to say that the IP address anded with the bitmask is the same as the bitmask.

This is why '!&0.255.255.0' does what we want. '&0.255.255.0' successfully matches if the IP address is exactly *.255.255.*, because both middle octets have all their bits set in the mask so they have to have all their bits set in the IP address, and because the first and last octets in the mask are 0, their value in the IP address isn't looked at. Then we negate this, so the lookup is considered to have failed if the bitmask matched, which would mean that Spamhaus returned results in 127.255.255.0/24.

I'm writing all of that out in detail because here is what the current Exim documentation says about negated DNS bitmask conditions:

Negation can also be used with a bitwise-and restriction. The dnslists condition with only be true if a result is returned by the lookup which, anded with the restriction, is all zeroes.

This is not how Exim behaves. If it was how Exim behaves, Spamhaus DBL lookups would not work correctly with '!&0.255.255.0'. DBL lookups return results in 127.0.1.0/24; if you bitwise-and that with 0.255.255.0, you get '0.0.1.0', which is not all zeroes.

(It could be useful to have a version of '&' that succeeded if any of the bits in the result were non-zero, but that's not what Exim has today, as discussed above.)

For reasons outside the scope of this entry, we recently switched from Spamhaus's traditional public DNS (what is now called the 'public mirrors') to an account with their Data Query Service. The DQS data can still be queried via DNS, which presents a problem: DNS queries have no way to carry any sort of access key with them. Spamhaus has solved this problem by embedding your unique access key in the zone name you must use. Rather than querying, say, zen.spamhaus.org, you query '<key>.zen.dq.spamhaus.net'. Because your DQS key is tied to your account and your account has query limits, you don't want to spread your DQS key around for other people to pick up and use.

We use the Exim mailer (which is more of a mailer construction kit out of the box). Exim has a variety of convenient features for using DNS (block) lists. One of them is that when Exim finds an entry in a DNS blocklist in an ACL, it sets some (Exim) variables that you can use later in various contexts, such as creating log messages. To more or less quote from the Exim documentation on (string) expansion variables:

$dnslist_domain
$dnslist_matched
$dnslist_text
$dnslist_value

When a DNS (black) list lookup succeeds, these variables are set to contain the following data from the lookup: the list’s domain name, the key that was looked up, the contents of any associated TXT record, and the value from the main A record. [...]

To make life easier on yourself, it's conventional to use these variables (among others) in things like SMTP error messages and headers that you add to messages:

deny hosts = !+local_networks
     message = $sender_host_address is listed \
               at $dnslist_domain: $dnslist_text
     dnslists = rbl-plus.mail-abuse.example

warn dnslists = weird.example
     add_header = X-Us-DNSBL: listed in $dnslist_domain

However, if you're using Spamhaus DQS, using $dnslist_domain as these examples do is dangerous. The DNS list domain will be the full domain, and that full domain will include your DQS access key, which you will thus be exposing in message headers and SMTP error messages. You probably don't want to do that.

(Certainly it feels like a bad practice to leak a theoretically confidential value into the world, even if the odds are that no one is going to pick it up and abuse it.)

You have two options. The first option is to simply hard code some appropriate name for the list instead of using $dnslist_domain. However, this only works if you're using a single DNS list in each ACL condition, instead of something where you check multiple DNS blocklists at once (with 'dnslists = a.example : b.example : c.example'). It's also a bit annoying to have to repeat yourself.

(This is what I did to our Exim configuration when I realized the problem.)

The second option is that Exim has a comprehensive string expansion language, so determined people can manipulate $dnslist_domain to detect that it contains your DQS key and remove it. The brute force way would be to use ${sg} (from expansion items) to replace your key with nothing, something like (this is untested):

${sg{$dnslist_domain}{<DQS key>}{}}

You could probably wrap this up in an Exim macro, call it 'DNSLIST_NAME', and then write ACLs as, say:

deny hosts = !+local_networks
     message = $sender_host_address is listed \
               at DNSLIST_NAME
     dnslists = rbl-plus.mail-abuse.example

(Because we're using ${sg}, we won't change the name of a DNSBL domain that doesn't contain the DQS key.)

This isn't terrible and it does cope with a single Exim ACL condition that checks multiple DNS blocklists.

The Amanda backup system is what we use to handle our backups. One of Amanda's core concepts is a 'dump cycle', the amount of time between normally scheduled full backups for filesystems. If you have a dumpcycle of 7 days and Amanda does a full backup of a filesystem on Monday, its normal schedule for the next full backup is next Monday. However, Amanda can 'promote' a full backup ahead of schedule if it believes there's room for the full backup in a given backup run. Promoting full backups is a good idea in theory because it reduces how much data you need to restore a filesystem.

The amanda.conf configuration file has a per-dumptype option that affects this:

maxpromoteday int

Default: 10000. The maximum number of day[s] for a promotion, set it 0 if you don't want promotion, set it to 1 or 2 if your disks get overpromoted.

As written, I find this a little bit opaque (to be polite). What maxpromoteday controls is the maximum of how many days ahead of the normal schedule Amanda will promote a full backup. For example, if you have a 7-day dump cycle, a maxpromoteday of 2, and did a full dump of a filesystem on Monday, the earliest Amanda will possibly schedule a 'promoted' full backup is two days before next Monday, so the coming Saturday or Sunday. By extension, if you set maxpromoteday to '0', Amanda will only consider promoting a full backup of a filesystem zero days ahead of schedule, which is to say 'not at all'. Any value larger than your 'dumpcycle' setting has no effect, because Amanda is already doing full backups that often and so a larger value doesn't add any extra constraints on Amanda's scheduling of full backups.

You might wonder why you'd want to set 'maxpromoteday' down to limit full backup promotions, and naturally there is a story here.

Amanda is a very old backup system, and although it's not necessarily used with physical tapes and tape robots today (our 'tapes' are HDDs), many of its behaviors date back to that era. While the modern version of Amanda can split up a single large backup of a single (large) filesystem across multiple 'tapes', what it refuses to do is to split such a backup across multiple Amanda runs. If a filesystem backup can't be completely written out to tape in the current Amanda run, any partially written amount is ignored; the entire filesystem backup will be (re)written in the next run, using up the full space. If Amanda managed to write 90% of your large filesystem to your backup media today, that 90% is ignored because the last 10% couldn't be written out.

The consequence of this is that if you're backing up large filesystems with Amanda, you really don't want to run out of tape space during a backup run because this can waste hundreds of gigabytes of backup space (or more, if you have multi-terabyte filesystems). In environments like ours where the 'tapes' are artificial and we have a lot of them available to Amanda (our tapes a partitions on HDDs and we have a dozen HDDs or more mounted on each backup server at any given time), the best way to avoid running out of tape space during a single Amanda run is to tell Amanda that it can use a lot of tapes, way more tapes than it should ever actually need.

(Even in theory, Amanda can't perfectly estimate how much space a given full or incremental backup will actually use and so it can run over the tape capacity you actually want it to use. In practice, in many environments you may have to tell Amanda to use 'server side estimates', where it guesses based on past backup behavior, instead of the much more time-consuming 'client side estimates', where it basically does an estimation pass over each filesystem to be backed up.)

However, if you tell Amanda it can use a lot of tapes in a standard Amanda setup, Amanda will see a vast expanse of available tape capacity and enthusiastically reach the perfectly rational conclusion that it should make use of that capacity by aggressively promoting full backups of filesystems (both small and large ones). This is very much not what you (we) actually want. We're letting Amanda use tons of 'tapes' to insure that it never wastes tape space, not so that it can do extra full backups; if Amanda doesn't need to use the tape space we don't want it to touch that tape space.

The easiest way for us to achieve this is to set 'maxpromoteday 0' in our Amanda configuration, at least for Amanda servers that back up very large filesystems (where the wasted tape space of an incompletely written backup could be substantial). Unfortunately I think you'll generally want to set this for all dump types in a particular Amanda server, because over-promotion of even small(er) filesystems could eat up a bunch of tape space that you want to remain unused.

(Amanda talks about 'dumps' because it started out on Unix systems where for a long time the filesystem backup program was called 'dump'. These days your Amanda filesystem backups are probably done with GNU Tar, although I think people still talk about things like 'database dumps' for backups.)

I mentioned on the Fediverse a while back that we have air conditioners on our internal network. Well, technically what we have on the internal network is separate (and optional) controller devices that connect to the physical AC units themselves, but as they say, this is close enough. Of course there's a story here:

Why do we have networked AC controllers? Well, they control portable AC units that are in our machine rooms for emergency use, and having their controllers on our internal network means we can possibly turn them on from home if the main room AC stops working out of hours, on weekends, etc.

(It would still be a bad time, just maybe a little less bad.)

Our machine rooms are old (cf) and so are their normal AC units. Over the years we've had enough problems with these AC units that we've steadily accumulated emergency measures. A couple of years ago, these emergency measures reached the stage of pre-deploying wheeled portable AC units with their exhaust hoses connected up to places where they could vent hot air that would take it outside of the machine room.

Like most portable ACs, these units are normally controlled in person from their front panels (well, top panels). However, these are somewhat industrial AC units and you could get optional network-accessible controllers for them; after thinking about it, we did and then hooked the controllers (and thus the ACs) up to our internal management network. As I mentioned, the use case for networked control of these AC units is to turn them on from home during emergencies. They don't have anywhere near enough cooling power to cover all of the systems we normally have running in our machine rooms, but we might be able to keep a few critical systems up rather than being completely down.

(We haven't had serious AC issues since we put these portable AC units into place, so we aren't sure how well they'd perform and how much we'd be able to keep up.)

These network controllers can get status information (including temperatures) from the ACs and have some degree of support for SNMP, so we could probably pull information from them for metrics purposes if we wanted to. Right now we haven't looked into this, partly because we have our own temperature monitoring and partly because I'm not sure I trust the SNMP server implementation to be free of bugs, memory leaks, and other things that might cause problems for the overall network controller.

(Like most little things, these network controllers are probably running some terrifyingly ancient Linux kernel and software stack. A quick look at the HTTP server headers says that it's running a clearly old version of nginx on Ubuntu, although it's slightly more recent than I expected.)

I recently wrote about the complexities of getting programs to report the TLS certificates they use, where I theorized about writing a script to scrape this information out of places like the Apache configuration files, and then today I realized the obvious specific approach for our environment:

Obvious realization is obvious: since we universally use Let's Encrypt with certbot and follow standard naming, I can just look in /etc/letsencrypt/live to find all live TLS certificates and (a) host name for them, for cross-checking against our monitoring.

Our TLS certificates usually have multiple names associated with them, only one of which is the directory name in /etc/letsencrypt/live. However, we usually monitor the TLS certificate under what we think of as the primary name, and in any case we can make this our standard Prometheus operating procedure.

In our Prometheus environment we create a standard label for the 'host' being monitored, including for metrics obtained through Blackbox. Given that Blackbox exposes TLS certificate metrics, we can use things like direct curl queries to Prometheus to verify that we have TLS certificate monitoring for everything in /etc/letsencrypt/live. The obvious thing to check is that we have a probe_ssl_earliest_cert_expiry metric with the relevant 'host' value for each Let's Encrypt primary name.

If we want to, we can go further by looking at probe_ssl_last_chain_info. This Blackbox metric directly exposes labels for the TLS 'subject' and 'subjectalternative', so we can in theory search them for either the primary name that Let's Encrypt will be using or for what we consider an important name to be covered. It appears that this wouldn't be needed to cover any additional TLS certificates for us, as we're already checking everything under its primary name.

(Well, we are after I found one omission in a manual check today.)

With the right tools (also), I don't need to make this a pre-written shell script that runs on each machine; instead, I can do this centrally by hand every so often. On the one hand this isn't as good as automating it, but on the other hand every bit of locally built automation is another bit of automation we have to maintain ourselves. We mostly haven't had a problem with tracking TLS certificates, and we have other things to notice failures.

(I should probably write a personal script to do this, just to capture the knowledge.)

One of the practical reasons that TLS certificates have dangerous expiry times is that in most environments, it's up to you to remember to add monitoring for each TLS certificate that you use, either as part of general purpose monitoring of the service or specific monitoring for certificate expiry. It would be nice if programs that used TLS certificates inherently monitored their expiry, but that's a fairly big change (for example, you have to decide how to send alerts about that information). A nominally easier change would be for programs routinely to be able to report what TLS certificates they're using, either as part of normal metrics and log messages or through some additional command line switch.

(If your program uses TLS certificates and it has some sort of built in way of reporting metrics, it would be very helpful to system administrators if it reported basic TLS certificate metrics like the 'notAfter' time.)

In a lot of programs, this would be relatively straightforward (in theory). A common pattern is for programs to read in all of the TLS certificates they're going to use on startup, before they drop privileges, which means that these programs reliably know what all of those certificates are (and some programs will abort if some TLS certificates can't be read). They could then report the TLS certificate file paths on startup, either as part of their regular startup or in a special 'just report configuration information' mode. In many cases, one could write your own script that scanned the program's configuration files and did a reasonably good job of finding all of the TLS certificate filenames (and you could then make it report the names those TLS certificates were for, and cross-check this against your existing monitoring).

(I should probably write such a script for our Apache environment, because adding TLS based virtual hosts and then forgetting to monitor them is something we could definitely do.)

However, not all programs are straightforward this way. There are some programs that can at least potentially generate the TLS certificate file name on the fly at runtime (for example, Exim's settings for TLS certificate file names are 'expanded strings' that might depend on connection parameters). And even usually straightforward programs like Apache can have conditional use of TLS certificates, although this probably will only leave you doing some extra monitoring of unused TLS certificates (let's assume you're not using SSLCertificateFile token identifiers). These programs would probably need to log TLS certificate filenames on their first use, assuming that they cache loaded TLS certificates rather than re-read them from scratch every time they're necessary.

There's also no generally obvious and good way to expose this information, which means that logging it or printing it out is only the first step and not necessarily deeply useful by itself. If programs put it into logs, people have to pull it out of logs; if programs report it from the command line, people need to write additional tooling. If a program has built in metrics that it exposes in some way, exposing metrics for any TLS certificates it uses is great, but most programs don't have their own metrics and statistics systems.

(Still, it would be nice if programs supported this first step.)

Over on the Fediverse, I shared an old story that's partly about (system) documentation, and it sparked a thought, which is that we (I) should write up a brief high level overview of our overall environment. This should probably be one level higher than an end of service writeup, which are focused on a specific service (if we write them at all). The reason to do this is because our regular documentation assumes a lot of context and part of that context is what our overall environment is. We know what the environment is because it's the water we work in, but a new person arriving here could very easily be lost.

What I'm thinking of is something as simple as saying (in a bit more words) that we store our data on a bunch of NFS fileservers and people get access to their home directories and so on by logging in to various multi-user Unix servers that all run Ubuntu Linux, or using various standard services like email (IMAP and webmail), Samba/CIFS file access, and printing. Our logins and passwords are distributed around as files from a central password server and a central NFS-mounted filesystem. There's some more that I would write here (including information about our networks) and I'd probably put in a bit more details about some names of the various servers and filesystems, but not too much more.

(At least not in the front matter. Obviously such an overview could get increasingly detailed in later sections.)

A bunch of this information is already on our support website in some form, but I feel the support website is both too detailed and not complete enough. It's too detailed because it's there to show people how to do things, and it's not complete because we deliberately omit some things that we consider implementation details (such as our NFS fileservers). A new person here should certainly read all the way through the support site sooner or later, but that's a lot of information to absorb. A high level overview is a quick start guide that's there to orient people and leave them with fewer moments of 'wait, you have a what?' or 'what is this even talking about?' as they're exposed to our usual documentation.

One reason to keep the high level overview at a high level is that the less specific it is, the less it's going to fall out of date as things change. Updating such a high level overview is always going to be low on the priority list, since it's almost never used, so the less updating it needs the better. Also, I can also write somewhat more detailed high level overviews of specific aspects or sub-parts of our environment, if I find myself feeling that the genuine high level version doesn't say enough. Another reason to keep it high level is to keep it short, because asking a new person to read a couple of pages (at most) as high level orientation is a lot better than throwing them into the deep end with dozens of pages and thousands of words.

(I'm writing this down partly to motivate myself to do this when we go back to work in the new year, even though it feels both trivial and obvious. I have to remind myself that the obvious things about our environment to me are that way partly because I'm soaking in it.)

I recently wrote about how getting out of being people's secondary authoritative DNS server is hard. In the process of that I said that there was a background Internet radiation of external machines throwing random DNS queries at us. Now that we've reduced the number of DNS zones that were improperly still pointing at us for historical reasons, I think I can finally see enough from our public authoritative DNS server's traffic to say something about that.

The rejected DNS queries we're seeing so far are a mixture of three types of queries. The first sort of query is for one of those DNS zones that used to be pointing to us but haven't been for long enough that people's DNS caches should have timed out by now. My best guess is that some systems simply hold on to DNS nameserver information for well over any listed TTLs for it. The amount of these queries has been going down for some time so it seems that eventually people do refresh their DNS information and stop poking us.

The second sort of query is for more or less random DNS names that have definitely never pointed at us, not infrequently in well known domains such as 'google.com' or 'googleapis.com', or a well known name like 'chrome.cloudflare-dns.com'. The source IPs for these queries are all over and they're generally low volume. Some IPs may be probing to see if we have any sort of open recursive resolver behavior, but others seem much more random, enough so that I wonder if the remote machines are experiencing some sort of corruption in the DNS server IP that they want to query (or perhaps their DNS lookup software or resolving DNS software is copying the NS record from one entry over to another).

(Sometimes people even make queries for things in the RFC 1918 portion of in-addr.arpa.)

The third and largest source of bad traffic is queries for what look like internal domains within at least one top level domain (and I'm going to name it, it's koenigmetall.com). On spot checking so far, all of the queries come from IP addresses that seem to be located in Romania. What I suspect here is a version of people using our 128.100/16 as internal IP address space. Our public authoritative DNS server is at the IP address 128.100.1.1, which is a very attractive IP to put something important on if you're using 128.100/16 internally. So I suspect that if someone were to inspect the internal DNS of the company in question, they'd find errant DNS NS and A records that said an internal DNS server for these internal zones was found at 128.100.1.1. Then queries theoretically to that internal DNS server are leaking onto the public Internet and reaching us, likely in a process similar to how people keep sending dynamic DNS updates to us (that entry is from 2021 but it's all still going on).

Without deliberately planning it, we've wound up running an assortment of DNS server software on an assortment of DNS servers. A lot of this involves history, so I might as well tell the story of that history in the process. This starts with our three sets of DNS servers: our internal DNS master (with a duplicate) that holds both the internal and external views of our zones, our resolving DNS servers (which use our internal zones), and our public authoritative DNS server (carrying our external zones, along with various relics of the past). These days we also have an additional resolving DNS server that resolves from outside our networks and so gives the people who can use it an external view of our zones.

In the beginning we ran Bind on everything, as was the custom in those days (and I suspect we started out without a separation between the three types of DNS servers, but that predates my time here), and I believe all of the DNS servers were Solaris. Eventually we moved the resolving DNS servers and the public authoritative DNS server to OpenBSD (and the internal DNS master to Ubuntu), still using Bind. Then OpenBSD switched which nameservers they liked from Bind to Unbound and NSD, so we went along with that. Our authoritative DNS server had a relatively easy NSD configuration, but our resolving DNS servers presented some challenges and we wound up with a complex Unbound plus NSD setup. Recently we switched our internal resolvers to using Bind on Ubuntu, and then we switched our public authoritative DNS server from OpenBSD to Ubuntu but kept it still with NSD, since we already had a working NSD configuration for it.

This has wound up with us running the following setups:

• Our internal DNS masters run Bind in a somewhat complex split horizon configuration.

• Our internal DNS resolvers run Bind in a simpler configuration where they act as internal authoritative secondary DNS servers for our own zones and as general resolvers.

• Our public authoritative DNS server (and its hot spare) run NSD as an authoritative secondary, doing zone transfers from our internal DNS masters.

• We have an external DNS resolver machine that runs Unbound in an extremely simple configuration. We opted to build this machine with Unbound because we didn't need it to act as anything other than a pure resolver, and Unbound is simple to set up for that.

At one level, this is splitting our knowledge and resources among three DNS servers rather than focusing on one. At another level, two out of the three DNS servers are being used in quite simple setups (and we already had the NSD setup written from prior use). Our only complex configurations are all Bind based, and we've explicitly picked Bind for complex setups because we feel we understand it fairly well from long experience with it.

(Specifically, I can configure a simple Unbound resolver faster and easier than I can do the same with Bind. I'm sure there's a simple resolver-only Bind configuration, it's just that I've never built one and I have built several simple and not so simple Unbound setups.)

Many, many years ago, my department operated one of the university's secondary authoritative DNS servers, which was used by most everyone with a university subdomain and as a result was listed as one of their DNS NS records. This DNs server was also the authoritative DNS server for our own domains, because this was in the era where servers were expensive and it made perfect sense to do this. At the time, departments who wanted a subdomain pretty much needed to have a Unix system administrator and probably run their own primary DNS server and so on. Over time, the university's DNS infrastructure shifted drastically, with central IT offering more and more support, and more than half a decade ago our authoritative DNS server stopped being a university secondary, after a lot of notice to everyone.

Experienced system administrators can guess what happened next. Or rather, what didn't happen next. References to our DNS server lingered in various places for years, both in the university's root zones as DNS glue records and in people's own DNS zone files as theoretically authoritative records. As late as the middle of last year, when I started grinding away on this, I believe that roughly half of our authoritative DNS server's traffic was for old zones we didn't serve and was getting DNS 'Refused' responses. The situation is much better today, after several rounds of finding other people's zones that were still pointing to us, but it's still not quite over and it took a bunch of tedious work to get this far.

(Why I care about this is that it's hard to see if your authoritative DNS server is correctly answering everything it should if things like tcpdumps of DNS traffic are absolutely flooded with bad traffic that your DNS server is (correctly) rejecting.)

In theory, what we should have done when we stopped being a university secondary authoritative DNS server was to switch the authoritative DNS server for our own domains to another name and another IP address; this would have completely cut off everyone else when we turned the old server off and removed its name from our DNS. In practice the transition was not clearcut, because for a while we kept on being a secondary for some other university zones that have long-standing associations with the department. Also, I think we were optimistic about how responsive people would be (and how many of them we could reach).

(Also, there's a great deal of history tied up in the specific name and IP address of our current authoritative DNS server. It's been there for a very long time.)

PS: Even when no one is incorrectly pointing to us, there's clearly a background Internet radiation of external machines throwing random DNS queries at us. But that's another entry.

Suppose that you have an internal master DNS server and a backup for that master server. The two servers are theoretically fed from the same data and so should have the same DNS zone contents, and especially they should have the same DNS zone SOAs for all zones in both of their internal and external views. They both run Bind and you use the Bind exporter, which provides the SOA values for every zone Bind is configured to be a primary or a secondary for. So you can write an alert with an expression like this:

bind_zone_serial{host="backup"}
  != on (view,zone_name)
    bind_zone_serial{host="primary"}

This is a perfectly good alert (well, alert rule), but it has lost all of the additional labels you might want in your alert. Especially, it has lost both host names. You could hard-code the host name in your message about the alert, but it would be nice to do better and propagate your standard labels into the alert. To do this you want to use one of group_left() and group_right(), but which one you want depends on where you want the labels to come from.

(Normally you have to chose between the two depending on which side has multiple matches, but in this case we have a one to one matching.)

For labels that are duplicated between both sides, the group_*() operators pick which side's labels you get, but backwards from their names. If you use group_right(), the duplicate label values come from the left; if you use group_left(), the duplicate label values come from the right. Here, we might change the backup host's name but we're probably not going to change the primary host's name, so we likely want to preserve the 'host' label from the left side and thus we use group_right():

bind_zone_serial{host="backup"}
  != on (view,zone_name)
    group_right (job,host,instance)
      bind_zone_serial{host="primary"}

One reason this little peculiarity is on my mind at the moment is that Cloudflare's excellent pint Prometheus rule linter recently picked up a new 'redundant label' lint rule that complains about this for custom labels such as 'host':

Query is trying to join the 'host' label that is already present on the other side of the query.

(It doesn't complain about job or instance, presumably because it understands why you might do this for those labels. As the pint message will tell you, to silence this you need to disable 'promql/impossible' for this rule.)

When I first saw pint's warning I didn't think about it and removed the 'host' label from the group_right(), but fortunately I actually tested what the result would be and saw that I was now getting the wrong host name.

(This is different from pulling in labels from other metrics, where the labels aren't duplicated.)

PS: I clearly knew this at some point, when I wrote the original alert rule, but then I forgot it by the time I was looking at pint's warning message. PromQL is the kind of complex thing where the details can fall out of my mind if I don't use it often enough, which I don't these days since our alert rules are relatively stable.

One of the reactions I saw to our move from OpenBSD to FreeBSD for firewalls was to wonder why we weren't moving all the way to nftables based Linux firewalls. It's true that this would reduce the number of different Unixes we have to operate and probably get us more or less state of the art 10G network performance. However, I have some negative views on the choice of PF versus nftables, both in our specific situation and in general.

(I've written about this before but it was in the implicit context of Linux iptables.)

In our specific situation:

• We have a lot of existing, relatively complex PF firewall rules; for example, our perimeter firewall has over 400 non-comment lines of rules, definitions, and so on. Translating these from OpenBSD PF to FreeBSD PF is easy, if it's necessary at all. Translating everything to nftables is a lot more work, and as far as I know there's no translation tool, especially not one that we could really trust. We'd probably have to basically rebuild each firewall from the ground up, which is both a lot of work and a high-stakes thing. We'd have to be extremely convinced that we had to do this in order to undertake it.

• We have a lot of well developed tooling around operating, monitoring, and gathering metrics from PF-based firewalls, most of it locally created. Much or all of this tooling ports straight over from OpenBSD to FreeBSD, while we have no equivalent tooling for nftables and would have to develop (or find) equivalents.

• We already know PF and almost all of that knowledge transfers over from OpenBSD PF to FreeBSD PF (and more will transfer with FreeBSD 15, which has some PF and PF syntax updates from modern OpenBSD).

In general (much of which also applies to our specific situation):

• There are a number of important PF features that nftables at best has in incomplete, awkward versions. For example, nftables' version of pflog is awkward and half-baked compared to the real thing (also). While you may be able to put together some nftables based rough equivalent of BSD pfsync, casual reading suggests that it's a lot more involved and complex (and maybe less integrated with nftables).

• The BSD PF firewall system is straightforward and easy to understand and predict. The Linux firewall system is much more complex and harder to understand, and this complexity bleeds through into nftables configuration, where you need to know chains and tables and so on. Much of this Linux complexity is not documented in ways that are particularly accessible.

• Nftables documentation is opaque compared to the BSD pf.conf manual page (also). Partly this is because there is no 'nftables.conf' manual page; instead, your entry point is the nft manual page, which is both a command line tool and the documentation of the format of nftables rules. I find that these are two tastes that don't go well together.

  (This is somewhat forced by the nftables decision to retain compatibility with adding and removing rules on the fly. PF doesn't give you a choice, you load your entire ruleset from a file.)

• nftables is already the third firewall rule format and system that the Linux kernel has had over the time that I've been writing Linux firewall rules (ipchains, iptables, nftables). I have no confidence that there won't be a fourth before too long. PF has been quite stable by comparison.

What I mostly care about is what I have to write and read to get the IP filtering and firewall setup that we want (and then understand it later), not how it gets compiled down and represented in the kernel (this has come up before). Assuming that the nftables backend is capable enough and the result performs sufficiently well, I'd be reasonably happy with a PF like syntax (and semantics) on top of kernel nftables (although we'd still have things like the pflog and pfsync issues).

Can I get things done in nftables? Certainly, nftables is relatively inoffensive. Do I want to write nftables rules? No, not really, no more than I want to write iptables rules. I do write nftables and iptables rules when I need to do firewall and IP filtering things on a Linux machine, but for a dedicated machine for this purpose I'd rather use a PF-based environment (which is now FreeBSD).

As far as I can tell, the state of Linux IP filtering documentation is partly a result of the fact that Linux doesn't have a unified IP filtering system and environment the way that OpenBSD does and FreeBSD mostly does (or at least successfully appears to so far). When the IP filtering system is multiple more or less separate pieces and subsystems, you naturally tend to get documentation that looks at each piece in isolation and assumes you already know all of the rest.

(Let's also acknowledge that writing good documentation for a complex system is hard, and the Linux IP filtering system has evolved to be very complex.)

PS: There's no real comparison between PF and the older iptables system; PF is clearly far more high level than you can reasonably do in iptables, which by comparison is basically an IP filtering assembly language. I'm willing to tentatively assume that nftables can be used in a higher level way than iptables can (I haven't used it for enough to have a well informed view either way); if it can't, then there's again no real comparison between PF and nftables.

A bit over a year ago I wrote about why we'd become interested in FreeBSD; to summarize, FreeBSD appeared promising as a better, easier to manage host operating system for PF-based things. Since then we've done enough with FreeBSD to have decided that we actively prefer it to OpenBSD. It's been relatively straightforward to convert our firewall OpenBSD PF rulesets to FreeBSD PF and the resulting firewalls have clearly better performance on our 10G network than our older OpenBSD ones did (with less tuning).

(It's possible that the very latest OpenBSD has significantly improved bridging and routing firewall performance so that it no longer requires the fastest single-core CPU performance you can get to go decently. But pragmatically it's too late; FreeBSD had that performance earlier and we now have more confidence in FreeBSD's performance in the firewall role than OpenBSD's.)

There are some nice things about FreeBSD, like root on ZFS, and broadly I feel that it's more friendly than OpenBSD. But those are secondary to its firewall network performance (and PF compatibility); if its network performance was no better than OpenBSD (or worse), we wouldn't be interested. Since it is better, it's now displacing OpenBSD for our firewalls and our latest VPN servers. We've stopped building new OpenBSD machines, so as firewalls come up for replacement they get rebuilt as FreeBSD machines.

(We have a couple of non-firewall OpenBSD machines that will likely turn into Ubuntu machines when we replace them, although we can't be sure until it actually happens.)

Would we consider going back to OpenBSD? Maybe, but probably not. Now that we've migrated a significant number of firewalls, moving the remaining ones to FreeBSD is the easiest approach, even if new OpenBSD firewalls would equal their performance. And the FreeBSD 10G firewall performance we're getting is sufficiently good that it leaves OpenBSD relatively little ground to exceed it.

(There are some things about FreeBSD that we're not entirely enthused about. We're going to be doing more firewall upgrades than we used to with OpenBSD, for one.)

PS: As before, I don't think there's anything wrong with OpenBSD if it meets your needs. We used it happily for years until we started being less happy with its performance on 10G Ethernet. A lot of people don't have that issue.

Over on the Fediverse, I mentioned a grump I have about containers:

As a sysadmin, containers irritate me because they amount to abandoning the idea of well done, well organized, well understood, etc installation of software. Can't make your software install in a sensible way that people can control and limit? Throw it into a container, who cares what it sprays where across the filesystem and how much it wants to be the exclusive owner and controller of everything in sight.

(This is a somewhat irrational grump.)

To be specific, it's by and large abandoning the idea of well done installs of software on shared servers. If you're only installing software inside a container, your software can spray itself all over the (container) filesystem, put itself in hard-coded paths wherever it feels like, and so on, even if you have completely automated instructions for how to get it to do that inside a container image that's being built. Some software doesn't do this and is well mannered when installed outside a container, but some software does and you'll find notes to the effect that the only supported way of installing it is 'here is this container image', or 'here is the automated instructions for building a container image'.

To be fair to containers, some of this is due to missing Unix APIs (or APIs that theoretically exist but aren't standardized). Do you want multiple Unix logins for your software so that it can isolate different pieces of itself? There's no automated way to do that. Do you run on specific ports? There's generally no machine-readable way to advertise that, and people may want you to build in mechanisms to vary those ports and then specify the new ports to other pieces of your software (that would all be bundled into a container image). And so on. A container allows you to put yourself in an isolated space of Unix UIDs, network ports, and so on, one where you won't conflict with anyone else and won't have to try to get the people who want to use your software to create and manage the various details (because you've supplied either a pre-built image or reliable image building instructions).

But I don't have to be happy that software doesn't necessarily even try, that we seem to be increasingly abandoning much of the idea of running services in shared environments. Shared environments are convenient. A shared Unix environment gives you a lot of power and avoids a lot of complexity that containers create. Fortunately there's still plenty of software that is willing to be installed on shared systems.

(Then there is the related grump that the modern Linux software distribution model seems to be moving toward container-like things, which has a whole collection of issues associated with it.)

For various reasons, I'm working to switch from wget to curl, and generally this has been going okay. However, I've now run into one situation where I don't know how to make curl do what I want. It is, of course, a project that doesn't bother to do easily-fetched downloads, but in a very specific way. In fact it's Django (again).

The Django URLs for downloads look like this:

https://www.djangoproject.com/download/5.2.8/tarball/

The way the websites of many projects turn these into actual files is to provide a filename in the HTTP Content-Disposition header in the reply. In curl, these websites can be handled with the -J (--remote-header-name) option, which uses the filename from the Content-Disposition if there is one.

Unfortunately, Django's current website does not operate this way. Instead, the URL above is a HTTP redirection to the actual .tar.gz file (on media.djangoproject.com). The .tar.gz file is then served without a Content-Disposition header as an application/octet-stream. Wget will handle this with --trust-server-names, but as far as I can tell from searching through the curl manpage, there is no option that will do this in curl.

(In optimistic hope I even tried --location-trusted, but no luck.)

If curl is directed straight to the final URL, 'curl -O' alone is enough to get the right file name. However, if curl goes through a redirection, there seems to be no option that will cause it to re-evaluate the 'remote name' based on the new URL; the initial URL and the name derived from it sticks, and you get a file unhelpfully called 'tarball' (in this case). If you try to be clever by running the initial curl without -O but capturing any potential redirection with "-w '%{redirect_url}\n'" so you can manually follow it in a second curl command, this works (for one level of redirections) but leaves you with a zero-length file called 'tarball' from the first curl.

It's possible that this means curl is the wrong tool for the kind of file downloads I want to do from websites like this, and I should get something else entirely. However, that something else should at least be a completely self contained binary so that I can easily drag it around to all of the assorted systems where I need to do this.

(I could always try to write my own in Go, or even take this as an opportunity to learn Rust, but that way lies madness and a lot of exciting discoveries about HTTP downloads in the wild. The more likely answer is that I hold my nose and keep using wget for this specific case.)

PS: I think it's possible to write a complex script using curl that more or less works here, but one of the costs is that you have to make first a HEAD and then a GET request to the final target, and that irritates me.

I have an unusual X desktop environment that has evolved over a long period, and as part of that I have an equally unusual and slowly evolved set of ways to handle URLs. By 'handle URLs', what I mean is going from an URL somewhere (email, text in a terminal, etc) to having the URL open in one of my several browser environments. Tied into this is handling non-URL things that I also want to open in a browser, for example searching for various sorts of things in various web places.

The simplest place to start is at the end. I have several browser environments and to go along with them I have a script for each that opens URLs provided as command line arguments in a new window of that browser. If there's no command line arguments, the scripts open a default page (usually a blank page, but for my main browser it's a special start page of links). For most browsers this works by running 'firefox <whatever>' and so will start the browser if it's not already running, but for my main browser I use a lightweight program that uses Firefox's X-based remote control protocol. which means I have to start the browser outside of it.

Layered on top of these browser specific scripts is a general script to open URLs that I call 'openurl'. The purpose of openurl is to pick a browser environment based on the particular site I'm going to. For example, if I'm opening the URL of a site where I know I need JavaScript, the script opens the URL in my special 'just make it work' JavaScript enabled Firefox. Most urls open in my normal, locked down Firefox. I configure programs like Thunderbird to open URLs through this openurl script, sometimes directly and sometimes indirectly.

(I haven't tried to hook openurl into the complex mechanisms that xdg-open uses to decide how to open URLs. Probably I should but the whole xdg-open thing irritates me.)

Layered on top of openurl and the specific browser scripts is a collection of scripts that read the X selection and do a collection of URL-related things with it. One script reads the X selection, looks for it being a URL, and either feeds the URL to openurl or just runs openurl to open my start page. Other scripts feed the URL to alternate browser environments or do an Internet search for the selection. Then I have a fvwm menu with all of these scripts in it and one of my fvwm mouse button bindings brings up this menu. This lets me select a URL in a terminal window, bring up the menu, and open it in either the default browser choice or a specific browser choice.

(I also have a menu entry for 'open the selection in my main browser' in one of my main fvwm menus, the one attached to the middle mouse button, which makes it basically reflexive to open a new browser window or open some URL in my normal browser.)

The other way I handle URLs is through dmenu. One of the things my dmenu environment does is recognize URLs and open them in my default browser environment. I also have short dmenu commands to open URLs in my other browser environments, or open URLs based on the parameters I pass the command (such as a 'pd' script that opens Python documentation for a standard library module). Dmenu itself can paste in the current X selection with a keystroke, which makes it convenient to move URLs around. Dmenu is also how I typically open a URL if I'm typing it in instead of copying it from the X selection, rather than opening a new browser window, focusing the URL bar, and entering the URL there.

(I have dmenu set up to also recognize 'about:*' as URLs and have various Firefox about: things pre-configured as hidden completions in dmenu, along with some commonly used website URLs.)

As mentioned, dmenu specifically opens plain URLs in my default browser environment rather than going through openurl. I may change this someday but in practice there aren't enough special sites that it's an issue. Also, I've made dedicated little dmenu-specific scripts that open up the various sites I care about in the appropriate browser, so I can type 'mastodon' in dmenu to open up my Fediverse account in the JavaScript-enabled Firefox instance.

Suppose, not hypothetically, that you have a very small DNS server for a captive network situation, where the DNS server exists only to give clients answers for a small set of hosts. One of the ways you can implement this is with an authoritative DNS servers, such as NSD, that simply has an extremely minimal set of DNS data. If you're using NSD for this, you might be curious how minimal you can be and how much you need to mimic ordinary DNS structure.

Here, by 'mimic ordinary DNS structure', I mean inserting various levels of NS records so there is a more or less conventional path of NS delegations from the DNS root ('.') down to your name. If you're providing DNS clients with 'dog.example.org', you might conventionally have a NS record for '.', a NS record for 'org.', and a NS record for 'example.org.', mimicking what you'd see in global DNS. Of course all of your NS records are going to point to your little DNS server, but they're present if anything looks.

Perhaps unsurprisingly, NSD doesn't require this and DNS clients normally don't either. If you say:

zone:
  name: example.org
  zonefile: example-stub

and don't have any other DNS data, NSD won't object and it will answer queries for 'dog.example.org' with your minimal stub data. This works for any zone, including completely made up ones:

zone:
  name: beyond.internal
  zonefile: beyond-stub

The actual NSD stub zone files can be quite minimal. An older OpenBSD NSD appears to be happy with zone files that have only a $ORIGIN, a $TTL, a '@ IN SOA' record, and what records you care about in the zone.

Once I thought about it, I realized I should have expected this. An authoritative DNS server normally only holds data for a small subset of zones and it has to be willing to answer queries about the data it holds. Some authoritative DNS servers (such as Bind) can also be used as resolving name servers so they'd sort of like to have information about at least the root nameservers, but NSD is a pure authoritative server so there's no reason for it to care.

As for clients, they don't normally do DNS resolution starting from the root downward. Instead, they expect to operate by sending the entire query to whatever their configured DNS resolver is, which is going to be your little NSD setup. In a number of configurations, clients either can't talk directly to outside DNS or shouldn't try to do DNS resolution that way because it won't work; they need to send everything to their configured DNS resolver so it can do, for example, "split horizon" DNS.

(Yes, the modern vogue for DNS over HTTPS puts a monkey wrench into split horizon DNS setups. That's DoH's problem, not ours.)

Since this works for a .net zone, you can use it to try to disable DNS over HTTPS resolvers in your stub DNS environment by providing a .net zone with 'use-application-dns CNAME .' or the like, to trigger at least Firefox's canary domain detection.

(I'm not going to address whether you should have such a minimal stub DNS environment or instead count on your firewall to block traffic and have a normal DNS environment, possibly with split horizon or response policy zones to introduce your special names.)

In a comment on my entry on how we reboot our machines right after updating their kernels, Jukka asked a good question:

While I do not know how many machines there are in your fleet, I wonder whether you do incremental rolling, using a small snapshot for verification before rolling out to the whole fleet?

We do this to some extent but we can't really do it very much. The core problem is that the state of almost all of our machines is directly visible and exposed to people. This is because we mostly operate an old fashioned Unix login server environment, where people specifically use particular servers (either directly by logging in to them or implicitly because their home directory is on a particular NFS fileserver). About the only genuinely generic machines we have are the nodes in our SLURM cluster, where we can take specific unused nodes out of service temporarily without anyone noticing.

(Some of these login servers in use all of the time; others we might find idle if we're extremely lucky. But it's hard to predict when someone will show up to try to use a currently empty server.)

This means that progressively rolling out a kernel update (and rebooting things) to our important, visible core servers requires multiple people-visible reboots of machines, instead of one big downtime when everything is rebooted. Generally we feel that repeated disruptions are much more annoying and disruptive overall to people; it's better to get the pain of reboot disruptions over all at once. It's also much easier to explain to people, and we don't have to annoy them with repeated notifications that yet another subset of our servers and services will be down for a bit.

(To make an incremental deployment more painful for us, these will normally have to be after-hours downtimes, which means that we'll be repeatedly staying late, perhaps once a week for three or four weeks as we progressively work through a rollout.)

In addition to the nodes of our SLURM cluster, there are a number of servers that can be rebooted in the background to some degree without people noticing much. We will often try the kernel update out on a few of them in advance, and then update others of them earlier in the day (or the day before) both as a final check and to reduce the number of systems we have to cover at the actual out of hours downtime. But a lot of our servers cannot really be tested much in advance, such as our fileservers or our web server (which is under constant load for reasons outside the scope of this entry). We can (and do) update a test fileserver or a test web server, but neither will see a production load and it's under production loads that problems are most likely to surface.

This is a specific example of how the 'cattle' model doesn't fit all situations. To have a transparent rolling update that involves reboots (or anything else that's disruptive on a single machine), you need to be able to transparently move people off of machines and then back on to them. This is hard to get in any environment where people have long term usage of specific machines, where they have login sessions and running compute jobs and so on, and where you have have non-redundant resources on a single machine (such as NFS fileservers without transparent failover from server to server).

About four years ago I wrote an entry about how your SMART drive database of attribute meanings needs regular updates. That entry was written on the occasion of updating the database we use locally on our Ubuntu servers, and at the time we were using a mix of Ubuntu 18.04 and Ubuntu 20.04 servers, both of which had older drive databases that probably dated from early 2018 and early 2020 respectively. It is now late 2025 and we use a mix of Ubuntu 24.04 and 22.04 servers, both of which have drive databases that are from after October of 2021.

Experienced system administrators know where this one is going: today I updated our SMART drive database again, to a version of the SMART database that was more recent than the one shipped with 24.04 instead of older than it.

It's a fact of life that people forget things. People especially forget things that are a long way away, even if they make little notes in their worklog message when recording something that they did (as I did four years ago). It's definitely useful to plan ahead in your documentation and write these notes, but without an external thing to push you or something to explicitly remind you, there's no guarantee that you'll remember.

All of which leads me to the view that it would be useful for us to have a long range calendar reminder system, something that could be used to set reminders for more than a year into the future and ideally allow us to write significant email messages to our future selves to cover all of the details (although there are hacks around that, such as putting the details on a web page and having the calendar mail us a link). Right now the best calendar reminder system we have is the venerable calendar, which we can arrange to email one-line notes to our general address that reaches all sysadmins, but calendar doesn't let you include the year in the reminder date.

(For SMART drive database updates, we could get away with mailing ourselves once a year in, say, mid-June. It doesn't hurt to update the drive database more than every Ubuntu LTS release. But there are situations where a reminder several years in the future is what we want.)

PS: Of course it's not particularly difficult to build an ad-hoc script system to do this, with various levels of features. But every local ad-hoc script that we write is another little bit of overhead, and I'd like to avoid that kind of thing if at all possible in favour of a standard solution (that isn't a shared cloud provider calendar).

A commentator on my entry on systemd-resolved's new DNS server delegation feature asked:

My memory might fail me here, but: wasn't something like this a feature introduced in ISC's BIND 8, and then considered to be a bad mistake and dropped again in BIND 9 ?

I don't know about Bind, but what I do know is that this feature is present in other DNS resolvers (such as Unbound) and that it has a variety of uses. Some of those uses can be substituted with other features and some can't be, at least not as-is.

The quick version of 'DNS server delegation' is that you can send all queries under some DNS zone name off to some DNS server (or servers) of your choice, rather than have DNS resolution follow any standard NS delegation chain that may or may not exist in global DNS. In Unbound, this is done through, for example, Forward Zones.

DNS server delegation has at least three uses that I know of. First, you can use it to insert entire internal TLD zones into the view that clients have. People use various top level names for these zones, such as .internal, .kvm, .sandbox (our choice), and so on. In all cases you have some authoritative servers for these zones and you need to direct queries to these servers instead of having your queries go to the root nameservers and be rejected.

(Obviously you will be sad if IANA ever assigns your internal TLD to something, but honestly if IANA allows, say, '.internal', we'll have good reason to question their sanity. The usual 'standard DNS environment' replacement for this is to move your internal TLD to be under your organizational domain and then implement split horizon DNS.)

Second, you can use it to splice in internal zones that don't exist in external DNS without going to the full overkill of split horizon authoritative data. If all of your machines live in 'corp.example.org' and you don't expose this to the outside world, you can have your public example.org servers with your public data and your corp.example.org authoritative servers, and you splice in what is effectively a fake set of NS records through DNS server delegation. Related to this, if you want you can override public DNS simply by having an internal and an external DNS server, without split horizon DNS; you use DNS server delegation to point to the internal DNS server for certain zones.

(This can be replaced with split horizon DNS, although maintaining split horizon DNS is its own set of headaches.)

Finally, you can use this to short-cut global DNS resolution for reliability in cases where you might lose external connectivity. For example, there are within-university ('on-campus' in our jargon) authoritative DNS servers for .utoronto.ca and .toronto.edu. We can use DNS server delegation to point these zones at these servers to be sure we can resolve university names even if the university's external Internet connection goes down. We can similarly point our own sub-zone at our authoritative servers, so even if our link to the university backbone goes down we can resolve our own names.

(This isn't how we actually implement this; we have a more complex split horizon DNS setup that causes our resolving DNS servers to have a complete copy of the inside view of our zones, acting as caching secondaries.)

Yesterday I wrote about restarting or redoing something after a systemd service restarts. The non-hypothetical situation that caused me to look into this was that after we applied a package update to one system, systemd-networkd on it restarted and wiped out some critical policy based routing rules. Since I vaguely remembered this happening before, I sighed and arranged to have our rules automatically reapplied on both systems with policy based routing rules, following the pattern I worked out.

Wait, two systems? And one of them didn't seem to have problems after the systemd-networkd restart? Yesterday I ignored that and forged ahead, but really it should have set off alarm bells. The reason the other system wasn't affected was I'd already solved the problem the right way back in March of 2024, when we first hit this networkd behavior and I wrote an entry about it.

However, I hadn't left myself (or my co-workers) any notes about that March 2024 fix; I'd put it into place on the first machine (then the only machine we had that did policy based routing) and forgotten about it. My only theory is that I wanted to wait and be sure it actually fixed the problem before documenting it as 'the fix', but if so, I made a mistake by not leaving myself any notes that I had a fix in testing. When I recently built the second machine with policy based routing I copied things from the first machine, but I didn't copy the true networkd fix because I'd forgotten about it.

(It turns out to have been really useful that I wrote that March 2024 entry because it's the only documentation I have, and I'd probably have missed the real fix if not for it. I rediscovered it in the process of writing yesterday's entry.)

I know (and knew) that keeping notes is good, and that my memory is fallible. And I still let this slip through the cracks for whatever reason. Hopefully the valuable lesson I've learned from this will stick a bit so I don't stub my toe again.

(One obvious lesson is that I should make a note to myself any time I'm testing something that I'm not sure will actually work. Since it may not work I may want to formally document it in our normal system for this, but a personal note will keep me from completely losing track of it. You can see the persistence of things 'in testing' as another example of the aphorism that there's nothing as permanent as a temporary fix.)

Every so often on the Fediverse, people ask for advice on a monitoring system to run on their machine (desktop or server), and some of the time Prometheus, and when it does I wind up making awkward noises. On the one hand, we run Prometheus (and Grafana) and are happy with it, and I run separate Prometheus setups on my work and home desktops. On the other hand, I don't feel I can recommend picking Prometheus for a basic single-machine setup, despite running it that way myself.

Why do I run Prometheus on my own machines if I don't recommend that you do so? I run it because I already know Prometheus (and Grafana), and in fact my desktops (re)use much of our production Prometheus setup (but they scrape different things). This is a specific instance (and example) of a general thing in system administration, which is that not infrequently it's simpler for you to use something you already know even if it's not necessarily an exact fit (or even a great fit) for the problem. For example, if you're quite familiar with operating PostgreSQL databases, it might be simpler to use PostgreSQL for a new system where SQLite could do perfectly well and other people would find SQLite much simpler. Especially if you have canned setups, canned automation, and so on all ready to go for PostgreSQL, and not for SQLite.

(Similarly, our generic web server hammer is Apache, even if we're doing things that don't necessarily need Apache and could be done perfectly well or perhaps better with nginx, Caddy, or whatever.)

This has a flipside, where you use a tool because you know it even if there might be a significantly better option, one that would actually be easier overall even accounting for needing to learn the new option and build up the environment around it. What we could call "familiarity-driven design" is a thing, and it can even be a confining thing, one where you shape your problems to conform to the tools you already know.

(And you may not have chosen your tools with deep care and instead drifted into them.)

I don't think there's any magic way to know which side of the line you're on. Perhaps the best we can do is be a little bit skeptical about our reflexive choices, especially if we seem to be sort of forcing them in a situation that feels like it should have a simpler or better option (such as basic monitoring of a single machine).

(In a way it helps that I know so much about Prometheus because it makes me aware of various warts, even if I'm used to them and I've climbed the learning curves.)

Once upon a time, my email handling was relatively simple. I wasn't on any big mailing lists, so I had almost everything delivered straight to my inbox (both in the traditional /var/mail mbox sense and then through to MH's own inbox folder directory). I did some mail filtering with procmail, but it was all for things that I basically never looked at, so I had procmail write them to mbox files under $HOME/.mail. I moved email from my Unix /var/mail inbox to MH's inbox with MH's inc command (either running it directly or having exmh run it for me). Rarely, I had a mbox file procmail had written that I wanted to read, and at that point I inc'd it either to my MH +inbox or to some other folder.

Later, prompted by wanting to improve my breaks and vacations, I diverted a bunch of mailing lists away from my inbox. Originally I had procmail write these diverted messages to mbox files, then later I'd inc the files to read the messages. Then I found that outside of vacations, I needed to make this email more readily accessible, so I had procmail put them in MH folder directories under Mail/inbox (one of MH's nice features is that your inbox is a regular folder and can have sub-folders, just like everything else). As I noted at the time, procmail only partially emulates MH when doing this, and one of the things it doesn't do is keep track of new, unread ('unseen') messages.

(MH has a general purpose system for keeping track of 'sequences' of messages in a MH folder, so it tracks unread messages based on what is in the special 'unseen' sequence. Inc and other MH commands update this sequence; procmail doesn't.)

Along with this procmail setup I wrote a basic script, called mlists, to report how many messages each of these 'mailing list' inboxes had in them. After a while I started diverting lower priority status emails and so on through this system (and stopped reading the mailing lists); if I got a type of email in any volume that I didn't want to read right away during work, it probably got shunted to these side inboxes. At some point I made mlists optionally run the MH scan command to show me what was in each inbox folder (well, for the inbox folders where this was potentially useful information). The mlists script was still mostly simple and the whole system still made sense, but it was a bit more complex than before, especially when it also got a feature where it auto-reset the current message number in each folder to the first message.

A couple of years ago, I switched the MH frontend I used from exmh to MH-E in GNU Emacs, which changed how I read my email in practice. One of the changes was that I started using the GNU Emacs Speedbar, which always displays a count of messages in MH folders and especially wants to let you know about folders with unread messages. Since I had the hammer of my mlists script handy, I proceeded to mutate it to be what a comment in the script describes as "a discount maintainer of 'unseen'", so that MH-E's speedbar could draw my attention to inbox folders that had new messages.

This is not the right way to do this. The right way to do this is to have procmail deliver messages through MH's rcvstore, which as a MH command can update the 'unseen' sequence properly. But using rcvstore is annoying, partly because you have to use another program to add the locking it needs, so at every point the path of least resistance was to add a bit more hacks to what I already had. I had procmail, and procmail could deliver to MH folder directories, so I used it (and at the time the limitations were something I considered a feature). I had a script to give me basic information, so it could give me more information, and then it could do one useful thing while it was giving me information, and then the one useful thing grew into updating 'unseen'.

And since I have all of this, it's not even worth the effort of switching to the proper rcvstore approach and throwing a bunch of it away. I'm always going to want the 'tell me stuff' functionality of my mlists script, so part of it has to stay anyway.

Can I see similarities between this and how various of our system tools have evolved, mutated, and become increasingly complex? Of course. I think it's much the same obvious forces involved, because each step seems reasonable in isolation, right up until I've built a discount environment that duplicates much of rcvstore.

Sidebar: an extra bonus bit of complexity

It turns out that part of the time, I want to get some degree of live notification of messages being filed into these inbox folders. I may not look at all or even many of them, but there are some periodic things that I do want to pay attention to. So my discount special hack is basically:

tail -f .mail/procmail-log |
  egrep -B2 --no-group-separator 'Folder: /u/cks/Mail/inbox/'

(This is a script, of course, and I run it in a terminal window.)

This could be improved in various ways but then I'd be sliding down the convoluted complexity slope and I'm not willing to do that. Yet. Give it a few years and I may be back to write an update.

About two years ago I wrote an entry about how my switch from reading email with exmh to reading it in GNU Emacs with MH-E had affected my email reading behavior more than I expected. As time has passed and I've made more extensive customizations to my MH-E environment, this has continued. One of the recent ways I've noticed is that I'm slowly making more and more use of the fact that GNU Emacs is a multi-window editor ('multi-frame' in Emacs terminology) and reading email with MH-E inside it still leaves me with all of the basic Emacs facilities. Specifically, I can create several Emacs windows (frames) and use this to be working in multiple MH folders at the same time.

Back when I used exmh extensively, I mostly had MH pull my email into the default 'inbox' folder, where I dealt with it all at once. Sometimes I'd wind up pulling some new email into a separate folder, but exmh only really giving me a view of a single folder at a time combined with a system administrator's need to be regularly responding to email made that a bit awkward. At first my use of MH-E mostly followed that; I had a single Emacs MH-E window (frame) and within that window I switched between folders. But lately I've been creating more new windows when I want to spend time reading a non-inbox folder, and in turn this has made me much more willing to put new email directly into different (MH) folders rather than funnel it all into my inbox.

(I don't always make a new window to visit another folder, because I don't spend long on many of my non-inbox folders for new email. But for various mailing lists and so on, reading through them may take at least a bit of time so it's more likely I'll decide I want to keep my MH inbox folder still available.)

One thing that makes this work is that MH-E itself has reasonably good support for displaying and working on multiple folders at once. There are probably ways to get MH-E to screw this up and run MH commands with the wrong MH folder as the current folder, so I'm careful that I don't try to have MH-E carry out its pending MH operations in two MH-E folders at the same time. There are areas where MH-E is less than ideal when I'm also using command-line MH tools, because MH-E changes MH's global notion of the current folder any time I have it do things like show a message in some folder. But at least MH-E is fine (in normal circumstances) if I use MH commands to change the current folder; MH-E will just switch it back the next time I have it show another message.

PS: On a purely pragmatic basis, another change in my email handling is that I'm no longer as irritated with HTML emails because GNU Emacs is much better at displaying HTML than exmh was. I've actually left my MH-E setup showing HTML by default, instead of forcing multipart/alternative email to always show the text version (my exmh setup). GNU Emacs and MH-E aren't up to the level of, say, Thunderbird, and sometimes this results in confusing emails, but it's better than it was.

(The situation that seems tricky for MH-E is that people sometimes include inlined images, for example screenshots as part of problem reports, and MH-E doesn't always give any indication that it's even omitting something.)

Not infrequently I wind up maintaining slowly growing lists of filtering rules to either allow good things or weed out bad things. Not infrequently, traffic can potentially match more than one filtering rule, either because it has multiple bad (or good) characteristics or because some of the match rules overlap. My usual habit has been to add new rules to the end of my rule lists (or the relevant section of them), so the oldest rules are at the top and the newest ones are at the bottom.

After writing about how access control rules need some form of usage counters, it's occurred to me that maybe I want to reverse this, at least in typical systems where the first matching rule wins. The basic idea is that the rules I'm most likely to want to drop are the oldest rules, but by having them first I'm hindering my ability to see if they've been made obsolete by newer rules. If an old rule matches some bad traffic, a new rule matches all of the bad traffic, and the new rule is last, any usage counters will show a mix of the old rule and the new rule, making it look like the old rule is still necessary. If the order was reversed, the new rule would completely occlude the old rule and usage counters would show me that I could weed the old rule out.

(My view is that it's much less likely that I'll add a new rule at the bottom that's completely ineffectual because everything it matches is already matched by something earlier. If I'm adding a new rule, it's almost certainly because something isn't being handled by the collection of existing rules.)

Another possible advantage to this is that it will keep new rules at the top of my attention, because when I look at the rule list (or the section of it) I'll probably start at the top. Currently, the top is full of old rules that I usually ignore, but if I put new rules first I'll naturally see them right away.

(I think that most things I deal with are 'first match wins' systems. A 'last match wins' system would naturally work right here, but it has other confusing aspects. I also have the impression that adding new rules at the end is a common thing, but maybe it's just in the cultural water here.)

Today, for reasons outside the scope of this entry, I decided to spend some time maintaining and pruning the access control rules for Wandering Thoughts, this blog. Due to the ongoing crawler plague (and past abuses), Wandering Thoughts has had to build up quite a collection of access control rules, which are mostly implemented as a bunch of things in an Apache .htaccess file (partly 'Deny from ...' for IP address ranges and partly as rewrite rules based on other characteristics). The experience has left me with a renewed view of something, which is that systems with access control rules need some way of letting you see which rules are still being used by your traffic.

It's in the nature of systems with access control rules to accumulate more and more rules over time. You hit another special situation, you add another rule, perhaps to match and block something or perhaps to exempt something from blocking. These rules often interact in various ways, and over time you'll almost certainly wind up with a tangled thicket of rules (because almost no one goes back to carefully check and revisit all existing rules when they add a new one or modify an existing one). The end result is a mess, and one of the ways to reduce the mess is to weed out rules that are now obsolete. One way a rule can be obsolete is that it's not used any more, and often these are the easiest rules to drop once you can recognize them.

(A rule that's still being matched by traffic may be obsolete for other reasons, and rules that aren't currently being matched may still be needed as a precaution. But it's a good starting point.)

If you have the necessary log data, you can sometimes establish if a rule was actually ever used by manually checking your logs. For example, if you have logs of rejected traffic (or logs of all traffic), you can search it for an IP address range to see if a particular IP address rule ever matched anything. But this requires tedious manual effort and that means that only determined people will go through it, especially regularly. The better way is to either have this information provided directly, such as by counters on firewall rules, or to have something in your logs that makes deriving it easy.

(An Apache example would be to augment any log line that was matched by some .htaccess rule with a name or a line number or the like. Then you could go readily through your logs to determine which lines were matched and how often.)

The next time I design an access control rule system, I'm hopefully going to remember this and put something in its logging to (optionally) explain its decisions.

(Periodically I write something that has an access control rule system of some sort. Unfortunately all of mine to date have been quiet on this, so I'm not at all without sin here.)

One of the things our Prometheus environment has is a notion of different sorts of alerts, and in particular of less important alerts that should go to a subset of people (ie, me). There are various reasons for this, including that the alert is in testing, or it concerns a subsystem that only I should have to care about, or that it fires too often for other people (for example, a reboot notification for a machine we routinely reboot).

For historical reasons, there are at least four different ways that this can be done in our Prometheus environment:

• a special label can be attached to the Prometheus alert rule, which is appropriate if the alert rule itself is in testing or otherwise is low priority.

• a special label can be attached to targets in a scrape configuration, although this has some side effects that can be less than ideal. This affects all alerts that trigger based on metrics from, for example, the Prometheus host agent (for that host).

• our Prometheus configuration itself can apply alert relabeling to add the special label for everything from a specific host, as indicated by a "host" label that we add. This is useful if we have so many exporters being scraped from a particular host, or if I want to keep metric continuity (ie, the metrics not changing their label set) when a host moves into production.

• our Alertmanager configuration can specifically route certain alerts about certain machines to the 'less important alerts' destination.

The drawback of these assorted approaches is that now there are at least three places to check and possibly to update when a host moves from being a testing host into being a production host. A further drawback is some of these (the first two) are used a lot more often than others of these (the last two). When you have multiple things, some of which are infrequently used, and fallible humans have to remember to check them all, you can guess what can happen next.

And that is the simple version of why alerts about one of our fileservers wouldn't have gone to everyone here for about the past year.

How I discovered the problem was that I got an alert about one of the fileserver's Prometheus exporters restarting, and decided that I should update the alert configuration to make it so that alerts about this service restarting only went to me. As I was in the process of doing this, I realized that the alert already had only gone to me, despite there being no explicit configuration in the alert rule or the scrape configuration. This set me on an expedition into the depths of everything else, where I turned up an obsolete bit in our general Prometheus configuration.

On the positive side, now I've audited our Prometheus and Alertmanager configurations for any other things that shouldn't be there. On the negative side, I'm now not completely sure that there isn't a fifth place that's downgrading (some) alerts about (some) hosts.

Normally I use Bash (and Readline in general) in my own environment, where I have a standard .inputrc set up to configure things to my liking (although it turns out that one particular setting doesn't work now (and may never have), and I didn't notice). However, sometimes I wind up using Bash in foreign environments, for example if I'm su'd to root at the moment, and when that happens the differences can be things that I get annoyed by. I spent a bit of today running into this again and being irritated enough that this time I figured out how to fix it on the fly.

The general Bash command to do readline things is 'bind', and I believe it accepts all of the same syntax as readline init files do, both for keybindings and for turning off (mis-)features like bracketed paste (which we dislike enough that turning it off for root is a standard feature of our install framework). This makes it convenient if I forget the exact syntax, because I can just look at my standard .inputrc and copy lines from it.

What I want to do is the following:

• Switch Readline to the Unix word erase behavior I want:

  set bind-tty-special-chars off
Control-w: backward-kill-word

  Both of these are necessary because without the first, Bash will automatically bind Ctrl-w (my normal word-erase character) to 'unix-word-rubout' and not let you override that with your own binding.

  (This is the difference that I run into all the time, because I'm very used to be able to use Ctrl-W to delete only the most recent component of a path. I think this partly comes from habit and partly because you tab-complete multi-component paths a component at a time, so if I mis-completed the latest component I want to Ctrl-W just it. M-Del is a standard Readline binding for this, but it's less convenient to type and not something I remember.)

• Make readline completion treat symbolic links to directories as if they were directories:

  set mark-symlinked-directories on

  When completing paths and so on, I mostly don't bother thinking about the difference between an actual directory (such as /usr/bin) and a symbolic link to a directory (such as /bin on modern Linuxes). If I type '/bi<TAB>' I want this to complete to '/bin/', not '/bin', because it's basically guaranteed that I will go on to tab-complete something in '/bin/'. If I actually want the symbolic link, I'll delete the trailing '/' (which does happen every so often, but much less frequently than I want to tab-complete through the symbolic link).

• Make readline forget any random edits I did to past history lines when I hit Return to finally do something:

  set revert-all-at-newline on

  The behavior I want from readline is that past history is effectively immutable. If I edit some bit of it and then abandon the edit by moving to another command in the history (or just start a command from scratch), the edited command should revert to being what I actually typed back when I executed it no later than when I hit Return on the current command and start a new one. It infuriates me when I cursor-up (on a fresh command) and don't see exactly the past commands that I typed.

  (My notes say I got this from Things You Didn't Know About GNU Readline.)

This is more or less in the order I'm likely to fix them. The different (and to me wrong) behavior of C-w is a relatively constant irritation, while the other two are less frequent.

(If this irritates me enough on a particular system, I can probably do something in root's .bashrc, if only to add an alias to use 'bind -f ...' on a prepared file. I can't set these in /root/.inputrc, because my co-workers don't particularly agree with my tastes on these and would probably be put out if standard readline behavior they're used to suddenly changed on them.)

(In other Readline things I want to remember, there's Readline's support for fishing out last or first or Nth arguments from earlier commands.)

We have a couple of general purpose, general access networks that anyone can use to connect their devices to; one is a wired network (locally, it's called our 'RED' network after the colour of the network cables used for it), and the other is a departmental wireless network that's distinct from the centrally run university-wide network. However, both of these networks have a requirement that we need to be able to more or less identify who is responsible for a machine on them. Currently, this is done through (IPv4) DHCP and registering the Ethernet address of your device. This is a problem for any IPv6 deployment, because the Android developers refuse to support DHCPv6.

We're starting to look more seriously at IPv6, including sort of planning out how our IPv6 subnets will probably work, so I came back to thinking about this issue recently. My conclusion and decision was to give up on letting Android devices use IPv6 on our networks. We can't use SLAAC (StateLess Address AutoConfiguration) because that doesn't require any sort of registration, and while Android devices apparently can use IPv6 Prefix Delegation, that would consume /64s at a prodigious rate using reasonable assumptions. We'd also have to build a system to do it. So there's no straightforward answer, and while I can think of potential hacks, I've decided that none of them are particular good options compared to the simple choice to not support IPv6 for Android by way of only supporting DHCPv6.

(Our requirement for registering a fixed Ethernet address also means that any device that randomizes its wireless Ethernet address on every connection has to turn that off. Hopefully all such devices actually have such an option.)

I'm only a bit sad about this, because you can only hope that a rock rolls uphill for so long before you give up. IPv6 is still not a critical thing in my corner of the world (as shown by how no one is complaining to us about the lack of it), so some phones continuing to not have IPv6 is not likely to be a big deal to people here.

(Android devices that can be connected to wired networking will be able to get IPv6 on some research group networks. Some research groups ask for their network to be open and not require pre-registration of devices (which is okay if it only exists in access-controlled space), and for IPv6 I expect we'll do this by turning on SLAAC on the research group's network and calling it a day.)

Over on the Fediverse, I said something:

It appears to have taken only a few hours (or at most a few hours) from putting a new resolving DNS server into production to seeing outside parties specifically probing it to see if it's an open resolver.

I assume people are snooping activity on authoritative DNS servers and going from there, instead of spraying targeted queries at random IPs, but maybe they are mass scanning.

There turns out to be some interesting aspects to these probes. This new DNS server has two network interfaces, both firewalled off from outside queries, but only one is used as the source IP on queries to authoritative DNS servers. In addition, we have other machines on both networks, with firewalls, so I can get a sense of the ambient DNS probes.

Out of all of these various IPs, the IP that the new DNS server used for querying authoritative DNS servers, and only that IP, very soon saw queries that were specifically tuned for it:

124.126.74.2.54035 > 128.100.X.Y.53: 16797 NS? . (19)
124.126.74.2.7747 > 128.100.X.Y.7: UDP, length 512
124.126.74.2.54035 > 128.100.X.Y.53: 17690 PTR? Y.X.100.128.in-addr.arpa. (47)

This was a consistent pattern from multiple IPs; they all tried to query for the root zone, tried to check the UDP echo port, and then tried a PTR query for the machine's IP itself. Nothing else saw this pattern; not the machine's other IP on a different network, not another IP on the same network, and so on. This pattern and the lack of this pattern to other IPs is what's led me to assume that people are somehow identifying probe targets based on what source IPs they seem making upstream queries.

(There are a variety of ways that you could do this without having special access to DNS servers. APNIC has long used web ad networks and special captive domains and DNS servers for them to do various sorts of measurements, and you could do similar things to discover who was querying your captive DNS servers.)

Suppose, not hypothetically, that you have an Unbound server with multiple network interfaces, at least two (which I will call A and B), and you'd like Unbound to listen on all of the interfaces. Perhaps these are physical interfaces and there are client machines on both, or perhaps they're virtual interfaces and you have virtual machines on them. Let's further assume that these are routed networks, so that in theory people on A can talk to IP addresses on B and vice versa.

The obvious and straightforward way to have Unbound listen on all of your interfaces is with a server stanza like this:

server:
  interface: 0.0.0.0
  interface: ::0
  # ... probably some access-control statements

This approach works 99% of the time, which is probably why it appears all over the documentation. The other 1% of the time is when a DNS client on network A makes a DNS request to Unbound's IP address on network B; when this happens, the network A client will not get any replies. Well, it won't get any replies that it accepts. If you use tcpdump to examine network traffic, you will discover that Unbound is sending replies to the client on network A using its network A IP address as the source address (which is the default behavior if you send packets to a network you're directly attached to; you normally want to use your IP on that network as the source IP). This will fail with almost all DNS client libraries because DNS clients reject replies from unexpected sources, which is to say any IP other than the IP they sent their query to.

(One way this might happen is if the client moves from network B to network A without updating its DNS configuration. Or you might be testing to see if Unbound's network B IP address answers DNS requests.)

The other way to listen on all interfaces in modern Unbound is to use 'interface-automatic: yes' (in server options), like this:

server:
  interface-automatic: yes

The important bit of what interface-automatic does for you is mentioned in passing in its documentation, and I've emphasized it here:

Listen on all addresses on all (current and future) interfaces, detect the source interface on UDP queries and copy them to replies.

As far as I know, you can't get this 'detect the source interface' behavior for UDP queries in any other way if you use 'interface: 0.0.0.0' to listen on everything. You get it if you listen on specific interfaces, perhaps with 'ip-transparent: yes' for safety:

server:
  interface: 127.0.0.1
  interface: ::1
  interface: <network A>.<my-A-IP>
  interface: <network B>.<my-B-IP>
  # insure we always start
  ip-transparent: yes

Since 'interface-automatic' is marked as an experimental option I'd love to be wrong, but I can't spot an option in skimming the documentation and searching on some likely terms.

(I'm a bit surprised that Unbound doesn't always copy the IP address it received UDP packets on and use that for replies, because I don't think things work if you have the wrong IP there. But this is probably an unusual situation and so it gets papered over, although now I'm curious how this interacts with default routes.)

This past Saturday (yesterday as I write this), a university machine room had an AC failure of some kind:

It's always fun times to see a machine room temperature of 54C and slowly climbing. It's not our machine room but we have switches there, and I have a suspicion that some of them will be ex-switches by the time this is over.

This machine room and its AC has what you could call a history; in 2011 it flooded partly due to an AC failure, then in 2016 it had another AC issue, and another in 2024 (and those are just the ones I remember and can find entries for).

Most of this machine room is a bunch of servers from another department, and my assumption is that they are what created all of the heat when the AC failed. Both we and the other department have switches in the room, but networking equipment is usually relatively low-heat compared to active servers. So I found it interesting that the temperature graph rises in a smooth arc to its maximum temperature (and then drops abruptly, presumably as the AC starts to get fixed). To me this suggests that many of the servers in the room kept running, despite the ambient temperature hitting 54C (and their internal temperatures undoubtedly being much higher). If some servers powered off from the heat, it wasn't enough to stabilized the heat level of the room; it was still increasing right up to when it started dropping rapidly.

(Servers may well have started thermally throttling various things, and it's possible that some of them crashed without powering off and thus potentially without reducing the heat load. I have second hand information that some UPS units reported battery overheating.)

It's one thing to be fairly confident that server thermal limits are set unrealistically high. It's another thing to see servers (probably) keep operating at 54C, rather than fall over with various sorts of failures. For example, I wouldn't have been surprised if power supplies overheated and shut down (or died entirely).

(I think desktop PSUs are often rated as '0C to 50C', but I suspect that neither end of that rating is actually serious, and this was over 50C anyway.)

I rather suspect that running at 50+C for a while has increased the odds of future failures and shortened the lifetime of everything in this machine room (our switches included). But it still amazes me a bit that things didn't fall over and fail, even above 50C.

(When I started writing this entry I thought I could make some fairly confident predictions about the servers keeping running purely from the temperature graph. But the more I think about it, the less I'm sure of that. There are a lot of things that could be going on, including server failures that leave them hung or locked up but still with PSUs running and pumping out heat.)

Some time back I read Simon Tatham's Policy of transience (via) and recognized both points of similarity and points of drastic departure between Tatham and I. Both Tatham and I use transient shell history, transient terminal and application windows (sort of for me), and don't save our (X) session state, and in general I am a 'disposable' usage pattern person. However, I depart from Tatham in that I have a permanently running browser and I normally keep my login sessions running until I reboot my desktops. But broadly I'm a 'transient' or 'disposable' person, where I mostly don't keep inactive terminal windows or programs around in case I might want them again, or even immediately re-purpose them from one use to another.

(I do have some permanently running terminal windows, much like I have permanently present other windows on my desktop, but that's because they're 'in use', running some program. And I have one inactive terminal window but that's because exiting that shell ends my entire X session.)

The big way that I depart from Tatham is already visible in my old desktop tour, in the form of a collection of iconified browser windows (in carefully arranged spots so I can in theory keep track of them). These aren't web pages I use regularly, because I have a different collection of schemes for those. Instead they're a collection of URLs that I'm keeping around to read later or in general to do something with. This is anathema to Tatham, who keeps track of URLs to read in other ways, but I've found that it's absolutely necessary for me.

Over and over again I've discovered that if something isn't visible to me, shoved in front of my nose, it's extremely likely to drop completely out of my mind. If I file email into a 'to be dealt with' or 'to be read later' or whatever folder, or if I write down URLs to visit later and explanations of them, or any number of other things, I almost might as well throw those things away. Having a web page in an iconified Firefox window in no way guarantees that I'll ever read it, but writing its URL down in a list guarantees that I won't. So I keep an optimistic collection of iconified Firefox windows around (and every so often I look at some of them and give up on them).

It would be nice if I didn't need to do this and could de-clutter various bits of my electronic life. But by now I've made enough attempts over a long enough period of time to be confident that my mind doesn't work that way and is unlikely to ever change its ways. I need active, ongoing reminders for things to stick, and one of the best forms is to have those reminders right on my desktop.

(And because the reminders need to be active and ongoing, they also need to be non-intrusive. Mailing myself every morning with 'here are the latest N URLs you've saved to read later' wouldn't work, for example.)

PS: I also have various permanently running utility programs and their windows, so my desktop is definitely not minimalistic. A lot of this is from being a system administrator and working with a bunch of systems, where I want various sorts of convenient fast access and passive monitoring of them.

When I wrote about how Exim's ${run ...} string expansion operator changed how it did quoting, I (sort of) mentioned that I found this when I tested a new version of Exim. Some people would do testing like this in a thorough, automated manner, but I don't go that far. Instead I have a written down test plan, with some resources set up for it in advance. Well, it's more accurate to say that I have test plans, because I have a separate test plan for each of our important mail servers because they have different features and so need different things tested.

In the beginning I simply tested all of the important features of a particular mail server by hand and from memory when I rebuilt it on a new version of Ubuntu. Eventually I got tired of having to reinvent my test process from scratch (or from vague notes) every time around (for each mail server), so I started writing it down. In the process of writing my test process down the natural set of things happened; I made it more thorough and systematic, and I set up various resources (like saved copies of the EICAR test file) to make testing more cut and paste. Having an organized, written down test plan, even as basic as ours is, has made it easier to test new builds of our Exim servers and made that testing more comprehensive.

I test most of our mail servers primarily by using swaks to send various bits of test email to them and then watching what happens (both in the swaks SMTP session and in the Exim logs). So a lot of the test plan is 'run this swaks command and ...', with various combinations of sending and receiving addresses, starting with the very most basic test of 'can it deliver from a valid dummy address to a valid dummy address'. To do some sorts of testing, such as DNS blocklist tests, I take advantage of the fact that all of the IP-based DNS blocklists we use include 127.0.0.2, so that part of the test plan is 'use swaks on the mail machine itself to connect from 127.0.0.2'.

(Some of our mail servers can apply different filtering rules to different local addresses, so I have various pre-configured test addresses set up to make it easy to test that per-address filtering is working.)

The actual test plans are mostly a long list of 'run more or less this swaks command, pointing it at your test server, to test this thing, and you should see the following result'. This is pretty close to cut and paste, which makes it relatively easy and fast for me to run through.

One qualification is that these test plans aren't attempting to be an exhaustive check of everything we do in our Exim configurations. Instead, they're mostly about making sure that the basics work, like delivering straightforward email, and that Exim can interact properly with the outside world, such as talking to ClamAV and rspamd or running external programs (which also tests that the programs themselves work on the new Ubuntu version). Testing every corner of our configurations would be exhausting and my feeling is that it would generally be pointless. Exim is stable software and mostly doesn't change or break things from version to version.

(Part of this is pragmatic experience with Exim and knowledge of what our configuration does conditionally and what it checks all of the time. If Exim does a check all of the time and basic mail delivery works, we know we haven't run into, say, an issue with tainted data.)

Suppose that you have a shiny new IAM system, and you take the 'access management' part of it seriously. Global access management is (or should be) simple; if you disable or suspect someone in your IAM system, they should wind up disabled everywhere. Well, they will wind up unable to authenticate. If they have existing credentials that are used without checking with your IAM system (including things like 'an existing SSH login'), you'll need some system to propagate the information that someone has been disabled in your IAM to consumers and arrange that existing sessions, credentials, and so on get shut down and revoked.

(This system will involve both IAM software features and features in the software that uses the IAM to determine identity.)

However, this only covers global access management. You probably have some things that only certain people should have access to, or that treat certain people differently. This is where our experiences with a non-IAM environment suggest to me that things start getting complex. For pure access, the simplest thing probably is if every separate client system or application has a separate ID and directly talks to the IAM, and the IAM can tell it 'this person cannot authenticate (to you)' or 'this person is disabled (for you)'. This starts to go wrong if you ever put two or more services or applications behind the same IAM client ID, for example if you set up a web server for one application (with an ID) and then host another application on the same web server because of convenience (your web server is already there and already set up to talk to the IAM and so on).

This gets worse if there is a layer of indirection involved, so that systems and application don't talk directly to your IAM but instead talk to, say, a LDAP server or a Radius server or whatever that's fed from your IAM (or is the party that talks to your IAM). I suspect that this is one reason why IAM software has a tendency to directly support a lot of protocols for identity and authentication.

(One thing that's sort of an extra layer of indirection is what people are trying to do, since they may have access permission for some things but not others.)

Another approach is for your IAM to only manage what 'groups' people are in and provide that information to clients, leaving it up to clients to make access decisions based on group membership. On the one hand, this is somewhat more straightforward; on the other hand, your IAM system is no longer directly managing access. It has to count on clients doing the right thing with the group information it hands them. At a minimum this gives you much less central visibility into what your access management rules are.

People not infrequently want complicated access control conditions for individual applications (including things like privilege levels). In any sort of access management system, you need to be able to express these conditions in rules. There's no uniform approach or language for expressing access control conditions, so your IAM will use one, your Unix systems will use one (or more) that you probably get to craft by hand using PAM tricks, your web applications will use one or more depending on what they're written in, and so on and so forth. One of the reasons that these languages differ is that the capabilities and concepts of each system will differ; a mesh VPN has different access control concerns than a web application. Of course these differences make it challenging to handle all of their access management in one single spot in an IAM system, leaving you with the choice of either not being able to do everything you want to but having it all in the IAM or having partially distributed access management.

The Exim mail server has, among other features, a string expansion language with quite a number of expansion operators. One of those expansion operators is '${run}', which 'expands' by running a command and substituting in its output. As is commonly the case, ${run} is given the command to run and all of its command line arguments as a single string, without any explicit splitting into separate arguments:

${run {/some/command -a -b foo -c ...} [...]}

Any time a program does this, a very important question to ask is how this string is split up into separate arguments in order to be exec()'d. In Exim's case, the traditional answer is that it was rather complicated and not well documented, in a way that required you to explicitly quote many arguments that came from variables. In my entry on this I called Exim's then current behavior dangerous and wrong but also said it was probably too late to change it. Fortunately, the Exim developers did not heed my pessimism.

In Exim 4.96, this behavior of ${run} changed. To quote from the changelog:

The ${run} expansion item now expands its command string elements after splitting. Previously it was before; the new ordering makes handling zero-length arguments simpler. The old ordering can be obtained by appending a new option "preexpand", after a comma, to the "run".

(The new way is more or less the right way to do it, although it can create problems with [[some sorts of command string expansions.)

This is an important change because this change is not backward compatible if you used deliberate quoting in your ${run} command string. For example, if you ever expanded a potentially dangerous Exim variable in a ${run} command (for example, one that might have a space in it), you previously had to wrap it in ${quote}:

${run {/some/command \
         --subject ${quote:$header_subject:} ...

(As seen in my entry on our attachment type logging with Exim.)

In Exim 4.96 and later, this same ${run} string expansion will add spurious quote marks around the email message's Subject: header as your program sees it. This is because ${quote:...} will add them, since you asked it to generate a quoted version of its argument, and then ${run} won't strip them out as part of splitting the command string apart into arguments because the command string has already been split before the ${quote:} was done. What this shows is that you probably don't need explicit quoting in ${run} command strings any more, unless you're doing tricky expansions with string expressions (in which case you'll have to switch back to the old way of doing it).

To be clear, I'm all for this change. It makes straightforward and innocent use of ${run} much safer and more reliable (and it plays better with Exim's new rules about 'tainted' strings from the outside world, such as the subject header). Having to remote my use of ${quote:...} is a minor price to pay, and learning this sort of stuff in advance is why I build test servers and have test plans.

(This elaborates on a Fediverse post of mine.)

Over on the Fediverse I said something about IAM:

My IAM choices appear to be "bespoke giant monolith" or "DIY from a multitude of OSS pieces", and the natural way of life appears to be that you start with the latter because you don't think you need IAM and then you discover maybe you have to blow up the world to move to the first.

At work we are the latter: /etc/passwd to LDAP to a SAML/OIDC server depending on what generation of software and what needs. With no unified IM or AM, partly because no rules system for expressing it.

Identity and Access Management (IAM) isn't the same thing as (single sign on) authentication, although I believe it's connected to authorization if you take the 'Access' part seriously, and also a bunch of IAM systems will also do some or all of authentication too so everything is in one place. However, all of these things can be separated, and in complex environments they are (for example, the university's overall IAM environment, also).

(If you have an IAM system you're presumably going to want to feed information from it to your authentication system, so that it knows who is (still) valid to authenticate and perhaps how.)

I believe that one thing that makes IAM systems complicated is interfacing with what could be called 'legacy systems', which in this context includes garden variety Unix systems. If you take your IAM system seriously, everything that knows about 'logins' or 'users' needs to somehow be drawing data from the IAM system, and the IAM system has to know how to provide each with the information it needs. Or alternately your legacy systems need to somehow merge local identity information (Unix home directories, UIDs, GIDs, etc) with the IAM information. Since people would like their IAM system to do it all, I think this is one driver of IAM system complexity and those bespoke giant monoliths that want to own everything in your environment.

(The reason to want your IAM system to do it all is that if it doesn't, you're building a bunch of local tools and then your IAM information is fragmented. What UID is this person on your Unix systems? Only your Unix systems know, not your central IAM database. For bonus points, the person might have different UIDs on different Unix systems, depending.)

If you start out with a green field new system, you can probably build in this central IAM from the start (assuming that you can find and operate IAM software that does what you want and doesn't make you back away in terror). But my impression is that central IAM systems are quite hard to set up, so the natural alternative is that you start without an IAM system and then are possibly faced with trying to pull all of your /etc/passwd, Apache authentication data, LDAP data, and so on into a new IAM system that is somehow going to take over the world. I have no idea how you'd pull off this transition, although presumably people have.

(In our case, we started our Unix systems well before IAM systems existed. There are accounts here that have existed since the 1980s, partly because professors and retired professors tend to stick around for a long time.)

The difficulty of moving our environment to anything like an IAM system leaves me looking at the whole thing from the outside. If we had to add an 'IAM system', it would likely be because something else we wanted to do needed to be fed data from some IAM system using some IAM protocol. The IAM system would probably not become the center of identity and access management, but just another thing that we pushed information into and updated information in.

We have a long standing system to monitor the temperatures of our machine rooms and alert us if there are problems. A recent discussion about the state of the temperature in one of them made me realize that we want to monitor and alert for two different problems, and because they're different we need two different sorts of alerts in our monitoring system.

The first, obvious problem is a machine room AC failure, where the AC shuts off or becomes almost completely ineffective. In our machine rooms, an AC failure causes a rapid and sustained rise in temperature to well above its normal maximum level (which is typically reached just before the AC starts its next cooling cycle). AC failures are high priority issues that we want to alert about rapidly, because we don't have much time before machines start to cook themselves (and they probably won't shut themselves down before the damage has been done).

The second problem is an AC unit that can't keep up with the room's heat load; perhaps its filters are (too) clogged, or it's not getting enough cooling from the roof chillers, or various other mysterious AC reasons. The AC hasn't failed and it is still able to cool things to some degree and keep the temperature from racing up, but over time the room's temperature steadily drifts upward. Often the AC will still be cycling on and off to some degree and we'll see the room temperature vary up and down as a result; at other things the room temperature will basically reach a level and more or less stay there, presumably with the AC running continuously.

One issue we ran into is that a fast triggering alert that was implicitly written for the AC failure case can wind up flapping up and down if insufficient AC has caused the room to slowly drift close to its triggering temperature level. As the AC works (and perhaps cycles on and off), the room temperature will shift above and then back below the trigger level, and the alert flaps.

We can't detect both situations with a single alert, so we need at least two. Currently, the 'AC is not keeping up' alert looks for sustained elevated temperatures with the temperature always at or above a certain level over (much) more time than the AC should take to bring it down, even if the AC has to avoid starting for a bit of time to not cycle too fast. The 'AC may have failed' alert looks for high temperatures over a relatively short period of time, although we may want to make this an average over a short period of time.

(The advantage of an average is that if the temperature is shooting up, it may trigger faster than a 'the temperature is above X for Y minutes' alert. The drawback is that an average can flap more readily than a 'must be above X for Y time' alert.)

We recently had a big downtime at work where part of the work was me doing a relatively complex and touchy thing. Naturally I made a checklist, but also naturally my checklist turned out to be incomplete, with some things I'd forgotten and some steps that weren't quite right or complete. This is a good illustration that checklists are hard to create.

Checklists are hard partly because they require us to try to remember, reconstruct, and understand everything in what's often a relatively complex system that is too big for us to hold in our mind. If your understanding is incomplete you can overlook something and so leave out a step or a part of a step, and even if you write down a step you may not fully remember (and record) why the step has to be there. My view is that this is especially likely in system administration where we may have any number of things that have been quietly sitting in the corner for some time, working away without problems, and so they've slipped out of our minds.

(For example, one of the issues that we ran into in this downtime was not remembering all of the hosts that ran crontab jobs that used one particular filesystem. Of course we thought we did know, so we didn't try to systematically look for such crontab jobs.)

To get a really solid checklist you have to be able to test it, much like all documentation needs testing. Unfortunately, a lot of the checklists I write (or don't write) are for one-off things that we can't really test in advance for various reasons, for example because they involve a large scale change to our live systems (that requires a downtime). If you're lucky you'll realize that you don't know something or aren't confident in something while writing the checklist, so you can investigate it and hopefully get it right, but some of the time you'll be confident you understand the problem but you're wrong.

Despite any imperfections, checklists are still a good thing. An imperfect written down checklist is better than relying on your memory and mind on the fly almost all of the time (the rare exceptions are when you wouldn't even dare do the operation without a checklist but an imperfect checklist tempts you into doing it and fumbling).

(You can try to improve the situation by keeping notes on what was missed in the checklist and then saving or publishing these notes somewhere. You can review these after the fact notes on what was missed in this specific checklist if you have to do the thing again, or look for specific types of things you tend to overlook and should specifically check for the next time you're making a checklist that touches on some area.)

Every so often I think about random things, and today's random thing was how our environment might look if it was rebuilt from scratch as a modern style greenfield development. One of the obvious assumptions is that it'd involve a lot of use of containers, which led me to wondering how you handle traditional Unix style login servers. This is a relevant issue for us because we have such traditional login servers and somewhat to our surprise, they still see plenty of use.

We have two sorts of login servers. There's effectively one general purpose login server that people aren't supposed to do heavy duty computation on (and which uses per-user CPU and RAM limits to help with that), and four 'compute' login servers where they can go wild and use up all of the CPUs and memory they can get their hands on (with no guarantees that there will be any, those machines are basically first come, first served; for guaranteed CPUs and RAM people need to use our SLURM cluster). Usage of these servers has declined over time, but they still see a reasonable amount of use, including by people who have only recently joined the department (as graduate students or otherwise).

What people log in to our compute servers to do probably hasn't changed much, at least in one sense; people probably don't log in to a compute server to read their mail with their favorite text mode mail reader (yes, we have Alpine and Mutt users). What people use the general purpose 'application' login server for likely has changed a fair bit over time. It used to be that people logged in to run editors, mail readers, and other text and terminal based programs. However, now a lot of logins seem to be done either to SSH to other machines that aren't accessible from the outside world or to run the back-ends of various development environments like VSCode. Some people still use the general purpose login server for traditional Unix login things (me included), but I think it's rarer these days.

(Another use of both sorts of servers is to run cron jobs; various people have various cron jobs on one or the other of our login servers. We have to carefully preserve them when we reinstall these machines as part of upgrading Ubuntu releases.)

PS: I believe the reason people run IDE backends on our login servers is because they have their code on our fileservers, in their (NFS-mounted) home directories. And in turn I suspect people put the code there partly because they're going to run the code on either or both of our SLURM cluster or the general compute servers. But in general we're not well informed about what people are using our login servers for due to our support model.

About five years ago I wrote an entry on what OSes we were using at the time. Five years is both a short time and a long time here, and in that time some things have changed.

Our primary OS is still Ubuntu LTS; it's our default and we use it on almost everything. On the one hand, these days 'almost everything' covers somewhat more ground than it did in 2020, as some machines have moved from OpenBSD to Ubuntu. On the other hand, as time goes by I'm less and less confident that we'll still be using Ubuntu in five years, because I expect Canonical to start making (more) unfortunate and unacceptable changes any day now. Our most likely replacement Linux is Debian.

CentOS is dead here, killed by a combination of our desire to not have two Linux variants to deal with and CentOS Stream. We got rid of the last of our CentOS machines last year. Conveniently, our previous commercial anti-spam system vendor effectively got out of the business so we didn't have to find a new Unix that they supported.

We're still using OpenBSD, but it's increasingly looking like a legacy OS that's going to be replaced by FreeBSD as we rebuild the various machines that currently run OpenBSD. Our primary interests are better firewall performance and painless mirrored root disks, but if we're going to run some FreeBSD machines and it can do everything OpenBSD can, we'd like to run fewer Unixes so we'll probably replace all of the OpenBSD machines with FreeBSD ones over time. This is a shift in progress and we'll see how far it goes, but I don't expect the number of OpenBSD machines we run to go up any more; instead it's a question of how far down the number goes.

(Our opinions about not using Linux for firewalls haven't changed. We like PF, it's just we like FreeBSD as a host for it more than OpenBSD.)

We continue to not use containers so we don't have to think about a separate, minimal Linux for container images.

There are a lot of research groups here and they run a lot of machines, so research group machines are most likely running a wide assortment of Linuxes and Unixes. We know that Ubuntu (both LTS and non-LTS) is reasonably popular among research groups, but I'm sure there are people with other distributions and probably some use of FreeBSD, OpenBSD, and so on. I believe there may be a few people still using Solaris machines.

(My office desktop continues to run Fedora, but I wouldn't run it on any production server due to the frequent distribution version updates. We don't want to be upgrading distribution versions every six months.)

Overall I'd say we've become a bit more of an Ubuntu LTS monoculture than we were before, but it's not a big change, partly because we were already mostly Ubuntu. Given our views on things like firewalls, we're probably never going to be all-Ubuntu or all-Linux.

I recently wrote about a new little script of mine, which I use to start programs in terminals in a way that I can interact with them (to simplify it). Much of what I start with this tool doesn't need to run in a terminal window at all; the actual program will talk directly to the X server or arrange to talk to my Firefox or the like. I could in theory start them directly from my X session startup script, as I do with other things.

The reason I haven't put these things in my X session startup is that running things in shell sessions in terminal windows is the easiest way to interact with them in all sorts of ways. It's trivial to stop the program or restart it, to look at its output, to rerun it with slightly different arguments if I need to, it automatically inherits various aspects of my current X environment, and so on. You can do all of these things with programs in ways other than using shell sessions in terminals, but it's generally going to be more awkward.

(For instance, on systemd based Linuxes, I could make some of these programs into systemd user services, but I'd still have to use systemd commands to manipulate them. If I run them as standalone programs started from my X session script, it's even more work to stop them, start them again, and so on.)

For well established programs that I expect to never restart or want to look at output from, I'll run them from my X session startup script. But for new programs, like these, they get to spend a while in terminal windows because that's the easiest way. And some will be permanent terminal window occupants because they sometimes produce (text) output.

On the one hand, using terminal windows for this is simple and effective, and I could probably make it better by using a multi-tabbed terminal program, with one tab for each program (or the equivalent in a regular terminal program with screen or tmux). On the other hand, it feels a bit sad that in 2025, our best approach for flexible interaction with a program and monitoring its output is 'put it in a terminal'.

(It's also irritating that with some programs, the easiest and best way to make sure that they really exit when you want them to shut down, rather than "helpfully" lingering on in various ways, is to run them from a terminal and then Ctrl-C them when you're done with them. I have to use a certain video conferencing application that is quite eager to stay running if you tell it to 'quit', and this is my solution to it. Someday I may have to figure out how to put it in a systemd user unit so that it can't stage some sort of great escape into the background.)

Recently I read Apps shouldn’t let users enter OpenSSL cipher-suite strings by Frank Denis (via), which advocates for providing at most a high level interface to people that lets them express intentions like 'forward secrecy is required' or 'I have to comply with FIPS 140-3'. As a system administrator, I've certainly been guilty of not keeping OpenSSL cipher suite strings up to date, so I have a good deal of sympathies for the general view of trusting the clients and the libraries (and also possibly the servers). But at the same time, I think that this approach has some issues. In particular, if you're only going to set generic intents, you have to trust that the programs and libraries have good defaults. Unfortunately, historically time when system administrators have most reached for setting specific OpenSSL cipher suite strings was when something came up all of a sudden and they didn't trust the library or program defaults to be up to date.

The obvious conclusion is that an application or library that wants people to only set high level options needs to commit to agility and fast updates so that it always has good defaults. This needs more than just the upstream developers making prompt updates when issues come up, because in practice a lot of people will get the program or library through their distribution or other packaging mechanism. A library that really wants people to trust it here needs to work with distributions to make sure that this sort of update can rapidly flow through, even for older distribution versions with older versions of the library and so on.

(For obvious reasons, people are generally pretty reluctant to touch TLS libraries and would like to do it as little as possible, leaving it to specialists and even then as much as possible to the upstream. Bad things can and have happened here.)

If I was doing this for a library, I would be tempted to give the library two sets of configuration files. One set, the official public set, would be the high level configuration that system administrators were supposed to use to express high level intents, as covered by Frank Denis. The other set would be internal configuration that expressed all of those low level details about cipher suite preferences, what cipher suites to use when, and so on, and was for use by the library developers and people packaging and distributing the library. The goal is to make it so that emergency cipher changes can be shipped as relatively low risk and easily backported internal configuration file changes, rather than higher risk (and thus slower to update) code changes. In an environment with reproducible binary builds, it'd be ideal if you could rebuild the library package with only the configuration files changed and get library shared objects and so on that were binary identical to the previous versions, so distributions could have quite high confidence in newly-built updates.

(System administrators who opted to edit these second set of files themselves would be on their own. In packaging systems like RPM and Debian .debs, I wouldn't even have these files marked as 'configuration files'.)

Recently on the Fediverse I posted a puzzle about a little shell script:

A silly little Unix shell thing that I've vaguely wanted for ages but only put together today. See if you can guess what it's for:

#!/bin/sh
trap 'exec $SHELL' 2
"$@"
exec $SHELL

(The use of this is pretty obscure and is due to my eccentric X environment.)

The actual version I now use wound up slightly more complicated, and I call it 'thenshell'. What it does (as suggested by the name) is to run something and then after the thing either exits or is Ctrl-C'd, it runs a shell. This is pointless in normal circumstances but becomes very relevant if you use this as the command for a terminal window to run instead of your shell, as in 'xterm -e thenshell <something>'.

Over time, I've accumulated a number of things I want to run in my eccentric desktop environment, such as my system for opening URLs from remote machines and my alert monitoring. But some of the time I want to stop and restart these (or I need to restart them), and in general I want to notice if they produce some output, so I've been running them in terminal windows. Up until now I've had to manually start a terminal and run these programs each time I restart my desktop environment, which is annoying and sometimes I forget to do it for something. My new 'thenshell' shell script handles this; it runs whatever and then if it's interrupted or exits, starts a shell so I can see things, restart the program, or whatever.

Thenshell isn't quite a perfect duplicate of the manual version. One obvious limitation is that it doesn't put the command into the shell's command history, so I can't just cursor-up and hit return to restart it. But this is a small thing compared to having all of these things automatically started for me.

(Actually, I think I might be able to get this into a version of thenshell that knows exactly how my shell and my environment handle history, but it would be more than a bit of a hack. I may still try it, partly because it would be nifty.)

I recently read z3bra's 2014 Avoid workspaces (via) which starts out with the tag "Virtual desktops considered harmful". At one level I don't disagree with z3bra's conclusion that you probably want flexible groupings of windows, and I also (mostly) don't use single-purpose virtual screens. But I do it another way, which I think is easier than z3bra's (2014) approach.

I've written about how I use virtual screens in my desktop environment, although a bit of that is now out of date. The short summary is that I mostly have a main virtual screen and then 'overflow' virtual screens where I move to if I need to do something else without cleaning up the main virtual screen (as a system administrator, I can be quite interrupt-driven or working on more than one thing at once). This sounds a lot like window groups, and I'm sure I could do it with them in another window manager. The advantage to me of fvwm's virtual screens is that it's very easy to move windows from one to another.

If I start a window in one virtual screen, for what I think is going to be one purpose, and it turns out that I need it for another purpose too, on another virtual screen, I don't have to fiddle around with, say, adding or changing its tags. Instead I can simply grab it and move it to the new virtual screen (or, for terminal windows and some others, iconify them on one screen, switch screens, and deiconify them). This makes it fast, fluid, and convenient to shuffle things around, especially for windows where I can do this by iconifying and deiconify them.

This is somewhat specific to (fvwm's idea of) virtual screens, where the screens have a spatial relationship to each other and you can grab windows and move them around to change their virtual screen (either directly or through FvwmPager). In particular, I don't have to switch between virtual screens to drag a window on to my current one; I can grab it in a couple of ways and yank it to where I am now.

In other words, it's the direct manipulation of window grouping that makes this work so nicely. Unfortunately I'm not sure how to get direct manipulation of currently not visible windows without something like virtual screens or virtual desktops. You could have a 'show all windows' feature, but that still requires bouncing between that all-windows view (to tag in new windows) and your regular view. Maybe that would work fluidly enough, especially with today's fast graphics.

Over on the Fediverse, Simon Tatham had a comment about (using) cloud identity providers that's sparked some discussion. Yesterday I wrote about the facets of identity providers. Today I'm sort of writing about why you might not want to run your own identity provider, despite the hazards of depending on the security of some outside third party. I'll do this by talking about what I see as being involved in the whole thing.

The hardcore option is to rely on no outside services at all, not even for multi-factor authentication. This pretty much reduces your choices for MFA down to TOTP and perhaps WebAuthn, either with devices or with hardware keys. And of course you're going to have to manage all aspects of your MFA yourself. I'm not sure if there's capable open source software here that will let people enroll multiple second factors, handle invalidating one, and so on.

One facet of being an identity provider is managing identities. There's a wide variety of ways to do this; there's Unix accounts, LDAP databases, and so on. But you need a central system for it, one that's flexible enough to cope with with real world, and that system is load bearing and security sensitive. You will need to keep it secure and you'll want to keep logs and audit records, and also backups so you can restore things if it explodes (or go all the way to redundant systems for this). If the identity service holds what's considered 'personal information' in various jurisdictions, you'll need to worry about an attacker being able to bulk-extract that information, and you'll need to build enough audit trails so you can tell to what extent that happened. Your identity system will need to be connected to other systems in your organization so it knows when people appear and disappear and can react appropriately; this can be complex and may require downstream integrations with other systems (either yours or third parties) to push updates to them.

Obviously you have to handle primary authentication yourself (usually through passwords). This requires you to build and operate a secure password store as well as a way of using it for authentication, either through existing technology like LDAP or something else (this may or may not be integrated with your identity service software, as passwords are often considered part of the identity). Like the identity service but more so, this system will need logs and audit trails so you can find out when and how people authenticated to it. The log and audit information emitted by open source software may not always meet your needs, in which case you may wind up doing some hacks. Depending on how exposed this primary authentication service is, it may need its own ratelimiting and alerting on signs of potential compromised accounts or (brute force) attacks. You will also definitely want to consider reacting in some way to accounts that pass primary authentication but then fail second-factor authentication.

Finally, you will need to operate the 'identity provider' portion of things, which will probably do either or both of OIDC and SAML (but maybe you (also) need Kerberos, or Active Directory, or other things). You will have to obtain the software for this, keep it up to date, worry about its security and the security of the system or systems it runs on, make sure it has logs and audit trails that you capture, and ideally make sure it has ratelimits and other things that monitor for and react to signs of attacks, because it's likely to be a fairly exposed system.

If you're a sufficiently big organization, some or all of these services probably need to be redundant, running on multiple servers (perhaps in multiple locations) so the failure of a single server doesn't lock you out of everything. In general, all of these expose you to all of the complexities of running your own servers and services, and each and all of them are load bearing and highly security sensitive, which probably means that you should be actively paying attention to them more or less all of the time.

If you're lucky you can find suitable all-in-one software that will handle all the facets you need (identity, primary authentication, OIDC/SAML/etc IdP, and perhaps MFA authentication) in a way that works for you and your organization. If not, you're going to have to integrate various different pieces of software, possibly leaving you with quite a custom tangle (this is our situation). The all in one software generally seems to have a reputation of being pretty complex to set up and operate, which is not surprising given how much ground it needs to cover (and how many protocols it may need to support to interoperate with other systems that want to either push data to it or pull data and authentication from it). As an all-consuming owner of identity and authentication, my impression is that such software is also something that's hard to add to an existing environment after the fact and hard to swap out for anything else.

(So when you pick an all in one open source software for this, you really have to hope that it stays good, reliable software for many years to come. This may mean you need to build up a lot of expertise before you commit so that you really understand your choices, and perhaps even do pilot projects to 'kick the tires' on candidate software. The modular DIY approach is more work but it's potentially easier to swap out the pieces as you learn more and your needs change.)

The obvious advantage of a good cloud identity provider is that they've already built all of these systems and they have the expertise and infrastructure to operate them well. Much like other cloud services, you can treat them as a (reliable) black box that just works. Because the cloud identity provider works at a much bigger scale than you do, they can also afford to invest a lot more into security and monitoring, and they have a lot more visibility into how attackers work and so on. In many organizations, especially smaller ones, looking after your own identity provider is a part time job for a small handful of technical people. In a cloud identity provider, it is the full time job of a bunch of developers, operations, and security specialists.

(This is much like the situation with email (also). The scale at which cloud providers operates dwarfs what you can manage. However, your identity provider is probably more security sensitive and the quality difference between doing it yourself and using a cloud identity provider may not be as large as it is with email.)

Over on the Fediverse, Simon Tatham had a comment about cloud identity providers, and this sparked some thoughts of my own. One of my thoughts is that in today's world, a sufficiently large organization may have a number of facets to its identity provider situation (which is certainly the case for my institution). Breaking up identity provision into multiple facets can leave it not clear if and to what extend you could be said to be using a 'cloud identity provider'.

First off, you may outsource 'multi-factor authentication', which is to say your additional factor, to a specialist SaaS provider who can handle the complexities of modern MFA options, such as phone apps for push-based authentication approval. This SaaS provider can turn off your ability to authenticate, but they probably can't authenticate as a person all by themselves because you 'own' the first factor authentication. Well, unless you have situations where people only authenticate via their additional factor and so your password or other first factor authentication is bypassed.

Next is the potential distinction between an identity provider and an authentication source. The identity provider implements things like OIDC and SAML, and you may have to use a big one in order to get MFA support for things like IMAP. However, the identity provider can delegate authenticating people to something else you run using some technology (which might be OIDC or SAML but also could be something else). In some cases this delegation can be quite visible to people authenticating; they will show up to the cloud identity provider, enter their email address, and wind up on your web-based single sign on system. You can even have multiple identity providers all working from the same authentication source. The obvious exposure here is that a compromised identity provider can manufacture attested identities that never passed through your authentication source.

Along with authentication, someone needs to be (or at least should be) the 'system of record' as to what people actually exist within your organization, what relevant information you know about them, and so on. Your outsourced MFA SaaS and your (cloud) identity providers will probably have their own copies of this data where you push updates to them. Depending on how systems consume the IdP information and what other data sources they check (eg, if they check back in with your system of record), a compromised identity provider could invent new people in your organization out of thin air, or alter the attributes of existing people.

(Small IdP systems often delegate both password validation and knowing who exists and what attributes they have to other systems, like LDAP servers. One practical difference is whether the identity provider system asks you for the password or whether it sends you to something else for that.)

If you have no in-house authentication or 'who exists' identity system and you've offloaded all of these to some external provider (or several external providers that you keep in sync somehow), you're clearly at the mercy of that cloud identity provider. Otherwise, it's less clear and a lot more situational as to when you could be said to be using a cloud identity provider and thus how exposed you are. I think one useful line to look at is to ask whether a particular identity provider is used by third party services or if it's only used to for that provider's own services. Or to put it in concrete terms, as an example, do you use Github identities only as part of using Github, or do you authenticate other things through your Github identities?

(With that said, the blast radius of just a Github (identity) compromise might be substantial, or similarly for Google, Microsoft, or whatever large provider of lots of different services that you use.)

OpenTelemetry ('OTel') is one of the current in things in the broad metrics and monitoring space. As I understand it, it's fundamentally a set of standards (ie, specifications) for how things can emit metrics, logs, and traces; the intended purpose is (presumably) so that people writing programs can stop having to decide if they expose Prometheus format metrics, or Influx format metrics, or statsd format metrics, or so on. They expose one standard format, OpenTelemetry, and then everything (theoretically) can consume it. All of this has come on to my radar because Prometheus can increasingly ingest OpenTelemetry format metrics and we make significant use of Prometheus.

If OpenTelemetry is just another metrics format that things will produce and Prometheus will consume just as it consumes Prometheus format metrics today, that seems perfectly okay. I'm pretty indifferent to the metrics formats involved, presuming that they're straightforward to generate and I never have to drop everything and convert all of our things that generate (Prometheus format) metrics to generating OpenTelemetry metrics. This would be especially hard because OpenTelemtry seems to require either Protobuf or (complex) JSON, while the Prometheus metrics format is simple text.

However, this is where I start getting twitchy. OpenTelemetry certainly gives off the air of being a complex ecosystem, and on top of that it also seems to be an application focused ecosystem, not a system focused one. I don't think that metrics are as highly regarded in application focused ecosystems as logs and traces are, while we care a lot about metrics and not very much about the others, at least in an OpenTelemtry context. To the extent that OpenTelemtry diverts people away from producing simple, easy to use and consume metrics, I'm going to wind up being unhappy with it. If what 'OpenTelemtry support' turns out to mean in practice is that more and more things have minimal metrics but lots of logs and traces, that will be a loss for us.

Or to put it another way, I worry that an application focused OpenTelemetry will pull the air away from the metrics focused things that I care about. I don't know how realistic this worry is. Hopefully it's not.

(Partly I'm underinformed about OpenTelemetry because, as mentioned I often feel disconnected from the mainstream of 'observability', so I don't particularly try to keep up with it.)

We mostly run systems, not applications, due to our generally different system administration environment. Many organizations instead run applications. Although these applications may be hosted on some number of systems, the organizations don't care about the systems, not really; they care about how the applications work (and the systems only potentially matter if the applications have problems). It's my increasing feeling that this has created differences in the general field of monitoring such systems (as well as alerting), which is a potential issue for us because most of the attention is focused on the application area of things.

When you run your own applications, you get to give them all of the 'three pillars of observability' (metrics, traces, and logs, see here for example). In fact, emitting logs is sort of the default state of affairs for applications, and you may have to go out of your way to add metrics (my understanding is that traces can be easier). Some people even process logs to generate metrics, something that's supported by various log ingestion pipelines these days. And generally you can send your monitoring output to wherever you want, in whatever format you want, and often you can do things like structuring them.

When what you run is systems, life is a lot different. Your typical Unix system will most easily provide low level metrics about things. To the extent that the kernel and standard applications emit logs, these logs come in a variety of formats that are generally beyond your control and are generally emitted to only a few places, and the overall logs of what's happening on the system are often extremely incomplete (partly because 'what's happening on the system' is a very high volume thing). You can basically forget about having traces. In the modern Linux world of eBPF it's possible to do better if you try hard, but you'll probably be building custom tooling for your extra logs and traces so they'd better be sufficiently important (and you need the relevant expertise, which may include reading kernel and program source code).

The result is that for people like us who run systems, our first stop for monitoring is metrics and they're what we care most about; our overall unstructured logs are at best a secondary thing, and tracing some form of activity is likely to be something done only to troubleshoot problems. Meanwhile, my strong impression is that application people focus on logs and if they have them, traces, with metrics only a distant and much less important third (especially in the actual applications, since metrics can be produced by third party tools from their logs).

(This is part of why I'm so relatively indifferent to smart log searching systems. Our central syslog server is less about searching logs and much more about preserving them in one place for investigations.)

At this point we've been running Grafana for quite some time (since late 2018), and (Grafana) Loki for rather less time and on a more ad-hoc and experimental basis. However, over time both have become 'legacy software' here, by which I mean that we (I) have frozen their versions and don't update them any more, and we (I) mostly or entirely don't touch their configurations any more (including, with Grafana, building or changing dashboards).

We froze our Grafana version due to backward compatibility issues. With Loki I could say that I ran out of enthusiasm for going through updates, but part of it was that Loki explicitly deprecated 'promtail' in favour of a more complex solution ('Alloy') that seemed to mostly neglect the one promtail feature we seriously cared about, namely reading logs from the systemd/journald complex. Another factor was it became increasingly obvious that Loki was not intended for our simple setup and future versions of Loki might well work even worse in it than our current version does.

Part of Grafana and Loki going without updates and becoming 'legacy' is that any future changes in them would be big changes. If we ever have to update our Grafana version, we'll likely have to rebuild a significant number of our current dashboards, because they use panels that aren't supported any more and the replacements have a quite different look and effect, requiring substantial dashboard changes for the dashboards to stay decently usable. With Loki, if the current version stopped working I'd probably either discard the idea entirely (which would make me a bit sad, as I've done useful things through Loki) or switch to something else that had similar functionality. Trying to navigate the rapids of updating to a current Loki is probably roughly as much work (and has roughly as much chance of requiring me to restart our log collection from scratch) as moving to another project.

(People keep mentioning VictoriaLogs (and I know people have had good experiences with it), but my motivation for touching any part of our Loki environment is very low. It works, it hasn't eaten the server it's on and shows no sign of doing that any time soon, and I'm disinclined to do any more work with smart log collection until a clear need shows up. Our canonical source of history for logs continues to be our central syslog server.)

Suppose, not entirely hypothetically, that you're going to need a 'VPN' system that authenticates through OIDC. What platforms do you need this VPN system to support? In our environment, the answer is that we have five platforms that we need to care about, and they're the obvious four plus one more: Windows, macOS, iOS, Android, and Linux.

We need to cover these five platforms because people here use our services from all of those platforms. Both Windows and macOS are popular on laptops (and desktops, which still linger around), and there's enough people who use Linux to be something we need to care about. On mobile devices (phones and tablets), obviously iOS and Android are the two big options, with people using either or both. We don't usually worry about the versions of Windows and macOS and suggest that people to stick to supported ones, but that may need to change with Windows 10.

Needing to support mobile devices unquestionably narrows our options for what we can use, at least in theory, because there are certain sorts of things you can semi-reasonably do on Linux, macOS, and Windows that are infeasible to do (at least for us) on mobile devices. But we have to support access to various of our services even on iOS and Android, which constrains us to certain sorts of solutions, and ideally ones that can deal with network interruptions (which are quite common on mobile devices in Toronto, as anyone who takes our subways is familiar with).

(And obviously it's easier for open source systems to support Linux, macOS, and Windows than it is for them to extend this support to Android and especially iOS. This extends to us patching and rebuilding them for local needs; with various modern languages, we can produce Windows or macOS binaries from modified open source projects. Not so much for mobile devices.)

In an ideal world it would be easy to find out the support matrix of platforms (and features) for any given project. In this world, the information can sometimes be obscure, especially for what features are supported on what platforms. One of my resolutions to myself is that when I find interesting projects but they seem to have platform limitations, I should note down where in their documentation they discuss this, so I can find it later to see if things have changed (or to discuss with people why certain projects might be troublesome).

In this modern age, more and more people are facing more and more pressure to have pervasive Multi-Factor Authentication, with every authentication your people perform protected by MFA in some way. I've come to feel that there are two broad approaches to achieving this and one of them is more realistic than the other, although it's also less appealing in some ways and less neat (and arguably less secure).

The 'proper' way to protect everything with MFA is to separately and individually add MFA to everything you have that does authentication. Ideally you will have a central 'single sign on' system, perhaps using OIDC, and certainly your people will want you to have only one form of MFA even if it's not all run through your SSO. What this implies is that you need to add MFA to every service and protocol you have, which ranges from generally easy (websites) through being annoying to people or requiring odd things (SSH) to almost impossible at the moment (IMAP, authenticated SMTP, and POP3). If you opt to set it up with no exemptions for internal access, this approach to MFA insures that absolutely everything is MFA protected without any holes through which an un-MFA'd authentication can be done.

The other way is to create some form of MFA-protected network access (a VPN, a mesh network, a MFA-authenticated SSH jumphost, there are many options) and then restrict all non-MFA access to coming through this MFA-protected network access. For services where it's easy enough, you might support additional MFA authenticated access from outside your special network. For other services where MFA isn't easy or isn't feasible, they're only accessible from the MFA-protected environment and a necessary step for getting access to them is to bring up your MFA-protected connection. This approach to MFA has the obvious problem that if someone gets access to your MFA-protected network, they have non-MFA access to everything else, and the not as obvious problem that attackers might be able to MFA as one person to the network access and then do non-MFA authentication as another person on your systems and services.

The proper way is quite appealing to system administrators. It gives us an array of interesting challenges to solve, neat technology to poke at, and appealingly strong security guarantees. Unfortunately the proper way has two downsides; there's essentially no chance of it covering your IMAP and authenticated SMTP services any time soon (unless you're willing to accept some significant restrictions), and it requires your people to learn and use a bewildering variety of special purpose, one-off interfaces and sometimes software (and when it needs software, there may be restrictions on what platforms the software is readily available on). Although it's less neat and less nominally secure, the practical advantage of the MFA protected network access approach is that it's universal and it's one single thing for people to deal with (and by extension, as long as the network system itself covers all platforms you care about, your services are fully accessible from all platforms).

(In practice the MFA protected network approach will probably be two things for people to deal with, not one, since if you have websites the natural way to protect them is with OIDC (or if you have to, SAML) through your single sign on system. Hopefully your SSO system is also what's being used for the MFA network access, so people only have to sign on to it once a day or whatever.)

In a comment on my entry on how WireGuard mesh networks need a provisioning system, vcarceler pointed me to Innernet (also), an interesting but opinionated provisioning system for WireGuard. However, two bits of it combined made me twitch a bit; Innernet only allows you to provision a given node once, and once a node is assigned an internal IP, that IP is never reused. This lack of support for re-provisioning machines would be a problem for us and we'd likely have to do something about it, one way or another. Nor is this an issue unique to Innernet, as a number of mesh network systems have it.

Our important servers have fixed, durable identities, and in practice these identities are both DNS names and IP addresses (we have some generic machines, but they aren't as important). We also regularly re-provision these servers, which is to say that we reinstall them from scratch, usually on new hardware. In the usual course of events this happens roughly every two years or every four years, depending on whether we're upgrading the machine for every Ubuntu LTS release or every other one. Over time this is a lot of re-provisionings, and we need the re-provisioned servers to keep their 'identity' when this happens.

We especially need to be able to rebuild a dead server as an identical replacement if its hardware completely breaks and eats its system disks. We're already in a crisis, we don't want to have a worse crisis because other things need to be updated because we can't exactly replace the server but instead have to build a new server that fills the same role, or will once DNS is updated, configurations are updated, etc etc.

This is relatively straightforward for regular Linux servers with regular networking; there's the issue of SSH host keys, but there's several solutions. But obviously there's a problem if the server is also a mesh network node and the mesh network system will not let it be re-provisioned under the same name or the same internal IP address. Accepting this limitation would make it difficult to use the mesh network for some things, especially things where we don't want to depend on DNS working (for example, sending system logs via syslog). Working around the limitation requires reverse engineering where the mesh network system stores local state and hopefully being able to save a copy elsewhere and restore it; among other things, this has implications for the mesh network system's security model.

For us, it would be better if mesh networking systems explicitly allowed this re-provisioning. They could make it a non-default setting that took explicit manual action on the part of the network administrator (and possibly required nodes to cooperate and extend more trust than normal to the central provisioning system). Or a system like Innernet could have a separate class of IP addresses, call them 'service addresses', that could be assigned and reassigned to nodes by administrators. A node would always have its unique identity but could also be assigned one or more service addresses.

(Of course our other option is to not use a mesh network system that imposes this restriction, even if it would otherwise make our lives easier. Unless we really need the system for some other reason or its local state management is explicitly documented, this is our more likely choice.)

PS: The other problem with permanently 'consuming' IP addresses as machines are re-provisioned is that you run out of them sooner or later unless you use gigantic network blocks that are many times larger than the number of servers you'll ever have (well, in IPv4, but we're not going to switch to IPv6 just to enable a mesh network provisioning system).