Back when I wrote about using tc to limit the outgoing bandwidth of a web server, I expressed a wish:

For our purposes it would be nice to do something sophisticated to aggregate all HTTPS requests from a single IP address together in a single 'flow' for tc-sfq(8). In theory this is possible with tc-flow(8) but in practice I can't figure out a command line that works right (despite consulting eg tc-sfb(8)'s example).

The good news is that I managed to figure out a syntax that tc would accept. The bad news is that I don't know if it does what I want, because I'm not sure how you see what connections are assigned to what flows.

I'll put forward two variations of tc-flow(8) commands. To start with, I'll set the stage. We start with our bandwidth limited class that will have a tc-sfq(8) qdisc below it, straight from the first entry:

tc class add dev eno1 parent 1: classid 1:10 htb rate 400mbit ceil 500mbit prio 10

Our first option is to do exactly what I said above, making the flow per-IP instead of per four-tuple, and attach this flow classification to our bandwidth limited qdisc.

tc filter add dev eno1 protocol ip parent 1:10 handle 20 flow hash keys src,dst,proto,proto-src divisor 4096

(The 'handle <something>' is apparently critical although I don't know why. I got this wisdom from here after extensive online searches.)

This theoretically makes flows be based on the source IP, destination IP, source port, and the protocol. In our case, for a HTTPS web server with us dealing with outgoing traffic, the source, protocol, and source port are all the same, so this only varies the flow by the destination IP. This is what we want to aggregate all connections from the same IP. Some sources will tell you to use 'perturb N' on this. We don't want to do this because we want flows to stay sticky and not get re-sorted every so often, and we want to set a very large divisor so that we maintain a lot of distinct flows to go with our (initially) large number of connections.

(The tc-sfb(8) manual page has an example like this, basing flows purely on the destination IP, but it's attached to the root qdisc.)

The other option is to be even more aggressive and theoretically group flows by the /24 of the destination IP address, and nothing else (since we're already theoretically only dealing with HTTPS responses sent by our web server, with the protocol, source IP, and source port all constant). This is done with the other sort of flow filter, a 'map' filter instead of a 'hash':

tc filter add dev eno1 protocol ip parent 1:10 handle 20 flow map key dst rshift 8 divisor 4096

If I'm understanding tc-flow(8) correctly, this takes the destination IP and right shifts it 8 bits (then divides the result by 4096, once again to try to have as many distinct flows as we could in theory have simultaneous connections). Dropping the right octet of a full IP address effectively reduces it to its /24. I think we could also use 'dst and 0xffffff00' to get basically the same effect, but maybe that would get more flow collisions when the dust settled.

I believe that we also need to tell our tc-sfq(8) queue how many flows it's supposed to have, matched with the number we used above:

tc qdisc add dev eno1 parent 1:10 handle 10: sfq flows 4096 divisor 4096

Unfortunately, I don't know if all of this works because I don't know of any way to see what flow a given connection is classified into (either before or after tc-sfq(8) gets at it). It could be that my tc-flow(8) usage isn't actually assigning flows the way I want (especially in the second case). It could also be that I need to put my flow filter somewhere else in the tc class hierarchy, or maybe somehow attach it directly to the tc-sfq(8) qdisc.

(Given the example in tc-sfb(8), where the filter is attached directly to the sfb qdisc, possibly I need to attach the flow filters to the sfq qdisc, with 'parent 10:' instead of 'parent 1:10'. This appears to be accepted by tc, although who knows if it's working.)

PS: Based on what 'iftop' is telling me, I'm relatively sure that my flow filter isn't actually working the way I want (with a flow filter either in this version or directly attached to the sfq qdisc). But it's hard to be sure.

I recently shared a war story about how we had load problems on a new web server but not an older one because of a missing firewall rule. In a comment, Aristotle Pagaltzis asked how we'd realized that we had a missing firewall rule. This is a good question, because the sequence of events involves a certain amount of luck and coincidence. So here's that story, which may be a useful example of system administration in action in practice (ie, it's kind of messy).

When we first deployed our new web server for a highly in-demand data set, Apache wasn't configured for very many concurrent connections and was immediately overloaded with HTTP requests, but we mostly shrugged. Because this upset our monitoring system and I did want to monitor the web server at least a bit, I turned up the Apache connection limits to an absurd number. This was mostly enough, and the web server's outgoing bandwidth jumped up to 1G wire rates, which at the time I thought was fine. Soon after that, an apparently unrelated machine began to have NFS performance problems, and after some work we realized that it was on the same 1G switch as the highly active web server and its network traffic was getting crowded out.

In a rush to limit the web server's bandwidth usage so the much more important other machine would stop having NFS problems, I hastily added some tc based bandwidth limits by hand, and by this I mean that I typed 'tc' commands in a shell session. A few days later, we worked out how to make mod_qos based limits work in Apache itself. One of the limits we imposed and fiddled with was a limit on the number of concurrent connections from a single IP address, which had a much bigger effect than I expected. The Apache error log showed that some IPs were hitting it quite a lot, and our metrics system said the number of concurrent Apache requests dropped dramatically afterward (to about half what they'd been before).

After the dust settled from the immediate crisis, we needed to decide if we were going to make the tc-based limits a permanent part of the machine's configuration (making it the first machine officially using tc in production, and we'd have to write something to install them on system startup) or if we'd rely entirely on mod_qos. While considering the tradeoffs involved, I remembered that we had a general purpose 'block brute force things' system on our perimeter firewall, so we should probably make it apply to HTTP and HTTPS requests to our new web server for extra insurance. Immediately after I did this, our metrics system showed another major drop in concurrent Apache connections (shrinking by half again).

(The perimeter firewall's system has a list of IP addresses that it applies the HTTP and HTTPS limits to, and the new web server, with a new IP address, wasn't in the list (until I added it).)

That major drop from the firewall rule was what sparked my realization of what was different that would explain why our main web server mostly hadn't been overwhelmed by this traffic, because that was the only thing out of all of our rate limiting changes that our main web server had in place. Our main web server had no tc limits and we'd turned off mod_qos several years ago and never revisited that change.

PS: We decided to keep both the Apache mod_qos limits and the tc based limits, partly for extra insurance. We've decided that we really don't want this particular web server to run over the bandwidth limits, so having two mechanisms that limits it means they'd both have to fail.

(We're not yet worried enough to look into FreeBSD PF's features for bandwidth limits, partly because if we got it wrong on the firewall, we could affect a lot more than this machine.)

(One comment.)

When I wrote about how early SunOS did diskless workstations, I got a good question in a comment:

Obviously this is slow, but having an indeterminate number of users sharing a circa 1982 mechanical hard drive sounds like a problematic amount of slow. Was this ever truly worth doing? I know hard drives were wildly expensive back then, but surely the work slowdown from having multiple users sharing a single hard drive in this fashion would mean the ROI for those hard drives would seem obvious?

One answer is that diskless workstations were not infrequently used in situations where there was no 'ROI' as such, for example for use by university graduate students (generally not in dedicated offices, unless you were very lucky, but instead in shared in terminal rooms). But in my view, a deeper answer is that there are two usage models of diskless workstations.

In one model, a diskless workstation was an inferior substitute for a workstation with a local disk. It had to do its disk IO over a slow shared 10MBit network connection to a server with mechanical HDDs that were used by multiple people (on all of those diskless workstations), which was obviously much worse performance than a local disk. But you saved the cost of the local disk, and perhaps you bought a low end workstation model (such as the basic Sun 3/50 instead of the better, faster 3/60) because they weren't going to be fast anyway.

In the other model, a diskless workstation was a superior replacement for a serial terminal, in much the same way that X terminals were later. Instead of a single text 'window' and no local computing, you gave people something with multiple windows, graphical capabilities, and some degree of local computing that could be faster than an (over)loaded central server. In the process you might save money on the central server, since it didn't need as much compute capacity as it would if everyone was directly logging in to it (although diskless workstations cost a lot more than serial terminals, so you probably weren't saving money overall).

(The other advantage of the diskless workstation model over the serial terminal model was it was more amenable to incremental upgrades, since you could buy better workstations (perhaps with disks) one by one. It was a "personal computer" model instead of a "terminal" model.)

These two models lead to different calculations of costs and benefits. In the first model, you're losing productivity but saving money on hardware, and the question is how much does the lost productivity actually cost you. You're probably going to give diskless workstations to people who don't have highly valuable productivity. Diskless workstations are a downgrade and workstations with disks will become a status symbol, a sign that you're important enough to call for the extra expense.

In the second model you're gaining productivity at the cost of spending more on hardware. the question is how much extra productivity do people gain compared to the extra cost of diskless workstations over serial terminals (possibly factoring in a cheaper server, fewer serial lines and serial port boards in the server, and so on). In some cases the productivity gains may be significant at relatively modest extra cost. Diskless workstations are an upgrade and a status symbol (compared to serial terminals).

Of course these models cross over somewhere, so you get to look at the relative payoffs and costs of serial terminals, diskless workstations, and workstations with local disks for different groups of people with different productivity payoffs. Once NFS and other shared, writable filesystems entered the picture, things got more complicated because even your 'local disk' workstations might be NFS mounting home directories, shared work areas, and so on, both for collaboration and so that people weren't tied to specific physical workstations.

(When X terminals arrived they added another section to the spectrum, with graphics but without local computation. This could make sense in a variety of ways; some people benefited from graphics but not local computation, and some people needed to do most or all of their compute on the same machine as their data (on HDDs), instead of hauling it back and forth over shared 10MBit networks with the network filesystem protocol of your choice.)

There's likely also a practical commercial aspect to diskless workstations. My impression is that Sun and other early Unix workstation vendors were relatively desperate to get their machines into places (Unix was a new thing, after all), so in the grand tradition of such things they created a low cost entry level version of their product to get their foot in the door, even if it wasn't all that great. If you could initially sell a company some base configuration diskless workstations and a server to go with them for cheap, maybe you could turn that into a later sale of better, more expensive hardware once the company got a taste of Unix.

(Sun would later continue this tradition by selling entry level hardware without hardware floating point.)

(One comment.)

Today, for reasons outside the scope of this entry, we needed to do unscheduled reboots on a number of Ubuntu 26.04 servers that people log in to and use. As is our usual process, we didn't reboot these on the spot; instead we ran 'shutdown -r +NN "<a message about the situation>"' so that people would have a little bit of warning because the impending shutdown would be periodically announced (by systemd, because this is systemd-based these days). Then, to our unpleasant surprise, we discovered that no announcements were happening. Shortly afterward we discovered that the venerable 'wall' program wasn't making announcements either.

Surprisingly, these turn out to be two separate issues. The wall issue is because starting in Debian 13 ('Trixie') and Ubuntu 25.10, the Debian and Ubuntu systemd is built without support for /var/run/utmp (aka /run/utmp), the traditional file recording who is logged in where; wall (which comes from the 'bsdutils' package) only looks in the utmp file. If there's no utmp file, wall is never going to do anything. If you need a wall equivalent, you'll need to write a script that gets the list of active user sessions with ptys and writes a message to them itself.

(For Debian Trixie dropping support for utmp, see eg this debian-devel thread. Apparently one reason for the change is that the utmp format has Y2038 problems. A replacement is available through the wtmpdb package and project, which also gives you a working 'last' command. You have to hook it up in your PAM configuration, and the Ubuntu 26.04 OpenSSH is built without wtmpdb support, so I believe you're going to be missing some information.)

The issue with shutdown not broadcasting messages appears to be because in Ubuntu 26.04 (with systemd 259.5), systemd's logind doesn't know what ttys people's SSH logins are using. You can see this with 'loginctl' or 'loginctl -j', which will have no TTY information for all SSH logins (although if you log in on the console, it will have that). This isn't the case in Ubuntu 24.04 (with systemd 255.4) or Fedora 43 (with systemd 258.9), although both versions are built with UTMP support (in theory this shouldn't matter, since logind's tty tracking is a separate thing). Logind's announcements of impending shutdowns only go to TTYs that it knows about, so since it doesn't know about any SSH login ptys, none of them get any announcements.

Update: Now that I pay attention, Fedora 43's systemd is older than Ubuntu 26.04's. However, Fedora 44 has systemd 259.7 (with UTMP enabled) and its 'loginctl' doesn't have this problem.

If you're using 'who' or 'w' on Ubuntu 26.04 (or at least the version of 'who' from GNU Coreutils, since the uutils version currently suffers from bug #2152801), you might notice that they do report pty information, at least if you have AppArmor disabled (as we do):

; who
cks      sshd pts/0   Jul 20 21:47 (...)
; lsb_release -r
Release:        26.04

This is because while the GNU Coreutils version of 'who' talks to systemd to try to get this information (through a set of systemd library APIs), if there's no TTY information for a session it will also look through /dev/pts to try to find a likely candidate. This works often enough that 'who' typically shows information for most interactive SSH sessions with a pty. The 'w' program, which comes from procps, has a similar fallback if systemd and utmp are both not reporting the tty. Since 'who' and 'w' use different approaches, on Ubuntu 26.04 one may report a tty that the other doesn't.

(Apparently the default Ubuntu 26.04 AppArmor profiles block access by 'who' to /run/systemd/sessions, which the systemd library API uses under the covers to get session information. Amusingly, this only affects 'who', not 'w', as 'w' has no specific AppArmor profile.)

(4 comments.)

I mentioned recently that we'd moved a very popular data set from our main web server to a new one that only handled that data set. On the new web server, we found it necessary to set an absurdly high Apache connection limit of 4,000 concurrent requests, because Apache could run out otherwise (and could even run out at 4,000, it was just less often).

Our main web server has a much lower concurrent connection limit but it didn't experience these problems (and not because we'd imposed connection limits in Apache; we'd turned that off in December of 2022). But as we (I) wrangled with the new web server to get it to stop running out of connections and so on, and even after I got mod_qos working on it, I never paused to ask myself why the new web server had so many problems with this when the main web server had run for years without explosions (or at least very infrequent ones).

(Part of this was because the main web server had started exploding sometimes; that was why we'd moved this data set to its own server. But even those explosions had been less severe than what I was seeing.)

The answer is that for years, our perimeter firewall has had per-IP connection rate limits for HTTP and HTTPS connection to our main web server (among other per-IP connection rate limits, for example for SSH connections). These predate the modern popularity of this data set and were added to stop other abuse, but it turns out that a lot of the connection volume for this data set was coming from a few IPs that were opening up a ton of rapid-fire and often simultaneous connections. Once we applied per-IP limits on both the number of simultaneous connections you could have (in mod_qos) and the rate at which you could make connections (in the firewall), the new web server's connection count dropped like a stone (but people still kept pulling data from it as fast as they could).

In retrospect, the change in the web server's behavior when we moved this data set to a new host was a signal. We'd had only occasional problems on the old web server host one (despite it being actively used for other things) and we had constant ones on the new server (dedicated only to the data set). So we could have asked what was different, and then investigated, and then found the perimeter firewall issue. But on the other hand, this is sort of hindsight bias speaking. Such changes in behavior are a signal, but as system administrators we're drowning in signals and we have to sort out what's meaningful and what's either a coincidence or a consequence of something else (for example, a sudden increase in demand for this data set, which would have also explained why we were suddenly seeing problems even on the main web server).

PS: There's some recent evidence that there was a real but temporary shift in demand for this data set, in large part from people (or software) that make extremely inefficient requests. If these people are done now, or have improved their software, that would be nice. Perhaps all of the connection blocking and limited bandwidth have encouraged them to download things only once and then keep local caches.

(2 comments.)

GNU Coreutils are what they sound like; a GNU version of a bunch of basic, core Unix programs such as 'mkdir', 'head', 'chmod', 'cp', 'mv', and so on. For a long time, talking about 'GNU Coreutils' was unnecessary and you could just talk about 'Coreutils'. Then some people decided to rewrite Coreutils in Rust, the uutils coreutils, which still wouldn't be very important for most people except that Canonical decided to make the Rust versions the default in Ubuntu 25.10 and then 26.04.

In theory the Rust coreutils aim for 100% compatibility with GNU Coreutils and anything to the contrary is a bug. In practice, I found an incompatibility almost immediately when testing 26.04 pre-release, Ubuntu bug reports are useless, and I was pretty certain that other people on our systems would run into other issues, so I decided that we would sit out this round of Canonical making 26.04 LTS people mandatory beta-testers of their current passion project.

Canonical doesn't make it easy to switch from Rust uutils to GNU Coreutils, but they do at least make it possible. The magic apt-get command you need is:

apt-get install coreutils-from-gnu coreutils-from-uutils- --allow-remove-essential

(Taken from here.)

If you do this, I suggest that you immediately do 'apt-mark hold coreutils-from-uutils' (you may not want to hold 'coreutils-from-gnu', since there might be bugfix updates to it for some reason, although the real programs are in the 'gnu-coreutils' package).

The reason you might want to do this is, well, let me quote a Fediverse post of mine:

Ubuntu: you can totally continue to use GNU Coreutils in 26.04 LTS.
Also Ubuntu: build-essential depends on the new Rust coreutils.

Yeah, that's not "you can totally continue to use GNU Coreutils", although it sure is tempting to build my own build-essential package with a different dependency.

What this means in practice is that if you install coreutils-from-gnu, the build-essential package is uninstalled if you have it installed, and if you install build-essential later, coreutils-from-gnu is uninstalled and coreutils-from-uutils (the Rust version) is reinstalled. If you're not paying close attention to all of the messages that an 'apt-get' is printing out, you might miss this (especially if the apt-get is happening in the middle of your general install framework). Then you will be surprised, as I was, when it turns out that your 26.04 systems have Rust coreutils despite you theoretically having switched.

Build-essential itself doesn't do much, although installing it is a convenient way to get some core software building tools (especially if you want to build Ubuntu packages, perhaps to make local changes). What really matters is that 'apt-get build-dep' will insist on installing build-essential, and you may want to do 'apt-get build-dep <some package>' for all sorts of reasons. For example, if you're going to build your own Emacs, 'apt-get build-dep emacs' is a convenient way to get most or all of the development packages it's going to want, rather than looking them up and getting each one yourself.

The build-essential dependency is explicit:

$ apt-cache show build-essential
[...]
Depends: libc6-dev | libc-dev, gcc (>= 4:14.2), g++ (>= 4:14.2), make, dpkg-dev (>= 1.22.11), coreutils-from-uutils

Not 'coreutils' (a meta-package that depends on either), not explicitly 'coreutils-from-uutils | coreutils-from-gnu', a direct, specific dependency on the Rust coreutils. This turns out to be a Canonical bodge from September 2025 that's not in the upstream package (via). Since this is an explicit dependency, dealing with it requires things like building your own version of build-essential that has a fixed dependency (perhaps with dgit).

There may be other packages with specific dependencies on 'coreutils-from-uutils', which is why I suggested you explicitly 'apt-mark hold' it. With the package held (or both coreutils-from-* packages held), 'apt-get install <some package>' will abort rather than flip your Coreutils setup around. Then at least you can find out which new package will make you unhappy with Canonical.

("apt-cache rdepends coreutils-from-uutils' doesn't show me anything on our Ubuntu 26.04 LTS machines, but I don't know if that's complete across the entire Ubuntu package set for 26.04.)

I mentioned recently (when I dug into systemd and your user D-Bus session bus) that some of my machines were set up so that they sometimes started another D-Bus session bus daemon for me. After having done some experimentation I can say that this isn't necessary (or really, proper) and I've now stopped doing it. You might wonder how I got myself into this situation, and that's a story of history.

On my primary desktops, I've never used any graphical login manager like gdm, xdm, or so on (which has long been sort of a heresy), and I've never run a standard desktop; instead I have my own window manager environment. This leaves me having to do a lot of things myself as part of starting the X server and my environment that a standard desktop and login environment takes care of for you.

D-Bus started being a thing in Linux before systemd. In those days, starting your user D-Bus session daemon was part of the jobs of your desktop environment, either internally or through files it put into the standard /etc/X11/xinit/xinitrc.d (where your graphical login manager of choice should pick them up, although I'm not sure how that works these days). Since I didn't have a desktop environment and was doing it all myself, I had to research what was normally run on session startup and duplicate it in my own shell scripts, and one of those things was running dbus-launch with the appropriate arguments. In the way I ran it, dbus-launch unconditionally starts a D-Bus session daemon and sets '$DBUS_SESSION_BUS_ADDRESS' to point to it.

This was fine in the pre-systemd days, when my regular console login didn't have a D-Bus session daemon started for it (or set up to be ready to start). Well, it was mostly fine, because the D-Bus session bus address was in /tmp, and things can happen to files in /tmp under various circumstances. But I think it only very rarely went wrong, enough that I didn't really notice.

When systemd started providing a D-Bus setup of its own, the proper official /etc/X11/xinit/xinitrc.d was changed so that it detected this and stepped out of the way and not started another D-Bus daemon (and desktop environments that did it all themselves internally were changed similarly). But my own scripts never had this in and never noticed, so when I logged in on the console systemd would set up the whole D-Bus stuff for me even though I was logging in on the console and then my xinit based scripts would promptly start another D-Bus daemon and override that.

(Well, systemd set up all of this provided that my session was in the right class.)

All of this shows one of the challenges of having your own desktop environment; it's on you to keep up with this sort of stuff, and you're probably not hooked into the information channels for it (as far as I know, the various desktop environment people talk to each other). There are probably other places where my environment has drifted away from how it should be.

PS: It's possible that I'll run into problems with my switch, because there's one potentially important thing that's different between the two approaches. The old approach started the D-Bus session daemon with a fully initialized environment (since I was starting it from my login shell after logging in), while the new one starts it with whatever minimal environment it gets from 'systemd --user'.

(One comment.)

Recently I was peripherally involved in a Fediverse discussion about (C's) argc and argv (the arguments to your main(), the start of a C program). Famously, argv[] is an array of pointers to your program's arguments (including the nominal name of the program), and it's sort of traditional to terminate it with a NULL pointer (although this isn't required by the Single Unix Standard; its execve() specification is silent on this). If you think about it, having both argc and a NULL-terminated argv is redundant, since you could determine one from the other. So me being me, I wondered how far back argc and argv went in Unix (and if argv was NULL terminated from the beginning). The answer turns out to be that they go all the way back to Research Unix V1, which is before C existed, and argv[] wasn't originally NULL terminated.

Update: Tony Finch pointed out that POSIX actually does specifically require argv[] to be NULL terminated (and the NULL not be counted in argc). See the comment for details.

The V1 exec(2) manual page is specific about both sides of the V1 exec() API (which is expressed in assembly language terms, since C wasn't invented yet). Exec() is called with a NULL-terminated array of pointers to the (zero-terminated) argument strings, but the invoked program receives an explicit count of the arguments along with an array of argument pointers, and the array is not listed as NULL-terminated. The V1 kernel source code for sysexec (in u2.s) doesn't appear to put in a final NULL pointer or any other pointer value after the regular argv[] pointers, so your program has to use argc to know when to stop.

The logic of this split between the exec() API and the API to programs is a bit clearer in the C code of the V4 exec() in sys/ken/sys1.c. Exec() needs to count the number of arguments in order to do things like allocate the correct size of argv[] array on the stack of the new program, and having created that count it might as well pass that to the new program as argc. However, if I'm reading the V4 exec() correctly, it adds a final '-1' right after the normal end of the argv[] array:

while(na--) {
  suword(ap=+2, c);
  do
    subyte(c++, *cp);
  while(*cp++);
}
suword(ap+2, -1);

This trailing -1 remains present all the way through the V6 exec() in sys/ken/sys1.c (and I don't know why it was -1 instead of 0; the V6 crt0.s doesn't seem to make any visible check for it).

Finally, in the V7 exece() in sys/sys.1, we get an actual NULL pointer at the end of argv[]. However, this is less of a terminator and more of a separator, because the addition of environment variables in V7 has turned argv[] into two arrays of pointers stacked on top of each other, one for the arguments and one for the user environment (which is also terminated with a NULL, because otherwise there's no way to tell). Based on how the C program startup libc/csu/crt0.s has a loop, I think that it finds the environment by walking the argv[] array to find the separator NULL, although the kernel is still providing argc as well as the argv[] array.

As far as I can tell, both System III and 4.2 BSD continue to add the separator NULL (it's more obvious in the 4.x BSD source, where there's an explicit copy of '0' into the user stack; in System III, it appears that the user stack section is pre-zeroed so the code just bumps the offset). BSD continued doing this at least as late as 4.3 BSD Reno (cf). Based on this repository, it appears that System V Release 2 for the Vax also separated argv[] and the environment with a NULL (cf vax/os/exec.c).

If there were Unix systems that later changed this to not have a separating NULL between argv[] and the environment (and thus not giving argv[] a terminating NULL), I don't know what they are. Instead, I suspect that either some C compilers on early non-Unix systems omitted the NULL at the end of their (made up) argv or that the ANSI C and POSIX people didn't want to explicitly require it.

Update: See above, POSIX does explicitly require NULL termination.

(Now you know why I was looking at exec() in early Unix and came to understand its argv size limit.)

(2 comments.)

When I wrote up how V7 gave us environment variables, I mentioned that up to V6, exec(2) had a limit of 510 bytes of command line arguments (including argv[0], the nominal name of your program). You can see the check in the V6 kernel exec() code in sys/ken/sys1.c (where it returns E2BIG in this case). You might wonder where this limit comes from and why.

When you exec() something, you discard your current process's memory and address space to create create a new one for the new program. Your current (user) memory includes the argv you're passing to exec(), so the kernel has to copy it from your user space into the kernel and then back, temporarily holding it in some sort of kernel memory. In a modern kernel you might dynamically allocate this kernel memory in exec() through the kernel equivalent of malloc(), but the Research Unix kernels were simple and didn't have that sort of thing. Instead, through Research Unix V6, they got their temporary scratch space for exec() by allocating a disk buffer, reusing a facility the kernel already needed. These disk buffers were 512 bytes long, which is more or less where the 510 byte limit on argument size comes from.

(I don't know why it's 510 bytes instead of 512; I've been unable to follow the code closely enough to see if it slips in a use of the last two bytes of the buffer for something else.)

You might innocently think that using a disk buffer just pushes the problem of dynamic allocation of (kernel) memory back one layer, to the disk buffer system. However, early Research Unix kernels are more brute force than that. The V6 kernel has a fixed (and limited) chunk of memory reserved for disk buffers, the buffers array in sys/dmr/bio.c, with its size set by NBUF in sys/param.h. The default NBUF isn't very large, but early Research Unix ran on small systems and had low limits in general (the same param.h sets a limit of 50 processes for the entire system).

This straightforward approach to exec() and disk buffers goes back to at least Research Unix V4 (I haven't looked earlier than that). In V7, the kernel implementation is rather more complex because it needs to handle environment variables too, but it still sort of uses the disk buffer trick. In order to get the extra space without using much extra RAM, V7 uses swap space, writing to it and reading back from it through V7's general disk buffer system (which probably often meant that the disk buffer you wrote to swap is still in RAM when you read it back shortly afterward as part of setting up the new process's memory). So to copy the exec() and exece() arguments, V7 allocates a disk buffer in swap space, copies from user space to the disk buffer until it fills up, flushes and releases the disk buffer, gets a new disk buffer for the new block of swap, and does it all over again.

(As an extra complication, V7 didn't have page based swapping, it only swapped whole programs. So during an execve(), V7 allocated swap space for a NCARGS sized 'program' and then used as much of it as necessary, one disk buffer at a time. If V7 couldn't allocate the necessary swap space during exece(), it paniced.)

PS: If you look at the V6 code for exec() carefully, you'll see one spot where it does 'suword(ap=+2, c);', which looks odd and wrong. That's because in V6 C, '=+' was how you wrote in-place arithmetic, instead of '+=' in V7 and the C we know today.

These days, a lot of things want you to have a (user) D-Bus session bus (to go with the system-wide one), which will listen for connections on some socket (the bus address). On a systemd based system, the normal D-Bus user session bus socket is /run/user/<uid>/bus, which you can see by logging in and doing, for example, 'echo $DBUS_SESSION_BUS_ADDRESS'. Suppose that you SSH in to some Linux machine that uses systemd, run this, see your expected D-Bus session bus address, and even use 'lsof' to see what's listening to it. Do you actually have a D-Bus session bus active?

Well, maybe, because these days your D-Bus session bus is a socket-activated systemd service. Specifically, it's a user socket and service that's managed by your per-user systemd instance, which is a systemd process running as 'systemd --user' under your uid. When this 'systemd --user' process starts, typically on your first login (including a SSH login), it will start listening on a bunch of sockets, including your standard D-Bus session bus socket, and it will insert a $DBUS_SESSION_BUS_ADDRESS into the 'systemd (user) service manager environment variables' with 'systemctl --user set-environment ...', where many things will then pull it back out (typically including your new SSH login).

Your actual D-Bus session bus and associated processes are only started by systemd if something actually tries to talk to the session bus. This will typically start 'dbus.service', but what that does varies from distribution to distribution. On Fedora, this runs dbus-broker-launch via /etc/systemd/user/dbus.service (which is actually a symlink to /usr/lib/systemd/user/dbus-broker.service), which I believe then starts dbus-daemon itself; on Ubuntu and Debian this directly runs dbus-daemon via /usr/lib/systemd/user/dbus.service. Your distance will likely vary on other distributions.

(The corollary to this is that 'systemctl --user set-environment' and friends aren't using your D-Bus session bus, because systemd does this without starting the the session bus. In Ubuntu 26.04 this communication is done through a systemd socket, /run/user/<uid>/systemd/private, that apparently uses a private API.)

If you SSH in to a server, look at your processes, and there's no dbus-daemon, I believe that you can be pretty sure that you don't have a D-Bus session bus operating yet. One corollary of this is that any surprising delays in logging in and starting your session definitely aren't D-Bus session bus problems, because clearly your session bus hasn't even been started.

(Well, assuming that you can deliberately start your session bus, for example by running 'dbus-monitor --session'. If your session bus refuses to start, that might be your problem. In my case, it's not.)

(This is the kind of thing that I want to write down in case I ever need it again, because I got confused about what was providing my session bus and whether it was activated or not.)

PS: Normally your session bus is shared between all logins, both on the local console and remotely over SSH, but this isn't required. It's possible to start another D-Bus session bus daemon and arrange for that session bus to be used by other processes. However I'm not sure you actually want to do this and it may be a mistake for some of my machines to (still) be set up this way.

(3 comments.)

An exciting change landed in the development version of Prometheus recently, making PromQL arithmetic expressions in time durations a standard feature instead of an experimental one. For me, duration arithmetic expressions by themselves aren't the truly interesting part. What's really exciting is that as part of this change, Prometheus has added some new PromQL functions, especially step().

(Although step() is gated behind an 'experimental PromQL functions' feature flag today, it will be made available as a standard function as part of duration expressions becoming a standard feature.)

For people who are familiar with Grafana, step() is the PromQL version of Grafana's $__interval interpolation variable. When you're in a range query, step() is how big the range step is, which was previously unavailable in PromQL even though Prometheus obviously knew this information. Prometheus has also added range(), which gives you the full size of the range duration (and the *_of() functions for further selection if, for example, the step() might be too small). Since PromQL now allows arithmetic expressions in durations, you can use step() in them, allowing you to write PromQL expressions like 'rate(your_metric[step()])', where the duration will automatically adjust to whatever the range step is.

Where step() is especially handy for me is when I'm doing ad-hoc graphs directly in Prometheus's web query interface (instead of wrestling with Grafana Explore). Previously I had to go through various increasingly elaborate processes to find out what the step value was for a given time range, so I could plug it into rate() or various *_over_time() things or the like. Now I can just ask for 'rate(...[step()])' and it will all work out right, and it will keep working right as I zoom the time scale in or out.

In theory one could replace various Grafana uses of $__interval in PromQL queries with step(). In practice this probably isn't worth it, unless you're running into problems with $__interval for some reason or maybe when you're writing completely new queries for new dashboards and so on. The minor advantage of using step() even in Grafana is that you can easily copy the query out of Grafana and put it directly into Prometheus or a query tool to see exactly what you're getting.

To use step() and friends in Prometheus 3.13, you need to enable some feature flags. But since this is going to be in the next version of Prometheus (unless something goes wrong and the Prometheus developers have to back out this change), I think it's pretty safe to turn on the necessary feature flags and start using step() and friends now, at least in ad-hoc poking around. Even if you have to stop using step() later, it will improve your experience today.

(This elaborates on a Fediverse post of mine.)

Over on the Fediverse, I had a little exchange recently:

[other person in a conversation]: I still haven’t forgiven Sun for NFS. No I’m not bitter.

@cks: It could have been worse, Sun could have stuck with nd.

What I was referencing in my post is a now obscure piece of cursed knowledge that I'm happy to share with you today.

Sun's workstations could boot without a local disk from very early on (because that made them cheaper, not because it made them better), but famously NFS only appeared in SunOS 2.0 (which required Sun to also create the idea of a virtual filesystem switch (VFS), which has appeared in basically every Unix since). The pre-NFS versions of SunOS operated without a local disk by using Sun's 'nd', the 'net(work) disk', which is basically what it sounds like.

SunOS nd(4) was a kernel block device (well, pseudo-device) that did its block IO through the network to the server kernel. As covered in nd(4), the same driver was used on both the client and the server, and the server handles everything in the kernel; nd(8) is only there for server setup purposes. The client wasn't configured with the server's information; instead it found the server through the simple approach of "[it] finds the server by broadcasting the initial request". As you can see from the fact that the manual pages I've linked to are for SunOS 3.0, SunOS kept the nd driver and infrastructure quite a long time after NFS was available (I'm not sure, but it might have only been dropped in SunOS 4).

(You can also see the SunOS 1.0 nd(4).)

By itself, the idea of a network disk device isn't particularly cursed. We used to run iSCSI based fileservers quite happily, there's a general ATA over Ethernet protocol that was at least a lot simpler than iSCSI, and Linux has DRBD (and there's probably others out there). What makes SunOS nd into something special is how it works, which comes from a specific limitation of SunOS covered in nd(4):

One last type of unit is provided for use by the server. These are called local units and are named /dev/ndl∗. The Sun physical disk sector 0 label only provides a limited number of partitions per physical disk (eight). Since this number is small and these partitions have somewhat fixed meanings, the nd driver itself has a subpartitioning capability built-in. This allows the large server physical disk partition (e.g. /dev/xy0g ) to be broken up into any number of diskless client partitions.

What this meant in practice was that on your server, you set up one giant partition and then manually decided on the starting and ending sectors for every nd 'disk' within that partition. Keeping track of all of these and making sure that they didn't overlap was your problem; as the nd(8) manual page dryly notes in the BUGS section, 'no sanity checking of disk partitions is done'.

For extra bonus problems, you might run out of available partitions to use on your server disk because you needed all of the available ones for regular filesystems and your swap area. If you were in this situation you could take the dangerous but necessary step of specifying your network disks using the special 'c' partition (cf dkinfo(8)), which was conventionally used to provide access to the entire disk. This was extra dangerous because you had to make sure that the nd disks you specified weren't overlapping into any regular partitions that you were using, since as nd(8) says, nd itself did no sanity checking. If you said sectors X to Y were network disk X, that's what they were, and goodness help you if some of them were also something else.

(I think this meant you could expose the server's /usr disk partition as a read-only 'public' nd device, so all your diskless clients could mount it, rather than having to put a separate copy of /usr into your nd area. This seems to be explicit considered in nd(4).)

Another charming thing about nd was that it didn't use UDP (or TCP). Instead it uses its own IP datagram protocol because, as covered in nd(4), "IP datagrams were chosen instead of UDP datagrams because only the IP header is checksummed, not the entire packet as in UDP" (and the manual page straight up says it was also done because the kernel internal interfaces were simpler). What this means is that the data sent through nd had no checksum protections on the wire, not even UDP's basic one; you were very much counting on absolutely nothing going wrong on your early 1980s Ethernet network.

(With early 1980s CPUs and so on, it presumably made a real performance difference to not checksum 1024 bytes of data on each packet. Early Sun workstations were not exactly performance powerhouses.)

All of this made using nd extra exciting and somewhat cursed. I don't think anyone really liked it back in the days, and people were happy to move to NFS, which used regular server filesystems and had much less of a chance to blow up your server and your clients in exciting ways.

(5 comments.)

Suppose, not hypothetically, that you have a situation where there's a number of distinct hosts backing some Git repository, such as all of the IP addresses of https.git.savannah.gnu.org, and one or more of them seem to be outdated or not working right. As far as I know, Git itself provides very few tools to examine or control which host the fetching process uses; the best option 'git fetch' has is to select only IPv4 or IPv6 hosts (well, IP addresses).

(Quite reasonably, git fetch's verbosity settings are focused on the Git side of things, not on the network side of things. The network is supposed to just work, or at least fail in an obvious way.)

Fortunately we can take advantage of the simple Git HTTP protocol to directly query every server to see the state of their repository (assuming that they respond). Specifically, we want to use dumb client reference discovery to see the commit ID of one or more references (most often branch heads) on each server. To do this we'll need some way of forcing a HTTPS server name to resolve to a specific IP address, but curl has this feature in the form of its '--resolve' command line option.

(Curl has two ways to remap a HTTP server name; for using a specific IP address, --resolve is easier or at least more obvious than --connect-to.)

So what we want is something like this (assuming we care about the state of the main branch; you can pick another one):

host=https.git.savannah.gnu.org
url=https://$host/git/emacs.git/info/refs
ipv4=$(dig +short a $host.)
# Curl requires IPv6 addresses as
# '[...]'.
ipv6=$(dig +short aaaa $host. |
       sed -e 's/^/[/' -e 's/$/]/')
for i in $ipv4 $ipv6; do
  echo $i:
  curl -sS -L --resolve $host:443:$i $url |
    grep refs/heads/master
done

(I'm using 'dig +short' in this example as the most convenient general way to get the IPv4 and IPv6 addresses of the host, without anything else.)

At the moment, this says that all of the IP addresses are actually responding to Curl and one IPv6 address is outdated (ie, it has a different commit ID for refs/heads/master, and that commit ID is an old one). I will leave a nicer output format as an exercise to the reader; this is a quick hack that I'm writing down in case I ever need it again (and I hope not to).

(One improvement would be a script that you ran in a repository so it could look up the current head commit and only show you mirror hosts that had a different commit ID for their head.)

Actually doing anything with this information is also left as an exercise to the reader. As far as I know, Git doesn't let you not connect to one specific IP address, so you're left with more system level things like blocking connections to the errant mirror host. Right now, I'm just going to remember to use 'git fetch -4 savannah' when fetching from the official GNU Emacs repository (and hope that no IPv4 mirror host goes bad).

(If you're operating mirror hosts you can use this approach to monitor whether all of the hosts are sufficiently up to date and check for persistently out of date hosts. Or you may have a better monitoring method, for example based on internal mirroring data.)

(2 comments.)

Suppose that you (I) write an Apache configuration stanza to make some settings conditional on the module that they're from, so you can enable and disable the module without blowing up your web server configuration (or having to edit it). Your version looks like this:

<IfModule mod_qos>
  QS_LocRequestLimitMatch "^...$" 1000
  QS_SrvMaxConnPerIP 8 100
</IfModule>

Unfortunately, this stanza isn't doing what you (I) think it is, although it looks like it's correct. As covered in the <IfModule> documentation, the name you give IfModule is either a module identifier or a module file name ('the file name of the module, at the time it was compiled'). The 'mod_qos' I've used here turns out to be neither; the correct module file name for mod_qos is 'mod_qos.c' (or at least I think it is), while the module identifier is 'qos_module'.

(As covered in the documentation for LoadModule, you can get the module identifier by looking at the first argument to LoadModule for the particular module. Although I believe that people writing Apache modules can do it differently if they want to, the standard form seems to be <whatever>_module. I don't know if there's any good way to get the module file name other than guessing it's the conventional name of the module (or its .so file) plus '.c'.)

The effect of an <IfModule name> for a name that's neither a module identifier nor a module file name is that you've completely disabled that stanza, since the <IfModule> can never match an enabled module. You might wonder how you can make this mistake, and in my case it's simple. If you started out with an unqualified set of (module) directives and you're adding the <IfModule> with the intention of then (temporarily) disabling the module, well, the module configuration will be ignored after your 'a2dismod' and Apache restart just as if you'd gotten it right. You'll only discover the mistake when you try to enable the module later (or copy the configuration stanza to another web server entirely), and that might be years later.

(In our case we 'temporarily' disabled mod_qos on our web server in December of 2022 and then never re-enabled it because our web server stopped getting obviously overloaded.)

Today I shared a brief war story on the Fediverse:

Today's new and exciting failure mode for a DHCP server handing out leases to dynamic clients: have something on your network that answers pings for absolutely every IP address (yes, it was broken). The ISC DHCP server pings what it thinks is a free IP before handing it out (to be sure), so if something answers all of those pings you have no 'free' IPs on your network and no one gets a dynamic IP.

(Technically this was like a day or two ago.)

The direct symptom of this in your ISC DHCP server logs is some log lines that look like this:

dhcpd[1656384]: Reclaiming abandoned lease 172.17.101.132.
[...]
dhcpd[1656384]: ICMP Echo reply while lease 172.17.101.132 valid.
dhcpd[1656384]: Abandoning IP address 172.17.101.132: pinged before offer

As ISC dhcpd documents (for example in dhcpd.conf's discussion of the 'ping-check' statement), by default dhcpd will ping an IP it's about to dynamically allocate to make sure it's unused. If something answers, dhcpd more or less gives up on the IP address (this doesn't happen for statically assigned IPs, at least according to the dhcpd.conf manual page). The consequence of this is that if you have such a 'screaming' machine, one that's answering ICMP pings for all IP addresses, dhcpd will conclude that your dynamic IP address pool is entirely exhausted and no dynamic client will be able to lease a new IP. For extra fun, apparently some clients will not accept a DHCP IP if there seems to be something else using it.

(I'm not sure what happens when clients are renewing leases.)

Such a screaming machine is obviously broken in some way and you need it off your network, but unless you get lucky, tracking it down may be hard. We were lucky that the machine was using its real MAC to answer all of the pings (which showed up all over the DHCP server's ARP table, among other places) and that MAC was registered with some useful and accurate additional information. Without that we would have been reduced to tracing through switch ARP tables (for switches smart enough to report that) and eventually unplugging sections of this particular network, which would have been pretty disruptive.

This particular network is port isolated, but that doesn't help here. Our DHCP server has to be able to reach the entire network and the entire network reach it, so its ARP requests flood through the network and anyone can answer them (and then its ICMP ping will be routed through the switch fabric to whatever port answered).

(3 comments.)

Suppose, not entirely hypothetically, that you're installing a server with two network ports and during the (Linux) installation, a third network device shows up with a funny name like 'enp1s0f4u1u2c2', which is a USB Ethernet device (despite you not having any such thing plugged in to the server's USB ports). To your further surprise, your server installer can even lease a DHCP IP on this interface, say "169.254.3.1". Congratulations, your server has a BMC, and this BMC probably speaks Redfish, which is sort of the modern, cloud influenced version of IPMI.

One of the things you'd like to do on a server with a BMC is have the server (the 'host') talk directly to the BMC, for example to get sensor information that only the BMC has or to configure the BMC. In the world of IPMI, people had to put together special methods to talk to the BMC, which required kernel drivers, extracting information from SMBIOS, and so on. This isn't the greatest, and is also not at all like how you talk to the management agent in cloud virtual machines, where you generally talk to the management agent by making HTTP requests to a special IP address. IPMI is in part a network protocol, but talking to a BMC using IPMI over the network is completely different from talking to it from the host server.

The IPMI protocol is an essentially custom UDP based thing, which made sense at the time. Redfish instead uses a HTTP REST based approach, partly because by the time Redfish was started, it was obvious that HTTP had become basically the universal protocol (and it was already in use for similar management purposes in cloud environments). So the natural way for a host server to talk to its Redfish based BMC is over some sort of network connection, instead of through some special out of band mechanism the way IPMI does. However, this requires a network interface that's directly connected to the BMC (and nothing else).

You could in theory wire up some sort of semi-virtual PCIe Ethernet device that was connected to the BMC on the other side. But that's complicated. Most BMCs support 'KVM over IP', and as part of that they need to provide virtual keyboard and mouse input, which these days is done by having the BMC present a (virtual) USB keyboard and mouse to the host. Many BMCs can also present USB storage media to the host, for install media. If a BMC is already presenting a bunch of virtual USB devices to the host, the obvious way to provide a network interface to the host for Redfish is through a virtual USB Ethernet device.

(I think all of these virtual USB devices are often presented on a virtual USB hub, and maybe even a virtual PCIe USB controller to go along with the virtual PCIe graphics card and maybe PCIe bridge. There's a lot of funny business that goes on to connect a BMC to the host system, never mind issues like how BMCs may control the host power.)

Since the host needs to have an IP address on this virtual USB Ethernet device to talk to the BMC at the other end, the BMC has a little DHCP server as well as its HTTP server. This internal HTTP server may or may not be the same as the BMC's regular management HTTP server. On some of our servers, this internal BMC HTTP server only answers Redfish requests and doesn't provide the normal BMC web interface that you can get on the BMC's management network interface (which also supports Redfish requests, of course).

(I think this is a sensible security decision on the BMC's part.)

It started with a discovery about my access to the official Emacs repository:

This is my face when my home desktop appears to be persistently talking to one instance of https.git.savannah.gnu.org that is many days out of date and out of sync on the GNU Emacs git repository. Yes, I know, volunteer organization, but how do you even troubleshoot that? At this rate I'm going to have to switch to the Github mirror even though the thought makes me spit reflexively.

(By 'how do you even troubleshoot that', I meant how might I figure out which mirror is out of date and report it. That DNS name has eight IPv4 addresses and eight IPv6 ones, although you can at least restrict Git to either IPv4 or IPv6.)

This led to me wishing for a way to conveniently pull the same branch from two different upstreams. To be specific, the experience I would like looks like this:

$ git status
On branch emacs-31
Your branch is up to date with 'origin/emacs-31'.
[...]
$ git pull
[pulls and updates my emacs-31 local checkout
from some reliable mirror]

$ git pull savannah
[pulls from the official repository and also
updates my emacs-31 local checkout]

As was pointed out to me by several people (once I read the git-pull manual page), you can get almost this experience with plain remotes, but on your non-default remote you have to remember to use a special form, 'git pull savannah emacs-31'. As far as I can see there's no way to tell 'git pull' to do this automatically, since 'git pull' goes from your local branch to the remote (and you can only have one remote).

(At this point you could make a git alias for this specific operation, perhaps called 'git alt-pull'.)

You can also merely fetch from the non-default upstream and then manually trigger the same nominal merge that 'git pull' would:

$ git fetch savannah
$ git merge --ff-only savannah/emacs-31

I also had a pseudo-clever idea that almost certainly won't work, as I can see better now that I've read a bit more about 'git pull':

I could manually edit .git/config so that both the savannah and github remotes updated the same local 'refs/remotes/origin/*' ref(s), but I suspect that this would go badly and also not necessarily ripple through to updating the on-disk state when I did a 'git pull' from the one that isn't listed as 'remote =' for the emacs-31 branch.

Manually switching the remote of the emacs-31 branch back and forth will do that but, well, annoyance.

Given how 'git pull' works, I believe this would give me the same 'you asked to pull ... but didn't specify a branch' error as 'git pull savannah' does in a standard configuration. It's possible that 'git fetch savannah' followed by 'git merge --ff-only' would work (if the shared remote HEAD was updated by my fetch), but that's already two commands and not much different than other options (at the cost of possibly confusing Git).

Manually switching the remote of my local branch back and forth can be done on the command line with git's general '-c <name>=<value>' setting for setting a configuration parameter:

$ git -c branch.emacs-31.remote=savannah status
On branch emacs-31
Your branch is ahead of 'savannah/emacs-31' by 17 commits.

Once again I could make a cover script that set this for all Git commands, or I think I could do it for specific commands through Git aliases (which I think would make it easier to pass command line arguments through compared to a 'git alt-pull' alias).

(The real answer is that I'm likely to switch more or less permanently to one of the mirrors and give up trying to directly fetch from savannah.gnu.org, which is apparently very overloaded and not all that healthy. One reason I cloned directly from savannah is that I had the impression that the mirrors could lag significantly behind, but on a spot check today the Github one was pretty up to date, with commits only an hour or two old. But at least going through this exercise has left me a bit more educated about some Git stuff.)

In a comment on my entry about how sometimes it's the network that's the problem, I was asked what networking workstations typically have on our networks (and if we still use 100 MBit for low speed things). The simple answer is that it's somewhat mixed but mostly 1G Ethernet, often with 1G uplinks from the switches that the office network jacks are connected to. But there's some stories involved.

As a university department, we have been in our current buildings for what is mostly multiple decades, and in some cases since the early 1980s (cf). In the oldest building (with our old machine room), the department's presence predates twisted pair network wiring entirely (there is still disconnected thicknet wiring around our offices); our other most used building predates Cat-6 cabling. As a result, all of this older space is wired with Cat-5 twisted pair and is only good for 1G Ethernet.

We started out using all of this wiring at 100 MBit (including our connection to the university backbone), and even as time went by we had a mixture of 100 MBit and 1G switches and connections. Due to cost reasons and sometimes wiring density problems, we only slowly migrated people's office network jacks from 100 MBit to 1G as we progressively replaced switches in our multi-switch setup. However, we did eventually move everything to 1G, including in our machine rooms (partly because our old 100 MBit switches were getting unreliable). I think most of our '1G' switches in wiring closets and so on are purely 1G, with no 10G uplink, so wired workstations on those 1G network jacks are sharing the uplink bandwidth.

Areas in buildings do get renovated from time to time, including parts of the department's space. For years, these renovations have typically included running new Cat-6 (or Cat-6A) wiring that is properly certified and tested for 10G-T (it's hard to significantly renovate space without needing to tear out the old wiring and then put in replacement wiring). Sometimes the people who are funding the space renovation really do want their desktops to have 10G connections available, and in that case the renovation funding will also include a collection of 10G-T switches for the other end of the office network jacks to plug in to (however many switches needed for however many true 10G office ports, which is often all of them). Otherwise, while we have 10G-T capable wiring in the walls, we only plug our end of the wires (in wiring closets and machine rooms) into 1G switches and everyone gets 1G.

(Newly built out space in new buildings is all wired with Cat-6A, of course, and we try to get funding for 10G-T switches to go with it. There's been some of this over the recent enough past, as the university does acquire a certain amount of new space over time.)

All of this is the theoretical answer. The practical answer is that a lot of people are using laptops (possibly with docking stations) or desktops with wireless built in, and many people default to using the wireless connection even if there's a network jack or three in their office. Typically, the only people who use actual wired connections are those that need their machines on one of our special research group private networks, instead of on our general 'random desktops and laptops' network. In some areas of the department, there's almost no use of wired network jacks because almost everyone is using laptops and the wireless.

(This makes wireless into critical network infrastructure.)

Here in 2026, I wouldn't try to use 100 MBit for anything, even things that don't need 1G, because I wouldn't trust it. In theory your 1G network interfaces and switches still support it (and maybe your 10G ones too, but don't count on it). In practice, it's been a fairly long time since 100 MBit hardware was in common use. Any switches or the like that you still have lying around are old, and these code paths and physical capabilities in network ports aren't likely to be heavily tested. You want to be running your networks in common and well supported setups, and today that means 1G is the minimum speed you should target.

(And if you have cabling that seems to only support 100 MBit, it's probably broken. Network cables do go bad over time.)

(One comment.)

Suppose, not hypothetically, that you have a web server that's using as much of your server's bandwidth as it can get and you would like it to use less bandwidth than that, so that you can get a word in edgewise (for backups, for example) or just because you don't feel like donating 1/10th of your outgoing bandwidth in apparent perpetuity to people who should be building local caches. There are various ways you might do this, for example using FreeBSD pf on your perimeter firewall, but the lowest impact and risk option is to do it on the (Linux) web server itself with tc(8), the Linux traffic control system. Conveniently I've already done a tiny bit with tc to fight bufferbloat latency.

There are probably a variety of ways to do this in tc(8), but what I'm using right now is mostly pulled from the Arch wiki. It goes like this:

1. We have to switch our device, which is eno1 for me, to using Hierarchy Token Bucket as its top level qdisc (queueing discipline), and in the process set a default class that otherwise unclassified packets will be assigned to.

   tc qdisc del dev eno1 root
   tc qdisc add dev eno1 root handle 1: htb default 30 r2q 1000
   

   Following the Arch example, the default class is 1:30 (the root handle of '1' combined with 'default 30'). Because I'm working with high bandwidth, I need to change r2q to make tc happy.

2. Set up a child class to limit bandwidth, here with a limit of more or less half of a 1G Ethernet. Because we're only doing one level bandwidth limiting (ie, we're not sub-dividing it), this can go right under the root parent ('1:').

   tc class add dev eno1 parent 1: classid 1:10 htb rate 400mbit ceil 500mbit prio 10
   

   Pick the bandwidth numbers to taste depending on your irritation (and the interface speed of your server, your outgoing bandwidth in general, and so on).

   The classid is somewhat arbitrary but as the Arch Wiki example shows, there can be a use for setting a numbering hierarchy if you have multiple levels of parent and child classes. In the Arch example, the top level bandwidth limited class is '1:1', then it has child classes '1:10', '1:20', and '1:30' (the default class for traffic).

3. Following the Arch Wiki example, we add a fair queueing qdisc below our bandwidth limit, so traffic flows that fall into this bandwidth limit are (hopefully) a bit better handled. I think this means that one HTTPS reply to a requester on a fast link won't starve all of the others.

   tc qdisc add dev eno1 parent 1:10 handle 10: sfq perturb 10
   

4. Filter outgoing HTTPS traffic into our bandwidth limiting class:

   tc filter add dev eno1 protocol ip parent 1: prio 1 u32 match ip sport 443 0xffff flowid 1:10
   

   This is a u32 match, and it requires some decoding to understand. A u32 match like this is fundamentally 'match <value>/<mask> at <offset>', and if we dump the raw form of this with 'tc filter show dev eno1', we'll get "match 01bb0000/ffff0000 at 20". The 'ip sport 443' format is simply tc-u32's friendlier way of encoding that (well, technically 'ip sport 443 0xffff', since the 0xffff is a load-bearing part of the shorthand). The tc-u32(8) manual page has various cautions about this matching, so I think if you really care you want to use iptables (or nftables) rules to set a firewall mark and then match on that with tc-fw(8).

5. Create our default catch-all class that effectively has no bandwidth limit, and as with our bandwidth limited class, also attach a tc-sfq(8) qdisc below it.

   tc class add dev eno1 parent 1: classid 1:30 htb rate 1gbit ceil 10gbit prio 1
   tc qdisc add dev eno1 parent 1:30 handle 30: sfq perturb 10
   

   We have two choices for 'prio'. The first option is that we can set this to 'prio 10', the same as our bandwidth limited class; in that case, I believe non-limited traffic will share bandwidth with the bandwidth limited HTTPS traffic, basically getting what's left over. Alternately, we can decide that we want non-limited traffic to have priority over bandwidth limited traffic, in which case we want a lower priority so its packets are sent first. That's what we're doing here.

   We need a classful qdisc, but we'd like one that will automatically use all of the bandwidth available to it. I've used tc-htb(8) here because it's what I was already using, but there are probably better options. The bandwidth I put here is simply 'really big', starting with the 1G interface rate.

This appears to work but the tc-htb(8) rate numbers may not correspond exactly to the wire bandwidth you (I) see. For our purposes, this is good enough; we're not trying to limit things to an exact MBytes/sec value, just something that's in the right ballpark.

Now that I've read the tc-u32(8) manual page, I'm pretty certain I wouldn't want to use it for anything significant where I cared deeply about what traffic was getting sorted into what class. Iptables or nftables is going to be much better at matching network traffic, especially in unusual and weird situations, and you can use that in traffic control rules through tc-fw(8). Here it doesn't matter too much if some unusual HTTPS traffic 'leaks' outside of my tc filter, as long as there's not too much of it (plus the whole server has an intrinsic bandwidth limit).

PS: For our purposes it would be nice to do something sophisticated to aggregate all HTTPS requests from a single IP address together in a single 'flow' for tc-sfq(8). In theory this is possible with tc-flow(8) but in practice I can't figure out a command line that works right (despite consulting eg tc-sfb(8)'s example). I'm sure the documentation makes sense to people who have a deep understanding of Linux traffic control, but I'm not such a person.

We've recently been having mysterious problems getting some of our backups to perform well. Also I recently wrote about how we'd wound up with a web server that frequently saturated its outgoing 1G interface with traffic (and it was a feature that it didn't have a faster network link). These two things turn out to not be as unrelated as we'd like, and there's a story or two there.

The problematic backups aren't done through our usual Amanda-based backup system; instead, a particular central machine pulls /var/log and other relevant files from FreeBSD and OpenBSD hosts (either through rsync or direct SSH access) and then writes them over NFS to one of our NFS fileservers (and then they get backed up by the Amanda backups). The problems manifested as terrible NFS performance; the central machine's load average would go to 50 or 70, everything NFS related on it would be slow, and so on. While we like NFS in general we've had quite a share of problems with it (cf, also), so we immediately assumed that this was yet another instance of an NFS problem. First we reduced the IO load from rsyncs in various ways (eg), then we spent quite a while digging at various NFS metrics, both from our metrics system and from the live system during a problem. Nothing really seemed to be the problem; the fileservers had perfectly good performance, the master machine had perfectly terrible actual NFS performance, and especially it didn't seem to be able to write the backup data very fast, in the range of a few MBytes a second.

Since I was pretty sure that the central machine could write to its local disk acceptably fast, I considered switching to a more complex backup scheme where we first rsync'd things to the local disk, packed up this directory tree into a tar archive, then scp'd it to the relevant fileserver, thereby bypassing any NFS write issues. To check that this would work acceptably fast, I started by scp'ing a test file from the central machine to the fileserver. To my surprise, the scp ran at only about 2 MBytes/s. More testing showed that scp's from this central machine to anywhere ran at 2 MBytes/sec at most, which did rather explain the NFS write problems (much like the lack of a CPU explains a server's failure to power on). At this point a penny dropped in my mind.

We're a university department, which means that we don't necessarily have the newest, shiniest stuff around and we keep things in service for a long time. One of the things we've kept in service is basic 1G switches, because quite a lot of servers don't need more than 1G (and as mentioned, sometimes it's a feature that a server only has 1G). But when I say 'basic 1G switches' I mean switches that are so old that all of their ports are 1G, including the port we use for 'uplinking' them into our overall 10G switch fabric (in contrast to modern 1G switches, many of which have one or two SFP+ ports that can run at 10G, or even a 10G-T port or two). This is fine for our normal 1G servers, which don't generate or receive much traffic even in aggregate, but it breaks down badly the moment you put a high volume 1G server on such a switch. For example, our new high volume web server, which was not only saturating its own outgoing 1G interface but was also saturating the 1G uplink from the switch it was connected to, a switch which unfortunately also had this critical central machine connected to it.

There's a programmer saying that if you think you have a compiler bug, you don't, you have a regular bug that you haven't spotted yet. This saying is almost always true, and it generalizes to other areas, like kernel bugs (also) or hardware problems, or networking problems. At our scale, modern networks are reliable, so if we don't have anything obvious wrong (for example, our monitoring system hasn't alerted us that some machine is unexpected at 100 MBit/sec), our network is almost certainly working fine. So for days it didn't occur to us to actually check. Of course the network was working fine, the network is always working fine. It obviously had to be NFS, especially since NFS has been flaky for us in the past. Except that sometimes it is the network and there's even a good explanation for it that becomes obvious once you realize it's the network.

This whole experience has given me some things to think about. On the one hand, years ago I wrote about how an obvious problem isn't necessarily obvious. There are a ton of things that can be wrong and you have to winnow through them somehow. On the other hand, I'm pretty sure that if we'd engaged in systematic troubleshooting from the ground up, we'd have found this pretty early on. For example, the USE method would have had us look at usage, saturation, and critically 'errors', which might well have caused us to look at TCP retransmits on the central machine (which were decidedly high).

Being systematic about troubleshooting is generally a good thing, but at the same time it's tedious. If the problem had really been a NFS problem (as it has been in the past) and I'd followed the USE method from the ground up, I'd have spent a chunk of time verifying that yes, the network was performing fine on both machines (along with the other things I looked at it, like NFS server metrics). Possibly what I should try to do is start out with likely guesses and then when they come up dry (eg, there are no obvious reasons for a NFS performance problem) and I'm getting frustrated, fall back to the USE method or something similar, even though it's possibly tedious.

(2 comments.)

Suppose that you're generating and transferring a file over ssh, for example to create a tar archive of something on a remote server and save it locally:

ssh [...] rem-server 'cd /var/log && tar -cf - .' >/backup/file.tar

If you're experiencing IO problems in this backup process, an interesting question is what buffer size ssh uses for its writes, perhaps because you'd like to make a few large writes to disk (for example, at the natural 128 KByte block size for your ZFS fileservers, even when you're writing over NFS) instead of a bunch of smaller ones.

The ssh manual page doesn't document anything about this, and there's no options to control it in either ssh or ssh_config(5). On Linux, running over TCP from a remote machine, the answer appears to be that ssh will normally do 32 KByte writes (or 64 KByte writes under some circumstances, possibly if the network is fast enough). It's possible ssh will write smaller buffers if the remote command generating the output can't keep up at full network bandwidth, and in general I suspect that there are a lot of things that can change this.

If this is important to you (and I'm not convinced it's important to us), you need to re-buffer the output from ssh in some way. The traditional way is to use dd, but you need to pick the right options to have dd reblock its input. There are other programs floating around but I don't know if any of them are standard.

(Since I looked it up, there's at least the Debian buffer package and mbuffer. There are probably others out there as well that I can't dig up in casual Internet searches.)

Finding out this information requires some way to trace ssh's system call activity. On Linux this is most easily done with 'strace', and you can narrow down all of the system calls that ssh does with 'strace -e trace-fd=5 ...'. You might think that you want to trace file descriptor 1 (standard output), but in fact current versions of OpenSSH ssh rewires its standard output on to file descriptor 5 and make file descriptor 1 point to /dev/null (which can be very confusing when you first encounter it).

(This is one of those things that I look into, don't find much, and then want to write down my negative results anyway for future use.)

Go's maps are famously implemented as hash tables, which is the only reasonable choice. The implementation has gotten somewhat more complicated since I looked at how maps store their values and keys due to the move to swiss tables, and these days you find the comments about how they work in internal/runtime/maps/map.go, but the core is still the same. Recently, a documentation commit landed in the Go development tree that opened my eyes to a bit of subtle complexity I hadn't considered before in Go's map implementation.

One of the things about hash tables is that they hash the value of keys down to some fixed size value in order to do operations more efficiently; in Go's current swiss tables, this is a 64-bit hash. Critically, the hash value of a key must be constant (which can be an issue in languages like Python that let you define a hash function at user level). You also want the actual value of keys to (only) compare equal when they are equal, which can also be a challenge in a language with user defined comparison functions, or just if you're dealing with NaNs.

Go has a quite broad definition of what's allowed as map keys; you can use any type that has == and != comparison operators defined. This includes pointers (which are directly comparable), arrays of pointers, and structs containing pointers (under the rule that a struct is comparable if all its fields are). However, Go pointers aren't guaranteed to be constant values, and today growing and shrinking a goroutine's stack will change some pointer values. This is a potential problem if you're hashing the current integer value of a pointer as part of a Go map key hash; you need that hash value to stay constant.

The documentation commit explains how Go deals with this today, primarily in its comment in map.go. When a Go value is stored as a map key, the Go compiler marks that value as 'escaping', which means that the value will be allocated in the heap instead of on the stack (along with anything it points to). Currently things in the heap never move, so once a key value is heap allocated, any pointers involved have a constant value and the key's hash value will never change.

As the comment notes, this is only done for keys that are getting stored in the map. Keys used for lookup or for delete will never be stored and so don't need to be specifically heap allocated. As the comment also notes:

If we are looking up a pointer which points to the stack, the hash value is ~irrelevant, as the key is guaranteed to not be in the map [...].

(This includes pointers in structs and so on.)

This map key hash stability requirement is a bit of a subtle constraint on any future Go garbage collector that works through copying values around (eg, also). Probably the simplest way to deal with it would be to mark heap pointers involved in map keys and then never copy or otherwise move them. Possibly you could do this on the fly during the garbage collection scanning process, since you need to trace through maps in general to mark their keys and values as used.

(Until I stumbled over this commit message and read into it more, I'd never thought about how the hash table stability requirement might clash with any sort of moving garbage collection mechanism.)

Suppose, not hypothetically, that you use rsync to push an encrypted backup file from the machine it's created on to a fileserver, where it will be backed up by your regular backup system. Because this encrypted backup file is backed up every day, you only need one copy of it on the filesystem, so you use and reuse a fixed name for the file. In other words, we're using rsync somewhat as if it was scp, but with better control over what remote files can be written and (not) read.

When you push (or pull) a file over rsync, rsync normally attempts to optimize what gets transferred by looking for common blocks in the file (how big a 'block' is depends on the file size, or you can fix it with the '--block-size' option, as covered in rsync(1)). This is a nice potential bandwidth saving, but it creates CPU and IO load on both ends as each of them checks their version of the file. In our specific case, we know that there aren't going to be common blocks; since the whole file is encrypted, it's basically random noise. Recently this backup process had some IO load problems, and today in the process of working on this I discovered rsync's '-W' option (also known as '--whole-file'). As the manual page explains, this 'disables rsync's delta-transfer algorithm'; in other words, it stops looking for common pieces between the two versions of the file. Rsync simply sends the whole file (and the receiver writes the whole file).

Since we know that today's encrypted file has no blocks in common with yesterday's encrypted file (well, it had better not if the encryption is working right), '-W' is exactly what we want to stop the receiving rsync daemon from doing unnecessary work (specifically, unnecessary IO). Effectively it turns the 'file copy' part of rsync into scp (although not literally; rsync will normally write the new version of the file to a temporary file and then replace the old version). Now that I know about -W, I'm going to be looking at some of our other uses of rsync to see if we might want to use it more widely.

(For example, we use rsync to back up /var/log from some FreeBSD hosts, and I'm pretty sure that's a good candidate for -W too.)

If you're using '-W', you want to avoid using '--checksum' and instead rely on the default 'quick check' of file size and modification time. This is because using --checksum requires rsync to read and checksum the whole file before the transfer starts (which is something that the manual page warns you about).

One lesson I've taken from today's experience is that when I use rsync, I should think about what I want to optimize (and what can actually be optimized). Rsync's default behavior is to optimize transfer bandwidth, but sometimes you have enough transfer bandwidth and you want to optimize for lower IO, lower CPU, or both (which is sort of our case for this encrypted backup file, with the extra issue that we know rsync can't reduce the transfer size). Alternately, sometimes you really want to squeeze the bandwidth and maybe '-S' and '-z' (and perhaps others) are what you want, even though you'll do more work on both ends.

(It's possible that rsync already has a clever encoding for runs of zero bytes and so '-S' doesn't save you any transfer bandwidth. I haven't tested.)

(This elaborates on a Fediverse post of mine.)

(3 comments.)

There are a variety of ways to automatically or semi-automatically install systems, especially Linux systems and especially over the network. People have built a whole raft of them over the years, often with relatively impressive capabilities. A number of them have significant levels of automation and control, including for things we might want like sophisticated automatic disk setup. Despite that, we're interested in approximately none of them. As a practical matter, we only want to use the standard, native install systems for whatever we're running, which in this case is the Ubuntu server installer. There are two reasons for this.

The obvious reason is that only native installers and install methods are officially supported by the Linux distribution (or whatever other free Unix we're using), and they're also the ones that are most used in practice. The native installers aren't always perfect, but other people work (often quite a lot) on making them work well, we can find plenty of people using them, and so on. If we use another installer, at the very least we're without support from the distribution. We're also likely to be off the beaten path, which means fewer people running into issues before us, fewer bug reports being filed and fixed, and so on.

The less obvious reason is that we do a certain amount of direct manual installation of servers for Ubuntu, FreeBSD, OpenBSD, and so on. The easiest way to do this is to use whatever native installer the distribution ships on their install ISO images. In the case of Ubuntu we customize this a bit, but none of our customizations are essential; we can and have installed systems straight from the main Ubuntu ISO images (and then possibly imported our customizations afterward). We could still do this even if we normally used a third party install system, but less would carry over between the two environments and we'd want to learn and stay familiar with both of them.

These issues aren't absolute blocks on using a third party installer system; we could deal with both of them. But we'd want to be getting something important from such a system, something that was enough of a gain to be worth the extra costs. I can think of situations which would be worth it but they haven't come up so far (at our modest scale and with only Ubuntu really in the picture for semi-automated installs, network installs, and so on).

(This is related to why it took us a relatively long time to build up a network install system, cf. There are specialized automated network install systems for Linux, but they run into this third party issue. Our current network install system for Ubuntu uses our Ubuntu ISO images that we also use for local installs, and that's a feature even though it creates complications.)

Once upon what's now a long while ago, we had a bunch of SunFire X2100 and X2200 1U servers (for example, they were used in our first generation ZFS fileserver). One of the reasons that I loved these servers back in the days was that for free, they came with a full IPMI/BMC setup that supported both KVM over IP and virtual media. I installed any number of servers this way from the comfort of my office, using the KVM over IP as if I was at the console (because I effectively was) and the IPMI virtual media to feed a local ISO image to the server I was installing. In time those servers went the way of all servers (which is to say, into the e-waste dumpster, although it took a while) and none of our later servers gave me that '(re)install from the office' experience with their limited BMCs. For years, we lived with doing in person physical installs and reinstalls of our servers (well, I lived with that, my co-workers don't care as much).

Recently we wound up with a bunch of servers in another building and a need to reinstall them all with Ubuntu 26.04. This pushed me into learning about UEFI network boot, working out how to network boot our customized Ubuntu ISO image, and then the realization that if we were reinstalling an existing, running server we could use a somewhat simpler kexec-based method. This has given us a reinstall experience (and sometimes an install experience) that looks a fair bit like the SunFire X2100 BMC based experience, and can be done from comfort of our office (instead of a noisy machine room a block or two away). My co-workers like it enough that we're talking about using our new network reinstall system even for servers in our main machine room.

On the one hand, it's nice that we can now have this experience with basically any system (it's nicer if it supports network booting, but reinstalls work without that). On the other hand, the experience is a lot more fragile than the BMC-based experience. Our network installs only works because the Ubuntu server installer supports access over SSH, it requires either a running system on the server or a whole collection of network booting infrastructure (including that it be enabled on the server). If the installer blows up (as it does too often in 26.04), you can't necessarily get access to the system or force a power cycle, and you may be stuck needing to go visit the server in person.

(The most practical version of the network install experience also relies on the server having enough RAM to download the ISO image. This seems to require 8 GB of RAM in common situations in my testing, which is far more than our SunFire X2100s normally had.)

This change from hardware support on limited machines to general software that more or less replicates the earlier experience (but imperfectly) feels like a pattern that's happened repeatedly. One advantage of the software version is that it generalizes, for example to virtual machines.

(Virtual machines can have a BMC-like experience, but it's all with a bespoke environment that's specific to that VM system. Network installs are generic across physical servers, virtual machines, and so on.)

PS: Of course the SunFire BMC used a Java applet, which was somewhat painful (cf). Modern BMCs thankfully use straightforward web stuff.

For a long time, I ignored kexec both for my own personal machines and at work. I knew it existed but I never attempted to use it. That changed recently when I realized we could use kexec to start a network (re)install environment without needing our servers to actually have network booting enabled (which they mostly don't currently, for historical reasons). This has led me to do some additional experimentation with kexec.

The most recent experiment was updating to a new Ubuntu kernel and then using 'kexec' to switch to it instead of 'reboot'. On Ubuntu, this can be pretty simple due to some short symlinks, more or less:

kexec -l /boot/vmlinuz --initrd /boot/initrd.img --append "$(cat /proc/cmdline)"
systemctl kexec

But I'm not sure we're going to actually use this in anything except very unusual circumstances.

The good side of using kexec instead of reboot is that you don't have to sit through your BIOS slowly fiddling around with hardware and then GRUB doing GRUB things. This gets your system back on the air anywhere between 'somewhat faster' and 'much faster', depending on how slow your BIOS firmware is (and also your GRUB timeout). Some of our servers have firmware that is pretty close to 'agonizingly slow', which is part of why we often have network booting disabled on them (this often skips a slow BIOS setup process for network cards, even if network booting was low on the boot priority).

But this good side is also the issue with not using reboot, because if you don't use reboot, you haven't actually tested and verified a cold boot situation. You don't know that your new GRUB boot entry for your new kernel works, and you don't know if your firmware is going to discover some hardware problem at an inconvenient time later on, instead of now during a planned downtime when you're available to deal with the situation. Of course probably the GRUB entry works and probably there's not going to be any surprises from the firmware (and maybe you can inspect the GRUB entry yourself to be sure).

For us, verifying the cold boot behavior (and having a simpler and less error prone 'switch to new kernel' process) is almost always going to be more important than a fast reboot. Now that we know about it, we may kexec into new kernels in exceptional circumstances, but I doubt we're ever going to do it routinely.

A while back I wrote about how claiming to be Googlebot is now a bad idea, where I noted that there were (reports of) malicious crawlers out there impersonating Googlebot and other legitimate big crawlers and at the same time, Google and other crawler operators published the IP address ranges their crawlers used. You could put these two together to block these impersonators:

Anything claiming to be Googlebot that is not from a listed Google IP is extremely suspicious and in this day and age of increasing anti-crawler defenses, blocking all 'Googlebot' activity that isn't from one of their listed IP ranges is an obvious thing to do.

After I wrote that entry, I quietly went and added support for blocking crawler impersonators to DWiki, the wiki-engine that powers Wandering Thoughts, and set it up for a few big crawlers with published IP address ranges. When I did this, I didn't expect to block very much, and for months that was indeed what happened; I'd get a few attempts once in a while. Then, this June, the floodgates opened.

For weeks, I've been seeing hundreds of requests a day claiming to be Googlebot (on a few days, thousands of requests). The requests come from a variety of IP addresses at a variety of providers, which I think are mostly or entirely cloud and hosting providers. The top sources by ASN are a rogue's gallery of places that I was already having problems with, like HostRoyale, M247, Latitude.sh, and Web2Objects. But AWS is in the collection as well, and there are a lot of other relatively mainstream providers. Many IPs seem to make only a few requests as Googlebot, and at least some of them immediately retry with another User-Agent value (which also generally doesn't work).

My guess is that this isn't a bunch of different abusive crawlers who've all spontaneously decided to try forging Googlebot to see if it gets them anywhere. Instead, I suspect that this is a large scale campaign by a single abusive crawler, run by people who can afford to obtain a lot of servers at a lot of different hosting providers (or who are prepared to commit various sorts of criminal fraud on a large scale). Ironically, if they'd picked a different tactic, I might not have noticed them among the background radiation of crawl attempts. Forging Googlebot and other known big crawlers is generally sufficiently rare that I actually bother looking at my logs to see it happening.

(I also suspect I'm not the only website this is happening to.)

PS: It feels somewhat ironic that this is happening at the same time as me wondering if I should allow Googlebot at all.

Recently an interesting series of commits landed in Prometheus with the goal of reducing the size of the Prometheus binary by allowing the Go linker to remove more unused code (something it's quite good at in general, although the linker is also deliberately limited in this). The commit with the message that's most informative about what is going on and why is discovery/gce: keep [Google Cloud] Compute SD client from defeating dead-code elimination, and you can read the full details in it. The short version is that if you're using certain sorts of reflection anywhere in your program, the Go linker won't remove exported (public) methods of any concrete type that's reachable through an interface. It doesn't matter how narrow the interface is (it can be the famous and minimal fmt.Stringer); the moment you combine reflection and a concrete type in an interface, the Go linker more or less stops throwing out unused functions and methods. Well, sort of, as the commit explains.

Unlike the standard Go toolchain not doing dead code elimination for package level variables with constant values, this isn't merely the linker deciding it's too much work to do this dead code elimination optimization. Instead it's at least partly a correctness issue. The problem for the Go linker is that reflect allows you to reach through any retained interface value to use any and all exported methods on the underlying concrete type of the value (and any types it contains), using things like Value.MethodByName() and Value.Call(). This makes it hard or impossible for the Go linker to know which exported methods are really dead and can never be reached at runtime.

(This has to apply to concrete types contained in top level concrete types because reflect can reach through structs, channels, maps, arrays, and so on to retrieve underlying types and values, and thus methods on those types.)

The current Go linker is actually doing more work and eliminating more dead code than the documentation requires it to. The documentation for Value.MethodByName() and friends say that they apply to all exported methods (possibly only of a given name), but apparently the linker will skip this for types that are never directly or indirectly boxed into an interface, because such types aren't reachable through reflect. Since all reflect functions that create a Type or a Value take an any (ie, 'interface{}') as their argument, you can't go from a value of a concrete type to either without putting the concrete type in an interface and triggering this. What this means in practice in a program where there's any use of reflect (including in some sub-dependency off in a corner) is that if you put a 'big' type with a lot of direct and indirect exported methods into an interface, all of those methods and all of their dependencies will have to be retained in the binary (and increase its size, possibly a lot), even if you only use a tiny subset of them.

(I believe this includes innocent looking things like merely printing such a 'big' concrete struct, which you might do for debugging purposes or because it has a String() method that does useful stuff. And of course JSON serialization uses interface values; json.Marshall() takes an 'any' as an argument, so there's your interface. While the json package uses reflect internally, it doesn't currently call any of the reflect methods that triggers this linker behavior.)

There are at least two ways around this, visible in the Compute service discovery commit and a similar Kubernetes commit. In the Kubernetes commit, a concrete top level Kubernetes struct was not retained in full in a Prometheus service discovery struct that would then be boxed into an interface; instead, only the methods on the Kubernetes struct that were actually needed were extracted and embedded into a new struct, so the Go linker only had to retained those methods and their code dependencies. In the more complex Compute commit, some processing had to be done dynamically using concrete types that had to be retained, so instead of putting the concrete types in a Prometheus struct (that would then be boxed as an interface inside the Prometheus code), the values of the concrete types were made inaccessible to reflect by putting them inside a function closure, and only the function closure was stored in the Prometheus struct.

One thing I take away from this is that one should avoid using the various reflect method-getting methods if at all possible, both in a program and especially in a package that you expect other people to use. If your package uses these internally, you're creating spooky action at a distance effects on the whole program (and you should probably mention this in your documentation).

PS: The Go linker's dead code elimination is (currently) discussed in general in a big comment in cmd/link/internal/ld/deadcode.go, which is worth reading for some details that I hadn't thought about until now, such as needing to retain all methods that might be reached through interfaces (which is necessary because you might wind up casting an interface value to another interface entirely, eg, also).

PPS: As mentioned in the Prometheus commits, one of the packages that uses reflect this way is go.yaml.in/yaml/v4. For the actual code and usage involved, see here and here, which seem like reasonably sensible uses to me, even if they have awkward consequences.

One answer to how your Ubuntu 26.04 server boots when its root filesystem is on a software RAID array is that it just does and you don't need to think about it. Unfortunately this wasn't the case in the pre-beta version. That's been fixed since (contrary to what I thought until now), but that raises the question of what changed between 24.04 LTS and 26.04 LTS to cause and then fix this.

At some point between 24.04 and 26.04, Canonical switched from booting servers using initramfs-tools (and an initial ramdisk built with them) to booting them with Dracut. Initial ramdisks built with initramfs-tools will automatically assemble all software RAID arrays they see, or possibly anything they see in their embedded /etc/mdadm/mdadm.conf file (it's not clear to me right now). Initramfses using Dracut will normally only assemble software RAID arrays that are explicitly specified with rd.md.uuid= arguments, regardless of what's in their /etc/mdadm.conf. The 26.04 pre-beta server installer didn't add any such rd.md.uuid arguments to the kernel command line, causing the obvious problems.

(At the time our workaround was to create an /etc/default/grub.d file that explicitly added the right rd.md.uuid argument to the kernel arguments. This had the UUID of the root filesystem's software RAID array hard-coded, which is the style of these files.)

Now that 26.04 has been released, if you install a 26.04 server system with a mirrored root filesystem and inspect the resulting server's /proc/cmdline, you'll still find a striking lack of rd.md.uuid parameters (or rd.auto, cf dracut.cmdline(7)). However, if you peek into the initial ramdisk itself, what you'll find is an /etc/cmdline.d/20-mdraid.conf that contains an rd.md.uuid setting. For example:

rd.md.uuid=838c6235:90e967bb:8e64be9f:3c38b0f8

(/etc/cmdline.d in the initramfs also contains a '20-root-dev.conf', at least on a test system. This is partially redundant with the kernel command line itself.)

As covered in the Dracut manual page, when the initramfs boots, Dracut (in the initramfs) will look at /etc/cmdline.d/ and effectively take them as extra Dracut-directed parameters. So apparently the installer has been updated to write this file, fixing the problem without making the fix visible in /proc/cmdline.

(The initramfs also has an /etc/mdadm.conf, which probably opens you up to the traditional issues with it, although some of them may not be an issue any more.)

This 20-mdraid.conf file only exists in the initramfs, not on the booted system, but the overall Ubuntu kernel update system arranges for each new kernel and initramfs to get this 20-mdraid.conf, so your newly installed kernel boots just as your old one did. (I don't know which bit of the overall setup is responsible for this; it might be Dracut itself.)

Many BMCs will tell you what hardware your server or other system has in it. This is a useful, even valuable function, but you shouldn't necessarily trust what your BMC is saying. We recently had a very vivid demonstration of that, where the BMC of a server confidently reported that the hardware had a number of inexpensive NVMe SSDs, a bunch of (expensive) RAM, and a pair of processors, but mysteriously the server wouldn't power on at all. When we opened up the server, we found no NVMe SSDs, no RAM, and especially no processors, which did rather explain the failure to power on.

Since the BMC in question offered us the option of downloading the raw SMBIOS information, my best guess is that the BMC gets its hardware inventory by having the main system's BIOS push the SMBIOS blob to it when the BIOS powers on or otherwise goes through a state change. The last time the system was powered on, it presumably had all of those NVMe SSDs, the RAM, and the processors, and now the BMC has that state latched until the next time the system can power up.

I was going to say that it made sense from a technical perspective too, but the more I think about it the less I'm completely convinced of that. I believe that some of the hardware bits in question report their status over I2C and other separate channels, not over the main system PCIe bus, so potentially they could be queried by the BMC through a direct BMC connection to appropriate I2C busses. It seems even NVMe devices can have separate I2C connections, so they could plausibly be inventoried separately. Whether this can be done with trickle power with the system powered off is another question. Perhaps what's going on is that the BMC can get hardware information independent of the system processors, but only when the system is powered up enough to supply power to everything, and that can't happen with no processors.

(This also shows vividly how a BMC can be a completely separate thing from the main system. There's no processors or RAM in the server, but the BMC is perfectly fine. It would be nice if the BMC could notice this and tell us, but I suppose this is a quite unusual situation that's not worth adding extra hardware or software for.)

Looking back, this isn't the only time I've seen this sort of BMC behavior. I've seen more than one type of server where some of the hardware sensors reported through IPMI over the management network were only available when the main system was powered on. In the case of sensors this was obvious because they reported no value when the main system was powered off. In the case of our hardware inventory, presumably the vendor decided that the value of the BMC reporting something when the main system was powered off outweighed the problem of it being potentially out of date.

PS: The interesting thing on top of all of this is that the BMC offered to let me configure the server's BIOS (through an entirely separate area of the BMC's web site, which switched to HTTP Basic Authentication). That's a neat trick when you have no processors, but maybe the BMC has software (and hardware) that can write to the NVRAM where the BIOS stores those parameters. It's also a bit disturbing when part of the BIOS parameters are processor dependent, but probably that's using the same BMC-cached information as the hardware inventory.

Suppose, unfortunately not hypothetically, that you have some machines that the Ubuntu 26.04 LTS installer kernel sometimes gets a kernel oops during a network based reinstall (we don't think this is a hardware flaw, but who knows; these machines were stable on 24.04). Further suppose that you're not network booting these machines but instead you're using kexec to boot them into the installer environment. This creates an awkward situation, where the over the network installer may have gotten far enough before the panic to have written over enough of the disk (with the previous install on it) so you can't reboot from it. Fortunately there is a way out, because you can contrive to kexec the installer environment again from within the installer environment.

(A kernel oops is not a kernel panic, and normally doesn't reboot the system unless you've made specific settings changes, which the Ubuntu 26.04 LTS installer environment hasn't done. In this case, that's a good thing for us.)

Once you've used 'kexec -l' to load the installer kernel and initrd with the required command line arguments, you have two choices for actually rebooting into the kexec kernel. You can use the plain and normal 'systemctl kexec', which will "shut down and reboot the system via kexec" (and is what you may have used to boot into the installer from the running system), or you can directly use 'kexec -e'. Based on our recent experiences, if you're rebooting the system because the kernel hit an oops, you probably want to use 'kexec -e', not 'systemctl kexec'.

The problem with a plain 'systemctl kexec' is that as the manual page tells you, it will try to go through the usual standard orderly shutdown process. If your system has hit a kernel oops, this process may not finish. Not even 'systemctl kexec --force' will necessarily finish, because that still tries to do some things in an orderly way. By contrast. 'kexec -e' proceeds immediately to the new kernel (as I believe 'systemctl kexec --force --force' also does, cf). Abruptly rebooting into the new kernel is not normally what you want to do when you're going to use things on the disk and so on later, but when you're already in the middle of the network installer environment, everything is going to get thrown away anyway and an orderly shutdown is pointless (and in this case, dangerous).

(An orderly shutdown at the end of the install process isn't pointless because the installer needs to finalize various things.)

We'll probably keep using 'systemctl kexec' when we're using kexec to go from an existing local install to booting the install environment over the network. It's always possible we'll change our mind and go back to the regular system (perhaps I made a mistake), and in that case an orderly shutdown is better.

One of my standard little scripts is something I call sshup, which waits for a machine to be 'up' by periodically checking to see if its SSH port is responding. As mentioned in my original entry on sshup, I actually have two versions of this script and recently I discovered that the difference is quietly important.

One version of the script uses Netcat, on our Ubuntu machines. The specific Netcat command line it uses is:

nc -w3 -q3 -z "$1" ssh >/dev/null 2>&1

(I didn't used to need the redirection, but Debian reverted a patch and Ubuntu 24.04 picked up the change.)

The technical effect of this is to try to connect to the SSH TCP port and exit once it's done that (or been rejected or timed out), with an appropriate exit status.

The other version doesn't use Netcat, but in Netcat terms what it does is the equivalent of:

nc -N "$1" ssh </dev/null >/dev/null >2&1

The technical effect of this is to try to connect to the server (if possible), immediately shut down the sending side of the TCP conversation, and wait until the server closes the connection completely.

If the server is working properly, these two versions have the same result. But if the server isn't working, the answer these give is different, with the second version being more useful. The problem with the first version is that it only checks if the kernel is willing to let you make a TCP connection to the SSH port; it doesn't check if the SSH daemon is actually responding (normally, the SSH daemon will print a banner, then see that there's nothing more to read from the network and close the connection). If the server is broken sufficiently so that the kernel will accept the connection but the SSH daemon won't run, the first version will tell me that the machine is up and the second one will correctly tell me I can't SSH in to the machine.

(Recently a server hung during shutdown in exactly this way, which is a story for another entry.)

This difference is also relevant for health and monitoring checks for services (and famously so). Connecting to a TCP port is the very basic step in a health check; after that is checking that you received a banner or a canned response from a service (for HTTP services, perhaps a '200 Okay' from a health check endpoint), and after that is checking that you can do something meaningful that's part of the service's regular activity. Of course the higher level you go in checking the service, the more specific your health check is to the service (whether you implement the check in code or in a configuration file that says what to look for). But at the same time, it's more meaningful because it comes closer and closer to what actually matters.

(But there's a tradeoff in how close you come versus how much work and so on it is. You make that tradeoff for individual services based partly on your knowledge of how that service works and what's likely to fail. For my 'is a machine up' checking script, seeing the SSH server banner is good enough basically all of the time. And if my sshup script says the machine is up but then I can't actually log in, that's telling me something useful too.)

(3 comments.)

Recently I wrote an entry about a simple but difficult wish I had for a certain sort of terminal pager that handled emoji. In comments, people suggested that the venerable less would do what I wanted, which was something I'd already tried and discovered it had behavior I didn't want. Except, well, let me quote my eventual comment:

It turns out that my testing of less's behavior was being contaminated by my usual $LESS settings, and I think that 'less -XRn' does more or less what I want (I was being fooled partly because my usual $LESS includes 'c', well, technically 'C').

I've been using less for a long time and for most of that time I've had the same set of less settings, burned into a $LESS environment variable set in my dotfiles, and also the same set of alternate less key bindings (where N and P move through files instead of their normal actions). I had long since forgotten the specifics of my $LESS settings when I tested less; my settings were just how less worked. And so I was fooled into initially thinking that less didn't work for what I wanted.

(I remember N and P partly because when I use less as root or on a machine I haven't particularly set up my account on, I don't have them and I miss them.)

My $LESS setting is far from the only environment variable I have set in my dotfiles. Without looking through them carefully I couldn't tell you what I have set, why I set them that way, and whether or not they're still correct or useful. The same is true for the configuration files for various programs, where I'm sure that surprising things lurk.

(This isn't the first time I've had old personal dotfiles cause problems.)

This is of course just the personal manifestation of a general problem we have in system administration. We set up a lot of settings and configurations that make sense and work at the time, and then we forget about them because they're in the background, just doing what we expect. They can readily become our mental image of how the program or system behaves, because it's certainly how it behaves for us. Then someday either they clash with what we want to do (as my standard $LESS did) or they don't work any more because things changed out from underneath them (cf).

I don't have any good answers to this. Some people will advocate not changing any settings from the defaults, but not only is that giving yourself countless little papercuts over time, it doesn't even necessarily protect you; programs can change their default behavior so something you've become accustomed to doesn't work any more.

(It's good to document the settings and why you made them, but that only helps with half of the problem. It's still easy to forget that you even have custom settings. Should you budget an hour every six months or whatever to look over and check your settings? Well, you can put it on the calendar, but I don't think it's going to work in practice, not after the first few times, because it's boring.)

(2 comments.)

One of the broadly accepted things among at least system administrators is that our own code is a liability (some programmers may resist this idea somewhat more). If your systems run on a collection of locally developed, bespoke programs, scripts, and so on, someone has to maintain and update all of that (I sort of wrote about this long ago, also). If you use standard programs instead (or at least as much standard code as possible), hopefully someone else does that, and you can choose which programs and what code to use based on how well maintained it seems to be at the moment.

(This breaks down when you're large enough, but I'm talking about more modest sized organizations or groups.)

It's recently struck me that the obvious extension of this is that configuration and configuration files are also a cost and a liability (partly as a result of thinking about the staff time cost of our email system). This is obvious when a configuration file embeds what is effectively code, such as Prometheus alerting rules (which have a whole array of clever tricks that you wind up learning), because code in any form is still code. But it goes beyond such embedded code into any configuration that you have to maintain. If you need a complex Bind configuration file or a complex DHCP setup or a tangled Apache configuration (perhaps for a very special server), some day they're probably going to have to be changed or at least understood again.

We can't operate with no configuration and we shouldn't try to. Under the right circumstances, code and configuration are both more of an asset than a liability. But I think we should be cautious about it. Just because we can create a complex configuration that does a lot doesn't mean that we should. It's tempting to create a complex configuration that does lots of things to meet your needs and support all the features you want, and at least for me there's an appealing problem solving aspect to it. But the result can be something that has long term costs and consequences.

Have I created some overly complicated configurations at work? Probably yes. These configurations have given us useful or important features, but they've also saddled us with complexity that we're now stuck with because we (the sysadmins) and other people have come to depend on the features the configurations create.

(Ultimately, one issue is that as a system administrator, it's hard to persuade myself to leave a problem unsolved. Simpler configurations usually mean living without things and we often put in the things for good reason, so simple configurations would require telling more people "you can't have that because we don't feel it's a good idea to support it, although we could".)

(3 comments.)

Ever since I started using PyPy and then had it quietly work fine for years, I've been keeping it in mind as a generally easy way to speed up any Python program that could benefit from a performance boost (which programs PyPy could accelerate sometimes surprised me). But that was in the era when most everything we had was Python 2 based. Now that I'm moving more and more things to Python 3, there's a little issue opening up that I've been thinking about. That issue is what versions of (C)Python PyPy supports, which is to say which language version it implements.

If you look at the PyPy site, you'll see that of Python 3 versions they support, the most recent one is 3.11, which the regular ('CPython') version of was released in late 2022. If you've followed a bit of PyPy development news, you may have read that the PyPy project is resource constrained and thus lags behind CPython for things like implementing language features or standard library changes. My understanding is that PyPy's timeline for implementing language features beyond 3.11 is "whenever we can but don't hold your breath".

On the one hand, Python 3.11 is a perfectly okay version of Python. On the other hand, Python 3 keeps adding more features in newer versions, including useful features. One area I've definitely noticed some changes is in type annotations and typing in general. Typing changes the code I write and I'm fairly certain that the sort of type hints that are (conveniently) available probably will do so too. And at a certain point, more and more code that I want to work with will require something more recent than Python 3.11. If I'm lucky, I'll get warnings about this; if I'm not, I'll some day discover that the reason pipx isn't upgrading various things I have installed through it is that the newer versions aren't compatible with Python 3.11.

(For all I know this is already happening. Or maybe pipx explicitly complains about this situation. Although I have some machines not using PyPy with pipx, and so far they have the same version of programs I install everywhere, such as Python LSP servers.)

In the long run I suspect that this is going to lead me to abandon my use of PyPy with pipx, and probably to step away from PyPy as Ubuntu LTS Python versions keep increasing beyond what PyPy supports. Right now, we have Ubuntu 22.04, Ubuntu 24.04, and Ubuntu 26.04 machines, which have Python 3.10, 3.12, and 3.14 respectively. PyPy's 3.11 doesn't look particularly bad (and it will be years before we stop having 22.04 machines). And of course Ubuntu 28.04 is a fairly long way away, and PyPy might increase its Python 3 version by then.

(This makes me feel a bit sad. PyPy has been good for me, and it's still a great option for Python 2, one that we may turn to sooner or later.)

A while back on the Fediverse, I mentioned something about our departmental web server:

Wow, people do some extremely slow downloads from work's main web server. We roll logs at midnight and Apache wrote the last log record to the old file at 03:58 (for a request that had started at 23:27).

(For our sins we've become a load-bearing source of ML image training data, for the "CIFAR" image dataset. We average 40 Mbytes/sec outgoing web server bandwidth most days.)

(This also got mentioned in passing in my entry on 24 hours of that server's logs. I call us 'load-bearing' not only because of how popular this dataset is but also because in the past the department has gotten plaintive emails from people about it when the dataset wasn't accessible.)

Recently, so many requests started arriving that they overwhelmed our main departmental web server and we decided to deal with the problem by moving this data to another physical machine with its own, dedicated web server (and then using HTTP redirections to push 'people', which is to say software, to fetch the data from that new web server).

(As is so often the case at universities, this isn't an officially published dataset that had been given a permanent URL. Instead, it's part of the home page of a now-departed graduate student. All of this home page data normally lives in a special set of filesystems on our NFS fileservers, but this dataset was so popular that the web server's kernel always had it cached in RAM.)

Now that the data is on its own server (which is still running Apache, because we use Apache for everything), this produced some interesting data, to wit:

How our role as a load-bearing source of ML image training data is going: we had to move the data to a special web server just for it, and that web server has been saturating its 1G network link since midnight local time. It has almost 4,000 connections (the maximum currently allowed).

In something that vaguely amazes me (because I'm sometimes stuck in the past), the machine is using under 10% CPU despite pushing wire rate TLS.

(That's 10% CPU on 4 core HT machine, so out of 8 nominal CPUs. It's a fairly old machine, a Dell R230.)

To be clear, that 4,000 connection limit isn't really a sensible one and you wouldn't normally do this for this sort of bulk server in our situation. At 4,000 connections, if we assume they're all downloading large multi-megabyte files (typically they are) and the available bandwidth is split evenly, each request is getting about 30 KBytes/sec of bandwidth. The reason we set such a high connection limit is because as far as I know, in Apache there's no way to reserve a certain number of connections for use by certain IP addresses, such as your monitoring system. We have a 4,000 connection limit to make it as unlikely as possible that our monitoring system is frozen out and triggers alarms.

Naturally people are not getting their data very fast from our server, but that's not our problem. The current server stats say that since restart Tuesday morning the average reply size is 14.9 MBytes and the average reply duration is a bit under four and a half minutes, which is an extremely unimpressive data rate.

People in that current 4k requests thundering herd are getting their data even slower, of course.

These are Apache's built in statistics, which report the state since server start. This server doesn't always have that many requests and run at wire saturation, although it's happened twice since Monday morning (and the first time was sustained for close to 32 hours). At quieter times the server can run at 'only' 70 to 80 MBytes/sec of outgoing traffic.

You might be surprised that such a popular server only has 1G networking. Well, there's a good reason for that:

Could we put a 10G-T network card in the current server for this dataset? Sure. And then we'd probably blow out all of our outgoing bandwidth to the university backbone. So no. That this server only has a 1G interface is a feature, not a bug.

There are a whole host of ways to limit overall server bandwidth, but the absolutely simplest and most foolproof way is to put the server on a limited-speed network connection. With a 1G network interface, this server is physically incapable of generating more than about 120 MBytes/sec of outgoing traffic. There are some side effects (logging in to the server and interacting with it is quite slow when it's running at saturation like this), but we can live with them.

(There's 2.5G and 5G networking these days, but we don't have the applicable switches and network cards, and given why this 1G saturation traffic seems to happen, I'm not in any rush to increase this server over 1G.)

Sidebar: Where the traffic comes from

As of about 19:40 when I took the stats, there had been 332,000 requests started since midnight local time, from 13,055 different IP addresses. The most prolific single requester is a single Google Cloud Platform IP that made 8,200 requests and probably received about 141 gigabytes of response data. This is far more data than the CIFAR dataset contains, so this IP address was clearly re-fetching data it had already fetched. Unfortunately I believe GCP doesn't charge for ingress traffic. A cluster of Azure IPs in the same /24 made thousands of requests each and the subnet received an aggregate of 1.8 TBytes of responses.

My suspicion is that these are processing jobs or training runs or the like being launched and re-launched repeatedly, with no effective local caching of the fetched data (either because the software doesn't cache at all or because everything is being done one at a time on ephemeral setups, so the just fetched and cached data is immediately discarded). In the past I've seen a similar pattern of high volume re-fetching from what was clearly a HPC cluster at a university.

Overall, 36.25% of the requests are from Azure, 9.89% from GCP, 6.78% from Alibaba IP address space (AS45102), and then everything is under 4% and I'm not going to bother writing it out.

(2 comments.)

In theory, network booting with UEFI is fairly straightforward. Unlike the earlier BIOS based network booting, UEFI defines network booting in the standard itself (although it's not required). In practice, vendors of x86 servers have found any number of creative ways to put stumbling blocks in your way in their firmware (aka 'BIOS'). Here's an incomplete list of reasons that your server might not be willing to do a UEFI network boot the way you expect it.

• The firmware might not even be in UEFI mode, in which case at best it's trying to do BIOS PXE boot, which requires a different set of support infrastructure on your boot server that may not be there (cf, also).

• There's no UEFI network booting option (or options) in the server's UEFI boot order.
• The UEFI network boot option is trying to boot from the wrong interface (on a server with multiple interfaces).
• The firmware only supports UEFI network booting using the built-in 1G network interfaces, not with your 10G-T add-in PCIe card that is actually connected to your network.

• UEFI network booting is enabled in the boot order, but your firmware settings have disabled network booting for all network devices that the firmware will try. Depending on the firmware, this might be a master option that applies to all onboard network interfaces or it might be a per-interface setting.

  (In some firmware this isn't even in the regular 'BIOS' setup screens, but is instead in a completely separate top level configuration system for (some) network devices.)

• Your firmware settings have the network devices in 'legacy' network boot mode (ie, PXE booting) despite your firmware being in UEFI mode, so the firmware won't touch them for UEFI network booting. Sometimes the firmware's setup screens will vaguely describe this as the device firmware being in 'legacy' or 'UEFI' modes, and not specifically mention network booting.

  You might ask why vendors would allow this configuration mismatch. I have no answer, I just know that at least one vendor does have UEFI firmware that will let you do this.

If the server's firmware has a specific 'PXE/UEFI network boot' option that it offers to you on startup (along with things like 'enter setup' or a regular boot menu), this boot option may or may not provide any useful information if the firmware decides it has no valid UEFI network device to boot from. You may be in a situation where you pound on F12 (or whatever key is applicable) but nothing happens and you drop through to whatever 'boot failed' option the firmware has available.

(If the firmware is feeling especially hostile, it will drop you into the UEFI shell [PDF], and see also the Arch wiki page.)

Similarly, if the server offers you a general one time boot menu, but when you pick it there's no UEFI network boot options, this could mean either that the general UEFI boot ordering in the firmware doesn't have network booting included or that the firmware thinks there's no valid UEFI network devices.

There are typically at least two ways to disable (UEFI) network booting on servers: you can take it out of the boot order, or you can tell the often built in network devices to not do network booting. If you've historically disabled network booting on your servers because you don't use it and typically all it did was add extra time to server boots as the firmware poked at network devices, you may have done either or both of these and now need to reverse either or both.

(In non-UEFI, BIOS MBR boot mode, network device firmware has historically fiddled around with PXE stuff even if the BIOS boot sequence wasn't going to wind up trying PXE booting. Or at least it did on some of our servers, so we often turned that off.)

All of this can leave you hunting all over your server firmware's setup screens in order to find whatever magic bit that's making the system not boot from the network the way you want it to. Good luck, but start by checking UEFI boot order and (network) device settings.

Yesterday I wrote an entry about why servers running Ubuntu can stall on boot for two minutes; the short version is that in 26.04, a network interface that either has no carrier or that has nothing else on it will cause systemd-networkd-wait-online to wait for two minutes in the hopes that this changes and the interface becomes healthy. My core diagnosis of the trigger for the problem was correct, but I had the wrong fix because in my testing, I made a classic mistake that's only really possible with virtual networking for virtual machines. There's an often invisible difference between an isolated virtual network interface and one that has no carrier, and I tested with an isolated interface, which stalls under some circumstances and fixed by some things, but not a virtual machine network interface with no carrier. The latter requires a more thorough fix.

I will put the answer at the front: you need to set every disconnected interface in your Netplan configuration to 'optional: true'. Setting 'ignore-carrier: true' isn't necessarily sufficient unless the interface has carrier. If you're mangling your 26.04 Netplan configuration anyway, I suggest you also set 'accept-ra: false' on every inactive interface (and maybe your active one too, unless you use IPv6 RAs). However, you should not set 'optional: true' on your main interface.

There are an assortment of things that will cause systemd-networkd-wait-online to think that it should wait until a particular interface is configured. A probably incomplete list of them is having a static IP address specified for an interface, having DHCPv4 or DHCPv6 enabled on the interface, and having IPv6AcceptRA=yes (to my surprise, although this is sort of covered in the documentation). Under normal circumstances, systemd-networkd will not do any of these things (or attempt them, for DHCP) if the interface has no carrier, which will cause the interface to not be configured. So it sounds like what you want is ConfigureWithoutCarrier=yes, so systemd-networkd configures the interface even without carrier and systemd-networkd-wait-online is happy. Unfortunately this has an important limitation spelled out in its documentation:

With this enabled, to make the interface enter the "configured" state, which is required to make systemd-networkd-wait-online work properly for the interface, all dynamic address configuration mechanisms like DHCP= and IPv6AcceptRA= (which is enabled by default in most cases) need to be disabled. [...]

(Although it's not really clear from the documentation, as far as I can see IPv6AcceptRA=yes is probably the default in many environments, as this manual page excerpt says.)

To disable this behavior when you have ConfigureWithoutCarrier=yes, in theory I believe you also need to set RequiredForOnline=no on the interface (or disable DHCP, IPv6 RA, and so on). Well, if you're working in a purely systemd-networkd world, which we're not on Ubuntu. Instead we're working in the world of Canonical Netplan.

Netplan doesn't simply run systemd-networkd-wait-online and let it decide what to do and what interfaces to consider. Instead, Netplan writes a drop in configuration file, /run/systemd/generator.late/systemd-networkd-wait-online.service.d/10-netplan.conf, and that configuration file tells systemd-networkd-wait-online specifically what interfaces to wait for and what (minimum) states they can be in; this minimum state is normally 'degraded' (as reported by networkctl. As far as I can see, Netplan will normally list and wait for every network interface that is mentioned in your Netplan configuration and that is not set as 'optional: yes'. Also, systemd-networkd-wait-online specifically waits for all of these interfaces to not be in the 'configuring' setup state (again, as reported by networkctl, and as more or less documented in its manual page).

If an interface has no carrier, it's permanently stuck in networkctl's 'no-carrier' state and will never reach the 'degraded' state. If this interface is listed in your Netplan configuration, Netplan's systemd-networkd-wait-online configuration file will say it has to reach 'degraded', and so systemd-networkd-wait-online will time out. This happens regardless of what other Netplan properties you have set for that no-carrier interface; if it's mentioned at all and is not 'optional: yes', you will always have a two minute timeout on boot. This includes an interface with just a do-nothing property like 'renderer: networkd'.

If you specify 'ignore-carrier: true' on an interface, Netplan will duly write ConfigureWithoutCarrier=yes to the applicable .network file. If an interface actually has carrier, this will make an interface with 'accept-ra: true' (or no mention of it) finish configuration immediately regardless of whether or not something responds; the interface will be "degraded" but also "configured" and the systemd-networkd-wait-online configuration Netplan has created will consider it ready. This is unlike the standard behavior covered in the manual page quote above, because Netplan likes to surprise people.

(If DHCP is on for the interface, systemd-networkd won't complete configuration until and unless it gets a DHCP answer, and systemd-networkd-wait-online will have to time out.)

Setting 'ignore-carrier: true' on an interface with a static IP in Netplan will allow that static IP to be configured even when there is no carrier, but it won't make that interface ready (by Netplan's standards) and so you'll still have that two minute timeout. The only way around the two minute timeout is to either not mention the interface at all in your Netplan configuration or to set it as 'optional: true'. Both of these will cause Netplan to not list the interface as an interface that has to reach the "degraded" state and "configured" status in the systemd-networkd-wait-online configuration file it writes.

(I don't think Netplan ever writes RequiredForOnline=no into a systemd-networkd .network file it creates.)

If you set all interfaces in your Netplan setup to 'optional: true', systemd-networkd-wait-online won't run at all under normal circumstances. This isn't obvious from the created configuration; Netplan will write a drop in configuration file for it that has no ExecStart= lines, which makes it seem like the standard .service file should take over and run systemd-networkd-wait-online in the standard way. However, the drop in file that Netplan writes does have a conditional dependency on /run/systemd/generator/network-online.target.wants/systemd-networkd-wait-online.service being a symbolic link, and Netplan doesn't set up that symbolic link, so the whole service is skipped. Given this, I think you almost certainly shouldn't set 'optional: true' on all interfaces, which means not setting it on your main one (the one you're actually using).

(That systemd-networkd-wait-online doesn't run at all is obvious if you look at the logs, which will report that it's not running because its conditions aren't met.)

Suppose, not hypothetically, that you have some standard physical servers. As is typical, the servers have more than one network interface (two is usually the basics even for 1U servers, and maybe you put in a 10G-T card in some), but you're only using one network interface on one network; the others are just sort of there. Recently you've started putting Ubuntu 26.04 LTS on them, and to your surprise (and displeasure) these machines stall for two minutes on boot, which they report as waiting for systemd-networkd-wait-online.

The direct culprit is that you haven't told Canonical Netplan to ignore the state of network carrier for all of the network interfaces that you aren't using and don't have plugged in to anything. This is the 'ignore-carrier' property, listed in Properties for all device types, and you need to set it on every network interface that is listed in /etc/netplan/*.yaml. However, why this happens and how we got here is a little bit convoluted (and it can happen in versions other than 26.04, under the right circumstances).

Update: My fix is incomplete. See How to correctly not wait for network carrier in Netplan.

Famously, Netplan doesn't actually configure your networks itself. Instead it creates a configuration for something else (a 'backend'), and on servers this is systemd-networkd. For every network listed in your Netplan configuration, Netplan will write a '10-netplan-<name>.network' file to /run/systemd/network/, with the appropriate contents taken from whatever your configuration says. As sort of covered in systemd-networkd-wait-online's documentation (when carefully combined with the description of network states in networkctl), this will cause systemd-networkd-wait-online to wait for every such interface to have carrier (up to its timeout), unless the .network file specifies ConfigureWithoutCarrier=yes, which Netplan sets only if you put the ignore-carrier property on the interface in your Netplan configuration.

(Under some circumstances Netplan will also write .link files for one or more interfaces.)

At install time, the Ubuntu server installer gives you options for what to do with each network interface it detects. If you explicitly configure an IP address or set an interface to do DHCP (or the installer sets it to do DHCP and you don't explicitly turn that off), the interface will of course appear in your installed system's Netplan configuration because it has to. However, if you explicitly disable an interface (or in some versions of the installer, the installer disables it because it detected no carrier and/or no DHCP), the Ubuntu server installer has traditionally not written any information about that interface to your installed server Netplan configuration. Since these additional interfaces weren't mentioned in Netplan at all, Netplan never wrote out .network files for them and systemd-networkd-wait-online never waited for carrier for them. Your server's Netplan file contained only the interfaces you actually used and you were happy even if there were no 'ignore-carrier: true' settings.

What's changed in Ubuntu 26.04 is that the Ubuntu server installer appears to write out an 'accept-ra: false' property for every interface it detects, even if you disabled the interface during network configuration in the installer. Since the installer itself now mentions every interface in your generated Netplan configuration, Netplan writes .network files for all of them to /run/systemd/network and you wind up 'waiting' for carrier on every interface on boot, causing a two minute delay, which is the default timeout used by systemd-networkd-wait-online (it's the --timeout option's default value). You can fix this either by setting 'ignore-carrier: true' on every such interface or by deleting all of those interfaces from your Netplan configuration.

(To confuse you, systemd will report during boot that systemd-networkd-wait-online has no timeout and it's technically correct. What systemd really means is that the systemd-networkd-wait-online.service file doesn't specify a TimeoutStartSec= setting and as 'Type=oneshot service, it doesn't get a default one. Instead the timeout is inside the systemd-networkd-wait-online program and is opaque to systemd (the PID 1 daemon that is trying to 'start' this weird service).)

Because I had to wrangle this, here is how to list every network interface and then set them to ignore carrier, assuming you're using Ubuntu 26.04 where they are already all listed in your Netplan configuration:

ifnames="$(ip --json link show | jq -r '.[] | select(.link_type == "ether") | .ifname')"
for i in $ifnames; do
    netplan set "ethernets.$i.ignore-carrier=true"
done

This isn't perfect and will match some virtual interfaces, but I leave dealing with that as an exercise to the reader, along with excluding interfaces that currently have carrier. For our purposes at install time, this is good enough. And yes, we turn off detecting carrier even on our active interface because we would rather have the machine come up to the extent that maybe we can log in on the console or the like.

(It would be nice to have a way to delete a particular interface entirely, but Netplan's command line interface doesn't provide any good way to do that.)

If you have NFS mounts on a client using a sufficiently recent kernel, such as that shipped with Ubuntu 26.04 LTS, and you inspect /proc/mounts (or 'mount -t nfs'), you'll probably discover a new NFS mount option that is listed for your mounts. For example:

[...]:/h/281 /h/281 nfs4 rw,[...],fatal_neterrors=none,[...]

You might be curious what this option is, especially if you have code that parses NFS (v4) options and sorts options into ones you care about and ones you don't (as we do). Unfortunately, as I write this about a year after the option was added to the Linux kernel, 'fatal_neterrors' isn't documented in any manual page that I can readily find. As far as I know, the only accessible documentation for it is in the commit that added it, which I'm just going to quote:

NFS: Add a mount option to make ENETUNREACH errors fatal

If the NFS client was initially created in a container, and that container is torn down, there is usually no possibility to go back and destroy any NFS clients that are hung because their virtual network devices have been unlinked.

Add a flag that tells the NFS client that in these circumstances, it should treat ENETDOWN and ENETUNREACH errors as fatal to the NFS client.

The option defaults to being on when the mount happens from inside a net namespace that is not "init_net".

As you can see in that commit, currently the only two values you can see for this are 'fatal_neterrors=none' or 'fatal_neterrors=ENETDOWN:ENETUNREACH'. As a mount option that you provide manually, you can also ask for 'default' or 'ENETUNREACH:ENETDOWN'.

If I'm reading the tea leaves correctly, this commit first appeared in Linux 6.15.0 (and so a 'fatal_neterrors' option for NFS mounts first started being reported in /proc/mounts then). This specific commit seems to be merely one of a whole series of net/sunrpc and fs/nfs commits that make ENETDOWN and ENETUNREACH errors as 'fatal' under the right magic circumstances, and I think that by 'fatal' what it means is that any client NFS request fails immediately and isn't ever retried. I don't think it makes the NFS mount itself disappear. Hopefully it will allow you to unmount the now permanently non-functional NFS mount.

(The current state of affairs is that all NFS mounts will report some value for the 'fatal_neterrors' option, although the value of 'none' is effectively the pre-6.15 state of affairs.)

PS: Probably the place where documentation will appear at some point is the upstream source for nfs(5), which will then wind up on man7.org as nfs(5).

We have a long established approach for configuring our Linux systems, and specifically our Ubuntu systems (since these days the only Linux we run is Ubuntu). Part of that has been that we use a standard /etc/resolv.conf that points to our local DNS resolvers (which have our split horizon DNS setup). However, as people may have noticed, Linux distributions these days are moving to systemd-resolved instead of a straight /etc/resolv.conf setup.

There are good reasons for this switch in general. Systemd-resolved can cope with a variety of situations and problems that trip up the standard resolv.conf setup and it cooperates better with other programs. Almost none of these apply in our particular situation for our own servers, which have a static network configuration, a static DNS resolution search domain, and our DNS resolvers already handle all of the split resolution (and we had to make this work well before systemd-resolved existed).

(The one aspect of systemd-resolved that matters to us is that if you have multiple DNS resolvers, as we do, resolved will rapidly fail over from one that isn't responding to one that is. Resolved can also cache answers locally but for us this isn't a feature and we'd turn it off.)

Although I switched to systemd-resolved on my own desktops (cf), our servers have traditionally stuck with our static /etc/resolv.conf setup and entirely turned off systemd-resolved (which means even its D-Bus interface isn't available, because we'd rather have D-Bus based DNS lookups fail entirely than have them return different results). This isn't the Ubuntu default (which has been systemd-resolved for a number of LTS releases). So far we've been able to get Ubuntu to accept this without complaint, but as time goes on I've been feeling more and more nervous about going the /etc/resolv.conf path, especially without a working D-Bus systemd-resolved setup.

The reason I'm nervous is the traditional issue of people writing and testing software only to and against the default system environment. Some day we're going to find some piece of software that simply assumes that it can make D-Bus DNS queries to systemd-resolved; if we're lucky, the software will explicitly state this as a requirement. And some day there will probably be software that relies on some aspect of systemd-resolved's behavior even while doing traditional non-D-Bus name resolution, such as expecting the special '_outbound' name to resolve (although that's not universally available even on systemd-resolved hosts).

It's possible to configure systemd-resolved so that it only handles DNS resolution for people who ask over D-Bus. But if you're going to do that, you need to make sure that the systemd-resolved configuration (for DNS resolvers and DNS search path) matches your /etc/resolv.conf, and once you have a matching configuration, perhaps it's simpler and less error-prone to use systemd-resolved for everything. That gets you (us) on to the theoretical happy path of using the setup that Ubuntu expects (at the cost of having to care about the DNS settings in /etc/netplan files, and possibly having to update them if you ever, say, change or add a DNS resolver).

Through Ubuntu 24.04 LTS, we explicitly masked and disabled systemd-resolved and used our /etc/resolv.conf. In our current experimental Ubuntu 26.04 LTS setup, we first arranged to configure systemd-resolved (via /etc/netplan) to match our resolv.conf so that D-Bus resolution would work if anyone explicitly tried to use it, and now we're trying out having systemd-resolved be the normal resolution method. This makes our 26.04 machines more like Ubuntu expects at the cost of being different from our 24.04 machines, and maybe exposing them to oddities in systemd-resolved's behaviour. We'll have to see how it goes (and we may go back to our /etc/resolv.conf ways).

(The one thing we don't do is have nss-resolve be used in /etc/nsswitch.conf. I don't even know if putting it there is stock Ubuntu 26.04 behavior, and I don't even use it on my desktops so I have no experience with it.)

PS: We're not planning to make any use of systemd-resolved DNS server delegations. All of that DNS steering is going to continue to live only on our DNS resolvers.

(One comment.)

If you look at it from the right angle, the existence of third party package systems that sit on top of Linux distributions is rather odd. After all, these distributions already have packaging systems, yet here people are, ignoring them and writing new ones. A while back on the Fediverse, I said something on this general topic:

I have feelings and some of the feelings are that everyone screwed this up, for actually natural reasons. I don't think any distro solved the 'how do we let people easily build and deploy software on us' problem, and so programmers do what programmers do and built themselves another layer of indirection to solve their immediate problem.

(I mean, ship an entire container running a web server to do what you could do with a CGI or some PHP files or ... this is my face, etc.)

(There are two sides of this, the system side (what I was talking about) and the user side.)

On the one hand this is perfectly natural for Linux distributions to do. They built their package system to manage their own components, with all of the features that are important for that, not to make it easy for programmers to create a package for something that was the contents of '/opt/<whatever>' and specified a few entry points and system dependencies. On the other hand, this is part of how we got Docker and also a bunch of third party package managers for things, because programmers really do want and need something that is that simple, and they're quite willing to write it themselves.

(I'm using Docker as an example because it has a simple system for specifying how to build your software and then declaring various entry points to it. This is a lot of what a package manager does but almost all of them are more complex.)

At the same time, what I said is too harsh on Linux distributions, because the two problems containerization solves are hard, complex ones. Containers provide a hard isolation boundary for changes to the system and avoid the need for a huge API for system things. My aside of 'a CGI or some PHP files' elides a huge level of complication for what a hypothetical programmer built system package of the same application would have to specify, and the API that specification implies. If you can hermetically package and then deploy a web application as some PHP files in /opt/whatever, the system has to have already specified a lot about how that's connected to everything, how installing your package may trigger dependencies of installing a web server and maybe configuring it for TLS and so on.

You could do a simple 'package and deploy' system, but it would be limited in what it could deploy, specifically in how it could connect the raw files it stuck in /opt/whatever to the system (because raw files on their own aren't all that useful). These limits would probably have driven programmers to reinvent many or all of the third party package and container systems that we have today.

(You absolutely would need 'easy and simple', because programmers voted with their feet that system package managers were too complicated to deal with. You've always been able to build your own local software as RPMs or .debs and manage them that way, but almost no one actually did that because it was too much of a pain.)

With that said, I do wonder how far you could get today on Linux if you had people provide 'entry points' in the form of systemd units in some standard place in their '/opt/<whatever>' directory tree tarball. Systemd units give you service activation, service dependencies, the equivalent of cron entries in timer units, and let you expose network services as socket units. You can't create fixed users and groups but you can ask for dynamic ones that are created on the fly, and systemd could probably have something to put all of your units under some UID that's not root and has less powers.

(Systemd has some general features for this such as portable services, but it envisions a container like experience where you build and ship a full OS image, even if it's a small one.)

(Sometimes I find that my off the cuff gut reactions aren't quite as well baked as I thought.)

(3 comments.)

In a comment on my entry on universities, email, and the issues of running things in house, I mentioned that our departmental email system has a non-trivial cost in both hardware and staff time. For the hardware costs I can easily count servers, but things are more fuzzy for the staff time side of things so I'm not going to try to come up with a number (especially because our email system itself is somewhat complex because we have a number of unusual features, such as our simple mailing lists). The bigger issue is that the basic time cost to maintain your email system is in some sense an illusion, or at least an inadequate measure.

The reality of modern email is that to run a modern email system you need to know about a lot of stuff, and worse the stuff that you need to know about keeps evolving. You can't set up a mail system and then walk away from it apart from software and hardware upgrades; instead, you have to keep on top of a perpetually changing and evolving landscape of anti-spam systems and especially what you need to keep your outgoing email being delivered. SPF, DKIM, DMARC, DMARC alignment, and so on are merely today's names; in a few years there will be more, different things that are necessary to know and deal with, which will require you to modify your mail system as part of its obvious time requirements.

Obtaining this knowledge and keeping up on developments in email is often more or less invisible time. Without it, your system may work today (especially if you set it up from a good guide) but things get riskier as time goes on. At best, you notice when outgoing email starts bouncing; at worse, you have no idea until people start complaining to you (if you're lucky) that email they send isn't getting received properly. Then, if you're lucky, you can find another guide for the current new reality of email; otherwise, you're in for research and learning on your own. To copy from an old story, your time spent might be half an hour to make the eventual change to your mail system but a day of research to know what to change and why.

(A similar thing exists with anti-spam and anti-malware handling for incoming email. In a modest sized environment (such as ours) you have no realistic choice but to outsource much of that to some free or paid piece of software, but that still leaves you to monitor the overall situation, determine when your current software is falling behind, and then find and configure new software.)

Of course you need to develop and maintain a certain amount of expertise with your mail software, but that's usually easier to keep current. Your mail software typically makes changes less frequently than the overall Internet mail environment does, and those changes will often be explicitly documented in release notes, news entries, and changelogs. Still, depending on how much complexity you opt for in your mail system, this may take a significant amount of time initially (learning Exim was in no way an overnight thing for me, and then I had to design and build our configuration because Exim is a construction kit).

(We have what I certainly hope is an unusually complicated mail system (also, also).)

If you have a multi-person system administration team and only one person from it goes through all of this and learns all about email (both things like DKIM and things like your local mail setup), what you've got is a single point of expertise. If that person is on vacation and a mail problem comes up, you kind of have a problem. To avoid this, you'll need more staff time for more people to be at least somewhat up to speed on email stuff and how to deal with problems. At one level this is no different than any other system you operate, but those systems probably evolve slower than Internet mail does and require less time to keep up on.

None of this is insurmountable. But it's not trivial either. Maintaining your own mail system (in the broad sense) is going to take a non-trivial amount of staff time for one or more people to keep up on Internet mail developments, monitor your mail system for signs of problems like too much incoming spam and malware getting through or too little outgoing mail reaching its destinations, and troubleshoot issues. This need for maintained expertise is part of why it can be simpler to outsource email to specialists who deal with it a lot.

(Also, as you deal with a larger population of people using your systems, you also need to worry about compromised accounts sending out bad stuff.)

(One comment.)

I've been using my home desktop for many years now, with a variety of disk drives, and over those years some of those disks have failed (without data loss, because I use software RAID mirrors). Some of the time, those failures have happened when the HDD was still under warranty, but despite that I've never bothered to try to return one for a warranty replacement. This is strikingly unlike work, where we routinely return failed drives that are still under warranty.

Historically there have been two things that changed my behavior at home. The first is that when a disk drive fails in my home system, my priority has always been to restore redundancy as fast as possible. Sending a drive in for warranty replacement and waiting for the new one would take weeks, and I don't want to run a non-redundant setup for that long, so I've always gone out and bought a replacement drive immediately. The second is that a lot of the time, the failed disk was old enough that the replacement HDD I'd be buying was bigger, which typically meant that I had little immediate use for a smaller than I now wanted warranty replacement drive.

(In retrospect, spare HDDs for backups were one use, but I'm lazy.)

Neither of those factors have historically applied at work. At work, we've always maintained a spares pool for disk drives, so we already have an immediate replacement for a failed drive and then the new warranty replacement drive can be cycled back in to the spares pool. As for the potential size mismatch, there are two mitigating factors; we haven't changed the size of our disk drives very much, and we can usually find a use for smaller drives as system disks and the like. System disks for our servers merely need to be 'big enough', and it's okay if they're mismatched in size (although we prefer them to be matched).

This generalizes for more than disk drives, but disk drives are the thing I've had fail at home so far. If I had a power supply failure or a CPU failure or the like in my home desktop, I wouldn't be able to wait for a warranty replacement; I'd be buying something right away.

(These days that argues strongly for keeping older, good enough hardware around to use as a fallback, as new hardware becomes increasingly unaffordable. If I had a spare desktop I could press into service at home, I could wait out a warranty replacement. Although my current home desktop is so old that any five year warranties expired years ago.)

At work, we make sure we have spares for servers and other important things even though it costs more, because we have to. One reason we can afford to do this is that we don't need all that many spares; a couple of disk drives or servers can be the spares for a whole collection of in production ones, on the assumption that we won't lose too many at once. At home, I have so few disk drives, desktops, and so on that a 'spares pool' would be much bigger relative to my active stuff.

(This is partly because I have only a single home desktop. There's lot of people with multiple home machines and so on.)

PS: Long warranties are still a useful signal from vendors even if I may not take advantage of them at home. A disk drive with only a three year warranty instead of a five year one may well last much longer than five years, but that difference is a suggestive message.

(2 comments.)

Back in 2022 I said that my Firefox addons hadn't changed in a while, and while that's still mostly true there's been some minor changes that I want to write down. Addons are central to my Firefox experience, but I don't change them very often partly because of that and partly because I'm much more conservative with Firefox than I am with things like Emacs packages. I'm generally happy with my Firefox environment and most of what I do with it is stop irritating changes and fix things that get turned off.

Of my old addons in my main Firefox setup, I still use Foxy Gestures, uBlock Origin, uMatrix (which is still not quite dead), Cookie AutoDelete, Stylus, Textern, Certainly Something, HTTP/2 Indicator, ClearURLs (although that may be ineffective these days), and Open in Browser, although I'm not sure that's doing anything for me. I no longer use HTTPS Everywhere. Because I switched from Firefox's default "storage partitioning" for cookies to the stricter "first party isolation", I had to switch from Cookie Quick Manager, which I don't think works with either of them, to Cookie Manager, which does work with first party isolation as far as I can tell. In my main Firefox, this is the only addon change, although in a sense there's another change in that I'm now pervasively using Stylus.

I have a second significant Firefox environment and in this environment I recently set up a container-based setup. This setup uses Multi-Account Containers for the basic multi-container support and FoxyTab to assign wildcarded domains to specific containers. This second Firefox now also has Cookie AutoDelete (it didn't used to), and also I've installed a few other addons over time: Copy Alt Text, Don't Fuck With Paste, Right-Click Borescope, and User-Agent Switcher and Manager. I have all of these in this Firefox because this Firefox allows JavaScript generally, which allows websites to do a lot more things that I want to override and examine. My normal Firefox blocks JavaScript, so websites can't readily do things like blocking paste. I don't use these addons very much, so they're mostly around in case I need them.

Although it's not a change in addons, a significant change in my Firefox usage is that my second Firefox environment has moved from being disposable (with everything discarded when Firefox shuts down) to being persistent, because I admitted to myself that that's what I was doing with it. This is part of what pushed me to add Cookie AutoDelete to this Firefox profile; even before I made it persistent, I admitted that I wasn't shutting it down very often and so cookies were increasingly lingering in it.

(4 comments.)

So today I had a learning experience:

Firefox Nightly apparently has decided to not allow extensions/addons to run for file:/// URLs and I am sorry, this completely breaks my environment (my startup page is a file:/// URL) and I find it infuriating. Maybe I can find some hidden toggle for this somewhere.

This is relevant to me because my start page is a local HTML file and I'm a long time user of Foxy Gestures (cf), which I expect to work on my home page. When it doesn't, I get upset.

This turns out to be bug 2034168, Add explicit permission for file:-access and implement extension.isAllowedFileSchemeAccess(), the components of which landed in Firefox Nightly on May 29th. Fortunately there's a way to allow extensions to still work on file: pages, but it's a bit annoying. Extensions can have per-extension additional permissions, and this change adds a new such permission, "Access local files on your computer". But since this is a per-extension setting, you have to go through each of your extensions (at least the ones this is relevant for) and turn it on, and this permission landed only with the change, so the first time you start Firefox all your extensions will be broken.

(One hopes you don't count on something like uBlock Origin to block JavaScript embedded in local HTML pages from doing undesired stuff. Well, and have them open in your Firefox so they'll be immediately (re)loaded when Firefox starts.)

I can't really blame the Firefox developers for this. The new situation is probably more secure and likely very few people will actually notice (and it's going to be in the release notes for Firefox 153, for those of you who read them). But this is one reason why I have a whole test procedure for my custom Firefox trunk builds before I actually use them for my regular browsing.

(At this point I will give a shout out to mozregression, which allowed me to easily find the exact point where this changed and thus find bug 2034168. If you try out Firefox Nightly on a regular basis, you might want to keep it around, perhaps via pipx.)

On the positive side, this may lead to Firefox addressing bug 1617594, Allow extensions to create tabs with file:/// URLs with an explicit permission, which would reduce the need for one of my custom Firefox changes. Although I already have that change and would probably keep carrying it in my custom Firefox builds, since I have other tweaks as well (cf).

PS: Firefox's motivation here seems to be not that local file-based content is untrusted but instead that it's highly trusted, and because it's highly trusted it's potentially dangerous to let extensions change what people see. Firefox has a similar but stronger block on extensions running on Firefox websites like addons.mozilla.org.

When I wrote about the Emacs packages I use, I mentioned that I had Embark installed but barely used it because I didn't understand much about how to really use it. One reason for that is that while there are a bunch of articles on the web about things you can do with Embark, all of the ones I've tried to read started out with complicated stuff involving other third party packages I didn't use, which caused me to tune out and stop reading. As sometimes happens, writing that entry caused me to poke at Embark some more and now I have a somewhat better understanding of it and some Embark tricks I want to remember.

(Now that I've made the effort to read it, Fifteen ways to use Embark has a bunch of useful examples that use only built in packages. Also, part of my confusion is that Embark actually does multiple things.)

With Embark, you start with a 'something' (what Embark calls a target) and then Embark lets you to do an assortment of things to it; some Embark writeups describe this as a middle mouse button context menu. There are at least two ways to get the 'something' (cf). In a regular buffer, it's whatever is at point (or the region if you have one active); in the minibuffer, it's whatever you're entering or completing. To add a bit of confusion for regular buffers, you can often change what Embark is acting on. For instance, if the GNU Emacs point (cursor) is on a word, Embark can act on the 'word' (in various ways depending on what it thinks the word is), the sentence it's part of, or the paragraph. What Embark can do depends on what sort of thing it has as its current target, so it offers you a completely different set of options for the name of an ELisp function than for a file name (see Default Actions for a very large list).

(When Embark starts in a situation where there are multiple options for the target, it will talk about 'shadowed targets at point' in the Embark buffer.)

One important 'action' that I want to remember that's always available is M-x, ie run a suitable (Lisp) command, and in fact a lot of your regular keybindings will apparently work. Not all commands will work right when run by Embark, but most of the ones you probably want to run will; see How does Embark call the actions? for the gory details.

To use Embark, you need a target. One way to get a target if you don't already have one at hand is to type it into a buffer, but another, better way is to use the minibuffer, by triggering some GNU Emacs command that will prompt you for the type of thing you're interested in. This leads to some of my 'stupid' Embark tricks (which are apparently perfectly normal). For example, suppose that I want to toggle the value of a GNU Emacs setting variable. Embark provides this as an action on variables, so the quick way to do this is 'C-h v', which will start minibuffer completion for variable names, then when I've picked the variable, trigger Embark and pick 't'. Similarly, you can set the value of a variable via Embark rather than having to remember 'M-x set-variable' and then completing the variable name anyway.

A bunch of Embark documentation talks about using Embark in the minibuffer because you changed your mind about what you want to do. You start out doing C-x C-f to open a new file and then you realize you want it in a new Emacs window so you can see your current file and the new one at the same time, so you use Embark to switch the result of C-x C-f to 'open file in new window'. Some specific options you can switch to are explicitly available, but in general you can switch to anything, although it's on you to make sure that your minibuffer completion makes sense for what you're switching to.

The logical extension of this is to not bother using or maybe even remembering C-x 4 C-f for 'open file in new window' and always using C-x C-f and then Embark to get it. Much as with my 'toggle a variable' example, you (I) are using C-x C-f as a way to generate file names for Embark to act on. Anything that generates file names in the minibuffer would do, but C-x C-f is a harmless thing if you hit RET by accident instead of triggering Embark. This is an intended use of Embark, per this Fediverse post, which I'll quote a bit of:

[...] The pattern is that any command that prompts you for Xs becomes an X manager. [...]

Functions, variables, files and directories, buffers, GNU Emacs packages, and so on, you can trigger something that prompts you for one of them, use all your completion features to fill it in, and then use Embark. You could even build a collection of personal commands (and keybindings) that only prompted you for the appropriate thing and then did nothing with the result.

A related trick is that you can use minibuffer completion to complete things you're writing in regular buffers, through Embark's action to insert text from the minibuffer into the regular buffer. Do you want to insert a file name into what you're writing? Use C-x C-f to trigger filename completion in the minibuffer and then Embark's general 'i' action to insert the result in your text. If you already have a completion setup with good completion for regular buffers (as I do), this is most useful for types of completion that aren't offered for your current buffer. In text buffers, this will be most of them; in code buffers this is likely to be things like file names.

(For file names specifically you can get the same completion option with Cape, although using your completion at point setup instead of minibuffer completion. But the Embark trick works for absolutely anything you can trigger a minibuffer completion for, including custom things.)

Another trick is that in minibuffer completion, Embark can also act on the current completion candidates, applying some action to all of them instead of just to one of them, the way it would if you finished completion. There's a number of actions Embark provides for acting on these groups, including exporting the current set of candidates to a buffer where you can further manipulate them in various ways that depend on the types of things (and whether you do an 'export' or a 'collect'). Embark also lets you create ad-hoc collections of things for it to act on. I'm writing about this because I looked it up but I don't think I'm likely to use this particular aspect of Embark very much, because it seems pretty fiddly.

(VOMPECCC: A Modular Completion Framework for Emacs has a discussion of the advantages of these Embark collection buffers.)

Unfortunately, using Embark in text mode buffers is somewhat fiddly because Embark often has unusual ideas of what a text word actually is. If you're lucky, Embark decides that it's an identifier and offers you various useful options (and also highlight other occurrences of the word). If you're not lucky, Embark will decide that your word is some other type of thing with a restricted set of actions; for example, 'minibuffer' (as a bare word) will be taken as an Emacs Lisp library, which has only a restricted list of actions. As far as I know there's built in way to change the type of thing or add an option to act on it as another type.

Since this is GNU Emacs, we can use violence, which is to say we can define a new sort of target, call it a 'word', and add a keymap for it that has specific bindings we want. This requires following the examples of both adding a new target and defining a keymap:

 (defvar-keymap embark-word-map
   :doc "Keymap for Embark actions on plain words."
   :parent embark-general-map
   ;; TODO: What should RET do?
   "o" 'occur
   "$" 'ispell-word
   "'" 'expand-abbrev
   "p" 'embark-previous-symbol
   "n" 'embark-next-symbol
   "c" 'capitalize-word
   "l" 'downcase-word
   "u" 'upcase-word
   "H" 'embark-toggle-highlight)
 (add-to-list 'embark-keymap-alist '(word . embark-word-map))

 (defun embark-target-word-at-point ()
   "Target a word at point but only in text mode buffers."
   (save-excursion
     (let* ((start (progn (skip-chars-backward "[:alnum:]") (point)))
          (end (progn (skip-chars-forward "[:alnum:]") (point)))
          (str (buffer-substring-no-properties start end)))
       (when (and (not (string-empty-p str))
                  (eq major-mode 'text-mode))
         `(word ,str ,start . ,end)))))
 (add-to-list 'embark-target-finders 'embark-target-word-at-point)

(In an ideal world this might also look to see if it was in comments or strings in a prog-mode buffer, but that's too much work for this quick hack.)

My 'word' target isn't quite as deluxe an experience as you get with identifiers, because identifiers and symbols will also lazily highlight all other occurrences in the buffer. But possibly you don't want that for plain words.

(Embark does say that it's primarily for minibuffer stuff, it's right in the name: "Emacs Mini-Buffer Actions Rooted in Keymaps".)

Sidebar: Giving Embark a connection to Flycheck

Embark ships with a connection to Flymake, so you can trigger Embark with point on a Flymake diagnostic and get some useful bindings. Because I default to Flycheck, I wired up the same thing for Flycheck, and to save other people having to do the work, here it is:

 (embark-define-overlay-target flycheck flycheck-overlay)
 (defvar-keymap embark-flycheck-map
   :doc "Keymap for Embark actions on Flycheck diagnostics."
   :parent embark-general-map
   "RET" 'flycheck-list-errors
   "e" 'flycheck-explain-error-at-point
   "h" 'flycheck-display-error-at-point
   "n" 'flycheck-next-error
   "p" 'flycheck-previous-error)

 (add-to-list 'embark-target-finders 'embark-target-flycheck-at-point)
 (add-to-list 'embark-keymap-alist '(flycheck . embark-flycheck-map))

Add more Flycheck bindings to taste, those seemed to be the obvious ones to me.

One piece of technology news of the time interval is that Google is no longer going to be providing Internet search, it's going to be providing 'answers'. On the Fediverse, I had a reaction to that:

Given Google's apparent change to what Google Search will be, I'm wondering if I should even allow Googlebot to crawl my techblog. If they're going to AI slop up anything they tell people after a search, I'd rather have my writing excluded entirely rather than be garbled.

(I have no idea how much traffic I get from people using Google Search, and if Google is going to do this I feel like I should punish them by pushing people away from them, even a little bit.)

To echo what Paul Cantrell said on the Fediverse, we tolerated and even embraced Googlebot crawling our websites as part of a social bargain. Allowing Googlebot was a big part of how people found our sites and our work, which is to say through searches. That social bargain has been fading as Google put more and more things inline, but at least they were still providing links and directly showing our words (I know, usually).

But now Google has said out loud that Googlebot is just the front end crawling ingester to an LLM system, much like all of the other LLM crawlers that are hammering our sites. Google is no longer in the search business, where they provide links to people; they're in the 'answers' business (which is to say, the probabilistic text generation business). Much as with HTTP requests from cloud provider IPs, this raises the question of whether we should care about allowing Googlebot to access our websites or whether it's now a source of undesired crawl load, or at least of no meaningful benefit to our sites.

(A hasty clarification: by 'we' I mean people running small web servers and web sites, as with requests from cloud provider IPs.)

It feels practically heretical to say this; as recently as last November I was saying that people needed Google to crawl them. However, here we are. It no longer feels at all obvious that I'm going to get future benefits from allowing Google to crawl Wandering Thoughts. For that matter I don't know if I'm getting current benefits (well, if people are in general), since you can't really tell any more when people come to your site from Google Search.

(If you can still tell from the Referer HTTP header, that suggests that almost no one is coming here from Google Search. Googlebot is still (re)crawling portions of Wandering Thoughts on an ongoing basis; in fact, now that I look it regularly fetches the front page, although it politely uses HTTP conditional GET.)

(5 comments.)

One of the problems I have when talking about 'vim' is that in practice, there are several versions of 'vim'. I don't mean 'version' in the sense of release numbers (although Vim has lots of releases and things change in them over time), but more in how people approach and use Vim. Even more than GNU Emacs, I feel that Vim has turned into an editor that has multiple faces, so the 'vim' that you use may not be the 'vim' that I use.

(Which version or face of Vim you see depends in part on what your system decided to set as Vim defaults, unless you're advanced enough in Vim things to have a personal configuration that overrides them all.)

I've seen people talk about at least three broad versions of Vim:

• The Vim with a bunch of plugins that is a full blown code editor or even IDE environment with LSP support (eg, also) and everything else you'd expect from such a thing. My impression is that this is surprisingly popular version of Vim, and I can see why; if you like the Vim editing style and find that LSP stuff fits into it for you, why not?

• The Vim with syntax highlighting, smart indentation (including language dependent things), and other features that you'd expect of a competent programming editor (one without IDE style integration and features).

• The Vim that's only slightly different from true original Vi, 'merely' fixing the things that make Vi a product of its time, such as only having a single level of undo. This is the Vim that I use.

  (People disagree about what in the original Vi needs fixing, and also how, cf undo.)

The Vim I use doesn't have syntax highlighting or smart indentation and I wouldn't use it if it did (or at least if those were on by default). But the Vim that other people experience does sometimes have those things, not necessarily by their decision but because that's how Vim comes set up on that system. Someday, someone may ship a system where Vim is pre-configured to provide the full IDE experience with LSP support already added and so on.

One of the issues with this is that because Vim can be configured to be quite different things, it's not necessarily obvious to people (me included, cf, also) how to change the Vim they get by default on a system to a Vim they want. People new to Vim may not even know it's possible, and people who know it's vaguely possible may conclude that it's too much work and they might as well switch editors, either to another version of Vi or to an entirely different editor.

At work, we have a lot of basic servers. These basic servers are 1U rack servers, with unexciting amounts of disk bays, RAM, and CPU performance, because we buy them to do straightforward jobs (for example, much of our mail system hardware is basic servers). Right now, these are all x86 machines, but like a lot of people I'd love it if the x86 architecture had real competition (even if returning to the days of a multi-architecture Unix environment wouldn't necessarily be fun). However, these days I don't think we're likely to see basic 1U ARM servers that we're interested in, and why that is comes down to modern hardware and modern basic server power usage.

At this point, the largest hardware difference between a reasonably performing basic x86 server and a reasonably performing basic ARM server will be the CPU (and surrounding chipset). Everyone uses the same RAM, the same PCIe busses, and quite possibly the same supporting chips for things like Ethernet and so on. And everyone is going into the same rack form factor 1U boxes with very similar power supplies and physical sizes.

My impression is that at the basic end, a comparably performing ARM server CPU is going to cost about the same as an x86 CPU (assuming you can get the former). My further impression is that the ARM CPU isn't going to have a significant power and heat advantage under normal usage, especially since modern basic servers seem to have pretty low power consumption in general (also). This means that I'd expect a basic ARM server to cost about the same as a basic x86 server and generate about the same heat and power load. Even if the ARM CPU is a bit cheaper, it's only one component out of many (and let's ignore the current inflation of RAM prices).

More powerful ARM servers (such as those used by the cloud vendors, or the ones with powerful CPUs that you can theoretically buy) can have a real advantage in some situations over x86 servers, so if you use some of those you might want some basic ARM servers for lighter weight work on the same architecture. But if your environment is otherwise free of ARM servers, and the basic ones cost the same as x86 servers, and maybe you have a bunch of x86 servers already, there doesn't seem to be much to attract people to ARM servers.

I feel a bit sad about this, because there's part of me that thinks it would be nifty to have some ARM servers around, but it certainly makes life easier to only have one CPU architecture for in our Unix machines. x86 CPUs are perfectly good, and these days all of the other parts are basically the same.

(Given this, I'd be somewhat surprised if very many vendors even offered basic ARM servers, especially for a competitive price. There are people selling 1U ARM servers but they're high core count things and I assume they cost accordingly. I don't know if anyone is making ARM servers in the 16 to 32 core range (or less), although there are definitely ARM CPUs that are that small. Although even the larger CPUs can have issues, although apparently that's on an older, lower performing ARM CPU.)

PS: In the past I've read that x86 CPUs no longer really have a power penalty because of their architecture; my impression is that this is less because dealing with the weird x86 stuff got more efficient and more that these days it's dwarfed by everything else going on in your typical complex, out of order cores.

(One comment.)

My Emacs configuration seems to have more or less settled down again after a flurry of changes, so it's time to update my previous list of Emacs packages that I use, so that I can come back to this entry later and see how things have changed over time. A bunch of things haven't changed since last time so I'm going to put the unchanged stuff at the bottom.

Currently I'm using Emacs 30.2 everywhere so some of the things I'm mentioning here are now built in, which is why I'm not restricting this to third party packages. As before I'm going to exclude dependencies that are automatically installed by the Emacs package system (since I don't use them, I just have them around in the background).

In no particular order:

• In text-mode, I use Flyspell (although there's a gotcha there). I've turned off Flyspell's default binding to M-TAB, because I find Flyspell basically useless for actually correcting words. Almost all of what I use Flyspell for is simply marking words it thinks are misspelled or mistakes so I can spot them (and then fix them by hand).

• I now use Eglot as my LSP server environment, which is built in to GNU Emacs. I use LSP servers for Go, Python, shell programming, and to some degree with C (and technically I have YAML and Rust stuff installed, but I don't really work with either in GNU Emacs).

  (I'm not sure a LSP server is all that useful for shell programming, or at least the LSP server I have.)

• For programming diagnostics, I primarily use Flycheck but I also have a Flymake configuration. My Flycheck configuration has flycheck-eglot for integration with Eglot, and also flycheck-golangci-lint (for Go) and flycheck-mypy (for Python) as supplemental linters that I turn on when I want them. For Flymake, I've installed flymake-shellcheck and flymake-flycheck (for when I want to throw the kitchen sink at something in Flymake).

• For Go programming I use go-mode. These days I no longer use supplemental packages for diagnostics; instead, all of that is handled through my fly* package setup, which primarily means the diagnostics provided by the gopls LSP server.

• For completion, I use a whole collection of packages with their own sets of configurations. The current list of main packages is the more or less standard-ish stack of vertico, marginalia, orderless, consult, and corfu, plus completion-preview-mode.

• After a significant amount of wrangling, I managed to get Tramp set up for my unusual environment (see my notes). I'm using it for more things than I entirely expected.

I have three partially used packages for displaying diagnostics and other things in code buffers to make them more visible:

• flyover is a package I just discovered today as my best Flycheck substitute for Flymake's 'flymake-show-diagnostics-at-end-of-line' setting, which I like as an option for shoving diagnostics in my face when I want to be sure I see them all.

• I've installed sideline and supporting packages sideline-flycheck, sideline-eglot, and sideline-flymake in case I some day decide I want a noisy programming mode environment (Eglot or otherwise) that shows LSP code actions, diagnostics and so on all over the place, as lsp-ui could be set to do.

  (I'd like some easy way of filtering Eglot's LSP code actions in sideline because otherwise it's too noisy most of the time.)

• I also have flycheck-inline installed because it's a less obtrusive version of what sideline gives me in Eglot buffers (where the full sideline experience would show me all code actions as well).

Packages that I've carried over unchanged from the previous late 2023 edition are:

• embark, which is in theory a great way to do all sorts of things with a few keystrokes and in practice I mostly use as a handy way to do 'reflow this region' when writing email. I have embark-consult installed as well.

  (I feel as if I should learn more about embark and how to use it well, but there's always so much to learn and remember about GNU Emacs and my configuration.)

• Magit for creating basically all of my Git commits. I mostly don't use Magit for other Git operations, but I consider it essential for easy and flexible Git commits (for example, selective commits). I'll sometimes start Emacs purely to make Git commits with Magit.
• git-timemachine to let me step through historical versions of Git-controlled files in Emacs.

• diminish to turn down the noise level of Emacs' modeline. I configure and use it through use-package so I usually don't think about it.

• backward-forward for easy, web-browser like jumping backward to where I was when I follow a reference to something in lsp-mode. I wrote an entry about jumping backward and forward.

• which-key, which gives me a prompt of what my next options are in multi-key sequences; I find this very useful for things I don't use regularly enough to have memorized or wired into my fingers already. Which-key is now built in to GNU Emacs.

• vundo to give me an easy way to navigate backward through Emacs' sometimes unpredictable undo stack. I know that there are more elaborate packages, like undo-tree, but vundo is quite simple and meets my desires.

• smartparens to make it less error prone to write and edit Lisp (I no longer have it enabled for Python because I found it too irritating). Smartparens isn't perfect for Lisp, but it's broadly better than trying to do it by hand. I don't use any key bindings for it or any of its smart commands (or its strict mode), I just let it automatically insert closing things for me. Some of its rearrangement commands might make my life easier, but life is full of Emacs things to learn.

  (One area of Lisp where smartparens falls down is single quotes, which in my Lisp are most often not paired but instead used to quote symbols. So every time I write "'thing" in Emacs Lisp I have to remove the trailing quote afterward. I'll live with it, though.)

• try, a handy way to try out an Emacs package without going through the effort to add it and then remove it again.

Things I'm not really using but still have installed (both carried over from last time):

• expand-region is a little package to expand the Emacs region out to cover increasingly big things. I use it partly for exactly that, but also partly as a way of seeing where, for example, Emacs considers the current Lisp s-expression or defun to end; if I expand the region to the entire s-expression, I can just look. I have this bound to C-=, which maybe I'll remember this time around.

  (In theory this is useful, in practice I keep forgetting I have it.)

• fold-this seemed potentially useful and I put together some bindings for it, but in practice I don't seem to touch it. I was planning to use it in conjunction with expand-region (as a quick way of selecting a region to fold).

  Folding feels like something that might be useful for navigating files or seeing an overview of their structure if I can figure out how to use it. But I'm not currently convinced it's the best option for this for me, instead of things like consult-imenu (although I'm not using that either).

I used to have evil installed but I removed it because I wasn't using it at all, partly due to its clash with my Emacs reflexes.

Some of these packages are probably out of date or not ideal, since I set a number of them up some time ago.

(Most of these packages are installed from MELPA, which means I'm generally getting frequent updates on the ones under active development and more or less the latest development version. So far this hasn't been a problem.)

In GNU Emacs, completion of things is a complex subject. There are at least two sorts of completion (in the minibuffer and in buffers you're editing) and many options for how things work. There's a whole ecology of third party packages for changing how both sorts of completion operate, some of which have become built in to GNU Emacs over time. For various reasons (cf) I'm going to write down my current setup for this.

For minibuffer completion I use:

• vertico to show more completion candidates, with some significant customizations and custom code to complete MH folder names, and also some forced completion categories.

• marginalia to add more information about completion candidates, except that I've then arranged for completion of GNU Emacs functions and variables to show less information because I mostly want to see their docstrings, not type and argument information.

  (Doing this isn't a standard marginalia option and instead requires somewhat annoying hacks.)

• orderless to provide more powerful completion options, except I use orderless in an unusually minimal configuration. Orderless technically can be used for completion in buffers, but I don't have that connected up in any useful way at the moment.

• consult to provide more context on various sorts of minibuffer completion results. Consult needs or benefits from having some additional packages wired up, like consult-flycheck, consult-flyspell, consult-eglot, and consult-eglot-embark, although I don't necessarily remember to use them very often.

For as I type in buffer completion, I use:

• completion-preview-mode gives me a shell style completion environment, where I can hit TAB at any time to complete the current prefix (mostly) and I can see what that prefix is as a little ahead-of-cursor annotation. It's really great and is almost always all I really want.

• corfu for 'completion at point' and to some degree as I type autocompletion, across both non-LSP and LSP modes (which is a change from before). I have a relatively restricted Corfu configuration which deliberately dials down how in my face it is. I turn on Corfu as you type autocompletion in prog-mode buffers but not in text mode buffers.

A lot of my completion configuration has been stable for a while. The big recent changes were switching to only using Corfu instead of a mix of Company and Corfu, and my discovery of completion-preview-mode.

This is a lot of packages to customize completion, but that's the modern GNU Emacs way; you have relatively narrowly focused packages that deal with one aspect of a large GNU Emacs feature. If you customize multiple aspects, you wind up with a lot of packages (both primary packages like consult and also secondary packages like consult-eglot that extend the primary package and connect it to other things).

An extremely common pattern for our Ubuntu servers is that they have exactly two disks and we want those two disks to be set up as mirrored system disks, with a UEFI boot partition in each and the root filesystem in a mirrored RAID array taking up all of the rest of the space (and for /boot/efi to come from the first disk). If a system has a single disk, we want the single-disk, non-mirrored version of this; if the system has more than one or two disks, the installer shouldn't try to do anything because we have an unusual system that needs hand holding; it should either stop to ask for help or abort.

As far as I can tell, this isn't something you can readily express in Ubuntu's server installer or really anything else that Canonical offers, although the server installer is entirely capable of creating this layout if you set it up by hand. I'm not sure it's very available in any Linux server installer, although I haven't looked outside of Ubuntu.

(Some installers can run scripts that can rewrite the installer instructions, so you could in theory write a script that sniffed around the system, detected everything, possibly did some initial setup, and then rewrote the installer instructions to exactly have what you wanted. This is what I will politely call a little bit too intricate for us to want to try to build our own trustworthy set of scripts to do this.)

Why I care about this is that by hand disk partitioning is probably the biggest time consumer when installing machines using our customized server ISO (and it's tediously boring). It's also probably the most critical thing that stops us from having a fully hands-off network booting installer. It would be nice if installers would someday do better with built in, fully supported and tested code (as opposed to a dangerous script we have to write ourselves and that's hard to thoroughly test).

Unfortunately I don't expect that to happen any time soon. My strong impression is that Canonical's focus is on cloud installation, where people don't use mirrored system disks and typically have very simple and uniform disk names, so you can predict (for example) that your entire fleet will have its system disk on /dev/sda or on disks from a particular maker. (Or perhaps you automatically generate a per-system installer configuration using knowledge of that system's hardware that's pulled from your inventory tracking system.)

PS: I'm relatively confident that I could write a suitable dangerous script to rewrite the Canonical server installer configuration file to do this under the right circumstances. I'm also confident that my co-workers would rightfully reject us using that script, because the various dangers aren't worth it at our scale. Always remember, cleverness is a trap.

Sidebar: Our assorted disk naming

Our systems are increasingly split between systems that have SATA SSDs and systems with NVMe SSDs (and then I test on virtual machines with virtio 'vdX' disks), which means that some systems will have 'sda' and 'sdb' and some systems will have 'nvme0n1' and 'nvme1n1'. An extra complication for all systems is that if we're installing from a USB stick (instead of, say, network booting), the USB stick will appear as an unpredictable /dev/sdX name. So even with all sdX disks, it's not actually true that 'sda' and 'sdb' are the system disks; one of them might be the USB stick.

(7 comments.)

I recently read Simon Tatham's Nitpicking the shell history scene in Tron: Legacy, where one thing that surprised Tatham was the film using 'login -n root' to become root instead of 'su. This surprised me because I found that perfectly ordinary, and this turns up both a bit of Unix history and a difference between modern Unixes.

Plain 'su' can let you become another user, including root, but what it explicitly doesn't do by default is create a new login shell for that user. If you do 'su root', the new root shell normally inherits most of your environment, your current directory, and so on. Sometimes this is what you want and sometimes you really want a new login environment, and originally in Unix how you got the latter was to run 'login' from your existing shell session (and this meant that login was setuid root, like su).

This split usage of su(1) and login(1) is present in Research Unix V7 (and for login goes back to at least V3), where the respective manual pages clearly say that su doesn't change your environment or your current directory, while login's normal use (from a shell) is to 'change from one user to another'. Similar wording remains in the 4.2 BSD su(1), but in System III, su(1) picked up an option to make the new shell a login shell (and it even describes the mechanism) and login(1) lost the ability to be run from a normal shell. The 4.3 BSD su(1) picked up the System III su change, but login(1) can still be used from a normal shell, and I believe this continued on the BSD lineage in general.

As you might expect, all of the modern versions of su across Linux and the free BSDs support starting a login shell (cf the normal Linux su (also), FreeBSD su(1), NetBSD su(1), and OpenBSD su(1)). On Linux and OpenBSD, login isn't setuid root and so can't be used from a regular shell environment to become a new user; your only option is su. On FreeBSD and NetBSD, login is still setuid root and can be used to switch to another account with a login shell, although this usage doesn't seem to be explicitly documented in either's manual page. Illumos (the open source successor of Solaris) also still supports using login from a command shell, and explicitly documents this in login(1).

(OpenBSD making login not be setuid fits their general security posture, since a setuid login has been a vector for security issues in the past. I can't easily find out if Linux versions of login were ever setuid.)

PS: It's possible that login is still setuid on some Linux distributions. The normal util-linux login specifically says that it doesn't work from a shell session, but the shadow-utils login may still, and some distributions might enable that.

(This sort of elaborates on a Fediverse post I made.)

Sidebar: The early history of su

The su command goes back to V1 Unix, but at the time it was only used to let you become root ('superuser', likely the source of the 'su' command name). We don't have much from V2 (well, sort of [PDF]), but in V3 su's manpage moves to section 8 (for 'administrative commands') as su(8), where it stayed in Research Unix V4, V5 (per the V5 manual [PDF]), and V6. Only in V7 does su gain the ability to change to any user and its manual page was moved to section 1 (for general commands) as su(1).

(Potentially of interest is this reconstruction of old Unix manual pages.)

(2 comments.)

The news of the time interval is a string of local privilege escalation vulnerabilities in Linux (in part in the kernel). We very much need the security boundary of Unix logins, and some of these vulnerabilities are mitigated or blocked by various Linux kernel security modules ('LSMs') (cf), so I've recently been thinking if we'd use AppArmor, the LSM that Ubuntu supports.

(AppArmor didn't block as many of the vulnerabilities as a proper SELinux setup did, but SELinux needs distribution buyin and that's not what Canonical provides.)

We've traditionally disabled AppArmor because it's had issues in our environment of NFS home directories in our own locations for them (also). So let's assume that AppArmor magically works now for NFS home directories and other directories (or can easily be set with tuning knobs), and still provides meaningful security afterward. Setting up AppArmor for our environment will take some amount of work (cf), so the question is how much protection against local privilege escalation we get.

Roughly speaking, our systems fall into two categories; systems that normal people can access and run programs on, and systems that are purely for services (including things such as IMAP mail). For services, in theory we (or the people writing AppArmor profiles) can work out what the services should be allowed to do and not do, and thus lock things down against local privilege escalations in kernel systems that the services shouldn't be touching anyway (and other vulnerabilities, such as information disclosure from reading files the service shouldn't be accessing). However, this protects against an unlikely set of chained issues, where there's both a vulnerability in a service itself and then an additional vulnerability in the kernel.

(If these issues aren't unlikely, we have bigger problems.)

That leaves the systems where normal people can run their own programs (which are the ones where we really need the security boundary of logins). On these systems we have to assume that an attacker can gain the ability to run relatively arbitrary programs, either by compromising an account outright or through, for example, a compromised package that people are using in the code they're writing for their research (or a compromised editor extension, or etc; there are lots of ways in). Since people are effectively running arbitrary code, we can't protect ourselves by having AppArmor restrict what specific programs can do the way we can on service-based machines. Instead, we have to find and inventory kernel features that people will never legitimately use, and then block them through AppArmor rules.

(This is how a strict SELinux setup appears to protect against the recent vulnerabilities; a normal login is simply not allowed to use, eg, RDS sockets.)

The Linux kernel has a lot of features and facilities, although some of them are blocked off because we don't allow user namespaces, and people doing CS research do a lot of things, some of them at least unusual. Could an AppArmor profile (or a set of them) be written so that people would be allowed access to what they use and not allowed access to things that they don't? Probably (although AppArmor is more focused on programs than on people, well, logins). Would we be able to find an out of the box set of AppArmor rules and so on that worked? Maybe, and this depends on exploits not being found in areas that people pretty much have to be given access to.

If we had a reliable set of AppArmor or SELinux profiles, we might well use them because it would be easy enough. Without a reliable set of AppArmor profiles, I'm not sure we'd try to build some ourselves unless we were desperate. And if we were going to do the work, it appears that we might get more results for less effort through things like explicitly blocking all the loadable kernel modules for Linux socket types that we don't use.

(Some people even block all kernel modules that their current configuration doesn't use. I'm not sure I'd go that far, but I suppose you can always un-block things like the netfilter modules if you turn out to want to add some nftables rules later.)

(3 comments.)

One of the things we operate for the research side of the department is an old fashioned general purpose web server, where everyone has a home page area of their own in the traditional '/~<login>/' style (cf). This web server has been there for a very long time, and one of the decisions that was made very early on was that for security reasons, the web server would not NFS mount people's regular home directories from our fileservers.

The traditional Apache way to do '/~<login>/' home pages is to have some location under your home directory that's exposed as your web home page area; the traditional name for this is 'public_html'. One alternative is to relocate this to a separate directory tree, but this directory tree is flat, which makes it awkward to have different pools of disk space for different people (which is absolutely required for us). Since we didn't want to use people's regular home directories for security reasons and we couldn't put everyone in one directory, we did the obvious hack: people have a different, special home directory on the web server. These home directories are in special 'webdir' filesystems on our fileservers, and these webdir filesystems are the only NFS filesystems that the web server NFS mounts.

The result is that everyone actually has two home directories in two different filesystems (although those two filesystems will come from the same ZFS pool). They have their regular home directory filesystem, which is accessible on our login and compute servers but not the web server, and their 'webdir' home directory, which is accessible everywhere. To make this more convenient to people, we create a 'public_html' symlink in people's regular home directories that points to the 'public_html' in their webdir home directory. If people have personally run web servers, these and their support files also live in the 'webdir' home directory, for relatively obvious reasons.

(We have a special short name form of people's home directories, so on the web server this short form points to their web home directory. The public_html symlink combined with this means that '/u/<login>/public_html' always refers to your web home page directory tree no matter what machine you're on.)

Because everyone's web home directory filesystem is in the same ZFS pool as their normal home directory filesystem, the web server still depends on all of our ZFS fileservers. Since our web server is reasonably active (also, also), it tends to react very rapidly to any NFS fileserver hiccups.

PS: The web home directory security decision predates me, so I don't know why it was made, but in my view it's a perfectly sensible decision. In general you should probably assume that your web server can be coaxed into reading and disclosing any Unix file that it has filesystem level access to. If you don't like the implications of this, you need to arrange for it to have access to fewer files. A dedicated set of filesystems is one relatively straightforward way to do that.

I wrote a bit ago about how our servers seem to have surprisingly low power consumption, where I looked at IPMI based power consumption information to look at their current, typical power usage and found that several of them were sitting at lower power usage than my desktops. I'm still interested in typical or average power usage for reasons beyond the scope of this entry, but now I'm also interested in maximum power usage that we've observed.

One reason to be interested in maximum power usage is that servers are often given relatively high capacity power supplies. A 600 watt power supply is on the small side for even a 1U server, and my impression is that 800 watt and even 1200 watt PSUs are reasonably normal in basic servers. On the one hand, the vendors building servers don't know what they're going to be used for, so they're likely to be conservative. On the other hand, that's a lot of spare power for a system that is typically using, say, 24 watts. A PSU that is that far under its rating may not be all that efficient (although the one 600 watt server PSU I looked at had a 'platinum' rating).

Obviously a server's PSU has to support not just its typical power draw but also its maximum power draw; if you idle at 107 watts but reach 500 watts or more at full load (as one of our servers does), then you need that big PSU. But if the maximum observed power draw is substantially lower, the PSU looks more like overkill.

Because we only collect IPMI sensor information every minute, I can't be confident that we've captured the absolute highest peak usage for any of our servers. If a server has a high power draw for only fifteen or thirty seconds or so, it would have to happen at just the right time for us to capture that in a sensor reading. But if the power draw lasts for more than a minute, it becomes increasingly likely that we'll capture it.

With that said, the regular 1U servers I have reliable IPMI power usage from top out at around 155 watts for a few servers (over the past year), and our NFS fileservers, which are loaded with SSDs, are under 100 watts. These machines don't seem to be putting much stress on their PSUs, to say the least (I think they all have 600 watt PSUs or better, ie bigger). I don't know what to make of this, or if it matters much from an efficiency point of view. Some extra percent of 25 watts is not necessarily a large number, and I don't know if reducing the PSU from 600 watts to, say, 400 watts would improve things much.

(Even if it did, I suspect that there's not enough of a market for it to make it an economical option for basic server vendors. I have a feeling that most people's servers are more loaded and more power-consuming than ours are.)

(2 comments.)

For reasons outside the scope of this entry, I recently dug into how the Go runtime did (Unix) signal handling on 64-bit x86 Linux. When I undertook this quest, I decided that the easiest way to navigate through the code of the Go runtime was to use the code navigation features exposed by the standard Go language server, gopls. In the process I was surprised by just how good its code navigation was, even in the Go runtime.

On Linux, Go's signal handling talks directly to the Linux kernel rather than going through the C library. As you can imagine, this is relatively architecture and Linux specific, as well as being relatively specific to Unix. The result is a tangle of OS and architecture specific code in a variety of signal related files in src/runtime. The first challenge for code navigation is picking out the right ones that apply to the environment you're interested with; in Go this is handled through build tags, which gopls understand. So gopls had no problem navigating from general Unix signal handling to setsig() in Linux-specific code and the 64-bit x86 Linux definition of the struct involved.

But that was only half the puzzle, because I was looking into how the Go runtime receives signals. This is done by 'sigtramp()' and the related function sigreturn__sigaction(), and it turns out that these functions are not defined in Go. All you'll find in Go is stubs of them at the start of os_linux.go. But gopls had no problems navigating from the stubs to the actual amd64 assembly version, despite the fact that the assembly version has an odd name, and then it was able to navigate from the assembly version of 'sigtramp()' back to the Go 'sigtrampgo()'.

(It turns out that one area where gopls is currently limited for Go assembly language is finding references for assembly language symbols. Fortunately I didn't need that here, all I needed was 'find definition'.)

Code navigation among Go code is not surprising, because that's what you expect from a (Go) language server like gopls. What surprised and impressed me is code navigation into and out of Go assembly code, where I was expecting to have to resort to manual searches with (rip)grep. This is almost certainly a relatively niche feature, yet gopls has basic support for it. This support doesn't come from the standard Go library; instead it's implemented specifically in gopls in internal/asm and internal/goasm.

Another nice trick that gopls can do (that I just investigated) is navigate from an interface or an interface method to everything in your codebase that implements the interface. This is done through (of course) the LSP 'find implementation' code navigation action (in Eglot in GNU Emacs, this is 'C-c i'). Gopls will also navigate backward from a concrete thing to all of the (in-scope) interfaces that it implements (again using 'find implementation'). Slightly inconveniently, if your thing has a String() method, this will report a number of interfaces in the Go standard library. Gopls currently includes non-exported interfaces in the standard library, which is technically correct but extra not useful.

(Specifically, currently this will include context.stringer and runtime.stringer, as well as fmt.Stringer, the public version (and expvar.Var, which has the same shape but incompatible return value requirements). I assume the Go runtime and standard library has multiple versions of this interface internally to limit cross-imports.)

Update: My GNU Emacs 'C-c i' key binding for Eglot's 'find implementation' command (eglot-find-implementation) is a custom personal key binding, not a standard one. Oops.

So what happened is that I converted a big pile of (highly untyped) Python 2 to Python 3 recently, and then I wanted to experiment with typing-heavy Python LSP servers in GNU Emacs, so I decided to try them out by experimentally adding some type annotations to DWiki, the aforementioned pile of untyped Python (and the code powering Wandering Thoughts). The experience was educational and taught me some new things about type annotations, but it also firmed up my view that typed Python code is different than untyped Python code (although not quite to the extent that they create a different language, as I sort of felt before). There are idioms that are perfectly natural in untyped Python that are pretty annoying to deal with in typed Python.

One of these idioms is dictionaries with multiple types of values. For instance, DWiki has a dictionary that is basically 'a collection of information about the HTTP request'. The authentic type of the values in this dictionary is "str | bool | SimpleCookie | dict[str, str]", which is to say that values can be any of a string, a boolean, a HTTP Cookie, or a dictionary of string key/value pairs. Of course, individual keys in the dictionary have a fixed type for their value; for example, the key 'request-fullpath' only ever has a string value, so in untyped Python code it's natural to write something like:

if reqdata['request-fullpath'] and \
   reqdata['request-fullpath'][-1] != '/':
    [...]

If you do this in typed Python, your type checker will almost certainly complain that this indexing isn't valid for booleans and HTTP Cookies. You need to either check or type-assert that the value is a string.

In untyped Python, this is a perfectly decent data structure (although it might not be good style). In typed Python, this is a bad data structure that will cause you pain. There are ways around the pain that preserve the underlying dictionary, but they exist almost entirely to pacify the type checker. A proper data structure in typed Python is not multi-typed like this, or at least it's not multi-typed with a lot of keys.

(One way is to use typing.TypedDict, but if you have a lot of keys it gets painful).

There's a good reason for this insistence in typed Python, because right now there's nothing preventing me from putting in the wrong type of value for a particular key in this dictionary. I could slip up and set some key that's supposed to have a string value to a boolean, or a key that's supposed to have a dictionary to a plain string. Typing can't detect those errors because any of those are valid for the dictionary in general, just not for that particular key. A proper data structure in typed Python is one where the type checker itself can check your invariants, so string values are separated from boolean values and so on. This would probably also be clearer code.

This is a general issue for any sort of variable-typed container object, return values, or the like. I saw a similar thing when typing my program that uses the email packages; the email packages have old-school polymorphic API return values that typing is not fond of and that required type checks or casts. This is relatively valid on the part of programs determining typing (they're unlikely to ever do full flow control analysis to determine actual types), and is clearly part of the style of typed Python.

(Another case of this in DWiki is that I have a general caching layer that uses pickle to store and retrieve arbitrary objects. The callers know what they're storing and retrieving under a particular key, but this isn't visible in any types I could assign.)

As far as I can see, typing also changes how you want to structure multi-file code with classes and other data structures. In untyped Python such as DWiki, it's natural to have one source file declare a data structure, create an instance of it, and pass it as an argument to a function (or a class) from another file that the first file imports. In typed Python, this doesn't work so well. Because everything that either takes data structures as arguments or returns them wants to name the data structure in type hints, you need the classes for those data structures to be eventually be accessible in everything that touches them, which means a tangle of circular imports.

(This is different from forward references in that the code that accepts instances of these data structures will normally never import the code that defines them, cf.)

Circular imports work, technically (as I've sort of written about before), but they make me unhappy. I lack enough experience with typed Python to know the correct approach, but it certainly feels like one should define as many data structures as possible in low level files that are relatively standalone so they can be imported into everything without circular imports. I'm not sure how this works once you want to put methods on your classes that take other classes as arguments and so on.

(Mypy has some suggestions but its answers don't make me feel happy.)

Another practical issue I ran into was that DWiki has a stack of middleware functions to fiddle with HTTP requests. All of the middleware functions take a standard set of four arguments, each with a specific type, and I have enough of theses functions that going through and adding the appropriate type annotation to each argument for each function (and the return value) was clearly a pain (in my experiment I only did this for a few). I found myself really wishing for a way to say that the function as a whole had a particular type shape, which would automatically infer the argument and return types. I think the proper way to do this is to pass each function fewer arguments (ideally one), but I'm not sure I like it (and the four arguments aren't tightly coupled to each other).

(I also wound up feeling that I should create a 'types.py' file that had all of the basic type definitions that didn't depend on classes and so on. This would be things like the shape of callable functions, that 'data about the HTTP request' dictionary, and so on. Many of these are used in multiple files in DWiki and this avoids various sorts of annoyances. I don't know if such a 'types.py' file is considered a code smell.)

I don't regret my scratch experiments with adding some types to DWiki (partly because I learned more useful things about Python typing), but it's clear that doing it properly is somewhere between infeasible and impossible (and Python typing acknowledges that this can be the case). A reasonable typed version of DWiki would be structured significantly differently, and getting from the current code to any new type-friendly structure would be a significant rewrite (which would fix some old mess but likely introduce new mess).

(The semi-typed results of my experimentation are messy enough that I'm to discard that copy of the source code.)

(I said something about type hints on the Fediverse and some interesting things came up in the replies, eg.)

(2 comments.)

Some languages have to make do with one LSP server. By contrast, Python has an embarrassment of riches; I know of at least five modern LSP servers for it. I've recently been experimenting with some of them in GNU Emacs, specifically Eglot, so before I forget I want to note down my views. The five Python things with LSP servers that I believe are modern and current are python-lsp-server ('pylsp'), Facebook's pyrefly, Astral's ty, Microsoft's pyright, and technically Astral's ruff.

The easiest to talk about is ruff, because it's not intended as a full-featured LSP server that does everything; instead it only does code diagnostics and formatting, and you need another LSP server for code navigation. Currently Eglot doesn't easily support multiple LSP servers and code navigation is a lot of what I care about, so direct use of ruff is off the table for me. Also off the table is pyright, since I don't have any interest in touching a Microsoft Python project or finding out how badly it works with anything other than VS Code (although there's basedpyright as a less-Microsofted pyright option).

Python-lsp-server is my default choice and is a solid basic LSP server with the code navigation features I normally care about, along with support for code diagnostics through either or both of mypy and ruff (via python-lsp-ruff). Python-lsp-server is also what I'd call a 'quiet' LSP server by default, without a lot of stuff popping up and being filled in in Eglot. It's supported by the community and is probably going to endure, but it's written in Python (so it's not the fastest thing) and my impression is that it's more focused on code navigation than on type checking your code. My view is that it's probably your best option if you have a lot of untyped Python code, which is my normal case.

(So after playing around with both ty and pyrefly for some time, I'm probably going to stick with python-lsp-server most of the time.)

Both ty and pyrefly are strongly into type checking and type annotations, in addition to supporting code navigation. Both support 'inlay hints' in Eglot, which fill in known or deduced types for you (and can also attach names to positional arguments in function calls; ty defaults this to on, pyrefly to off). There are some differences in what types they fill in, for example ty will tell me 'Unknown' for types while pyrefly is silent about them (with no inlay hint), and I suspect that there are differences in what types they deduce for things. I don't have enough experience with Python type checking to have strong opinions on the general choice between ty and pyrefly. Both support more or less all LSP code navigation features (ty's LSP documentation, pyrefly's LSP documentation), with pyrefly currently having one more supported navigation ('go to implementations', which lets you find the reimplementation of methods in sub-classes, and now that I've tried it that's kind of handy and it's not currently supported by python-lsp-server).

(Eglot allows you to easily toggle inlay hints off and on with 'eglot-inlay-hints-mode', in case you don't like the noise of them but do want, for example, pyrefly's code navigation. I'm not sure how much unwanted type diagnostics and notes pyrefly or ty will spit out at you on untyped, anarchic Python code bases.)

As before, I think setting up Python LSP support in GNU Emacs is worth it, especially if you're working with typed Python and pick a good LSP server for this. LSP server code navigation is really quite nice and will work across files in your Python project (and pyrefly's support for 'find everything that overrides this method' is handy if you have that kind of code base).

(GNU Emacs can do some amount of code navigation in Python code without a LSP, but you want to create and maintain a tags table and in brief experimentation the experience is not as smooth and more annoying.)

If you want the most deluxe Eglot based Python LSP experience, I think you want to set up pyrefly with however many inlay hints you want. Since I slogged through the effort to determine what special Eglot configuration you need for this, I will save people the effort:

(setq-default eglot-workspace-configuration
   '([...]
     :python (:analysis (:inlayHints (:callArgumentNames "partial")))
    )
 )

As (sort of) covered in pyrefly's LSP documentation, pyrefly doesn't use its own name for these settings, it uses names that pyright apparently originated. Fortunately Eglot will send (all of) your settings to whatever LSP you're currently running, regardless of their names. I believe you can also configure this in per-project configuration files, which would also let you entirely disable pyrefly type checking in places where you don't want it (per the configuration documentation).

(Some bits of the pyrefly experience in GNU Emacs will get more deluxe in GNU Emacs 31, when Eglot will acquire support for reporting things like call and type hierarchies.)

Sidebar: A brief experience with basedpyright

I ran a little poll on the Fediverse and a surprising number of people (to me) turned out to use pyright or basedpyright, so I gave it a try. The result is, effectively, a failure for my code. Even code that I thought was well typed and free of problems came out full of diagnostics in basedpyright's default configuration. It does have more or less the same code navigation features as pyrefly, but for me the cost of getting them is too high.

But if you want to write extremely strictly typed and careful Python code, basedpyright will make you do it (assuming you make it have no errors and keep its strict default settings).

(The poll also suggested that very few people use pyrefly, which surprised me a bit.)

I've recently come up with what I expect would be a quite good anti-robot, anti-crawler tactic, which I will give the snappy label and summary of "robots don't POST". Simply require a HTTP cookie to see your web pages and then if visitors don't have the cookie, put up an interstitial page with a HTML form that requires them to POST it to get the cookie. All the form need is a "click me to get your entrance cookie", because right now, few or no robots or crawlers will make that HTTP POST request; they only do HTTP GETs. To distract bad crawlers you might need some other links on the interstitial page, optionally going to content tarpits.

(If you're going to do this in practice you'll want to exempt syndication feed requests and perhaps requests from bingbot, Googlebot, and so on. Although maybe not Googlebot any more.)

The obvious problem with this technique is that if people start doing it in any quantity, the "robots don't POST" thing won't last. Bad crawlers will start hitting POST endpoints for forms that just have a "click me" button, and then POST endpoints for forms that have an "I am human" tick box to mark or a field to fill in or whatever the elaboration people come with is, and so on. Bad crawlers are in an arms race with websites and this is a problem.

Arms races require two active participants. An inactive participant in an arms race usually loses by default. In today's environment with aggressively bad crawlers, you can't simply set up a website and walk away from it, not if you want it to survive; you're forced to participate in the arms race. Your website may be static but your operation of your website increasingly can't be, not unless you want to wake up one day and discover that you don't have a website, you have a smoking hole in the ground and perhaps a big bandwidth bill from your hosting provider.

I don't have any answers to this. Instead, it feels like this whole situation is another obstacle in the way of people having their own low-attention websites (after the comment spammers made it impossible to have your own low-attention comment system). Someone has to pay attention, so that's either you or someone you outsource it to, and that someone is most likely going to need to be paid sooner or later.

(There are exceptions, but they're rare. Also, if you run your own website you sort of have to maintain the software involved, but automatic updates (and static websites) have mostly made that easier.)

(6 comments.)

In some environments, you want to connect programs together with mutual authentication and encryption of their traffic (so each end can trust the other and the traffic is immune to easy eavesdropping). If the programs are talking to each other over TCP, there's a well developed solution for this in the form of mutual TLS (mTLS) (although you'll probably get to enjoy the fun of running your own private Certificate Authority). But if you're using UDP, things are less clear. When this came up recently in a Fediverse discussion I was a peripheral part of, it occurred to me that we already have an existing, well regarded, UDP-based mechanism for authentication and encryption in the form of WireGuard.

(Yes, there's QUIC, but that still leaves you with TLS and it gives you a reliable stream model instead of a UDP model.)

WireGuard is normally used to create general networking connections between two machines, a connection that other programs can use to pass whatever traffic they want. But in theory it doesn't have to be used this way. There are purely user-level WireGuard libraries, and if you have a suitable library, you can have the program it's embedded in receive and handle the packets from inside the WireGuard connection, without injecting them into the operating system and exposing them to other programs (and it can also send its own packets back). You're probably going to need your own user level IP implementation to make the WireGuard library happy, and you may want or need a user level UDP or TCP implementation to make handling your own traffic simpler, but all of those are available if you hunt around.

What you get out of this is a well regarded protocol with simple, straightforward authentication, and to some degree it handles out of order packet delivery and packet loss, which is presumably something you care about if you picked UDP to start with. You don't have to deal with the complexities of any variant of TLS, you don't need a private Certificate Authority, and you can always directly know what one program is willing to talk to, because you have a list of public keys.

The drawback of this is that you have to put it together yourself. If you can find a suitable QUIC library (eg), it should do all of this for you in a hopefully straightforward API that looks a lot like (TCP based) TLS. The one potential drawback of the QUIC approach is that I believe it's only a stream based protocol without UDP-like, out of order delivery (and possible packet loss). If what you want is 'an authenticated, encrypted stream but over UDP', then QUIC is probably more like this than WireGuard. If what you want is 'authenticated, encrypted UDP', then WireGuard might be closer to that than QUIC.

Sidebar: WireGuard, QUIC, and out of order packets

As far as I can tell from reading the basic WireGuard protocol summary, each WireGuard encrypted packet is independent. While packets are sort of ordered by a counter, WireGuard can handle them out of order and will accept them out of order within a window (cf). If you're sending UDP datagram traffic within WireGuard, I believe this means that your underlying system can still have the UDP properties of non-blocking, non-sequential receives.

As I understand it, a single QUIC connection carries multiple streams within it and each of these streams runs independently, so packet loss and delays on one stream don't affect other streams. However, I believe packet loss and packet reordering will block a single stream because it's, well, a stream.

(9 comments.)

In a comment on this recent entry, I was asked if we still considered Unix logins to be a serious security boundary. This is a sensible question; there are a horde of Linux local privilege escalation vulnerabilities going around right now (and one FreeBSD one for spice), and in general (some) security people have been saying for years that once an attacker had local code execution, the game was over. Our answer is that yes, we consider it a serious security boundary, and if that situation ever changed we'd need a drastically different system environment from our current environment.

Our current environment has shared NFS fileservers where people keep all their files and data, shared login servers for both general usage and compute, a (shared) SLURM computer cluster, and a reasonably flexible shared web server environment where people can run programs. While some people are still using our login servers interactively, others are running software (such as VSCode) that connects to them somewhat behind the scenes and uses them to run tools. All of this is critically dependent on the security provided by Unix logins; if Unix logins weren't a real security boundary any more, anyone on any of these machines could read other people's files or run programs as them.

Since these machines are all shared machines with multiple people logged in at once, switching to Kerberos authenticated NFS wouldn't solve the problem. If we assume that attackers can merely become any other person, then they can gain access to the Kerberos tickets of anyone else who's currently logged in and access their files. If we assume that attackers can compromise root, then all bets are off and once a person has used that machine it can't be trusted for any future use (since the attacker could have compromised programs to capture the login credentials of future people logging in).

Basically, if you lose the security boundaries of Unix logins, you lose shared machines. You need to create a new environment without sharing (or with sharing boundaries that people can't break out of). Today, it appears that the only way to do that securely is a separate virtual machine for each person, with Kerberos authentication to our NFS fileservers (given some of the Linux security issues, containers are clearly not good enough). I'm not sure how you manage a SLURM cluster in this environment, but it certainly wouldn't be the straightforward way we do it today.

This would be a drastic change for people here and it would also be a significant increase in resource requirements (since realistic virtual machines are much more heavyweight than even full login sessions). We couldn't leave 'your' virtual machine (or machines) running all the time (we have too many people using our systems for that), so you'd have to use some web interface to request it be started with some resource allocation. Managing, maintaining, and updating these virtual machine images and running VMs would be at least a bit painful, and people would probably experience more disruption in their activities. Some things would become effectively impossible, such as running CGIs on our web server.

(4 comments.)

One of the divisions in GNU Emacs people is between using Flymake, which is built into GNU Emacs and is well supported by other standard GNU Emacs packages such as Eglot, and using Flycheck. I've used Flycheck for a long time (cf) and recently tried using Flymake, which has given me some pragmatic opinions for my own usage.

(For non GNU Emacs people, Flymake and Flycheck both exist to present (and to some extent detect) 'diagnostics' about your code or whatever file you're editing.)

For me, Flymake and Flycheck are about as good as each other, at least in LSP based environments and Emacs Lisp. Flymake is better integrated into Eglot and can make errors more visible, Flycheck comes with more keybindings by default, and I go back and forth about how I feel about their modelines (after I diminished Flymake's verbose modeline name down to 'FlyM' and changed the colours a bit). Why I prefer Flycheck is that it's more flexible in one way that matters to me.

My particular taste with checkers is that by default I only want to see actual errors (or relatively strong style issues), but I want to have access to linters that express views I may not agree with in order to see what they say and maybe fix some things they complain about. This way I can keep my code free of real, core issues (that are reported by the error linters) and have a nice clear modeline showing '0' issues (and not have to remember how many baseline non-issues a file has), while still being able to conveniently see style issues if I want to consider them.

As far as I can tell, Flymake has no built in support for (easily) changing what sources of diagnostics it draws on. Things are just magically supposed to get it right, which is fine if they actually do but sub-optimal if they don't. One case where they don't necessarily is in Eglot, where as far as I know the normal diagnostics will only come from the LSP server you're running and will cover only what it provides. Even in cases where it's possible, changing what diagnostics you get from a LSP server isn't simple.

Perhaps because you can switch Flycheck checkers around, there are a bunch of third party Flycheck packages that support optional Go and Python style checkers (and some for other languages). Flymake has some third party checkers, but not really in the way Flycheck does (and what third party checkers it has can be rather out of date). The Flycheck situation is convenient and useful for me, because it means I can easily run (for example) golangci-lint against my Go code within the Flycheck framework with all sorts of jump to complaint support.

(There is an adapter to connect Flycheck checkers to Flymake, but as far as I know you're still left without a convenient way to pick your checker.)

Although Flycheck is my default, I've kept my Flymake configuration around and wired up some personal functions so that I can switch back and forth (either buffer locally or globally). Sometimes I flip over to Flymake to see what it says or use some of its other features.

(There's also Flycheck's comparison page with Flymake. A bunch of the differences that Flycheck lists aren't important to me, partly because I don't use GNU Emacs to edit everything in sight so the large collection of languages and configuration files that Flycheck supports aren't as important.)

PS: I'm dating this in the title because both Flymake and Flycheck have changed over time. My impression is that Flymake stagnated for a while, putting Flycheck clearly ahead in those days, but that things are more even today (especially in LSP environments, where both are getting the same diagnostics from the LSP server).

(One comment.)

Suppose, hypothetically, that you want to have a personal copy of the content of some website that you feel is important (to you). There are perfectly good reasons to want such a copy; websites go away all the time on the Internet, and not everyone is online all of the time. It's generally possible to do this (and it's certainly possible to do this with Wandering Thoughts), but there's some things the hypothetical you is going to more or less need to do. These things will be work, but that's the difference between successfully getting a personal copy and turning a brute force crawler lose and then getting ratelimited and blocked. It's also the difference between being polite and being rude, and hopefully you care about that.

(With the increasing decay of Internet search engines, you might also want to build your own personal index of useful website content.)

First, you need to work out the URLs for the real content of the website. Many websites of interest have some mixture of real pages and various sorts of indexes and other aggregations of those real pages, and it's not uncommon for the index pages to outnumber the real pages, sometimes vastly. Your personal copy of the website contents doesn't need all of those index pages, you probably don't want them because they'll inflate the size of your copy, and the website itself will probably be unhappy that you're fetching a ton of redundant index pages.

(The amount of index pages varies with site design. Static sites are usually much friendlier than dynamic sites because it's more work to have a lot of index pages in a static site.)

If you're extremely lucky, the website will have an accurate, up to date (XML) sitemap and will put a tag mentioning this in the HTTML <head> of its pages. If you're not so lucky you will have to manually look around to see if it has any particular index pages that you can mine for URLs (eg) and then work out what additional links and pages you need to also fetch to get what you consider a full copy (for example, to also get comments or 'talk' pages or the like, or to fetch images used in the web pages). In less friendly cases you'll have to go through a whole collection of category pages to accumulate the URLs.

(It's possible that the website supports paged syndication feeds and you can go back through its syndication feed to collect a full set of initial URLs, but I suspect that's not any more likely than a discoverable sitemap.)

Having accumulated your list of URLs, it's time to start fetching them, respectfully. Respectful fetching means doing two things: working slowly, and having an honest HTTP User-Agent. Working slowly means that getting a full copy will take a significant amount of time, but unless you think the website is going to go away tomorrow, you have that time. By 'slowly' I mean a request rate of one every 30 seconds or every minute, and if you get HTTP 429s or other indications of rate limits, you should slow down, even if you think this is absurdly slow. In my view, an honest HTTP User-Agent admits to what you're doing and optionally names the software you're using to do the fetching, because the web site operator cares much more about why these requests are happening than that you're using curl, wget, or whatever to make them.

(You especially shouldn't pretend to be a regular browser, or directly use a headless one. In these days of aggressive stealth crawlers, that makes you look extremely suspicious and may well get you blocked rapidly.)

Once you start fetching, you should monitor your fetching for problem indicators. Basically anything other than a HTTP 200 success may be a sign that either you have the wrong URLs or that you're in some way not welcome to do what you're doing. Continuing despite a spate of HTTP redirections or HTTP errors isn't particularly useful for your content copying project; you're only going to have to weed the results out of your copy.

(Also, continuing when a website is telling you 'no' is being rude. You're saying that your desires are more important than the website's views, and this generally makes you a certain sort of person.)

What all of this will get you is a personal copy of the website's content, possibly in addition to a skeletal set of index pages that you can use to navigate through it (you collected these pages when you built the initial URL set). It won't get you a complete archive of the website in HTML form that you could stick up somewhere else. A full website archive is a different thing, one that websites may be much more hostile to depending (in part) on how much redundant content you will wind up crawling in order to assemble your 'complete' version.

(Even if what you want is a full archive of everything, including index pages, starting with the important content first gets you the important content if something goes wrong.)

PS: Wandering Thoughts has a sitemap, which I bashed together many years ago to make Google happy and then found it was convenient for testing because it gave me a list of all pages that I really cared about the HTML rendering of. Interested parties can access it by putting a '?sitemap' on any directory URL. It's not (currently) in the HTML <head> of any pages because when I set it up, that wasn't really a thing. Given the modern web environment, I'm not certain I'll ever make it visible in the HTML <head> because I'm not certain I want to hand every abusive crawler a nice obvious map to the juicy bits.

(I have no idea how long it's been since Google accessed the sitemap; I suspect it's been years. But then, I increasingly don't care about Googlebot, although that's another entry.)

(2 comments.)

For reasons beyond the scope of this entry, I've wound up thinking about how stable or unstable the Unix landscape has been 'recently' (which means for more than a decade, and especially as compared with the 1990s and early 00s). I've written about aspects of this before, such as the fading out of multi-architecture Unix environments. In thinking about it more after my Fediverse post, I've come to feel that the Unix environment has still been changing but in places where I'm not as conscious of it.

The biggest change is probably the growth of cloud Unix, which I could characterize as "Unix machines on demand". In practice, cloud Unix is a whole new Unix environment that is quite different from traditional Unix, with different tools and especially different practices. Some of the practices are (sort of) extensions from old fashioned large scale Unix administration but many aren't really. I'm aware of cloud Unix and this gulf between operating it and what we do if I think about it, but I don't usually.

Cloud Unix practices spill over into what people want to do outside of the cloud, in the form of things like containers. Operating software through containers is quite different from traditional Unix system administration, especially if responsibility for the containers themselves gets moved from the system administration team to other people.

(There's also the idea of immutable systems created through declarative means, which isn't mainstream but also isn't a tiny corner any more. You can find plenty of people using Unix this way on servers and even desktops.)

I think that all of this has led to a significant change in how people experience Unix. Increasingly, Unix is either a desktop environment (not necessarily a graphical one, consider WSL in Windows) or a backend target; it's not something you explicitly (remotely) log in to very much. We've seen less and less direct use of our login servers and more use that is, for example, modern desktop IDEs starting remote sessions to run development tools on our servers. If VSCode could start SLURM jobs for people, some people here might never explicitly log in to our compute servers. I personally still log in to lots of remote Unix machines, but I'm increasingly an exception.

(I can't throw stones here since I recently carefully set up my desktop GNU Emacs so I could run remote LSP servers (and Git) through Tramp.)

A quiet but significant development is that after narrowing to x86 in practice for a while, Unix is moving back to being multi-architecture. There are a steadily increasing number of ARM servers and ARM devices that run Linux and other Unixes and that you'll find in the wild, primarily in clouds and as small Unix computers that you might put on your network to do specific jobs. It's plausible that some day we'll also get RISC-V servers and devices, or see ARM on general (Unix) desktops. People now routinely care about multi-architecture support for languages, compilers, distributions, and so on, where I think ten or twenty years ago that was a relatively niche concern.

(We've actually looked at small ARM-based Unix devices repeatedly and passed on trying to actually operate any of them for various reasons. Moderate sized, general purpose ARM servers don't seem to really be a thing so far, but maybe someday.)

In Linux, systemd is a drastic (and good) change on how init systems worked and how you interact with them, and makes that part of system administration relatively different from the pre-systemd days. Although I don't know when exactly it happened, the BSDs have gone through a similar evolution that regularized and improved the old ad-hoc BSD init system, making it rather easier to operate. This is probably the most dramatic change a system administrator from 2006 would notice if you jumped them 20 years ahead to today (and had them work with on premise servers without containerization).

There are certainly things that are part of my day to day use or at least administration of Unix that weren't there a decade or two ago. Even on old fashioned on premise servers, there's a lot more JSON and YAML than there used to be, partly because JSON has become the universal program-readable output format that everyone can agree on (and good tools, such as jq, have become widely available). But broadly, I feel that Unix has carried on being Unix and the experience of logging in and using the environment hasn't changed dramatically. If anything, different Unixes have become more similar, partly because lots of Unixes use the same programs (such as Bash and vim) and partly because Unixes have converged on common options for common programs (through both POSIX and pressure from people using them).

(Bash and vim aren't necessarily the default experience on all Unixes, but they're commonly available, partly because people want them.)

PS: The switch from X to Wayland is (still) a change that's in progress, but at the same time it's broadly supposed to be an invisible one to most people. Whether it should count as a change in Unix I will leave up to you.

Sidebar: My history with universal dotfiles

A long time ago I tried to have universal dotfiles for my shell environment across all of the multiple Unixes that I then had accounts on. The result was complicated, with lots of per-Unix and per-group settings. Today, I'm relatively certain that I could do a version for the surviving Unixes and system environments (and accounts) that had almost no conditionals. Some of this is through Unixes converging, some of it is through vendors with weird Unixes going away or becoming irrelevant to me (I'm unlikely to ever log in to another AIX machine, or a Solaris family one), and some of it through a relative convergence in how to administer machines.

(7 comments.)

I have a long standing personal program to display MIME formatted email messages in the terminal in a sensible way (it was mentioned in this old entry on my email tools and its comments). For a long time this was a Python 2 program, using the Python 2 version of the email package. Recently, I moved this program to Python 3 as part of my sudden enthusiasm for Python 3 conversions, using the Python 3 version of email and its sub-packages. In the process I have wound up with some notes and opinions on practical use of the Python 3 email packages.

(The Python 2 version of email had its own quirks and oddities, but I worked all of those out that hard way years ago, have mostly forgotten them since, and they're not interesting any more now that the era of Python 2 is over.)

The Python 3 email documentation will tell you that the modern interface for email messages is email.message.EmailMessage. The older email.message.Message is (theoretically) only there for Python 3.2 compatibility and you should ignore its methods and use only the EmailMessage methods. This is not entirely the case. If you look behind the curtain, you'll discover that many of the EmailMessage APIs for reading message contents are in fact Message APIs with masks on, and especially they're various masks for Message.get_payload(). That get_payload() isn't obsolete in practice matters, because it turns out that get_payload() is the only way to do certain things you (I) need.

As with decoding email headers, my strong impression is that the entire set of email parsing and message reading APIs are only really designed to deal with well formed email messages with fully correct MIME. This isn't what you find out in the real world, both due to programs being imperfect and also due to things like other mail systems sending you a bounce message that includes a message/rfc822 version of the original message where the other mail system has retained all of the message headers, including the Content-Type that says the original message was a multipart/alternative, but has replaced the entire body of the message with '(Body suppressed)'. As far as I can tell, there's no EmailMessage API that will give you (just) the body text of that (malformed) message/rfc822; your only way to dig it out is to use the older Message.get_payload() API.

(That bounce example is a real case that I've seen.)

At the same time, EmailMessage.get_content() is a handy API that does a lot of the work for you for things like extracting a de-mangled, Unicode version of a text part (or anything that's sufficiently text-like, although you will get back a bytes thing instead of a str and then decode it yourself). So I use get_content() as much as possible but some things have to fall back to get_payload(). The one thing I'm cautious about with get_content() is that it has a cheerful trust in the asserted character set encoding of the MIME part, when I'm pretty certain that some mail creation programs blithely assume you'll typically interpret stuff as UTF-8 (especially if it has no type specified, which in theory means ASCII).

(get_payload() will also probably give you heartburn if you're trying to use typing, but this is a general email problem with API typing.)

The email package parses your messages with stuff in email.parser, which has some additional notes on how it theoretically parses things. Some of these notes are experimentally false, especially the one for message/delivery-status. The actual story is in comments in the source code:

message/delivery-status contains blocks of headers separated by a blank line. We'll represent each header block as a separate nested message object, but the processing is a bit different than standard message/* types because there is no body for the nested messages. A blank line separates the subparts.

Although the actual text of a message/delivery-status part is plain text (admittedly in a specific format, in theory), the parsed version is a multipart EmailMessage object containing a series of text/plain EmailMessage children, where the actual contents are in the headers of those text/plain children (and the 'body' is empty). The best way to extract the actual contents as text to print or process them is to use EmailMessage.as_string() on each child. This is quite confusing if you expect a message/delivery-status to have obvious contents or to match the documentation (and EmailMessage.get_content() doesn't work right on the multipart parent object; this may be a bug that will be fixed at some point).

PS: The reason you don't want to use .as_string() on text or broken MIME parts is that MIME parts have headers, namely the various Content- ones, and .as_string() will give you those headers as well as the text you want. There's no option in the EmailMessage API to not get the headers.

Sidebar: Types for email stuff

Because sometimes I get enthusiasms, I added types to my program that's using email. It was somewhat painful and the kind of thing that you describe after the fact as "a valuable learning experience". In order for future me to not lose that learning experience, here's some notes.

My first problem was that often, mypy inferred that something was an email.message.Message instead of an email.message.EmailMessage; the latter is a subclass of the former. Much of this could be fixed with isinstance() to create type narrowing. I found the most convenient way to do this to be an assert(), for example:

prs = email.parser.BytesParser(policy=...)
m = prs.parse(fp)
assert(isinstance(m, EmailMessage))
[...]

Here I know that email.parser.BytesParser will return an EmailMessage because that's what my policy is set up to do (cf), but mypy can't see that.

A more involved situation is the return value of Message.get_payload(), which mypy typically typed as including 'list[Message]' when I know that what I have is a 'list[EmailMessage]'. Fixing this requires typing.cast():

def showalternative(p: EmailMessage) -> None:
  m = p.get_payload()
  if isinstance(m, str):
    [...]
    return

  assert(isinstance(m, list)) # for safety
  m = typing.cast(list[EmailMessage], m)
  [...]

You need to use typing.cast() to correct mypy's idea of the member type of a list or other container.

(Technically mypy and any other type checker that does similar inference. I don't know my way around the Python typechecker landscape, although I've wound up with a few of them installed.)

(2 comments.)

In a comment on my entry on universities, email, and the issues of running things in house, I mentioned that our departmental email system has a non-trivial cost in hardware alone to keep going. To better illustrate that, I'll describe all of the servers that our email system currently requires (because it's more than one). Some of these servers exist for historical reasons and may go away at some point, but many of them don't.

Currently, we have:

• A server as our external mail gateway (our DNS MX target). This is separate from other mail servers because it's much simpler to configure and operate this way.

• A server for the (FOSS) anti-spam and anti-virus software we use (and everyone needs some version of). This could be folded into the mail gateway server (and it was in our recent backup MX, but we weren't sure about the software's resource usage and system impact when we set it up. Keeping it separate also means we can move it to a new OS version for more up to date software without having to worry about any changes in new versions of the mailer that the mail gateway runs.

• A server for our central mail machine that handles all aspects of email to local addresses, which for various reasons (cf) can include sending email to the outside world. This machine doesn't store any email locally; instead, to simplify slightly, email lives on our general purpose NFS fileservers.

• A separate server to handle forwarding known spam to outside email addresses. We're required to support this by people using our email system and we found it necessary to put this work on a separate machine.

• A server to handle unauthenticated mail submission from inside our networks. Separating mail submission from the central mail machine makes for a simpler configuration for both (eg), and we historically started with only an unauthenticated mail submission machine.

• A fairly powerful server to handle IMAP and authenticated SMTP submission, which these days also has /var/mail (where all our inboxes live) on local storage and thus also acts as a NFS server.

• A server for a webmail frontend (to our IMAP server). We put this on a separate server than IMAP for multiple reasons, including resource usage and that it decouples the OS and packaged software version requirements of our webmail (for instance, certain versions of PHP and Apache) from everything else.

We've found it very important for practical reasons to use separate IP addresses for different sorts of outgoing email (also). We can do this on a single machine (and we do), but in many ways it's simpler to use separate machines for different sorts of email. It's also simpler to handle things like rate limits if we use different machines for things that need different rate limits.

All of these servers rely on existing elements of our general infrastructure, such as our general purpose NFS fileservers, our local DNS resolvers, and our system of propagating account information. I hope that at some point in the future our IMAP server machine will also wind up relying on our local OIDC identity provider (and indirectly on the LDAP server it uses), but that's currently not possible in practice. I'm mentioning these because a stand-alone mail environment would require some equivalent of all of them; you have to store mailboxes somewhere, get account and authentication information, do DNS resolution, and so on.

Most of these servers are 'basic' 1U servers, which these days means that they have 16 GB to 32 GB of RAM, a mirrored pair of SATA SSDs, a reasonable CPU, and traditionally cost a few thousand dollars each if bought new (their prices are probably higher at the moment). These specifications are good enough that we don't have to worry about the exact resource requirements of each server's job (although we made sure to give the anti-spam software machine 32 GB of RAM and a decent CPU). If we used smaller machines we'd have to be more careful; I'm pretty sure that not all of these roles would be happy with only 8 GB of RAM in practice (much less 4 GB). Basic 1U servers used to be cheaper, and these days we've got a stock of older servers that are good enough for these jobs. But if we were setting up a green field environment from scratch and had to buy all of these new, five or six servers (possibly plus a spare) would be a non-trivial cost.

(Because we're using the same sort of servers for these as we use for everything else, there's no dedicated spare for specific machines; we have spare server hardware in general.)

The one server that is an exception is our IMAP server. The current version has 64 GB, four relatively large SATA SSDs, a decent CPU, and 10G-T networking, and because it's so important we have a spare server ready to be pressed into use immediately in case of a hardware failure. The current hardware is old enough that we'd like to replace it, this time with more memory (so more things get cached) and NVMe SSDs instead of SATA ones. Unfortunately, in the current environment the price quotes we got are jaw dropping and unpleasant (especially since we have to buy two of the basic server to have a spare, although we don't need two sets of the NVMe drives).

All of this serves a department with somewhat over a thousand active people, about 1.5 TBytes of inboxes (if we talk about the likely uncompressed size; since we use ZFS for /var/mail, we have compression turned on), and an inbound mail volume that is probably around 10,000 messages a day. As mail system sizes go, this is modest.

(We have several thousand inboxes (and Unix accounts to go with them), but many of them are inactive for various reasons. The size distribution of inboxes is also extremely uneven, as you might guess.)

(Publication of this entry was delayed by me getting distracted and forgetting to actually publish it last night. I didn't realize it was still sitting in my drafts area until I noticed the stray editor window just now.)

(One comment.)

On the Fediverse, I had a simple sounding wish:

This is my face that I need a simple pagination program for Unix (show a page, pause, hit CR to show the next page) that is Unicode and emoji aware, so that it knows how long lines with them are. AFAIK less can't be used for this when I don't want it to ever clear the screen, just to keep printing the next page for however long.

(... because you have to know how long lines of text are so that you know when you've printed a full page.)

(So what I'd like is 'cat, but paginated'.)

This sounds like a simple, easy wish. Some of my readers are now laughing flatly, because it's not. In fact I believe it's impossible to write a simple general program to do this; you need either terminal program specific knowledge or to do some relatively extreme tricks as you print text.

Once upon a time, physical terminals and thus terminal programs were simple. They showed a set of characters in a monospaced grid, commonly with bytes mapping one to one to displayed characters (I'm ignoring DEC's double-sized character escape sequences). In this world, 'cat but paginated' is relatively simple, and indeed I have a program that does exactly this job; the only real complexity is handling tabs (where you have to work out what the next tab stop is in order to correctly track the width of the line).

(You have to track the width of a line because you need to know when the line you're printing spills over to a second physical line despite the lack of a newline character.)

The first problem that terminal programs give a pagination program in the non-Latin world is over-sized characters. Latin text has relatively simple character shapes that are easy to read at modest font sizes, but other scripts and other sorts of characters have much more complex shapes that are hard to read if you squeeze them into the same monospaced grid block as a Latin character at a given point size. So some of the time, some terminal programs don't; they render the characters larger. Which characters are rendered larger? It depends on the terminal program and, I think, the font (and certainly the character; see Let's Stop Ascribing Meaning to Code Points).

The second problem is emoji. Emoji are one of the common cases of Unicode characters combining together, or more exactly I should say Unicode code points. Famously, many flag emoji are actually two emoji put together. For example, the Canadian flag emoji, 🇨🇦, is the 🇨 emoji followed by the 🇦 emoji (CA is the ISO country code for Canada). Whether this renders as a Canadian flag or as C followed by A depends on whether the terminal program and the font rendering environment knows about this specific combination and is willing to turn it into a flag.

(There can be multiple reasons for not rendering an emoji flag as a flag, including that sometimes flags are new and sometimes flags are politically charged, for example 🇹🇼 or 🇵🇸. I would not be surprised if in some environments, one or both of those flags is not rendered as a flag, but as two emoji characters.)

As a practical example, in my X environment the only terminal program that combined 🇨 and 🇦 to make a Canadian flag is konsole. None of xterm, rxvt-unicode, or to my surprise gnome-terminal did (gnome-terminal does render many emoji, but apparently not flags or even emoji characters). What this means is that how many displayed characters this sequence takes up depends on the terminal program. A pagination program that assumes it's some fixed width is guaranteed to be wrong some of the time.

(Emoji rendering can also be an example of wider character rendering. In konsole, 🇨 is as wide as two regular Latin characters; in the other three terminal programs, it's single width. My graphical GNU Emacs also combines emoji characters to make flags and displays emoji characters as double width in a monospace environment. In gnome-terminal, emoji that are displayed properly (as emoji) are typically double width.)

If you want your pagination program to strictly print output without manipulating things through cursor positioning, I'm not sure what a good way to handle this is. From where I sit, it certainly looks like a program that satisfied my simple sounding wish would have to hard code knowledge of how various terminal programs I use render emoji.

(I'd be remiss if I didn't point to It’s Not Wrong that "🤦🏼‍♂️".length == 7.)

PS: The 'less' pager seems to be able to cope with this, so it's possible in general if I'm willing to give up my quixotic wish for a 'cat with pagination' instead of something that clears and overwrites the screen, ruining my scrollback.

PPS: Arguably the correct place for this sort of pagination is in the terminal program itself, but xterm doesn't do that and I'm very attached to xterm. Also, I'm not sure if any existing good X terminal program does this today.

(11 comments.)

For reasons beyond the scope of this entry, today I was curious how much power our servers were using. If you have sufficiently fancy PDUs, I think you can get per-outlet measurements that are trustworthy, but we don't have such PDUs. Instead, the best I can do is look at the information that some of our servers report through IPMI. I'm not sure how accurate the IPMI information is, but at least some of the numbers seem plausible, so I'll assume that it's not massively off. The results surprise me with how low they were.

Most of our servers are basic 1U servers with a pair of SATA SSDs. They're typically not very active, and the not particularly active servers that report IPMI power usage are reporting anything from 22 watts to 26 watts right now. A few servers also have a pair of HDDs, and one has four SSDs and is typically active; all of them report 44 watts right now. Our NFS fileservers currently have 24 SSDs in each and are reporting a range of power usage from 47 watts to 62 watts. One interesting case is our perimeter firewall, where we have the active server and an identical running hot spare, both 1U servers. The active server is at 52 watts (and handling about 40 Mbytes/sec of network traffic); the hot spare is idle at 26 watts.

We have a few compute servers that report power information through their IPMI, and the ones that are currently active are reporting the highest power usage. However, even this isn't spectacularly high. The most power hungry machine is a GPU SLURM node, where its IPMI is reporting 330 watts of total power while its GPU is claiming about 166 watts of power draw (its CPU is busy too).

Some of our servers don't report power usage in their IPMI sensor data but will report it through the web interface of their BMCs. I checked two of them, both powerful 1U servers that are essentially identical to each other. The one that is our primary login server is reporting an average of 149 watts and a peak of 195 watts over the past hour. The SLURM compute node, which is currently in active use, averaged 501 watts in the past hour with a peak slightly higher (when not in use it appears to idle around 107 watts).

One of the reason these numbers surprise me is that many of the idle numbers are lower than my desktops. I have a mental image of servers as not being particularly low power or power efficient, just as they're not particularly quiet, but that seems to be wrong. I suppose it's not too odd that people making 1U servers care about power usage and power density, since that's definitely a concern in general in data centers, it just hadn't really occurred to me before.

(Our own use of 1U servers is not particularly constrained by power and cooling.)

(3 comments.)

Every so often, I want (or need) to make modifications to programs in an Ubuntu package, and often the programs are written in C (and these days I'm using dgit to manipulate the package). One of my challenges when I do this is that I generally don't start out knowing where and how to change the code to do what I want; instead, I have to navigate around an unfamiliar code base and work out enough of its structure to find the specific bit of code I need to change.

These days, the dominant way to get smart code navigation and other code knowledge things is through LSP servers and clients. A variety of modern and semi-modern languages have LSP servers that you can immediately use in your editor of choice and then navigate around random code bases with handy features like 'find definition' and 'find references' (for example, Go, Python, and Rust). Unfortunately, C isn't such a language. In the general case, understanding C code requires knowing how it's compiled, and that means you often have to tell C LSP servers this information. Well, specifically you have to tell this stuff to clangd, the dominant LSP server for C and C++.

(There's also ccls, which may work out part of this information on its own, but it seems to be less popular and I have no experience with it.)

Fortunately for people like me, there is a simple way to gather this compilation information even if the program's build system doesn't do it for you, and that's Bear (which is available as a standard Ubuntu package for extra convenience). Bear operates as a front-end on however you normally build your program; you build your program (or collection of programs) with 'bear -- <build command>', and Bear monitors compiler execution and records everything. This is slower than a normal build (sometimes significantly so), but you get a compilation database out of it and then you can use LSP tooling to jump around the source code.

(My understanding is that gcc, clang, and so on can generate this compilation information if they're asked, and modern build systems often ask them to do so, but an old fashioned build system using things like 'make' won't include the magic compiler options necessary. Possibly you can include them yourself by hand, but Bear takes care of the work for you.)

Somewhat to my surprise, Bear not only works with programs built by 'make', it also works when you build Debian or Ubuntu packages under Bear with 'bear -- dpkg-buildpackage -uc -b'. If you're building a substantial package (such as Dovecot), you're definitely going to notice the slowdown, but you do get LSP based code intelligence out of it (and you only have to do this once, not every time you change the code).

(Under some circumstances you may have to edit the generated compile_commands.json to take out gcc options that clang doesn't support, but fortunately the JSON file is in a human friendly format where each compiler option is on its own line. Possibly there's a way to manipulate the Debian/Ubuntu package build process to not use such options in the first place.)

Building Debian and Ubuntu packages contaminate your source directory, so once you've run a build under Bear to generate the compile_commands.json file, you need to move the file to safety and then reset your source directory somehow. If you're using dgit (which I very much think you should be), I believe this can be done with a variant of the standard dgit source directory reset instructions:

git clean -xdf -e compile_commands.json
git reset --hard

The process I suspect I'm going to follow in future dgit modifications of Ubuntu packages is to set up the package with dgit, build it once under Bear in unmodified state, rm the generated .deb and .ddeb files, and then start poking around the source code with LSP intelligence to find where I need to make my modifications (and then commit them and do a dgit build as usual).

(This elaborates on some Fediverse posts.)

(3 comments.)

DWiki is the pile of code that underlies Wandering Thoughts. It started out many years ago as a Python 2 program (partly because there was no Python 3 at the time), and it stayed that way for a long time, making it the most significant and by far the most substantial Python 2 program I still cared deeply about. Years ago I said I'd port it to Python 3 someday and somewhat to my surprise, that day has now come (well, it came yesterday).

The direct trigger was discovering that Python 3.13 had dropped 2to3, which made me feel that I should run 2to3 over DWiki's current Python 2 code base while I still could (I had an old conversion from many years ago, but that converted code base was very out of date). One thing led to another, as it often does with me, and I wound up doing a full port and then putting it into production, which is to say serving this blog. I suspect that part of me just felt it was time.

(The 2to3 removal is in the Python 3.13 release notes, and it comes after 2to3 and its infrastructure were deprecated in 3.11 for reasonable reasons.)

As I expected years ago, the stuff that 2to3 could handle was the easy part. Much of the actual work of the port was sorting out the boundary between Unicode strings and byte strings in a Python 3 world. Some of this would have been easier if I'd found PEP 3333 earlier and followed it in my own discount WSGI implementation, but a bunch of it I had to find the hard way, by trying things and having them blow up, sometimes in production.

(I wound up in the same place as PEP 3333 just from the inherent requirements of the web. For example, the HTTP Content-Length is in octets, so if you're using it to read a POST body, the object you're reading from has to be providing bytes. And it turns out that you can't write HTTP headers to a text mode file object because that will turn \r\n sequences into \n, which will make things unhappy with you.)

Not all of the changes were at the IO boundaries of DWiki (and the IO boundaries themselves weren't always simple or obvious). Python 3's handling of cryptographic hashes requires bytes, which rippled through to several places where I use them in DWiki (and the hmac API changed a bit, which wasn't fixed up by 2to3). Python 3 also really wants your regular expressions to be in r"..." strings, because otherwise it will complain about you using regular expression backslash escapes like '\s' that aren't string backslash escapes.

I don't have a DWiki test suite, but long ago I built scripts that would crawl and collect all real pages from an old and a new version of DWiki. I originally used these to check for changes in how pages got rendered when I changed the wikitext processing code (often I wanted no changes), but this time around I was able to use them to verify that the Python 3 DWiki could at least render all existing pages into essentially the same thing (there were \r\n sequences that turned into \n instead of being passed through, but that's probably a good change). But that still left things like writing comments, and also the two sets of code involved in how DWiki runs in production instead of in testing.

I probably wouldn't have tried to do this if I hadn't had a relatively substantial block of free time. It took me more or less all day yesterday to get up to the current production state, with a lot of back and forth, experimentation, and tweaking. There was a lot of code and problem context that I might not have retained if I'd had to slice my work up into half hour or hour long chunks of work, and once I started running the Python 3 version as the live server I was relatively committed to fixing any problems that came up on the spot.

(I could have rolled back to the Python 2 version but it would have been at least a bit awkward for various reasons, including a pickle format change.)

The current Python 3 DWiki code still needs additional cleanups, partly to undo unnecessary 2to3 changes like changing 'for ... in dct.keys():' to 'for ... in list(dct.keys()):'. But it's running stably now for, well, not quite 24 hours yet but for at least a bunch of all of the typical traffic that Wandering Thoughts gets. Probably there aren't any remaining Unicode conversion issues, although re-reading one of my old entries makes me feel I should audit every use of EnvironmentError when dealing with files.

(2to3 appears to always put list() around things that changed to return generators in Python 3. Sometimes this is important, but it's not necessary if the result is only being used in a 'for'.)

I also want to think about what Unicode error handling to use in various circumstances, although these days I'm inclined to be draconian. For example, if someone tries to write a comment with invalid UTF-8, I probably don't want to backslash escape the invalid bits, so the default 'replace' handling is fine (in my case, this comes from using urllib to decode POST bodies). And currently all of the existing content in Wandering Thoughts is UTF-8 clean, at least as far as I can tell.

(The whole Unicode and bytes issue is something where types would be handy (or an option to turn off all of Python 3's implicit conversions), but adding typing to DWiki's 'originated in Python 2' codebase is both a lot of work and also extremely messy, because it uses things in ways that mypy is already unhappy about.)

PS: The Github version of DWiki is now significantly out of date and I'm probably not going to update it for reasons that don't fit in the margins of this entry.

Sidebar: The Python 3 WSGI rules in a nutshell

To summarize PEP 3333 in my own way, HTTP headers are Unicode strings, ie str, but must be limited to iso-8859-1 characters (at least when you write them). The wsgi.input file object produces bytes and your HTTP response body is also bytes. In a CGI environment, you read from sys.stdin.buffer and your WSGI CGI implementation writes to sys.stdout.buffer (including the headers, after encoding to iso-8859-1).

If your WSGI implementation is talking to a network socket, you can and must leave the network socket as a binary file object. In my case, this generally means wsgi.input is created with 'os.fdopen(fd, "rb")'.

For a while, one of my little irritations with my Emacs environment was that sometimes, when I fired up Emacs to edit some code and then quit out of it, Emacs would complain that there was still an ispell process running and ask me what to do with it. This was especially mysterious to me as I don't normally use flyspell-prog-mode (I find it too irritating for general use). Recently I got sufficiently irritated to use a combination of the ELisp debugger and strategic '(message ...)' usage to track this down, which initially looked like one issue and actually turned out to be another one that I discovered only as part of writing this entry.

One of the major modes in GNU Emacs is text-mode. I have a text-mode hook, probably like many people, and one of the things it does is turn on flyspell-mode in that buffer, which causes flyspell to invoke ispell and thus start an ispell process. It's also my custom from long ago to set the default major mode of buffers to text-mode (the out of the box default is fundamental-mode). If I'm editing something and it's not program source code, it's almost always text and having to say 'M-x text-mode' all the time is the kind of annoyance GNU Emacs is designed to erase.

When I used debug-on-entry to find out where the ispell process was starting from, it pointed to my text-mode hook. At first I theorized that code buffers were starting out in the default mode (and thus triggering my text-mode hook) before being switched to their proper mode, but strategic use of '(message ...)' in my text-mode hook revealed that it was actually being triggered on a scratch buffer for Flycheck. So I switched my theory to Flycheck creating scratch buffers without specifying their mode, so they would up in the default major-mode, which for normal setups is fundamental-mode but for me is text-mode, triggering my text-mode hook and starting ispell.

Except I looked at the Flycheck source and this is wrong. Here, let me quote a small bit:

(define-derived-mode flycheck-error-message-mode text-mode
  "Flycheck error messages"
  "Major mode for extended error messages.")

Flycheck explicitly derives the mode for some of its scratch buffers from text-mode, which of course means that they run text-mode hooks. This is a perfectly reasonable thing to do in general, since text-mode is the appropriate mode in general for, well, text, but it leads me to today's GNU Emacs learning experience which is that text-mode hooks may run in surprising buffers, not just text files I'm visiting and editing. I shouldn't put anything in my text-mode hook that I want only for real text files that I'm editing, at least not without guarding it somehow. One of those things is flyspell, not just because of its side effects of starting an ispell process but also because I don't particularly want flyspell to mark 'misspelled' words in, for example, Flycheck diagnostics.

(Flyspell's markings also get in the way of mouse based copy and paste.)

My solution was to guard what my text-mode hook did so that it only happens in buffers associated with a file:

(defun cks/text-mode-hook ()
  (when buffer-file-name
     ....))

It's possible that some day I'll want my text-mode setup in an anonymous buffer, but until that day I'll leave such scratch buffers alone. I could probably do a bit better by looking for buffer names that start and end with * (this is the usual GNU Emacs naming convention for explicit scratch buffers), but that would take a bit more work.

(Although not much more, now that I've found string-prefix-p and string-suffix-p.)

It's not uncommon that people using filesystems want to map from an internal object number (an 'inode number' for normal filesystems, an object id or object number in ZFS) to a path. ZFS itself wants to do this efficiently for things like 'zfs diff' and the 'zpool status' report on what files are damaged. To help with this, ZFS stores the likely parent object for every normal filesystem object. If you use zdb to do a sufficiently verbose dump of any particular object, you can find this as the 'parent' attribute.

If you want to do this mapping yourself, you can use zdb or something like it to manually follow these 'parent' pointers (and also look up the name of everything in its parent directory). However, that would require high privileges, and ZFS doesn't want to make things like 'zpool status' require that, so the kernel and libzfs expose an API for this. In libzfs, this is 'zpool_obj_to_path()', which uses the kernel's ZFS_IOC_OBJ_TO_PATH ioctl(). Because it's intended for internal usage, this API doesn't take a pool and filesystem name (in addition to the object ID); instead it takes a pool handle and a dataset ID. It's up to callers, such as 'zpool status', to do the mapping.

(One reason you might want to go from an inode number (object id) to a path is that various things only give you inode numbers, such as NFS v4 locks on Linux NFS servers. Or you might have NFS activity tracing software that can only reliably report the inode number of files and directories that people are using heavily.)

In OpenZFS, years ago someone wrote a command that used this libzfs API to do all the work for us, zfs_ids_to_path (also). Like the API, this requires the dataset ID. Helpfully we don't need to use 'zdb' to get this; instead we can ask 'zfs list' for it. This gives us:

# zfs list -o name,objsetid ssddata/homes
NAME           OBJSETID
ssddata/homes       431
# zfs_ids_to_path 431 1920047
/homes/cks/.rcenv

Illumos and FreeBSD don't ship a version of zfs_ids_to_path, but the source code is sufficiently small and self contained that you could probably compile it yourself.

(Although my test FreeBSD 15 instance doesn't have the libshare.h header that's needed by libzfs.h, presumably through a packing mistake.)

If you needed to do this frequently and found it annoying to look up the dataset ID every time, I believe that it wouldn't be too hard to work out and write the code you needed in order to go from a name like 'ssddata/homes' to a pool object and a dataset ID. Sorting through, for example, the source code for 'zfs list' might take some work (there's a whole collection of callbacks and so on), but it's doable (and perhaps someday people will write a slightly handier version).

(The lazy person can write a front end script today that combines 'zfs list' with zfs_ids_to_path.)

The Linux kernel's netconsole is a kernel module that will "log kernel printk messages over UDP" to a remote system, which makes it another form of kernel (message) console. These days it can be activated either on boot or after boot, and in the past I've had mixed views of it. However, I recently had a nice experience with netconsole that's left me more well inclined to it in specific situations.

A while back, my home desktop started locking up every once in a while. Several years ago my home desktop had a somewhat similar problem that was due to hardware issues, but the lockups this time were different, in that the machine would lock up for a bit and then reboot on its own. Local logs showed nothing, but I happen to have another machine sitting around so I thought I might as well try netconsole again. These days netconsole can be enabled on the fly:

modprobe netconsole
cd /sys/kernel/config/netconsole
mkdir heedra
cd heedra
echo em0 >dev_name
echo 192.168.X.Y >remote_ip
echo 1 >enabled

(This other machine is called heedra for obscure reasons.)

On the other machine I ran a simple script to capture output inside a screen session:

#!/bin/sh
while :; do
   nc --recv-only -u -l 6666 |
      tee $HOME/work/h-logs/netconsole
done

(The advantage of --recv-only is that nc won't complain if I hit CR a few times in the screen session to create blank lines, so new messages are more obvious.)

After a while, my home desktop locked up again and rebooted soon afterward. When I checked the netconsole log file on the other machine, I discovered that I had actually captured kernel log messages, and reasonably useful ones at that.

The kernel logs revealed that this appears to be a kernel 'soft lockup', where all cores had gone to 100% system usage during what appears to be TLB flushes or cross-core kernel communication. In several of the kernel stack backtraces, bpf_trace_run4 appears, so I suspect that there's an uncommon eBPF locking race or issue that's infrequently tickled by the eBPF metrics gathering programs I normally run on my desktop.

(It's probably not from the eBPF programs systemd uses for network access control, since those are used widely.)

Capturing these kernel messages doesn't give me a solution, but at least it gives me a way forward if the lockups get too frequent and annoying (I can try disabling my eBPF metrics collectors). And I couldn't have gotten these messages with anything else except a serial console, which I don't have available on my home desktop and anyway would have needed a second machine in physical proximity (which is awkward in my home setup).

My understanding is that netconsole isn't quite as reliable as a serial console for getting last gasp kernel panic messages out, since you need more kernel pieces to still be working to transmit network packets. But it's more reliable than anything short of a serial console, and serial consoles are generally in short supply on modern desktops and desktop-like things (including hand-built SLURM nodes). For one off, small scale use my listening script would be fine, although if we needed to use it on a larger scale, we'd need some infrastructure to collect netconsole logs from multiple machines.

(Some suggestions for that are in the comments on my earlier entry.)

Suppose that you are a Python code reformatter, and someone hands you the following snippet of Python code to act on:

if something:
    blah blah blah
    [...]
    final-line
some-statement

[... more statements ...]

Here's the question: should you reindent 'some-statement' so that it's part of the 'if' block?

One answer is that you absolutely should not. The current code is valid Python code, and you are a reformatter for style, not to correct (presumed) errors. Since this is valid code, you should re-flow line wrapping and so on within blocks, but not change what block valid code is part of.

Another answer is that maybe the person writing this code made a mistake. Style wise, it's common to add a blank line between the end of an indented block and following code; the lack of a blank line suggests that a mistake was made. So maybe you should reindent 'some-statement' to where it properly should be, especially if you have a style rule that says that there should be blank lines in this sort of situation.

(Of course, you could also opt to add the blank line that your style guide says should be there and not change what block a statement goes in. But we're in heuristics territory here.)

If you're a heuristic reformatter, your opinion may change depending on what the 'final-statement' is. For instance, if the final statement in the if block is 'return', it is pretty obvious that there's not supposed to be anything after it. Anything after it is dead code, which would be a different and less likely error. So you should leave 'some-statement' alone and it's valid style to not have a blank line between the last statement in the 'if' block and 'some-statement'.

Python doesn't have all that many statements that definitively end blocks, but it does have some that are extremely suggestive. Consider this pattern of code:

try:
   something
except SomeError:
   pass
some-statement

The pass statement is a no-op, not something that affects control flow, so it's perfectly valid to have statements after a 'pass'; they will be executed normally. At the same time it's commonly used this way when there's not going to be anything after it, so a heuristic Python code formatter that moved 'some-statement' up into the 'except' would make lots of people unhappy.

One such heuristic Python code reformatter is the one used in GNU Emacs in both its conventional python-mode (which 'parses' Python code with regular expressions) and python-ts-mode (which fully parses Python code with a tree-sitter grammar). I'm not sure if these are the same reformatters, but they have the same effects. This particular reformatter heuristic turns out to be the root cause of my Python code reformatting glitches.

(In fact the GNU Emacs Python code reformatting appears to take a 'pass' as a hard end of block and will out-dent anything after it, regardless of which this does to control flow. If you add a 'pass' in the middle of a function and reflow with M-q, GNU Emacs will happily make all statements after it module level ones.)

I experimented with some stand-alone Python code formatters I had sitting around, and none of them behaved this way, which I guess isn't surprising (I tried black, ruff, and yapf). Since the normal pylsp Python LSP server relies on one of them for code reformatting (which one depends on your configuration), this also means LSP-driven code reformatting won't do this. It's possible that only GNU Emacs has this (arguably incorrect) heuristic reformatting.

(I was led to discover all of this by a comment ae left on my earlier entry about Python 2 LSP problems.)

PS: There are other heuristic decisions you can make depending on what 'some-statement' is and where it currently is in the overall block. For example, if 'some-statement' is the last statement in a function and in a 'return', then it's almost certainly correct in its current place. But these heuristics multiply endlessly.

(6 comments.)

Recently, I decided to take my long standing, perfectly good GNU Emacs lsp-mode setup and completely replace it with Eglot, the now built in GNU Emacs LSP solution. At one level I didn't have any particularly strong specific reason to switch; I started by trying out Eglot after switching entirely to Corfu then just kept going to see how far I could get towards a good Eglot environment. The result is perfectly good and some things work better (Eglot will do 'complete to common prefix' in Go and Python modes) but it took more than a little bit of yak shaving to get here.

At another level, lsp-mode with lsp-ui is what I'd call a busy interface, with all sorts of things going on, and these days I've decided that I want a quieter LSP experience. Eglot is famously more minimal and quiet than lsp-mode, although you can and should augment Eglot's interface with additional packages. I could have tamed lsp-ui more with additional settings and fiddling, but switching to Eglot took care of all of that all at once, with other benefits. Overall I'm happy to have switched, although it was more work than I was entirely expecting.

(Should you switch? I don't know, but if you stick with GNU Emacs and use it in the modern way, I think you will sooner or later.)

As I've described in an earlier entry, Eglot's minimalism is because it's a modern GNU Emacs package that expects you to fill in features with other packages that interact with it through standard Emacs Lisp APIs. This means that for a good (but non-busy) LSP experience in Eglot, I needed to hook up a variety of additional things.

• Corfu just worked for completion; my general Corfu settings were fine.
• To get a good cross reference setup where I could get lsp-ui like previews of references to something, I needed to connect consult to the general Emacs xref system by setting 'xref-show-xrefs-function' to 'consult-xref'.

• I went back and forth between Flycheck with flycheck-eglot and Flymake before eventually settling on Flycheck. Flymake is better integrated with Eglot (in a way that I notice a bit) but I can make Flycheck work well enough and I prefer it in general. Eglot normally automatically puts buffers into flymake-mode, so to shut that off I do (in my use-package declaration for Eglot):

  :config
  (add-to-list 'eglot-stay-out-of 'flymake)
  

  And then to automatically activate flycheck-eglot:

  :hook
  (eglot-managed-mode . (lambda () (if (eglot-managed-p) (flycheck-eglot-mode 1))))
  

  (In theory flycheck-eglot has a global mode, in practice it didn't work out reliably for me and the brute force of a hook was the easiest approach.)

Eglot has some configuration settings that you'll want to experiment with. I found that I wanted 'eglot-extend-to-xref' to be 't', partly because that makes M-? find other uses in my own project of whatever external thing I've jumped to.

Eglot doesn't ship with any key bindings and I definitely needed some, partly to make LSP code actions more accessible. Since it's early in my Eglot usage, my key bindings are probably going to change, but my current set are:

("C-c r" . eglot-rename)
("C-c o" . eglot-code-action-organize-imports)
("C-c h" . eldoc)
("C-c a" . eglot-code-actions)
("C-c q" . eglot-code-action-quickfix)
("C-M-<mouse-2>" . eglot-code-actions-at-mouse)

The mouse binding exists because of one way flycheck-eglot isn't as fully hooked into Eglot as I'd wish, but it turns out to be generally convenient for access to LSP 'code actions'.

(I have deliberately not bound eglot-format to anything. In Go, the one language where I would trust LSP-driven code formatting, I already go-mode's gofmt command that I'm accustomed to using. I also don't expect to use the LSP 'organize imports' often, but maybe in Python.)

This is in addition to key bindings for other packages, such as Flymake, where in order to get nice navigation of Flymake reports, I needed to set up a key binding for consult-flymake along with a few others for Flymake functions. This became a somewhat unnecessary side trip when I went back to Flycheck, but since I built a working Flymake setup, I'm keeping it for any time when I want to use Flymake instead.

Looking back, I'd estimate that most of my work in switching from lsp-mode to Eglot wasn't in configuring Eglot, it was in configuring other packages. But to say it that way makes it sound more straightforward than it was. The actual process involved a lot of looking around for additional packages, trying things out, discovering things that didn't work for me, and so on (and some amount of backtracking, like my adventures with Flymake). To be fair, this is more or less what I went through with lsp-mode when I first set it up.

Eglot officially recommends that you start it by hand (cf), but I'm too lazy for that. Instead, as I did with lsp-mode, I arranged to start it automatically for local files in the relevant modes.

(use-package eglot
  :defer t
  :init
  (defun eglot-ensure-local-only ()
    "Enable Eglot only on local buffers."
    (unless (file-remote-p default-directory) (eglot-ensure)))
  :hook
  (python-mode . eglot-ensure-local-only)
  (go-mode . eglot-ensure-local-only)
  [...]

One potential limitation of eglot-ensure as compared to eglot is that if you have multiple LSP servers for a particular language (such as 'pylsp' and 'ruff' for Python), eglot-ensure just picks the default one while eglot offers you a choice. To change afterward, you need to shut down the current LSP server and invoke 'eglot'.

(There's a program to multiplex LSP servers (discussion) if I ever want to run several at once.)

LSP servers can offer you a profusion of 'code actions'. Sadly Eglot doesn't make these particularly conveniently accessible (but then neither did my lsp-mode setup), although I hacked around that with a mouse binding (mentioned above). At one level this is technically fair and correct, because LSP servers only offer you code actions when you ask (and code actions are specific to a particular spot). Eglot also doesn't give you any way of filtering what specific code actions it will show you out of a potentially long server list that you find mostly irrelevant (and some, not working), which sadly makes them rather 'busy' for both Go and Python.

Once I had a basic Eglot setup working, I had a fun time learning how to disable some checkers in pylsp, the Python LSP server I use, because my tastes are strongly against style-based linters in 'present all the time' diagnostics. Lsp-mode provides convenient controls to turn off, for example, diagnostics from the 'mccabe' complexity linter. With Eglot, I got to learn all about user specified workspace configuration, which is definitely the morally correct approach to this but which is much more complex. Here, let me show you:

(setq-default eglot-workspace-configuration
   '(:pylsp (:plugins (:mccabe (:enabled :json-false)
                       :pylint (:enabled :json-false)
                       :pylsp_mypy (:enabled :json-false)
                       :mypy (:enabled :json-false)
                       :pycodestyle (:enabled :json-false))
                      )))

Yes, sometimes the mypy stuff is "pylsp_mypy" and sometimes it's just "mypy". This is an internal pylsp detail that Eglot makes you learn. Also, that 'setq-default' is load bearing; you can't use setq.

I find it unfortunate that Eglot doesn't have any convenient way to temporarily set LSP server parameters for a project. If you have specific settings, your life will be much easier if you put them in a correctly formatted .dir-locals.el file, which may look like this:

(( nil
   . ((eglot-workspace-configuration
       . ( :gopls (:analyses
             (:unusedresult :json-false
              :QF1012 :json-false
	      :fmtappendf :json-false)))))))

(As you can tell, what you need to set varies from LSP server to LSP server. Gopls for Go is completely different than pylsp. This is a directory local setting for me rather than a global one because they only mis-fire on some of my code.)

If you want to change these settings on the fly, Eglot has documentation on that but it's not fun to deal with. If you sometimes want to turn on mypy for your Python (LSP) code but not always, as I do, you'll get to use 'dir-locals-set-class-variables' to set up a new class, then use a function that looks like this:

 (defun cks/mypy-enable ()
   "Set Python eglot workspace configuration to enable mypy."
   (interactive)
   (let ((server (eglot--current-server-or-lose)))
     (dir-locals-set-directory-class
        (project-root (eglot--project server))
                      'cks-mypy-enabled)
     (eglot-signal-didChangeConfiguration server)))

That this elaborate process is required is an accurate reflection of reality. Eglot is running one LSP server (per language) across your entire 'project' (directory tree), and settings for that LSP apply to all files you're editing in the project, so it can't have any notion of file or buffer local LSP server settings; they have to be project wide. By extension, setting 'eglot-workspace-configuration' through conventional means is a bad idea; that makes it a buffer local variable, which does nothing useful and will only confuse you.

Sidebar: My journey with Flymake and Flycheck in Eglot

Eglot works better with Flymake than with Flycheck and flycheck-eglot, at least currently. Specifically, with Flymake, Emacs will put a button 2 popup menu on the note itself with any LSP server driven corrections (usually a 'quickfix' LSP code action), but with Flycheck, all you get is the error being marked and you have to look for and trigger LSP code actions in another way. I initially switched to Flymake because of this, but Flymake took me some effort to configure so that I liked it.

However, after switching from Flycheck to Flymake, I found that there were still some things that Flycheck did better and sometimes I wanted Flycheck instead. So I retained my Flycheck setup as well (with flycheck-eglot too), which was convenient when the flycheck-eglot author came up with a nice workaround for my issue.

There's stuff to use Flycheck checkers in Flymake but I haven't done much experimentation with it, although I installed the package and set up some support infrastructure. My impression is that Flycheck has a larger collection of checkers than Flymake does and it's easier to shuffle among them. In theory a LSP server should make all other checkers unimportant, but in practice not so, especially if you want to sometimes invoke 'linter' level checkers.

I do sort of miss Flymake's 'show diagnostics at end of line' option, because it was a good way to make LSP diagnostics glaringly obvious, for times when I want that. There's flycheck-inline, but that only displays the current warning when you're on it, not all of the warnings when you scroll through. Sideline with sideline-flycheck has the same limitation but in my view a better UI experience.

I still have a certain amount of Python 2 code, both for work and for personal projects (for example, DWiki, the wiki software behind this blog; it will be Python 3 someday, but not so far). For a long time, I've preferred to do any significant editing of Python code in GNU Emacs, my normal choice for a superintelligent editor, and for a while, I've used LSP based Python editing. There's a very old LSP server for Python 2, but all of the Python LSP servers you actually want to use are specifically for Python 3, and recently I hit a problem that made me turn off the Python 2 LSP server. Since then I've been editing my Python 2 code (cautiously) with pylsp (my normal Python 3 LSP server) and recently, a little bit with 'ruff'. Somewhat to my surprise, this has more or less worked.

My minimum standard for more or less working is that the LSP doesn't malfunction obviously or deluge me with errors and other diagnostics that aren't applicable because it's applying Python 3 rules to Python 2 code. It's even better if the LSP can actually identify real problems, such as misspelled variable names or function names, and recently I've had pylsp do that for some of my code (code that was never tested or used, or I'd have found the problems much earlier; possibly this is a sign that I should have deleted the code instead of fixing it).

(The LSP server does obviously complain about Python 2 code that's using 'print' as a statement, since it's invalid Python 3 syntax, but this is easily fixed even in Python 2 code, and I want to fix it in anything I intend to maintain.)

Much of my Python 2 code mixes spaces and tabs for indentation, and I expected this to upset the Python 3 LSP servers. To my surprise, it hasn't for either pylsp or ruff. Although I can't tell for sure, I think that they're even still correctly interpreting the result (in terms of indentation levels and so on), or at least they're not complaining about syntax errors or other things I'd expect them to if they had the wrong idea of the code's structure.

(Parts of GNU Emacs' python-mode do seem to get confused and (re)indent stuff incorrectly in my old school Python 2 code with 8 space indents and real tabs, which is somewhat surprising. But I guess very few people are editing Python 2 code with tabs in GNU Emacs these days.)

I've done some testing, and as far as I can tell LSP features like 'go to definition' and 'find references' more or less work as I'd expect them to in pylsp. In my (GNU Emacs) environment I think pylsp is limited to cross references within the set of Python files that the editor has loaded and told it about, but within that it's handy.

All of this makes it clearly worthwhile to me to keep LSP stuff enabled for my Python 2 code and to continue to use a superintelligent editor for editing it (although I still make quick changes to Python 2 code with vim). Which is good, because it's also easier and sometimes I'm lazy.

(Work still has Python 2 programs because those programs are load bearing and doesn't particularly need to change, at least most of the time. Could we port them to Python 3? Sure. Could we be sure they didn't have lurking Unicode issues or other problems? No, not necessarily. I did one Python 2 to Python 3 conversion for a load bearing set of programs, our suite of ZFS management tools (including our spares management system), and it was somewhat nerve wracking.)

PS: In my current GNU Emacs environment using Eglot, I don't think the LSP server is called when I hit TAB or M-q (based on the server events reported by eglot-events-buffer), so it's not going to be involved in any rerun of my problem with lsp-mode and the Python 2 LSP server. The LSP server will reindent and reflow the entire file (Emacs buffer), but I have to very specifically ask it to do that. If I have Eglot ask pylsp to reformat a function (selected as a region), pylsp ends back a null result, which I believe means 'no changes', so perhaps pylsp is throwing up its hands at my mixed tabs and spaces indentation.

(2 comments.)

Tramp is a famous and often praised GNU Emacs system for editing remote files; lots of people will call it one of Emacs' compelling features. I've always had a decidedly different view of Tramp because Tramp has mostly not worked for me in opaque ways. I recently took another run at getting Tramp working (so I could have an informed opinion on why I'm not a fan), and in the process I've learned a bunch of things that I don't want to forget.

Although Tramp has a bunch of ways to get access to files remotely ('methods' in Tramp jargon), the dominant way is for Tramp to SSH in to the remote system and do stuff. In order to work with your remote shell, Tramp really wants your login on the remote system to have a conventional shell environment, ideally one that uses the Bourne shell (especially Bash).

(But see Remote shell setup hints and the Tramp FAQ.)

In specific, Tramp has requirements for its ssh method in a stock setup:

• Your shell must have a relatively conventional shell prompt. Defining this is beyond the scope of this entry; see the definition of tramp-shell-prompt-pattern in tramp.el.
• Your shell must accept and use backslash quoting of more or less arbitrary characters in command lines.
• Your shell login can't pause to ask questions; it can produce some additional output but it needs to drop you to a shell prompt (that Tramp can recognize).

All of these are required because with the 'ssh' method, Tramp ssh's in and starts a full login session, then switches to /bin/sh (or the Tramp remote shell you've set) with some special things that will let it reliably recognize its own Tramp (shell) prompts. Using the 'sshx' method can bypass a lot of this because with it, Tramp directly runs /bin/sh without going through your remote login session. I believe sshx is also often going to be faster, at the cost of not establishing all of the environment variables and so on that your login session would (including your remote shell's normal $PATH).

If your login shell environment doesn't match all of these you're going to have a varying amount of problems, especially with the 'ssh' method. If you have an unconventional prompt, you can sort of fix it, but a shell with different quoting rules will be painful. Tramp has some mechanisms to deal with additional questions but my impression is that they're at least a slog (see parts of Remote shell setup).

(Since I went through this, to deal with quoting issues you need to redefine tramp-end-of-output to something that doesn't require quoting that your shell doesn't support, and then make sure that your tramp-shell-prompt-pattern matches it in addition to everything else. The only characters that won't be quoted with backslashes by GNU Emacs are -, ., /, 0-9. and a-zA-Z (this is deep in shell-quote-argument). There are some things that may break inside GNU Emacs and Tramp if you do this but I haven't had any problems yet.)

If you ask Tramp to use the (remote) $PATH your remote environment sets up, it must be able to run '/bin/sh -l -c ...' in a way that successfully runs the command string without having your .profile blow things up, despite your .profile probably not being able to detect this. This is typically triggered by you putting 'tramp-own-remote-path' somewhere in tramp-remote-path (either the global version or a connection profile). Because Tramp is that way, the remote path is not part of the predefined connection information that you can set directly.

Despite Tramp carefully initializing your remote login session (if you use 'ssh'), Tramp then normally ignores your remote $PATH and instead generates its own, based on tramp-remote-path. Various bits of Tramp documentation will imply that you can use '~' in things you add to tramp-remote-path (cf some of the examples), but as far as I can tell this is what you would call inoperable. As part of connection setup, Tramp reduces tramp-remote-path down to the directories that exist on the remote machine, and the mechanism Tramp uses for this appears to be incompatible with the use of either '~' or environment variables like '$HOME'.

(Tramp does this path check using the tramp-bundle-read-file-names defconst and you can read what that expands to in order to see the details, along with the tramp-get-remote-path function and the stuff it calls. Since the shell snippet Tramp sends to the remote end quotes all of the directory names it checks, whether or not the remote shell supports '~' is irrelevant and it won't expand $HOME for you. It's possible that this is a bug and Tramp will get fixed some day, but don't hold your breath.)

There's no particularly good fix to this that I know of; instead, I think you have two options. The first is to make tramp-own-remote-path work (it probably will if you use a conventional shell and .profile), add it to tramp-remote-path, and set up and handle your $PATH properly in each machine's .profile. This is probably the better option if you can arrange it, in part because you probably want a correctly set remote $PATH for when you're logged in to the machine directly. The second option, suitable only if you have a common home directory name pattern or two across all your machines, is to add all likely directories to your tramp-remote-path in whatever variations of your home directory you might have:

(dolist (pe '("/home/cks/go/bin" "/u/cks/go/bin" ....))
  (add-to-list 'tramp-remote-path pe))

(Or you could write an ELisp function that generated the list from multiple sublists, one for things relative to your home directory and one a list of possible home directories.)

Many modern Unix systems in standard configurations will make your home directory be /home/<login>, so you can cover all of them by a few paths in tramp-remote-path. Well, assuming you have the same login on all of them. Otherwise, you'll probably have to venture into the world of connection local variables and profiles.

When changing tramp-remote-path there is something very important that can cause you (me) a great deal of frustration if you don't know the full story. At the very end of Tramp's documentation on remote programs, there is this critically important bit:

When remote search paths are changed, local Tramp caches must be recomputed. To force Tramp to recompute afresh, call M-x tramp-cleanup-this-connection RET or friends (see Cleanup remote connections).

If you're me, you might innocently think that it's safe to, for example, set or modify tramp-remote-path before you make any connections. This is false, and calling tramp-cleanup-this-connection is not sufficient to force 'local Tramp caches' to be recomputed. In fact, not even quitting and restarting Emacs will do so. Tramp maintains a persistent file based cache of information about each host you've ever connected to, including the remote $PATH it determined at the time of the first connection (with the first connection's tramp-remote-path), and it will use that cached remote $PATH value until and unless you clear the entire cache by, for example, deleting ~/.emacs.d/tramp (with Emacs not running), or you use tramp-cleanup-all-connections, which I think is probably sufficient.

Given its persistent and dangerous effects, you might want to disable this Tramp cache file. The fine documentation asserts that you can do this by setting tramp-persistency-file-name to nil. This appears to be technically correct but practically inoperative, because you cannot customize the variable to nil (only to a filename) or usefully setq it before Tramp is loaded. You can only setq it to nil (and have it stick) after Tramp is loaded (and you probably also want to invoke tramp-cleanup-all-connections to get rid of anything Tramp may have loaded).

Tramp isn't a mode and so doesn't have any hook that fires when it loads and starts to activate, which would be the right time to augment tramp-remote-path, clear any cached data Tramp loaded, and so on. This is unfortunate but use-package provides a way to work around it:

(use-package tramp
  :defer t
  ;; :config will be run right after Tramp loads.
  :config
  (cks/tramp-setup)
  )

This appears to reliably fire as I start to enter '/sshx:' or '/ssh:' or what have you.

(The manual version of this would be to directly use eval-after-load, but I might as well stick with use-package even if that's what use-package is using under the (macro) hood.)

When it works, Tramp can be pretty magical. However, my voyage of getting to this point was anything but smooth, and parts of it were extremely frustrating. That part was the part with the Tramp file cache, which made various changes to tramp-remote-path have no effect and then sometimes have effect and then go back to having no effect because I wasn't religiously clearing and removing the cache.

(Tramp badly needs a command that reports all of the relevant parameters for the current connection, such as the current remote path that Tramp is using. I could probably put my own version together with enough determination, but I shouldn't have to.)

PS: This entry was written in my working Tramp configuration from my home desktop, but I'm not sure I'm going to bother doing this again (I normally write entries in vim on the host that Wandering Thoughts is on). The red squiggles under (potentially) misspelled words are sort of nice, but on the other hand I turn out to have lots of vim reflexes for writing Wandering Thoughts entries.

(The reflexes aren't triggered by writing in general, because these days I write a lot of email in GNU Emacs and that goes fine.)

Many Bourne shells go slightly beyond the POSIX sh specification to also support a '-l' option that makes the shell act as a 'login shell'. POSIX's omission of -l isn't only because it doesn't really talk about login shells at all, it's also because Unix has a special way of marking login shells that goes back very far in its history. The -l option isn't necessarily what login and sshd and so on use, it's something that you can use if you specifically want to get a login shell in an unusual circumstance.

Bourne shells also have a '-c <command string>' option that causes the shell to execute the command string rather than be interactive (this is a long standing option that is in POSIX). It may surprise you to hear that most or all Bourne shells that support -l also allow you to use -l and -c together. Basically all Bourne shells interpret this as first executing your .profile and so on, then executing the command string instead of going interactive. One use for this is to non-interactively run a command line in the context of your fully set up shell, with $PATH and other environment variables ready for use.

Now, suppose (not hypothetically) that you have some things in your .profile that you would like to not run in this situation. Perhaps they're unnecessary and expensive, or perhaps they cause problems in the rest of your environment if they're run outside the context of a genuine login shell. It would be nice for your .profile to be able to detect this. Unfortunately, as far as I know at the moment this is impossible in general.

If you're using Bash specifically (and /bin/sh is Bash), Bash will set '$BASH_EXECUTION_STRING' in this case when invoked as either 'bash -l -c ...' or 'sh -l -c'. As far as I know, this is the only common Bourne compatible shell that provides any way to detect this. I've been unable to find any other shell that provides any indicator of this, neither as environment variables nor as, for example setting '$*' or '$0' to any special values.

(I checked Dash on Ubuntu LTS, which is their standard /bin/sh, the OpenBSD ksh (which is also /bin/sh), and FreeBSD /bin/sh (also a descendant of Kenneth Almquist's shell, like Dash. Fedora Linux uses Bash as /bin/sh.)

If you use Bash as your login shell but /bin/sh is something else (and things are specifically doing '/bin/sh -l -c ...'), you can rename your .profile to be .bash_profile, so it will only be read by Bash. If you have common setup stuff, you could put it in .profile and source that from your .bash_profile (although Bash has some complicated tangles around startup files, also).

If what you really care about is running your .profile a second time inside a first session, you can set a marker environment variable at the end of your .profile and then have your .profile look to see if the environment variable is set. If it is, you can skip re-doing things. This doesn't help if some clever program is doing 'ssh yourhost sh -l -c ....' in order to run some command line with your full environment set up (or if a graphical login environment does this, partly because of the Unix shell initialization problem). Under some circumstances you can detect this in your .profile because it won't be connected to a tty and you can check this with, for example, 'test -t 0'. Under other circumstances, you may have a tty despite being 'non interactive' in practice.

(It's possible that there is some generally available marker that I'm not aware of. If so, I'd be happy to hear about it.)

(3 comments.)

All Linux distributions (and all systems) face the need to generate SSH host keys when your system gets installed. One traditional way this was done was if the system started and discovered it had no SSH host keys, it would generate new ones. One way this was handy was that if you wanted to generate new SSH host keys for some reason, you could remove the existing ones and either reboot or restart the SSH daemon (which would usually trigger this).

As I found out the hard way the other day, some Linux distributions don't do this any more. In particular, Ubuntu doesn't. If you remove your SSH host keys, your SSH daemon will refuse to (re)start, and as far as I know there's no convenient, simple way to regenerate the necessary keys. If you make this mistake (as I did), you'll get to have fun looking up the ssh-keygen arguments you need (and then typing them in on the system console or a serial connection).

Before I started writing this entry, I would have guessed that this was common behavior across multiple distributions, because in this day and age it makes sense for your SSH keys to be set up in the installer rather than (possibly) on system boot, in a situation where the kernel's random number generation may not have accumulated much entropy. However, it turns out that Fedora doesn't behave like this.

Fedora's OpenSSH package has an entire set of systemd units and a script to generate SSH host keys if any of them are missing. Fedora has a templated sshd-keygen@service, which uses /usr/libexec/openssh/sshd-keygen to generate a host key of the appropriate type if it doesn't exist. Then Fedora's sshd.service unit 'wants' sshd-keygen.target, which in turn wants sshd-keygen@rsa.service, sshd-keygen@ecdsa.service, and sshd-keygen@ed25519.service, so before sshd starts, any missing host keys will be generated (whether or not your specific SSH server configuration uses them).

Since Ubuntu usually follows Debian, I assume Debian also doesn't automatically regenerate SSH host keys (and if it does, it doesn't seem to use the approach Fedora does). Fedora derived enterprise distributions probably follow Fedora, but I'm not even going to look. Other distributions may go either way, there probably isn't anything you could describe as a standard approach for this.

In the future, if I want to reset an Ubuntu machine's SSH host keys, the simplest thing for me will be to copy the Fedora sshd-keygen over to the system and run it (since my desktops are Fedora, I have convenient access to it). On a quick scan, the script itself is distribution-independent, so in theory you (I) could fish it out of Fedora in advance and stash a copy somewhere.

(Especially for servers, there's an argument that a missing SSH host key should be a fatal error for sshd, not something you should automatically fix up, since something is obviously badly wrong. If you generate new SSH host keys anyway so maybe people can SSH in to check the server, what you're effectively doing is training people to accept mismatched host keys in times of problems.)

Update: In a comment, Andreas pointed out 'ssh-keygen -A', which does exactly this system host key regeneration.

(6 comments.)

Over on the Fediverse, I shared a little story:

The current state of my GNU Emacs yak shaving:
; wc -l .emacs
1550 .emacs

Some of that is comments. Some of that is personal functions that I should move out to other files so I can have only use-package stuff in my .emacs. And some of it is large blocks for lsp-mode and company and some other stuff I'm probably never going to use again, which I should drop.

I got into my .emacs situation despite using use-package, which is the usual way people recommend to tame your Emacs configuration. Today I dealt with the whole thing by splitting my .emacs up into separate files, which is much better in general even if it's a bit more annoying in some ways.

My .emacs had accumulated a number of things over time, probably like many long term Emacs users. Besides infrastructure for use-package, it also had general Emacs settings I want, little personal commands and functions, simple use-package declarations, and a number of large, complex use-package declarations for packages that I need to significantly customize and tweak (and where I had lots of comments about the situation, written for my future self). Plus it also had commented out remains of experiments with things like origami-mode.

Some of these things I could remove to cut down my .emacs size, but a lot of them are intrinsically large, especially various use-package declarations. Things like Eglot, Corfu, and Vertico aren't really small packages with little to configure and adjust; they touch core areas of my Emacs experience where I have some strong opinions that don't match their default configurations. Making them work how I want them to is not necessarily a tiny process of one or two customizations. Using use-package basically encourages putting those customizations inline, as part of the overall use-package declaration, including little helper functions.

(Plus, modern modular things like Eglot require a bunch of additional packages for the full experience, and a bunch of small use-package declarations add up, especially when I add comments about why I have them.)

I took two approaches in my split. For personal functions and key bindings, I set up a number of new personal packages in ~/share/elisp and configured them in new use-package blocks (which also let me set hooks and establish key bindings). Then I took existing large use-package declarations (and small ones tied to them) and moved them all to a collection of separate files that I directly 'load-file' in my .emacs. The separate files are organized by general purpose; I have one for LSP stuff, one for all aspects of completion, one for flymake and flycheck, and one for MH-E. I left unrelated small use-package declarations (many for small packages) in my .emacs rather than try to push them to a 'miscellaneous' file where I'd probably forget about them.

(The resulting .emacs file has 18 use-package declarations left, five of which are for personal things and some of which are present purely so I can ':diminish' their modeline markers.)

What I take from this is that use-package is a perfectly good way to keep things organized but, somewhat obviously, it in no way guarantees that they will stay small or that I will refrain from adding packages.

Sidebar: use-package, load paths, load-file, and require

I don't have my ~/share/elisp directory tree on my Emacs load path, although maybe I should. For my collection of personal functions that I set up with use-package, I used ':load-path':

(use-package cks-misc
  :load-path "~/share/elisp"
  :commands (....)
  [...]
  )

(These things have no use-package usage inside themselves, they're just ELisp functions and so on.)

For the use-package declarations I moved to other files, I put them in ~/share/elisp/startup and did, eg:

(load-file "~/share/elisp/startup/lsp-startup.el")

It's probably more Emacs-proper to put things on my load path and then require the relevant name, but the load-file approach works, it directly expresses what I'm doing, and it hopefully makes it clear to future me that these aren't anything like normal packages.

Incidentally, as I discovered in the process of writing this entry but my readers may already know, when you use use-package's ':load-path', the directory is permanently added to 'load-path'; it's not just used once for this use-package (this is sort of spelled out in the documentation if you read carefully). I'm still going to use ':load-path' on everything that 'needs' it, although now I'm more tempted than before to extend 'load-path' to my ~/share/elisp in my .emacs setup code.

We had a weekend long power shutdown this past weekend in the building with our main machine room. As is our custom, we powered off the servers before hand (on Friday evening, with some surprises), and then turned them back on this morning. This isn't the first time we've gone through such a power shutdown (although usually they're shorter), and over time we've written checklists and lessons learned for these things. This time was no exception, so I wrote checklists for both powering down everything and powering it back up (well in advance for once). Then we collectively looked at my nice, detailed, step by step power on checklist and ripped it up.

The issue is that powering things up in our environment is not really an orderly, step by step process. A lot of our systems are both core things and relatively independent of each other, and while there are ordering dependencies (our fileservers have to be up before any NFS client, for example, and the DNS resolvers need to be up before the fileservers), they're small and at the start. Even in the ordering there's a lot that can be done at once, such as booting up all of the fileservers at once.

This structure, or lack of it, doesn't particularly fit in the traditional checklist format and process, which sort of assumes that you have a real order to things. Our power up process is more anarchic than that; at best it proceeds in stages, and even then there are multiple stages that can be done at once (such as turning on most of the firewalls and turning on the fileservers; neither depends on the other). Adding to the mix is the potential need to either troubleshoot things like failed PDUs or non-booted switches, or to decide to defer them to later.

This isn't the first time I've written up a power up list and had it more or less abandoned in practice (and our retrospective 'what actually happened' worklogs even talked about it). This is just the first time I've really admitted it up front.

I'm not sure what the best form of documentation is for our orderly cold start power up requirements, but it's certainly not a detailed checklist or anything that claims to be a linear narrative. Maybe what we want to do is simply list what everything requires, starting from the machines that don't require anything. Then everyone involved can look at what has all its requirements satisfied and go for it.

(A complication is that there are also some things that are ideally started early but if they're having problems it's not critical. For example, it's nice to have our central syslog server up early to collect everyone's logs right from the start, but it's not essential in the way that, say, our NFS fileservers or our local DNS resolvers are.)

(2 comments.)

Not content with blowing up my in-buffer LSP completion, I decided to follow it up by first trying out Eglot and then more or less switching from my relatively long standing use of lsp-mode. In the process I've wound up with some opinions on the contrast between lsp-mode and Eglot. I will give you the summary up front.

If you're just starting out with GNU Emacs and you want to have a functional, nice LSP based development environment without going through a long voyage of discovery, install and use lsp-mode, company-mode, and probably lsp-ui-mode (it comes with lsp-mode). If you stick with GNU Emacs you'll eventually want to move to Eglot, but that's for later.

To understand why I say this requires a voyage into the history of GNU Emacs, at least as I understand it.

GNU Emacs has always been in part a programming environment for creating UIs for editing text, especially code. However, for a long time many of the basic native ELisp pieces involved in doing this were relatively monolithic and weren't designed to be extensively modified and customized. If you wanted a modified version of something that GNU Emacs had a basic ELisp version of, you usually didn't hook into the native version; instead you had to replace it entirely with your own version (possibly copying and modifying the original Emacs ELisp code). One area where this was the case was in-buffer completion (especially autocompletion), which gave us third party monolithic packages like auto-complete and company(-mode) that a decade ago were your best or only choices. Lsp-mode dates from this era (its first commits were in 2016) and unsurprisingly, it's a monolith that implements many UI features itself (and it integrates with company-mode, also a monolith).

Somewhat recently (I'm not sure when it started), GNU Emacs has been modularizing many of these internal features, creating APIs that let people hook into aspects of (for one prominent example), completion (also, also). Modern GNU Emacs has adopted what you could call a "Unix tools" approach, where you have small, contained packages that handle one aspect of something and work by connecting themselves to these API points. Sometimes this results in very small, modest packages but even packages that take on bigger jobs are more smaller and more limited than past monoliths. Partly this is because they don't have to do everything themselves; they can leave various things as a problem for other people. Is Corfu giving you only limited completions in some programming language? That's not Corfu's problem, you need something else to create completion data.

(When back in the day it was Company's problem, more or less, and Company had to get a bunch of people to write a bunch of things to provide completion data.)

Eglot is a GNU Emacs package for this modern GNU Emacs world, which is why people say it's smaller than lsp-mode and also 'better' or 'more Emacsy'. It's smaller, more limited, and more Emacsy because it relies on standard GNU Emacs facilities that can now be customized and improved by other packages, rather than implementing its own nicer versions of those facilities the way lsp-mode does. Do you want nice autocompletion? That's not Eglot's problem, you can set up corfu yourself. Do you want nice 'go to definition' and 'see (other) references'? That's also not Eglot's problem, see consult-xref. Would you like to display code action possibilities on the right side? You probably want sideline. And so on.

Eglot's choice has a good side and a bad side. The good side is that it's part of this powerful, capable modern Emacs ecology of relatively narrow, focused packages. As you adopt the versions of these packages that you like, these packages improve things all across GNU Emacs, including in Eglot, because they're hooking into those general Emacs features and APIs. Corfu isn't just autocompletion for LSP buffers, it's potentially autocompletion for everything. And your Eglot environment inherits the other general improvements you make in your overall GNU Emacs environment. The whole thing gives you compounding effects from individual improvements.

This good side is why I think you'll wind up with Eglot if you stay with GNU Emacs. Over the long term you get a lot of power from moving into the modern Emacs ecology of narrowly focused but general purpose packages, and the more packages you adopt the more appealing Eglot is as part of that ecology (and the more foreign lsp-mode and company-mode become, and the more attractive it becomes to move to your standard packages). This is more or less my path to Eglot, and I wouldn't be here if I hadn't already adopted a whole collection of packages.

The bad side is that to get a decently nice Eglot experience, you also need a bunch of other packages. This means that you have to hear about those packages, experiment to decide which ones you like, learn how to set them up for your tastes, and so on. Until you do so, your LSP editing will be left with the relatively bare bones base GNU Emacs experience for completion, cross references, and other things. This is functional but by modern standards, not all that appealing. Even once you have all the packages you have to learn how to connect them all up to Eglot; lacking that knowledge at the time is why I bounced off Eglot in an earlier experiment with it.

(You could adopt someone else's modern GNU Emacs configuration, but your tastes may not be their tastes and anyway, that way you're effectively adopting a black box that you don't (yet) understand. I'm not sure this is meaningfully better than using lsp-mode, and lsp-mode will probably be better documented than the combination you've been given.)

Another issue is that integrated packages like lsp-mode and company tend to give you a better, more pleasant experience for some things. For one painful example, Eglot's approach to configuring what LSP servers support is general and clearly the proper way to do it, but lsp-mode's approach is much easier to use. Turning off pylsp's 'mccabe' code complexity metrics is simple in lsp-mode and an extended voyage of discover in Eglot (at least for me). You may discover that it's beyond your (current) GNU Emacs capabilities to do some things in Eglot that are relatively straightforward in lsp-mode.

(This is kind of the extended version of something I said on the Fediverse.)

One of the recent commotions in my corner of the tech sphere was over an incident where a piece of software deleted a company's production database and all of its backups. The software got all of the backups too because, I'll quote:

[Their SaaS provider] stores volume-level backups in the same volume — a fact buried in their own documentation that says "wiping a volume deletes all backups" — [...]

A lot of people were horrified, but I had some sympathies with the SaaS provider. An important thing about backups is how backups work depends on what you're trying to recover from, and for certain sorts of disasters and recoveries, this decision is perfectly sensible. For a SaaS company, they also depend on customer support needs and what customers are going to want, and the decision can also make sense from that perspective.

In this case, the obvious question is whether the SaaS provider is trying to protect customers from loss of data in the volume or from deliberate deletion of the volume. If what you're protecting people from is an accidental 'DROP TABLE' or an accidental 'rm' (or an accidental overwrite of something important), then in volume backups such as ZFS snapshots make perfect sense. We use ZFS snapshots ourselves for this purpose on some filesystems (although they're not our only form of backups). As a bonus, restores are much faster than external backups. However, backups tied directly to the volume aren't a good ideal if what you're protecting people against is deletion of the volume itself.

(The SaaS provider itself might be concerned about loss of the volume from things other than deliberate deletion, but this isn't a concern customers want to have; they want to pretend that the SaaS provider has 100% reliable handling of volumes until they delete them. Of course, this can lead to unpleasant customer surprises if something goes wrong, which is why wise customers have completely external backups so they don't have to trust the SaaS provider and the SaaS provider's cloud vendor. The people this happened to were not wise customers, but if you've heard of this incident, you already knew that.)

If a SaaS provider wants to potentially protect people from deliberate deletion of a volume, there are a bunch of tradeoffs. For example, you're probably charging people for out of volume backups in some way, which means that if people really want to delete an unused volume, they also want to delete its backups so they're not being charged for those either. If you surface an option for 'also delete backups of this volume' so that people deleting volumes can handle the situation right away and aren't surprised by charges later, what you're surfacing is an easy total data loss option; people will reflexively say "yes" and wipe out their backups too.

(After all, typically people who delete volumes think they're doing the right thing at the time. Software agents don't think but they're generally going to behave in the same way.)

The harder you make it to delete volume backups, the more you're going to annoy some of your customers who really do want to delete their volume backups (or perhaps many of your customers, since you'd hope that almost all volume deletions are customers making the right choice and they probably don't want the backups either). At a certain point, a SaaS provider might take a rational look at their data on what people are deleting and what they're recovering from (and customer support calls), and conclude that hard to delete volume backups aren't worth it because customers don't use the extra resilience and are annoyed by the side effects of it. Perhaps you can design both your systems and your charging to get around this, but it's more product development work and if you're a SaaS company, you have a lot of other product development work you could be doing and that other work may have much higher value to your company.

(Convenient, easily accessible in volume backups may also have side effects. The space consumption side effects of ZFS backups are why we don't use them pervasively for all of our fileserver ZFS filesystems.)

Locally we use external backups, but this is because we're operating physical storage and so we have to be concerned about all sorts of catastrophic things happening to it. Our external backups are slower to restore from for in-volume damage like deleted files, but we have to make that tradeoff because we absolutely have to be able to recover from a total loss of a ZFS filesystem, ZFS pool, or an entire fileserver (or our entire machine room).

We have a whole building, weekend long power shutdown in the building with our machine room that officially starts tomorrow (Saturday) morning at 5am, which is the motivation for our newly added temporary backup MX. Because we like to be in control of both the shutdown and the startup of our machines, we turn machines off in advance for scheduled outages (there's not much we can do about unscheduled ones). For various reasons we did the shutdown earlier this evening.

(One reason to start machines under controlled circumstances is that sometimes hardware fails, things go wrong, or you discover unfortunate aspects of your environment (also). At least these days we've mostly learned lessons from previous power shutdowns and startups, although there are aspects I hadn't fully absorbed and will write about later.)

During the shutdown, something surprising happened, which is that all of our ZFS fileservers came back to life. We definitely ran 'poweroff' on each of them and they were off the network for some amount of time, but then my co-workers doing work in the machine room noticed that they were all powered back on. We ran 'poweroff' on the rebooted servers and they shut down properly, rather than rebooting, so that part's not the problem. After some discussion we decided to deal with the immediate problem by pulling their power plugs, so they can't come back on even if something on board wants them to (all of these servers have BMCs).

One of the things we did between the fileservers shutting down and them coming back up is that I ran fping to scan the subnet they're on, to see if we'd missed shutting down any machines (and this fping run showed that none of them were on the network at the time). The host I ran fping from was on the same network and would have still had the MAC addresses of the fileservers in its ARP cache, so it could have directly unicast packets to the MAC.

One theory we have is that this triggered some sort of 'Wake on LAN' power up behavior. I wasn't pinging with a WoL 'Magic Packet', but as covered in sources like the Linux ethtool(8) manual pages, your hardware may potentially support a whole host of WoL mechanisms, including 'unicast messages'. This sounds like it might cause a server to wake up if its network interface receives a packet to its hardware MAC. Such as, for example, an ICMP ping packet that didn't need an ARP because the sending host already knew the target's MAC.

(I can't find much documentation on what these Wake on LAN options mean, but see eg here, this chipset documentation, or FreeBSD's ifconfig and its 'wol' options.)

When the power shutdown is over and we bring the fileservers back up on Monday, we'll be looking at what 'ethtool' reports as their Wake on LAN settings. Since they have fully capable BMCs, we may want to force all of them to have no Wake on LAN active at all. Certainly it seems undesired to have them potentially powering up based on just receiving packets, since there's a whole host of ways they could receive traffic.

PS: We haven't seen this in past power shutdowns, but our fileserver hardware was refreshed between the last one and now.

(2 comments.)

I've said before that Python virtual environments can be moved around. Well, technically that entry said 'usually', but in practice I don't remember the limitations I mentioned in that entry. And that is how a while back I renamed the top level directory of a Django virtual environment that I'd also installed the Python LSP server into, and then yesterday I was rather puzzled when I tried some Django development and GNU Emacs gave me a weird error and didn't start my LSP environment.

(Fortunately what I was really doing was seeing how my new Corfu based lsp-mode completion would behave with some Python code.)

The issue is simple: every (Python) program installed into your venv's bin/ directory starts with '#!/path/to/venv/bin/python3', including programs like pylsp, the Python LSP server. They have to do this because they need to run the venv's Python, but that means that they're locked to the original filesystem location of the venv. If you move the venv, either there will be no 'python3' at that path for them to run or worse, you'll be pointing into and using a different venv. Programs outside the venv aren't normally affected, because they're directly using the venv's bin/python3 and the Python interpreter makes that work.

(In my case in GNU Emacs, there was no python3 at the path that pylsp was pointing to, so it failed to start with a weird system message. With no LSP server, Emacs' lsp-mode threw up its hands and gave up.)

Incidentally, this includes the venv's 'pip'. If its '#!' line points to what is now another venv's Python, I believe 'pip install <whatever>' will wind up installing <whatever> into that other venv, not the one you think you're in. This could be anywhere from confusing to somewhat disastrous, depending on what the alternate venv is. Venv name reuse may seem unlikely, but it depends on what your venv naming is like; a worst case option would be something like 'dev-venv' and 'prod-venv', where you remove the old 'prod-venv' venv and rename the 'dev-venv' top level directory to 'prod-venv' (then create a new 'dev-venv' sooner or later).

So far I haven't stubbed my toe on this in anything critical, but it's definitely something I need to remember and it may change how I set up and (don't) move venvs. If I'm going to move venvs very much, it'd be tempting to write something that fixed up all of the '#!' lines in a venv's 'bin/' directory.

(There may already be tools out there that do this, but I'd have to find one of them and Internet search is increasingly bad.)

Somewhat recently I read this article on a modular completion framework for GNU Emacs (via) and expressed a thought on the Fediverse:

If lsp-mode in GNU Emacs supported corfu in addition to (or instead of) company-mode, I would probably switch from company to corfu just to have a unified completion environment. But I don't think as-you-type completion with LSP is supported in anything except company-mode, and I'm not moving to eglot (I looked once and rejected it).

Oh well, maybe someday I can unify things a bit more. (Or I will get annoyed with as-you-type completion.)

(I already use corfu for general completion, with company-mode only used in lsp-mode buffers.)

I was wrong; corfu does support as you type completion. Corfu calls this auto completion and doesn't enable it by default, but you can change that if you want, either locally to specific buffers or generally. Today I gave things a try and after the dust has settled, I've switched entirely to corfu, even in lsp-mode, with some additional changes.

As I discovered when I first explored as you type autocompletion in GNU Emacs, I like seeing the completion information but what I don't want is to have my keystrokes stolen just because some autocomplete information showed up. Corfu's default keybindings steal common keys that I might want to type while programming, such as RETURN, TAB, and cursor up and down; this is perfectly reasonable in corfu's normal environment where you have to manually trigger completion, but isn't what I want with autocomplete on. Corfu makes life slightly more difficult for me by using '<remap>' in its corfu-map local keybindings, so I have to unset them by hand:

  (keymap-unset corfu-map "RET" 'remove)
  (keymap-unset corfu-map "TAB" 'remove)
  (keymap-unset corfu-map "<up>" 'remove)
  (keymap-unset corfu-map "<down>" 'remove)
  (keymap-unset corfu-map "<remap> <next-line>" 'remove)
  (keymap-unset corfu-map "<remap> <previous-line>" 'remove)

Then, as with company-mode, I bind C-RET, C-TAB, C-<up>, and C-<down> to do these actions, which are corfu-insert, corfu-complete, corfu-previous, and corfu-next respectively. I also made my wheel mouse scroll up and down through the selections.

While I was fiddling around in corfu, I made the fortuitous discovery of completion-preview-mode. The visually obvious thing completion-preview mode does for me is that it shows the current completion prefix ahead of what I'm typing (if there is one). The non-obvious thing it does is that I can immediately hit TAB to complete to that prefix (the same way a single tab works in shell filename completion). This completion sort of works even even in lsp-mode, where corfu's normal completion expansion gives up entirely. Initially I thought that completion-preview could successfully complete prefixes in lsp-mode, but I was being fooled by how often the prefix was the first completion.

With completion-preview showing me basic information about what I can immediately complete, I decided to slow down how soon corfu's auto-completion popup appears. If I want to trigger it early, I can always hit M-TAB. My current value for 'corfu-auto-delay' is 0.5 (seconds).

The remaining fix needed is that lsp-mode is extremely attached to company-mode. If you have company-mode installed lsp-mode will activate it in buffers, and if you don't have it installed, lsp-mode will complain. This behavior can be turned off by setting the somewhat oddly named lsp-completion-provider variable to ':none' from its default value of ':capf'. Despite capf being a standard GNU Emacs jargon, lsp-mode really means 'company' here. No doubt there's some history involved.

(It's not clear to me if corfu makes some use of company-mode if it's available.)

Although I had to shave a certain amount of yaks to get here, I feel glad to have switched to only using Corfu. Company-mode is a perfectly fine autocompletion environment and I was happy with it for years, but I didn't use it everywhere and once I added corfu I was juggling two sets of reflexes, one for corfu M-TAB initiated completion in places like Emacs Lisp and the other for company autocompletion in LSP buffers. Every so often I'd hit M-TAB in an LSP buffer out of reflex, and sometimes that got confusing. Now I only have one set of reflexes.

One tricky bit of using autocompletion in corfu is that you can't change the value of 'corfu-auto' on the fly. What matters is its value when corfu-mode starts. Fortunately we can use brute force; if we assume that we're only going to change corfu-auto when corfu-mode is on, we can write functions like:

 (defun corfu-enable-auto ()
   "Enable corfu auto-completion in this buffer."
   (interactive)
   (setq-local corfu-auto t)
   (corfu-mode -1)
   (corfu-mode 1))
 (defun corfu-disable-auto ()
   "Disable corfu auto-completion in this buffer."
   (interactive)
   (setq-local corfu-auto nil)
   (corfu-mode -1)
   (corfu-mode 1))

There is probably a better way to do this, and possibly I should turn the mode off before changing the corfu-auto value. Also, don't forget to make these interactive functions, as I did in the first version I wrote.

(Well, it's GNU Emacs, we can always read the source, also, and then duplicate what the source is doing when it goes into or out of corfu-mode. But those are internal details that might change, while having corfu-mode redo its setup should always work.)

PS: Someday I would like to make corfu complete prefixes properly in lsp-mode (which would probably also fix completion-preview, and even company-mode, since they all have the same problem), but that's another and bigger problem. For today I'm happy to have switched.

Over on the Fediverse, I said something:

If your idea of a good configuration file format is JSON, you are not a daemon I will ever run voluntarily.

This is not very much of a subtoot of ISC Kea. If we ever have to replace the traditional ISC DHCP server with anything, it will not be with Kea.

If your program's configuration file format is JSON, you're openly advertising that you care far more about programming convenience in reading and loading your configuration file than you do about the people operating your software. "You can generate our JSON with software from something else", yeah, no. You've told me what your priorities are and I'm going to believe you. I would rather run software that actually cares about the people running it.

JSON is a perfectly good format for your internal configuration data store, what you transform a configuration file into and then save for your software's future convenience. It's not a configuration file format, and if you use it as such, you're basically forcing people to write your compiled configuration storage format themselves. The result is a configuration file only in a narrow technical sense that it is a file you force people to supply to configure your software. You could tell them to compile C or their language of choice into a shared .so file that you will load as a plugin to configure things, or to write a Python, Perl, Lua, or JavaScript file (depending on your implementation language) that you will load and execute to create the configuration, and call all of those 'configuration files', and it would not be too far off from the JSON case.

(One of my Python programs can get its configuration from a pickled configuration object loaded from a file. That is a file and it has the program's configuration in it, but I would never call it a configuration file.)

Why all of this matters is something I said on the Fediverse and have said before (more or less):

I should say this out loud: a program's configuration files and configuration file format is part of its user interface. Much like other user interfaces, you cannot necessarily use a generic 'UI' for your configuration files without inflicting pain on people operating your software.

Yes, this means that sometimes you have to design and build your own configuration file format, much like you may have to build other UIs for your program.

(See also.)

If your configuration user interface is JSON, you're making a statement about what and who you care about. You may also be making a statement about how you more or less require your software to be used, and how you expect people to deploy it. Certainly various people are going to read things into your choice, whether or not that's your genuine intentions, because people do that.

Pragmatically, I expect that almost no one is writing those JSON configurations and configuration files by hand. Instead they're probably generating them through a program or translating them from some (slightly) more approachable format, like YAML (which is only mildly better, but at least it has comments and an explicit multi-line structure). I'm sure there are multiple YAML to JSON translators, and some of them probably can take some sort of schema along with the input file, so you can get useful syntax errors when you make certain sorts of mistakes in your configuration.

(This is probably the route we would take if we absolutely had to run such a program.)

(8 comments.)

I mentioned before that I've been switching some libvirt-based virtual machines to UEFI. I've recently had to do some more things there, which has led me to discover what's important about the XML parts of your libvirt machine definitions for this. Or at least, what's important if you use virt-manager to change things.

(There's a long story that boils down to libvirt external snapshots not playing well with virtual CD-ROMs, BIOS PXE booting being annoying, and UEFI Secure Boot causing the Ubuntu 26.04 GRUB to refuse to touch the Ubuntu 22.04 installer kernel.)

As mentioned in the previous entry, what determines whether the machine boots into UEFI or BIOS is whether or not the <os> XML node has a "firmware='efi'" attribute set on it. Once you have UEFI firmware, the <os> XML node can have a '<firmware>' node with some '<feature>' nodes that tell it what to do about Secure Boot:

 <os firmware='efi'>
   <type arch='x86_64' machine='pc-q35-9.2'>hvm</type>
   <firmware>
     <feature enabled='yes' name='enrolled-keys'/>
     <feature enabled='yes' name='secure-boot'/>
   </firmware>
 </os>

By itself this isn't a fully specified UEFI set of attributes, because you need <loader> and <nvram> elements as well, and these vary based on your secure-boot and enrolled-keys settings.

Conveniently for me, if you edit your XML in virt-manager, don't have (or remove) the <loader> and <nvram> elements, and then pick the 'Apply' button, virt-manager will pick appropriate values for you based on your settings for Secure Boot (or the lack of it). This can be used when you're turning off Secure Boot (or turning it on), or when you're moving from BIOS to UEFI.

(This might also happen if you use 'virsh edit', but I haven't tested that. But I suspect it's virt-manager doing some convenient magic for you.)

So the easy way to convert a machine from BIOS booting to UEFI, with or without secure boot, is to add "firmware='efi'" to the <os> attribute and past in an appropriate <firmware> block. The block above is for full Secure Boot. For full lack of Secure Boot, I want:

   <firmware>
     <feature enabled='no' name='enrolled-keys'/>
     <feature enabled='no' name='secure-boot'/>
   </firmware>

Apparently if you flip around between Secure Boot and non-Secure Boot, you may want to reset your NVRAM file. One way to do this is to remove the relevant NVRAM file that I will find in /var/lib/libvirt/qemu/nvram/. Another way is to use --reset-nvram with 'virsh start', eg 'virsh start foo --reset-nvram'. You can also use --reset-nvram with 'virsh snapshot-revert', and I may be doing that someday.

(You don't need to reset the NVRAM file when going from BIOS to UEFI because BIOS doesn't have a NVRAM file. If you go from UEFI to BIOS and then back to UEFI, probably you want to reset your NVRAM, but also maybe you want two separate VMs instead of switching between BIOS and UEFI all the time.)

I recently read Geoff Huston's Revocation of X.509 certificates, which in part talks about OCSP's failure. One of the pragmatic reasons for OCSP being dead is that Chrome dropped support for it more than a decade ago. Specifically, Chrome's replacement for certificate revocation was for Chrome to have an internal set of revoked certificates. Recently, Firefox has adopted a similar approach (with a different technical implementation).

One of my views of this is that it shows browsers recognizing and accepting that if they want something, they have to do it themselves and they can't rely on the behavior of outside parties, especially the behavior of a lot of outside parties. Another way to put it is that browsers can change themselves to get something done but they often have a hard time getting other people to change.

OCSP had two groups of outside parties; Certificate Authorities for direct CA OCSP checks, and web servers for OCSP stapling, and in the end browsers clearly couldn't rely on either group. In my own experience, direct use of CA OCSP checks by Firefox failed so often because of problems with CA OCSP servers that turning it off was my first reaction any time I ran into a TLS problem (cf). When you think about it, browsers clearly couldn't count on other parties to run high volume, critical services with no economic model that were guaranteed to be both reliable and private.

(The kindest thing you can say about OCSP is that it was created in a long ago world where probably no one expected that HTTPS would become as prevalent and as critical as it has. In a world where HTTPS was only used when paying for your shopping cart and interacting with parts of your bank, both the volume and the privacy impacts of OCSP would be much, much lower.)

The answer to the problems with direct OCSP checks with Certificate Authorities was supposed to be OCSP Stapling. However, this had its own problem, which was that for it to really work, all (HTTPS) web servers had to upgrade. This was never really likely to happen, especially on a timely basis, and it probably became obvious fairly soon that it wasn't going to happen in practice (partly because it's hard, also).

So one way to view Chrome's decision to drop support for OCSP (and quite early) was a recognition that they couldn't count on any other party to handle certificate revocation for them. If Chrome wanted certificate revocation to work, they had to own their own mechanism for it (even if that mechanism was only used to a limited extent for high priority revocations). Browsers building their own mechanism also meant that browsers could handle the situation where a Certificate Authority was slow to handle a revocation for one reason or another, since the revocation data doesn't have to come only from CAs.

(The browsers require Certificate Authorities to promptly handle revocations, but if a CA doesn't do it in practice, resolving this is generally a long process involving people arguing over things, not an immediate thing where browsers remove the Certificate Authority. Immediate removal is reserved for a crisis, such as the Certificate Authority being compromised entirely.)

PS: For similar reasons I think that browsers relying on DNSSEC for TLS security properties in modern web PKI is a non-starter, even beyond all of the other DNSSEC problems in practice.

I recently wrote about all of the various steps of a UEFI network install, where you have a whole collection of DHCP, GRUB fetching things via TFTP and HTTP, and so on, all to boot into your Ubuntu server install ISO image. Specifically, all of the GRUB stuff and much of the complicated DHCP stuff is there because we have to load the installer's kernel and initial ramdisk. Our primary usage for UEFI network installs is to reinstall physical servers that are now in inconvenient locations, so eventually it occurred to me that if we already have running Linux systems, there are simpler ways to boot into a specific kernel and initramfs with specific command line arguments. One way is to add new GRUB boot entries, and another way is kexec.

If we're already using a local kernel and initramfs, it might be convenient to get rid of the need for a DHCP server too, by copying the network parameters from the currently running server and embedding them in both the kernel boot parameters and, more importantly, the cloud-init files that the installer will use. To do this, we need to embed the cloud-init files in the initramfs (and then point to them with 'ds=nocloud;s=/whatever' in the kernel command lines). Well, that's the theory, but it turns out that this is not quite the practice.

The problem is that contrary to what you (I) might think, the Ubuntu server installer is not running from the initramfs. Instead, the initramfs constructs an in-memory root filesystem from various squashfs filesystem images that it gets from /casper on the installer ISO. As part of the initramfs boot, Casper mounts the ISO image (either via NFS or via a HTTP copy), finds those files on it in /casper, and then uses these files to construct the root filesystem that will then have the ISO image (still) mounted in it when Casper pivots the system into running from it. This means that while it's readily possible to add files to the initramfs, your added files are immediately discarded when Casper pivots to its pre-built root filesystem. Since the squashfs filesystem images come from the ISO image, they're generic across your systems and you can't use them to embed per-system configurations.

(In the process of this pivot, Casper will do things like switch to a standard systemd init environment.)

To deal with Casper dropping the initramfs, we must arrange to copy our injected initramfs contents into the root filesystem that Casper builds before Casper pivots into it and discards the initramfs (as far as I know, there's no way to access the initramfs after this, especially with it pre-mounted so that your cloud-init file can be immediately read). Sadly Casper makes this complicated and potentially specific to the Ubuntu server installer you're using.

As part of the Casper initramfs process, Casper will run a collection of scripts from /scripts/casper-bottom, so ideally we can just add our own script to that and have it copy things from the initramfs to appropriate places in /root (the real root filesystem to be). Unfortunately, Casper doesn't scan this directory for scripts to run; instead what scripts to run (in what order) is handled by /scripts/casper-bottom/ORDER (this is the standard Casper way and is used for other Casper 'directories of scripts'). So we have to add our script and also replace the ORDER file from the ISO's initrd with one that includes our script.

A Linux kernel initramfs is a collection of cpio archives, with the last archive (usually) compressed. You can put your own uncompressed cpio archive on the front, or (usually) compress your own cpio archive with the same compression method as the compressed archive and stick it on at the end. Files in later cpio archives overwrite files from earlier cpio archives, and since we need to overwrite /scripts/casper-bottom/ORDER, we have to put our cpio archive at the end. Starting no later than Ubuntu 22.04 LTS, the standard installers all have the last cpio archive compressed with zstd, so that's also what we need to compress our own cpio archive.

(I believe there are potentially tricky issues with sticking compressed archives together this way, which I will leave to others to investigate. I made a 26.04 version work without problems but that could have been luck.)

To make this less annoying, we can use two local cpio archives. One archive contains only our additions and changes to /scripts/casper-bottom; it's zstd compressed and goes on the end of the initramfs, and we can even prepare generic, amended initramfs images with this already pre-built. Then the only per-machine addition we need to build is our cloud-init configuration files, which can go into an uncompressed cpio archive that we put on the front of our initramfs (perhaps the prepared, modified initramfs). This will give us a full initramfs that we can use as kexec's '--initrd' argument (or set up in a GRUB entry).

(This is not quite enough by itself to enable a DHCP-less network boot and install, because we also have to configure the system's IP address and other details in Casper itself via the 'ip=' command line argument; see casper(7) for the format of that. With a proper ip= setting, Casper can find the ISO image and mount it, and with a proper cloud-init injected into the initramfs and then the installer root filesystem, the server installer will properly set up networking and keep it up so that you can go through the normal over the network installer operation.)

PS: Apparently I will go through quite a lot to not have to maintain and update DHCP server entries, even through scripts that the future me might have fun writing.

I recently mentioned that we'd built a backup MX server due to concerns prompted by a scheduled power outage. In a comment on that entry, Greg A. Woods said something that I broadly agree with:

I think backup MX hosts are, generally speaking, a bad idea in modern times (even going back a couple of decades).

[...]

The added maintenance overhead and headache of keeping a full-time backup MX host running and reliably forwarding ALL email it collects, and reliably rejecting all email it should reject, isn't usually worth the bother.

One reason that we implemented a backup MX is that this isn't our experience. Our backup MX was easy to build and is essentially trivial to keep in reliable operation. However, this isn't because we have some special trick to running backup MXes; instead, it's because we have a general mail architecture that enables it.

Many, many years ago we moved from a mail architecture that was essentially monolithic to one that had an external MX gateway that was stuck in front of our central mail server. This transition involved creating what I call a 'white-box' mailer environment, where knowledge of things like valid local addresses and domains was materialized in text files and reusable in many contexts. Our spam and virus filtering is also done with FOSS components, which we can more or less run as many copies of as we like.

So our backup MX is essentially a clone of our regular external MX gateway machine, except that it has the MTA and the anti-spam stuff on the same machine (and we may do this for the next version of the external MX gateway, now we know more about how much load the anti-spam stuff creates). The backup MX server uses the same white-box mail information that our external MX gateway machine does, and we arranged for it to sit in a network environment where it could deliver accepted mail straight to our central mail server (instead of later delivering it to the normal external MX gateway, which would have added more hops and more redundant spam checking).

(All of the changes from the regular external MX gateway were things that we already had in operation on other machines and needed only modest tweaks to deal with the unique parts of this one.)

This is only possible because we already had all of the pieces. We have a general framework for installing and operating servers, we had an external MX gateway separate from the main mail system, that external MX gateway didn't rely on internal services to do things like validate addresses, and we didn't have commercial software involved that might have had license restrictions that prevented us from running an extra copy on our new backup MX.

We're also making life easier on ourselves by only running this backup MX temporarily, and with a configuration for valid email addresses, spam settings, and so on that is effectively frozen because all of the machines and services that could change any of that are powered off. That way we don't have to worry about what happens if the network connection between the backup MX and us gets blocked and the backup MX starts drifting out of sync on what email addresses are valid and so on.

If we hadn't already moved from a monolithic black-box mailer environment to a multi-machine white box one, building and running a backup MX host would have had all of the issues that Greg A. Woods identified. The existence of some of these issues is part of why spammers like to probe your backup MX. Also, in general I still agree with my old entry on the case against a full time backup MX, although modern email makes me nervous about the potential for aggressive mail delivery timeouts.

(In my old terminology, what we've built is technically a redundant MX. But that's a happy accident of the available network connectivity where this machine is going to be located for the power outage, and it could have had to deliver mail to our regular external MX gateway.)

There are at least three ways that x86 machines can try to boot from the network; BIOS PXE boot, UEFI PXE boot, and UEFI HTTP boot. All of them start by the machine asking a DHCP server for what it should boot, and all of them require different answers from the DHCP server. If you want to support more than one network booting option, your DHCP server needs to give each sort of client the right answer for it, which generally means you have to tell the DHCP server how to tell the types of clients apart.

(If you have all modern machines you can probably get away with only supporting UEFI PXE booting, which will simplify your life slightly.)

The DHCP server we use is the standard and now old-fashioned ISC DHCP server. There are a variety of guides for how to configure your ISC DHCP server for multiple types of network booting, but for various reasons I'm writing my own. This one is actually tested in real use (I've booted machines all three ways from this configuration).

When DHCP clients send out network booting requests, they include two important pieces of information, their "vendor class identifier" and their 'architecture'; these are DHCP option code 60 and DHCP option code 93 respectively. The vendor class identifier is a string and the architecture is a 16-bit integer. ISC DHCP has names for both options, vendor-class-identifier and pxe-system-type respectively (cf), although the latter appears to be recent enough that a lot of Internet writeups think you have to define it yourself in your dhcpd.conf, eg:

option pxe-arch code 93 = unsigned integer 16;

Since I didn't read up on all of this before this entry, my dhcpd.conf contains this superstition and I haven't (yet) tested a version without it.

If all you care about is UEFI x86 systems, you can use the vendor class identifier to tell apart UEFI PXE booting and UEFI HTTP booting. In PXE booting, it starts with 'PXEClient', and in HTTP booting, it starts with 'HTTPClient'. This results in a configuration snippet that looks like this:

class "pxeclients" {
  # TFTP
  match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
  next-server X.Y.Z.Q;
  filename "/grub/shimx64.efi";
}
class "httpclients" {
  match if substring (option vendor-class-identifier, 0, 10) = "HTTPClient";
  # the v-c-i in the reply is required
  option vendor-class-identifier "HTTPClient";
  filename "http://X.Y.Z.Q/grub/shimx64.efi";
}

If you also want to handle BIOS PXE systems, you need something more complicated, because both BIOS PXE and UEFI PXE have a vendor class identifier that starts with 'PXEClient'. You can be more precise by matching more of the vendor class identifier because it also includes an 'Arch:XXXXX' string (cf), but I think it's simpler to switch to using the 'architecture' number (which is what the 'Arch:' part is telling you anyway). The official list of architecture types is IANA's Processor Architecture Types, and one thing to know when reading it is that 'x64' is 64-bit x86, not Itanium. In practice with x86, what you'll see is 0x00 (BIOS PXE), 0x07 (UEFI PXE), and 0x10 (UEFI HTTP). In your dhcpd.conf, this looks like:

if (option pxe-arch = 00:10) {
  # The v-c-i is required
  option vendor-class-identifier "HTTPClient";
  filename "http://X.Y.Z.Q/grub/shimx64.efi";
} else if (option pxe-arch = 00:07) {
  next-server X.Y.Z.Q;
  filename "/grub/shimx64.efi";
} else {
  next-server X.Y.Z.Q;
  filename "/pxe/lpxelinux.0";
}

(Technically I should check pxe-arch for the last clause.)

I believe you can use the official 'pxe-system-type' here instead of my self-defined version, but I'm copying this example straight from my known-working dhcpd.conf. Also, as covered in dhcp-eval, possibly this would be more clearly written as a switch statement. I may experiment with both changes later, but this is what's working for me today.

(See also my entry on the various steps of a network install from an Ubuntu server ISO, which discusses the shimx64.efi and lpxelinux.0 bits a bit more.)

(2 comments.)

Suppose, not hypothetically, that you have a locally customized Ubuntu server install ISO image (and have for a while), and you also now have a number of UEFI based machines that it would be convenient to (re)install over the network without having to visit them in person (and they don't have IPMIs/BMCs that support virtual media). It turns out that you can take an Ubuntu ISO and install from it over the network, but how the various steps and stages connect together isn't obvious. Here are my notes on this, before I forget them all. I'll assume that you already have a modern Ubuntu server installer configuration setup, but you can also do this with a stock ISO image that will walk you through the full set of server installer questions.

The process of booting your ISO over the network goes like this (including recommended things):

1. Your UEFI based server sends out a DHCP request that includes, among other things, its request for one of the UEFI network booting options.
2. Your DHCP server answers with the server's IP and either a HTTP URL (for UEFI HTTP boot) to shimx64.efi, or a TFTP server and the (TFTP) path to shimx64.efi. Most of your machines will probably want the TFTP option. Provided that your DHCP server gave the server you're installing a usable gateway, this TFTP and HTTP server doesn't have to be on the same network as the server you're network installing.
3. Shimx64.efi will load grubx64.efi (which must be the signed grubnetx64.efi) from the same server and (relative) path as it was loaded, eg if shimx64.efi was loaded from '/inst/2604/grub/shimx64.efi', it will load '/inst/2604/grub/grubx64.efi'.

   The shimx64.efi and grub(net)x64.efi don't have to be from the Ubuntu version you're booting, but your grubx64.efi should match the GRUB modules you're going to use with it. You probably want to use the latest GRUB you can conveniently get your hands on.

4. GRUB will load '/grub/grub.cfg' and various other things in '/grub' from your TFTP or HTTP server. Unlike the shimx64 to grubx64 transition, GRUB (at least the Ubuntu version) insists on using an absolute path, not one relative to the directory it was loaded from. GRUB will expect to find various things in, for example, '/grub/x86_64-efi/'.

   In your /grub/grub.cfg, you can switch all future accesses to HTTP by using '(http)' in future references to things, perhaps with a prefix:

   set http=(http)/inst/
   

   Your grub.cfg can be universal for all of your machines, or you can go on to load a machine-specific one using some GRUB variables:

   source $http/grub/by-net/$net_default_ip
   

   (This trick comes from a co-worker, not me.)

   Some GRUB documentation will claim that GRUB will automatically search for a variety of grub.cfg names that are derived from the machine's IP address and other parameters. This is experimentally false for the Ubuntu 26.04 UEFI grubnetx64.efi; my server logs show no attempts for anything other than '/grub/grub.cfg'.

5. Whatever GRUB configuration file you use now loads the appropriate installer ISO's kernel and initrd, ideally over HTTP instead of TFTP because you switched above. You can get both of these from the /casper directory on the ISO (along with things you don't need). Once you've put these where you want them, you can specify them as, say:

   linux $http/casper/vmlinuz ip=dhcp [other options to come] ---
   initrd $http/casper/initrd
   

   Because the Ubuntu ISO's initrd contains kernel modules, it's specifically tied to the ISO's kernel; you have to use a matching pair and can't just swap in a more modern kernel with better hardware support for your hardware.

6. GRUB boots the installer kernel with the installer initrd, which makes its own DHCP request (and hopefully gets the same IP back), because once booted into Linux you no longer get to use UEFI services and the UEFI-obtained DHCP stuff. If you forgot to put 'ip=dhcp' into the kernel command line, the Ubuntu server installer initrd won't do DHCP, won't set up any networking, and everything else will fail.

   (It would be nice if the kernel automatically inherited all of the UEFI IP settings, including the TFTP or HTTP server information, but as far as I can tell it doesn't.)

7. The initrd 'mounts' the ISO. You have two options for how this is done, which are covered in the casper(7) manual page. Either the .iso image itself can be fetched over HTTP, stuffed into RAM, and mounted as a ramdisk image, or you can NFS mount an extracted directory tree version of it from a suitable server (perhaps the very install server that you've been TFTP'ing and HTTP'ing from so far; GRUB's $net_default_server variable may be convenient for this).

   The simpler option is configuring a NFS mount. This is done with the (kernel) command line options:

   netboot=nfs nfsroot=W.X.Y.Z:/some/path/
   

   To fetch the ISO from a URL, the kernel command line parameter is 'iso-url=http://...', but by itself this will probably fail because the default ramdisk is too small. So instead you need to also specify a bigger ramdisk (the size appears to be superstition, cf, but it works for Ubuntu 26.04 beta):

   root=/dev/ram0 ramdisk_size=1500000 iso-url=....
   

   A potential advantage of directly loading the ISO is that once it's loaded, you don't really have to care about the network connection to the install server. With a NFS mount, if something resets the networking you're really up the creek. On the other hand, the NFS mount starts quickly and means you don't have to care about things like ramdisk sizes and how much RAM your servers have.

8. Something fetches your your installer configuration quite early on (I think it may be the installer proper, not the initrd). If you don't provide a configuration, all you've done is network booted the stock install ISO and it's now going to sit there asking you to interact with the installer on the system console (which might be good enough if the server has a BMC with KVM over IP support). To either automatically install your system or to allow you remote SSH access to the installer, you need a cloud-init configuration. I believe that you can use the version you've embedded in your ISO image with your regular ds= parameter, but you may find it more convenient to fetch it via HTTP with more kernel command line parameters:

   "ds=nocloud-net;s=http://..."
   

   (You have to put this in quotes or GRUB will break it at the ';'.)

   If your install isn't fully automated and you want remote access to it to configure the interactive sections, your cloud-init user-data must include a chpasswd section for the user 'installer', or a ssh_authorized_keys with an appropriate key (which will again be used for 'installer').

   (I found this long ago from here.)

   (It's possible that you can configure a kernel serial console and then use IPMI Serial Over LAN to talk to the installer, if you have an IPMI with SoL support but no KVM over IP.)

9. The Ubuntu server installer will start up as normal, just as if it was booted from a real ISO, except that when the installer gets to configuring the network, it will reset networking and proceed according to your default configured networking, if any. This makes it critical to set 'dhcpv4: true' (or 'dhcpv6: true' if you're that sort of person) in your installer configuration, because otherwise your server will drop off the network, probably breaking its (network) install, especially if you opted to NFS-mount the ISO image's directory tree instead of fetching the ISO into RAM.

Provided that you've configured an appropriate cloud-init password or SSH key, you can SSH in to your network-booted server as 'installer' and be put in the regular server ISO installer environment, where you can go through whatever interactive steps you normally would with an in-person install. You'll want to use a big window and it needs to be a modern terminal program like gnome-terminal (don't try this with xterm). If you set 'network' as one of your interactive sections and you don't want to keep using DHCP in the installed system, you can switch from getting networking through DHCP to the same networking being set statically.

(You can also switch from DHCP to a static networking setup after the system has booted into its new local Ubuntu install; your install DHCP server is probably not going anywhere.)

Some of the kernel parameters here are confusing, because some of the time they can be interpreted by the kernel and some of the time they're ignored by the kernel and interpreted by things like casper(7). This is the case with the 'ip=' parameter, which in theory can be interpreted by the kernel but in practice is interpreted by Casper, with a different syntax. Since I just went on an extended digging session to find this out, I will tell you that the syntax Casper actually accepts for 'ip=' is the extended syntax used for klibc's ipconfig in its -d argument, because if your 'ip=' is something complicated, Casper winds up more or less passing it to ipconfig.

(This contradicts the Casper manual page but I extracted the 26.04 /casper/initrd to find this out. Not that it really matters, because in practice you mostly have to have DHCP working to get UEFI to network boot and then to keep your running install ISO on the network so you can talk to it.)

The minimal-changes version of going from an Ubuntu (server) ISO image to a booting it over the network is the iso-url option, although you will need to extract /casper/vmlinuz and /casper/initrd from the ISO. This avoids setting up NFS service on your install server, and also avoids having to unpack the ISO (which is easy enough with the right tools, but you have to know what the right tools are). My personal view is that I prefer the NFS option, and if you're the right kind of person you can use Apache Alias directives to serve /casper right out of the ISO's extracted directory tree rather than copy them into your web server area.

PS: It's possible to do much the same with a BIOS PXE booting server, but you have to use PXELINUX instead of GRUB (in practice you'll want to use the 'lpxelinux.0' variant that understands HTTP). Once you're at the stage of loading and booting the kernel, everything is the same; you need to boot the /casper vmlinuz and initrd, with the same kernel command line options as in the UEFI case. The one gotcha is that you can't use the syslinux INITRD directive because it messes up the kernel command line.

If you need to (re)install a large collection of servers or servers in inconvenient locations for physical access, booting them from the network in order to install them is something that you might be quite interested in. In the pre-UEFI PC 'BIOS' era of MBR booting, this was often called PXE booting, but UEFI changes things around.

UEFI firmware typically has built in support for networking, which is to say that there are UEFI protocols (function calls) for doing common things with the network (also, also). In practice this means that bootloaders and other things don't have to embed their own code to deal with the network (or their own network card drivers); provided that they don't exit from the UEFI preboot environment, they can just use UEFI services. In typical Linux environments, this will handle everything up until the kernel starts with its initial ramdisk (GRUB will load the kernel and initramfs over the network using UEFI services).

As covered in UEFI HTTP Boot, UEFI provides two ways to do network booting. Both ways start with the UEFI firmware doing DHCP to get an initial chunk of information, either by IPv4 or IPv6. In the standard and widely supported way, your DHCP server answers with (among other things) a next-server setting that points to a TFTP server and a 'filename' setting that is the initial EFI file to load and boot from that TFTP server. If you're using UEFI Secure Boot, this EFI file must be signed, so for x86 Linux with GRUB it's typically the (signed) shimx64.efi that you'd use locally (which will then boot 'grubx64.efi', which must really be the (signed) 'grubnetx64.efi'). My understanding is that this looks a lot like old fashioned PXE booting with minor differences in file names, configuration files, and so on.

The other, modern option is to skip using TFTP and load the EFI boot file over HTTP, hence UEFI HTTP Boot; this was apparently added in UEFI 2.5, from 2015. The UEFI firmware signals that it's doing a HTTP boot instead of a TFTP boot by setting special options in its DHCP request; it requests a special architecture and puts special things in its DHCP 'vendor class identifier'. If your DHCP server and your overall environment supports this boot option, you'll reply with a DHCP 'filename' option that is the URL of what to start booting from (often shimx64.efi again) and a special 'vendor class identifier' marker of your own to tell the UEFI firmware that this is a HTTP boot reply.

(See here, here, and the end of here for various DHCP server incantations using either the advertised client DHCP architecture or its vendor class identifier.)

Although the UEFI standard's description of UEFI HTTP Boot is somewhat unclear, it clearly envisions that HTTP boot can be used to 'boot' not just EFI programs but also disk images and even ISOs. These will be set up by UEFI firmware as a (UEFI) RAM disk. How your system installer accesses this ISO RAM image after the installer's kernel has started and UEFI firmware services aren't available any more is up to it.

UEFI HTTP booting has a variety of appealing features, like not using TFTP and supporting DNS (and everything that comes with that), and in modern UEFI firmware you apparently don't even need DHCP if you configure everything in the UEFI boot variables (cf, also). However, it has the potentially significant drawback of being modern, which means that older UEFI firmware (which you may have on systems you're now retaining) may either not support it at all or may have bugs and flaky behavior related to it. For that matter, even your modern UEFI firmware may not be entirely free of bugs, especially if you want to do more exotic things like directly boot an ISO image.

If you're already going to get as much as possible of the installer from your HTTP server, my view is that you might as well enable UEFI HTTP booting in your DHCP server. It probably won't hurt and it may enable somewhat better network booting, especially across subnet boundaries. Although ideally you won't be loading very much via TFTP anyway.

We have an extended power outage coming up, one that's long enough that I think we want a backup MX that can stay up during it. I've been building out a stand-alone duplicate of our current inbound mail gateway, and today I added a lower priority DNS MX record that points to it. What happened next is predictable:

This is my absolutely not surprised face that mere moments after I add a secondary MX to one of our zones, various IPs show up to poke its SMTP port, despite our primary MX being up (and the backup MX not actually running a SMTP server right now).

Admittedly, there's a reason for some use of our backup MX, which I discovered after I started the MTA on the backup MX machine:

Oops, I have to retract some of my 'the spammers are showing up on schedule' snark, because our primary MX greylists people sometime and if the primary MX is 4xx'ing things, trying the backup MX is reasonable.

(But surprise, the backup MX will greylist you too because our MXes are running the same configuration.)

Another case where things appear to have more or less legitimately shifted over to the backup MX was when one particular amazonses.com IP address opened up so many simultaneous connections to our primary MX that our primary MX started giving that IP temporary failures on connection. Trying the backup MX when the primary MX gives you an immediate 4xx is reasonable.

As far as I can tell, this isn't general SMTP probes against DNS names or IP addresses:

I did some more digging using our firewall PF logs and it appears pretty definite that some people showed up to do SMTP authentication probes only after this host appeared in DNS MX and got a TLS certificate. It's possible that the TLS certificate is the trigger for SMTP auth attempts, but they're very bad SMTP auth attempts (they aren't starting TLS, for a start, and this backup MX doesn't do SMTP auth).

Most of the sending machines that showed up were clearly bad, and many of them were rejected (or at the least sent things that got extremely high spam scores). Very few of them showed any signs of having tried to contact our primary MX. All of this matches what I think of as the expected behavior, where spammers and other bad actors hope that your backup MX is less well protected than your primary MX, so they prefer to talk to it if they can in the hopes that they'll get more bad stuff through.

(I've heard this story about backup MXes for a long time, but I never had a backup MX around to see this happen. It's nice, in a way, to have this story confirmed right in front of me.)

Some sending machines are more mysterious. For example, one outlook.com machine contacted the backup MX instead of the primary MX for no clear reason that I can see. These days, it's entirely possible that there was a transient network glitch on the path between that machine and us when it was trying to contact the primary MX, so it tried the secondary after its first connection glitched out.

Given all of this, if I was building a backup MX for full time use, it would be tempting to build a system where the MTA (mail server) was only enabled once the backup MX detected that the primary MX wasn't responding. Depending on taste, I could make the backup MX's MTA generate 4xx errors on connection or simply have it not be running at all so people got 'connection refused' if they tried. Checking once a minute or once every few minutes would be fine for our intended uses.

(In our planned one time use, we'll just enable and disable the backup MX's MTA by hand.)

(3 comments.)

One of the slow moving pieces of TLS news is that Google is effectively requiring everyone to stop issuing TLS certificates that can officially be used for "Client Authentication" (although the actual wording may have walked this back a bit). Certificate Authorities can create new roots that can be used to issue TLS certificates that are officially usable for client authentication, but Let's Encrypt isn't currently planning to do this. This was announced last year and then slowed down a bit this year, but it's still happening.

As part of the TLS handshake, a TLS client can optionally present a TLS certificate of its own to the TLS server. Officially, this TLS certificate and its entire certificate chain must be marked as being authorized for this purpose in an 'extended key usage (EKU)', just as TLS certificates used to identify servers must be marked as being authorized for this purpose. When people like Let's Encrypt talk about 'removing TLS Client Certificate support', what they're talking about is no longer issuing TLS certificates with this client authorization EKU.

However, TLS certificates are TLS certificates, regardless of what EKUs they are or aren't marked with. As a result there's nothing that stops servers from validating a TLS certificate presented to them by a TLS client whether or not it has a client EKU. In particular, you can almost certainly simply collect the TLS certificate without validating it, then turn around and ask your TLS library to validate it as if it was a TLS server certificate (or in general, accept either a TLS client certificate or a TLS server certificate). I expect that more or less any TLS library will let you do this; Go certainly will.

Various protocols and systems that are used by various people want and require TLS clients to present TLS certificates that will be used to validate the TLS client, with these TLS certificates being public ones obtained from trusted Certificate Authorities. In theory these TLS certificates should have the 'client authentication' EKU set. In practice, that relies on people being able to obtain such TLS certificates without difficulty. If it becomes difficult or impossible to obtain (public) TLS certificates with the client authentication EKU, the easiest thing for everyone involved to do is to change their server code so that it (also) accepts TLS certificates with the readily available 'server' EKU set, which Let's Encrypt and lots of other people will issue to people.

(This is especially likely in FOSS projects, where the people running clients and servers don't particularly have any budget to go out and find someone who will sell them TLS client certificates.)

I'm pretty certain that I've seen news flying by about at least one project that was starting to accept TLS server certificates from TLS clients, although I can't find it now in some Internet searches (and I foolishly didn't save a reference to it). I expect more projects and systems to do it in the future. It's really the inevitable result of no one blinking on this. We'll know that the cookie has really crumbled when commercial service providers start accepting TLS server certificates from TLS clients for purposes such as authenticating inbound SMTP mail (assuming this hasn't already happened).

All of this illustrates a fundamental issue with TLS security in practice, which I can summarize as the traffic is going to flow. No TLS security measure that prevents desired traffic from happening will survive in the real world. And generally the solution that will survive and thrive is whatever is easiest (here, using public TLS server certificates instead of trying to set up your own CA and certificate issuance infrastructure that will work across multiple organizations). Is this a good outcome for public TLS in general? Probably not. But it is what it is.

(3 comments.)

A while back I said something on the Fediverse:

Do I care enough about a couple of milliseconds¹ to make a program I'm considering my first attempt at a Rust program, or do I do it in Go, where I'm confident I can write it without irritation?

¹ This program will be quite short running, so the big difference I expect is in startup times. Go's runtime is (much) more heavyweight (and makes more system calls) than a basic Rust program's 'runtime'.

This is an example of what you'd call a superstition. I assumed that Go had a detectable runtime startup overhead, since Go has to initialize a bunch of things, including a garbage collection and its concurrency system (which involves some background goroutines), and Rust didn't. Eventually I found hyperfine (most basic Unix timing tools can't measure things in the microsecond range) and got around to actually timing things on the machine that I care about.

I've already spoiled the answer, which is that on the machine I care about in this case, any difference in startup time between a 'hello world' program in Rust and Go is down in the noise. Perhaps there is a ten or twenty microsecond difference in timing, but perhaps not and it's an artifact of scheduling, CPU caches, physical memory layout, and other random variations you experience in anything on a normal Unix system. A 'hello world' program written in pure C is typically faster than both the Rust and the Go programs by a visible amount of microseconds, but hyperfine also says it has a higher variation in timing.

(I also compared things to a Python hello world program, which as expected takes many times longer to run than the C, Rust, or Go programs. On this machine, the Python program runs in 13 milliseconds or so as compared to less than a millisecond for all the others.)

The machine I care about here is a FreeBSD machine. But I also use Go on Linux machines, so I pulled all of my test programs over to a pretty capable Linux machine and ran them there, and the results surprised me again. On several Linux systems, the Go hello world runs appreciably slower than the Rust hello world program (and the C hello world program remains faster). Typical hyperfine results say the Go program takes roughly twice as long as the Rust program, and it is indeed in the range of a millisecond or more of difference.

This gives me more to think about (and wonder about). I'm probably still going to stick to Go, but at least now I know that as of now (with the current state of Go and Rust), Rust does indeed seem to have appreciably less runtime startup overhead on Linux, but not on FreeBSD. If I'm trying to shave even a single millisecond off the runtime of something, I probably want Rust instead of Go.

(This assumes that the rest of the code will be equally fast in Rust and Go, which may or may not be true in practice. Without writing the same program in both good Go and good Rust, a real comparison is pretty hard.)

Also, on both FreeBSD and Linux, a statically linked C executable runs appreciably faster than a dynamically linked one. How much faster depends on the OS. Unsurprisingly, a statically linked Rust executable runs appreciably faster than the dynamically linked one that is the default 'rustc' result and that I was using above; on both Linux and FreeBSD, a statically linked Rust 'hello world' is faster than the dynamically linked C one (but not as fast as the statically linked C one). I generated the statically linked executable with 'rustc -O -C target-feature=+crt-static'.

The Go executables were all statically linked, since this is the Go default on both OSes and a simple 'hello world' program doesn't do anything that would force Go to dynamically link things.

(See also my Fediverse thread.)

In a comment on a recent entry, Verisimilitude noticed a feature that I quietly added to here not too long ago:

I've noticed the Add Comment button is now conditionally excluded; that's a neat trick.

I've long had precautions against comment spam and they've mostly worked. But not entirely, and so there have always been some network areas that I disallowed comments from even if they didn't run into those precautions. And if a (bad) network area was a sufficiently high source of automatically blocked comment spam attempts, I would add it to the list of blocked areas in case the software doing the comment spam got smart enough to get past my other precautions.

For a long time the only thing this blocked was direct access to the specific URLs used to write comments here (where the 'add comment' links point to). Then, recently, I realized that it made very little sense to give people and their software the link then block them when they used the link, and it would be better not to give them the link in the first place (as well as still blocking direct access). Among other things, I can hope that this stops software from crawling Wandering Thoughts to collect all the 'add comment' links that it will hit later through, for example, a proxy network.

Adding this feature was made easier because DWiki, the wiki software behind Wandering Thoughts, already had a permissions system for whether or not people could leave comments (and who could). As part of that permission system, DWiki had always done the obvious thing of not generating an 'Add Comment' link unless you had commenting permission. So all I had to do was extend the permissions check a little bit.

(The actual implementation has a collection of markers that can be set during processing of the request to influence what additional links are provided and not provided. For example, if you're a known robot, you don't see links to my syndication feeds because I don't allow known robots to request those. So I have a whole set of what is effectively middleware that scrutinizes the request and decides what should be allowed and not allowed, and then the final, low level dynamic page rendering looks at the result and includes or doesn't include various things.)

So if you visit Wandering Thoughts entries and they don't include an 'add comment' link, that's a sign that something about your request is making my anti-various-things precautions block comments (it might be your IP address or it might be something else).

The general idea strikes me as obvious in retrospect. If you're going to block direct use of something for some request source, you almost certainly want to not serve links to it either. And it's probably a better and less frustrating for any innocent bystanders caught up in a 'you can't comment' area. Previously it would have looked like they can comment, but any attempt would fail; now, they don't see the link at all so they can't get mysterious failures.

(Fortunately, DWiki always blocked all access to the 'add comment' link, even the initial one, so no one ever faced the really frustrating experience of writing a comment only to have posting it fail mysteriously.)

(6 comments.)